  High-fidelity pedestrian tracking in real-life conditions has been an
important tool in fundamental crowd dynamics research allowing to quantify
statistics of relevant observables including walking velocities, mutual
distances and body orientations. As this technology advances, it is becoming
increasingly useful also in society. In fact, continued urbanization is
overwhelming existing pedestrian infrastructures such as transportation hubs
and stations, generating an urgent need for real-time highly-accurate usage
data, aiming both at flow monitoring and dynamics understanding. To
successfully employ pedestrian tracking techniques in research and technology,
it is crucial to validate and benchmark them for accuracy. This is not only
necessary to guarantee data quality, but also to identify systematic errors.
  In this contribution, we present and discuss a benchmark suite, towards an
open standard in the community, for privacy-respectful pedestrian tracking
techniques. The suite is technology-independent and is applicable to academic
and commercial pedestrian tracking systems, operating both in lab environments
and real-life conditions. The benchmark suite consists of 5 tests addressing
specific aspects of pedestrian tracking quality, including accurate crowd flux
estimation, density estimation, position detection and trajectory accuracy. The
output of the tests are quality factors expressed as single numbers. We provide
the benchmark results for two tracking systems, both operating in real-life,
one commercial, and the other based on overhead depth-maps developed at TU
Eindhoven. We discuss the results on the basis of the quality factors and
report on the typical sensor and algorithmic performance. This enables us to
highlight the current state-of-the-art, its limitations and provide
installation recommendations, with specific attention to multi-sensor setups
and data stitching.
