  Multiple-object tracking (MOT) is a challenging task that requires
simultaneous reasoning about location, appearance, and identity of the objects
in the scene over time. Our aim in this paper is to move beyond
tracking-by-detection approaches, that perform well on datasets where the
object classes are known, to class-agnostic tracking that performs well also
for unknown object classes.To this end, we make the following three
contributions: first, we introduce {\em semantic detector queries} that enable
an object to be localized by specifying its approximate position, or its
appearance, or both; second, we use these queries within an auto-regressive
framework for tracking, and propose a multi-query tracking transformer
(\textit{MQT}) model for simultaneous tracking and appearance-based
re-identification (reID) based on the transformer architecture with deformable
attention. This formulation allows the tracker to operate in a class-agnostic
manner, and the model can be trained end-to-end; finally, we demonstrate that
\textit{MQT} performs competitively on standard MOT benchmarks, outperforms all
baselines on generalised-MOT, and generalises well to a much harder tracking
problems such as tracking any object on the TAO dataset.
