  Vision-based teleoperation offers the possibility to endow robots with
human-level intelligence to physically interact with the environment, while
only requiring low-cost camera sensors. However, current vision-based
teleoperation systems are designed and engineered towards a particular robot
model and deploy environment, which scales poorly as the pool of the robot
models expands and the variety of the operating environment increases. In this
paper, we propose AnyTeleop, a unified and general teleoperation system to
support multiple different arms, hands, realities, and camera configurations
within a single system. Although being designed to provide great flexibility to
the choice of simulators and real hardware, our system can still achieve great
performance. For real-world experiments, AnyTeleop can outperform a previous
system that was designed for a specific robot hardware with a higher success
rate, using the same robot. For teleoperation in simulation, AnyTeleop leads to
better imitation learning performance, compared with a previous system that is
particularly designed for that simulator. Project page:
https://yzqin.github.io/anyteleop/.
