  Visual localization is the task of estimating the camera pose from which a
given image was taken and is central to several 3D computer vision
applications. With the rapid growth in the popularity of AR/VR/MR devices and
cloud-based applications, privacy issues are becoming a very important aspect
of the localization process. Existing work on privacy-preserving localization
aims to defend against an attacker who has access to a cloud-based service. In
this paper, we show that an attacker can learn about details of a scene without
any access by simply querying a localization service. The attack is based on
the observation that modern visual localization algorithms are robust to
variations in appearance and geometry. While this is in general a desired
property, it also leads to algorithms localizing objects that are similar
enough to those present in a scene. An attacker can thus query a server with a
large enough set of images of objects, \eg, obtained from the Internet, and
some of them will be localized. The attacker can thus learn about object
placements from the camera poses returned by the service (which is the minimal
information returned by such a service). In this paper, we develop a
proof-of-concept version of this attack and demonstrate its practical
feasibility. The attack does not place any requirements on the localization
algorithm used, and thus also applies to privacy-preserving representations.
Current work on privacy-preserving representations alone is thus insufficient.
