  Cross-view image geo-localization aims to determine the locations of
street-view query images by matching with GPS-tagged reference images from
aerial view. Recent works have achieved surprisingly high retrieval accuracy on
city-scale datasets. However, these results rely on the assumption that there
exists a reference image exactly centered at the location of any query image,
which is not applicable for practical scenarios. In this paper, we redefine
this problem with a more realistic assumption that the query image can be
arbitrary in the area of interest and the reference images are captured before
the queries emerge. This assumption breaks the one-to-one retrieval setting of
existing datasets as the queries and reference images are not perfectly aligned
pairs, and there may be multiple reference images covering one query location.
To bridge the gap between this realistic setting and existing datasets, we
propose a new large-scale benchmark -- VIGOR -- for cross-View Image
Geo-localization beyond One-to-one Retrieval. We benchmark existing
state-of-the-art methods and propose a novel end-to-end framework to localize
the query in a coarse-to-fine manner. Apart from the image-level retrieval
accuracy, we also evaluate the localization accuracy in terms of the actual
distance (meters) using the raw GPS data. Extensive experiments are conducted
under different application scenarios to validate the effectiveness of the
proposed method. The results indicate that cross-view geo-localization in this
realistic setting is still challenging, fostering new research in this
direction. Our dataset and code will be released at
\url{https://github.com/Jeff-Zilence/VIGOR}
