  We present an interesting and challenging dataset that features a large
number of scenes with messy tables captured from multiple camera views. Each
scene in this dataset is highly complex, containing multiple object instances
that could be identical, stacked and occluded by other instances. The key
challenge is to associate all instances given the RGB image of all views. The
seemingly simple task surprisingly fails many popular methods or heuristics
that we assume good performance in object association. The dataset challenges
existing methods in mining subtle appearance differences, reasoning based on
contexts, and fusing appearance with geometric cues for establishing an
association. We report interesting findings with some popular baselines, and
discuss how this dataset could help inspire new problems and catalyse more
robust formulations to tackle real-world instance association problems. Project
page:
$\href{https://caizhongang.github.io/projects/MessyTable/}{\text{MessyTable}}$
