  Tidying up a messy table may appear simple for humans, but articulating clear
criteria for tidiness is challenging due to the ambiguous nature of common
sense reasoning. Large Language Models (LLMs) have proven capable of capturing
common sense knowledge to reason over this vague concept of tidiness. However,
they alone may struggle with table tidying due to the limited grasp on the
spatio-visual aspects of tidiness. In this work, we aim to ground the
common-sense concept of tidiness within the context of object arrangement. Our
survey reveals that humans usually factorize tidiness into semantic and
visual-spatial tidiness; our grounding approach aligns with this decomposition.
We connect a language-based policy generator with an image-based tidiness score
function: the policy generator utilizes the LLM's commonsense knowledge to
cluster objects by their implicit types and functionalities for semantic
tidiness; meanwhile, the tidiness score function assesses the visual-spatial
relations of the object to achieve visual-spatial tidiness. Our tidiness score
is trained using synthetic data generated cheaply from customized random walks,
which inherently encode the order of tidiness, thereby bypassing the need for
labor-intensive human demonstrations. The simulated experiment shows that our
approach successfully generates tidy arrangements, predominately in 2D, with
potential for 3D stacking, for tables with various novel objects.
