  For robots to operate in a three dimensional world and interact with humans,
learning spatial relationships among objects in the surrounding is necessary.
Reasoning about the state of the world requires inputs from many different
sensory modalities including vision ($V$) and haptics ($H$). We examine the
problem of desk organization: learning how humans spatially position different
objects on a planar surface according to organizational ''preference''. We
model this problem by examining how humans position objects given multiple
features received from vision and haptic modalities. However, organizational
habits vary greatly between people both in structure and adherence. To deal
with user organizational preferences, we add an additional modality,
''utility'' ($U$), which informs on a particular human's perceived usefulness
of a given object. Models were trained as generalized (over many different
people) or tailored (per person). We use two types of models: random forests,
which focus on precise multi-task classification, and Markov logic networks,
which provide an easily interpretable insight into organizational habits. The
models were applied to both synthetic data, which proved to be learnable when
using fixed organizational constraints, and human-study data, on which the
random forest achieved over 90% accuracy. Over all combinations of $\{H, U,
V\}$ modalities, $UV$ and $HUV$ were the most informative for organization. In
a follow-up study, we gauged participants preference of desk organizations by a
generalized random forest organization vs. by a random model. On average,
participants rated the random forest models as 4.15 on a 5-point Likert scale
compared to 1.84 for the random model
