  The rise of generalist large-scale models in natural language and vision has
made us expect that a massive data-driven approach could achieve broader
generalization in other domains such as continuous control. In this work, we
explore a method for learning a single policy that manipulates various forms of
agents to solve various tasks by distilling a large amount of proficient
behavioral data. In order to align input-output (IO) interface among multiple
tasks and diverse agent morphologies while preserving essential 3D geometric
relations, we introduce morphology-task graph, which treats observations,
actions and goals/task in a unified graph representation. We also develop
MxT-Bench for fast large-scale behavior generation, which supports procedural
generation of diverse morphology-task combinations with a minimal blueprint and
hardware-accelerated simulator. Through efficient representation and
architecture selection on MxT-Bench, we find out that a morphology-task graph
representation coupled with Transformer architecture improves the multi-task
performances compared to other baselines including recent discrete
tokenization, and provides better prior knowledge for zero-shot transfer or
sample efficiency in downstream multi-task imitation learning. Our work
suggests large diverse offline datasets, unified IO representation, and policy
representation and architecture selection through supervised learning form a
promising approach for studying and advancing morphology-task generalization.
