  Visual planning methods are promising to handle complex settings where
extracting the system state is challenging. However, none of the existing works
tackles the case of multiple heterogeneous agents which are characterized by
different capabilities and/or embodiment. In this work, we propose a method to
realize visual action planning in multi-agent settings by exploiting a roadmap
built in a low-dimensional structured latent space and used for planning. To
enable multi-agent settings, we infer possible parallel actions from a dataset
composed of tuples associated with individual actions. Next, we evaluate
feasibility and cost of them based on the capabilities of the multi-agent
system and endow the roadmap with this information, building a capability
latent space roadmap (C-LSR). Additionally, a capability suggestion strategy is
designed to inform the human operator about possible missing capabilities when
no paths are found. The approach is validated in a simulated burger cooking
task and a real-world box packing task.
