  The rapid growth of automated and autonomous instrumentations brings forth an
opportunity for the co-orchestration of multimodal tools, equipped with
multiple sequential detection methods, or several characterization tools to
explore identical samples. This can be exemplified by the combinatorial
libraries that can be explored in multiple locations by multiple tools
simultaneously, or downstream characterization in automated synthesis systems.
In the co-orchestration approaches, information gained in one modality should
accelerate the discovery of other modalities. Correspondingly, the
orchestrating agent should select the measurement modality based on the
anticipated knowledge gain and measurement cost. Here, we propose and implement
a co-orchestration approach for conducting measurements with complex
observables such as spectra or images. The method relies on combining
dimensionality reduction by variational autoencoders with representation
learning for control over the latent space structure, and integrated into
iterative workflow via multi-task Gaussian Processes (GP). This approach
further allows for the native incorporation of the system's physics via a
probabilistic model as a mean function of the GP. We illustrated this method
for different modalities of piezoresponse force microscopy and micro-Raman on
combinatorial $Sm-BiFeO_3$ library. However, the proposed framework is general
and can be extended to multiple measurement modalities and arbitrary
dimensionality of measured signals. The analysis code that supports the funding
is publicly available at https://github.com/Slautin/2024_Co-orchestration.
