  Optimizing robotic action parameters is a significant challenge for
manipulation tasks that demand high levels of precision and generalization.
Using a model-based approach, the robot must quickly reason about the outcomes
of different actions using a predictive model to find a set of parameters that
will have the desired effect. The model may need to capture the behaviors of
rigid and deformable objects, as well as objects of various shapes and sizes.
Predictive models often need to trade-off speed for prediction accuracy and
generalization. This paper proposes a framework that leverages the strengths of
multiple predictive models, including analytical, learned, and simulation-based
models, to enhance the efficiency and accuracy of action parameter
optimization. Our approach uses Model Deviation Estimators (MDEs) to determine
the most suitable predictive model for any given state-action parameters,
allowing the robot to select models to make fast and precise predictions. We
extend the MDE framework by not only learning sim-to-real MDEs, but also
sim-to-sim MDEs. Our experiments show that these sim-to-sim MDEs provide
significantly faster parameter optimization as well as a basis for efficiently
learning sim-to-real MDEs through finetuning. The ease of collecting sim-to-sim
training data also allows the robot to learn MDEs based directly on visual
inputs and local material properties.
