  We need to look at our shoelaces as we first learn to tie them but having
mastered this skill, can do it from touch alone. We call this phenomenon
"sensory scaffolding": observation streams that are not needed by a master
might yet aid a novice learner. We consider such sensory scaffolding setups for
training artificial agents. For example, a robot arm may need to be deployed
with just a low-cost, robust, general-purpose camera; yet its performance may
improve by having privileged training-time-only access to informative albeit
expensive and unwieldy motion capture rigs or fragile tactile sensors. For
these settings, we propose "Scaffolder", a reinforcement learning approach
which effectively exploits privileged sensing in critics, world models, reward
estimators, and other such auxiliary components that are only used at training
time, to improve the target policy. For evaluating sensory scaffolding agents,
we design a new "S3" suite of ten diverse simulated robotic tasks that explore
a wide range of practical sensor setups. Agents must use privileged camera
sensing to train blind hurdlers, privileged active visual perception to help
robot arms overcome visual occlusions, privileged touch sensors to train robot
hands, and more. Scaffolder easily outperforms relevant prior baselines and
frequently performs comparably even to policies that have test-time access to
the privileged sensors. Website: https://penn-pal-lab.github.io/scaffolder/
