  The advent of tactile sensors in robotics has sparked many ideas on how
robots can leverage direct contact measurements of their environment
interactions to improve manipulation tasks. An important line of research in
this regard is that of grasp force control, which aims to manipulate objects
safely by limiting the amount of force exerted on the object. While prior works
have either hand-modeled their force controllers, employed model-based
approaches, or have not shown sim-to-real transfer, we propose a model-free
deep reinforcement learning approach trained in simulation and then transferred
to the robot without further fine-tuning. We therefore present a simulation
environment that produces realistic normal forces, which we use to train
continuous force control policies. An evaluation in which we compare against a
baseline and perform an ablation study shows that our approach outperforms the
hand-modeled baseline and that our proposed inductive bias and domain
randomization facilitate sim-to-real transfer. Code, models, and supplementary
videos are available on https://sites.google.com/view/rl-force-ctrl
