  Robots should understand both semantics and physics to be functional in the
real world. While robot platforms provide means for interacting with the
physical world they cannot autonomously acquire object-level semantics without
needing human. In this paper, we investigate how to minimize human effort and
intervention to teach robots perform real world tasks that incorporate
semantics. We study this question in the context of visual servoing of mobile
robots and propose DIViS, a Domain Invariant policy learning approach for
collision free Visual Servoing. DIViS incorporates high level semantics from
previously collected static human-labeled datasets and learns collision free
servoing entirely in simulation and without any real robot data. However, DIViS
can directly be deployed on a real robot and is capable of servoing to the
user-specified object categories while avoiding collisions in the real world.
DIViS is not constrained to be queried by the final view of goal but rather is
robust to servo to image goals taken from initial robot view with high
occlusions without this impairing its ability to maintain a collision free
path. We show the generalization capability of DIViS on real mobile robots in
more than 90 real world test scenarios with various unseen object goals in
unstructured environments. DIViS is compared to prior approaches via real world
experiments and rigorous tests in simulation. For supplementary videos, see:
\href{https://fsadeghi.github.io/DIViS}{https://fsadeghi.github.io/DIViS}
