  We explore the origin of the Galactic high latitude black hole X-ray binary
XTE J1118+480, and in particular its birth location and the magnitude of the
kick received by the black hole upon formation in the supernova explosion. We
constrain the age of the companion to the black hole using stellar evolution
calculations between 2 Gyr and 5 Gyr, making an origin in a globular cluster
unlikely. We therefore argue that the system was born in the Galactic disk and
the supernova propelled it in its current high latitude orbit. Given the
current estimates on its distance, proper motion and radial velocity, we
back-trace the orbit of XTE J1118+480 in the Galactic potential to infer the
peculiar velocity of the system at different disk crossings over the last 5
Gyr. Taking into account the uncertainties on the velocity components, we infer
an average peculiar velocity of 183 \pm 31 km/s. The maximum velocity which the
binary can acquire by symmetric supernova mass loss is about 100 km/s, which is
2.7 sigma away from the mean of the peculiar velocity distribution. We
therefore argue that an additional asymmetric kick velocity is required. By
considering the orientation of the system relative to the plane of the sky, we
derive a 95% probability for a non null component of the kick perpendicular to
the orbital plane of the binary. The distribution of perpendicular velocities
is skewed to lower velocities with an average of 93^{+55}_{-60} km/s.
