  Adaptive optics (AO) systems deliver high-resolution images that may be ideal
for precisely measuring positions of stars (i.e. astrometry) if the system has
stable and well-calibrated geometric optical distortions. A calibration unit,
equipped with back-illuminated pinhole mask, can be utilized to measure
instrumental optical distortions. AO systems on the largest ground-based
telescopes, such as the W. M. Keck Observatory and the Thirty Meter Telescope
require pinhole positions known to 20 nm to achieve an astrometric precision of
0.001 of a resolution element. We characterize a photo-lithographic pinhole
mask and explore the systematic errors that result from different experimental
setups. We characterized the nonlinear geometric distortion of a simple imaging
system using the mask; and we measured 857 nm RMS of optical distortion with a
final residual of 39 nm (equivalent to 20 {\mu}as for TMT). We use a sixth
order bivariate Legendre polynomial to model the optical distortion and allow
the reference positions of the individual pinholes to vary. The nonlinear
deviations in the pinhole pattern with respect to the manufacturing design of a
square pattern are 47.2 nm +/- 4.5 nm (random) +/- 10.8 nm (systematic) over an
area of 1788 mm$^2$. These deviations reflect the additional error induced when
assuming the pinhole mask is manufactured perfectly square. We also find that
ordered mask distortions are significantly more difficult to characterize than
random mask distortions as the ordered distortions can alias into optical
camera distortion. Future design simulations for astrometric calibration units
should include ordered mask distortions. We conclude that photo-lithographic
pinhole masks are >10 times better than the pinhole masks deployed in first
generation AO systems and are sufficient to meet the distortion calibration
requirements for the upcoming thirty meter class telescopes.
