  Recent technological developments have spurred great advances in the
computerized tracking of joints and other landmarks in moving animals,
including humans. Such tracking promises important advances in biology and
biomedicine. Modern tracking models depend critically on labor-intensive
annotated datasets of primary landmarks by non-expert humans. However, such
annotation approaches can be costly and impractical for secondary landmarks,
that is, ones that reflect fine-grained geometry of animals, and that are often
specific to customized behavioral tasks. Due to visual and geometric ambiguity,
nonexperts are often not qualified for secondary landmark annotation, which can
require anatomical and zoological knowledge. These barriers significantly
impede downstream behavioral studies because the learned tracking models
exhibit limited generalizability. We hypothesize that there exists a shared
representation between the primary and secondary landmarks because the range of
motion of the secondary landmarks can be approximately spanned by that of the
primary landmarks. We present a method to learn this spatial relationship of
the primary and secondary landmarks in three dimensional space, which can, in
turn, self-supervise the secondary landmark detector. This 3D representation
learning is generic, and can therefore be applied to various multiview settings
across diverse organisms, including macaques, flies, and humans.
