  Modern machine learning systems have demonstrated substantial abilities with
methods that either embrace or ignore human-provided knowledge, but combining
benefits of both styles remains a challenge. One particular challenge involves
designing learning systems that exhibit built-in responses to specific abstract
stimulus patterns, yet are still plastic enough to be agnostic about the
modality and exact form of their inputs. In this paper, we investigate what we
call The Ungrounded Alignment Problem, which asks How can we build in
predefined knowledge in a system where we don't know how a given stimulus will
be grounded? This paper examines a simplified version of the general problem,
where an unsupervised learner is presented with a sequence of images for the
characters in a text corpus, and this learner is later evaluated on its ability
to recognize specific (possibly rare) sequential patterns. Importantly, the
learner is given no labels during learning or evaluation, but must map images
from an unknown font or permutation to its correct class label. That is, at no
point is our learner given labeled images, where an image vector is explicitly
associated with a class label. Despite ample work in unsupervised and
self-supervised loss functions, all current methods require a labeled
fine-tuning phase to map the learned representations to correct classes.
Finding this mapping in the absence of labels may seem a fool's errand, but our
main result resolves this seeming paradox. We show that leveraging only letter
bigram frequencies is sufficient for an unsupervised learner both to reliably
associate images to class labels and to reliably identify trigger words in the
sequence of inputs. More generally, this method suggests an approach for
encoding specific desired innate behaviour in modality-agnostic models.
