  Search-based dialog models typically re-encode the dialog history at every
turn, incurring high cost. Curved Contrastive Learning, a representation
learning method that encodes relative distances between utterances into the
embedding space via a bi-encoder, has recently shown promising results for
dialog modeling at far superior efficiency. While high efficiency is achieved
through independently encoding utterances, this ignores the importance of
contextualization. To overcome this issue, this study introduces
triple-encoders, which efficiently compute distributed utterance mixtures from
these independently encoded utterances through a novel hebbian inspired
co-occurrence learning objective in a self-organizing manner, without using any
weights, i.e., merely through local interactions. Empirically, we find that
triple-encoders lead to a substantial improvement over bi-encoders, and even to
better zero-shot generalization than single-vector representation models
without requiring re-encoding. Our code
(https://github.com/UKPLab/acl2024-triple-encoders) and model
(https://huggingface.co/UKPLab/triple-encoders-dailydialog) are publicly
available.
