  Estimation of a speaker's direction and head orientation with binaural
recordings can be a critical piece of information in many real-world
applications with emerging `earable' devices, including smart headphones and
AR/VR headsets. However, it requires predicting the mutual head orientations of
both the speaker and the listener, which is challenging in practice. This paper
presents a system for jointly predicting speaker-listener head orientations by
leveraging inherent human voice directivity and listener's head-related
transfer function (HRTF) as perceived by the ear-mounted microphones on the
listener. We propose a convolution neural network model that, given binaural
speech recording, can predict the orientation of both speaker and listener with
respect to the line joining the two. The system builds on the core observation
that the recordings from the left and right ears are differentially affected by
the voice directivity as well as the HRTF. We also incorporate the fact that
voice is more directional at higher frequencies compared to lower frequencies.
