  Chatbots built upon language models have exploded in popularity, but they
have largely been limited to synchronous, turn-by-turn dialogues. In this paper
we present a simple yet general method to simulate real-time interactive
conversations using pretrained text-only language models, by modeling timed
diarized transcripts and decoding them with causal rejection sampling. We
demonstrate the promise of this method with two case studies: instant messenger
dialogues and spoken conversations, which require generation at about 30 tok/s
and 20 tok/s respectively to maintain real-time interactivity. These
capabilities can be added into language models using relatively little data and
run on commodity hardware.
