  We aim to help users communicate their intent to machines using flexible,
adaptive interfaces that translate arbitrary user input into desired actions.
In this work, we focus on assistive typing applications in which a user cannot
operate a keyboard, but can instead supply other inputs, such as webcam images
that capture eye gaze or neural activity measured by a brain implant. Standard
methods train a model on a fixed dataset of user inputs, then deploy a static
interface that does not learn from its mistakes; in part, because extracting an
error signal from user behavior can be challenging. We investigate a simple
idea that would enable such interfaces to improve over time, with minimal
additional effort from the user: online learning from user feedback on the
accuracy of the interface's actions. In the typing domain, we leverage
backspaces as feedback that the interface did not perform the desired action.
We propose an algorithm called x-to-text (X2T) that trains a predictive model
of this feedback signal, and uses this model to fine-tune any existing, default
interface for translating user input into actions that select words or
characters. We evaluate X2T through a small-scale online user study with 12
participants who type sentences by gazing at their desired words, a large-scale
observational study on handwriting samples from 60 users, and a pilot study
with one participant using an electrocorticography-based brain-computer
interface. The results show that X2T learns to outperform a non-adaptive
default interface, stimulates user co-adaptation to the interface, personalizes
the interface to individual users, and can leverage offline data collected from
the default interface to improve its initial performance and accelerate online
learning.
