  Automation of tasks can have critical consequences when humans lose agency
over decision processes. Deep learning models are particularly susceptible
since current black-box approaches lack explainable reasoning. We argue that
both the visual interface and model structure of deep learning systems need to
take into account interaction design. We propose a framework of collaborative
semantic inference (CSI) for the co-design of interactions and models to enable
visual collaboration between humans and algorithms. The approach exposes the
intermediate reasoning process of models which allows semantic interactions
with the visual metaphors of a problem, which means that a user can both
understand and control parts of the model reasoning process. We demonstrate the
feasibility of CSI with a co-designed case study of a document summarization
system.
