  Human-Computer Interaction has been shown to lead to improvements in machine
learning systems by boosting model performance, accelerating learning and
building user confidence. In this work, we aim to alleviate the expectation
that human annotators adapt to the constraints imposed by traditional labels by
allowing for extra flexibility in the form that supervision information is
collected. For this, we propose a human-machine learning interface for binary
classification tasks which enables human annotators to utilise counterfactual
examples to complement standard binary labels as annotations for a dataset.
Finally we discuss the challenges in future extensions of this work.
