  Constructive feedback is an effective method for improving critical thinking
skills. Counter-arguments (CAs), one form of constructive feedback, have been
proven to be useful for critical thinking skills. However, little work has been
done for constructing a large-scale corpus of them which can drive research on
automatic generation of CAs for fallacious micro-level arguments (i.e. a single
claim and premise pair). In this work, we cast providing constructive feedback
as a natural language processing task and create Riposte!, a corpus of CAs,
towards this goal. Produced by crowdworkers, Riposte! contains over 18k CAs. We
instruct workers to first identify common fallacy types and produce a CA which
identifies the fallacy. We analyze how workers create CAs and construct a
baseline model based on our analysis.
