  In this paper, we describe an approach for modelling causal reasoning in
natural language by detecting counterfactuals in text using multi-head
self-attention weights. We use pre-trained transformer models to extract
contextual embeddings and self-attention weights from the text. We show the use
of convolutional layers to extract task-specific features from these
self-attention weights. Further, we describe a fine-tuning approach with a
common base model for knowledge sharing between the two closely related
sub-tasks for counterfactual detection. We analyze and compare the performance
of various transformer models in our experiments. Finally, we perform a
qualitative analysis with the multi-head self-attention weights to interpret
our models' dynamics.
