  The field of natural language processing has reached breakthroughs with the
advent of transformers. They have remained state-of-the-art since then, and
there also has been much research in analyzing, interpreting, and evaluating
the attention layers and the underlying embedding space. In addition to the
self-attention layers, the feed-forward layers in the transformer are a
prominent architectural component. From extensive research, we observe that its
role is under-explored. We focus on the latent space, known as the Activation
Space, that consists of the neuron activations from these feed-forward layers.
In this survey paper, we review interpretability methods that examine the
learnings that occurred in this activation space. Since there exists only
limited research in this direction, we conduct a detailed examination of each
work and point out potential future directions of research. We hope our work
provides a step towards strengthening activation space analysis.
