  The power of word embeddings is attributed to the linguistic theory that
similar words will appear in similar contexts. This idea is specifically
invoked by noting that "you shall know a word by the company it keeps," a quote
from British linguist J.R. Firth who, along with his American colleague Zellig
Harris, is often credited with the invention of "distributional semantics."
While both Firth and Harris are cited in all major NLP textbooks and many
foundational papers, the content and differences between their theories is
seldom discussed. Engaging in a close reading of their work, we discover two
distinct and in many ways divergent theories of meaning. One focuses
exclusively on the internal workings of linguistic forms, while the other
invites us to consider words in new company - not just with other linguistic
elements, but also in a broader cultural and situational context. Contrasting
these theories from the perspective of current debates in NLP, we discover in
Firth a figure who could guide the field towards a more culturally grounded
notion of semantics. We consider how an expanded notion of "context" might be
modeled in practice through two different strategies: comparative
stratification and syntagmatic extension
