  Implicit Personalization (IP) is a phenomenon of language models inferring a
user's background from the implicit cues in the input prompts and tailoring the
response based on this inference. While previous work has touched upon various
instances of this problem, there lacks a unified framework to study this
behavior. This work systematically studies IP through a rigorous mathematical
formulation, a multi-perspective moral reasoning framework, and a set of case
studies. Our theoretical foundation for IP relies on a structural causal model
and introduces a novel method, indirect intervention, to estimate the causal
effect of a mediator variable that cannot be directly intervened upon. Beyond
the technical approach, we also introduce a set of moral reasoning principles
based on three schools of moral philosophy to study when IP may or may not be
ethically appropriate. Equipped with both mathematical and ethical insights, we
present three diverse case studies illustrating the varied nature of the IP
problem and offer recommendations for future research. Our code and data are at
https://github.com/jiarui-liu/IP.
