  Agency, the capacity to proactively shape events, is central to how humans
interact and collaborate. While LLMs are being developed to simulate human
behavior and serve as human-like agents, little attention has been given to the
Agency that these models should possess in order to proactively manage the
direction of interaction and collaboration. In this paper, we investigate
Agency as a desirable function of LLMs, and how it can be measured and managed.
We build on social-cognitive theory to develop a framework of features through
which Agency is expressed in dialogue - indicating what you intend to do
(Intentionality), motivating your intentions (Motivation), having self-belief
in intentions (Self-Efficacy), and being able to self-adjust (Self-Regulation).
We collect a new dataset of 83 human-human collaborative interior design
conversations containing 908 conversational snippets annotated for Agency
features. Using this dataset, we develop methods for measuring Agency of LLMs.
Automatic and human evaluations show that models that manifest features
associated with high Intentionality, Motivation, Self-Efficacy, and
Self-Regulation are more likely to be perceived as strongly agentive.
