Embedded Agency
  Traditional models of rational action treat the agent as though it is cleanly
separated from its environment, and can act on that environment from the
outside. Such agents have a known functional relationship with their
environment, can model their environment in every detail, and do not need to
reason about themselves or their internal parts.
  We provide an informal survey of obstacles to formalizing good reasoning for
agents embedded in their environment. Such agents must optimize an environment
that is not of type "function"; they must rely on models that fit within the
modeled environment; and they must reason about themselves as just another
physical system, made of parts that can be modified and that can work at cross
purposes.
