  Any finite-dimensional $p$-adic representation of the absolute Galois group
of a $p$-adic local field with imperfect residue field is characterized by its
arithmetic and geometric Sen operators defined by Sen and Brinon. We generalize
their construction to the fundamental group of a $p$-adic affine variety with a
semi-stable chart, and prove that the module of Sen operators is canonically
defined, independently of the choice of the chart. Our construction relies on a
descent theorem in the $p$-adic Simpson correspondence developed by Tsuji. When
the representation comes from a $\mathbb{Q}_p$-representation of a $p$-adic
analytic group quotient of the fundamental group, we describe its Lie algebra
action in terms of the Sen operators, which is a generalization of a result of
Sen and Ohkubo. These Sen operators can be extended continuously to certain
infinite-dimensional representations. As an application, we prove that the
geometric Sen operators annihilate locally analytic vectors, generalizing a
result of Pan.
