  We show that every finitely generated group G with an element of order at
least $(5rank(G))^{12}$ admits a locally finite directed Cayley graph with
automorphism group equal to G. If moreover G is not generalized dihedral, then
the above Cayley directed graph does not have bigons. On the other hand, if G
is neither generalized dicyclic nor abelian and has an element of order at
least $(2rank(G))^{36}$, then it admits an undirected Cayley graph with
automorphism group equal to G. This extends classical results for finite groups
and free products of groups. The above results are obtained as corollaries of a
stronger form of rigidity which says that the rigidity of the graph can be
observed in a ball of radius 1 around a vertex. This strong rigidity result
also implies that the Cayley (di)graph covers very few (di)graphs. In
particular, we obtain Cayley graphs of Tarski monsters which essentially do not
cover other quasi-transitive graphs.
  We also show that a finitely generated group admits a locally finite labelled
unoriented Cayley graph with automorphism group equal to itself if and only if
it is neither generalized dicyclic nor abelian with an element of order greater
than 2.
