  Ruberman in the 90's showed that the group of exotic diffeomorphisms of
closed 4-manifolds can be infinitely generated. We provide various results on
the question of when such infinite generation can localize to a smaller
embedded submanifold of the original manifold. Our results include: (1) All
known infinitely generated groups of exotic diffeomorphisms of 4-manifolds
detected by families Seiberg-Witten theory do not localize to any topologically
(locally-flatly) embedded rational homology balls in the ambient 4-manifold.
(2) Many exotic diffeomorphisms cannot be obtained as Dehn twists along
homology spheres (under mild assumptions). (3) There is no contractible
4-manifolds with Seifert fibered boundary that have a universal property for
exotic diffeomorphisms analogous to a universal cork. In addition, there is no
universal compact 4-manifold $W$ such that the set of exotic diffeomorphisms of
a 4-manifold can localize to an embedding of $W$. (4) Certain infinite
generations of exotic diffeomorphism groups do localize to a non-compact subset
$V$ with a small Betti number, but not to any compact subset of $V$. (5) An
analogous result holds for mapping class groups of 4-manifolds.
