  For the spherical mean operators $\mathcal{A}_t$ in $\mathbb{R}^d$, $d\ge 2$,
we consider the maximal functions $M_Ef =\sup_{t\in E} |\mathcal{A}_t f|$, with
dilation sets $E\subset [1,2]$. In this paper we give a surprising
characterization of the closed convex sets which can occur as closure of the
sharp $L^p$ improving region of $M_E$ for some $E$. This region depends on the
Minkowski dimension of $E$, but also other properties of the fractal geometry
such as the Assouad spectrum of $E$ and subsets of $E$. A key ingredient is an
essentially sharp result on $M_E$ for a class of sets called (quasi-)Assouad
regular which is new in two dimensions.
