  The minimal degree of a permutation group $G$ is the minimum number of points
not fixed by non-identity elements of $G$. Lower bounds on the minimal degree
have strong structural consequences on $G$. Babai conjectured that if a
primitive coherent configuration with $n$ vertices is not a Cameron scheme,
then its automorphism group has minimal degree $\geq cn$ for some constant
$c>0$. In 2014, Babai proved the desired lower bound on the minimal degree of
the automorphism groups of strongly regular graphs, thus confirming the
conjecture for primitive coherent configurations of rank 3.
  In this paper, we extend Babai's result to primitive coherent configurations
of rank 4, confirming the conjecture in this special case. The proofs combine
structural and spectral methods.
