  We apply moderate-high-energy inelastic neutron scattering (INS) measurements
to investigate Yb$^{3+}$ crystalline electric field (CEF) levels in the
triangular spin-liquid candidate YbMgGaO$_4$. Three CEF excitations from the
ground-state Kramers doublet are centered at the energies $\hbar \omega$ = 39,
61, and 97\,meV in agreement with the effective \mbox{spin-1/2} $g$-factors and
experimental heat capacity, but reveal sizable broadening. We argue that this
broadening originates from the site mixing between Mg$^{2+}$ and Ga$^{3+}$
giving rise to a distribution of Yb--O distances and orientations and, thus, of
CEF parameters that account for the peculiar energy profile of the CEF
excitations. The CEF randomness gives rise to a distribution of the effective
spin-1/2 $g$-factors and explains the unprecedented broadening of low-energy
magnetic excitations in the fully polarized ferromagnetic phase of YbMgGaO$_4$,
although a distribution of magnetic couplings due to the Mg/Ga disorder may be
important as well.
