  Ever since the discovery of the charge density wave (CDW) transition in the
kagome metal CsV$_3$Sb$_5$, the nature of its symmetry breaking is under
intense debate. While evidence suggests that the rotational symmetry is already
broken at the CDW transition temperature ($T_{\rm CDW}$), an additional
electronic nematic instability well below $T_{\rm CDW}$ has been reported based
on the diverging elastoresistivity coefficient in the anisotropic channel
($m_{E_{2g}}$). Verifying the existence of a nematic transition below $T_{\rm
CDW}$ is not only critical for establishing the correct description of the CDW
order parameter, but also important for understanding low-temperature
superconductivity. Here, we report elastoresistivity measurements of
CsV$_3$Sb$_5$ using three different techniques probing both isotropic and
anisotropic symmetry channels. Contrary to previous reports, we find the
anisotropic elastoresistivity coefficient $m_{E_{2g}}$ is
temperature-independent, except for a step jump at $T_{\rm CDW}$. The absence
of nematic fluctuations is further substantiated by measurements of the
elastocaloric effect, which show no enhancement associated with nematic
susceptibility. On the other hand, the symmetric elastoresistivity coefficient
$m_{A_{1g}}$ increases below $T_{\rm CDW}$, reaching a peak value of 90 at $T^*
= 20$ K. Our results strongly indicate that the phase transition at $T^*$ is
not nematic in nature and the previously reported diverging elastoresistivity
is due to the contamination from the $A_{1g}$ channel.
