  Two-dimensional conformal field theories with Virasoro symmetry generically
contain a Schwarzian sector. This sector is related to the near-horizon region
of the near-extremal BTZ black hole in the holographic dual. In this work we
generalize this picture to CFTs with higher spin conserved currents. It is
shown that the partition function in the near-extremal limit agrees with that
of BF higher spin gravity in AdS$_2$ which is described by a generalized
Schwarzian theory. We also provide a spectral decomposition of Schwarzian
partition functions via the $\mathcal{W}_N$ fusion kernel and consider
supersymmetric generalizations.
