  We theoretically investigate the enhancement of surface enhanced Raman
spectroscopy (SERS) using hyperbolic stratified nanostructures and compare to
metal nanoresonators. The photon Green function of each nanostructure within
its environment is first obtained from a semi-analytical modal theory, which is
used in a quantum optics formalism of the molecule-nanostructure interaction to
model the SERS spectrum. An intuitive methodology is presented for calculating
the single molecule enhancement factor (SMEF), which is also able to predict
known experimental SERS enhancement factors of an example gold nano-dimer. We
elucidate the important figures-of-merit of the enhancement and explore these
for different designs. We find that the use of hyperbolic stratified materials
can enhance the photonic local density of states (LDOS) by close to 2 times in
comparison to pure metal nanostructures, when both designed to work at the same
operating wavelengths. However, the increased LDOS is accompanied by higher
electric field concentration within the lossy hyperbolic material, which leads
to increased quenching that serves to reduce the overall detected SERS
enhancement in the far field. For nanoresonators with resonant localized
surface plasmon wavelengths in the near-infrared, the SMEF for the hyperbolic
stratified nanostructure is approximately an order of magnitude lower than the
pure metal counterpart. Conversely, we show that by detecting the Raman signal
using a near-field probe, hyperbolic materials can provide an improvement in
SERS enhancement compared to using pure metal nanostructures when the probe is
sufficiently close (<50 nm) to the Raman active molecule at the plasmonic
hotspot.
