  The traditional, geometrical rotating vector model (RVM) has proved
particularly poor at capturing the polarization sweeps of the young energetic
and millisecond pulsars detected by \textit{Fermi}. We augment this model by
including finite altitude effects using a swept back vacuum dipole geometry. By
further including the effects of orthogonal mode jumps, multiple emission
altitudes, open zone growth via y-point lowering, and interstellar scattering,
we show that a wide range of departures from RVM can be modeled well while
retaining a geometrical picture. We illustrate these effects by fitting six
\textit{Fermi}-detected pulsars (J0023$+$0923, J1024$-$0719, J1744$-$1134,
J1057$-$5226, J1420$-$6048, and J2124$-$3358) and we describe how such modeling
can improve our understanding of their emission geometry.
