  Recent measurements of Galactic polarized dust emission have found a nonzero
$TB$ signal, a correlation between the total intensity and the $B$-mode
polarization component. We present evidence that this parity-odd signal is
driven by the relative geometry of the magnetic field and the filamentary
interstellar medium in projection. Using neutral hydrogen morphology and Planck
polarization data, we find that the angle between intensity structures and the
plane-of-sky magnetic field orientation is predictive of the signs of Galactic
$TB$ and $EB$. Our results suggest that magnetically misaligned filamentary
dust structures introduce nonzero $TB$ and $EB$ correlations in the dust
polarization, and that the intrinsic dust $EB$ can be predicted from
measurements of dust $TB$ and $TE$ over the same sky mask. We predict
correlations between $TE$, $TB$, $EB$, and $EE/BB$, and confirm our predictions
using synthetic dust polarization maps from magnetohydrodynamic simulations. We
introduce and measure a scale-dependent effective magnetic misalignment angle,
$\psi_\ell^{dust} \sim 5^\circ$ for $100 \lesssim \ell \lesssim 500$, and
predict a positive intrinsic dust $EB$ with amplitude $\left<D_\ell^{EB}\right>
\lesssim 2.5~\mu\mathrm{K^2_{CMB}}$ for the same multipole range at 353 GHz
over our sky mask. Both the sign and amplitude of the Galactic $EB$ signal can
change with the sky area considered. Our results imply that searches for parity
violation in the cosmic microwave background must account for the nonzero
Galactic $EB$ and $TB$ signals, necessitating revision of existing analyses of
the evidence for cosmic birefringence.
