  Experiments at the interface of quantum-optics and chemistry have revealed
that strong coupling between light and matter can substantially modify chemical
and physical properties of molecules and solids. While the theoretical
description of such situations is usually based on non-relativistic quantum
electrodynamics, which contains quadratic light-matter coupling terms, it is
commonplace to disregard these terms and restrict to purely bilinear couplings.
In this work we clarify the physical origin and the substantial impact of the
most common quadratic terms, the diamagnetic and self-polarization terms, and
highlight why neglecting them can lead to rather unphysical results.
Specifically we demonstrate its relevance by showing that neglecting it leads
to the loss of gauge invariance, basis-set dependence, disintegration (loss of
bound states) of any system in the basis set-limit, unphysical radiation of the
ground state and an artificial dependence on the static dipole. Besides
providing important guidance for modeling strongly coupled light-matter
systems, the presented results do also indicate under which conditions those
effects might become accessible.
