  We develop the formalism required to study the nonlinear interaction of modes
in rotating Newtonian stars in the weakly nonlinear regime. The formalism
simplifies and extends previous treatments. At linear order, we elucidate and
extend slightly a formalism due to Schutz, show how to decompose a general
motion of a rotating star into a sum over modes, and obtain uncoupled equations
of motion for the mode amplitudes under the influence of an external force.
Nonlinear effects are added perturbatively via three-mode couplings. We
describe a new, efficient way to compute the coupling coefficients, to zeroth
order in the stellar rotation rate, using spin-weighted spherical harmonics.
  We apply this formalism to derive some properties of the coupling
coefficients relevant to the nonlinear interactions of unstable r-modes in
neutron stars, postponing numerical integrations of the coupled equations of
motion to a later paper. From an astrophysical viewpoint, the most interesting
result of this paper is that many couplings of r-modes to other rotational
modes (modes with zero frequencies in the non-rotating limit) are small: either
they vanish altogether because of various selection rules, or they vanish to
lowest order in the angular velocity. In zero-buoyancy stars, the coupling of
three r-modes is forbidden entirely and the coupling of two r-modes to one
hybrid rotational mode vanishes to zeroth order in rotation frequency. In
incompressible stars, the coupling of any three rotational modes vanishes to
zeroth order in rotation frequency.
