  We study the excitation and damping of tides in close binary systems,
accounting for the leading order nonlinear corrections to linear tidal theory.
These nonlinear corrections include two distinct effects: three-mode nonlinear
interactions and nonlinear excitation of modes by the time-varying
gravitational potential of the companion. This paper presents the formalism for
studying nonlinear tides and studies the nonlinear stability of the linear
tidal flow. Although the formalism is applicable to binaries containing stars,
planets, or compact objects, we focus on solar type stars with stellar or
planetary companions. Our primary results include: (1) The linear tidal
solution often used in studies of binary evolution is unstable over much of the
parameter space in which it is employed. More specifically, resonantly excited
gravity waves are unstable to parametric resonance for companion masses M' >
10-100 M_Earth at orbital periods P = 1-10 days. The nearly static equilibrium
tide is, however, parametrically stable except for solar binaries with P < 2-5
days. (2) For companion masses larger than a few Jupiter masses, the dynamical
tide causes waves to grow so rapidly that they must be treated as traveling
waves rather than standing waves. (3) We find a novel form of parametric
instability in which a single parent wave excites a very large number of
daughter waves (N = 10^3[P / 10 days]) and drives them as a single coherent
unit with growth rates that are ~N times faster than the standard three wave
parametric instability. (4) Independent of the parametric instability, tides
excite a wide range of stellar p-modes and g-modes by nonlinear inhomogeneous
forcing; this coupling appears particularly efficient at draining energy out of
the dynamical tide and may be more important than either wave breaking or
parametric resonance at determining the nonlinear dissipation of the dynamical
tide.
