  In hierarchical triple systems, the inner binary is slowly perturbed by a
distant companion, giving rise to large-scale oscillations in eccentricity and
inclination, known as von-Zeipel-Lidov-Kozai (ZLK) oscillations. Stable systems
with a mild hierarchy, where the period ratio is not too small, require an
additional corrective term, known as the Brown Hamiltonian, to adequately
account for their long-term evolution. Although the Brown Hamiltonian has been
used to accurately describe the highly eccentric systems on circulating orbits
where the periapse completes a complete revolution, the analysis near its
elliptical fixed points had been overlooked. We derive analytically the
modified fixed points including the Brown Hamiltonian and analyse its librating
orbits (where the periapse motion is limited in range). We compare our result
to the direct three-body integrations of millions of orbits and discuss the
regimes of validity. We numerically discover the regions of orbital
instability, allowed and forbidden librating zones with a complex, fractal,
structure. The retrograde orbits, where the mutual inclination is $\iota > 90\
\rm deg$, are more stable and allowed to librate for larger areas of the
parameter space. We find numerical fits for the librating-circulating boundary.
Finally, we discuss the astrophysical implications for systems of satellites,
stars and compact objects. In a companion paper (paper II), we apply our
formalism to the orbits of irregular satellites around giant planets.
