  In this paper, we build up Hill-type formula for linear Hamiltonian systems
with Lagrangian boundary conditions, which include standard Neumann, Dirichlet
boundary conditions. Such a kind of boundary conditions comes from the brake
symmetry periodic orbits in $n$-body problem naturally. The Hill-type formula
connects the infinite determinant of the Hessian of the action functional with
the determinant of matrices which depend on the monodromy matrix and boundary
conditions. Consequently, we derive the Krein-type trace formula and give
nontrivial estimation for the eigenvalue problem. Combined with the Maslov-type
index theory, we give some new stability criteria for the brake symmetry
periodic solutions of Hamiltonian systems. As an application, we study the
linear stability of elliptic relative equilibria in planar $3$-body problem.
