  The expected Euler characteristic (EEC) method is an integral-geometric
method used to approximate the tail probability of the maximum of a random
field on a manifold. Noting that the largest eigenvalue of a real-symmetric or
Hermitian matrix is the maximum of the quadratic form of a unit vector, we
provide EEC approximation formulas for the tail probability of the largest
eigenvalue of orthogonally invariant random matrices of a large class. For this
purpose, we propose a version of a skew-orthogonal polynomial by adding a side
condition such that it is uniquely defined, and describe the EEC formulas in
terms of the (skew-)orthogonal polynomials. In addition, for the classical
random matrices (Gaussian, Wishart, and multivariate beta matrices), we analyze
the limiting behavior of the EEC approximation as the matrix size goes to
infinity under the so-called edge-asymptotic normalization. It is shown that
the limit of the EEC formula approximates well the Tracy-Widom distributions in
the upper tail area, as does the EEC formula when the matrix size is finite.
