  In this paper, we consider a nonlinear integral eigenvalue problem, which is
a reformulation of the transmission eigenvalue problem arising in the inverse
scattering theory. The boundary element method is employed for discretization,
which leads to a generalized matrix eigenvalue problem. We propose a novel
method based on the spectral projection. The method probes a given region on
the complex plane using contour integrals and decides if the region contains
eigenvalue(s) or not. It is particularly suitable to test if zero is an
eigenvalue of the generalized eigenvalue problem, which in turn implies that
the associated wavenumber is a transmission eigenvalue. Effectiveness and
efficiency of the new method are demonstrated by numerical examples.
