  Capture-recapture experiments are widely used to estimate the abundance of a
finite population. Based on capture-recapture data, the empirical likelihood
(EL) method has been shown to outperform the conventional conditional
likelihood (CL) method. However, the current literature on EL abundance
estimation ignores behavioral effects, and the EL estimates may not be stable,
especially when the capture probability is low. We make three contributions in
this paper. First, we extend the EL method to capture-recapture models that
account for behavioral effects. Second, to overcome the instability of the EL
method, we propose a penalized EL (PEL) estimation method that penalizes large
abundance values. We then investigate the asymptotics of the maximum PEL
estimator and the PEL ratio statistic. Third, we develop standard
expectation-maximization (EM) algorithms for PEL to improve its practical
performance. The EM algorithm is also applicable to EL and CL with slight
modifications. Our simulation and a real-world data analysis demonstrate that
the PEL method successfully overcomes the instability of the EL method and the
proposed EM algorithm produces more reliable results than existing optimization
algorithms.
