  We study the possibility that the $\gamma$-ray loud active galactic nuclei
(AGN) are the sources of ultra-high energy cosmic rays (UHECR), through the
correlation analysis of their locations and the arrival directions of UHECR. We
use the $\gamma$-ray loud AGN with $d\le 100 {\rm Mpc}$ from the second Fermi
Large Area Telescope AGN catalog and the UHECR data with $E\ge 55 {\rm EeV}$
observed by Pierre Auger Observatory. The distribution of arrival directions
expected from the $\gamma$-ray loud AGN is compared with that of the observed
UHECR using the correlational angular distance distribution and the
Kolmogorov-Smirnov test. We conclude that the hypothesis that the $\gamma$-ray
loud AGN are the dominant sources of UHECR is disfavored unless there is a
large smearing effect due to the intergalactic magnetic fields.
