  The recent measurement of muon anomalous magnetic dipole moment (muon $g-2$)
suggests that there might exist new physics that dominantly interacts with
muons. The observed gamma-ray excess from Fermi-LAT indicates that dark matter
annihilates into a specific charged fermions. We propose a successful model
simultaneously to explain the Fermi-LAT GeV gamma-ray excess and sizable muon
$g-2$ with a modular $A_4$ symmetry. Due to nature of this symmetry, our DM
only interacts with pairs of muon and we explain sizable muon $g-2$ without
suffering from constraints of any lepton flavor violations. We numerically show
our allowed spaces on each measurements of Fermi-LAT, relic density of DM and
muon $g-2$, randomly scanning our input parameters.
