  Within the framework of leading power factorization formalism of
nonrelativistic quantum chromodynamics, we calculate the jet fragmentation
function for $J/\psi$ production in proton-proton (pp) collisions ranging from
$\sqrt{s}=500$ GeV to $13$ TeV. The reasonable agreements between theory and
experimental data indicate that $J/\psi$ production within a jet is mainly
dominated by gluon fragmentation. Such a mechanism can be further tested by the
predicted jet transverse momentum and radius dependence of jet fragmentation
function. Based on the satisfying description of pp baseline, we carry out the
first theoretical investigation on medium modification on $J/\psi$ production
within jet in heavy-ion collisions at the Large Hadron Collider, using a linear
Boltzmann transport model combined with hydrodynamics for the simulation of
jet-medium interaction. The consistency with the experimental measurement on
nuclear modification factor $R_{\text{AA}}$ by CMS collaboration reveals that
the gluon jet quenching is the driving force for the suppression of $J/\psi$
production in jet. Furthermore, we make predictions for the dependence of
$R_{\text{AA}}$ on the jet transverse momentum and jet radius $R$, which can be
tested in future measurements to further constrain the flavor dependence of jet
quenching.
