  The semiclassical general formula for the probability of radiation of twisted
photons by ultrarelativistic scalar and Dirac particles moving in the
electromagnetic field of a general form is derived. This formula is the analog
of the Baier-Katkov formula for the probability of radiation of one plane wave
photon with the quantum recoil taken into account. The derived formula is used
to describe the radiation of twisted photons by charged particles in undulators
and laser waves. Thus, the general theory of undulator radiation of twisted
photons and radiation of twisted photons in the nonlinear Compton process is
developed with account for the quantum recoil. The explicit formulas for the
probability to record a twisted photon are obtained in these cases. In
particular, we found that the quantum recoil and spin degrees of freedom
increase the radiation probability of twisted photons in comparison with the
formula for scalar particles without recoil. In the range of applicability of
the semiclassical formula, the selection rules for undulator radiation
established in the purely classical framework are not violated. The
manifestation of the blossoming out rose effect in the nonlinear Compton
process in a strong laser wave with circular polarization and in the wiggler
radiation is revealed. Several examples are studied: the radiation of MeV
twisted photons by $180$ GeV electrons in the wiggler; the radiation of twisted
photons by $256$ MeV electrons in strong electromagnetic waves produced by the
CO$_2$ and Ti:Sa lasers; and the radiation of MeV twisted photons by $51.1$ MeV
electrons in the electromagnetic wave generated by the FEL with photon energy
$1$ keV.
