  Previous studies of the modulation instability (MI) of continuous waves (CWs)
in a two-core fiber (TCF) did not consider effects caused by co-propagation of
the two polarized modes in a TCF that possesses birefringence, such as
cross-phase modulation (XPM), polarization-mode dispersion (PMD), and
polarization-dependent coupling (PDC) between the cores. This paper reports an
analysis of these effects on the MI by considering a linear-birefringence TCF
and a circular-birefringence TCF, which feature different XPM coefficients. The
analysis focuses on the MI of the asymmetric CW states in the TCFs, which have
no counterparts in single-core fibers. We find that, the asymmetric CW state
exists when its total power exceeds a threshold (minimum) value, which is
sensitive to the value of the XPM coefficient. We consider, in particular, a
class of asymmetric CW states that admit analytical solutions. In the anomalous
dispersion regime, without taking the PMD and PDC into account, the MI gain
spectra of the birefringent TCF, if scaled by the threshold power, are almost
identical to those of the zero-birefringence TCF. However, in the normal
dispersion regime, the power-scaled MI gain spectra of the birefringent TCFs
are distinctly different from their zero-birefringence counterparts, and the
difference is particularly significant for the circular-birefringence TCF,
which takes a larger XPM coefficient. On the other hand, the PMD and PDC only
exert weak effects on the MI gain spectra. We also simulate the nonlinear
evolution of the MI of the CW inputs in the TCFs and obtain a good agreement
with the analytical solutions.
