  We consider the problem of distinguishing, with minimum probability of error,
two optical beam-splitter channels with unequal complex-valued reflectivities
using general quantum probe states entangled over M signal and M' idler mode
pairs of which the signal modes are bounced off the beam splitter while the
idler modes are retained losslessly. We obtain a lower bound on the output
state fidelity valid for any pure input state. We define number-diagonal signal
(NDS) states to be input states whose density operator in the signal modes is
diagonal in the multimode number basis. For such input states, we derive series
formulas for the optimal error probability, the output state fidelity, and the
Chernoff-type upper bounds on the error probability. For the special cases of
quantum reading of a classical digital memory and target detection (for which
the reflectivities are real valued), we show that for a given input signal
photon probability distribution, the fidelity is minimized by the NDS states
with that distribution and that for a given average total signal energy N_s,
the fidelity is minimized by any multimode Fock state with N_s total signal
photons. For reading of an ideal memory, it is shown that Fock state inputs
minimize the Chernoff bound. For target detection under high-loss conditions, a
no-go result showing the lack of appreciable quantum advantage over coherent
state transmitters is derived. A comparison of the error probability
performance for quantum reading of number state and two-mode squeezed vacuum
state (or EPR state) transmitters relative to coherent state transmitters is
presented for various values of the reflectances. While the nonclassical states
in general perform better than the coherent state, the quantitative performance
gains differ depending on the values of the reflectances.
