  We present the source separation framework SCARLET for multi-band images,
which is based on a generalization of the Non-negative Matrix Factorization to
alternative and several simultaneous constraints. Our approach describes the
observed scene as a mixture of components with compact spatial support and
uniform spectra over their support. We present the algorithm to perform the
matrix factorization and introduce constraints that are useful for optical
images of stars and distinct stellar populations in galaxies, in particular
symmetry and monotonicity with respect to the source peak position. We also
derive the treatment of correlated noise and convolutions with band-dependent
point spread functions, rendering our approach applicable to coadded images
observed under variable seeing conditions. SCARLET thus yields a PSF-matched
photometry measurement with an optimally chosen weight function given by the
mean morphology in all available bands. We demonstrate the performance of
SCARLET for deblending crowded extragalactic scenes and on an AGN jet -- host
galaxy separation problem in deep 5-band imaging from the Hyper Suprime-Cam
Stategic Survey Program. Using simulations with prominent crowding we show that
SCARLET yields superior results to the HSC-SDSS deblender for the recovery of
total fluxes, colors, and morphologies. Due to its non-parametric nature, a
conceptual limitation of SCARLET is its sensitivity to undetected sources or
multiple stellar population within detected sources, but an iterative strategy
that adds components at the location of significant residuals appears
promising. The code is implemented in Python with C++ extensions and is
available at https://github.com/fred3m/scarlet
