  We present a new additive image factorization technique that treats images to
be composed of multiple latent specular components which can be simply
estimated recursively by modulating the sparsity during decomposition. Our
model-driven {\em RSFNet} estimates these factors by unrolling the optimization
into network layers requiring only a few scalars to be learned. The resultant
factors are interpretable by design and can be fused for different image
enhancement tasks via a network or combined directly by the user in a
controllable fashion. Based on RSFNet, we detail a zero-reference Low Light
Enhancement (LLE) application trained without paired or unpaired supervision.
Our system improves the state-of-the-art performance on standard benchmarks and
achieves better generalization on multiple other datasets. We also integrate
our factors with other task specific fusion networks for applications like
deraining, deblurring and dehazing with negligible overhead thereby
highlighting the multi-domain and multi-task generalizability of our proposed
RSFNet. The code and data is released for reproducibility on the project
homepage.
