  Hyperspectral image fusion (HIF) is critical to a wide range of applications
in remote sensing and many computer vision applications. Most traditional HIF
methods assume that the observation model is predefined or known. However, in
real applications, the observation model involved are often complicated and
unknown, which leads to the serious performance drop of many advanced HIF
methods. Also, deep learning methods can achieve outstanding performance, but
they generally require a large number of image pairs for model training, which
are difficult to obtain in realistic scenarios. Towards these issues, we
proposed a self-regression learning method that alternatively reconstructs
hyperspectral image (HSI) and estimate the observation model. In particular, we
adopt an invertible neural network (INN) for restoring the HSI, and two
fully-connected network (FCN) for estimating the observation model. Moreover,
\emph{SoftMax} nonlinearity is applied to the FCN for satisfying the
non-negative, sparsity and equality constraints. Besides, we proposed a local
consistency loss function to constrain the observation model by exploring
domain specific knowledge. Finally, we proposed an angular loss function to
improve spectral reconstruction accuracy. Extensive experiments on both
synthetic and real-world dataset show that our model can outperform the
state-of-the-art methods
