This repository contains the PyTorch codes for paper "Deep Hyperspectral Image Fusion Network with Iterative Spatio-Spectral Regularization" (IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING (IEEE TCI), VOL. 6, 2022) by Tao Huang, Weisheng Dong, Xin Li.
Physical acquisition of high-resolution hyperspectral images (HR-HSI) has remained difficult, despite its potential of resolving material-related ambiguities in vision applications. Deep hyperspectral image fusion, aiming at reconstructing an HR-HSI from a pair of low-resolution hyperspectral image (LRHSI) and high-resolution multispectral image (HR-MSI), has become an appealing computational alternative. Existing fusion methods either rely on hand-crafted image priors or treat fusion as a nonlinear mapping problem, ignoring important physical imaging models. In this paper, we propose a novel regularization strategy to fully exploit the spatio-spectral dependency by a spatially adaptive 3D filter. Moreover, the joint exploitation of spatio-spectral regularization and physical imaging models inspires us to formulate deep hyperspectral image fusion as a differentiable optimization problem. We show how to solve this optimization problem by an end-to-end training of a model-guided unfolding network named DHIF-Net. Unlike existing works of simply concatenating spatial with spectral regularization, our approach aims at an end-to-end optimization of iterative spatio-spectral regularization by multistage network implementations. Our extensive experimental results on both synthetic and real datasets have shown that our DHIF-Net outperforms other competing methods in terms of both objective and subjective visual quality.
-
Requirements are Python 3 and PyTorch 1.7.0.
-
Download this repository via git
git clone https://github.com/TaoHuang95/DHIF-Net
or download the [zip file] manually.
-
Training simulation model
-
Put hyperspectral image datasets (Ground truth) and RGB datasets into corrsponding path, i.e., 'CAVE/Data/Train/HSI (RGB)'.
-
Run CAVE/Train.py.
-
-
Testing on simulation data [Checkpoint]
- Run CAVE/Test.py to reconstruct 12 synthetic datasets. The results will be saved in 'CAVE/Result/' in the MAT File format.
If you find our work useful for your research, please consider citing the following papers :)
@article{huang2022deep,
title={Deep hyperspectral image fusion network with iterative spatio-spectral regularization},
author={Huang, Tao and Dong, Weisheng and Wu, Jinjian and Li, Leida and Li, Xin and Shi, Guangming},
journal={IEEE Transactions on Computational Imaging},
volume={8},
pages={201--214},
year={2022},
publisher={IEEE}
}
Tao Huang, Xidian University, Email: thuang_666@stu.xidian.edu.cn, thuang951223@163.com
Weisheng Dong, Xidian University, Email: wsdong@mail.xidian.edu.cn
Xin Li, West Virginia University, Email: xin.li@ieee.org