[TCSVT 2022] Exploring the Relationship between Center and Neighborhoods: Central Vector Oriented Self-Similarity Network for Hyperspectral Image Classification

Mingsong Li, Yikun Liu, Guangkuo Xue, Yuwen Huang, and Gongping Yang

Time Lab, SDU

---

This repository is the official implementation of our paper: Exploring the Relationship between Center and Neighborhoods: Central Vector Oriented Self-Similarity Network for Hyperspectral Image Classification, IEEE Transactions on Circuits and Systems for Video Technology (TCSVT) 2022.

Contents

1. Brief Introduction
2. Environment
3. Data Sets and File Hierarchy
4. Implementations of Compared Methods
5. Citation
6. License and Acknowledgement

Brief Introduction

To mine the spectral-spatial information of target pixel in hyperspectral image classification (HSIC), convolutional neural network (CNN)-based models widely adopt patch-based input pattern, where a patch represents its central pixel and the neighbor pixels play auxiliary roles in the classification process. However, compared to the central pixel, its neighbor pixels often have different contributions for classification. Although many existing patch-based CNNs could adaptively emphasize the spatial neighbor information, most of them ignore the latent relationship between the center pixel and its neighbor pixels. Moreover, efficient spectral-spatial feature extraction has been a difficult yet vital topic for HSIC. To address the mentioned problems, a central vector oriented self-similarity network (CVSSN) is proposed for HSIC. Specifically, based on two similarity measures, we firstly design an adaptive weight addition based spectral vector self-similarity module (AWA-SVSS) in input space and a Euclidean distance based feature vector self-similarity module (ED-FVSS) in feature space to fully mine the central vector oriented spatial relationships. Besides, a spectral-spatial information fusion module (SSIF) is formulated as a new pattern to fuse the central 1D spectral vector and the corresponding 3D patch for efficient spectral-spatial feature learning of the subsequent modules. Moreover, we implement a channel spatial separation convolution module (CSS-Conv) and a scale information complementary convolution module (SIC-Conv) for efficient spectral-spatial feature learning. Extensive experimental results on four popular HSI data sets demonstrate the effectiveness and efficiency of the proposed method compared with other state-of-the-art methods. The source code is available at https://github.com/lms-07/CVSSN.

CVSSN Framework

AWA-SVSS Module	ED-FVSS Module

Environment

• The software environment is Ubuntu 18.04.5 LTS 64 bit.
• This project is running on a single Nvidia GeForce RTX 3090 GPU based on Cuda 11.0.
• We adopt Python 3.8.5, PyTorch 1.8.1+cu111.
• The py+torch combination may not be limited by our adopted one.

Data Sets and File Hierarchy

Four popular HSI data sets are adopted in our experiments, i.e., Indian Pines (IP), Kennedy Space Center (KSC), University of Pavia (UP), and University of Houston 13 (UH). The first three data sets could be accessed through link1, and the UH data set through link2. Our project is organized as follows:

CVSSN
|-- process_xxx     // main files 1) cls for two classic methods 2) dl for eight dl
|                      based methods 3) disjoint for the disjoint dataset (UH)
|-- data                    
|   |-- IP
|   |   |-- Indian_pines_corrected.mat
|   |   |-- Indian_pines_gt.mat
|   |-- KSC
|   |   |-- KSC.mat
|   |   |-- KSC_gt.mat
|   |-- UP
|   |   |-- PaviaU.mat
|   |   |-- PaviaU_gt.mat
|   |-- HU13_tif
|   |   |--Houston13_data.mat
|   |   |--Houston13_gt_train.mat
|   |   |--Houston13_gt_test.mat
|-- model           // seven dl based compared methods and our proposed method
|-- output
|   |-- cls_maps    // classification map visualizations 
|   |-- results     // classification result files
|-- src             // source files
|-- utils           // data loading, processing, and evaluating
|-- visual          // cls maps visual

Implementations of Compared Methods

For comparisons, our codebase also includes related compared methods.

• RF, PyTorch version, sklearn-based
• SVM, PyTorch version, sklearn-based
• ContextNet, Going Deeper With Contextual CNN for Hyperspectral Image Classification TIP 2017, our corrected version from an HSIC Tool Codebase, https://github.com/eecn/Hyperspectral-Classification
• RSSAN, Residual Spectral–Spatial Attention Network for Hyperspectral Image Classification TGRS 2020, from an unofficial PyTorch implementation, https://github.com/lierererniu/RSSAN-Hyperspectral-Image
• SSTN, Spectral-Spatial Transformer Network for Hyperspectral Image Classification: A Factorized Architecture Search Framework TGRS 2021, from official PyTorch version, https://github.com/zilongzhong/SSTN/blob/main/NetworksBlocks.py
• SSAN, Spectral–Spatial Attention Network for Hyperspectral Image Classification TGRS 2019, our unofficial PyTorch implementation based on TensorFlow version, https://github.com/XiangtaoZheng/SSAN
• SSSAN, Spectral–Spatial Self-Attention Networks for Hyperspectral Image Classification TGRS 2021, our unofficial PyTorch implementation based on the part of source Keras code from the author Dr. Xuming Zhang
• SSAtt, Hyperspectral Image Classification With Attention-Aided CNNs TGRS 2020, from a PyTorch implementation, https://github.com/weecology/DeepTreeAttention/blob/main/src/models/Hang2020.py
• A2S2K-ResNet, Attention-Based Adaptive Spectral-Spatial Kernel ResNet for Hyperspectral Image Classification TGRS 2020, from official PyTorch version, https://github.com/suvojit-0x55aa/A2S2K-ResNet/blob/master/A2S2KResNet/A2S2KResNet.py

Citation

Please kindly cite our work if this work is helpful for your research.

[1] M. Li, Y. Liu, G. Xue, Y. Huang and G. Yang, "Exploring the Relationship Between Center and Neighborhoods: Central Vector Oriented Self-Similarity Network for Hyperspectral Image Classification," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 4, pp. 1979-1993, April 2023.

BibTex entry:

@article{li2022exploring,
   title={Exploring the Relationship between Center and Neighborhoods: Central Vector Oriented Self-Similarity Network for Hyperspectral Image Classification},
   author={Li, Mingsong and Liu, Yikun and Xue, Guangkuo and Huang, Yuwen and Yang, Gongping},
   journal={IEEE Transactions on Circuits and Systems for Video Technology},
   year={2023},
   volume={33},
   number={4},
   pages={1979-1993},
   publisher={IEEE}
}

Contact information

If you have any problem, please do not hesitate to contact us msli@mail.sdu.edu.cn.

License and Acknowledgement

This project is released under GPLv3 license.

• We would like to thank the Hyperspectral Image Analysis group and the NSF Funded Center for Airborne Laser Mapping (NCALM) at the University of Houston for providing the UH dataset used in this work.
• Part of our HSIC framework is referred to HybridSN, A2S2K-ResNet, and CNN_Enhanced_GCN. Please also follow their licenses. Thanks for their awesome works.
• Among the adopted compared methods, we also would like to thank Assistant Professor Xiangtao Zheng and Dr. Xuming Zhang for providing the source tensorflow code of SSAN and the part of source keras code of SSSAN, respectively.