A hybrid generative adversarial network for weakly-supervised cloud detection in multispectral images

Jun Li, Zhaocong Wu*, Qinghong Sheng, Bo Wang, Zhongwen Hu, Shaobo Zheng, Gustau Camps-Valls, Matthieu Molinier, Jing M. Chen (Editor)

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

36 Citations (Scopus)

Abstract

Cloud detection is a crucial step in the optical satellite image processing pipeline for Earth observation. Clouds in optical remote sensing images seriously affect the visibility of the background and greatly reduce the usability of images for land applications. Traditional methods based on thresholding, multi-temporal or multi-spectral information are often specific to a particular satellite sensor. Convolutional Neural Networks for cloud detection often require labeled cloud masks for training that are very time-consuming and expensive to obtain. To overcome these challenges, this paper presents a hybrid cloud detection method based on the synergistic combination of generative adversarial networks (GAN) and a physics-based cloud distortion model (CDM). The proposed weakly-supervised GAN-CDM method (available online https://github.com/Neooolee/GANCDM) only requires patch-level labels for training, and can produce cloud masks at pixel-level in both training and testing stages. GAN-CDM is trained on a new globally distributed Landsat 8 dataset (WHUL8-CDb, available online doi:https://doi.org/10.5281/zenodo.6420027) including image blocks and corresponding block-level labels. Experimental results show that the proposed GAN-CDM method trained on Landsat 8 image blocks achieves much higher cloud detection accuracy than baseline deep learning-based methods, not only in Landsat 8 images (L8 Biome dataset, 90.20% versus 72.09%) but also in Sentinel-2 images (“S2 Cloud Mask Catalogue” dataset, 92.54% versus 77.00%). This suggests that the proposed method provides accurate cloud detection in Landsat images, has good transferability to Sentinel-2 images, and can quickly be adapted for different optical satellite sensors.

Original languageEnglish
Article number113197
JournalRemote Sensing of Environment
Volume280
DOIs
Publication statusPublished - Oct 2022
MoE publication typeA1 Journal article-refereed

Funding

We would like to thank Zhe Zhu and Shi Qiu (University of Connecticut) for providing the Fmask 4.0 validation dataset. The authors are grateful for the Sentinel-2 data services from Copernicus Open Access Hub. This work is supported in part by the Natural Science Foundation of Jiangsu Province (No. BK20220888 ), in part by Fundamental Research Funds for the Central Universities (No. NT2022022 ), in part by the National Key Laboratory Foundation 2021-JCJQ-LB-006 (Grant No. 6142411442120 ) and in part by the National Key Laboratory Foundation (Contract No. 6142411203416 ). Gustau Camps-Valls would like to acknowledge the support from the European Research Council (ERC) under the ERC Synergy Grant USMILE (grant agreement 855187) and DEEPCLOUD project funded by the Spanish Ministry of Science, Innovation and Universities (MCIU/AEI/FEDER, UE) PID2019-109026RB-I00 . The work of Matthieu Molinier is part of the Finnish Flagship Programme FCAI: Finnish Center for Artificial Intelligence, by the Academy of Finland. We would like to thank Zhe Zhu and Shi Qiu (University of Connecticut) for providing the Fmask 4.0 validation dataset. The authors are grateful for the Sentinel-2 data services from Copernicus Open Access Hub. This work is supported in part by the Natural Science Foundation of Jiangsu Province (No. BK20220888), in part by Fundamental Research Funds for the Central Universities (No. NT2022022), in part by the National Key Laboratory Foundation 2021-JCJQ-LB-006 (Grant No. 6142411442120) and in part by the National Key Laboratory Foundation (Contract No. 6142411203416). Gustau Camps-Valls would like to acknowledge the support from the European Research Council (ERC) under the ERC Synergy Grant USMILE (grant agreement 855187) and DEEPCLOUD project funded by the Spanish Ministry of Science, Innovation and Universities (MCIU/AEI/FEDER, UE) PID2019-109026RB-I00. The work of Matthieu Molinier is part of the Finnish Flagship Programme FCAI: Finnish Center for Artificial Intelligence, by the Academy of Finland.

Keywords

  • Cloud detection
  • Cloud distortion model
  • Deep learning
  • Generative adversarial networks (GAN)
  • Remote sensing

Fingerprint

Dive into the research topics of 'A hybrid generative adversarial network for weakly-supervised cloud detection in multispectral images'. Together they form a unique fingerprint.

Cite this