Binary patterns encoded convolutional neural networks for texture recognition and remote sensing scene classification

Rao Muhammad Anwer, Fahad Shahbaz Khan, Joost van de Weijer, Matthieu Molinier, Jorma Laaksonen

    Research output: Contribution to journalArticleScientificpeer-review

    33 Citations (Scopus)

    Abstract

    Designing discriminative powerful texture features robust to realistic imaging conditions is a challenging computer vision problem with many applications, including material recognition and analysis of satellite or aerial imagery. In the past, most texture description approaches were based on dense orderless statistical distribution of local features. However, most recent approaches to texture recognition and remote sensing scene classification are based on Convolutional Neural Networks (CNNs). The de facto practice when learning these CNN models is to use RGB patches as input with training performed on large amounts of labeled data (ImageNet). In this paper, we show that Local Binary Patterns (LBP) encoded CNN models, codenamed TEX-Nets, trained using mapped coded images with explicit LBP based texture information provide complementary information to the standard RGB deep models. Additionally, two deep architectures, namely early and late fusion, are investigated to combine the texture and color information. To the best of our knowledge, we are the first to investigate Binary Patterns encoded CNNs and different deep network fusion architectures for texture recognition and remote sensing scene classification. We perform comprehensive experiments on four texture recognition datasets and four remote sensing scene classification benchmarks: UC-Merced with 21 scene categories, WHU-RS19 with 19 scene classes, RSSCN7 with 7 categories and the recently introduced large scale aerial image dataset (AID) with 30 aerial scene types. We demonstrate that TEX-Nets provide complementary information to standard RGB deep model of the same network architecture. Our late fusion TEX-Net architecture always improves the overall performance compared to the standard RGB network on both recognition problems. Furthermore, our final combination leads to consistent improvement over the state-of-the-art for remote sensing scene classification.
    Original languageEnglish
    Pages (from-to)74–85
    Number of pages12
    JournalISPRS Journal of Photogrammetry and Remote Sensing
    Volume138
    DOIs
    Publication statusPublished - 1 Apr 2018
    MoE publication typeA1 Journal article-refereed

    Fingerprint

    remote sensing
    Remote sensing
    textures
    Textures
    texture
    Neural networks
    Fusion reactions
    fusion
    Antennas
    patch use
    aerial photography
    computer vision
    satellite imagery
    statistical distribution
    Network architecture
    statistical distributions
    Computer vision
    learning
    education
    imagery

    Keywords

    • remote sensing
    • deep learning
    • scene classification
    • local binary patterns
    • texture analysis

    Cite this

    @article{a62ae0a58f454259ac85c32d7f52ab77,
    title = "Binary patterns encoded convolutional neural networks for texture recognition and remote sensing scene classification",
    abstract = "Designing discriminative powerful texture features robust to realistic imaging conditions is a challenging computer vision problem with many applications, including material recognition and analysis of satellite or aerial imagery. In the past, most texture description approaches were based on dense orderless statistical distribution of local features. However, most recent approaches to texture recognition and remote sensing scene classification are based on Convolutional Neural Networks (CNNs). The de facto practice when learning these CNN models is to use RGB patches as input with training performed on large amounts of labeled data (ImageNet). In this paper, we show that Local Binary Patterns (LBP) encoded CNN models, codenamed TEX-Nets, trained using mapped coded images with explicit LBP based texture information provide complementary information to the standard RGB deep models. Additionally, two deep architectures, namely early and late fusion, are investigated to combine the texture and color information. To the best of our knowledge, we are the first to investigate Binary Patterns encoded CNNs and different deep network fusion architectures for texture recognition and remote sensing scene classification. We perform comprehensive experiments on four texture recognition datasets and four remote sensing scene classification benchmarks: UC-Merced with 21 scene categories, WHU-RS19 with 19 scene classes, RSSCN7 with 7 categories and the recently introduced large scale aerial image dataset (AID) with 30 aerial scene types. We demonstrate that TEX-Nets provide complementary information to standard RGB deep model of the same network architecture. Our late fusion TEX-Net architecture always improves the overall performance compared to the standard RGB network on both recognition problems. Furthermore, our final combination leads to consistent improvement over the state-of-the-art for remote sensing scene classification.",
    keywords = "remote sensing, deep learning, scene classification, local binary patterns, texture analysis",
    author = "Anwer, {Rao Muhammad} and Khan, {Fahad Shahbaz} and {van de Weijer}, Joost and Matthieu Molinier and Jorma Laaksonen",
    note = "Project: 112403",
    year = "2018",
    month = "4",
    day = "1",
    doi = "10.1016/j.isprsjprs.2018.01.023",
    language = "English",
    volume = "138",
    pages = "74–85",
    journal = "ISPRS Journal of Photogrammetry and Remote Sensing",
    issn = "0924-2716",
    publisher = "Elsevier",

    }

    Binary patterns encoded convolutional neural networks for texture recognition and remote sensing scene classification. / Anwer, Rao Muhammad; Khan, Fahad Shahbaz; van de Weijer, Joost; Molinier, Matthieu; Laaksonen, Jorma.

    In: ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 138, 01.04.2018, p. 74–85.

    Research output: Contribution to journalArticleScientificpeer-review

    TY - JOUR

    T1 - Binary patterns encoded convolutional neural networks for texture recognition and remote sensing scene classification

    AU - Anwer, Rao Muhammad

    AU - Khan, Fahad Shahbaz

    AU - van de Weijer, Joost

    AU - Molinier, Matthieu

    AU - Laaksonen, Jorma

    N1 - Project: 112403

    PY - 2018/4/1

    Y1 - 2018/4/1

    N2 - Designing discriminative powerful texture features robust to realistic imaging conditions is a challenging computer vision problem with many applications, including material recognition and analysis of satellite or aerial imagery. In the past, most texture description approaches were based on dense orderless statistical distribution of local features. However, most recent approaches to texture recognition and remote sensing scene classification are based on Convolutional Neural Networks (CNNs). The de facto practice when learning these CNN models is to use RGB patches as input with training performed on large amounts of labeled data (ImageNet). In this paper, we show that Local Binary Patterns (LBP) encoded CNN models, codenamed TEX-Nets, trained using mapped coded images with explicit LBP based texture information provide complementary information to the standard RGB deep models. Additionally, two deep architectures, namely early and late fusion, are investigated to combine the texture and color information. To the best of our knowledge, we are the first to investigate Binary Patterns encoded CNNs and different deep network fusion architectures for texture recognition and remote sensing scene classification. We perform comprehensive experiments on four texture recognition datasets and four remote sensing scene classification benchmarks: UC-Merced with 21 scene categories, WHU-RS19 with 19 scene classes, RSSCN7 with 7 categories and the recently introduced large scale aerial image dataset (AID) with 30 aerial scene types. We demonstrate that TEX-Nets provide complementary information to standard RGB deep model of the same network architecture. Our late fusion TEX-Net architecture always improves the overall performance compared to the standard RGB network on both recognition problems. Furthermore, our final combination leads to consistent improvement over the state-of-the-art for remote sensing scene classification.

    AB - Designing discriminative powerful texture features robust to realistic imaging conditions is a challenging computer vision problem with many applications, including material recognition and analysis of satellite or aerial imagery. In the past, most texture description approaches were based on dense orderless statistical distribution of local features. However, most recent approaches to texture recognition and remote sensing scene classification are based on Convolutional Neural Networks (CNNs). The de facto practice when learning these CNN models is to use RGB patches as input with training performed on large amounts of labeled data (ImageNet). In this paper, we show that Local Binary Patterns (LBP) encoded CNN models, codenamed TEX-Nets, trained using mapped coded images with explicit LBP based texture information provide complementary information to the standard RGB deep models. Additionally, two deep architectures, namely early and late fusion, are investigated to combine the texture and color information. To the best of our knowledge, we are the first to investigate Binary Patterns encoded CNNs and different deep network fusion architectures for texture recognition and remote sensing scene classification. We perform comprehensive experiments on four texture recognition datasets and four remote sensing scene classification benchmarks: UC-Merced with 21 scene categories, WHU-RS19 with 19 scene classes, RSSCN7 with 7 categories and the recently introduced large scale aerial image dataset (AID) with 30 aerial scene types. We demonstrate that TEX-Nets provide complementary information to standard RGB deep model of the same network architecture. Our late fusion TEX-Net architecture always improves the overall performance compared to the standard RGB network on both recognition problems. Furthermore, our final combination leads to consistent improvement over the state-of-the-art for remote sensing scene classification.

    KW - remote sensing

    KW - deep learning

    KW - scene classification

    KW - local binary patterns

    KW - texture analysis

    UR - http://www.scopus.com/inward/record.url?scp=85042190618&partnerID=8YFLogxK

    U2 - 10.1016/j.isprsjprs.2018.01.023

    DO - 10.1016/j.isprsjprs.2018.01.023

    M3 - Article

    VL - 138

    SP - 74

    EP - 85

    JO - ISPRS Journal of Photogrammetry and Remote Sensing

    JF - ISPRS Journal of Photogrammetry and Remote Sensing

    SN - 0924-2716

    ER -