GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicing

Yuxiu Hua, Rongpeng Li, Zhifeng Zhao, Xianfu Chen, Honggang Zhang

    Research output: Contribution to journalArticleScientificpeer-review

    Abstract

    Network slicing is a key technology in 5G communications system. Its purpose is to dynamically and efficiently allocate resources for diversified services with distinct requirements over a common underlying physical infrastructure. Therein, demand-aware resource allocation is of significant importance to network slicing. In this paper, we consider a scenario that contains several slices in a radio access network with base stations that share the same physical resources (e.g., bandwidth or slots). We leverage deep reinforcement learning (DRL) to solve this problem by considering the varying service demands as the environment state and the allocated resources as the environment action. In order to reduce the effects of the annoying randomness and noise embedded in the received service level agreement (SLA) satisfaction ratio (SSR) and spectrum efficiency (SE), we primarily propose generative adversarial network-powered deep distributional Q network (GAN-DDQN) to learn the action-value distribution driven by minimizing the discrepancy between the estimated action-value distribution and the target action-value distribution. We put forward a reward-clipping mechanism to stabilize GAN-DDQN training against the effects of widely-spanning utility values. Moreover, we further develop Dueling GAN-DDQN, which uses a specially designed dueling generator, to learn the action-value distribution by estimating the state-value distribution and the action advantage function. Finally, we verify the performance of the proposed GAN-DDQN and Dueling GAN-DDQN algorithms through extensive simulations.

    Original languageEnglish
    JournalIEEE Journal on Selected Areas in Communications
    DOIs
    Publication statusAccepted/In press - 2019
    MoE publication typeA1 Journal article-refereed

    Fingerprint

    Reinforcement learning
    Base stations
    Resource allocation
    Communication systems
    Bandwidth

    Keywords

    • 5G
    • deep reinforcement learning
    • distributional reinforcement learning
    • GAN
    • generative adversarial network
    • network slicing

    Cite this

    @article{a627bd42f2654462b74524ff120b7268,
    title = "GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicing",
    abstract = "Network slicing is a key technology in 5G communications system. Its purpose is to dynamically and efficiently allocate resources for diversified services with distinct requirements over a common underlying physical infrastructure. Therein, demand-aware resource allocation is of significant importance to network slicing. In this paper, we consider a scenario that contains several slices in a radio access network with base stations that share the same physical resources (e.g., bandwidth or slots). We leverage deep reinforcement learning (DRL) to solve this problem by considering the varying service demands as the environment state and the allocated resources as the environment action. In order to reduce the effects of the annoying randomness and noise embedded in the received service level agreement (SLA) satisfaction ratio (SSR) and spectrum efficiency (SE), we primarily propose generative adversarial network-powered deep distributional Q network (GAN-DDQN) to learn the action-value distribution driven by minimizing the discrepancy between the estimated action-value distribution and the target action-value distribution. We put forward a reward-clipping mechanism to stabilize GAN-DDQN training against the effects of widely-spanning utility values. Moreover, we further develop Dueling GAN-DDQN, which uses a specially designed dueling generator, to learn the action-value distribution by estimating the state-value distribution and the action advantage function. Finally, we verify the performance of the proposed GAN-DDQN and Dueling GAN-DDQN algorithms through extensive simulations.",
    keywords = "5G, deep reinforcement learning, distributional reinforcement learning, GAN, generative adversarial network, network slicing",
    author = "Yuxiu Hua and Rongpeng Li and Zhifeng Zhao and Xianfu Chen and Honggang Zhang",
    year = "2019",
    doi = "10.1109/JSAC.2019.2959185",
    language = "English",
    journal = "IEEE Journal on Selected Areas in Communications",
    issn = "0733-8716",
    publisher = "IEEE Institute of Electrical and Electronic Engineers",

    }

    GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicing. / Hua, Yuxiu; Li, Rongpeng; Zhao, Zhifeng; Chen, Xianfu; Zhang, Honggang.

    In: IEEE Journal on Selected Areas in Communications, 2019.

    Research output: Contribution to journalArticleScientificpeer-review

    TY - JOUR

    T1 - GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicing

    AU - Hua, Yuxiu

    AU - Li, Rongpeng

    AU - Zhao, Zhifeng

    AU - Chen, Xianfu

    AU - Zhang, Honggang

    PY - 2019

    Y1 - 2019

    N2 - Network slicing is a key technology in 5G communications system. Its purpose is to dynamically and efficiently allocate resources for diversified services with distinct requirements over a common underlying physical infrastructure. Therein, demand-aware resource allocation is of significant importance to network slicing. In this paper, we consider a scenario that contains several slices in a radio access network with base stations that share the same physical resources (e.g., bandwidth or slots). We leverage deep reinforcement learning (DRL) to solve this problem by considering the varying service demands as the environment state and the allocated resources as the environment action. In order to reduce the effects of the annoying randomness and noise embedded in the received service level agreement (SLA) satisfaction ratio (SSR) and spectrum efficiency (SE), we primarily propose generative adversarial network-powered deep distributional Q network (GAN-DDQN) to learn the action-value distribution driven by minimizing the discrepancy between the estimated action-value distribution and the target action-value distribution. We put forward a reward-clipping mechanism to stabilize GAN-DDQN training against the effects of widely-spanning utility values. Moreover, we further develop Dueling GAN-DDQN, which uses a specially designed dueling generator, to learn the action-value distribution by estimating the state-value distribution and the action advantage function. Finally, we verify the performance of the proposed GAN-DDQN and Dueling GAN-DDQN algorithms through extensive simulations.

    AB - Network slicing is a key technology in 5G communications system. Its purpose is to dynamically and efficiently allocate resources for diversified services with distinct requirements over a common underlying physical infrastructure. Therein, demand-aware resource allocation is of significant importance to network slicing. In this paper, we consider a scenario that contains several slices in a radio access network with base stations that share the same physical resources (e.g., bandwidth or slots). We leverage deep reinforcement learning (DRL) to solve this problem by considering the varying service demands as the environment state and the allocated resources as the environment action. In order to reduce the effects of the annoying randomness and noise embedded in the received service level agreement (SLA) satisfaction ratio (SSR) and spectrum efficiency (SE), we primarily propose generative adversarial network-powered deep distributional Q network (GAN-DDQN) to learn the action-value distribution driven by minimizing the discrepancy between the estimated action-value distribution and the target action-value distribution. We put forward a reward-clipping mechanism to stabilize GAN-DDQN training against the effects of widely-spanning utility values. Moreover, we further develop Dueling GAN-DDQN, which uses a specially designed dueling generator, to learn the action-value distribution by estimating the state-value distribution and the action advantage function. Finally, we verify the performance of the proposed GAN-DDQN and Dueling GAN-DDQN algorithms through extensive simulations.

    KW - 5G

    KW - deep reinforcement learning

    KW - distributional reinforcement learning

    KW - GAN

    KW - generative adversarial network

    KW - network slicing

    UR - http://www.scopus.com/inward/record.url?scp=85076390888&partnerID=8YFLogxK

    U2 - 10.1109/JSAC.2019.2959185

    DO - 10.1109/JSAC.2019.2959185

    M3 - Article

    AN - SCOPUS:85076390888

    JO - IEEE Journal on Selected Areas in Communications

    JF - IEEE Journal on Selected Areas in Communications

    SN - 0733-8716

    ER -