TACT: A Transfer Actor-Critic Learning Framework for Energy Saving in Cellular Radio Access Networks

R Li, Z Zhao, Xianfu Chen, J Palicot, H Zhang

    Research output: Contribution to journalArticleScientificpeer-review

    53 Citations (Scopus)

    Abstract

    Recent works have validated the possibility of improving energy efficiency in radio access networks (RANs), achieved by dynamically turning on/off some base stations (BSs). In this paper, we extend the research over BS switching operations, which should match up with traffic load variations. Instead of depending on the dynamic traffic loads which are still quite challenging to precisely forecast, we firstly formulate the traffic variations as a Markov decision process. Afterwards, in order to foresightedly minimize the energy consumption of RANs, we design a reinforcement learning framework based BS switching operation scheme. Furthermore, to speed up the ongoing learning process, a transfer actor-critic algorithm (TACT), which utilizes the transferred learning expertise in historical periods or neighboring regions, is proposed and provably converges. In the end, we evaluate our proposed scheme by extensive simulations under various practical configurations and show that the proposed TACT algorithm contributes to a performance jumpstart and demonstrates the feasibility of signi?cant energy ef?ciency improvement at the expense of tolerable delay performance
    Original languageEnglish
    Pages (from-to)2000-2011
    Number of pages11
    JournalIEEE Transactions on Wireless Communications
    Volume13
    Issue number4
    DOIs
    Publication statusPublished - 2014
    MoE publication typeA1 Journal article-refereed

    Fingerprint

    Energy Saving
    Base stations
    Energy conservation
    Traffic
    Traffic Dynamics
    Markov Decision Process
    Reinforcement learning
    Network Design
    Reinforcement Learning
    Expertise
    Learning Process
    Energy Efficiency
    Energy Consumption
    Energy efficiency
    Forecast
    Speedup
    Energy utilization
    Converge
    Minimise
    Configuration

    Keywords

    • Radio access networks
    • base stations
    • sleeping mode
    • green communications
    • energy saving
    • reinforcement learning
    • transfer learning
    • actor-critic algorithm

    Cite this

    @article{af77c0691a2a4c469aca7d6f5ca71e5d,
    title = "TACT: A Transfer Actor-Critic Learning Framework for Energy Saving in Cellular Radio Access Networks",
    abstract = "Recent works have validated the possibility of improving energy efficiency in radio access networks (RANs), achieved by dynamically turning on/off some base stations (BSs). In this paper, we extend the research over BS switching operations, which should match up with traffic load variations. Instead of depending on the dynamic traffic loads which are still quite challenging to precisely forecast, we firstly formulate the traffic variations as a Markov decision process. Afterwards, in order to foresightedly minimize the energy consumption of RANs, we design a reinforcement learning framework based BS switching operation scheme. Furthermore, to speed up the ongoing learning process, a transfer actor-critic algorithm (TACT), which utilizes the transferred learning expertise in historical periods or neighboring regions, is proposed and provably converges. In the end, we evaluate our proposed scheme by extensive simulations under various practical configurations and show that the proposed TACT algorithm contributes to a performance jumpstart and demonstrates the feasibility of signi?cant energy ef?ciency improvement at the expense of tolerable delay performance",
    keywords = "Radio access networks, base stations, sleeping mode, green communications, energy saving, reinforcement learning, transfer learning, actor-critic algorithm",
    author = "R Li and Z Zhao and Xianfu Chen and J Palicot and H Zhang",
    note = "Project code: 82164",
    year = "2014",
    doi = "10.1109/TWC.2014.022014.130840",
    language = "English",
    volume = "13",
    pages = "2000--2011",
    journal = "IEEE Transactions on Wireless Communications",
    issn = "1536-1276",
    publisher = "IEEE Institute of Electrical and Electronic Engineers",
    number = "4",

    }

    TACT: A Transfer Actor-Critic Learning Framework for Energy Saving in Cellular Radio Access Networks. / Li, R; Zhao, Z; Chen, Xianfu; Palicot, J; Zhang, H.

    In: IEEE Transactions on Wireless Communications, Vol. 13, No. 4, 2014, p. 2000-2011.

    Research output: Contribution to journalArticleScientificpeer-review

    TY - JOUR

    T1 - TACT: A Transfer Actor-Critic Learning Framework for Energy Saving in Cellular Radio Access Networks

    AU - Li, R

    AU - Zhao, Z

    AU - Chen, Xianfu

    AU - Palicot, J

    AU - Zhang, H

    N1 - Project code: 82164

    PY - 2014

    Y1 - 2014

    N2 - Recent works have validated the possibility of improving energy efficiency in radio access networks (RANs), achieved by dynamically turning on/off some base stations (BSs). In this paper, we extend the research over BS switching operations, which should match up with traffic load variations. Instead of depending on the dynamic traffic loads which are still quite challenging to precisely forecast, we firstly formulate the traffic variations as a Markov decision process. Afterwards, in order to foresightedly minimize the energy consumption of RANs, we design a reinforcement learning framework based BS switching operation scheme. Furthermore, to speed up the ongoing learning process, a transfer actor-critic algorithm (TACT), which utilizes the transferred learning expertise in historical periods or neighboring regions, is proposed and provably converges. In the end, we evaluate our proposed scheme by extensive simulations under various practical configurations and show that the proposed TACT algorithm contributes to a performance jumpstart and demonstrates the feasibility of signi?cant energy ef?ciency improvement at the expense of tolerable delay performance

    AB - Recent works have validated the possibility of improving energy efficiency in radio access networks (RANs), achieved by dynamically turning on/off some base stations (BSs). In this paper, we extend the research over BS switching operations, which should match up with traffic load variations. Instead of depending on the dynamic traffic loads which are still quite challenging to precisely forecast, we firstly formulate the traffic variations as a Markov decision process. Afterwards, in order to foresightedly minimize the energy consumption of RANs, we design a reinforcement learning framework based BS switching operation scheme. Furthermore, to speed up the ongoing learning process, a transfer actor-critic algorithm (TACT), which utilizes the transferred learning expertise in historical periods or neighboring regions, is proposed and provably converges. In the end, we evaluate our proposed scheme by extensive simulations under various practical configurations and show that the proposed TACT algorithm contributes to a performance jumpstart and demonstrates the feasibility of signi?cant energy ef?ciency improvement at the expense of tolerable delay performance

    KW - Radio access networks

    KW - base stations

    KW - sleeping mode

    KW - green communications

    KW - energy saving

    KW - reinforcement learning

    KW - transfer learning

    KW - actor-critic algorithm

    U2 - 10.1109/TWC.2014.022014.130840

    DO - 10.1109/TWC.2014.022014.130840

    M3 - Article

    VL - 13

    SP - 2000

    EP - 2011

    JO - IEEE Transactions on Wireless Communications

    JF - IEEE Transactions on Wireless Communications

    SN - 1536-1276

    IS - 4

    ER -