Stochastic power adaptation with multi-agent reinforcement learning for cognitive wireless mesh networks

Xianfu Chen, Z. Zhao, H. Zhang

    Research output: Contribution to journalArticleScientificpeer-review

    39 Citations (Scopus)

    Abstract

    As the scarce spectrum resource is becoming overcrowded, cognitive radio indicates great flexibility to improve the spectrum efficiency by opportunistically accessing the authorized frequency bands. One of the critical challenges for operating such radios in a network is how to efficiently allocate transmission powers and frequency resource among the secondary users (SUs) while satisfying the quality-of-service constraints of the primary users. In this paper, we focus on the noncooperative power allocation problem in cognitive wireless mesh networks formed by a number of clusters with the consideration of energy efficiency. Due to the SUs' dynamic and spontaneous properties, the problem is modeled as a stochastic learning process. We first extend the single-agent Q-learning to a multiuser context, and then propose a conjecture-based multiagent Q-learning algorithm to achieve the optimal transmission strategies with only private and incomplete information. An intelligent SU performs Q-function updates based on the conjecture over the other SUs' stochastic behaviors. This learning algorithm provably converges given certain restrictions that arise during the learning procedure. Simulation experiments are used to verify the performance of our algorithm and demonstrate its effectiveness of improving the energy efficiency.
    Original languageEnglish
    Pages (from-to)2155-2166
    Number of pages12
    JournalIEEE Transactions on Mobile Computing
    Volume12
    Issue number11
    DOIs
    Publication statusPublished - 2013
    MoE publication typeA1 Journal article-refereed

    Fingerprint

    Wireless mesh networks (WMN)
    Reinforcement learning
    Learning algorithms
    Energy efficiency
    Radio receivers
    Cognitive radio
    Power transmission
    Frequency bands
    Quality of service
    Experiments

    Keywords

    • Communication/Networking and Information Technology
    • Computer Systems Organization
    • distributed networks
    • energy-aware systems
    • hardware
    • network architecture and design
    • power management

    Cite this

    @article{f22532d8420241788429d0c0a3233757,
    title = "Stochastic power adaptation with multi-agent reinforcement learning for cognitive wireless mesh networks",
    abstract = "As the scarce spectrum resource is becoming overcrowded, cognitive radio indicates great flexibility to improve the spectrum efficiency by opportunistically accessing the authorized frequency bands. One of the critical challenges for operating such radios in a network is how to efficiently allocate transmission powers and frequency resource among the secondary users (SUs) while satisfying the quality-of-service constraints of the primary users. In this paper, we focus on the noncooperative power allocation problem in cognitive wireless mesh networks formed by a number of clusters with the consideration of energy efficiency. Due to the SUs' dynamic and spontaneous properties, the problem is modeled as a stochastic learning process. We first extend the single-agent Q-learning to a multiuser context, and then propose a conjecture-based multiagent Q-learning algorithm to achieve the optimal transmission strategies with only private and incomplete information. An intelligent SU performs Q-function updates based on the conjecture over the other SUs' stochastic behaviors. This learning algorithm provably converges given certain restrictions that arise during the learning procedure. Simulation experiments are used to verify the performance of our algorithm and demonstrate its effectiveness of improving the energy efficiency.",
    keywords = "Communication/Networking and Information Technology, Computer Systems Organization, distributed networks, energy-aware systems, hardware, network architecture and design, power management",
    author = "Xianfu Chen and Z. Zhao and H. Zhang",
    note = "Project code: 43135",
    year = "2013",
    doi = "10.1109/TMC.2012.178",
    language = "English",
    volume = "12",
    pages = "2155--2166",
    journal = "IEEE Transactions on Mobile Computing",
    issn = "1536-1233",
    publisher = "IEEE Institute of Electrical and Electronic Engineers",
    number = "11",

    }

    Stochastic power adaptation with multi-agent reinforcement learning for cognitive wireless mesh networks. / Chen, Xianfu; Zhao, Z.; Zhang, H.

    In: IEEE Transactions on Mobile Computing, Vol. 12, No. 11, 2013, p. 2155-2166.

    Research output: Contribution to journalArticleScientificpeer-review

    TY - JOUR

    T1 - Stochastic power adaptation with multi-agent reinforcement learning for cognitive wireless mesh networks

    AU - Chen, Xianfu

    AU - Zhao, Z.

    AU - Zhang, H.

    N1 - Project code: 43135

    PY - 2013

    Y1 - 2013

    N2 - As the scarce spectrum resource is becoming overcrowded, cognitive radio indicates great flexibility to improve the spectrum efficiency by opportunistically accessing the authorized frequency bands. One of the critical challenges for operating such radios in a network is how to efficiently allocate transmission powers and frequency resource among the secondary users (SUs) while satisfying the quality-of-service constraints of the primary users. In this paper, we focus on the noncooperative power allocation problem in cognitive wireless mesh networks formed by a number of clusters with the consideration of energy efficiency. Due to the SUs' dynamic and spontaneous properties, the problem is modeled as a stochastic learning process. We first extend the single-agent Q-learning to a multiuser context, and then propose a conjecture-based multiagent Q-learning algorithm to achieve the optimal transmission strategies with only private and incomplete information. An intelligent SU performs Q-function updates based on the conjecture over the other SUs' stochastic behaviors. This learning algorithm provably converges given certain restrictions that arise during the learning procedure. Simulation experiments are used to verify the performance of our algorithm and demonstrate its effectiveness of improving the energy efficiency.

    AB - As the scarce spectrum resource is becoming overcrowded, cognitive radio indicates great flexibility to improve the spectrum efficiency by opportunistically accessing the authorized frequency bands. One of the critical challenges for operating such radios in a network is how to efficiently allocate transmission powers and frequency resource among the secondary users (SUs) while satisfying the quality-of-service constraints of the primary users. In this paper, we focus on the noncooperative power allocation problem in cognitive wireless mesh networks formed by a number of clusters with the consideration of energy efficiency. Due to the SUs' dynamic and spontaneous properties, the problem is modeled as a stochastic learning process. We first extend the single-agent Q-learning to a multiuser context, and then propose a conjecture-based multiagent Q-learning algorithm to achieve the optimal transmission strategies with only private and incomplete information. An intelligent SU performs Q-function updates based on the conjecture over the other SUs' stochastic behaviors. This learning algorithm provably converges given certain restrictions that arise during the learning procedure. Simulation experiments are used to verify the performance of our algorithm and demonstrate its effectiveness of improving the energy efficiency.

    KW - Communication/Networking and Information Technology

    KW - Computer Systems Organization

    KW - distributed networks

    KW - energy-aware systems

    KW - hardware

    KW - network architecture and design

    KW - power management

    U2 - 10.1109/TMC.2012.178

    DO - 10.1109/TMC.2012.178

    M3 - Article

    VL - 12

    SP - 2155

    EP - 2166

    JO - IEEE Transactions on Mobile Computing

    JF - IEEE Transactions on Mobile Computing

    SN - 1536-1233

    IS - 11

    ER -