Stochastic power adaptation with multi-agent reinforcement learning for cognitive wireless mesh networks

Xianfu Chen, Z. Zhao, H. Zhang

Research output: Contribution to journalArticleScientificpeer-review

38 Citations (Scopus)

Abstract

As the scarce spectrum resource is becoming overcrowded, cognitive radio indicates great flexibility to improve the spectrum efficiency by opportunistically accessing the authorized frequency bands. One of the critical challenges for operating such radios in a network is how to efficiently allocate transmission powers and frequency resource among the secondary users (SUs) while satisfying the quality-of-service constraints of the primary users. In this paper, we focus on the noncooperative power allocation problem in cognitive wireless mesh networks formed by a number of clusters with the consideration of energy efficiency. Due to the SUs' dynamic and spontaneous properties, the problem is modeled as a stochastic learning process. We first extend the single-agent Q-learning to a multiuser context, and then propose a conjecture-based multiagent Q-learning algorithm to achieve the optimal transmission strategies with only private and incomplete information. An intelligent SU performs Q-function updates based on the conjecture over the other SUs' stochastic behaviors. This learning algorithm provably converges given certain restrictions that arise during the learning procedure. Simulation experiments are used to verify the performance of our algorithm and demonstrate its effectiveness of improving the energy efficiency.
Original languageEnglish
Pages (from-to)2155-2166
Number of pages12
JournalIEEE Transactions on Mobile Computing
Volume12
Issue number11
DOIs
Publication statusPublished - 2013
MoE publication typeA1 Journal article-refereed

Fingerprint

Wireless mesh networks (WMN)
Reinforcement learning
Learning algorithms
Energy efficiency
Radio receivers
Cognitive radio
Power transmission
Frequency bands
Quality of service
Experiments

Keywords

  • Communication/Networking and Information Technology
  • Computer Systems Organization
  • distributed networks
  • energy-aware systems
  • hardware
  • network architecture and design
  • power management

Cite this

@article{f22532d8420241788429d0c0a3233757,
title = "Stochastic power adaptation with multi-agent reinforcement learning for cognitive wireless mesh networks",
abstract = "As the scarce spectrum resource is becoming overcrowded, cognitive radio indicates great flexibility to improve the spectrum efficiency by opportunistically accessing the authorized frequency bands. One of the critical challenges for operating such radios in a network is how to efficiently allocate transmission powers and frequency resource among the secondary users (SUs) while satisfying the quality-of-service constraints of the primary users. In this paper, we focus on the noncooperative power allocation problem in cognitive wireless mesh networks formed by a number of clusters with the consideration of energy efficiency. Due to the SUs' dynamic and spontaneous properties, the problem is modeled as a stochastic learning process. We first extend the single-agent Q-learning to a multiuser context, and then propose a conjecture-based multiagent Q-learning algorithm to achieve the optimal transmission strategies with only private and incomplete information. An intelligent SU performs Q-function updates based on the conjecture over the other SUs' stochastic behaviors. This learning algorithm provably converges given certain restrictions that arise during the learning procedure. Simulation experiments are used to verify the performance of our algorithm and demonstrate its effectiveness of improving the energy efficiency.",
keywords = "Communication/Networking and Information Technology, Computer Systems Organization, distributed networks, energy-aware systems, hardware, network architecture and design, power management",
author = "Xianfu Chen and Z. Zhao and H. Zhang",
note = "Project code: 43135",
year = "2013",
doi = "10.1109/TMC.2012.178",
language = "English",
volume = "12",
pages = "2155--2166",
journal = "IEEE Transactions on Mobile Computing",
issn = "1536-1233",
publisher = "Institute of Electrical and Electronic Engineers IEEE",
number = "11",

}

Stochastic power adaptation with multi-agent reinforcement learning for cognitive wireless mesh networks. / Chen, Xianfu; Zhao, Z.; Zhang, H.

In: IEEE Transactions on Mobile Computing, Vol. 12, No. 11, 2013, p. 2155-2166.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Stochastic power adaptation with multi-agent reinforcement learning for cognitive wireless mesh networks

AU - Chen, Xianfu

AU - Zhao, Z.

AU - Zhang, H.

N1 - Project code: 43135

PY - 2013

Y1 - 2013

N2 - As the scarce spectrum resource is becoming overcrowded, cognitive radio indicates great flexibility to improve the spectrum efficiency by opportunistically accessing the authorized frequency bands. One of the critical challenges for operating such radios in a network is how to efficiently allocate transmission powers and frequency resource among the secondary users (SUs) while satisfying the quality-of-service constraints of the primary users. In this paper, we focus on the noncooperative power allocation problem in cognitive wireless mesh networks formed by a number of clusters with the consideration of energy efficiency. Due to the SUs' dynamic and spontaneous properties, the problem is modeled as a stochastic learning process. We first extend the single-agent Q-learning to a multiuser context, and then propose a conjecture-based multiagent Q-learning algorithm to achieve the optimal transmission strategies with only private and incomplete information. An intelligent SU performs Q-function updates based on the conjecture over the other SUs' stochastic behaviors. This learning algorithm provably converges given certain restrictions that arise during the learning procedure. Simulation experiments are used to verify the performance of our algorithm and demonstrate its effectiveness of improving the energy efficiency.

AB - As the scarce spectrum resource is becoming overcrowded, cognitive radio indicates great flexibility to improve the spectrum efficiency by opportunistically accessing the authorized frequency bands. One of the critical challenges for operating such radios in a network is how to efficiently allocate transmission powers and frequency resource among the secondary users (SUs) while satisfying the quality-of-service constraints of the primary users. In this paper, we focus on the noncooperative power allocation problem in cognitive wireless mesh networks formed by a number of clusters with the consideration of energy efficiency. Due to the SUs' dynamic and spontaneous properties, the problem is modeled as a stochastic learning process. We first extend the single-agent Q-learning to a multiuser context, and then propose a conjecture-based multiagent Q-learning algorithm to achieve the optimal transmission strategies with only private and incomplete information. An intelligent SU performs Q-function updates based on the conjecture over the other SUs' stochastic behaviors. This learning algorithm provably converges given certain restrictions that arise during the learning procedure. Simulation experiments are used to verify the performance of our algorithm and demonstrate its effectiveness of improving the energy efficiency.

KW - Communication/Networking and Information Technology

KW - Computer Systems Organization

KW - distributed networks

KW - energy-aware systems

KW - hardware

KW - network architecture and design

KW - power management

U2 - 10.1109/TMC.2012.178

DO - 10.1109/TMC.2012.178

M3 - Article

VL - 12

SP - 2155

EP - 2166

JO - IEEE Transactions on Mobile Computing

JF - IEEE Transactions on Mobile Computing

SN - 1536-1233

IS - 11

ER -