Power entangling and matching in cognitive wireless mesh networks by applying conjecture based multi-agent QQ-learning approach

As the scarce spectrum resource is becoming over-crowded, cognitive wireless mesh networks express great flexibility to improve the spectrum utilization by opportunistically accessing the authorized frequency bands. One of the critical challenges for realizing such networks is how to adaptively match transmit powers and allocate frequency resources among secondary users (SUs) of the licensed frequency bands whilst maintaining the Quality-of-Service (QoS) requirement of the primary users (PUs), even in mutually entangled interference environment. In this paper, we discuss the non-cooperative power allocation matching problem in cognitive wireless mesh networks formed by a number of clusters with the consideration of energy efficiency. Due to the secondary users' selfish and spontaneous features, the problem is modeled as a stochastic learning process. We extend the conventional single-agent Q-learning to a multi-user context, coined as QQ-learning, using the framework of stochastic games. Within the multi-agent QQ-learning processes, a learning SU performs Q-function updates based on the conjecture about the other SUs' behaviors. This learning algorithm provably converges given certain restrictions that arise during learning procedure. Numerical experiments are used to verify the performance of our algorithm and demonstrate its effectiveness of improving the energy efficiency.