Applying multi-agent Q-learning scheme in cognitive wireless mesh networks for green communications

Cognitive wireless mesh networks have great potential to green communication. One of the critical challenges for realizing such networks is how to adaptively match transmit powers and allocate frequency resources among secondary users of the licensed frequency bands whilst maintaining the quality-of-service (QoS) requirement of the primary users, even in mutually entangled interferences environment. In this paper, we discuss the power assignment matching problem in cognitive wireless mesh networks formed by a number of clusters with the consideration of the Signal-to-Interference-plus-Noise Ratio (SINR) requirement of each secondary user (SU), the mean-squared error (MSE) constraint by the primary users, and the energy efficiency. Due to the secondary users' selfish and spontaneous features, the problem is modeled as a stochastic non-cooperative game. We extend the conventional single-agent Q-learning to a non-cooperative multi-agent learning context, using the framework of stochastic non-cooperative games. Within the multi-agent Q-learning processes, a learning SU maintains Q-functions over joint actions set, and performs updating based on the conjecture about the other SUs' behaviors over the current Q-values. Numerical experiments on a hybrid CogMesh consisting two SUs and one specific PU suggest validity and efficiency of the proposed algorithm.