Greenly offloading traffic in stochastic heterogeneous cellular networks

This paper puts forwards an on-line reinforcement learning framework for the problem of traffic offloading in a stochastic Markovian heterogeneous cellular network (HCN), where the time-varying traffic demand of mobile terminals (MTs) can be offloaded from macrocells to small-cells. Our aim is to minimize the average energy consumption of the HCN while maintaining the Quality-of-Service (QoS) experienced by MTs. For each cell (i.e., a macrocell or a small-cell), the energy consumption is determined by its system load which is coupled with the system loads served in other cells due to the sharing over a common frequency band. We model the energy-aware traffic offloading in such HCNs as a constrained Markov decision process (C-MDP). The statistics of the C-MDP depends on a selected traffic offloading strategy and thus, the actions performed by a network controller have a long-term impact on the network state evolution. Based on the traffic demand observations and the traffic offloading operations, the controller gradually optimizes the strategy with no prior knowledge of the process statistics. Numerical experiments are conducted to show the effectiveness of the proposed learning framework in balancing the tradeoff between energy saving and QoS satisfaction.