Multi-Agent Meta-Reinforcement Learning for Optimized Task Scheduling in Heterogeneous Edge Computing Systems

Mobile edge computing (MEC) brings the potential to address the ever increasing computation demands from the mobile users (MUs). In addition to local processing, the resource-constrained MUs in an MEC system can also offload computation to the nearby servers for remote execution. With the explosive growth of mobile devices, computation offloading faces the challenge of spectrum congestion, which in turn deteriorates the overall quality of computation experience. This paper hence investigates computation task scheduling in a heterogeneous cellular and WiFi MEC system. Such a system provides both licensed and unlicensed spectrum opportunities. Due to the sharing of communication and computation resources as well as the uncertainties, we formulate the problem of computation task scheduling among the competing MUs in a stationary heterogeneous edge computing system as a non-cooperative stochastic game. We propose an approximation-based multi-agent Markov decision process without the global system state observations, under which a multi-agent proximal policy optimization (PPO) algorithm is derived to solve the corresponding Nash equilibrium. When expanding to a non-stationary heterogeneous edge computing system, the obtained algorithm suffers from the slow convergence due to constrained adaptability. Accordingly, we explore meta-learning and propose a multi-agent meta-PPO algorithm, which rapidly adapts the control policy learning to the non-stationarity. Numerical experiments demonstrate performance gains from our proposed algorithms.