Decentralized Deep Reinforcement Learning for Delay-Power Tradeoff in Vehicular Communications

Xianfu Chen, Celimuge Wu, Honggang Zhang, Yan Zhang, Mehdi Bennis, Heli Vuojala

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

Abstract

This paper targets at the problem of radio resource management for expected long-term delay-power tradeoff in vehicular communications. At each decision epoch, the road side unit observes the global network state, allocates channels and schedules data packets for all vehicle user equipment-pairs (VUE-pairs). The decision-making procedure is modelled as a discrete-time Markov decision process (MDP). The technical challenges in solving an optimal control policy originate from highly spatial mobility of vehicles and temporal variations in data traffic. To simplify the decision-making process, we first decompose the MDP into a series of per-VUE-pair MDPs. We then propose an online long short-term memory based deep reinforcement learning algorithm to break the curse of high dimensionality in state space faced by each per-VUE-pair MDP. With the proposed algorithm, the optimal channel allocation and packet scheduling decision at each epoch can be made in a decentralized way in accordance with the partial observations of the global network state at the VUE-pairs. Numerical simulations validate the theoretical analysis and show the effectiveness of the proposed online learning algorithm.

Original languageEnglish
Title of host publication2019 IEEE International Conference on Communications, ICC 2019
PublisherInstitute of Electrical and Electronic Engineers IEEE
Number of pages6
ISBN (Electronic)978-1-5386-8088-9
ISBN (Print)978-1-5386-8089-6
DOIs
Publication statusPublished - 2019
MoE publication typeA4 Article in a conference publication
Event2019 IEEE International Conference on Communications, ICC 2019 - Shanghai, China
Duration: 20 May 201924 May 2019

Conference

Conference2019 IEEE International Conference on Communications, ICC 2019
CountryChina
CityShanghai
Period20/05/1924/05/19

Fingerprint

Reinforcement learning
Communication
Learning algorithms
Decision making
Scheduling
Computer simulation

Cite this

Chen, X., Wu, C., Zhang, H., Zhang, Y., Bennis, M., & Vuojala, H. (2019). Decentralized Deep Reinforcement Learning for Delay-Power Tradeoff in Vehicular Communications. In 2019 IEEE International Conference on Communications, ICC 2019 Institute of Electrical and Electronic Engineers IEEE. https://doi.org/10.1109/ICC.2019.8761949
Chen, Xianfu ; Wu, Celimuge ; Zhang, Honggang ; Zhang, Yan ; Bennis, Mehdi ; Vuojala, Heli. / Decentralized Deep Reinforcement Learning for Delay-Power Tradeoff in Vehicular Communications. 2019 IEEE International Conference on Communications, ICC 2019. Institute of Electrical and Electronic Engineers IEEE, 2019.
@inproceedings{467762fc03324803a555095f1fe52ce7,
title = "Decentralized Deep Reinforcement Learning for Delay-Power Tradeoff in Vehicular Communications",
abstract = "This paper targets at the problem of radio resource management for expected long-term delay-power tradeoff in vehicular communications. At each decision epoch, the road side unit observes the global network state, allocates channels and schedules data packets for all vehicle user equipment-pairs (VUE-pairs). The decision-making procedure is modelled as a discrete-time Markov decision process (MDP). The technical challenges in solving an optimal control policy originate from highly spatial mobility of vehicles and temporal variations in data traffic. To simplify the decision-making process, we first decompose the MDP into a series of per-VUE-pair MDPs. We then propose an online long short-term memory based deep reinforcement learning algorithm to break the curse of high dimensionality in state space faced by each per-VUE-pair MDP. With the proposed algorithm, the optimal channel allocation and packet scheduling decision at each epoch can be made in a decentralized way in accordance with the partial observations of the global network state at the VUE-pairs. Numerical simulations validate the theoretical analysis and show the effectiveness of the proposed online learning algorithm.",
author = "Xianfu Chen and Celimuge Wu and Honggang Zhang and Yan Zhang and Mehdi Bennis and Heli Vuojala",
year = "2019",
doi = "10.1109/ICC.2019.8761949",
language = "English",
isbn = "978-1-5386-8089-6",
booktitle = "2019 IEEE International Conference on Communications, ICC 2019",
publisher = "Institute of Electrical and Electronic Engineers IEEE",
address = "United States",

}

Chen, X, Wu, C, Zhang, H, Zhang, Y, Bennis, M & Vuojala, H 2019, Decentralized Deep Reinforcement Learning for Delay-Power Tradeoff in Vehicular Communications. in 2019 IEEE International Conference on Communications, ICC 2019. Institute of Electrical and Electronic Engineers IEEE, 2019 IEEE International Conference on Communications, ICC 2019, Shanghai, China, 20/05/19. https://doi.org/10.1109/ICC.2019.8761949

Decentralized Deep Reinforcement Learning for Delay-Power Tradeoff in Vehicular Communications. / Chen, Xianfu; Wu, Celimuge; Zhang, Honggang; Zhang, Yan; Bennis, Mehdi; Vuojala, Heli.

2019 IEEE International Conference on Communications, ICC 2019. Institute of Electrical and Electronic Engineers IEEE, 2019.

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

TY - GEN

T1 - Decentralized Deep Reinforcement Learning for Delay-Power Tradeoff in Vehicular Communications

AU - Chen, Xianfu

AU - Wu, Celimuge

AU - Zhang, Honggang

AU - Zhang, Yan

AU - Bennis, Mehdi

AU - Vuojala, Heli

PY - 2019

Y1 - 2019

N2 - This paper targets at the problem of radio resource management for expected long-term delay-power tradeoff in vehicular communications. At each decision epoch, the road side unit observes the global network state, allocates channels and schedules data packets for all vehicle user equipment-pairs (VUE-pairs). The decision-making procedure is modelled as a discrete-time Markov decision process (MDP). The technical challenges in solving an optimal control policy originate from highly spatial mobility of vehicles and temporal variations in data traffic. To simplify the decision-making process, we first decompose the MDP into a series of per-VUE-pair MDPs. We then propose an online long short-term memory based deep reinforcement learning algorithm to break the curse of high dimensionality in state space faced by each per-VUE-pair MDP. With the proposed algorithm, the optimal channel allocation and packet scheduling decision at each epoch can be made in a decentralized way in accordance with the partial observations of the global network state at the VUE-pairs. Numerical simulations validate the theoretical analysis and show the effectiveness of the proposed online learning algorithm.

AB - This paper targets at the problem of radio resource management for expected long-term delay-power tradeoff in vehicular communications. At each decision epoch, the road side unit observes the global network state, allocates channels and schedules data packets for all vehicle user equipment-pairs (VUE-pairs). The decision-making procedure is modelled as a discrete-time Markov decision process (MDP). The technical challenges in solving an optimal control policy originate from highly spatial mobility of vehicles and temporal variations in data traffic. To simplify the decision-making process, we first decompose the MDP into a series of per-VUE-pair MDPs. We then propose an online long short-term memory based deep reinforcement learning algorithm to break the curse of high dimensionality in state space faced by each per-VUE-pair MDP. With the proposed algorithm, the optimal channel allocation and packet scheduling decision at each epoch can be made in a decentralized way in accordance with the partial observations of the global network state at the VUE-pairs. Numerical simulations validate the theoretical analysis and show the effectiveness of the proposed online learning algorithm.

UR - http://www.scopus.com/inward/record.url?scp=85070188547&partnerID=8YFLogxK

U2 - 10.1109/ICC.2019.8761949

DO - 10.1109/ICC.2019.8761949

M3 - Conference article in proceedings

AN - SCOPUS:85070188547

SN - 978-1-5386-8089-6

BT - 2019 IEEE International Conference on Communications, ICC 2019

PB - Institute of Electrical and Electronic Engineers IEEE

ER -

Chen X, Wu C, Zhang H, Zhang Y, Bennis M, Vuojala H. Decentralized Deep Reinforcement Learning for Delay-Power Tradeoff in Vehicular Communications. In 2019 IEEE International Conference on Communications, ICC 2019. Institute of Electrical and Electronic Engineers IEEE. 2019 https://doi.org/10.1109/ICC.2019.8761949