Policy-based and QoE-aware content delivery using Q-learning method

Olli Mämmelä, Faqir Zarrar Yousaf, Petteri Mannersalo, Johannes Lessmann

Research output: Contribution to journalArticleScientificpeer-review

2 Citations (Scopus)

Abstract

With the increasing popularity of various video services, video content is becoming a dominant traffic type in mobile networks. This poses a serious challenge to mobile network operators as well as service providers when it comes to delivering video content in a controllable and resource-efficient way to multiple users. Meeting various quality of experience and quality of service requirements is a difficult task especially in a wireless environment where several different priority based user classes can be included. This paper proposes an intelligent and context-aware application level fair scheduler, which is based on reinforcement learning and which can dynamically adjust relevant scheduling parameters in reaction to specific events or context information. The implemented Q-learning method is analyzed with reference to the delivery of progressive video streaming services employed by the likes of YouTube, Daily Motion, etc. In this regard, we study the performance observed by the end users in a scenario where the backhaul link in a mobile network infrastructure may become congested. Using the application level scheduler to intelligently orchestrate between multiple concurrent flows will minimize the number of buffer starvation events and thus enable smooth playback in cases where a pure TCP based delivery would fail. We also demonstrate the effectiveness of the Q-learning based scheduler to provide service separation between the user classes and fairness within a user class.
Original languageEnglish
Pages (from-to)315-342
JournalWireless Personal Communications
Volume83
Issue number1
DOIs
Publication statusPublished - 2015
MoE publication typeA1 Journal article-refereed

Fingerprint

Wireless networks
Video streaming
Reinforcement learning
Quality of service
Scheduling

Keywords

  • reinforcement learning
  • q-learning
  • qoE-aware content delivery
  • application level scheduling
  • video content delivery

Cite this

Mämmelä, Olli ; Yousaf, Faqir Zarrar ; Mannersalo, Petteri ; Lessmann, Johannes. / Policy-based and QoE-aware content delivery using Q-learning method. In: Wireless Personal Communications. 2015 ; Vol. 83, No. 1. pp. 315-342.
@article{23baee60b34d44bbadb2ecfe8ab9bbc1,
title = "Policy-based and QoE-aware content delivery using Q-learning method",
abstract = "With the increasing popularity of various video services, video content is becoming a dominant traffic type in mobile networks. This poses a serious challenge to mobile network operators as well as service providers when it comes to delivering video content in a controllable and resource-efficient way to multiple users. Meeting various quality of experience and quality of service requirements is a difficult task especially in a wireless environment where several different priority based user classes can be included. This paper proposes an intelligent and context-aware application level fair scheduler, which is based on reinforcement learning and which can dynamically adjust relevant scheduling parameters in reaction to specific events or context information. The implemented Q-learning method is analyzed with reference to the delivery of progressive video streaming services employed by the likes of YouTube, Daily Motion, etc. In this regard, we study the performance observed by the end users in a scenario where the backhaul link in a mobile network infrastructure may become congested. Using the application level scheduler to intelligently orchestrate between multiple concurrent flows will minimize the number of buffer starvation events and thus enable smooth playback in cases where a pure TCP based delivery would fail. We also demonstrate the effectiveness of the Q-learning based scheduler to provide service separation between the user classes and fairness within a user class.",
keywords = "reinforcement learning, q-learning, qoE-aware content delivery, application level scheduling, video content delivery",
author = "Olli M{\"a}mmel{\"a} and Yousaf, {Faqir Zarrar} and Petteri Mannersalo and Johannes Lessmann",
note = "Project code: 104139",
year = "2015",
doi = "10.1007/s11277-015-2395-1",
language = "English",
volume = "83",
pages = "315--342",
journal = "Wireless Personal Communications",
issn = "0929-6212",
publisher = "Springer",
number = "1",

}

Mämmelä, O, Yousaf, FZ, Mannersalo, P & Lessmann, J 2015, 'Policy-based and QoE-aware content delivery using Q-learning method', Wireless Personal Communications, vol. 83, no. 1, pp. 315-342. https://doi.org/10.1007/s11277-015-2395-1

Policy-based and QoE-aware content delivery using Q-learning method. / Mämmelä, Olli; Yousaf, Faqir Zarrar; Mannersalo, Petteri; Lessmann, Johannes.

In: Wireless Personal Communications, Vol. 83, No. 1, 2015, p. 315-342.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Policy-based and QoE-aware content delivery using Q-learning method

AU - Mämmelä, Olli

AU - Yousaf, Faqir Zarrar

AU - Mannersalo, Petteri

AU - Lessmann, Johannes

N1 - Project code: 104139

PY - 2015

Y1 - 2015

N2 - With the increasing popularity of various video services, video content is becoming a dominant traffic type in mobile networks. This poses a serious challenge to mobile network operators as well as service providers when it comes to delivering video content in a controllable and resource-efficient way to multiple users. Meeting various quality of experience and quality of service requirements is a difficult task especially in a wireless environment where several different priority based user classes can be included. This paper proposes an intelligent and context-aware application level fair scheduler, which is based on reinforcement learning and which can dynamically adjust relevant scheduling parameters in reaction to specific events or context information. The implemented Q-learning method is analyzed with reference to the delivery of progressive video streaming services employed by the likes of YouTube, Daily Motion, etc. In this regard, we study the performance observed by the end users in a scenario where the backhaul link in a mobile network infrastructure may become congested. Using the application level scheduler to intelligently orchestrate between multiple concurrent flows will minimize the number of buffer starvation events and thus enable smooth playback in cases where a pure TCP based delivery would fail. We also demonstrate the effectiveness of the Q-learning based scheduler to provide service separation between the user classes and fairness within a user class.

AB - With the increasing popularity of various video services, video content is becoming a dominant traffic type in mobile networks. This poses a serious challenge to mobile network operators as well as service providers when it comes to delivering video content in a controllable and resource-efficient way to multiple users. Meeting various quality of experience and quality of service requirements is a difficult task especially in a wireless environment where several different priority based user classes can be included. This paper proposes an intelligent and context-aware application level fair scheduler, which is based on reinforcement learning and which can dynamically adjust relevant scheduling parameters in reaction to specific events or context information. The implemented Q-learning method is analyzed with reference to the delivery of progressive video streaming services employed by the likes of YouTube, Daily Motion, etc. In this regard, we study the performance observed by the end users in a scenario where the backhaul link in a mobile network infrastructure may become congested. Using the application level scheduler to intelligently orchestrate between multiple concurrent flows will minimize the number of buffer starvation events and thus enable smooth playback in cases where a pure TCP based delivery would fail. We also demonstrate the effectiveness of the Q-learning based scheduler to provide service separation between the user classes and fairness within a user class.

KW - reinforcement learning

KW - q-learning

KW - qoE-aware content delivery

KW - application level scheduling

KW - video content delivery

U2 - 10.1007/s11277-015-2395-1

DO - 10.1007/s11277-015-2395-1

M3 - Article

VL - 83

SP - 315

EP - 342

JO - Wireless Personal Communications

JF - Wireless Personal Communications

SN - 0929-6212

IS - 1

ER -