Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning

Xianfu Chen, Honggang Zhang, Celimuge Wu, Shiwen Mao, Yusheng Ji, Mehdi Bennis

    Research output: Contribution to journalArticleScientificpeer-review

    25 Citations (Scopus)

    Abstract

    To improve the quality of computation experience for mobile devices, mobile-edge computing (MEC) is a promising paradigm by providing computing capabilities in close proximity within a sliced radio access network (RAN), which supports both traditional communication and MEC services. Nevertheless, the design of computation offloading policies for a virtual MEC system remains challenging. Specifically, whether to execute a computation task at the mobile device or to offload it for MEC server execution should adapt to the time-varying network dynamics. This paper considers MEC for a representative mobile user in an ultradense sliced RAN, where multiple base stations (BSs) are available to be selected for computation offloading. The problem of solving an optimal computation offloading policy is modeled as a Markov decision process, where our objective is to maximize the long-term utility performance whereby an offloading decision is made based on the task queue state, the energy queue state as well as the channel qualities between mobile user and BSs. To break the curse of high dimensionality in state space, we first propose a double deep Q -network (DQN)-based strategic computation offloading algorithm to learn the optimal policy without knowing a priori knowledge of network dynamics. Then motivated by the additive structure of the utility function, a Q -function decomposition technique is combined with the double DQN, which leads to a novel learning algorithm for the solving of stochastic computation offloading. Numerical experiments show that our proposed learning algorithms achieve a significant improvement in computation offloading performance compared with the baseline policies.

    Original languageEnglish
    Article number8493155
    Pages (from-to)4005-4018
    Number of pages14
    JournalIEEE Internet of Things Journal
    Volume6
    Issue number3
    DOIs
    Publication statusPublished - 1 Jun 2019
    MoE publication typeNot Eligible

    Fingerprint

    Reinforcement learning
    Mobile devices
    Base stations
    Learning algorithms
    Time varying networks
    Electron energy levels
    Servers
    Decomposition
    Communication

    Keywords

    • deep reinforcement learning
    • Electronic mail
    • Heuristic algorithms
    • Markov decision process
    • Mobile handsets
    • mobile-edge computing
    • Network slicing
    • network virtualization
    • Q-function decomposition.
    • radio access networks
    • Servers
    • Stochastic processes
    • Task analysis
    • Wireless communication

    Cite this

    Chen, Xianfu ; Zhang, Honggang ; Wu, Celimuge ; Mao, Shiwen ; Ji, Yusheng ; Bennis, Mehdi. / Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning. In: IEEE Internet of Things Journal. 2019 ; Vol. 6, No. 3. pp. 4005-4018.
    @article{2cbd84cbeb9a405fb83ee2c95bf26f2f,
    title = "Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning",
    abstract = "To improve the quality of computation experience for mobile devices, mobile-edge computing (MEC) is a promising paradigm by providing computing capabilities in close proximity within a sliced radio access network (RAN), which supports both traditional communication and MEC services. Nevertheless, the design of computation offloading policies for a virtual MEC system remains challenging. Specifically, whether to execute a computation task at the mobile device or to offload it for MEC server execution should adapt to the time-varying network dynamics. This paper considers MEC for a representative mobile user in an ultradense sliced RAN, where multiple base stations (BSs) are available to be selected for computation offloading. The problem of solving an optimal computation offloading policy is modeled as a Markov decision process, where our objective is to maximize the long-term utility performance whereby an offloading decision is made based on the task queue state, the energy queue state as well as the channel qualities between mobile user and BSs. To break the curse of high dimensionality in state space, we first propose a double deep Q -network (DQN)-based strategic computation offloading algorithm to learn the optimal policy without knowing a priori knowledge of network dynamics. Then motivated by the additive structure of the utility function, a Q -function decomposition technique is combined with the double DQN, which leads to a novel learning algorithm for the solving of stochastic computation offloading. Numerical experiments show that our proposed learning algorithms achieve a significant improvement in computation offloading performance compared with the baseline policies.",
    keywords = "deep reinforcement learning, Electronic mail, Heuristic algorithms, Markov decision process, Mobile handsets, mobile-edge computing, Network slicing, network virtualization, Q-function decomposition., radio access networks, Servers, Stochastic processes, Task analysis, Wireless communication",
    author = "Xianfu Chen and Honggang Zhang and Celimuge Wu and Shiwen Mao and Yusheng Ji and Mehdi Bennis",
    year = "2019",
    month = "6",
    day = "1",
    doi = "10.1109/JIOT.2018.2876279",
    language = "English",
    volume = "6",
    pages = "4005--4018",
    journal = "IEEE Internet of Things Journal",
    issn = "2327-4662",
    publisher = "IEEE Institute of Electrical and Electronic Engineers",
    number = "3",

    }

    Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning. / Chen, Xianfu; Zhang, Honggang; Wu, Celimuge; Mao, Shiwen; Ji, Yusheng; Bennis, Mehdi.

    In: IEEE Internet of Things Journal, Vol. 6, No. 3, 8493155, 01.06.2019, p. 4005-4018.

    Research output: Contribution to journalArticleScientificpeer-review

    TY - JOUR

    T1 - Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning

    AU - Chen, Xianfu

    AU - Zhang, Honggang

    AU - Wu, Celimuge

    AU - Mao, Shiwen

    AU - Ji, Yusheng

    AU - Bennis, Mehdi

    PY - 2019/6/1

    Y1 - 2019/6/1

    N2 - To improve the quality of computation experience for mobile devices, mobile-edge computing (MEC) is a promising paradigm by providing computing capabilities in close proximity within a sliced radio access network (RAN), which supports both traditional communication and MEC services. Nevertheless, the design of computation offloading policies for a virtual MEC system remains challenging. Specifically, whether to execute a computation task at the mobile device or to offload it for MEC server execution should adapt to the time-varying network dynamics. This paper considers MEC for a representative mobile user in an ultradense sliced RAN, where multiple base stations (BSs) are available to be selected for computation offloading. The problem of solving an optimal computation offloading policy is modeled as a Markov decision process, where our objective is to maximize the long-term utility performance whereby an offloading decision is made based on the task queue state, the energy queue state as well as the channel qualities between mobile user and BSs. To break the curse of high dimensionality in state space, we first propose a double deep Q -network (DQN)-based strategic computation offloading algorithm to learn the optimal policy without knowing a priori knowledge of network dynamics. Then motivated by the additive structure of the utility function, a Q -function decomposition technique is combined with the double DQN, which leads to a novel learning algorithm for the solving of stochastic computation offloading. Numerical experiments show that our proposed learning algorithms achieve a significant improvement in computation offloading performance compared with the baseline policies.

    AB - To improve the quality of computation experience for mobile devices, mobile-edge computing (MEC) is a promising paradigm by providing computing capabilities in close proximity within a sliced radio access network (RAN), which supports both traditional communication and MEC services. Nevertheless, the design of computation offloading policies for a virtual MEC system remains challenging. Specifically, whether to execute a computation task at the mobile device or to offload it for MEC server execution should adapt to the time-varying network dynamics. This paper considers MEC for a representative mobile user in an ultradense sliced RAN, where multiple base stations (BSs) are available to be selected for computation offloading. The problem of solving an optimal computation offloading policy is modeled as a Markov decision process, where our objective is to maximize the long-term utility performance whereby an offloading decision is made based on the task queue state, the energy queue state as well as the channel qualities between mobile user and BSs. To break the curse of high dimensionality in state space, we first propose a double deep Q -network (DQN)-based strategic computation offloading algorithm to learn the optimal policy without knowing a priori knowledge of network dynamics. Then motivated by the additive structure of the utility function, a Q -function decomposition technique is combined with the double DQN, which leads to a novel learning algorithm for the solving of stochastic computation offloading. Numerical experiments show that our proposed learning algorithms achieve a significant improvement in computation offloading performance compared with the baseline policies.

    KW - deep reinforcement learning

    KW - Electronic mail

    KW - Heuristic algorithms

    KW - Markov decision process

    KW - Mobile handsets

    KW - mobile-edge computing

    KW - Network slicing

    KW - network virtualization

    KW - Q-function decomposition.

    KW - radio access networks

    KW - Servers

    KW - Stochastic processes

    KW - Task analysis

    KW - Wireless communication

    UR - http://www.scopus.com/inward/record.url?scp=85055025077&partnerID=8YFLogxK

    U2 - 10.1109/JIOT.2018.2876279

    DO - 10.1109/JIOT.2018.2876279

    M3 - Article

    AN - SCOPUS:85055025077

    VL - 6

    SP - 4005

    EP - 4018

    JO - IEEE Internet of Things Journal

    JF - IEEE Internet of Things Journal

    SN - 2327-4662

    IS - 3

    M1 - 8493155

    ER -