Anomaly detection from server log data: A case study

Sami Nousiainen, Jorma Kilpi, Paula Silvonen, Mikko Hiirsalmi

    Research output: Book/ReportReport

    1 Citation (Scopus)

    Abstract

    This study focuses on the analysis of server log data and the detection and potential prediction of anomalies related to the monitored servers. The issue is relevant in many mission-critical systems consisting of multiple servers. There it is favourable to be able detect and even foresee problems to be able to react promptly and apply required corrections to the system. In this study, we have done off-line analyses based on pre-recorded data. In reality, if the objective is to come up with solutions for detecting anomalies in real-time, additional requirements and constraints would be imposed on the algorithms to be used. For example, in on-line situation, higher requirements on the performance of the algorithm and on the amount of historical data available for the algorithm would exist. However, we do not address those issues in this preliminary study. In addition to the analysis of real data, we have interviewed experts that are working on the server-related issues on a daily basis. Based on those discussions, we have tried to formulate practical cases, for which some algorithms and tools could provide practical utility.
    Original languageEnglish
    Place of PublicationEspoo
    PublisherVTT Technical Research Centre of Finland
    Number of pages46
    ISBN (Electronic)978-951-38-7289-2
    Publication statusPublished - 2009
    MoE publication typeNot Eligible

    Publication series

    SeriesVTT Tiedotteita - Research Notes
    Number2480
    ISSN1235-0605

    Fingerprint

    Servers

    Keywords

    • anomaly detection
    • data mining
    • machine learning
    • SOM
    • self-organizing map
    • IT monitoring
    • server log file
    • CPU
    • memory
    • process

    Cite this

    Nousiainen, S., Kilpi, J., Silvonen, P., & Hiirsalmi, M. (2009). Anomaly detection from server log data: A case study. Espoo: VTT Technical Research Centre of Finland. VTT Tiedotteita - Research Notes, No. 2480
    Nousiainen, Sami ; Kilpi, Jorma ; Silvonen, Paula ; Hiirsalmi, Mikko. / Anomaly detection from server log data : A case study. Espoo : VTT Technical Research Centre of Finland, 2009. 46 p. (VTT Tiedotteita - Research Notes; No. 2480).
    @book{48dbee710a724d2fb2b7cee6c0367d9c,
    title = "Anomaly detection from server log data: A case study",
    abstract = "This study focuses on the analysis of server log data and the detection and potential prediction of anomalies related to the monitored servers. The issue is relevant in many mission-critical systems consisting of multiple servers. There it is favourable to be able detect and even foresee problems to be able to react promptly and apply required corrections to the system. In this study, we have done off-line analyses based on pre-recorded data. In reality, if the objective is to come up with solutions for detecting anomalies in real-time, additional requirements and constraints would be imposed on the algorithms to be used. For example, in on-line situation, higher requirements on the performance of the algorithm and on the amount of historical data available for the algorithm would exist. However, we do not address those issues in this preliminary study. In addition to the analysis of real data, we have interviewed experts that are working on the server-related issues on a daily basis. Based on those discussions, we have tried to formulate practical cases, for which some algorithms and tools could provide practical utility.",
    keywords = "anomaly detection, data mining, machine learning, SOM, self-organizing map, IT monitoring, server log file, CPU, memory, process",
    author = "Sami Nousiainen and Jorma Kilpi and Paula Silvonen and Mikko Hiirsalmi",
    note = "Project code: 13674",
    year = "2009",
    language = "English",
    series = "VTT Tiedotteita - Research Notes",
    publisher = "VTT Technical Research Centre of Finland",
    number = "2480",
    address = "Finland",

    }

    Nousiainen, S, Kilpi, J, Silvonen, P & Hiirsalmi, M 2009, Anomaly detection from server log data: A case study. VTT Tiedotteita - Research Notes, no. 2480, VTT Technical Research Centre of Finland, Espoo.

    Anomaly detection from server log data : A case study. / Nousiainen, Sami; Kilpi, Jorma; Silvonen, Paula; Hiirsalmi, Mikko.

    Espoo : VTT Technical Research Centre of Finland, 2009. 46 p. (VTT Tiedotteita - Research Notes; No. 2480).

    Research output: Book/ReportReport

    TY - BOOK

    T1 - Anomaly detection from server log data

    T2 - A case study

    AU - Nousiainen, Sami

    AU - Kilpi, Jorma

    AU - Silvonen, Paula

    AU - Hiirsalmi, Mikko

    N1 - Project code: 13674

    PY - 2009

    Y1 - 2009

    N2 - This study focuses on the analysis of server log data and the detection and potential prediction of anomalies related to the monitored servers. The issue is relevant in many mission-critical systems consisting of multiple servers. There it is favourable to be able detect and even foresee problems to be able to react promptly and apply required corrections to the system. In this study, we have done off-line analyses based on pre-recorded data. In reality, if the objective is to come up with solutions for detecting anomalies in real-time, additional requirements and constraints would be imposed on the algorithms to be used. For example, in on-line situation, higher requirements on the performance of the algorithm and on the amount of historical data available for the algorithm would exist. However, we do not address those issues in this preliminary study. In addition to the analysis of real data, we have interviewed experts that are working on the server-related issues on a daily basis. Based on those discussions, we have tried to formulate practical cases, for which some algorithms and tools could provide practical utility.

    AB - This study focuses on the analysis of server log data and the detection and potential prediction of anomalies related to the monitored servers. The issue is relevant in many mission-critical systems consisting of multiple servers. There it is favourable to be able detect and even foresee problems to be able to react promptly and apply required corrections to the system. In this study, we have done off-line analyses based on pre-recorded data. In reality, if the objective is to come up with solutions for detecting anomalies in real-time, additional requirements and constraints would be imposed on the algorithms to be used. For example, in on-line situation, higher requirements on the performance of the algorithm and on the amount of historical data available for the algorithm would exist. However, we do not address those issues in this preliminary study. In addition to the analysis of real data, we have interviewed experts that are working on the server-related issues on a daily basis. Based on those discussions, we have tried to formulate practical cases, for which some algorithms and tools could provide practical utility.

    KW - anomaly detection

    KW - data mining

    KW - machine learning

    KW - SOM

    KW - self-organizing map

    KW - IT monitoring

    KW - server log file

    KW - CPU

    KW - memory

    KW - process

    M3 - Report

    T3 - VTT Tiedotteita - Research Notes

    BT - Anomaly detection from server log data

    PB - VTT Technical Research Centre of Finland

    CY - Espoo

    ER -

    Nousiainen S, Kilpi J, Silvonen P, Hiirsalmi M. Anomaly detection from server log data: A case study. Espoo: VTT Technical Research Centre of Finland, 2009. 46 p. (VTT Tiedotteita - Research Notes; No. 2480).