Anomaly detection from server log data: A case study

Sami Nousiainen, Jorma Kilpi, Paula Silvonen, Mikko Hiirsalmi

Research output: Book/ReportReport

1 Citation (Scopus)

Abstract

This study focuses on the analysis of server log data and the detection and potential prediction of anomalies related to the monitored servers. The issue is relevant in many mission-critical systems consisting of multiple servers. There it is favourable to be able detect and even foresee problems to be able to react promptly and apply required corrections to the system. In this study, we have done off-line analyses based on pre-recorded data. In reality, if the objective is to come up with solutions for detecting anomalies in real-time, additional requirements and constraints would be imposed on the algorithms to be used. For example, in on-line situation, higher requirements on the performance of the algorithm and on the amount of historical data available for the algorithm would exist. However, we do not address those issues in this preliminary study. In addition to the analysis of real data, we have interviewed experts that are working on the server-related issues on a daily basis. Based on those discussions, we have tried to formulate practical cases, for which some algorithms and tools could provide practical utility.
Original languageEnglish
Place of PublicationEspoo
PublisherVTT Technical Research Centre of Finland
Number of pages46
ISBN (Electronic)978-951-38-7289-2
Publication statusPublished - 2009
MoE publication typeNot Eligible

Publication series

SeriesVTT Tiedotteita - Research Notes
Number2480
ISSN1235-0605

Fingerprint

Servers

Keywords

  • anomaly detection
  • data mining
  • machine learning
  • SOM
  • self-organizing map
  • IT monitoring
  • server log file
  • CPU
  • memory
  • process

Cite this

Nousiainen, S., Kilpi, J., Silvonen, P., & Hiirsalmi, M. (2009). Anomaly detection from server log data: A case study. Espoo: VTT Technical Research Centre of Finland. VTT Tiedotteita - Research Notes, No. 2480
Nousiainen, Sami ; Kilpi, Jorma ; Silvonen, Paula ; Hiirsalmi, Mikko. / Anomaly detection from server log data : A case study. Espoo : VTT Technical Research Centre of Finland, 2009. 46 p. (VTT Tiedotteita - Research Notes; No. 2480).
@book{48dbee710a724d2fb2b7cee6c0367d9c,
title = "Anomaly detection from server log data: A case study",
abstract = "This study focuses on the analysis of server log data and the detection and potential prediction of anomalies related to the monitored servers. The issue is relevant in many mission-critical systems consisting of multiple servers. There it is favourable to be able detect and even foresee problems to be able to react promptly and apply required corrections to the system. In this study, we have done off-line analyses based on pre-recorded data. In reality, if the objective is to come up with solutions for detecting anomalies in real-time, additional requirements and constraints would be imposed on the algorithms to be used. For example, in on-line situation, higher requirements on the performance of the algorithm and on the amount of historical data available for the algorithm would exist. However, we do not address those issues in this preliminary study. In addition to the analysis of real data, we have interviewed experts that are working on the server-related issues on a daily basis. Based on those discussions, we have tried to formulate practical cases, for which some algorithms and tools could provide practical utility.",
keywords = "anomaly detection, data mining, machine learning, SOM, self-organizing map, IT monitoring, server log file, CPU, memory, process",
author = "Sami Nousiainen and Jorma Kilpi and Paula Silvonen and Mikko Hiirsalmi",
note = "Project code: 13674",
year = "2009",
language = "English",
series = "VTT Tiedotteita - Research Notes",
publisher = "VTT Technical Research Centre of Finland",
number = "2480",
address = "Finland",

}

Nousiainen, S, Kilpi, J, Silvonen, P & Hiirsalmi, M 2009, Anomaly detection from server log data: A case study. VTT Tiedotteita - Research Notes, no. 2480, VTT Technical Research Centre of Finland, Espoo.

Anomaly detection from server log data : A case study. / Nousiainen, Sami; Kilpi, Jorma; Silvonen, Paula; Hiirsalmi, Mikko.

Espoo : VTT Technical Research Centre of Finland, 2009. 46 p. (VTT Tiedotteita - Research Notes; No. 2480).

Research output: Book/ReportReport

TY - BOOK

T1 - Anomaly detection from server log data

T2 - A case study

AU - Nousiainen, Sami

AU - Kilpi, Jorma

AU - Silvonen, Paula

AU - Hiirsalmi, Mikko

N1 - Project code: 13674

PY - 2009

Y1 - 2009

N2 - This study focuses on the analysis of server log data and the detection and potential prediction of anomalies related to the monitored servers. The issue is relevant in many mission-critical systems consisting of multiple servers. There it is favourable to be able detect and even foresee problems to be able to react promptly and apply required corrections to the system. In this study, we have done off-line analyses based on pre-recorded data. In reality, if the objective is to come up with solutions for detecting anomalies in real-time, additional requirements and constraints would be imposed on the algorithms to be used. For example, in on-line situation, higher requirements on the performance of the algorithm and on the amount of historical data available for the algorithm would exist. However, we do not address those issues in this preliminary study. In addition to the analysis of real data, we have interviewed experts that are working on the server-related issues on a daily basis. Based on those discussions, we have tried to formulate practical cases, for which some algorithms and tools could provide practical utility.

AB - This study focuses on the analysis of server log data and the detection and potential prediction of anomalies related to the monitored servers. The issue is relevant in many mission-critical systems consisting of multiple servers. There it is favourable to be able detect and even foresee problems to be able to react promptly and apply required corrections to the system. In this study, we have done off-line analyses based on pre-recorded data. In reality, if the objective is to come up with solutions for detecting anomalies in real-time, additional requirements and constraints would be imposed on the algorithms to be used. For example, in on-line situation, higher requirements on the performance of the algorithm and on the amount of historical data available for the algorithm would exist. However, we do not address those issues in this preliminary study. In addition to the analysis of real data, we have interviewed experts that are working on the server-related issues on a daily basis. Based on those discussions, we have tried to formulate practical cases, for which some algorithms and tools could provide practical utility.

KW - anomaly detection

KW - data mining

KW - machine learning

KW - SOM

KW - self-organizing map

KW - IT monitoring

KW - server log file

KW - CPU

KW - memory

KW - process

M3 - Report

T3 - VTT Tiedotteita - Research Notes

BT - Anomaly detection from server log data

PB - VTT Technical Research Centre of Finland

CY - Espoo

ER -

Nousiainen S, Kilpi J, Silvonen P, Hiirsalmi M. Anomaly detection from server log data: A case study. Espoo: VTT Technical Research Centre of Finland, 2009. 46 p. (VTT Tiedotteita - Research Notes; No. 2480).