A big data architecture for large scale security monitoring

Samuel Marchal; Xiuyan Jiang; Radu State; Thomas Engel

doi:10.1109/BigData.Congress.2014.18

A big data architecture for large scale security monitoring

Samuel Marchal^*
, Xiuyan Jiang
, Radu State
, Thomas Engel

^*Corresponding author for this work

Not published at VTT

University of Luxembourg
University of Lorraine

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

115 Citations (Scopus)

Abstract

Network traffic is a rich source of information for security monitoring. However the increasing volume of data to treat raises issues, rendering holistic analysis of network traffic difficult. In this paper we propose a solution to cope with the tremendous amount of data to analyse for security monitoring perspectives. We introduce an architecture dedicated to security monitoring of local enterprise networks. The application domain of such a system is mainly network intrusion detection and prevention, but can be used as well for forensic analysis. This architecture integrates two systems, one dedicated to scalable distributed data storage and management and the other dedicated to data exploitation. DNS data, NetFlow records, HTTP traffic and honeypot data are mined and correlated in a distributed system that leverages state of the art big data solution. Data correlation schemes are proposed and their performance are evaluated against several well-known big data framework including Hadoop and Spark.

Original language	English
Title of host publication	Proceedings 2014 IEEE International Congress on Big Data
Publisher	IEEE Institute of Electrical and Electronic Engineers
Pages	56-63
Number of pages	8
ISBN (Electronic)	978-1-4799-5057-7
DOIs	https://doi.org/10.1109/BigData.Congress.2014.18
Publication status	Published - 22 Sept 2014
MoE publication type	A4 Article in a conference publication
Event	3rd IEEE International Congress on Big Data, BigData Congress 2014 - Anchorage, United States Duration: 27 Jun 2014 → 2 Jul 2014

Conference

Conference	3rd IEEE International Congress on Big Data, BigData Congress 2014
Country/Territory	United States
City	Anchorage
Period	27/06/14 → 2/07/14

Access to Document

10.1109/BigData.Congress.2014.18

Cite this

@inproceedings{e816675d3a6e4ec1ae0a6ae2c8089fb1,

title = "A big data architecture for large scale security monitoring",

abstract = "Network traffic is a rich source of information for security monitoring. However the increasing volume of data to treat raises issues, rendering holistic analysis of network traffic difficult. In this paper we propose a solution to cope with the tremendous amount of data to analyse for security monitoring perspectives. We introduce an architecture dedicated to security monitoring of local enterprise networks. The application domain of such a system is mainly network intrusion detection and prevention, but can be used as well for forensic analysis. This architecture integrates two systems, one dedicated to scalable distributed data storage and management and the other dedicated to data exploitation. DNS data, NetFlow records, HTTP traffic and honeypot data are mined and correlated in a distributed system that leverages state of the art big data solution. Data correlation schemes are proposed and their performance are evaluated against several well-known big data framework including Hadoop and Spark.",

author = "Samuel Marchal and Xiuyan Jiang and Radu State and Thomas Engel",

year = "2014",

month = sep,

day = "22",

doi = "10.1109/BigData.Congress.2014.18",

language = "English",

pages = "56--63",

booktitle = "Proceedings 2014 IEEE International Congress on Big Data",

publisher = "IEEE Institute of Electrical and Electronic Engineers",

address = "United States",

note = "3rd IEEE International Congress on Big Data, BigData Congress 2014 ; Conference date: 27-06-2014 Through 02-07-2014",

}

Marchal, S, Jiang, X, State, R & Engel, T 2014, A big data architecture for large scale security monitoring. in Proceedings 2014 IEEE International Congress on Big Data. IEEE Institute of Electrical and Electronic Engineers, pp. 56-63, 3rd IEEE International Congress on Big Data, BigData Congress 2014, Anchorage, United States, 27/06/14. https://doi.org/10.1109/BigData.Congress.2014.18

A big data architecture for large scale security monitoring. / Marchal, Samuel; Jiang, Xiuyan; State, Radu et al.
Proceedings 2014 IEEE International Congress on Big Data. IEEE Institute of Electrical and Electronic Engineers, 2014. p. 56-63.

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

TY - GEN

T1 - A big data architecture for large scale security monitoring

AU - Marchal, Samuel

AU - Jiang, Xiuyan

AU - State, Radu

AU - Engel, Thomas

PY - 2014/9/22

Y1 - 2014/9/22

N2 - Network traffic is a rich source of information for security monitoring. However the increasing volume of data to treat raises issues, rendering holistic analysis of network traffic difficult. In this paper we propose a solution to cope with the tremendous amount of data to analyse for security monitoring perspectives. We introduce an architecture dedicated to security monitoring of local enterprise networks. The application domain of such a system is mainly network intrusion detection and prevention, but can be used as well for forensic analysis. This architecture integrates two systems, one dedicated to scalable distributed data storage and management and the other dedicated to data exploitation. DNS data, NetFlow records, HTTP traffic and honeypot data are mined and correlated in a distributed system that leverages state of the art big data solution. Data correlation schemes are proposed and their performance are evaluated against several well-known big data framework including Hadoop and Spark.

AB - Network traffic is a rich source of information for security monitoring. However the increasing volume of data to treat raises issues, rendering holistic analysis of network traffic difficult. In this paper we propose a solution to cope with the tremendous amount of data to analyse for security monitoring perspectives. We introduce an architecture dedicated to security monitoring of local enterprise networks. The application domain of such a system is mainly network intrusion detection and prevention, but can be used as well for forensic analysis. This architecture integrates two systems, one dedicated to scalable distributed data storage and management and the other dedicated to data exploitation. DNS data, NetFlow records, HTTP traffic and honeypot data are mined and correlated in a distributed system that leverages state of the art big data solution. Data correlation schemes are proposed and their performance are evaluated against several well-known big data framework including Hadoop and Spark.

UR - https://www.scopus.com/pages/publications/84923911158

U2 - 10.1109/BigData.Congress.2014.18

DO - 10.1109/BigData.Congress.2014.18

M3 - Conference article in proceedings

AN - SCOPUS:84923911158

SP - 56

EP - 63

BT - Proceedings 2014 IEEE International Congress on Big Data

PB - IEEE Institute of Electrical and Electronic Engineers

T2 - 3rd IEEE International Congress on Big Data, BigData Congress 2014

Y2 - 27 June 2014 through 2 July 2014

ER -

A big data architecture for large scale security monitoring

Abstract

Conference

Access to Document

Other files and links

Fingerprint

Cite this