Designing a Data Management Pipeline for Pervasive Sensor Communication Systems

Pervasive sensor systems offer unbounded possibilities for monitoring and tracking objects, machines, and spaces. To maximize the benefit from a sensor system, sensor data requires efficient preprocessing and analysis. Big data techniques make distributed processing of huge amounts of data fast and cost-effective, making them a practical necessity for sensor data. However, the real-time requirements and the sheer velocity and volume of data from large sensor systems require a dedicated approach to designing the data processing pipeline. This paper discusses viewpoints and requirements for designing a sensor data pipeline, with specific focus on data input, live preprocessing, and storage.