Abnormality detection using SOM modeling

We describe work aimed at applying neural network methods to detect abnormal
conditions in quality measurements of an industrial chemical process. The methods predict the future values of the target variables based on multivariable histories of past sensor readings. A waste water cleaning plant using an activated sludge cleaning method served as our test environment for the methods.

Our aim has been to develop methods that may be implemented as online analysis tools at the process control environments and are capable of being interfaced with existing data sources. Previously we have tested Multi-Layered Perceptron networks on this problem and have described a demonstration system indicating how the measurements can be stored in a RapidBase main memory active database and how the neural network models may be used online for creating predictions for future values. In this report we continue the previously reported analysis work with extended measurement data series and investigate especially the suitability of the Self-Organizing Map (SOM) architecture.

We have used a time-lagged delay line for embedding the temporal histories of the measurement time series and correlation analysis and statistical independency testing for detecting the most useful input variables for the analysis. Correlation analysis suggests that typically only the most recent measurements correlate with the future values.

In order to detect abnormal behaviour we have tought a SOM with data describing
normally operating process. The learned map has been used to detect abnormalities in new data by monitoring the shortest distance of these new samples from the learned SOM codebook. Calibration data is used to select an optimal classification threshold. Experimental tests suggest that typically only short term predictions are possible in the case of the studied process. The best classification accuracies reach 80-90% levels. Additionally we have used SOM to predict the future values by teaching it with a random sample of all the measurements and then creating an index from the SOM neurons to the list of teaching samples best matching that neuron. This index is used for nearest
neighbour prediction while monitoring new data samples. In experimental tests the predictions were found to be promising for short intervals but lost accuracy when the prediction interval was made larger. The classification accuracies achieved were a little better than with the previously mentioned QE-monitoring results but the false positive rates were condiderably more severe.