Regular Decomposition of Multivariate Time Series and Other Matrices

We describe and illustrate a novel algorithm for clustering a large number of time series into few 'regular groups'. Our method is inspired by the famous Szemerédi's Regularity Lemma (SRL) in graph theory. SRL suggests that large graphs and matrices can be naturally 'compressed' by partitioning elements in a small number of sets. These sets and the patterns of relations between them present a kind of structure of large objects while the more detailed structure is random-like. We develop a maximum likelihood method for finding such 'regular structures' and applied it to the case of smart meter data of households. The resulting structure appears as more informative than a structure found by k-means. The algorithm scales well with data size and the structure itself becomes more apparent with bigger data size. Therefore, our method could be useful in a broader context of emerging big data