Generate multiple image from multivariate time series

[hvsw]

Description

I was wondering, is fitted.shape equal to (6, 6, 6) because it fitted and transformed for each feature and generated an image for each one, so I have 6 6x6 images?

Steps/Code to Reproduce

This code shows what I mean. fitted.shape is (6, 6, 6).

from pyts.image import GramianAngularField
gasf = GramianAngularField(image_size=6, method='summation')
cueSamples = numpy.zeros((6, 314)) # shape = (6, 314)

fitted = gasf.fit_transform(cueSamples)
fitted.shape # (6, 6, 6)

Versions

NumPy 1.20.2
SciPy 1.6.2
Scikit-Learn 0.24.1
Numba 0.53.1
Pyts 0.11.0

[johannfaouzi]

Indeed, each "row" of cueSamples is independently transformed into an image. In the article introducing this transformation, the authors worked on univariate time series only. The Gramian Angular Field relies on the polar coordinate representation of a univariate time series, so I don't think it would be trivial to generalize for multivariate time series: you can't have a single scalar to define the "angle" of a vector in N dimensions (with N > 2). You could use the cosine similarity to have an angle between two vectors (a vector would represent the different features of a multivariate time series at a given time point):

GramianAngularField actually expects a 2D-array as input to represent several univariate time series and not a multivariate time series, thus the transformation of each row independently into an image.

Hope this helps you a bit. If you would like to try the cosine similarity for multivariate time series, I could help you with the code if needed.

[hvsw]

Thank you @johannfaouzi, that's super helpful!

I'm interested in the cosine similarity you mentioned - some context: I'm working on training CNNs with EEG data for BCI systems for my undergrad thesis. Multivariate Time Series Data Transformation for Convolutional Neural Network describes a simple way of getting 1 single image by simply appending the variables images(in my case possibly channels/trials images).

So, what you're suggesting is another option to "join the data", but instead of appending images, we generate a single matrix having the angles, i.e. instead of having N trials/channels images we'd have a matrix with the angle between the trials/channels as a single scalar and generate 1 single image from that matrix "having all the information"?

[johannfaouzi]

Indeed I think that there are two main approaches:

• Generating an image for each channel. In this case, you let your model deal with the possible correlation between the different images. It seems to me that this approach is the most suitable if your algorithm is a CNN (or a deep learning algorithm in general) since the selling point of deep learning is to let the model extract / learn the relevant information. You could also do this with non-deep machine learning where you would train a model for each channel and output a single prediction using some sort of voting/averaging at the end.
• Generating a single image for all the channels. In this case, you "hand-craft" the possible interactions between the different channels. Suitable approach for non-deep machine learning, but it may require some kind of expertise to generate a single relevant image to your use case.

You will find attached a notebook in which I tried several approaches based on cosine similarity (or linear kernel). Here is also a Google Colab link.

Hope this helps you a bit.

Transforming multivariate time series into images.ipynb.zip

[Fiwzea]

@johannfaouzi
Hi dear, I am also working on time series classification.
I have my 6 input time series data and I want to converted into images using GAF and MTF, as you know that these two methods were designed for univariate input but mine I have 6 input (accelerometer in x,y & z and gyroscope x,y & z), so I need to modify the GAF method to but for multi-variate . I read your suggestion above but i am wondering of how did you reach that, like did you use the average or Concatenation? Also, which package or library did you use? Really appreciate your helping

[Fiwzea]

@hvsw
Hi dear, I am also working on time series classification.
I have my 6 input time series data and I want to converted into images using GAF and MTF, as you know that these two methods were designed for univariate input but mine I have 6 input (accelerometer in x,y & z and gyroscope x,y & z), so I need to modify the GAF method to but for multi-variate . I read your suggestion above but i am wondering of how did you reach that, like did you use the average or Concatenation? Also, which package or library did you use? Did you use concatenation or average. Really appreciate your helping