Problem processing rows of uneven number of elements

[nevr1106]

I found a problem with the lvm_read.read() function when processing a labview data file. Further testing shows that it cannot process rows of uneven number of elements.

Error prompted:
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (20224,) + inhomogeneous part.

Source of the problem:
lvm_data[s]['data'] = np.asarray(lvm_data[s]['data'])

if the lvm_data[s]['data'] is like this:

[[1,2,3], [1,2], [1,2,3],...]

Then it will produce this error.

Solution:
Based on solution proposed here: ayhan, the problem code can be changed to the following:
lvm_data[s]['data'] = np.asarray(list(itertools.zip_longest(*lvm_data[s]['data'], fillvalue=0))).T

[jankoslavic]

Thank you @nevr1106! Are uneven number of elements supported by LVM?
As far as I am aware besides data one row is reserved for comments (which is in general string).
Can you please give more information on your motivation for the need?

[nevr1106]

The dataset I had problem with is shaped like this:

i th row: feature 1-10 at frequency 0.5s, feature 10-15 at frequency 1s, comment at frequency 1s

I do think the added comment is the trigger of the problem. The dataset with empty comment does not produce the same error.

[jankoslavic]

Ok. Thanks for the explanation. Can you please attach the sample file?

[nevr1106]

Here is a dummy dataset that will trigger this error.

dummy_dataset.zip

[jankoslavic]

@nevr1106 I have prepared a PR: #13
I am not perfectly happy with this as it not an elegant/simple solution.
Your test file not only has comments it also has uneven data. Is this a real case that you had with your measurements?
PS: the comments are saved with the key "comments"

Please check it out and give feedback.

[nevr1106]

Yes, it is masked data from a real testing facility. The uneven data comes from sensors operating at different sampling frequency.

[jankoslavic]

dear @nevr1106 have you further tested this implementation? It is a relatively big change and I am slightly hesitant in merging it... :)