Can not query multiple symbols if their columns are stored with different data types

[briancappello]

This is perhaps more of a limitation than a bug?

Let's say you create two TBKs, one stored with 64bit values and the other 32bit:

from pymarketstore import Client, DataShape
client.create('AMD/1D/OHLCV', DataShape([
  ('Open', 'float64'),
  ('High', 'float64'),
  ('Low', 'float64'),
  ('Close', 'float64'),
  ('Volume', 'int64'),
]))
client.create('NVDA/1D/OHLCV', DataShape([
  ('Open', 'float32'),
  ('High', 'float32'),
  ('Low', 'float32'),
  ('Close', 'float32'),
  ('Volume', 'int32'),
]))

If you then query for these together:

client.query('AMD,NVDA/1D/OHLCV')

The response format does not support this. (pymarketstore will blow up attempting to decode the raw response data.) The reason being, there's an implicit assumption built into NumpyMultiDataset that all datatypes must be the same:

NumpyDataset:
  ColumnNames: ['Open', 'Volume']
  ColumnTypes: ['float32', 'int32']
  ColumnData:
    Open: [AMD opens] + [NVDA opens]
    Volume: [AMD volumes] + [NVDA volumes]
StartIndex: indexes into ColumnData for each symbol
Lengths: number of bars in ColumnData for each symbol

ColumnData is simply bytes, so marketstore will happily return 32bit values and 64bit values in sequence, however, the client has no way to know that the data types vary by symbol, and as such fails to correctly index into the ColumnData arrays.

Querying for the symbols individually works fine, with marketstore returning the correct data types for each.