Crashing due to malloc failure

[svendaj]

Trying to run example_model.py on Windows 10, 8GB RAM notebook results in following crash:

(numer.ai) C:\Users\lsadmin\Documents\numer.ai\example-scripts>python example_model.py
Downloading dataset files...
2022-05-17 16:01:22,904 INFO numerapi.utils: target file already exists
2022-05-17 16:01:22,906 INFO numerapi.utils: download complete
2022-05-17 16:01:24,126 INFO numerapi.utils: target file already exists
2022-05-17 16:01:24,126 INFO numerapi.utils: download complete
2022-05-17 16:01:25,315 INFO numerapi.utils: target file already exists
2022-05-17 16:01:25,315 INFO numerapi.utils: download complete
2022-05-17 16:01:26,494 INFO numerapi.utils: target file already exists
2022-05-17 16:01:26,494 INFO numerapi.utils: download complete
2022-05-17 16:01:28,128 INFO numerapi.utils: target file already exists
2022-05-17 16:01:28,128 INFO numerapi.utils: download complete
Reading minimal training data
Traceback (most recent call last):
File "C:\Users\lsadmin\Documents\numer.ai\example-scripts\example_model.py", line 52, in
validation_data = pd.read_parquet('v4/validation.parquet',
File "C:\Users\lsadmin\anaconda3\envs\numer.ai\lib\site-packages\pandas\io\parquet.py", line 493, in read_parquet
return impl.read(
File "C:\Users\lsadmin\anaconda3\envs\numer.ai\lib\site-packages\pandas\io\parquet.py", line 240, in read
result = self.api.parquet.read_table(
File "pyarrow\array.pxi", line 767, in pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow\table.pxi", line 1996, in pyarrow.lib.Table._to_pandas
File "C:\Users\lsadmin\anaconda3\envs\numer.ai\lib\site-packages\pyarrow\pandas_compat.py", line 789, in table_to_blockmanager
blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
File "C:\Users\lsadmin\anaconda3\envs\numer.ai\lib\site-packages\pyarrow\pandas_compat.py", line 1135, in _table_to_blocks
result = pa.lib.table_to_blocks(options, block_table, categories,
File "pyarrow\table.pxi", line 1356, in pyarrow.lib.table_to_blocks
File "pyarrow\error.pxi", line 116, in pyarrow.lib.check_status
pyarrow.lib.ArrowMemoryError: malloc of size 4209552448 failed

Any idea why or how to overcome? Failed malloc is exactly 4GB, can the example run on machine like above?

[1NoobDev]

Your device doesn't meet the required system specifications as described in the FAQ:

Minimum: 16GB RAM and 4 core (Intel i5) / 6 cores (AMD Ryzen 5).
Recommended: 32GB RAM and 8 core (Intel i7/i9) / 12 cores (AMD Ryzen 7/9).

You could try to change the feature_metadata on line 44, to small:
features = feature_metadata["feature_sets"]["medium"] # get the medium feature set

Or overcome by adding more RAM in your device.