MemoryError when loading large number of datasets

[jakeatmsft]

Describe the bug
My training file is loading multiple datasets into memory using "create_tabular_dataset_from_parquet_files" function then using rm(dataset);gc(); to release the dataset. Error thrown even after datasets are removed and garbage collection invoked.

driver_log:
Error in py_call_impl(callable, dots$args, dots$keywords) :
MemoryError: Engine process terminated. This is most likely due to system running out of memory. Please retry with increased memory. |session_id=ba9874d8-f32a-4568-b75c-650f6747ef4e

Detailed traceback:
File "/azureml-envs/azureml_505e9e90fca3d01d6df21acd71c8c832/lib/python3.6/site-packages/azureml/data/_loggerfactory.py", line 126, in wrapper
return func(*args, **kwargs)
File "/azureml-envs/azureml_505e9e90fca3d01d6df21acd71c8c832/lib/python3.6/site-packages/azureml/data/dataset_factory.py", line 122, in from_parquet_files
partition_format)
File "/azureml-envs/azureml_505e9e90fca3d01d6df21acd71c8c832/lib/python3.6/site-packages/azureml/data/dataset_factory.py", line 134, in _from_parquet_files
validate or _is_inference_required(set_column_types))
File "/azureml-envs/azureml_505e9e90fca3d01d6df21acd71c8c832/lib/python3.6/site-packages/azureml/data/dataset_factory.py", line 768, in _transform_and_validate
'Make sure the path is accessible and contains
Calls: source ... create_tabular_dataset_from_parquet_files -> -> py_call_impl

Expected behavior
rm(); gc(); should release memory associated with in-memory dataset such that memoryerror does not occur.

70_driver_log (1).txt