Endurance Test!

[shahrokhDaijavad]

As we discussed issue #388 in the internal repo, we want to test if the framework has a memory leak or not. To do this, we use the noop transform and a large dataset (like test set 3 that has 1500 zip files) and 1) Run ingest2parquet on it first and 2) Try the noop transform while monitoring the memory usage of the laptop and see if it reaches a flat plateau or not.

[shivdeep-singh-ibm]

I tested with a set of 1483 files on (32GB, 4CPUs)and used traceback library to check for memory leak. I ran 10 iterations and observed the memory usage which was peaking around 4 GB. There were no obvious signs of a memory leak in this test.

[shivdeep-singh-ibm]

I ran another test of 1483 files on podman VM with different memory configurations. The results are below.
It seems that it needed around 4GB of available memory to run successfully for these 1483 files.

CPUs	Total Memory	Memory Used by Ray	Transform	Files Processed Successfully	Total Files	Status of JOB
4	8GB	4.2GB	NOOP	1483	1483	Passed
4	6GB	3 GB	NOOP	910	1483	Crashed
4	4GB	2GB	NOOP	504	1483	Crashed

[blublinsky]

@shahrokhDaijavad, @shivdeep-singh-ibm This is an important piece of info not only for us but also for potential users. Can we please:

1. Bring all of the results together in a separate document inside the project
2. Once 1 is complete, remove this issue

[shahrokhDaijavad]

I agree, @blublinsky. I will create an md file called memorytest under doc with this information and link from the mac.md file to it.

[blublinsky]

@shahrokhDaijavad great. Should it be memory? or endurance?

[shahrokhDaijavad]

@blublinsky It's a combination of testing for memory leak (which peaks and flattens around 4GB, i.e., no leak) and endurance that shows with smaller memory (4GB and 6GB total memory), it is still possible to process 500 or 900 files successfully before it crashes. I will explain in the readme.