Worker not releasing memory from Python process

[mpinkerton-oasis]

The Python process seems to hold on to the memory used for generating the analysis inputs. This should be freed up, and in particular made available to the ktools model execution.

According to @ndaqjn this can be easilly verified with the PiWind model.

[sambles]

Just checked and it does seem to be leaking memory, this might be worth a try:
https://docs.python.org/3/library/tracemalloc.html

[mpinkerton-oasis]

Note from @OmegaDroid
his came down to 3 issues,

1. The billiard implementation of thread pools requires pool.terminate to be called otherwise the processes are kept around
2. The pandas memory leak in the issue link previously. I added the monkey patch suggested in the issue which seems to do the trick
3. I added some explicit garbage collection around file generation
   I've also made the commands calls in the task worker create a sub process calling the oasislmf executabe so it shouldnt use any memory anymore once complete, I'm having issues getting it running locally atm so this hasnt been tested fully yet but ill open a pr for it

[mpinkerton-oasis]

@OmegaDroid could you add the link for the Pandas issue.

[OmegaDroid]

pandas-dev/pandas#2659

[sambles]

1 & 2 done, adding an issue under OasisLmf for 3 and closing this issue

OasisLMF/OasisLMF#490