Post-processing timeout/memory error for large runs

[aspeake]

Describe the bug
I have encountered issues post-processing large resstock runs (~1.5MM sims) on Eagle.

Initial error in postprocessing.out:
slurmstepd: error: Detected 1 oom-kill event(s) in step 7788440.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.

This error was circumvented by requesting bigmem nodes, however, I have timed out running --postprocessonly, requiring a large number of bigmem nodes which prevented it from being scheduled in a timely manner.

To Reproduce

1. Project yaml
2. Run --postprocessonly
   2.1 If requesting bigmem nodes, the job will take a very long time to schedule
   2.2 if requesting standard nodes, the out-of-memory error will likely occur

Logs
Memory error: /lustre/eaglefs/projects/scout/flex_measures/flex_full/postprocessing_202110291115.out
Time out error: /lustre/eaglefs/projects/scout/flex_measures/flex_full/postprocessing_202111080407.out

Platform:

• Simulation platform: Eagle
• BuildStockBatch version, branch, or sha: restructure-v3-flex (https://github.com/NREL/buildstockbatch/tree/restructure-v3-flex)
• resstock or comstock repo version, branch, or sha: restructure-v3-flex (https://github.com/NREL/resstock/tree/restructure-v3-flex)

[rajeee]

I think the OOM error is occuring because the postprocessing attempts to compose a single gigantic results_csv dataframe here:

buildstockbatch/buildstockbatch/postprocessing.py

Line 267 in c8d759e

results_df = pd.DataFrame(dpouts).rename(columns=to_camelcase)

If we can break these to process only one results_csv_jobx at a time, it should not result in OOM.