Singularity image collision on download

[nmerket]

Describe the bug
If you ask for a openstudio version that doesn't exist in the /shared-projects/buildstock/singularity_images/ folder on Eagle, it'll try to download it from the appropriate S3 bucket. This is only supposed to happen once in the sampling step. However it seems to be happening in each individual job:

DEBUG:2022-01-05 13:16:03:buildstockbatch.base:Using OpenStudio version: 3.3.0 with SHA: ad235ff36e
DEBUG:2022-01-05 13:16:03:__main__:Output directory = /scratch/pwhite/atlanta_tp_runs/atlanta_2010_cold_long
DEBUG:2022-01-05 13:16:04:__main__:Downloading singularity image: https://s3.amazonaws.com/openstudio-builds/3.3.0/OpenStudio-3.3.0.ad235ff36e-Singularity.simg
DEBUG:2022-01-05 13:16:32:__main__:Downloaded singularity image to /scratch/pwhite/atlanta_tp_runs/atlanta_2010_cold_long/openstudio.simg

This is causing several independent jobs to try to write to the same file at the same time and causing corrupted singularity images and simulation failures.

To Reproduce
Steps to reproduce the behavior:

1. Submit a project on Eagle using an OpenStudio version for which we don't already have one downloaded in the shared-projects folder.
2. Not sure if it matters, but the sampler used was the precomputed one.

@whiphi92 can you fill this part in? 👇

Platform (please complete the following information):

• Simulation platform: Eagle
• BuildStockBatch version, branch, or sha:
• resstock or comstock repo version, branch, or sha:
• Local Desktop OS: [e.g. Windows, Mac, Linux, especially important if running locally]

Additional context
Add any other context about the problem here.

[whiphi92]

Platform:

Simulation platform: Eagle
BuildStockBatch version, branch, or sha: [develop] Merge pull request #251 from NREL/os330
resstock or comstock repo version, branch, or sha: origin/resilience
Local Desktop OS: Windows

[afontani]

@whiphi92: Is this issue fixed?

[whiphi92]

@afontani I have not ran into it again - @nmerket do you know?