You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
This bit of code in the postprocessing doesn't have a switch on it and we really really truly need these parquet files for vizstock in particular. The aggregated timeseries files that can be produced from them are useless to us at this point for production runs of ComStock, and soon ResStock.
Describe the solution you'd like
A flag to disable this behavior.
Describe alternatives you've considered
Making the flag default to not clean up, but I think that's not really reasonable...
Additional context
I've spend over 160 hours 'fixing' this for the V1 ComStock run by re-processing over a PB of double-zipped data. I look forward to never doing this ever again.
The text was updated successfully, but these errors were encountered:
This "fix" seems easy enough, but I'd be interested to understand better what you're trying to do first to see if there's a better way to address the issue.
And, yeah, we really should fix that double zip nightmare.
Is your feature request related to a problem? Please describe.
This bit of code in the postprocessing doesn't have a switch on it and we really really truly need these parquet files for vizstock in particular. The aggregated timeseries files that can be produced from them are useless to us at this point for production runs of ComStock, and soon ResStock.
Describe the solution you'd like
A flag to disable this behavior.
Describe alternatives you've considered
Making the flag default to not clean up, but I think that's not really reasonable...
Additional context
I've spend over 160 hours 'fixing' this for the V1 ComStock run by re-processing over a PB of double-zipped data. I look forward to never doing this ever again.
The text was updated successfully, but these errors were encountered: