New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory errors with refactored code #1580
Comments
I also suspect that the I'll want to keep at the least the fit residuals and the return status message. I'll remove the rest (perhaps as an option since I think your use case is probably on the extreme end). Some people may want all the fit info details. |
Just curious -- what |
11x11. If I switch to 5x5, I'd roughly expect to get to 4x more sources... |
...assuming only one footprint, of course, which is probably an underestimate |
Reducing |
I'm trying with a hack, changing: fit_info = self.fitter.fit_info.copy() to fit_info = {key: self.fitter.fit_info.get(key)
for key in
('param_cov', 'fvec', 'fun', 'ierr', 'status')
} |
I did some testing, and I don't think the My next suspect is the PSF model. Are you using a |
Could you please send me your input PSF model? |
yes, I'm using a webbpsf model. Can be reproduced with: import webbpsf
obsdate = '2022-08-28'
nrc = webbpsf.NIRCam()
nrc.load_wss_opd_by_date(f'{obsdate}T00:00:00')
nrc.filter = 'F405N'
nrc.detector = 'NRCA5'
grid = nrc.psf_grid(num_psfs=16, all_detectors=False, verbose=True, save=True)
psf_model = grid I think... I haven't tested this; in production, the obsdate and some other variables come from FITS headers EDIT: tested, this works now. |
Thanks. Your PSF model is ~20 MB. 12_000 of them is ~233 GB (just for the PSF models, not the data, results, etc.). So that seems to be the culprit. The code returns a copy of the fit models. But it's copying the entire model. For the |
@keflavich #1581 should fix your memory issues with |
Thanks. Past 15k already, so it looks like an improvement. |
Hm, still died, but got a lot further:
Any idea for further workarounds? Splitting up the image sounds like a possible, but very annoying, way to get around this. Increasing memory isn't really practical |
@larrybradley I'd recommend reopening this one; it's not fully solved. |
Yes, I'm working on some improvements now. |
Thanks. I'll test 'em right away! |
#1586 is another big reduction in memory for |
I had 2 of 3 filters work, and pretty fast! One still failed:
Notably, this is at a later stage, so maybe this is solvable by other means |
ok, I thought they had completed, but it looks like all runs failed somewhere in the |
Looking at the source code for Here's a record of my failures:
For context, I'm first running a basic PSFPhotometry run, then subsequently an I was running IterativePSFPhotometry with:
so maybe I can shrink the background area a bit and see if it completes. |
ah, another data point: I was making the model image (residual image) with 11x11 patches, not 5x5. |
I don't how I also reduced the memory footprint of Are you using source grouping and have very large groups? I'm wondering if that could be an issue. Large groups should be avoided because it requires fitting a very large multi-dimensional parameter space (which can be slow and error prone, and probably memory intensive). |
No, I disabled the grouper, so it's not source grouping. I'll see if this works better now, post #1604. |
As noted in another thread, I'm consistently getting out-of-memory errors when running the new
PSFPhotometry
fitter.My fitting runs have died at the following stages as ID'd by the progressbar:
These are pretty consistent endpoints.
I suspect the problem is that
fit_info
is being stored in memory. iirc,fit_info
includes at least one, and maybe several, copies of the data. Can we minimized thefit_info
before storing it? I think onlyparam_cov
is used downstream?Note that I have 256GB of memory allocated for these runs, which imo is a very large amount to dedicate to photometry of a single JWST field-of-view.
The text was updated successfully, but these errors were encountered: