Unable to do SNAP Workflow with Pyrate

[pbrotoisworo]

Describe the bug
SNAP workflow is incomplete and doesn't represent GAMMA format properly. Some discussion on SNAP forums here. I've been using your repo's gamma test data as a reference as well as the discussion in the forum.

For now, I'm testing PyRate on Google Colab due to current storage limitations with my Windows workstation. I process the SLC data on my Windows machine then upload the processed data to a Colab instance. PyRate on Colab seems to have compiled correctly. I run python3 -m pytest tests/test_timeseries.py and it shows it passes the tests.

I'm testing with a very small dataset (3 images total) just so it's quicker to upload to Colab and debug. After much tinkering I only got as far as prepifg. It cannot generate one of the coherence files even though I can see the input .coh file and it is in the coh_list file which I inputted in the conf file. pyrate prepifg subsequently fails because it cannot find the missing .coh file.

I'm wondering if I can get some insight into how to fix this? I hope to create and share a SNAP workflow for PyRate.

Cheers.

To Reproduce

SNAP PROCESSING

1. Apply Orbit
2. TOPSAR Split
3. Backgeocoding
4. Interferogram
5. Deburst
6. Topophase Removal
7. Multilook
8. Filter
9. SNAPHU Export
10. Unwrapping
11. SNAPHU Import
12. Terrain Correction
13. Save DEM as GAMMA; Save Unw phase, coherence as GAMMA for PyRate

ADDITIONAL FIXING THAT I NEEDED TO DO
I have a Python script that automates the following items:

14. Rename Unw Phase .rslc file to {reference_date}-{secondary_date}.unw
15. Rename coherence .rslc file to {reference_date}-{secondary_date}.coh
16. Add datum and ellipsoid data to DEM.par file
17. Add missing data to slc header files. There is only frequency, incident_angle, and date.
18. Add missing data to DEM header file. Including origin coords, pixel spacing in decimal degrees, crs information.

Expected behavior
Use SNAP to export as "Gamma for PyRate" format and then use with PyRate with no problems.

Screenshots

Input data for Pyrate using a custom workflow:

Desktop (please complete the following information):

• OS (PyRate): Ubuntu 18.04.5 LTS (Google Colab)
• OS (SNAP): Windows 8
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context

Log for conv2tif:

18:49:33 main:105 2715 INFO 0/0 Verbosity set to INFO.
18:49:33 shared:1400 2715 INFO 0/0 Running 'conv2tif' step in serial
18:49:33 conv2tif:78 2715 INFO 0/0 Converting input interferograms to geotiff
18:49:33 conv2tif:87 2715 INFO 0/0 Running geotiff conversion in serial
18:49:33 conv2tif:115 2715 WARNING 0/0 Full-res geotiff already exists in out/interferogram_dir/20210607-20210619_unw.tif! Returning existing geotiff!
18:49:33 conv2tif:115 2715 WARNING 0/0 Full-res geotiff already exists in out/interferogram_dir/20210619-20210701_unw.tif! Returning existing geotiff!
18:49:33 conv2tif:115 2715 WARNING 0/0 Full-res geotiff already exists in out/coherence_dir/20210619-20210701_coh.tif! Returning existing geotiff!
18:49:33 conv2tif:115 2715 WARNING 0/0 Full-res geotiff already exists in out/geometry_dir/elevation_rslc.tif! Returning existing geotiff!
18:49:33 conv2tif:69 2715 INFO 0/0 Finished 'conv2tif' step

Log for prepifg

18:49:35 main:105 2730 INFO 0/0 Verbosity set to INFO.
18:49:35 shared:1400 2730 INFO 0/0 Running 'prepifg' step in serial
18:49:35 gdal_python:247 2730 INFO 0/0 Writing geotiff: out/interferogram_dir/20210607-20210619_ifg.tif
18:49:35 gdal_python:247 2730 INFO 0/0 Writing geotiff: out/interferogram_dir/20210619-20210701_ifg.tif
18:49:36 gdal_python:247 2730 INFO 0/0 Writing geotiff: out/geometry_dir/dem.tif
Traceback (most recent call last):
  File "/usr/local/bin/pyrate", line 33, in <module>
    sys.exit(load_entry_point('Py-Rate==0.6.0', 'console_scripts', 'pyrate')())
  File "/usr/local/lib/python3.7/dist-packages/pyrate/main.py", line 111, in main
    prepifg.main(params)
  File "/usr/local/lib/python3.7/dist-packages/pyrate/prepifg.py", line 102, in main
    do_prepifg(process_ifgs_paths, exts, params)
  File "/usr/local/lib/python3.7/dist-packages/pyrate/prepifg.py", line 172, in do_prepifg
    _prepifg_multiprocessing(m_path, exts, params)
  File "/usr/local/lib/python3.7/dist-packages/pyrate/prepifg.py", line 304, in _prepifg_multiprocessing
    coherence_thresh=coherence_thresh)
  File "/usr/local/lib/python3.7/dist-packages/pyrate/core/prepifg_helper.py", line 199, in prepare_ifg
    out_driver_type=driver_type, hdr=header, coherence_path=coherence_path, coherence_thresh=coherence_thresh
  File "/usr/local/lib/python3.7/dist-packages/pyrate/core/gdal_python.py", line 194, in crop_resample_average
    src_ds, src_ds_mem = _setup_source(input_tif)
  File "/usr/local/lib/python3.7/dist-packages/pyrate/core/gdal_python.py", line 329, in _setup_source
    src_ds = gdal.Open(input_tif)
  File "/usr/local/lib/python3.7/dist-packages/osgeo/gdal.py", line 3019, in Open
    return _gdal.Open(*args)
RuntimeError: out/coherence_dir/20210607-20210619_coh.tif: No such file or directory

[adeane-ga]

Thanks for submitting the issue @pbrotoisworo, this helps.

Regarding the SNAP output file name issue, where you had to manually change the files, we will pass this on to the SNAP developers for a change.

On the other issue you are having regarding the coherence file not being found, we will work on this as we can.

But could I request some information from the processing configuration you had?

1. The .conf file.
2. Your coherence.list file (the list of input coherence file paths)
3. Also a print out of your coherence file directory.

Thanks.