How to set ISCE stackSentinel.py parameters in order to run MiaplPy? #60

leeyhgeo · 2023-08-14T18:16:40Z

Hello,

I used to analyze Sentinel1 images with MintPy and I found that MiaplPy seems to need different input. However, I'm confused about how to set stackSentinel.py parameters so that I can get coregistered SLC data in full resolution. Is there any document/discussion that could be referenced? (I can hardly find one so far)

Thank you

mirzaees · 2023-08-17T16:30:15Z

Hi @geoglee
MiaplPy can read data from 'interferogram' and 'slc' workflow of stackSentinel.py

If your current directory is the project directory, then setting the autoPath to yes would read all data automatically.
You might need to rename your template to reflect your project, for example if your project directory name is 'testProject' then rename your template to testProject.template:

./testProject/testProject.template

MiaplPy/miaplpy/defaults/miaplpyApp.cfg

Line 22 in 3771804

miaplpy.load.autoPath = auto # [yes, no] auto for no

Also you can manually set path to data as bellow in your template file:

MiaplPy/miaplpy/defaults/auto_path.py

Lines 21 to 38 in 3771804

    
           miaplpy.load.metaFile       = ${PROJECT_DIR}/reference/IW*.xml 
        
           miaplpy.load.baselineDir    = ${PROJECT_DIR}/baselines 
        
           miaplpy.load.slcFile        = ${PROJECT_DIR}/merged/SLC/*/*.slc.full 
        
           miaplpy.load.unwFile        = ${WORK_DIR}/inverted/interferograms_${int_type}/*/fi*.unw 
        
           miaplpy.load.corFile        = ${WORK_DIR}/inverted/interferograms_${int_type}/*/fi*.cor 
        
           miaplpy.load.connCompFile   = ${WORK_DIR}/inverted/interferograms_${int_type}/*/*.unw.conncomp 
        
           miaplpy.load.ionoFile       = None 
        
           miaplpy.load.intFile        = None  
        
           miaplpy.load.demFile        = ${PROJECT_DIR}/merged/geom_reference/hgt.rdr.full 
        
           miaplpy.load.lookupYFile    = ${PROJECT_DIR}/merged/geom_reference/lat.rdr.full 
        
           miaplpy.load.lookupXFile    = ${PROJECT_DIR}/merged/geom_reference/lon.rdr.full 
        
           miaplpy.load.incAngleFile   = ${PROJECT_DIR}/merged/geom_reference/los.rdr.full 
        
           miaplpy.load.azAngleFile    = ${PROJECT_DIR}/merged/geom_reference/los.rdr.full 
        
           miaplpy.load.shadowMaskFile = ${PROJECT_DIR}/merged/geom_reference/shadowMask.rdr.full 
        
           miaplpy.load.waterMaskFile  = ${PROJECT_DIR}/merged/geom_reference/waterMask.rdr.full 
        
           miaplpy.load.bperpFile      = None

leeyhgeo · 2023-08-20T17:09:08Z

Thank you so much @mirzaees !
I've tried your suggestion but I ran into some path problems. I hope I could solve it soon.
At the same time, I'm using stackStripMap.py to process ERS and Envisat data.
Is it possible as well to run products from stackStripMap.py in MiaplPy?

mirzaees · 2023-08-20T18:20:20Z

you are very welcome @geoglee
I may be able to help you if you send the error you get and your template

Regarding stripmap, yes, I have run it for TerraSAR data before and it worked
however I have never tested ERS or Envisat, let me know if you encounter a problem

Jesse-kearse · 2023-09-24T23:49:05Z

Hello

I have been using MintPy and MiaplPy, and have managed to get good results. Recently, I have had an issue when loading an SLC stack.

The SLC stack was generated with the following parameters:
stackSentinel.py -s /insar/raw/track_08/ -d /insar/raw/dem/dem.wgs84 -b '-38.16 -37.53 176.07 177.73' -n '1 2' -m 20180112 -a ./aux/ -o ./orbits -C NESD -W slc

I ran the MiaplPy software with the suggested command "miaplpyApp.py kawerau.cfg --runfiles --dir ."

the prepc_slc_isce.py application crashes, and I am unsure why. I thought it may be due to

"miaplpy.subset.lalo" setting, so I tried with various extents and with no subset, and it was the same result
"miaplpy.load.startDate" or "maiplpy.load.endDate" so I tried with and without dates and the result was the same.

The only difference I could think of between this SLC stack and other SLC stack that worked fine with miaplpy, is co-registration method: "NESD" was used here, and I had used "geometry" in the past with the SLC stack that were loaded into Miaplpy without issue. Perhaps the the choice of number of swaths could be a factor? I am stuck as to the issue.

Below is the report of the run, is there anything else I could provide to help diagnosing the issue?

Thanks for any help

--RUN-at-2023-09-25 11:35:24.796217--
Current directory: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy
Run routine processing with arg_parser.py on steps: ['load_data', 'phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction']
Remaining steps: ['phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction']
Project name: kawerau

20230925:113524 * miaplpyApp.py kawerau.cfg --runfiles --dir .
Project name: kawerau
Go to work directory: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy
copy default template file /home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/defaults/smallbaselineApp.cfg to work directory
read custom template file: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy/kawerau.cfg
update default template based on input custom template
No new option value found, skip updating /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy/smallbaselineApp.cfg
copy kawerau.cfg to inputs directory for backup.
copy smallbaselineApp.cfg to inputs directory for backup.
copy kawerau.cfg to pic directory for backup.
copy smallbaselineApp.cfg to pic directory for backup.
read default template file: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy/smallbaselineApp.cfg
copy default template file /home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/miaplpy/objects/../defaults/miaplpyApp.cfg to work directory
update default template based on input custom template
miaplpy.multiprocessing.numProcessor: auto --> 40
miaplpy.load.processor: auto --> isce
miaplpy.load.autoPath: auto --> yes
miaplpy.load.slcFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/SLC//.slc.full
miaplpy.load.startDate: auto --> 20180112
miaplpy.load.endDate: auto --> 20200114
miaplpy.load.metaFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/reference/IW.xml
miaplpy.load.baselineDir: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/baselines
miaplpy.load.demFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/hgt.rdr.full
miaplpy.load.lookupYFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/lat.rdr.full
miaplpy.load.lookupXFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/lon.rdr.full
miaplpy.load.incAngleFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/los.rdr.full
miaplpy.load.azAngleFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/los.rdr.full
miaplpy.load.shadowMaskFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/shadowMask.rdr.full
miaplpy.load.waterMaskFile: auto --> None
miaplpy.load.unwFile: auto --> ./inverted/interferograms_single_reference//fine.unw
miaplpy.load.corFile: auto --> ./inverted/interferograms_single_reference//fine*.cor
miaplpy.load.connCompFile: auto --> ./inverted/interferograms_single_reference//.unw.conncomp
miaplpy.subset.lalo: auto --> -38.117:-37.995,176.651:176.913
mintpy.load.processor: auto --> isce
copy miaplpyApp.cfg to inputs directory for backup.
read default template file: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy/miaplpyApp.cfg
SAR platform/sensor : None
processor: isce
check auto path setting for Univ of Miami users for processor: isce

prepare metadata files for isce products
prep_slc_isce.py -s /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/SLC -f .slc.full -m /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/reference/IW1.xml -b /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/baselines -g /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference --force
extract metadata from ISCE/topsStack xml file: /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/reference/IW1.xml
extract metadata from geometry files: ['hgt.rdr.full', 'lat.rdr.full', 'lon.rdr.full', 'los.rdr.full']
extract metadata from geometry files: ['hgt.rdr', 'lat.rdr', 'lon.rdr', 'los.rdr']
Traceback (most recent call last):
File "/home/kearsejess/conda/envs/mintpy_new/bin/prep_slc_isce.py", line 8, in
sys.exit(main())
^^^^^^
File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/miaplpy/prep_slc_isce.py", line 362, in main
metadata = extract_isce_metadata(inps.metaFile,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/miaplpy/prep_slc_isce.py", line 145, in extract_isce_metadata
metadata_multi_looked = isce_utils.extract_geometry_metadata(geom_dir, metadata_multi_looked)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/utils/isce_utils.py", line 559, in extract_geometry_metadata
data = readfile.read(geom_file, box=box)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/utils/readfile.py", line 310, in read
data, atr = read_binary_file(fname, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/utils/readfile.py", line 679, in read_binary_file
data = read_binary(
^^^^^^^^^^^^
File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/utils/readfile.py", line 1914, in read_binary
data = np.fromfile(fname, dtype=data_type, count=count).reshape(-1, widthnum_band)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: cannot reshape array of size 15521796 into shape (43903)

Jesse-kearse · 2023-09-26T02:52:30Z

P.S. I am running the latest version of MiaplPy V0.2.1, which was downloaded ~20092023

mirzaees · 2023-09-26T22:28:49Z

Hi @Jesse-kearse
Looks like your program crashes when reading geometry files. One thing that comes to my mind is the *.rsc files, If you have run mintpy before this, it has created *.rsc metadata in the geometry directory which are associated with multilooked files. now you are reading the full res and miaplpy tries to create *.rsc for full res but it first checks whether they exist in the directory. So please delete them (all *.rsc files in merged/geom_reference directory and merged/SLC/dates/* directory) and try again. let me know if that solves the issue

Jesse-kearse · 2023-09-26T22:31:26Z

Hi @mirzaees , yes I had run mintpy on the same SLC stack prior to miaplpy. Thanks for your advice! I will try your suggestion and let you know how it goes.

Cheers

Jesse-kearse · 2023-09-26T22:58:53Z

hi @mirzaees , Actually, I had not run mintpy on this particular SLC stack, and there were no .rsc files in the /merged/geom_reference/ dir or the /merged/SLC/*/ directory. Any other ideas?

Thanks again for your help.

mirzaees · 2023-09-27T01:04:27Z

Hi @Jesse-kearse,
I need a subset from your data as a test to run it and see if I can reproduce your error. Can you provide me your merged folder with geom-reference, and 2 dates in SLC and reference+baseline folders in a zipfile somewhere?

gjustin40 · 2023-09-27T03:47:27Z

I also have a similar issue. @mirzaees
I created a stack using ISCE2, and ultimately, the problem is that the slc.full data is missing.

miaplpy.load.slcFile        = ../merged/SLC/*/*.slc.full  #[path2slc_file]

After running ISCE2 command, the slc.full file was not generated.
The ISCE2 command I executed is as follows:

dem.py -a stitch -b {bbox lat/lon} -r -s 1 -c
stackSentinel.py -s SLC/ -d DEM/*.dem.wgs84 -a AuxDir/ -o Orbits/ -b {bbox lat/lon} -c 2
# run all run_files

I successfully used MintPy with the files generated by above command.

I know there are a lot of example of command to make run_files in ISCE,
What command should I execute in ISCE2 to generate slc.full files?

Thanks

Jesse-kearse · 2023-09-27T03:57:08Z

Hi @mirzaees

Thank you for offering to take a look.

I have created the a zip archive "SLC_zip.zip" that contains the files you requested. You can download with the following command:

rsync -av --progress rsync://kearsejess@core.geo.vuw.ac.nz/kearsejess/

password: gJbmvnmtUU

Thanks again.

mirzaees · 2023-09-27T22:33:27Z

Hi @Jesse-kearse
Thanks for providing your data. which workflow of stackSentinel.py have you used? if that is interferogram or slc, then it should work the way it is without any touching but from your data I realize you may have done some separate/post processing or the merging step was not complete. that is because your multilooked version of geometry files are not compatible with their shape in VRT. Also if you display them with view.py of mintpy, you get a similar error. Anyways I removed that part of the code that relies on multilooked version and should not be a problem any more, just reinstall with:

cd MiaplPy; git pull
python -m pip install .

one more issue is about the baselines you provided which seems to be a raster file for each date. however, miaplpy assumes text files in the following format if your reference date is 20180112:

20180112_20180301/20180112_20180301.txt
20180112_20180313/20180112_20180313.txt

I don't know why your baselines are like this but stackSentinel.py should create those text files. You will still get error if your baselines are not in that format

mirzaees · 2023-09-27T22:37:57Z

Hi @gjustin40
As long as the full res VRT files exist and you have not deleted the other folders in the project, you are fine and it should be able to read them. However, as I mentioned above, if you have run mintpy before miaplpy, clean all .rsc metadata and run miaplpy again

Jesse-kearse · 2023-09-27T23:03:08Z

Thanks for taking the time to look at the files, @mirzaees , I really appreciate your help.

The workflow I used was SLC: "stackSentinel.py -s /insar/raw/track_08/ -d /insar/raw/dem/dem.wgs84 -b '-38.16 -37.53 176.07 177.73' -n '1 2' -m 20180112 -a ./aux/ -o ./orbits -C NESD -W slc"

I realise now that the post processing that I had done with StaMPS (which includes a multi-looking step) was probably the reason for the modified geometry files. But with your recent update this will no longer be an issue! thanks, I will git pull and re-install.

The baseline files I sent were from ./merged/baselines/ dir. Sorry I misinterpreted which baseline information you wanted. The files in ./baselines/ dir are in the correct format: 20180112_20180301/20180112_20180301.txt, so that should be fine.

I will let you know how it goes.

Thanks again.

Jesse-kearse · 2023-09-27T23:27:39Z

Following git pull and re-installation it was a successful completion of "miaplpyApp file.cfg --runfiles --dir ."

Thanks @mirzaees

mirzaees · 2023-09-28T00:25:47Z

You are very welcome @Jesse-kearse
I will close this issue

leeyhgeo · 2023-10-24T17:29:31Z

Hi @geoglee MiaplPy can read data from 'interferogram' and 'slc' workflow of stackSentinel.py

If your current directory is the project directory, then setting the autoPath to yes would read all data automatically. You might need to rename your template to reflect your project, for example if your project directory name is 'testProject' then rename your template to testProject.template:

./testProject/testProject.template

MiaplPy/miaplpy/defaults/miaplpyApp.cfg

Line 22 in 3771804

miaplpy.load.autoPath = auto # [yes, no] auto for no

Also you can manually set path to data as bellow in your template file:

MiaplPy/miaplpy/defaults/auto_path.py

Lines 21 to 38 in 3771804

miaplpy.load.metaFile = ${PROJECT_DIR}/reference/IW*.xml

miaplpy.load.baselineDir = ${PROJECT_DIR}/baselines

miaplpy.load.slcFile = ${PROJECT_DIR}/merged/SLC/*/*.slc.full

miaplpy.load.unwFile = ${WORK_DIR}/inverted/interferograms_${int_type}/*/fi*.unw

miaplpy.load.corFile = ${WORK_DIR}/inverted/interferograms_${int_type}/*/fi*.cor

miaplpy.load.connCompFile = ${WORK_DIR}/inverted/interferograms_${int_type}/*/*.unw.conncomp

miaplpy.load.ionoFile = None

miaplpy.load.intFile = None

miaplpy.load.demFile = ${PROJECT_DIR}/merged/geom_reference/hgt.rdr.full

miaplpy.load.lookupYFile = ${PROJECT_DIR}/merged/geom_reference/lat.rdr.full

miaplpy.load.lookupXFile = ${PROJECT_DIR}/merged/geom_reference/lon.rdr.full

miaplpy.load.incAngleFile = ${PROJECT_DIR}/merged/geom_reference/los.rdr.full

miaplpy.load.azAngleFile = ${PROJECT_DIR}/merged/geom_reference/los.rdr.full

miaplpy.load.shadowMaskFile = ${PROJECT_DIR}/merged/geom_reference/shadowMask.rdr.full

miaplpy.load.waterMaskFile = ${PROJECT_DIR}/merged/geom_reference/waterMask.rdr.full

miaplpy.load.bperpFile = None

@mirzaees Sorry for the really late reply.

I run my stackSentinel.py with the following parameter stackSentinel.py -s ~/slc -o ~/orbits -a ~/S1_auxiliary_files -d ~/dem/dem.wgs84 -c 3 -z 3 -r 15 -b '22.3 24.4 119.9 120.6' and this is how it looks like under the folder ${PROJECT_DIR}/merged/SLC/*

I cannot find any files end with *.slc.full, and so as files under the geom_reference folder as below

I wonder if the problem arises from my multilooking settings of -z 3 -r 15 above, or maybe other issues.

Thanks!

mirzaees · 2023-10-26T17:24:16Z

@geoglee
I don't see any problem with your isce outputs,
as long as the vrt files exist in your folder, miaplpy will read the data.
you need to set up your template similar to the example above.
I need to see your error to be able to help you, can you run and send me your error message?

Jesse-kearse · 2023-10-26T20:14:23Z

@geoglee

If you only have the "merged" directory and sub directories, you may be missing the files that the .slc.full.vrt files are referencing. In my workflow, .slc.full.vrt files point to burst.slc.vrt files in the "coreg_secondarys" directory, one above "merged". The raw data is contained in the burst.slc files within the same directory. I ran into this issue by mistake when I deleted a whole bunch of files (to gain extra disc space) thinking I only needed the merged// files. You can check by opening the *.slc.full.vrt files and checking the path in

Jesse-kearse · 2023-10-26T20:15:43Z

...checking the path in SourceFilename relativeToVRT

leeyhgeo · 2023-10-28T16:55:03Z

@mirzaees @Jesse-kearse

Thank you a lot for helping me! For the path problem, awkwardly, I rechecked the template and just found that I didn't turn on the auto path function since I thought the default setting (auto) was yes till I noticed the comment behind miaplpy.load.autoPath. As I turn on the auto path function, Miaplpy seems to work right now (though it's still running). Hope it can operate normally till the final step. I'll report the outcome later.

Another question about the stackSentinel.py settings. I'm not sure whether multilooking (-z, -r) and num_connections (-c) will affect the Miaplpy processing results. Any ideas?

Besides, I already put the keyword "Sen" into the file name of my template, but the message still outputs SAR platform/sensor : None. How to set it properly?

mirzaees closed this as completed Sep 28, 2023

mirzaees reopened this Oct 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to set ISCE stackSentinel.py parameters in order to run MiaplPy? #60

How to set ISCE stackSentinel.py parameters in order to run MiaplPy? #60

leeyhgeo commented Aug 14, 2023

mirzaees commented Aug 17, 2023

leeyhgeo commented Aug 20, 2023

mirzaees commented Aug 20, 2023

Jesse-kearse commented Sep 24, 2023

Jesse-kearse commented Sep 26, 2023

mirzaees commented Sep 26, 2023 •

edited

Jesse-kearse commented Sep 26, 2023

Jesse-kearse commented Sep 26, 2023

mirzaees commented Sep 27, 2023

gjustin40 commented Sep 27, 2023

Jesse-kearse commented Sep 27, 2023

mirzaees commented Sep 27, 2023

mirzaees commented Sep 27, 2023

Jesse-kearse commented Sep 27, 2023

Jesse-kearse commented Sep 27, 2023

mirzaees commented Sep 28, 2023

leeyhgeo commented Oct 24, 2023 •

edited

mirzaees commented Oct 26, 2023

Jesse-kearse commented Oct 26, 2023

Jesse-kearse commented Oct 26, 2023

leeyhgeo commented Oct 28, 2023

How to set ISCE stackSentinel.py parameters in order to run MiaplPy? #60

How to set ISCE stackSentinel.py parameters in order to run MiaplPy? #60

Comments

leeyhgeo commented Aug 14, 2023

mirzaees commented Aug 17, 2023

leeyhgeo commented Aug 20, 2023

mirzaees commented Aug 20, 2023

Jesse-kearse commented Sep 24, 2023

Jesse-kearse commented Sep 26, 2023

mirzaees commented Sep 26, 2023 • edited

Jesse-kearse commented Sep 26, 2023

Jesse-kearse commented Sep 26, 2023

mirzaees commented Sep 27, 2023

gjustin40 commented Sep 27, 2023

Jesse-kearse commented Sep 27, 2023

mirzaees commented Sep 27, 2023

mirzaees commented Sep 27, 2023

Jesse-kearse commented Sep 27, 2023

Jesse-kearse commented Sep 27, 2023

mirzaees commented Sep 28, 2023

leeyhgeo commented Oct 24, 2023 • edited

mirzaees commented Oct 26, 2023

Jesse-kearse commented Oct 26, 2023

Jesse-kearse commented Oct 26, 2023

leeyhgeo commented Oct 28, 2023

mirzaees commented Sep 26, 2023 •

edited

leeyhgeo commented Oct 24, 2023 •

edited