Subset HyP3 datasets and create horz and vert displacement on Jupyter notebook

[suatcoskun]

I am using HyP3 and Jupyter notebook for MintPy processing. I prepared my results for both track ( asc and desc). But I did not crop these datasets for the same aoi because of Jupyter notebook selected aoi with manual box we can not enter coordinates. Then I tried to run asc_desc2horz_vert.py I faced lookup file problem . As you know if you are using Jupyter notebook with HyP3 datasets MintPy didn't create lookup file because of input datasets have already geocoded. Do you have any advice for subset HyP3 datasets for known coordinates and create horz and vert displacement on Jupyter notebook?

Best
Suat

[welcome[bot]]

👋 Thanks for opening your first issue here! Please filled out the template with as much details as possible. We appreciate that you took the time to contribute!
Make sure you read our contributing guidelines.

[yunjunz]

I am not super familiar with notebooks from various sources, please post your detailed questions.

MintPy is well documented on the code level. I would recommend: locating the API in the code, looking at comments next to the code for example usage.

[suatcoskun]

@yunjunz

If I am giving some extra explanation about my question; I am using Hyp3 on demand datasets for processing. ASF’s opensarlab jupyter notebook is very useful tool for MintPy. When I am preparing stacks I used “Prepare_HyP3_InSAR_Stack_for_MintPy.ipynb” . As you know in step 3 “Subset the Stack and Cleanup Unused Files” you can subset or crop your datasets. At this point when your subseting the timeseries to own AOI you can use area of interest selector (interactive plot) so you can not input specific coordinates. This situation caused to create different boundaries aoi for descending and ascending orbits. At the end of the processing when you are applying asc_desc2horz_vert.py code i took a error massage about for unmatched aoi boundaries. After that I used for data with different spatial resolution and coverage geocode.py but ıt dosent work for me because of I have no geometryradar.h5 and lookup files. As I know if you are using Jupyter notebook with HyP3 datasets MintPy didn't create lookup file because of input datasets have already geocoded.

In a result I have two question the first one is;
İs it possible to crop desc and asc orbits for the same specific boundaries with well known coordinates ASF’s opensarlab jupyter notebook?

The second question is;
Do you know another way to create horz and vertical displacement in ASF’s opensarlab jupyter notebook?

Thank you for interesting…

Best
Suat

[yunjunz]

This sounds like a question for the hyp3 team, please contact them for more constructive suggestions.

[jhkennedy]

This notebook is maintained by the OpenScienceLab team. @Alex-Lewandowski, can your team take a look? 🤔 We now have at least three different HyP3+Mintpy notebooks so we should pick back up #1004

[jhkennedy]

Here's the notebook in question for anyone interested:
https://github.com/ASFOpenSARlab/opensarlab-notebooks/blob/master/SAR_Training/English/Master/Prepare_HyP3_InSAR_Stack_for_MintPy.ipynb

[Alex-Lewandowski]

Hi @suatcoskun , (@yunjunz and @jhkennedy, please correct me if I am wrong) I do not believe you can combine ascending and descending orbits in the same SBAS analysis because you'd end up with much too large of a perpendicular baseline. We should mention that in the notebooks.

That issue aside, the subsetting code should allow you to subset a stack with a mix of orbit directions to the same AOI, so there is likely an issue in the code. I will take a look at this today.

[Alex-Lewandowski]

@suatcoskun I've have been unable to duplicate the issue you are facing. Would you please provide some more information on your data set and workflow?

Did you run through the notebook twice, once for each orbit direction, downloading everything to the same directory and subsetting?

I could see that causing subsetting issues if the datasets were in different UTM zones because of how the notebook globs all the tiffs in the data directory. The first data set would potentially be reprojected and subset multiple times, going back and forth between lat/lon and 2 different UTM zones.

I think the easiest way to get to the bottom of this is to look at your datasets in both orbit directions with an understanding of where you downloaded and subset them.

It is now possible to share your HyP3 data with other Hyp3 users. All you need is a HyP3 username and project name. If your data have not yet expired, please provide your HyP3 username and the name of the HyP3 project you are having trouble running through the notebook. You can email me directly at aflewandowski@alaska.edu to avoid publicly posting those details.

[suatcoskun]

Hi @Alex-Lewandowski , I am currently working on a landslide with seasonal HyP3 datasets. I have processed these datasets on Opensarlab Jupyter notebook. I used your notebooks to create the time series analysis (prep_hyp3_mintpy.ipynb and MintPy_Time_Series.ipynb) for ascending orbits, I have prepared data sets between June 2018 and November 2023 in path 145. And for descending orbit, the same parameters were set but the path was chosen as 50. I can not share my datasets with you because I do not have an active project.
I went through the notebook for each orbit and created time series. But I created a different subfolder for each orbit. In my opinion, prep_hyp3 notebooks do not allow us to subset ascending and descending orbit directions for the same AOI because of we used an interactive selector, but I think we need wkt, shp or geojson input options to crop stacks.
One more explanation about my workflow: After completing all the steps in the notebook, I used a terminal to create (.kml) time series and (.shp) exports for Qgis. (e.g.: save_kmz_timeseries.py and save_qgis.py) But when I tried to run asc_desc2horz_vert.py in the terminal with the necessary inputs, I got this error message;

I sent my jupyter notenook accounts to your email adress. When your are sign in notebook you will see devre_asc and devre_desc folders for the same aoi.

Best
Suat

[Alex-Lewandowski]

Hi @suatcoskun,

Thank you for providing those details. I accessed your server, and it looks like you have copied the entire notebook repo into /home/jovyan/tutorials, and are working out of that location.

The original clone of that repo, located in /home/jovyan/notebooks is managed with nbgitpuller, and automatically synced with the remote repo every time you re-start your OSL server. The copy of the repo that you made is never being updated, and so you are working with outdated code in an outdated conda environment.

In the past month and a half, there have been significant changes to the OSL MintPy notebooks and the conda environment used to run them.

To move forward, I suggest that you:

1. Reprocess your HyP3 projects to start with un-subset data in their original projections
   1. You can request a short-term HyP3 quota increase, if needed
2. Build the new osl_mintpy conda environment using the /home/jovyan/conda_environments/Create_OSL_Conda_Environments.ipynb notebook
   1. I recommend selecting the locked version of the environment when presented with the option
3. Run the regularly synced, up-to-date, clones of the MintPy notebooks located in the /home/jovyan/notebooks/SAR_Training/English/Master directory

If you continue having trouble after that, please let us know. It is possible that there is something else causing the trouble, but working with an outdated environment and notebooks are likely culprits.

Your suggestion to add additional options for selecting an AOI is a great idea! I will add a story to our work backlog to make that update. The team may not have a chance to implement this before February.

[suatcoskun]

Thank you @Alex-Lewandowski for the warning about folder navigation to process datasets. I will pay attention to this point. I worked out a little bit more my datasets to create horizontal and vertical displacement and I tried to test your suggestion. Finally, I understood asc_desc2horz_vert.py was created by ISCE outputs and designed for different workflow. If you tried to run this code for Hyp3 datasets it doesn't work because some outputs don't exist in HyP3. Maybe at a later time code will designed again and will be suitable for HyP3.

Best..
Suat

[ditafaith]

Hi @Alex-Lewandowski @yunjunz
I also work with HyP3 products and continue the process using MintPy in my PC.
The problem is converting UTM to lat/lon before stack processing. If I want to geocode, MintPy asked the lookup file, which I don't know how to obtain from HyP3.
I would be glad if you can give me a hint for using hyp3 products for mintpy.