[best practices] Processing by orbit #81
Comments
|
Hi Ivan, Your own validation results will be very welcome in this regard, thanks a lot ! It is quite easy to run start-maja after separating the swaths in different folders before launching start-maja, but we should think of implementing what you suggest as an option. Our development pipeline is quite full, so it might take a while. Best regards, |
|
Hi Olivier and thanks for the swift reply, you are right, I should have been more specific. Our remarks refer to cloud detection accuracy.
This is reasonable and we want to take advantage of this. However, this case is about a tile where most of the time the different orbits come one after the other when sorted by date-time (R050 > R093 > R050 > R093 > R050 > R050 > R093 > etc). The important part is that one of the orbits has half the area as NODATA and the other is full. (I see that MAJA in NOMINAL mode uses a single previous product. Could it use more?)
We already have an L1C set separated in different ways (year, tile, etc) but now we added the orbit, too. We are now running MAJA for each orbit separately and will come back with results. PS: The specific tile we are experimenting with is 34SFJ and the two orbits are R050 and R093. |
|
To answer your question, MAJA uses only one previous level 2 product, but this product conveys information from a lot of previous images. We use a composite image at 240m resolution, which for each pixel contains the most recent cloud free observation, and it is used as a reference to compare with the new image to classify. This composite is updated at each new image with the new cloud free pixels, and stored in the level 2 (the private part). As a result, if each half of the image is observed on a different swath, it should not be an issue. |
|
Dear Olivier, catching up to my colleague @c00kiemon5ter, we would like to inform you that our experiments on tile 34SFJ (for year 2019), indicated that using both orbits at the same MAJA run, indeed gave better results for most scenes, concerning cloud detection. However the first remark that Ivan pointed out, referring to much better cloud detection results on another tile (34SEG) is still valid and after further investigation we think that this can be directly associated to differences of the image's landscape. In particular 34SFJ includes the largest agricultural plain of Greece, i.e., great heterogeneity in dark/bright values throughout the year and frequent alternation between those in just a few meters. MAJA 4.2.1 gives us a lot of false positive pixels classified as clouds, that are actually clear (cloud-free) observations of crop parcels, even in (almost) cloud-free images. e.g. True color composite of T34SFJ_20191016 & detected clouds in red (cloud mask >0) from CLM_R1 product of MAJA 4.2.1. What is also interesting to point out is that the false-positive issue concerning agricultural areas is not observed in that extent in some results we produced using MAJA 3.2.2 version (via Sen4Cap). It seems that older versions produce cloud mask at a coarser resolution and though this comes with a worse and expanded delineation result of the cloud figures, at the same time there are significantly fewer issues concerning these false positives (salt & pepper) pixels detected as clouds. e.g. Cloud masks from versions 3.2.2 & 4.2.1 on a True color composite of T34SFJ_20190718. We read here (https://labo.obs-mip.fr/multitemp/maja-4-2-is-available-and-open-source/) that 4.2 version gives the possibility of processing cloud detection at 120m (instead of 240m), is this the case when using ''start-maja'' for 4.2.1 version? We have also observed a significant difference in the size of CLM products between 4.2 and 3.2 versions. Thank you in advance. Best Regards, |
|
Thanks Christina. |
|
We run MAJA v4.2.1 (official binary release) on Ubuntu 18.04 (a Vagrant VM), in a dedicated miniconda environment setup with The command we use after having activated the conda environment is the following: $ cat config.ini
[Maja_Inputs]
exeMaja=/opt/maja/v4.2.1/bin/maja
repWork=/data/work
repGipp=/data/gipp
repMNT=/data/dtm
repL1=/data/L1C
repL2=/data/L2A
#repCAMS=/data/CAMS
[DTM_Creation]
repRAW=/data/dtm/raw
repGSW=/data/dtm/gswI will later upload the vagrant recipe and let you know, so that you can replicate the exact environment.
Does this mean that MAJA v4.2 uses 120m resolution to create the cloud masks by default (while previous releases -v3.2.2 in particular- used 240m)?
If there is something we can help with, let us know and we can have a run with different params/settings. |
|
Dear @c00kiemon5ter |
|
Olivier and Jerome Greetings from Athens, May I ask if you had any news regarding the false positives overestimation for the higher resolution Cloud Mask for v 4.2 ? Thank you once again, |
|
Dear Christina, At line 234: In the meantime, we found that some other parameters in the GIPPs are not well adapted to the 120m resolution (which affect the reflectances rather than the cloud mask). For now, we'd recommend you to keep processing products at 240m, and we'll reset this 240 resolution as the default one until the release of Maja 4.4 (we will have fixed all this issues until then). Hope this help, |
|
Hello @jerome-colin (also, following up from #82 (comment)) Only The same holds for this repository, searching for So, I think we are already processing at 240m resolution. Are we missing some other configuration option? |
|
Hi @c00kiemon5ter, You may still guess the resolution of your DTM from the size of some files. As an example, for your 34SFJ tile, the file That's obviously not satisfying, and we'll modify the code to let the user set this coarse resolution explicitly (eg. add a --L2CoarseResolution option to startmaja), probably for release 4.4. Jerome |
Hello, I want to ask a question regarding best practices around processing tiles and specifically focusing on tile-orbits.
We made a small run of MAJA and processed some L1C products and got great results. Processing a different tile we got less accurate results. We went back to see what had happened and what came to our mind is the mix of the orbits. The first experiment was on a tile that happened to always have products be on the same orbit, but the second tile had mixed orbits.
Sen4CAP, using MAJA as the processor to produce L2A products, internally separates tiles per orbit and uses a tile of the same orbit as the previous product.
With those observation, would it be better to separate processing per tile and orbit or let Maja pick any tile as a previous product, regardless of the tile orbit?
We are running a test case for that second experiment, but this time with separate L1C products per orbit. I will come back with some results and new observations, but your experience and observations would certainly be valuable here.
PS: I am using MAJA v4.2.1 through StartMaja
On v4.2.1, StartMaja will just get the closest product to be used as the previous. This seems like a good place to implement a better heuristic that can pick a previous product based on some strategy (ie, the closest of the last 4 with the least cloud-coverage.)
https://github.com/CNES/MAJA/blob/4.2.1/StartMaja/Chain/Workplan.py#L259-L266
The text was updated successfully, but these errors were encountered: