Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ERROR (Start_MAJA) - First backward processing was unsuccessful, check MAJA installation #54

Open
egregr opened this issue Apr 2, 2020 · 9 comments
Open

ERROR (Start_MAJA) - First backward processing was unsuccessful, check MAJA installation #54

egregr opened this issue Apr 2, 2020 · 9 comments

Comments

@egregr
Copy link

egregr commented Apr 2, 2020

  1. Install MAJA
    Version: 3.3.2 [TM]
  2. Files in Start-MAJA/
    Common/ , folders.txt , prepare_mnt , start_maja.py , userconf/ , cams_download/ , prepare_dtm , Readme.md
  3. Folders and files added in Start-MAJA

  • DTM/S2__TEST_AUX_REFDE2_T31TFJ_0001 containing
    S2__TEST_AUX_REFDE2_T31TFJ_0001.DBL.DIR
    S2__TEST_AUX_REFDE2_T31TFJ_0001.HDR
    These are the result of the 'Create DTM' step applied on L1C data
    The L1C data was taken by performing
    python ./peps_download.py -c S2ST -l --lon 4.8 --lat 43.97 -a peps.txt -d 2015-11-01 -f 2015-12-01 -w /var/local/maja/L1C_DATA/4-8_43-97
    The folder /var/local/maja/L1C_DATA/4-8_43-97 contains
    S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE
    S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.zip
    S2A_MSIL1C_20151120T104332_N0204_R008_T31TFJ_20151120T104750.SAFE
    S2A_MSIL1C_20151120T104332_N0204_R008_T31TFJ_20151120T104750.zip
    S2A_MSIL1C_20151127T103352_N0204_R108_T31TFJ_20151127T103440.SAFE
    S2A_MSIL1C_20151127T103352_N0204_R108_T31TFJ_20151127T103440.zip
    S2A_MSIL1C_20151130T104412_N0204_R008_T31TFJ_20151130T104646.SAFE
    S2A_MSIL1C_20151130T104412_N0204_R008_T31TFJ_20151130T104646.zip

  • GIPP_S2AS2B/SENTINEL2_TM/ containing files S2B_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF
    S2A_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_CKEXTL_S_ALLSITES_10001_20190626_21000101.EEF
    S2A_TEST_GIP_CKEXTL_S_ALLSITES_10001_20190626_21000101.EEF S2B_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF
    S2A_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_CKQLTL_S_ALLSITES_10001_20190626_21000101.EEF
    S2A_TEST_GIP_CKQLTL_S_ALLSITES_10001_20190626_21000101.EEF S2B_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
    S2A_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_L2COMM_L_ALLSITES_10001_20190626_21000101.EEF
    S2A_TEST_GIP_L2COMM_L_ALLSITES_10001_20190626_21000101.EEF S2B_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
    S2A_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF S2__TEST_GIP_L2SITE_S_ALLSITES_00001_20190626_21000101.EEF

  • LUT_SENT2_NOCAMS/LUTs containing (https://zenodo.org/record/3368623)
    S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR
    S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR
    S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
    S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR
    S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
    S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR
    S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR
    S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR
    S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR
    S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR

  • TMP_FILES/

This is the folder.txt file:
repCode=/var/local/maja/Start-MAJA
repWork=/var/local/maja/Start-MAJA/TMP_FILES
repL1 =/var/local/maja/L1C_DATA
repL2 =/var/local/maja/L2A_DATA
repMaja=/var/local/maja/bin/maja
repCAMS=/var/local/maja/Start-MAJA/CAMS

I encounter an error when performing start_maja.py

python ./start_maja.py -f folders.txt -g GIPP_S2AS2B/SENTINEL2_TM -l LUT_SENT2_NOCAMS/LUTs -t 31TFJ -s 4-8_43-97 -d 20151101 -e 20151201

2020-04-02 15:59:24,574 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_31TFJ____T20151117.DBL.DIR not found
2020-04-02 15:59:24,574 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//SENTINEL2?_20151117-_T31TFJ_C_V not found
2020-04-02 15:59:24,575 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_31TFJ____T20151120.DBL.DIR not found
2020-04-02 15:59:24,575 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//SENTINEL2?_20151120-_T31TFJ_C_V not found
2020-04-02 15:59:24,576 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_31TFJ____T20151127.DBL.DIR not found
2020-04-02 15:59:24,576 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//SENTINEL2?_20151127-_T31TFJ_C_V not found
2020-04-02 15:59:24,577 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_31TFJ____T20151130.DBL.DIR not found
2020-04-02 15:59:24,577 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//SENTINEL2?_20151130-_T31TFJ_C_V not found
2020-04-02 15:59:24,577 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2020-04-02 15:59:24,577 - Start-Maja - INFO - => processing date 20151117
2020-04-02 15:59:24,577 - Start-Maja - INFO - dates to process in backward mode :
2020-04-02 15:59:24,577 - Start-Maja - INFO - -- 20151117 : /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE
2020-04-02 15:59:24,577 - Start-Maja - INFO - -- 20151120 : /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151120T104332_N0204_R008_T31TFJ_20151120T104750.SAFE
2020-04-02 15:59:24,577 - Start-Maja - INFO - -- 20151127 : /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151127T103352_N0204_R108_T31TFJ_20151127T103440.SAFE
2020-04-02 15:59:24,577 - Start-Maja - INFO - -- 20151130 : /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151130T104412_N0204_R008_T31TFJ_20151130T104646.SAFE
2020-04-02 15:59:24,581 - Start-Maja - INFO - #################################
2020-04-02 15:59:24,581 - Start-Maja - INFO - #################################
2020-04-02 15:59:24,581 - Start-Maja - INFO - processing /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE in backward mode
2020-04-02 15:59:24,581 - Start-Maja - INFO - Initialisation mode with backward is longer
2020-04-02 15:59:24,581 - Start-Maja - INFO - MAJA logfile: /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE.log
2020-04-02 15:59:24,582 - Start-Maja - INFO - #################################
2020-04-02 15:59:24,586 - Start-Maja - INFO - => processing date 20151120
2020-04-02 15:59:24,587 - Start-Maja - INFO - MAJA command failed : /var/local/maja/bin/maja -i /var/local/maja/Start-MAJA/TMP_FILES/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//in -o /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM/ -m L2BACKWARD -ucs /var/local/maja/Start-MAJA/TMP_FILES/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//userconf --TileId 31TFJ >/var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE.log 2>&1
2020-04-02 15:59:24,587 - Start-Maja - ERROR - First backward processing was unsuccessful, check MAJA installation
@octoflar
Copy link

octoflar commented Apr 2, 2020

I had the same issue two weeks ago. Meanwhile Maja 3.3.5 is available but I could not test yet whether the same issue occurs again.

@olivierhagolle
Copy link
Contributor

Hi, thanks for submitting the issue.

MAJA 3.3.5 will not solve the issue, it only improves performances on some Linux distributions such as RH7. Could you please provide the log file from MAJA ?

/var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE.log

Thanks,
Olivier

@egregr
Copy link
Author

egregr commented Apr 3, 2020

/var/local/maja/bin/maja-processing-3.3.2: error while loading shared libraries: libMajaChain.so.3.3: cannot open shared object file

@petket-5
Copy link

petket-5 commented Apr 3, 2020

There seems to be some problem with your maja installation. Do you have any additional PATH or LD_LIBRARY_PATH variables set (maybe in your ~/.bashrc)?

To debug, please run ldd /var/local/maja/lib/libMajaChain.so.3.3 and post the output here. Also, what OS are you running on?

Kind regards,
Peter

@octoflar
Copy link

octoflar commented Apr 6, 2020 •
edited

I have the same issue. Typing

python ./start_maja.py -f myfolders.txt -g ../gipp/SENTINEL2_NATIF -l ../luts/LUT_MAJA_3_TM_CAMS -t 32UNE -s Hamburg -d 20190714 -e 20190717

processes the first date successfully, but then aborts with an error:

2020-04-06 08:20:18,076 - Start-Maja - INFO -  /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2?_OPER_SSC_L2VALD_32UNE____T20190714.DBL.DIR not found
2020-04-06 08:20:18,076 - Start-Maja - INFO -  /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//SENTINEL2?_20190714-*_T32UNE_C_V* not found
2020-04-06 08:20:18,079 - Start-Maja - INFO -  /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2?_OPER_SSC_L2VALD_32UNE____T20190717.DBL.DIR not found
2020-04-06 08:20:18,079 - Start-Maja - INFO -  /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//SENTINEL2?_20190717-*_T32UNE_C_V* not found
2020-04-06 08:20:18,079 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2020-04-06 08:20:18,114 - Start-Maja - INFO - => processing date 20190714
2020-04-06 08:20:18,116 - Start-Maja - INFO - dates to process in backward mode :
2020-04-06 08:20:18,116 - Start-Maja - INFO - -- 20190714 : /home/janwevers/worldcover/proc/l1c/Hamburg/S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE
2020-04-06 08:20:18,118 - Start-Maja - INFO - -- 20190717 : /home/janwevers/worldcover/proc/l1c/Hamburg/S2B_MSIL1C_20190717T104029_N0208_R008_T32UNE_20190717T142616.SAFE
2020-04-06 08:20:18,622 - Start-Maja - INFO - #################################
2020-04-06 08:20:18,622 - Start-Maja - INFO - #################################
2020-04-06 08:20:18,622 - Start-Maja - INFO - processing /home/janwevers/worldcover/proc/l1c/Hamburg/S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE in backward mode
2020-04-06 08:20:18,622 - Start-Maja - INFO - Initialisation mode with backward is longer
2020-04-06 08:20:18,622 - Start-Maja - INFO - MAJA logfile: /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log
2020-04-06 08:20:18,622 - Start-Maja - INFO - #################################
2020-04-06 08:53:12,119 - Start-Maja - INFO - => processing date 20190717
2020-04-06 08:53:12,721 - Start-Maja - INFO - MAJA command failed : /home/janwevers/worldcover/maja/bin/maja  -i /home/janwevers/worldcover/proc/tmp/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//in -o /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF/ -m L2BACKWARD -ucs /home/janwevers/worldcover/proc/tmp/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//userconf --TileId 32UNE >/home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log 2>&1
2020-04-06 08:53:12,871 - Start-Maja - ERROR - First backward processing was unsuccessful, check MAJA installation

The log file from processing the first date is attached.

S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log

@egregr
Copy link
Author

egregr commented Apr 6, 2020

Indeed the problem was that certain soft paths to the libraries were broken i.e. all the libraries were in the folder /var/local/maja/lib/ but were not found

@octoflar
Copy link

octoflar commented Apr 6, 2020

When I re-issue the failed command manually

/home/janwevers/worldcover/maja/bin/maja  -i /home/janwevers/worldcover/proc/tmp/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//in -o /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF/ -m L2BACKWARD -ucs /home/janwevers/worldcover/proc/tmp/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//userconf --TileId 32UNE

I obtain the log output attached.

out.log

@octoflar
Copy link

octoflar commented Apr 6, 2020

The problem does not occur for the TM version, which I use now instead of no_TM.

@olivierhagolle
Copy link
Contributor

Good, we managed to have you using the TM version ;)

I do not think it was related to the version but rather to the way it had been installed. but who knows.
Olivier

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@olivierhagolle @petket-5 @octoflar @egregr