MODIS problem under Mac OS 11 - Big Sur

[vanderleidebastiani]

Hi,

I updated my MacBook to Big Sur and now runGdal not work.

The package sf have some problems with installation under Big Sur. I followed the instructions on this issue (r-spatial/sf#1536) to install the sf and worked for me. I installed MODIS via devtools::install_github("MatMatt/MODIS", ref = "develop").

I also checked if I had GDAL installed with HDF4 support (https://cornelllabofornithology.github.io/ebird-best-practices/covariates.html#covariates-dl). It looks ok.

In terminal:

gdal-config --formats
"derived gtiff hfa mem vrt aaigrid adrg aigrid airsar arg blx bmp bsb cals ceos ceos2 coasp cosar ctg dimap dted e00grid elas envisat ers fit gff gsg gxf hf2 idrisi ignfheightasciigrid ilwis ingr iris iso8211 jaxapalsar jdem kmlsuperoverlay l1b leveller map mrf msgn ngsgeoid nitf northwood pds prf r raw rmf rs2 safe saga sdts sentinel2 sgi sigdem srtmhgt terragen til tsx usgsdem xpm xyz zmap rik ozi grib eeda plmosaic rda wcs wms wmts daas rasterlite mbtiles pdf webp epsilon dods openjpeg jpeg2000 netcdf hdf5 hdf4 gif jpeg png pcraster fits pcidsk postgisraster"

In R:
library(sf)
"Linking to GEOS 3.8.1, GDAL 3.2.0, PROJ 6.3.2"

library(MODIS)
"ok"

MODIS:::checkTools("GDAL")
"Checking availability of GDAL:
OK, GDAL 3.2.0 found!"

However, then I try rum runGdal show the following:

MODIS:::runGdal(...)
"GDAL not installed or configured, read in '?MODISoptions' for help"

MODIS:::checkHdf4Driver()
"HDF4 driver seems to be lacking, please install GDAL with HDF4 support."

Is there any way to fix this?

Best, Vanderlei

[jeroen]

This problem is now fixed on CRAN. You can now install the sf and rgdal packages from CRAN in the regular way on MacOS big sur.

[fdetsch]

@vanderleidebastiani Can you give it another try?
@jeroen Glad to hear this has now been fixed, thanks!

[vanderleidebastiani]

Hi @jeroen and @fdetsch

I removed and reinstall rgdal (current version 1.5.18), sf (0.9.6) and MODIS (1.2.3) via CRAN and apparently it worked appropriately. I needed to define a new timeout to download the packages, in my case options(timeout = 240). However, the function runGdal is taken a long time to download, and I am not sure if it is in fact working.

Best, Vanderlei

[vanderleidebastiani]

Hi @jeroen and @fdetsch

I am working on two Macs with MacOS 11. I can run the runGdal function just in one of them. When the function runGdal does not work, it takes a long time in the download process and never ends. I can not find the exact difference between them, but the Mac that it works with uses bash as the default shell. The Mac that does not work uses zsh (Z shell) as the default shell (see https://support.apple.com/en-us/HT208050). Can that make difference?

Best, Vanderlei

[mickayla32]

Hi @jeroen and @fdetsch

I removed and reinstall rgdal (current version 1.5.18), sf (0.9.6) and MODIS (1.2.3) via CRAN and apparently it worked appropriately. I needed to define a new timeout to download the packages, in my case options(timeout = 240). However, the function runGdal is taken a long time to download, and I am not sure if it is in fact working.

Best, Vanderlei

Hi Vanderlei,
I was having the same original problem as you (thank you for posting this!) Would you mind sharing how you correctly removed and reinstalled rdgal, sf, and MODIS?
Thank you very much,
Mickayla

[vanderleidebastiani]

Hi @mickayla32

I typed:
remove.packages(c("sf", "MODIS", "rgdal"))

However, the runGdal function is still not working.

Best, Vanderlei

[fdetsch]

@vanderleidebastiani Does runGdal() create any new files in getOption("MODIS_localArcPath")? If not, this would point towards issues with getHdf(), which is running under the hood.

[vanderleidebastiani]

Hi @fdetsch

I tried with the example of the getHdf function. It creates some folders and a file .hdf, but the file have always zero bytes. Could you help me with that?

Some additional check:

MODISoptions()
File '~/.MODIS_Opts.R' does not exist. Create it now to make settings permanent? [y/n]: n

STORAGE:

---

localArcPath : /private/var/folders/q_/jbn0s0r90ss2_0yxltdcz0zm0000gn/T/RtmpCtrNLZ/MODIS_ARC
outDirPath : /private/var/folders/q_/jbn0s0r90ss2_0yxltdcz0zm0000gn/T/RtmpCtrNLZ/MODIS_ARC/PROCESSED

DOWNLOAD:

---

MODISserverOrder : LPDAAC, LAADS
dlmethod : auto
stubbornness : high
wait : 0.5
quiet : TRUE

PROCESSING:

---

GDAL : 3.1.1
MRT : Enabled
pixelSize : asIn
outProj : asIn
resamplingType : NN
dataFormat : GTiff
cellchunk : 1

MODIS:::checkTools("GDAL")
OK, GDAL 3.1.1 found!

MODIS:::checkTools("WGET")
Nothing was returned

Best, Vanderlei

[fdetsch]

Are you positive that login credentials in ~/.netrc are correct?

[vanderleidebastiani]

@fdetsch

I am working on two Macs with macOS 11. One of them is OK, I have done a mistake and change the credenctials. Thanks for the tip. The other it keeps not working, the downloading never finish. I think that it can be a PATH problem, but I am not able to fix this.

In Terminal:
~echo $0
-zsh
~R CMD
-looks ok

In R/RStudio:

system("echo $0")
sh
system("R CMD")
sh: R: command not found
Warning message:
In system("R CMD") : error in running command

Then I change the PATH with Sys.getenv to include "/Library/Frameworks/R.framework/Resources"

Sys.getenv("PATH")
[1] "/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin:/opt/local/bin:/Applications/RStudio.app/Contents/MacOS/postback"
Sys.setenv(PATH="/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin:/opt/local/bin:/Applications/RStudio.app/Contents/MacOS/postback:/Library/Frameworks/R.framework/Resources")
system("R CMD")
-looks ok

I tried to include the PATH permanently, so I edited ./bash_profile (additionally I edited it on ./zshrc)

export PATH="/Library/Frameworks/R.framework/Resources:$PATH"

However, it does not make any effect. The PATH does not recognize after I restart R, it goes back to initial settings. Anyway the function runGdal nor works on this computer. I have not idea if it is in fact the problem, but this is the difference that I can find at the moment.

Do I need to include PATH in another file or I made some mistake in the way to include it?

Best, Vanderlei

[fdetsch]

For me as a non-Mac person, this is extremely hard to debug. Since empty files are created, I would assume that there's something wrong with the file download, some proxy or firewall issue maybe. Can you

• debug(MODIS:::ModisFileDownloader),
• run an arbitrary getHdf() command,
• and then go through debug mode line by line?

Like this, you should be able to find the exact point at which file download fails. Maybe this will provide some insights..

[vanderleidebastiani]

Hi @fdetsch

I tried to debug ModisFileDownloader in both macs that I use. I notice that the functions follow distinct paths between computers. The Mac that the download can be complete use the function curl::curl_download to try download, while the Mac that not complete the download try to download via utils::download.file. Using download.file I think that have some authentication problems (I rechecked my user and pass, this is not a problem).

To fix this issue I need to force the option dlmethod with MODISoptions.

The steps to fix:

• MODISoptions(dlmethod = "curl") (answer yes)
• Remove any file at the path indicated in localArcPath

Thanks for the help.

Best, Vanderlei

[fdetsch]

... which brings up two questions:

• Why is it behaving differently on the two Macs? Have you maybe had another download method permanently set on the Mac on which this was not working?
• Can we fix the reported authentication problems in ModisFileDownloader() → download.file()? I am not much of a Mac person, so any help would be highly appreciated.

[vanderleidebastiani]

@fdetsch

I have not set a permanent download method. The mac that the download work without set a download method a warning message is showed ("sh: wget: command not found"). This message is not a problem, the download can be completed and it looks like a message from the bash shell. I think that this difference is because of the difference between the shell, I think that I have a PATH problem in the mac where the download can not finish without set the download method. I opened a question in stackoverflow (https://stackoverflow.com/questions/65013779/shell-path-to-r-under-macos-big-sur) to try to fix, but until now it has no answer.

Best, Vanderlei

[fdetsch]

Great, thanks. I'd appreciate if you could keep us updated!

[fdetsch]

Managed to narrow this down to some kind of change in wget LAADS download behavior.

On the command line, LP DAAC download works as expected and a ~20 MB .hdf file is downloaded:

wget --user=<your_user> --password=<your_pw> --no-check-certificate https://e4ftl01.cr.usgs.gov/MOLT/MOD13A3.006/2020.01.01/MOD13A3.A2020001.h21v09.006.2020034152427.hdf

By contrast, LAADS download downloads a ~12 KB file only:

wget --user=<your_user> --password=<your_pw> --no-check-certificate https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD13A3/2020/001/MOD13A3.A2020001.h21v09.006.2020034152427.hdf

Any wget redirect/authentication etc. experts around here, @MatMatt maybe? I am fairly sure that once this is solved, 0 KB file download issues will be resolved.

[MatMatt]

When I try to access the two files on a browser the LP DAAC can be downloaded, but for the LAADS url I get an error, is that to be expected (due to login) or is there an issue with the server/service? Access stops here https://ladsweb.modaps.eosdis.nasa.gov/archive/

[fdetsch]

You can go to https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD13A3/2020/001 and then download the above file manually. This should redirect you to urs.earthdata.nasa.gov where you need to sign in before you can download the file.