CAMS near-real-time data for the current day (last 24h)

[jprankl]

Dear Oliver,

I am currently integrating CAMS data into our MAJA processing pipeline. In principle, I am using the provided scripts to incrementally download CAMS data from ECMWF (MARS) and process new tiles. I found out that CAMS data for the past 24h is not available, but it seems (at least some) data is accessible from an ECMWF FTP server.

So, my question is if it is possible to immediately process Sentinel tiles when they are available (about 6h after sensing) using CAMS?

Is the needed CAMS (surface, model, pressure) data available on the FTP server? Can that data be used?

Best regards!
Johann

[jprankl]

Dear @olivierhagolle and @petket-5,

it seems there is a new API where cams data can be accessed (https://ads.atmosphere.copernicus.eu) and at least the ECMWF public web API is stopped on June 30th, 2021.

Is it possible/ recommended to use the new API? How would the request look like?

Thanks!

[olivierhagolle]

Yes, the change comes with short notice :(
The team is currently working hard to adapt to the new API.
Olivier

[jprankl]

Thanks for the fast reply. Indeed the ECMWF mail was short notice and even a little bit confusing to me.

[jprankl]

Dear @olivierhagolle and @petket-5 ,

yesterday I did a quick test with the new cams downloader from Maja 4.2.7. Two questions came up:

• the size of the files increased from 55MB (Maja 3), 156MB (Maja 4) to 1.5GB new Maja 4.2.7 ADS API
• and more important for me, now the last 7 days are not available, compared to the 24h of the old ECMWF API

I am using Maja in a (touched) real-time style, so the last 7 days are important for me. Do you know about that issue? Is that because of the new API? What would you suggest, do I need to cope without CAMS data?

Thanks in advance!
Johann

[jerome-colin]

Dear @jprankl,
I did several tests on different tiles and got the same issue : no CAMS found for recent days, while I can get them through the cdf api. There's most probably a bug, I'm opening an issue.
Thanks for raising this problem, I'll come back to you asap.
Jerome

[jerome-colin]

Hi @jprankl ,
Unfortunately, the problem doesn't seem to be a bug, but a license issue. While the aerosols AOD and mixing ratios are computed by CAMS, the relative humidity is a meteorological field from ECMWF. And since ECMWF doesn't belong to the general Copernicus agreement, the access to this data is delayed by 5 days. That's a pretty serious obstacle to activating the --cams option for near real time processing. We'll let you know if we find a workaround.

[jprankl]

Hi @jerome-colin ,
many thanks for your investigation!
Is the use of humidity new in 4.2.7? At least in Maja 3 and the old API I did not have that issue.
Would be cool if you find a workaround and let me know.
Thanks!

[jerome-colin]

We've been using relative humidity with cams for more than 2 years, but I never tried near real time runs for algorithm validation. I cannot tell whether it comes from the latest API or if we just never came into this issue because of the time period of the contexts we usually processed. ECMWF data are accessible via institutional agreements, but even so it'll need some additional tuning from our side to plug an additional data source.
Jerome

[jprankl]

o.k., please let me know if you find some options.
thanks for your effort!