bug: NCEP/prep_bufr code outdated, fails to compile with gfortran

[mjs2369]

Describe the bug

The code in this directory is very outdated and difficult to use and produces some compile time errors with more recent versions of gfortran. This is not DART code, and has been fragile in the past (similar to the JPL module code). In order to change this code, we need to have extensive tests.

This code is not built with the dart build system (mkmf, quickbuild). Instead, there's two scripts for building, one being bash and the other cshell: DART/observations/obs_converters/NCEP/prep_bufr/install.sh and DART/observations/obs_converters/NCEP/prep_bufr/convert_bufr/convert_bufr.csh.

The following files also include the use of outdated subroutines argc and getarg, which were replaced by the fortran intrinsics everywhere else in the DART code:
observations/obs_converters/NCEP/prep_bufr/convert_bufr/arg_test.f and observations/obs_converters/NCEP/prep_bufr/convert_bufr/stat_test.f:
observations/obs_converters/NCEP/prep_bufr/convert_bufr/arg_test.f

Finally, grabbufr.f fails to compile with gfortran 12.2.0 due to type mismatch errors (see screenshot). It does compile with intel.

1. List the steps someone needs to take to reproduce the bug.
   Run "./install.sh" and then "./convert_bufr/convert_bufr.csh" using gfortran. To do so, you must edit convert_bufr.csh to set gfortran as the Fortran compiler.
2. What was the expected outcome?
   Three executables are produced: stat_test.x, arg_test.x, and grabbufr.x
3. What actually happened?
   Grabbufr.f fails to compile with multiple errors

Error Message

See screenshot

Which model(s) are you working with?

NCEP observation converter

Screenshots

Version of DART

v11.0.2

Have you modified the DART code?

No

Build information

Please describe:

1. The machine you are running on (e.g. windows laptop, NCAR supercomputer Cheyenne).
2. The compiler you are using (e.g. gnu, intel).

All machines, gfortran 12.2.0

[mjs2369]

Including comments from Nancy in our discussion of the prepbufr section of the WRF/DART Tutorial:

""
This process relies heavily on scripting, which is complex. Moving some of the details into a source program would help simplify this process a lot.  (see below.)

There are several complications here - which a redesign could help:

1. This converter consists of 3 separate programs. It could be only one which would simplify a lot of issues. Two of the programs read bufr files and write a temporary ascii intermediate file. The last program reads that temp file and creates an obs_seq file.
2. The input prepqm files are in 6 hour chunks, with times centered on 0Z (midnight), 6Z, etc. One annoying thing is that observations exactly at the boundaries (3Z, 9Z, etc) can show up in either file. So typically the scripts run the converter on the current input file and also the next one to scoop up only the obs exactly on the boundary.
3. These files have times in them but no date info. The date info is only present in the filename. because the original program only opened prepqm.in as input, the dates are all handled in the scripts. It would be clearer if the program constructed the filenames and opened them directly.
4. To assimilate at 0Z with a 6 hour window, you will need observations from (21Z day 1 plus 1 second) to (3Z day 2). Because of how the programs were written, there is a 03Z version of the source that adds 24 hours to the times for all obs between midnight and 3Z on the second day.  so hours go from 0 to 27.  Those are handled in the second executable.  if these programs were combined, and if the program opened the files with their original dates in the names, this complexity can be removed.
5. Because of different users, these programs were constructed to either output 6 hour files or combine 4 into daily files. That seems like an unneeded complication - pick one and support that.
   ""

[nancycollins]

as i mentioned in the standup, years ago (egads, 2013 apparently?) i made a simple pass through the prepbufr.f code to enable additional obs types and add more comments. i found a copy of it and it's in my home directory on derecho if you want to see/use it. i didn't tackle outputting our format obs directly from it but that could be added where it does the ascii write. we'd have to add a quickbuild to pull in our other source code - right now it's a standalone compile.

[mjs2369]

Many of the prepqm files are “cosblocked”, which makes it so you can't untar them. On cheyenne there used to be a cosconvert program to fix the file, but this is not on Derecho. The majority of these prepqm files are therefore currently unusable. I followed the instructions here to build the program, but it was missing necessary files. Nancy was able to get the source code over email and build successfully. She is awaiting a response to her question of if we could put the source code some place users could access it.

[mgharamti]

Marlee, I've worked quite a bit on these prepbufr files and I used to unblock them routinely. I didn't use any special tool to do that. Below is a sample to unblock a file. You can also refer to one of my obs processing scripts for more details: /glade/campaign/cisl/dares/gharamti/wrf_dart/jeddah_rain/progs/prepare_obs.csh
Hope this helps!

   echo 'block'          >! in
   echo 'prepbufr.in'    >> in
   echo 'prepqm.blocked' >> in
   ${DART_DIR}/observations/obs_converters/NCEP/prep_bufr/exe/cword.x < in
   ${MOVE} prepqm.blocked prepqm.in
   ${REMOVE} in prepbufr.in

   ${DART_DIR}/observations/obs_converters/NCEP/prep_bufr/exe/prepbufr.x

[nancycollins]

moha - i think that's a different type of blocking. cword.x works on individual prepqm files that are in bufr format. the tar files that are archived on the RDA can't be untarred into individual prepqm.in files until they're converted with 'cosconvert'. the source says it has something to do with cray blocking.

n.

[mgharamti]

I get it now. Are we talking about this:
/glade/u/home/rdadata/bin/ch/cosconvert -b rawfile data.tar

[mjs2369]

I get it now. Are we talking about this: /glade/u/home/rdadata/bin/ch/cosconvert -b rawfile data.tar

@mgharamti yes that is it

[mjs2369]

There are many issues with the documentation for PROGRAM prepbufr (https://docs.dart.ucar.edu/en/latest/observations/obs_converters/NCEP/prep_bufr/prep_bufr.html).

We want to combine this document with the information about prepbufr in the WRF-DART tutorial. We will then remove that section from the tutorial and just have it point to the prepbufr program page so we don't have to maintain two separate docs for this and because it is difficult to follow the instructions and get all the necessary info having to go back and forth between these two documents.

We will also need to address several issues with the PROGRAM prepbufr page (DART/observations/obs_converters/NCEP/prep_bufr/prep_bufr.rst), including but not limited to:

• Link to an NCAR RDA page that can't be found line 183
• Outdated/incorrect list of instructions for navigating the NSF NCAR/NCEP reanalysis archive, starting on line 154. Inventories tab does not exist, website seems to have gotten a reorganization

[mgharamti]

Marlee, thanks for working on this. It does seem like a mess. I agree regarding updating the documentation.
Nancy, it's good that you have the code to compile and generate a cosconvert. I already copied it and kept it in a safe directory for my future use. PS: Your home directory scared me!

[nancycollins]

over the years i've stuffed things in my home dir for other folks to pick up. it does seem a bit out of control! maybe a house-cleaning is in order...

[nancycollins]

i got a note back from bob dattore, who wrote the cosconvert.c source. he says:

To your cosconvert question: you may do whatever you like with the code. There is no license, or you could say that it is licensed under the Creative Commons license with no restrictions if you need? My name is in there only so someone can contact me with questions or problems. You are absolutely free to remove that information if you want.

there's a copy of the source in my home dir on derecho, so whoever works on the prepbufr code can put a copy where ever seems appropriate without worrying about licensing issues.

[hkershaw-brown]

@mjs2369 thanks for working through this.

The dart docs point to https://rda.ucar.edu/datasets/ds090.0/ which
"This historical dataset is recommended for ancillary use only and not as a primary research dataset. It has likely been superseded by a newer, improved dataset."

There are newer versions of the dataset, e.g.
https://rda.ucar.edu/datasets/ds337.0/
which look like they may provide netcdf versions of the data.
"The data provided here are also available in NetCDF and ASCII formats, which can be accessed by following the "Get a subset" link on the ds337.0 data access page. The NetCDF datafiles are converted from PREPBUFR format using the pb2nc utility in the Model Evaluation Tools (MET) software package."

Before bashing though cosconvert.c, let's regroup and see what data we should be using.

[nancycollins]

i think the question was if we'd point users to the cosconvert source or not. the sentiment is no - that's an RDA problem. so there's nothing to do there.

as far as the netcdf format, in observations/obs_converters/NCEP/netcdf is a version of a netcdf to obs converter that was started by soyoung but never finished. the README says some of the needed info was missing at the time the code was written. if the netcdf files now have what is needed, maybe look there to see if it's worth starting from it or from scratch. the main program is 'convert_pb_netcdf' which could be renamed to be clearer but i think stands for 'convert prepbufr netcdf files to obs_seq'.

edit: changed what i think the program name stands for. it reads already converted netcdf and outputs obs_seq

[hkershaw-brown]

I think a question is also, should we point people to https://rda.ucar.edu/datasets/ds337.0/ rather than https://rda.ucar.edu/datasets/ds090.0/

[hkershaw-brown]

Timeout on this issue.
The prepbufr library included with dart is at least 15 years old. Lets talk to someone who works with these observations before going any further with the code in the dart repo.
e.g. https://github.com/NOAA-EMC/NCEPLIBS-bufr