New readers from MM satellite group #112

rrbuchholz · 2023-05-04T19:15:54Z

Adding in:
Tropomi L2 reader
OMPS L3 reader
MOPITT L3 reader
and adding to __init__.py file

* Update raqms.py tracking problems in colab * Update __init__.py * Update __init__.py * Update __init__.py * trying to make Kaggle import work * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update setup.cfg * Update setup.cfg * Update __init__.py * Update raqms.py adjusting timestamp. Needed for pairing. Unsure if this was a temporary fix sort of thing * Update raqms.py * Update raqms.py * Update raqms.py * Update omps_nadir.py * Update omps_nadir.py * data handling issue fix * Update omps_nadir.py * Update omps_nadir.py * add trimmed file reader for earthcube notebook * Update raqms.py * Update raqms.py * Update raqms.py * merge fixes * trying to fix for merge * Adding MOPITT read * tropomi no2 reader * rename * adding hdf close file * omps level 3 reader * Update raqms.py remove double of _fix_pres --------- Co-authored-by: mebruckner <48494069+mebruckner@users.noreply.github.com> Co-authored-by: Maggie Bruckner <mbruckner@raqms-ops.ssec.wisc.edu> Co-authored-by: Meng Li <mengli@MengM1.local>

zmoon · 2023-05-08T17:08:08Z

@dwfncar @rrbuchholz do you have/want any tests for any of these?

zmoon · 2023-05-08T17:10:56Z

3.6 failing due to #114 , but there are also some comments from flake8 to address, and some formatting stuff.
Also looks like we need to add h5py to the docs env.

rrbuchholz · 2023-05-10T17:35:14Z

I have an example Jupyter notebook for MOPITT in the MELODIES MONET develop_satellite branch that needs to be pulled.
I also think Meng has added a Tropomi example and Maggie has added the OMPS L3 example in MELODIES MONET.

zmoon · 2023-05-10T17:49:49Z

@rrbuchholz that'll be great to have those MM examples, but I was referring here to smaller test cases for the MONETIO readers to add to the MONETIO test suite. It's ok if we don't have, just thought I would ask.

zmoon · 2023-06-05T18:40:19Z

Data file @rrbuchholz shared now available at https://csl.noaa.gov/groups/csl4/modeldata/melodies-monet/data/example_observation_data/satellite/MOP03JM-201701-L3V95.9.3.he5

zmoon

Some initial comments.

monetio/sat/_mopitt_l3_mm.py

monetio/sat/_omps_l3_mm.py

monetio/sat/_tropomi_l2_no2_mm.py

text formatting differences Co-authored-by: Zachary Moon <zmoon92@gmail.com>

text formatting using consistent language Co-authored-by: Zachary Moon <zmoon92@gmail.com>

using consistent language in text descriptions Co-authored-by: Zachary Moon <zmoon92@gmail.com>

Code semantics Co-authored-by: Zachary Moon <zmoon92@gmail.com>

Remove development debugging Co-authored-by: Zachary Moon <zmoon92@gmail.com>

monetio/sat/_mopitt_l3_mm.py

with (somewhat) informative message containing the options

zmoon · 2023-09-11T21:15:33Z

monetio/sat/_tropomi_l2_no2_mm.py

+            ds.attrs["quality_flag"] = varname
+            ds.attrs["quality_thresh_min"] = variable_dict[varname]["quality_flag_min"]


These are dataset attrs so they will be overwritten if there is more than one variable in variable_dict.

True. This is an old version of the TROPOMI reader. I add a new dataset attribute 'var_applied' to define the variables list for each quality flag in the new reader

@mlirenzhenmayi I don't see this in the current version of the code.

zmoon · 2023-09-11T21:16:44Z

monetio/sat/_tropomi_l2_no2_mm.py

+                values[quality_flag <= quality_thresh_min] = np.nan
+
+
+def open_dataset(fnames, variable_dict, debug=False):


The variable_dict is necessary for the fn to be useful. Is this something that will always be the same? If so, could include in the code sort of like in the MOPITT one.

and/or document in the docstring the format

fn always change with the date and vary one by one. It seems that there's no string always the same.

By the way, by "fn" I meant function, not filename. I was wondering if we could set it up so you could pass varnames like the MOPITT one, instead of the variable dict. The variable dict info could be embedded in the module, if it is fixed for a certain variable.

* Update raqms.py tracking problems in colab * Update __init__.py * Update __init__.py * Update __init__.py * trying to make Kaggle import work * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Update setup.cfg * Update setup.cfg * Update __init__.py * Update raqms.py adjusting timestamp. Needed for pairing. Unsure if this was a temporary fix sort of thing * Update raqms.py * Update raqms.py * Update raqms.py * Update omps_nadir.py * Update omps_nadir.py * data handling issue fix * Update omps_nadir.py * Update omps_nadir.py * add trimmed file reader for earthcube notebook * Update raqms.py * Update raqms.py * Update raqms.py * merge fixes * trying to fix for merge * Adding MOPITT read * tropomi no2 reader * rename * adding hdf close file * omps level 3 reader * Update raqms.py remove double of _fix_pres * personal changes for using wisc wrf-chem output * Update _mopitt_l3_mm.py updating to read multiple variables in rather than just 1 * Update _mopitt_l3_mm.py fixing small error * Update _mopitt_l3_mm.py Removing loading of multiple variables in read_mopittdataset to figure out why apriori_col isn't loading right * Update _mopitt_l3_mm.py Fix consistency error in naming for apriori_prof, adding back in initial update for reading in many variables from file in one go * correct merge conflicts * add updated version to correct merge conflicts * fix merge conflicts * finish merge conflict fixes * typo * forgot a comma * fix merge conflicts by copying newer version of reader in * updated mopitt reader to add all variables needed for pairing and column calculation * remove reference to a reader not included Removed call to a reader that is only in personal branch --------- Co-authored-by: R Buchholz <rrbuchholz@users.noreply.github.com> Co-authored-by: Maggie Bruckner <mbruckner@raqms-ops.ssec.wisc.edu> Co-authored-by: rrbuchholz <buchholz@ucar.edu> Co-authored-by: Meng Li <mengli@MengM1.local>

rrbuchholz · 2024-05-02T19:33:36Z

Hi @zmoon! Just wondering what you need from us in the satellite team to get this pull request merged? Would be great to try and have it done before the summer release of MELODIES MONET. (It's been a year since we opened this!)

zmoon · 2024-05-03T21:06:38Z

Hi @rrbuchholz , very sorry, it's been on my list to get back to @mlirenzhenmayi

flake8 doesn't like them

this way the current test still works

seems to be more likely what we want though I'm not sure why setting 'quality_flag' to the var name is helpful

zmoon · 2024-05-05T23:50:03Z

@rrbuchholz and @mlirenzhenmayi I added some basic tests for the other two. I also made a few changes here and there (e.g. adding attributes to variables), maybe you can check that your workflows still work and/or review.

applied to all data vars

zmoon self-requested a review May 8, 2023 17:07

zmoon added 3 commits May 30, 2023 09:50

format

f423e27

please flake8

2ca5fc2

No top-level h5py import; some formatting

db5c69a

zmoon added 2 commits June 5, 2023 15:10

Initial mop test

85b0a00

Test da extractor

e0778ac

zmoon reviewed Jun 5, 2023

View reviewed changes

zmoon and others added 10 commits June 5, 2023 16:00

better msg

c3b917c

Add h5py to dev env

793e923

Merge branch 'noaa-oar-arl:develop' into develop

a04f468

Update monetio/sat/_tropomi_l2_no2_mm.py

2df1d76

text formatting differences Co-authored-by: Zachary Moon <zmoon92@gmail.com>

Update monetio/sat/_omps_l3_mm.py

0fda080

text formatting using consistent language Co-authored-by: Zachary Moon <zmoon92@gmail.com>

Update monetio/sat/_omps_l3_mm.py

38fc2f1

using consistent language in text descriptions Co-authored-by: Zachary Moon <zmoon92@gmail.com>

Update monetio/sat/_mopitt_l3_mm.py

9598318

Code semantics Co-authored-by: Zachary Moon <zmoon92@gmail.com>

Update monetio/sat/_mopitt_l3_mm.py

eb82711

Remove development debugging Co-authored-by: Zachary Moon <zmoon92@gmail.com>

Update monetio/sat/_mopitt_l3_mm.py

fece70d

Remove development debugging Co-authored-by: Zachary Moon <zmoon92@gmail.com>

Adding units and long_name

07eaf82

zmoon reviewed Sep 11, 2023

View reviewed changes

monetio/sat/_mopitt_l3_mm.py Show resolved Hide resolved

zmoon added 3 commits September 11, 2023 14:29

Style; fix file list/str

f90380f

Test MOPITT open_dataset too

961554f

Let exceptions be raised; check varname

86f8b15

with (somewhat) informative message containing the options

zmoon reviewed Sep 11, 2023

View reviewed changes

File lock for MOPITT test file

be7a9a7

rrbuchholz and others added 4 commits February 22, 2024 13:14

Merge branch 'noaa-oar-arl:develop' into develop

3655f4e

Merge branch 'noaa-oar-arl:develop' into develop

7ba5a77

Merge branch 'noaa-oar-arl:develop' into develop

3cc8a91

rrbuchholz mentioned this pull request Apr 18, 2024

Satellite team tasks NOAA-CSL/MELODIES-MONET#256

Open

zmoon and others added 18 commits May 3, 2024 15:13

Merge branch 'develop' into develop

afc51e3

Auto format

2789f2e

No ##

72febb9

flake8 doesn't like them

Allow single string varname

be143a3

this way the current test still works

Variable attrs instead

dc7a2de

seems to be more likely what we want though I'm not sure why setting 'quality_flag' to the var name is helpful

Sanity check

0648ffe

sp

95cee34

Allow passing Path

d6b5d59

dims and such; get time from data

9a141eb

Initial tropomi test

bcb5b3c

Apply dataset fill value

24c6ec9

Check min thing works

b6a3f17

Only modify logging config in debug case

198c51e

Update docstring

dabc84f

Switch y and x

952b7ee

Add attrs; time as coord

33c7734

Initial OMPS test using local files

f4f4580

Use uploaded data

aeb9296

zmoon force-pushed the develop branch from e760bb2 to aeb9296 Compare May 5, 2024 23:36

zmoon and others added 3 commits May 5, 2024 18:00

Back to single quality flag var

10cd0fc

applied to all data vars

Allow passing vn(s); doc variable_dict a bit

90bb5a8

Merge branch 'noaa-oar-arl:develop' into develop

899e530

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New readers from MM satellite group #112

New readers from MM satellite group #112

rrbuchholz commented May 4, 2023 •

edited by zmoon

zmoon commented May 8, 2023

zmoon commented May 8, 2023 •

edited

rrbuchholz commented May 10, 2023 •

edited

zmoon commented May 10, 2023 •

edited

zmoon commented Jun 5, 2023

zmoon left a comment

zmoon Sep 11, 2023

mlirenzhenmayi Mar 28, 2024

zmoon May 3, 2024

zmoon Sep 11, 2023

mlirenzhenmayi Mar 28, 2024

zmoon May 3, 2024

rrbuchholz commented May 2, 2024 •

edited

zmoon commented May 3, 2024

zmoon commented May 5, 2024 •

edited

		ds.attrs["quality_flag"] = varname
		ds.attrs["quality_thresh_min"] = variable_dict[varname]["quality_flag_min"]

		values[quality_flag <= quality_thresh_min] = np.nan


		def open_dataset(fnames, variable_dict, debug=False):

New readers from MM satellite group #112

Are you sure you want to change the base?

New readers from MM satellite group #112

Conversation

rrbuchholz commented May 4, 2023 • edited by zmoon

zmoon commented May 8, 2023

zmoon commented May 8, 2023 • edited

rrbuchholz commented May 10, 2023 • edited

zmoon commented May 10, 2023 • edited

zmoon commented Jun 5, 2023

zmoon left a comment

Choose a reason for hiding this comment

zmoon Sep 11, 2023

Choose a reason for hiding this comment

mlirenzhenmayi Mar 28, 2024

Choose a reason for hiding this comment

zmoon May 3, 2024

Choose a reason for hiding this comment

zmoon Sep 11, 2023

Choose a reason for hiding this comment

mlirenzhenmayi Mar 28, 2024

Choose a reason for hiding this comment

zmoon May 3, 2024

Choose a reason for hiding this comment

rrbuchholz commented May 2, 2024 • edited

zmoon commented May 3, 2024

zmoon commented May 5, 2024 • edited

rrbuchholz commented May 4, 2023 •

edited by zmoon

zmoon commented May 8, 2023 •

edited

rrbuchholz commented May 10, 2023 •

edited

zmoon commented May 10, 2023 •

edited

rrbuchholz commented May 2, 2024 •

edited

zmoon commented May 5, 2024 •

edited