Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LakeCat published comparison #92

Merged
merged 2 commits into from
Aug 30, 2021
Merged

LakeCat published comparison #92

merged 2 commits into from
Aug 30, 2021

Conversation

debboutr
Copy link
Contributor

This PR updates the ListPublishedUnpublishedFiles.py script to examine and update what StreamCat Final_Table_Names are up on the FTP site. I have added a print statement that will show any FinalTables that are published in StreamCat but not in LakeCat.

I also took a look at potentially consolidating the ControlTables for both repos and have found the following inconsistencies that need to be addressed in this PR or potentially others (comment below):

  • should we write out control table w/ updated "Published" values? tracking issues? reading then writing will convert all "Conversion" cells to be of dtype float64
  • columns align -- LKCAT just missing "use_mask" and "Published" <- no masks in LKCAT, "Published" could be used for both
  • "Rckdep" in LKCAT is "RckDep" in StreamCat
  • BFI uses "bfi48grd.tif" in StreamCat but in LKCAT "bfi.tif"
  • "Tmin", "Tmean", "Tmax", "precip" in StreamCat are "Tmin8110", "Tmean8110", "Tmax8110", "precip8110" in LKCAT
  • /lookup tables still point to L: drive in LKCAT <- would be nice to be unified
  • "PIP_2017" is duplicated in metric name in LkCat but has different LandscapeLayer / FullTableName values!
  • "sw_flux" is duplicated in StreamCat control table -- all attrs the same
  • the following are run in LKCAT, but not in StreamCat <- REVERSE of what the script is looking for
MetricName
TMEANPY_2017
TMEANPW_2017
PSUMPY_2017
PSUM6_2017
PIP_2017
TIP_2017
PCVPY_2017
MercDep

@debboutr debboutr changed the title discover diffs between LKCAT and STRMCAT -- add print stmnt for LakeCat published comparison LakeCat published comparison Aug 27, 2021
@debboutr debboutr linked an issue Aug 27, 2021 that may be closed by this pull request
@debboutr debboutr changed the base branch from master to dev August 30, 2021 19:10
@debboutr debboutr merged commit 5685be1 into dev Aug 30, 2021
@debboutr debboutr mentioned this pull request Aug 30, 2021
1 task
debboutr added a commit that referenced this pull request Aug 30, 2021
@debboutr debboutr deleted the list-unpublished-lakecat branch September 4, 2021 00:13
@debboutr debboutr mentioned this pull request Sep 4, 2021
3 tasks
mhweber pushed a commit that referenced this pull request Dec 15, 2021
* add print stmnt for LakeCat published comparison (#92)

Co-authored-by: Debbout <Debbout.Rick@epa.gov>

* Stream temp modelling (#102)

* updates for nlcd 2019 rasters

* update os.environ addition for arcpy, control table needs  col to be float

* update control table

* run values at 0 causing problems

* add temp-modelling layers

* update check_metrics

* remove duplicative lookup tables

* update docstring for check_metrics command

* add ability for alternate control table

* implement click command

* add temp-modelling layers

* rename arg

* add ability to use absolute path on landscape layer

* add z_stats for public viewing

* add script for float precision (#114)

* add script for float precision

* precision title change

* adjust control table

* update Published in Control, use HTTPS/bs4 for identifying published

Co-authored-by: Debbout <Debbout.Rick@epa.gov>
Co-authored-by: Tgephart <gephart.tyler@epa.govgit config --global --list configc --global --listgit config --global user.email gephart.tyler@epa.gov>

Co-authored-by: Debbout <Debbout.Rick@epa.gov>
Co-authored-by: Tgephart <gephart.tyler@epa.govgit config --global --list configc --global --listgit config --global user.email gephart.tyler@epa.gov>
debboutr added a commit that referenced this pull request Jan 13, 2022
* add print stmnt for LakeCat published comparison (#92)

Co-authored-by: Debbout <Debbout.Rick@epa.gov>

* Stream temp modelling (#102)

* updates for nlcd 2019 rasters

* update os.environ addition for arcpy, control table needs  col to be float

* update control table

* run values at 0 causing problems

* add temp-modelling layers

* update check_metrics

* remove duplicative lookup tables

* update docstring for check_metrics command

* add ability for alternate control table

* implement click command

* add temp-modelling layers

* rename arg

* add ability to use absolute path on landscape layer

* add z_stats for public viewing

* add script for float precision (#114)

* add script for float precision

* precision title change

* adjust control table

* update Published in Control, use HTTPS/bs4 for identifying published

Co-authored-by: Debbout <Debbout.Rick@epa.gov>
Co-authored-by: Tgephart <gephart.tyler@epa.govgit config --global --list configc --global --listgit config --global user.email gephart.tyler@epa.gov>

* convert polys to crs of points instead of vice-versa

* formatting

* remove exit statement

Co-authored-by: Debbout <Debbout.Rick@epa.gov>
Co-authored-by: Tgephart <gephart.tyler@epa.govgit config --global --list configc --global --listgit config --global user.email gephart.tyler@epa.gov>
mhweber pushed a commit that referenced this pull request Feb 7, 2022
* remove year from NLCD lookup table

* Point poly update (#115)

* add print stmnt for LakeCat published comparison (#92)

Co-authored-by: Debbout <Debbout.Rick@epa.gov>

* Stream temp modelling (#102)

* updates for nlcd 2019 rasters

* update os.environ addition for arcpy, control table needs  col to be float

* update control table

* run values at 0 causing problems

* add temp-modelling layers

* update check_metrics

* remove duplicative lookup tables

* update docstring for check_metrics command

* add ability for alternate control table

* implement click command

* add temp-modelling layers

* rename arg

* add ability to use absolute path on landscape layer

* add z_stats for public viewing

* add script for float precision (#114)

* add script for float precision

* precision title change

* adjust control table

* update Published in Control, use HTTPS/bs4 for identifying published

Co-authored-by: Debbout <Debbout.Rick@epa.gov>
Co-authored-by: Tgephart <gephart.tyler@epa.govgit config --global --list configc --global --listgit config --global user.email gephart.tyler@epa.gov>

* convert polys to crs of points instead of vice-versa

* formatting

* remove exit statement

Co-authored-by: Debbout <Debbout.Rick@epa.gov>
Co-authored-by: Tgephart <gephart.tyler@epa.govgit config --global --list configc --global --listgit config --global user.email gephart.tyler@epa.gov>

* always write out control file from ListPublishedUnpublishedFiles.py

Co-authored-by: Debbout <Debbout.Rick@epa.gov>
Co-authored-by: Tgephart <gephart.tyler@epa.govgit config --global --list configc --global --listgit config --global user.email gephart.tyler@epa.gov>
mhweber added a commit that referenced this pull request Mar 21, 2022
* remove year from NLCD lookup table

* Point poly update (#115)

* add print stmnt for LakeCat published comparison (#92)

Co-authored-by: Debbout <Debbout.Rick@epa.gov>

* Stream temp modelling (#102)

* updates for nlcd 2019 rasters

* update os.environ addition for arcpy, control table needs  col to be float

* update control table

* run values at 0 causing problems

* add temp-modelling layers

* update check_metrics

* remove duplicative lookup tables

* update docstring for check_metrics command

* add ability for alternate control table

* implement click command

* add temp-modelling layers

* rename arg

* add ability to use absolute path on landscape layer

* add z_stats for public viewing

* add script for float precision (#114)

* add script for float precision

* precision title change

* adjust control table

* update Published in Control, use HTTPS/bs4 for identifying published

Co-authored-by: Debbout <Debbout.Rick@epa.gov>
Co-authored-by: Tgephart <gephart.tyler@epa.govgit config --global --list configc --global --listgit config --global user.email gephart.tyler@epa.gov>

* convert polys to crs of points instead of vice-versa

* formatting

* remove exit statement

Co-authored-by: Debbout <Debbout.Rick@epa.gov>
Co-authored-by: Tgephart <gephart.tyler@epa.govgit config --global --list configc --global --listgit config --global user.email gephart.tyler@epa.gov>

* always write out control file from ListPublishedUnpublishedFiles.py

* Added functions to load existing StreamCat ftp csv tables to the Stre… (#117)

* Added functions to load existing StreamCat ftp csv tables to the StreamCat API database

* fix to populate tables function

* Added function to list published / unpublished metrics in API

* Updated delete table function and control table

* updates to API

* updates for StreamCat_API.py and ControlTable

* updated control table

Co-authored-by: debboutr <debboutr@gmail.com>
Co-authored-by: Debbout <Debbout.Rick@epa.gov>
Co-authored-by: Tgephart <gephart.tyler@epa.govgit config --global --list configc --global --listgit config --global user.email gephart.tyler@epa.gov>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

discover diffs between LAKECAT and STREAMCAT published files
1 participant