Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

first draft of splitting NWSS signals #1946

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Conversation

dsweber2
Copy link
Contributor

Description

This splits the dataset based on the provider and normalization (not every pair is actually present), and adds the metric signals. The resulting signals are called:

  • pcr_conc_smoothed_CDC_VERILY_flow-population
  • detect_prop_15d_CDC_VERILY_flow-population
  • percentile_CDC_VERILY_flow-population
  • ptc_15d_CDC_VERILY_flow-population
  • pcr_conc_smoothed_CDC_VERILY_microbial
  • detect_prop_15d_CDC_VERILY_microbial
  • percentile_CDC_VERILY_microbial
  • ptc_15d_CDC_VERILY_microbial
  • pcr_conc_smoothed_NWSS_flow-population
  • detect_prop_15d_NWSS_flow-population
  • percentile_NWSS_flow-population
  • ptc_15d_NWSS_flow-population
  • pcr_conc_smoothed_NWSS_microbial
  • detect_prop_15d_NWSS_microbial
  • percentile_NWSS_microbial
  • ptc_15d_NWSS_microbial
  • pcr_conc_smoothed_WWS_microbial
  • detect_prop_15d_WWS_microbial
  • percentile_WWS_microbial
  • ptc_15d_WWS_microbial

Some of these can have negative values; for e.g. ptc_15d, the values are small enough that I expect these may actually be exponents. Still looking into why the concentration data has negative values, which are too large to make sense as exponents.

Fixes

  • Fix generate_weights in the case where some weights are negative, and added a test for it

@dsweber2 dsweber2 changed the title first draft of splitting signals first draft of splitting NWSS signals Feb 23, 2024
@dsweber2
Copy link
Contributor Author

dsweber2 commented Mar 11, 2024

Minh confirmed that this runs on her machine and that the output looks reasonable. A couple of things to do before we merge this:

  • figure out the right cut-off for sig digits so that the noise is removed
  • seeing what values the formerly negative concentrations became
  • write docs similar to this PR

@dsweber2 dsweber2 force-pushed the splittingNWSSSignals branch 3 times, most recently from df5aa19 to e86e5fa Compare March 29, 2024 18:36
@dsweber2 dsweber2 requested a review from dshemetov April 16, 2024 18:40
@dsweber2
Copy link
Contributor Author

Currently not passing because of the same update that made nssp tests fail. Once that's merged and this is rebased it will pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants