Add Maine to Spotlight #562

terryttsai · 2022-05-05T00:41:55Z

Description of the change

Add Maine to Spotlight, the copy and data is not accurate right now.

Type of change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Configuration change (adjusts configuration to achieve some end related to functionality, development, performance, or security)

Related issues

Closes #541

Checklists

Development

These boxes should be checked by the submitter prior to merging:

Manual testing against realistic data has been performed locally

Code review

These boxes should be checked by reviewers prior to merging:

This pull request has a descriptive title and information useful to a reviewer
This pull request has been moved out of a Draft state, has no "Work In Progress" label, and has assigned reviewers
Potential security implications or infrastructural changes have been considered, if relevant

coveralls · 2022-05-05T18:47:58Z

Pull Request Test Coverage Report for Build 2303760533

9 of 9 (100.0%) changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+4.6%) to 88.889%

Totals
Change from base Build 2303067202:	4.6%
Covered Lines:	324
Relevant Lines:	353

💛 - Coveralls

terryttsai · 2022-05-05T19:01:11Z

spotlight-client/src/contentApi/sources/us_me.ts

+import programRegionsTopology from "./usNdProgramRegions";
+
+// localities for both sentencing and probation
+const judicialDistricts = [


What is the process for updating the sources file for a new state?

Is there an authoritative source for correctness?

Ideally this has already been created somewhere within the data platform as part of the ingest setup. In the past there has not really been a clear and obvious source for it, I've just had to hunt around for it. Now that we have State Pods I would say you should ask Kris or Daniela (I don't know how much of the ME ingest she has handed off yet)

When I did this for ID/TN I ran this query in BQ and just used those:

SELECT supervision_type, district FROM `recidiviz-staging.public_dashboard_views.supervision_population_by_district_by_demographics` WHERE state_code = 'US_ME' group by supervision_type, district order by supervision_type, district;

It looks like for probation they aren't hydrated yet, but for parole there are some that you could use.

I don't think I have bigquery access to this project? What are you getting?

Oh oops, here are the results!

I've added those in before, however I find it strange that the data I fetch gives me the same numbers for both "ALL" and "EXTERNAL_UNKNOWN", and I think Maine doesn't have parole, yet district breakdowns are all in the "PAROLE" category...

The end result being in the Sentencing by district and Probation by district charts, selecting "All" and "Unknown" gives me the same result, and selecting a district gives me "No Data".

Fetching sentence_type_by_district_by_demographics gives me no district breakdowns, and fetching supervision_population_by_district_by_demographics gives me district breakdowns only in the Parole category. Is this data that should be available?

@terryttsai thanks for calling this out, these are great questions. Can you upload screenshots or videos of this behavior?

@hobuobi maybe you can help here? Or we can involve Kris/Daniela?

spotlight-client/src/contentApi/sources/us_me.ts

macfarlandian

any idea what the data prognosis is for Maine? this is fine with me if it unblocks development but obviously it's completely broken ... if we at least have a listing of what metrics we expect to actually have it might be worth culling the rest of them now?

spotlight-client/src/contentApi/sources/us_me.ts

lilidworkin

this is fine with me if it unblocks development but obviously it's completely broken

@macfarlandian can you help us out a bit further here? Obviously this is me and @terryttsai 's first time doing this so we're learning the process. In the task @hobuobi has said that all data for the BASELINE metrics should be available. Does that not seem to be the case? (I haven't checked this out or ran it locally)

@terryttsai maybe what you could do is go through the app and start a doc where you record some screenshots and thoughts about what seems to be working, what's broken, etc, and then we can involve the ME state pod?

lilidworkin · 2022-05-09T15:16:24Z

spotlight-client/src/DataStore/utils.ts

@@ -30,6 +30,7 @@ export function getTenantFromDomain(): TenantId | undefined {
  // production domains
  if (domain.endsWith(".gov")) {
    // TODO(#530): Add ID here
+    // TODO(#): Add ME here


Let's create an actual issue to track this?

Created an issue that mirrors #540: #569 and updated the code to point to it.

Looks like this is what code we need to update to point to the correct production domain?

lilidworkin · 2022-05-09T15:23:16Z

spotlight-client/src/contentApi/sources/us_me.ts

+import programRegionsTopology from "./usNdProgramRegions";
+
+// localities for both sentencing and probation
+const judicialDistricts = [


@terryttsai thanks for calling this out, these are great questions. Can you upload screenshots or videos of this behavior?

@hobuobi maybe you can help here? Or we can involve Kris/Daniela?

lilidworkin · 2022-05-09T15:24:36Z

spotlight-client/src/contentApi/sources/us_me.ts

+      ],
+    },
+  },
+  // TODO(#540): Update localities


Is this the right TODO for this?

Hmm I see this in both us_id.ts and us_tn.ts, but it points to #540. @colincadams is this the right TODO?

macfarlandian · 2022-05-09T16:41:51Z

wow sorry @lilidworkin my local environment was broken and I assumed it was a data issue but in fact it was my own configuration error! The only metrics that are actually "totally broken" are the two recidivism sections, same as we have seen with the other new Spotlight states so far; if that's a similar known issue then I guess we're actually in pretty good shape

terryttsai · 2022-05-09T20:05:33Z

Also "How many people end up back in prison?" and "How has the recidivism rate changed over time?" are missing since I get a null response from the server when I look up "recidivism_rates_by_cohort_by_year".

macfarlandian

LGTM pending Lili's comments

lilidworkin · 2022-05-09T21:08:41Z

@hobuobi who should we talk to about this missing data in the screenshots above?

lilidworkin · 2022-05-10T16:32:52Z

@terryttsai if you address my remaining two comments about Zenhub issues, then I think we should land this, and then once it's in staging we can work with the state pod to address the data issues?

terryttsai changed the title ~~Add Maine to Spotlight with bunk source data to start~~ [WIP] Add Maine to Spotlight with bunk source data to start May 5, 2022

terryttsai changed the title ~~[WIP] Add Maine to Spotlight with bunk source data to start~~ [WIP] Add Maine to Spotlight May 5, 2022

terryttsai changed the title ~~[WIP] Add Maine to Spotlight~~ Add Maine to Spotlight May 5, 2022

terryttsai marked this pull request as ready for review May 5, 2022 18:56

terryttsai commented May 5, 2022

View reviewed changes

terryttsai requested review from colincadams and macfarlandian May 5, 2022 19:01

macfarlandian reviewed May 6, 2022

View reviewed changes

spotlight-client/src/contentApi/sources/us_me.ts Outdated Show resolved Hide resolved

macfarlandian reviewed May 6, 2022

View reviewed changes

spotlight-client/src/contentApi/sources/us_me.ts Outdated Show resolved Hide resolved

lilidworkin reviewed May 9, 2022

View reviewed changes

terryttsai requested a review from macfarlandian May 9, 2022 20:06

macfarlandian approved these changes May 9, 2022

View reviewed changes

lilidworkin approved these changes May 10, 2022

View reviewed changes

terryttsai added 7 commits May 10, 2022 15:41

Add Maine to Spotlight with bunk source data to start

020f85a

Update us_me.ts

468b48f

Update snapshot

ddc19e2

Add facilities and remove parole references

e449b9d

Removed unused sections

956ed86

Add districts

abe3df9

update TODO

42564ef

terryttsai force-pushed the terry/add-maine branch from 81503ad to 42564ef Compare May 10, 2022 22:42

add docLink and docName

5829dac

terryttsai merged commit 7167ce1 into main May 10, 2022

terryttsai deleted the terry/add-maine branch May 10, 2022 23:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Maine to Spotlight #562

Add Maine to Spotlight #562

terryttsai commented May 5, 2022 •

edited

coveralls commented May 5, 2022 •

edited

terryttsai May 5, 2022

macfarlandian May 6, 2022

colincadams May 6, 2022

terryttsai May 6, 2022

colincadams May 6, 2022

terryttsai May 6, 2022 •

edited

terryttsai May 6, 2022 •

edited

terryttsai May 6, 2022 •

edited

lilidworkin May 9, 2022

terryttsai May 9, 2022

macfarlandian left a comment

lilidworkin left a comment

lilidworkin May 9, 2022

terryttsai May 10, 2022

lilidworkin May 9, 2022

lilidworkin May 9, 2022

terryttsai May 10, 2022

macfarlandian commented May 9, 2022

terryttsai commented May 9, 2022

macfarlandian left a comment

lilidworkin commented May 9, 2022

lilidworkin commented May 10, 2022

Add Maine to Spotlight #562

Add Maine to Spotlight #562

Conversation

terryttsai commented May 5, 2022 • edited

Description of the change

Type of change

Related issues

Checklists

Development

Code review

coveralls commented May 5, 2022 • edited

Pull Request Test Coverage Report for Build 2303760533

💛 - Coveralls

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

terryttsai May 6, 2022 • edited

Choose a reason for hiding this comment

terryttsai May 6, 2022 • edited

Choose a reason for hiding this comment

terryttsai May 6, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

macfarlandian left a comment

Choose a reason for hiding this comment

lilidworkin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

macfarlandian commented May 9, 2022

terryttsai commented May 9, 2022

macfarlandian left a comment

Choose a reason for hiding this comment

lilidworkin commented May 9, 2022

lilidworkin commented May 10, 2022

terryttsai commented May 5, 2022 •

edited

coveralls commented May 5, 2022 •

edited

terryttsai May 6, 2022 •

edited

terryttsai May 6, 2022 •

edited

terryttsai May 6, 2022 •

edited