[US_OR] Switching some supervision and incarceration periods (Recidiv…

…iz/recidiviz-data#28530) ## Description of the change Originally we used location type to determine if a period is a supervision or incarceration period because they use the same tables and the CURRENT_STATUS does not have the same level of information/accuracy. However, some periods in Oregon have a CURRENT_STATUS of IN (adult in custody/facility) or LC (local jail) with a location that is not a prison or jail (usually a community location). OR says these should still be counted as Incarceration Periods if the CURRENT_STATUS is IN or LC, so this change removed those periods from supervision periods and added them to incarceration periods. The supervision period results went from 2448054 to 2447002 and the incarceration period results went from 2356469 to 2368316. ## Type of change > All pull requests must have at least one of the following labels applied (otherwise the PR will fail): | Label | Description | |----------------------------- |----------------------------------------------------------------------------------------------------------- | | Type: Bug | non-breaking change that fixes an issue | | Type: Feature | non-breaking change that adds functionality | | Type: Breaking Change | fix or feature that would cause existing functionality to not work as expected | | Type: Non-breaking refactor | change addresses some tech debt item or prepares for a later change, but does not change functionality | | Type: Configuration Change | adjusts configuration to achieve some end related to functionality, development, performance, or security | | Type: Dependency Upgrade | upgrades a project dependency - these changes are not included in release notes | ## Related issues Closes #XXXX ## Checklists ### Development **This box MUST be checked by the submitter prior to merging**: - [x] **Double- and triple-checked that there is no Personally Identifiable Information (PII) being mistakenly added in this pull request** These boxes should be checked by the submitter prior to merging: - [x] Tests have been written to cover the code changed/added as part of this pull request ### Code review These boxes should be checked by reviewers prior to merging: - [x] This pull request has a descriptive title and information useful to a reviewer - [x] Potential security implications or infrastructural changes have been considered, if relevant GitOrigin-RevId: c908e9ba8d75e38f4d771afc6d0008b81981f04c
Recidiviz · May 11, 2024 · 40650d3 · 40650d3
1 parent ed4e2b6
commit 40650d3
Show file tree

Hide file tree

Showing 4 changed files with 17 additions and 8 deletions.
diff --git a/recidiviz/ingest/direct/regions/us_or/ingest_mappings/us_or_Incarceration_Period.yaml b/recidiviz/ingest/direct/regions/us_or/ingest_mappings/us_or_Incarceration_Period.yaml
@@ -51,8 +51,11 @@ output:
               $mappings:
                 StateIncarcerationType.COUNTY_JAIL:
                   - L # Local Control Jail
+                  - N # Non DOC Agency (jail)
                 StateIncarcerationType.STATE_PRISON:
                   - I # Institution
+                StateIncarcerationType.INTERNAL_UNKNOWN:
+                  - C # Community (OR has number of period where status is IN/LC (in facility or jail) but location is community - these should be incarceration periods)
           admission_date: MOVE_IN_DATE
           release_date: MOVE_OUT_DATE
           facility: FACILITY
@@ -97,6 +100,7 @@ output:
                   - CRTR # REVOKE/TERM MISD PROB SUPV - ORDER REQ'D - more in TBMOVE (only 2 instances)
                   - ABSC # Absconded Supervision
                   - VIOL #  VIOLATION
+                  - WRNT # Warrant - showing up in incarceration periods with community locations
                 StateIncarcerationPeriodAdmissionReason.STATUS_CHANGE: # check these
                   - JAIL # House in jail
                   - INTX # Inpatient Alcohol and Drug Treatment
@@ -128,6 +132,8 @@ output:
                   - INBH # BHS Intake
                   - INMA # Intake Medical Move
                 StateIncarcerationPeriodAdmissionReason.INTERNAL_UNKNOWN:
+                  - ISP # waiting answer from OR, only coming in when periods with community location have incarceration (IN/LC) current_status
+                  - DAYR # waiting answer from OR, only coming in when periods with community location have incarceration (IN/LC) current_status
                   - PARO # Parole - Parole Order Required (only 2 instances and in 1989)
                   - POST # SGL Prison/LC Sentence Expired (only 3 instances from data errors)
                   - WARR # Warrant for Non-Compliance (1 instance)
@@ -242,6 +248,7 @@ output:
                   ## OR data does not have frequent admission/release reasons and they more often match to release reason, so the view does such
                   ## but results in some erroneous values for release reason I.E. ABSC, as a release vs admission reason.
                   - ABSC # Absconded Supervision   (31 instances)
+                  - WRNT # Warrant - showing up in incarceration periods with community locations
                   - VIOL # PA/PPS/PR/LV Viol. to DOC or Local Control OR being admitted to DOC/LC on additional sentences.
                   - PROB # Probation-Status Change Only (23 instances) ** look into why it's showing up
                   - DETA # Detainer
@@ -250,6 +257,7 @@ output:
                   - REVP # Offender Sentenced to Federal Prison (only 4 instances)
                   - LOCL # Released to Local Control
                   - JAIL # House in jail
+                  - REVO # PAROLE REVOKED BY BOARD- DOC REQUIRED (most instances before 2000 - showing up on periods marked community location with IN/LC current_status)
                   - CRTR # REVOKE/TERM MISD PROB SUPV - ORDER REQ'D (only 2 instances)
                   - BNPB # Conv. to Bench Prob. Court Order Req'd. (only 13)
                   - EDIS # EARNED DISCHARGE (only 2 instances)

diff --git a/recidiviz/ingest/direct/regions/us_or/ingest_mappings/us_or_Supervision_Period.yaml b/recidiviz/ingest/direct/regions/us_or/ingest_mappings/us_or_Supervision_Period.yaml
@@ -59,8 +59,6 @@ output:
                   - SL # Second Look Juvenile (means they were let out at half of sentence generally) - closest to Post Prison sentence
                 StateSupervisionPeriodSupervisionType.INTERNAL_UNKNOWN:
                   - LV # Leave - transition from DOC to Supervision (doesn't fit into our enums)
-                  - LC # Local Control #TODO(#26389): Remove post refactor - technically an incarceration sentence but has location type of community
-                  - IN # Adult in Custody (AIC) #TODO(#26389): Remove post refactor - technically an incarceration sentence but has location type of community
                   - DI # Discharge - these are primarily data errors from OR
                   - AP # Admission Pending (1 instance)
                   - PP # ParolePending - Not used (1 instance)

diff --git a/recidiviz/ingest/direct/regions/us_or/ingest_views/view_Incarceration_Period.py b/recidiviz/ingest/direct/regions/us_or/ingest_views/view_Incarceration_Period.py
@@ -200,13 +200,15 @@
     ON transfers.RECORD_KEY = releases.RECORD_KEY
     AND transfers.CUSTODY_NUMBER = releases.CUSTODY_NUMBER
     AND transfers.ADMISSION_NUMBER = releases.ADMISSION_NUMBER
-  WHERE LOCATION_TYPE IN ('L', 'I') # Institution or Jail 
-  AND high_level_transfers.RESPONSIBLE_DIVISION IN ('I', 'L') # Only counting as Incarceration Period if custodial authority is jail or prison facility
+  WHERE (LOCATION_TYPE IN ('L', 'I') # Institution or Jail 
+  AND high_level_transfers.RESPONSIBLE_DIVISION IN ('I', 'L')) # Only counting as Incarceration Period if custodial authority is jail or prison facility
+  OR CURRENT_STATUS IN ('IN', 'LC')
 ), merging_units1 AS (
   # Because schema has housing unit but not cell number, we want to squash cell transfers happening within same units.
   SELECT
     periods.RECORD_KEY,
     PERIOD_ID,
+    CURRENT_STATUS, 
     LOCATION_TYPE, 
     IF(FACILITY = LAST_FACILITY AND UNIT_NUMBER = LAST_UNIT, null, MOVE_IN_DATE) AS MOVE_IN_DATE,
     MOVE_OUT_DATE,
@@ -228,6 +230,7 @@
   SELECT
     RECORD_KEY,
     PERIOD_ID,
+    CURRENT_STATUS,
     LOCATION_TYPE, 
     LAST_VALUE(MOVE_IN_DATE ignore nulls) OVER (periods_for_units range between UNBOUNDED preceding and current row) AS MOVE_IN_DATE,
     MOVE_OUT_DATE,

diff --git a/recidiviz/ingest/direct/regions/us_or/ingest_views/view_Supervision_Period.py b/recidiviz/ingest/direct/regions/us_or/ingest_views/view_Supervision_Period.py
@@ -172,10 +172,10 @@
     AND transfers.CUSTODY_NUMBER = releases.CUSTODY_NUMBER
     AND transfers.ADMISSION_NUMBER = releases.ADMISSION_NUMBER
   WHERE LOCATION_TYPE NOT IN ('L', 'I') # Institution or Jail 
-  AND high_level_transfers.RESPONSIBLE_DIVISION NOT IN ('I', 'L') # Only counting as Incarceration Period if custodial authority is jail or prison facility
-), 
-adding_supervision_level AS (
-  # Joining in supervision level where the date is between MOVE_IN_DATE and MOVE_OUT_DATE.
+  AND high_level_transfers.RESPONSIBLE_DIVISION NOT IN ('I', 'L') -- Don't want Supervision Period where custodial authority is jail or prison facility
+  AND CURRENT_STATUS NOT IN ('IN', 'LC') -- Some CURRENT_STATUS of IN/LC (facility/jail) have LOCATION_TYPE type of community so we want to make sure to exclude those
+), adding_supervision_level AS (
+  # Joining in custody level where the date is between MOVE_IN_DATE and MOVE_OUT_DATE.
   SELECT 
     periods.RECORD_KEY,
     PERIOD_ID,