Skip to content

Commit

Permalink
[US_AZ] Refactor incarceration periods view based on initial validati…
Browse files Browse the repository at this point in the history 
…on (Recidiviz/recidiviz-data#29313)

## Description of the change

See full notes from validation process
[here](https://docs.google.com/document/d/135KIzywjfdxt-3S1qEDCtWneAnOtsMclY01P30EQR80/edit).

There were two main issues with the incarceration periods view that
arose while I was comparing the set of people who we believed were
incarcerated on 6/15 of the last 5 years with the set of people ADCRR
believed were incarcerated on those same dates.

1. Recidiviz is not properly closing “periods” that were opened and
closed on the same day, particularly when there are an odd number of
movements tracked on that one day.
2. Recidiviz is not always correctly tracking movements that happen
after an escorted leave of any kind. These escorted leaves include trips
to court and the hospital.

There are a number of examples of each of these situations in the notes
document that I have confirmed are fixed by this change. This change
also greatly simplifies the logic of the view so that further tightening
of this validation will be easier.

One thing that became clear after this change is that the process for
revocations in AZ often goes like this: a person is out on supervision,
and is involved in some sort of incident that results in a `Temporary
Placement` in a facility, presumably while the incident is reviewed.
After that initial placement, there can be another movement signifying
that the person's supervision was revoked. For this reason, we see
`Parole Revoked`-type movements as "releases" not infrequently. This
initial period of `Temporary Placement` should possibly be mapped to
some investigative status when it is followed by a revocation.

Tests:

- [x] external ID fields are unique
- [x] run_mappings script passes
- [x] ingest view query is deterministic (in progress!)
- [x] sandbox ingest
[pipeline](https://console.cloud.google.com/dataflow/jobs/us-west1/2024-04-24_11_53_36-10222558064572225174?project=recidiviz-staging)
succeeds (in progress)

## Type of change

> All pull requests must have at least one of the following labels
applied (otherwise the PR will fail):

| Label | Description |
|-----------------------------
|-----------------------------------------------------------------------------------------------------------
|
| Type: Bug | non-breaking change that fixes an issue |
| Type: Feature | non-breaking change that adds functionality |
| Type: Breaking Change | fix or feature that would cause existing
functionality to not work as expected |
| Type: Non-breaking refactor | change addresses some tech debt item or
prepares for a later change, but does not change functionality |
| Type: Configuration Change | adjusts configuration to achieve some end
related to functionality, development, performance, or security |
| Type: Dependency Upgrade | upgrades a project dependency - these
changes are not included in release notes |

## Related issues

Related to Recidiviz/recidiviz-data#28522
Related to Recidiviz/recidiviz-data#28906
Related to Recidiviz/recidiviz-data#28907

## Checklists

### Development

**This box MUST be checked by the submitter prior to merging**:
- [ ] **Double- and triple-checked that there is no Personally
Identifiable Information (PII) being mistakenly added in this pull
request**

These boxes should be checked by the submitter prior to merging:
- [ ] Tests have been written to cover the code changed/added as part of
this pull request

### Code review

These boxes should be checked by reviewers prior to merging:

- [ ] This pull request has a descriptive title and information useful
to a reviewer
- [ ] Potential security implications or infrastructural changes have
been considered, if relevant

GitOrigin-RevId: 34d6eb6d828aafe6e7ed99154526baef79d34a1f
  • Loading branch information
@elisegonzal
elisegonzal authored and Helper Bot committed May 10, 2024
1 parent 11fa787 commit ca05752
Show file tree
Hide file tree
Showing 2 changed files with 208 additions and 168 deletions.
237 changes: 124 additions & 113 deletions recidiviz/ingest/direct/regions/us_az/ingest_mappings/us_az_incarceration_periods.yaml
View file
Expand Up @@ -36,190 +36,201 @@ output:
$raw_text: admission_reason
$mappings:
StateIncarcerationPeriodAdmissionReason.NEW_ADMISSION:
- New Commitment
- Recommitment
- Intake New Offender
- New Commitment
- ISC Commitment
StateIncarcerationPeriodAdmissionReason.RETURN_FROM_ESCAPE:
- Return from AWOL
- Return from Escape
StateIncarcerationPeriodAdmissionReason.RETURN_FROM_TEMPORARY_RELEASE:
- Return from Escorted Leave
- Temporary Release Return
- Return from Unescorted Compassionate Leave
- Return from Court
- Return from Hospital
- Return from Court
- Return from Unescorted Compassionate Leave
- Temporary Release Return
- Return from Bond
StateIncarcerationPeriodAdmissionReason.REVOCATION:
- Return from Abscond
- TIS Release Return
- Condition of Probation
- ERCD Release Return
- Community Supervision Revoked
- Parole Revoked-Regular
- Received CC/CS
- Return Deport 1/2 Term
- Provisional Rel Rescinded/Revoked
- Mandatory Release Res/Rev
- Reinstated - RUAP
- Home Arrest Rescinded/Revoked
- Return from AWOL
- SB1291 Violated
- Work Furlough Rescinded/Revoked
- Drug Transition Violated
- Parole Revoked/41.1604.11K
- RTRN CUSTDY ITP
- Discretionary Release Returnee
- Drug Transition Violated
- TIS Release Return
- ERCD Release Return
- Mandatory Release Res/Rev
- Received CC/CS
- Return from Abscond
- SB1291 Violated
- Reopen Last Prison Episode
- Re-Open Last Prison Episode
- Revocation
- SHOCK Incarceration Violator
- Work Furlough Rescinded/Revoked
- Condition of Probation
StateIncarcerationPeriodAdmissionReason.STATUS_CHANGE:
- Close Supervision Episode - Active Inmate
- Close Supervision Episode - Leave Prison Episode Active
- Received After 20 Days
- STATUS_CHANGE
- Assign to Central Office
StateIncarcerationPeriodAdmissionReason.TEMPORARY_RELEASE:
- Out to Court
- Out to Hospital
- Escorted Leave
- Unescorted Compassionate Leave
StateIncarcerationPeriodAdmissionReason.TEMPORARY_CUSTODY:
- Temporary Placement
- SHOCK Incarceration Commitment
StateIncarcerationPeriodAdmissionReason.TRANSFER:
- Received From
- Transfer In - Unit Arrival
- Transferred To
- Overcrowding
- Received from Overcrowding
- Transfer In - Prison/CIP Arrival
- Transfer In - Unit Arrival
StateIncarcerationPeriodAdmissionReason.TRANSFER_FROM_OTHER_JURISDICTION:
- ISC Commitment
- Transfer Out - Prison/CIP Departure
- Transfer WICC/ICC
- Return from WICC/ICC
- Transfer Out - Unit Departure
- Out of State/Federal
StateIncarcerationPeriodAdmissionReason.RETURN_FROM_ERRONEOUS_RELEASE:
- Release in Error Return
- Deactivated in Error
- Reopen Last Prison Episode
StateIncarcerationPeriodAdmissionReason.ESCAPE:
- Escaped
- AWOL/Abscond
- Temp Absent to Escape Status
StateIncarcerationPeriodAdmissionReason.SANCTION_ADMISSION:
- SHOCK Incarceration Commitment
- SHOCK Incarceration Violator
StateIncarcerationPeriodAdmissionReason.INTERNAL_UNKNOWN:
- Duplicate ADC Number Activation
- Return from WICC/ICC
- Return Deport 1/2 Term
- RTRN CUSTDY ITP
- Out on Bond
- In Transit T
- Transit Driver
- Rule 11
- Temp Absent to Escape Status
- Class 1 Date Adjustment
- Received After 20 Days
- Duplicate ADC Number Activation
- Close Supervision Episode - Active Inmate
- NULL - Movement Code Correction - No Status Change
release_date: release_date
release_reason:
$enum_mapping:
$raw_text: release_reason
$mappings:
StateIncarcerationPeriodReleaseReason.COMMUTED:
- Commutation
StateIncarcerationPeriodReleaseReason.CONDITIONAL_RELEASE:
- Home Arrest 41-1604.13
- Provisional Release
- WK F 41-160411C # Work Furlough
- Discretionary Release
- Flat Discretionary Release
StateIncarcerationPeriodReleaseReason.COMPASSIONATE:
- Unescorted Compassionate Leave
StateIncarcerationPeriodReleaseReason.COURT_ORDER:
- Release by Court
- Out on Bond
StateIncarcerationPeriodReleaseReason.DEATH:
- Death
StateIncarcerationPeriodReleaseReason.ESCAPE:
- Escaped
- AWOL/Abscond
- Releasee Abscond
- Return from Escape
StateIncarcerationPeriodReleaseReason.EXECUTION:
- Execution
StateIncarcerationPeriodReleaseReason.PARDONED:
- Full Pardon
StateIncarcerationPeriodReleaseReason.RELEASED_FROM_ERRONEOUS_ADMISSION:
- Activated In Error
- Return to Custody in Error (Inactive)
StateIncarcerationPeriodReleaseReason.RELEASED_FROM_TEMPORARY_CUSTODY:
- Shock Incarceration Release
- SHOCK Incarceration End
- Activated In Error
StateIncarcerationPeriodReleaseReason.RELEASED_IN_ERROR:
- Deactivated in Error
- Supervision Closed in Error
- Release in Error Return
- Deactivated in Error
StateIncarcerationPeriodReleaseReason.RELEASED_TO_SUPERVISION:
- Community Supervision Release
- Correctional Release Center
- Correctional Release Center Release
- DC to Probation
- Drug Transition Release
- Parole (Early/Emergency)
- Absolute Discharge # Earned Release to Probation
- NEW CODE - Earned Release
- Paroled - Regular
- Standard Transition Release
- DC to Probation
- Mandatory Release
- Correctional Release Center
- Regular Parole - Juvenile
- Received CC/CS
- Provisional Release
- Discretionary Release
- Parole (Early/Emergency)
- Drug Transition Release
- Flat Discretionary Release
- WK F 41-160411C # Work release
- Home Arrest 41-1604.13
StateIncarcerationPeriodReleaseReason.SENTENCE_SERVED:
- Absolute Discharge
- Close Prison Episode
- Community Supervision End Date
- End of Supervision
- Mandatory Release
- NEW CODE - Earned Release
- OLD/NEW CODE - Expiration
- Standard Transition Release
- End of Supervision
- Shock Incarceration Release
- SHOCK Incarceration End
StateIncarcerationPeriodReleaseReason.STATUS_CHANGE:
- Close Supervision Episode - Leave Prison Episode Active
# These appear when someone has been placed in temporary custody
# due to some infraction and has their supervision revoked soon after.
- In Custody - Other
- STATUS_CHANGE
- Temporary Placement
- TIS Release Return
- Return from Abscond
- Community Supervision Revoked
- Parole Revoked-Regular
- Provisional Rel Rescinded/Revoked
- Home Arrest Rescinded/Revoked
- Mandatory Release Res/Rev
- Temporary Release Return
- Return Community Service
- Return from AWOL
- SB1291 Violated
- Work Furlough Rescinded/Revoked
- Drug Transition Violated
- Condition of Probation
- Reopen Last Prison Episode
- Parole Revoked/41.1604.11K
- RTRN CUSTDY ITP
- ERCD Release Return
- Return from Bond
- Discretionary Release Returnee
- 41-1604.16 Revoked
- Re-Open Last Prison Episode
- Close Supervision Episode - Active Inmate
- Reinstated - RUAP
- Recommitment
- New Commitment
StateIncarcerationPeriodReleaseReason.TEMPORARY_RELEASE:
- Out to Court
- Out on Bond
- Escorted Leave
- Out to Hospital
- Temporary Placement
- Return from Hospital
- Return from Court
- Return from Escorted Leave
- Unescorted Compassionate Leave
- Return from Unescorted Compassionate Leave
- Temporary Release
- Escorted Leave
StateIncarcerationPeriodReleaseReason.TRANSFER:
- ISC Commitment
- Overcrowding
- Transferred To
- Transfer Out - Prison/CIP Departure
- Transfer Out - Unit Departure
- Transfer WICC/ICC
- Transfer/Return Interstate
- Overcrowding
- Received from Overcrowding
- Transfer In - Unit Arrival
- Transfer WICC/ICC
- Transfer In - Prison/CIP Arrival
- Transfer Out - Prison/CIP Departure
- Received From
- Return from WICC/ICC
- Assign to Central Office
StateIncarcerationPeriodReleaseReason.TRANSFER_TO_OTHER_JURISDICTION:
- In Transit T
- Out of State/Federal
- Release ISC
StateIncarcerationPeriodReleaseReason.VACATED:
- Admin Action Case Closed
- Warrant Quashed
StateIncarcerationPeriodReleaseReason.INTERNAL_UNKNOWN:
# These are often admission reasons, and they appear as release reasons
# because the certain admissions are regularly duplicated in the data. These
# account for .7% of release reasons in the last 10 years, with at most 2% of
# mini-span release reasons appearing in this list in any of the last
# 10 years.
- Assign to Central Office
- Community Supervision Revoked
- Condition of Probation
- Deport 1/2 Term
- Discretionary Release Returnee
- Drug Transition Violated
- ERCD Release Return
- Home Arrest Rescinded/Revoked
- Mandatory Release Res/Rev
- New Commitment
- Parole Revoked-Regular
- Parole Revoked/41.1604.11K
- Provisional Rel Rescinded/Revoked
- Transit Driver
- 41-1604.16 Revoked
- REIN FROM SUSP.
- Received CC/CS
- Received From
- Received from Overcrowding
- Recommitment
- Reinstated - RUAP
- Reopen Last Prison Episode
- Return Community Service
- ISC Commitment
- Transfer/Return Interstate
- Return Deport 1/2 Term
- Return from AWOL
- Return from Abscond
- Return from Bond
- Return from Court
- Return from Escape
- Return from Escorted Leave
- Return from Hospital
- Return from Unescorted Compassionate Leave
- Return from WICC/ICC
StateIncarcerationPeriodReleaseReason.INTERNAL_UNKNOWN:
- In Transit T
- Transit Driver
- Admin Action Case Closed
- Close Prison Episode
- Class 1 Date Adjustment
- Received After 20 Days
- NULL - Movement Code Correction - No Status Change
- Rule 11
- SB1291 Violated
- TIS Release Return
- Temporary Release Return
- Work Furlough Rescinded/Revoked
- SHOCK Incarceration Commitment
- REIN FROM SUSP.
facility: facility
housing_unit: housing_unit
# All housing and custody level mappings rely on the assumption that a documented
Expand Down

0 comments on commit ca05752

Please sign in to comment.