Feature/edan verifier utility #531

EMaslowskiQ · 2023-01-31T18:49:21Z

Creation of EDAN verifier utility.

Accessed through endpoint: /verifiers/edan
Uses Workflows for managing execution
Uses WorkflowReports for storing results
Merged changes made to 'develop' and tested

* WIP snapshot of EDAN verifier * compares subject units and identifiers outputting results to CSV * currently runs as a test case in EdanCollection.test.ts

* WIP commit capturing refactor of EDAN verifier * lacks cleanup and testing

* WIP commit for EDAN Verifier utility * cleaned out legacy code prior to refactor * limiting returned EDAN records to 1, otherwise throw error * fix issues with ARK ids not being outputted if also set to preferred * added basic determination of DPO vs. EDAN subject sources * support for name comparisons * support for updating Packrat units from EDAN

* WIP commit for EDAN Verifier utility * support for name comparisons * support for updating Packrat units from EDAN * improved messaging for automatic/manual fixing

* WIP commit for EDAN verifier * added EDAN MDM column to ouput * added ability to run a specific subject ID for debugging

* WIP commit for EDAN verifier * moved EDAN verifier into own class derived from VerifierBase * cleaned up how EDAN units are collected * opting for comparing abbreviations when using fetchFromNamedSearch and receiving multiple hits

* WIP commit for EDAN verifier * removed fallback for checking multiple EDAN units and opting for comparing abbreviations * verifier now accepts EdanVerifierConfig object with settings * moved all temporary debug statements to standard LOG

* EDAN Verifier updated to support running from any endpoint (/verifier/edan) * support query params for basic configuration * returns a CSV file if given param of: returnFile=true

* using WorkflowEngine for running the EDAN verifier (WorkflowVerifier) * full support of query params for configuring verifier (accepts: objectId, limit, returnFile, detailedLogs) * HTTP response includes hyperlink for downloading the report * added 'Name' property to WorkflowReport to assist in filename creation

* cleaned up Packrat schema. comment prevented rebuilding the DB with initdb * created utility routine that gathers all SystemObject details (via GraphQL) and returns as NDJSON. Not accessible.

* tested against 'develop' branch * added cleaner/styled output for response

* updated model graph (mwb) to reflect changes in DB

* fixed eslint error with resolving VerifierBase

jahjedtieson

Your first PR! What a great milestone!

The changes to server/tests/jest.config.js must not be committed. Doing so will disable regression tests from being run, by default, by other users, and by GitHub CI.
We don't add server/db/sql/models/Packrat.mwb.bak to our source tree.
server/utils/verifiers/VerifierBase.ts's replacePackratUnit() is doing the wrong thing. Instead of updating Unit records, it needs to update Subject.idUnit (i.e point the subject at the right unit).
I've asked a number of inline questions/provided feedback.

server/db/api/WorkflowReport.ts

server/http/index.ts

server/http/routes/download.ts

server/http/routes/verifiers.ts

jahjedtieson · 2023-01-31T23:24:50Z

server/utils/verifiers/VerifierBase.ts

+                }
+
+                // if identifier exists in our database (value & type) then store it
+                const identifiers: DBAPI.Identifier[] | null = await DBAPI.Identifier.fetchFromIdentifierValue(content);


I'm confused by the intention of this code. fetchFromIdentifierValue is going to get all identifiers that match the specified value ... this may include matches for identifiers with the same value, but applied to other system objects, and/or of random identifier types. In other words, identifiers are not guaranteed to have a unique value....

server/utils/verifiers/VerifierBase.ts

server/utils/verifiers/EdanVerifier.ts

server/http/routes/dataQueries.ts

* formatting and cleanup for submission

…ithsonian/dpo-packrat into feature/EDAN-verifier-utility

jahjedtieson

Please remove server/db/sql/models/Packrat.mwb.bak from this PR.

* changed LOG type top eSYS for better grouping * clearer definition of VerifierBase types and their use * comments describing future improvements

EMaslowskiQ added 16 commits January 18, 2023 15:11

Utility:

5e6a368

* WIP snapshot of EDAN verifier * compares subject units and identifiers outputting results to CSV * currently runs as a test case in EdanCollection.test.ts

Utility:

cb5de99

* WIP commit capturing refactor of EDAN verifier * lacks cleanup and testing

Utility:

c4afeb7

* WIP commit for EDAN Verifier utility * support for name comparisons * support for updating Packrat units from EDAN * improved messaging for automatic/manual fixing

Utility:

017d3b5

* WIP commit for EDAN verifier * added EDAN MDM column to ouput * added ability to run a specific subject ID for debugging

Utility:

48514ed

* WIP commit for EDAN verifier * moved EDAN verifier into own class derived from VerifierBase * cleaned up how EDAN units are collected * opting for comparing abbreviations when using fetchFromNamedSearch and receiving multiple hits

Utility:

1a61589

* WIP commit for EDAN verifier * removed fallback for checking multiple EDAN units and opting for comparing abbreviations * verifier now accepts EdanVerifierConfig object with settings * moved all temporary debug statements to standard LOG

Utility:

dca915c

* EDAN Verifier updated to support running from any endpoint (/verifier/edan) * support query params for basic configuration * returns a CSV file if given param of: returnFile=true

Utility:

b7ca8a7

* cleaned up Packrat schema. comment prevented rebuilding the DB with initdb * created utility routine that gathers all SystemObject details (via GraphQL) and returns as NDJSON. Not accessible.

Merge remote-tracking branch 'origin/develop'

8d93a6c

Utility:

4d5ff0f

* tested against 'develop' branch * added cleaner/styled output for response

Utility:

aa291a6

* updated model graph (mwb) to reflect changes in DB

Utility:

0fb99f6

* fixed eslint error with resolving VerifierBase

Merge branch 'develop' into feature/EDAN-verifier-utility

73b3a0e

Rename verifierBase.ts to VerifierBase.ts

ef4002d

jahjedtieson requested changes Jan 31, 2023

View reviewed changes

EMaslowskiQ added 2 commits January 31, 2023 19:16

Utility:

7dbe9dc

* formatting and cleanup for submission

Merge branch 'feature/EDAN-verifier-utility' of https://github.com/Sm…

94af4b6

…ithsonian/dpo-packrat into feature/EDAN-verifier-utility

jahjedtieson reviewed Feb 1, 2023

View reviewed changes

EMaslowskiQ marked this pull request as draft February 1, 2023 01:54

Utility:

756198b

* changed LOG type top eSYS for better grouping * clearer definition of VerifierBase types and their use * comments describing future improvements

EMaslowskiQ force-pushed the develop branch from cac0d06 to b05e532 Compare January 24, 2024 22:01

EMaslowskiQ force-pushed the develop branch from 6563288 to 178c52e Compare February 1, 2024 03:51

EMaslowskiQ force-pushed the develop branch 2 times, most recently from 008e5d6 to aa7535c Compare May 13, 2024 15:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/edan verifier utility #531

Feature/edan verifier utility #531

EMaslowskiQ commented Jan 31, 2023

jahjedtieson left a comment

jahjedtieson Jan 31, 2023

jahjedtieson left a comment

Feature/edan verifier utility #531

Are you sure you want to change the base?

Feature/edan verifier utility #531

Conversation

EMaslowskiQ commented Jan 31, 2023

jahjedtieson left a comment

Choose a reason for hiding this comment

jahjedtieson Jan 31, 2023

Choose a reason for hiding this comment

jahjedtieson left a comment

Choose a reason for hiding this comment