Autocreate Zenodo DOIs and create shared access links #5879

kathy-t · 2024-05-06T18:01:07Z

Description

Updated PR Description

Zenodo swagger PR: https://github.com/dockstore/swagger-java-zenodo-client/pull/21

This PR has been updated with the feedback from the story time discussion.

This PR requires the creation of a Dockstore-owned personal access token on Zenodo as well as a Dockstore community on Zenodo. This personal access token is used to automatically create DOIs.

Automatic DOI Creation Behaviour

DOIs are automatically created for valid published tags for Workflow-based entries on refresh, GitHub App release, and on publish. During automatic DOI creation, we skip creating a snapshot.
The DOI is only created the first time it is invoked. If tags are recreated, we do not create a new DOI. This can be an enhancement if we want to go down that route.
- I originally didn't do it on publish but after manual testing, the user experience to automatically create DOIs for a new workflows that aren't originally published felt clunky. For example, a user creates a new GitHub app workflow that isn't published so no DOIs are created. They publish it. They must now generate push events for DOIs to be automatically created.
There is no option to opt-out.
A new Doi entity is added so that we can track multiple DOIs from multiple sources (user, GitHub, Dockstore). This means that users can have both user-generated and Dockstore-generated DOIs, along with GitHub DOIs.
- I have deprecated the old DOI columns instead of removing them so that there is no down time during the release. There is no trigger to keep the new and old columns in sync. Is it worth it to add a trigger?
Note that we don't allow the deletion of entries that were previously published. Since the auto DOI creation only occurs for published tags, this means that DOIs in our database won't be orphaned.
Tags with only Dockstore-generated DOIs can be deleted since they are not frozen...this can result in an orphaned DOI.

Dockstore Zenodo Community

New DOIs, whether user or Dockstore generated, are added to the Dockstore Zenodo community. We do not add old records to the community retro-actively.

Shared Access Links

Three endpoints are added to manage shared access links that allow users to edit the Dockstore-generated DOIs. The access link returns an ID and a token. When this token is appended to the Zenodo DOI URL, the user is able to edit the record if they are logged into Zenodo. I did not store this token in the database because I feared that we could accidentally expose it. Only the link ID is stored and when requested, we can retrieve the token from Zenodo. This editing feature will likely not be used often.
The link has edit permissions and no expiration date. There is an endpoint to delete the access link.
The access link ID is stored in the concept DOI object because this link is applicable to all versions of the DOI

Hoverfly Tests

Created a few Zenodo tests using Hoverfly. These tests aren't super complex due to the limitations of mocked responses, but it's something 🙂
Created a new testing profile for tests that use Hoverfly because it needs the localtruststore in CircleCI

Follow-up

Hook up to the UI: https://ucsc-cgl.atlassian.net/browse/SEAB-6462

Draft PR Description

Currently in draft mode to get feedback as I figure out some Hoverfly test weirdness in CircleCI.

Zenodo swagger PR: dockstore/swagger-java-zenodo-client#21

This PR adds a enableAutomaticDoiCreation boolean to Entry that is set to true by default.

When set to true, it automatically creates DOIs using Dockstore's own Zenodo personal access token, specified in the web.yml. The auto-creation occurs during refresh and GitHub App release for valid, published tags and involves snapshotting the version prior to creating the DOI. This boolean can be updated using a new endpoint, allowing users to opt-out in the UI. For GitHub app entries, they will need to specify it in their .dockstore.yml.

Should this feature include hosted entries?

This boolean can be updated using a new endpoint, allowing users to opt-out in the UI. For GitHub app entries, they will need to specify it in their .dockstore.yml.

This PR also adds an endpoint to create a shared access link with edit permissions so entry owners can modify their DOIs if they want. There is an option to set an expiration date, but I set it to never expire. The idea is that in the UI, the entry owner can click a button that creates this shared access link. After the first time, the UI can display this link somewhere so the user can modify their DOI. UI PR will be in follow-up ticket.

Should we allow the user to delete this access link?

This PR also adds DOIs to the Dockstore community, using the community ID specified in the web.yml. This is for all DOIs, whether Dockstore-created or end-user-created.

Some questions and thoughts
There is a potential exception if an end user has already requested a DOI for an entry version and our Dockstore user tries to create a DOI for a different version because our Dockstore user does not have permissions to modify some one else's DOI. Due to this, there is a check to see if the entry has 0 DOIs or if all DOIs are Dockstore-owned to ensure that no exception occurs. I think there's a bigger conversation here about how we want to handle this situation so this PR takes the simplest route for now, i.e., not allowing the mixing of end-user-created and Dockstore-created DOIs. Below are some thoughts about it, but I think we should have a follow-up ticket to decide what to do.

Do we solve this with communities? Using a one-time endpoint, we could try to add all DOIs to the Dockstore community. This would give the Dockstore Zenodo user permission to edit the DOI and create versions for DOIs that are owned by end users.
Do we allow multiple users to create DOIs and we store multiple DOIs? (similar ticket https://ucsc-cgl.atlassian.net/browse/DOCK-1922)
How about the reverse where the Dockstore Zenodo user creates the DOI first and the end user wants to create their own DOI?
- Shared access links would allow them to create new versions of the DOI, but they would not "own" the DOI

Review Instructions
Describe if this ticket needs review and if so, how one may go about it in qa and/or staging environments.
For example, a ticket based on Security Hub, Snyk, or Dependabot may not need review since those services
will generate new warnings if the issue has not been resolved properly. On the other hand, an infrastructure
ticket that results in visible changes to the end-user will definitely require review.
Many tickets will likely be between these two extremes, so some judgement may be required.

Issue
https://ucsc-cgl.atlassian.net/browse/SEAB-5929
https://ucsc-cgl.atlassian.net/browse/SEAB-6371

Security and Privacy

If there are any concerns that require extra attention from the security team, highlight them here and check the box when complete.

Security and Privacy assessed

e.g. Does this change...

Any user data we collect, or data location?
Access control, authentication or authorization?
Encryption features?

Please make sure that you've checked the following before submitting your pull request. Thanks!

Check that you pass the basic style checks and unit tests by running mvn clean install
Ensure that the PR targets the correct branch. Check the milestone or fix version of the ticket.
Follow the existing JPA patterns for queries, using named parameters, to avoid SQL injection
If you are changing dependencies, check the Snyk status check or the dashboard to ensure you are not introducing new high/critical vulnerabilities
Assume that inputs to the API can be malicious, and sanitize and/or check for Denial of Service type values, e.g., massive sizes
Do not serve user-uploaded binary images through the Dockstore API
Ensure that endpoints that only allow privileged access enforce that with the @RolesAllowed annotation
Do not create cookies, although this may change in the future
If this PR is for a user-facing feature, create and link a documentation ticket for this feature (usually in the same milestone as the linked issue). Style points if you create a documentation PR directly and link that instead.

… hoverfly cert

kathy-t · 2024-05-06T18:11:19Z

dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ZenodoHelper.java

@@ -87,33 +261,25 @@ public static ZenodoDoiResult registerZenodoDOI(ApiClient zenodoClient, Workflow
                returnDeposit = depositApi.createDeposit(deposit);
                depositionID = returnDeposit.getId();
                depositMetadata = returnDeposit.getMetadata();
-                // Set the attribute that will reserve a DOI before publishing


Removed this because the DOI is automatically reserved the first time a deposit is created eblondel/zen4R#37 (comment)

kathy-t · 2024-05-06T18:11:52Z

dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ZenodoHelper.java

                // Retrieve the DOI so we can use it to create a Dockstore alias
                // to the workflow; we will add that alias as a Zenodo related identifier
-                // TODO clean this after https://github.com/dockstore/swagger-java-zenodo-client/pull/20/files is merged and released


Tackled this TODO comment due to updating the zenodo swagger version

denis-yuen · 2024-05-07T14:39:14Z

Due to this, there is a check to see if the entry has 0 DOIs or if all DOIs are Dockstore-owned to ensure that no exception occurs. I think there's a bigger conversation here about how we want to handle this situation so this PR takes the simplest route for now, i.e., not allowing the mixing of end-user-created and Dockstore-created DOIs.

I think this makes sense, basically allowing users to go from automatic to user-created but not vice versa

Do we allow multiple users to create DOIs and we store multiple DOIs?

A good observation, I guess we basically had this problem in some way already.

How about the reverse where the Dockstore Zenodo user creates the DOI first and the end user wants to create their own DOI?

I think we should allow this, there's a good chance the user may not know about the automatic doi feature until after it issues one, so we need to allow for this direction (automatic to end-user created)

denis-yuen

Will double-back, but starting the conversation

.circleci/config.yml

denis-yuen · 2024-05-07T14:44:52Z

dockstore-common/src/main/java/io/dockstore/common/yaml/Service12.java

@@ -30,6 +30,8 @@ public class Service12 extends AbstractYamlService implements Workflowish {

    private DescriptorLanguageSubclass subclass;

+    private boolean enableAutomaticDoiCreation = true;


nitpick, since this is the default, maybe this should be flipped to disableAutomaticDoiCreation or enableEndUserDoiCreation

denis-yuen · 2024-05-07T14:47:24Z

dockstore-webservice/generated/src/main/resources/pom.xml

@@ -108,7 +108,7 @@
    <dependency>
      <groupId>io.dockstore</groupId>
      <artifactId>swagger-java-zenodo-client</artifactId>
-      <version>2.0.2</version>
+      <version>2.0.3-zeta.0</version>


Just setting reminder to fix before merging

denis-yuen · 2024-05-07T14:48:46Z

dockstore-webservice/src/main/java/io/dockstore/webservice/core/VersionMetadata.java

@@ -78,6 +78,14 @@ public class VersionMetadata {
    @Enumerated(EnumType.STRING)
    protected Version.DOIStatus doiStatus;

+    @Column()
+    @Schema(description = "The token for the Zenodo access link to edit the Dockstore-owned DOI")
+    protected String doiEditAccessLinkToken;


There's a relationship between these two columns that can be enforced through a check constraint

svonworl · 2024-05-07T15:01:26Z

This PR also adds an endpoint to create a shared access link with edit permissions so entry owners can modify their DOIs if they want. There is an option to set an expiration date, but I set it to never expire. The idea is that in the UI, the entry owner can click a button that creates this shared access link. After the first time, the UI can display this link somewhere so the user can modify their DOI. UI PR will be in follow-up ticket.

Should we allow the user to delete this access link?

Whatever the editing mechanism, if it "leaks" (as the above link sounds like it could), it'd be great if there was a way to neutralize it.

svonworl · 2024-05-07T15:15:14Z

Regarding the concept of auto-snapshotting before generating the DOI, some thoughts:

Will automatically snapshotting all tags mess up anyone's release flow? Once the version is snapshotted, it's frozen. Our assumption is that a github tag is a permanent fixture, but does everyone use it that way? For example, imagine a repo where the stable tag is attached to the latest stable release and moves around. The first stable tag would be snapshotted, and then Dockstore would reject subsequent retaggings. Do people do stuff like that, and if so, should we support it?

If the answer to the last question above is yes, we might need to figure out what to do if a user accidentally/inadvertently freezes a version before they opt-out.

svonworl · 2024-05-07T15:30:47Z

...ore-webservice/src/main/java/io/dockstore/webservice/resources/AbstractWorkflowResource.java

@@ -667,6 +680,10 @@ private boolean createWorkflowsAndVersionsFromDockstoreYml(List<? extends Workfl
                        Workflow workflow = createOrGetWorkflow(workflowType, repository, user, workflowName, wf, gitHubSourceCodeRepo);
                        WorkflowVersion version = addDockstoreYmlVersionToWorkflow(repository, gitReference, dockstoreYml, gitHubSourceCodeRepo, workflow, latestTagAsDefault, yamlAuthors);

+                        // Automatically register a DOI if it's enabled and the workflow is published and the version is a valid tag
+                        if (canAutomaticallyCreateDockstoreOwnedDoi(workflow, version)) {
+                            automaticallyRegisterDockstoreOwnedZenodoDOI(workflow, version, user, this);


This new DOI creation code appears to run in the same database transaction that will create the version. So, if the transaction is rolled back or fails, a DOI will be created that references a version that doesn't exist in the Dockstore database. Recommend that the DOI creation code is executed after successful version creation.

svonworl · 2024-05-07T15:38:00Z

Regarding multiple DOIs, we've been discussing at least three sources of DOIs:

github-generated
user-generated Zenodo
dockstore-generated Zenodo

If you count "generate our own" custom DOIs, four sources. :)

Would be great if we could track all of them. Time for a new "DOI" database entity?

This reverts commit 78a7a52.

kathy-t · 2024-05-23T18:24:58Z

dockstore-webservice/src/main/java/io/dockstore/webservice/core/VersionMetadata.java

    protected String doiURL;

+    @ManyToMany(fetch = FetchType.EAGER, cascade = CascadeType.ALL)


This is a many to many relationship because I recall during the DOI harvesting storytime that @denis-yuen said we shouldn't have duplicate DOIs if we assign multiple workflows to a single DOI

denis-yuen · 2024-05-24T14:23:35Z

Is it worth it to add a trigger?

Does not seem like it, we can schedule a 1.16.1 ticket for example to delete the old columns

denis-yuen

some initial questions, this one is a big one so will probably do one (at least) more pass

denis-yuen · 2024-05-24T14:42:04Z

dockstore-integration-testing/src/test/java/io/dockstore/client/cli/EntryResourceIT.java

@@ -46,6 +48,7 @@
 @ExtendWith(SystemStubsExtension.class)
 @ExtendWith(MuteForSuccessfulTests.class)
 @ExtendWith(TestStatus.class)
+@Tag(HoverflyTest.NAME)


Is there a reason this is tagged when there are no other changes in the file?

There's a test in the file that uses hoverfly, but I realized it's an ORCID test so I'm going to move it to ORCIDIT

dockstore-integration-testing/src/test/java/io/dockstore/webservice/TokenResourceIT.java

dockstore-integration-testing/src/test/java/io/dockstore/webservice/ZenodoIT.java

denis-yuen · 2024-05-24T14:45:56Z

dockstore-integration-testing/src/test/java/io/dockstore/webservice/ZenodoIT.java

+        assertTrue(workflow.getConceptDois().isEmpty());
+        assertTrue(tagVersion.getDois().isEmpty());
+
+        // Publish workflow, should automatically create DOIs


How many versions does this go back?
Will likely be answered later

On publish, it will look at all versions

dockstore/dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java

Line 760 in 7d1b34a

publishedWorkflow.getWorkflowVersions().forEach(version -> automaticallyRegisterDockstoreOwnedZenodoDOI(workflow, version, user, this));

See comment elsewhere, can be follow-up ticket

denis-yuen · 2024-05-24T14:47:10Z

dockstore-webservice/generated/src/main/resources/pom.xml

@@ -108,7 +108,7 @@
    <dependency>
      <groupId>io.dockstore</groupId>
      <artifactId>swagger-java-zenodo-client</artifactId>
-      <version>2.0.2</version>
+      <version>2.0.3-zeta.2</version>


Probably need a real release for this

Reminder for this before merging

denis-yuen · 2024-05-24T14:48:36Z

dockstore-webservice/src/main/java/io/dockstore/webservice/core/Doi.java

+    private DoiCreator creator;
+
+    @Column(nullable = false)
+    @Schema(description = "The DOI name", requiredMode = RequiredMode.REQUIRED)


can add examples to openapi, I think this is the 10.1080/10509585.2015.1092083 bit?

Yeah, I'll add an example. I think the existing DOI column calls it doiUrl which is a little misleading because it's not a URL. Figured name was more appropriate based on the description on doi.org

A DOI name is a digital identifier of an object, any object — physical, digital, or abstract. DOIs solve a common problem: keeping track of things. Things can be matter, material, content, or activities.

dockstore-webservice/src/main/java/io/dockstore/webservice/core/Entry.java

denis-yuen · 2024-05-24T14:51:27Z

dockstore-webservice/src/main/java/io/dockstore/webservice/core/VersionMetadata.java

    protected String doiURL;

+    @ManyToMany(fetch = FetchType.EAGER, cascade = CascadeType.ALL)


Does it also work out to a limit of three here?

Could you please elaborate on https://github.com/dockstore/dockstore/pull/5879/files/6333c924f9561b0443c2134ca3993a16c58adaa1#r1612139110 with a concrete example maybe? Want to make sure I understand what we're trying to prevent...

Could you please elaborate on https://github.com/dockstore/dockstore/pull/5879/files/6333c924f9561b0443c2134ca3993a16c58adaa1#r1612139110 with a concrete example maybe? Want to make sure I understand what we're trying to prevent...

With @coverbeck's GitHub DOI harvesting work, if a multi-workflow repository has a GitHub DOI then we are going to assign it to all workflows in the repository.

Suppose that the DOI is 10.foo/bar and we have workflow A and workflow B that belong to the same repository. I believe the scenario we are trying to prevent is having duplicate DOI entities in our table, like two instances of DOI 10.foo/bar, one for workflow A and one for workflow B. We want a single DOI entity that is referenced by both workflow A and workflow B

denis-yuen · 2024-05-24T14:55:16Z

dockstore-webservice/src/main/resources/migrations.1.16.0.xml

@@ -17,7 +17,7 @@

 <databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd"
-                   context="1.15.0">
+                   context="1.16.0">


Hmmm, wonder if this means we will need to re-migrate qa

Hmm, I'm not sure, but doesn't hurt. Realized while writing the migrations that this value wasn't accurate

These changes do mean we'll have downtime when we deploy 1.16, right? The old web service couldn't run against the new schema? If yes, please update the 1.16 checklist.

These changes do mean we'll have downtime when we deploy 1.16, right? The old web service couldn't run against the new schema? If yes, please update the 1.16 checklist.

I don't think there should be any downtime because the old DOI columns still exist (they're just deprecated in this PR)

coverbeck

My comment about the number of DOIs that could get issued by Zenodo might require a bit work if we decide to tackle it, so I sort skimmed the rest of the PR. I'll do a more thorough review once we decide how to handle that.

dockstore-webservice/src/main/java/io/dockstore/webservice/core/Entry.java

coverbeck · 2024-05-24T16:58:07Z

dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ZenodoHelper.java

+    private static String checkAndGetDockstoreVersionDoiName(Workflow workflow) {
+        Optional<String> existingDockstoreVersionDoiURL = getAnExistingDOIForWorkflow(workflow, DoiCreator.DOCKSTORE);
+        if (existingDockstoreVersionDoiURL.isEmpty()) {
+            LOG.error(NO_DOCKSTORE_DOI);


It could be useful to log the workflow, although maybe we'll already know it from earlier in the logs.

dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ZenodoHelper.java

coverbeck · 2024-05-24T17:14:35Z

dockstore-integration-testing/src/test/resources/fixtures/createDepositResponse.json

@@ -0,0 +1,34 @@
+{
+  "conceptrecid": {{ randomIntegerRange 10000 99999 }},


Neat (and the Request.Path[3] further as well)! How did I not know about this? Where is the documented? I couldn't find it in a quick google search.

It's documented here 🙂 https://docs.hoverfly.io/en/latest/pages/keyconcepts/templating/templating.html

coverbeck · 2024-05-24T17:17:57Z

dockstore-webservice/src/main/resources/migrations.1.16.0.xml

@@ -17,7 +17,7 @@

 <databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd"
-                   context="1.15.0">
+                   context="1.16.0">


These changes do mean we'll have downtime when we deploy 1.16, right? The old web service couldn't run against the new schema? If yes, please update the 1.16 checklist.

dockstore-webservice/src/main/java/io/dockstore/webservice/core/Entry.java

svonworl · 2024-05-25T00:57:05Z

dockstore-webservice/src/main/java/io/dockstore/webservice/core/VersionMetadata.java

    protected String doiURL;

+    @ManyToMany(fetch = FetchType.EAGER, cascade = CascadeType.ALL)


Could you please elaborate on https://github.com/dockstore/dockstore/pull/5879/files/6333c924f9561b0443c2134ca3993a16c58adaa1#r1612139110 with a concrete example maybe? Want to make sure I understand what we're trying to prevent...

svonworl · 2024-05-25T01:10:10Z

...ore-webservice/src/main/java/io/dockstore/webservice/resources/AbstractWorkflowResource.java

                        WorkflowVersion version = addDockstoreYmlVersionToWorkflow(repository, gitReference, dockstoreYml, gitHubSourceCodeRepo, workflow, latestTagAsDefault, yamlAuthors);
-
+                        workflowVersionId.set(version.getId());


As part of https://ucsc-cgl.atlassian.net/browse/SEAB-6449, I can modify TransactionHelper to include a method that allows us to run a method in a transaction that returns a result. Then, we don't need to track the ids with the AtomicLongs if we opt not to.

kathy-t · 2024-05-27T20:00:25Z

dockstore-webservice/src/main/java/io/dockstore/webservice/core/Version.java

@@ -304,7 +307,6 @@ public void setDirtyBit(boolean dirtyBit) {
    void updateByUser(final Version<?> version) {
        this.getVersionMetadata().hidden = version.isHidden();
        this.setDoiStatus(version.getDoiStatus());
-        this.setDoiURL(version.getDoiURL());


This code allows user to modify the DOI URL. I don't believe this is used and we probably shouldn't allow the user to edit DOI data so I removed this line

denis-yuen

Looks good as a first past, will be good to iterate/extend

denis-yuen · 2024-05-28T15:13:44Z

dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java

@@ -398,6 +374,10 @@ private Workflow refreshWorkflow(User user, Long workflowId, Optional<String> ve
        // Update file formats in each version and then the entry
        FileFormatHelper.updateFileFormats(existingWorkflow, newWorkflow.getWorkflowVersions(), fileFormatDAO, true);

+        for (WorkflowVersion workflowVersion: existingWorkflow.getWorkflowVersions()) {


I'm a little doubtful about the utility of automatically creating a DOI for all versions of a workflow without limit.
Thinking things like a hard limit of the most recent 30 versions, or 20 tags (since only tags should be eligible), etc.
Partly a performance thing, can be a follow-up ticket. Think manually testing with a fork of the gatk repo for example.

I changed it so it only looks at the 10 most recent tags

denis-yuen · 2024-05-28T15:14:24Z

dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java

@@ -807,6 +757,7 @@ public Workflow publish(@ApiParam(hidden = true) @Parameter(hidden = true, name
        checkNotArchived(workflow);

        Workflow publishedWorkflow = publishWorkflow(workflow, request.getPublish(), userDAO.findById(user.getId()));
+        publishedWorkflow.getWorkflowVersions().forEach(version -> automaticallyRegisterDockstoreOwnedZenodoDOI(workflow, version, user, this));
        Hibernate.initialize(publishedWorkflow.getWorkflowVersions());


Ditto with above comment

denis-yuen · 2024-05-28T15:14:45Z

dockstore-integration-testing/src/test/java/io/dockstore/webservice/ZenodoIT.java

+        assertTrue(workflow.getConceptDois().isEmpty());
+        assertTrue(tagVersion.getDois().isEmpty());
+
+        // Publish workflow, should automatically create DOIs


See comment elsewhere, can be follow-up ticket

dockstore-webservice/src/main/java/io/dockstore/webservice/core/Doi.java

coverbeck · 2024-05-29T18:17:12Z

dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ZenodoHelper.java

+                registerZenodoDOI(zenodoClient, workflow, workflowVersion, workflowOwner, authenticatedResourceInterface,
+                        DoiCreator.DOCKSTORE);
+            } catch (CustomWebApplicationException e) {
+                LOG.error("Could not automatically register DOI for {}", workflowNameAndVersion(workflow, workflowVersion), e);


This is going to fail silently for the end user and we probably won't notice it either. I wonder if it's worth capturing this somehow?

A CloudWatch Alarm based on this message?

Add it as a warning to validations, if possible?

dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ZenodoHelper.java

dockstore-webservice/src/main/java/io/dockstore/webservice/core/Entry.java

dockstore-webservice/src/main/java/io/dockstore/webservice/core/Doi.java

pom.xml

svonworl · 2024-05-30T00:26:36Z

dockstore-webservice/src/main/java/io/dockstore/webservice/core/Doi.java

+
+    public enum DoiType {
+        CONCEPT,
+        VERSION


If I understand correctly, "concept" and "version" are Zenodo concepts. The DOI standard has no such abstractions, you can point a DOI at whatever you want. No change necessary here.

svonworl · 2024-05-30T00:49:03Z

dockstore-webservice/src/main/java/io/dockstore/webservice/core/Entry.java

+    @MapKeyEnumerated(EnumType.STRING)
+    @Size(max = MAX_NUMBER_OF_DOI_CREATORS)
+    @Schema(description = "The Digital Object Identifier (DOI) representing all of the versions of your workflow")
+    private Map<DoiCreator, Doi> conceptDois = new HashMap<>();


From a maintainability/extensibility point of view, I'd probably make this a Set, because the above Map implementation forecloses supporting a future scheme where an entry/version could have multiple DOIs from the same creator (could happen someday, maybe an "UNKNOWN" creator). But, it's not wired into the database representation, which wouldn't make it super hard to change. I'm registering this thought here more to document it, less to press for a change...

sonarcloud · 2024-05-31T16:25:09Z

Quality Gate failed

Failed conditions
79.4% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

denis-yuen · 2024-05-31T17:14:46Z

dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ZenodoHelper.java

@@ -839,6 +844,11 @@ public static void checkCanRegisterDoi(Workflow workflow, WorkflowVersion workfl
            throw new CustomWebApplicationException(String.format("Could not generate DOI for %s. %s", workflowNameAndVersion, FROZEN_VERSION_REQUIRED), HttpStatus.SC_BAD_REQUEST);
        }

+        if (workflowVersion.isHidden()) {
+            LOG.error("{}: Could not generate DOI for {}. {}", user.getUsername(), workflowNameAndVersion, UNHIDDEN_VERSION_REQUIRED);
+            throw new CustomWebApplicationException(String.format("Could not generate DOI for %s. %s", workflowNameAndVersion, UNHIDDEN_VERSION_REQUIRED), HttpStatus.SC_BAD_REQUEST);


Could add the explanation for the user message though we'll have it in the logs. UI requires a snapshot anyway, so not big deal

kathy-t added 11 commits May 1, 2024 13:02

Add flag for automatic doi creation by dockstore zenodo account

0f3515f

Merge branch 'develop' into feature/zenodo-prototype

1dca34c

Use existing community object

f77e4a9

Use 2.0.3-zeta.0 tag for zenodo client

ce8b4ce

Allow automatic DOIs for services

be9664d

Add hoverfly tests for automatic DOI creation - GH apps

1cafb53

Weird, how about this

24ed60e

Try extending BaseIT to mimic WebhookIT which doesn't seem to use the…

421c78c

… hoverfly cert

Remove unused imports

7278ab4

Use hoverfly cert for integration tests

7749335

Revert hoverfly fix attempts

3261305

kathy-t self-assigned this May 6, 2024

kathy-t commented May 6, 2024

View reviewed changes

kathy-t mentioned this pull request May 6, 2024

Add link sharing endpoint and modify community object for automatic DOI creation dockstore/swagger-java-zenodo-client#21

Merged

9 tasks

kathy-t requested review from svonworl, denis-yuen, coverbeck and hyunnaye May 6, 2024 18:20

kathy-t added 4 commits May 6, 2024 16:16

Create hoverfly-tests profile

7202265

Fix typos

b57a4b7

Exclude

11bf333

Disable migrations cache

22b5075

denis-yuen reviewed May 7, 2024

View reviewed changes

svonworl reviewed May 7, 2024

View reviewed changes

Comment out problematic line to see what happens

78a7a52

Revert "Comment out problematic line to see what happens"

afec59e

This reverts commit 78a7a52.

kathy-t commented May 23, 2024

View reviewed changes

denis-yuen reviewed May 24, 2024

View reviewed changes

coverbeck reviewed May 24, 2024

View reviewed changes

kathy-t added 6 commits May 24, 2024 16:46

Fix orcid export tests

299b8ff

Move ORCID hoverfly tests from EntryResourceIT to OrcidIT

88612e6

Fix more tests: don't allow DOI editing

665e809

Remove hoverfly tag from EntryResourceIT

5af2333

Remove unused import

fff2878

Fix checkstyle

7d1b34a

svonworl reviewed May 25, 2024

View reviewed changes

kathy-t added 2 commits May 27, 2024 15:19

PR feedback

bf4431c

Merge branch 'develop' into feature/zenodo-prototype

3c63413

kathy-t commented May 27, 2024

View reviewed changes

kathy-t requested review from coverbeck, denis-yuen and svonworl May 27, 2024 20:00

denis-yuen approved these changes May 28, 2024

View reviewed changes

coverbeck approved these changes May 29, 2024

View reviewed changes

kathy-t mentioned this pull request May 29, 2024

DOCK-2523: Fix formatting of conceptDOI #5894

Open

10 tasks

Update zenodo client version

37fee58

svonworl approved these changes May 30, 2024

View reviewed changes

kathy-t added 4 commits May 30, 2024 11:16

PR feedback

2a92f49

Fix tests, more cleanup

810394a

Merge branch 'develop' into feature/zenodo-prototype

58bb2b5

Fix tests

f5f000f

kathy-t requested review from coverbeck and denis-yuen May 31, 2024 15:34

denis-yuen approved these changes May 31, 2024

View reviewed changes

		@@ -30,6 +30,8 @@ public class Service12 extends AbstractYamlService implements Workflowish {

		private DescriptorLanguageSubclass subclass;

		private boolean enableAutomaticDoiCreation = true;

		protected String doiURL;

		@ManyToMany(fetch = FetchType.EAGER, cascade = CascadeType.ALL)

		@@ -0,0 +1,34 @@
		{
		"conceptrecid": {{ randomIntegerRange 10000 99999 }},

		WorkflowVersion version = addDockstoreYmlVersionToWorkflow(repository, gitReference, dockstoreYml, gitHubSourceCodeRepo, workflow, latestTagAsDefault, yamlAuthors);

		workflowVersionId.set(version.getId());

Autocreate Zenodo DOIs and create shared access links #5879

Are you sure you want to change the base?

Autocreate Zenodo DOIs and create shared access links #5879

Conversation

kathy-t commented May 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denis-yuen commented May 7, 2024

denis-yuen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svonworl commented May 7, 2024

svonworl commented May 7, 2024 • edited

Choose a reason for hiding this comment

svonworl commented May 7, 2024 • edited

Choose a reason for hiding this comment

denis-yuen commented May 24, 2024

denis-yuen left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coverbeck left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denis-yuen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented May 31, 2024

Quality Gate failed

Choose a reason for hiding this comment

kathy-t commented May 6, 2024 •

edited

svonworl commented May 7, 2024 •

edited

svonworl commented May 7, 2024 •

edited

denis-yuen left a comment •

edited