From ed714b4f4e1389b0f776eb682bee1fba38d8fb0d Mon Sep 17 00:00:00 2001 From: Marcos Lopez Gonzalez Date: Fri, 24 Jan 2020 10:54:22 +0100 Subject: [PATCH] Collections catalogue (#168) * [maven-release-plugin] prepare for next development iteration * Update gbif-doi to version 2.7 * [maven-release-plugin] prepare release registry-2.120 * [maven-release-plugin] prepare for next development iteration * Upgrade to API with fixed notification_addresses key. https://github.com/gbif/portal-feedback/issues/2046 * Changes to endorsement email. Requested in https://github.com/gbif/portal-feedback/issues/2126 * Omit repeated Download objects from DatasetOccurrenceDownloadUsage responses. Resolves #134. * Update download-query-tools to support huge downloads with many taxa. * Released versions. * [maven-release-plugin] prepare release registry-2.121 * [maven-release-plugin] prepare for next development iteration * Hack XML validation test to pass, avoiding redirect to HTTPS for the DC schema. * [maven-release-plugin] prepare release registry-2.122 * [maven-release-plugin] prepare for next development iteration * Update API version, for download predicate limits/changes. https://github.com/gbif/occurrence/issues/50 * Always include ENDORSE link in new publisher emails. * added earthCape installation type * updated gbif-common-ws version * updated common-mybatis version * [maven-release-plugin] prepare release registry-2.123 * [maven-release-plugin] prepare for next development iteration * updated gbif-api version * updated gbif-api version * [maven-release-plugin] prepare release registry-2.124 * [maven-release-plugin] prepare for next development iteration * SQLDownloadRequest was replaced with SqlDownloadRequest * Implement search by installation type. * Allow editors to see their organization's shared token. Resolves #121. * Align DataCite metadata with citation guidelines. Resolves #137. * Allow dataset editors to edit default-term.gbif.org machine tags. Resolves 120. * Fix copy-paste error. * Allow deleting default-term machine tags. Resolves #120. * Add missing Liquibase change. * pipelines history tracking service migrated to the registry * tests pipeline process ws * adding a crawlall endpoint and supporting ther platform parameter * moving page size to constant * cleanup * Correction to test. * cleanup * added tests pipelines * javadoc * pipelinesModule changed not to install postal service * Check node permissions when setting endorsement. Resolves #140. * Released version. * removing datasetTilte from PipelineProcess + pipelines enums added to ws * updated gbif-api version * fixed enumeration resource test * pipelines history: added tests + small fixes * added metrics to pipelines history * added pipelines properties to test resource * index url for pipelines metrics * added log * changed metrics type handler not to store empty values in DB * fix metrics url * fixed url creation for pipelines metrics * last attempt throws exception if not found * not throwing exception when a crawl dataset fails * updated versions of gbif-api and postal-service * modified loops for crawl all and rerun all pipelines when dataset fails * added logs * cleanup * [maven-release-plugin] prepare release registry-2.125 * [maven-release-plugin] prepare for next development iteration * fix bug in rerun all pipelines steps * fix loop run and crawl all pipelines * crawAll and runAll pipelines executed async * less verbose logs * [maven-release-plugin] prepare release registry-2.126 * [maven-release-plugin] prepare for next development iteration * replaced insert with upsert to create pipelines history process to avoid concurrency issues when calling from crawler * [maven-release-plugin] prepare release registry-2.127 * [maven-release-plugin] prepare for next development iteration * handling TO_VERBATIM step by transform it into a for specific step for ABCD, DWCA and XML * using latest api that has the TO_VERBATIM step * [maven-release-plugin] prepare release registry-2.128 * [maven-release-plugin] prepare for next development iteration * Update postal-service to 0.38 * Fix compilation error in DatasetResource, StartCrawlMessage constructor parameters * Changed mybatis.version to the old (TIMESTAMP issue) * Improve DatasetProcessStatusIT * adapted dataset process status to new mybatis version * pipelines history ordered by created date * deffensive checks for ES metrics * [maven-release-plugin] prepare release registry-2.129 * [maven-release-plugin] prepare for next development iteration * #152 returning json response when steps are null * pipeline steps ordered in SQL query * updated gbif-api version * changed the check of input params in pipelines history * get DOI URL decoded for citations * Decoding DOI URL in citation * updated gbif-api version * [maven-release-plugin] prepare release registry-2.130 * [maven-release-plugin] prepare for next development iteration * #156 Refactor, fix geoLocation mapping part * Improve DatasetProcessStatusIT * #156 refactoring and geoLocation mapping * Reorganize classes in registry-doi * Replace DataCiteConverter with specific ones DownloadConverter or DatasetConverter * Fix DownloadConverter#truncateDescriptionDCM and tests * Refactor DatasetConverter * Refactor DatasetConverter * Improve DatasetConverterTest, add RegistryDoiUtils * Refactor DownloadConverter * Refactor DownloadConverterTest * Fix RegistryDoiUtilsTest date problem * Fix DatasetConverterTest and DownloadConverterTest date issue * CustomDownloadDataCiteConverter * Improve language mapping for DatasetConverter * [maven-release-plugin] prepare release registry-2.131 * [maven-release-plugin] prepare for next development iteration * fix bug when running all and crawling all datasets * [maven-release-plugin] prepare release registry-2.132 * [maven-release-plugin] prepare for next development iteration * added checks for empty metrics from ES in pipelines history * Fixed pipelines message order, monitoring and index prefix for doOnAll * added number of records to pipeline process + fix new steps * cleaned import * added number of records in pipeline process * added number of records in pipeline process * added log for number of records in pipeline process * Updated gbif-postal-service.version * updated gbif-api version * [maven-release-plugin] prepare release registry-2.133 * [maven-release-plugin] prepare for next development iteration * small refactor * Update README.md * Refactor NodeIT * Refactor NetworkEntityTest#testUpdate * Refactor NetworkEntityTest * Fix LenientAssert * Cleanup NodeResource * Reformat ws/security package * changed ES metrics type handler to avoid issues with unexpected values * run all pipelines and crawl all now include sampling event datasets too * [maven-release-plugin] prepare release registry-2.134 * [maven-release-plugin] prepare for next development iteration * [maven-release-plugin] prepare release registry-2.135 * [maven-release-plugin] prepare for next development iteration * [maven-release-plugin] prepare release registry-2.136 * [maven-release-plugin] prepare for next development iteration * [maven-release-plugin] prepare release registry-2.137 * [maven-release-plugin] prepare for next development iteration * fixed runAll and crawAll for pipelines * [maven-release-plugin] prepare release registry-2.138 * [maven-release-plugin] prepare for next development iteration * added PipelineProcessView to show a custom view in the registry-console * added checklist datasets to runAll and crawlAll + datasetTitle to process * added checklist datasets to runAll and crawlAll + datasetTitle to process * changed test DOIs * added checks for number of records in pipelines process * added checks for number of records in pipelines process * crawling all datasets since even some METADATA only datasets are associated to occurrence records * crawAll includes now all datasets * [maven-release-plugin] prepare release registry-2.139 * [maven-release-plugin] prepare for next development iteration * Reorder filters in RegistryWsServletListener, EditorFilter must be the last one * Add additional checks to EditorAuthorizationFilter * EditorAuthorizationFilter improve user is null case * EditorAuthorizationFilter improvements * EditorAuthorizationFilter change regex pattern in order to match the whole path * EditorAuthorizationFilter check methods return void * EditorAuthorizationFilter exclude endorsement and machine tags * added filter to exclude some datasets in crawAll and runAll pipelines * [maven-release-plugin] prepare release registry-2.140 * [maven-release-plugin] prepare for next development iteration * added workaround to ignore Optional values in pipelines history * revert workaround PipelinesAbdcMessage * updated gbif-postal-service version * pipelines history minor changes * updated postal-service version * updated postal-service version * [maven-release-plugin] prepare release registry-2.141 * [maven-release-plugin] prepare for next development iteration * ingestion service that merges crawl and pipelines history * ingestion service that merges crawl and pipelines history * ingestion service that merges crawl and pipelines history * removed MetricsHandler and added tests * test versions * test versions * fixed test * added remarks in PipelineProcessMapper.xml + fix tests * fix ingestion history when pipeline process doesn't exist * updated cloudera version * changes type of steps to run to be text instead of enum * minor improvements pipelines history * #159 Skeleton code for Index Herbariorum synchronization * improved response of run pipeline attempt + improved order of pipeline history * taking basicRecordsCountAttempted as number of records for verbatimToInterpreted step * updated versions to release (including cdh 5.12.0) * [maven-release-plugin] prepare release registry-2.142 * [maven-release-plugin] prepare for next development iteration * fix sorting in pipelines history * fix sorting in pipelines history * [maven-release-plugin] prepare release registry-2.143 * [maven-release-plugin] prepare for next development iteration * optimized method to get ingestion history to do less queries since this method is used very often by the UI * fix case when there is no dataset process statues in ingestion history * fix case when there is no dataset process statues in ingestion history * fix case when there is no dataset process statues in ingestion history * adapted classes for the http calls + entity converter + github client + extended grscicoll model * sync staff + refactor to make it easier to test * sync staff + refactor to make it easier to test * added tests * added tests * added cliSyncApp skeleton * removed lombok builders in entities used in WS because they need public constructor * CliSyncApp + tests * github issues assignees externalized to properties + fixes format diff file * added failed actions + improvements * fix test * added links to entities in GH issues + mapping IH countries to our enum * rollback test * mapping countries from IH to our enum + gh issues links + tests * improved country mapping * minor fixes * issues moved out from diff finder + issues for fails + using map for matches * config file for tests * changed config test * check for duplicate codes in grscicoll + added search by code and name * code unique * made GrSciColl entities machine taggable * adding identifiers manually to person in IH-sync * removed files pushed by mistake * removed check for duplicate codes + added numberSpecimens to collections * removed TODO Co-authored-by: GBIF Jenkins Bot Co-authored-by: Matt Blissett Co-authored-by: Mikhail Podolskiy Co-authored-by: Federico Mendez Co-authored-by: Nikolay Volik Co-authored-by: Tim Robertson --- pom.xml | 1 + registry-collections-sync/README.md | 3 + registry-collections-sync/pom.xml | 119 ++++++ .../registry/collections/sync/CliSyncApp.java | 107 +++++ .../registry/collections/sync/SyncConfig.java | 100 +++++ .../collections/sync/diff/DiffResult.java | 119 ++++++ .../sync/diff/DiffResultExporter.java | 197 +++++++++ .../sync/diff/DiffResultHandler.java | 244 +++++++++++ .../sync/diff/EntityConverter.java | 404 ++++++++++++++++++ .../sync/diff/IndexHerbariorumDiffFinder.java | 193 +++++++++ .../sync/diff/StaffDiffFinder.java | 362 ++++++++++++++++ .../registry/collections/sync/diff/Utils.java | 57 +++ .../sync/grscicoll/GrSciCollHttpClient.java | 230 ++++++++++ .../sync/http/BasicAuthInterceptor.java | 23 + .../collections/sync/http/SyncCall.java | 44 ++ .../collections/sync/ih/IHEntity.java | 10 + .../collections/sync/ih/IHHttpClient.java | 76 ++++ .../collections/sync/ih/IHInstitution.java | 46 ++ .../collections/sync/ih/IHMetadata.java | 11 + .../registry/collections/sync/ih/IHStaff.java | 37 ++ .../sync/notification/GithubClient.java | 69 +++ .../collections/sync/notification/Issue.java | 17 + .../sync/notification/IssueFactory.java | 228 ++++++++++ .../collections/sync/SyncConfigTest.java | 30 ++ .../sync/diff/EntityConverterTest.java | 289 +++++++++++++ .../diff/IndexHerbariorumDiffFinderTest.java | 339 +++++++++++++++ .../sync/diff/StaffDiffFinderTest.java | 368 ++++++++++++++++ .../src/test/resources/sync-config.yaml | 13 + .../identity/util/PasswordEncoderTest.java | 1 + .../main/resources/liquibase/065-ih-sync.xml | 133 ++++++ .../src/main/resources/liquibase/master.xml | 1 + .../gbif/registry/ws/client/GenericTypes.java | 2 +- .../ws/client/collections/BaseClient.java | 161 +++++++ .../ws/client/collections/BaseCrudClient.java | 45 -- ...> BaseExtendedCollectionEntityClient.java} | 50 +-- .../collections/CollectionWsClient.java | 12 +- .../collections/InstitutionWsClient.java | 22 +- .../ws/client/collections/PersonWsClient.java | 22 +- .../registry/events/VarnishPurgeListener.java | 24 +- .../registry/persistence/WithMyBatis.java | 2 +- .../mapper/collections/BaseMapper.java | 25 ++ .../mapper/collections/CollectionMapper.java | 15 +- .../mapper/collections/CrudMapper.java | 17 - .../mapper/collections/InstitutionMapper.java | 15 +- .../mapper/collections/PersonMapper.java | 6 +- .../BaseCollectionEntityResource.java | 379 ++++++++++++++++ .../collections/BaseCrudResource.java | 97 ----- .../collections/CollectionResource.java | 38 +- ... => ExtendedCollectionEntityResource.java} | 160 ++----- .../collections/InstitutionResource.java | 38 +- .../resources/collections/PersonResource.java | 60 ++- .../mapper/collections/CollectionMapper.xml | 105 ++++- .../mapper/collections/InstitutionMapper.xml | 93 +++- .../mapper/collections/PersonMapper.xml | 128 ++++++ .../{CrudTest.java => BaseTest.java} | 119 +++++- .../registry/collections/CollectionIT.java | 120 ++++-- ...java => ExtendedCollectionEntityTest.java} | 76 +--- .../collections/IdentifierResolverIT.java | 8 +- .../registry/collections/InstitutionIT.java | 88 ++-- .../gbif/registry/collections/PersonIT.java | 65 ++- .../mapper/CollectionMapperTest.java | 54 ++- .../mapper/InstitutionMapperTest.java | 36 +- 62 files changed, 5376 insertions(+), 577 deletions(-) create mode 100644 registry-collections-sync/README.md create mode 100644 registry-collections-sync/pom.xml create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/CliSyncApp.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/SyncConfig.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/DiffResult.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/DiffResultExporter.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/DiffResultHandler.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/EntityConverter.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/IndexHerbariorumDiffFinder.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/StaffDiffFinder.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/Utils.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/grscicoll/GrSciCollHttpClient.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/http/BasicAuthInterceptor.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/http/SyncCall.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHEntity.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHHttpClient.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHInstitution.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHMetadata.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHStaff.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/notification/GithubClient.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/notification/Issue.java create mode 100644 registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/notification/IssueFactory.java create mode 100644 registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/SyncConfigTest.java create mode 100644 registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/diff/EntityConverterTest.java create mode 100644 registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/diff/IndexHerbariorumDiffFinderTest.java create mode 100644 registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/diff/StaffDiffFinderTest.java create mode 100644 registry-collections-sync/src/test/resources/sync-config.yaml create mode 100644 registry-liquibase/src/main/resources/liquibase/065-ih-sync.xml create mode 100644 registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseClient.java delete mode 100644 registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseCrudClient.java rename registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/{BaseExtendableCollectionEntityClient.java => BaseExtendedCollectionEntityClient.java} (50%) create mode 100644 registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/BaseMapper.java delete mode 100644 registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/CrudMapper.java create mode 100644 registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/BaseCollectionEntityResource.java delete mode 100644 registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/BaseCrudResource.java rename registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/{BaseExtendableCollectionResource.java => ExtendedCollectionEntityResource.java} (58%) rename registry-ws/src/test/java/org/gbif/registry/collections/{CrudTest.java => BaseTest.java} (50%) rename registry-ws/src/test/java/org/gbif/registry/collections/{BaseCollectionTest.java => ExtendedCollectionEntityTest.java} (75%) diff --git a/pom.xml b/pom.xml index 0eaf0c2312..f10daa5e7b 100644 --- a/pom.xml +++ b/pom.xml @@ -60,6 +60,7 @@ registry-surety registry-ws registry-ws-client + registry-collections-sync + + junit + junit + test + + + + diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/CliSyncApp.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/CliSyncApp.java new file mode 100644 index 0000000000..0457f03433 --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/CliSyncApp.java @@ -0,0 +1,107 @@ +package org.gbif.registry.collections.sync; + +import org.gbif.api.model.collections.Collection; +import org.gbif.api.model.collections.Institution; +import org.gbif.api.model.collections.Person; +import org.gbif.registry.collections.sync.diff.*; +import org.gbif.registry.collections.sync.grscicoll.GrSciCollHttpClient; +import org.gbif.registry.collections.sync.ih.IHHttpClient; +import org.gbif.registry.collections.sync.ih.IHInstitution; + +import java.nio.file.Paths; +import java.util.List; +import java.util.concurrent.CompletableFuture; + +import com.beust.jcommander.JCommander; +import com.beust.jcommander.Parameter; +import lombok.extern.slf4j.Slf4j; + +import static org.gbif.registry.collections.sync.diff.DiffResult.FailedAction; + +@Slf4j +public class CliSyncApp { + + public static void main(String[] args) { + // parse args + CliArgs cliArgs = new CliArgs(); + JCommander.newBuilder().addObject(cliArgs).build().parse(args); + + SyncConfig config = + SyncConfig.fromFileName(cliArgs.confPath) + .orElseThrow(() -> new IllegalArgumentException("No valid config provided")); + + // load the data from the WS + log.info("Loading IH"); + IHHttpClient ihHttpClient = IHHttpClient.create(config.getIhWsUrl()); + CompletableFuture> ihInstitutionsFuture = + CompletableFuture.supplyAsync(ihHttpClient::getInstitutions); + + GrSciCollHttpClient grSciCollHttpClient = GrSciCollHttpClient.create(config); + log.info("Loading Institutions"); + CompletableFuture> institutionsFuture = + CompletableFuture.supplyAsync(grSciCollHttpClient::getInstitutions); + + log.info("Loading Collections"); + CompletableFuture> collectionsFuture = + CompletableFuture.supplyAsync(grSciCollHttpClient::getCollections); + + log.info("Loading Persons"); + CompletableFuture> personsFuture = + CompletableFuture.supplyAsync(grSciCollHttpClient::getPersons); + + CompletableFuture.allOf( + ihInstitutionsFuture, institutionsFuture, collectionsFuture, personsFuture) + .join(); + + List ihInstitutions = ihInstitutionsFuture.join(); + List institutions = institutionsFuture.join(); + List collections = collectionsFuture.join(); + List persons = personsFuture.join(); + + // create an entity converter to use in the diff finder process + EntityConverter entityConverter = + EntityConverter.builder() + .countries(ihHttpClient.getCountries()) + .creationUser(config.getRegistryWsUser()) + .build(); + + // look for differences + log.info("Looking for differences"); + DiffResult diffResult = + IndexHerbariorumDiffFinder.builder() + .ihInstitutions(ihInstitutions) + .ihStaffFetcher(ihHttpClient::getStaffByInstitution) + .institutions(institutions) + .collections(collections) + .persons(persons) + .entityConverter(entityConverter) + .build() + .find(); + + // handle results + List fails = + DiffResultHandler.builder() + .diffResult(diffResult) + .config(config) + .grSciCollHttpClient(grSciCollHttpClient) + .build() + .handle(); + + // add fails to result + log.info("{} operations failed updating the registry", fails.size()); + diffResult.setFailedActions(fails); + + log.info("Diff result: {}", diffResult); + + // save results to a file + if (config.isSaveResultsToFile()) { + DiffResultExporter.exportResultsToFile( + diffResult, Paths.get("ih_sync_result_" + System.currentTimeMillis())); + } + } + + private static class CliArgs { + @Parameter(names = {"--config", "-c"}) + private String confPath; + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/SyncConfig.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/SyncConfig.java new file mode 100644 index 0000000000..4005fb155f --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/SyncConfig.java @@ -0,0 +1,100 @@ +package org.gbif.registry.collections.sync; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Paths; +import java.util.List; +import java.util.Optional; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.ObjectReader; +import com.fasterxml.jackson.dataformat.yaml.YAMLFactory; +import com.google.common.base.Strings; +import lombok.Getter; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; + +@Getter +@Setter +@Slf4j +public class SyncConfig { + + private static final ObjectMapper YAML_MAPPER = new ObjectMapper(new YAMLFactory()); + private static final ObjectReader YAML_READER = YAML_MAPPER.readerFor(SyncConfig.class); + + private String registryWsUrl; + private String registryWsUser; + private String registryWsPassword; + private String ihWsUrl; + private NotificationConfig notification; + private boolean saveResultsToFile; + private boolean dryRun; + private boolean sendNotifications; + + @Getter + @Setter + public static class NotificationConfig { + private String githubWsUrl; + private String githubUser; + private String githubPassword; + private String ihPortalUrl; + private String registryPortalUrl; + private List ghIssuesAssignees; + } + + public static Optional fromFileName(String configFileName) { + if (Strings.isNullOrEmpty(configFileName)) { + log.error("No config file provided"); + return Optional.empty(); + } + + File configFile = Paths.get(configFileName).toFile(); + SyncConfig config; + try { + config = YAML_READER.readValue(configFile); + } catch (IOException e) { + log.error("Couldn't load config from file {}", configFileName, e); + return Optional.empty(); + } + + if (config == null) { + return Optional.empty(); + } + + // do some checks for required fields + if (Strings.isNullOrEmpty(config.getRegistryWsUrl()) + || Strings.isNullOrEmpty(config.getIhWsUrl())) { + throw new IllegalArgumentException("Registry and IH WS URLs are required"); + } + + if (!config.isDryRun() + && (Strings.isNullOrEmpty(config.getRegistryWsUser()) + || Strings.isNullOrEmpty(config.getRegistryWsPassword()))) { + throw new IllegalArgumentException( + "Registry WS credentials are required if we are not doing a dry run"); + } + + if (config.isSendNotifications()) { + if (config.getNotification() == null) { + throw new IllegalArgumentException("Notification config is required"); + } + + if (!config.getNotification().getGithubWsUrl().endsWith("/")) { + throw new IllegalArgumentException("Github API URL must finish with a /."); + } + + if (Strings.isNullOrEmpty(config.getNotification().getGithubUser()) + || Strings.isNullOrEmpty(config.getNotification().getGithubPassword())) { + throw new IllegalArgumentException( + "Github credentials are required if we are not ignoring conflicts."); + } + + if (Strings.isNullOrEmpty(config.getNotification().getRegistryPortalUrl()) + || Strings.isNullOrEmpty(config.getNotification().getIhPortalUrl())) { + throw new IllegalArgumentException("Portal URLs are required"); + } + } + + return Optional.of(config); + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/DiffResult.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/DiffResult.java new file mode 100644 index 0000000000..957a07aff4 --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/DiffResult.java @@ -0,0 +1,119 @@ +package org.gbif.registry.collections.sync.diff; + +import org.gbif.api.model.collections.Collection; +import org.gbif.api.model.collections.CollectionEntity; +import org.gbif.api.model.collections.Institution; +import org.gbif.api.model.collections.Person; +import org.gbif.registry.collections.sync.ih.IHEntity; +import org.gbif.registry.collections.sync.ih.IHInstitution; +import org.gbif.registry.collections.sync.ih.IHStaff; + +import java.util.List; + +import lombok.AllArgsConstructor; +import lombok.Builder; +import lombok.Data; +import lombok.Singular; + +@Data +@Builder +public class DiffResult { + + @Singular(value = "institutionNoChange") + private List institutionsNoChange; + + @Singular(value = "institutionToCreate") + private List institutionsToCreate; + + @Singular(value = "institutionToUpdate") + private List> institutionsToUpdate; + + @Singular(value = "collectionNoChange") + private List collectionsNoChange; + + @Singular(value = "collectionToUpdate") + private List> collectionsToUpdate; + + @Singular(value = "outdatedInstitution") + private List> outdatedIHInstitutions; + + @Singular(value = "conflict") + private List> conflicts; + + @Singular(value = "action") + private List failedActions; + + @Data + @AllArgsConstructor + @Builder + public static class EntityDiffResult { + private T oldEntity; + private T newEntity; + private StaffDiffResult staffDiffResult; + + public boolean isEmpty() { + return oldEntity == null && newEntity == null && staffDiffResult.isEmpty(); + } + } + + @Data + @Builder + public static class StaffDiffResult { + private T entity; + + @Singular(value = "personNoChange") + private List personsNoChange; + + @Singular(value = "personToCreate") + private List personsToCreate; + + @Singular(value = "personToUpdate") + private List personsToUpdate; + + @Singular(value = "personToRemoveFromEntity") + private List personsToRemoveFromEntity; + + @Singular(value = "outdatedStaff") + private List> outdatedStaff; + + @Singular(value = "conflict") + private List> conflicts; + + public boolean isEmpty() { + return personsNoChange.isEmpty() + && personsToCreate.isEmpty() + && personsToUpdate.isEmpty() + && personsToRemoveFromEntity.isEmpty() + && outdatedStaff.isEmpty() + && conflicts.isEmpty(); + } + } + + @Data + @AllArgsConstructor + public static class PersonDiffResult { + private Person oldPerson; + private Person newPerson; + } + + @Data + @AllArgsConstructor + public static class FailedAction { + private Object entity; + private String message; + } + + @Data + @AllArgsConstructor + public static class IHOutdated { + private T ihEntity; + private R grSciCollEntity; + } + + @Data + @AllArgsConstructor + public static class Conflict { + private T ihEntity; + private List grSciCollEntities; + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/DiffResultExporter.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/DiffResultExporter.java new file mode 100644 index 0000000000..db070fdf5a --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/DiffResultExporter.java @@ -0,0 +1,197 @@ +package org.gbif.registry.collections.sync.diff; + +import org.gbif.api.model.collections.Collection; +import org.gbif.api.model.collections.CollectionEntity; +import org.gbif.api.model.collections.Institution; +import org.gbif.registry.collections.sync.diff.DiffResult.EntityDiffResult; + +import java.io.BufferedWriter; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.LocalDateTime; +import java.util.List; + +import lombok.AccessLevel; +import lombok.NoArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import static org.gbif.registry.collections.sync.diff.DiffResult.PersonDiffResult; +import static org.gbif.registry.collections.sync.diff.DiffResult.StaffDiffResult; + +/** Exports a {@link DiffResult} to a file. */ +@Slf4j +@NoArgsConstructor(access = AccessLevel.PRIVATE) +public class DiffResultExporter { + + private static final String SECTION_SEPARATOR = + "##########################################################################"; + private static final String SUBSECTION_SEPARATOR = + "--------------------------------------------------------------------------"; + private static final String LINE_STARTER = ">"; + private static final String SIMPLE_INDENT = "\t"; + private static final String DOUBLE_INDENT = "\t\t"; + + public static void exportResultsToFile(DiffResult diffResult, Path filePath) { + + try (BufferedWriter writer = Files.newBufferedWriter(filePath)) { + + printWithNewLineAfter(writer, "IH Sync " + LocalDateTime.now()); + printWithNewLineAfter(writer, "Summary:"); + printWithNewLineAfter(writer, SUBSECTION_SEPARATOR); + printWithNewLineAfter( + writer, "Institutions No Change: " + diffResult.getInstitutionsNoChange().size()); + printWithNewLineAfter( + writer, "Institutions To Create: " + diffResult.getInstitutionsToCreate().size()); + printWithNewLineAfter( + writer, + "Institutions To Update: " + + diffResult.getInstitutionsToUpdate().size() + + printOnlyStaffUpdates(diffResult.getInstitutionsToUpdate())); + printWithNewLineAfter( + writer, "Collections No Change: " + diffResult.getCollectionsNoChange().size()); + printWithNewLineAfter( + writer, + "Collections To Update: " + + diffResult.getCollectionsToUpdate().size() + + printOnlyStaffUpdates(diffResult.getCollectionsToUpdate())); + printWithNewLineAfter( + writer, "Outdated institutions: " + diffResult.getOutdatedIHInstitutions().size()); + printWithNewLineAfter(writer, "General Conflicts: " + diffResult.getConflicts().size()); + printWithNewLineAfter( + writer, + "Failed Actions (updates or conflict notifications that failed): " + + diffResult.getFailedActions().size()); + writer.newLine(); + writer.newLine(); + + // Institutions + printSection(writer, "Institutions No Change", diffResult.getInstitutionsNoChange()); + printSection(writer, "Institutions to Create", diffResult.getInstitutionsToCreate()); + printSectionTitle( + writer, + "Institutions to Update: " + + diffResult.getInstitutionsToUpdate().size() + + printOnlyStaffUpdates(diffResult.getInstitutionsToUpdate())); + for (EntityDiffResult diff : diffResult.getInstitutionsToUpdate()) { + writer.write(LINE_STARTER); + printWithNewLineAfter(writer, "UPDATE DIFF:"); + printWithNewLineAfter(writer, SIMPLE_INDENT + "OLD: " + diff.getOldEntity()); + printWithNewLineAfter(writer, SIMPLE_INDENT + "NEW: " + diff.getNewEntity()); + printStaffDiffResult(writer, diff.getStaffDiffResult()); + } + + // Collections + printSection(writer, "Collections No Change", diffResult.getCollectionsNoChange()); + printSectionTitle( + writer, + "Collections to Update: " + + diffResult.getCollectionsToUpdate().size() + + printOnlyStaffUpdates(diffResult.getInstitutionsToUpdate())); + for (EntityDiffResult diff : diffResult.getCollectionsToUpdate()) { + writer.write(LINE_STARTER); + printWithNewLineAfter(writer, "UPDATE DIFF:"); + printWithNewLineAfter(writer, SIMPLE_INDENT + "OLD: " + diff.getOldEntity()); + printWithNewLineAfter(writer, SIMPLE_INDENT + "NEW: " + diff.getNewEntity()); + printStaffDiffResult(writer, diff.getStaffDiffResult()); + } + + // Outdated + printSection(writer, "Outdated Institutions", diffResult.getOutdatedIHInstitutions()); + + // Conflicts + printSection(writer, "General Conflicts", diffResult.getConflicts()); + + // fails + printSection(writer, "Failed Actions", diffResult.getFailedActions()); + + } catch (Exception e) { + log.warn("Couldn't save diff results", e); + } + } + + private static String printOnlyStaffUpdates( + List> entityDiffResult) { + long onlyStaffUpdate = + entityDiffResult.stream() + .filter(d -> d.getNewEntity() == null && d.getOldEntity() == null) + .count(); + + return " ( " + onlyStaffUpdate + " only Staff update)"; + } + + private static void printSectionTitle(BufferedWriter writer, String title) throws IOException { + writer.write(title); + writer.newLine(); + writer.write(SECTION_SEPARATOR); + writer.newLine(); + writer.write(LINE_STARTER); + } + + private static void printSubsectionTitle(BufferedWriter writer, String title) throws IOException { + writer.write(DOUBLE_INDENT + title); + writer.newLine(); + writer.write(DOUBLE_INDENT + SUBSECTION_SEPARATOR); + writer.newLine(); + writer.write(DOUBLE_INDENT + LINE_STARTER); + } + + private static void printSection(BufferedWriter writer, String title, List collection) + throws IOException { + writer.newLine(); + printSectionTitle(writer, title + ": " + collection.size()); + printCollection(writer, collection); + writer.newLine(); + } + + private static void printSubsection(BufferedWriter writer, String title, List collection) + throws IOException { + writer.newLine(); + printSubsectionTitle(writer, title + ": " + collection.size()); + printCollectionSubsection(writer, collection); + writer.newLine(); + } + + private static void printStaffDiffResult( + BufferedWriter writer, StaffDiffResult staffDiffResult) throws IOException { + printWithNewLineAfter(writer, ">>> Differences in Associated Staff"); + + printSubsection(writer, "Staff No Change", staffDiffResult.getPersonsNoChange()); + printSubsection( + writer, "Staff to Create and add to the entity", staffDiffResult.getPersonsToCreate()); + printSubsection( + writer, "Staff to Remove from entity", staffDiffResult.getPersonsToRemoveFromEntity()); + + printSubsectionTitle(writer, "Staff to Update: " + staffDiffResult.getPersonsToUpdate().size()); + for (PersonDiffResult staffUpdate : staffDiffResult.getPersonsToUpdate()) { + writer.write(LINE_STARTER); + printWithNewLineAfter(writer, DOUBLE_INDENT + "STAFF DIFF:"); + printWithNewLineAfter(writer, DOUBLE_INDENT + "OLD: " + staffUpdate.getOldPerson()); + printWithNewLineAfter(writer, DOUBLE_INDENT + "NEW: " + staffUpdate.getNewPerson()); + } + + printSubsection(writer, "Outdated Staff", staffDiffResult.getOutdatedStaff()); + printSubsection(writer, "Staff Conflicts", staffDiffResult.getConflicts()); + } + + private static void printCollection(BufferedWriter writer, List collection) + throws IOException { + for (T e : collection) { + writer.write(LINE_STARTER); + printWithNewLineAfter(writer, e.toString()); + } + } + + private static void printCollectionSubsection(BufferedWriter writer, List collection) + throws IOException { + for (T e : collection) { + writer.write(DOUBLE_INDENT + LINE_STARTER); + printWithNewLineAfter(writer, DOUBLE_INDENT + e.toString()); + } + } + + private static void printWithNewLineAfter(BufferedWriter writer, String text) throws IOException { + writer.write(text); + writer.newLine(); + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/DiffResultHandler.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/DiffResultHandler.java new file mode 100644 index 0000000000..cfa26a78d0 --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/DiffResultHandler.java @@ -0,0 +1,244 @@ +package org.gbif.registry.collections.sync.diff; + +import org.gbif.api.model.collections.Collection; +import org.gbif.api.model.collections.CollectionEntity; +import org.gbif.api.model.collections.Institution; +import org.gbif.api.model.collections.Person; +import org.gbif.registry.collections.sync.SyncConfig; +import org.gbif.registry.collections.sync.grscicoll.GrSciCollHttpClient; +import org.gbif.registry.collections.sync.notification.GithubClient; +import org.gbif.registry.collections.sync.notification.Issue; +import org.gbif.registry.collections.sync.notification.IssueFactory; + +import java.util.*; +import java.util.function.BiConsumer; +import java.util.function.Function; +import java.util.stream.Collectors; +import javax.annotation.Nullable; + +import lombok.Builder; +import lombok.extern.slf4j.Slf4j; + +import static org.gbif.registry.collections.sync.diff.DiffResult.EntityDiffResult; +import static org.gbif.registry.collections.sync.diff.DiffResult.FailedAction; +import static org.gbif.registry.collections.sync.diff.DiffResult.PersonDiffResult; +import static org.gbif.registry.collections.sync.diff.DiffResult.StaffDiffResult; + +/** + * Handles the results stored in a {@link DiffResult}. This class is responsible to make the + * necessary updates in GrSciColl and notify the existing conflicts. + */ +@Slf4j +public class DiffResultHandler { + private final DiffResult diffResult; + private final SyncConfig config; + private final GrSciCollHttpClient grSciCollHttpClient; + private final IssueFactory issueFactory; + @Nullable private GithubClient githubClient; + + @Builder + private DiffResultHandler( + DiffResult diffResult, SyncConfig config, GrSciCollHttpClient grSciCollHttpClient) { + this.diffResult = Objects.requireNonNull(diffResult); + this.config = Objects.requireNonNull(config); + this.grSciCollHttpClient = Objects.requireNonNull(grSciCollHttpClient); + + if (config.isSendNotifications()) { + this.githubClient = GithubClient.create(config); + } + + this.issueFactory = IssueFactory.fromConfig(config.getNotification()); + } + + public List handle() { + if (config.isDryRun() && !config.isSendNotifications()) { + log.info("Skipping results handler. Dry run and ignore conflicts are both set to true."); + return Collections.emptyList(); + } + + List fails = new ArrayList<>(); + + // Institutions to create + for (Institution institutionToCreate : diffResult.getInstitutionsToCreate()) { + executeOrCreateFail( + () -> grSciCollHttpClient.createInstitution(institutionToCreate), + e -> + new FailedAction( + institutionToCreate, "Failed to create institution: " + e.getMessage())) + .ifPresent(fails::add); + } + + // Institutions to update + for (EntityDiffResult diff : diffResult.getInstitutionsToUpdate()) { + executeOrCreateFail( + () -> grSciCollHttpClient.updateInstitution(diff.getNewEntity()), + e -> + new FailedAction( + diff.getNewEntity(), "Failed to update institution: " + e.getMessage())) + .ifPresent(fails::add); + + // staff + fails.addAll( + handleStaffDiff( + diff.getStaffDiffResult(), + grSciCollHttpClient::addPersonToInstitution, + grSciCollHttpClient::removePersonFromInstitution)); + } + + // collections to update + for (EntityDiffResult diff : diffResult.getCollectionsToUpdate()) { + executeOrCreateFail( + () -> grSciCollHttpClient.updateCollection(diff.getNewEntity()), + e -> + new FailedAction( + diff.getNewEntity(), "Failed to update collection: " + e.getMessage())) + .ifPresent(fails::add); + + // staff + fails.addAll( + handleStaffDiff( + diff.getStaffDiffResult(), + grSciCollHttpClient::addPersonToCollection, + grSciCollHttpClient::removePersonFromCollection)); + } + + // issues and conflicts + fails.addAll(createInstitutionIssues()); + + // fails + if (!fails.isEmpty()) { + // create issue + createIssue(issueFactory.createFailsNotification(fails)).ifPresent(fails::add); + } + + return fails; + } + + private List handleStaffDiff( + StaffDiffResult staffDiffResult, + BiConsumer addPersonAction, + BiConsumer removePersonAction) { + List fails = new ArrayList<>(); + + for (Person personToCreate : staffDiffResult.getPersonsToCreate()) { + executeOrCreateFail( + () -> { + UUID createdKey = grSciCollHttpClient.createPerson(personToCreate); + addPersonAction.accept(createdKey, staffDiffResult.getEntity().getKey()); + }, + e -> new FailedAction(personToCreate, "Failed to add person: " + e.getMessage())) + .ifPresent(fails::add); + } + + for (PersonDiffResult personDiff : staffDiffResult.getPersonsToUpdate()) { + executeOrCreateFail( + () -> { + grSciCollHttpClient.updatePerson(personDiff.getNewPerson()); + // add identifiers if needed + personDiff.getNewPerson().getIdentifiers().stream() + .filter(i -> i.getKey() == null) + .forEach( + i -> + grSciCollHttpClient.addIdentifierToPerson( + personDiff.getNewPerson().getKey(), i)); + }, + e -> + new FailedAction( + personDiff.getNewPerson(), "Failed to update person: " + e.getMessage())) + .ifPresent(fails::add); + } + + for (Person personToRemove : staffDiffResult.getPersonsToRemoveFromEntity()) { + executeOrCreateFail( + () -> + removePersonAction.accept( + personToRemove.getKey(), staffDiffResult.getEntity().getKey()), + e -> new FailedAction(personToRemove, "Failed to remove person: " + e.getMessage())) + .ifPresent(fails::add); + } + + // staff issues + fails.addAll(createStaffIssues(staffDiffResult)); + + return fails; + } + + private Optional executeOrCreateFail( + Runnable runnable, Function failCreator) { + if (config.isDryRun()) { + return Optional.empty(); + } + + try { + runnable.run(); + } catch (Exception e) { + return Optional.of(failCreator.apply(e)); + } + + return Optional.empty(); + } + + private List createInstitutionIssues() { + List fails = new ArrayList<>(); + if (!config.isSendNotifications()) { + log.debug("Ignore conflicts flag enabled. Ignoring conflict."); + return fails; + } + + List issues = + diffResult.getOutdatedIHInstitutions().stream() + .map( + o -> + issueFactory.createOutdatedIHInstitutionIssue( + o.getGrSciCollEntity(), o.getIhEntity())) + .collect(Collectors.toList()); + + issues.addAll( + diffResult.getConflicts().stream() + .map(c -> issueFactory.createConflict(c.getGrSciCollEntities(), c.getIhEntity())) + .collect(Collectors.toList())); + + issues.forEach(i -> createIssue(i).ifPresent(fails::add)); + + return fails; + } + + private List createStaffIssues( + StaffDiffResult staffDiffResult) { + List fails = new ArrayList<>(); + if (!config.isSendNotifications()) { + log.debug("Ignore conflicts flag enabled. Ignoring conflict."); + return fails; + } + + List issues = + staffDiffResult.getOutdatedStaff().stream() + .map( + o -> + issueFactory.createOutdatedIHStaffIssue( + o.getGrSciCollEntity(), o.getIhEntity())) + .collect(Collectors.toList()); + + issues.addAll( + staffDiffResult.getConflicts().stream() + .map(c -> issueFactory.createStaffConflict(c.getGrSciCollEntities(), c.getIhEntity())) + .collect(Collectors.toList())); + + issues.forEach(i -> createIssue(i).ifPresent(fails::add)); + return fails; + } + + private Optional createIssue(Issue issue) { + if (!config.isSendNotifications()) { + return Optional.empty(); + } + + try { + githubClient.createIssue(issue); + } catch (Exception e) { + return Optional.of(new FailedAction(issue, "Failed to create isssue: " + e.getMessage())); + } + + return Optional.empty(); + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/EntityConverter.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/EntityConverter.java new file mode 100644 index 0000000000..b905dc5783 --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/EntityConverter.java @@ -0,0 +1,404 @@ +package org.gbif.registry.collections.sync.diff; + +import org.gbif.api.model.collections.Collection; +import org.gbif.api.model.collections.*; +import org.gbif.api.model.registry.Identifiable; +import org.gbif.api.model.registry.Identifier; +import org.gbif.api.vocabulary.Country; +import org.gbif.api.vocabulary.IdentifierType; +import org.gbif.registry.collections.sync.ih.IHInstitution; +import org.gbif.registry.collections.sync.ih.IHStaff; + +import java.lang.reflect.InvocationTargetException; +import java.math.BigDecimal; +import java.math.RoundingMode; +import java.net.URI; +import java.util.*; +import java.util.regex.Pattern; + +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Strings; +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; +import lombok.Builder; +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.beanutils.BeanUtils; + +import static org.gbif.registry.collections.sync.diff.Utils.encodeIRN; + +/** Converts IH insitutions to the GrSciColl entities {@link Institution} and {@link Collection}. */ +@Slf4j +public class EntityConverter { + + private static final Pattern WHITESPACE = Pattern.compile("[\\s+]"); + private static final Map COUNTRY_MANUAL_MAPPINGS = new HashMap<>(); + private final Map countryLookup; + private String creationUser; + + static { + COUNTRY_MANUAL_MAPPINGS.put("U.K.", Country.UNITED_KINGDOM); + COUNTRY_MANUAL_MAPPINGS.put("UK", Country.UNITED_KINGDOM); + COUNTRY_MANUAL_MAPPINGS.put("Scotland", Country.UNITED_KINGDOM); + COUNTRY_MANUAL_MAPPINGS.put("Alderney", Country.UNITED_KINGDOM); + COUNTRY_MANUAL_MAPPINGS.put("Congo Republic (Congo-Brazzaville)", Country.CONGO); + COUNTRY_MANUAL_MAPPINGS.put("Republic of Congo-Brazzaville", Country.CONGO); + COUNTRY_MANUAL_MAPPINGS.put( + "Democratic Republic of the Congo", Country.CONGO_DEMOCRATIC_REPUBLIC); + COUNTRY_MANUAL_MAPPINGS.put("Italia", Country.ITALY); + COUNTRY_MANUAL_MAPPINGS.put("Ivory Coast", Country.CÔTE_DIVOIRE); + COUNTRY_MANUAL_MAPPINGS.put("Laos", Country.LAO); + COUNTRY_MANUAL_MAPPINGS.put("Republic of Korea", Country.KOREA_SOUTH); + COUNTRY_MANUAL_MAPPINGS.put("Republic of South Korea", Country.KOREA_SOUTH); + COUNTRY_MANUAL_MAPPINGS.put("São Tomé e Príncipe", Country.SAO_TOME_PRINCIPE); + COUNTRY_MANUAL_MAPPINGS.put("Slovak Republic", Country.SLOVAKIA); + } + + @Builder + private EntityConverter(List countries, String creationUser) { + this.creationUser = creationUser; + countryLookup = mapCountries(countries); + + if (countryLookup.size() != countries.size()) { + log.warn("We couldn't match all the countries to our enum"); + } + } + + @VisibleForTesting + static Map mapCountries(List countries) { + // build map with the titles of the Country enum + Map titleLookup = + Maps.uniqueIndex(Lists.newArrayList(Country.values()), Country::getTitle); + + Map mappings = new HashMap<>(); + + countries.forEach( + c -> { + Country country = titleLookup.get(c); + + // we first try manual mappings + country = COUNTRY_MANUAL_MAPPINGS.get(c); + + if (country == null) { + country = Country.fromIsoCode(c); + } + if (country == null) { + country = Country.fromIsoCode(c.replaceAll("\\.", "")); + } + if (country == null && c.contains(",")) { + country = titleLookup.get(c.split(",")[0]); + } + if (country == null) { + country = + Arrays.stream(Country.values()) + .filter(v -> c.contains(v.getTitle())) + .findFirst() + .orElse(null); + } + if (country == null) { + country = + Arrays.stream(Country.values()) + .filter(v -> v.getTitle().contains(c)) + .findFirst() + .orElse(null); + } + if (country == null) { + country = + Arrays.stream(Country.values()) + .filter(v -> v.name().replaceAll("_", " ").equalsIgnoreCase(c)) + .findFirst() + .orElse(null); + } + + if (country != null) { + mappings.put(c.toLowerCase(), country); + } + }); + + return mappings; + } + + public Country matchCountry(String country) { + return countryLookup.get(country.toLowerCase()); + } + + public Institution convertToInstitution(IHInstitution ihInstitution) { + return convertToInstitution(ihInstitution, null); + } + + public Institution convertToInstitution(IHInstitution ihInstitution, Institution existing) { + Institution institution = new Institution(); + + if (existing != null) { + try { + BeanUtils.copyProperties(institution, existing); + } catch (IllegalAccessException | InvocationTargetException e) { + log.warn("Couldn't copy institution properties from bean: {}", existing); + } + } + + getStringValue(ihInstitution.getOrganization()).ifPresent(institution::setName); + institution.setCode(ihInstitution.getCode()); + institution.setIndexHerbariorumRecord(true); + institution.setNumberSpecimens(Math.toIntExact(ihInstitution.getSpecimenTotal())); + setLocation(ihInstitution, institution); + + setAddress(institution, ihInstitution); + institution.setEmail(getIhEmails(ihInstitution)); + institution.setPhone(getIhPhones(ihInstitution)); + getIhHomepage(ihInstitution).ifPresent(institution::setHomepage); + + addIdentifierIfNotExists(institution, encodeIRN(ihInstitution.getIrn()), creationUser); + + return institution; + } + + private static void setLocation(IHInstitution ihInstitution, Institution institution) { + if (ihInstitution.getLocation() != null) { + IHInstitution.Location location = ihInstitution.getLocation(); + if (location.getLat() != null) { + BigDecimal lat = BigDecimal.valueOf(location.getLat()).setScale(6, RoundingMode.HALF_DOWN); + if (lat.compareTo(BigDecimal.valueOf(-90)) >= 0 + && lat.compareTo(BigDecimal.valueOf(90)) <= 0) { + institution.setLatitude(lat); + } else { + log.info( + "Invalid lat coordinate {} for instittuion with IRN {}", + location.getLat(), + ihInstitution.getIrn()); + } + } + + if (location.getLon() != null) { + BigDecimal lon = BigDecimal.valueOf(location.getLon()).setScale(6, RoundingMode.HALF_DOWN); + if (lon.compareTo(BigDecimal.valueOf(-180)) >= 0 + && lon.compareTo(BigDecimal.valueOf(180)) <= 0) { + institution.setLongitude(lon); + } else { + log.info( + "Invalid lon coordinate {} for instittuion with IRN {}", + location.getLon(), + ihInstitution.getIrn()); + } + } + } + } + + public Collection convertToCollection(IHInstitution ihInstitution, Collection existing) { + Collection collection = new Collection(); + + if (existing != null) { + try { + BeanUtils.copyProperties(collection, existing); + } catch (IllegalAccessException | InvocationTargetException e) { + log.warn("Couldn't copy collection properties from bean: {}", existing); + } + } + + getStringValue(ihInstitution.getOrganization()).ifPresent(collection::setName); + collection.setCode(ihInstitution.getCode()); + collection.setIndexHerbariorumRecord(true); + + setAddress(collection, ihInstitution); + + collection.setEmail(getIhEmails(ihInstitution)); + collection.setPhone(getIhPhones(ihInstitution)); + getIhHomepage(ihInstitution).ifPresent(collection::setHomepage); + + addIdentifierIfNotExists(collection, encodeIRN(ihInstitution.getIrn()), creationUser); + + return collection; + } + + public Person convertToPerson(IHStaff ihStaff) { + return convertToPerson(ihStaff, null); + } + + public Person convertToPerson(IHStaff ihStaff, Person existing) { + Person person = new Person(); + + if (existing != null) { + try { + BeanUtils.copyProperties(person, existing); + } catch (IllegalAccessException | InvocationTargetException e) { + log.warn("Couldn't copy person properties from bean: {}", existing); + } + } + + buildFirstName(ihStaff).ifPresent(person::setFirstName); + getStringValue(ihStaff.getLastName()).ifPresent(person::setLastName); + getStringValue(ihStaff.getPosition()).ifPresent(person::setPosition); + + if (ihStaff.getContact() != null) { + getFirstString(ihStaff.getContact().getEmail()).ifPresent(person::setEmail); + getFirstString(ihStaff.getContact().getPhone()).ifPresent(person::setPhone); + getFirstString(ihStaff.getContact().getFax()).ifPresent(person::setFax); + } + + if (ihStaff.getAddress() != null) { + Address mailingAddress = new Address(); + getStringValue(ihStaff.getAddress().getStreet()).ifPresent(mailingAddress::setAddress); + getStringValue(ihStaff.getAddress().getCity()).ifPresent(mailingAddress::setCity); + getStringValue(ihStaff.getAddress().getState()).ifPresent(mailingAddress::setProvince); + getStringValue(ihStaff.getAddress().getZipCode()).ifPresent(mailingAddress::setPostalCode); + + if (!Strings.isNullOrEmpty(ihStaff.getAddress().getCountry())) { + Country mailingAddressCountry = matchCountry(ihStaff.getAddress().getCountry()); + mailingAddress.setCountry(mailingAddressCountry); + if (mailingAddressCountry == null) { + log.warn( + "Country not found for {} and IH staff {}", + ihStaff.getAddress().getCountry(), + ihStaff.getIrn()); + } + } + + person.setMailingAddress(mailingAddress); + } + + addIdentifierIfNotExists(person, encodeIRN(ihStaff.getIrn()), creationUser); + + return person; + } + + private Optional buildFirstName(IHStaff ihStaff) { + StringBuilder firstNameBuilder = new StringBuilder(); + if (!Strings.isNullOrEmpty(ihStaff.getFirstName())) { + firstNameBuilder.append(ihStaff.getFirstName()).append(" "); + } + if (!Strings.isNullOrEmpty(ihStaff.getMiddleName())) { + firstNameBuilder.append(ihStaff.getMiddleName()); + } + + String firstName = firstNameBuilder.toString(); + if (Strings.isNullOrEmpty(firstName)) { + return Optional.empty(); + } + + return Optional.of(firstName.trim()); + } + + private void setAddress(Contactable contactable, IHInstitution ih) { + Address physicalAddress = null; + Address mailingAddress = null; + if (ih.getAddress() != null) { + physicalAddress = new Address(); + getStringValue(ih.getAddress().getPhysicalStreet()).ifPresent(physicalAddress::setAddress); + getStringValue(ih.getAddress().getPhysicalCity()).ifPresent(physicalAddress::setCity); + getStringValue(ih.getAddress().getPhysicalState()).ifPresent(physicalAddress::setProvince); + getStringValue(ih.getAddress().getPhysicalZipCode()) + .ifPresent(physicalAddress::setPostalCode); + + if (!Strings.isNullOrEmpty(ih.getAddress().getPhysicalCountry())) { + Country physicalAddressCountry = matchCountry(ih.getAddress().getPhysicalCountry()); + physicalAddress.setCountry(physicalAddressCountry); + if (physicalAddressCountry == null) { + log.warn( + "Country not found for {} and IH institution {}", + ih.getAddress().getPhysicalCountry(), + ih.getIrn()); + } + } + + mailingAddress = new Address(); + getStringValue(ih.getAddress().getPostalStreet()).ifPresent(mailingAddress::setAddress); + getStringValue(ih.getAddress().getPostalCity()).ifPresent(mailingAddress::setCity); + getStringValue(ih.getAddress().getPostalState()).ifPresent(mailingAddress::setProvince); + getStringValue(ih.getAddress().getPostalZipCode()).ifPresent(mailingAddress::setPostalCode); + + if (!Strings.isNullOrEmpty(ih.getAddress().getPostalCountry())) { + Country mailingAddressCountry = matchCountry(ih.getAddress().getPostalCountry()); + mailingAddress.setCountry(mailingAddressCountry); + if (mailingAddressCountry == null) { + log.warn( + "Country not found for {} and IH institution {}", + ih.getAddress().getPhysicalCountry(), + ih.getIrn()); + } + } + } + contactable.setAddress(physicalAddress); + contactable.setMailingAddress(mailingAddress); + } + + private static List getIhEmails(IHInstitution ih) { + if (ih.getContact() != null && ih.getContact().getEmail() != null) { + return parseStringList(ih.getContact().getEmail()); + } + return Collections.emptyList(); + } + + private static List getIhPhones(IHInstitution ih) { + if (ih.getContact() != null && ih.getContact().getPhone() != null) { + return parseStringList(ih.getContact().getPhone()); + } + return Collections.emptyList(); + } + + private static List parseStringList(String stringList) { + String listNormalized = stringList.replaceAll("\n", ","); + return Arrays.asList(listNormalized.split(",")); + } + + private static Optional getIhHomepage(IHInstitution ih) { + if (ih.getContact() == null || ih.getContact().getWebUrl() == null) { + return Optional.empty(); + } + // when there are multiple URLs we try to get the first one + Optional webUrlOpt = getFirstString(ih.getContact().getWebUrl()); + + if (!webUrlOpt.isPresent()) { + return Optional.empty(); + } + + // we try to clean the URL... + String webUrl = WHITESPACE.matcher(webUrlOpt.get()).replaceAll(""); + + try { + return Optional.of(URI.create(webUrl)); + } catch (Exception ex) { + log.warn("Couldn't parse the contact webUrl {} for IH institution {}", webUrl, ih.getCode()); + return Optional.empty(); + } + } + + private static Optional getFirstString(String stringList) { + if (Strings.isNullOrEmpty(stringList)) { + return Optional.empty(); + } + + String firstValue = null; + if (stringList.contains(",")) { + firstValue = stringList.split(",")[0]; + } else if (stringList.contains(";")) { + firstValue = stringList.split(";")[0]; + } else if (stringList.contains("\n")) { + firstValue = stringList.split("\n")[0]; + } + + if (Strings.isNullOrEmpty(firstValue)) { + return Optional.empty(); + } + + return Optional.of(firstValue.trim()); + } + + private static void addIdentifierIfNotExists(Identifiable entity, String irn, String user) { + if (!containsIrnAsIdentifier(entity, irn)) { + // add identifier + Identifier ihIdentifier = new Identifier(IdentifierType.IH_IRN, irn); + ihIdentifier.setCreatedBy(user); + entity.getIdentifiers().add(ihIdentifier); + } + } + + private static boolean containsIrnAsIdentifier(Identifiable entity, String irn) { + return entity.getIdentifiers().stream().anyMatch(i -> Objects.equals(irn, i.getIdentifier())); + } + + private static Optional getStringValue(String value) { + if (Strings.isNullOrEmpty(value)) { + return Optional.empty(); + } + return Optional.of(value); + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/IndexHerbariorumDiffFinder.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/IndexHerbariorumDiffFinder.java new file mode 100644 index 0000000000..ed6e15fb15 --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/IndexHerbariorumDiffFinder.java @@ -0,0 +1,193 @@ +package org.gbif.registry.collections.sync.diff; + +import org.gbif.api.model.collections.Collection; +import org.gbif.api.model.collections.*; +import org.gbif.api.model.registry.LenientEquals; +import org.gbif.registry.collections.sync.ih.IHInstitution; +import org.gbif.registry.collections.sync.ih.IHStaff; + +import java.util.*; +import java.util.function.Function; +import java.util.stream.Collectors; + +import lombok.Builder; +import lombok.extern.slf4j.Slf4j; + +import static org.gbif.registry.collections.sync.diff.DiffResult.EntityDiffResult; +import static org.gbif.registry.collections.sync.diff.Utils.encodeIRN; +import static org.gbif.registry.collections.sync.diff.Utils.isIHOutdated; +import static org.gbif.registry.collections.sync.diff.Utils.mapByIrn; + +/** + * A synchronization utility that will ensure GRSciColl is up to date with IndexHerbariorum. This + * operates as follows: + * + *
    + *
  • Retrieve all Herbaria from IndexHerbariorum + *
  • For each entity locate the equivalent Institution or Collection in GRSciColl using the IH + * IRN + *
  • If the entity exists and they differ, update GrSciColl + *
  • If the entity does not exist, insert it as an institution and with an identifier holding + * the IH IRN + *
+ * + *

A future version of this may allow editing of IH entities in GRSciColl. Under that scenario + * when entities differ more complex logic is required, likely requiring notification to GRSciColl + * and IH staff to resolve the differences. + */ +@Slf4j +public class IndexHerbariorumDiffFinder { + + private final List ihInstitutions; + private final Function> ihStaffFetcher; + private final Map> institutionsByIrn; + private final Map> collectionsByIrn; + private final EntityConverter entityConverter; + private final StaffDiffFinder staffDiffFinder; + + @Builder + private IndexHerbariorumDiffFinder( + List ihInstitutions, + Function> ihStaffFetcher, + List institutions, + List collections, + List persons, + EntityConverter entityConverter) { + this.ihInstitutions = ihInstitutions; + this.ihStaffFetcher = ihStaffFetcher; + this.entityConverter = entityConverter; + this.institutionsByIrn = mapByIrn(institutions); + this.collectionsByIrn = mapByIrn(collections); + this.staffDiffFinder = + StaffDiffFinder.builder() + .allGrSciCollPersons(persons) + .entityConverter(entityConverter) + .build(); + } + + public DiffResult find() { + DiffResult.DiffResultBuilder diffResult = DiffResult.builder(); + + for (IHInstitution ihInstitution : ihInstitutions) { + + // locate potential matches in GrSciColl + Match match = + findMatches(institutionsByIrn, collectionsByIrn, encodeIRN(ihInstitution.getIrn())); + + if (match.onlyOneInstitutionMatch()) { + Institution existing = match.institutions.iterator().next(); + log.info("Institution {} matched with IH {}", existing.getKey(), ihInstitution.getCode()); + + if (isIHOutdated(ihInstitution, existing)) { + diffResult.outdatedInstitution(new DiffResult.IHOutdated<>(ihInstitution, existing)); + continue; + } + + // we look for differences between entities + Institution institution = entityConverter.convertToInstitution(ihInstitution, existing); + EntityDiffResult entityDiff = + checkEntityDiff(ihInstitution, institution, existing); + + if (entityDiff.isEmpty()) { + diffResult.institutionNoChange(existing); + } else { + diffResult.institutionToUpdate(entityDiff); + } + } else if (match.onlyOneCollectionMatch()) { + Collection existing = match.collections.iterator().next(); + log.info("Collection {} matched with IH {}", existing.getKey(), ihInstitution.getCode()); + + if (isIHOutdated(ihInstitution, existing)) { + diffResult.outdatedInstitution(new DiffResult.IHOutdated<>(ihInstitution, existing)); + continue; + } + + // we look for differences between entities + Collection collection = entityConverter.convertToCollection(ihInstitution, existing); + EntityDiffResult entityDiff = + checkEntityDiff(ihInstitution, collection, existing); + + if (entityDiff.isEmpty()) { + diffResult.collectionNoChange(existing); + } else { + diffResult.collectionToUpdate(entityDiff); + } + } else if (match.noMatches()) { + log.info("New institution to create for IH: {}", ihInstitution.getCode()); + // create institution + Institution institution = entityConverter.convertToInstitution(ihInstitution); + + institution.setContacts( + ihStaffFetcher.apply(ihInstitution.getCode()).stream() + .map(entityConverter::convertToPerson) + .collect(Collectors.toList())); + + diffResult.institutionToCreate(institution); + + } else { + // Conflict that needs resolved manually + log.info( + "Conflict. {} institutions and {} collections are candidate matches in registry for {}: ", + match.institutions, + match.collections, + ihInstitution.getOrganization()); + + diffResult.conflict(new DiffResult.Conflict<>(ihInstitution, match.getAllMatches())); + } + } + + return diffResult.build(); + } + + private & Contactable> + EntityDiffResult checkEntityDiff(IHInstitution ihInstitution, T newEntity, T existing) { + + EntityDiffResult.EntityDiffResultBuilder updateDiffBuilder = EntityDiffResult.builder(); + if (!newEntity.lenientEquals(existing)) { + updateDiffBuilder.newEntity(newEntity).oldEntity(existing); + } + + // look for differences in staff + log.info("Syncing staff for IH institution {}", ihInstitution.getCode()); + DiffResult.StaffDiffResult staffDiffResult = + staffDiffFinder.syncStaff( + newEntity, ihStaffFetcher.apply(ihInstitution.getCode()), existing.getContacts()); + updateDiffBuilder.staffDiffResult(staffDiffResult); + + return updateDiffBuilder.build(); + } + + private Match findMatches( + Map> institutions, + Map> collections, + String irn) { + Match match = new Match(); + match.institutions = institutions.getOrDefault(irn, Collections.emptySet()); + match.collections = collections.getOrDefault(irn, Collections.emptySet()); + + return match; + } + + private static class Match { + Set institutions; + Set collections; + + boolean onlyOneInstitutionMatch() { + return institutions.size() == 1 && collections.isEmpty(); + } + + boolean onlyOneCollectionMatch() { + return collections.size() == 1 && institutions.isEmpty(); + } + + boolean noMatches() { + return institutions.isEmpty() && collections.isEmpty(); + } + + List getAllMatches() { + List all = new ArrayList<>(institutions); + all.addAll(collections); + return all; + } + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/StaffDiffFinder.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/StaffDiffFinder.java new file mode 100644 index 0000000000..a719f239ce --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/StaffDiffFinder.java @@ -0,0 +1,362 @@ +package org.gbif.registry.collections.sync.diff; + +import org.gbif.api.model.collections.CollectionEntity; +import org.gbif.api.model.collections.Person; +import org.gbif.api.vocabulary.Country; +import org.gbif.registry.collections.sync.ih.IHStaff; + +import java.util.*; +import java.util.function.BiPredicate; +import java.util.function.Function; +import java.util.regex.Pattern; +import java.util.stream.Collectors; + +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Strings; +import lombok.Builder; +import lombok.Data; + +import static org.gbif.registry.collections.sync.diff.DiffResult.StaffDiffResult; +import static org.gbif.registry.collections.sync.diff.Utils.encodeIRN; +import static org.gbif.registry.collections.sync.diff.Utils.isIHOutdated; +import static org.gbif.registry.collections.sync.diff.Utils.mapByIrn; + +class StaffDiffFinder { + + private static final String EMPTY = ""; + private static final Pattern WHITESPACE = Pattern.compile("[\\s]"); + + private final EntityConverter entityConverter; + private final List allGrSciCollPersons; + private final Map> grSciCollPersonsByIrn; + + @Builder + private StaffDiffFinder(EntityConverter entityConverter, List allGrSciCollPersons) { + this.entityConverter = entityConverter; + this.allGrSciCollPersons = allGrSciCollPersons; + this.grSciCollPersonsByIrn = mapByIrn(allGrSciCollPersons); + } + + private static final Function CONCAT_IH_NAME = + s -> { + StringBuilder fullNameBuilder = new StringBuilder(); + if (!Strings.isNullOrEmpty(s.getFirstName())) { + fullNameBuilder.append(s.getFirstName()); + } + if (!Strings.isNullOrEmpty(s.getMiddleName())) { + fullNameBuilder.append(s.getMiddleName()); + } + if (!Strings.isNullOrEmpty(s.getLastName())) { + fullNameBuilder.append(s.getLastName()); + } + + String fullName = fullNameBuilder.toString(); + + if (Strings.isNullOrEmpty(fullName)) { + return null; + } + + return WHITESPACE.matcher(fullName).replaceAll(EMPTY); + }; + + private static final Function CONCAT_IH_FIRST_NAME = + s -> { + StringBuilder firstNameBuilder = new StringBuilder(); + if (!Strings.isNullOrEmpty(s.getFirstName())) { + firstNameBuilder.append(s.getFirstName()); + } + if (!Strings.isNullOrEmpty(s.getMiddleName())) { + firstNameBuilder.append(s.getMiddleName()); + } + + String firstName = firstNameBuilder.toString(); + + if (Strings.isNullOrEmpty(firstName)) { + return null; + } + + return firstName.trim(); + }; + + private static final Function CONCAT_PERSON_NAME = + p -> { + StringBuilder fullNameBuilder = new StringBuilder(); + if (!Strings.isNullOrEmpty(p.getFirstName())) { + fullNameBuilder.append(p.getFirstName()); + } + if (!Strings.isNullOrEmpty(p.getLastName())) { + fullNameBuilder.append(" ").append(p.getLastName()); + } + + String fullName = fullNameBuilder.toString(); + + if (Strings.isNullOrEmpty(fullName)) { + return null; + } + + return WHITESPACE.matcher(fullName).replaceAll(EMPTY); + }; + + public StaffDiffResult syncStaff( + T entity, List ihStaffList, List contacts) { + + StaffDiffResult.StaffDiffResultBuilder diffResult = + StaffDiffResult.builder().entity(entity); + + List contactsCopy = new ArrayList<>(contacts); + for (IHStaff ihStaff : ihStaffList) { + // try to find a match in the GrSciColl contacts + Set matches = matchWithContacts(ihStaff, contactsCopy); + + if (matches.isEmpty()) { + // no match among the contacts. We check now in all the GrSciColl persons. + Set globalMatches = matchGlobally(ihStaff, allGrSciCollPersons); + if (globalMatches.isEmpty()) { + // we create a new person and add it to the entity + Person person = entityConverter.convertToPerson(ihStaff); + diffResult.personToCreate(person); + } else if (globalMatches.size() > 1) { + // conflict + diffResult.conflict(new DiffResult.Conflict<>(ihStaff, new ArrayList<>(globalMatches))); + } else { + // there is one match. + Person globalMatch = globalMatches.iterator().next(); + compareStaff(ihStaff, globalMatch, diffResult); + } + } else if (matches.size() > 1) { + // conflict + contactsCopy.removeAll(matches); + diffResult.conflict(new DiffResult.Conflict<>(ihStaff, new ArrayList<>(matches))); + } else { + // there is one match + Person existing = matches.iterator().next(); + contactsCopy.remove(existing); + compareStaff(ihStaff, existing, diffResult); + } + } + + // remove from the GrSciColl entity the persons that don't exist in IH + diffResult.personsToRemoveFromEntity(contactsCopy); + + return diffResult.build(); + } + + private void compareStaff( + IHStaff ihStaff, Person existing, StaffDiffResult.StaffDiffResultBuilder diffResult) { + + if (isIHOutdated(ihStaff, existing)) { + // add issue + diffResult.outdatedStaff(new DiffResult.IHOutdated<>(ihStaff, existing)); + return; + } + + Person person = entityConverter.convertToPerson(ihStaff, existing); + if (!person.lenientEquals(existing)) { + diffResult.personToUpdate(new DiffResult.PersonDiffResult(existing, person)); + } else { + diffResult.personNoChange(person); + } + } + + private Set matchGlobally(IHStaff ihStaff, List grSciCollPersons) { + // first try with IRNs + Set matchesWithIrn = + grSciCollPersonsByIrn.getOrDefault(encodeIRN(ihStaff.getIrn()), Collections.emptySet()); + + if (!matchesWithIrn.isEmpty()) { + return matchesWithIrn; + } + + // we try to match with fields + return matchWithFields(ihStaff, grSciCollPersons, 11); + } + + private Set matchWithContacts(IHStaff ihStaff, List grSciCollPersons) { + // try to find a match by using the IRN identifiers + String irn = encodeIRN(ihStaff.getIrn()); + Set irnMatches = + grSciCollPersons.stream() + .filter( + p -> + p.getIdentifiers().stream() + .anyMatch(i -> Objects.equals(irn, i.getIdentifier()))) + .collect(Collectors.toSet()); + + if (!irnMatches.isEmpty()) { + return irnMatches; + } + + // no irn matches, we try to match with the fields + return matchWithFields(ihStaff, grSciCollPersons, 10); + } + + @VisibleForTesting + Set matchWithFields(IHStaff ihStaff, List persons, int minimumScore) { + if (persons.isEmpty()) { + return Collections.emptySet(); + } + + StaffNormalized ihStaffNorm = buildIHStaffNormalized(ihStaff, entityConverter); + + int maxScore = 0; + Set bestMatches = new HashSet<>(); + for (Person person : persons) { + StaffNormalized personNorm = buildGrSciCollPersonNormalized(person); + int equalityScore = getEqualityScore(ihStaffNorm, personNorm); + + if (equalityScore < minimumScore) { + continue; + } + + if (equalityScore > maxScore) { + bestMatches.clear(); + bestMatches.add(person); + maxScore = equalityScore; + } else if (equalityScore > 0 && equalityScore == maxScore) { + bestMatches.add(person); + } + } + + return bestMatches; + } + + private static StaffNormalized buildIHStaffNormalized( + IHStaff ihStaff, EntityConverter entityConverter) { + StaffNormalized.StaffNormalizedBuilder ihBuilder = + StaffNormalized.builder() + .fullName(CONCAT_IH_NAME.apply(ihStaff)) + .firstName(CONCAT_IH_FIRST_NAME.apply(ihStaff)) + .lastName(ihStaff.getLastName()) + .position(ihStaff.getPosition()); + + if (ihStaff.getContact() != null) { + ihBuilder + .email(ihStaff.getContact().getEmail()) + .phone(ihStaff.getContact().getPhone()) + .fax(ihStaff.getContact().getFax()); + } + + if (ihStaff.getAddress() != null) { + ihBuilder + .street(ihStaff.getAddress().getStreet()) + .city(ihStaff.getAddress().getCity()) + .state(ihStaff.getAddress().getState()) + .zipCode(ihStaff.getAddress().getZipCode()) + .country(entityConverter.matchCountry(ihStaff.getAddress().getCountry())); + } + + return ihBuilder.build(); + } + + private static StaffNormalized buildGrSciCollPersonNormalized(Person person) { + StaffNormalized.StaffNormalizedBuilder personBuilder = + StaffNormalized.builder() + .fullName(CONCAT_PERSON_NAME.apply(person)) + .firstName(person.getFirstName()) + .lastName(person.getLastName()) + .position(person.getPosition()) + .email(person.getEmail()) + .phone(person.getPhone()) + .fax(person.getFax()); + + if (person.getMailingAddress() != null) { + personBuilder + .street(person.getMailingAddress().getAddress()) + .city(person.getMailingAddress().getCity()) + .state(person.getMailingAddress().getProvince()) + .zipCode(person.getMailingAddress().getPostalCode()) + .country(person.getMailingAddress().getCountry()); + } + + return personBuilder.build(); + } + + private static int getEqualityScore(StaffNormalized staff1, StaffNormalized staff2) { + BiPredicate compareStrings = + (s1, s2) -> { + if (!Strings.isNullOrEmpty(s1) && !Strings.isNullOrEmpty(s2)) { + return s1.equalsIgnoreCase(s2); + } + return false; + }; + + BiPredicate compareNamePartially = + (s1, s2) -> { + if (!Strings.isNullOrEmpty(s1) && !Strings.isNullOrEmpty(s2)) { + return s1.startsWith(s2) || s2.startsWith(s1); + } + return false; + }; + + int score = 0; + if (compareStrings.test(staff1.email, staff2.email)) { + score += 10; + } + + if (compareStrings.test(staff1.fullName, staff2.fullName)) { + score += 10; + } else { + if (compareStrings.test(staff1.firstName, staff2.firstName)) { + score += 5; + } else if (compareNamePartially.test(staff1.firstName, staff2.firstName)) { + score += 4; + } + + if (compareStrings.test(staff1.lastName, staff2.lastName)) { + score += 5; + } else if (compareNamePartially.test(staff1.lastName, staff2.lastName)) { + score += 4; + } + } + + // at least the name or the email should match + if (score == 0) { + return score; + } + + if (compareStrings.test(staff1.phone, staff2.phone)) { + score += 3; + } + if (staff2.country != null && staff1.country == staff2.country) { + score += 3; + } + if (compareStrings.test(staff1.city, staff2.city)) { + score += 2; + } + if (compareStrings.test(staff1.position, staff2.position)) { + score += 2; + } + if (compareStrings.test(staff1.fax, staff2.fax)) { + score += 1; + } + if (compareStrings.test(staff1.street, staff2.street)) { + score += 1; + } + if (compareStrings.test(staff1.state, staff2.state)) { + score += 1; + } + if (compareStrings.test(staff1.zipCode, staff2.zipCode)) { + score += 1; + } + + return score; + } + + /** Contains all the common field between IH staff and GrSciColl persons. */ + @Data + @Builder + private static class StaffNormalized { + private String fullName; + private String firstName; + private String lastName; + private String email; + private String phone; + private String fax; + private String position; + private String street; + private String city; + private String state; + private String zipCode; + private Country country; + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/Utils.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/Utils.java new file mode 100644 index 0000000000..3858ec0eb4 --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/diff/Utils.java @@ -0,0 +1,57 @@ +package org.gbif.registry.collections.sync.diff; + +import org.gbif.api.model.collections.CollectionEntity; +import org.gbif.api.model.registry.Identifiable; +import org.gbif.api.vocabulary.IdentifierType; +import org.gbif.registry.collections.sync.ih.IHEntity; + +import java.time.LocalDate; +import java.time.ZoneOffset; +import java.util.*; + +import lombok.AccessLevel; +import lombok.NoArgsConstructor; + +@NoArgsConstructor(access = AccessLevel.PRIVATE) +public class Utils { + + private static final String GRSCICOLL_MIGRATION_USER = "registry-migration-grbio.gbif.org"; + + /** + * Encodes the IH IRN into the format stored on the GRSciColl identifier. E.g. 123 -> + * gbif:ih:irn:123 + */ + public static String encodeIRN(String irn) { + return "gbif:ih:irn:" + irn; + } + + /** + * Checks if a {@link CollectionEntity} is more up to date than a IH entity based in the modified + * date. We don't take into account the GrSciColl modifications made during the initial migration. + */ + public static boolean isIHOutdated(IHEntity ihEntity, CollectionEntity grSciCollEntity) { + return grSciCollEntity != null + && grSciCollEntity.getModified() != null + && !GRSCICOLL_MIGRATION_USER.equals(grSciCollEntity.getModifiedBy()) + && ihEntity != null + && grSciCollEntity + .getModified() + .toInstant() + .isAfter( + LocalDate.parse(ihEntity.getDateModified()) + .atStartOfDay() + .toInstant(ZoneOffset.UTC)); + } + + public static Map> mapByIrn( + List entities) { + Map> mapByIrn = new HashMap<>(); + entities.forEach( + o -> + o.getIdentifiers().stream() + .filter(i -> i.getType() == IdentifierType.IH_IRN) + .forEach( + i -> mapByIrn.computeIfAbsent(i.getIdentifier(), s -> new HashSet<>()).add(o))); + return mapByIrn; + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/grscicoll/GrSciCollHttpClient.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/grscicoll/GrSciCollHttpClient.java new file mode 100644 index 0000000000..571246f9e9 --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/grscicoll/GrSciCollHttpClient.java @@ -0,0 +1,230 @@ +package org.gbif.registry.collections.sync.grscicoll; + +import org.gbif.api.model.collections.Collection; +import org.gbif.api.model.collections.Institution; +import org.gbif.api.model.collections.Person; +import org.gbif.api.model.common.paging.PagingResponse; +import org.gbif.api.model.registry.Identifier; +import org.gbif.api.vocabulary.Country; +import org.gbif.registry.collections.sync.SyncConfig; +import org.gbif.registry.collections.sync.http.BasicAuthInterceptor; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import java.util.UUID; + +import com.fasterxml.jackson.core.JsonParser; +import com.fasterxml.jackson.databind.DeserializationContext; +import com.fasterxml.jackson.databind.JsonDeserializer; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.module.SimpleModule; +import okhttp3.OkHttpClient; +import retrofit2.Call; +import retrofit2.Retrofit; +import retrofit2.converter.jackson.JacksonConverterFactory; +import retrofit2.http.*; + +import static org.gbif.registry.collections.sync.http.SyncCall.syncCall; + +/** A lightweight GRSciColl client. */ +public class GrSciCollHttpClient { + + private final API api; + + private GrSciCollHttpClient(String grSciCollWsUrl, String user, String password) { + Objects.requireNonNull(grSciCollWsUrl); + + ObjectMapper mapper = new ObjectMapper(); + SimpleModule module = new SimpleModule(); + module.addDeserializer(Country.class, new IsoDeserializer()); + mapper.registerModule(module); + + OkHttpClient.Builder okHttpClientBuilder = new OkHttpClient.Builder(); + + if (user != null && password != null) { + okHttpClientBuilder.addInterceptor(new BasicAuthInterceptor(user, password)).build(); + } + + Retrofit retrofit = + new Retrofit.Builder() + .client(okHttpClientBuilder.build()) + .baseUrl(grSciCollWsUrl) + .addConverterFactory(JacksonConverterFactory.create(mapper)) + .build(); + api = retrofit.create(API.class); + } + + public static GrSciCollHttpClient create(String grSciCollWsUrl, String user, String password) { + return new GrSciCollHttpClient(grSciCollWsUrl, user, password); + } + + public static GrSciCollHttpClient create(String grSciCollWsUrl) { + return new GrSciCollHttpClient(grSciCollWsUrl, null, null); + } + + public static GrSciCollHttpClient create(SyncConfig syncConfig) { + if (syncConfig.isDryRun()) { + return new GrSciCollHttpClient(syncConfig.getRegistryWsUrl(), null, null); + } + return new GrSciCollHttpClient( + syncConfig.getRegistryWsUrl(), + syncConfig.getRegistryWsUser(), + syncConfig.getRegistryWsPassword()); + } + + /** Returns all institutions in GrSciCol. */ + public List getInstitutions() { + List result = new ArrayList<>(); + + boolean endRecords = false; + int offset = 0; + while (!endRecords) { + PagingResponse response = syncCall(api.listInstitutions(1000, offset)); + endRecords = response.isEndOfRecords(); + offset += response.getLimit(); + result.addAll(response.getResults()); + } + + return result; + } + + public void createInstitution(Institution institution) { + syncCall(api.createInstitution(institution)); + } + + public void updateInstitution(Institution institution) { + syncCall(api.updateInstitution(institution.getKey(), institution)); + } + + /** Returns all institutions in GrSciCol. */ + public List getCollections() { + List result = new ArrayList<>(); + + boolean endRecords = false; + int offset = 0; + while (!endRecords) { + PagingResponse response = syncCall(api.listCollections(1000, offset)); + endRecords = response.isEndOfRecords(); + offset += response.getLimit(); + result.addAll(response.getResults()); + } + + return result; + } + + public void updateCollection(Collection collection) { + syncCall(api.updateCollection(collection.getKey(), collection)); + } + + /** Returns all persons in GrSciCol. */ + public List getPersons() { + List result = new ArrayList<>(); + + boolean endRecords = false; + int offset = 0; + while (!endRecords) { + PagingResponse response = syncCall(api.listPersons(1000, offset)); + endRecords = response.isEndOfRecords(); + offset += response.getLimit(); + result.addAll(response.getResults()); + } + + return result; + } + + public UUID createPerson(Person person) { + return syncCall(api.createPerson(person)); + } + + public void updatePerson(Person person) { + syncCall(api.updatePerson(person.getKey(), person)); + } + + public void addIdentifierToPerson(UUID personKey, Identifier identifier) { + syncCall(api.addIdentifierToPerson(personKey, identifier)); + } + + public void addPersonToInstitution(UUID personKey, UUID institutionKey) { + syncCall(api.addPersonToInstitution(institutionKey, personKey)); + } + + public void removePersonFromInstitution(UUID personKey, UUID institutionKey) { + syncCall(api.removePersonFromInstitution(institutionKey, personKey)); + } + + public void addPersonToCollection(UUID personKey, UUID collectionKey) { + syncCall(api.addPersonToCollection(collectionKey, personKey)); + } + + public void removePersonFromCollection(UUID personKey, UUID collectionKey) { + syncCall(api.removePersonFromCollection(collectionKey, personKey)); + } + + private interface API { + @GET("institution") + Call> listInstitutions( + @Query("limit") int limit, @Query("offset") int offset); + + @POST("institution") + Call createInstitution(@Body Institution institution); + + @PUT("institution/{key}") + Call updateInstitution(@Path("key") UUID key, @Body Institution institution); + + @GET("collection") + Call> listCollections( + @Query("limit") int limit, @Query("offset") int offset); + + @PUT("collection/{key}") + Call updateCollection(@Path("key") UUID key, @Body Collection collection); + + @GET("person") + Call> listPersons( + @Query("limit") int limit, @Query("offset") int offset); + + @POST("person") + Call createPerson(@Body Person person); + + @PUT("person/{key}") + Call updatePerson(@Path("key") UUID key, @Body Person person); + + @POST("person/{key}/identifier") + Call addIdentifierToPerson(@Path("key") UUID personKey, @Body Identifier identifier); + + @POST("{institutionKey}/contact") + Call addPersonToInstitution( + @Path("institutionKey") UUID institutionKey, @Body UUID personKey); + + @DELETE("{institutionKey}/contact/{personKey}") + Call removePersonFromInstitution( + @Path("institutionKey") UUID institutionKey, @Path("personKey") UUID personKey); + + @POST("{collectionKey}/contact") + Call addPersonToCollection( + @Path("collectionKey") UUID collectionKey, @Body UUID personKey); + + @DELETE("{collectionKey}/contact/{personKey}") + Call removePersonFromCollection( + @Path("collectionKey") UUID collectionKey, @Path("personKey") UUID personKey); + } + + /** Adapter necessary for retrofit due to versioning. */ + private static class IsoDeserializer extends JsonDeserializer { + @Override + public Country deserialize(JsonParser jp, DeserializationContext ctxt) throws IOException { + try { + if (jp != null && jp.getTextLength() > 0) { + return Country.fromIsoCode(jp.getText()); + } else { + return Country.UNKNOWN; // none provided + } + } catch (Exception e) { + throw new IOException( + "Unable to deserialize country from provided value (not an ISO 2 character?): " + + jp.getText()); + } + } + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/http/BasicAuthInterceptor.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/http/BasicAuthInterceptor.java new file mode 100644 index 0000000000..5c7194ee09 --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/http/BasicAuthInterceptor.java @@ -0,0 +1,23 @@ +package org.gbif.registry.collections.sync.http; + +import java.io.IOException; + +import okhttp3.*; + +/** Interceptor for the {@link OkHttpClient} to add basic auth in all the requests. */ +public class BasicAuthInterceptor implements Interceptor { + + private String credentials; + + public BasicAuthInterceptor(String user, String password) { + this.credentials = Credentials.basic(user, password); + } + + @Override + public Response intercept(Chain chain) throws IOException { + Request request = chain.request(); + Request authenticatedRequest = + request.newBuilder().header("Authorization", credentials).build(); + return chain.proceed(authenticatedRequest); + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/http/SyncCall.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/http/SyncCall.java new file mode 100644 index 0000000000..f838d1187f --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/http/SyncCall.java @@ -0,0 +1,44 @@ +package org.gbif.registry.collections.sync.http; + + +import java.io.IOException; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import retrofit2.Call; +import retrofit2.HttpException; +import retrofit2.Response; + +/** + * Utility class to perform synchronous call on Retrofit services. + */ +public class SyncCall { + + private static final Logger LOG = LoggerFactory.getLogger(SyncCall.class); + + /** + * Private constructor. + */ + private SyncCall() { + //DO NOTHING + } + + /** + * Performs a synchronous call to {@link Call} instance. + * @param call to be executed + * @param content of the response object + * @return the content of the response, throws an {@link HttpException} in case of error + */ + public static T syncCall(Call call) { + try { + Response response = call.execute(); + if (response.isSuccessful()) { + return response.body(); + } + LOG.error("Service responded with an error {}", response); + throw new HttpException(response); // Propagates the failed response + } catch (IOException ex) { + throw new IllegalStateException("Error executing call", ex); + } + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHEntity.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHEntity.java new file mode 100644 index 0000000000..acc803a8d3 --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHEntity.java @@ -0,0 +1,10 @@ +package org.gbif.registry.collections.sync.ih; + +/** Tag interface for IH entities. */ +public interface IHEntity { + + String getIrn(); + + String getDateModified(); + +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHHttpClient.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHHttpClient.java new file mode 100644 index 0000000000..41cb795ffe --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHHttpClient.java @@ -0,0 +1,76 @@ +package org.gbif.registry.collections.sync.ih; + +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; + +import lombok.Data; +import retrofit2.Call; +import retrofit2.Retrofit; +import retrofit2.converter.jackson.JacksonConverterFactory; +import retrofit2.http.GET; +import retrofit2.http.Query; + +import static org.gbif.registry.collections.sync.http.SyncCall.syncCall; + +/** Lightweight IndexHerbariorum client. */ +public class IHHttpClient { + + private final API api; + + private IHHttpClient(String ihWsUrl) { + Objects.requireNonNull(ihWsUrl); + + Retrofit retrofit = + new Retrofit.Builder() + .baseUrl(ihWsUrl) + .addConverterFactory(JacksonConverterFactory.create()) + .build(); + api = retrofit.create(API.class); + } + + public static IHHttpClient create(String ihWsUrl) { + return new IHHttpClient(ihWsUrl); + } + + public List getInstitutions() { + return syncCall(api.listInstitutions()).getData(); + } + + public List getStaffByInstitution(String institutionCode) { + return syncCall(api.listStaff(institutionCode)).getData(); + } + + public List getCountries() { + return syncCall(api.listCountries()).getData(); + } + + private interface API { + @GET("institutions") + Call listInstitutions(); + + @GET("staff/search") + Call listStaff(@Query("code") String institutionCode); + + @GET("countries") + Call listCountries(); + } + + @Data + private static class InstitutionWrapper { + private IHMetadata meta; + private List data; + } + + @Data + private static class StaffWrapper { + private IHMetadata meta; + private List data = new ArrayList<>(); + } + + @Data + private static class CountryWrapper { + private IHMetadata meta; + private List data = new ArrayList<>(); + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHInstitution.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHInstitution.java new file mode 100644 index 0000000000..a2955390fc --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHInstitution.java @@ -0,0 +1,46 @@ +package org.gbif.registry.collections.sync.ih; + +import lombok.Data; + +/** Models an Index Herbariorum institution. */ +@Data +public class IHInstitution implements IHEntity { + + private String irn; + private String organization; + private String code; + private String division; + private String department; + private long specimenTotal; + private Address address; + private Contact contact; + private Location location; + private String dateModified; + + @Data + public static class Address { + private String physicalStreet; + private String physicalCity; + private String physicalState; + private String physicalZipCode; + private String physicalCountry; + private String postalStreet; + private String postalCity; + private String postalState; + private String postalZipCode; + private String postalCountry; + } + + @Data + public static class Contact { + private String phone; + private String email; + private String webUrl; + } + + @Data + public static class Location { + private Double lat; + private Double lon; + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHMetadata.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHMetadata.java new file mode 100644 index 0000000000..3de90a024c --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHMetadata.java @@ -0,0 +1,11 @@ +package org.gbif.registry.collections.sync.ih; + +import lombok.Data; + +/** Models the Index Herbariorum metadata that is used in the WS responses. */ +@Data +public class IHMetadata { + private int hits; + private int code; + private String message; +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHStaff.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHStaff.java new file mode 100644 index 0000000000..948861b3e1 --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/ih/IHStaff.java @@ -0,0 +1,37 @@ +package org.gbif.registry.collections.sync.ih; + +import lombok.Data; + +/** Models an Index Herbariorum staff. */ +@Data +public class IHStaff implements IHEntity { + + private String irn; + private String code; + private String lastName; + private String middleName; + private String firstName; + private String birthDate; + private String correspondent; + private String position; + private String specialities; + private Address address; + private Contact contact; + private String dateModified; + + @Data + public static class Address { + private String street; + private String city; + private String state; + private String zipCode; + private String country; + } + + @Data + public static class Contact { + private String phone; + private String email; + private String fax; + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/notification/GithubClient.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/notification/GithubClient.java new file mode 100644 index 0000000000..fbb9fff1d7 --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/notification/GithubClient.java @@ -0,0 +1,69 @@ +package org.gbif.registry.collections.sync.notification; + +import org.gbif.registry.collections.sync.SyncConfig; +import org.gbif.registry.collections.sync.http.BasicAuthInterceptor; + +import java.util.List; +import java.util.Objects; + +import okhttp3.OkHttpClient; +import retrofit2.Call; +import retrofit2.Retrofit; +import retrofit2.converter.jackson.JacksonConverterFactory; +import retrofit2.http.Body; +import retrofit2.http.POST; + +import static org.gbif.registry.collections.sync.http.SyncCall.syncCall; + +/** Lightweight client for the Github API. */ +public class GithubClient { + + private final API api; + private final List assignees; + + private GithubClient(String githubWsUrl, String user, String password, List assignees) { + Objects.requireNonNull(githubWsUrl); + Objects.requireNonNull(user); + Objects.requireNonNull(password); + + OkHttpClient okHttpClient = + new OkHttpClient.Builder().addInterceptor(new BasicAuthInterceptor(user, password)).build(); + + Retrofit retrofit = + new Retrofit.Builder() + .client(okHttpClient) + .baseUrl(githubWsUrl) + .addConverterFactory(JacksonConverterFactory.create()) + .build(); + api = retrofit.create(API.class); + this.assignees = assignees; + } + + public static GithubClient create(String githubWsUrl, String user, String password) { + return new GithubClient(githubWsUrl, user, password, null); + } + + public static GithubClient create(SyncConfig syncConfig) { + Objects.requireNonNull(syncConfig); + Objects.requireNonNull(syncConfig.getNotification()); + return new GithubClient( + syncConfig.getNotification().getGithubWsUrl(), + syncConfig.getNotification().getGithubUser(), + syncConfig.getNotification().getGithubPassword(), + syncConfig.getNotification().getGhIssuesAssignees()); + } + + public void createIssue(Issue issue) { + if (assignees != null && !assignees.isEmpty()) { + // we use the assignees from the config if they were set + issue.setAssignees(assignees); + } + + syncCall(api.createIssue(issue)); + } + + private interface API { + @POST("issues") + Call createIssue(@Body Issue issue); + } +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/notification/Issue.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/notification/Issue.java new file mode 100644 index 0000000000..0cc9668e10 --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/notification/Issue.java @@ -0,0 +1,17 @@ +package org.gbif.registry.collections.sync.notification; + +import java.util.List; + +import lombok.AllArgsConstructor; +import lombok.Builder; +import lombok.Data; + +@Data +@Builder +@AllArgsConstructor +public class Issue { + private String title; + private String body; + private List assignees; + private List labels; +} diff --git a/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/notification/IssueFactory.java b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/notification/IssueFactory.java new file mode 100644 index 0000000000..c3fccfcc5a --- /dev/null +++ b/registry-collections-sync/src/main/java/org/gbif/registry/collections/sync/notification/IssueFactory.java @@ -0,0 +1,228 @@ +package org.gbif.registry.collections.sync.notification; + +import org.gbif.api.model.collections.Collection; +import org.gbif.api.model.collections.CollectionEntity; +import org.gbif.api.model.collections.Institution; +import org.gbif.api.model.collections.Person; +import org.gbif.registry.collections.sync.diff.DiffResult; +import org.gbif.registry.collections.sync.ih.IHEntity; +import org.gbif.registry.collections.sync.ih.IHInstitution; +import org.gbif.registry.collections.sync.ih.IHStaff; + +import java.net.URI; +import java.util.Collections; +import java.util.List; +import java.util.function.UnaryOperator; + +import static org.gbif.registry.collections.sync.SyncConfig.NotificationConfig; + +/** Factory to create {@link Issue}. */ +public class IssueFactory { + + private static final String IH_OUTDATED_TITLE = "The IH %s with IRN %s is outdated"; + private static final String ENTITY_CONFLICT_TITLE = + "IH %s with IRN %s matches with multiple GrSciColl entities"; + private static final String FAILS_TITLE = + "Some operations have failed updating the registry in the IH sync"; + private static final String NEW_LINE = "\n"; + private static final String CODE_SEPARATOR = "```"; + private static final List DEFAULT_LABELS = Collections.singletonList("GrSciColl-IH sync"); + + private static final UnaryOperator PORTAL_URL_NORMALIZER = + url -> { + if (url != null && url.endsWith("/")) { + return url.substring(0, url.length() - 1); + } + return url; + }; + + private final NotificationConfig config; + private final String ihInstitutionLink; + private final String ihStaffLink; + private final String registryInstitutionLink; + private final String registryCollectionLink; + private final String registryPersonLink; + + private IssueFactory(NotificationConfig config) { + this.config = config; + this.ihInstitutionLink = + PORTAL_URL_NORMALIZER.apply(config.getIhPortalUrl()) + "/ih/herbarium-details/?irn=%s"; + this.ihStaffLink = + PORTAL_URL_NORMALIZER.apply(config.getIhPortalUrl()) + "/ih/person-details/?irn=%s"; + this.registryInstitutionLink = + PORTAL_URL_NORMALIZER.apply(config.getRegistryPortalUrl()) + "/institution/%s"; + this.registryCollectionLink = + PORTAL_URL_NORMALIZER.apply(config.getRegistryPortalUrl()) + "/collection/%s"; + this.registryPersonLink = + PORTAL_URL_NORMALIZER.apply(config.getRegistryPortalUrl()) + "/person/%s"; + } + + public static IssueFactory fromConfig(NotificationConfig config) { + return new IssueFactory(config); + } + + public static IssueFactory fromDefaults() { + NotificationConfig config = new NotificationConfig(); + config.setGhIssuesAssignees(Collections.emptyList()); + return new IssueFactory(config); + } + + public Issue createOutdatedIHStaffIssue(Person grSciCollPerson, IHStaff ihStaff) { + return createOutdatedIHEntityIssue( + grSciCollPerson, ihStaff.getIrn(), ihStaff.toString(), EntityType.IH_STAFF); + } + + public Issue createOutdatedIHInstitutionIssue( + CollectionEntity grSciCollEntity, T ihInstitution) { + return createOutdatedIHEntityIssue( + grSciCollEntity, + ihInstitution.getIrn(), + ihInstitution.toString(), + EntityType.IH_INSTITUTION); + } + + private Issue createOutdatedIHEntityIssue( + CollectionEntity grSciCollEntity, + String irn, + String ihEntityAsString, + EntityType entityType) { + + // create body + StringBuilder body = new StringBuilder(); + body.append("The IH ") + .append(createLink(irn, entityType)) + .append(":") + .append(formatEntity(ihEntityAsString)) + .append("has a more up-to-date ") + .append( + createLink(grSciCollEntity.getKey().toString(), getRegistryEntityType(grSciCollEntity))) + .append(" in GrSciColl:") + .append(formatEntity(grSciCollEntity.toString())) + .append("Please check which one should remain and sync them in both systems."); + + if (entityType != EntityType.IH_STAFF) { + body.append(" Remember to sync the associated staff too."); + } + + return Issue.builder() + .title(String.format(IH_OUTDATED_TITLE, entityType.title, irn)) + .body(body.toString()) + .assignees(config.getGhIssuesAssignees()) + .labels(DEFAULT_LABELS) + .build(); + } + + public Issue createConflict(List entities, IHInstitution ihInstitution) { + return createConflict(entities, ihInstitution, EntityType.IH_INSTITUTION); + } + + public Issue createStaffConflict(List persons, IHStaff ihStaff) { + return createConflict(persons, ihStaff, EntityType.IH_STAFF); + } + + private Issue createConflict( + List entities, IHEntity ihEntity, EntityType ihEntityType) { + // create body + StringBuilder body = new StringBuilder(); + body.append("The IH ") + .append(createLink(ihEntity.getIrn(), ihEntityType)) + .append(":") + .append(formatEntity(ihEntity)) + .append("have multiple candidates in GrSciColl: " + NEW_LINE); + entities.forEach( + e -> + body.append(createLink(e.getKey().toString(), getRegistryEntityType(e), true)) + .append(formatEntity(e))); + body.append("A IH ") + .append(ihEntityType.title) + .append(" should be associated to only one GrSciColl entity. Please resolve the conflict."); + + return Issue.builder() + .title(String.format(ENTITY_CONFLICT_TITLE, ihEntityType.title, ihEntity.getIrn())) + .body(body.toString()) + .assignees(config.getGhIssuesAssignees()) + .labels(DEFAULT_LABELS) + .build(); + } + + public Issue createFailsNotification(List fails) { + // create body + StringBuilder body = new StringBuilder(); + body.append( + "The next operations have failed when updating the registry with the IH data. Please check them: "); + + fails.forEach( + f -> + body.append(NEW_LINE) + .append("Error: ") + .append(f.getMessage()) + .append(NEW_LINE) + .append("Entity: ") + .append(f.getEntity())); + + return Issue.builder() + .title(FAILS_TITLE) + .body(body.toString()) + .assignees(config.getGhIssuesAssignees()) + .labels(DEFAULT_LABELS) + .build(); + } + + private String formatEntity(Object entity) { + return NEW_LINE + + CODE_SEPARATOR + + NEW_LINE + + entity.toString() + + NEW_LINE + + CODE_SEPARATOR + + NEW_LINE; + } + + private String createLink(String id, EntityType entityType) { + return createLink(id, entityType, false); + } + + private String createLink(String id, EntityType entityType, boolean omitText) { + String linkTemplate; + if (entityType == EntityType.IH_INSTITUTION) { + linkTemplate = ihInstitutionLink; + } else if (entityType == EntityType.IH_STAFF) { + linkTemplate = ihStaffLink; + } else if (entityType == EntityType.INSTITUTION) { + linkTemplate = registryInstitutionLink; + } else if (entityType == EntityType.COLLECTION) { + linkTemplate = registryCollectionLink; + } else { + linkTemplate = registryPersonLink; + } + + URI uri = URI.create(String.format(linkTemplate, id)); + String text = omitText ? uri.toString() : entityType.title; + + return "[" + text + "](" + uri.toString() + ")"; + } + + private EntityType getRegistryEntityType(CollectionEntity entity) { + if (entity instanceof Institution) { + return EntityType.INSTITUTION; + } else if (entity instanceof Collection) { + return EntityType.COLLECTION; + } else { + return EntityType.PERSON; + } + } + + private enum EntityType { + IH_INSTITUTION("institution"), + IH_STAFF("staff"), + INSTITUTION("institution"), + COLLECTION("collection"), + PERSON("person"); + + String title; + + EntityType(String title) { + this.title = title; + } + } +} diff --git a/registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/SyncConfigTest.java b/registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/SyncConfigTest.java new file mode 100644 index 0000000000..5600d150de --- /dev/null +++ b/registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/SyncConfigTest.java @@ -0,0 +1,30 @@ +package org.gbif.registry.collections.sync; + +import java.nio.file.Paths; + +import org.junit.Test; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +/** Tests the {@link SyncConfig}. */ +public class SyncConfigTest { + + private static final String CONFIG_TEST_PATH = "src/test/resources/sync-config.yaml"; + + @Test + public void loadConfigTest() { + String path = Paths.get(CONFIG_TEST_PATH).toFile().getAbsolutePath(); + SyncConfig config = SyncConfig.fromFileName(path).orElse(null); + + assertNotNull(config); + assertNotNull(config.getRegistryWsUrl()); + assertNotNull(config.getIhWsUrl()); + assertTrue(config.isSaveResultsToFile()); + assertTrue(config.isDryRun()); + assertTrue(config.isSendNotifications()); + assertNotNull(config.getNotification()); + assertFalse(config.getNotification().getGhIssuesAssignees().isEmpty()); + } +} diff --git a/registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/diff/EntityConverterTest.java b/registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/diff/EntityConverterTest.java new file mode 100644 index 0000000000..a8e245948a --- /dev/null +++ b/registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/diff/EntityConverterTest.java @@ -0,0 +1,289 @@ +package org.gbif.registry.collections.sync.diff; + +import org.gbif.api.model.collections.Address; +import org.gbif.api.model.collections.Collection; +import org.gbif.api.model.collections.Institution; +import org.gbif.api.model.collections.Person; +import org.gbif.api.model.registry.Identifier; +import org.gbif.api.vocabulary.Country; +import org.gbif.api.vocabulary.IdentifierType; +import org.gbif.api.vocabulary.collections.InstitutionType; +import org.gbif.registry.collections.sync.ih.IHHttpClient; +import org.gbif.registry.collections.sync.ih.IHInstitution; +import org.gbif.registry.collections.sync.ih.IHStaff; + +import java.math.BigDecimal; +import java.net.URI; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; + +import org.junit.Ignore; +import org.junit.Test; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +/** Tests the {@link EntityConverter}. */ +public class EntityConverterTest { + + private static final String IRN_TEST = "1"; + private static final EntityConverter entityConverter = + EntityConverter.builder() + .countries(Arrays.asList("U.K.", "U.S.A.", "United Kingdom", "United States")) + .creationUser("test-user") + .build(); + + @Test + public void institutionConversionFromExistingTest() { + // Existing + Institution existing = new Institution(); + existing.setCode("UARK"); + existing.setName("University of Arkansas OLD"); + existing.setType(InstitutionType.OTHER_INSTITUTIONAL_TYPE); + existing.setLatitude(new BigDecimal("36.0424")); + existing.setLongitude(new BigDecimal("-94.1624")); + existing.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, Utils.encodeIRN(IRN_TEST))); + + Address address = new Address(); + address.setCity("foo"); + address.setProvince("Arkansas"); + address.setCountry(Country.UNITED_STATES); + existing.setMailingAddress(address); + + // IH Institution + IHInstitution ih = getIhInstitution(); + + // Expected + Institution expected = new Institution(); + expected.setCode("UARK"); + expected.setName("University of Arkansas"); + expected.setType(InstitutionType.OTHER_INSTITUTIONAL_TYPE); + expected.setIndexHerbariorumRecord(true); + expected.setLatitude(BigDecimal.valueOf(30d)); + expected.setLongitude(BigDecimal.valueOf(-80d)); + expected.setNumberSpecimens(1000); + expected.setEmail(Arrays.asList("uark@uark.com", "uark2@uark.com")); + expected.setPhone(Arrays.asList("123", "456", "789")); + expected.setHomepage(URI.create("http://www.a.com")); + expected.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, Utils.encodeIRN(IRN_TEST))); + expected.setMailingAddress(getExpectedMailingAddress()); + expected.setAddress(getExpectedAddress()); + + // When + Institution converted = entityConverter.convertToInstitution(ih, existing); + + // Expect + assertTrue(converted.lenientEquals(expected)); + } + + @Test + public void institutionConversionTest() { + // IH Institution + IHInstitution ih = getIhInstitution(); + + // Expected + Institution expected = new Institution(); + expected.setCode("UARK"); + expected.setName("University of Arkansas"); + expected.setIndexHerbariorumRecord(true); + expected.setLatitude(BigDecimal.valueOf(30d)); + expected.setLongitude(BigDecimal.valueOf(-80d)); + expected.setNumberSpecimens(1000); + expected.setEmail(Arrays.asList("uark@uark.com", "uark2@uark.com")); + expected.setPhone(Arrays.asList("123", "456", "789")); + expected.setHomepage(URI.create("http://www.a.com")); + expected.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, Utils.encodeIRN(IRN_TEST))); + expected.setMailingAddress(getExpectedMailingAddress()); + expected.setAddress(getExpectedAddress()); + + // When + Institution converted = entityConverter.convertToInstitution(ih); + + // Expect + assertTrue(converted.lenientEquals(expected)); + } + + @Test + public void collectionConversionFromExistingTest() { + // Existing + Collection existing = new Collection(); + existing.setCode("code"); + existing.setName("old name"); + existing.setEmail(Collections.singletonList("bb@bb.com")); + existing.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, Utils.encodeIRN(IRN_TEST))); + + Address address = new Address(); + address.setCity("foo"); + address.setProvince("Arkansas"); + address.setCountry(Country.AFGHANISTAN); + existing.setMailingAddress(address); + + // IH Institution + IHInstitution ih = getIhInstitution(); + + // Expected + Collection expected = new Collection(); + expected.setCode("UARK"); + expected.setName("University of Arkansas"); + expected.setIndexHerbariorumRecord(true); + expected.setEmail(Arrays.asList("uark@uark.com", "uark2@uark.com")); + expected.setPhone(Arrays.asList("123", "456", "789")); + expected.setHomepage(URI.create("http://www.a.com")); + expected.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, Utils.encodeIRN(IRN_TEST))); + expected.setMailingAddress(getExpectedMailingAddress()); + expected.setAddress(getExpectedAddress()); + expected.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, Utils.encodeIRN(IRN_TEST))); + + // When + Collection converted = entityConverter.convertToCollection(ih, existing); + + // Expect + assertTrue(converted.lenientEquals(expected)); + } + + @Test + public void personConversionFromExistingTest() { + // Existing + Person existing = new Person(); + existing.setFirstName("John Wayne"); + existing.setEmail("wayne@test.com"); + existing.setPosition("Dummy"); + existing.setFax("0000"); + + Address address = new Address(); + address.setCity("foo"); + address.setProvince("Arkansas"); + address.setCountry(Country.AFGHANISTAN); + existing.setMailingAddress(address); + + // IH Staff + IHStaff ihStaff = getIhStaff(); + + // Expected + Person expected = new Person(); + expected.setFirstName("First M."); + expected.setLastName("Last"); + expected.setEmail("a@a.com"); + expected.setPhone("123"); + expected.setFax("987"); + expected.setPosition("Collections Manager"); + expected.setMailingAddress(getExpectedAddress()); + expected.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, Utils.encodeIRN(IRN_TEST))); + + // When + Person converted = entityConverter.convertToPerson(ihStaff, existing); + + // Expect + assertTrue(converted.lenientEquals(expected)); + } + + @Test + public void personConversionTest() { + // IH Staff + IHStaff ihStaff = getIhStaff(); + + // Expected + Person expected = new Person(); + expected.setFirstName("First M."); + expected.setLastName("Last"); + expected.setEmail("a@a.com"); + expected.setPhone("123"); + expected.setFax("987"); + expected.setPosition("Collections Manager"); + expected.setMailingAddress(getExpectedAddress()); + expected.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, Utils.encodeIRN(IRN_TEST))); + + // When + Person converted = entityConverter.convertToPerson(ihStaff); + + // Expect + assertTrue(converted.lenientEquals(expected)); + } + + @Ignore("Manual test") + @Test + public void testCountryMapping() { + IHHttpClient ihHttpClient = IHHttpClient.create("http://sweetgum.nybg.org/science/api/v1/"); + List countries = ihHttpClient.getCountries(); + + Map mappings = EntityConverter.mapCountries(countries); + + assertEquals(countries.size(), mappings.size()); + } + + private IHStaff getIhStaff() { + IHStaff s = new IHStaff(); + s.setIrn(IRN_TEST); + s.setCode("UARK"); + s.setLastName("Last"); + s.setMiddleName("M."); + s.setFirstName("First"); + s.setPosition("Collections Manager"); + + IHStaff.Address address = new IHStaff.Address(); + address.setStreet("street"); + address.setCity("FAYETTEVILLE"); + address.setState("state"); + address.setCountry("U.S.A."); + address.setZipCode("zip"); + s.setAddress(address); + + IHStaff.Contact contact = new IHStaff.Contact(); + contact.setEmail("a@a.com\nb@b.com"); + contact.setPhone("123,456\n789"); + contact.setFax("987;654"); + s.setContact(contact); + + return s; + } + + private IHInstitution getIhInstitution() { + // IH Institution + IHInstitution ih = new IHInstitution(); + ih.setIrn(IRN_TEST); + ih.setCode("UARK"); + ih.setOrganization("University of Arkansas"); + ih.setSpecimenTotal(1000); + + IHInstitution.Address ihAddress = new IHInstitution.Address(); + ihAddress.setPhysicalStreet("street"); + ihAddress.setPhysicalState("state"); + ihAddress.setPhysicalZipCode("zip"); + ihAddress.setPhysicalCity("FAYETTEVILLE"); + ihAddress.setPhysicalCountry("United States"); + ihAddress.setPostalCity("FAYETTEVILLE"); + ihAddress.setPostalCountry("U.K."); + ih.setAddress(ihAddress); + + IHInstitution.Location location = new IHInstitution.Location(); + location.setLat(30d); + location.setLon(-80d); + ih.setLocation(location); + + IHInstitution.Contact contact = new IHInstitution.Contact(); + contact.setEmail("uark@uark.com\nuark2@uark.com"); + contact.setPhone("123,456\n789"); + contact.setWebUrl("http://www. a.com;http://www.b.com"); + ih.setContact(contact); + return ih; + } + + private Address getExpectedMailingAddress() { + Address expectedMailingAddress = new Address(); + expectedMailingAddress.setCity("FAYETTEVILLE"); + expectedMailingAddress.setCountry(Country.UNITED_KINGDOM); + return expectedMailingAddress; + } + + private Address getExpectedAddress() { + Address expectedAddress = new Address(); + expectedAddress.setAddress("street"); + expectedAddress.setProvince("state"); + expectedAddress.setPostalCode("zip"); + expectedAddress.setCity("FAYETTEVILLE"); + expectedAddress.setCountry(Country.UNITED_STATES); + return expectedAddress; + } +} diff --git a/registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/diff/IndexHerbariorumDiffFinderTest.java b/registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/diff/IndexHerbariorumDiffFinderTest.java new file mode 100644 index 0000000000..db9a66ec68 --- /dev/null +++ b/registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/diff/IndexHerbariorumDiffFinderTest.java @@ -0,0 +1,339 @@ +package org.gbif.registry.collections.sync.diff; + +import org.gbif.api.model.collections.Address; +import org.gbif.api.model.collections.Collection; +import org.gbif.api.model.collections.CollectionEntity; +import org.gbif.api.model.collections.Institution; +import org.gbif.api.model.registry.Identifier; +import org.gbif.api.vocabulary.Country; +import org.gbif.api.vocabulary.IdentifierType; +import org.gbif.api.vocabulary.collections.InstitutionType; +import org.gbif.registry.collections.sync.ih.IHInstitution; +import org.gbif.registry.collections.sync.ih.IHStaff; + +import java.math.BigDecimal; +import java.sql.Date; +import java.time.LocalDateTime; +import java.time.ZoneOffset; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.UUID; +import java.util.function.Function; +import java.util.stream.Collectors; + +import com.google.common.collect.ImmutableList; +import lombok.Builder; +import lombok.Data; +import org.junit.Test; + +import static org.gbif.registry.collections.sync.diff.Utils.encodeIRN; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertTrue; + +/** Tests the {@link IndexHerbariorumDiffFinder}. */ +public class IndexHerbariorumDiffFinderTest { + + private static final String IRN_TEST = "1"; + private static final String IRN_TEST_2 = "2"; + + private static final Function> EMPTY_STAFF = (p) -> Collections.emptyList(); + private static final EntityConverter ENTITY_CONVERTER = + EntityConverter.builder() + .countries(Arrays.asList("U.K.", "U.S.A.", "United Kingdom", "United States")) + .creationUser("test-user") + .build(); + + @Test + public void syncInstitutionsTest() { + TestEntity institutionToCreate = createInstitutionToCreate(); + TestEntity institutionNoChange = createInstitutionNoChange(); + TestEntity institutionToUpdate = createInstitutionToUpdate(); + + List allTestEntities = + ImmutableList.of(institutionToCreate, institutionNoChange, institutionToUpdate); + List grSciCollInstitutions = + allTestEntities.stream() + .filter(e -> e.getEntity() != null) + .map(e -> (Institution) e.getEntity()) + .collect(Collectors.toList()); + + List ihInstitutions = + allTestEntities.stream() + .filter(e -> e.getIhInstitution() != null) + .map(TestEntity::getIhInstitution) + .collect(Collectors.toList()); + + DiffResult result = + IndexHerbariorumDiffFinder.builder() + .ihInstitutions(ihInstitutions) + .ihStaffFetcher(EMPTY_STAFF) + .institutions(grSciCollInstitutions) + .entityConverter(ENTITY_CONVERTER) + .build() + .find(); + + assertEquals(1, result.getInstitutionsToCreate().size()); + assertEquals(1, result.getInstitutionsNoChange().size()); + assertEquals(1, result.getInstitutionsToUpdate().size()); + assertTrue(result.getOutdatedIHInstitutions().isEmpty()); + assertTrue(result.getCollectionsToUpdate().isEmpty()); + assertTrue(result.getCollectionsNoChange().isEmpty()); + + assertFalse(grSciCollInstitutions.contains(result.getInstitutionsToCreate().get(0))); + assertEquals(institutionNoChange.entity, result.getInstitutionsNoChange().get(0)); + assertNotEquals( + result.getInstitutionsToUpdate().get(0).getNewEntity(), + result.getInstitutionsToUpdate().get(0).getOldEntity()); + assertTrue(result.getInstitutionsToUpdate().get(0).getNewEntity().isIndexHerbariorumRecord()); + } + + @Test + public void syncCollectionsTest() { + TestEntity collectionsNoChange = createCollectionNoChange(); + TestEntity collectionsToUpdate = createCollectionToUpdate(); + + List allTestEntities = ImmutableList.of(collectionsNoChange, collectionsToUpdate); + List grSciCollCollections = + allTestEntities.stream() + .filter(e -> e.getEntity() != null) + .map(e -> (Collection) e.getEntity()) + .collect(Collectors.toList()); + + List ihInstitutions = + allTestEntities.stream() + .filter(e -> e.getIhInstitution() != null) + .map(TestEntity::getIhInstitution) + .collect(Collectors.toList()); + + DiffResult result = + IndexHerbariorumDiffFinder.builder() + .ihInstitutions(ihInstitutions) + .ihStaffFetcher(EMPTY_STAFF) + .collections(grSciCollCollections) + .entityConverter(ENTITY_CONVERTER) + .build() + .find(); + + assertEquals(1, result.getCollectionsNoChange().size()); + assertEquals(1, result.getCollectionsToUpdate().size()); + assertTrue(result.getOutdatedIHInstitutions().isEmpty()); + assertTrue(result.getInstitutionsToUpdate().isEmpty()); + assertTrue(result.getInstitutionsNoChange().isEmpty()); + assertTrue(result.getInstitutionsToCreate().isEmpty()); + + assertEquals(collectionsNoChange.entity, result.getCollectionsNoChange().get(0)); + assertNotEquals( + result.getCollectionsToUpdate().get(0).getNewEntity(), + result.getCollectionsToUpdate().get(0).getOldEntity()); + assertTrue(result.getCollectionsToUpdate().get(0).getNewEntity().isIndexHerbariorumRecord()); + } + + @Test + public void outdatedInstitutionConflictTest() { + IHInstitution outdatedIh = new IHInstitution(); + outdatedIh.setIrn(IRN_TEST); + outdatedIh.setDateModified("2018-01-01"); + + Institution upToDateInstitution = new Institution(); + upToDateInstitution.setKey(UUID.randomUUID()); + upToDateInstitution.setModified(Date.from(LocalDateTime.now().toInstant(ZoneOffset.UTC))); + upToDateInstitution + .getIdentifiers() + .add(new Identifier(IdentifierType.IH_IRN, encodeIRN(IRN_TEST))); + + DiffResult result = + IndexHerbariorumDiffFinder.builder() + .ihInstitutions(Collections.singletonList(outdatedIh)) + .ihStaffFetcher(EMPTY_STAFF) + .institutions(Collections.singletonList(upToDateInstitution)) + .entityConverter(ENTITY_CONVERTER) + .build() + .find(); + + assertEquals(1, result.getOutdatedIHInstitutions().size()); + } + + @Test + public void outdatedCollectionConflictTest() { + IHInstitution outdatedIh = new IHInstitution(); + outdatedIh.setIrn(IRN_TEST); + outdatedIh.setDateModified("2018-01-01"); + + Collection upToDateCollection = new Collection(); + upToDateCollection.setKey(UUID.randomUUID()); + upToDateCollection.setModified(Date.from(LocalDateTime.now().toInstant(ZoneOffset.UTC))); + upToDateCollection + .getIdentifiers() + .add(new Identifier(IdentifierType.IH_IRN, encodeIRN(IRN_TEST))); + + DiffResult result = + IndexHerbariorumDiffFinder.builder() + .ihInstitutions(Collections.singletonList(outdatedIh)) + .ihStaffFetcher(EMPTY_STAFF) + .collections(Collections.singletonList(upToDateCollection)) + .entityConverter(ENTITY_CONVERTER) + .build() + .find(); + + assertEquals(1, result.getOutdatedIHInstitutions().size()); + } + + @Test + public void multipleMatchesConflictTest() { + IHInstitution ih1 = new IHInstitution(); + ih1.setIrn(IRN_TEST); + ih1.setCode("A"); + IHInstitution ih2 = new IHInstitution(); + ih2.setIrn(IRN_TEST_2); + ih2.setCode("B"); + + Institution i1 = new Institution(); + i1.setKey(UUID.randomUUID()); + i1.setCode("i1"); + i1.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, encodeIRN(IRN_TEST))); + + Institution i2 = new Institution(); + i2.setKey(UUID.randomUUID()); + i2.setCode("i2"); + i2.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, encodeIRN(IRN_TEST))); + + Collection c1 = new Collection(); + c1.setKey(UUID.randomUUID()); + c1.setCode("c1"); + c1.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, encodeIRN(IRN_TEST_2))); + + Collection c2 = new Collection(); + c2.setKey(UUID.randomUUID()); + c2.setCode("c2"); + c2.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, encodeIRN(IRN_TEST_2))); + + DiffResult result = + IndexHerbariorumDiffFinder.builder() + .ihInstitutions(Arrays.asList(ih1, ih2)) + .ihStaffFetcher(EMPTY_STAFF) + .institutions(Arrays.asList(i1, i2)) + .collections(Arrays.asList(c1, c2)) + .entityConverter(ENTITY_CONVERTER) + .build() + .find(); + + assertEquals(2, result.getConflicts().size()); + } + + private TestEntity createInstitutionToCreate() { + IHInstitution ih = new IHInstitution(); + ih.setCode("foo"); + ih.setOrganization("foo"); + ih.setSpecimenTotal(1000); + + return TestEntity.builder().ihInstitution(ih).build(); + } + + private TestEntity createInstitutionNoChange() { + Institution i = new Institution(); + i.setKey(UUID.randomUUID()); + i.setCode("bar"); + i.setName("bar"); + i.setIndexHerbariorumRecord(true); + i.setNumberSpecimens(1000); + i.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, Utils.encodeIRN(IRN_TEST))); + + IHInstitution ih = new IHInstitution(); + ih.setIrn(IRN_TEST); + ih.setCode("bar"); + ih.setOrganization("bar"); + ih.setSpecimenTotal(1000); + + return TestEntity.builder().entity(i).ihInstitution(ih).build(); + } + + private TestEntity createInstitutionToUpdate() { + Institution i = new Institution(); + i.setKey(UUID.randomUUID()); + i.setCode("UARK"); + i.setName("University of Arkansas OLD"); + i.setType(InstitutionType.HERBARIUM); + i.setIndexHerbariorumRecord(false); + i.setLatitude(new BigDecimal("36.0424")); + i.setLongitude(new BigDecimal("-94.1624")); + i.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, Utils.encodeIRN(IRN_TEST_2))); + + Address address = new Address(); + address.setCity("FAYETTEVILLE"); + address.setProvince("Arkansas"); + address.setCountry(Country.UNITED_STATES); + i.setMailingAddress(address); + + IHInstitution ih = new IHInstitution(); + ih.setIrn(IRN_TEST_2); + ih.setCode("UARK"); + ih.setOrganization("University of Arkansas"); + ih.setSpecimenTotal(1000); + + IHInstitution.Address ihAddress = new IHInstitution.Address(); + ihAddress.setPhysicalCity("FAYETTEVILLE"); + ihAddress.setPhysicalCountry("U.S.A."); + ihAddress.setPostalCity("FAYETTEVILLE"); + ihAddress.setPostalCountry("U.S.A."); + ih.setAddress(ihAddress); + + IHInstitution.Location location = new IHInstitution.Location(); + location.setLat(30d); + location.setLon(-80d); + ih.setLocation(location); + + IHInstitution.Contact contact = new IHInstitution.Contact(); + contact.setEmail("uark@uark.com"); + ih.setContact(contact); + + return TestEntity.builder().entity(i).ihInstitution(ih).build(); + } + + private TestEntity createCollectionNoChange() { + Collection c = new Collection(); + c.setKey(UUID.randomUUID()); + c.setCode("A"); + c.setIndexHerbariorumRecord(true); + c.setEmail(Collections.singletonList("aa@aa.com")); + c.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, Utils.encodeIRN(IRN_TEST))); + + IHInstitution ih = new IHInstitution(); + ih.setIrn(IRN_TEST); + ih.setCode("A"); + IHInstitution.Contact contact = new IHInstitution.Contact(); + contact.setEmail("aa@aa.com"); + ih.setContact(contact); + + return TestEntity.builder().entity(c).ihInstitution(ih).build(); + } + + private TestEntity createCollectionToUpdate() { + Collection c = new Collection(); + c.setKey(UUID.randomUUID()); + c.setCode("B"); + c.setEmail(Collections.singletonList("bb@bb.com")); + c.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, Utils.encodeIRN(IRN_TEST_2))); + + IHInstitution ih = new IHInstitution(); + ih.setIrn(IRN_TEST_2); + ih.setCode("A"); + + IHInstitution.Contact contact = new IHInstitution.Contact(); + contact.setEmail("bb@bb.com"); + contact.setPhone("12345"); + ih.setContact(contact); + + return TestEntity.builder().entity(c).ihInstitution(ih).build(); + } + + @Builder + @Data + private static class TestEntity { + CollectionEntity entity; + IHInstitution ihInstitution; + } +} diff --git a/registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/diff/StaffDiffFinderTest.java b/registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/diff/StaffDiffFinderTest.java new file mode 100644 index 0000000000..9df539ea81 --- /dev/null +++ b/registry-collections-sync/src/test/java/org/gbif/registry/collections/sync/diff/StaffDiffFinderTest.java @@ -0,0 +1,368 @@ +package org.gbif.registry.collections.sync.diff; + +import org.gbif.api.model.collections.Address; +import org.gbif.api.model.collections.Institution; +import org.gbif.api.model.collections.Person; +import org.gbif.api.model.registry.Identifier; +import org.gbif.api.vocabulary.Country; +import org.gbif.api.vocabulary.IdentifierType; +import org.gbif.registry.collections.sync.ih.IHStaff; + +import java.sql.Date; +import java.time.LocalDateTime; +import java.time.ZoneOffset; +import java.util.*; +import java.util.stream.Collectors; + +import com.google.common.collect.ImmutableList; +import lombok.Builder; +import lombok.Data; +import org.junit.Test; + +import static org.gbif.registry.collections.sync.diff.Utils.encodeIRN; + +import static org.junit.Assert.*; + +/** Tests the {@link StaffDiffFinder}. */ +public class StaffDiffFinderTest { + + private static final EntityConverter ENTITY_CONVERTER = + EntityConverter.builder() + .countries(Arrays.asList("U.K.", "U.S.A.", "United Kingdom", "United States")) + .creationUser("test-user") + .build(); + private static final StaffDiffFinder STAFF_DIFF_FINDER = + StaffDiffFinder.builder() + .entityConverter(ENTITY_CONVERTER) + .allGrSciCollPersons(Collections.emptyList()) + .build(); + private static final String IRN_TEST = "1"; + private static final Institution DEFAULT_INSTITUTION = new Institution(); + + static { + DEFAULT_INSTITUTION.setKey(UUID.randomUUID()); + DEFAULT_INSTITUTION.setCode("code"); + } + + @Test + public void syncStaffTest() { + TestStaff staffToUpdate = createTestStaffToUpdate(); + TestStaff staffToCreate = createTestStaffToCreate(); + TestStaff staffToDelete = createTestStaffToDelete(); + TestStaff staffNoChange = createTestStaffNoChange(); + List allStaff = + ImmutableList.of(staffToUpdate, staffToCreate, staffToDelete, staffNoChange); + + List ihStaff = + allStaff.stream() + .filter(i -> i.getIhStaff() != null) + .map(TestStaff::getIhStaff) + .collect(Collectors.toList()); + List grSciCollPersons = + allStaff.stream() + .filter(i -> i.getPerson() != null) + .map(TestStaff::getPerson) + .collect(Collectors.toList()); + + DiffResult.StaffDiffResult result = + STAFF_DIFF_FINDER.syncStaff(DEFAULT_INSTITUTION, ihStaff, grSciCollPersons); + assertEquals(1, result.getPersonsToCreate().size()); + assertEquals(1, result.getPersonsToUpdate().size()); + assertEquals(1, result.getPersonsNoChange().size()); + assertEquals(1, result.getPersonsToRemoveFromEntity().size()); + + assertEquals(staffNoChange.person, result.getPersonsNoChange().get(0)); + assertNotEquals( + result.getPersonsToUpdate().get(0).getNewPerson(), + result.getPersonsToUpdate().get(0).getOldPerson()); + assertEquals(staffToDelete.person, result.getPersonsToRemoveFromEntity().get(0)); + assertFalse(grSciCollPersons.contains(result.getPersonsToCreate().get(0))); + } + + @Test + public void outdatedIHConflictTest() { + IHStaff outdatedStaff = new IHStaff(); + outdatedStaff.setIrn(IRN_TEST); + outdatedStaff.setDateModified("2018-01-01"); + + Person upToDatePerson = new Person(); + upToDatePerson.setKey(UUID.randomUUID()); + upToDatePerson.setModified(Date.from(LocalDateTime.now().toInstant(ZoneOffset.UTC))); + upToDatePerson.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, encodeIRN(IRN_TEST))); + + DiffResult.StaffDiffResult result = + STAFF_DIFF_FINDER.syncStaff( + DEFAULT_INSTITUTION, + Collections.singletonList(outdatedStaff), + Collections.singletonList(upToDatePerson)); + + assertEquals(1, result.getOutdatedStaff().size()); + } + + @Test + public void syncWithGlobalMatchUpdateTest() { + IHStaff s = new IHStaff(); + s.setCode("UARK"); + s.setLastName("Last"); + s.setMiddleName("M."); + s.setFirstName("First"); + s.setPosition("Collections Manager"); + + IHStaff.Contact contact = new IHStaff.Contact(); + contact.setEmail("a@a.com"); + s.setContact(contact); + + Person existing = new Person(); + existing.setKey(UUID.randomUUID()); + existing.setFirstName("First"); + existing.setPosition("foo"); + existing.setEmail("a@a.com"); + + StaffDiffFinder staffDiffFinder = + StaffDiffFinder.builder() + .entityConverter(ENTITY_CONVERTER) + .allGrSciCollPersons(Collections.singletonList(existing)) + .build(); + + DiffResult.StaffDiffResult diffResult = + staffDiffFinder.syncStaff( + DEFAULT_INSTITUTION, Collections.singletonList(s), Collections.emptyList()); + + assertEquals(1, diffResult.getPersonsToUpdate().size()); + assertEquals( + s.getPosition(), diffResult.getPersonsToUpdate().get(0).getNewPerson().getPosition()); + assertNotNull(diffResult.getPersonsToUpdate().get(0).getNewPerson().getKey()); + } + + @Test + public void syncWithGlobalMatchCreateTest() { + IHStaff s = new IHStaff(); + s.setCode("UARK"); + s.setLastName("Last"); + s.setMiddleName("M."); + s.setFirstName("First"); + s.setPosition("Collections Manager"); + + IHStaff.Contact contact = new IHStaff.Contact(); + contact.setEmail("a@a.com"); + s.setContact(contact); + + StaffDiffFinder staffDiffFinder = + StaffDiffFinder.builder() + .entityConverter(ENTITY_CONVERTER) + .allGrSciCollPersons(Collections.emptyList()) + .build(); + + DiffResult.StaffDiffResult diffResult = + staffDiffFinder.syncStaff( + DEFAULT_INSTITUTION, Collections.singletonList(s), Collections.emptyList()); + + assertEquals(1, diffResult.getPersonsToCreate().size()); + assertNull(diffResult.getPersonsToCreate().get(0).getKey()); + } + + @Test + public void multipleMatchesWithIrnConflictTest() { + IHStaff s = new IHStaff(); + s.setIrn(IRN_TEST); + + Person p1 = new Person(); + p1.setKey(UUID.randomUUID()); + p1.setEmail("aa@aa.com"); + p1.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, encodeIRN(IRN_TEST))); + + Person p2 = new Person(); + p2.setKey(UUID.randomUUID()); + p2.setEmail("bb@bb.com"); + p2.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, encodeIRN(IRN_TEST))); + + DiffResult.StaffDiffResult result = + STAFF_DIFF_FINDER.syncStaff( + DEFAULT_INSTITUTION, Collections.singletonList(s), Arrays.asList(p1, p2)); + + assertEquals(1, result.getConflicts().size()); + } + + @Test + public void multipleMatchesWithFieldsConflictTest() { + IHStaff s = new IHStaff(); + s.setIrn(IRN_TEST); + s.setFirstName("Name"); + s.setLastName("Last"); + + Person p1 = new Person(); + p1.setKey(UUID.randomUUID()); + p1.setFirstName("Name Last"); + + Person p2 = new Person(); + p2.setKey(UUID.randomUUID()); + p2.setFirstName("Name"); + p2.setLastName("Last"); + + DiffResult.StaffDiffResult result = + STAFF_DIFF_FINDER.syncStaff( + DEFAULT_INSTITUTION, Collections.singletonList(s), Arrays.asList(p1, p2)); + + assertEquals(1, result.getConflicts().size()); + } + + @Test + public void matchWithFieldsTest() { + // IH Staff + IHStaff s = new IHStaff(); + s.setFirstName("First"); + s.setMiddleName("M."); + s.setLastName("Last"); + s.setPosition("Manager"); + + IHStaff.Address ihAddress = new IHStaff.Address(); + ihAddress.setStreet(""); + ihAddress.setCity("Fayetteville"); + ihAddress.setState("AR"); + ihAddress.setCountry("U.S.A."); + ihAddress.setZipCode("72701"); + s.setAddress(ihAddress); + + IHStaff.Contact contact = new IHStaff.Contact(); + contact.setPhone("[1] 479 575 4372"); + contact.setEmail("a@a.com"); + s.setContact(contact); + + // GrSciColl persons + Person p1 = new Person(); + p1.setFirstName("First M."); + p1.setEmail("a@a.com"); + + Person p2 = new Person(); + p2.setPosition("Manager"); + Address address = new Address(); + address.setCountry(Country.UNITED_STATES); + p2.setMailingAddress(address); + + // When + Set persons = STAFF_DIFF_FINDER.matchWithFields(s, Arrays.asList(p1, p2), 0); + + // Expect + assertEquals(1, persons.size()); + assertTrue(persons.contains(p1)); + + // GrSciColl persons + p1 = new Person(); + p1.setFirstName("First"); + + p2 = new Person(); + p2.setPosition("Manager"); + address = new Address(); + address.setCountry(Country.UNITED_STATES); + p2.setMailingAddress(address); + + // When + persons = STAFF_DIFF_FINDER.matchWithFields(s, Arrays.asList(p1, p2), 0); + + // Expect + assertEquals(1, persons.size()); + assertTrue(persons.contains(p1)); + + // GrSciColl persons + p1 = new Person(); + p1.setFirstName("Fir"); + p1.setPosition("Manager"); + + p2 = new Person(); + p2.setLastName("Last"); + + // When + persons = STAFF_DIFF_FINDER.matchWithFields(s, Arrays.asList(p1, p2), 0); + + // Expect + assertEquals(1, persons.size()); + assertTrue(persons.contains(p1)); + } + + private TestStaff createTestStaffToUpdate() { + Person p = new Person(); + p.setKey(UUID.randomUUID()); + p.setFirstName("First M. Last"); + p.setPosition("Director"); + p.setPhone("[1] 479/575-4372"); + p.setEmail("a@uark.edu"); + Address mailingAddress = new Address(); + mailingAddress.setCity("FAYETTEVILLE"); + mailingAddress.setProvince("Arkansas"); + mailingAddress.setCountry(Country.UNITED_STATES); + p.setMailingAddress(mailingAddress); + + IHStaff s = new IHStaff(); + s.setCode("UARK"); + s.setLastName("Last"); + s.setMiddleName("M."); + s.setFirstName("First"); + s.setPosition("Professor Emeritus"); + + IHStaff.Address address = new IHStaff.Address(); + address.setStreet(""); + address.setCity("Fayetteville"); + address.setState("Arkansas"); + address.setCountry("U.S.A."); + s.setAddress(address); + + IHStaff.Contact contact = new IHStaff.Contact(); + contact.setEmail("a@uark.edu"); + s.setContact(contact); + + return TestStaff.builder().person(p).ihStaff(s).build(); + } + + private TestStaff createTestStaffNoChange() { + Person p = new Person(); + p.setKey(UUID.randomUUID()); + p.setEmail("foo@foo.com"); + p.getIdentifiers().add(new Identifier(IdentifierType.IH_IRN, encodeIRN(IRN_TEST))); + + IHStaff s = new IHStaff(); + s.setIrn(IRN_TEST); + IHStaff.Contact contact = new IHStaff.Contact(); + contact.setEmail("foo@foo.com"); + s.setContact(contact); + + return TestStaff.builder().person(p).ihStaff(s).build(); + } + + private TestStaff createTestStaffToDelete() { + Person p = new Person(); + p.setKey(UUID.randomUUID()); + p.setFirstName("extra person"); + return TestStaff.builder().person(p).build(); + } + + private TestStaff createTestStaffToCreate() { + IHStaff s = new IHStaff(); + s.setCode("UARK"); + s.setLastName("Last"); + s.setMiddleName("M."); + s.setFirstName("First"); + s.setPosition("Collections Manager"); + + IHStaff.Address address = new IHStaff.Address(); + address.setStreet(""); + address.setCity("Fayetteville"); + address.setState("AR"); + address.setCountry("U.S.A."); + address.setZipCode("72701"); + s.setAddress(address); + + IHStaff.Contact contact = new IHStaff.Contact(); + contact.setPhone("[1] 479 575 4372"); + contact.setEmail("a@uark.edu"); + s.setContact(contact); + + return TestStaff.builder().ihStaff(s).build(); + } + + @Builder + @Data + private static class TestStaff { + Person person; + IHStaff ihStaff; + } +} diff --git a/registry-collections-sync/src/test/resources/sync-config.yaml b/registry-collections-sync/src/test/resources/sync-config.yaml new file mode 100644 index 0000000000..6ad8d0a0c6 --- /dev/null +++ b/registry-collections-sync/src/test/resources/sync-config.yaml @@ -0,0 +1,13 @@ +registryWsUrl: http://test.com/ +ihWsUrl: http://test2.com/ +saveResultsToFile: true +dryRun: true +sendNotifications: true +notification: + registryPortalUrl: https://a.com/ + ihPortalUrl: https://b.com/ + githubWsUrl: https://c.com/ + githubUser: bar + githubPassword: fake + ghIssuesAssignees: + - foo diff --git a/registry-identity/src/test/java/org/gbif/identity/util/PasswordEncoderTest.java b/registry-identity/src/test/java/org/gbif/identity/util/PasswordEncoderTest.java index eea85439b2..9ab6db823c 100644 --- a/registry-identity/src/test/java/org/gbif/identity/util/PasswordEncoderTest.java +++ b/registry-identity/src/test/java/org/gbif/identity/util/PasswordEncoderTest.java @@ -51,4 +51,5 @@ public void testWithSalting() throws Exception { String encoded2 = encoder.encode(password, encoded1); // encode again reading the salt genarate above assertEquals(encoded1, encoded2); // verify they } + } diff --git a/registry-liquibase/src/main/resources/liquibase/065-ih-sync.xml b/registry-liquibase/src/main/resources/liquibase/065-ih-sync.xml new file mode 100644 index 0000000000..0615f418e8 --- /dev/null +++ b/registry-liquibase/src/main/resources/liquibase/065-ih-sync.xml @@ -0,0 +1,133 @@ + + + + + + + + diff --git a/registry-liquibase/src/main/resources/liquibase/master.xml b/registry-liquibase/src/main/resources/liquibase/master.xml index b6eb01ecc7..05faeae5d2 100644 --- a/registry-liquibase/src/main/resources/liquibase/master.xml +++ b/registry-liquibase/src/main/resources/liquibase/master.xml @@ -68,4 +68,5 @@ + diff --git a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/GenericTypes.java b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/GenericTypes.java index 4a8be2780f..dcb7f421f7 100644 --- a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/GenericTypes.java +++ b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/GenericTypes.java @@ -41,7 +41,7 @@ /** * Package access utility to provide generic types. */ -class GenericTypes { +public class GenericTypes { public static final GenericType> PAGING_NODE = new GenericType>() { }; diff --git a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseClient.java b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseClient.java new file mode 100644 index 0000000000..a8a383ad17 --- /dev/null +++ b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseClient.java @@ -0,0 +1,161 @@ +package org.gbif.registry.ws.client.collections; + +import org.gbif.api.model.collections.CollectionEntity; +import org.gbif.api.model.common.paging.Pageable; +import org.gbif.api.model.common.paging.PagingResponse; +import org.gbif.api.model.registry.*; +import org.gbif.api.service.collections.CrudService; +import org.gbif.api.service.registry.IdentifierService; +import org.gbif.api.service.registry.MachineTagService; +import org.gbif.api.service.registry.TagService; +import org.gbif.api.vocabulary.TagName; +import org.gbif.api.vocabulary.TagNamespace; +import org.gbif.registry.ws.client.GenericTypes; +import org.gbif.ws.client.BaseWsGetClient; +import org.gbif.ws.client.QueryParamBuilder; + +import java.io.IOException; +import java.util.List; +import java.util.UUID; +import javax.annotation.Nullable; +import javax.validation.constraints.NotNull; +import javax.ws.rs.core.MediaType; + +import com.sun.jersey.api.client.GenericType; +import com.sun.jersey.api.client.WebResource; +import com.sun.jersey.api.client.filter.ClientFilter; +import org.codehaus.jackson.map.ObjectMapper; + +/** Base ws client for {@link CollectionEntity}. */ +public abstract class BaseClient< + T extends CollectionEntity & Taggable & MachineTaggable & Identifiable> + extends BaseWsGetClient + implements CrudService, TagService, MachineTagService, IdentifierService { + + protected static final GenericType> LIST_TAG = new GenericType>() {}; + protected static final GenericType> LIST_IDENTIFIER = + new GenericType>() {}; + private final GenericType> pagingType; + private final ObjectMapper mapper = new ObjectMapper(); + + protected BaseClient( + Class resourceClass, + WebResource resource, + @Nullable ClientFilter authFilter, + GenericType> pagingType) { + super(resourceClass, resource, authFilter); + this.pagingType = pagingType; + } + + @Override + public UUID create(@NotNull T entity) { + return post(UUID.class, entity, "/"); + } + + @Override + public void delete(@NotNull UUID uuid) { + delete(String.valueOf(uuid)); + } + + @Override + public void update(@NotNull T entity) { + put(entity, String.valueOf(entity.getKey())); + } + + @Override + public int addIdentifier(@NotNull UUID key, @NotNull Identifier identifier) { + return post(Integer.class, identifier, String.valueOf(key), "identifier"); + } + + @Override + public void deleteIdentifier(@NotNull UUID key, int identifierKey) { + delete(String.valueOf(key), "identifier", String.valueOf(identifierKey)); + } + + @Override + public List listIdentifiers(@NotNull UUID key) { + return get(LIST_IDENTIFIER, null, null, (Pageable) null, String.valueOf(key), "identifier"); + } + + @Override + public int addTag(@NotNull UUID key, @NotNull String value) { + return addTag(key, new Tag(value)); + } + + @Override + public int addTag(@NotNull UUID key, @NotNull Tag tag) { + return post(Integer.class, tag, String.valueOf(key), "tag"); + } + + @Override + public void deleteTag(@NotNull UUID key, int tagKey) { + delete(String.valueOf(key), "tag", String.valueOf(tagKey)); + } + + @Override + public List listTags(@NotNull UUID uuid, @Nullable String s) { + return get(LIST_TAG, null, null, (Pageable) null, String.valueOf(uuid), "tag"); + } + + @Override + public int addMachineTag(UUID targetEntityKey, MachineTag machineTag) { + // allow post through varnish (no chunked encoding needed) + int tagId; + try { + tagId = getResource() + .path(targetEntityKey.toString()) + .path("machineTag") + .type(MediaType.APPLICATION_JSON) + .post(Integer.class, mapper.writeValueAsBytes(machineTag)); + + } catch (IOException e) { + throw new IllegalStateException(e); + } + return tagId; + } + + @Override + public int addMachineTag(@NotNull UUID targetEntityKey, @NotNull TagName tagName, @NotNull String value) { + return addMachineTag(targetEntityKey, new MachineTag(tagName, value)); + } + + @Override + public int addMachineTag(@NotNull UUID targetEntityKey, @NotNull String namespace, @NotNull String name, @NotNull String value) { + return addMachineTag(targetEntityKey, new MachineTag(namespace, name, value)); + } + + @Override + public void deleteMachineTag(UUID targetEntityKey, int machineTagKey) { + delete(targetEntityKey.toString(), "machineTag", String.valueOf(machineTagKey)); + } + + @Override + public void deleteMachineTags(@NotNull UUID targetEntityKey, @NotNull String namespace) { + delete(targetEntityKey.toString(), "machineTag", String.valueOf(namespace)); + } + + @Override + public void deleteMachineTags(@NotNull UUID targetEntityKey, @NotNull TagNamespace tagNamespace) { + delete(targetEntityKey.toString(), "machineTag", tagNamespace.getNamespace()); + } + + @Override + public void deleteMachineTags(@NotNull UUID targetEntityKey, @NotNull String namespace, @NotNull String name) { + delete(targetEntityKey.toString(), "machineTag", String.valueOf(namespace), String.valueOf(name)); + } + + @Override + public void deleteMachineTags(@NotNull UUID targetEntityKey, @NotNull TagName tagName) { + delete(targetEntityKey.toString(), "machineTag", tagName.getNamespace().getNamespace(), tagName.getName()); + } + + @Override + public List listMachineTags(UUID targetEntityKey) { + return get(GenericTypes.LIST_MACHINETAG, null, null, (Pageable) null, targetEntityKey.toString(), "machineTag"); + } + + public PagingResponse listByMachineTag(String namespace, @Nullable String name, @Nullable String value, @Nullable Pageable page) { + return get(pagingType, null, QueryParamBuilder.create("machineTagNamespace", namespace, "machineTagName", name, "machineTagValue", value).build(), page); + } + +} diff --git a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseCrudClient.java b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseCrudClient.java deleted file mode 100644 index 347755e4e8..0000000000 --- a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseCrudClient.java +++ /dev/null @@ -1,45 +0,0 @@ -package org.gbif.registry.ws.client.collections; - -import org.gbif.api.model.collections.CollectionEntity; -import org.gbif.api.model.common.paging.PagingResponse; -import org.gbif.api.service.collections.CrudService; -import org.gbif.ws.client.BaseWsGetClient; - -import java.util.UUID; -import javax.annotation.Nullable; -import javax.validation.constraints.NotNull; - -import com.sun.jersey.api.client.GenericType; -import com.sun.jersey.api.client.WebResource; -import com.sun.jersey.api.client.filter.ClientFilter; - -/** Base ws client for {@link CollectionEntity}. */ -public abstract class BaseCrudClient extends BaseWsGetClient - implements CrudService { - - private final GenericType> pagingType; - - protected BaseCrudClient( - Class resourceClass, - WebResource resource, - @Nullable ClientFilter authFilter, - GenericType> pagingType) { - super(resourceClass, resource, authFilter); - this.pagingType = pagingType; - } - - @Override - public UUID create(@NotNull T entity) { - return post(UUID.class, entity, "/"); - } - - @Override - public void delete(@NotNull UUID uuid) { - delete(String.valueOf(uuid)); - } - - @Override - public void update(@NotNull T entity) { - put(entity, String.valueOf(entity.getKey())); - } -} diff --git a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseExtendableCollectionEntityClient.java b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseExtendedCollectionEntityClient.java similarity index 50% rename from registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseExtendableCollectionEntityClient.java rename to registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseExtendedCollectionEntityClient.java index 5e4a1f70b0..914131b367 100644 --- a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseExtendableCollectionEntityClient.java +++ b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/BaseExtendedCollectionEntityClient.java @@ -5,13 +5,10 @@ import org.gbif.api.model.collections.Person; import org.gbif.api.model.common.paging.Pageable; import org.gbif.api.model.registry.Identifiable; -import org.gbif.api.model.registry.Identifier; -import org.gbif.api.model.registry.Tag; +import org.gbif.api.model.registry.MachineTaggable; import org.gbif.api.model.registry.Taggable; import org.gbif.api.model.registry.search.collections.KeyCodeNameResult; import org.gbif.api.service.collections.ContactService; -import org.gbif.api.service.registry.IdentifierService; -import org.gbif.api.service.registry.TagService; import java.util.List; import java.util.UUID; @@ -26,17 +23,15 @@ * Base ws client for {@link CollectionEntity} that are also {@link Taggable}, {@link Identifiable} * and {@link Contactable}. */ -public abstract class BaseExtendableCollectionEntityClient - extends BaseCrudClient implements TagService, IdentifierService, ContactService { +public abstract class BaseExtendedCollectionEntityClient< + T extends CollectionEntity & Taggable & MachineTaggable & Identifiable & Contactable> + extends BaseClient implements ContactService { protected static final GenericType> LIST_PERSON = new GenericType>() {}; - protected static final GenericType> LIST_TAG = new GenericType>() {}; - protected static final GenericType> LIST_IDENTIFIER = - new GenericType>() {}; protected static final GenericType> LIST_KEY_CODE_NAME = new GenericType>() {}; - protected BaseExtendableCollectionEntityClient( + protected BaseExtendedCollectionEntityClient( Class resourceClass, WebResource resource, @Nullable ClientFilter authFilter, @@ -58,39 +53,4 @@ public void addContact(@NotNull UUID entityKey, @NotNull UUID contactKey) { public void removeContact(@NotNull UUID entityKey, @NotNull UUID contactKey) { delete(String.valueOf(entityKey), "contact", String.valueOf(contactKey)); } - - @Override - public int addIdentifier(@NotNull UUID key, @NotNull Identifier identifier) { - return post(Integer.class, identifier, String.valueOf(key), "identifier"); - } - - @Override - public void deleteIdentifier(@NotNull UUID key, int identifierKey) { - delete(String.valueOf(key), "identifier", String.valueOf(identifierKey)); - } - - @Override - public List listIdentifiers(@NotNull UUID key) { - return get(LIST_IDENTIFIER, null, null, (Pageable) null, String.valueOf(key), "identifier"); - } - - @Override - public int addTag(@NotNull UUID key, @NotNull String value) { - return addTag(key, new Tag(value)); - } - - @Override - public int addTag(@NotNull UUID key, @NotNull Tag tag) { - return post(Integer.class, tag, String.valueOf(key), "tag"); - } - - @Override - public void deleteTag(@NotNull UUID key, int tagKey) { - delete(String.valueOf(key), "tag", String.valueOf(tagKey)); - } - - @Override - public List listTags(@NotNull UUID uuid, @Nullable String s) { - return get(LIST_TAG, null, null, (Pageable) null, String.valueOf(uuid), "tag"); - } } diff --git a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/CollectionWsClient.java b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/CollectionWsClient.java index 8383370386..ebb9474867 100644 --- a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/CollectionWsClient.java +++ b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/CollectionWsClient.java @@ -19,7 +19,7 @@ import com.sun.jersey.api.client.filter.ClientFilter; import com.sun.jersey.core.util.MultivaluedMapImpl; -public class CollectionWsClient extends BaseExtendableCollectionEntityClient implements CollectionService { +public class CollectionWsClient extends BaseExtendedCollectionEntityClient implements CollectionService { private static final GenericType> PAGING_COLLECTION = new GenericType>() {}; @@ -37,13 +37,19 @@ protected CollectionWsClient( @Override public PagingResponse list( - @Nullable String query, @Nullable UUID institutionKey, @Nullable UUID contactKey, @Nullable Pageable pageable - ) { + @Nullable String query, + @Nullable UUID institutionKey, + @Nullable UUID contactKey, + @Nullable String code, + @Nullable String name, + @Nullable Pageable pageable) { return get(PAGING_COLLECTION, null, QueryParamBuilder.create("institution", institutionKey) .queryParam("contact", contactKey) .queryParam("q", query) + .queryParam("code", code) + .queryParam("name", name) .build(), pageable); } diff --git a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/InstitutionWsClient.java b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/InstitutionWsClient.java index ea3c019e2a..5419bbbfbb 100644 --- a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/InstitutionWsClient.java +++ b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/InstitutionWsClient.java @@ -19,7 +19,7 @@ import com.sun.jersey.api.client.filter.ClientFilter; import com.sun.jersey.core.util.MultivaluedMapImpl; -public class InstitutionWsClient extends BaseExtendableCollectionEntityClient +public class InstitutionWsClient extends BaseExtendedCollectionEntityClient implements InstitutionService { private static final GenericType> PAGING_INSTITUTION = @@ -38,12 +38,20 @@ protected InstitutionWsClient( @Override public PagingResponse list( - @Nullable String query, @Nullable UUID contactKey, @Nullable Pageable pageable - ) { - return get(PAGING_INSTITUTION, - null, - QueryParamBuilder.create("q", query).queryParam("contact", contactKey).build(), - pageable); + @Nullable String query, + @Nullable UUID contactKey, + @Nullable String code, + @Nullable String name, + @Nullable Pageable pageable) { + return get( + PAGING_INSTITUTION, + null, + QueryParamBuilder.create("q", query) + .queryParam("contact", contactKey) + .queryParam("code", code) + .queryParam("name", name) + .build(), + pageable); } @Override diff --git a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/PersonWsClient.java b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/PersonWsClient.java index 026dd9f396..735a1324fe 100644 --- a/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/PersonWsClient.java +++ b/registry-ws-client/src/main/java/org/gbif/registry/ws/client/collections/PersonWsClient.java @@ -3,14 +3,17 @@ import org.gbif.api.model.collections.Person; import org.gbif.api.model.common.paging.Pageable; import org.gbif.api.model.common.paging.PagingResponse; +import org.gbif.api.model.registry.Identifier; import org.gbif.api.model.registry.search.collections.PersonSuggestResult; import org.gbif.api.service.collections.PersonService; +import org.gbif.api.service.registry.IdentifierService; import org.gbif.registry.ws.client.guice.RegistryWs; import org.gbif.ws.client.QueryParamBuilder; import java.util.List; import java.util.UUID; import javax.annotation.Nullable; +import javax.validation.constraints.NotNull; import javax.ws.rs.core.MultivaluedMap; import com.google.inject.Inject; @@ -19,11 +22,13 @@ import com.sun.jersey.api.client.filter.ClientFilter; import com.sun.jersey.core.util.MultivaluedMapImpl; -public class PersonWsClient extends BaseCrudClient implements PersonService { +public class PersonWsClient extends BaseClient implements PersonService, IdentifierService { private static final GenericType> PAGING_PERSON = new GenericType>() {}; private static final GenericType> LIST_PERSON_SUGGEST = new GenericType>() {}; + protected static final GenericType> LIST_IDENTIFIER = + new GenericType>() {}; @Inject protected PersonWsClient(@RegistryWs WebResource resource, @Nullable ClientFilter authFilter) { @@ -54,4 +59,19 @@ public List suggest(@Nullable String q) { queryParams.putSingle("q", q); return get(LIST_PERSON_SUGGEST, null, queryParams, null, "suggest"); } + + @Override + public int addIdentifier(@NotNull UUID key, @NotNull Identifier identifier) { + return post(Integer.class, identifier, String.valueOf(key), "identifier"); + } + + @Override + public void deleteIdentifier(@NotNull UUID key, int identifierKey) { + delete(String.valueOf(key), "identifier", String.valueOf(identifierKey)); + } + + @Override + public List listIdentifiers(@NotNull UUID key) { + return get(LIST_IDENTIFIER, null, null, (Pageable) null, String.valueOf(key), "identifier"); + } } diff --git a/registry-ws/src/main/java/org/gbif/registry/events/VarnishPurgeListener.java b/registry-ws/src/main/java/org/gbif/registry/events/VarnishPurgeListener.java index 699c06fc8f..b5f60d6339 100644 --- a/registry-ws/src/main/java/org/gbif/registry/events/VarnishPurgeListener.java +++ b/registry-ws/src/main/java/org/gbif/registry/events/VarnishPurgeListener.java @@ -224,14 +224,14 @@ private void cascadeDatasetChange(Dataset ... datasets) { purger.purge(path("dataset", d.getParentDatasetKey())); } } - purger.ban(String.format("dataset/%s/constituents", purger.anyKey(parentKeys))); + purger.ban(String.format("dataset/%s/constituents", VarnishPurger.anyKey(parentKeys))); // /installation/{d.installationKey}/dataset BAN - purger.ban(String.format("installation/%s/dataset", purger.anyKey(instKeys))); + purger.ban(String.format("installation/%s/dataset", VarnishPurger.anyKey(instKeys))); // /organization/{d.publishingOrganizationKey}/publishedDataset BAN // /organization/{d.installation.organizationKey}/hostedDataset BAN - purger.ban(String.format("organization/%s/(published|hosted)Dataset", purger.anyKey(orgKeys))); + purger.ban(String.format("organization/%s/(published|hosted)Dataset", VarnishPurger.anyKey(orgKeys))); // /node/{d.publishingOrganization.endorsingNodeKey}/dataset BAN - purger.ban(String.format("node/%s/dataset", purger.anyKey(nodeKeys))); + purger.ban(String.format("node/%s/dataset", VarnishPurger.anyKey(nodeKeys))); // /network/{any UUID}/constituents BAN purger.ban(String.format("network/.+/constituents")); } @@ -242,7 +242,7 @@ private void cascadeOrganizationChange(Organization ... orgs) { for (Organization o : orgs) { nodeKeys.add(o.getEndorsingNodeKey()); } - purger.ban(String.format("node/%s/organization", purger.anyKey(nodeKeys))); + purger.ban(String.format("node/%s/organization", VarnishPurger.anyKey(nodeKeys))); } private void cascadeInstallationChange(Installation ... installations) { @@ -251,7 +251,7 @@ private void cascadeInstallationChange(Installation ... installations) { for (Installation i : installations) { keys.add(i.getOrganizationKey()); } - purger.ban(String.format("organization/%s/installation", purger.anyKey(keys))); + purger.ban(String.format("organization/%s/installation", VarnishPurger.anyKey(keys))); // /node/{i.organization.endorsingNodeKey}/installation BAN Set nodekeys = new UUIDHashSet(); @@ -259,26 +259,28 @@ private void cascadeInstallationChange(Installation ... installations) { Organization o = organizationService.get(orgKey); nodekeys.add(o.getEndorsingNodeKey()); } - purger.ban(String.format("%node/%s/organization", purger.anyKey(nodekeys))); + purger.ban(String.format("%node/%s/organization", VarnishPurger.anyKey(nodekeys))); } private void cascadePersonChange(Person ... persons) { Set collectionKeys = new UUIDHashSet(); for (Person p : persons) { - List collections = collectionService.list(null, null, p.getKey(), null).getResults(); + List collections = + collectionService.list(null, null, p.getKey(), null, null, null).getResults(); collections.forEach(c -> collectionKeys.add(c.getKey())); } Set institutionKeys = new UUIDHashSet(); for (Person p : persons) { - List institutions = institutionService.list(null, p.getKey(), null).getResults(); + List institutions = + institutionService.list(null, p.getKey(), null, null, null).getResults(); institutions.forEach(i -> institutionKeys.add(i.getKey())); } // /collection/{collectionKey}/contact BAN - purger.ban(String.format("collection/%s/contact", purger.anyKey(collectionKeys))); + purger.ban(String.format("collection/%s/contact", VarnishPurger.anyKey(collectionKeys))); // /institution/{institutionKey}/contact BAN - purger.ban(String.format("institution/%s/contact", purger.anyKey(institutionKeys))); + purger.ban(String.format("institution/%s/contact", VarnishPurger.anyKey(institutionKeys))); } /** diff --git a/registry-ws/src/main/java/org/gbif/registry/persistence/WithMyBatis.java b/registry-ws/src/main/java/org/gbif/registry/persistence/WithMyBatis.java index a76fe9290d..3fa48faaf1 100644 --- a/registry-ws/src/main/java/org/gbif/registry/persistence/WithMyBatis.java +++ b/registry-ws/src/main/java/org/gbif/registry/persistence/WithMyBatis.java @@ -205,7 +205,7 @@ public static List listMachineTags(MachineTaggableMapper machineTagg return machineTaggableMapper.listMachineTags(targetEntityKey); } - public static PagingResponse listByMachineTag( + public static PagingResponse listByMachineTag( MachineTaggableMapper mapper, String namespace, @Nullable String name, @Nullable String value, @Nullable Pageable page ) { Preconditions.checkNotNull(page, "To list by machine tag you must supply a page"); diff --git a/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/BaseMapper.java b/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/BaseMapper.java new file mode 100644 index 0000000000..c644c48822 --- /dev/null +++ b/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/BaseMapper.java @@ -0,0 +1,25 @@ +package org.gbif.registry.persistence.mapper.collections; + +import org.gbif.api.model.registry.Identifiable; +import org.gbif.api.model.registry.MachineTaggable; +import org.gbif.api.model.registry.Taggable; +import org.gbif.registry.persistence.mapper.IdentifiableMapper; +import org.gbif.registry.persistence.mapper.MachineTaggableMapper; +import org.gbif.registry.persistence.mapper.TaggableMapper; + +import java.util.UUID; + +import org.apache.ibatis.annotations.Param; + +/** Generic mapper for CRUD operations. Initially implemented for collections. */ +public interface BaseMapper + extends TaggableMapper, IdentifiableMapper, MachineTaggableMapper { + + T get(@Param("key") UUID key); + + void create(T entity); + + void delete(@Param("key") UUID key); + + void update(T entity); +} diff --git a/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/CollectionMapper.java b/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/CollectionMapper.java index 2bd12ea76b..971e4d3d44 100644 --- a/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/CollectionMapper.java +++ b/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/CollectionMapper.java @@ -3,8 +3,6 @@ import org.gbif.api.model.collections.Collection; import org.gbif.api.model.common.paging.Pageable; import org.gbif.api.model.registry.search.collections.KeyCodeNameResult; -import org.gbif.registry.persistence.mapper.IdentifiableMapper; -import org.gbif.registry.persistence.mapper.TaggableMapper; import java.util.List; import java.util.UUID; @@ -12,20 +10,21 @@ import org.apache.ibatis.annotations.Param; -/** - * Mapper for {@link Collection} entities. - */ -public interface CollectionMapper - extends CrudMapper, ContactableMapper, TaggableMapper, IdentifiableMapper { +/** Mapper for {@link Collection} entities. */ +public interface CollectionMapper extends BaseMapper, ContactableMapper { List list(@Nullable @Param("institutionKey") UUID institutionKey, @Nullable @Param("contactKey") UUID contactKey, @Nullable @Param("query") String query, + @Nullable @Param("code") String code, + @Nullable @Param("name") String name, @Nullable @Param("page") Pageable page); long count(@Nullable @Param("institutionKey") UUID institutionKey, @Nullable @Param("contactKey") UUID contactKey, - @Nullable @Param("query") String query); + @Nullable @Param("query") String query, + @Nullable @Param("code") String code, + @Nullable @Param("name") String name); /** * A simple suggest by title service. diff --git a/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/CrudMapper.java b/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/CrudMapper.java deleted file mode 100644 index 262e6330d6..0000000000 --- a/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/CrudMapper.java +++ /dev/null @@ -1,17 +0,0 @@ -package org.gbif.registry.persistence.mapper.collections; - -import java.util.UUID; - -import org.apache.ibatis.annotations.Param; - -/** Generic mapper for CRUD operations. Initially implemented for collections. */ -public interface CrudMapper { - - T get(@Param("key") UUID key); - - void create(T entity); - - void delete(@Param("key") UUID key); - - void update(T entity); -} diff --git a/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/InstitutionMapper.java b/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/InstitutionMapper.java index 6c77f66377..d6b2ab582c 100644 --- a/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/InstitutionMapper.java +++ b/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/InstitutionMapper.java @@ -3,8 +3,6 @@ import org.gbif.api.model.collections.Institution; import org.gbif.api.model.common.paging.Pageable; import org.gbif.api.model.registry.search.collections.KeyCodeNameResult; -import org.gbif.registry.persistence.mapper.IdentifiableMapper; -import org.gbif.registry.persistence.mapper.TaggableMapper; import java.util.List; import java.util.UUID; @@ -12,16 +10,19 @@ import org.apache.ibatis.annotations.Param; -/** - * Mapper for {@link Institution} entities. - */ -public interface InstitutionMapper extends CrudMapper, ContactableMapper, TaggableMapper, IdentifiableMapper { +/** Mapper for {@link Institution} entities. */ +public interface InstitutionMapper extends BaseMapper, ContactableMapper { List list(@Nullable @Param("query") String query, @Nullable @Param("contactKey") UUID contactKey, + @Nullable @Param("code") String code, + @Nullable @Param("name") String name, @Nullable @Param("page") Pageable page); - long count(@Nullable @Param("query") String query, @Nullable @Param("contactKey") UUID contactKey); + long count(@Nullable @Param("query") String query, + @Nullable @Param("contactKey") UUID contactKey, + @Nullable @Param("code") String code, + @Nullable @Param("name") String name); /** * A simple suggest by title service. diff --git a/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/PersonMapper.java b/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/PersonMapper.java index d32e4c844f..7c4266f3bf 100644 --- a/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/PersonMapper.java +++ b/registry-ws/src/main/java/org/gbif/registry/persistence/mapper/collections/PersonMapper.java @@ -10,10 +10,8 @@ import org.apache.ibatis.annotations.Param; -/** - * Mapper for {@link Person} entities. - */ -public interface PersonMapper extends CrudMapper { +/** Mapper for {@link Person} entities. */ +public interface PersonMapper extends BaseMapper { List list(@Nullable @Param("institutionKey") UUID institutionKey, @Nullable @Param("collectionKey") UUID collectionKey, diff --git a/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/BaseCollectionEntityResource.java b/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/BaseCollectionEntityResource.java new file mode 100644 index 0000000000..1937eecd22 --- /dev/null +++ b/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/BaseCollectionEntityResource.java @@ -0,0 +1,379 @@ +package org.gbif.registry.ws.resources.collections; + +import org.gbif.api.model.collections.CollectionEntity; +import org.gbif.api.model.common.paging.Pageable; +import org.gbif.api.model.common.paging.PagingRequest; +import org.gbif.api.model.common.paging.PagingResponse; +import org.gbif.api.model.registry.*; +import org.gbif.api.service.collections.CrudService; +import org.gbif.api.service.registry.IdentifierService; +import org.gbif.api.service.registry.MachineTagService; +import org.gbif.api.service.registry.TagService; +import org.gbif.api.vocabulary.TagName; +import org.gbif.api.vocabulary.TagNamespace; +import org.gbif.registry.events.ChangedComponentEvent; +import org.gbif.registry.events.collections.DeleteCollectionEntityEvent; +import org.gbif.registry.persistence.WithMyBatis; +import org.gbif.registry.persistence.mapper.IdentifierMapper; +import org.gbif.registry.persistence.mapper.MachineTagMapper; +import org.gbif.registry.persistence.mapper.TagMapper; +import org.gbif.registry.persistence.mapper.collections.BaseMapper; +import org.gbif.registry.ws.guice.Trim; +import org.gbif.registry.ws.security.EditorAuthorizationService; +import org.gbif.ws.server.interceptor.NullToNotFound; + +import java.util.List; +import java.util.Optional; +import java.util.UUID; +import javax.annotation.Nullable; +import javax.annotation.security.RolesAllowed; +import javax.validation.Valid; +import javax.validation.constraints.NotNull; +import javax.validation.groups.Default; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import javax.ws.rs.core.SecurityContext; + +import com.google.common.eventbus.EventBus; +import org.apache.bval.guice.Validate; +import org.mybatis.guice.transactional.Transactional; + +import static org.gbif.registry.ws.security.UserRoles.ADMIN_ROLE; +import static org.gbif.registry.ws.security.UserRoles.GRSCICOLL_ADMIN_ROLE; +import static org.gbif.registry.ws.security.UserRoles.GRSCICOLL_EDITOR_ROLE; + +import static com.google.common.base.Preconditions.checkArgument; + +/** Base class to implement the CRUD methods of a {@link CollectionEntity}. */ +public abstract class BaseCollectionEntityResource< + T extends CollectionEntity & Taggable & Identifiable & MachineTaggable> + implements CrudService, TagService, IdentifierService, MachineTagService { + + private final BaseMapper baseMapper; + private final Class objectClass; + private final TagMapper tagMapper; + private final MachineTagMapper machineTagMapper; + private final IdentifierMapper identifierMapper; + private final EventBus eventBus; + final EditorAuthorizationService userAuthService; + + protected BaseCollectionEntityResource( + BaseMapper baseMapper, + TagMapper tagMapper, + MachineTagMapper machineTagMapper, + IdentifierMapper identifierMapper, + EditorAuthorizationService userAuthService, + EventBus eventBus, + Class objectClass) { + this.baseMapper = baseMapper; + this.tagMapper = tagMapper; + this.machineTagMapper = machineTagMapper; + this.identifierMapper = identifierMapper; + this.eventBus = eventBus; + this.objectClass = objectClass; + this.userAuthService = userAuthService; + } + + @POST + @Trim + @Validate + @Transactional + @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) + public UUID create(@NotNull T entity, @Context SecurityContext security) { + entity.setCreatedBy(security.getUserPrincipal().getName()); + entity.setModifiedBy(security.getUserPrincipal().getName()); + return create(entity); + } + + @DELETE + @Path("{key}") + @Validate + @Transactional + @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) + public void delete(@PathParam("key") @NotNull UUID key, @Context SecurityContext security) { + T entityToDelete = get(key); + entityToDelete.setModifiedBy(security.getUserPrincipal().getName()); + update(entityToDelete); + + delete(key); + } + + @Transactional + @Validate + @Override + public void delete(@NotNull UUID key) { + T objectToDelete = get(key); + baseMapper.delete(key); + eventBus.post(DeleteCollectionEntityEvent.newInstance(objectToDelete, objectClass)); + } + + @GET + @Path("{key}") + @Nullable + @NullToNotFound + @Validate(validateReturnedValue = true) + @Override + public T get(@PathParam("key") @NotNull UUID key) { + return baseMapper.get(key); + } + + @PUT + @Path("{key}") + @Validate + @Transactional + @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) + public void update( + @PathParam("key") @NotNull UUID key, + @NotNull @Trim T entity, + @Context SecurityContext security) { + checkArgument( + key.equals(entity.getKey()), "Provided entity must have the same key as the resource URL"); + entity.setModifiedBy(security.getUserPrincipal().getName()); + update(entity); + } + + @POST + @Path("{key}/identifier") + @Trim + @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) + public int addIdentifier( + @PathParam("key") @NotNull UUID entityKey, @NotNull Identifier identifier, @Context SecurityContext security + ) { + identifier.setCreatedBy(security.getUserPrincipal().getName()); + return addIdentifier(entityKey, identifier); + } + + @Validate(groups = {PrePersist.class, Default.class}) + @Override + public int addIdentifier(@NotNull UUID entityKey, @Valid @NotNull Identifier identifier) { + int identifierKey = WithMyBatis.addIdentifier(identifierMapper, baseMapper, entityKey, identifier); + eventBus.post(ChangedComponentEvent.newInstance(entityKey, objectClass, Identifier.class)); + return identifierKey; + } + + @DELETE + @Path("{key}/identifier/{identifierKey}") + @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) + @Transactional + @Override + public void deleteIdentifier( + @PathParam("key") @NotNull UUID entityKey, + @PathParam("identifierKey") int identifierKey + ) { + WithMyBatis.deleteIdentifier(baseMapper, entityKey, identifierKey); + eventBus.post(ChangedComponentEvent.newInstance(entityKey, objectClass, Identifier.class)); + } + + @GET + @Path("{key}/identifier") + @Nullable + @Validate(validateReturnedValue = true) + @Override + public List listIdentifiers(@PathParam("key") @NotNull UUID key) { + return WithMyBatis.listIdentifiers(baseMapper, key); + } + + @POST + @Path("{key}/tag") + @Trim + @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) + public int addTag(@PathParam("key") @NotNull UUID entityKey, @NotNull Tag tag, @Context SecurityContext security) { + tag.setCreatedBy(security.getUserPrincipal().getName()); + return addTag(entityKey, tag); + } + + @Override + public int addTag(@NotNull UUID key, @NotNull String value) { + Tag tag = new Tag(); + tag.setValue(value); + return addTag(key, tag); + } + + @Validate(groups = {PrePersist.class, Default.class}) + @Override + public int addTag(@NotNull UUID entityKey, @Valid @NotNull Tag tag) { + int tagKey = WithMyBatis.addTag(tagMapper, baseMapper, entityKey, tag); + eventBus.post(ChangedComponentEvent.newInstance(entityKey, objectClass, Tag.class)); + return tagKey; + } + + @DELETE + @Path("{key}/tag/{tagKey}") + @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) + @Transactional + @Override + public void deleteTag(@PathParam("key") @NotNull UUID entityKey, @PathParam("tagKey") int tagKey) { + WithMyBatis.deleteTag(baseMapper, entityKey, tagKey); + eventBus.post(ChangedComponentEvent.newInstance(entityKey, objectClass, Tag.class)); + } + + @GET + @Path("{key}/tag") + @Nullable + @Validate(validateReturnedValue = true) + @Override + public List listTags(@PathParam("key") @NotNull UUID key, @QueryParam("owner") @Nullable String owner) { + return WithMyBatis.listTags(baseMapper, key, owner); + } + + /** + * Adding most machineTags is restricted based on the namespace. + * + * For some tags, it is restricted based on the editing role as usual. + * + * @param targetEntityKey key of target entity to add MachineTag to + * @param machineTag MachineTag to add + * @param security SecurityContext (security related information) + * @return key of MachineTag created + */ + @POST + @Path("{key}/machineTag") + @Trim + @Transactional + public int addMachineTag(@PathParam("key") UUID targetEntityKey, @NotNull @Trim MachineTag machineTag, + @Context SecurityContext security) { + + if (security.isUserInRole(GRSCICOLL_ADMIN_ROLE) + || userAuthService.allowedToModifyNamespace(security.getUserPrincipal(), machineTag.getNamespace()) + || (security.isUserInRole(GRSCICOLL_EDITOR_ROLE) + && TagNamespace.GBIF_DEFAULT_TERM.getNamespace().equals(machineTag.getNamespace()) + && userAuthService.allowedToModifyDataset(security.getUserPrincipal(), targetEntityKey)) + ) { + machineTag.setCreatedBy(security.getUserPrincipal().getName()); + return addMachineTag(targetEntityKey, machineTag); + } else { + throw new WebApplicationException(Response.Status.FORBIDDEN); + } + } + + @Validate(groups = {PrePersist.class, Default.class}) + @Override + public int addMachineTag(UUID targetEntityKey, @Valid MachineTag machineTag) { + return WithMyBatis.addMachineTag(machineTagMapper, baseMapper, targetEntityKey, machineTag); + } + + @Override + public int addMachineTag(@NotNull UUID targetEntityKey, @NotNull String namespace, @NotNull String name, @NotNull String value) { + MachineTag machineTag = new MachineTag(); + machineTag.setNamespace(namespace); + machineTag.setName(name); + machineTag.setValue(value); + return addMachineTag(targetEntityKey, machineTag); + } + + @Override + public int addMachineTag(@NotNull UUID targetEntityKey, @NotNull TagName tagName, @NotNull String value) { + MachineTag machineTag = MachineTag.newInstance(tagName, value); + return addMachineTag(targetEntityKey, machineTag); + } + + /** + * The webservice method to delete a machine tag. + * Ensures that the caller is authorized to perform the action by looking at the namespace. + */ + @DELETE + @Path("{key}/machineTag/{machineTagKey: [0-9]+}") + @Consumes(MediaType.WILDCARD) + public void deleteMachineTag(@PathParam("key") UUID targetEntityKey, @PathParam("machineTagKey") int machineTagKey, + @Context SecurityContext security) { + + Optional optMachineTag = WithMyBatis.listMachineTags(baseMapper, targetEntityKey).stream() + .filter(m -> m.getKey() == machineTagKey).findFirst(); + + if (optMachineTag.isPresent()) { + MachineTag machineTag = optMachineTag.get(); + + if (security.isUserInRole(GRSCICOLL_ADMIN_ROLE) + || userAuthService.allowedToModifyNamespace(security.getUserPrincipal(), machineTag.getNamespace()) + || (security.isUserInRole(GRSCICOLL_EDITOR_ROLE) + && TagNamespace.GBIF_DEFAULT_TERM.getNamespace().equals(machineTag.getNamespace()) + && userAuthService.allowedToModifyDataset(security.getUserPrincipal(), targetEntityKey)) + ) { + deleteMachineTag(targetEntityKey, machineTagKey); + + } else { + throw new WebApplicationException(Response.Status.FORBIDDEN); + } + } else { + throw new WebApplicationException(Response.Status.NOT_FOUND); + } + } + + /** + * Deletes the MachineTag according to interface without security restrictions. + * + * @param targetEntityKey key of target entity to delete MachineTag from + * @param machineTagKey key of MachineTag to delete + */ + @Override + public void deleteMachineTag(@PathParam("key") UUID targetEntityKey, @PathParam("machineTagKey") int machineTagKey) { + WithMyBatis.deleteMachineTag(baseMapper, targetEntityKey, machineTagKey); + } + + /** + * The webservice method to delete all machine tag in a namespace. + * Ensures that the caller is authorized to perform the action by looking at the namespace. + */ + @DELETE + @Path("{key}/machineTag/{namespace}") + @Consumes(MediaType.WILDCARD) + public void deleteMachineTags(@PathParam("key") UUID targetEntityKey, @PathParam("namespace") String namespace, + @Context SecurityContext security) { + if (!security.isUserInRole(GRSCICOLL_ADMIN_ROLE) + && !userAuthService.allowedToModifyNamespace(security.getUserPrincipal(), namespace)) { + throw new WebApplicationException(Response.Status.FORBIDDEN); + } + deleteMachineTags(targetEntityKey, namespace); + } + + @Override + public void deleteMachineTags(@NotNull UUID targetEntityKey, @NotNull TagNamespace tagNamespace) { + deleteMachineTags(targetEntityKey, tagNamespace.getNamespace()); + } + + @Override + public void deleteMachineTags(@NotNull UUID targetEntityKey, @NotNull String namespace) { + WithMyBatis.deleteMachineTags(baseMapper, targetEntityKey, namespace, null); + } + + /** + * The webservice method to delete all machine tag of a particular name in a namespace. + * Ensures that the caller is authorized to perform the action by looking at the namespace. + */ + @DELETE + @Path("{key}/machineTag/{namespace}/{name}") + @Consumes(MediaType.WILDCARD) + public void deleteMachineTags(@PathParam("key") UUID targetEntityKey, @PathParam("namespace") String namespace, + @PathParam("name") String name, @Context SecurityContext security) { + if (!security.isUserInRole(GRSCICOLL_ADMIN_ROLE) + && !userAuthService.allowedToModifyNamespace(security.getUserPrincipal(), namespace)) { + throw new WebApplicationException(Response.Status.FORBIDDEN); + } + deleteMachineTags(targetEntityKey, namespace, name); + } + + @Override + public void deleteMachineTags(@NotNull UUID targetEntityKey, @NotNull TagName tagName) { + deleteMachineTags(targetEntityKey, tagName.getNamespace().getNamespace(), tagName.getName()); + } + + @Override + public void deleteMachineTags(@NotNull UUID targetEntityKey, @NotNull String namespace, @NotNull String name) { + WithMyBatis.deleteMachineTags(baseMapper, targetEntityKey, namespace, name); + } + + @GET + @Path("{key}/machineTag") + @Override + public List listMachineTags(@PathParam("key") UUID targetEntityKey) { + return WithMyBatis.listMachineTags(baseMapper, targetEntityKey); + } + + public PagingResponse listByMachineTag(String namespace, @Nullable String name, @Nullable String value, + @Nullable Pageable page) { + page = page == null ? new PagingRequest() : page; + return WithMyBatis.listByMachineTag(baseMapper, namespace, name, value, page); + } +} diff --git a/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/BaseCrudResource.java b/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/BaseCrudResource.java deleted file mode 100644 index ab93345250..0000000000 --- a/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/BaseCrudResource.java +++ /dev/null @@ -1,97 +0,0 @@ -package org.gbif.registry.ws.resources.collections; - -import org.gbif.api.model.collections.CollectionEntity; -import org.gbif.api.service.collections.CrudService; -import org.gbif.registry.events.collections.DeleteCollectionEntityEvent; -import org.gbif.registry.persistence.mapper.collections.CrudMapper; -import org.gbif.registry.ws.guice.Trim; -import org.gbif.ws.server.interceptor.NullToNotFound; - -import java.util.UUID; -import javax.annotation.Nullable; -import javax.annotation.security.RolesAllowed; -import javax.validation.constraints.NotNull; -import javax.ws.rs.*; -import javax.ws.rs.core.Context; -import javax.ws.rs.core.SecurityContext; - -import com.google.common.eventbus.EventBus; -import org.apache.bval.guice.Validate; -import org.mybatis.guice.transactional.Transactional; - -import static org.gbif.registry.ws.security.UserRoles.ADMIN_ROLE; -import static org.gbif.registry.ws.security.UserRoles.GRSCICOLL_ADMIN_ROLE; - -import static com.google.common.base.Preconditions.checkArgument; - -/** Base class to implement the CRUD methods of a {@link CollectionEntity}. */ -public abstract class BaseCrudResource implements CrudService { - - private final CrudMapper crudMapper; - private final EventBus eventBus; - private final Class objectClass; - - protected BaseCrudResource(CrudMapper crudMapper, EventBus eventBus, Class objectClass) { - this.crudMapper = crudMapper; - this.eventBus = eventBus; - this.objectClass = objectClass; - } - - @POST - @Trim - @Validate - @Transactional - @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) - public UUID create(@NotNull T entity, @Context SecurityContext security) { - entity.setCreatedBy(security.getUserPrincipal().getName()); - entity.setModifiedBy(security.getUserPrincipal().getName()); - return create(entity); - } - - @DELETE - @Path("{key}") - @Validate - @Transactional - @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) - public void delete(@PathParam("key") @NotNull UUID key, @Context SecurityContext security) { - T entityToDelete = get(key); - entityToDelete.setModifiedBy(security.getUserPrincipal().getName()); - update(entityToDelete); - - delete(key); - } - - @Transactional - @Validate - @Override - public void delete(@NotNull UUID key) { - T objectToDelete = get(key); - crudMapper.delete(key); - eventBus.post(DeleteCollectionEntityEvent.newInstance(objectToDelete, objectClass)); - } - - @GET - @Path("{key}") - @Nullable - @NullToNotFound - @Validate(validateReturnedValue = true) - @Override - public T get(@PathParam("key") @NotNull UUID key) { - return crudMapper.get(key); - } - - @PUT - @Path("{key}") - @Validate - @Transactional - @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) - public void update( - @PathParam("key") @NotNull UUID key, - @NotNull @Trim T entity, - @Context SecurityContext security) { - checkArgument( - key.equals(entity.getKey()), "Provided entity must have the same key as the resource URL"); - entity.setModifiedBy(security.getUserPrincipal().getName()); - update(entity); - } -} diff --git a/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/CollectionResource.java b/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/CollectionResource.java index d5c741ca1c..9efaca3935 100644 --- a/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/CollectionResource.java +++ b/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/CollectionResource.java @@ -7,18 +7,16 @@ import org.gbif.api.model.registry.search.collections.KeyCodeNameResult; import org.gbif.api.service.collections.CollectionService; import org.gbif.registry.persistence.mapper.IdentifierMapper; +import org.gbif.registry.persistence.mapper.MachineTagMapper; import org.gbif.registry.persistence.mapper.TagMapper; import org.gbif.registry.persistence.mapper.collections.AddressMapper; import org.gbif.registry.persistence.mapper.collections.CollectionMapper; +import org.gbif.registry.ws.security.EditorAuthorizationService; import java.util.List; import java.util.UUID; import javax.annotation.Nullable; -import javax.ws.rs.Consumes; -import javax.ws.rs.GET; -import javax.ws.rs.Path; -import javax.ws.rs.Produces; -import javax.ws.rs.QueryParam; +import javax.ws.rs.*; import javax.ws.rs.core.Context; import javax.ws.rs.core.MediaType; @@ -38,16 +36,30 @@ @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.APPLICATION_JSON) @Path(GRSCICOLL_PATH + "/collection") -public class CollectionResource extends BaseExtendableCollectionResource +public class CollectionResource extends ExtendedCollectionEntityResource implements CollectionService { private final CollectionMapper collectionMapper; @Inject - public CollectionResource(CollectionMapper collectionMapper, AddressMapper addressMapper, - IdentifierMapper identifierMapper,TagMapper tagMapper, EventBus eventBus) { - super(collectionMapper, addressMapper, collectionMapper, tagMapper, collectionMapper, identifierMapper, collectionMapper, - eventBus, Collection.class); + public CollectionResource( + CollectionMapper collectionMapper, + AddressMapper addressMapper, + IdentifierMapper identifierMapper, + TagMapper tagMapper, + MachineTagMapper machineTagMapper, + EventBus eventBus, + EditorAuthorizationService userAuthService) { + super( + collectionMapper, + addressMapper, + tagMapper, + identifierMapper, + collectionMapper, + machineTagMapper, + eventBus, + Collection.class, + userAuthService); this.collectionMapper = collectionMapper; } @@ -55,11 +67,13 @@ public CollectionResource(CollectionMapper collectionMapper, AddressMapper addre public PagingResponse list(@Nullable @QueryParam("q") String query, @Nullable @QueryParam("institution") UUID institutionKey, @Nullable @QueryParam("contact") UUID contactKey, + @Nullable @QueryParam("code") String code, + @Nullable @QueryParam("name") String name, @Nullable @Context Pageable page) { page = page == null ? new PagingRequest() : page; query = query != null ? Strings.emptyToNull(CharMatcher.WHITESPACE.trimFrom(query)) : query; - long total = collectionMapper.count(institutionKey, contactKey, query); - return new PagingResponse<>(page, total, collectionMapper.list(institutionKey, contactKey, query, page)); + long total = collectionMapper.count(institutionKey, contactKey, code, name, query); + return new PagingResponse<>(page, total, collectionMapper.list(institutionKey, contactKey, query, code, name, page)); } @GET diff --git a/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/BaseExtendableCollectionResource.java b/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/ExtendedCollectionEntityResource.java similarity index 58% rename from registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/BaseExtendableCollectionResource.java rename to registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/ExtendedCollectionEntityResource.java index ae99eaaf30..01ebcc5ee6 100644 --- a/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/BaseExtendableCollectionResource.java +++ b/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/ExtendedCollectionEntityResource.java @@ -6,20 +6,14 @@ import org.gbif.api.model.collections.Person; import org.gbif.api.model.registry.*; import org.gbif.api.service.collections.ContactService; -import org.gbif.api.service.registry.IdentifierService; -import org.gbif.api.service.registry.TagService; import org.gbif.registry.events.ChangedComponentEvent; import org.gbif.registry.events.collections.CreateCollectionEntityEvent; import org.gbif.registry.events.collections.UpdateCollectionEntityEvent; -import org.gbif.registry.persistence.WithMyBatis; -import org.gbif.registry.persistence.mapper.IdentifiableMapper; -import org.gbif.registry.persistence.mapper.IdentifierMapper; -import org.gbif.registry.persistence.mapper.TagMapper; -import org.gbif.registry.persistence.mapper.TaggableMapper; +import org.gbif.registry.persistence.mapper.*; import org.gbif.registry.persistence.mapper.collections.AddressMapper; import org.gbif.registry.persistence.mapper.collections.ContactableMapper; -import org.gbif.registry.persistence.mapper.collections.CrudMapper; -import org.gbif.registry.ws.guice.Trim; +import org.gbif.registry.persistence.mapper.collections.BaseMapper; +import org.gbif.registry.ws.security.EditorAuthorizationService; import java.util.List; import java.util.UUID; @@ -29,17 +23,15 @@ import javax.validation.constraints.NotNull; import javax.validation.groups.Default; import javax.ws.rs.*; -import javax.ws.rs.core.Context; import javax.ws.rs.core.MediaType; import javax.ws.rs.core.Response; -import javax.ws.rs.core.SecurityContext; import com.google.common.eventbus.EventBus; import org.apache.bval.guice.Validate; import org.mybatis.guice.transactional.Transactional; -import static org.gbif.registry.ws.security.UserRoles.GRSCICOLL_ADMIN_ROLE; import static org.gbif.registry.ws.security.UserRoles.ADMIN_ROLE; +import static org.gbif.registry.ws.security.UserRoles.GRSCICOLL_ADMIN_ROLE; import static com.google.common.base.Preconditions.checkArgument; @@ -47,33 +39,45 @@ * Base class to implement the main methods of {@link CollectionEntity} that are also @link * * Taggable}, {@link Identifiable} and {@link Contactable}. * * * - *

It inherits from {@link BaseCrudResource} to test the CRUD operations. + *

It inherits from {@link BaseCollectionEntityResource} to test the CRUD operations. */ -public abstract class BaseExtendableCollectionResource - extends BaseCrudResource implements TagService, IdentifierService, ContactService { +public abstract class ExtendedCollectionEntityResource< + T extends CollectionEntity & Taggable & Identifiable & MachineTaggable & Contactable> + extends BaseCollectionEntityResource implements ContactService { - private final CrudMapper crudMapper; + private final BaseMapper baseMapper; private final AddressMapper addressMapper; - private final TaggableMapper taggableMapper; + private final ContactableMapper contactableMapper; private final TagMapper tagMapper; - private final IdentifiableMapper identifiableMapper; + private final MachineTagMapper machineTagMapper; private final IdentifierMapper identifierMapper; - private final ContactableMapper contactableMapper; private final EventBus eventBus; private final Class objectClass; - protected BaseExtendableCollectionResource(CrudMapper crudMapper, AddressMapper addressMapper, - TaggableMapper taggableMapper, TagMapper tagMapper, - IdentifiableMapper identifiableMapper, IdentifierMapper identifierMapper, - ContactableMapper contactableMapper, EventBus eventBus, Class objectClass) { - super(crudMapper, eventBus, objectClass); - this.crudMapper = crudMapper; + protected ExtendedCollectionEntityResource( + BaseMapper baseMapper, + AddressMapper addressMapper, + TagMapper tagMapper, + IdentifierMapper identifierMapper, + ContactableMapper contactableMapper, + MachineTagMapper machineTagMapper, + EventBus eventBus, + Class objectClass, + EditorAuthorizationService userAuthService) { + super( + baseMapper, + tagMapper, + machineTagMapper, + identifierMapper, + userAuthService, + eventBus, + objectClass); + this.baseMapper = baseMapper; this.addressMapper = addressMapper; - this.taggableMapper = taggableMapper; + this.contactableMapper = contactableMapper; this.tagMapper = tagMapper; - this.identifiableMapper = identifiableMapper; + this.machineTagMapper = machineTagMapper; this.identifierMapper = identifierMapper; - this.contactableMapper = contactableMapper; this.eventBus = eventBus; this.objectClass = objectClass; } @@ -95,14 +99,23 @@ public UUID create(@Valid @NotNull T entity) { } entity.setKey(UUID.randomUUID()); - crudMapper.create(entity); + baseMapper.create(entity); + + if (!entity.getMachineTags().isEmpty()) { + for (MachineTag machineTag : entity.getMachineTags()) { + checkArgument(machineTag.getKey() == null, "Unable to create a machine tag which already has a key"); + machineTag.setCreatedBy(entity.getCreatedBy()); + machineTagMapper.createMachineTag(machineTag); + baseMapper.addMachineTag(entity.getKey(), machineTag.getKey()); + } + } if (!entity.getTags().isEmpty()) { for (Tag tag : entity.getTags()) { checkArgument(tag.getKey() == null, "Unable to create a tag which already has a key"); tag.setCreatedBy(entity.getCreatedBy()); tagMapper.createTag(tag); - taggableMapper.addTag(entity.getKey(), tag.getKey()); + baseMapper.addTag(entity.getKey(), tag.getKey()); } } @@ -111,7 +124,7 @@ public UUID create(@Valid @NotNull T entity) { checkArgument(identifier.getKey() == null, "Unable to create an identifier which already has a key"); identifier.setCreatedBy(entity.getCreatedBy()); identifierMapper.createIdentifier(identifier); - identifiableMapper.addIdentifier(entity.getKey(), identifier.getKey()); + baseMapper.addIdentifier(entity.getKey(), identifier.getKey()); } } @@ -142,7 +155,7 @@ public void update(@Valid @NotNull T entity) { updateAddress(entity.getAddress(), entityOld.getAddress()); // update entity - crudMapper.update(entity); + baseMapper.update(entity); // check if we can delete the mailing address if (entity.getMailingAddress() == null && entityOld.getMailingAddress() != null) { @@ -210,87 +223,4 @@ public List listContacts(@PathParam("key") @NotNull UUID key) { return contactableMapper.listContacts(key); } - @POST - @Path("{key}/identifier") - @Trim - @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) - public int addIdentifier( - @PathParam("key") @NotNull UUID entityKey, @NotNull Identifier identifier, @Context SecurityContext security - ) { - identifier.setCreatedBy(security.getUserPrincipal().getName()); - return addIdentifier(entityKey, identifier); - } - - @Validate(groups = {PrePersist.class, Default.class}) - @Override - public int addIdentifier(@NotNull UUID entityKey, @Valid @NotNull Identifier identifier) { - int identifierKey = WithMyBatis.addIdentifier(identifierMapper, identifiableMapper, entityKey, identifier); - eventBus.post(ChangedComponentEvent.newInstance(entityKey, objectClass, Identifier.class)); - return identifierKey; - } - - @DELETE - @Path("{key}/identifier/{identifierKey}") - @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) - @Transactional - @Override - public void deleteIdentifier( - @PathParam("key") @NotNull UUID entityKey, - @PathParam("identifierKey") int identifierKey - ) { - WithMyBatis.deleteIdentifier(identifiableMapper, entityKey, identifierKey); - eventBus.post(ChangedComponentEvent.newInstance(entityKey, objectClass, Identifier.class)); - } - - @GET - @Path("{key}/identifier") - @Nullable - @Validate(validateReturnedValue = true) - @Override - public List listIdentifiers(@PathParam("key") @NotNull UUID key) { - return WithMyBatis.listIdentifiers(identifiableMapper, key); - } - - @POST - @Path("{key}/tag") - @Trim - @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) - public int addTag(@PathParam("key") @NotNull UUID entityKey, @NotNull Tag tag, @Context SecurityContext security) { - tag.setCreatedBy(security.getUserPrincipal().getName()); - return addTag(entityKey, tag); - } - - @Override - public int addTag(@NotNull UUID key, @NotNull String value) { - Tag tag = new Tag(); - tag.setValue(value); - return addTag(key, tag); - } - - @Validate(groups = {PrePersist.class, Default.class}) - @Override - public int addTag(@NotNull UUID entityKey, @Valid @NotNull Tag tag) { - int tagKey = WithMyBatis.addTag(tagMapper, taggableMapper, entityKey, tag); - eventBus.post(ChangedComponentEvent.newInstance(entityKey, objectClass, Tag.class)); - return tagKey; - } - - @DELETE - @Path("{key}/tag/{tagKey}") - @RolesAllowed({ADMIN_ROLE, GRSCICOLL_ADMIN_ROLE}) - @Transactional - @Override - public void deleteTag(@PathParam("key") @NotNull UUID entityKey, @PathParam("tagKey") int tagKey) { - WithMyBatis.deleteTag(taggableMapper, entityKey, tagKey); - eventBus.post(ChangedComponentEvent.newInstance(entityKey, objectClass, Tag.class)); - } - - @GET - @Path("{key}/tag") - @Nullable - @Validate(validateReturnedValue = true) - @Override - public List listTags(@PathParam("key") @NotNull UUID key, @QueryParam("owner") @Nullable String owner) { - return WithMyBatis.listTags(taggableMapper, key, owner); - } } diff --git a/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/InstitutionResource.java b/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/InstitutionResource.java index 98286c289f..f9750db622 100644 --- a/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/InstitutionResource.java +++ b/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/InstitutionResource.java @@ -7,18 +7,16 @@ import org.gbif.api.model.registry.search.collections.KeyCodeNameResult; import org.gbif.api.service.collections.InstitutionService; import org.gbif.registry.persistence.mapper.IdentifierMapper; +import org.gbif.registry.persistence.mapper.MachineTagMapper; import org.gbif.registry.persistence.mapper.TagMapper; import org.gbif.registry.persistence.mapper.collections.AddressMapper; import org.gbif.registry.persistence.mapper.collections.InstitutionMapper; +import org.gbif.registry.ws.security.EditorAuthorizationService; import java.util.List; import java.util.UUID; import javax.annotation.Nullable; -import javax.ws.rs.Consumes; -import javax.ws.rs.GET; -import javax.ws.rs.Path; -import javax.ws.rs.Produces; -import javax.ws.rs.QueryParam; +import javax.ws.rs.*; import javax.ws.rs.core.Context; import javax.ws.rs.core.MediaType; @@ -38,27 +36,43 @@ @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.APPLICATION_JSON) @Path(GRSCICOLL_PATH + "/institution") -public class InstitutionResource extends BaseExtendableCollectionResource +public class InstitutionResource extends ExtendedCollectionEntityResource implements InstitutionService { private final InstitutionMapper institutionMapper; @Inject - public InstitutionResource(InstitutionMapper institutionMapper, AddressMapper addressMapper, IdentifierMapper identifierMapper, - TagMapper tagMapper, EventBus eventBus) { - super(institutionMapper, addressMapper, institutionMapper, tagMapper, institutionMapper, identifierMapper, institutionMapper, - eventBus, Institution.class); + public InstitutionResource( + InstitutionMapper institutionMapper, + AddressMapper addressMapper, + IdentifierMapper identifierMapper, + TagMapper tagMapper, + MachineTagMapper machineTagMapper, + EditorAuthorizationService userAuthService, + EventBus eventBus) { + super( + institutionMapper, + addressMapper, + tagMapper, + identifierMapper, + institutionMapper, + machineTagMapper, + eventBus, + Institution.class, + userAuthService); this.institutionMapper = institutionMapper; } @GET public PagingResponse list(@Nullable @QueryParam("q") String query, @Nullable @QueryParam("contact") UUID contactKey, + @Nullable @QueryParam("code") String code, + @Nullable @QueryParam("name") String name, @Context Pageable page) { page = page == null ? new PagingRequest() : page; query = query != null ? Strings.emptyToNull(CharMatcher.WHITESPACE.trimFrom(query)) : query; - long total = institutionMapper.count(query, contactKey); - return new PagingResponse<>(page, total, institutionMapper.list(query, contactKey, page)); + long total = institutionMapper.count(query, contactKey, code, name); + return new PagingResponse<>(page, total, institutionMapper.list(query, contactKey, code, name, page)); } @GET diff --git a/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/PersonResource.java b/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/PersonResource.java index c45ed9dc5a..9d80f8e5b9 100644 --- a/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/PersonResource.java +++ b/registry-ws/src/main/java/org/gbif/registry/ws/resources/collections/PersonResource.java @@ -4,13 +4,20 @@ import org.gbif.api.model.common.paging.Pageable; import org.gbif.api.model.common.paging.PagingRequest; import org.gbif.api.model.common.paging.PagingResponse; +import org.gbif.api.model.registry.Identifier; +import org.gbif.api.model.registry.MachineTag; import org.gbif.api.model.registry.PrePersist; +import org.gbif.api.model.registry.Tag; import org.gbif.api.model.registry.search.collections.PersonSuggestResult; import org.gbif.api.service.collections.PersonService; import org.gbif.registry.events.collections.CreateCollectionEntityEvent; import org.gbif.registry.events.collections.UpdateCollectionEntityEvent; +import org.gbif.registry.persistence.mapper.IdentifierMapper; +import org.gbif.registry.persistence.mapper.MachineTagMapper; +import org.gbif.registry.persistence.mapper.TagMapper; import org.gbif.registry.persistence.mapper.collections.AddressMapper; import org.gbif.registry.persistence.mapper.collections.PersonMapper; +import org.gbif.registry.ws.security.EditorAuthorizationService; import java.util.List; import java.util.UUID; @@ -18,11 +25,7 @@ import javax.validation.Valid; import javax.validation.constraints.NotNull; import javax.validation.groups.Default; -import javax.ws.rs.Consumes; -import javax.ws.rs.GET; -import javax.ws.rs.Path; -import javax.ws.rs.Produces; -import javax.ws.rs.QueryParam; +import javax.ws.rs.*; import javax.ws.rs.core.Context; import javax.ws.rs.core.MediaType; @@ -46,17 +49,30 @@ @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.APPLICATION_JSON) @Path(GRSCICOLL_PATH + "/person") -public class PersonResource extends BaseCrudResource implements PersonService { +public class PersonResource extends BaseCollectionEntityResource implements PersonService { private final PersonMapper personMapper; private final AddressMapper addressMapper; + private final IdentifierMapper identifierMapper; + private final TagMapper tagMapper; + private final MachineTagMapper machineTagMapper; private final EventBus eventBus; @Inject - public PersonResource(PersonMapper personMapper, AddressMapper addressMapper, EventBus eventBus) { - super(personMapper, eventBus, Person.class); + public PersonResource( + PersonMapper personMapper, + AddressMapper addressMapper, + IdentifierMapper identifierMapper, + TagMapper tagMapper, + MachineTagMapper machineTagMapper, + EventBus eventBus, + EditorAuthorizationService userAuthService) { + super(personMapper, tagMapper, machineTagMapper, identifierMapper, userAuthService, eventBus, Person.class); this.personMapper = personMapper; this.addressMapper = addressMapper; + this.identifierMapper = identifierMapper; + this.tagMapper = tagMapper; + this.machineTagMapper = machineTagMapper; this.eventBus = eventBus; } @@ -93,6 +109,34 @@ public UUID create(@Valid @NotNull Person person) { person.setKey(UUID.randomUUID()); personMapper.create(person); + if (!person.getMachineTags().isEmpty()) { + for (MachineTag machineTag : person.getMachineTags()) { + checkArgument(machineTag.getKey() == null, "Unable to create a machine tag which already has a key"); + machineTag.setCreatedBy(person.getCreatedBy()); + machineTagMapper.createMachineTag(machineTag); + personMapper.addMachineTag(person.getKey(), machineTag.getKey()); + } + } + + if (!person.getTags().isEmpty()) { + for (Tag tag : person.getTags()) { + checkArgument(tag.getKey() == null, "Unable to create a tag which already has a key"); + tag.setCreatedBy(person.getCreatedBy()); + tagMapper.createTag(tag); + personMapper.addTag(person.getKey(), tag.getKey()); + } + } + + if (!person.getIdentifiers().isEmpty()) { + for (Identifier identifier : person.getIdentifiers()) { + checkArgument( + identifier.getKey() == null, "Unable to create an identifier which already has a key"); + identifier.setCreatedBy(person.getCreatedBy()); + identifierMapper.createIdentifier(identifier); + personMapper.addIdentifier(person.getKey(), identifier.getKey()); + } + } + eventBus.post(CreateCollectionEntityEvent.newInstance(person, Person.class)); return person.getKey(); } diff --git a/registry-ws/src/main/resources/org/gbif/registry/persistence/mapper/collections/CollectionMapper.xml b/registry-ws/src/main/resources/org/gbif/registry/persistence/mapper/collections/CollectionMapper.xml index 77100f33c2..d68b408f65 100644 --- a/registry-ws/src/main/resources/org/gbif/registry/persistence/mapper/collections/CollectionMapper.xml +++ b/registry-ws/src/main/resources/org/gbif/registry/persistence/mapper/collections/CollectionMapper.xml @@ -5,6 +5,8 @@ + + @@ -12,18 +14,19 @@ + - key, code, name, description, content_type, active, personal_collection, doi, homepage, catalog_url, api_url, + key, code, name, description, content_type, active, personal_collection, doi, email, phone, homepage, catalog_url, api_url, preservation_type, accession_status, institution_key, mailing_address_key, address_key, - created_by, modified_by, created, modified + created_by, modified_by, created, modified, index_herbariorum_record, number_specimens - c.key, c.code, c.name, c.description, c.content_type, c.active, c.personal_collection, c.doi, c.homepage, + c.key, c.code, c.name, c.description, c.content_type, c.active, c.personal_collection, c.doi, c.email, c.phone, c.homepage, c.catalog_url, c.api_url, c.preservation_type, c.accession_status, c.institution_key, c.mailing_address_key, - c.address_key, c.created_by, c.modified_by, c.created, c.modified, c.deleted + c.address_key, c.created_by, c.modified_by, c.created, c.modified, c.deleted, c.index_herbariorum_record, c.number_specimens @@ -35,6 +38,8 @@ #{active,jdbcType=BOOLEAN}, #{personalCollection,jdbcType=BOOLEAN}, #{doi,jdbcType=VARCHAR}, + #{email,jdbcType=ARRAY,typeHandler=StringArrayTypeHandler}, + #{phone,jdbcType=ARRAY,typeHandler=StringArrayTypeHandler}, #{homepage,jdbcType=VARCHAR}, #{catalogUrl,jdbcType=VARCHAR}, #{apiUrl,jdbcType=VARCHAR}, @@ -46,7 +51,9 @@ #{createdBy,jdbcType=VARCHAR}, #{modifiedBy,jdbcType=VARCHAR}, now(), - now() + now(), + #{indexHerbariorumRecord,jdbcType=BOOLEAN}, + #{numberSpecimens,jdbcType=INTEGER} @@ -57,6 +64,8 @@ active = #{active,jdbcType=BOOLEAN}, personal_collection = #{personalCollection,jdbcType=BOOLEAN}, doi = #{doi,jdbcType=VARCHAR}, + email = #{email,jdbcType=ARRAY,typeHandler=StringArrayTypeHandler}, + phone = #{phone,jdbcType=ARRAY,typeHandler=StringArrayTypeHandler}, homepage = #{homepage,jdbcType=VARCHAR}, catalog_url = #{catalogUrl,jdbcType=VARCHAR}, api_url = #{apiUrl,jdbcType=VARCHAR}, @@ -67,7 +76,9 @@ address_key = #{address.key,jdbcType=INTEGER}, modified_by = #{modifiedBy,jdbcType=VARCHAR}, modified = now(), - deleted = null + deleted = null, + index_herbariorum_record = #{indexHerbariorumRecord,jdbcType=BOOLEAN}, + number_specimens = #{numberSpecimens,jdbcType=INTEGER} @@ -122,6 +133,12 @@ AND ccp.collection_person_key = #{contactKey,jdbcType=OTHER} + + AND c.code = #{code,jdbcType=VARCHAR} + + + AND c.name = #{name,jdbcType=VARCHAR} + ORDER BY ts_rank_cd(c.fulltext_search, query) DESC, c.created DESC, c.key LIMIT #{page.limit} OFFSET #{page.offset} @@ -141,6 +158,12 @@ AND c.institution_key = #{institutionKey,jdbcType=OTHER} + + AND c.code = #{code,jdbcType=VARCHAR} + + + AND c.name = #{name,jdbcType=VARCHAR} + AND ccp.collection_person_key = #{contactKey,jdbcType=OTHER} @@ -206,6 +229,7 @@ tag_key = tag.key AND collection_key = #{targetEntityKey,jdbcType=OTHER} AND tag_key = #{tagKey,jdbcType=INTEGER} + @@ -237,6 +261,75 @@ INNER JOIN identifier i ON i."key" = ci.identifier_key WHERE regexp_replace(i.identifier, 'http://', '') = regexp_replace(#{identifier,jdbcType=VARCHAR}, 'http://', ''); + + + + + INSERT INTO collection_machine_tag(collection_key,machine_tag_key) + VALUES( + #{targetEntityKey,jdbcType=OTHER}, + #{machineTagKey,jdbcType=INTEGER} + ) + + + + + + + DELETE FROM machine_tag USING collection_machine_tag + WHERE + machine_tag_key = machine_tag.key AND + collection_key = #{targetEntityKey,jdbcType=OTHER} AND machine_tag_key = #{machineTagKey,jdbcType=INTEGER} + + + + DELETE FROM machine_tag USING collection_machine_tag + WHERE + machine_tag_key = machine_tag.key AND + collection_key = #{targetEntityKey,jdbcType=OTHER} AND + namespace = #{namespace} + + AND name = #{name} + + + + + + + + + + + + INSERT INTO institution_machine_tag(institution_key,machine_tag_key) + VALUES( + #{targetEntityKey,jdbcType=OTHER}, + #{machineTagKey,jdbcType=INTEGER} + ) + + + + + + + DELETE FROM machine_tag USING institution_machine_tag + WHERE + machine_tag_key = machine_tag.key AND + institution_key = #{targetEntityKey,jdbcType=OTHER} AND machine_tag_key = #{machineTagKey,jdbcType=INTEGER} + + + + DELETE FROM machine_tag USING institution_machine_tag + WHERE + machine_tag_key = machine_tag.key AND + institution_key = #{targetEntityKey,jdbcType=OTHER} AND + namespace = #{namespace} + + AND name = #{name} + + + + + + + + + + INSERT INTO collection_person_identifier(collection_person_key,identifier_key) + VALUES( + #{targetEntityKey,jdbcType=OTHER}, + #{identifierKey,jdbcType=INTEGER} + ) + + + + + + + DELETE FROM identifier USING collection_person_identifier + WHERE + identifier_key = identifier.key AND + collection_person_key = #{targetEntityKey,jdbcType=OTHER} AND identifier_key = #{identifierKey,jdbcType=INTEGER} + + + + + + + + INSERT INTO collection_person_tag(collection_person_key,tag_key) + VALUES( + #{targetEntityKey,jdbcType=OTHER}, + #{tagKey,jdbcType=INTEGER} + ) + + + + + + + DELETE FROM tag USING collection_person_tag + WHERE + tag_key = tag.key AND + collection_person_key = #{targetEntityKey,jdbcType=OTHER} AND tag_key = #{tagKey,jdbcType=INTEGER} + + + + + + INSERT INTO collection_person_machine_tag(collection_person_key,machine_tag_key) + VALUES( + #{targetEntityKey,jdbcType=OTHER}, + #{machineTagKey,jdbcType=INTEGER} + ) + + + + + + + DELETE FROM machine_tag USING collection_person_machine_tag + WHERE + machine_tag_key = machine_tag.key AND + collection_person_key = #{targetEntityKey,jdbcType=OTHER} AND machine_tag_key = #{machineTagKey,jdbcType=INTEGER} + + + + DELETE FROM machine_tag USING collection_person_machine_tag + WHERE + machine_tag_key = machine_tag.key AND + collection_person_key = #{targetEntityKey,jdbcType=OTHER} AND + namespace = #{namespace} + + AND name = #{name} + + + + + + + +