Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable and refine the aggregation subsystem based importer integration test after installing Hive on CI test cluster #1360

Open
marekhorst opened this issue Jun 29, 2022 · 0 comments

Comments

@marekhorst
Copy link
Member

marekhorst commented Jun 29, 2022

After integrating the https://github.com/openaire/iis/blob/marekhorst_1298_refactor_iis_content_importer branch with #1361 pull request we end up with one eu.dnetlib.iis.wf.importer.content.WorkflowTest#testImportContentUrlWorkflowWithHiveBasedAggregationSubsystemAsBackend() test as disabled because it can't be run on CI test cluster due to missing Hive service. It works on the main IIS CDH5 cluster though.

Apart from enabling the test once the hive service is installed on CI test cluster one another issue should be addressed related to the workflow.xml definition (link currently pointing to the branch but it will be merged with IIS master branch eventually) which should be modified, along with the build system configuration in iis-wf/pom.xml file, to avoid setting jdbc-url and hive_metastore_uris to explicit values bound to a particular environment.

The oozie workflow, by design, should remain environment-agnostic and it should be possible to rely on specific placeholders predefined for a given environment in ~/.iis/integration-test.properties file stored in an environment just as it is already done for various env specific properties such as sparkExecutorMemory or oozieServiceLoc.

Currently defined values which are currently pointing to the main hadoop cluster and which should be addressed:

  • jdbc-url: jdbc:hive2://iis-cdh5-test-m3.ocean.icm.edu.pl:10000
  • hive_metastore_uris: thrift://iis-cdh5-test-m3.ocean.icm.edu.pl:9083

which should be placed among the connection properties dedicated for each individual properly at: https://git.icm.edu.pl/openaire/iis-deployment/-/tree/master/connection-properties

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant