Skip to content

Latest commit

 

History

History
760 lines (595 loc) · 41.3 KB

development-environment.asciidoc

File metadata and controls

760 lines (595 loc) · 41.3 KB
Raw

Development Environment

TinkerPop is fairly large body of code spread across many modules and covering multiple programming languages. Despite this complexity, it remains relatively straightforward a project to build. This following subsections explain how to configure a development environment for TinkerPop.

conspiracy

System Configuration

At a minimum, development of TinkerPop requires Java 8 but it is preferable to use Java 11 cross-compiled to Java 8 (the cross-compilation happens automatically as part of the build). Starting with 3.7.0, you can also build with Java 17. Note, however, that there are some issues with deep reflection so there are --add-opens JVM options included in the pom files to enable this to work. Maven (requiring a minimum of Maven 3.5.3+) is used as the common build system, which even controls the builds of non-JVM GLVs such as gremlin-python. Java and Maven are described as a "minimum" for a development environment, because they will only build JVM portions of TinkerPop and many integration tests will not fire with this simple setup. It is possible to get a clean and successful build with this minimum, but it will not be possible to build non-JVM aspects of the project and those will go untested.

To gain the ability to execute all aspects of the TinkerPop build system, other environmental configurations must be established. Those prerequisites are defined in the following subsections.

As of TinkerPop 3.5.5, environment configuration for Gremlin Language Variants can be optional, as Docker becomes a system requirement to build and test GLVs inside of Maven. Please make sure Docker is installed and running on your system. You will need to install both Docker Engine and Docker Compose, which are included in Docker Desktop.

Important
 Use Java 11 for documentation generation with bin/process-docs.sh and for other build features outside of the basic mvn clean install sort of function.
Important
 For those who intend to offer a contribution, building with a minimal configuration may not be sufficient when submitting a pull request. Consider setting up the full environment.
Tip
 Consider using SDKMAN! to manage Java and Maven versions and environments.
Note
 For those using Windows, efforts have been made to keep the build OS independent, but, in practice, it is likely that TinkerPop’s build system will only allow for a minimum build at best.

Refer to Building On Windows section for more details.

Groovy Environment

Groovy is not used in the standard build, but when generating documentation it does require the loading of a Gremlin Console instance. The Gremlin Console is Groovy-based and the documentation bootstrapping loads TinkerPop plugins which requires proper configuration of Graph/Ivy dependency loaders as described in the Gremlin Applications Section of the Reference Documentation.

The base configuration described in that link may need to be modified if there is a desire to work with the Gremlin Console (for documentation generation or just general testing) in a way that utilizes SNAPSHOT releases in the Apache Snapshots Repository. In that case, the grapeConfig.xml will need to include a resolver for that repository and the basic Ivy configuration will look as follows:

<ivysettings>
  <settings defaultResolver="downloadGrapes"/>
  <resolvers>
    <chain name="downloadGrapes" returnFirst="true">
      <filesystem name="cachedGrapes">
        <ivy pattern="${user.home}/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml"/>
        <artifact pattern="${user.home}/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision](-[classifier]).[ext]"/>
      </filesystem>
      <ibiblio name="localm2" root="${user.home.url}/.m2/repository/" checkmodified="true" changingPattern=".*" changingMatcher="regexp" m2compatible="true"/>
      <ibiblio name="jcenter" root="https://jcenter.bintray.com/" m2compatible="true"/>
      <ibiblio name="ibiblio" m2compatible="true"/>
      <ibiblio name="apache-snapshots" root="http://repository.apache.org/snapshots/" m2compatible="true"/>
    </chain>
  </resolvers>
</ivysettings>

The above configuration is just a modification of the default. Perhaps the most lean common configuration might just be:

<ivysettings>
  <settings defaultResolver="downloadGrapes"/>
  <resolvers>
    <chain name="downloadGrapes">
      <ibiblio name="local" root="file:${user.home}/.m2/repository/" m2compatible="true"/>
      <ibiblio name="central" root="https://repo1.maven.org/maven2/" m2compatible="true"/>
    </chain>
  </resolvers>
</ivysettings>

In the above case, the configuration largely relies on the standard Maven builds to create a well cached .m2 directory. Under typical development circumstances, SNAPSHOT will find themselves deployed there locally and that is all that will be required for Grape to do its work.

As a final word, it is important to take note of the order used for these references as Grape will check them in the order they are specified and depending on that order, an artifact other than the one expected may be used which is typically an issue when working with SNAPSHOT dependencies.

Documentation Environment

The documentation generation process is not Maven-based and uses shell scripts to process the project’s asciidoc. The scripts should work on Mac and Linux. Javadocs should be built using Java 11.

Tip
 We recommend performing documentation generation on Linux. For the scripts to work on Mac, you will need to install GNU versions of the utility programs via homebrew, e.g.grep, awk, sed, findutils, and diffutils.

To generate documentation, it is required that Hadoop 3.3.x is running in pseudo-distributed mode. Be sure to set the HADOOP_GREMLIN_LIBS environment variable as described in the reference documentation. It is also important to set the CLASSPATH to point at the directory containing the Hadoop configuration files, like mapred-site.xml.

The /etc/hadoop/yarn-site.xml file prefers this configuration over the one provided in the Hadoop documentation referenced above:

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
  </property>
  <property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>4</value>
  </property>
</configuration>

The /etc/hadoop/mapred-site.xml file prefers the following configuration:

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapred.map.tasks</name>
    <value>4</value>
  </property>
  <property>
    <name>mapred.reduce.tasks</name>
    <value>4</value>
  </property>
  <property>
    <name>mapreduce.job.counters.limit</name>
    <value>1000</value>
  </property>
  <property>
    <name>mapreduce.jobtracker.address</name>
    <value>localhost:9001</value>
  </property>
  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>2048</value>
  </property>
  <property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>4096</value>
  </property>
  <property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx2048m</value>
  </property>
  <property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx4096m</value>
  </property>
</configuration>

Also note that awk version 4.0.1 is required for documentation generation. The YARN recipe also uses the zip program to create an archive so that needs to be installed, too, if you don’t have it already.

The Hadoop 3.3.x installation instructions call for installing pdsh but installing that seems to cause permission problems when executing sbin/start-dfs.sh. Skipping that prerequisite seems to solve the problem.

Documentation can be generated locally with:

bin/process-docs.sh

Documentation is generated to the target/docs directory. It is also possible to generate documentation locally with Docker. docker/build.sh -d.

Note
 The installation of plugins sometimes fails in this step with the error: Error grabbing grapes - download failed. It often helps in this case to delete the directories for the dependencies that cannot be downloaded in the .m2 (~/.m2/) and in the grapes (~/.groovy/grapes/) cache. E.g., if the error is about asm#asm;3.2!asm.jar, then remove the asm/asm sub directory in both directories.
Note
 Unexpected failures with OLAP often point to a jar conflict that arises in scenarios where Hadoop or Spark dependencies (or other dependencies for that matter) are modified and conflict. It is not picked up by the enforcer plugin because the inconsistency arises through plugin installation in Gremlin Console at document generation time. Making adjustments to the various paths by way of the <manifestEntries> on the jar given the functionality provided by the DependencyGrabber class which allows you to manipulate (typically deleting conflicting files from /lib and /plugin) plugin loading will usually resolve it, though it could also be a more general environmental problem with Spark or Hadoop. The easiest way to see the error is to simply run the examples in the Gremlin Console which more plainly displays the error than the failure of the documentation generation process.

To generate the web site locally, there is no need for any of the above infrastructure. Site generation is a simple shell script:

bin/generate-home.sh

The site will be generated to the target/site/home directory.

Python Environment

As of TinkerPop 3.2.2, the build optionally requires Python to build the gremlin-python module. If Python is not installed, TinkerPop will still build with Maven, but native Python tests and Java tests that require Python code will be skipped. Developers should also install pip and virtualenv (version 15.0.2 - older versions may cause build failures).

The build expects Python a python3 installation which should be 3.5.3 or better. Python also tests kerberos and therefore requires:

sudo apt install libkrb5-dev krb5-user

As of TinkerPop 3.5.5, gremlin-python uses Docker for all tests inside of Maven, and Python installation will not be required to run gremlin-python through Maven. Please make sure Docker is installed and running on your system.

Once the Python environment is established, the full building and testing of gremlin-python may commence. It can be done manually from the command line with:

mvn clean install -Pglv-python

which enables the "glv-python" Maven profile or in a more automated fashion simply add a .glv file to the root of the gremlin-python module which will signify to Maven that the environment is Python-ready. The .glv file need not have any contents and is ignored by Git. A standard mvn clean install will then build gremlin-python in full.

The .glv file in gremlin-python also activates the "console-integration-tests" Maven profile to run gremlin-console integration tests. Alternatively, this profile can be activated manually. Note that unlike gremlin-python the tests are actually integration tests and therefore must be actively switched on with -DskipIntegrationTests=false:

mvn clean install -pl gremlin-console -DskipIntegrationTests=false
Tip
 For those who do not have a full Maven environment, please see this section for how Docker can be used to help run tests.
Tip
 Consider use of pyenv to better manage Python versions and environments.

See the Release Environment section for more information on release manager configurations.

DotNet Environment

The build optionally requires .NET SDK (>=8.0) to work with the gremlin-dotnet module. If .NET SDK is not installed, TinkerPop will still build with Maven, but .NET projects will be skipped.

As of TinkerPop 3.5.5, gremlin-dotnet uses Docker for running all test projects inside of Maven, and .NET SDK will not be required to run gremlin-dotnet tests through Maven. Please make sure Docker is installed and running on your system.

gremlin-dotnet can be built and tested from the command line with:

mvn clean install -Pgremlin-dotnet

which enables the "gremlin-dotnet" Maven profile or in a more automated fashion simply add a .glv file to the src and test directories of the gremlin-dotnet module which will signify to Maven that the environment is .NET-ready. The .glv file need not have any contents and is ignored by Git. A standard mvn clean install will then build gremlin-dotnet in full.

In order to pack the Gremlin.Net.Template project, it is also necessary to install Mono. The template can still be built and tested without Mono but packing will be skipped. To pack the template (which will also download the NuGet CLI tool) the nuget property has to be set:

mvn clean install -Dnuget
Tip
 For those who do not have a full Maven environment, please see this section for how Docker can be used to help run tests.

See the Release Environment section for more information on release manager configurations.

JavaScript Environment

When building gremlin-javascript, mvn command will include a local copy of Node.js runtime and npm inside your project using com.github.eirslett:frontend-maven-plugin plugin. This copy of the Node.js runtime will not affect any other existing Node.js runtime instances in your machine.

To run the development and build scripts of gremlint and its corresponding web page docs/gremlint, Node.js and npm have to be installed. When generating or publishing the TinkerPop website, the docs/gremlint web page has to be built. Consequently, the scripts bin/generate-home.sh and bin/publish-home.sh require that Node.js and npm are installed. Version 8.x or newer of npm is required. This is covered in more detail in the Site section.

As of TinkerPop 3.5.5, gremlin-javascript uses Docker for all tests inside of Maven. Please make sure Docker is installed and running on your system.

Important
 Beware of unexpected or unwanted changes on package-lock.json files when committing and merging.
Tip
 For those who do not have a full Maven environment, please see this section for how Docker can be used to help run tests.
Tip
 Consider using nvm to manage node.js versions and environments.

See the Release Environment section for more information on release manager configurations.

Go Environment

The build optionally requires Go (>=1.21) to work with the gremlin-go module. Creating an empty .glv file will enable running of tests inside of Maven. If .glv file does not exist, TinkerPop will still build with Maven, but Go projects will be skipped.

gremlin-go can be built and tested from the command line with:

mvn clean install -pl gremlin-go

Alternatively, after installing Go, gremlin-go can be built from the command line with:

go build

Docker allows you to test the driver without installing any dependencies. The following command can be used to run docker:

docker-compose up --exit-code-from gremlin-go-integration-tests

See the Release Environment section for more information on release manager configurations.

Docker Environment

The build optionally requires Docker to build Docker images of Gremlin Server and Gremlin Console. The Docker images can be built from the command line with:

mvn clean install -pl gremlin-server,gremlin-console -DdockerImages

which enables the "docker-images" Maven profile.

If confronted with "Permission denied" errors on Linux, it may be necessary to do the following:

sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker
sudo chmod 666 /var/run/docker.sock

As of TinkerPop 3.5.5, a docker image of the Gremlin Server will be built automatically with mvn clean install, which is use for GLV tests inside of Docker. To skip building this image, append the -DskipImageBuild flag to Maven commands.

Release Environment

This section is only useful to TinkerPop release managers and describes prerequisites related to deploying an official release of TinkerPop.

Maven needs to be configured to deploy maven artifacts. Apache LDAP credentials can be used for this. Release managers should encrypt their Apache LDAP password as described in the Apache Maven docs. The encrypted password can then be configured in the settings.xml as described in the section "Set up your development environment" of this Apache Infra article. This configuration will be used by mvn deploy.

For Python releases, uploading to pypi uses twine which is automatically installed by the build process in maven. Twine refers to HOME/.pypirc file for configuration on the pypi deploy environments and username and password combinations. The file typically looks like this:

[distutils]
index-servers=
    pypi
    pypitest

[pypitest]
username = <username>
password =

[pypi]
username = <username>
password =

The approach above assumes basic authentication and a blank password will in .pyirc will have Maven prompt for a password. That said, it is preferred that your account be secured with 2FA, in which case, basic authentication will no longer be allowed. Instead, an API token is required. It can be generated under your account settings and then the username in .pyirc is set to token and the password to the token value. Please see PyPI documentation for more details.

For .NET releases, install Mono. The release process is known to work with 6.12.0, so it is best to probably install that version. Release managers should probably also do an install of nuget 3.4.4 as it will help with environmental setup. To get an environment ready to deploy to NuGet, it is necessary to have a NuGet API key. First, create an account with nuget and request that a PMC member add your account to the Gremlin.Net and the Gremlin.Net.Template package in nuget so that you can deploy. Next, generate an API key for your account on the nuget website. The API key should be added to NuGet.Config with the following:

mono nuget.exe setApiKey [your-api-key]

This should update ~/.config/NuGet/NuGet.Config a file with an entry containing the encrypted API key. On mvn deploy, this file will be referenced on the automated nuget push.

To deploy JavaScript / TypeScript artifacts on the npm registry, the release manager must set the authentication information on the ~/.npmrc file. The easiest way to do that is to use the npm adduser command. This must be done only once, as the auth token doesn’t have an expiration date and it’s stored on your file system. If this account is newly created then request that a PMC member add your account to the "gremlin" package on npm.

Deploying Docker images to Docker Hub requires an account that is a member of the TinkerPop organization. So if you don’t already have an account on Docker Hub then create one and request that a PMC member adds your account to the TinkerPop organization. Afterwards, authentication information needs to be added to the ~/.docker/config.json file. This information can simply be added with the docker login command which will ask for credentials. This must be done only once. Finally, docker push can be used to push images to Docker Hub which will be done automatically on mvn deploy or it can be triggered manually with mvn docker:push.

Building and Testing

The following commands are a mix of Maven flags and shell scripts that handle different build operations

  • Build project: mvn clean install

    • Build a specific module (e.g. gremlin-server) within the project: mvn clean install -pl gremlin-server

    • Build without assertions for "iterator leaks" which are enabled by default: mvn clean install -DtestIteratorLeaks=false

    • Specify specific tests in a TinkerPop Suite to run with the GREMLIN_TESTS environment variable, along with the Maven project list argument, e.g.:

      export GREMLIN_TESTS='org.apache.tinkerpop.gremlin.process.traversal.step.map.PathTest$Traversals,org.apache.tinkerpop.gremlin.process.traversal.PathTest'
mvn -Dmaven.javadoc.skip=true --projects tinkergraph-gremlin test

    • Clean the .groovy/grapes/org.apache.tinkerpop directory on build: mvn clean install -DcleanGrapes

    • Turn off "heavy" logging in the "process" tests: mvn clean install -DargLine="-DmuteTestLogs=true"

    • The test suite for neo4j-gremlin is disabled by default - to turn it on: mvn clean install -DincludeNeo4j

  • Regenerate toy graph data (only necessary given changes to IO classes): mvn clean install -Dio from tinkergraph-gremlin directory

    • If there are changes to the Gryo format, it may be necessary to generate the Grateful Dead dataset from GraphSON (see IoDataGenerationTest.shouldWriteGratefulDead)

  • Start Gremlin Server with Docker using the standard test configuration: docker/gremlin-server.sh

  • Check license headers are present: mvn apache-rat:check

  • Build AsciiDocs (see Documentation Environment): bin/process-docs.sh

    • Build AsciiDocs (but don’t evaluate code blocks): bin/process-docs.sh --dryRun

    • Build AsciiDocs (but don’t evaluate code blocks in specific files): bin/process-docs.sh --dryRun docs/src/reference/the-graph.asciidoc,docs/src/tutorial/getting-started,…​

    • Build AsciiDocs (but evaluate code blocks only in specific files): bin/process-docs.sh --fullRun docs/src/reference/the-graph.asciidoc,docs/src/tutorial/getting-started,…​

    • Process a single AsciiDoc file: docs/preprocessor/preprocess-file.sh `pwd`/gremlin-console/target/apache-tinkerpop-gremlin-console-*-standalone "" "*" `pwd`/docs/src/xyz.asciidoc

  • Build JavaDocs/JSDoc: mvn process-resources -Djavadoc

    • Javadoc to target/site/apidocs directory

    • JSDoc to the gremlin-javascript/src/main/javascript/gremlin-javascript/doc/ directory

  • Specify the seed used for Random in tests mvn clean install -DtestSeed - useful when a test fails, the seed will be printed in the build output so that the test can run with the same version of random (look for "TestHelper" logger in output)

  • Check for newer dependencies: mvn versions:display-dependency-updates or mvn versions:display-plugin-updates

  • Check the effective pom.xml: mvn -pl gremlin-python -Pglv-python help:effective-pom -Doutput=withProfilePom.xml

  • Deploy JavaDocs/AsciiDocs: bin/publish-docs.sh svn-username

  • Integration Tests: mvn verify -DskipIntegrationTests=false

    • Execute with the -DincludeNeo4j option to include neo4j-gremlin tests. As of 3.7.0, transactional tests on Gremlin Server, Gremlin Driver, and GLVs will run automatically with integration tests against the TinkerTransactionGraph.

    • Execute with the -DuseEpoll option to try to use Netty native transport (works on Linux, but will fallback to Java NIO on other OS).

  • Benchmarks: mvn verify -DskipBenchmarks=false

    • Reports are generated to the console and to gremlin-tools/gremlin-benchmark/target/reports/benchmark.

  • Test coverage report: mvn clean install -Dcoverage - note that the install is necessary because report aggregation is bound to that part of the lifecycle.

    • Reports are generated to gremlin-tools/gremlin-coverage/target/site.

  • cd site

    • Generate web site locally: bin/generate-home.sh

    • Publish web site: bin/publish-home.sh <username>

Building On Windows

The following steps must be taken in order to build TinkerPop on Windows:

  1. Install winutils for Hadoop

    • Download the latest version of winutils binaries for Hadoop. The binaries winutils.exe and hadoop.dll are required.

      • Note that these libraries require Microsoft Visual C Redistributable 2015-2022 to be installed. We've tested it on Windows 10 and Windows 11 with Microsoft Visual C Redistributable 2015-2022 14.32.31326.

    • Place contents of the bin folder on your local driver in the following folder structure:

      • e.g. hadoop-3.3.1/bin/winutils.exe

    • Set HADOOP_HOME to point to the hadoop-3.3.1 folder

    • Add %HADOOP_HOME%\bin to your PATH

  2. Run mvn clean install from root of tinkerpop

  3. Follow IDE specific steps if applicable:

You should now be able to work with TinkerPop on Windows.

Docker Integration

TinkerPop provides a shell script, that can start several build tasks within a Docker container. The required Docker images will be built automatically if they don’t exist yet. Thus the first invocation of the Docker script is expected to take some time.

The script can be found under PROJECT_HOME/docker/build.sh. The following tasks are currently supported:

  • run standard test suite

  • run integration tests

  • build Java docs

  • build user docs

A list of command line options is provided by docker/build.sh --help. The container will install, configure and start all required dependencies, such as Hadoop.

By default, this script will run every module in the project. However, if you are planning on working on just a small set of the modules (e.g. the GLVs) then you can use the script options to reduce the modules included by specifically selecting which modules you want. This behavior is currently supported for the non-Java GLVs and gremlin-console. This option will include only the selected modules as well as gremlin-server, gremlin-test, neo4j-gremlin and all their dependencies. This is the minimum set of modules required to build and test the GLVs.

Build and test gremlin-python and gremlin-go with minimal Gremlin Server dependencies
./docker/build.sh --tests --integration-tests --python --golang

Options can be passed to Docker by setting the TINKERPOP_DOCKER_OPTS environment variable. A speed boost can be gained at the expense of memory by using tmpfs and the special directory /usr/src/tinkermem.

Build in-memory
TINKERPOP_DOCKER_OPTS="--tmpfs /usr/src/tinkermem:exec,mode=0755,rw,noatime,size=2000m"
Disable IPv6 for Hadoop
TINKERPOP_DOCKER_OPTS="--sysctl net.ipv6.conf.all.disable_ipv6=1 --sysctl net.ipv6.conf.default.disable_ipv6=1"

A custom maven settings.xml can be supplied, for example, to point to a local proxy. Copy the settings.xml to the PROJECT_HOME/ directory. The Docker script will detect and copy it to the running container.

If the container is used to generate the user docs, it will start a web server and show the URL that is used to host the HTML docs.

After finishing all tasks, the script will immediately destroy the container.

Docker can also be helpful to developers who do not want to run tests from a Maven environment, which may be a bit opaque when dealing with test failures and largely unhelpful for debugging. This situation is typically case for developers doing work on Gremlin Language Variants (e.g. Python). To help alleviate this problem, developers can start a standalone Gremlin Server with its standard test configuration that is used in the standard Maven build.

Generally speaking, most developers will want to test their code against the latest build of Gremlin Server in the TinkerPop repository. To do that, first be sure to build a Docker image of the current code:

mvn clean install -DskipTests

Next, generate the a Docker image for Gremlin Server with:

mvn clean install -pl :gremlin-server -DdockerImages -DskipTests
Important
 If changes are made to the repository that need to be reflected in the Gremlin Server Docker image then the old image should be removed and then the above commands re-executed.

Finally, start the server with:

docker/gremlin-server.sh

Starting Gremlin Server this way makes it possible to run Gremlin Language Variant tests without Maven (for example, directly from a debugger) which should greatly reduce development friction for these environments.

It is also possible to specify the exact version of Gremlin Server to run with the test configuration. This version should be an existing Docker image version and must be an explicit version that maps to an actual TinkerPop artifact:

docker/gremlin-server.sh 3.4.2

To be a bit more clear, the version can not be a Docker tag like "latest" because there is no such TinkerPop artifact that has been published with that version number.

Testing Sub-Modules with Docker

Currently the modules gremlin-go, gremlin-javascript, gremlin-dotnet, gremlin-python and gremlin-console can be tested through Docker.

Please make sure Docker is installed and running on your system. You will need to install both Docker Engine and Docker Compose, which are included in Docker Desktop.

The following environment variables used by Docker Compose will automatically be set when running through Maven.

The docker compose environment variable GREMLIN_SERVER specifies the Gremlin server docker image to use, i.e. an image with the tag tinkerpop/gremlin-server:$GREMLIN_SERVER, and is a required environment variable. This also requires the specified docker image to exist, either locally or in Docker Hub.

Running mvn clean install -pl gremlin-server -DskipTests -DskipIntegrationTests=true -Dci -am in the main tinkerpop directory will automatically build a local SNAPSHOT Gremlin server image. If your OS Platform cannot build a local SNAPSHOT Gremlin server through maven, it is recommended to use the latest released server version from Docker Hub (do not use GREMLIN_SERVER=latest, use actual version number, e.g. GREMLIN_SERVER=3.5.x or GREMLIN_SERVER=3.6.x).

The docker compose environment variable HOME specifies the user home directory for mounting volumes during test image set up. This variable is set by default in Unix/Linux, but will need to be set for Windows, for example, run $env:HOME=$env:USERPROFILE in PowerShell.

There are different ways to launch the test suite and set the GREMLIN_SERVER environment variable depending on your Platform:

  • Run Maven commands, e.g. mvn clean install inside of project folder e.g. tinkerpop/gremlin-go, or mvn clean install -pl gremlin-go inside of tinkerpop (platform-agnostic - recommended)

  • Add GREMLIN_SERVER=<server-image-version> and HOME=<user-home-directory> to an .env file inside project folder and run docker-compose up --exit-code-from gremlin-go-integration-tests (Platform-agnostic).

  • Run GREMLIN_SERVER=<server-image-version> docker-compose up --exit-code-from gremlin-go-integration-tests in Unix/Linux.

  • Run $env:GREMLIN_SERVER="<server-image-version>";$env:HOME=$env:USERPROFILE;docker-compose up --exit-code-from gremlin-go-integration-tests in Windows PowerShell.

You should see exit code 0 upon successful completion of the test suites. Run docker-compose down to remove the service containers (not needed if you executed Maven commands or run.sh), or docker-compose down --rmi all to remove the service containers while deleting all used images.

Note for running docker with MacOS on ARM processors: Docker’s performance is extremely poor on ARM Mac’s in its default configuration. It is recommended to enable both the "New Virtualization Framework" and "VirtioFS" under Docker Desktop Settings → Experimental Features.

Intellij Usage

Most core TinkerPop developers are using Intellij for their work so this section helps describe the mechanisms for best working with it as an IDE.

Setup

Installation and basic configuration of Intellij is beyond the scope of this writing and it is assumed that the TinkerPop GitHub repository has been cloned and the root of the repository is open in Intellij. From there, we can begin to look at configuration options specifically relevant to TinkerPop itself.

TinkerPop has a module called gremlin-shaded which contains shaded dependencies for some libraries that are widely used and tend to introduce conflicts. To ensure that Intellij properly interprets this module after importing the Maven pom.xml perform the following steps:

  1. Build gremlin-shaded from the command line with mvn clean install.

  2. Right-click on the gremlin-shaded module in the project viewer of Intellij and select "Remove module". If this menu option is not available (as is the case in newer versions of Intellij - first noticed in 13.1.5), then open the "Maven Projects" side panel, right click the gremlin-shaded module and select "Ignore Project".

  3. In the "Maven Projects" Tool window and click the tool button for "Reimport All Maven projects" (go to View | Tool Windows | Maven Projects on the main menu if this panel is not activated).

  4. At this point it should be possible to compile and run the tests within Intellij, but in the worst case, use File | Invalidate Caches/Restart to ensure that indices properly rebuild.

Note that it may be necessary to re-execute these steps if the gremlin-shaded pom.xml is ever updated.

You will initially see lots of errors related to the Gremlin lexer/parser. The gremlin-language module requires ANTLR processing. While this processing is configured to execute with Maven, it can also be setup to generate parser files within Intellij itself on command:

  1. Install the ANTLR4 Grammar Plugin for Intellij

  2. Right-click on the Gremlin.g4 file and "Configure ANTLR"

  3. Set "Output directory where all output is generated" to target/generated-sources/antlr4

  4. Set "Grammar file encoding" to utf-8

  5. Set "Package/namespace for the generated code" to org.apache.tinkerpop.gremlin.language.grammar

  6. Set "Language" to Java

  7. Set "Case transformation in the Preview window" should be "Leave as-is"

  8. The "generate parse tree listener" should be unchecked and the "generate parse tree visitor" should be checked.

With these settings it should be possible to right-click Gremlin.g4 and "Generate ANTLR Recognizer" which will place the generated code in where specified at target/generated-sources/antlr4. Be sure to right-click the antlr4 directory and "Mark directory as" "Generated Sources Root" which should allow Intellij to recognize it.

The gremlin-groovy module uses a Java annotation processor to help support DSLs. Annotation processing in Intellij should be set up by the Maven import, but if this is set up incorrectly you will see a cryptic error message when building: java: Compilation failed: internal java compiler error. To fix this, search for the Intellij setting Annotation Processors. Make sure that annotation processing is enabled for gremlin-groovy but disabled for gremlin-annotations. This should fix the internal java compile error. Next you may see compile errors complaining that the jsr223 classes __, `CredentialTraversal, CredentialTraversalSource, and DefaultCredentialTraversal cannot be found. Those classes are generated by annotations. To fix these errors, make sure mark the directory gremlin-groovy/target/generated-sources/annotations as "Generated Sources Root".

Developers working on the neo4j-gremlin module should enabled the include-neo4j Maven profile in Intellij. This will ensure that tests will properly execute within the IDE.

Debugging

It is generally assumed that JVM-based debugging of TinkerPop code in Intellij is a relatively straightforward task for most developers, but it is worth pointing out a few important points related to it and to drill into some specifics for the non-JVM languages.

Java

There are generally no complexities to running the debugger for any JVM-based test in the repository, but the following tips are helpful to know when doing so:

  • It is not possible to run the tests in gremlin-test without an Graph implementation. For example, it is not possible to just right-click org.apache.tinkerpop.gremlin.process.traversal.step.filter.CoinTest and then select Debug 'CoinTest'. Instead, running that test would require opening tinkergraph-gremlin and executing the TinkerGraphProcessStandardTest (as an example) which runs the full Gremlin test suite to include CoinTest.

  • To run just CoinTest, set a GREMLIN_TESTS environment variable with the fully qualified path name to the test configuration in Intellij. Note that the fully qualified test name is really org.apache.tinkerpop.gremlin.process.traversal.step.filter.CoinTest$Traversals.

  • Gherkin tests have a similar pattern in that they require a Graph implementation to execute them. Therefore, debugging entails going to tinkergraph-gremlin and running TinkerGraphFeatureTest in the debugger.

  • It is possible to filter the Gherkin tests by adding a system property to the debug configuration that specifies the tags to use or ignore. For example to just run the coin() tests: -Dcucumber.filter.tags="@StepCoin"

Python

Debugging Python within this mainly JVM-based project structure requires a bit of configuration. The following steps will help get Intellij prepared for this task:

  1. Install the Python plugin from JetBrains which should provide PyCharm like functionality.

  2. Right-click the "tinkerpop" top-level module in Intellij’s project explorer and "Open Module Settings".

  3. Select "Platform Settings | SDKs" and then click the "+" to "Add Python SDK…​"

  4. Choose a "Virtualenv Environment" and a "New environment". Set the "Location" to "<project-root>/gremlin-python/src/main/python". Select a "Base interpreter" that matches the version required by gremlin-python and click "OK".

  5. Select "Project Settings | Modules" and then select "gremlin-python" from the listing. Change the "Module SDK" to the newly added Python SDK.

  6. Open a terminal to gremlin-python/scr/main/python and do venv/bin/pip3 install -e . to pull in all of the gremlin-python dependencies.

At this stage, it should be possible to run unit tests in Python:

  1. Pull down the drop down in the toolbar for "Run/Debug Configurations" and select "Edit Configurations…​"

  2. Click the "+" to "Python Tests | pytest" and browse to a test to supply the "Script path".

  3. Go to "Python interpreter" and select "Use SDK of module" and choose "gremlin-python" in the drop-down and click OK.

  4. The test should not appear in the "Run/Debug Configurations" and can be executed.

Note
 When the Debug button is pressed, Intellij may display a notification that using the debugger requires some additional downloads - confirm those installations as required.

Many of the tests in gremlin-python require Gremlin Server. They are effectively integration tests. TinkerPop makes it easy to debug these tests by providing a Docker based test server which is rigged up with all the configurations required for the tests to execute. Start this server with docker/gremlin-server.sh -n where the -n will enable Neo4j for transaction based tests.