Troubleshooting Neo4jExecutor

Troubleshooting dotmotif.Neo4jExecutor

The Neo4j executor has a lot of moving parts, so here are some frequently encountered issues and how to solve them.

Best-Practices

Memory Management

Memory variables should almost always be set explicitly; Neo4jExecutors cannot currently "guess" how much memory you want it to use.

• max_memory: How much RAM to use (maximum). Suggested value is "XG", where X is the integer number of gigabytes of RAM your machine has, minus a bit. (So for a 128GB machine, 120G seems like a good start.)
• initial_memory: How big the JVM stack should be to start. (You can generally ignore this if you don't know what it means.)

Important Warnings

When creating a new Executor, specify `max_memory` and `initial_memory` so that the Executor can expand to fill the available space. But...a few warnings! **If you set `max_memory` too high, your container will silently fail (as the container will try to allocate too much memory, and then do whatever it is that the JVM does instead of segfaulting noisily).

initial_memory should be set high enough that the JVM doesn't have to reprovision memory too many times; but be aware that this amount of memory will be unavailable to the rest of your system while the executor is alive.

Errors when starting a new Executor

• I get a warning that "host port 7474 is already in use."

  You may already have a running Neo4jExecutor or Neo4j database container which is using the 7474 port. Check with docker ps.

• The executor waits for a long time, and then tells me it failed to reach the Neo4j server.

  This means that the executor tried to create a new docker container, but was unable to reach it.

Errors with executor responses

• After I run Executor.find, the result list is empty! But I know that my graph contains that motif!

  The .find() returns a cursor to your results, not your results themselves. Please consider the following:

  E = Neo4jExecutor(...)
  
  results = E.find(motif)
  A = results.to_table()
  B = results.to_table()

  A will contain all of your results; B will be EMPTY, since you already "used up" your results in the first call to to_table. You can learn more about cursors here. Once you have assigned these values to A, you can reuse them as many times as you like.

  Common Follow-up Question: WHYYY DO YOU DO THIS

  In some cases, you may receive too many results to easily process (many gigabytes of results). In these cases, you will want to 'stream' the results instead of getting a list of all of them. Here, next(results) is your friend!

  If you prefer that the executor return a nicely parsed table of results, you can pass cursor=False to find. You will then get back a Python data structure instead of a cursor.

• The executor waits for a long time, and then tells me it failed to reach the Neo4j server.

  This means that the executor tried to create a new docker container, but was unable to reach it. Make sure you have Docker installed, that you have a Neo4j image either local or reachable over the net, and that your Docker container can be accessed by the current user. Once you have run DotMotif for the first time (or have installed the Neo4j image manually), it will run entirely offline in subsequent runs.

The executor successfully creates immediately, but my first query fails. (Subsequent queries within a few seconds may also fail.)

Versions of dotmotif including and following 0.4.3 will wait for the Docker container to spin up by default before the Executor creation completes successfully. If you are using an earlier version of DotMotif, or you have passed the wait_for_boot=False argument to the Neo4jExecutor constructor, the initialization will complete while the container continues to boot in the background.

This is desired behavior if you anticipate a long-running process taking place in between executor creation and your first query. But in all other cases, you should leave this flag set to True.