CAVE is a computational infrastructure for immediate and reproducible connectome analysis in up-to petascale datasets (~ $1mm^3$) while proofreading and annotating is ongoing. CAVE consists of a number of services and database management systems that are listed below. It was initially developed as a collaboration between the electron microscopy group at the Allen Institute for Brain Science and Sebastian Seung's lab at Princeton University. It is most fully described in a pre-print available here, though aspects of the PyChunkedGraph service were described also in earlier works (Dorkenwald et al. 2022a and Dorkenwald et al. 2022b). It is use by several large scale published electron microscopy efforts, including the MICrONS project, flywire, fanc, and h01, and several unpublished datasets.

A service for storing basic metadata about data, presently datastack, aligned_volume, permission_group and table_mapping. Repository

A service for controlling authorization on top of authentication provided by Google OAuth. Repository

We deploy a second version of auth as "sticky" auth which has stick_sessions turned on. This is useful for certain operations that require sticky sessions, such as the flask-admin interface, whereas the other pods for auth do not require it and can distribute traffic more easily. There are no code base changes, only deployment changes.

A service for storing/serving presently supported annotation schemas. Note this is also a library that is utilized by AnnotationEngine and MaterializationEngine to generate database schemas. Repository

A service for posting and retrieving json states. Intended to provide support for neuroglancer link shortening. Backed by Google Datastore (future version to place json in bucket storage). Repository

A service for creating, reading and updating annotations on aligned volumes, independent of segmentation. Backed by postgreSQL and postGIS. Repository

A service to keep a set of annotations up to date with a segmentation. Shared codebase also contains an API to query these annotations, and a worker to process these annotations. Backed by postgreSQL and postGIS, workflow powered by celery with a redis broker. Repository

A service to dynamically proofread and track a supervoxel graph of a segmentation. Stores graph in a dynamic oct-tree backed by Google BigTable. Abbreviation PCG. Transmits activity via google pub-sub, presently consumed by Meshing and L2cache. Repository

A service to track, serve and regenerate meshes from a dynamic segmentation (shared codebase with the PCG). Has a worker component as well, driven by google pubsub. Repository

A service to track, serve and regenerate summary statistics of “level 2” fragments of segmentation in the PCG. Has a worker component for calculating statistics,driven by google pub-sub. Repository

A flask app service for creating neuroglancer links with annotation points to guide proofreaders to tips and branches. Guidebook utilizes PCG awfare skeletonization procedures via pcg_skel (see below). Repository

A flask app for serving dash apps that are protected by auth. Currently only serving dash-connectivity-viewer apps, provides interactive analysis UI of connectivity and cell types, along with dynamic neuroglancer link generation. Repository, Repository

An app designed for FlyWire to track proofreading progress. Repository

TBD. Repository (not public and not generalized past flywire)

This is a deprecated service for redirecting traffic to a google bucket which is not publicly readable. Requests are rerouted to a configurable set of google buckets, with an authorization header that is attached by the server to allow the user to get access to the bucket. The resulting data is simply forwarded back to the user. Connections to these endpoints are protected by the authorization service and so access can be limited in this way. Allows neuroglancer communication. Repository (not public repo)

Interface library for AnnotationEngine and MaterializationEngine to interface with SQLalchemy based dynamic databases. Repository

Library for automating creation of neuroglancer links from annotations as dataframes, wrapper over neuroglancer python. Repository

Python library for accessing various microservice endpoints. Repository, Documentation

Python library that wraps around cloudvolume to provide analysis and visualization functionality on top of meshes and skeleton. Repository, Documentation

Library for running skeletonization that integrates the chunkedgraph level2 graph, and the l2cache. Repository, Documentation

These tools integrate with CAVE and are useful for CAVE users. They are developed and maintained by members of the community and not the CAVE team.

Python library for accessing images, segmentation, meshes and skeletons from precomputed and CAVE (i.e. graphene protocol) sources. Repository, Documentation

Python library wrapped around many other libraries that gives analysis tools for analyzing neurons. Integrates with cloud-volume and caveclient. Focused mostly on fly neurons, but provides general blender integration for example. Repository, Documentation

CAVE uses kubernetes for deployment. We developed a bash script based system of templates and config files to deploy CAVE. Repository, Instructions