v0.16.0

Relational database schemas (#1020 via #1159)

Subgraph data is now stored in Postgres using relational schemas. Instead of a storing all entities of a subgraph in a single table using JSONB, each entity type gets a dedicated table, with each field being stored in its own column.

This has several advantages:

1. Reading and writing are faster, more scalable.
2. Entity data can be validated and enforced better, preventing bugs in subgraphs.
3. Introspecting the data is much easier when debugging.

The stricter validation of entities written to the store may cause subgraphs that work today to fail when being deployed. A note from @lutter, who implemented this feature:

As part of the switch to the relational storage scheme, subgraph entities are validated more strictly than they were in the past. In particular, the store will reject attempts to store entities that have null in fields that are marked as non-null in the GraphQL schema. If your subgraph reports such errors after you deploy a new version, you can address that either by marking the field as nullable in the GraphQL schema by leaving the ! off its type, or by changing your mappings to ensure that the field is always set to a non-null value.

Note: This only affects new subgraph deployments. Existing deployments will keep using the JSONB schema. On the hosted service we will slowly clean up unused deployments over time until we can drop the old JSONB-based schema entirely.

Start block for data sources (#1262, #1294, #1320)

Until now, subgraphs were always indexing from the genesis block. This involved scanning the entire chain for relevant blocks, logs and traces, even if the subgraph contracts were only deployed recently.

This was fine initially. However, with more expensive features—such as block and call handlers—being added and more advanced subgraphs being developed, it has often become desirable to skip irrelevant old blocks entirely to speed up indexing.

This is now possible. As of this release, data sources can specify an optional startBlock number in the manifest:

dataSources:
  - name: Gravity
    source:
      address: '0x2E645469f354BB4F5c8a05B3b30A929361cf77eC'
      abi: Gravity
      startBlock: 6000000

The subgraph will then start indexing from this block. If there are multiple data sources with or without start blocks, the earliest of these blocks is used as the starting point. The absence of a startBlock is equivalent to 0 (aka the genesis block).

Other changes

GraphQL

• Fix querying interfaces by id (#1204)
• Fix following interface references (#1303 via #1307)
• Support where filter on relationship fields (#1212)
• Validate @derivedFrom directives on subgraph schemas (#1247)

Ethereum

• Add an Ethereum contract call cache to speed up indexing (#1230 via #1270)
• Detect failed Solidity assertions in try_ calls (#1162)
• Detect Bad instruction fd errors in try_ calls (#1327)
• Allow recursive data source template creation within the same block (#1185)
• Handle Alchemy timeouts in eth_getLogs gracefully (#1242)
• Make eth_getLogs scanning more efficient (#1260)
• Only fetch transaction receipts when necessary (#1308, #1326)

Mappings

• Reduce memory usage and number of threads by using one thread per mapping instead of one per data source (#1281)

Database

• Allow subgraphs to make progress while Postgres is autovacuuming the subgraphs table (#1175)
• Use a single connection pool per node (#1232)
• Preload pg_stat_statements in Docker Compose setup (#1163)
• Fix default Postgres port in the README (#1181, thanks @akeem!)

Monitoring

• Add Prometheus instrumentation support (#1259 via #1264)

Configuration

• Add new GRAPH_IPFS_SUBGRAPH_LOADING_TIMEOUT environment variable for loading subgraph manifests from IPFS (#1091, #1238, #1244)
• Add optional limit for GraphQL/WS operations using the GRAPH_GRAPHQL_MAX_OPERATIONS_PER_CONNECTION environment variable (#1160)

Misc

• Don't skip the genesis block when indexing (#1295 via #1297)
• Allow Docker image to wait for other services using full URLs (#1275)
• Fix Docker builds by passing --locked to Cargo
• Support multiple Ethereum networks in Docker image by sperating networks in the ethereum environment variable with spaces (#1332)
• Fix yarn commands in README (#1282, thanks @ana0!)
• Update dependencies (bs58, chrono, clap, crossbeam-channel, graphql-parser, hex, hex-literal, hyper, indexmap, jsonrpc-core, jsonrpc-http-server, lru_time_cache, num-bigint, serde, serde_derive, serde_yaml)
• Remove dependencies on itertools and sentry

v0.15.1

• Fix WASM memory allocation bug (#1154).
• Fix incorrect contract call reversion detection caused by ambiguity between Ganache and Parity (#1155).

Note: Make sure to also read the release notes for v0.15.0.

v0.15.0

CAUTION: If you are running your own Graph Node, make sure to run 0.14.0 for a week or two and deploy a new version of your subgraphs at least once before switching to 0.15.0. The 0.14.0 release gradually removes the public.entities table in favor of per-subgraph entities tables. Switching to 0.15.0 too early may break existing subgraphs and require you to redeploy and reindex immediately.

Fallible contract calls (#1139)

Calls to contract functions can fail due to assertions in the contract. Until now there was no way to handle this in subgraphs gracefully. This release introduces new try_someContractFunction call variants that return a result object with reverted and value fields. These can then be used in mappings to handle call failures.

An example can be found in the documentation.

Top-level templates (#1085)

Data source templates have been moved to a top-level templates field in the manifest. The data source templates documentation has been updated accordingly.

Top-level templates simplify creating new data sources at runtime: templates can now be referred to from all data sources and can create new data sources from other templates as well.

This also affects subgraph validation and code generation in a few ways. While Graph Node >= 0.15.0 still allows subgraphs with nested templates to run, Graph CLI now rejects such subgraphs. Code generation now puts all generated template classes into a single templates.ts file.

For more details, see the original issue.

Docker changes

• Set default Ethereum network name to mainnet in docker-compose.yml (#1086).
• Add setup.sh script for Linux host IP detection. Run this before the docker-compose up and it will inject the host IP address into docker-compose.yml (#1123).

Other changes

• Add support for arrays of Ethereum tuples / Solidity structs (#1119).
• Optimize performance of serializing data from Rust to AssemblyScript (#1145).
• Support decoding Ethereum strings with broken UTF-8 encoding (#1152).
• Bump default IPFS timeout to 60s (#1091).
• Fetch blocks from Ethereum if loading them from the cache fails (#1108).
• Reduce memory usage of filtering blocks with call traces (#1110).
• Add GRAPH_TOKIO_THREAD_COUNT environment variable (#1117, see environment variable docs).
• Add GRAPH_NODE_ID environment variable as an alternative to --node-id (#1136, see environment variable docs).
• Fix block/call handlers from dynamic data sources not being loaded on node restart (#1087).
• Fix ingesting blocks with no transactions (#1099).
• Fix indexing the genesis block with Ganache (#1094).
• Fix skipping call traces without insufficient data (#1104).
• Fix Postgres indexes for string fields to support values of arbitrary length (#1138).
• Update codebase to Rust 1.37 (#1118).
• Update dependencies.

v0.14.0

v0.14.0

v0.13.2

v0.13.2

v0.13.1

v0.13.1

v0.13.0

v0.13.0

v0.12.0

Changes

• Add a log.log host export for the new graph-ts logging API (see the AssemblyScript API documentation for more information) (#599, #943).
• Measure the effect of tokio and database contention in subgraph indexing (#917, #918, #921, #926).
• Document the templates field in the manifest spec (#910, contribution by @dOrgJelli).
• Don't block chain head updates when there is back-pressure from block streams (#929).
• Write block ingestion and GraphQL query activity to Elasticsearch (#928).
• Add a database index for event sources in the event_meta_data (#945).
• Add database indexes for reference fields (#916, #944).
• Allow trace logging in production builds (#948).
• Bump tokio from 0.1.15 to 0.1.20 (#946).
• Increase thread pool size to 100 (#946).
• Make block stream errors non-fatal to allow to retry fetching blocks (#947).

v0.11.0

Block, transaction and call handlers

Until now, the only triggers for indexing were events. This release adds support for triggering based on blocks and transactions/calls in the form of blockHandlers and callHandlers (in addition to the existing eventHandlers).

From the documentation:

While events provide an effective way to collect relevant changes to the state of a contract, many contracts avoid generating logs to optimize gas costs. In these cases, a subgraph can subscribe to calls made to the data source contract. This is achieved by defining call handlers referencing the function signature and the mapping handler that will process calls to this function. To process these calls, the mapping handler will receive an EthereumCall as an argument with the typed inputs to and outputs from the call. Calls made at any depth in a transaction's call chain will trigger the mapping, allowing activity with the data source contract through proxy contracts to be captured.

Regarding block handlers:

In addition to subscribing to contract events or function calls, a subgraph may want to update its data as new blocks are appended to the chain. To achieve this a subgraph can run a function after every block or after blocks that match a predefined filter.

For more information about how to define and write call and block handlers, please refer to the documentation.

Note: This feature requires Parity archive nodes with the trace API enabled.

Adaptive block range scanning

In order to deal with Infura's limit of returning a maximum of 1000 matches for eth_getLogs, the block range based scanning for events is now adapting the size of the block ranges being scanned when necessary. After a problematic block range has been processed, the block range size returns to the original ETHEREUM_BLOCK_RANGE_SIZE value.

Efficient entityCount querying

Until now, querying the entityCount of a subgraph deployment via the subgraph of subgraphs at /subgraphs lazily counted the entities of the deployment. This becomes increasingly expensive as subgraphs grow in size.

Starting with this release, the entityCount is stored in the deployment and is updated whenever changes are made to the entities of the subgraph, including when operations are being reverted due to a block reorg. This makes querying entityCount efficient.

Other changes

• Use a global database lock to prevent Graph Nodes from running migrations in parallel (#922).
• Capture and log potential thread and database contentions (#917, #918, #921).
• Revert dynamically created data sources that are affected by block reorgs (#890, #907).
• Add log message codes to subgraph and query logs for analytics (#915).
• Hide URL passwords from logs (#849, #853).
• Remove subgraph_list JSON-RPC method (#899).

v0.10.0

Data source templates / dynamic data sources

Also referred to as dynamic contract subscriptions, as this is currently the main use case.

This feature supports creating new data sources from templates while indexing the subgraph. The motivation behind this is to provide a natural way of indexing registry/factory contracts that reference many other (sub)contracts.

See Define a Subgraph: Dynamic Data Sources in the docs for more details.

Details about the implementation are described in the pull request that introduced the feature:

When starting a subgraph, the node now not only loads the manifest, it also loads the dynamic data sources from the store and fetches their files from IPFS. It then injects the data source instances into the subgraph instance before starting the indexing.

Whenever we we process a block and mappings request new data sources, these data sources are collected and, after having processed the block, are instantiated from templates. We then process the current block again but only with those new data sources. The entity operations from this are merged into the ones we already have for the block. After that, the dynamic data sources are persisted by adding the data sources to the subgraph instance and by adding entity operations to store them in the db (for 1.).

If new data sources have been added in a block, the block stream is restarted to incorporate events that are relevant for the new data sources going forward. We figured this would be easier to do (and it is) than modifying the block stream while it is already running.

Anonymous events

Anonymous Solidity events are used by projects like Maker. Supporting them requires filtering events not by their usual signature (e.g. Transfer(address,address)) but by their topic 0 value.

This version adds support for that by allowing event handlers to specify the topic0 value to filter by. For more information see Define a Subgraph: Anonymous Events in the docs.

Full support for GraphQL fragments

This releases finishes the support for GraphQL fragments both inline and spread in.

Query complexity limiting

Graph Node now supports limiting complexity of queries both in terms of potential number of entities returned and query depth, similar to how GitHub do it in their GraphQL API.

The following environment variables can be used for this:

• GRAPH_GRAPHQL_MAX_COMPLEXITY — limits the complexity of queries (default: unlimited).
• GRAPH_GRAPHQL_MAX_DEPTH — limits the depth of queries (default and maximum: 255).

Entity table splitting

Starting with this release, Graph Node stores entities and their history in a Postgres schema dedicated to the subgraph deployment. Every subgraph deployment now gets their own entities table, which speeds up queries significantly. Existing entity data in the global entities table can still be queried as before.

Other changes

• Fix list equality filters.
• Log smart contract call times.
• Reduce subgraph logging overall to reduce noise around event processing.
• Switch to the latest IPFS client version for HTTPS support.
• Add environment variables to control the block range size used while scanning the history of the chain (ETHEREUM_PARALLEL_BLOCK_RANGES) as well as the number of parallel block ranges to scan (ETHEREUM_BLOCK_RANGE_SIZE).