Skip to content

Latest commit

 

History

History
364 lines (306 loc) · 22.5 KB

session-and-channel-pool-configuration.md

File metadata and controls

364 lines (306 loc) · 22.5 KB
Raw

Session Pool and Channel Pool Configuration

NOTE: This document contains detailed background information on how the Cloud Spanner client library works internally. This document is intended for expert users who want a more in-depth understanding of how the library works. It is not necessary to read or understand this document in order to use the client library.

Introduction

The Cloud Spanner Java client uses gRPC for communication with Cloud Spanner. gRPC uses 'channels', which are roughly equivalent to TCP connections. gRPC channels are handled by the core gRPC library. The gRPC library creates and maintains channel pools that are used for communication with Cloud Spanner. The client application provides configuration for how those channels should be created, how many should be created etc. This document describes the default configuration that the Cloud Spanner Java client uses for channels and sessions, and what configuration options can be tuned for applications that require a higher throughput than is possible with the default configuration.

Sessions

Cloud Spanner uses sessions for all data operations. A session belongs to a specific database. Executing a query or starting a transaction will use a session. One session can only execute one transaction at any time. This means that an application will need as many sessions as the number of concurrent transactions that the application will execute. Note that executing a query without a transaction on Cloud Spanner will automatically cause Cloud Spanner to use a single-use read-only transaction for the query. The session pool keeps track of the metric MAX_IN_USE_SESSIONS to help users understand how many sessions are used by the application.

Admin operations such as creating instances and databases, but also all DDL statements, do not require a session.

gRPC Channels

gRPC channels are used by the Cloud Spanner client for communication. One gRPC channel is roughly equivalent to a TCP connection. One gRPC channel can handle up to 100 concurrent requests. This means that an application will need at least as many gRPC channels as the number of concurrent requests the application will execute, divided by 100.

Each Spanner client creates a new channel pool. Each DatabaseClient that is created by the same Spanner client will use the same channel pool.

The Cloud Spanner client will affiliate each session with one channel. This ensures that the same channel is used for each request that uses the same session, which improves affinity to backend servers.

Session Pool and Channel Pool

The Java client creates and maintains a session pool for each DatabaseClient that is created by the client. The session pool configuration can be customized by the client application.

The relationship between channel pools and session pools is:

  1. A client application creates a Spanner client. A Spanner client is specific to a Google Cloud project and holds the credentials that should be used to connect to Cloud Spanner. A Spanner client can create one or more DatabaseClients for data operations, and DatabaseAdminClients and InstanceAdminClients for admin operations.
  2. A Spanner client creates one channel pool that will be shared by all DatabaseClients that are created by this Spanner client.
  3. DatabaseClients are specific to a single database. A DatabaseClient contains the session pool that is specific for that database.
  4. Multiple session pools (DatabaseClients) that are created by the same Spanner client will share the same channel pool.

Default Configuration and Behavior

Session and Channel Pool

The Java Spanner client will by default:

  1. Create a gRPC channel pool of fixed size with 4 channels (configurable).
  2. Each DatabaseClient that is created by a Spanner client will create a session pool with MinSessions=100 and MaxSessions=400. These values are configurable.
  3. The initial 100 sessions that are created by the pool are evenly distributed over the channels. This means that the pool will execute 4 (varies with the number of channels that have been configured) BatchCreateSessions RPCs, each using a separate channel. A channel pool hint is added to all of the sessions that are created to ensure that all subsequent RPCs on those sessions use the same channel. This behavior is fixed and cannot be changed through configuration.
  4. The sessions that are returned by the initial BatchCreateSessions RPCs are added in random order to the session pool. This gives a uniform distribution of channels across the session pool. This behavior is not configurable.
  5. The session pool will dynamically create new sessions if all sessions that are currently in the pool have been checked out and a new session is requested by the application. Session creation is executed asynchronously using the internal thread pool that is also used for gRPC invocations (see below). The thread that requested a session will be blocked until either at least one new session has been created, or another thread returns a session to the pool.
  6. Growth of the session pool is done in steps of 25 sessions (not configurable). Each step of 25 sessions is created in a single BatchCreateSessions RPC. The channel that is used for this RPC is changed round-robin for each new growth RPC. This means that the first ‘growth RPC’ will create slight overuse of the first channel, and that it will even out as more growth happens. This behavior is not configurable.
  7. Sessions are taken from and returned to the session pool in LIFO order. This behavior is fixed and not configurable.
  8. The session pool does not prepare read/write transactions on the sessions. Instead, the client library will inline the BeginTransaction option with the first statement in a transaction. This behavior is not configurable.

The above values that are marked as configurable can only be changed at creation, and are not modifiable after a session pool / channel pool has been created.

gRPC Libraries

The gRPC libraries will:

  1. Create the actual channel pool with NumChannels (configurable) and maintain this. Dropped channels etc. are handled transparently by the gRPC library.
  2. Create an internal thread pool that is used for RPC invocations. The thread pool will scale dynamically with actual usage. Each RPC invocation is handled by a separate thread, and threads in the pool will be reused. This means that the thread pool will at most contain as many threads as there have been parallel RPC invocations. It will automatically scale down the number of threads if there are less RPC invocations for a while (scale down happens after 60 seconds).
  3. The gRPC thread pool is shared among all gRPC channels created in the same JVM.
  4. The above can be configured by setting a custom ChannelProvider in the SpannerOptions. Using this requires in-depth knowledge of the gRPC libraries.

Tuning Options

More Pods/VMs vs Larger Session Pools

High throughput systems that need to scale can choose between creating a smaller number of large pods/VMs for the client application with more memory and CPU and a large session pool, or creating a larger number of smaller pods/VMs with a smaller session pool.

There’s no fundamental difference in performance between the two choices, as long as the size and available resources of a single pod/VM is aligned with the size of the session pool and expected QPS that it will receive, and the total size of the cluster is able to scale to a size that can handle the expected maximum number of transactions.

Note that:

  • The MaxSessions setting for a pod is the maximum number of parallel transactions or queries that a pod could theoretically execute.
  • The number of gRPC channels that is required is the maximum number of parallel requests/100 that a pod could theoretically execute. One channel can handle multiple requests from different transactions concurrently.
  • Setting a high value for MaxSessions on a pod with a small number of CPUs, limited memory and/or limited amount of network bandwidth therefore does not make sense, as that pod will not have the resources available to execute a large number of parallel transactions.
  • Setting a high value for MinSessions for a pod that does not execute many parallel transactions will cause that pod to spend resources to keep those sessions alive.
  • A rule of thumb is that having more than 100 sessions per CPU core in a pod/VM is unusual, as it would mean that one CPU core is expected to execute more than 100 concurrent transactions.

Session Pool and Channel Pool Options

Note: All the below considerations are at the level of a single pod/VM.

When considering whether you need to increase MaxSessions or NumChannels, you should always consider whether one single pod might execute more than a certain number of parallel transactions or queries, and not the total number of parallel transactions your system might have.

Num Channels

The NumChannels option in the Spanner options determines how many channels the underlying gRPC channel pool will contain. The default is 4. One gRPC channel can multiplex at most 100 parallel RPC invocations. The default 4 channels align with the default MaxSessions=400, as the MaxSessions value also determines the maximum number of parallel queries/transactions that can be executed by a single database client.

NumChannels should be at least MaxSessions / 100. E.g. MaxSessions=1000 => NumChannels=10.

Spanner clients that create multiple different session pools (e.g. for multi-tenancy purposes) should set NumChannels to at least the maximum number of concurrent requests/100 that the Spanner client will execute. You should also consider lowering the value for MinSessions and MaxSessions for these clients. The value for both should be set to the maximum number of concurrent transactions that one database client will execute. Keeping a large number of unused sessions alive will unnecessarily consume resources.

Increasing the number of channels beyond MaxSessions/100 on high throughput systems can reduce p95/p99 latencies, as the probability of random channel congestion is reduced. Creating more channels means creating more TCP connections, which will increase memory usage slightly. Creating too many channels can cause some channels to be used too little. This can cause Google infrastructure to close infrequently used channels, causing higher latencies as channels more often need to be re-created.

MinSessions and MaxSessions

MinSessions determines the number of sessions that are always created for a session pool. The default is 100. This number should be high enough to serve the normal application load for that pod/database pair. Each transaction, including single-use read-only transactions, requires a session.

MaxSessions determines the maximum number of sessions that will ever be created by the pool on that pod. The default is 400. This number should be high enough to serve the highest number of parallel transactions that is expected on a single pod/database pair.

It is recommended to set MinSessions=MaxSessions on high throughput systems to prevent unnecessary up- and downscaling of the session pool.

Setting MinSessions to a much larger value than the number of transactions that a pod/database pair will ever execute in parallel will degrade performance. Most of these sessions will not be used, and the client library will have to actively keep these alive. This will cost additional resources both on the client and the server.

Life of a Request

An application that uses the Cloud Spanner Java client library will execute queries and transactions using the public API of the client library. This section explains what happens internally in the client library during such a request.

Life of a Single Query

Executing a single query or read operation using the Java client library uses a 'single use read context' like this:

DatabaseClient client =
    spanner.getDatabaseClient(DatabaseId.of("my-project", "my-instance", "my-database"));
try (ResultSet resultSet =
    client.singleUse().executeQuery(Statement.of("select col1, col2 from my_table"))) {
  while (resultSet.next()) {
    // use the results.
  }
}

This will cause the following to happen internally in the client library:

  1. The singleUse() method call returns a ReadContext. Internally, this method checks out a session from the session pool. The call to check out a session from the pool is always non-blocking, and if no session is available in the pool, the read context will instead be assigned a reference to a future session. Checking out a session from the pool can also initiate an RPC to create more sessions in the pool, if all sessions in the pool at that moment are in use. This RPC will be executed in the background using the default gRPC thread pool.
  2. The executeQuery(..) call returns a ResultSet. This call is also non-blocking and will prepare the request that is needed to execute the query. The actual RPC invocation to execute the query will be delayed until the first call to ResultSet#next().
  3. The first call to ResultSet#next() will:
    1. If necessary, block until the session that was checked out in step 1 comes available.
    2. Invoke the ExecuteStreamingSql RPC on Cloud Spanner. The RPC invocation will use a thread from the default gRPC thread pool. The stream that is returned by this RPC will fill the result set with data.
    3. Subsequent calls to ResultSet#next() will return data row-by-row that is yielded by the stream that was returned in step 3.ii.
  4. Closing the result set will return the session that was checked out in step 1 to the pool. It is therefore good practice to use all result sets in a try-with-resources block to ensure it is always closed. Failing to close a result set will cause a session leak. The result set is also automatically closed when all data has been consumed.

Life of a Read/Write Transaction

Executing a read/write transaction using the Java client library uses a TransactionRunner like this:

client
    .readWriteTransaction()
    .run(
        transaction -> {
          try (ResultSet resultSet =
              transaction.executeQuery(Statement.of("select col1, col2 from my_table"))) {
            while (resultSet.next()) {
              // use the results.
            }
          }
          return transaction.executeUpdate(
              Statement.newBuilder("update my_table set col2=@value where col1=@id")
                  .bind("value")
                  .to("new-value")
                  .bind("id")
                  .to(1L)
                  .build());
        });

This will cause the following to happen internally in the client library:

  1. The readWriteTransaction() method call returns a TransactionRunner. Internally, this method checks out a session from the session pool. The call to check out a session from the pool is always non-blocking, and if no session is available in the pool, the transaction runner will instead be assigned a reference to a future session. Checking out a session from the pool can also initiate an RPC to create more sessions in the pool, if all sessions in the pool at that moment are in use. This RPC will be executed in the background using the default gRPC thread pool.
  2. The TransactionRunner#run(..) method will execute the given user code in a read/write transaction. The user function that is passed to the TransactionRunner will receive a reference to a TransactionContext. All statements in the transaction should use this TransactionContext.
  3. The client library will not invoke the BeginTransaction RPC on Cloud Spanner. Instead, the client library will include a BeginTransaction option with the first statement that is executed using the TransactionContext. In the above example, that is the executeQuery call. All subsequent statements in the same transaction will use the transaction identifier that was returned by the first statement.
  4. The executeQuery(..) call returns a ResultSet. This call is also non-blocking and will prepare the request that is needed to execute the query. The actual RPC invocation to execute the query will be delayed until the first call to ResultSet#next().
  5. The first call to ResultSet#next() will:
    1. If necessary, block until the session that was checked out in step 1 comes available.
    2. Invoke the ExecuteStreamingSql RPC on Cloud Spanner. This RPC will include the BeginTransaction option. The RPC invocation will use a thread from the default gRPC thread pool. The stream that is returned by this RPC will fill the result set with data.
    3. Subsequent calls to ResultSet#next() will return data row-by-row that is yielded by the stream that was returned in step 5.ii.
  6. The executeUpdate() call returns an update count. This call is blocking and will directly execute the update statement using the ExecuteSql RPC. The RPC invocation uses a thread from the default gRPC thread pool.
  7. The TransactionRunner will automatically commit the transaction if the supplied user code finished without any errors. The Commit RPC that is invoked uses a thread from the default gRPC thread pool.

Session Leak

A DatabaseClient object of the Client Library has a limit on the number of maximum sessions. For example the default value of MaxSessions in the Java Client Library is 400. You can configure these values at the time of creating a Spanner instance by setting custom SessionPoolOptions. When all the sessions are checked out of the session pool, every new transaction has to wait until a session is returned to the pool. If a session is never returned to the pool (hence causing a session leak), the transactions will have to wait indefinitely and your application will be blocked.

Common Root Causes

The most common reason for session leaks in the Java client library are:

  1. Not closing a ResultSet that is returned by executeQuery. Always put ResultSet objects in a try-with-resources block, or take other measures to ensure that the ResultSet is always closed.
  2. Not closing a ReadOnlyTransaction when you no longer need it. Always put ReadOnlyTransaction objects in a try-with-resources block, or take other measures to ensure that the ReadOnlyTransaction is always closed.
  3. Not closing a TransactionManager when you no longer need it. Always put TransactionManager objects in a try-with-resources block, or take other measures to ensure that the TransactionManager is always closed.

As shown in the example below, the try-with-resources block releases the session after it is complete. If you don't use try-with-resources block, unless you explicitly call the close() method on all resources such as ResultSet, the session is not released back to the pool.

DatabaseClient client =
    spanner.getDatabaseClient(DatabaseId.of("my-project", "my-instance", "my-database"));
try (ResultSet resultSet =
    client.singleUse().executeQuery(Statement.of("select col1, col2 from my_table"))) {
  while (resultSet.next()) {
    // use the results.
  }
}

Debugging and Resolving Session Leaks

Logging

Enabled by default, the logging option shares warn logs when you have exhausted >95% of your session pool. This could mean two things, either you need to increase the max sessions in your session pool (as the number of queries run using the client side database object is greater than your session pool can serve) or you may have a session leak.

To help debug which transactions may be causing this session leak, the logs will also contain stack traces of transactions which have been running longer than expected. The logs are pushed to a destination based on how the log exporter is configured for the host application.

final SessionPoolOptions sessionPoolOptions = 
   SessionPoolOptions.newBuilder().setWarnIfInactiveTransactions().build()

final Spanner spanner =
        SpannerOptions.newBuilder()
            .setSessionPoolOption(sessionPoolOptions)
            .build()
            .getService();
final DatabaseClient client = spanner.getDatabaseClient(databaseId);

// Example Log message to warn presence of long running transactions
// Detected long-running session <session-info>. To automatically remove long-running sessions, set SessionOption ActionOnInactiveTransaction 
// to WARN_AND_CLOSE by invoking setWarnAndCloseIfInactiveTransactions() method. <Stack Trace and information on session>
Automatically clean inactive transactions

When the option to automatically clean inactive transactions is enabled, the client library will automatically spot problematic transactions that are running for extremely long periods of time (thus causing session leaks) and close them. The session will be removed from the pool and be replaced by a new session. To dig deeper into which transactions are being closed, you can check the logs to see the stack trace of the transactions which might be causing these leaks and further debug them.

final SessionPoolOptions sessionPoolOptions = 
   SessionPoolOptions.newBuilder().setWarnAndCloseIfInactiveTransactions().build()

final Spanner spanner =
        SpannerOptions.newBuilder()
            .setSessionPoolOption(sessionPoolOptions)
            .build()
            .getService();
final DatabaseClient client = spanner.getDatabaseClient(databaseId);

// Example Log message for when transaction is recycled
// Removing long-running session <Stack Trace and information on session>