🐛 Bug Report: MaxListenersExceededWarning from TechDocs

[drodil]

📜 Description

Error and possible memory leak detected in the plugin-techdocs-backend. This has started to happen after updating from 1.25.2 to 1.26.4.

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. Use emitter.setMaxListeners() to increase limit

We are using the AWS S3 Publisher for TechDocs in production but local for local development, and it's failing there as well.

👍 Expected behavior

No error

👎 Actual Behavior with Screenshots

[0] (node:93627) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. Use emitter.setMaxListeners() to increase limit
[0]     at genericNodeError (node:internal/errors:984:15)
[0]     at wrappedFn (node:internal/errors:538:14)
[0]     at _addListener (node:events:592:17)
[0]     at Socket.addListener (node:events:610:10)
[0]     at Readable.on (node:internal/streams/readable:1126:35)
[0]     at Socket.socketListenerWrap [as on] (node:_http_server:1188:54)
[0]     at createEventStream (/backstage/node_modules/@backstage/plugin-techdocs-backend/src/service/router.ts:346:15)
[0]     at <anonymous> (/backstage/node_modules/@backstage/plugin-techdocs-backend/src/service/router.ts:243:55)
[0]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

👟 Reproduction steps

Just run the backend for a while and click around the UI. Good to run with to get the backtrace:

export NODE_OPTIONS='--trace-warnings'

📃 Provide the context for the Bug.

No response

🖥️ Your Environment

OS: Darwin 23.4.0 - darwin/arm64
node: v18.19.0
yarn: 1.22.19
cli: 0.26.4 (installed)
backstage: 1.26.4

Dependencies:
@backstage/app-defaults 1.5.4
@backstage/backend-app-api 0.6.2, 0.7.1
@backstage/backend-common 0.21.7
@backstage/backend-defaults 0.2.17
@backstage/backend-dev-utils 0.1.4
@backstage/backend-openapi-utils 0.1.10
@backstage/backend-plugin-api 0.6.17
@backstage/backend-tasks 0.5.22
@backstage/backend-test-utils 0.3.7
@backstage/catalog-client 1.6.4
@backstage/catalog-model 1.4.5
@backstage/cli-common 0.1.13
@backstage/cli-node 0.2.5
@backstage/cli 0.26.4
@backstage/config-loader 1.8.0
@backstage/config 1.2.0
@backstage/core-app-api 1.12.4
@backstage/core-compat-api 0.2.4
@backstage/core-components 0.14.5
@backstage/core-plugin-api 1.9.2
@backstage/dev-utils 1.0.31
@backstage/e2e-test-utils 0.1.1
@backstage/errors 1.2.4
@backstage/eslint-plugin 0.1.7
@backstage/frontend-plugin-api 0.6.4
@backstage/integration-aws-node 0.1.12
@backstage/integration-react 1.1.26
@backstage/integration 1.10.0
@backstage/plugin-api-docs 0.11.4
@backstage/plugin-app-backend 0.3.65
@backstage/plugin-app-node 0.1.17
@backstage/plugin-auth-backend-module-atlassian-provider 0.1.9
@backstage/plugin-auth-backend-module-aws-alb-provider 0.1.9
@backstage/plugin-auth-backend-module-azure-easyauth-provider 0.1.0
@backstage/plugin-auth-backend-module-bitbucket-provider 0.1.0
@backstage/plugin-auth-backend-module-cloudflare-access-provider 0.1.0
@backstage/plugin-auth-backend-module-gcp-iap-provider 0.2.12
@backstage/plugin-auth-backend-module-github-provider 0.1.14
@backstage/plugin-auth-backend-module-gitlab-provider 0.1.14
@backstage/plugin-auth-backend-module-google-provider 0.1.14
@backstage/plugin-auth-backend-module-guest-provider 0.1.3
@backstage/plugin-auth-backend-module-microsoft-provider 0.1.12
@backstage/plugin-auth-backend-module-oauth2-provider 0.1.14
@backstage/plugin-auth-backend-module-oauth2-proxy-provider 0.1.10
@backstage/plugin-auth-backend-module-oidc-provider 0.1.8
@backstage/plugin-auth-backend-module-okta-provider 0.0.10
@backstage/plugin-auth-backend 0.22.4
@backstage/plugin-auth-node 0.4.12
@backstage/plugin-auth-react 0.1.1
@backstage/plugin-catalog-backend-module-aws 0.3.12
@backstage/plugin-catalog-backend-module-msgraph 0.5.25
@backstage/plugin-catalog-backend-module-scaffolder-entity-model 0.1.15
@backstage/plugin-catalog-backend-module-unprocessed 0.4.4
@backstage/plugin-catalog-backend 1.21.1
@backstage/plugin-catalog-common 1.0.22
@backstage/plugin-catalog-graph 0.4.4
@backstage/plugin-catalog-node 1.11.1
@backstage/plugin-catalog-react 1.11.3
@backstage/plugin-catalog-unprocessed-entities-common 0.0.1
@backstage/plugin-catalog-unprocessed-entities 0.2.3
@backstage/plugin-catalog 1.19.0
@backstage/plugin-devtools-backend 0.3.3
@backstage/plugin-devtools-common 0.1.9
@backstage/plugin-devtools 0.1.13
@backstage/plugin-events-backend 0.3.4
@backstage/plugin-events-node 0.3.3
@backstage/plugin-home-react 0.1.12
@backstage/plugin-home 0.7.3
@backstage/plugin-kubernetes-common 0.7.5
@backstage/plugin-notifications-backend 0.2.0
@backstage/plugin-notifications-common 0.0.3
@backstage/plugin-notifications-node 0.1.3
@backstage/plugin-notifications 0.2.0
@backstage/plugin-org 0.6.24
@backstage/plugin-permission-backend 0.5.41
@backstage/plugin-permission-common 0.7.13
@backstage/plugin-permission-node 0.7.28
@backstage/plugin-permission-react 0.4.22
@backstage/plugin-proxy-backend 0.4.15
@backstage/plugin-scaffolder-backend-module-azure 0.1.9
@backstage/plugin-scaffolder-backend-module-bitbucket-cloud 0.1.7
@backstage/plugin-scaffolder-backend-module-bitbucket-server 0.1.7
@backstage/plugin-scaffolder-backend-module-bitbucket 0.2.7
@backstage/plugin-scaffolder-backend-module-confluence-to-markdown 0.2.18
@backstage/plugin-scaffolder-backend-module-cookiecutter 0.2.41
@backstage/plugin-scaffolder-backend-module-gerrit 0.1.9
@backstage/plugin-scaffolder-backend-module-gitea 0.1.7
@backstage/plugin-scaffolder-backend-module-github 0.2.7
@backstage/plugin-scaffolder-backend-module-gitlab 0.3.3
@backstage/plugin-scaffolder-backend 1.22.4
@backstage/plugin-scaffolder-common 1.5.1
@backstage/plugin-scaffolder-node-test-utils 0.1.3
@backstage/plugin-scaffolder-node 0.4.3
@backstage/plugin-scaffolder-react 1.8.4
@backstage/plugin-scaffolder 1.19.3
@backstage/plugin-search-backend-module-catalog 0.1.23
@backstage/plugin-search-backend-module-elasticsearch 1.4.0
@backstage/plugin-search-backend-module-techdocs 0.1.22
@backstage/plugin-search-backend-node 1.2.21
@backstage/plugin-search-backend 1.5.7
@backstage/plugin-search-common 1.2.11
@backstage/plugin-search-react 1.7.10
@backstage/plugin-search 1.4.10
@backstage/plugin-signals-backend 0.1.3
@backstage/plugin-signals-node 0.1.3
@backstage/plugin-signals-react 0.0.3
@backstage/plugin-signals 0.0.5
@backstage/plugin-techdocs-backend 1.10.4
@backstage/plugin-techdocs-node 1.12.3
@backstage/plugin-techdocs-react 1.2.3
@backstage/plugin-techdocs 1.10.4
@backstage/plugin-user-settings-backend 0.2.16
@backstage/plugin-user-settings 0.8.5
@backstage/release-manifests 0.0.11
@backstage/repo-tools 0.8.0
@backstage/test-utils 1.5.4
@backstage/theme 0.5.3
@backstage/types 1.1.1
@backstage/version-bridge 1.0.8

👀 Have you spent some time to check if this bug has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

Are you willing to submit PR?

None

[drodil]

Might be related: #20698 and #20700

[freben]

There's a couple of obvious close listeners being added:

When creating event streams:

backstage/plugins/techdocs-backend/src/service/router.ts

Line 346 in 32cc4e6

res.socket?.on('close', () => {

In the cache middleware:

backstage/plugins/techdocs-backend/src/cache/cacheMiddleware.ts

Line 70 in 32cc4e6

socket.on('close', async hadError => {

Just noting quickly, haven't dug deep. In the traceback, the first one seems to be the culprit. I guess it gets called several times for a given socket?

[drodil]

@freben it seems to be related to the buildLogTransport someway, described very well in the #20968. We forked the techdocs backend plugin completely to be able to override this (until we have support for #24587 or some other proper fix) with the following:

 const buildLogTransport = {
          pipe: () => {},
          log: () => {},
          on: () => {},
          emit: () => {},
          write: () => {},
          once: () => {},
          _writableState: {
            objectMode: true,
          },
        };

After adding that to the router options, the error is now gone and memory usage seems to be stable. Probably this needs a bit closer look why is this adding so many listeners to the socket..

Additionally this is related to new backend system update we did quite lately and we had this workaround already in use in the old backend.

[freben]

I'm a bit surprised then - in the current iteration of the code after #20700, no fallback is made if buildLogTransport is not passed in. But the stream transport IS, no matter what. Does this keep happening if you do NOT specify a buildLogTransport at all?

[drodil]

Yeah it does. Passing this custom transport seems the only way to fix this