Kafka circulars

[dakota002]

Description

Adds a table-streams event for Circulars over Kafka

This is currently a draft as there is either an issue in a dependency (possibly in cpp-zst-node16-fork or bindings) or in how we are building out these table-streams functions that is causing errors.

Currently investigating how cpp-zst-node16-fork uses the bindings package and why it works in some projects and not others

Related Issue(s)

Resolves #686

Testing

To get this to work in local dev setting there are a few things that need to be modified under node_modules:

• @remix-run/dev/dist/devServer_unstable/index.js -> line 139 sets stdio: pipe, change this to

      stdin: process.stdin,
      stdout: process.stdout,

in order to use the plugin-lambda-invoker. Relates to remix-run/remix#9293

• Add the following lines to the try array defined on the defaults object in bindings/bindings.js (currently trying to resolve this):

// Not needed anymore
      ['module_root', 'node_modules', 'cppzst-node-16-fork', 'build', 'Debug', 'bindings'],
      ['module_root', 'node_modules', 'cppzst-node-16-fork', 'build', 'Release', 'bindings'],

Once those are in place, test by:

• Start a basic kafka consumer pointed to the dev server
• press i to trigger the invoker
• select @tables-streams circulars-kafka-distribution

Edit: I may have resolved the zstd issue

[lpsinger]

The build job is failing with C++ compilation errors when building cpp-zst-node16-fork. Based on the compilation errors, I suspect that fixing it might be as simple as adding -std=c++11 to the compiler flags, though I don't know how to do that with node-gyp off the top of my head.

As for the binding path issues, the fundamental problem is that this Zstandard compressor/decompressor library contains a native Node addon which is our bundler (esbuild) does not know how to handle. I think it should be sufficient to copy the native addons to the appropriate build output directory in https://github.com/nasa-gcn/gcn.nasa.gov/blob/main/esbuild.js.

However, given the installation challenges, it may be worthwhile checking if there is a pure JavaScript or a WASM implementation of Zstandard that we could use instead, which would be trivial to install and bundle/tree-shake correctly.

[dakota002]

I think I got it working. The esbuild solution was a much easier find than the previous issues I was encountering. I adapted this https://esbuild.github.io/content-types/#file based on a comment I came across somewhere.

Edit: I am unsure about how this will function when deployed, there is a generated file that may need to be in a different directory than where it is currently getting placed

[dakota002]

Marked as ready too soon, need to fix something else first

[lpsinger]

Rather than add a new Lambda, I would rather that we use the existing Lambda in app/table-streams/circulars. Look at app/scheduled/circulars for ideas about how to structure it. I imagine something roughly like:

// app/table-streams/circulars/actions/search.ts
export async function searchAction(...) {
  ...
}

// app/table-streams/circulars/actions/kafka
export async function kafkaAction(...) {
}

// app/table-streams/circulars/index.ts
import { kafkaAction } from './actions/kafka'
import { searchAction } from './actions/search'
import { createTriggerHandler } from '~/lib/lambdaTrigger.server'

export const handler = createTriggerHandler(
  async (record: DynamoDBRecord) => {
    Promise.all([kafkaAction, searchAction].map((action) => action(record)))
  }
)

[lpsinger]

Please close the Kafka connection only when the Lambda runtime is terminating. That way we can amortize the cost of opening the connection across multiple Lambda invocations, only paying it once per cold start.

[dakota002]

Is this best done by not calling the disconnect?

[lpsinger]

You probably should call disconnect when the Lambda runtime is terminating gracefully. For Python Lambdas, AWS sends a SIGTERM signal to the function. The documentation doesn't say that it does that for Node.js Lamdbas, but it probably does.

[dakota002]

This might be a useful example, I will see if I can implement something similar

[dakota002]

Actually I am not sure if this will work as we think. Without the disconnect, the function hangs and throws an error

I'll do some more testing though to be sure

[dakota002]

Okay, I have something in the latest commit that should work. The listener gets added in, but locally there is no SIGTERM in sandbox. I'm not 100% confident in this implementation, but it should work

[lpsinger]

Does this work now?

[lpsinger]

@dakota002, does this work now?

Note that you will need #2296 so that our build pipeline installs libraries for the correct OS and CPU.

[dakota002]

@dakota002, does this work now?

Note that you will need #2296 so that our build pipeline installs libraries for the correct OS and CPU.

I need to commit some of my local changes and run some testing on this, but I will let you know shortly! It was working as expected before these updates too

[dakota002]

@lpsinger This still doesn't disconnect locally. It may work when deployed though since there will be different emitted events and not just an invoked function

[lpsinger]

@lpsinger This still doesn't disconnect locally. It may work when deployed though since there will be different emitted events and not just an invoked function

Does it disconnect locally if you hook into the SIGTERM signal?