feat(openai): support streamed responses for completions and chat completions

[sabrenner]

What does this PR do?

Adds support for streamed responses for chat.completions and legacy completions calls. There are a few callouts for this PR:

1. Implementation detail: For openai versions >=4.1 <4.13, the async iterator used for generating streamed responses returns Buffer objects from the next invocation. This is only an issue because some of these buffers contain half a chunk (ie data: {id: '...', choices: [{ delta: 'hi, this is a, and following chunks contain the rest. Similarly, some chunks will also contain two object chunks in one buffer. To handle this, I wrote a small helper to take all of the buffers, concatenate them, and process them as specified in the OpenAI API documentation (which are just SSE, so they all start with data: , and the last element is data: [DONE]). For openai versions >=4.13.0, the elements returned from the next invocation are plain JSON objects.
2. Testing: The only way I figured out to stream responses using nock was through fs.createReadStream with a data file. If there's a more efficient way of doing this, please let me know! But, I think this should be sufficient enough for the handful of tests this change needs.

Motivation

Adds missing support for streamed responses for chat.completions and legacy completions.

More OpenAI Follow-Ups

1. General refactor of the integration, to make it and its tests cleaner and easier to read and contribute to.

Plugin Checklist

• Unit tests.
• TypeScript definitions.
• TypeScript tests.
• API documentation.
• CircleCI jobs/workflows.
• Plugin is exported.

Additional Notes

[github-actions]

Overall package size

Self size: 6.53 MB
Deduped: 60.59 MB
No deduping: 60.87 MB

Dependency sizes

name	version	self size	total size
@datadog/native-iast-taint-tracking	2.1.0	14.91 MB	14.92 MB
@datadog/native-appsec	7.1.1	14.39 MB	14.4 MB
@datadog/pprof	5.3.0	9.85 MB	10.22 MB
protobufjs	7.2.5	2.77 MB	6.56 MB
@datadog/native-iast-rewriter	2.3.1	2.15 MB	2.24 MB
@opentelemetry/core	1.14.0	872.87 kB	1.47 MB
@datadog/native-metrics	2.0.0	898.77 kB	1.3 MB
@opentelemetry/api	1.8.0	1.21 MB	1.21 MB
import-in-the-middle	1.7.4	70.19 kB	739.86 kB
msgpack-lite	0.1.26	201.16 kB	281.59 kB
opentracing	0.14.7	194.81 kB	194.81 kB
semver	7.5.4	93.4 kB	123.8 kB
pprof-format	2.1.0	111.69 kB	111.69 kB
@datadog/sketches-js	2.1.0	109.9 kB	109.9 kB
lodash.sortby	4.7.0	75.76 kB	75.76 kB
lru-cache	7.14.0	74.95 kB	74.95 kB
ignore	5.2.4	51.22 kB	51.22 kB
int64-buffer	0.1.10	49.18 kB	49.18 kB
shell-quote	1.8.1	44.96 kB	44.96 kB
istanbul-lib-coverage	3.2.0	29.34 kB	29.34 kB
tlhunter-sorted-set	0.1.0	24.94 kB	24.94 kB
limiter	1.1.5	23.17 kB	23.17 kB
dc-polyfill	0.1.4	23.1 kB	23.1 kB
retry	0.13.1	18.85 kB	18.85 kB
jest-docblock	29.7.0	8.99 kB	12.76 kB
crypto-randomuuid	1.0.0	11.18 kB	11.18 kB
path-to-regexp	0.1.7	6.78 kB	6.78 kB
koalas	1.0.2	6.47 kB	6.47 kB
module-details-from-path	1.0.3	4.47 kB	4.47 kB

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.20%. Comparing base (3db62e3) to head (cc4352a).
Report is 2 commits behind head on master.

❗ Current head cc4352a differs from pull request most recent head 71bf20c

Please upload reports for the commit 71bf20c to get more accurate results.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4308      +/-   ##
==========================================
- Coverage   85.94%   82.20%   -3.75%     
==========================================
  Files         117      245     +128     
  Lines        4384    10350    +5966     
  Branches       33       33              
==========================================
+ Hits         3768     8508    +4740     
- Misses        616     1842    +1226     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-05-24 18:31:48

Comparing candidate commit 71bf20c in PR branch sabrenner/openai-streaming with baseline commit 95b5a41 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 259 metrics, 7 unstable metrics.

[tlhunter]

Is this literally the streaming protocol that OpenAi uses?

[jjxct]

yes, for the older versions