Unexpected token 'export' from formdata-node when using openai/shims/node

[deedubFICTIV]

Confirm this is a Node library issue and not an underlying OpenAI API issue

• This is an issue with the Node library

Describe the bug

Issues experienced while running unit tests in our nodejs backend.
I first encountered the fetch is not defined bug, which led me to #304 and was able to move past the issue with import 'openai/shims/node'

But now I'm receiving the following errors:

    Details:

    /Users/person/Developer/repos/my-apis/node_modules/formdata-node/lib/esm/index.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){export * from "./FormData.js";
                                                                                      ^^^^^^

    SyntaxError: Unexpected token 'export'

      4 | import 'formdata-polyfill'
      5 | import 'whatwg-fetch'
    > 6 | import 'openai/shims/node'
        | ^
      7 |
      8 | import joi from 'joi'
      9 | import OpenAI from 'openai'

      at Runtime.createScriptFromCode (node_modules/jest-config/node_modules/jest-runtime/build/index.js:1505:14)
      at Object.<anonymous> (node_modules/openai/src/_shims/node-runtime.ts:5:1)
      at Object.<anonymous> (node_modules/openai/src/shims/node.ts:4:1)

I've also added

/**
 * @jest-environment jsdom
 */
import 'formdata-polyfill'
import 'whatwg-fetch'

But I'll still get the same SyntaxError: Unexpected token 'export' error.

I've updated the transformIgnorePatterns value to include "/node_modules/(?!formdata-node)" as well as "/node_modules/(?!openai/node/shims)" and many combinations. Still no luck.

yarn v1.22.4
typescript v5.1.6
jest v29.7.0

To Reproduce

yarn add openai

I've been able to reproduce it with as little as just the imports.
Add the following imports to a file that has unit tests (or a logic/utils file used by a file being tested)

import 'openai/shims/node'

import OpenAI from 'openai'

run the tests on that file

Code snippets

No response

OS

macOS

Node version

Node v16.20.0 (also tested with v18.19.0 same results)

Library version

openai v4.20.0

[deedubFICTIV]

Updating to v3.3.0 got me past the issues I was having.

I'll leave this open in case someone wants to look into why those issues were experienced with v4.20.0, but for those stuck where I was, v3.3.0 will likely be what you need

[rattrayalex]

Can you share tsconfig, package.json, and jest config?

If you are using @jest-environment jsdom, there's a good chance that you actually want to use import 'openai/shims/node' and add a global fetch polyfill, for example with undici, instead. But it depends on your setup.

Note that v3.3.0 has many downsides, like no auto-retry, outdated/incorrect types, no support for streaming, etc.

[deedubFICTIV]

I did use openai/shims/node which got me past the fetch error.

Here's the tsconfig.json file:

// tsconfig.json
{
    "compilerOptions": {
        /* Basic Options */
        //"incremental": true,                   /* Enable incremental compilation */
        "target": "es2019" /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017', 'ES2018', 'ES2019' or 'ESNEXT'. */,
        "module": "commonjs" /* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', or 'ESNext'. */,
        // "lib": [],                             /* Specify library files to be included in the compilation. */
        "allowJs": true /* Allow javascript files to be compiled. */,
        "checkJs": false /* Report errors in .js files. */,
        "jsx": "react" /* Specify JSX code generation: 'preserve', 'react-native', or 'react'. */,
        "declaration": true /* Generates corresponding '.d.ts' file. */,
        "declarationMap": true /* Generates a sourcemap for each corresponding '.d.ts' file. */,
        "sourceMap": true /* Generates corresponding '.map' file. */,
        // "outFile": "./",                       /* Concatenate and emit output to single file. */
        // "outDir": "./dist",                        /* Redirect output structure to the directory. */
        "rootDir": "." /* Specify the root directory of input files. Use to control the output directory structure with --outDir. */,
        "composite": true /* Enable project compilation */,
        // "tsBuildInfoFile": "./",               /* Specify file to store incremental compilation information */
        // "removeComments": true,                /* Do not emit comments to output. */
        // "noEmit": true,                        /* Do not emit outputs. */
        // "importHelpers": true,                 /* Import emit helpers from 'tslib'. */
        // "downlevelIteration": true,            /* Provide full support for iterables in 'for-of', spread, and destructuring when targeting 'ES5' or 'ES3'. */
        // "isolatedModules": true,               /* Transpile each file as a separate module (similar to 'ts.transpileModule'). */

        /* Strict Type-Checking Options */
        // "strict": true,                           /* Enable all strict type-checking options. */
        // "noImplicitAny": true,                 /* Raise error on expressions and declarations with an implied 'any' type. */
        "strictNullChecks": true /* Enable strict null checks. */,
        "strictFunctionTypes": true /* Enable strict checking of function types. */,
        "strictBindCallApply": true /* Enable strict 'bind', 'call', and 'apply' methods on functions. */,
        "strictPropertyInitialization": true /* Enable strict checking of property initialization in classes. */,
        "noImplicitThis": true /* Raise error on 'this' expressions with an implied 'any' type. */,
        "alwaysStrict": true /* Parse in strict mode and emit "use strict" for each source file. */,

        /* Additional Checks */
        // "noUnusedLocals": true,                /* Report errors on unused locals. */
        // "noUnusedParameters": true,            /* Report errors on unused parameters. */
        // "noImplicitReturns": true,             /* Report error when not all code paths in function return a value. */
        // "noFallthroughCasesInSwitch": true,    /* Report errors for fallthrough cases in switch statement. */

        /* Module Resolution Options */
        "moduleResolution": "node" /* Specify module resolution strategy: 'node' (Node.js) or 'classic' (TypeScript pre-1.6). */,
        "baseUrl": "./" /* Base directory to resolve non-absolute module names. */,
        "paths": {
            /* A series of entries which re-map imports to lookup locations relative to the 'baseUrl'. */
            "@fictiv/*": ["./packages/*"]
        },
        // "rootDirs": [],                        /* List of root folders whose combined content represents the structure of the project at runtime. */
        "typeRoots": [
            "node_modules/@types"
        ] /* List of folders to include type definitions from. */,
        // "types": [],                           /* Type declaration files to be included in compilation. */
        "allowSyntheticDefaultImports": true /* Allow default imports from modules with no default export. This does not affect code emit, just typechecking. */,
        "esModuleInterop": true /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */,
        // "preserveSymlinks": true,              /* Do not resolve the real path of symlinks. */
        // "allowUmdGlobalAccess": true,          /* Allow accessing UMD globals from modules. */

        /* Source Map Options */
        // "sourceRoot": "",                      /* Specify the location where debugger should locate TypeScript files instead of source locations. */
        // "mapRoot": "",                         /* Specify the location where debugger should locate map files instead of generated locations. */
        // "inlineSourceMap": true                /* Emit a single file with source maps instead of having a separate file. */,
        // "inlineSources": true,                 /* Emit the source alongside the sourcemaps within a single file; requires '--inlineSourceMap' or '--sourceMap' to be set. */

        /* Experimental Options */
        // "experimentalDecorators": true,        /* Enables experimental support for ES7 decorators. */
        // "emitDecoratorMetadata": true,         /* Enables experimental support for emitting type metadata for decorators. */

        /* Advanced Options */
        "forceConsistentCasingInFileNames": true /* Disallow inconsistently-cased references to the same file. */,
        "resolveJsonModule": true,
        "skipLibCheck": true
    },
    "exclude": [
        "**/node_modules/**/*",
        "**/dist/**/*",
        "./_templates/**/*",
        "./bin/**",
        "./coverage/**",
        "./eslint-custom-rules/**",
        "./babel.config.js"
    ]
}

And here's package.json which includes the jest config (I removed scripts and other private info)

// package.json
{
    "version": "1.0.0",
    "license": "UNLICENSED",
    "private": true,
    "workspaces": [
        "packages/*",
        "services/*"
    ],
    "jest": {
        "preset": "ts-jest",
        "collectCoverageFrom": [
            "services/**/*.{js,ts,tsx}",
            "packages/**/*.{js,ts,tsx}"
        ],
        "transform": {
            "^.+\\.(ts|tsx)$": [
                "ts-jest",
                {
                    "isolatedModules": true
                }
            ]
        },
        "coveragePathIgnorePatterns": [
            "/node_modules/",
            "/db/",
            "/dist/",
            "__mocks__",
            "__tests__",
            "test-helpers/",
            "/.build/",
            "packages/static-data/"
        ],
        "moduleNameMapper": {
            "~/graphql-server/(.*)$": "<rootDir>/services/graphql-server/src/$1",
            "~/zip-file-service/(.*)$": "<rootDir>/services/zip-file-service/src/$1",
            "~/zapier-events/(.*)$": "<rootDir>/services/zapier-events/src/$1",
            "~/kinesis-monitoring/(.*)$": "<rootDir>/services/kinesis-monitoring/src/$1",
            "^uuid$": "uuid"
        },
        "transformIgnorePatterns": [
            "/node_modules/(?!jest-when|formdata-node)"
        ],
        "testPathIgnorePatterns": [
            "<rootDir>/dist/",
            "/dist/",
            "/node_modules/",
            "/.build/",
            "<rootDir>/*/.*/__tests__/test-helpers"
        ],
        "moduleDirectories": [
            "node_modules",
            "src"
        ],
        "moduleFileExtensions": [
            "ts",
            "tsx",
            "js",
            "jsx",
            "json"
        ],
        "modulePathIgnorePatterns": [
            "<rootDir>/services/src/*/.*/__mocks__",
            "/dist/"
        ],
        "testEnvironment": "node",
        "setupFilesAfterEnv": [
            "<rootDir>/packages/lib-test-helpers/src/unitTestSetup.js"
        ],
        "reporters": [
            "default",
            "jest-junit"
        ],
        "coverageReporters": [
            "text",
            "html"
        ],
        "snapshotSerializers": [
            "<rootDir>/node_modules/@whoaa/jest-mock-knex/serializer"
        ],
        "roots": [
            "<rootDir>",
            "<rootDir>/packages/",
            "<rootDir>/services/"
        ],
        "testTimeout": 300000
    },
    "jest-junit": {
        "outputDirectory": "./reports",
        "outputName": "jest.xml"
    },
    "devDependencies": {
        "@aws-sdk/client-s3": "~3.379.1",
        "@pdftron/pdfnet-node": "^9.2.0",
        "@types/aws-lambda": "^8.10.85",
        "@types/bunyan": "^1.8.6",
        "@types/express": "^4.17.2",
        "@types/express-serve-static-core": "^4.17.1",
        "@types/jest": "~27.4.1",
        "@types/jsforce": "^1.11.0",
        "@types/jsonwebtoken": "^7.2.7",
        "@types/lodash": "^4.14.133",
        "@types/node": "~13.7.0",
        "@types/puppeteer": "~5.4.3",
        "@typescript-eslint/eslint-plugin": "^5.61.0",
        "@typescript-eslint/parser": "^5.61.0",
        "@whoaa/jest-mock-knex": "^1.20.0",
        "add": "^2.0.6",
        "apollo-cache-inmemory": "^1.1.12",
        "apollo-client": "^2.2.8",
        "apollo-link": "^1.2.1",
        "apollo-link-http": "^1.5.3",
        "aws-sdk": "~2.1374.0",
        "brfs": "^2.0.2",
        "chalk": "^2.4.1",
        "concurrently": "^6.5.1",
        "copy-webpack-plugin": "^9.0.0",
        "db-migrate": "^0.11.13",
        "db-migrate-pg": "^1.2.2",
        "delay": "^4.3.0",
        "dogapi": "^2.8.4",
        "eslint": "~8.47.0",
        "eslint-config-prettier": "~8.5.0",
        "eslint-import-resolver-custom-alias": "~1.3.2",
        "eslint-plugin-fictiv-apis": "file:./eslint-custom-rules",
        "eslint-plugin-import": "~2.28.1",
        "eslint-plugin-jest": "~27.2.3",
        "eslint-plugin-no-only-tests": "~3.1.0",
        "eslint-plugin-node": "~11.1.0",
        "eslint-plugin-prettier": "~5.0.0",
        "eslint-plugin-react": "~7.33.2",
        "eslint-plugin-simple-import-sort": "~10.0.0",
        "expect-more-jest": "~5.5.0",
        "graphql": "^15.5.1",
        "graphql-fields": "^1.0.2",
        "graphql-tools": "^4.0.7",
        "jest": "~29.7.0",
        "jest-junit": "~16.0.0",
        "jest-when": "~3.5.1",
        "js-yaml": "~4.1.0",
        "json-loader": "^0.5.4",
        "lerna": "~6.6.2",
        "mock-knex": "~0.4.10",
        "mockdate": "^3.0.2",
        "node-loader": "^2.0.0",
        "node-loader-relative": "^0.0.3",
        "nodemon": "^3.0.1",
        "nope-sorry": "^1.0.0",
        "npm-conf": "^1.1.3",
        "prettier": "^3.0.2",
        "quick-lru": "^4.0.1",
        "raw-loader": "^0.5.1",
        "rimraf": "^3.0.1",
        "serverless": "~3.34.0",
        "serverless-http": "~3.2.0",
        "serverless-offline": "~12.0.4",
        "serverless-package-python-functions": "^0.7.0",
        "serverless-plugin-git-variables": "^5.2.0",
        "serverless-webpack": "~5.13.0",
        "stripe": "^12.6.0",
        "supertest": "^6.3.3",
        "transform-loader": "^0.2.4",
        "ts-jest": "^29.1.1",
        "ts-loader": "^9.2.6",
        "ts-node": "^10.9.1",
        "typescript": "~5.1.6",
        "utility-types": "^3.10.0",
        "webpack": "~5.82.0",
        "webpack-node-externals": "~3.0.0",
        "yaml-jest": "~1.2.0",
        "yargs": "^17.3.1",
        "yarn": "^1.21.1",
        "yml-loader": "^2.1.0"
    },
    "resolutions": {
        "pg": "^8.11.1",
        "brfs": "^2.0.2"
    },
    "dependencies": {
        "@opensearch-project/opensearch": "^2.0.0",
        "@easypost/api": "^5.10.1",
        "ajv": "^6.12.3",
        "analytics-node": "6.2.0",
        "apollo-server-express": "^3.12.0",
        "aws-sdk": "~2.1374.0",
        "aws-opensearch-connector": "^1.1.0",
        "axios": "~1.6.2",
        "base64-url": "^2.0.0",
        "bluebird": "^3.5.0",
        "bunyan": "^1.8.10",
        "bunyan-middleware": "^1.0.1",
        "cli": "^1.0.1",
        "cognito-express": "~3.0.2",
        "cookie-parser": "1.4.5",
        "cors": "^2.8.3",
        "dataloader": "^1.4.0",
        "decimal.js": "^7.2.0",
        "dotenv": "^4.0.0",
        "dynamodb-data-types": "^3.0.0",
        "dynamodb-migrations": "^0.0.10",
        "express": "^4.18.2",
        "express-jwt-authz": "^1.0.0",
        "graphql": "^15.5.1",
        "graphql-tools": "^4.0.7",
        "graphql-apollo-errors": "^2.0.3",
        "graphql-custom-types": "^1.5.1",
        "graphql-extensions": "^0.10.10",
        "graphql-resolve-batch": "^1.0.2",
        "graphql-type-json": "^0.3.1",
        "helmet": "^4.1.1",
        "i18n-iso-countries": "6.0.0",
        "is-uuid": "^1.0.2",
        "joi": "17.2.1",
        "jsforce": "^1.11.1",
        "jsonwebtoken": "9.0.0",
        "jsx-pdf": "whoaa512/jsx-pdf#58feebf",
        "knex": "^0.21.21",
        "libphonenumber-js": "^1.9.44",
        "lodash": "^4.17.21",
        "moment": "~2.29.4",
        "moment-timezone": "~0.5.37",
        "nocache": "^2.1.0",
        "odoo-connect": "https://github.com/fictiv/odoo-connect.git#v1.3.0",
        "openai": "3.3.0",
        "p-iteration": "^1.1.8",
        "p-map": "^2.1.0",
        "p-props": "^2.0.0",
        "p-reduce": "^1.0.0",
        "pdfmake": "^0.2.7",
        "pdf-merger-js": "^4.3.0",
        "pg": "^8.11.1",
        "postal-codes-js": "2.4.0",
        "psl": "1.8.0",
        "raven": "^2.6.0",
        "@sentry/node": "5.15.4",
        "source-map-support": "^0.4.15",
        "stripe": "^8.154.0",
        "url-join": "^4.0.0",
        "uuid": "^2.0.1",
        "jwks-rsa": "^2.0.5",
        "json2csv": "6.0.0-alpha.2",
        "qrcode": "^1.5.1",
        "@sideway/address": "4.1.2",
        "xml-js": "1.6.11",
        "xml2js": "~0.6.2",
        "vwo-node-sdk": "^1.62.0",
        "whatwg-fetch": "3.6.19",
        "tough-cookie": "^4.1.3",
        "request": "^2.88.2"
    }
}

[rattrayalex]

Ah I'm so sorry, I meant openai/shims/web not openai/shims/node in that case.

[deedubFICTIV]

To be clear, this was being used in a backend nodeJS environment. Would I still want to use the web shim?

[rattrayalex]

Yes, you can as long as you have access to the global fetch function – it'll just use that. However, file uploads will require you to also polyfill FormData, File, and other classes – I think undici exposes these.

[RobertHaslinger]

I first encountered the fetch is not defined bug, which led me to #304 and was able to move past the issue with import 'openai/shims/node'

I also encountered the same problem with:

• node: 20.10.0
• openai: 4.24.1
• jest: 29.4.2
• testEnvironment: 'node'

I managed to get the tests working by first removing the line import 'openai/shims/node' again.
Then, to fix the "ReferenceError: fetch is not defined" error, I did the following:

1. Install undici as devDependency (6.2.1 in my case)
2. Add the following to a jest.setup.js file (see ReferenceError: ReadableStream is not defined mswjs/msw#1934 (comment)):

//jest.setup.js
const { TextDecoder, TextEncoder, ReadableStream } = require("node:util")

Object.defineProperties(globalThis, {
    TextDecoder: { value: TextDecoder },
    TextEncoder: { value: TextEncoder },
    ReadableStream: { value: ReadableStream },
})

const { Blob, File } = require("node:buffer")
const { fetch, Headers, FormData, Request, Response } = require("undici")

Object.defineProperties(globalThis, {
    fetch: { value: fetch, writable: true },
    Blob: { value: Blob },
    File: { value: File },
    Headers: { value: Headers },
    FormData: { value: FormData },
    Request: { value: Request },
    Response: { value: Response },
})

3. Reference this setup in the jest.config.js:

//jest.config.js
module.exports = {
    setupFilesAfterEnv: ['./jest.setup.js'],
    ...other config
}

[Fiattarone]

Right on @RobertHaslinger, nailed it. I just needed Request and Response in my case, but great fix.