node-fetch v2 does not work but node-fetch v3 works

[sibelius]

Reproduction

I'm doing the same request using node-fetch v2 and node-fetch v3

here are the response headers for v2

{
date: [ 'Tue, 27 Sep 2022 10:56:44 GMT' ],
    server: [ 'Apache' ],
    'content-length': [ '318' ],
    connection: [ 'close' ],
    'content-type': [ 'text/html; charset=iso-8859-1' ]
    }

here are the response headers for v3

{
    'cache-control': [ 'no-store, no-cache, must-revalidate' ],
    connection: [ 'close' ],
    'content-encoding': [ 'gzip' ],
    'content-length': [ '60' ],
    'content-type': [ 'text/html; charset=UTF-8' ],
    date: [ 'Tue, 27 Sep 2022 10:52:20 GMT' ],
    expires: [ 'Thu, 19 Nov 1981 08:52:00 GMT' ],
    pragma: [ 'no-cache' ],
    server: [ 'Apache' ],
    'set-cookie': [
      'frontend=593da7b9ba1979630c43c1a9084c0163; expires=Tue, 27-Sep-2022 11:52:20 GMT; Max-Age=3600; path=/; domain=www.example.com; HttpOnly',
      'frontend_cid=l9Z7tarPrOb8YEM5; expires=Tue, 27-Sep-2022 11:52:20 GMT; Max-Age=3600; path=/; domain=www.example.com; secure; HttpOnly'
    ],
    vary: [ 'Accept-Encoding' ],
    'x-content-type-options': [ 'nosniff' ],
    'x-xss-protection': [ '1; mode=block' ]
  }

Expected behavior

it should work on both node-fetch v2 and v3.
I can't upgrade because ESM breaks my project and many packages

Screenshots

I can share the script that I'm using to test, ping me on twitter https://twitter.com/sseraphini, it has some sensitive data

Your Environment

software	version
node-fetch	2.6.7 and 3.2.10
node	16.17.0
npm	8.15.0
Operating System	macosx

Additional context

it also works with native node 18 fetch implementation

the request headers for node fetch v2

{
accept: [
      'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
    ],
    'accept-language': [ 'en-US,en;q=0.9,pt;q=0.8,und;q=0.7,fr;q=0.6' ],
    'cache-control': [ 'max-age=0' ],
    'sec-ch-ua': [
      '"Google Chrome";v="105", "Not)A;Brand";v="8", "Chromium";v="105"'
    ],
    'sec-ch-ua-mobile': [ '?0' ],
    'sec-ch-ua-platform': [ '"macOS"' ],
    'sec-fetch-dest': [ 'document' ],
    'sec-fetch-mode': [ 'navigate' ],
    'sec-fetch-site': [ 'none' ],
    'sec-fetch-user': [ '?1' ],
    'upgrade-insecure-requests': [ '1' ],
    'User-Agent': [ 'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)' ],
    'Accept-Encoding': [ 'gzip,deflate' ],
    Connection: [ 'close' ]
}

{
    accept: 'application/json',
    'accept-encoding': 'gzip, deflate, br',
    authorization: 'sexylovepix',
    connection: 'close',
    'content-length': '68',
    'content-type': 'application/json',
    'user-agent': 'node-fetch',
  },

[jimmywarting]

without going into depth of what headers are being sent & received on v2 vs v3 and the differences, you say it works with native node 18 fetch implementation... Are there any particular reason why you can't use that instead?
no dependency === better IMO.

All doe... NodeJS fetch impl. still lacks formdata decoding (.formdata()) and allowing you to create file/blobs backed up by the file system... on the plus side undici (NodeJS) is accepting 3th party blobs like fetch-blob as well.

Also is there any reason why you can't lazy import('node-fetch') from commonjs? ESM works in commonjs too.
There are a few solution covered over at: #1279

---

I also built this lazy-resolver a long time a go.
The magic of it is that you can remove all those promise chains and get rid of all those .then(dothis) callbacks so it looks more like a sync operation.

const fetch = resolve(import('node-fetch')).default

fetch('https://httpbin.org/get?items=4&items=2')
  .json()
  .args
  .items
  .map(n => ~~n * 4)
  .forEach(n => console.log(n))

[jimmywarting]

yea. you can't use require() on a esm only module...

try this instead:

const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args));

[sibelius]

the problem is that I can't fix @sentry/cli

[sibelius]

another problem is that we bundle our code using @webpack, and it transforms all import to require

[jimmywarting]

the problem is that I can't fix @sentry/cli

Have you tried sending them a PR?

In other case a solution like this could help to prevent bundlers to rewrite your import statement back to require

const _importDynamic = new Function('modulePath', 'return import(modulePath)')

async function fetch(...args) {
  const {default: fetch} = await _importDynamic('node-fetch'))
  return fetch(...args))
}

We haven't seen any reason as to why you should be bundle server side stuff.

[sibelius]

we bundle backend to create a smaller lambda code for serveless

it tree shaking a lot of code

we go from 2gb to 20mb

[jimmywarting]

It almost looks like you are sending/receiving different request/response header depending on if you are using v2 vs v3

Maybe there is some details in the response body that can clarify what is wrong with the request?
sec-fetch-xxx looks like a bunch of headers being made by the browser?

[jimmywarting]

i figured out that they do send out different response depending on what user-agent you are using...
it seems as if they have banned node-fetch/* as a bad bot or something.

if i send a 'user-agent': 'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)' then i get a 403 Forbidden text/html response
(it's not even a json response so calling await res.json() will throw a error)

In v3 we changed the user agent to just be: node-fetch to not leak as much information about what version we are using

but if i send 'user-agent': 'random' then i get back 400 Bad Request (saying that i'm just missing some credentials.
{"error":"Invalid Request","description":"Invalid authorization header: ","field":"authorization header"}

So I believe that changing the user-agent to some web ish agent will work OK.

(don't use the others api for ill intention)

[sibelius]

that's a great debug

thanks for the help

[LinusU]

I can recommend the following user agent:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36

some background: Automattic/node-canvas#2104 (comment)

@jimmywarting probably too late to do in version 2 or 3 now, but maybe we want to change to that one as a static string for v4?

[jimmywarting]

v3 have a more static agent

node-fetch/src/request.js

Line 278 in 6f72caa

headers.set('User-Agent', 'node-fetch');