Build fails after with "Unexpected token" in async-requires after upgrade to 2.0.84

[dlindahl]

Description

Running rm -rf .cache && rm -rf public && gatsby-build works fine under 2.0.83. After upgrading to 2.0.84, an error is thrown, aborting the build.

Steps to reproduce

The only I am taking is running gatsby build. After downgrading back to 2.0.83, the issue disappears.

Expected result

The build should complete successfully

Actual result

The build quits with the following error:

success onPostBootstrap — 0.202 s

info bootstrap finished - 6.171 s


error Generating JavaScript bundles failed


  Error: ./.cache/async-requires.js 8:11
  Module parse failed: Unexpected token (8:11)
  You may need an appropriate loader to handle this file type.
  | exports.components = {
  |   "component---node-modules-gatsby-plugin-offline-app-shell-js": function componentNodeModulesGatsbyPluginOfflineAppShellJs() {
  >     return import("/Users/dereklindahl/Work/APP/node_modules/gatsby-plugin-offline/app-shell.js"
  |     /* webpackChunkName: "component---node-modules-gatsby-plugin-offline-app-shell-js" */
  |     );
   @ ./.cache/production-app.js 18:0-45 21:23-36 26:23-36

Environment

npx gatsby info -- --clipboard                 

  System:
    OS: macOS High Sierra 10.13.6
    CPU: (4) x64 Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
    Shell: 5.4.2 - /usr/local/bin/zsh
  Binaries:
    Node: 10.14.0 - ~/.nodenv/versions/10.14.0/bin/node
    Yarn: 1.12.3 - /usr/local/bin/yarn
    npm: 6.4.1 - ~/.nodenv/versions/10.14.0/bin/npm
  Languages:
    Python: 2.7.10 - /usr/bin/python
  Browsers:
    Chrome: 71.0.3578.98
    Firefox: 63.0.3
    Safari: 12.0.2
  npmPackages:
    gatsby: 2.0.84 => 2.0.84 
    gatsby-image: ^2.0.25 => 2.0.25 
    gatsby-mdx: ^0.2.0 => 0.2.0 
    gatsby-plugin-canonical-urls: ^2.0.8 => 2.0.8 
    gatsby-plugin-google-analytics: ^2.0.8 => 2.0.8 
    gatsby-plugin-manifest: ^2.0.13 => 2.0.13 
    gatsby-plugin-netlify: ^2.0.6 => 2.0.6 
    gatsby-plugin-netlify-cache: ^1.0.0 => 1.0.0 
    gatsby-plugin-offline: ^2.0.21 => 2.0.21 
    gatsby-plugin-react-helmet: ^3.0.5 => 3.0.5 
    gatsby-plugin-sharp: ^2.0.17 => 2.0.17 
    gatsby-plugin-sitemap: ^2.0.4 => 2.0.4 
    gatsby-plugin-sri: ^1.0.4 => 1.0.4 
    gatsby-plugin-styled-components: ^3.0.4 => 3.0.4 
    gatsby-plugin-zopfli: ^1.0.2 => 1.0.2 
    gatsby-source-filesystem: ^2.0.12 => 2.0.12 
    gatsby-transformer-sharp: ^2.1.10 => 2.1.10 

I saw #10038 which sounded familiar, but my Webpack config is uninteresting:

// gatsby-node.js
exports.onCreateWebpackConfig = ({ actions }) => {
  actions.setWebpackConfig({
    module: {
      rules: [
        {
          test: /\.ogv$/,
          use: [
            {
              loader: require.resolve(`url-loader`),
              options: { limit: 10000, name: 'static/[name]-[hash].[ext]' }
            }
          ]
        }
      ]
    },
    resolve: {
      alias: {
        '@': path.resolve(__dirname, 'src/components')
      },
      modules: [path.resolve(__dirname, 'src'), 'node_modules']
    }
  })
}

And commenting that block out does not fix the error.

If I replace gatsby-plugin-offline with gatsby-plugin-remove-serviceworker, the problem remains, but with a different async import.

FWIW, I do not notice a difference in the contents of async-require.js in the 2.0.83 or 2.0.84 builds and upgrading gatsby-plugin-offline made no difference as well.

[ooloth]

I'm experiencing a similar error after upgrading from 2.0.62 to 2.0.91.

In my case, gatsby develop works fine, but gatsby build errors out on either the template.js page (if I include it) or 404.js (if I remove the createPages snippet from gatsby-node):

error Generating JavaScript bundles failed

Error: ./.cache/async-requires.js 8:11
  Module parse failed: Unexpected token (8:11)
  You may need an appropriate loader to handle this file type.
  | exports.components = {
  |   "component---src-templates-template-js": function componentSrcTemplatesTemplateJs() {
  >     return import("/Users/michael/Sites/projects/gatsby-starter/src/templates/template.js"
  |     /* webpackChunkName: "component---src-templates-template-js" */  |     );
   @ ./.cache/production-app.js 18:0-45 21:23-36 26:23-36

or

error Generating JavaScript bundles failed

Error: ./.cache/async-requires.js 8:11
  Module parse failed: Unexpected token (8:11)
  You may need an appropriate loader to handle this file type.
  | exports.components = {
  |   "component---src-pages-404-js": function componentSrcPages404Js()   {
  >     return import("/Users/michael/Sites/projects/gatsby-starter/src/pages/404.js"
  |     /* webpackChunkName: "component---src-pages-404-js" */
  |     );
   @ ./.cache/production-app.js 18:0-45 21:23-36 26:23-36

Everything worked fine before upgrading gatsby. 🤷‍♂️

[sidharthachatterjee]

@dlindahl @ooloth

Can you please link to minimal reproductions of this?

[magnusarinell]

I'm getting the same error using Gatsby v 2.0.55 where package-lock.json is disabled in .npmrc. The site is built from a clean workspace daily. One day it worked, the next it didn't. I'm guessing the error is related to some transient dependency which has changed.

[cyrildurand]

same issue when I upgrade gatsby from v2.0.91 to v2.0.93 (v2.0.92) doesn't exist

[madelyneriksen]

Hi, chiming in to say that I am currently also experiencing this issue, but am struggling to build a minimal reproduction for it.

Currently running an npm update on my repository does cause the build to fail, but it does not for my personal website.

I'll keep digging around until I can find what's causing it or someone else figures it out. If I can get it isolated cleanly I'll post back here.

Thanks!

[omrllm]

Same here!
I have typed npm update -g npm to get the npm 6.7.0 version and I have gatsby 2.0.98.

gatsby plugin offline version -> 2.0.21

Repository with this problem: there you go.

[DanielRuf]

Also can you list the installed dependencies with npm ls and run node --version?

[DanielRuf]

Bisecting might be helpful here too. I'll test the repo in a few minutes.

Ok same error here. I'll bisect this.

It seems it happens in all versions so it is a plugin or another dependency probably.
Will test further.

[DanielRuf]

I think I found the cause. I'll provide a patch and PR.

[DanielRuf]

@omrllm (a patch for gatsby 2.0.60)

patch-package
--- a/node_modules/gatsby/dist/internal-plugins/query-runner/pages-writer.js
+++ b/node_modules/gatsby/dist/internal-plugins/query-runner/pages-writer.js
@@ -86,9 +86,9 @@ const preferDefault = m => m && m.default || m
     let asyncRequires = `// prefer default export if available
 const preferDefault = m => m && m.default || m
 \n`;
-    asyncRequires += `exports.components = {\n${components.map(c => `  "${c.componentChunkName}": () => import("${(0, _path.joinPath)(c.component)}" /* webpackChunkName: "${c.componentChunkName}" */)`).join(`,\n`)}
+    asyncRequires += `exports.components = {\n${components.map(c => `  "${c.componentChunkName}": () => require("${(0, _path.joinPath)(c.component)}" /* webpackChunkName: "${c.componentChunkName}" */)`).join(`,\n`)}
 }\n\n`;
-    asyncRequires += `exports.data = () => import("${(0, _path.joinPath)(program.directory, `.cache`, `data.json`)}")\n\n`;
+    asyncRequires += `exports.data = () => require("${(0, _path.joinPath)(program.directory, `.cache`, `data.json`)}")\n\n`;
 
     const writeAndMove = (file, data) => {
       const destination = (0, _path.joinPath)(program.directory, `.cache`, file);

[DanielRuf]

By changing the import to require it should work. There is probably just a missing loader but why use import here anyway when the ESM way just produces issues and mixing exports with import is never a good idea.

[DanielRuf]

+ gatsby@2.0.98
+ gatsby-image@2.0.28
+ gatsby-plugin-google-analytics@2.0.10
+ gatsby-transformer-sharp@2.1.12
+ gatsby-plugin-sharp@2.0.18
+ gatsby-source-filesystem@2.0.17
added 9 packages from 3 contributors, removed 4 packages, updated 92 packages and audited 43569 packages in 200.269s

diff 23.cache/ .cache/
Only in 23.cache/: .sonarlint
Common subdirectories: 23.cache/__tests__ and .cache/__tests__
Common subdirectories: 23.cache/caches and .cache/caches
Common subdirectories: 23.cache/commonjs and .cache/commonjs
diff 23.cache/data.json .cache/data.json
1c1
< {"pages":[{"componentChunkName":"component---src-pages-index-js","jsonName":"index","path":"/"},{"componentChunkName":"component---src-pages-404-js","jsonName":"404-html-516","path":"/404.html"},{"componentChunkName":"component---src-pages-404-js","jsonName":"404-22d","path":"/404/"},{"componentChunkName":"component---src-pages-about-js","jsonName":"about-f34","path":"/about/"},{"componentChunkName":"component---src-pages-contact-js","jsonName":"contact-26a","path":"/contact/"}],"dataPaths":{"404-22d":"657/path---404-22-d-bce-yc2HAWbdDECy3NCKIhFOCg1lY8","404-html-516":"84/path---404-html-516-62a-yc2HAWbdDECy3NCKIhFOCg1lY8","about-f34":"691/path---about-f-34-4c2-WV9OHhcgC975Z2f0az9WK5Dpl0Y","contact-26a":"662/path---contact-26-a-cd9-SNoLKPyPsqQ59X6yAuuT79ALOJc","index":"612/path---index-6a9-j0JKW3rrllGOOtWKwyNn0ujHMk"}}
\ No newline at end of file
---
> {"pages":[{"componentChunkName":"component---src-pages-index-js","jsonName":"index","path":"/"},{"componentChunkName":"component---src-pages-404-js","jsonName":"404-html-516","path":"/404.html"},{"componentChunkName":"component---src-pages-404-js","jsonName":"404-22d","path":"/404/"},{"componentChunkName":"component---src-pages-about-js","jsonName":"about-f34","path":"/about/"},{"componentChunkName":"component---src-pages-contact-js","jsonName":"contact-26a","path":"/contact/"}],"dataPaths":{"404-22d":"657/path---404-22-d-bce-yc2HAWbdDECy3NCKIhFOCg1lY8","404-html-516":"84/path---404-html-516-62a-yc2HAWbdDECy3NCKIhFOCg1lY8","about-f34":"691/path---about-f-34-4c2-WV9OHhcgC975Z2f0az9WK5Dpl0Y","contact-26a":"662/path---contact-26-a-cd9-SNoLKPyPsqQ59X6yAuuT79ALOJc","index":"770/path---index-6a9-dVi4vZoL0B52PVt3C79b9kQk"}}
\ No newline at end of file
diff 23.cache/default-html.js .cache/default-html.js
4,31c4,29
< export default class HTML extends React.Component {
<   render() {
<     return (
<       <html {...this.props.htmlAttributes}>
<         <head>
<           <meta charSet="utf-8" />
<           <meta httpEquiv="x-ua-compatible" content="ie=edge" />
<           <meta
<             name="viewport"
<             content="width=device-width, initial-scale=1, shrink-to-fit=no"
<           />
<           {this.props.headComponents}
<         </head>
<         <body {...this.props.bodyAttributes}>
<           {this.props.preBodyComponents}
<           <noscript key="noscript" id="gatsby-noscript">
<             This app works best with JavaScript enabled.
<           </noscript>
<           <div
<             key={`body`}
<             id="___gatsby"
<             dangerouslySetInnerHTML={{ __html: this.props.body }}
<           />
<           {this.props.postBodyComponents}
<         </body>
<       </html>
<     )
<   }
---
> export default function HTML(props) {
>   return (
>     <html {...props.htmlAttributes}>
>       <head>
>         <meta charSet="utf-8" />
>         <meta httpEquiv="x-ua-compatible" content="ie=edge" />
>         <meta
>           name="viewport"
>           content="width=device-width, initial-scale=1, shrink-to-fit=no"
>         />
>         {props.headComponents}
>       </head>
>       <body {...props.bodyAttributes}>
>         {props.preBodyComponents}
>         <noscript key="noscript" id="gatsby-noscript">
>           This app works best with JavaScript enabled.
>         </noscript>
>         <div
>           key={`body`}
>           id="___gatsby"
>           dangerouslySetInnerHTML={{ __html: props.body }}
>         />
>         {props.postBodyComponents}
>       </body>
>     </html>
>   )
Common subdirectories: 23.cache/fragments and .cache/fragments
Common subdirectories: 23.cache/json and .cache/json
diff 23.cache/navigation.js .cache/navigation.js
37c37
< const onPreRouteUpdate = location => {
---
> const onPreRouteUpdate = (location, prevLocation) => {
39c39
<     apiRunner(`onPreRouteUpdate`, { location })
---
>     apiRunner(`onPreRouteUpdate`, { location, prevLocation })
43c43
< const onRouteUpdate = location => {
---
> const onRouteUpdate = (location, prevLocation) => {
45c45
<     apiRunner(`onRouteUpdate`, { location })
---
>     apiRunner(`onRouteUpdate`, { location, prevLocation })
136c136
<     onPreRouteUpdate(props.location)
---
>     onPreRouteUpdate(props.location, null)
140c140
<     onRouteUpdate(this.props.location)
---
>     onRouteUpdate(this.props.location, null)
145c145
<       onRouteUpdate(this.props.location)
---
>       onRouteUpdate(this.props.location, prevProps.location)
151c151
<       onPreRouteUpdate(this.props.location)
---
>       onPreRouteUpdate(this.props.location, prevProps.location)
diff 23.cache/static-entry.js .cache/static-entry.js
55c55,59
<     <meta name="generator" content={`Gatsby ${gatsbyVersion}`} />,
---
>     <meta
>       name="generator"
>       content={`Gatsby ${gatsbyVersion}`}
>       key={`generator-${gatsbyVersion}`}
>     />,
354,360c358,366
<   const bodyScripts = scripts.filter(s => s.rel !== `prefetch`).map(s => {
<     const scriptPath = `${__PATH_PREFIX__}/${JSON.stringify(s.name).slice(
<       1,
<       -1
<     )}`
<     return <script key={scriptPath} src={scriptPath} async />
<   })
---
>   const bodyScripts = scripts
>     .filter(s => s.rel !== `prefetch`)
>     .map(s => {
>       const scriptPath = `${__PATH_PREFIX__}/${JSON.stringify(s.name).slice(
>         1,
>         -1
>       )}`
>       return <script key={scriptPath} src={scriptPath} async />
>     })

Diff after the updater and a new build run @madelyneriksen

[DanielRuf]

patch-package
--- a/node_modules/gatsby/dist/internal-plugins/query-runner/pages-writer.js
+++ b/node_modules/gatsby/dist/internal-plugins/query-runner/pages-writer.js
@@ -88,9 +88,9 @@ const preferDefault = m => m && m.default || m
     let asyncRequires = `// prefer default export if available
 const preferDefault = m => m && m.default || m
 \n`;
-    asyncRequires += `exports.components = {\n${components.map(c => `  "${c.componentChunkName}": () => import("${(0, _path.joinPath)(c.component)}" /* webpackChunkName: "${c.componentChunkName}" */)`).join(`,\n`)}
+    asyncRequires += `exports.components = {\n${components.map(c => `  "${c.componentChunkName}": () => require("${(0, _path.joinPath)(c.component)}" /* webpackChunkName: "${c.componentChunkName}" */)`).join(`,\n`)}
 }\n\n`;
-    asyncRequires += `exports.data = () => import(/* webpackChunkName: "pages-manifest" */ "${(0, _path.joinPath)(program.directory, `.cache`, `data.json`)}")\n\n`;
+    asyncRequires += `exports.data = () => require(/* webpackChunkName: "pages-manifest" */ "${(0, _path.joinPath)(program.directory, `.cache`, `data.json`)}")\n\n`;
 
     const writeAndMove = (file, data) => {
       const destination = (0, _path.joinPath)(program.directory, `.cache`, file);

This should work for .98

[DanielRuf]

Maybe I am on the wrong track, the change produces failing tests #11331

Not yet sure why.

[omrllm]

So it still doesn't work, right?

[DanielRuf]

The build script succeeds with my proposed change. But I'm not sure if this is the right solution. So please test and review it.

[DanielRuf]

Hm, or is it due to the new preset?
69faca0#diff-a30bb413b08d8091d9187909b256c943

Is the plugins array correct?

Does the issue also happen in a fresh new Gatsby project and can it be reproduced with the CI tests?

[DanielRuf]

I npm update and the problem went away

Please provide a list of the installed dependencies (before and after the update).

[mehrad-meraji]

@DanielRuf, I can't reproduce it I guess it was a fluke I am still getting the error.

[DanielRuf]

A workaround is to install the dependencies with Yarn instead of npm which seems to work after importing the npm lockfile.

[cyrildurand]

You can find my npm ls with gatsby v2.0.91 (build succeed) and v2.0.93 (build failed) here : https://gist.github.com/cyrildurand/f4b70abff19288117ea3996500532774

I still have the problem with gatsby 2.0.103

[DanielRuf]

Did you try to install the dependencies with yarn too?

[seonim-ryu]

@cyrildurand
Did you encounter this error when you installed npm?

I also had the same issue, but upgrading arcon's version to 6.0 worked well.

[paulmelnikow]

I think this may be the issue: webpack/webpack#8656.

Added: Symptoms match and timing fits, too.

[DanielRuf]

@paulmelnikow good catch. And this is why I recommend using CJS if we do not have to use ESM - it is still not 100% reliable in bundlers. And the other way should still work.

So pinning webpack in the root package.json should work?

[DanielRuf]

And this js also why SemVer in the JS ecosystem is totally broken imho. The automatic updates of (deep) deps because the lockfiles only work in the root level.

[DanielRuf]

To be clear, acorn and the way npm resolves the deptree are the issue and cause.

webpack/webpack#8656 (comment)

I can reproduce this (another reason why we still use yarn at work).

[DanielRuf]

To be exact, it is a npm issue.

https://npm.community/t/packages-with-peerdependencies-are-incorrectly-hoisted/4794/4

https://npm.community/t/release-npm-6-8-0-next-0/5058

Solution: install latest npm release.

[paulmelnikow]

I'm not sure I agree with that assessment. This may manifest in npm and not yarn due to differences in the way resolution takes place, however 4.29.3 is a perfectly correct version to install when a package has declared a dependency on ^4.12.0. That is what the caret means. If Gatsby wants to lock a specific version it is welcome to do so, and in that case npm will honor it.

Webpack is a dependency of Gatsby, not a peer dependency.

[DanielRuf]

The issue is was the hoisting in npm (different in Yarn) and that acorn could not be loaded properly due to this. See Tobias community post and the linked comment.

[paulmelnikow]

I feel like I need you to explain that better. I skimmed those threads but don’t see how any it applies here.

I don’t know what is causing the bug in Webpack; however if we agree Gatsby shouldn’t use 4.29.3, the caret dependency needs to be changed.

[DanielRuf]

It works with yarn, it is just an issue with npm, in combination with specific dependencies and the deptree calculation. See the PR of Tobias.

See https://github.com/npm/cli/pull/147/files

[paulmelnikow]

Aiiiee gotcha. It's that peer dependencies bug in npm which is what's causing webpack 4.29 not to work correctly.

What we can prevent is the installation of webpack 4.29. And I'm not sure there is an easy way of an end user locking the webpack version. npm doesn't provide a way to do it, so users would have to use something like the third-party tool npm-force-resolutions.

See npm/cli#152; it doesn't look like we can expect a quick resolution.

[paulmelnikow]

Now that 2.0.118 ships a band-aid, npm users should be okay, and obviously can’t use webpack 4.29.x regardless.

Am I right that yarn users can use resolutions to force gatsby to use a later, out-of-range version, if they want?

[DSchau]

@paulmelnikow correct - but really shouldn't be necessary and I'm not sure there's a particular necessary benefit in doing so.

We'll update the pinned dependency as soon as we are able to (following the npm issue now), so it should be but a blip for npm users in particular.

Thanks for the fix!

Now I think I can close this out :)

[paulmelnikow]

Confirmed fixed with the 2.0.118 release!

[dlindahl]

This was amazing to follow. Thank you all for the amazing work!

[wardpeet]

Hey all! I wonder if you guys could give npm install gatsby@webpack-acorn a spin. We want to upgrade webpack to the latest version but we're not sure if this is still an issue. I couldn't reproduce it but we rather want to be safe than sorry.

[paulmelnikow]

Hi @wardpeet! Thanks for reaching out.

I created a branch here: badges/shields#3572

It looks like it's still an issue: https://circleci.com/gh/badges/shields/57401

Steps to reproduce locally would be to clone that branch and run npm ci followed by npm run build. Feel free to do that if you'd like, or ping me and I can update the PR branch.

[wardpeet]

@paulmelnikow thanks so much! Could you also tell me what node & npm version you using so I can run it with your setup as well in case it works for me.

[paulmelnikow]

This is what I've got locally:

~/c/shields (bump-webpack-rc|✔) $ node --version
v10.13.0
~/c/shields (bump-webpack-rc|✔) $ npm --version
6.9.0

In CI it's also happening in Node 8 (not sure the exact npm version).

The problem is reproducible in both environments.

Thanks for looking into it!