Lazy image processing in gatsby develop

[wardpeet]

Released in 2.30: https://www.gatsbyjs.com/docs/reference/release-notes/v2.30#query-on-demand-and-lazy-images-generally-available

---

Hi 👋,

Gatsby is a great tool to develop your website. We've got some feedback that the more image-heavy your website gets, the slower gatsby develop. I've got good news for all of you, we've got an experimental feature (behind a flag) that only does image processing when an image gets requested by the browser.

To use it, make sure you have gatsby@^2.28.0 and gatsby-plugin-sharp@^2.10.0 and add LAZY_IMAGES to flags in gatsby-config.js:

// In your gatsby-config.js
module.exports = {
  // your existing config
  flags: {
    LAZY_IMAGES: true,
  },
}

If you want to opt-out - set the LAZY_IMAGES flag to false.

We'll only do lazy images processing during "develop". On "build" we process all images that haven't been processed yet. Note that there might still be issues, so when you switch between develop & build, so be careful when you deploy your site from your machine.

Please comment below if you encounter any bugs/issues... Please react with an 👍 emoji if you like this feature.

PS. You might also be interested in Query on-demand in gatsby develop that applies a similar mechanism but on query running. In fact - it would be awesome to check them both at the same time and play with enabling one or the other (and both for most "laziness") and let us know which of the two does make a bigger impact for you.

Happy dev'ing!

[josephmarkus]

This is great news. It is encouraging to see that the issue is being worked on.

I noticed an issue with a message when the build starts:

info [gatsby-plugin-sharp] The lazy image processing experiment is enabled, to disable it please set
success onPreInit - 0.038s

Looks like part of the message is missing

Looking forward to seeing feature land for everyone

[wardpeet]

That's a cli issue :'( Thanks for surfacing!

[sandren]

The text is no longer cut off, but it still seems wrong. It says info [gatsby-plugin-sharp] The lazy image processing experiment is enabled, to disable it please set experimentalDisableLazyProcessing to false in pluginOptions. However, if disable is set to false, wouldn't that effectively leave the option enabled?

[wardpeet]

Haha yes that should be set to true to disable it

[sandren]

I've tested this feature for local development of https://coggscircus.com/ and enabling lazy image processing reduced "server boot finished" time from 481.303s to 112.760s (running gatsby clean before each gatsby develop), a 77% reduction in the initial startup time, which is an even better result than what I experienced testing the query on demand feature. Incredible! 😍

Using both lazy image processing and query on demand together produced the best results, reducing the "server boot finished" time further still to only 58.620s, or an 88% reduction in the initial startup time. However, I think lazy image processing is responsible for most of the reductions (at least for a site of this size) because delaying queries also delays image processing.

Based on these experiences, I will probably use this lazy image processing feature on all existing and future Gatsby projects, but wait to use the query on demand feature until it's a bit more polished.

[wardpeet]

Thanks for the great feedback. Do you know where the other 58s come from? Is it sourcing from your cms? If you could paste in the gatsby logs that would be amazing!

[sandren]

Yes, it's mostly sourcing data from Craft CMS and Shopify. There are also some inefficiencies related to some pages querying more information than is strictly necessary and filtering it down to what is needed. This was related to a limitation of gatsby-source-graphql. Once the site migrates to the new gatsby-source-craft, I expect that number will come down a bit.

Here is the log of a fresh gatsby develop after a gatsby clean using lazy image processing, but not query on demand: https://gist.github.com/sandren/79a270e3eb0bf4c86e59f5cfbccb89f0

[wardpeet]

Thanks for the log, it seems like shopify takes ages

[jasonrundell]

A heads-up for those using this and found that some images load and others don't: check your file names. I ran into an issue where some images from our CMS were being processed at build time, but a lot of them were not and it was because a publisher had uploaded images with the @ in the file name e.g. illustration-1-a@3x.png which then was encoded and turned into this in graphql: illustration-1-a%25403x.webp.

So watch out for filename encoding issues! :)

[KyleAMathews]

This is a bug vs. master — found the problem (also noticed it on an image on my blog) and put up a PR with the fix #28449

[czabaj]

Beware 🚨

I've just spent several hours investigating a strange problem with gatsby-plugin-manifest, where it failed to process an icon with

Error: Input file contains unsupported image format

This error occurred only in CI on Linux 4.15 Debian GNU/Linux 9 (stretch) 9 (stretch), local build on macOS was OK (one of the nasties kind of errors).

It smelled like this issue, but adding SHARP_IGNORE_GLOBAL_LIBVIPS=true did not helped. The environment also does not contained the libvips program which was cause of many similar problems that the issues of lovel/sharp tracks.

In the end, I simply installed the latest versions of gatsby-plugin-sharp@2.9.1 instead of lazy-images, and everything now just works.

Here are my gatsby dependencies

// dependencies
  "gatsby": "2.27.4",
  "gatsby-image": "2.6.0",
  "gatsby-plugin-react-helmet": "3.5.0",
  "gatsby-cli": "2.14.1",
// dev dependencies
  "gatsby-plugin-extract-schema": "0.0.5",
  "gatsby-plugin-graphql-loader": "1.0.2",
  "gatsby-plugin-manifest": "2.7.0",
  "gatsby-plugin-offline": "3.5.0",
  "gatsby-plugin-sass": "2.6.0",
  "gatsby-plugin-sharp": "lazy-images", // change to "2.9.1" helped here
  "gatsby-plugin-sitemap": "2.7.0",
  "gatsby-plugin-svgr": "2.0.2",
  "gatsby-plugin-typescript": "2.7.0",
  "gatsby-source-filesystem": "2.6.1",
  "gatsby-transformer-sharp": "2.7.0",

And the stats of problematic CI environment

  System:
    OS: Linux 4.15 Debian GNU/Linux 9 (stretch) 9 (stretch)
    CPU: (36) x64 Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
    Memory: 37.84 GB / 68.69 GB
    Container: Yes
    Shell: 4.4.12 - /bin/bash
  Binaries:
    Node: 12.20.0 - /usr/local/bin/node
    Yarn: 1.22.5 - /usr/local/bin/yarn
    npm: 6.14.8 - /usr/local/bin/npm
  Utilities:
    Make: 4.1 - /usr/bin/make
    GCC: 6.3.0 - /usr/bin/gcc
    Git: 2.11.0 - /usr/bin/git
    Mercurial: 4.0 - /usr/bin/hg
    Subversion: 1.9.5 - /usr/bin/svn
  Languages:
    Bash: 4.4.12 - /bin/bash
    Perl: 5.24.1 - /usr/bin/perl
    Python: 2.7.13 - /usr/bin/python
    Python3: 3.5.3 - /usr/bin/python3

The environment spins up circleci/node:erbium docker image and yes, it runs on circle CI.

The version of sharp is the same, sharp@0.26.3 for gatsby-plugin-sharp@lazy-images and gatsby-plugin-sharp@2.9.1 there must be some kind of error in lazy-images which affects also gatsby-plugin-manifest (it runs onPostBootstrap so very early before other possible errors).

[vladar]

Can you share your icon image?

[czabaj]

Sorry, I'm busy, tomorrow I'll have more time and will test the setting in the docker image. If I succeed to reproduce the issue, I'll let you know.

I tried all image formats supported by gatsby-plugin-manifest, so I think I can safely cross out the image as a problem source.

[czabaj]

Just tested gatsby-starter-default in incriminated docker image with dependencies set to exact versions as described in an issue and everything works. Then tested lazy-images in my (unfortunately private) work repo with today's fresh Gatsby dependencies and it fails again, even I changed the icon to brand new SVG.

Well, to be correct, the error for SVG is a little bit different

vips_value_hash: no case for NULL
	type 64, gchararray
	generic 53, GParamString
vips_value_hash: no case for 0.000000
	type 60, gdouble
	generic 109, GParamDouble
vips_value_hash: no case for FALSE
	type 20, gboolean
	generic 76, GParamBoolean
vips_value_hash: no case for ((VipsAccess) VIPS_ACCESS_RANDOM)
	type 76458320, VipsAccess
	generic 80, GParamEnum
vips_value_hash: no case for FALSE
	type 20, gboolean
	generic 76, GParamBoolean
error Command failed with exit code 1.

I encountered the same errors from sharp previously when tested SVG icon.

Again, there is no issue with most recent version of gatsby-plugin-sharp@2.10.0

It could be a concurrency related problem, the docker image on my machine is very slow, although it should have 4 cores attached, in CI, we have an 8-core machine 🤔

[vladar]

We moved this feature from the canary to the released version of plugin sharp (behind a flag). So, can you try it with gatsby-plugin-sharp@2.10.0 and flag LAZY_IMAGES set to true? (see the updated description on how to setup the flag)

[czabaj]

All green ✔️

[LekoArts]

Thanks everyone for testing! This was enabled by default in 2.30: https://www.gatsbyjs.com/docs/reference/release-notes/v2.30#query-on-demand-and-lazy-images-generally-available