WebP images don’t work in og:image (external link previews)

[da2x]

Expected behaviour

Toots with WebP images in their OpenGraph thumbnail image in WebP format should work as a preview image in Mastodon. (<meta name="og:image" content="https://example.com/image.webp">)

Actual behaviour

The toot is shown with the default image. The external image (WebP format) fails to load.

Steps to reproduce the problem

Share any link from e.g. reddit.com (any thread that’s shown in their interface with an image) or ctrl.blog and open the toot in the web interface.

Specifications

Firefox 81 on Fedora 32 and Windows 10.

Related

#14819 SVG support

[Gargron]

I can't fix this, imagemagick lacks proper webp support

[SuperSandro2000]

Maybe ImageMagick needs an update because on their website they list quite a few options for webp. https://imagemagick.org/script/webp.php

[charlag]

That kinda sucks from the user perspective when you can't upload pics from e.g. reddit and you don't know why, regardless of whether it should be served to the clients as webp or not

[ayushnix]

I can't fix this, imagemagick lacks proper webp support

Does imagemagick properly support webp now?

[pauloxnet]

I can't fix this, imagemagick lacks proper webp support

It seems to me that imagemagick now support well the webp format since a while
https://imagemagick.org/script/webp.php

It's also one of the main format advertise in the imagemagick home Page
https://imagemagick.org/

[pauloxnet]

I think the support has been added in 379a7a7

@da2x @Gargron this issue can be closed.

[pauloxnet]

Can anyone close this issue?

Here my last toot as demonstration that mastodon now can correctly render preview card using og:image file in WebP format
https://fosstodon.org/@paulox/109395028585159669

[THeK3nger]

It is actually still broken for me. I cannot see the og:image preview in my blog's links (nor in the previous example).

(Even third-party clients cannot load them. Does it still depends on the main Mastodon code?)

EDIT: Yep. At least on v4.0.2, if I convert the og:image to png everything works. With webp, I get nothing.

[quicoto]

@THeK3nger I was looking for a way to show bigger image preview and stumbled across your issue

I'm using webp and I do see a preview in 4.1.0

Example post: https://ricard.social/@ricard/109992959202878541

[LeoEurope]

Link previews for many sites don’t show the OpenGraph Image because the image is WebP or AVIF. See also: #18762

• Take this link from The Guardian: https://www.theguardian.com/environment/2023/apr/24/european-countries-pledge-huge-expansion-of-north-sea-wind-farms When posted to Mastodon it looks like this: https://mstdn.social/@TheEuropeanNetwork/110254671877149700 because the OG image is WebP.

• Or this link from Euronews, it claims to send a JPG but some browser detection actually makes it send a Webp: https://www.euronews.com/next/2023/05/22/this-british-start-up-has-designed-an-umbrella-like-landing-system-to-make-satellites-reus When posted to Mastodon it looks like this: https://mstdn.social/@TheEuropeanNetwork/110413336447544332

• Or this link from the Financial Times: https://www.ft.com/content/358e382d-ba54-46d0-8b34-22910dbe7cec When posted to Mastodon it looks like this: https://mstdn.social/@TheEuropeanNetwork/110077571317654160 because the OG image is AVIF.

Now, there are other issues I don’t understand but are perhaps related.

• This article serves a PNG as OG Image: https://www.spiegel.de/international/europe/britain-in-crisis-the-uk-faces-a-steep-climb-out-of-a-deep-hole-a-6b61dc6f-e33f-46f3-bd27-743364dd675c but it doesn’t come through when posted in Mastodon: https://mstdn.social/@TheEuropeanNetwork/110225054846515314

• Another example with a PNG: https://www.wired.co.uk/article/qanon-conspiracy-bodegraven-netherlands vs https://mstdn.social/@TheEuropeanNetwork/110186271930242074

• This article serves a JPG as OG Image: https://www.rte.ie/news/2023/0416/1377262-finland-reactor/ but it doesn’t come through when posted in Mastodon: https://mstdn.social/@TheEuropeanNetwork/110212988501999628

• This article serves a JPG as OG Image: https://www.bloomberg.com/news/features/2023-05-16/from-traffic-choked-brussels-a-model-for-driving-less but it doesn’t come through when posted in Mastodon: https://mstdn.social/@TheEuropeanNetwork/110395364546446410

• This article's og:image works fine in the web client, the official Mastodon app, and the MetaText app, but it doesn't work in the Ice Cubes app. It may be an app specific issue, though. https://euobserver.com/green-economy/157251

Solution?
ImageMagick now supports both WebP and AVIF but the server doesn't, and many Mastodon clients may not. The use of WebP and AVIF will only increase in future. Is it an idea to require ImageMagick with WebP and AVIF support (it looks like a compile time option) and convert all WebP and AVIF images to PNG (or even high quality JPG) until support for those formats in clients has improved?

A solution for non-standard or buggy JPGs might be to have ImageMagick rewrite the JPGs without further compression. I assume it's currently already stripping privacy sensitive information (EXIF) from images anyway.

[tomresing]

Is it possible that the solution is as simple as including the instructions to download and install libwebp?

I'm a novice at ImageMagick and Mastodon installation, but after I installed ImageMagick on my device it didn't support WebP until I also installed libwebp as suggested in the ImageMagick community.

[tomresing]

I don't see libwebp listed in the system packages in the Mastodon install documentation. I do see libwebp in the list of available packages for both Debian and Ubuntu.

[c960657]

@LeoEurope The issue with The Guardian is not related to WebP – see #26219.

Now, there are other issues I don’t understand but are perhaps related.

Some of them are skipped, because the file size of the Open Graph image is above the 1 MB limit. I think it would be reasonable to increase this limit a bit.

[LeoEurope]

@LeoEurope The issue with The Guardian is not related to WebP – see #26219.

That could play a role, yes. But I doubt it's the only one. Check this image link from The Guardian: https://i.guim.co.uk/img/media/b5bd441e246054953dbc21d57ab9667093036ead/0_346_6048_3628/master/6048.jpg?width=620&dpr=1&s=none The URL suggests that it's a JPG but Firefox actually receives it as an AVIF file. If I open the same link in an older version of Safari that I have lying around it's sent as a JPG.

That suggests that, server side, some sniffing is happening to send an WebP or AVIF file even if the extension is .jpg. The Mastodon server might be receiving an AVIF and doesn't know what to do with it.

[c960657]

The server returns different file formats depending the Accept header. This is why Firefox receives an AVIF file.

$ curl -s -H 'Accept: image/avif, *' -I 'https://i.guim.co.uk/img/media/b5bd441e246054953dbc21d57ab9667093036ead/0_346_6048_3628/master/6048.jpg?width=620&dpr=1&s=none' | grep -i content-type
content-type: image/avif

$ curl -s -I 'https://i.guim.co.uk/img/media/b5bd441e246054953dbc21d57ab9667093036ead/0_346_6048_3628/master/6048.jpg?width=620&dpr=1&s=none' | grep -i content-type
content-type: image/jpeg

[tomresing]

The server returns different file formats depending the Accept header. This is why Firefox receives an AVIF file.

$ curl -s -H 'Accept: image/avif, *' -I 'https://i.guim.co.uk/img/media/b5bd441e246054953dbc21d57ab9667093036ead/0_346_6048_3628/master/6048.jpg?width=620&dpr=1&s=none' | grep -i content-type
content-type: image/avif

$ curl -s -I 'https://i.guim.co.uk/img/media/b5bd441e246054953dbc21d57ab9667093036ead/0_346_6048_3628/master/6048.jpg?width=620&dpr=1&s=none' | grep -i content-type
content-type: image/jpeg

Very helpful to see the difference in the responses from the two requests for the same resource. Do we know which accept header Mastodon is sending? Do you have a similar request that returns the size of the image?

[LeoEurope]

The server returns different file formats depending the Accept header. This is why Firefox receives an AVIF file.

$ curl -s -H 'Accept: image/avif, *' -I 'https://i.guim.co.uk/img/media/b5bd441e246054953dbc21d57ab9667093036ead/0_346_6048_3628/master/6048.jpg?width=620&dpr=1&s=none' | grep -i content-type
content-type: image/avif

$ curl -s -I 'https://i.guim.co.uk/img/media/b5bd441e246054953dbc21d57ab9667093036ead/0_346_6048_3628/master/6048.jpg?width=620&dpr=1&s=none' | grep -i content-type
content-type: image/jpeg

Very helpful to see the difference in the responses from the two requests for the same resource. Do we know which accept header Mastodon is sending? Do you have a similar request that returns the size of the image?

That's a good point, the find by @c960657 makes me think a quick first solution would be to have the Mastodon server only send image/jpg and image/png accept headers when getting the Preview image until WebP and AVIF support is fixed.

Over time it makes sense to also support WebP and AVIF as that will have considerable impact on storage use by all servers.

[c960657]

Do we know which accept header Mastodon is sending?

It doesn't send any.

Do you have a similar request that returns the size of the image?

The file size is bytes can be measured like this:

$ curl -s 'https://i.guim.co.uk/img/media/b5bd441e246054953dbc21d57ab9667093036ead/0_346_6048_3628/master/6048.jpg?width=620&dpr=1&s=none' | wc -c
   48074

[tomresing]

@c960657 is the header passed from the Mastodon client through to the end web server then? I'm not sure why you mentioned that that it sends back different image formats based on the header. It might be that the fix @LeoEurope suggests is the PR needed to resolve this issue, but it seems like you're suggesting the normalize URL PR you've submitted will solve the issue without a header change.

[Sushubh]

at my end... og:image webp images seem to work when the image is on the same domain as the shared link. if the image is third party, it does not appear. third party jpegs however do appear so the issue does not seem to be third party image. just a fyi.

[phocks]

Link previews for many sites don’t show the OpenGraph Image because the image is WebP or AVIF. See also: #18762

* Take this link from The Guardian: https://www.theguardian.com/environment/2023/apr/24/european-countries-pledge-huge-expansion-of-north-sea-wind-farms When posted to Mastodon it looks like this: https://mstdn.social/@TheEuropeanNetwork/110254671877149700 because the OG image is WebP.

* Or this link from Euronews, it claims to send a JPG but some browser detection actually makes it send a Webp: https://www.euronews.com/next/2023/05/22/this-british-start-up-has-designed-an-umbrella-like-landing-system-to-make-satellites-reus When posted to Mastodon it looks like this: https://mstdn.social/@TheEuropeanNetwork/110413336447544332

* Or this link from the Financial Times: https://www.ft.com/content/358e382d-ba54-46d0-8b34-22910dbe7cec When posted to Mastodon it looks like this: https://mstdn.social/@TheEuropeanNetwork/110077571317654160 because the OG image is AVIF.

Now, there are other issues I don’t understand but are perhaps related.

* This article serves a PNG as OG Image: https://www.spiegel.de/international/europe/britain-in-crisis-the-uk-faces-a-steep-climb-out-of-a-deep-hole-a-6b61dc6f-e33f-46f3-bd27-743364dd675c but it doesn’t come through when posted in Mastodon: https://mstdn.social/@TheEuropeanNetwork/110225054846515314

* Another example with a PNG: https://www.wired.co.uk/article/qanon-conspiracy-bodegraven-netherlands vs https://mstdn.social/@TheEuropeanNetwork/110186271930242074

* This article serves a JPG as OG Image: https://www.rte.ie/news/2023/0416/1377262-finland-reactor/ but it doesn’t come through when posted in Mastodon: https://mstdn.social/@TheEuropeanNetwork/110212988501999628

* This article serves a JPG as OG Image: https://www.bloomberg.com/news/features/2023-05-16/from-traffic-choked-brussels-a-model-for-driving-less but it doesn’t come through when posted in Mastodon: https://mstdn.social/@TheEuropeanNetwork/110395364546446410

* This article's og:image works fine in the web client, the official Mastodon app, and the MetaText app, but it doesn't work in the Ice Cubes app. It may be an app specific issue, though. https://euobserver.com/green-economy/157251

Solution? ImageMagick now supports both WebP and AVIF but the server doesn't, and many Mastodon clients may not. The use of WebP and AVIF will only increase in future. Is it an idea to require ImageMagick with WebP and AVIF support (it looks like a compile time option) and convert all WebP and AVIF images to PNG (or even high quality JPG) until support for those formats in clients has improved?

A solution for non-standard or buggy JPGs might be to have ImageMagick rewrite the JPGs without further compression. I assume it's currently already stripping privacy sensitive information (EXIF) from images anyway.

This seems like the only good solution from @LeoEurope. Simply convert the webp avif etc images to jpegs and be done with it.

[phocks]

The server returns different file formats depending the Accept header. This is why Firefox receives an AVIF file.

$ curl -s -H 'Accept: image/avif, *' -I 'https://i.guim.co.uk/img/media/b5bd441e246054953dbc21d57ab9667093036ead/0_346_6048_3628/master/6048.jpg?width=620&dpr=1&s=none' | grep -i content-type
content-type: image/avif

$ curl -s -I 'https://i.guim.co.uk/img/media/b5bd441e246054953dbc21d57ab9667093036ead/0_346_6048_3628/master/6048.jpg?width=620&dpr=1&s=none' | grep -i content-type
content-type: image/jpeg

@c960657 anyone know where the code is that fetches the images in the codebase? Should be a pretty easy fix?

Edit: I found this. Surely it wouldn't be just as simply as removing the image/webp from here. og spec says it only works with jpg png and gif

og:image:type - MIME type of the image. One of image/jpeg, image/gif or image/png

mastodon/app/models/preview_card.rb

Lines 37 to 42 in d8c4284

	class PreviewCard < ApplicationRecord
	include Attachmentable
	IMAGE_MIME_TYPES = ['image/jpeg', 'image/png', 'image/gif', 'image/webp'].freeze
	LIMIT = 2.megabytes

I'll keep searching. I guess it doesn't explain the avif images. And I guess this file does something with the images in the database or something. Not sure, I don't really know Ruby that well.

Edit 2: I found this. Might be something. Might have a look later when I'm free.

mastodon/app/services/fetch_link_card_service.rb

Lines 129 to 130 in d8c4284

	@card.embed_url = (url + embed[:url]).to_s
	@card.image_remote_url = (url + embed[:url]).to_s

I think we're probably on the right track here. Just need to find where it actually constructs the http request right?

mastodon/app/lib/link_details_extractor.rb

Lines 113 to 131 in d8c4284

	def to_preview_card_attributes
	{
	title: title || '',
	description: description || '',
	image_remote_url: image,
	image_description: image_alt || '',
	type: type,
	link_type: link_type,
	width: width || 0,
	height: height || 0,
	html: html || '',
	provider_name: provider_name || '',
	provider_url: provider_url || '',
	author_name: author_name || '',
	author_url: author_url || '',
	embed_url: embed_url || '',
	language: language,
	published_at: published_at.presence,
	}

[LeoEurope]

Link previews for many sites don’t show the OpenGraph Image because the image is WebP or AVIF. See also: #18762

* Take this link from The Guardian: https://www.theguardian.com/environment/2023/apr/24/european-countries-pledge-huge-expansion-of-north-sea-wind-farms When posted to Mastodon it looks like this: https://mstdn.social/@TheEuropeanNetwork/110254671877149700 because the OG image is WebP.

* Or this link from Euronews, it claims to send a JPG but some browser detection actually makes it send a Webp: https://www.euronews.com/next/2023/05/22/this-british-start-up-has-designed-an-umbrella-like-landing-system-to-make-satellites-reus When posted to Mastodon it looks like this: https://mstdn.social/@TheEuropeanNetwork/110413336447544332

* Or this link from the Financial Times: https://www.ft.com/content/358e382d-ba54-46d0-8b34-22910dbe7cec When posted to Mastodon it looks like this: https://mstdn.social/@TheEuropeanNetwork/110077571317654160 because the OG image is AVIF.

Now, there are other issues I don’t understand but are perhaps related.

* This article serves a PNG as OG Image: https://www.spiegel.de/international/europe/britain-in-crisis-the-uk-faces-a-steep-climb-out-of-a-deep-hole-a-6b61dc6f-e33f-46f3-bd27-743364dd675c but it doesn’t come through when posted in Mastodon: https://mstdn.social/@TheEuropeanNetwork/110225054846515314

* Another example with a PNG: https://www.wired.co.uk/article/qanon-conspiracy-bodegraven-netherlands vs https://mstdn.social/@TheEuropeanNetwork/110186271930242074

* This article serves a JPG as OG Image: https://www.rte.ie/news/2023/0416/1377262-finland-reactor/ but it doesn’t come through when posted in Mastodon: https://mstdn.social/@TheEuropeanNetwork/110212988501999628

* This article serves a JPG as OG Image: https://www.bloomberg.com/news/features/2023-05-16/from-traffic-choked-brussels-a-model-for-driving-less but it doesn’t come through when posted in Mastodon: https://mstdn.social/@TheEuropeanNetwork/110395364546446410

* This article's og:image works fine in the web client, the official Mastodon app, and the MetaText app, but it doesn't work in the Ice Cubes app. It may be an app specific issue, though. https://euobserver.com/green-economy/157251

Solution? ImageMagick now supports both WebP and AVIF but the server doesn't, and many Mastodon clients may not. The use of WebP and AVIF will only increase in future. Is it an idea to require ImageMagick with WebP and AVIF support (it looks like a compile time option) and convert all WebP and AVIF images to PNG (or even high quality JPG) until support for those formats in clients has improved?
A solution for non-standard or buggy JPGs might be to have ImageMagick rewrite the JPGs without further compression. I assume it's currently already stripping privacy sensitive information (EXIF) from images anyway.

This seems like the only good solution from @LeoEurope. Simply convert the webp avif etc images to jpegs and be done with it.

There's now PR #30090 about replacing ImageMagick with libvips, which might solve the underlying issue of WebP and AVIF support.

I'd suggest to use this bug to solve any WebP/AVIF problems that may still occur after libvips has landed and consider using #27370 as a temporary stopgap by sending HTTP AcceptHeaders to stop servers sending WebP and AVIF until this is solved (if necessary).

[phocks]

I have done some poking around in the Mastodon code and it seems I've traced some preview card issues to an unlikely source.

def body_with_limit(limit = 1.megabyte)

If an article page's html content is larger than 1MB (like Wired and The New Yorker) then an error is raised and the preview card doesn't show.

If we set the limit to 2MB the preview card displays as normal.

ps. your site html really shouldn't be more than 1MB yet here we are

mastodon/app/lib/request.rb

Line 236 in 2da2a1d

def body_with_limit(limit = 1.megabyte)

(from my toot https://bne.social/@phocks/112458999544660059)

I think bumping this up to 2MB makes sense in 2024

[phocks]

OK I've created a pull request. Seems like the truncated_body function was adding the chunk of html to the contents and THEN stopping after that, which meant that the contents.bytesize was ALWAYS going to be over the limit and so the exception is always raised when the html is over 1MB (such as New Yorker and Wired articles).

This just makes the break before adding the chunk, so contents remains just under the limit.

Who do I need to ping to get this approved? @Gargron ? @renchap ?

Cheers!

[renchap]

Looks like there is a lot of confusion going on here.

Take this link from The Guardian: https://www.theguardian.com/environment/2023/apr/24/european-countries-pledge-huge-expansion-of-north-sea-wind-farms

Fixed by #26219

Or this link from Euronews, it claims to send a JPG but some browser detection actually makes it send a Webp: https://www.euronews.com/next/2023/05/22/this-british-start-up-has-designed-an-umbrella-like-landing-system-to-make-satellites-reus When posted to Mastodon it looks like this: https://mstdn.social/@TheEuropeanNetwork/110413336447544332

The image is correctly a JPEG, it appears as a WebP in your browser because your browser sends Accept headers. The issue here is not related to the image format

Or this link from the Financial Times: https://www.ft.com/content/358e382d-ba54-46d0-8b34-22910dbe7cec When posted to Mastodon it looks like this: https://mstdn.social/@TheEuropeanNetwork/110077571317654160 because the OG image is AVIF.

There are no OG or oembed tags for this URL, this is why there is no preview image

This article serves a PNG as OG Image: https://www.spiegel.de/international/europe/britain-in-crisis-the-uk-faces-a-steep-climb-out-of-a-deep-hole-a-6b61dc6f-e33f-46f3-bd27-743364dd675c but it doesn’t come through when posted in Mastodon: https://mstdn.social/@TheEuropeanNetwork/110225054846515314

The Opengraph image is more than 1 MB, so it is ignored by Mastodon. We might bump this once the libvips PR is merged

Another example with a PNG: https://www.wired.co.uk/article/qanon-conspiracy-bodegraven-netherlands vs https://mstdn.social/@TheEuropeanNetwork/110186271930242074

Same

This article serves a JPG as OG Image: https://www.rte.ie/news/2023/0416/1377262-finland-reactor/ but it doesn’t come through when posted in Mastodon: https://mstdn.social/@TheEuropeanNetwork/110212988501999628

Same, with a 5 MB image

This article serves a JPG as OG Image: https://www.bloomberg.com/news/features/2023-05-16/from-traffic-choked-brussels-a-model-for-driving-less but it doesn’t come through when posted in Mastodon: https://mstdn.social/@TheEuropeanNetwork/110395364546446410

May be blocked by Bloomberg's bot detection (need to investigate more)

This article's og:image works fine in the web client, the official Mastodon app, and the MetaText app, but it doesn't work in the Ice Cubes app. It may be an app specific issue, though. https://euobserver.com/green-economy/157251

Link preview generation is made on the backend, so not sure what is happening here, but this is not a Mastodon issue

As you can see, none of those are caused by having a WebP of AVIF image, neither by HTML documents being over the 1 MB limit.

We will consider upping the OG image limit once #30090 is merged.