Image resolution limitations - an increasing problem.

[enn-nafnlaus]

Reopening discussion over maximum image resolutions.

There's recently been a big influx of people from Birdsite, and with them, a lot of artists - who are disappointed when they get here and find that the resolution of their images has been crushed. Example:

https://fosstodon.org/web/@arvalis@mastodon.social/109311924798595398

Comment example: "Kind of a bummer to have to send folks offsite to actually see the artwork. Feels very 2008."

While in previous discussions (#7242) the talk was of quotas, rather than waiting on quotas, I propose a simple solution that should be a trivial change:

1. Ditch the resolution limit and stick only with file size limits - the aspect that actually matters. Excepting at the extreme low end of the CRF range, higher jpeg compression is better for image quality than lower resolutions, for a given file size. Let people choose that, and don't forcably rescale an image that's under the size limit.

Beyond that, another possible change (not as trivial, but simpler than quotas):

2. Drop the current image max file size down, but then give an option to upload "high quality" images (say double the limit). Make the process to do so involve clicking through more steps (including at least one "are you sure?" dialog that mentions the disadvantages of large images) so that most people will skip it unless they specifically need the detail. I strongly suspect the savings on dropping the default max image size down would outweigh the costs of allowing people to override it when they specifically need the resolution.

Let's make Mastodon a welcoming place for artists! :)

(I might be willing to code at least #1 myself if there were consensus on this issue)

[AlanBell]

There is also a very big influx of people posting important pictures of their cats and what they had for lunch - with very little concern for image quality and cellphone cameras with potentially lots of pixels. Maybe it could be a per-instance server admin preference, so an art focused server could have more fidelity.

[enn-nafnlaus]

My point is that the current technical approach is not appropriate. Limiting images by resolution but not file size makes no sense at all. It should be seen as a bug, not fit for purpose. Hence solution #1.

And solution #2 would likely outright reduce space requirements.

Certainly it can be server-specific, but again: the current implementation should be seen as a bug, not a feature. It makes no sense. Image resolution is not the same thing as file size.

[AlanBell]

I am thinking this could be useful for non-dinosaur drawings, things like JWST images, but I am not convinced I need more than full HD resolution without going out to where the image is hosted.

[enn-nafnlaus]

Again, Alan: I'm talking about a bug. The current implementation - limiting by resolution, not file size - makes no sense. It kills quality for no reason at all. Why are you arguing in favour of killing quality for no reason at all?

So, just did some testing.

First, the nominal 8MB limit isn't even relevant, as Mastodon mangles them to far lower than that limit. So first, there should also be a file size limit that actually reflects the reality of what Mastodon is working with.

Now, for a test. Start with this 4912x7360 image of a boy running with jpeg quality setting = 86%.

https://www.pexels.com/photo/boy-running-during-sunset-1416736/

The image is 3,7MB.

If we take a closeup on his feet, here's what they look like:

And if we lower the JPEG quality settings to ~33, the image drops to 1MB, and the result looks like:

But Mastodon's mangling brings it down to ~470k. So we'll half the image res and keep JPEG quality settings about the same to achieve that, and we get.

But here's what Mastodon does to it. Needlessly.

I'll repeat: this is a bug. It's hurting quality in a way that it in no way has to. The above two images are the same file size, but Mastodon's is much lower quality.

So for starters, we should do #1: fix the bug. And I'm willing to help with this if need be.

And I'd also recommend doing #2. Slightly lowering the default quality further, but giving an option (with more user effort) to override it to a higher quality, so the net average file size is the same, so there's no harm to the network. Again, I must stress those last five words: no harm to the network. Only benefit and happier users.

Lastly, I think letting servers pick their own file size limits (call it "# 3") would be optimal. Let me repeat: not resolution limits, but rather file size limits. The thing that actually matters.

[enn-nafnlaus]

And actually, one more addendum: I hope it does this already (haven't looked at the code), but in case it doesn't: images should be keyed by thumbnail hash, so that when the same image gets reposted, it doesn't consume extra space and bandwidth. If it's not already doing this, then this should be fixed, and the freed-up space and bandwidth can go toward higher-resolution images.

It's time we step up from the image quality of 1 1/2 decades ago, with smarter image handling. And at the very least, fixing bug #1 above should happen.

[AlanBell]

The code is here https://github.com/mastodon/mastodon/blob/main/app/javascript/mastodon/utils/resize_image.js
What it does is render the image to a canvas, resizing if it is big, then it uploads that canvas as a PNG This strips out EXIF data puts it in a format everyone can render and totally ignores the file size and compression that may have been applied before it was uploaded. In essence Mastodon shares a picture of the picture, so the original compression isn't helpful (probably harmful in fact).
Not trying to be argumentative here, just looking at how it works and what compromises have been selected that may or may not be currently valid.

[AlanBell]

ah no, that probably isn't the code any more. still poking about trying to understand this stuff . . .

[Gargron]

Client-sizing only happens when the image exceeds the server-side processing limit. Processing an image consumes RAM proportional to the dimensions of the image. The processing limit is 4096x4096 pixels.

[enn-nafnlaus]

4096x4096 isn't a problem. The problem is this hard, fast rule:

const newWidth = Math.round(Math.sqrt(MAX_IMAGE_PIXELS * (width / height)));
const newHeight = Math.round(Math.sqrt(MAX_IMAGE_PIXELS * (height / width)));

... where MAX_IMAGE_PIXELS is so low, yielding e.g. 2000x1000 images. When the same file size can be achieved much better with a higher resolution but a lower jpeg quality setting.

(On top of the other issues mentioned above).

On the topic of server processing: I can dig through the code, but it's better to ask: is there an image hashing check? Or is every image treated as unique, even though people frequently repost the same image over and over? If so, that's low-hanging fruit to reduce load pretty significantly. In a simple a-hash implementation, thumbnail an image to greyscale, hash the thumbnail, do a hash lookup on the server, and if it matches an existing hash, replace all references for the new image to that of the preexisting one, and stop all further processing. p-hash, d-hash or w-hash for better results.

[Gargron]

On the topic of server processing: I can dig through the code, but it's better to ask: is there an image hashing check? Or is every image treated as unique, even though people frequently repost the same image over and over? If so, that's low-hanging fruit to reduce load pretty significantly.

Yes, every image is treated as unique, but this is far from low-hanging fruit. It would be a pretty big task to modify the database schema to support reusing images, you need to devise a migration plan for millions of rows with zero downtime, and if you're introducing extra tables to join on, you might lose very valuable performance if you're not careful, since media attachments are everywhere. And do you hash on the input or output? The conversions are not deterministic, so while you may get away with deduplicating files uploaded on your server by hashing on input, every file you receive over federation which is already processed differently will still generate a separate entry. Sure there are gains to be made even then, but you'll also want to delete files no longer referenced by anybody, and that might end up a very expensive query to run too. Nothing is impossible but I wouldn't call it low hanging fruit. Also, if you use one of those perceptual hashes on the input, how do you guarantee the one image you already processed was the highest-quality one?

... where MAX_IMAGE_PIXELS is so low, yielding e.g. 2000x1000 images. When the same file size can be achieved much better with a higher resolution but a lower jpeg quality setting.

The images are basically never displayed at sizes larger than that, though.

[enn-nafnlaus]

you need to devise a migration plan for millions of rows with zero downtime

This is sort of derailing from the original topic. But I don't see why new images can't just be versioned differently from old images and get their own handler. No migration of old images at all (at least not unless there's demand to do so to free up old resources, wherein they could be migrated one-at-a-time). Hash on input. Strict similarity requirements, and res/CRF stored with the hash. Handle on a per-server basis, not federated. KISS principle. The potential processing savings isn't just small, it's pretty huge, because most images on social media get pretty heavily duplicated, with the differences mainly being res, jpeg noise, etc.

That said, I agree that it's certainly a larger issue than just:

"The images are basically never displayed at sizes larger than that, though."

Well, 15 years ago that wouldn't be a problem. It's 2022 now though. Those images are tiny; wouldn't even fill an iPhone's screen. That sort of quality is a deterrent to participation. Fixing this doesn't sound like a complicated task at all. I'm even willing to look into it myself, if that's what it takes to get it fixed. I just don't want to start working on it if for some reason people are going to insist on low-res low-quality images for no gain whatsoever.

And for what it's worth, people don't just view images inline. (A) If they don't care, they scroll past. (B) If they want to see it better, they click. (C) If they want to see more detail, they expand it further. Except right now people can't go past (B). They literally can't get more detail. Even if said detail is important to the image (small text, etc).

[Vapok]

I can say, with certainty, that the explosion in users over the past week, has definitely come with a fair share of people complaining about image quality, GIF quality, and lack of GIPHY...

and the reality is, the current method that is being used in all of the current and past mentality about the above items is making it so that hard drives are actually filling up with duplicates of these images (especially the GIFs) because people are downloading many variations of the same GIF in order to upload because that is now how Gen-Z and the 22nd century have conversations.

I'd much rather, have great image quality for artistic images, and not have a single GIPHY stored in my file system, because the UI supports browser rendering of a GIPHY or TENOR link.

And if admins having to deal with API keys seems like a lot of work, I'd much rather deal with an API key once, than deal with people complaining about lack of image support in the UI.

[ldexterldesign]

Hi @enn-nafnlaus,

Thanks for follow up

I created this, which @Gargron closed and opened that - in the interests of noise is there a reason you've created something new here?

Cheers

[enn-nafnlaus]

Because I'm not proposing quotas, and the previous issue was closed.

[renchap]

I am used to working with a lot of photos (I created notos.co) and I am familiar with these issues.

A few ideas from my experience:

• browser-side image resizing is powerful but quality can really suffer. If you do this, I suggest using image-blob-reduce / pica (pica is the base brick for the other one), it can really produce better results and is still very fast (it uses WASM)
• server-side processing can be far less resource-heavy by using vips instead of imagemagick, see this suggestion for more details: Consider: Switching from ImageMagick to Vips #20269
• thumbnail generation can be done dynamically nowadays, I am using imgproxy on some very heavy websites and it is amazing. Combined with a cache/CDN, it allows you to no longer generate any size-specific images, and deliver automatic format to the user (it will return a webp image if the browser supports it). This should probably not be mandatory, but could be an option very very useful for large instances.

[enn-nafnlaus]

Thanks for your comment, Renchap - very good points.

At risk of distracting further from the simple fact that we're currently ruining image quality for no good reason via overaggressive downscaling (on the basis that people don't actually care, except they actually do) and it shouldn't be a complicated fix: I'd like to proffer yet another way to improve quality and/or reduce bandwidth: AVIF.

Pros:

• Literally half the size of JPEG at low bitrates (plus has a lot of other features, like animation, alpha transparency, HDR, high bit depth, etc). The difference decreases at high bitrates.
• It has big backers - Netflix, Facebook, Apple, Google, etc.
• A number of major websites are using it, including Vimeo, Etsy, Cloudflare, etc.
• Chrome has had support for for 2 years, Firefox for 1 year, Webkit/Safari for desktop for half a year and on iOS for a few months.
• There's a browser-capable WebAssembly encoder/decoder (https://www.npmjs.com/package/@saschazar/wasm-avif) with a compatible license, so images can be encoded client-side before sending, and even if the browser doesn't support AVIF, can be converted to a supported format for display.
• WebAssembly has been supported by browsers for about 5 years now, so we're talking really legacy stuff (like IE) in terms of a lack of support. Still, servers could generate and cache JPEGs for such legacy clients as needed.
• No need to change anything about preexisting stored JPEG images - just to switch to defaulting to preferring AVIF.

Cons:

• There is a patent troll (Sisvel) trying to charge for AV1, the codec on which AVIF was based. AOMedia responded with a patent defense programme, and Google seems to basically be ignoring Sisvel. Sisvel is currently only trying to charge device manufacturers, and has ruled out ever trying to charge content creators. Sisvel has also not attempted to expand beyond AV1 to AVIF, being focused on the streaming video market
• Slow. Though settings-adjustable, so it would be possible to give clients the option to upload higher-quality images if they were willing to wait a few seconds to encode them.
• No progressive-mode (not a big deal)

An alternative is JPEG XL.

Pros:

• While AVIF is better at low bitrates, JPEG XL at moderate and high bitrates - in general, across most of the spectrum.
• JPEG XL can losslessly transcode existing JPEG files, and restore them byte-for-byte at will. Doing so gets about 20% extra compression (compression from scratch is higher)
• JPEG XL is significantly faster than AVIF, both for encoding and decoding.
• Has a cool progressive-saliency mode, where (if used) it doesn't just load detail as it comes in (like progressive JPEG), but sends the detail that matters most first.
• No patent trolls.

Cons:

• No browser supports it natively, however (except in some cases by enabling certain flags). Google has actually gone backwards and removed their experimental JPEG XL flag.
• Not sure if there's a preexisting webassembly package for browser-side JPEG XL<->JPEG conversion (there is a Node.js package that might be a good starting point, however).

Again, though: I don't want this to distract too much from the main point, which is that we're pointlessly hurting quality (relative to file size) by overaggressive scaling, and this matters, and it should be fixed.

[eobet]

I also noticed images getting resized pretty much no matter what I did, and turns out, I was nowhere near guessing the maximum resolution currently allowed, which is 2.3 MP (there's digital cameras from the 90s with that resolution). 😅

Someone in Discord pointed out that the lowest resolution Mac you can buy has 4480x2520 and lots of PC users have 4K monitors.

Anyway, I like to author my own image files, and being somewhat responsible, I try to always keep them under 2Mb (often 1Mb looks good if I only author 2K, but that still breaks Mastodon's 2.3MP limit).

Here's an example I created after I learned about this limit (please click on it to see the 200% screenshot I took):

Both are crops from 1.5Mb large files (but wildly different resolutions), and I hope you can guess which one I'd rather upload to Mastodon.

Anyway, yes, migrating artists are beginning to note (even though the tech people who actually do the coding might not):

https://mastodon.social/@arvalis/109311924664462793

(Now, I may have posted a reply in that thread that I found the size limit acceptable, and I guess I do... but things could always improve and also, I'm not an artist.)

EDIT: Oh, and what I also don't understand at all is that this MP limit is enforced on PNG files as well... but they stay PNG... so, I guess not a lot of people upload PNG files, but you're apparently allowed to throw a several Mb large PNG on Mastodon without any real benefit (since scaling will ruin the lossless nature of it to a certain degree anyway).

[enn-nafnlaus]

So, I did an AVIF test using avifenc --min 0 --max 63 -a end-usage=q -a tune=ssim -j 4 on my several-years-old laptop, to compare performance and quality to (A) properly-compressed JPEG, and (B) what Mastodon is currently doing to shrink them.

So, as a reminder, I took a 4912x7360 test image, uploaded it to Mastodon, downloaded it (467k), cropped out the feet, and got this mess:

JPEG handled better - only doing a 50% scale and a ~33 CRF, we get this 470k file, which is significantly improved.

But what about AVIF? So, we can choose the balance between encoding speed (which in Mastodon's case could be done on the user end in WebAssembly on the user's browser using the above preexisting library) and quality for a given size; s=0 means slowest, s=10 means fastest. So I'll include speeds and qualities - cq-level=0 means lossless, while higher values mean lower qualities.

Keep in mind however that this is a huge 4912x7360 image; since the backend can't even handle more than 4096x4096, the worst case speeds would be half this, and most images much much less.

That said, let's start. But first, the original:

Now midspeed, at the Mastodon file size

AVIF s=6 cq-level=32
size: 468k
time: 7,5s

Almost perfect (some very slight quality loss on the socks). Now fastest:

AVIF s=10 cq-level=37
size: 463k
time: 3,9s

Again, big improvement over the proper JPEG, and no comparison vs. the Mastodon JPEG.

Now let's go down to half the size. First, midspeed:

AVIF s=6 cq-level=46
size: 233k
time: 6,9s

The socks are washed out, but there's no JPEG blockiness; I'd rank this about the same in subjective appearance as the proper JPEG that's twice as large. But vs. the Mastodon mangled JPEG, it's obviously far superior.

What about faster?

AVIF s=10 cq-level=37
size: 234k
time: 3,3s

Honestly, not much different than the slower version.

So, what about 1/4th the size of the Mastodon file? This time, let's start with the fast version:

AVIF s=10 cq-level=60
size: 119k
time: 2,6s

I'd call this slightly worse than the Mastodon version, but not hugely. But again, a quarter the size.

And midspeed?

AVIF s=6 cq-level=58
size: 112k
time: 6,7s

Much better - I'd call this similar to or slightly better than the Mastodon version. Under a quarter of the size.

And even slower?

AVIF s=4 cq-level=55
size: 117k
time: 26,5s

Even nicer!

Basically: AVIF would let one get at least 4x better quality or 4x less bandwidth than the current system, and at least 2x better quality or 2x less bandwidth than a properly handled JPEG (which I'll reiterate, we should do regardless).

The time/quality tradeoff also plays naturally into "giving people a higher-quality upload option. So for example, we could default to s=10 cg-level=40 and get nearly double the quality at less than half the bandwidth with a compression time of under 2s for a max res (4096x4096) image, a fraction of a second for most images (could be done in the background). But if someone chose a high-quality image, we could default to s=5 cg-level=32, which would take 6-7 seconds for a max 4096x4096 image, and the result would be the same size as a Mastodon image today, but vastly higher quality. Said delay would discourage people from using this option unless they actually needed it.

Net result: Mastodon's image bandwidth and storage usage is basically halved, normal image quality doubles, and high quality image quality becomes truly high quality.

Unknowns / complications:

• These tests were done with avifenc. I have no clue the performance of the WebAssembly browser encoder/decoder; I'd guess it's somewhat slower, which may require adjusting the above parameters. (Then again, these tests were also done on a couple-year-old laptop)

• For non-up-to-date browsers that don't have native AVIF support, I'd also have to test how WebAssembly decoding would affect browsing performance. Probably fine, but would need testing.

• I don't know what percentage of browsers are "legacy" (like IE) that can't support WebAssembly and would need JPEGs served to them. And beyond that, how much caching generated JPEGs would benefit serving legacy clients.

So, again, don't let this distract from the need to fix how Mastodon ruins images (relative to file size) at present by seriously overshrinking them. I just bring it up for the possibility of freeing up significant space and bandwidth on servers while simultaneously offering significantly better image quality.

(I imagine that how videos are handled could probably also be worth a relook... not sure how much space/bandwidth is images vs. videos at present)

[jmaris]

Perhaps it is also worth considering switching from JPEG to WEBP, as I believe it could allow for some considerable space savings server side while preserving more quality.

[enn-nafnlaus]

WebP would be sort of an intermediary step. Chrome has had it basically forever, Opera nearly as long, but it's only been in Safari and Firefox for a couple years, with various limitations. Never in legacy browsers like IE. So we'd still need a browser-side converter and/or, where conversion can't be done, the server to serve jpegs to legacy clients. So the same issues come up for dealing with legacy clients. That said, vs. AVIF, those couple extra years definitely do make a difference:

In terms of a general distribution of browsers, here's AVIF support (these numbers will of course all grow with time - particularly Safari on IOS, whose support was only added recently, but which should spread quickly):

Here's WebP support:

And here's WebAssembly support (for client-side converters, e.g. AVIF encoders/decoders, WebP encoders, etc):

WebP is generally considered better compression than JPEG (though the real-world amount is disputed), but not to the standards of AVIF or JPEG XL.

In these discussions it would be of use to get a good sense of what is the typical distribution of image requests and how high the typical per-request processing needs are. Because that will determine the impact of occasionally having to do a serverside conversion for a legacy client - the combination of (A) how rare such clients are, times (B) the difference in processing time if you have to convert a file serverside, times (B) what are the odds of getting a cache miss for a (temporarily) cached converted file.

... On the other hand...

Even in the case of a large number of legacy browsers requesting diverse files that can't benefit from a reasonable-sized cache - which is unlikely to be the case, but assume it is - if you have to store legacy file formats alongside a modern file format to have them always at the ready, you could still serve modern file formats to the vast majority of clients. Which would mean lower bandwidth and disk IO demands on the server. Aka, you'd increase storage requirements but decrease other server requirements. And it would certainly be possible to store lower-quality images for legacy clients than for modern clients.

So basically: any discussions of serving alternative file formats are data-dependent (unlike, to reiterate, discussions of fixing the current JPEG behavior :) ).

[renchap]

IMO any additional format should not be handled by Mastodon itself, but by a dynamic image proxy like I mentioned before, using the Accept header to deliver the best format accepted by the client (see here for imgproxy).

[enn-nafnlaus]

IMO any additional format should not be handled by Mastodon itself, but by a dynamic image proxy like I mentioned before, using the Accept header to deliver the best format accepted by the client (see here for imgproxy).

Agreed. With the caveat that the discussion of the odds of a cache hit and the processing impact of a cache miss still applies.

It should also be noted that (to resurrect an earlier discussion) any identification of duplicates would further reduce proxy conversion & caching needs.

But none of these things are "all or nothing":

• Fixing current JPEG sizing / compression can be done on its own
• Having a lower standard default quality but a (less convenient) option to upload high-quality images can be done on its own.
• Storing incoming files as e.g. AVIF and serving them through imgproxy for legacy clients can be done on its own
• Converting files to e.g. AVIF clientside to avoid needing a serverside conversion can be done at any point after that***
• Identifying image duplicates with fingerprinting can be done at any time independent of the four steps above

Nothing is all-or-nothing. Each step would provide a significant improvement to performance, quality, or both.

*** Actually, I'm starting to re-evaluate the importance of this aspect. Surely the amount of server resources for serving an image are far higher than for processing its upload - even with a processing-intense conversion - given how many times each image has to be served... Plus, serverside processing can run at low priority. Again, though, it all depends on the data...

[31SFX4]

Rather than any hard-coded limit, better allow the instance administrator to set the size for their instance. This would probably be fairly trivial to implement by making max_image_size a configuration option.

You're right that artists want bigger pictures, but many admins want to reduce storage and processing power as this costs a lot of money.

Giving admins the option to set it for their instance makes it possible that certain instances for artists have higher limits, while those that are run by volunteers (who cover their own cost) make them smaller.

This also then makes all discussion about the best size here unnecessary, because that will be up to the instance administrators. And the requirements are likely to change as technology advances.

[enn-nafnlaus]

This also then makes all discussion about the best size here unnecessary,

This thread isn't about "what's the best size". :) The discussions here are about:

1. Fixing a bug in how JPEGs are made that is throwing away quality for zero gain whatsoever (or contrarily: wasting server resources for nothing at all): that is, to say, an overreliance on decreasing resolution to control file size when you get better results by reducing the JPEG quality factor. A related bug is the fact that it'll do the same with PNGs but still store them as (huge) PNG files.

2. Other things that can be done to even further improve quality and/or reduce server needs: AVIF, WebP, identifying duplicates, etc.

Nobody in this thread has been against letting servers configure parameters :)

[31SFX4]

@enn-nafnlaus

The whole issue is called "image resolution limitations" and in the original post you argue that the resolution should be higher, so don't understand what you mean by "this isn't about 'what's the best size'".

The thread has then drifted off to a lot of other ideas (like file formats) that perhaps should be moved to separate issues?

My suggestion was about the original issue (resolution) and suggest a solution by making it easily configurable. As far as I understand the code, it is not configurable at the moment (unless the admin is willing to change the code on their instance, which brings all sorts of problems).

[enn-nafnlaus]

The whole issue, smatthie, is that the image resolutions are for no good reason. The actual limitation should be file sizes. The resolution limits are an attempt to mimick a file size limitation, but they're a bad attempt to do so.

I'll repeat, nobody here has come out against letting servers configure their own file size limits - although it may not be as simple as you're picturing, as Mastodon is federated, so if people start sharing files on one instance they're going to propagate to people on others. Still probably a good idea, but one does have to think through the consequences.

ED, re: the below: I'm not trying to be aggressive here; I would just have appreciated if you had read more than the title. At least having read the first post.

[31SFX4]

@enn-nafnlaus Right. Can you tune down the aggressive tone please? I made a suggestion in good faith, take t or leave it, but don't attack me for suggesting it.
I'm out.

[eishiya]

As a pixel artist, I often want to upload PNG images that have large resolutions (pixel art scaled up with Nearest Neighbor, which keeps it sharp) but very small file sizes due to them having small palettes. Mastodon scales them down and makes them blurry. The same thing happens when I share my B&W artwork, which by its nature makes quite small PNGs. In many cases, the "reduced" image actually ends up having a larger file size than the original! Any kind of reprocessing or compression is likely to kill the sharpness of pixel art, so it's important to keep the file as-is unless it poses an actual problem.

So, I'd love to see a file size limit with either no resolution limit or a very high resolution limit (many browsers choke on very large images even if the file size is small, so a limit of something like 10kx10k might be reasonable - but this limit should be left up to instance admins).

[enn-nafnlaus]

A good point, eishiya - I had been idly thinking of something similar. Files below a certain size could just be get kept as-is with no processing whatsoever (or further lossless-compressed if possible?). In an ideal case, if we had options for both standard and high-quality uploads (the latter requiring extra steps or delays to discourage overuse):

If the uploaded image is less than a fixed amount, leave it as-is or further lossless-compress:

• Standard quality: <150K
• High quality: <300K

Otherwise, if compressing with 95% of the original quality would be less than...

• Standard quality: <250K: Compress with 95% of the original quality
• High quality: <500K: Compress with 95% of the original quality

Otherwise, use whatever compression level is necessary to compress to a target of approximately:

• Standard quality: 250K
• High quality: 500K

Potentially server-configurable.

A currently uploaded high-res JPEG is shrunk to the ballpark of 500K, so we'd be looking to roughly half the server image resources (since the vast majority of images would be standard quality), while simultaneously - via eliminating overaggressive scaling and the use of more modern formats - greatly improving image quality, esp. where it's needed most.

What size are your PNGs, out of curiosity? And what would your take be on further lossless compression of your small images if it would save space? Also, Gargron says there's a 4096x4096 processing limit - do you find that it a serious issue? Note that we basically never approach that today due to the overaggressive cropping.

There's another question that comes up here, which is of metadata, whether it should be stripped or kept. I'm in the "keep" camp, but I understand there's reasonable arguments for stripping too.

[enn-nafnlaus]

Just heard back from an instance admin:

https://fosstodon.org/@jonn@social.doma.dev/109353969124435471

Text: hundreds of megabytes (500 MB), Videos: thousands of megabytes (5 GB), Images: tens of thousands of megabytes (25 GB).

So images are the real culprit here. If we were to halve image storage requirements we could nearly double capacity! Doing all of the steps described above (including fingerprinting to eliminate dupes) could probably cut image storage size by an order of magnitude while simultaneously improving quality.

[eobet]

As I have said, if user want's to upload 50Mb file he should be able to do it. Server just resizes it and instance could have config flag to save the original (or not). UI could provide link to the original file if it is found.

Dividing content to different services is not going to work when Twitter provides single service that could do all easier.

Since size is the main stopper for post migration, imagine what users uploading 50Mb files would do to that! But I already suggested paying for that feature, so why not do the same for huge files for those who want it?

Btw, Twitter isn't the only service, there's also Instagram, Flickr etc which are image focused... and unlike those services which are closed and isolated, I believe Pixelfed which I mentioned above can integrate with Mastodon since they're both based on ActivityPub...

[enn-nafnlaus]

You're making the major error of assuming that every image would be 50MB. That's like the top 1% of image sizes that people post, vs, say, memes and whatnot. Fixing how we currently (mishandle) images would free up far more bandwidth than allowing the comparably rare case of 50MB uploads.

That said, if desired, I fully support the notion of making users jump through added hoops to upload large files vs. casually posting images (extra click-throughs or doing manual steps on their end), to discourage needless posting of oversized images. One could certainly make an extra "hires image" checkbox on the button and/or an "Are you sure?" dialog, but an even simpler way might be to force them to convert to WebP or AVIF on their end - since very few people work with those formats natively, and they're also well-compressed formats. So just put smaller file size resolutions on JPEGs and PNGs but much larger limits on WebP or AVIF, as an option for those who need the resolution, and rely on the extra step to discourage casual users.

[TeroOjala]

As I have said, if user want's to upload 50Mb file he should be able to do it. Server just resizes it and instance could have config flag to save the original (or not). UI could provide link to the original file if it is found.
Dividing content to different services is not going to work when Twitter provides single service that could do all easier.

Since size is the main stopper for post migration, imagine what users uploading 50Mb files would do to that! But I already suggested paying for that feature, so why not do the same for huge files for those who want it?

Btw, Twitter isn't the only service, there's also Instagram, Flickr etc which are image focused... and unlike those services which are closed and isolated, I believe Pixelfed which I mentioned above can integrate with Mastodon since they're both based on ActivityPub...

Just exclude the originals from migration and transfer only resized files so receiving instance doesn't have to reprocess everything.

Twitter is not the only service, but clearly it's the main competitor for Mastodon and Mastodon tries to emulate Twitter-experience. It just falls far behind Twitter in media handling.

[nazgum]

just to throw my name in the hat for this issue, im in the process of moving over from twitter as an indie game dev and looking to share mostly gameplay photos and gifs; and the poor image quality was immediately noticeable and off-putting.

cheers,

[eobet]

Is this topic broad enough to include movies and GIFs, or should I open a new one?

When I author really optimized webm files and upload them, they get compressed to hell, and spat out by Mastodon (a 2.4MB webm became a 800k mp4).

But when I upload a 31MB mp4 file because I'm tired of wasting my time on this, Mastodon gave me a 32MB mp4 back.

I mean... 😵

[ldexterldesign]

👋

@enn-nafnlaus thanks for running with this, since #7242 back in 2018 (almost 5 years ago 🤯) - it's clear you've considered a lot

I think it would be useful to move on from this issue ASAP, using @enn-nafnlaus's proposed #1:

While in previous discussions (#7242) the talk was of quotas, rather than waiting on quotas, I propose a simple solution that should be a trivial change:

1. Ditch the resolution limit and stick only with file size limits - the aspect that actually matters. Excepting at the extreme low end of the CRF range, higher jpeg compression is better for image quality than lower resolutions, for a given file size. Let people choose that, and don't forcibly rescale an image that's under the size limit.

... then focus on #11808:

Originals won't be kept unless a media quota system is implemented. @Gargron - #7242 (comment)

Ultimately, it's worrying file originals aren't properly handled

Once control (for admins) and transparency (for users) exist - in terms of file uploads - then everyone should be a lot happier

--

I can see why it is discussed here but I think the nuances of file formats/recoding/display should be left to later / another issue, where I hope "progressive enhancement" is more of a consideration and less of a worry

CC @mhkhung - thanks for PR 🙏🤞

Tangentially, I raised #17037 in 2021

I see this issue is going the same route while losing hope of addressing the original problem (i.e. "As a user, when I upload images, I want them to display in full resolution so users can read text") 🫠

--

Aside, I see this is the 5th most commented issue in the repo' so I hope owners/contributors are watching carefully, even if they don't seem to be adding (much) input/direction..? 🤷

Hope this helps

Regards

[Metamere]

As an artist, this is a critical issue for me. I've noticed that my large resolution, small file size abstract art .PNG images get resized down to a small fraction of the resolution, losing a lot of detail, while increasing the file size considerably. This just doesn't make sense. I could easily upload 39.9 MB videos all day long on Mastodon, yet it's impossible for me to upload a sub-one-megabyte .PNG that's sized to fit modern high resolution displays. Based on everything in this thread, I find it rather concerning that progress on correcting this issue wasn't started a long time ago. It makes it feel like the arts and sustainability aren't priorities for those who are developing this project.

Edit: As an example, I uploaded an .PNG image a couple of weeks ago that was sized to fit a 4K display, at 3766x2596. It was automatically resized to 1734x1195. The file size changed from 115.78 KB to 154.22 KB. So 33% larger file size, all for only 21% as many pixels as the original. Very discouraging.

[Shanesan]

If the chief idea is that "people don't really notice" or something, I notice these very poor quality uploads easily on a phone screen. No matter what the tech limitations are, that shouldn't happen.

So if we're looking for votes or something here as it seems that quality is optional, I vote to change the current solution to nearly any choice above, or at least have an option to move to a choice above if your server can handle the requirements.

[ghost]

It seems like the limitation also affects caching remote content.
Follow some artists who use a different Fedi software where such a limit does not exist.
So, they are uploading great-quality pictures, but Mastodon refuses to handle them at all.

[eobet]

It seems like the limitation also affects caching remote content. Follow some artists who use a different Fedi software where such a limit does not exist. So, they are uploading great-quality pictures, but Mastodon refuses to handle them at all.

Can I ask if that is Pixelfed or something else?

[ghost]

This specific example is from an Akkoma instance, https://fedi.absturztau.be to be specific.
Picture is 3420x5135, 1.3MB

But I have seen it with pixelfed and calckey too.
Guess it happens whenever someone "dares" to upload a photo with more than 2MP to an instance that does not resize the image.

The remote image will not be loaded into the local cache and the media proxy itself also just returns 404

[TeroOjala]

This specific example is from an Akkoma instance, https://fedi.absturztau.be to be specific. Picture is 3420x5135, 1.3MB

But I have seen it with pixelfed and calckey too. Guess it happens whenever someone "dares" to upload a photo with more than 2MP to an instance that does not resize the image.

The remote image will not be loaded into the local cache and the media proxy itself also just returns 404

I also confirm this. When I modified my instance to allow bigger images/videos, it broke compatibility to other Mastodon-based instances. Media handling is really shitty in Mastodon.

[tallship]

As an artist, this is a critical issue for me. I've noticed that my large resolution, small file size abstract art .PNG images get resized down to a small fraction of the resolution, losing a lot of detail, while increasing the file size considerably. This just doesn't make sense. I could easily upload 39.9 MB videos all day long on Mastodon, yet it's impossible for me to upload a sub-one-megabyte .PNG that's sized to fit modern high resolution displays. Based on everything in this thread, I find it rather concerning that progress on correcting this issue wasn't started a long time ago. It makes it feel like the arts and sustainability aren't priorities for those who are developing this project.

Edit: As an example, I uploaded an .PNG image a couple of weeks ago that was sized to fit a 4K display, at 3766x2596. It was automatically resized to 1734x1195. The file size changed from 115.78 KB to 154.22 KB. So 33% larger file size, all for only 21% as many pixels as the original. Very discouraging.

Kinda bass ackwards. Not surprisingly, yet another reason people are increasingly migrating elsewhere - other Fediverse server platforms.

The part that really got me here was:

So 33% larger file size, all for only 21% as many pixels as the original. Very discouraging.

That's pretty bad. But fortunately, even as more folks consider mastodon as deprecated, every other platform accommodates, so it almost doesn't even matter anymore.

[RokeJulianLockhart]

I support solution #1. I don't think there should be a limit, since high resolution screens of nowadays demand high resolution images, and anything that can be vectoral generally is too, due to the ubiquity of support for .svg in browsers.

Can we make a poll for which to use, despite whatever technical decision eventually becomes consensus amongst the developers? I think it would be useful to show what the users and admins actually want, despite its feasibility.

[ghost]

Could we please stop to make a political debate about it and just give control to the instance admins?

[Juppstein]

Not sure where you see a political debate but otoh I support your suggestion to implement this as a per-instance setting.

[Shanesan]

I haven't seen you post here in a little bit @enn-nafnlaus. Any new perspectives on this? Also @stnfrd there is debate (it's not political) because "just give control to the instance admins" essentially means either "program everything" or "make images a hard-mode self-configuration which may not be compatible with other instances" and that's not a good solution. There's a lot of different formats for images that have all sorts of different pros and cons, and making sure each one works and supporting several standards over the long-term is not a great option with limited programmers (and if there were plenty of programmers, we'd have seen pull requests here). Minimizing choice that works for most people with a backup for servers that can't handle the load would probably be best.

I support the AVIF proposal at a maximum of 5K resolution (issue #21872) with thumbnail -> full window (2073600? Same as old max) -> "original" (14745600) renders specifically because:

1. AVIF is much faster to decode & smaller in bytes vs JPEG-XL (though XL can be more lossless), while being comparable in transcoding effort (source).
2. You can view AVIF's in all supported browsers, the majority of used operating systems outside the browser without additional software, and viewable on most current image editing software.

I have the following concerns:

1. Saving every single "original" 5K render on every federated instance seems like a waste, especially with how often it will actually be viewed. Perhaps the "original" should only pull "on request" and then stores it for next time.
2. Though I wish the time to render was faster, most instances will not suffer from the rendering timeframe. In reality, the only places who would have a rendering bottleneck may be, unsurprisingly, mastodon.social. A flag should be added to the code to have an option for a quicker and dirtier method (perhaps a fallback to how Mastodon is doing it now) until a solution is proposed.

I don't believe these concerns should hold up a release (except for the flag in concern 2).

I'm of the hope someone will take this and run with it, submitting a pull request and it hopefully being approved. Even if it's not approved (and from what I've read here, there's whatabouts, not actual technical hurdles), I plan to use it.

[eobet]

Just a FYI for this thread... Gargron confirmed on Discord that this PR is on track for 4.2:

#23726

[ghost]

@Shanesan @Juppstein

You are right, calling it political was a bit too much, I am sorry for that.
Just a bit annoyed about stuff like this #20255 (comment)

Whatever we do and discuss here, the idea about switching to a different format, the complaints that the reencoding of mastodon is actually making files bigger than they were before.

Nobody in the team ever entered this discussion, it's again a decision made mostly without us.

However, I still favour giving control over the limit to the admins.
Glitch-soc shows that it is relatively easy and doable and would solve some (not all) issues on the admin side.

Also, the compatibility part is more of an issue now, which would be fixed by it and not the other way around.
The fediverse is more than Mastodon, so we need to expect images bigger than the Mastodon limits anyway, which is currently causing issues, as I've mentioned in #20255 (comment)

Or, let's think about the other side.
If I am unable to reencode all those pictures and videos, I should be able to lower the limit if I want to.

Sure some users might don't like that, but it's kinda what the feidverse is about. Every instance is different and has different benefits and issues and you choose which you like the most.

[humantex]

I think there may be some confusion by some of the comments, and maybe some clarification would be helpful. From the initial post and follow ups from some of the devs, the overview and complaint has to do with proposed changes to image/video parameters for the maximum allowable file size and resolution limits, but there are 3 distinct considerations at play and lots more behind that, and some posters here may be mixing apples with oranges to muddy it even further.

There are other underlying considerations, but the 3 groups of size/resolution parameters at the uploading and processing point are usually: 1. System defaults, 2. Minimum/preset limits (i.e., requirements to accept a file), and 3. Maximum allowable limits. Underneath those are the core abilities available at the server level, for what coded components and libraries will be used to perform all of the media-related duties for what and how it will store/render/transfer/compress/encrypt/etc., whatever media files are allowed into storage, and in what compatible formats they can be processed and presented. Those components will dictate what any/all the settable parameters could ever be, and usually, not all possibilities are available to every component code that's selectable by the devs.

From what I've gathered so far, beyond those parts comes everything necessary to share the original data between instances, and I think that's the part that has the greatest impact on how it will all work without causing serious incompatibilities and glitches going forward. If there are no reasonable presets in place to handle the Fediverse aspects, then I'll assume that users will be right back to filing a new bug report about how that piece is now just as big of an issue as this one.

As an generic example... Instance A is using the defaults for all media formats, regardless of what those defaults might be. Instance B's admin has set all parameters to their max level (again, regardless of what they'd be), and Instance C has everything set to the minimums. Every instance may have flawless performance in how it handles the incoming/storing/rendering of every file. The client software for each user may also flawlessly process and present those same files to everyone who's on the same instance -but- only Instance C would be certain of presenting their stored files to any/every other instance's users without fail, unless there are an additional set of reasonable size/resolution presets to mandate some norms that won't fail on every bit of shared media that's stored outside the absolute minimums.

There must be some shared parameters that can be served to any other requesting instance, and my suggestion would be that a choice has to be agreed upon on, on exactly where that compatibility layer needs to be (either client or server), and that there's also a failover and an override at the requesting instance for failed transfers or bad conversions if they'd ever happen, i.e., imageMagick timed out on a lengthy downsize and convert process, and no file was created or it was corrupted. Each instance must assume there will be an agreed upon and de facto standard that will always allow a reasonable quality representation of the original media. Wherever a file is served from isn't as important, but that the Fediverse as a whole has a good, known, written-in-stone-for-this-version sharing standard that will work - regardless of any instance's specific media-handling setup.

If the intent should always be to provide the best quality representation of the original to the client, then devs shouldn't be pushed to get lost in the minutiae of how to get it there - fully formed in the first go. They should be allowed to put something reasonable in place first, and do it in such a way that it can be modified and fine-tuned without scraping major parts of the base code. Start somewhere that can be advanced upon if it needs it, but if there's no reasonable instance sharing standards in place to begin with, it might be doomed before it's fully out of the box.

[cooljeanius]

I support solution #1. I don't think there should be a limit, since high resolution screens of nowadays demand high resolution images, and anything that can be vectoral generally is too, due to the ubiquity of support for .svg in browsers.

Note that svg support is a separate issue: #6569

[MichaelZuo]

It's still unclear why exactly handling of normal PNG images is broken.

I don't understand the libraries enough to understand why exactly the code degrades their resolution and simultaneously increases file size, can someone explain?

[j12i]

Just posting the exact pixel size limitation here (before #23726), as it's hard to find that number with a web search. Relevant for when you want to resize your images yourself before upload to prevent Mastodon from mangling them.
The number of pixels an image can have on Mastodon currently is 2073600.¹ (Derived from "Full HD", 1920*1080, but you can have other aspect ratios.)

edit: I wasn't aware the PR went through already. So if you're on Mastodon 4.2+, the number is 8294400.

Footnotes

1. source in the code ↩

[eobet]

Has anyone tested what happens when you go over the new limitations?

I don't want to spam my timeline, but I noticed that when I uploaded something that went over the limitation, I was resampled straight back to the old 2 megapixel limitation... but if I upload something under the new 8 megapixel limitation, the resolution is kept (although the image is always somewhat aggressively recompressed).

[vmstan]

Closing as completed with #23726 being merged and available in Mastodon 4.2, if there are specific implementation issues please open a new issue.