Side note: Uploading pictures is so much faster since the migration. It’s almost like having pictures didn’t have to be a PITA 🤔
Heh, welllll… I just tried to upload one. I get an HTTP 400 from the post to the /pictrs/image endpoint and the response body is Request error: error sending request for url (http://pictrs:8080/image): operation timed out. I thought maybe because I was pasting a largish image so I tried a 350kb or so JPG and clicked the upload button …nope, same deal. Oh, and the UI tries to parse the body of the error response as JSON so it pops up a red box with SyntaxError: JSON.parse: unexpected character at line 1 column 1of the JSON data.
Thanks for your efforts and yay, that works here too! I can paste them even. Nice. Interestingly, I copied a JPG by right clicking in Firefox and it wound up as a PNG when I pasted that. But maybe that’s the browser? Still, it’d be neat if the site converts to something more efficient like WebP and maybe caps the res at 4k …though maybe storage is cheap enough at this point that we could each put a full res SotD from our phones for the next decade without really being a burden? …and now I wonder if the transcoding can be done client side via wasm or something. Yawn, I should sleep, rambling.
Oh! One thing though: earlier I tried pasting my SotD to another Lemmy instance to see if that works and it did fine. I didn’t post it and cancelled the preview. However the image stuck around. Wondered if and how those get mopped up, maybe a cron job? It seems to be there still some hours later, might check tomorrow just out of curiosity.
(heh I think you restarted while I was trying to post this so will also be interesting if the image persists, though presumably shouldn’t be orphaned now that it’s referenced in a comment?)
Yes, I restarted haha. I took a backup now that everything is working. It turns out the problem was caused by ME earlier today, changing the icon. I changed the banner, too (it’s Tabac now), but that didn’t hurt anything. Anyway, I had to modify the frickin’ database to get things working again. Ridiculous. Now I’m going to be nervous any time I want to change the banner/icon.
Your points regarding pict-rs (the software that handles pictures for lemmy) – to my knowledge nothing gets deleted. It’s a big of a problem long term, but like you said storage is cheap. One day we can even move all of that data and host is somewhere else if storage becomes a problem. Pict-rs is mostly clogged up by thousands/millions of thumbnails. It keeps thumbnails from every federated post in addition to the uploads from our users. Our pict-rs folder is currently sitting at 8-9GB.
I’d suggest WebP with a quality of 85 and max dimensions of 3840x3840 (because 4k is 3840x2160).
That takes djundjila’s photo for today (a 4236x2677 jpeg) from 1.9Mb down to 360Kb for a 3840x2427 WebP.
You might also consider AVIF as that’s widely supported as well and looking like it’ll probably become the new standard but for now is a bit CPU intensive until CPUs start adding SIMD for it. Too bad JPEGXL lost momentum, it was kinda better (and less CPU intensive). But if the server doesn’t mind chewing on it …that same SotD as a quality 66 AVIF is just 160Kb, eg. less than a tenth the size.
The weird part is I downloaded a photo that had been uploaded here, and re-uploaded it as the icon/banner (it doesn’t let you use a URL for the icon/banner). So that picture exists multiple times in our storage. It also crashed pict-rs, which in turn doesn’t let Lemmy boot. Initially my browser downloaded a .webp, but I changed the URL in order to download it in .jpg. I don’t have a good program that can edit .webp. So, maybe the problem was that I saved it as a .jpg, when it fact it was a .webp? The banner worked just fine, so I’m not sure. I only cropped the banner image, whereas I resized the icon.
Sorry, I just peeked at the config options and have never run any of this so no idea about the interaction with Lemmy or how to troubleshoot that. What I’d do though is spin up a toy instance on another subdomain/locally in a container and try to reproduce it there. Maybe with a copy of the data from production if that’s easy enough to pull off.
But ya only in retrospect - I wouldn’t have expected changing the banner or icon to be a risky operation.
Heh, welllll… I just tried to upload one. I get an HTTP 400 from the post to the /pictrs/image endpoint and the response body is
Request error: error sending request for url (http://pictrs:8080/image): operation timed out. I thought maybe because I was pasting a largish image so I tried a 350kb or so JPG and clicked the upload button …nope, same deal. Oh, and the UI tries to parse the body of the error response as JSON so it pops up a red box withSyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data.I’ll take a look later at why this might be. There are some limits in place to prevent huge files but I’m sure that wasn’t it.
Edit: it was pict-rs crashing on account of me uploading a new icon for the instance. Seriously, what the heck.
I got the same error message. I restarted everything and now I can upload no problem.
Why did this happen? I don’t know. Will it continue to happen in the future? I sure hope not!
Thanks for your efforts and yay, that works here too! I can paste them even. Nice. Interestingly, I copied a JPG by right clicking in Firefox and it wound up as a PNG when I pasted that. But maybe that’s the browser? Still, it’d be neat if the site converts to something more efficient like WebP and maybe caps the res at 4k …though maybe storage is cheap enough at this point that we could each put a full res SotD from our phones for the next decade without really being a burden? …and now I wonder if the transcoding can be done client side via wasm or something. Yawn, I should sleep, rambling.
Oh! One thing though: earlier I tried pasting my SotD to another Lemmy instance to see if that works and it did fine. I didn’t post it and cancelled the preview. However the image stuck around. Wondered if and how those get mopped up, maybe a cron job? It seems to be there still some hours later, might check tomorrow just out of curiosity.
(heh I think you restarted while I was trying to post this so will also be interesting if the image persists, though presumably shouldn’t be orphaned now that it’s referenced in a comment?)
Yes, I restarted haha. I took a backup now that everything is working. It turns out the problem was caused by ME earlier today, changing the icon. I changed the banner, too (it’s Tabac now), but that didn’t hurt anything. Anyway, I had to modify the frickin’ database to get things working again. Ridiculous. Now I’m going to be nervous any time I want to change the banner/icon.
Your points regarding pict-rs (the software that handles pictures for lemmy) – to my knowledge nothing gets deleted. It’s a big of a problem long term, but like you said storage is cheap. One day we can even move all of that data and host is somewhere else if storage becomes a problem. Pict-rs is mostly clogged up by thousands/millions of thumbnails. It keeps thumbnails from every federated post in addition to the uploads from our users. Our pict-rs folder is currently sitting at 8-9GB.
Just poked around because I was curious, it does look like the config for pict-rs lets you specify a format to convert things to and a max size/quality: https://git.asonix.dog/asonix/pict-rs/src/branch/main/pict-rs.toml#L237
I’d suggest WebP with a quality of 85 and max dimensions of 3840x3840 (because 4k is 3840x2160).
That takes djundjila’s photo for today (a 4236x2677 jpeg) from 1.9Mb down to 360Kb for a 3840x2427 WebP.
You might also consider AVIF as that’s widely supported as well and looking like it’ll probably become the new standard but for now is a bit CPU intensive until CPUs start adding SIMD for it. Too bad JPEGXL lost momentum, it was kinda better (and less CPU intensive). But if the server doesn’t mind chewing on it …that same SotD as a quality 66 AVIF is just 160Kb, eg. less than a tenth the size.
For more info about quality settings: https://www.industrialempathy.com/posts/avif-webp-quality-settings/
Nice, I’ll look into this stuff. Pretty cool.
The weird part is I downloaded a photo that had been uploaded here, and re-uploaded it as the icon/banner (it doesn’t let you use a URL for the icon/banner). So that picture exists multiple times in our storage. It also crashed pict-rs, which in turn doesn’t let Lemmy boot. Initially my browser downloaded a .webp, but I changed the URL in order to download it in .jpg. I don’t have a good program that can edit .webp. So, maybe the problem was that I saved it as a .jpg, when it fact it was a .webp? The banner worked just fine, so I’m not sure. I only cropped the banner image, whereas I resized the icon.
Sorry, I just peeked at the config options and have never run any of this so no idea about the interaction with Lemmy or how to troubleshoot that. What I’d do though is spin up a toy instance on another subdomain/locally in a container and try to reproduce it there. Maybe with a copy of the data from production if that’s easy enough to pull off.
But ya only in retrospect - I wouldn’t have expected changing the banner or icon to be a risky operation.