Hello. I will be taking fedia.io offline on Saturday, Feb 3 to rehome it to a different server. Since moving fedia.io to a container-based platform, federation has been very unreliable and slow. I'll be moving to a non-container-based/bare metal install, which should address the federation issues.
I’m blocking probably hundreds of mags (all mags on Cloudflare sites like lemmy.world). The results of search queries are rich in posts from blocked mags. When I visit the post the sidebar shows that indeed that mag is blocked....
When I look at https://fedia.io/all/newest right now there seems to be very little getting through today, with only 3 new threads in the past 10 hours....
I am going to apply this upgrade this weekend. I spent a few hours looking through logs this morning but didn't see anything obvious (there are a breathtaking amount of logs) so gave up and restarted the containers. Will look again when it fails after I make log storage persistent
I tried to block a couple communities from reaching my timeline and got the 500 error. These are the URLs the browser tried to visit when clicking the “block” button:...
Uploading GIFs on this site has been hit-or-miss for the few months I've been here, but where they used to fail with an error page a while back, the GIF is now quietly discarded from the thread. Posting GIFs to this instance is broken from kbin.social as well (where the whole post fails) so it's not a front-end issue.
Looking at the (short!) list of defederated instances, I tried looking through this magazine to read about what caused some of the defederation actions -- unfortunately it looks like some of that information might've been lost in the mbin migration and subsequent cleanup?...
Most were suspended for child porn at various points - though I think it was not necessarily something they condone, but lemmy has damn near no ability to moderate or remove images once they land on a server
Hi all. I am going to be taking fedia.io offline for a bit today to move to a new server, switch to a containerized version, and some other enhancements. Thanks for patience.
Ok. Maintenance completed. I've moved fedia.io to a container-based setup that is now running on my main infrastructure - a 48 core/96 thread AMD epyc zen4 genoa with 256GB of DDR5 ECC ram and 2x4TB nvme SSDs, backed by a dedicated database server with the exact same specs, on a 10gbps network.
I was happily working on a different set of servers when apparently the database server for fedia.io became unresponsive. Once I realized it, I forced a reboot and everything is back to normal. Now that the code and database are stable(ish), I will be moving the site to actual server hardware in the coming days while I have...
I think the issue now is with the containerized version of mbin. I tried saving money by running it in my container host, but it's clear to me that is not going to work out. So I will be rolling it back to a bare metal server :/
@jerry My comments and boosts from before the downtime last week have disappeared from this instance. They still appear on other instances, and they are still counted in the comment section on magazines here but are invisible (the comment count on my profile only includes the ones I made after the server came back online).
@iamnomad some of existing data was so malformed that I couldn’t easily fix it in a way that would allow the site to continue operating. I have been (slowly) attempting to piece it back together, though I am not convinced it’ll work in many cases.
Hello everyone. I will be taking fedia.io offline in a few hours (timing dependent on work meetings) to apply the latest mbin updates, which should, among other things, fix 2FA....
Can you give it another try? It looks to be correlated with php running out of memory (not sure why your session would require more that 512MB of ram, but lets confirm that is the case first.
ok - I bumped up the memory limit in php to 1gb and that appears to have fixed the problem. I have no idea yet why it takes so much memory, though. But I'm glad it worked
I did not update last night - there are many in flight code changes and I’d prefer not to cause more problems, so letting that settle down for a bit before I update.
I spent an incredible amount of time over the past few days unf*cking the fedia.io database. It should be clean now and that should both dramatically improve performance and fix all or most all of the remaining error 500’s
As part of the database clean up, I had to remove many corrupt records, so the answer is yes there are missing posts and comments, however I do have backups have I have been attempting to find ways to reinsert them without recreating the problems. I’ve not had much luck with that yet.
Scheduled downtime for fedia.io
Hello. I will be taking fedia.io offline on Saturday, Feb 3 to rehome it to a different server. Since moving fedia.io to a container-based platform, federation has been very unreliable and slow. I'll be moving to a non-container-based/bare metal install, which should address the federation issues.
Lemmy once again isn't federating properly?
I get the threads, but there's no votes, no replies. Nothing. None of my comments show up anywhere if I look at the original instances....
Periodic issues connecting to https://fedia.io
Hi,...
Bug: blocks have no effect on search-generated timelines
I’m blocking probably hundreds of mags (all mags on Cloudflare sites like lemmy.world). The results of search queries are rich in posts from blocked mags. When I visit the post the sidebar shows that indeed that mag is blocked....
Federation is more broken than usual
When I look at https://fedia.io/all/newest right now there seems to be very little getting through today, with only 3 new threads in the past 10 hours....
error 500 when trying to block some communities
I tried to block a couple communities from reaching my timeline and got the 500 error. These are the URLs the browser tried to visit when clicking the “block” button:...
Test
Flaky GIF uploads
Uploading GIFs on this site has been hit-or-miss for the few months I've been here, but where they used to fail with an error page a while back, the GIF is now quietly discarded from the thread. Posting GIFs to this instance is broken from kbin.social as well (where the whole post fails) so it's not a front-end issue.
Several hour delay on both incoming and outgoing federation
Sorry to bother you again jerry 😅...
403 forbidden when DMing a Lemmy user
When I try to DM someone on the jlai.lu lemmy instance, I get “403 Forbidden”....
Request: pinned thread tracking instance defederation
Looking at the (short!) list of defederated instances, I tried looking through this magazine to read about what caused some of the defederation actions -- unfortunately it looks like some of that information might've been lost in the mbin migration and subsequent cleanup?...
Fedia.io downtime for maintenance
Hi all. I am going to be taking fedia.io offline for a bit today to move to a new server, switch to a containerized version, and some other enhancements. Thanks for patience.
It's always something... [sorry for the downtime]
I was happily working on a different set of servers when apparently the database server for fedia.io became unresponsive. Once I realized it, I forced a reboot and everything is back to normal. Now that the code and database are stable(ish), I will be moving the site to actual server hardware in the coming days while I have...
Fedia submissions stopped federating out again?
I've noticed that recent submissions made in both @LoLFanArt and @FloatingIsFun don't seem to have federated out:...
Brief maintenance outage evening of Oct 31
Hello everyone. I will be taking fedia.io offline in a few hours (timing dependent on work meetings) to apply the latest mbin updates, which should, among other things, fix 2FA....
Is there a problem with 2FA?
Is anyone else having the QR image not generate when trying to setup two-factor authentication?
error 500 when visiting the “we’re back” thread and also notifications
I cannot view the “we’re back” thread -- just get the error 500....
We’re back
I spent an incredible amount of time over the past few days unf*cking the fedia.io database. It should be clean now and that should both dramatically improve performance and fix all or most all of the remaining error 500’s