jerry avatar

jerry Admin

@jerry@fedia.io

Yet more stability issues with fedia.io

Yesterday, the fedia.io server locked up. I was able to reboot it remotely and it came up clean. After less than an hour, the server froze again. This happened several more times throughout the day. Unfortunately, there were no logs recording what happened, and nothing on the console - just frozen hardware....

jerry Admin ,
jerry avatar

I’ll check

jerry Admin ,
jerry avatar

It’s because I blocked lemmynsfw after it was reported for the 746272635th time.

jerry Admin , (edited )
jerry avatar

Is there a port of interstellar to Linux or Windows? Unfortunately I don’t have any android devices to test with.

Edit: never mind. I found them

jerry Admin ,
jerry avatar

Ok, I checked and creating new oauth clients is not restricted to administrators. I am trying to find a way to run interstellar so I can test and figure out what is wrong.

jerry Admin ,
jerry avatar

ok - I was able to find and fix the problem. it should work now.

jerry Admin ,
jerry avatar

I was mistaken - lemmynsfw isn't bblocked. There are only 3 blocked: lemmy.one, lemmy.today, lemmy.froztbyte.dev. Are you proposing that lemmy.ml, lemmygrad, and hexbear get the block?

jerry Admin ,
jerry avatar

lemmy.froztbyte.dev is/was sending malformed messages that caused this the queue runners to crap out. The others were for persistent harassment of one of our trans members.

jerry Admin ,
jerry avatar

Hi all. I am looking into why this is happening.

Apologies for the recent downtime

The server fedia.io had been running on started developing stability problems overnight from Thursday April 4 to Friday April 5. By Saturday (today), the system was completely unbootable. After attempting to resolve the hardware issue with Hetzner (the ISP) for about 6 hours, I gave up and moved the site to a new server. All...

jerry Admin ,
jerry avatar

Report at least one of the posts from any spam account you see. I try to suspend and delete them as I see them, but alas I am not omniscient and don’t know where all the spam accounts are

jerry Admin ,
jerry avatar

thanks. I think I've gotten rid of all the spam accounts, posts, and replies from the recent issue, but there may be more I am not finding.

jerry Admin ,
jerry avatar

It looks like a Miltonia

jerry Admin ,
jerry avatar

It’s really beautiful, whatever it is

jerry Admin ,
jerry avatar

Best of luck. I tried a keeping a few of those over the years without much success. Hopefully you can do better.

jerry OP Admin ,
jerry avatar

As a bonus, the most excellent mbin folk pointed out a problem in the database and that has additionally helped dramatically reduce the load

jerry OP Admin ,
jerry avatar

Hello. It is not. I think it is the largest. I am not sure there is an official instance for mbin the way there is for kbin or mastodon, etc

jerry OP Admin ,
jerry avatar

I am not - I have tried to help, and have been helped by the developers though

Fedia.io Downtime Saturday, March 9

I am going to be rehoming fedia.io to a new server. Today, fedia.io runs on shared app server and shared database server. I will be moving fedia.io to a dedicated 48 core/96 thread AMD epyc server with 256GB of RAM, which will house both the app and the database. This is less about fedia.io and more about the other apps...

jerry Admin ,
jerry avatar

it may be related to the ongoing federation problems. https://fedia.io/m/french@sopuli.xyz is working now for me. there are no posts in it yet on our instance because no one had subscribed to it until now.

jerry Admin ,
jerry avatar

I’ll take a look

jerry Admin ,
jerry avatar

Ok. I don’t know why this works, but if you search for the magazine (ie search for oregon@lemmy.world) it’ll return a result. After that, the links you posted starts working.

jerry Admin ,
jerry avatar

The Florida one does because I followed it and there has been posts since then

jerry Admin ,
jerry avatar

Also, I checked the source and there have been no new posts since. I'll keep an eye on it.

jerry Admin ,
jerry avatar

I am not sure what is going on with some of the lemmy.world communities - some do seem to be having issues like netsec, while others like https://fedia.io/m/news@lemmy.world are humming along very well. I'll keep poking at it.

jerry Admin ,
jerry avatar

actually, I'm wrong - the posts that appeared after I followed it are weeks old. I can't explain that.

jerry OP Admin ,
jerry avatar

The server is busily processing the 1,200,000 messages that queued up over the past 20 hours. It’s died 3 times in the past few minutes, so I’m not optimistic about how long this will take

jerry OP Admin ,
jerry avatar

Ah - that is what we’re here for. I know kbin has had a cloud of uncertainty around it. Did something recently happen on kbin.social?

jerry OP Admin ,
jerry avatar

Up to 1,700,000 in the queue 😱

jerry OP Admin ,
jerry avatar

The good news is that I think I figured out where the problematic messages are coming from. Now I have to figure out what it is about them.

jerry OP Admin ,
jerry avatar

it took 3 days to process the backlog, but it's caught up now and I've not seen any re-occurrence of the prior problem.

Request for errors on fedia.io

Hi all. I was recently made aware that people have been getting error 429's and other error 500's when visiting fedia.io. My hope/expectation is that those will no longer happen now that I've moved back to a bare metal install, but if you do experience that, please comment below, or if that doesn't work, send me an emai to...

jerry OP Admin ,
jerry avatar

Thank you. I will investigate

jerry OP Admin ,
jerry avatar

Thanks. I missed that and will go apply the upgrade now.

jerry OP Admin ,
jerry avatar

ok, fix applied.

jerry OP Admin ,
jerry avatar

I applied a fix that (hopefully) resolves this new problem. Please let me know if you continue to see it.

jerry OP Admin ,
jerry avatar

ok, I think this should work now

jerry OP Admin ,
jerry avatar

yes, there was a problem with rabbitmq that I found/resolved.

jerry OP Admin ,
jerry avatar

I did find a problem in the server, which I fixed. I am hopeful for no more problems.

jerry OP Admin ,
jerry avatar

With any luck, this will take care of the 429 errors too. I actually didn’t realize that was happening until I read the thread you linked to above.

jerry OP Admin ,
jerry avatar

Thank you for this. It is behind nginx, so I’ll likely have to do this

jerry Admin ,
jerry avatar

I will lift the ban again. This has happened twice in the past, and each time within ~12 hours, I've had more child porn landing on the server from them. If it happens again though, I will likely leave it that way.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • Mordhau
  • WatchParties
  • Rutgers
  • jeremy
  • Lexington
  • cragsand
  • mead
  • RetroGamingNetwork
  • loren
  • steinbach
  • xyz
  • PowerRangers
  • AnarchoCapitalism
  • kamenrider
  • supersentai
  • WarhammerFantasy
  • itdept
  • AgeRegression
  • mauerstrassenwetten
  • MidnightClan
  • space_engine
  • learnviet
  • bjj
  • Teensy
  • khanate
  • electropalaeography
  • neondivide
  • fandic
  • All magazines