Federation is more broken than usual

ciferecaNinjo , 4 months ago

Not sure if this is related but I just received a notification for a new reply and it turned out to be a quite old comment from 5 months ago:

https://fedia.io/m/homeimprovement@lemmy.world/t/173551/Most-chimneys-in-my-neighborhood-are-open-thus-water-enters#entry-comment-3109852

I block all mags from all Cloudflare instances. So sometime after posting to homeimprovement@lemmy.world I blocked it. I still get notifications (which is good), then I unblock it just to read the response then re-block. That’s not likely related, but something happened in the past couple days that apparently caused a “seen” flag to get unset. It’s not in itself worth chasing up but maybe it’s a clue as to whatever happened.

jerry Admin , 4 months ago

Hi all. I am working on it. I am trying to figure out what the problem is this time rather than just restarting the containers.

e-five , 4 months ago

if there's anything people can help with don't hesitate to ask in matrix!

One thing I wanted to raise to your attention was asdfzdfj's change to persist logging for docker setups which might make it easier to debug if your logs aren't persisted between container restarts

I mentioned in matrix but just in case since it's a bit of a gotcha for people that run in docker:

as a headsup for anyone running docker, there was a recent change to persist storage of logging just merged. it requires creating a directory and changing permissions on it. the admin guide was updated here but you may need to mkdir -p storage/logs and sudo chown $USER:$USER storage/logs where you run docker from by next sync/release

this may or may not be relevant to you depending on if sync'ing would change your compose setup

jerry Admin , 4 months ago

I am going to apply this upgrade this weekend. I spent a few hours looking through logs this morning but didn't see anything obvious (there are a breathtaking amount of logs) so gave up and restarted the containers. Will look again when it fails after I make log storage persistent

DarkThoughts , 4 months ago

Instances like feddit.de are like almost 2 weeks behind now, with the last threads being from right at the end of the Holidays.

e-five , 4 months ago

It's also possible there are no local subscribers, or weren't until now. It's one of the things we need to fix after updating subscriber counts to show accurate information rather than just local information, as it's unclear if a magazine has no local subscribers anymore

DarkThoughts , 4 months ago

I'm subbed to https://fedia.io/m/dach@feddit.de and it has not caused any federation of the newly submitted stuff since then.

e-five , 4 months ago

Hmm... looking at their instance list and stuff, I get the feeling they might consider fedia an "inactive" instance (based on the information being out of date), which is a bug lemmy has where after upgrades it marks instances as inactive and never updates it again. I will try to get in contact with their admins to confirm

Edit: quick edit, speaking with another german instance owner they said they have the same problem with feddit.de right now and have already spoken to the admins and already taken steps to reset active state but it isn't happening, perhaps more debugging is needed or maybe this is the lemmy federation issue people have been discussing

Edit edit: they say they're waiting for https://github.com/LemmyNet/lemmy/issues/4288#issuecomment-1879052639 to be merged which looks hopefully soon

kbal OP , 4 months ago

Same problem is back again I think.

kbal OP , 4 months ago

I assumed that it had stopped at some point ten hours before I noticed, but no. It's still going. It's still 10 hours behind as if we're experiencing some kind of time warp.

osaerisxero , 4 months ago

This might be the same issue that brought kbin.social to its knees, only fedia/mbin is giving up instead of the infinite retries which are crippling the other platform.

DarkThoughts , 4 months ago

One comment in one of the kbin threads of a guy who hosts his own forked instance said it's probably caused by a Lemmy update, which I guess makes sense. kbin and mbin aren't the same code wise (can't even use the nsfw unblur here), and kbin is also a version ahead of mbin so they might just handle the same issue differently. A lot of the magazines / communities I checked have their latest threads from like a week ago when viewed through fedia. kbin ironically has similar examples, but different ones to fedia. Whatever the cause, it obviously shines a negative light on any 'bin instance and to a degree onto the fediverse. For me, I would really like to avoid having to deal with the shitty Lemmy UI and support the Lemmy devs directly.

e-five , 4 months ago

I wouldn't exactly say mbin is behind kbin, as things have diverged much since the fork. For example, mbin has moderators from lemmy, and lemmy shows mbin moderators. As of latest, mbin gets moderator reports from lemmy (and maybe the other way, I didn't closely review that PR), things like sensitive groups are marked correctly, subscriber and follower counts come from source... Still, there's a lot of work to do for the user experience, many people have made that clear lately, so I'm hoping mbin will be able to gain the contributors needed to make the changes people want to see

Also, yea a lot of /kbin tweaks won't work. I do want to make an eye toggle button to make visible all nsfw on a page, but that might be hindered a bit due to infinite scroll (as well as a user setting to just have it shown by default). In the meantime though, it's just the structure of data changing from javascript to css, so it's always possible to do. I wrote a tampermonkey script able to do so as an example:

// ==UserScript==
// @name         Mbin tweak
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  Unhide NSFW entries automatically
// @match        https://fedia.io/*
// @require      https://greasyfork.org/scripts/12228/code/setMutationHandler.js
// ==/UserScript==

checkThem([].slice.call(document.querySelectorAll('input[type="checkbox"].sensitive-state')));

setMutationHandler(document, 'input[type="checkbox"].sensitive-state', checkThem);

function checkThem(nodes) {
    nodes.forEach(function(n) { n.checked = true });
}

based on this answer on how to check all checkboxes on a page

chris , 4 months ago

Yeah, latest threads from some of my subs are from yesterday. @jerry