@hrefna@hachyderm.io cover
@hrefna@hachyderm.io avatar

hrefna

@hrefna@hachyderm.io

SRE at Google. Queer. Poly :potion_polyamory: Trans :verified_trans: :nonbinary_potion: Engineer. Ace :flag_ace: Member of AWU-CWA. #ActuallyAutistic :rainbowinfinity: #UnionStrong

Opinions my own. Does not suffer fools gladly.

This profile is from a federated server and may be incomplete. View on remote instance

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

Cis people sitting with each other and saying "protect trans kids" to each other and then undermining that in every interaction with a trans person or in any opportunity to actually support trans people is…

…well, it's definitely a thing.

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

I find it worthwhile to go back now and again and look through analyses of why a piece of software succeeded or failed.

It's always particularly illuminating to look at three things:

  1. The role of leadership in the failure.
  2. How much leadership got paid regardless/what happened to leadership afterwards.
  3. What happened to the rank-and-file who actually build the software.

The story is always basically the same, with a few variations on the themes.

https://kotaku.com/how-biowares-anthem-went-wrong-1833731964

hrefna OP ,
@hrefna@hachyderm.io avatar

People will occasionally poke at Google's developers for something about one of Google's products and its like… I agree with you, as do most developers I know (not that I talk about specifics publicly in the moment).

You have to look above the team who is doing the work. They've probably already said their piece and were simply ignored.

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

This statement is going to live rent-free in my head for a long time to come, I suspect.

hrefna OP ,
@hrefna@hachyderm.io avatar

@adamhotep

That irony was not lost on me in the slightest.

@tchambers

hrefna OP ,
@hrefna@hachyderm.io avatar

@maegul

A few things.

First, that is laughably far from where we are today.

Second, to the degree we are there it has very little to do with ActivityPub and more to do with people believing in the vision of the fediverse.

Finally, given the lack of support for (and the massive resistance toward) quote-boosting, differing formats between platforms, etc it's hard to see how you can claim that the vision of not using screenshots is something that we are approaching.

@adamhotep @tchambers

hrefna OP ,
@hrefna@hachyderm.io avatar

@maegul

Further driving home that last point is the outright hostility in certain corners toward "bridge" technologies or towards anything that might look like it hypothetically may make it onto a major (or minor but wannabe major) social media platform.

@adamhotep @tchambers

hrefna OP ,
@hrefna@hachyderm.io avatar

@tchambers

One of about twenty examples, and another example of it being solved by people working around the protocol more than they work through the protocol.

Even then getting here required a lot people doing a lot of fighting over the course of multiple years, a lot of burnout, and involved a lot of derision and attacking of people who come from different online cultural experiences.

@maegul @adamhotep

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

Part of my frustration with and one of the things I find baffling giving everything else in it: the lack of tools for backpressure.

Backpressure is fundamental in building reliable distributed systems (c.f., Notes on Distributed Systems for Young Bloods). From a C2S perspective I get why it wouldn't need to be specified, but from a S2S federated protocol perspective its absence is frustrating.

All that it says is to take care not to overwhelm others and a bit on rate limits

hrefna OP ,
@hrefna@hachyderm.io avatar

@jenniferplusplus Yes. Yes. A million times yes.

There's no sync capability despite that we're basically building a distributed database where we are trying to have a sync'd state.

So if you fall off for a few hours what is expected to happen?

Everyone's favorite thing from C++: Unspecified.

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

Making LLMs sound more human while being a spicy version of Bob/Clippy is the opposite of what we should be doing, but here we are.

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

Why is every single gif search system a nightmare?

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

Man who does no real work claims that those who are actually working are not doing any real work. Film at 11.

hrefna OP ,
@hrefna@hachyderm.io avatar

Evidently "real work" is defined as "making enough money that the project doesn't get cancelled" which uh.

So my work is only 'real' if my project isn't cancelled… and who does the cancelling? The same people who you want to keep around. I see.

hrefna OP ,
@hrefna@hachyderm.io avatar

I recognize that the current party line among VCs is that we're all overstaffed and could be run more efficiently with half the people (c.f., twitter) but every time that is tried in practice it goes… poorly. Because they have absolutely no clue what real work actually looks like.

You would think they would figure it out eventually, but to them it is just an academic exercise in shuffling money around instead of involving tens or hundreds of thousands of lives.

tchambers , (edited ) to Random stuff
@tchambers@indieweb.social avatar

@mike- new numbers, but ones that tell the same story we discussed on your podcast: on my server, we have about 1 moderator to 500 users. Here are the new numbers on how that dynamic plays out for major social platforms in the EU:

"X does have the worst ratio of moderation staff to users, at 1/60,249, with LinkedIn coming in second (1/41,652), then TikTok (1/22,586) and Meta (1/17,600)."

https://www.socialmediatoday.com/news/x-has-significantly-fewer-moderation-staff/714650/

hrefna ,
@hrefna@hachyderm.io avatar

@thisismissem

Yes, this is an important distinction.

It should also be noted that this is not an apples-to-apples distinction:

  • How many moderator hours are being used? This is going to be largely homogenous between the major platforms, it is not homogenous for the fediverse.

  • How many moderators are paid? This is going to be largely homogenous between major platforms, it is not homogenous for the fediverse

  • How much duplication of effort is there?

etc.

@fediplomacy @tchambers

hrefna ,
@hrefna@hachyderm.io avatar

@tchambers

With respect: you are dipping into wishful thinking and/or propaganda.

What do these things look like for the ten largest servers, which make up 80% of the non-threads userbase?

You may also spend more time and know your users better, but that is not a uniform value.

You can argue that there are many advantages to the way that the fediverse handles these things, but comparing with the raw numbers from the EU report is not how you do that.

@thisismissem @fediplomacy

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

Basically:

If what you want is compatibility with mastodon and a clear upgrade path the amount of work to do that and maintain that at the database and protocol level is phenomenally nontrivial and it is both tedious and time consuming, with virtually nothing out there already built to make it easier

Further, you are then contributing to enshrining mastodon's interpretations as Correct™ even where they diverge from the protocol as written.

That can all be fine, but recognize the situation.

jenniferplusplus , to Random stuff
@jenniferplusplus@hachyderm.io avatar

Given how many fedi servers are out there, it's interesting to me that I can only find one written in Ruby. You'd think that a long running successful project would just naturally produce resources that other projects can use. But that doesn't seem to have happened.

hrefna ,
@hrefna@hachyderm.io avatar

@jenniferplusplus This is purely guesswork, but I suspect part of it is that golang is a relatively popular shop language (along with Java, C++, C#, etc)—the languages that people use to Get Things Done™ and you barely have to justify using them—and its structure encourages a kind of modularity that is appealing.

It just fits well as a "good enough" choice that is accessible for a lot of people.

Outside of Ruby, Rails is basically multidimensionally the wrong choice for this kind of project.

hrefna ,
@hrefna@hachyderm.io avatar

@jenniferplusplus I mean to be slightly cynical I don't think rails is the correct choice for basically anything, so I'm not sure my evaluation there is going to be particularly fair :p

More seriously: IMO part of this is Ruby has been in a sort of decline for a long time as one of the major languages. Since before it was chosen for mastodon (c.f. https://www.tiobe.com/tiobe-index/ruby/ and https://octoverse.github.com/2022/top-programming-languages).

There's also been the, uh, trajectory of DHH.

So at a guess: others are just more appealing.

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

Really I'm of the view that a hard fork of mastodon is (probably) not the right solution, but it's likely my view on that is for different reasons than a lot of other people who have talked about it (and also substantially similar in some respects, but with some core differences).

But if you want to do the work and it energizes you then go forth. It just seems to be more hassle than I suspect it will be worth versus just building a ground-up implementation (which is already plenty of work).

polotek , to Random stuff
@polotek@social.polotek.net avatar

“We have a culture of vibrant, open discussion that enables us to create amazing products and turn great ideas into action,” he said in the memo, which the company posted online. “But ultimately we are a workplace and our policies and expectations are clear: this is a business.”
https://www.washingtonpost.com/technology/2024/04/22/google-nimbus-israel-protest-fired-workers/

hrefna ,
@hrefna@hachyderm.io avatar

@sgf

I suspect those stories are relatively common in part because SREs feel a sense of obligation to each other more than anything.

I don't want to let down my team, who is responsible. My team has secondaries and a sister team in EMEA who will be the ones to pay the price. Google won't, my EMEA sister team will. My other coworkers will

Google won't care unless all of us go out together, but that won't break immediately

There are ethical solutions, but they require ground work

@polotek

hrefna ,
@hrefna@hachyderm.io avatar

@sgf

This is one reason I eschew "activism" (even where I acknowledge it has a place) in favor of organizing. Because without the power what happens is everyone gets fried, there's a bad press cycle, and then the world moves on.

The power doesn't have to be on a critical area, but it does need density and to be about something that these companies care about. 100 people across 100 teams is an easier target for them than 30 people on one team with a deadline for a major event.

@polotek

hrefna ,
@hrefna@hachyderm.io avatar

@polotek

No, it is definitely not unique at all.

It's not the caring about the job, I'm trying to make a more nuanced point than that that I think may agree with what you are saying: it's that you care about your coworkers, who are the only ones who are negatively impacted unless you've built density.

Basically I'd argue: it isn't the feeling of responsibility for production, but that you need to do the work to get the entire team on the line together for it to matter to the company.

@sgf

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

"I think it's great that people are realizing if their social media app isn't talking to the fedi they needn't bother making it."

This is a bad take and you should feel bad for making it.

Snark aside: there's nothing magical about the "fediverse" and there's a lot to improve. Some of the ways that the fediverse has traditionally improved involved people going off and doing their own thing and finding ways to make it fit later.

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

If your solution to a problem is "everyone works for free until we obtain the glorious utopian vision" then you don't have a solution, you have a second problem.

devnull , to Random stuff
@devnull@crag.social avatar

So someone deleted their Mastodon account and now my instance is receiving an Undo(Announce) for everything they've ever done.

This ... is ... concerning, considering one should assume a single Delete(Actor) should automatically handle all of the side effects?

I'm not particularly worried about the noise, bandwidth, or cycles, it's mostly just out of principle that I think this is weird.

I think @hrefna tooted about this awhile back too.

hrefna ,
@hrefna@hachyderm.io avatar

@devnull Yep. Under the ActivityPub there's no inferences that deleting the actor deletes things downstream from the actor, despite this being a sensible position.

So you get variable implementations that either:

  1. Assume that deleting the actor is sufficient.
  2. Undo everything, which is also not in accordance with the spec but fits with how AS frames things.
  3. Delete literally everything (technically twice), which fits with how AP is written.
  4. Just not propagate account deletions
devnull , to Random stuff
@devnull@crag.social avatar

When @BeAware asks for help scaling out Mastodon because his SINGLE USER INSTANCE is falling over, and he reveals that he's paying for an 8 vCPU server with 16GB of memory, and all the comments are talking about tweaking postgres.

What the flying fuck.

hrefna ,
@hrefna@hachyderm.io avatar

@devnull There's something of a tendency to try to compensate for slowness in the rest of the stack with the database and it can almost work if you are willing to throw enough money at vertically scaling the database.

Which is a shame, because mastodon then proceeds to NOT DO THE THINGS that would allow the database to scale more cleanly or cheaply.

hrefna ,
@hrefna@hachyderm.io avatar

@devnull I commented awhile back that I can't tell what mastodon is actually targeting, because it doesn't seem to scale up nor down particularly well.

hrefna ,
@hrefna@hachyderm.io avatar

@devnull

Yuuuuup. For the vast majority of PSQL applications it's:

  • Reduce your number of connections, use pooling.
  • Use batching where you can.
  • Reuse results where you can.
  • Be particularly mindful of when you are doing deletes.
  • Make sure you are keeping up on database maintenance and that vacuum isn't too far behind.

For most problems that's it.

@ryansingel @baris

devnull , (edited ) to Random stuff
@devnull@crag.social avatar

Federated NextDoor? @evan

ducks

Could be a neat use case for location based notes outside of the hashtag purview.

hrefna ,
@hrefna@hachyderm.io avatar

@devnull WE WERE ALL HAVING A GOOD DAY JULIAN

hrefna ,
@hrefna@hachyderm.io avatar

@jenniferplusplus

Jokes aside: I think the generalized idea has merit, but I don't think we have the tooling or the capability to get anywhere close to a way to do it ethically at this time, and the problem of intersecting communities is one where IMO the fediverse is interesting in theory but challenging in practice.

In the "running something for my community" sense, however, there's a lot of merit.

@devnull

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

The things I don't like about the discussion on whether this is a state actor behind the backdoor are:

  • It doesn't change the response for pretty much anyone except a narrow group of professionals. Ultimately I don't know that it matters for most of us if this was a state attacker or some kid who wants a way to get op privileges.

  • It distracts from next steps.

  • Would they think that if the actor were named John? Will this increase suspicion of anyone with a "foreign" sounding name?

hrefna OP ,
@hrefna@hachyderm.io avatar

@joeyh Which is absolutely useless analysis from the standpoint of determining how to prevent it in the future, and is also not useful even to speculate about unless you are like quite literally working for a very, very narrow group of security professionals.

It also costs a trivial amount of money to fund this sort of thing. Like you don't need a state actor, a small company could do it trivially.

hrefna OP ,
@hrefna@hachyderm.io avatar

@joeyh k. Your disagreement and $5 will get me a cup of coffee.

Random dude who doesn't seem to understand threat assessment and who works on low-level utilities has opinions that are different from security professionals, security-adjacent professionals, and SREs. Film at 11.

hrefna OP ,
@hrefna@hachyderm.io avatar

@xdydx

k, and your complete inability to understand second person pronouns and use of "assumptions about gender" (which I didn't, thank you very much, unless it's changed very recently) also indicates that you are a person I do not need to be talking to either, regardless of your credentials.

Have a nice life.

@joeyh

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

Tired: "70% of bugs are caused by memory safety issues"

Wired: "70% of bugs are caused by scope creep"

Inspired: "70% of bugs are caused by overworking and underfunding developers."

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

The protocol as it now sits will not keep you "safe" from threads in any meaningful way.

Repeat. After. Me.

The protocol as it now sits will not keep you "safe" from threads in any meaningful way.

I don't mean as in "it will not protect from a malevolent actor" sense. I mean in an ordinary, reasonable behavior sense.

not only does not have the tools for this, it makes assumptions that are fundamentally opposed to the kinds of protections that people seem to be seeking.

1/

hrefna OP ,
@hrefna@hachyderm.io avatar

Domain blocking is not a feature in ActivityPub

Authorized fetch is arguably in violation of the required treatment of public objects

Public objects must be shared without authentication and when it was asked to change this the response from members of the SWICG was, at best, hostile

There's a requirement for dereferencable ids that makes this all fundamentally harder

Forwarding from the inbox removes control

The protocol conflates ids with objects and either can be sent in any case

2/

hrefna OP ,
@hrefna@hachyderm.io avatar

Because blocks aren't shared with anyone in the base design it means no one else can enforce your blocks. Even other well behaved servers that you trust.

I can go on.

Mastodon or Akkoma or whatever may have ameliorated these concerns today, but that's an implementation detail that there's a 99% probability is an undocumented implementation detail.

New implementations aren't beholden to them and may not know about them even if they want to be well behaved.

3/

hrefna OP ,
@hrefna@hachyderm.io avatar

This should (SHOULD) make you rather uncomfortable.

There is no way, in the protocol, and in fact the protocol's platonic ideal, has no requirement that they even behave consistently or inform you of what their behaviors are. If the server you are on were to switch tomorrow to another that doesn't support domain blocking and everyone is using the platonic ideal of the S2S and the C2S protocols you'd have literally no way of knowing it.

Again, not hostile, intrinsically and by design.

4/

hrefna OP ,
@hrefna@hachyderm.io avatar

How do we fix this?

IMO:

We need implementations to define subdialects and for ways to communicate about those subdialects.

We need a real extension mechanism that isn't just "this is how you get something into the AS namespace."

We need to create documented norms about when we do or do not share objects vs. links.

We need to push back on situations where the protocol is unsafe and call out examples.

We need to acknowledge the problems and see where we can find a way forward.

5/

hrefna OP ,
@hrefna@hachyderm.io avatar

One can argue that a protocol should not have to carry some of these aspects, but my point is more subtle than that:

AP gets involved in these areas and expresses opinions about them, making it the domain of AP. To me it is ridiculous to make strong statements about how things must be and then turn around and say "but the consequences of this are not my business."

Even if it isn't within the formal definition of the protocol, we need these things to be attached to how we think of AP.

6/

hrefna OP ,
@hrefna@hachyderm.io avatar

What are some steps you can take?

  • Support projects that think about safety and acknowledge these kinds of problems. Letterbook is an example, but there are others.

  • Pressure implementations to adopt a practice of publishing their subdialects and formalizing them with test suites. Work to improve the general state of conformance testing.

  • Pressure (or join) SWICG to either work to improve this situation or get out of the way of those who do work to improve this situation.

etc.

7/

hrefna OP ,
@hrefna@hachyderm.io avatar

You can also support developers/projects who are doing work to make the situation better in specific areas.

Regardless of the how, we need active energy and committed attention. It's also something that no amount of "fediblocking" or "harassing Eugen/others to block threads on m.s/others" will accomplish or aid.

This is not something that will be repaired by default or that we as members of the fediverse should ignore.

This is something where we need you to take an active role.

8/8

hrefna , to Random stuff
@hrefna@hachyderm.io avatar

If you think that treads is an existential threat to the fediverse because of EEE and yet you think a mass-blocking campaign will do anything to move the needle there, I have some shares of a bridge to sell you.

Lovely bridge, overlooks the ocean, located in Colorado.

This isn't to say that you should federate with threads. I don't care and you may have very good reasons to defederate from them or even encourage a mass-block campaign.

Let's just not kid ourselves that it will slow down EEE.

hrefna OP ,
@hrefna@hachyderm.io avatar

Things I worry about from Meta when it comes to EEE:

  • A branded client
  • A server
  • Privileged access to the w3c
  • Undocumented or weakly documented protocol changes; particularly the kind that is delivered without tests or that change the conformance criteria is subtle ways

Things I bet that Meta does not worry about when it comes to EEE:

  • Being unable to follow members of less than 1/3rd of the smaller servers on the fediverse.
hrefna OP ,
@hrefna@hachyderm.io avatar

"But we need to block them for our own safety."

I'm not talking about that, and you may have the best reasons in the world personally. I'm talking about what happens after the initial breaking in phase. About EEE.

This isn't about whether you block, this is about why people are being told that they need to block or how people analyze meta being here.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • Mordhau
  • WatchParties
  • Rutgers
  • steinbach
  • Lexington
  • cragsand
  • mead
  • RetroGamingNetwork
  • mauerstrassenwetten
  • loren
  • xyz
  • PowerRangers
  • AnarchoCapitalism
  • kamenrider
  • supersentai
  • itdept
  • neondivide
  • space_engine
  • AgeRegression
  • WarhammerFantasy
  • Teensy
  • learnviet
  • bjj
  • khanate
  • electropalaeography
  • MidnightClan
  • jeremy
  • fandic
  • All magazines
    •