Lemmy Server Performance

6 readers

1 users here now

lemmy_server uses the Diesel ORM that automatically generates SQL statements. There are serious performance problems in June and July 2023 preventing Lemmy from scaling. Topics include caching, PostgreSQL extensions for troubleshooting, Client/Server Code/SQL Data/server operator apps/sever operator API (performance and storage monitoring), etc.

founded 1 year ago

MODERATORS

RoundSparrow@lemmy.ml

RoundSparrow@bulletintree.com

Lemmy PERFORMANCE CRISIS: popular instances with many remote instances following big communities could stop federating outbound for Votes on comments/posts - code path identified (lemmy.ml)

submitted 1 year ago* (last edited 1 year ago) by RoundSparrow@lemmy.ml to c/lemmyperformance@lemmy.ml

14 comments fedilink hide all child comments

I spent several hours tracing in production (updating the code a dozen times with extra logging) to identify the actual path the lemmy_server code uses for outbound federation of votes to subscribed servers.

Major popular servers, Beehaw, Leemy.world, Lemmy.ml - have a large number of instance servers subscribing to their communities to get copies of every post/comment. Comment votes/likes are the most common activity, and it is proposed that during the PERFORMANCE CRISIS that outbound vote/like sharing be turned off by these overwhelmed servers.

pull request for draft:

https://github.com/LemmyNet/lemmy/compare/main...RocketDerp:lemmy_comment_votes_nofed1:no_federation_of_votes_outbound0

EDIT: LEMMY_SKIP_FEDERATE_VOTES environment variable

you are viewing a single comment's thread
view the rest of the comments

[–] RoundSparrow@lemmy.ml 0 points 1 year ago (1 children)

I have no idea why the file is named "mod.rs", as normal non-admin non-moderator users seem to go through this code path.

[–] kkard2@lemmy.ml 0 points 1 year ago (1 children)

this is a rust thing: https://stackoverflow.com/questions/26435102/in-rust-what-is-the-purpose-of-a-mod-rs-file

[–] RoundSparrow@lemmy.ml 0 points 1 year ago* (last edited 1 year ago) (1 children)

ok, I figured out how to get Rust to match the enum, is there a way to do this with match instead of if statements?

+    if let AnnouncableActivities::UndoVote(_) = activity {
+      warn!("zebratrace310 SKIP UndoVote");
+    } else if let AnnouncableActivities::Vote(_) = activity {
+      warn!("zebratrace310A SKIP Vote");
+    } else {
+      warn!("zebratrace311 send");
+      AnnounceActivity::send(activity.clone().try_into()?, community, context).await?;
+    };

Code seems to work great, blocks UndoVote/Vote but does the send on comment reply.

[–] kkard2@lemmy.ml 1 points 1 year ago (1 children)

more "correct" way would be this:

match activity {
    AnnouncableActivities::UndoVote(_) => warn!("zebratrace310 SKIP UndoVote"),
    AnnouncableActivities::Vote(_) => warn!("zebratrace310A SKIP Vote"),
    _ => {
        warn!("zebratrace311 send");
        AnnounceActivity::send(activity.clone().try_into()?, community, context).await?;
    },
}

here it is in the rust book: https://doc.rust-lang.org/stable/book/ch06-02-match.html

[–] RoundSparrow@lemmy.ml 1 points 1 year ago

ool. I found the syntax for multiple hits, so I was looking for:

match activity {
    AnnouncableActivities::UndoVote(_)  |
    AnnouncableActivities::Vote(_) => {
        warn!("zebratrace310 SKIP federating Vote/UndoVote");
    },
    _ => {
        warn!("zebratrace311 send");
        AnnounceActivity::send(activity.clone().try_into()?, community, context).await?;
    },
}

Thank you.