Completely off topic. You've linked to another post. I follow it, and end up on a different server, where I don't have an account. I wonder if there is a possible solution.
this post was submitted on 22 Jun 2023
4 points (100.0% liked)
Lemmy Server Performance
6 readers
1 users here now
Lemmy Server Performance
lemmy_server uses the Diesel ORM that automatically generates SQL statements. There are serious performance problems in June and July 2023 preventing Lemmy from scaling. Topics include caching, PostgreSQL extensions for troubleshooting, Client/Server Code/SQL Data/server operator apps/sever operator API (performance and storage monitoring), etc.
founded 1 year ago
MODERATORS
There is an open GitHub issue on that topic, and /c/lemmyfederaiton or /c/lemmycode community for discussion.
For anyone else looking, here is the issue: https://github.com/LemmyNet/lemmy/issues/3259
Thanks, I didn't have the link handy.
Batching the inserts up only kicks the can down the road a few weeks. We need a 500x improvement in insertion time.
The proposal has been raised (by me) to move all federation out of lemmy_server into a different service and have a queue in there. I think that opens up to people working and updating the code better. The email systems I have worked with that have a database storage backend have used their own MTA service, not run in the main app's core. I also think Reddit does data acceptance before it gets to PostgreSQL too - as I've seen comments get backed up when one of their servers or services goes offline.
Or, build some sort of in memory buffer that bulks the actions, and commits to the DB in batches over time/size a-la AWS Kinesis Firehose.
Every action gets recorded in memory, in their own separate buffers (I.E. one for post, one for comment, one for like/dislike, etc.) and is supplemented to the results on each request. When the buffer reaches X MB in size, it is then batch insert/update’ed into the database as to reduce the overhead?
Well, I'm pulling out "like/dislike" (votes), because I consider it less of a priority. The actual comments and postings are taking over 1.0 second to INSERT, but that's the bulk of the site's purpose - to share actual content. If the likes lag by 15 seconds, is that such a big deal?
The way I'd imagine this working is the data is supplemented (augmented) into the query results.
Basically, there'd be a new cache / injection layer that sits between the application and the database. Instead of application directly working with the existing ORM to work with the DB (I'm not actually sure how Rust does this, so I'm just speaking in broader terms), the application would work against this layer that creates the buffer, and interface with the ORM. Then, on write actions, it fills up the buffer until buffer fills up or some time has past before bulk performing the write action; on read actions, it interfaces with ORM, and then weave the buffered data into the response back to the application.
Thus, from the user's perspective, nothing should be changed, and they wouldn't know any wiser.
view more: next ›