this post was submitted on 30 Dec 2023
41 points (100.0% liked)

Lemmy

496 readers
2 users here now

Everything about Lemmy; bugs, gripes, praises, and advocacy.

For discussion about the lemmy.ml instance, go to !meta@lemmy.ml.

founded 4 years ago
MODERATORS
nutomic@lemmy.ml
41
Work-around for the federation bug found in 0.19.0 and 0.19.1 (sub.wetshaving.social)
submitted 10 months ago* (last edited 10 months ago) by walden@sub.wetshaving.social to c/lemmy@lemmy.ml
7 comments fedilink hide all child comments
 

I've found the following work-around works pretty well. If you host an instance that's currently on 0.19.0 or 0.19.1, consider implementing this.

There are two bugs that this helps with:

Work-around:
Create cronjobs that restart the Lemmy container every 6 hours (but not at midnight). The following example is used for a Debian system running Lemmy in Docker.

Type crontab -e into the terminal Add something like the following:

~~0 1 * * * docker container restart lemmy-lemmy-1
0 7 * * * docker container restart lemmy-lemmy-1
0 13 * * * docker container restart lemmy-lemmy-1
0 19 * * * docker container restart lemmy-lemmy-1~~

3 1-23/6 * * * docker container restart lemmy-postgres-1 && sleep 60 && docker container restart lemmy-lemmy-1

By restarting the container every 6 hours, outbound federation continues to work. There may still be some delays, but everything gets cleared up regularly.

By telling it what time to restart (0100, 0700, 1300, and 1900 as opposed to "every 6 hours"), it avoids restarting at midnight. This avoids the second bug.

My instance has been doing this for enough days where I'm confident that it's working. You can check your federation status here. Note that it's normal for there to be 0 up-to-date instances and a lot of lagging instances. As long as they sometimes turn "up to date", then everything is getting caught up.

top 7 comments
sorted by: hot top controversial new old
[–] RedKrieg@lemmy.redkrieg.com 16 points 10 months ago* (last edited 10 months ago) (1 children)

I like to do things just off the top of the hour, since top of the hour is when many maintenance crons run. If you're running a modern cron daemon, you can rewrite that as:

3 1-23/6 * * * docker container restart lemmy-lemmy-1

https://crontab.guru/#3_1/6___*

[–] walden@sub.wetshaving.social 6 points 10 months ago* (last edited 10 months ago)

I figured someone more familiar with crontab would chime in. Thanks for the recommendation!

Edit: I've updated my cronjob to https://crontab.guru/#7_1-23/6___* and I'll keep an eye out to make sure it works on Debian 12.

[–] BlueEther@no.lastname.nz 4 points 10 months ago

The only thing that doesn't seem to federate on my 0.19.1 instance is new subscriptions, but in saying that it looks like the BE restarted an hour ago

[–] SeeJayEmm@lemmy.procrastinati.org 2 points 10 months ago (1 children)

I didn't know how this tool is supposed to work but it always fails for me but my instance appears to be working ok.

https://phiresky.github.io/lemmy-federation-state

[–] walden@sub.wetshaving.social 2 points 10 months ago

Interesting. Yeah I tried too and it doesn't work for lemmy.procrastinati.org

[–] bjoern_tantau@swg-empire.de 1 points 10 months ago* (last edited 10 months ago) (1 children)

Funnily enough, since I've started to regularly check if my comments made it out I haven't had any problems. Schrödinger's bug.

Edit: Then again, after checking with this post, although my comment made it out just fine, some comments haven't arrived on my instance. I'll try to restart.

[–] walden@sub.wetshaving.social 2 points 10 months ago

Comments from other instances should arrive to yours no problem, as the bug only seems to affect outgoing stuff. If the other instance is running 0.19.0+, then of course those comments might be stuck.