Between 19:45 UTC and 19:50 UTC, there was a mistake in how information was stored temporarily (cached) on Beehaw. This mistake could have allowed some people to see and use other people's accounts without permission.

If you were using the website during that time, please check that your account settings and email address are still correct. Also, make sure that any posts or actions you made during that time are still connected to your account.

It's important to note that we don't have any proof that this error was actually used by anyone to do anything bad during the short time it happened.

top 16 comments

sorted by: hot top controversial new old

[–] remington 16 points 2 years ago (1 children)

I can verify that @Penguincoder@beehaw.org is one of our volunteer sysadmins and this incident was witnessed by me.

[–] Helix 8 points 2 years ago (1 children)

same, same :p

[–] remington 9 points 2 years ago

I can, also, verify that @Helix@beehaw.org is another volunteer sysadmin here at Beehaw.

[–] argv_minus_one 15 points 2 years ago

That's one of the two hard problems of programming: cache invalidation, naming things, and off-by-one errors.

[–] fishy_2_0 8 points 2 years ago

thanks for alerting everyone to this aswell as reacring quickly

[–] Cougar 6 points 2 years ago (1 children)

Do we know what could have caused this cache error?

[–] Penguincoder 10 points 2 years ago (1 children)

Yes; configuration settings for the web server involving improving performance. Those settings have been changed back to the previous, non-issue ones. So this should not occur at this time, or again.

[–] Cougar 10 points 2 years ago (1 children)

Is this a mistake that's easy to do for an inexperienced instance admin or just a consequence of too much fiddling and shouldn't be an issue for other instances?

[–] Penguincoder 14 points 2 years ago (1 children)

Was a result of too much fiddling. Attempting to gain even better performance from a bottleneck issue due to recent user influx. It was not an error in the Lemmy instance or Lemmy-UI but rather the web server front-end misconfiguration.

[–] karce 6 points 2 years ago* (last edited 2 years ago) (2 children)

Thanks for working through these issues and improving performance of the website! Very appreciated. I've been tempted recently to create my own Lemmy instance, was this a problem with an nginx configuration option? How much does Beehaw deviate from a standard Lemmy deployment?

Feel free to answer vaguely if you don't feel comfortable with giving away the details : )

[–] Helix@feddit.de 6 points 2 years ago (2 children)

was this a problem with an nginx configuration option?

Basically, this was proxy_cache_key being configured incorrectly. If you don't use the proxy_cache you should be fine.

The only thing we changed from the norm is ulimits and some nginx settings. If we figure out what works well, we'll probably create a post about how to host lemmy. If you stick to the defaults, you'll be mostly fine if your instance isn't as big as Beehaw's.

[–] karce 3 points 2 years ago

Awesome! Thanks for the response. I'd love a post sometime on hosting Lemmy. I'd find it very interesting and useful!

[–] nutomic@lemmy.ml 2 points 2 years ago (1 children)

Im also considering to setup nginx caching for lemmy.ml. Did you find a configuration which works?

[–] Helix@feddit.de 4 points 2 years ago (1 children)

Not yet. Session tracking in Lemmy is pretty hard to proxy, I'll have to dive into the code to figure out why.

[–] nutomic@lemmy.ml 3 points 2 years ago

Have a look at this: https://github.com/LemmyNet/lemmy-ansible/pull/75

Sending proper cache-control headers from Lemmy will require some big code changes though.

[–] Penguincoder 3 points 2 years ago

Yes.

Deviation is pretty minimal, related to configuration and customization.