this post was submitted on 14 Jun 2023
193 points (100.0% liked)

Blahaj Lemmy Meta

86 readers
4 users here now

Blåhaj Lemmy is a Lemmy instance attached to blahaj.zone. This is a group for questions or discussions relevant to either instance.

founded 2 years ago
MODERATORS
 

Apologies to all for today's performance issues and downtime.

Logging for our lemmy instance filled our main partition and brought everything to a halt. Our lemmy instance shares hardware with blahaj.zone, and the significant growth in lemmy over the last 10 days has caused the logs to grow in size exponentially! As a result of the logs filling up the main partition, both lemmy.blahaj.zone and blahaj.zone itself went down.

We've freed up some space, upgraded the hardware and spun everything up again, and it looks everything is back to normal.

Apologies for the inconvenience, and welcome back to everyone!

top 35 comments
sorted by: hot top controversial new old
[–] spaduf@lemmy.blahaj.zone 25 points 1 year ago (1 children)

Do we currently have any sort of donation page?

[–] ada@lemmy.blahaj.zone 32 points 1 year ago (6 children)

Not yet. We're talking about if/how we want to do it though

[–] vinceman@lemmy.blahaj.zone 17 points 1 year ago

Please let us know. Have 0 issue kicking ina a few bucks.

[–] Wereduck@lemmy.blahaj.zone 11 points 1 year ago

I'm 100% down to kick in. Open Collective or Liberapay might be a way to avoid Patreon's cut. But I'd definitely donate to a Patreon too!

[–] EnderWi99in@kbin.social 8 points 1 year ago

Put something up and I'll contribute! Could do a Patreon. Would love any partnership opportunities with some of the other API's that were left in the dust too like Apollo and RIF.

[–] emi@lemmy.blahaj.zone 4 points 1 year ago (1 children)

I would suggest https://opencollective.com/ as it seems to be a popular choice for similar communities when it comes to funding operations.

[–] CoachDom@lemmy.blahaj.zone 2 points 1 year ago

Or liberapay

[–] superb@lemmy.blahaj.zone 3 points 1 year ago

I would be happy to donate!

[–] Cevilia@lemmy.blahaj.zone 1 points 1 year ago

Please do. I'd gladly sign up for it.

Gimme that Blahaj Gold! (/joke - don't wanna make this a them-and-us thing)

[–] colorfulmoth@lemmy.blahaj.zone 23 points 1 year ago (1 children)

Nice to hear that everything is ok. Wouldn't it be a good idea to shit out the logs to another form of long term storage? That could help during the influx of Reddit refugees.

[–] ada@lemmy.blahaj.zone 54 points 1 year ago (1 children)

Yeah absolutely. The main issue here is that this lemmy instance was spun up as a side project for me and until 10 days ago, I was basically the only user. We never planned for a reddit migration and 1400 user signups in a single week, so things are not currently configured optimally for two active instances.

We're aware of it though, and will get everything humming along smoothly, once we've had a chance to catch up :)

[–] EnderWi99in@kbin.social 13 points 1 year ago

Thanks for letting us all come hang out with you!

[–] yaycupcake@lemmy.blahaj.zone 16 points 1 year ago

Thanks so much for your efforts and transparency!

[–] jherazob 13 points 1 year ago (1 children)

Servers going down because logs fill a disk is one common Sysadmin experience, achievement unlocked! 😄

Thanks for doing the work 👍

[–] ada@lemmy.blahaj.zone 8 points 1 year ago (1 children)

I mean, if we expected lemmy to grow in to an instance 3 times the size of blahaj.zone, we'd have done things differently from the start, but suffice to say, we didn't expect Blahaj Lemmy to be anything but a bit of a sandbox for me to play with, so we made some initial choices that didn't scale well with sudden unexpected growth :)

[–] jherazob 2 points 1 year ago

And that too is an achievement unlocked! 😄

Been there, done that, got some premature white hairs :P

[–] Nolando@lemmy.blahaj.zone 12 points 1 year ago* (last edited 1 year ago) (1 children)

Good to be back, and glad to hear it was only a slow hug-of-death. Any consideration in upgrading the server version to 17.4?

Edit: are there any alternative channels where server status is posted?

[–] ada@lemmy.blahaj.zone 30 points 1 year ago (2 children)

Any consideration in upgrading the server version to 17.4?

Yep. That's on the radar. We're trying to get a kbin instance up and running too, and we'll most likely upgrade lemmy once that has been done.

are there any alternative channels where server status is posted?

Not yet. A week ago, it was basically just me on this instance. Now it's me and my 1400 closest friends :)

[–] vinceman@lemmy.blahaj.zone 9 points 1 year ago

Thanks Ada. Lol

[–] DpwnShift@kbin.social 7 points 1 year ago

Hey, it's me, your best (new) friend!

[–] AlexTheTurtle@lemmy.blahaj.zone 11 points 1 year ago (1 children)

I was touching grass today so i didn't even notice. Keep up the good work!

[–] ada@lemmy.blahaj.zone 17 points 1 year ago (1 children)

I don't understand. What's grass? :P

[–] AlexTheTurtle@lemmy.blahaj.zone 17 points 1 year ago (1 children)

Green stuff in dirt that brainwashes you into thinking happy thoughts.

[–] ada@lemmy.blahaj.zone 17 points 1 year ago (2 children)

i HavE to say, it reaLly works 100 Percent. This was NOT sponsored by the grass lobby.

[–] jdp23@lemmy.blahaj.zone 4 points 1 year ago

Yeah, seem unlikely.

[–] Cassa@lemmy.blahaj.zone 9 points 1 year ago

As a reddit refugee I'm incredibly thankful for this amazing place

Thank you Ada ❤️❤️❤️ This place is awesome!

[–] binaryphile@kbin.social 7 points 1 year ago

Don't apologize. Instead, thank people for their patience.

Thank you so much 🙏

[–] zbyte64@lemmy.blahaj.zone 2 points 1 year ago

Sounds like success to me.

[–] CrimsonOnoscopy 2 points 1 year ago (1 children)

Consider installing a lot rotator maybe.

[–] ada@lemmy.blahaj.zone 14 points 1 year ago (1 children)

The real issue is simply that this instance was set up more as experiment initially, with basically 1 user, and suddenly it's got 1500 users. The growth caught us by surprise, and it was never set up with this growth rate in mind. Got it under control now, and we will be keeping an eye on future growth

[–] CrimsonOnoscopy 4 points 1 year ago

Cool; Good luck!

[–] Dfoxe@lemmy.blahaj.zone 2 points 1 year ago (1 children)

Do we have a status page that can be viewed in case of an outage? Either way, appreciate all the hard work you're doing! As someone in the tech sector, I know that's a huge surprise right xD, I understand the difficulties of what you're doing. I'm just glad to have such a great place to call home. If you do set up any kind of donation I know many of us are going to be willing to share to keep such a great and safe place running.

[–] ada@lemmy.blahaj.zone 9 points 1 year ago

Alas, there is no status page. The consequences of a community that grew almost overnight :)

We will probably get a donations page up soon, but we want to make sure we do that correctly if we're going to do it.

load more comments
view more: next ›