hey folks, here's another meta-post. this one isn't specifically in response to the massive surge of users, but the surge is fortuitously timed because i've been intending to give a good idea of what our financial stability is like. as a reminder, we're 100%-user funded. everything you donate to us specifically goes to the website, or any outside labor we pay to do something for us.
thanks to your generous support, we're pretty confident we have passed our current break-even point for this month, at which we wouldn't eventually need to pay out of our own pockets to keep the site running. that point in our estimation is about $26 a month or $312 a year. (please ignore OC's estimated yearly budget--we don't determine it lol)
our expenses are currently:
- $18/mo toward our host, Digital Ocean. (yesterday we upgraded from DO's $12 tier to its $18 tier to mitigate traffic issues and lag, and it's really worked out!)
- $2/mo for weekly backups
- $4/mo for daily snapshots of the website, which would allow us to restore the website in between the weekly backups if need be.
for a total of at least $26/mo in expenses. this may vary from month to month though, so we're baking in a bit of uncertainty with our estimation.
we currently have, for the month of June:
- $70/mo in recurring donations (at least for June)
- $200 this month in one-time donations
for a total of $270 this month. our total balance now stands at $331.31.
that balance means we now have about a year months of reserves currently, if we received no other donations and have no unexpected expenses.[^1] the recurring donations put us well into the green at this point.
this is good! everything past our break-even point each month is, to be clear, money we can save and put toward scaling up our infrastructure. there is no downside to donating after we've already met our "goal" of basic financial stability. doing so will have pretty straightforward practical implications for you: fewer 500s, 503s, better image support (this takes a lot of space!), and the website generally being run on more than potato hardware.[^2] if you'd like to do so in light of this information, our OpenCollective page is this post's link. thanks folks!
[^1]: we will have at least one upcoming expense but its size is TBD, and so is how we'll pay for it [^2]: especially during times like now, where we've likely been getting thousands or tens of thousands of hits an hour
Absolutely, and at a minimum, I'd suggest implementing the 3-2-1 backup strategy, widely regarded as the "gold standard" for maintaining data integrity. This involves keeping a total of three backups: two stored locally, but on different devices, and one off-site backup (ideally kept in a secure location such as a bank safe deposit box, with hard drives rotated every two weeks).
That's definitely where I went wrong. The sad part is I did make an effort to backup the mc server weekly. I had a cron job that would bring the minecraft server down, export the SQL db's for plugins, and copy the world / other config files into a tar.gz file and copy it to a hard drive raid also running on the server. I knew having raid wasn't a backup in itself, and also running it on the same server wasn't doing me any favors. The problem was the archive files were incomplete, but I didn't realize that until the SSD died and I went to restore the archive file. Still not really sure why they weren't complete archive files. I still have them, just can't open them. It's weird though, when I first set it up they totally worked.
Indeed, one challenging aspect of data backup is the necessity to confirm that the backup is restorable. This adds another layer of complexity to system administration. However, with the aid of a well-crafted script to handle backup verification, reliability of backup restoration can be ensured, potentially through a method such as hash comparison.
Backups can sometimes give you a deceptive feeling of safety. If you neglect regular verification, you might find that your backup fails just when you need it most for a restoration.
Hashing! Why didn't I think of that when I created the original backup script haha. Low key might go write a new backup script and incorporate that. I'm running another private mc server for some friends, this time in the cloud cause I didn't want to deal with another crash. I would like to bring it back to running in my garage sometime.
In your situation, I'd recommend creating a script that performs the following steps sequentially:
Regarding Borg backup, while it offers robust verification and security mechanisms, I've personally found its performance lacking, though your experience might differ. I would recommend thoroughly reviewing its documentation and conducting some trial runs prior to integrating it into a production environment.
Ahh okay. I do enjoy writing programs myself where I can so I'll probably still do it myself, but good to have options. Since it's just the MC server I'm truly worried about, I don't care too much about the performance within reason. I like the idea of a text file with corresponding hashes. Keep it simple! Or as my old shop teacher used to say, KISS, keep it simple, stupid.
I really recommend you to use Borg Backup instead. It has incremental encryption which is really useful, and generally does its job well.
For what it's worth, I've had issues with data corruption restoring with Borg. Defeating the purpose as a backup. I've used Restic happily and restores very well.
This underscores why I often advocate for simplicity. It's preferable to use trusted tools, like LUKS for filesystem-level encryption and straightforward solutions like the Ext4 filesystem. This aligns with the mentality of "If it's not broken, don't fix it", which is particularly relevant in a backup context. The fewer modifications you make to the backup software, the greater your confidence that it won't let you down in a critical moment.
Fwiw, I've used restic as well. But there was a discussion a while back about very large backups corrupting. I don't know what the right answer is for large backups where duplication is prohibitively costly, but for personal use I do a restic/borg type chunking repo plus a vanilla rsync to a copy on write filesystem.
I'll check it out, thanks!