hey folks, here's another meta-post. this one isn't specifically in response to the massive surge of users, but the surge is fortuitously timed because i've been intending to give a good idea of what our financial stability is like. as a reminder, we're 100%-user funded. everything you donate to us specifically goes to the website, or any outside labor we pay to do something for us.

thanks to your generous support, we're pretty confident we have passed our current break-even point for this month, at which we wouldn't eventually need to pay out of our own pockets to keep the site running. that point in our estimation is about $26 a month or $312 a year. (please ignore OC's estimated yearly budget--we don't determine it lol)

our expenses are currently:

$18/mo toward our host, Digital Ocean. (yesterday we upgraded from DO's $12 tier to its $18 tier to mitigate traffic issues and lag, and it's really worked out!)
$2/mo for weekly backups
$4/mo for daily snapshots of the website, which would allow us to restore the website in between the weekly backups if need be.

for a total of at least $26/mo in expenses. this may vary from month to month though, so we're baking in a bit of uncertainty with our estimation.

we currently have, for the month of June:

$70/mo in recurring donations (at least for June)
$200 this month in one-time donations

for a total of $270 this month. our total balance now stands at $331.31.

that balance means we now have about a year months of reserves currently, if we received no other donations and have no unexpected expenses.[^1] the recurring donations put us well into the green at this point.

this is good! everything past our break-even point each month is, to be clear, money we can save and put toward scaling up our infrastructure. there is no downside to donating after we've already met our "goal" of basic financial stability. doing so will have pretty straightforward practical implications for you: fewer 500s, 503s, better image support (this takes a lot of space!), and the website generally being run on more than potato hardware.[^2] if you'd like to do so in light of this information, our OpenCollective page is this post's link. thanks folks!

[^1]: we will have at least one upcoming expense but its size is TBD, and so is how we'll pay for it [^2]: especially during times like now, where we've likely been getting thousands or tens of thousands of hits an hour

all 50 comments

sorted by: hot top controversial new old

[–] alyaza 45 points 2 years ago* (last edited 2 years ago) (3 children)

minor PSA: we splurged a bit thanks to your donations and got slightly better hardware. you've probably noticed a more fluid website that's erroring way less accordingly. enjoy! i'll update this post's numbers tomorrow to give you a reflection of the new break-even. thanks for using the site folks

[–] Mersampa 11 points 2 years ago (1 children)

It will be interesting to see how many users stick around. It's busy right now but in a month you may be able to backtrack to the previous hardware.

[–] alyaza 22 points 2 years ago (1 children)

well, in a month we might be dealing with a second wave of users (and i think we won't be short on people who will stick around), but yeah point taken lol. it does help we're also not using a complex set-up, so "scaling up" isn't a big commitment right now

[–] Mersampa 14 points 2 years ago (1 children)

I think it's normal after a big surge to see participation drop off. If you can hit the critical mass to keep content flowing, that would be awesome.

And at least in the short term, it's appreciated that the site is now running much smoother! That's also important for retention of users :)

[–] alyaza 12 points 2 years ago (1 children)

oh yeah no we'll definitely see some (probably a lot of) dropoff, but if even 20% of this wave sticks around and starts contributing at all that will add several times over what our previous userbase was. we were pretty close to a point as a community where there'd always be activity of some kind, and this event has pushed us well over that threshold

[–] Mersampa 6 points 2 years ago

Haha yes, it's like a big foot reaching out to give reddit a kick in the pants: https://the-federation.info/node/details/25274

[–] uthredii 6 points 1 year ago (1 children)

I would like to know to what extent you are able to scale up the hardware. Is there a theoretical limit in the number of users that can handled?

[–] remington 9 points 1 year ago

There is room to scale the VPS immensely. However, we made a small bump to the next tier yesterday and things are working very well now.

Is there a theoretical limit in the number of users that can handled?

Maybe...I don't know.

load more comments (1 replies)

[–] GuyDudeman 26 points 1 year ago

At the risk of sounding like Roy from Ted Lasso.... This is FUCKING REFRESHING. This is how it should be! Thank you so much for the transparency, and for all the work you're doing to implement this instance. I'll definitely be sticking around, and hopefully contributing whatever I can. This is amazing. Please keep it up!

[–] Crotaro 25 points 1 year ago (1 children)

I discovered this whole thing just a few days ago (of course from that reddit post) and I love how this has such strong early-internet vibes! I mean where anymore do you ever read the owner of a site casually go "Yeah, since we are a couple hundred dollars in surplus, I was able to splurge on new hardware so that you'll see loading errors a little less often because the server is no longer powered just by my child being strapped into an oversized hamster wheel"?

[–] SubArcticTundra@lemmy.ml 7 points 1 year ago (1 children)

Do you think you could link me to that Reddit post? I'm out of the loop

[–] SaintPaul 9 points 1 year ago (2 children)

Here it is

https://reddit.com/r/apolloapp/comments/13ws4w3/had_a_call_with_reddit_to_discuss_pricing_bad/

load more comments (2 replies)

[–] arcticpiecitylights 24 points 1 year ago

I'm new to BeeHaw but its really encouraging to see such transparency and, frankly, competent admin communication. Thanks for sharing this!

[–] axonPotential 17 points 2 years ago

Massively appreciate this level of transparency on the instance's finances. To me, that's a promising sign

[–] TheTrueLinuxDev 15 points 2 years ago (7 children)

Scaling up can become quickly cost-prohibitive with large-scale servers. I've noted that the most affordable option with Digital Ocean, at $12/month, offers only a basic droplet with 1 vCPU, 2GB of RAM, and 50GB of SSD storage. When you consider a higher-end configuration with 16GB of RAM, 8 vCPUs, and 320GB of SSD storage at $96/month, it may not seem economical, especially as storage and backup needs increase with server scaling.

As an alternative approach to minimize costs while scaling, consider purchasing used servers from platforms like eBay and setting up a small-scale hosting operation in your garage. While this route does introduce overheads like business internet services and electricity costs, along with regular maintenance such as HDD replacements, it could be more cost-effective in the long run.

For instance, you could acquire a server on eBay for about $300, offering 20 CPU cores, 64GB of RAM, and 8TB of SAS HDD storage. Comparatively, a similar setup on Digital Ocean would cost around $544/month or $6528/year, making the used server a strong competitor against cloud services.

Just some food for thought if you're contemplating scaling in the future.

[–] nutomic@lemmy.ml 12 points 2 years ago

Digitalocean is pretty expensive. For example on hetzner.com you can get 8 vCPU for 15 euros per month. OVH should be similar. So thats really affordable if you have hundreds or thousands of users and a couple of donors.

[–] Gaywallet 10 points 2 years ago (1 children)

While I'd be more than happy to purchase and use actual server space for doing this kind of thing, I'm not nearly tech saavy enough to actually run that. I would also worry about bandwidth considerations and other issues. Perhaps there are people willing to contribute to the cause to find affordable ways to run a website like this, but being accessible by multiple people to do things like power cycle the server, not being concerned about our personal IPs being attacked, and having access to support I think is worth the extra cost.

There is a point at which something akin to this needs to be done for financial reasons, but I think we're pretty far from that being a reality and while it relies on slightly more donations from our lovely userbase, even at $96/mo that's not that much money to collect when we have the thousands of users that would require that kind of hardware.

[–] TheTrueLinuxDev 7 points 2 years ago* (last edited 2 years ago) (2 children)

I appreciate your thoughtful response to my comment. There are indeed several strategies that can be employed to decrease bandwidth and storage costs. Leveraging a Content Delivery Network, such as the free service provided by Cloudflare, can help mitigate these costs by caching your webpages and images. As for the cost of internet service, it greatly varies based on your location. If you're located closer to the internet backbone, the likelihood of finding a more reasonably priced business internet plan increases.

While it may seem premature at this stage, I firmly believe in the success of this website, even in the face of numerous failures in this space.

Edited to add -

You've correctly highlighted the potential threat of attacks on your server. Cloudflare is known for its prowess in mitigating substantial distributed denial-of-service (DDoS) attacks and could be an excellent security asset in this context.

Regarding the value of professional support, I acknowledge that the cost can often be justified. My suggestions are merely alternatives, providing you with additional options should you require them.

[–] Gaywallet 6 points 2 years ago (2 children)

Very much appreciate the knowledge and responses! We're definitely in need of systems administrators who have time and energy to expend in terms of helping us reach something sustainable both financially and technically speaking.

[–] Penguincoder 8 points 2 years ago (1 children)

Echoing thetruelinuxdev, I'd be willing to help however I can. I also have a personal server just sitting mostly idle I could try to use for this stuff, but that of course depends on you all as the admins of the instance trusting some random on the internet vs a VPS provider like DO. I do have Linux Sysadmin and other technology experience. Giving money is easier but might not be as impactful as helping with infrastructure, to me.

[–] Gaywallet 7 points 2 years ago (2 children)

Thank you! We will definitely take any and all suggestions and offers into consideration 🥰

[–] Penguincoder 7 points 2 years ago* (last edited 2 years ago)

Just PM for resume 🤣 but seriously, while I know I can't donate as much time as some of you already do, for what I can do, I'd rather support a community effort like this with my labor rather than that other-site.

[–] spaghetti_carbanana 5 points 2 years ago

I’ll throw my hat in here too, I have capacity on my in-home infrastructure and will also be colocating a couple servers in a proper datacentre over the coming months. I back up everything to disks, minimum 2 week on-disk retention, then duplicate it onto tapes (don’t laugh - they’re rated for 30 years cold storage). I keep yearly backups infinitely, monthlys for 1-2 years, weeklys for 6 months iirc. Based in Australia if that helps. 10+ years experience as a sysadmin/cyber sec professional. Feel free to DM if I can help the cause as I’m happy to donate capacity

[–] TheTrueLinuxDev 5 points 2 years ago

While I may not be able to provide financial assistance, I'm more than willing to lend my expertise in system administration and development, should you find it beneficial.

[–] Parsnip8904 5 points 2 years ago (1 children)

Using cloudflare as proxy would essentially mean letting them MITM all the traffic though right? All things considered they're one of the trustworthy companies but is there some other alternative that you can basically self host?

[–] TheTrueLinuxDev 5 points 1 year ago (3 children)

Cloudflare can't be accurately labeled as a Man-in-the-Middle (MitM), given its integral role in the service stack. The same logic would falsely accuse platforms like Linode, AWS, and Azure of the same. Moreover, self-hosting is entirely feasible. The main challenge arises from Internet Service Providers, which often restrict upload speeds unjustifiably. I highlight this to explain why it typically becomes more economical to locate closer to the internet backbone, where the cost and the plan tend to be more reasonable.

load more comments (3 replies)

[–] CobolSailor 6 points 2 years ago (1 children)

I did that in my garage a few years back to tinker around with linux servers. Just don't make the same mistake I did and buy the cheapest SSD on eBay. My friend group still hasn't forgiven me for letting our minecraft server get corrupted lol.

[–] TheTrueLinuxDev 7 points 2 years ago (1 children)

Absolutely, and at a minimum, I'd suggest implementing the 3-2-1 backup strategy, widely regarded as the "gold standard" for maintaining data integrity. This involves keeping a total of three backups: two stored locally, but on different devices, and one off-site backup (ideally kept in a secure location such as a bank safe deposit box, with hard drives rotated every two weeks).

[–] CobolSailor 5 points 2 years ago (1 children)

That's definitely where I went wrong. The sad part is I did make an effort to backup the mc server weekly. I had a cron job that would bring the minecraft server down, export the SQL db's for plugins, and copy the world / other config files into a tar.gz file and copy it to a hard drive raid also running on the server. I knew having raid wasn't a backup in itself, and also running it on the same server wasn't doing me any favors. The problem was the archive files were incomplete, but I didn't realize that until the SSD died and I went to restore the archive file. Still not really sure why they weren't complete archive files. I still have them, just can't open them. It's weird though, when I first set it up they totally worked.

[–] TheTrueLinuxDev 7 points 2 years ago (1 children)

Indeed, one challenging aspect of data backup is the necessity to confirm that the backup is restorable. This adds another layer of complexity to system administration. However, with the aid of a well-crafted script to handle backup verification, reliability of backup restoration can be ensured, potentially through a method such as hash comparison.

Backups can sometimes give you a deceptive feeling of safety. If you neglect regular verification, you might find that your backup fails just when you need it most for a restoration.

[–] CobolSailor 5 points 2 years ago (6 children)

Hashing! Why didn't I think of that when I created the original backup script haha. Low key might go write a new backup script and incorporate that. I'm running another private mc server for some friends, this time in the cloud cause I didn't want to deal with another crash. I would like to bring it back to running in my garage sometime.

[–] TheTrueLinuxDev 5 points 2 years ago (1 children)

In your situation, I'd recommend creating a script that performs the following steps sequentially:

Accumulate the hashes (ideally SHA-256 or SHA-512) of all files you plan to backup.
Create a text file that lists the file paths and their corresponding hashes.
Initiate the backup process.
Upon completion of the backup, restore the data to an auxiliary drive.
Compare the hashes from the text file to those of the restored data.
Alert you about whether the process was successful or not.

Regarding Borg backup, while it offers robust verification and security mechanisms, I've personally found its performance lacking, though your experience might differ. I would recommend thoroughly reviewing its documentation and conducting some trial runs prior to integrating it into a production environment.

load more comments (1 replies)

load more comments (5 replies)

load more comments (4 replies)

[–] eggsnham07 15 points 2 years ago

Thanks for the quick approval!

Really hope this becomes the new reddit!

[–] thefingolfin 14 points 2 years ago

Started a small monthly contribution, hopefully will help scale to what will likely be a lot of increased usage

[–] OofShoot 13 points 2 years ago

It's going to be entertaining seeing the new numbers after you update this post.

[–] jherazob 11 points 1 year ago

That's... surprisingly low! Nice! Now i find myself wondering about the financials of something like the biggest Mastodon instances...

[–] RomanceDailies 10 points 2 years ago

I’m now contributing a little bit. :)

[–] dax 9 points 2 years ago* (last edited 2 years ago)

super happy I set up a recurring donation last week - I had a feeling you guys were going to get hammered at some point!

I'm coming in at about 78$ a month for mydroplet; a s-4vcpu-8gb and a singular 250gb volume, including taxes. I still think paying someone to host it remotely is a better deal - but I also live on an island and we lose power multiple times a year and I don't want my inability to power things impact my users.

[–] karce 8 points 1 year ago (2 children)

Is it possible to donate Cryptocurrency? I'd love to give XMR (Monero) so I don't have to reveal any details about myself, etc.

[–] alyaza 6 points 1 year ago* (last edited 1 year ago) (3 children)

Is it possible to donate Cryptocurrency?

at least currently we're tethered to OpenCollective (and not really in a position to extend ourselves across even more stuff to maintain) so if they don't accept it then no, that's just a limitation we'll have to live with. that said, i believe OpenCollective allows for anonymous donation--we have many Guest contributions--so that may be suitable for your purposes if you can do crypto to fiat conversion

load more comments (3 replies)

load more comments (1 replies)

[–] CobolSailor 7 points 2 years ago (3 children)

That's great to hear! I'm still trying to wrap my head around the federation concept itself so if you don't mind, I have a question for you. In the event the site did go under (which it seems like is not the case, yay!) what happens to our account? Will we have to create a new account on another instance? If so, will we be able to use the same username?

[–] Mersampa 10 points 2 years ago* (last edited 2 years ago) (7 children)

I like the email analogy.

When you want to use email, you sign up for an account. It doesn't matter if you go to gmail or hotmail or something else, you can still communicate with anyone else that has email.

Now if your username for email is fred@hotmail.com and then that server went down, you would no longer have access to your account, people couldn't contact you. If you then decided to set up a gmail account, you would only be able to get the account fred@gmail.com if no one else had taken it. The name "fred" is not reserved for you across all email providers.

Federated platforms like Lemmy (which Beehaw is part of) work the same way. Your account is not @CobolSailor but @CobolSailor@beehaw.org. If Beehaw went under and no longer existed, you can go and sign up on another server, e.g. lemmy.ml. But you could only use @CobolSailor@lemmy.ml if it was not taken by someone else, it's not reserved for you across all servers - and can't be, because there is no central server keeping track.

Does that explain it?

[–] CobolSailor 8 points 2 years ago

The email analogy just made the whole federation click in my brain. Thank you for the well explained response!

[–] ScaryGary@mastodon.social 6 points 2 years ago

@Mersampa @CobolSailor
As someone who has like 1 hours worth of experience here, that cleared a lot up for me about how this works actually. Cheers!

load more comments (5 replies)

load more comments (2 replies)