this post was submitted on 11 May 2024
269 points (100.0% liked)

Technology

1083 readers
42 users here now

This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.


Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.


Rules:

1: All Lemmy rules apply

2: Do not post low effort posts

3: NEVER post naziped*gore stuff

4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.

5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)

6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist

7: crypto related posts, unless essential, are disallowed

founded 5 years ago
MODERATORS
 

A week of downtime and all the servers were recovered only because the customer had a proper disaster recovery protocol and held backups somewhere else, otherwise Google deleted the backups too

Google cloud ceo says "it won't happen anymore", it's insane that there's the possibility of "instant delete everything"

top 48 comments
sorted by: hot top controversial new old
[–] harry315@feddit.de 129 points 6 months ago (3 children)

Remember people: The cloud is just someone else's computer.

[–] dan1101@lemm.ee 23 points 6 months ago (1 children)

Yeah there's that, and the fact that you have no control over how much the bill will be each renewal period. Those two things kept me off the cloud for anything important.

[–] cmnybo@discuss.tchncs.de 4 points 6 months ago (1 children)

Most cloud providers have a way to set limits. Make sure you learn how to set appropriate limits to avoid unexpected bills.

[–] IronKrill@lemmy.ca 4 points 6 months ago (1 children)

The limits don't matter if the provider raises their price next month.

[–] Natanael@slrpnk.net 3 points 6 months ago

And some functions don't support hard limits, you'd have to set up a script monitoring load and literally take down your service if you get near the max

https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1

[–] imnotfromkaliningrad@lemmy.ml 11 points 6 months ago (1 children)

thats why i am trying to explain to my family since forever. their answer always amounts to something like "it would be illegal for them to look at my data!" like those companies would care. .

[–] umbrella@lemmy.ml 3 points 6 months ago

in many cases "looking at my data!" is in their TOS

[–] delirious_owl@discuss.online 9 points 6 months ago

Unless its a self-hosted cloud. Then its your own computers

[–] MossyFeathers@pawb.social 96 points 6 months ago* (last edited 6 months ago) (5 children)

They said the outage was caused by a misconfiguration that resulted in UniSuper’s cloud account being deleted, something that had never happened to Google Cloud before.

Bullshit. I've heard of people having their Google accounts randomly banned or even deleted before. Remember when the Terraria devs cancelled the Stadia port of Terraria because Google randomly banned their account and then took weeks to acknowledge it? The only reason why Google responded so quickly to this is because the super fund manages over $100b and could sue the absolute fuck out of Google.

[–] Pechente@feddit.de 30 points 6 months ago (1 children)

This happened to me years ago. Suddenly got a random community guidelines violation on YouTube for a 3 second VFX shot that was not pornographic or violent and that I owned all the rights to. After that my whole Google account was locked down. I never found out what triggered this response and I could never resolve the issue with them since I only ever got automated responses. Fuck Google.

load more comments (1 replies)
[–] umbrella@lemmy.ml 2 points 6 months ago* (last edited 6 months ago)

one of my accounts was locked for no reason once. i apparently did well to not trust important data to them anymore.

load more comments (3 replies)
[–] GolfNovemberUniform@lemmy.ml 33 points 6 months ago (3 children)

Tbh I do not understand why would a company keep their data on a service like Google Cloud

[–] Chozo@fedia.io 22 points 6 months ago (4 children)

Money. It's a lot cheaper to let somebody else maintain your systems than to pay somebody to create and maintain your own, directly.

[–] Tryptaminev@lemm.ee 10 points 6 months ago (1 children)

If you are a small company then yes. But i would argue that for larger companies this doesn't hold true. If you have 200 employees you'll need an IT department either way. You need IT expertise either way. So having some people who know how to plan, implement and maintain physical hardware makes sense too.

There is a breaking point between economics of scale and the added efforts to coordinate between your company and the service provider plus paying that service providers overhead and profits.

[–] matti@sopuli.xyz 4 points 6 months ago (1 children)

If coordinating with service providers is hard for a firm, I would argue the cost effective answer isn't "let's do all this in house". Many big finance firms fall in this trap of thinking it's cheaper to build v buy, and that's how you get everyone building their own worse versions of everything. Whether your firm is good at the markets or kitchens or travel bookings, thinking you can efficiently in-source tech is a huge fallacy.

[–] Tryptaminev@lemm.ee 3 points 6 months ago

it is not about it being hard. It simply creates effort to coordinate. And this effort needs to be considered. If you do things externally that means there is two PMs to pay, you need QMs on both sides, you need two legal/contract teams, you need to pay someone in procurement and someone in sales...

I agree with you that doing software inhouse when there is good options on the market is usually not a good idea. But for infrastructure i don't see there to be as much of an efficiency loss. Especially as you very much need experts on how to set things up in a cloud environment and you better look carefully at how many resources you need to not overpay huge amounts.

[–] GolfNovemberUniform@lemmy.ml 3 points 6 months ago* (last edited 6 months ago) (1 children)

No I meant that Google Cloud is very invasive. Why not to use a more ethical provider?

[–] allywilson@lemmy.ml 6 points 6 months ago (1 children)

Why do you think it's invasive? How do you quantify which providers are less invasive?

[–] GolfNovemberUniform@lemmy.ml 7 points 6 months ago (2 children)

Google is one of the most privacy invasive companies in the world. And judging by encryption standards, terms of service and privacy policies

[–] settoloki@lemmy.one 9 points 6 months ago (1 children)

Are you sure you've not just read bad stuff without verification on the internet and feel the need to chime in on something you don't fully understand?

[–] GolfNovemberUniform@lemmy.ml 4 points 6 months ago* (last edited 6 months ago) (1 children)

Yes. I read Google's policies many times.

[–] settoloki@lemmy.one 10 points 6 months ago (1 children)

Me too as a programmer that uses Google cloud to store government information. Which bit of the policy says they are going to access your data, shouldn't take you long to link it to me if you read them as much as you say. Unless what you're actually doing is spreading misinformation and bullshit.

[–] ReversalHatchery 4 points 6 months ago (3 children)

I'm not the one who you were responding to, but considering google's history, I don't believe anything they claim, because they have lied so many times in the past, and because every "privacy guarantee" they provide is practically unprovable. It's nothing more than wishful thinking to think that google does nothing with government data stored with them, with google classroom data of millions of children, and others. They have shown that they can't be trusted.

[–] settoloki@lemmy.one 7 points 6 months ago (1 children)

If they lied about this and are accessing very confidential information I think my company would sue the giblets off Google.

You need to remember we are talking about Google Cloud, the enterprise services they offer and not Gmail and search engines.

[–] ReversalHatchery 1 points 6 months ago (1 children)

I only have one question: how will your company find out?

[–] settoloki@lemmy.one 1 points 6 months ago (1 children)

Same way companies know they've been hacked. I'm making the assumption you're non technical, given the question. But there are many ways such as access logs, server monitoring etc

[–] ReversalHatchery 1 points 6 months ago (1 children)

But there are many ways such as access logs, server monitoring etc

Which are all in the control of the company running the servers. If we trust the company, we can trust them giving honest information on these, but if we don't trust the company.. they could just redact logs or even straight out fake them

[–] settoloki@lemmy.one 1 points 6 months ago

I think you live in a fantasy world fella. Also server monitoring isn't done by Google, it's don't by another 3rd party company.

[–] pupbiru@aussie.zone 5 points 6 months ago* (last edited 6 months ago)

b2b and audited security standards are a whole different thing - you deal with finance and health you’ve gotta prove to a 3rd party over and over that you have controls and technology in place to make sure you aren’t lying

this isn’t consumer BS

[–] GolfNovemberUniform@lemmy.ml 4 points 6 months ago

This. Even if by some miracle Google isn't accessing everything on corporate cloud, it is an evil company and the policy can change. It's a very untrustworthy and unreliable base for a business. And I'm not even talking about the fact that businesses that pay for the cloud are financially supporting Google

[–] pupbiru@aussie.zone 3 points 6 months ago

and you know the security standards that are achievable on google cloud entirely negate your point right? their cloud offering is a totally different beast

[–] PowerCrazy@lemmy.ml 3 points 6 months ago

It's absolutely not. If you are at any kind of scale whatsoever, your yearly spend will be a minimum of 2x at a cloud provider rather then creating and operating the same system locally including all the employees, contracts, etc.

load more comments (1 replies)
[–] Kit@lemmy.blahaj.zone 5 points 6 months ago

G Suite is a legitimate option for small-medium businesses. It's seen as the cheaper, simpler option versus Azure. I usually recommend it for nonprofits as they have a decent free option for 501c3 orgs.

[–] KarnaSubarna@lemmy.ml 3 points 6 months ago (1 children)

Money and Time – It's rather easier/cheaper for Organizations nowadays to outsource a part of infra to Cloud service providers.

[–] GolfNovemberUniform@lemmy.ml 2 points 6 months ago

I meant Google Cloud, not cloud outsourcing itself

[–] heluecht@pirati.ca 23 points 6 months ago (1 children)

@Moonrise2473 Regardless of one thinks about "cloud" solutions, this is a good example, why you always should have an offsite backup.

[–] Hirom 5 points 6 months ago (2 children)

They had backups at multiple locations, and lost data at multiple (Google Cloud) locations because of the account deletion.

They restored from backups stored at another provider. It may have been more devastating if they relied exclusively on google for backups. So having an "offsite backup" isn't enough in some cases, that offsite location need to be at a different provider.

[–] heluecht@pirati.ca 6 points 6 months ago (1 children)

@Hirom With "offsite" I mean either a different cloud provider or own hardware (if you hold your regular data at some cloud provider, like in this case).

[–] Hirom 1 points 6 months ago

That would indeed be a good backup strategy, but better be specific. "Offsite" may be interpreted in different ways.

[–] Tangentism@lemmy.ml 2 points 6 months ago (1 children)

It may have been more devastating if they relied exclusively on google for backups.

Which is why having any data, despite the number of backups, on a cloud provider shouldn't be seen as off-site.

Only when it is truly outside their ecosphere and cannot be touched by them should it be viewed as such.

If that company didn't have such resilience built into their backup plan, they would be toast with a derisory amount of compensation from Google.

[–] Hirom 2 points 6 months ago (1 children)

Having a backup at a cloud provider is fine, as long as there is at least one other backup that isn't with this provider.

Cloud provider seems to do a good job protecting against hardware failure, but can do poorly with arbitrary account bans, and sometimes have mishaps due to configuration problems.

Whereas a DIY backup solution is often more subject to hardware problems (disk failure, fire, flooding, theft, ...), but there's no risk of account problem.

A mix is fine to protect against different kind of issues.

[–] Tangentism@lemmy.ml 2 points 6 months ago* (last edited 6 months ago) (1 children)

as long as there is at least one other backup that isn't with this provider.

Which is exactly what I was saying.

Any services used with a cloud provider should be treated as 1 entity, no matter how many geo-locations they claim your data is backed up to because they are a single point from which all those can be deleted.

When I was last involved in a companies backups, we had a fire safe in the basement, we had an off-site location with another fire safe & third copies would go off to another company that provided a backup storage solution so for all backups to be deleted, someone had to go right out of their way to do so. Not just a simple deletion of our account & all backups are wiped.

That company had the foresight to do something similar & it's saved them. [edited - was on the tube when I wrote this and didnt see the autocorrect had put 'comment', not 'company']

[–] Hirom 2 points 6 months ago (1 children)

Okay, I misinterpreted your comment.

[–] Tangentism@lemmy.ml 1 points 6 months ago

No, it's all good. We're on the same page about disaster recovery!

[–] Simon@lemmy.dbzer0.com 10 points 6 months ago

Just an FYI in case you don't follow Cloud news but Google has deleted customers accounts on multiple occasions and has been for literal years. This time they just did it to someone large enough to make the news. I work in SRE and no longer recommend GCP to anyone.

[–] autotldr@lemmings.world 1 points 6 months ago

This is the best summary I could come up with:


More than half a million UniSuper fund members went a week with no access to their superannuation accounts after a “one-of-a-kind” Google Cloud “misconfiguration” led to the financial services provider’s private cloud account being deleted, Google and UniSuper have revealed.

Services began being restored for UniSuper customers on Thursday, more than a week after the system went offline.

Investment account balances would reflect last week’s figures and UniSuper said those would be updated as quickly as possible.

In an extraordinary joint statement from Chun and the global CEO for Google Cloud, Thomas Kurian, the pair apologised to members for the outage, and said it had been “extremely frustrating and disappointing”.

“These backups have minimised data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration,” the pair said.

“Restoring UniSuper’s Private Cloud instance has called for an incredible amount of focus, effort, and partnership between our teams to enable an extensive recovery of all the core systems.


The original article contains 412 words, the summary contains 162 words. Saved 61%. I'm a bot and I'm open source!