Technology

Insider from OpenAI PR department speaks out!

How did he calculate the 70% chance?

Maybe they asked ChatGPT?

This is a horoscope trick. They can always say AI destroyed humanity.

Trump won in 2016 and there was Cambridge Analytica doing data analysis: AI technology destroyed humanity!

Israel used AI-guided missiles to attack Gaza: AI destroyed humanity!

Whatever. You can point at whatever catastrophe and there is always AI behind because already in 2014 AI is a basic technology used everywhere.

The person who predicted 70% chance of AI doom is Daniel Kokotajlo, who quit OpenAI because of it not taking this seriously enough. The quote you have there is a statement by OpenAI, not by Kokotajlo, this is all explicit in the article. The idea that this guy is motivated by trying to do marketing for OpenAI is just wrong, the article links to some of his extensive commentary where he is advocating for more government oversight specifically of OpenAI and other big companies instead of the favorable regulations that company is pushing for. The idea that his belief in existential risk is disingenuous also doesn't make sense, it's clear that he and other people concerned about this take it very seriously.

It isn’t AI itself, it’s AI as a vector for corporate recklessness.

This. 1000% this. Many of Issac Asimov novels warned about this sort of thing too; as did any number of novels inspired by Asimov.

It's not that we didn't provide the AI with rules. It's not that the AI isn't trying not to harm people. It's that humans, being the clever little things we are, are far more adept at deceiving and tricking AI into saying things and using that to justify actions to gain benefit.

...Understandably this is how that is being done. By selling AI that isn't as intelligent as it is being trumpeted as. As long as these corporate shysters can organize a team to crap out a "Minimally Viable Product" they're hailed as miracle workers and get paid fucking millions.

Ideally all of this should violate the many, many laws of many, many civilized nations...but they've done some black magic with that too; by attacking and weakening laws and institutions that can hold them liable for this and even completely ripping out or neutering laws that could cause them to be held accountable by misusing their influence.

I don't think your assumption holds. Corporations are not, as a rule, incompetent - in fact, they tend to be really competent at squeezing profit out of anything. They are misaligned, which is much more dangerous.

I think the more likely scenario is also more grim:

AI actually does continue to advance and gets better and better displacing more and more jobs. It doesn't happen instantly so barely anything gets done. Some half-assed regulations are attempted but predictably end up either not doing anything, postponing the inevitable by a small amount of time, or causing more damage than doing nothing would. Corporations grow in power, build their own autonomous armies, and exert pressure on governments to leave them unregulated. Eventually all resources are managed by and for few rich assholes, while the rest of the world tries to survive without angering them.
If we're unlucky, some of those corporations end up being managed by a maximizer AGI with no human supervision and then the Earth pretty much becomes an abstract game with a scoreboard, where money (or whatever is the equivalent) is the score.

Limitations of human body act as an important balancing factor in keeping democracies from collapsing. No human can rule a nation alone - they need armies and workers. Intellectual work is especially important (unless you have some other source of income to outsource it), but it requires good living conditions to develop and sustain. Once intellectual work is automated, infrastructure like schools, roads, hospitals, housing cease to be important for the rulers - they can give those to the army as a reward and make the rest of the population do manual work. Then if manual work and policing through force become automated, there is no need even for those slivers of decency.
Once a single human can rule a nation, there is enough rich psychopaths for one of them to attempt it.

There are also other AI-related pitfalls that humanity may fall into in the meantime - automated terrorism (e.g. swarms of autonomous small drones with explosive charges using face recognition to target entire ideologies by tracking social media), misaligned AGI going rogue (e.g. the famous paperclip maximizer, although probably not exactly this scenario), collapse of the internet due to propaganda bots using next-gen generative AI... I'm sure there's more.

Yes. We need human responsibility for everything what AI does. It's not the technology that harms but human beings and those who profit from it.

Yes, it's very concerning and frustrating that more people don't understand the risks posed by AI. It's not about AI becoming sentient and destroying humanity, it's about humanity using AI to destroy itself. I think this fundamental misunderstanding of the problem is the reason why you get so many of these dismissive "AI is just techbro hype" comments. So many people are genuinely clueless about a) how manipulative this technology already is and b) the rate at which it is advancing.

Calling LLMs, "AI" is one of the most genius marketing moves I have ever seen. It's also the reason for the problems you mention.

I am guessing that a lot of people are just thinking, "Well AI is just not that smart... yet! It will learn more and get smarter and then, ah ha! Skynet!" It is a fundamental misunderstanding of what LLMs are doing. It may be a partial emulation of intelligence. Like humans, it uses its prior memory and experiences (data) to guess what an answer to a new question would look like. But unlike human intelligence, it doesn't have any idea what it is saying, actually means.

@millie @floofloof this is so well articulated I can't stand it. I want to have it printed out and hand it to anyone who asks me anything about AI. Thank you for this!

And these current LLMs aren’t just gonna find sentience for themselves. Sure they’ll pass a Turing test but they aren’t alive lol

Habsburg-AI? Do you have an idea on how much you made me laugh in real life with this expression??? It's just... perfect! Model degeneration is a lot like what happened with the Habsburg family's genetic pool.

When it comes to hallucinations in general, I got another analogy: someone trying to use a screwdriver with nails, failing, and calling it a hallucination. In other words I don't think that the models are misbehaving, they're simply behaving as expected, and that any "improvement" in this regard is basically a band-aid being added to humans to a procedure that doesn't yield a lot of useful outputs to begin with.

And that reinforces the point from your last paragraph - those people genuinely believe that, if you feed enough data into a L"L"M, it'll "magically" become smart. It won't, just like 70kg of bees won't "magically" think as well as a human being would. The underlying process is "dumb".

Energy restrictions actually could be pretty easily worked around using analog converting methods. Otherwise I agree completely though, and what's the point of using energy on useless tools. There's so many great things that AI is and can be used for, but of course like anything exploitable whatever is "for the people" is some amalgamation of extracting our dollars.

The funny part to me is that like mentioned "beautiful" AI cabins that are clearly fake -- there's this weird dichotomy of people just not caring/too ignorant to notice the poor details, but at the same time so many generative AI tools are specifically being used to remove imperfection during the editing process. And that in itself is something that's too bad, I'm definitely guilty of aiming for "the perfect composition" but sometimes nature and timing forces your hand which makes the piece ephemeral in a unique way. Shadows are going to exist, background subjects are going to exist.

The current state of marketed AI is selling the promise of perfection, something that's been getting sold for years already. Just now it's far easier to pump out scam material with these tools, something that gets easier with each advancement in these sorts of technologies, and now with more environmental harm than just a victim of a predator.

It really sucks being an optimist sometimes.

It could be only hype. But I don't entirely agree. Personally, I believe we are only a few years away from AGI. Will it come from OpenAI and LLMs? Maybe, but it will likely come from something completely different. Like it or not, we are within spitting distance of a true Artificial Intelligence, and it will shake the foundations of the world.

Your scenario 1 is the actual danger. It's not that AI will outsmart us and kill us. It's that AI will trick us into trusting them with more responsibility than the AI can responsibly handle, to disastrous results.

It could be small scale, low stakes stuff, like an AI designing a menu that humans blindly cook. Or it could be higher stakes stuff that actually does things like affect election results, crashes financial markets, causes a military to target the wrong house, etc. The danger has always been that humans will act on the information provided by a malfunctioning AI, not that AI and technology will be a closed loop with no humans involved.

I think much of it comes from "futurologists" spending too much time smelling each others' farts. These AI guys think so very much of themselves.

The only danger to humans is humans.

I’m sorry, but this is a really dumb take that borders on climate change denial logic. A sufficiently large comet is an existential threat to humanity. You seem to have this optimistic view that humanity is invincible against any threat but itself, and I do not think that belief is justified.

People are right to be very skeptical about OpenAI and “techbros.” But I fear this skepticism has turned into outright denial of the genuine risks posed by AGI.

I find myself exhausted by this binary partitioning of discourse surrounding AI. Apparently you have to either be a cult member who worships the coming god of the singularity, or think that AI is either impossible or incapable of posing a serious threat.

But if AI learns from us...

I think you're right on the money when it comes to the real dangers, especially your first bullet point. I don't necessarily agree with your napkin maths. If the virtual neurons are used in a more efficient way, that could make up for a lot versus human neuron count.

The human brain has an estimated 100 trillion neuronal connections - so probably a good order of magnitude estimation for the parameter count of an AGI model.

Yeah, but a lot of those do things unrelated to higher reasoning. A small monkey is smarter than a moose, despite the moose obviously having way more synapses.

I don't think you can rely on this kind of argument so heavily. A brain isn't a muscle.

Yeah I'll admit I was freaked out at the beginning. So I learned about models, used them, and got familiar with them. Now I'm less freaked out and more "oh my god so many people are going to get scammed/tricked".

Go on Facebook and you'll see it's a good 50-70% AI garbage now. My favorite are "log cabin" and kitchen posts that are just images of them with blanket titles like "wish I lived here" with THOUSANDS of comments of people saying "YES" or "it's so beautiful". Of course it is it has no supports! The cabinets are held up by nothing! There are 9 kinds of lanterns and most are floating. Jesus people are not ready for it.

The guy complaining left the company:

Fed up, Kokotajlo quit the firm in April, telling his team in an email that he had "lost confidence that OpenAI will behave responsibly" as it continues trying to build near-human-level AI.

I don't think that he stands to benefit.

He also didn't say that OpenAI was on the brink of having something like this either.

Like, I don't think all the fighting at OpenAI and people being ejected and such is all a massive choreographed performance. I think that there have been people who really strongly disagree with each other.

I absolutely think that AGI has the potential to post existential risks to humanity. I just don't think that OpenAI is anywhere near building anything capable of that. But if you're trying to build towards such a thing, the risks are something that I think a lot of people would keep in mind.

I think that human level AI is very much technically possible. We can do it ourselves, and we have hardware with superior storage and compute capacity. The problem we haven't solved is the software side. And I can very easily believe that we may get there not all that far in the future. Years or decades, not centuries down the road.

When I think of AI ruining humanity, this is how I picture it