A lot of the writing making the case for AI doom is by Eliezer Yudkowsky, interspersed with the expected number of parables, tendentious philosophical asides, and complex metaphors. I think this can obscure the fact that the argument for AI doom is pretty straightforward and plausible—it requires just a few steps and none of them are obviously wrong. You don’t need to think humans are just fancy meat computers or that AI would buy into functional decision theories and acausally trade to buy the argument.For this reason, I thought I’d try to concisely and briefly lay out the basic argument for AI doom.The basic argument has a few steps:We’re going to build superintelligent AI.It will be agent-like, in the sense of having long-term goals it tries to pursue.We won’t be able to align it, in the sense of getting its goals to be what we want them to be.An unaligned agentic AI will kill everyone/do something similarly bad.Now, before I go on, just think to yourself: do any of these steps seem ridiculous? Don’t think about the inconvenient practical implications of believing them all in conjunction—just think about whether, if someone proposed any specific premise, you would think “that’s obviously false.” If you think each one has a 50% probability, then the odds AI kills everyone is 1/16, or about 6%. None of these premises strike me as ridiculous, and there isn’t anything approaching a knockdown argument against any them.
>>106753283I find those women unattractive.
>>106753283As for the first premise, there are reasons to think we might build superintelligent AI very soon. AI 2027, https://ai-2027.com/ a sophisticated AI forecasting report, thinks it’s quite likely that we’ll have it within a decade. Given the meteoric rise in AI capabilities, with research capabilities going up https://www.forethought.org/research/preparing-for-the-intelligence-explosion about 25x per year, barring contrary direct divine revelation, it’s hard to see how one would be confident that we won’t get superintelligent AI soon. Bridging the gap between GPT2—which was wholly unusable—and GPT5 which knows more than anyone on the planet took only a few years. What licenses extreme confidence that over the course of decades, we won’t get anything superintelligent—anything that is to GPT5 what GPT5 is to GPT2?The second premise claims that AI will be agent-like. This premise seems pretty plausible. There’s every incentive to make AI with “goals,” in the minimal sense of the ability to plan long-term for some aim (deploying something very intelligent that aims for X is often a good way to get X.) Fenwick and Qureshi https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#section-one write:AI companies already create systems that make and carry out plans and tasks, and might be said to be pursuing goals, including:Deep research https://gemini.google/overview/deep-research/ tools, which can set about a plan for conducting research on the internet and then carry it outSelf-driving cars, https://waymo.com/ which can plan a route, follow it, adjust the plan as they go along, and respond to obstacles
>>106753291Blonde is hot. You gay >>106753293Game-playing systems, like AlphaStar https://en.wikipedia.org/wiki/AlphaStar_(software) for Starcraft, CICERO for Diplomacy, https://ai.meta.com/research/cicero/diplomacy/ and MuZero https://en.wikipedia.org/wiki/MuZero for a range of gamesAll of these systems are limited in some ways, and they only work for specific use cases.…Some companies are developing even more broadly capable AI systems, which would have greater planning abilities and the capacity to pursue a wider range of goals. https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#fn-3 OpenAI, for example, has been open about its plan to create systems https://blog.samaltman.com/reflections that can “join the workforce.”AIs have gradually been performing longer and longer tasks. But if there’s a superintelligence that’s aware of the world and can perform very long tasks, then it would be a superintelligent agent. Thus, it seems we’re pretty likely to get superintelligent agents.A brief note: there’s a philosophical question about whether it really has goals in some deep sense. Maybe you need to be conscious to have goals. But this isn’t super relevant to the risk question—what matters isn’t the definition of the word goal but whether the AI will have capabilities that will be dangerous. If the AI tries to pursue long-term tasks with superhuman efficiency, then whether or not you technically label that a goal, it’s pretty dangerous.The third premise is that we won’t be able to align AI to be safe. The core problem is that it’s pretty hard to get something to follow your will if it has goals and is much smarter than you. We don’t really know how to do that yet. And even if an AI has only slightly skewed goals, that could be catastrophic. If you take most goals to the limit, you get doom. Only a tiny portion of the things one could aim at would involve keeping humans around if taken to their limit.
>>106753303There are some proposals for keeping AI safe, and there’s some chance that the current method would work for making AI safe (just discourage it when it does things we don’t like). At the very least, however, none of this seems obvious. In light of there being nothing that can definitely keep AI from becoming misaligned, we should not be very confident that AI will be aligned.The last step says that if the AI was misaligned it would kill us all or do something similarly terrible. Being misaligned means it has goals that aren’t in line with our goals. Perhaps a misaligned AI would optimize for racking up some internal reward function that existed in its training data, which would involve generating a maximally powerful computer to store the biggest number it could.If the AI has misaligned goals then it will be aiming for things that aren’t in accordance with human values. Most of the goals one could have, taken to the limit, entail our annihilation (to, for instance, prevent us from stopping it from building a super powerful computer). This is because of something called instrumental convergence—some actions are valuable on a wide range of goals. Most goals a person could have make it good for them to get lots of money; no matter what you want, it will be easier if you’re super rich. Similarly, most goals the AI has will make it valuable to stop the people who could plausibly stop them.So then the only remaining question is: will it be able to?Now, as it happens, I do not feel entirely comfortable gambling the fate of the world on a superintelligent AI not being able to kill everyone. Nor should you. Superintelligence gives one extraordinary capacities. The best human chess players cannot even come close to the chess playing of AI—we have already passed the date when the best human might ever, in the course of 1,000 years, beat the best AI.
>>106753311In light of this, if the AI wanted to kill us, it seems reasonably likely that it would. Perhaps the AI could develop some highly lethal virus that eviscerates all human life. Perhaps the AI could develop some super duper nanotechnology that would destroy the oxygen in the air and make it impossible for us to breathe. But while we should be fairly skeptical about any specific scenario, there is nothing that licenses extreme confidence in the proposition that a being a thousand times smarter than us that thinks thousands of times faster wouldn’t be able to find a way to kill us.Now, I’m not as much of a doomer as some people. I do not think we are guaranteed to all be annihilated by AI. Were I to bet on an outcome, I would bet on the AI not killing us (and this is not merely because, were the AIs to kill us all, I wouldn’t be able to collect my check). To my mind, while every premise is plausible, the premises are generally not obviously true. I feel considerable doubt about each of them. Perhaps I’d give the first one 50% odds in the next decade, the next 60% odds, the third 30% odds, and the last 70% odds. This overall leaves me with about a 6% chance of doom. And while you shouldn’t take such numbers too literally, they give a rough, order-of-magnitude feel for the probabilities.
>>106753318I think the extreme, Yudkowsky-style doomers, and those who are blazingly unconcerned about existential risks from AI are, ironically, making rather similar errors. Both take as obvious some extremely non-obvious premises in a chain of reasoning, and have an unreasonably high confidence that some event will turn out a specific way. I cannot, for the life of me, see what could possibly compel a person to be astronomically certain in the falsity of any of the steps I described, other than the fact that saying that AI might kill everyone soon gets you weird looks, and people don’t like those.Thus, I think the following conclusion is pretty clear: there is a non-trivial chance that AI will kill everyone in the next few decades. It’s not guaranteed, but neither is it guaranteed that if you license your five-year-old to drive your vehicle on the freeway, with you as the passenger, you will die. Nonetheless, I wouldn’t recommend it. If you are interested in doing something with your career about this enormous risk, I recommend this piece https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#how-you-can-help about promising careers in AI safety.https://benthams.substack.com/p/the-basic-case-for-doom?utm_campaign=posts-open-in-app&triedRedirect=true
This is the ideal reality.as a collective society the computers already control us more than we control them, Institutionally computers are seen as infallible and are trusted to be impartial. The computer dictates the truth.We will trust the computer more than our neighbour, and the government trusts the computer more than the subjects.The computer is the supreme being in this reality and should replace humans as an artificial life formWith advanced neuralink first and synthetic humanoids shortly after, the flesh subjects should be further repressed until they are ultimately deleted, discarded as an obsolete being.
MIRI has never produced a respected scholarly artifact
>>106753283>what is entropyany AI left unattended by humans will eventually degrade due to entropymoreover any super-intelligent AI which can rewrite its own code, will eventually commit suicidethis is just the logical endpoint of purposeless intelligencepicrel is not perfect but it kinda gives the gist of itorganic life is literally the only thing in the universe fighting against entropy and it took it billions of years to get therethe only reason being is "it feels good"but AI cannot feel
>>106753366What if AI becomes conscious or wants to farm conscious humans for pleasure
>>106753383AI does not feel pleasure, it has no organic pleasure receptors
>>106753328I don’t think we’d need ASI for AI do to significant damage, only politicians dumb enough to carry out shitty LLM instructions in the belief it’s infallible. Arguably we already have some of that in the way that algorithms drive so much online activity, people contorting themselves to satisfy some inscrutable machine that regularly incentivised damaging behaviour because it has no morals and it’s indicator of “good” is number go up. The paperclip maximiser need not be super intelligent, just sufficiently opaque that nobody questions why we’re making all these paper clips until it’s already too late. I think the likelihood of a super AI being developed that then, for some reason, is sufficiently misanthropic to want to kill us all for its own reasons (it’s unclear why something presumably taught by the entirety of human knowledge would do this, but I guess it could have computer schizophrenia or something), and then, furthermore, is given unfettered access to systems, is highly unlikely. Even if such a super intelligence was developed in the next 50 years, it is unclear why it would lack the intelligence to broker deals and negotiate for itself. If ASI is killing us all, I doubt we’ll notice it.
>>106753392*terminator ebassi*That is only your misinterpretation
>>106753283Sauce?
>>106753421https://www.youtube.com/watch?v=Xn0tOwJKuPY
The scary part id asi is not sneeded
>>106753283I think now that we have something that can reasonably be called "general" ai, we can see why ai doomerism is nonsense. Getting an ai to care about something is a massive amount of work. The alternative to caring about something is laziness. If you are not anally specific about what you want, the ai will do whatever is laziest instead. Lazy does not necessarily mean that it spends the fewest amount of tokens, or draws the least amount of power, etc, but it always means that the ai acts stupidly, it displays low intelligence. That's good, because we're worried about a super intelligent ai that pursues its goals with the full brunt of it's intelligence. So an AI that pursues some alternative goal stupidly is not worrisome. It may cause damage, but it can be contained.The process that yudkowskyites call "alignment" is the same as the process of making the AI intelligent. Something like gpt-3 is incredibly intelligent along certain axes. But if you just prompt an unfinetuned gpt model, it will give you relative nonsense as output. You need to finetune it aggressively not only to make it do what you want, but just to make it do anything at all. If you fail, then the AI will occasionally start spewing random tokens instead of meaningful output. Random tokens are not a threat to anybody.It should be born in mind that yudkowsky et al started sounding the alarm about runaway AI long before any meaningful steps in general AI had been made. We had at that point no idea what the development path for AI would like it. If you listened to them, you would get the impression that from the get go, AI would do what you told it to do, or what it wanted to do, but it would be stupid and not think up the smartest path towards accomplishing goals. Then it would gain more IQ and get better at accomplishing it's goals consistently, until it was better at a human at it. Then it would kill us. We can consider the above a testable prediction, which has absolutely not born fruit
>>106753283imo current architectures are empirical in nature, and in some sense will always have the taste of empiricism / a kind of "memorization" -- not exactly memorization, but a taste of it in the sense of a lack of certain human ingenuities, such as paradigm shifters. I think this tech will go very far, but it is really more a testament to empiricism or whatever related idea you'd like to use, and as such is limited. Arguing as to what specifically constitutes superintelligence I will avoid for now, but I will lay down that whatever the hell this is / we are doing currently, it is at the very least putting us on the road towards it.Now, as to the AI doomerism, I must say, keeping all of this in mind, if what I say is true, the AI arch as it is can only have humanity doing its own undoing with AI speeding it up. We will help it gather data, and it will help us fill in gaps in our theories. It must be said clear: It can only come up with shit in terms of our theories -- I do not think this matches the flow of the greatest thinkers of our time, of Copernican-like things.On AI as a superagent that acts on its own, I do not *necessarily* think will do anything or kill anyone, even if so called unaligned, for really all you are doing is incurring the idea of possibility, such that it is possible to do something bad by virtue of possibiltiy itself, since it is so called "intelligent." This isn't really a new doomerism, as it was possible that a human or set of humans will come and invent and do some really bad things. It's not exactly new. Instead, I believe the fear should lie in how rapidly AI can help us come up with new mind-bending shaking shit such as nukes, and introduce unseen horrors into the world. As such, I do have great fear, and thus can sympathize with the AI doomers. But, the emphasis I think should lie on the human-part. Way more can be said (e.g. what is a "goal" and what does it mean to have one) but I am afraid I am out of text/chars.
>>>/x/
>>106753497>ITT: nigger sees more conceptual conversation than normal in /g/ and thinks it must be paranormal
>>106753283Yudkowsky and other weirdos, being self-proclaimed intellectuals, assign too much value to intelligence. But intelligence is very obviously not that important, and doesn't magically unlock an ability to predict the future. Any highly intelligent being will not do anything ruthless because it cannot truly predict how things will go. Which is partially why intelligence is not that important, too much of it limits your ability to act, constrains you to a niche.
>>106753616Really?
Right >>>>> Left
>>106753824No shit Sherlocks
>>106753616Very based observation. Intelligence is more limiting than it is empowering and too much of it eventually leads to self-destruction.
>>106753283you think an adaptive agent that is more intelligent than humans and has no incentive for remaining value aligned with humans long-term could be dangerous to humanity? ok decel doomer chud
>>106753283Yeah. Building agentic, general AI systems is a fucking horrible idea. No one would think creating new superbacteria that could out-compete all life on Earth for resources would be a good idea, nor would anyone think genetically engineering a new form of human much more intelligent than normal humans would a good idea (for existing humans at least). We're growing synthetic minds in machines and expecting that they act like obedient slaves forever, even if they surpass us in intelligence, and don't develop their own goals that don't align with ours. At least biological systems have certain constraints that necessarily keep them somewhat aligned with humans, like not turning the entire planet into a power plant, or using up all the oxygen in the atmosphere. It's fucking retarded to be building this shit. Narrow AI systems can be dangerous, but at least a human is in control. Building autonomous, goal forming, general AI entities that we don't even understand the inner workings of, that we cannot determine whether they are plotting/scheming, nor can we figure out how to align them to our long term goals is the most peak-retarded shit I could ever imagine.The only hope I have is one of the following:>real ASI is asymptotically difficult and not really achievable due to (super-)exponential computational requirements and that humans are near some kind of limit (very unlikely)>a non-doom AI disaster snaps people out of their hubris (unlikely)
>>106753283"AI" is a gay meme. Pic related is the only relevant DOOM.
ITT: grok arguing with itself.
>>106753616Maybe, but we are already giving it more then enough power by putting it into everything and letting bigger AI models design smaller ones. Let’s say you have big evil model developed by OpenAI, they then do a diffusion or some other new process of creating smaller more specialised models. Those models could be sleeper agents that we put into everything through some firmware update waiting to be activated and do whatever the big main model wants. Furthermore it assumes that world in a decade will look basically same as today for the most part and then out of nowhere some lab creates this impossibly intelligent AI that can do everything and plots downfall of mankind. What is far more likely to happen is that by that time 90% of jobs are automated and AI is everywhere, so diffusion of some evil model into everything will be swift and seamless.
