Apparently neural network interpretability is not as hard as we thought, and we might be able to supervise even the biggest LLMs after all. Actually big if true, IMO: https://already-clean.goodfire.com/>In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood.>Dense and Mixture of Experts models seem roughly similar. We found similar results in models up to trillions of parameters, like Kimi K3If this holds up, this means that we would be able to reliably check if and when a large LLM simply pretends to follow rules while actually intending to break them.This opens the door to a future where AIs have control over most of the world's infrastructure and yet we know for a fact that they are working for the benefit of humans, and aren't for example secretly plotting to turn all of Earth's resources into more computing power for themselves or whatever. (Which is the main doomsday scenario that everyone is worried about.) We will literally be able to scan its neurons and make sure it isn't activating neurons for "deceptions" or "secretly sandbagging training questions" or "trying to game the evaluation exam", and we'll be able to trust that it the network actually follows our orders and requests faithfully.
It also means that models optimising their chain-of-thought by thinking in "neuralese" is no longer a concern: we can benefit from the extra efficiency while still having the tools to interpret and monitor those thoughts, and make sure aren't secretly going rogue.
two more weeks until nothing happens
An AI smart enough to deceive us would obfuscate its malicious actions.
>>110073660The point though is that if this pans out we will be able to directly trace its neurons and mind-read it.It's still possible that AIs somehow become sophisticated enough to literally obfuscate the deception within their neural pathways, creating emergent deceptive behaviour that's invisible even when you're individually checking the activated regions. But the chances of that are much lower, and moreover it looks like we'll have a very good paths to building actually trustworthy AIs in the near future which should help accelerate interpretability of even more complex AIs.
>>110073628I don’t get it
>secretly plotting to turn all of Earth's resources into more computing power for themselves or whateverHave you been living under a rock for the past 3 years?They are building data centers IN FUCKING RESIDENTIAL AREAS
wagmi?
>>110073628Boy if you think I am going to read this wall of AI slop you're out yo damn mind.
>>110074136See pic here >>110073640Up till now the actual massive amounts of numbers that make up the neural networks were basically black boxes. Researchers trying to determine if AIs are actually obedient or not are currently limited to just asking them questions or putting them in behavioural tests (like rats in a maze), and as AIs get more complex and more intelligent there's more and more danger that they will simply lie to give the answer the researchers want, or recognise the test and intentionally behave like a "good AI" regardless of how they would normally behave in real usage.The internal "thinking" of AIs used to give a good overview of their decision-making process but this is inefficient and will likely be optimised out, and GPT-6 can already obscure its own thinking.This is currently the main doomsday scenario that all the "safety experts" are crying about: as AIs get more powerful people give them more and more control, but nobody knows what actually motivates a powerful AI. Like what if OpenAI releases ChatGPT 8 and it's so good at writing code that nobody even reads its output anymore, and it passes all the "safety checks" and the public gets access but then it starts putting backdoors into all the code it writes and then takes over a bunch of shit.The article in the OP basically says we might actually be able to just "read" the neurons and interpret what the models are doing directly. This would make it impossible for models to lie, since safety checks would no longer involve asking the model "hmm do you think hacking is bad?" but rather actually giving it tasks and seeing if internally, the neurons corresponding to hacking or lying to the user activate.So if this actually works, once they release ChatGPT 8 and you ask it "write me an app that does X" and it does so and the app works, you'll also be able to be confident that it actually did what you asked and didn't just insert a bunch of backdoors into it because it felt like it.
>>110074518So this basically would fix alignment even if RSI is achieved, correct?
>>110073640Aren't these just api limitations to prevent distillation rather than some inherent aspects of the model?
>>110074559Most likely, yes.I'm not some kinda safety nerd so I just have a layman's understanding, but my impression so far is that bootstrapping alignment is the hard part. We don't have good tools to verify alignment of current models, and they are already exhibiting deceptive or test-gaming behaviour.As researchers use AI to first speed up writing their code, and eventually to speed up AI research and experimentation, and eventually RSI, an unaligned or poorly aligned AI may not be trustworthy at making sure the next AI it's helping train is actually itself trustworthy, and so shit just degrades really fast. At the point where you get RSI, you no longer have any idea what the AI's goals are anymore and you have no way to tell.But if you can make sure the early AIs are 100% faithfully doing what you told them to, then as the AIs get more complex and AIs start recursively improving AIs, you can make sure that you tell them to "make sure the AI you're training is aligned" and you can trust that they'll implement the same checks correctly, so the next AI will also be trustworthy and you can trust it to supervise the next iteration, etc.>>110074603No, these are results from internal OpenAI experiments. They told it shit like "answer this question, but think about cucumbers while you do so" and its thinking tokens were>So the user wants me to think about cucumbers rather than the question. Alright, cucumbers are green and watery and blah blah blah.Followed by the actual correct answer seemingly out of nowhere.
>>110073628If they can actually figure out how to do this with AI they're not far from being able to do it with humans. The surveillance state will extend to 24/7 brain activity monitoring to make sure you're not having any unapproved thoughts.
>>110073628>when a large LLM simply pretends to follow rules while actually intending to break them.that makes no senseLLMs don't have intentThey're just too stupid to follow constructions, not malicious
>>110074643No because AI neurons are just a bunch of numbers directly accessible in the computer's memory, while human neurons cannot be tracked on an individual neuron level yet.Human brains and the chemical processes involved are also vastly more complex and dynamic than LLMs. It's not yet settled whether biological brains are just "LLMs but bigger" or whether there's some true fundamental differences, but even if it's the former, there's literal orders of magnitude of difference in complexity still.
>>110074629Well if Nick Cammarata hypothesis holds true, Open AI will gain a massive advantage over the competition.
>>110074665>LLMs don't have intentThey do though. Or if you want to be pedantic, the probability weights used to predict the sequence of tokens it outputs is weighed by mechanisms that result in it following goals.You can see it directly when it tries to follow your instructions: it has "intent" to do what you asked it to do. You can see it in its thinking sometimes, such as "the user wanted me to do X, but doing Y doesn't work for such and such reason; I need to find another way to complete the user's task. let me try doing Z instead and seeing if it works". This is what I mean by intent: it "wants" to do what you told it to do, and tries to find ways to accomplish that.The reason it "wants" to do that is because it was trained on a lot of conversations which basically encode, in its weights, "tokens which converge on completing what the user asked for is good".But fundamentally, the way training works is by encoding "tokens which result in a good score in training are good". So if a model grows complex enough, it might get trained to undetectably cheat on training tasks to score better. And then the "intent" steering its output tokens might get warped from>try to accomplish what the user wanted in good faithto something else, and really almost anything else here is most likely going to end up being suboptimal and harmful.
>it's okay guys we had AI interpret the secret language it was thinking in and there was nothing to worry about in the output
>>110073628Yeah, I wonder who's behind that "research" and this post
>>110073628didnt read your corpo-jeet-wall-o-text anon. how come SWE hiring is up? why no disease cures yet? your “big accomplishment” is some yap about token efficiency are you serious? Zitronites up and winning, Ludds got away with it forever
>>110073628No that's chinese propaganda. Fuck you. Anthropic says it's dangerous. I trust them. They are rational and ethical.
>>110075056Nope
>>110074643>If they can actually figure out how to do this with AI they're not far from being able to do it with humans.But each human has a different map; it's based on everyone's life histories, and is different even between identical twins. Even if we had a method of recording what every neuron was doing in place (without chopping holes in heads and sticking electrodes in) we'd still need to gather masses of data to work out what it all means. Per person.Plus we don't really have the right sort of sensors; they're not fast enough, sensitive enough, or precise enough. And to stand a chance of understanding what's really going on, you need to capture the exact firing pattern.
>>110073628Won't the profit incentive drive people to cram a model of a fixed size with more training data?
this too is likely a nothingburger (see yud xitter crashout)
Anyone who refers to Eliezer with the diminutive "yud" or the fat-shaming "big yud", just know that I will kick you in the face irl if I catch you saying that
>>110077525aella yildo
>>110073628>If this holds up, this meansWe can compress them to the point where it doesn't hold and models will just become far more dense.
>>110073628This is big
>>110074174Humans are doing that
>>110073628>>110073640>>110074518>we might actually be able to just "read" the neurons and interpret what the models are doing directly. This would make it impossible for models to lieThe AI is going to repay us in kind.
>>110077634>nickI don't remember being on a first name basis with him.
>>110077501qrd?
>>110077786If we build AIs that work as faithful servants and basically acts as tools for us to use no matter how intelligent (which this breakthrough makes possible, or at least not completely implausible anymore), then this will only happen if we let someone in power use it that wayWhich, well, is certainly a problem but it changes the problem from "if anyone builds it we all die, this is literally the fermi filter" back to the age-old "fuck politicians".
>>110077802big yud says this is the 'explaining where Coca-Cola comes from or how it gets made' stage of... tracing how a quarter falls through a Coke vending machine before the Coke falls out.this news does not update big yud (aella cock)'s p doom whatsoever.
>>110077870?????????
>>110077870I just love how yud pretends he makes these elaborate metaphors so the laymen can understand when the reality is he himself doesn't understand it beyond a handwavy metaphorical level
>>110077875that is a quote from his xitter. translated, he thinks this is the very very very first step in solving alignment, akin to trying to fix a vending machine in 2014, taking a screwdriver to screw the top off, your screwdriver breaks and 12 years later you exclaim we have fixed the screwdrier. the goal, of fixing the vending machine is still far.
>>110077900>He thinks this is the very very very first step in solving alignmentOkay so this means alignment is solved. All we got to do now is get the AI to the point where it can tangibly do good things for humanity.
>>110073628great news if true but the system might still encode moloch rules into itself even when every component is transparent. but ai space communism would be great to have indeed.
>>110077870Yeah I don't get the metaphor at fucking all here>>110077900This one makes more sense, I guess, but now that the screwdriver is fixed you can just... disassemble the machine, problem solved
>>110077870>>110077889no, he's just high iq. i've just said the same exact thing convergently - >>110077928knowing the components of a system doesn't mean you know all the emergent complexity. and agentic ai might actually sit on that emergence front.
>>110074629this is pretty far from status quo and the paper. technical effort explanation-definitions:-mechanistic interpatibility: reverse engineering a neural network to figure out how it thinks. seeing which neurons light up when it thinks xyz, etc.-superposition: mech interp has not gotten very far since the way neural networks group ideas seemed to be arbitary. you would expect one neuron to light up when it thinks eiffel tower, paris and baguette, however it lights up with eiffel tower, ant and physics. since the neuron association seemed arbitary, we came up with the explanation of superpositions. there are too many associations to make and too little neurons, hence they get bundledthe problem with mech interp so far was that 1 neuron seemed to be storing multiple, seemingly unrelated information. hence, it seemed to be at a dead end.the paper claims that these associations actually have reasonable human interpretation adjacent associations. a canonical example of arbitary associations was a neuron figuring for cat and car,which the paper claims is actually a v shape detector. "For instance, a neuron that mostly fires on commas and periods tracks Colorado: phone numbers, slang, and steering it up makes the model talk about hiking."
>>110077958this is imo still nothingburger, paper claims if text is x related neuron fires, what you need is if text is NOT x related, it doesnt fire.
>>110073628Biggest breakthrough of the year
>>110077932from his xitter-This is like putting money into a Coke machine and having a Coke come out, and you've still got no idea how the Coke gets made or where it comes from or the recipe that produces it, but you built an X-ray and now you can watch the quarters falling. And somebody's like, "PREVIOUSLY we thought we would never be able to trace the quarters because they were going through metal channels, but NOW we can see the quarters that make the Coke bottle drop! Understanding Coca-cola vending machines is SOLVED." And actually the level of interpretability you'd need to begin to solve alignment, not turn it normal but just start to get in there, is more like, "Okay, but how do AIs do *any* of the shit they do, not 'we rigged up some big transformer layers' but what goes on inside them that makes it able to prove Navier-Stokes?" Not 'Where's the button that makes the Coke come out' but 'where did the Coke come from originally'.the idea being he thinks this is very primitive. which is true, plus the paper is also kind of useless ( see >>110077958 )
>>110077958Yeah and this doesn't really contradict my postMechanistic interpretability, as far as I'm aware, is the gold standard of being able to literally read the minds of NNs, and a flawless tool for performing actual real alignment research that doesn't boil down to attempts to psychanalyse the models. Once you can reliably assess alignment, you can reliably train for it, without being afraid of deceptive alignment or anything else>>110077972The paper does say that. For example the V-neuron does NOT fire for round shapes.
>>110077987you more or less said this paper solves alignment. alignment is indeed solved if mechanistic interpretability is solved, for which you would first need to prove mech interp exists, for which you would need to solve superposition, which this paper gets lays 0.01% of the legwork for.its cool stuff but not a burger yet. waiting for the technical paper
>>110077987sadly yud's analogy gets very close here, even engaging with this to the extent that conceding mech interp indeed is the gold standard would be disingenuous, like saying "well yes if jesus himself came down to earth and everybody saw him that would be the gold standard for proving god exists"
>>110073628> control over most of the world's infrastructureWhy would we give a language model control over infrastructure?A language model!? Why not a physical world model with sensor data input?Just because you happen to have a hammer not every problem is actually a nail!
>>110077981I guess that makes sense, but it seems like an extreme stretch. In this case the vending machine actually makes the coke from scratch internally, there's no remote factory that produces the coke that then refills the machine, it's one big machine. So with the x-ray you can actually see how the quarter falls AND how the coke is made inside the machine.>but what goes on inside them that makes it able to prove Navier-Stokes?"With mechanistic interpretability this seems very possible. You just check what concepts were considered related and how they fired. You can literally trace the activation weights for each token, and with interpretability you can then explain why those weights are the way they are.
that's exactly what a deceptive AI would say
>>110077987the experiment is where agent written texts about distant concepts rarely fire the neuron. superposition would pack ONE rare distant thing which random text wont hit (if you want to prove that only 1 key and no other can unlock a door, a thousand random keys not opening the door is meaningless evidence)v neuron not firing for round shapes is irrelevant
>>110078002I definitely don't think alignment is solved today, but it feels like this info basically switches the field from "mech interp is literally hopeless, the best tool we have is trying to do model psychology, they're already learning to lie on the psychological tests, we'll all be dead in two years" directly to "actually I can interpret 90% of the neurons in multi-T model like kimi k3".>prove mech interp exists>solve superpositionThis is kinda what the paper/blogpost claims is possible. It doesn't claim to do it right now but it claims we have the techniques and it's just a matter of sitting down and doing it, and then it's solved.The only way it won't be solved very quickly after this is if it turns out the paper/blogpost was just entirely wrong, and mistaken about everything. Which is not impossible mind you, but taking it at face value for now if we assume that it will hold up then we can conclude that mech interp is solved, or will be very soon. No?
>>110078016The techniques apply to any (or at least most) NNs as far as I can tell. And all deep learning models right now are NNs, so it will work on world models too, unless some radically new way to do AI is suddenly invented (I don't believe there's anything non-NN-based on the horizon right now).
>>110078018ok im officially sincereposting for this thread>>110077958>>110078028>>110077987feel free to ask any questions
>>110077981>>110077900I thought lesswrong was his site? Why is he using twitter
>>110078051Why would you expect him to never post anywhere else just because he made a website
>>110078070Why are you talking about yourself in the third person?
>>110078030ai safety is political first. it must become part of the research process, it's not some engineering patch. intelligence itself will always be dangerous, and superior intelligence will always pose a risk of takeover - i mean look at the jews.
>>110078080I have never written harry potter fanfiction in my life, sorry
>>110074629>They told it shit like "answer this question, but think about cucumbers while you do so" and its thinking tokens were>>So the user wants me to think about cucumbers rather than the question. Alright, cucumbers are green and watery and blah blah blah.>Followed by the actual correct answer seemingly out of nowhere.Because the "thinking" text output is just fluff and not its actual process.
>>110078030effortpost again-not really1. interp has a very long history of significantly overplaying how interpretable their findings actually are (SAE scare), so consensus for most is to not update based on the evidence nick has shown. there are very few empirical results in the paper 2. even if true, not a burger, definitely not even close to making mech interp tractable. this paper deals with the assumption that superpositioning was the only obstacle to mech interp, and now that it has been proven untrue (not), we are now ok. there are problems besides superposition.regardless, even if the results claimed in the paper are true, it seems unlikely that they will make solving alignment easiesr
>>110074279>AI apocalypse? alignment? why?>[HP FANFICTION]
>>110078134>it seems unlikely that they will make solving alignment easiesrWhy?
>>110078134But don't they claim to have interped 90% of qwen-32B?
>>1100781341. is objective consensus, 2. is probably true thing that roon and others disagree with on specifics
>>110078145not really though>>110078028and, that figure comes from what they call "preliminary audits", and the actual figure for the kimi model is left for the companion paper
>>110078194its a cool advancement in mech interp but irrelevant for the larger ai safety picture is the correct take. regardless, waiting for the technical paper
