This is nuts
>>110231849Anthropic is gonna IPO and have $200-$400 billion in cash to play around with. They’re gonna figure out RSI. This is it.
>>110232021This is it; AGI? Should I better airgap my devices before November 16?
no THIS is nuts*unzips*
>>110232136Haven't you graduated from kindergarden yet?
>>110232152yes sir
>>110231849
>>110231849So besides number go up what does it say?idgaf really since I already know it's IPO hype scamming. But what are they saying and more importantly what are the leaving out?
>>110232021It fixes repetition loops in qwen 0.8B model. It could not improve another smaller model that had no repetition loops.
>>110232191Once upon a time, it couldn’t do math.
>>110232136Alright, now cough for me.
>>110231849what's the cache?
>>110231849I dont get what its saying
>>110232965he's just saying older models couldn't do it, but the new one canbut instead of a gradual improvement it went from not being able to do it at all to really good at doing it
I asked gpt to look this up for me and it is kind of incredible
>>110233033No, I think its to do with this
I'm still convinced that nothing about that the public is told about LLMs is true and they are far simpler than you are led to believe.
>>110233071This is pretty fucking cool, good for consumer hardware.
>>110231849Question, isn’t this basically just new way of doing distillation, but different? You get smarter model to build dumber smaller model but you can’t make better model from worse model, just like with distillation. Sure distillation is completely different thing but in terms of the wider real world effect it’s like distillation.
>>110233033Isn’t this basically almost every capability? It can’t do X and then suddenly it mastered X in no time?
Blah blah this just means that the people who get paid literal millions of dollars to teach the computer software program how to do tasks are doing it.
>>110233071What was properly done:>big llm generates answers / questions>feeds the questions to little llm>scores how good the answers are based on own result>modify the weights of little llm a little, store vector>repeat>move in the direction of better resultsSo they're lying it uses gradients or the LLM is lying about using them.
>>110233135this exactly.They're such a bunch of crooks.
>>110231849Every month they tell you this shit, and yet look at the actual results. A 13B model today is what, 10% "better" than a 13B model from 3 years ago? Wow, AGI is here sirs.
>>110233601CAAAAAAAAAAAAAAAT
>>110233135so this is actually bad for anthropic/openai because it means way better specialized/smaller models that run locally?
>>110233622Yes.
>>110233135theoretically you should be able to do this on a model the same size as the one you're working on, now that i think about it, i see no reason why you cant reverse it and have a smaller model working on a larger one... i'm not sure if this is actually rsi... unless you can tell the ai to simply do a breakthrough and make the model better... god this is rsi isnt it
>>110233585retard
>>110233705Wow look at all this useless stupid shit, yet it still can't count letters, still repeats itself, still hallucinates, still hasn't improved anyone's lives, still hasn't increased productivity, still hasn't given a single useful program. Very good sir, thank you very much, very worth RAM costing $2000.
>>110232218It still can't. We need extensive harnesses and tool calls and tools to make it simulate being able to do math.
>>110233917You’re fucking delusional
>>110233175There are multiple parts to this claim. The main and most important claim is it can understand what the weights compute. This is massive because instead of just editing weights, it means it can write code to compute functions the model needed to approximate in order to get to the solution. This is a new form of automated data engineering, basically, but if this can run over a few rounds without breaking, it means we could make (I exaggerate here but imagine) an entire LLM into just a fully deterministic, very fast pure code solution with small gaussian models to patch up only those parts of language we can't model with a well understood deterministic function. 4GB ram 4000 tok/s on core 2 duo opus 5.5-level performance.For research, it means building hitherto undiscovered relations between elements: you train a black box model and use the LLM to write the deterministic code it has learned to, e.g., classify signals in an oxford nanopore device. This could lead directly to using oxford nanopore with proteins without needing to do the weird decade-long research they're on about now, revolutionizing proteomics overnight.The weight edit shit is more for fluff.
>>110231849deez nuts
>>110233932You can test it any time if you have api access.I have a PhD in deep learning and I've been working on deep learning, language models, and LLMs (building and training them at first, then only finetuning because the former job type disappeared if you're not a poo with a bought degree, then only using the APIs because companies didn't want to spend money on serving their own models anymore due to marketing issues when they can't tell customers that "we're using the latest gpt 6.9 ass straw").My models still run millions of conversations a day at multiple client sites despite not having trained shit since 2023.These days my main task is literally to benchmark these frontier models with various kinds of harnesses or none, on internal datasets and tasks that are business-oriented. These days it's legal documents (contract understanding and modification). There are often very basic math involved (if the effective date is 2026-01-15 and the contract ends in 18 months, what date is the contract end? If there are 5 contractors at 20 hours per week max for this 1 year contract, what is the maximum contract value?), they still get that shit way wrong very often.
>>110233585Don't be a fucking idiot, training AI from zero is expensive AF. That new trick will cut costs so much is unbelievable.
>>110233071What the fuck? Does it work with DiT models, VAE?
>>110233940>The main and most important claim is it can understand what the weights compute.I agree, it seems like this would be the kind of thing an llm would be good at, by pure statistics, if it can do this i see no reason why it can't do it to itself
>>110233975Not so, as the implied computation becomes more complex from depth and distribution of the function over more neurons in a layer, one might expect this stops working. Some functions will be computed "on the side" on the "first X neurons of the first 7 layers" while other neurons would compute other functions of different depths, so you can't go per-layer either.However a "puzzle solving method" (identify the easiest functions first and retrain the model, for example) should be able to get quite far, just not all the way. The question is "how far can we go". It's very exciting (if all this is true).
>>110233974They tested with a simple ppo-trained deep rl system so in principle it should work.>>110233970Only if "AI" means "frontier LLM" to you. The vast majority of useful models are quite cheap to train, 1m of train time on 1 consumer GPU gets you a very capable classifier on complex classes on full video data.
>>110231849>blue checkmark niggerbuy an ad
>>110231849They simply fed it common neural network examples as training data.
>>110233992I disagree because I saw it in a dream, we'll see who's right (me, i saw it in a dream)
>>110233071The crazy shit is that they didn't need to do anything elaborated.They literally copypasted the weights lmaoFucking insane, we are probably going to have Opus 5.5 level local models running in pretty standard consumer grade hardware. Albeit they would need to be extremely specialized and fine-tuned for a single task.
>>110234016no
>>110231849i don't know how clankers can look at billions of raw numbers and say, "Yep, this is wrong; I'll fix it" lmao I hope this is true so AI can hurry up and kill us all
>>110233992shut up, Gemma.
>>110234156It can't really, that was just fluff because nobody's getting funded to do things not directly related to LLMs nowadays.
>>110234156the idea that it just "magically knows" the weights sounds like bullshit. The entire point of LLMs is that they have only the correct weights adjusted during attention based on the actual answer you want, so I would assume the LLM's process is1. run a test prompt2. judge the output quality3. look at the difference in the weight4. change the prompt5. repeat6. based on the weights that changed in each prompt, adjust them in the direction of better outputin which case the improvements might be very real but the claims surrounding them are being intentionally dishonest about what is actually happening
>>110234165AI psychosis is a hell of a drug>if it's smarter than my tiny sub-65 IQ, it must be AI
>>110233071>0.8blmao evenwe can also cure cancer in mice in 20 different ways
>>110233175You are describing distillation which has been done for years
>>110233940The domain specific models you mentioned out of for example genetics or chemistry aren't llms and they require training.They claim the llm did it because it """understands""" somehow. This is totally implausible and pure hype.
>>110234211Well yes, that's is what I mean. The only difference to what deepseek does to western models in this case the model which is distilled from is told to cooperate.
>>110234220>The domain specific models you mentioned out of for example genetics or chemistry aren't llms and they require training.Correct. (I will add that pLMs and dna language models are completely worthless scams and don't even being to work, while the bespoke models are amazing and solve great problem classes we had no solution without them).>They claim the llm did it because it """understands""" somehow. This is totally implausible and pure hype.Agreed, but I don't care about that part. They have other issues with the writeup, for example they didn't correctly ensure the LLM did not have tool access (among other things) in the section they claim that tools are forbidden, they only prompted "no tools pl0x" and called it a day. That is, the "researchers" are sloppy as fuck, if they even wrote any of that or did the research themselves. It explains why everything in the writeup is written all retarded.That said, someone else on X was "able to reproduce" and it's easy enough to test (I plan to do that later today). The problem isn't "can it understand" but, is it capable of extracting the function learned by the weights, which does not require understanding: just pattern matching, as there are plenty of esoteric coding schemes for various functions, especially popular in signal processing and math applications for example. Think diophantine sets that can be recast in equations whose zeros are equivalent to computer programs, or circuit functions (or/and/xor/nand) being encoded on analog signal processors, etc.
do you subhuman niggers even read anything past the headline, or even read the llm slopped summary of it?>hey go fix this model>model improves... due to no infinite loops>model is significantly worse compared to the pre-changes model which didn't loop when it was given a proper harnessastra made the model WORSE, not better. the whole thing relied completely on qwen going in circles due to a bug which astra fixed in the model rather than in the harnessfucking vantablack subsaharan niggers, the lot of you
>>110234306>vantablack subsaharan niggersI'm picturing that in my head rn
>>110234306Easy to test, if the """improved""" model with a harness performs better than the stock one.
Just a speculation on my part, but I think that while it might seemingly improve something on the surface, it fucks the model in many ways that are impossible to foresee. There's a reason the models are trained with insanely many small steps and that's because no one knows how they work.
>>110234365No, the reason for the many small steps is because the stochastic effect of using subgradient estimators allow jittering the model out of jagged local minima.
