discuss
>>110286055RSI.
>>110286085That’s just the public model.
>>110286055>robotics: 64.9%lmao we really are gonna be able to say "ChatGPT, finish this building" in a year aren't we
>>110286055exponentially more compute for incremental improvementsdead end technology.
>>110286247You people have been saying this for 3 years…
>>110286247you've been saying for this years now as models have gone from not knowing how to make a basic website to writing operating systems autonomously and creating frontier math breakthroughs. where does it end?
>>110286272yes, and its been true for years
>>110286247We're getting exponential improvement on benchmarks over time, just as compute scales. So it's more like linear, which is worrying, no?
>>110286055Really fuckin expensive
>>110286276the real dead technology has been math all along..!
>>110286272>>110286276nonoi have been saying that llms *will* hit a hard wall some time in the future.that future is now. and the maths behind it are very simple.first- whats a context?its the number of tokens taken into account in training, and thats why thats the limit of tokens that can be taken into account when inferring.now, the model is created by estimating the relation between each token during training.that can be abstracted into determining the relation bw two tokens. the exact architecture doesnt matter, attention heads or large conceptual models and whatnot. the essence boils down to establishing the relation between a group of tokens of size up to context sizenow lets consider a context of size 4, with 4 tokens well call a, b, c and d.to create a model out of your dataset you will be considering the relation between a and b, a and c, a and d, b and c, b and d, c and d.thats six relations. the actual formula is that you sum up the numbers from 1 up to (context size - 1).so context size of 4 gives us 3 + 2 + 1 , or 6.now lets increase the context size to 8. thats 7 + 6 + 5 + 4 + 3 + 2 + 1, or 28.now lets increase the context size again, to 12. thats 11 + 10 + 9 + 8 + 7 + 6 + 5 + 4 + 3 + 2 + 1, or 66.you see the pattern?each increase of the context size, by the same amount, and working on the same data,costs more in terms of compute.what i didnt know in 2023, is exactly when a continued expansion would become more expensive than its worthnow i believe that moment was reached somewhere in the later half of 2025
>the exact architecture doesnt matter, attention heads or large conceptual models and whatnot.actuallythis doesnt even need to concern llms, nns or even dlthis is a law that concerns a parent superset even higher, this concerns machine learning as a wholethese are the maths of training a statistically self calibrating system vs increasing the number of inputs considered
