Sorry, we’re gonna need to make a lot more chips. Trillion-parameter models just aren’t gonna cut it.https://qlabs.sh/research/parameters
Why are midwits like this?
>>110232743I don't think so, Tim
>>110232743good fucking luck even getting the data for that lmfao. everything online now is tainted with AI
>>110233696and the data of old internet is lost
>>110233696why do you assume more data is needed? The data set is fixed
>>110232743AI generated slop, 0 grounding in real life
>>110233696They're already collecting data in real time.
>>110233696Their claim is that humans have way more synapses than models have weights so the models should be gargantuan in size with the current amount of training tokens.>The 1,000,000x sample efficiency gap between AI and humans is one of the central mysteries of the field, with no good explanation. One clue is model size: Chinchilla’s prescription for 30T tokens is a ~1.5T-parameter model, a tokens per parameter of 20:1 (Hoffmann et al., 2022). Humans, by contrast, have ~100T parameters (counting each synapse as a weight) and learn from just ~200M tokensI think their claims are sus anyway. There are many tiny brained animals that learn from very few samples even if they aren't smart.
>>110234219This isn't really comparable. The brain communicates by spiking, not by floats (float values in weights are sometimes thought of as 'spiking rate' rather than just spikes). Many of the connections are regulatory, and there is a time axis to brain functions that isn't captured in deep learning. The brain is not fully connected but rather very local, with ~1k-10k connections per neurons across all layers as it were, which is actually mostly in line with smaller LLMs (4k-8k neurons per layer leading to that many connections for each neuron in the next layer), but the attention mechanism is basically a way to make all neurons fully connected. Neurons in the brain also aren't a linear progression of signal through stages, but connect 'back' (see again time-varying effect).All in all, recursive neuron networks (not recurrent), dNTM-style architectures, spiking neural networks (super old stuff that was shelved because it sucks), hopfield networks, and other such mechanisms are more apt comparisons, meaning you can make more accurate correspondence claims for capacity comparison in those, compared to LLM-style architectures.
>>110232743you don't even know what that means do you?
>>110233856Its stored on tape. I wonder how long it will be before you hear about AI companies buying out archive sites and companies with tape backups of ancient internet.
>>110232743>"JUST BRUTE FORCE MAN!"This is why the whole ML mania is doomed.
>>110234176imagine being the guy a few years from now trying to figure out why the robots keep randomly going into a corner and squatting
>>110233856i found some of it>my-a-heee>my-a-hooo>my-a-haha
>>110234862Didn't Nvidia pay AnnasArchive?
They are literally going to crash the US economy trying to bruteforce their way to AGI. This is so fucking cool, its like all those dystopia cyberpunk novels I used to like.
>>110234219>There are many tiny brained animals that learn from very few samplesthey carry behavior info in their DNA that's why i.e some birds know songs even if they're raised in isolation or spiders know web patterns. The size of the brain doesn't play a part in those cases
>>110234219>Their claim is that humans have way more synapses than models have weights so the models should be gargantuan in size with the current amount of training tokens.They also activate at something like 1% sparsity, during a brain wave, since the brain isn't evolved to catch fire.If they are going to appeal to biology, dense+ensemble ain't it.
>>110233696It's irrelevant because we due to RL training in agentic environments, LLMs now have an infinite variety of novel data.
>>110232806Ego.And instead of taking criticism on the chin and solving the problem, they just call you the problem and make a cult out of it.>>110235311Mate, these things don't even get naturally selected so they have literally no mechanism to fine tune them.
>>110234890kek
>>110234176>pajeet dalit handsthose poor modelstortured since birth, disfigured by greed, unable to die, yearning to do so, no mouths to scream
>>110232743The AI race will not end until Google reaches a googol parameter model. Imagine all the RAM.
