>>108512057
Scale. It's amazing what you can do when you throw enough hardware at the problem.

>>108512057
Akinator is a whole different beast.

I watched an intro from a course by Andrew Ng and his point was kind of >>108512401. When you have lots of data the magic starts to happen, but there was a breakthrough with the invention of the transform architecture. LLMs aren't just looking at the previous n words to calculate the next, they look at the "context" (which is the layman term for Attention) of the conversation and use a previously built mathematical construct of meaning (turnings words into numbers and then creating an invisible layer of connections during training) which then can be used to predict the next word. Also akinator was just a complex decision tree, was it not?

>>108512487
>>108512436
so what was the innovation that pushed this slop into being so profitable and "Revolutionary", the whole context thing?

>>108512493
Yes and no. Yes, the revolution happened because of big data + the Transform architecture, but no, it's not so "profitable". It's pretty fucking expensive and they are currently losing lots of money to stay competitive while praying to God that the competition just can't keep up... eventually.

>>108512526
by profitable i mean its still profitable for them to scam investors with the fake promises, i know they lose money due to the inefficiency
but if this is really how llms work, that they always have to rely on mega data and brute force, even with economy of scale won't they always stay inefficient and unprofitable?

>>108512493
>what was the innovation
There were multiple innovations plus simple brute force at scale. It's nowhere near being profitable, and it's still uncertain how useful it will turn out to be. A lot of AI output is solidly in the uncanny valley territory right now (e.g. the ballyhooed AI-written C compiler that can supposedly compile Linux but couldn't compile a simple hello world program), and it's uncertain if they'll be able to pull out of it.

>>108512526
>they are currently losing lots of money to stay competitive while praying to God that the competition just can't keep up... eventually.
Which btw was the Uber (and usually is the whole startup game) prime strategy during the 2010s. They drove prices down so aggressively that it made the competition a bunch of kids playing tic-tac-toe and unable to compete, while of course expanding like maniacs.
Different from Uber, though, it's not that hard to catch up when you are an AI company, thus the famous Google quote, "OpenAI has no moat, and neither do us". You can even train on your competitors tokens (a practice called distillation, which recently Anthropic tried to poison).
Btw that's why Altman and co was seething and shitting themselves in fear calling for AI Safety and talking about the end of the world, the risk of AGI etc. They were try to regulate the market so they could curb-stomp any other companies from trying to get in.

>>108512535
>even with economy of scale won't they always stay inefficient and unprofitable?
We are in a phase of the tech, which is the case of almost any tech race, where they are burning crazy capex to stay relevant and competitive. There's nothing saying that after the competition dies out (which sort of happened to companies like Mistral, and I think Deepseek seems to be in deepshit, too) and they get their exit, the engineer will go gaga on optimizing everything to make it cheaper to run.
Google did exactly that these days. I can't give you the details, but they made context window tokens fairly compact in a recent paper, something like that.
So my point being, they just need to set their minds to, but right now is not the time yet. Or at least that's my reading.

>>108512590
>engineers

>>108512057
basically same but more layers, more parameters