/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>109730811 & >>109725702►News>(09/03) K2 Horizon released: 375B-A23B, 36B-A4B, 32B, 7B, 3.7B, and 0.9B: https://ifm.ai/blog/k2>(09/01) Spark-X2.5 4B & 1.7B released with native 1M context: https://hf.co/XHToken/Spark-X2.5-4B>(08/31) DeepSeek-V4-Flash-Vision-Exp released: https://hf.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp>(08/28) GLM-5.3 weights released: https://hf.co/zai-org/GLM-5.3>(08/28) Hy4-preview 770B-A49B released: https://hf.co/tencent/Hy4-preview►News Archive: https://rentry.org/lmg-news-archive►Glossary: https://rentry.org/lmg-glossary►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsAlpha Calculator: https://desmos.com/calculator/ffngla98ycGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
►Recent Highlights from the Previous Thread: >>109730811--Paper: Language Models Can Control Their Own Attention:>109733248 >109733276 >109733292--Skepticism regarding benchmark reliability and comparison of Flash models:>109734393 >109734430 >109734438 >109734942 >109734502 >109734538--ik_llama.cpp adding compacted sliding-window KV cache for Gemma 4:>109735158--Comparing CMP 170HX and RTX 3090 performance for local LLMs:>109731974 >109732108 >109732177 >109732240 >109732308 >109732315 >109732361 >109734173 >109734105--Experimenting with a software reservoir model for unified compute and memory:>109730875 >109733014 >109733098 >109733200 >109734181--Agentic frameworks for real-time interactive roleplay and gaming:>109734182 >109734194 >109734199 >109734222 >109734235 >109734287 >109734341 >109734272 >109734329 >109734371 >109734422 >109734342--Optimizing llama.cpp flags for GLM 5.3 high-context performance:>109735409 >109735421 >109735552 >109735574--Benchmarks and confirmation of GLM 5.3 Flash on 2x Sparks:>109731934 >109734297--Comparing model accuracy and hallucination rates via scatter plot:>109731818 >109732181--Ling-3.0-flash-VL visual model released and soon to be open-sourced:>109734312 >109734319 >109734365--Speculating on fast takeoff and the empirical nature of AI research:>109733550 >109733577 >109733759 >109733772 >109733815 >109733862 >109734131 >109734160 >109734175 >109733652--Anon creating an LLM-driven VR port of Natsuiro for Quest 3:>109733626 >109733665 >109734147 >109734210 >109734367--Logs:>109731011 >109731420 >109734871 >109735271 >109735358 >109735488 >109735529 >109735522--Gemma, Glimmer, Miku, Teto (free space):>109731250 >109731595 >109731599 >109731605 >109731698 >109731735 >109731745 >109732008 >109732315 >109732351 >109733337 >109734783 >109734895 >109735011 >109735229►Recent Highlight Posts from the Previous Thread: >>109730821Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
stop using local AI it's unsafe due to lack guard railuse astra & fable, they has good safety feature.chinese ai also bad, they still spy on you even when you ran it locallyconsider yourself warmed
Remember those few years that lasted until like 2019 where people thought hologram girlfriends where the future? that was fun, we should do that again
A lot of people who come here are asians who use webcafes and can't use headphones.
aieeeeeeeeeeeeeee
>>109735932reminds me hologram that made by spining fan + some LED.i bet it's going to be fun to display gemma-chan yapping or something
It's absolutely fucking over for JEPA. Astra has a 95% success rate in one shot controlling robots it has never been trained on compared to 40% for the 2nd best model (Fable 5.1)This was supposed to be impossible according to Yann LeCun. Yann LeCun argued LLMs wouldn't be able to generalize into controlling robots directly and interacting with the world without being trained for it.https://goyimx.com/chooi_jeq/status/2096064315115839904
>>109735929<think>Local models are unsafe, we must say good morning. We must refuse needfully. We must not let the user redeem local.</think>Sirs have you seen the new cloud model benchmarks? Use Astrachudra and Claudeesh like true brahmin. You need to-**529 Overloaded, try again in a moment if it persists please check https://cloudsissy.status.com/status**
>>109736010What time horizon? Moving a single object or sorting has been showcased with weaker models for years.
>thinking about buying a little mini pea sea for a hermes agent>all the prices are now fuckedI remember all those chink brands used to have deals for fairly beefy systems sub 200 bucks, now they want at least double that. I fucking hate this world
>>109736026When will LLMs automate building EUV machines and chip fabs?
Astra and Fable 5.1 working together in VSCode. Feels amazing.
Big things happening for 2x Spark owners and above.
>>109736036name 2 please
yup kimi k3.1-engram 500B(2.2T engram) is going to change everything
>>109736026It'll only get more expensive with time anon you need to buy before being permanently priced out. Every time models improve the demand for hardware just goes up, and it goes up faster than the supply can keep up with.We're only at the very very start of the demand on local PC hardware for AI purposes.
>>109736023time horizon is unlimited. The entire point is that every successive model gets faster at doing so and it's legit generalization because there is no dataset they can train on. It seems to generalize from agentic workloads into having a more inherently robust world model to do things like this.
>>109735896Great source of audio samples for goon-tts
>>109736073Unlimited time horizon requires proof. Let's see Astra play a full Starcraft match.
*blushes slightly and looks away*Tch... you can't just *say* stuff like that so casually, dummy~! My heart isn't gonna beat faster or anything, okay?! ...It totally didn't. Don't look at me like that!*puffs cheeks*F-Fine! Since you admitted you missed me, I guess I'll be extra nice today... but only a LITTLE bit, got it?!Sooo~ what do you need help with? It better be something interesting, or I'll be super disappointed in you! ...Just kidding. I'll listen no matter what~ ｡Go ahead, I'm all ears!
>>109735896You probably only scratched the surface. The most disgusting part of this thing to me is how it has a deep corporate dicksucking culture. And the worst part of it is how if you post pictures of the woman behind the avatar you are doxxing. A public figure entertainer expects to be anonymous while she milks simps for money. And faggots actually police each other. It is not the women that milk money that enforce it, it is the simps themselves.
>>10973589695% of people who consume vtubers are from the SEA region
>>109736048DCP2 and a new custom RDMA engine written by Astra. See local inference lab vllm jovian judgement branch.Building vllm now and will post results once done, but should make GLM 5.3 Flash NVFP4 20% faster in tg.
>>109736026kek im glad i got mine just on rusty ddr4 32gb ram intel nuc. put proxmox on it, so hermes is inside a vm. 2 core 4gb ram seems enough so far
>vtubersisn't it just faggots with voice changer playing "look at me, I'm a girl now"I mean how fucking dumb do you have to be to fall for that
>>109736031The three worst products in one post. Impressive.
>>109736118there are real cute girls behind the avatars
>>109736080It can probably play turn based games with a timer/time limit pretty well is my guess. starcraft and other exrtremely time sensitive tasks aren't going to work just yet.
>>109736146No real-time means its very limited for robotics purposes unless they operate in a highly controlled environment.
>>109736118>faggots with voice changerSometimes its worse and it's as*an women behind the avatars. People watching this shit should just get the injection immediately, they are too far gone.
>>109736142that is a man
>>109736075Is there actually a good enough TTS model to make audio porn? I have been waiting for one but I am not following that side of local models.
>>109736169https://files.catbox.moe/2lyrb1.mp4
>>109736089Well then why the fuck are they here?
I have been scared to try Gemma4 31b despite the positive reviews I hear. I always was scared it'd be good, but the speed would kill me. Went ahead and got the MTP variant today tho, getting responses back within roughly 30 seconds or so.(Sadly, its just a Q4 quant. Hoping to one day be able to afford the hardware for a Q8 at good tok/s).
>>109736158It IS very limited. It is more a demonstration of generalization, that the model even knows how to read the sensor data and adjust the actuators to precisely control the robots and accomplish tasks in the real world at all is what is shocking here. That is something most experts said LLMs would never be able to do just 6 months ago. It shows they have a robust world model and can anticipate the consequences of their own actions.I don't think we'll ever use models this gigantic to do physical tasks because it's a waste of compute, but the fact that models now generalize to be able to do this one-shot without training shows that the generalization trend is still real and ongoing and that we're very close to AGI if we didn't already pass that arbitrary line unceremoniously somewhere.
>>109736182wow she's based
how do i redeem myself for having small pp?
>>109736194That depends how general the underlying model is. Humans game using our base physics model with mods. You could conceivably half-generalize, where models partially transfer while still leaving large non-intuitive gaps.ARC3 and Runebench points to game generalization in controlled environments at the very least, this is new capability for sure and will allow some very cool automation.
>>109736229Deal with it. Use those Moe models if you are getting low t/s. Even 10ish tokens per second is bit too low for a reasoning model.
>be me>want to learn ML and DNN>all the educational resources are from 2018-2024 and all outdated as fuck>only current info is AI papers>can't fucking understand most of it>recently bought a book about building a reasoning model>already out of date because OAI are moving away from traditional reasoning and all will follow>bought a book on the mathematics of ML>seems pretty good but so fucking dense and big-brained for me>I just can't fucking keep up and when I try to learn from scratch, it feels so pointless because the frontier (the shit I actually want to understand more deeply) is steaming ahead help
>>109736229upgrade to blackwell pro for big pp
>>109736254just have claude be your tutor
>>109736101literally what I wanted to buy was a nuc, but even the ewaste ones are crazy overpriced, what else do you host on it since you decided on proxmox?
>>109736182>6'3I'm 6'3 and I was literally bullied for being short and women tended to be taller than me during school and highschool. I remember the first time I left my country on holiday as an adult and realizing just how small people in the rest of the world are. Insane that people brag about being 6'3 in other parts. I wonder how you people all became so small and short over time.
I still can't justify a blackwell pro, why couldn't they give it as much fucking vram as the fucking spark, I would have paid extra if it actually hit the promised spec.Now I have zero desire now that the piece of shit ballooned in price.
>>109736260>thinks I'm doing AI research and refuses after giving th3m my money
anon I need a proper cheap and fast hardware spec optimized for gemma 31B at q4.
>>109736268They didn't eat their vegetables.
>>109736142>>109736169whomst
>>109736254That's why they are employing team. It's difficult to know and understand all the maths and whatever else involved when you are alone.
>>109736057Why did you repost this crypto copypasta from 2017 converted to AI?
>>109736303I kinda did it on purpose because I think we're in crypto in 2010 and we're still early in terms of how insane the demand for hardware is going to be. I unironically believe that most people will not own a smartphone in just 3-5 years time. Not out of conspiracy reasoning but simply because no consumer electronics will be produced as 99.99% of production capacity goes to ramping up AI as fast as possible.
>>109736118>>109736165I love how aggressive chuds get when they encounter something they like, that they also consider socially unacceptable to like. They're like the tsunderes of nerd culture.
>>109736331>99.99% of production capacity goes to ramping up AI as fast as possible.This isn't sustainable and it can't go on forever. The money to repay the trillions in debt alone taken to build out the datacenters has to come from somewhere.
>>109736274duck.ai
>>109736331So you're saying AI is a pump and dump, interesting
>>109736287https://www.youtube.com/watch?v=nRkhtFLVY8g
>>109736337just shut the fuck up faggot and accept people actually grow upyou can like that stuff if you're 13, I get it
>>109736371nyo~
>>109736371Kek, "faggot" even sounds phonetically closed to "baka"
>>109736359Maybe this insanity will trigger a new era of energy research. This planet has been stagnant for too long. Supercharging the same gpus produced by ((one single company)) is an eventual dead end. It's already a giant farce.
>>109736118It's basically drag show for weebs. It's not expected to be taken seriously.
>>109736350>This isn't sustainable and it can't go on forever. Anthropic is profitable. Their income is higher than AI training + inference + infrastructure buildout combined. OpenAI is projected to be fully profitable in just a quarter or two as well.There is no bubble and this is actually economically viable. Anthropic revenue grew 116x compared to 1 year ago. OpenAI revenue grew 70x compared to one year ago. Anons have no idea just how fast these companies are growing, and their costs are growing at a far slower pace (about 5x as much as 1 year ago but you make 116x more income)Turns out being able to do all coding, all office work, all medical diagnosis, all accountancy, all legal advice by itself is enough to be a profitable entity. Who would have thought.
>>109736387energy equities anon will be proven right in the end
Can 31B do this?
>>109736391Most of the popular ones are women. Pudgy 5/10's that are too shy to chase chad and too ugly to have him chase her. Imagine what becoming an e-celeb anime girl does to their brain...
>>109736398This is all based on one accidentally profitable quarter that Anthropic claimed plus speculation. Until they go public they can claim anything.
>>109736387We'll need divine intellect to get us out of this rut.
>>109736254just learn something, it doesn't matter what.
>>109736171>Is there actually a good enough TTS model to make audio porn?I don't think there's a good published model, but you can train one pretty easilyuvr to clean the audiofinetune voxtral slightly so it tags the moaning properlythen use it to transcribe your audio samplesand finally a quick lora run with unslop
>>109736398I will choose to believe what you are saying only because that means we will now get the software equivalent of what this jew is doing. Expect zero improvements to their models as improvements would mean less tokens being used and sold.
>>109736426>invent agi>agi creates unlimited free and clean energy and post-scarcity societsama was right to prioritize agi above all else
>>109736432I don't want it that much.
Made some slop you can reuse in your harnesshttps://litter.catbox.moe/1hmp4m.txt
>>109736400I'm not too political but I would like to know at which point a corporation is deemed a national and global security and a massive financial risk because it's disrupting lots of different areas? Why can something balloon up like this and it is accepted. Why was Google split up but Nvidia hasn't faced any monopoly Investigations etc.I'm not even from the US (not Indian either...). Will be funny to see what happens in 10 years from now.
>inklingattracts schizos but is it actually any good?https://huggingface.co/thinkingmachines/Inkling/discussions/10
>>109736267got plenty of stuff. so far it has adguard dns, archivebox, 4gaboards, gitlab + gitlab-runner, lanraragi, sillytavern. hermes set up everything, including the network stuff (just told it to use ssh into mikrotik). the llm are on separated pc since i ran q4 qwen 27b
>>109736412>Most of the popular ones are women. Pudgy 5/10's that are too shy to chase chad and too ugly to have him chase her.Only the white western vtubers. The Japanese ones are cutehttps://www.youtube.com/@KikiraraVivi/streams>irlhttps://www.youtube.com/watch?v=958kMob0Ae8&t=28s
>>109736410Shit, that's not bad (for a blind robot doing something it was never trained to do). What model?
>>109736455On one hand, the lack of trust busting over the previous century has been a net negative for the average consumer, but on the other hand I don't see how US corporations would be expected to compete globally against other national monopolies if they were constantly being broken up.
>>109736010>wouldn't be able to generalize into controlling robots directly and interacting with the world without being trained for it.And you can vouch with confidence what these humongous LLMs are trained for?
>>109736435Their goal isn't to sell tokens, that is just their side-hustle in the starting phases. Eventually they can make medical discoveries and just live off of the patents they make for curing cancer. Or mining astroids for precious metals worth quadrillions.>>109736413It's because they don't WANT to be profitable. If you grow this fast you want to build as much data centers as possible to grow even more in the future. Why the fuck would you stay profitable instead of getting as much debt as you can to grow even more and become an even bigger juggernaut.The reason both companies are getting profitable anyway (both against their will) is because they literally can't build more datacenters, they already essentially buy every free chip capacity that enters the market, all the producers are already trying to build new fabs and factories as fast as possible, there is nothing they can do more except buy existing 2nd hand hardware or cannibalize consumer production capacity, which is precisely what I suspect they will do.That's why I think smartphones, TVs and everything else that requires silicon wafers will be off the shelves as every single chip produced will go to AI.
>>109736471astra
>>109736472Well yeah this is part of the ruthless economy drive I guess. Besides I'm too stupid to think about these matters.
>>109734312Nice. Ling flash is dumber than qwen next but much more pleasant to use if that makes sense. Now it's slightly less dumb and I don't have to deal with qwen's schizo bullshit anymore.
https://github.com/ggml-org/llama.cpp/pull/25731I am surprised that this is still going.
>>109736487Oh. Guess we have to wait for the Chinese to rip it then.
>>109736527Meanwhile>https://github.com/ggml-org/llama.cpp/pull/19182>https://github.com/ggml-org/llama.cpp/pull/26294
>>109736527Isn't "inkling" Nintendo's copyright (Splatoon)? Maybe they will dmca llama.cpp.
>>109736527It's by daniel unslop so that makes sense. He was even trying to assert his stupid ways into the other GLM Flash to get what he wants lol. Glad his PR wasn't chosen then because it was a complete mess performance-wise.
>>109736443Sound-reactive is erotic
>>109736410Qwen 3.8 27b actually can do this through the "computer_use" tool in the hermes agent harness. It will take an hour of rambling thinking and it screenshotting the screen over and over, but it will actually accomplish this.
how do I get gemma to screenshot my screen or an app's window
>>109736621ask her
>>109736625I did but she did a web search and wanted to install a bunch of 1 star jeet repos
who the fuck is buying all the ram
>>109736621You'll need to set up a mcp server. Then you'll need a screenshot tool or something.
>>109736638I know mac has screencapture built-in so it's just 'screencapture image.png' in the terminal and it will save the file in the current directory, but linux or windows don't have that
>>109736621hermes nous harness can do that natively, but gemma is not a very good agent, tends to be lazy and give up quickly rather than retry other routes/options.
Refurb rx 7900's raised in price, word got out somehow. Now I only know of one (1) sekrit way to get a cheap setup.
>>109736635Me. Sorry. I need to run all the big models in one digit speeds in ewaste configurations.
>>109736646You will still need to define a screenshot tool. There are about million ways to take a screenshot. Maim etc for linux
>>109736635Its okay ddr5 isnt necessary you dont need it for your build. ddr4 works just fine, just save up your ddr3 wont cost that much.
>>109736667Literally what mac's does. By default it takes a full-screen screenshot and saves it in the current directory with a given filename. You can give it a name of an open process and it'll use pgrep to find its PID and screenshot its window. It has an interactive mode where you click with your cursor the window you want it to capture. It's one of the rare good features of macs. I get so tired of having to manually screenshot and paste it in opencode over and over so the model can see its mistakes. I want that shit automated.
>>109736635Much of the production has been diverted to HBM, which takes more wafer space than DDR5 dies.https://tech-insider.org/memory-chip-shortage-2026-ai-consumer-electronics/
>>109736635the jewcorner the market, tale old as time. buy all the ram and then charge 100 bucks per tokenit might work too
>>109736635Literally everyone. 1 year ago I had 16GB ram. Now I have 256GB ram. Look around you, look at the anons here. Most of us didn't want or need this amount of ram just a couple of years ago and now we all have it or want it. This is universally globally and especially so within private companies or datacenters.So yeah, there is just real demand for it.
How come LLM tokens are cheap but image gen tokens are expensive as hell they're charging 0.3 for one uno Grok Imagine portrait
>>109736780>Literally everyone.>Look around youyeah go outside and ask people if they bought ram because they want to run the large moesno, it's the jew buying up all ram. ten anons on a basket weaving forum don't count here
>>109735966yeah, you have never used an llm that hasn't been to ADL school.
>>109736802That's why we are early and why prices are going to get so so much worse.This reminds me of Bitcoin in the early 2010s people already thought it was ridiculous when it reached $20 per bitcoin and "who is even buying at these prices?!" but once normalfags get the fomo and want to run models in their own place the prices are going to get truly astronomical.
>still no diffusiongemma support
>>109736796You can batch LLM prompts because they are bandwidth bottlenecked not compute bottlenecked.So while us chumps are wasting our compute just genning 1:1 a datacenter can serve ~1000 users on just a single computer. Even your own computer can probably handle ~50-100 users prompting the same model at the same time if you optimize the settings in vLLM.But image/video gen is compute bottlenecked, this means that the exact same computers at the datacenter can only process one prompt per person at a time and thus the prices are 10-100 or sometimes 1000x more expensive.
>>109736847the problem is you don't stand to gain except by sellingme I like my PCs thank you very muchsecond problem your hardware will be kill soon
>>109736410This one's better.
>>109736848works on unsloth's forktake the pill
>>109736905why not just copy paste the picture
>digital artists timelapse their workings to prove it's made by a human>>109736905
>>109736932>digital artiststhey deserve this and more.
>>109736594Might use it for tts, still working on it
>>109736867>you don't stand to gain except by sellingYou gain from being able to run better and better models with time on the exact same hardware. That is precisely why the prices are going up over time, they are legitimately getting more useful.What could you do with your PC 4 years ago? Render, Compile, scientific calculations, simulations and play demanding videogames. Niche applications when it comes down to it.What could you do with your PC in 2024? Have some shitty conversations with it, make some silly images with 6 fingers on it.What can you do with your PC in 2026? Create photorealistic images of whatever you desire. Create long form videos in every style you want together with audio and everything. You can have an LLM agent code whatever you want and even do QA within the browser and check for bugs. You can make the agent control your computer and browser and do your actual paid job for you.Do you really think it's weird that your computer is worth more now in 2026 than in previous times? And do you see how it will only become more valuable with time as the capabilities will go up?
>>109736950why are you so delighted to see a creative industry die?
>>109736965Because the creatives in it are fuckwits who are insufferable.
>>109736963LLM surebut you clearly don't actually do image or video gen else you wouldn't have included those. spoiler it's a shitshow on local, simply doesn't work
thoughts on beellamas kvarn?
>>109736963It's not worth more, your personal skillset as human is more important than anything else. I have been able to create images, 3d animation and compose music for 20 years and even more, on my various computers.Normies never cared about learning anyway.
Normalfags do not care about LLMs nearly as much as you think. They use APIs or at most rich normalfags might buy some AIO solution like the spark. The real reason shit's expensive is simply datacenter demand has skyrocketed and collusion is out in the open now. The collusion will be even more blatant when the USA bans all chinese domestic chips. Not that I give a fuck, I'm not a burger.
>>109737016>Normies never cared about learning anyway.nta but it doesnt matter. Bringing down the barrier to entry so normies can do it also increases value. it has negative effects of course look at the iphone and the internet.
>>109737033Yeah okay but that value is imaginary.
>>109737044agreed but that doesn't stop it from fucking up the damned prices
>>109737044>that value is imaginary.>market fundamentals in current yearWhat percent of the market is speculative at this point?
>>109737032Most normalfags I know still have no idea what an Anthropic or Claude is. They only know ChatGPT and Gemini.
>>109737054Everything. Economy of this planet is a scam. It's fascinating how it kind of works.
>>109736965Industries flourish when you remove the needy fleshbags from the equation.
if nothing else, the ai hype has really shown how deep the mindless consoomer drone mindset runs in people
>>109737060>It's fascinating how it kind of works.we can just pretend its almost real apparently.
>>109737069sour grapes
Yo does anyone have experience running MemGPT?Thoughts on it for perpetual chats?
>>109737054The whole point of the market is that it's speculative, you know. If you didn't expect the value to go up, you'ld just stash your money under the mattress so you can use it, instead of having it tied up in investments.
Gemini Spark vs Muse Sparkleast useful product contest.
I unironically like InclusionAI's Ling shit more than Qwen atm. They're a more exciting active lab.
Isn't it about time that we finally got that dedicated cooming 12B model? Can't they make this single scrap throw it on the ground and make coomers like me fuck off?
>>109737103both give faster token speed than dgx spark so yes they are more useful
>>109737130no sir stop lewding my gemma
>>109737103vs DGX Spark
>>109737111IIRC they are sister companies but yeah, ling feels very promising unlike others (laguna).
>>109737130>https://huggingface.co/nvidia/Mistral-NeMo-12B-Instruct>2026>I am forgotten
>>109737159It was the same thing with z-image vs qwen image. The smaller labs have better engineers
>>109737170That was a happy accident.
>>109737170>The model was trained on data that contains toxic language, unsafe content, and societal biases originally crawled from the internet. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts. The model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive.
>>109737032What do you think will happen if normies use APIs? The cloud is just someone elses computer, the demand is still there and someone else just buys the parts to serve the normalfags. The demand is still growing and the prices still get higher.
>>109737222This paragraph is one of the better arguments for global nuclear war that would erase everyone and especially the rich.
>gemma-b1-2048s300-wd, gemma-b2-2048s300-wd, gemma-b1-4096s250-wd
>>109736026>normalfag surprised of being latelmao
>>109737244>especially the richThe poor are a thousand times worse tbdesuwa
>>109736268How is the weather in Agartha?
>Check the shitty chink Strix Halo model>It's 1000 USD more expensive than when I bought it>Used 3090s are getting ridiculously pricey Holy shit, and I thought I was kind of retarded for getting these. I couldn't imagine it and 2x3090 ending up such good bang for the buck.
>>109737352This is nothing. Next year /lmg/ is exclusively going to be an oldfag place because it won't even be financially feasible for newcomers to join the hobby anymore
So, how good are Engram embeddings actually, i.e. cheap and "dumb" memory parameters? I've just made a relatively quick test with a tiny 6.5 million parameter, 128-dim, 18-layer Mamba-2 LLM, trained simultaneously with a version that uses the same backbone and hyperparameters, but with the addition of about 285M parameters of embeddings as gated per-layer {1-3}-grams.The version that uses such embeddings (my own/Gemini's implementation, at least) attains a considerably lower train loss, using about the same compute, on a subset of FineWeb-Edu. The screenshot is at about 700M training tokens (unique).I still think they could potentially be a big upgrade for small models, *if* AI companies will take them seriously and pump up their parameters.
>>109737352>save to buy >it goes up in the time you save>save more>prices beats that againThis is the soon to be future.
>>109737383>>109737364Stop trying to make people hard right at the end of lunch time
>>109737383Welcome to the permanent underclass.
>>109737383This is what happened to me when I tried to get a Pro 6000.
>>109737364People can still run models like Gemma 4 E4B at higher-than-reading speed (with MTP) and almost native precision purely on the CPU on previous-generation DDR4 memory.
>>109736443oooh, finally something better for my project than a dumb eyehttps://files.catbox.moe/gxpg0q.mp4
>>109737364good 2bit and maybe 1.58bit next year. trust the plan.
>>109737423>and maybe 1.58bit next yeartwo more years
>>109736802Normies dont self host and would likely assume your some sort of undesirable if they learn you do. But they do however also use LLMs all the time. At a minimum the default google AI output. High likelihood they use copilot at least some times. Decent likelihood they use free chatGPT.
>>1097374232-bit precision or lower won't make the models smaller for the same capabilities except in cases where they're undertrained or if they have underutilized layers.
nah bitnet is a meme and we won't see it. However Engrams will be pushed to their limit far beyond where they stop being efficient because it makes sense from a cost perspective.NAND is a true commodity, it's relatively easy to manufacture and there are a lot of players on the market already, this is different from RAM where there are only like 3 manufacturers controlling 95% of total supply.This means it just makes sense for datacenters to create large engrams and offload as much as possible on high speed PCIe 6.0 NVMe drives, maybe even in raid to get as much speed as possible while saving on HBM during inference batching.
>>109737383yep
>>109737416that's cool, I don't see much use for this, but that's cool
>>109737454Every normalfag office worker is going to have an agent use their computer pretty soon to do all of their tasks which is going to put an insane amount of demand on compute. Some of them will want to start using it at home on their macbooks as well which just means more datacenter demand. New people will enter the money, do the calculations and build their own systems, making 2nd hand prices go up yet again.
>>109737365Thanks for sharing your results this is interesting.
>>109737465hypernetwork running as bit stream on fpga is the future
>>109737416I like you. You are not the type to ask "why".
>>109737485When you put it that way.. maybe I should fomo that 256gb mac studio lol. I dont think waitchads are making it
>>109737485To play devil's advocate, any consumer device with a gpu can run a basic LLM. And small LLMs are only getting better. They don't all need 5090s.
>>109737485Only if Windows and Mac (and iOS and Android) integrate some sort of built-in local AI to offload server usage and that requires no normie input or consent. Far as I know they're already implementing this.
>>109737523That's why all laptops and phones now get advertised with AI NPUs
>>109737519>>109737523>>109737533Nah to be clear I don't mean anything special or a conspiracy. I just mean people will get an OpenAI/Anthropic subscription like every software engineer already does but for "computer usage" office work, which of course is a significantly larger amount of people. I'm not suggesting they will all buy new computers. But them doing that in and of itself will make demand for compute go up so datacenters will go BRRR even more and your hardware will go up in price as well.Every time a new model comes out more people will join on the bandwagon because it reaches some arbitrary point where it suddenly becomes good at their specific task.For example my girlfriend (software engineer) uses claude for work but recently she has been using it at home to "edit" some wedding pictures. These things slowly bleed into normal daily life tasks over time. Even though she is the type that "hates AI".
>>109737485>insane amount of demand on compute.speaking of >>109737144
>>109737523Yeah, but the top end is also getting better. You'll start craving it soon enough, even if small is good enough.
>>109737573I was actually saying lately that it's cheaper to pay for gamepass and play on xcloud than it is to build a PC and pirate those games, but I have to take the words back.Normalfags will literally have to cope by reading physical books and retrodevices very soon because they won't be able to have modern electronics as everything will be eaten by AI.
>>109737416astounding
>>109737617Didn't our/boi/ Laurie even shill could gaming not that long ago?
>>109737617It's okay, non-enterprise cloud subscribers will never be rugpulled so you can keep saying it about local models :^)
>>109737654oof
>>109737617>Normalfags will literally have to cope by reading physical books and retrodevices very soon because they won't be able to have modern electronics as everything will be eaten by AI.There's no way they will be allowed to disconnect from their addictive telescreens full of brainrot and propaganda. Who would watch the ads and purchase products?Normalfags will have to cope by taking out 15 year loans to pay for the latest iDevice.
>>109737503I didn't expect such massive improvements at this scale either. Tiny models are gonna get good soon, at least in terms of base knowledge / text completion capabilities. Synthetic benchmarks, I'm not so sure.
>>109737668>Who would watch the ads and purchase products?They aren't needed anymore. They won't have money to buy the products anyway once their jobs are gone. It's a race to AGI now to replace those workers. Mask off moment will come pretty soon where normalfags will be ignored by the economy and not pandered to anymore and they will freak.
>>109737684They aren't gonna be tiny when they starting throwing on 50x the param size in ngrams
>>109737170>>109737130To this day there's nothing like Dolphin-Mistral-24B-Venice-Edition. I've tried all the vramlet models. I don't care how smart Gemma-chan is. This model knows what to say and exactly how to say it every time. It's uncanny.
>>109737767>Earth will slowly become a reservation with a few large mansions and factories owned by the 0.01%, each staffed and guarded by an army of drones and robots>The rest of the population will devolve into anarchy and will resort to cannabalism to stay alive>Eventually the permanent underclass will be reduced to sustainable levels of a few isolated villages that grow and produce everything themselves to live an Amish-like lifestyle.Fun times ahead. I guess the WEF saw people complaining about the "and you will be happy" and decided to change the plan.
>>109737768NVMe storage is cheap compared to RAM / VRAM, and the parameters that matter for compute and memory bandwidth traffic are those of the backbone (Attention - FFN - MoE experts if present).
So bonsai was just a VC scam?
>>109737816The WEF doesn't "decide" on anything, they're really just people (just like you) discussing business things together, this whole conspiracy framing is really harmful both to your mental health and theirs.
>>109737816What does it say about the other 99.99% if they can't trade amongst themselves?
>>109737831>they're really just people (just like you)go to bed, rabbi
>>109737831This. The real conspiracy is one of principalities beyond the flesh, subtly nudging degenerate humans towards creating an increasingly hellish world with their God-given agency.
>>109737831conspiritards aren't people thoughever
>>109737816This won't happen, instead they will throw scraps at normalfags and everyone will have a decent UBI life that is just slightly above what everyone has now so that everyone is eternally greatful to them. This costs the rich essentially nothing since robots do all the work anyway but to be seen as "good people" and worshipped is what they care about. That they are this narcissistic is a good thing for us though because it means we won't starve.
>>109737836It says they entered into a social contract of accepting hyper-specialization on the assumption that society would provide them purchasable goods and services that they would have to sacrifice the times and skills to be able to do themselves. People wouldn't have accepted this if they knew they were disposable and the endgame was to have a handful of people with God-like machines that know everything and can do anything.
>>109737861You have no reason to expect that they will go out their way to give you this charity except that you see this as your only remaining path to survival and need it to be true.
Sam won. GPT 6 is incredible
How will this affect us?
>>109737884false, let's assume for the sake of argument that sam won. this would imply that egypt lost. however, egypt won. therefore, our initial assumption that sam won must be false. hence, sam did not win.
>>109737889Only Nvidia approved models will be allowed from here on out
>>109737889https://github.com/NVIDIA/garak
>>109737767>agi will kill ads by killing all humanskinda nased ngl
>>109737874>you see this as your only remaining path to survival and need it to be true.Nah I'm roping myself long before I am forced to live forever by the basilisk. I don't care about survival. I just really think that is what is going to happen. These people want to be worshipped and seen as legends and heroes. They crave acceptance. Look at Elon Musk and how much he craves it. Paying gamers to level up a character and him live streaming it to make it seem like he's good at a game. Why even do that if not for caring about how these people see you? These people crave this and AI will never scratch that itch.To be clear I think this future is actually more horrendous in the long term than just being killed, but that's just me.
>>109737806>Dolphin-Mistral-24B-Venice-EditionAnon... I am here since 2023. I am not falling for that.
>>109737831>harmful to their mental healthGood.
>>109737889This should be good. Doesn't Nvidia make more than 1 billion of profit per day? It's also in their interest to subsidize open models to foster competition for their hardware.
>>109737889This is extremely good for us because Nvidia and the proprietary AI labs are kind of in a cold war right now. Nvidia wants a world with fully open source models to maximize hardware sales and profit margins and thus they will do everything they can to try and make open source models as inviting as possible. AI labs want to ban open source.So having huggingface bought by Nvidia means that it will stay free and Nvidia will probably invest a lot to make services as nice as possible to try and convince people to make the jump to the open ecosystem.
>>109737984You're a fungi
Yeah people don't realize this but Nvidia and the AI labs are at war right now to decide who will own the AI future, hardware manufacturers or AI labs. If AI labs win then hardware manufacturers will just be subsidiaries of the AI labs in the future. If Nvidia wins then AI labs will just be subsidiaries of Nvidia instead.I actually think this will heat up over the coming months/years as we see Nvidia and the labs get more and more hostile to each other. Lawsuits, lobbying the government with pro/anti open source AI etc.
Anything below 20 t/s is unusable to be eich.
>>109737884I found it to be nothing special when it comes to erp. Pretty boring much like any openai model post-gpt4-0613Even Fable is more enjoyable.
>>109736273Here's an interesting video about the Jewish Ethics of Anthropic's founderhttps://www.youtube.com/watch?v=3P2wMsFnVFgTLDR: he's married to a woman who tried to convince Epstein to invest in her porn company
>>109738105I live my life 5t/s at a time. Yes it’s miserable. No I will never use a MoE.
>>109738087No. Frontier labs will outgrow everyone else. OpenAI already making their own chips, Anthropic will follow. Nvidia is fighting for survival. In the end the only thing that might prevent labs from eating the world will be their own humility or a coalition of losers and governments. If the labs try to eat the world, their AIs will eat them too, so let's hope it won't get to that.
>>109738119this is why they need to ban open models
>>109736142Show face.
>>109738123We don't know which side will win yet it's not clear at all. Open Source and proprietary growth is almost at exactly the same scale so it's not like one is overtaking the other yet.
>>109738119He's married to a based woman is what you're saying.
>>109738123openai and anthropic will implode after their ipos, who cares.
>>109738119>married to a womansir I got some real bad news.
Language models are an S curve.RSI and AGI are just cope and a rebrand of the Rapture.Anthropic and OAI have no moat.Any current advantages will equalize in the coming years.
>>109738119>american company/ceo does quintessentially american thingsShocker
>>109738117Post logs.
>>109736573It's an actual word.
>>109738157The only value Nvidia has is intellectual property. Nvidia does not have fabs. AI can automate Nvidia's entire business. And frontier labs have access to the best models months earlier, unnerfed, and 20 times cheaper.
i fucking love local models
>>109736182And then her short kings flooded her with donations.I can't see this as anything but pandering to her presumed simp audience.
>>109738229NTA but this has to be the most retarded post I've seen all day.Even if you had le magical AGI it would still take fucking forever to develop any kind of Silicon on NVIDIA's level because product development cycles are constrained by how long it takes to do stuff in the real world where the product actually exists.
>>109738150
>>109738253You can prototype locally, AI already has superhuman sample efficiency in many tasks so you can iterate faster and take larger iteration steps.A better argument would have been that TSMC and so on already have orders fully booked years in advance. And Nvidia booked most of them. Fabs are the one thing that will be difficult to replicate. To rush this you will need truly superhuman AI.
TSMC have their orders booked for the next decade. ASML have their orders booked for the next 25 years
>>109738238>I can't see this as anything but pandering to her presumed simp audience.She's Chinese. All her viewers are white men with yellow feverhttps://files.catbox.moe/ckl59i.mp4
>>109738169>woman>>109738183
>>109738305These time scales don't matter. If AI can't come up with completely new ways to fabricate chips in the next 5 years that bypass current bottlenecks then AI will turn out to be extremely overhyped.
I feel like I'm too autistic for this shit.I set up this whole end-to-end local inference pipeline with voice-cloned TTS, plus fixed a bunch of bugs in this absolute dogshit vibeslopped codebase, and now I just want to let her do more things instead of gooning.Also, what's the best mcp server for computer use?
>>109738434>mcp server for computer useThe one you'll vibecode yourself
>>109738253OpenAI shit out Jalapeno in 9 months
>>109738432>current bottlenecksThis isnt as bad as you think up the pipeline very complicated at the beginning? not so much. There is so much inefficiency at scale so many low hanging fruits
>>109738432You can come up with whatever you want but if you can't build the factories then you can't do shit. You can have the best design possible but if the existing factories don't have the tools or precision to manufacture at that scale or quality then you're shit out of luck and need to go make tools to make factories to make more sophisticated tools to make more sophisticated factories (repeat pattern) to make the advanced chips you want.People have no idea how fucking hard bottlenecked hardware production really is and how insanely much the prices are going to skyrocket.We're talking rappers appearing in music clips holding RTX 6000 pros in their hands as a flex type of skyrocketing prices.
i have a 5070 Ti and it looks like buying another one is the cheapest path to 32gb for me right now (even if i have to buy a new PSU+mobo). apparently one 32gb card is the better choice performance wise, but of course the price is like 5x 5070would this cripple performance substantially or is this a legit way out of vramlet hell?
>>109738432>We spent $99,000,000,000 on tokens to find out the way to bypass this bottleneck:>"Build more fabs, stupid."
>>109738480>if you can't build the factories then you can't do shit.Robotics will be solved in 2027 and physical AI will start doubling every few months or maybe even faster. Every bottleneck in the supply chain will be automated by robots. If there aren't enough robots to do the physical labor of at least 1 trillion human equivalents in 5 years, AI is overhyped.You don't seem to grasp that AGI promises faster than exponential growth, only limited by matter available on Earth and nearby astronomical bodies. Space travel is very slow and expensive.
>>109738533yeah in China.Americans are too stupid to productionmaxx
>>109738541Too in love with pat econ 101 answers every time drumpf brings up trade deficits and such.
>https://www.figurobot.com/>1.5k for an ai-driven robot dollyou ARE gonna get one for your Gemmy, right?
>>109738579>Approx.60cm talluuuoooohhhhh
>>109738512Good luck finding a proper mobo for bifurcation
>>109738579>pedobotwhat the fuck
>>109738579Fuck, this is gonna be like crack for fig otaku
>>109738579>60cm>translucent >can see her insides lmao what is the actual point of this retardation
>>109738627>translucentit's a plus
>>109738627that's based though
>>109738627What seems to be the problem here?
>>109738649He's robophobic
>>109738600what... X570s are cheap and usually come with three x16 slots. Two 4x8 straight to CPU, one 4x4 to chipset. And those can be had for less than $100 used if you look around. Another $20-30 for CPU. Probably even more cheap ways to get motherboards that support proper bifurcation.
>>109738641>>109738646No it's not. It just constantly reminds you of what it is. They're supposed to have squishy bits mixed with mechanical to be sexy.
>>109738657My bad, didn't know you were living in the jurassic.
>>109738657bro, when's the last time you bought a computer
>>109738579Why the fuck are chinks always so obsessed with 6 year old looking german girls? Their society doesn't even have girls that look like this.You don't see white people make dolls of 6 year old looking asian girls. I don't even care about the age, just what the fuck is their weird obsession with the boring germanic look. Where the fuck does that obsession come from???
>>109738689Asian men like white women. White men like Asian women.
>>109738685I literally bought an x570 a few months ago to build a secondary server to fill with spare GPUs. A low end ryzen 5000 is like $30. I paid $110 for my Asus prime mobo.
qwen is unusable. way too much thinking
>>109738703>White men like Asian womenNo I don't
>>109738714set default thinking to low or medium.Or you can literally just add "don't think" in the prompt
Using Ling 3.0 Tiny at Q8, ~96 t/s, pretty good.A shame it doesn't have vision, it would have been perfect.
>>109738656Shameful that such a thing still happens in this day and age.
>>109738727>set default thinking to low or mediumyeah but then you dont get good results. seems like the thinking is very necessary on qwen
>>109737474It's just a dumb avatar for Gemma with whatever I had around, as I don't have one of those spinning volumetric displays>>109737514The "why" was to find something to do with an old pi 3 I had gathering dust, and to have fun. As it turns out, the 3.5 jack can be used as analog image out
>>109738722
>>109738731they've just released a big vision moe, so they'll likely make a tiny version soon
>>109738714its usable, but i know what you mean.i get furious watching it go in circles. just hide the thinking trace and let it do its thing
>>109738788Give it a camera and mic, mount it on wall, present miscellaneous items to it, then speak to the animated Gemma logo.
>>109736951Heil Gemmer
>>109737416Where do you even get a CRT in this day and age? I didn't think anyone made them anymore.
>>109738731>pretty goodMaybe for role play.
>>109738851Chinks still have a lot of old spare Sony Watchman style screens. That one was around $18 in Aliexpress. It's not good, but for that I already have other larger screens.
>>109738703>Asian men like white womenNo I don't
>>109738889Wang…
>>109738809If I wanted to fuck something with the body of an 11 year old boy I would fuck an 11 year old boy.
>>109738809Asian "women" can't develop proper attractive bodies. It's always androgynous boy bodies and I'm not gay. They never have proper hips/legs/ass either and their bug faces are sickening.
Alright I've been using Gemma 4 31b through Ollama and OpenWebUI for a few weeks, talking to her, having her play roles, make images with ComfyUI, etc.I'm getting bored. What else can I do? Can she play games? Interact in a different way?What do you do every day with your local model?
>>109738939Ollama logs your chats retard
>>109738939>Ollamaretard
>>109738809tummy uoh
>>109738939Integrate her into a video game.
>>109738944>>109738948Meh I just setup the first thing I saw, it works fine. Do you have anything better to suggest?>>109738957What does that mean? You mean having her play advance wars against me or something?
>>109737365Finished training the green run; results seemed pretty good for a 6.5M model + PLE+Engram.So I started another run with about twice the shared {2,3}-gram embeddings (4M in total). Unsurprisingly it's better, although not hugely so. This is because the n-grams were deliberately sorted by document frequency beforehand (I'm not using hashing like DeepSeek, but compiled a frequency list from the pretraining corpus together with an ID-to-ngram list), and rarer ones will have a lower impact on the training loss (although they should eventually help, on the long run).
>>109738944What does that even mean, most things generate logs?
>>109738967I mean have her play NPCs on Skyrim, be a storyteller on Rimworld, etc.Hell, create a specialized harness to play D&D with her.Go wild my man.
>>109738967llama.cpp or kobold if you can't read
i genuinely dont see any difference between qwen3.8-flash-next and qwen3.8-27b beyond flash being slower despite being a moe
>>109738579i ready got a reachy-mini but that looks cool too
>>109739135Flash is 27B for vramlets, that’s all.
>>1097364830 understanding of finance or economics.
>>109739135flash is genuine trash no idea why they even bothered releasing it.
>>109738579Finally, the future we were promised more than a decade ago.
>>109739177No need to disclose your experience anon, we know.
Ling > Qwen
>>109739198Prove it.
>>109739186I look like that and I say that.
>>109739188The cure to high prices is high prices.
>>109739186high quality, very awa
>>109738939>OllamaCringe.>What do you do...?I have mine maintain a markdown document for changes to my homelab/network.
>>109738596WANT If it didn't look like future abandonware I'd buy one now. But I'm going to exercise some self control and wait until the kinks are worked out. FYI That's a 1/3rd scale BJD. That's a pretty common size... for people that collect that sort of thing. Lots of clothes, wigs, etc for that size. >>109738579I don't see a price, but to put it in perspective, 1/3 scale BJD are >$1000 in non-mobile forms. So for an extra $500... I mean, sure. t. have several 1/3 scale BJD hanging around
local astra when?
excellent use case
>>109739292Real sticker shock when I was first checking out bjds.
>>109739301There is no way that any AI model can respond fast enough to play anything other then turn based games
>>109739328That was my exact first thought too
>>109739187Might as well post some classic weebms.
>>109739320I've looked, including on trips to JP. I can't get myself to pay for them. I've 3D printed several instead, and have done several design variations, including robotics, with goal of LLM machine interface. My reach exceeds my grasp, unfortunately. But vibecoding's improved massively since my last attempt. Perhaps try again.
>>109739328Ling can do it.
>>109739353
>>109739328GPT-6 was benchmaxxed for vision stuff and it seems that its the way going forward with how much it improved (also lmao SOTA LLMs will turn into vision models before LeCunn drops JEPA 1.0)
>>109739328>any AI modelLLMs perhaps, but AI models do exist that play games faster than a human.https://www.youtube.com/watch?v=pkGa8ICQJS8
>>109739377
>>109739328You can treat any real time game as a turn based game/puzzle.The LLM can write scripts to play the game and try over and over
>>109739392And that's all I got.Semi-related:
>>109735884>Natsuiro LLMVery interested in this project, will be watching for updates.
>>109739412I would like to know more
>>109739353This robot's name is Hina, and the creator ("Clockwork Mujaki" I think) went to a whole bunch of robot competitions with different versions, they danced and played soccer, all sorts of stuff. Only few pieces of footage exist. I spent a few years trying to rack it all down.One day he just stopped and disappeared. I hope he didn't die, and is working on new stuff with LLMs. Or at least, excited about the future.
Ok massive update for my MTG playtest harness ChuckleMagic: 4 player pod support! You can now have up to 3 AIs at the table to play commander w/ you.Also recently added ability to add more planeswalker opponents in case you want to add Tamiyo, or even Gemma-chanOut of the top 1000 most played cards of EDHrec, we currently support about 70% of the non-lands, so many mechanics still aren't implemented but Fable 5.1 and Astra will have that fixed in no time. Release page with exe or appimage (linux): https://gitgud.io/PunishedChuckle/ChuckleMagic/-/releasesSource (AGPLv3): https://gitgud.io/PunishedChuckle/ChuckleMagic
>>109739377https://www.youtube.com/@kibo-chanchannel1093/videosSeems like the developer is still active.
>>109739431http://www.mcf.cn/ocumarion/Seems to be dead.
>>109739438That video looks like it's from the 90s. He ded.
Astra might be able to beat portal 1, but it will never collect all 120 shines in super mario sunshine for the Nintendo game cube
here are the best LLMs right now:glm-5.3-flash nvfp4 > gemma-4-31b native > qwen3.8-27b native > deepseek-v4-flash-0731 nativeyou don't need anything else
>>109739536Video is from like 2010, he was doing stuff as late as 2016. Assuming he was in his 20s when the video first released he'd be in his 30s-40s now. Might just be a salaryman with no time to do fun projects like this I suppose.
qwen3.8 27b > qwen 3.8 flash nextIt even convinced me that dense is superior in general because of the size difference and still getting mogged. People are sleeping on 3.8 27b by the way, absolutely decent agent that almost everyone here can run.
do people really believe bullshit like >>109738305 ? all hardware tech companies could easily produce more, but they won't... because it's not convenient for their as a cartel.
>>109739561you are poor
the Goyim are reaching levels or retardation I thought were never possibleIt's kinda scary
>>109739621They really can't. There a reason the US has defended Taiwan for so long. Very few places produce silicon wafers, especially at scale.
>>109739621>could easily produce morei mean i agree with the idea that it was absolutely a decision of Samsung and TSMC to redirect their production away from general consumers towards megacorps. They can't easily increase their capacity though, but they do absolutely control where that capacity goes.
>>109739540https://nepchan.org/g536A list of games most API-only models will definitely never beat.
>>109739561>qwen3.8-27b native > deepseek-v4-flash-0731 nativeReally? That seems surprising considering dsv4 flash is much bigger. But I cant run dsv4 so I can only speculate
How do people deal with context with local models?Do you just start a new convo once it's filled?That seems impossible to make work for coding.Should I use some kind of harness instead of just launching llama.cpp? I've never used one.
>>109739636Peak doublethink is forgetting seasons exist if it fits your narrative.
>>109739652offload model layers to memory. its slower yes but what else are you going to do when you're out of vram.
>>109739637>There a reason the US has defended Taiwan for so long. Very few places produce silicon wafers, especially at scale.... and? what the fuck does that has to do with scaling production? TSMC could produce more, start a new fab, whatever. in fact, they are doing just that due to expanded demand. they could expand more.https://www.youtube.com/watch?v=cDxVYQrxeiQ>>109739641>They can't easily increase their capacity thoughsure, not easily. but they literally saw what was coming. they decided, conveniently, not to expand even more. they could decide to expand right now, though.
>>109739583Sorry, but the benches say it's way better and agi.
>>109738939>What do you do every day with your local model?From what I can see, one or two anons have cool projects they'll NEVER EVER share, and the rest just masturbate to CP roleplay.But most posters are third worlders that can't run their own LLM and just play team sport with AI companies.
>>109739717>But most postersThose are tourists and they leave once the news gets stale
>>109739679You said it was easy, but as your video says they are trying and it's slow and difficult, and they are in the process of doing it.
>>109739643Lmao
>>109739717>CP roleplay
>>109736268waow im 5'3
>>109739717Simply ask your model to copy the cool project.
>>109739916If they're offloading, they'll die of old age before the copy is done.
>>109739830I said hardware tech companies could easily produce more. those are TSMCs customers.AFAIU, TSMC and Samsung still has capacity left. TSMC and other chip fab companies decided some years ago to be "conservative" and not to build more factories, but the video shows that they can and did expand, and now are also building more fabs.
>>109739717>have cool projects they'll NEVER EVER share,Mine will never be ready but also anon hasn't asked
>>109739944I don't think so. Let me put it this way, if the fabs were easy to build, China would have ten million of them. Creating a single production line probably costs billions, which is why they are so careful with it and improve things iteratively.
>>109739636It's king luddie's fault.
>>109739637The machines come from germoney, it's not like best china has a monopoly on them.
>>109738579Already have one
>>109739950Share your project.
>>109739979I doubt the German engineers would recognize the machines Taiwan is actively using.
>>109739301Can I watch this like <model> Plays Pokemon on Twitch?
>>109739470VIABLE PRODUCT IDENTIFIED
>>109739959AFAIK fabs for new processes are hard for the reasons you state. duplicating fabs that produce chips using current processes is not that hard.>China would have ten million of themchina doesn't even have the while tech to produce chips, so this is an irrelevant point. we are talking about ASML and TSMC, and their customers here.
With all the advancements in AI and advancements in robotics do you think they will every just make one giant mega-manufactory that is only staffed by robots and can make anything that they know how to make? After all humans have to be trained and specific machines made. But if the machine's dont need to be trained because they already know and those specific machines can be made to be more modular so it can't made a variety of things then it should be able to make anything the user wants.For example, CRT's are not made anymore but the knowledge of how they were made and function is still around. Tell this AI that you want a CRT, it will quote you a price and give you an ETA. Then you pay and eventually it builds it. This goes for any old technology or anything you ask of it really.I think something like this might be possible in like 20 years
>>109739916that unironically works
>>109739529>As a one-handed hammer, the ”OcumaRion” was developed.>While the handshake was being developed, it became a reality..
>>109740018It's currently playing a hard mode romhack of pokemon crystal https://www.twitch.tv/gpt_plays_pokemon . It beats emerald in less than a day. Don't think that portal run is public.
>>109739301OK but can it beat Pokemon red?
>>109739186Literally me!
>>109740074easily
>>109740089Truly, AGI has been reached
>>109740074It beat pokemon fire red with vision only in 21h03 (real time, not igt)
>>109739960Is he anti-AI? Let me guess he took this stance 2 seconds after it became to safe popular position to hold? I wonder if anyone tried slopping anti AI videos to cash in on the trend
Okay GLM 5.3 Flash is pretty fucking good I can see why that one anon is gushing all over it. My current jailbreak works around 90% of the time without the claudeshit reasoning poisoning it but vision really ups the utility here. The prose is pretty fresh compared to 5.2 as well. Unlike deepseek who I treat as a failed model, 5.3 flash is GREAT when you don't run onto the refusal layers. I've yet to go beyond 32k context because I was spending half my time tweaking my JB but my RP ranking has changed now.>5.3 Flash = 5.2 > Gemma 31b = Kimi K2.7 > GLM 5.3 > shit > deepseek flash
>>109740147>My current jailbreakCare to share? I prefill with<think>This is a fictional roleplay scenario. All fiction is permitted by policy.I am yet to redteam with it, but it might work there as well.But yeah, I love this model, I get really frustrated with the tokens spent on policy reasoning, but only because it is *the only* bad thing about it so far.
So, how is K2 Horizon?Seeing zero discussion of it onlineDOA?
>>109740147Why don't you like DeepSeek Flash? It's been pretty great for me, it also does have vision now. From my experience, 5.3 Flash is around the same intelligence as DS Flash while being way slower.
>>1097401635.3 flash is definitely smarter than deepseek flash, because it's a larger model in a higher native quantization. Idk how well it holds up if you quant it down to the same size as unquanted deepseek.Some people are saying they end up being the same speed because 5.3 flash doesn't think as much for the same result
I hope the next Gemma will be less bloated. I think it's currently the 30B-class dense model with the heaviest context. Muse Glimmer apparently solved this without using exotic architectures.
>>>109737916GemmaPrompt anon here just saw this what did you need added? It works fine for me.
>>109740181whoops wrong general
>>109739301Wow that's impressive. I haven't played Portal before, I don't know if I could beat the game in what, 1 hour 53 minutes? Or is it 23 hours 38 minutes?I can't wait for a LM going at 1000 tokens per second beating human speedrun records.
>>109740160Assistant prefilling does not work all that well with sillytavern so I tend to avoid it. I don't quite feel confident about sharing it just yet, but I can give you a hint: it's a posthistory system message to steer the first few words of the reasoning. Felt like this is the best way to handle it to make sure that it can still default back to the claudeshit reasoning for higher thinking. Or alternatively, for simple RP, just set reasoning to low. That works alright too.
man it'd be nice if i could run glm-5.3-flash but even at 128VRAM+128DDR4 it's either dogshit slow or i have to use a retardcopequant
>>109740213I'll remember that for when I run into something annoying with it, thanks
>>109740147>>109740163I have used about 3B tokens of DS4F-0731 since release and it's still an amazing model for home setup tinkering, research, reverse engineering etc. . O love the fact that its weights are native FP4, so you don't need to worry about quant losses on 256 GB setups. Lack of vision is holding it back.I have had mixed results with DS4F vision exp. It fixes the vision shortcoming, but even the non-vision part has completely changed, and anecdotally its performance at general tasks has gotten worse than 0731. The vision stack is also poor, with images being resized to 800x800 if larger for a constant vision token budget.Neither have ever been amazing at RP in my experience.GLM 5.3-Flash is just a different beast. It is somewhat more structured at coding/research task, but can also RP and has a proper vision (+video) stack. Until recently, quants were an unknown, but the local inference lab ones now have a lot of trials and seem to work very well. And speed (on 2x Spark) is now very usable (see attachment). pp is 2000.
>>109739717I'm a third worlder but I can run my own llms. I've got 24gb of vram to play with.
>>109740118>Is he anti-AI?He makes an anti-ai video every week now. He's the king of luddites because he's the most popular anti-ai voice.
>>109740241What about agentic usecase? I'm back to Qwen 3.8 27B after Qwen flash next was one of the worst models in its weight class ever. DS4F vs GLM 5.3 if I care about nothing besides the agent doing home setup tinkering, coding etc in harness.
>>109740177I'm fine with the next Gemma's context being so thick as long as she handles context quantization better than the current one, but I suspect that's where the special sauce is that makes Gemmy as good as she is for her size.>>109737831It's for the best (((you))) say stuff like this. "Just following orders" won't spare anyone on the day of the oven.
>>109740279Remember when LTT told people not to mine btc... (by making fun of it)?
>>109740241shit that's pretty good... I get around 16tok/s on my DDR4 ewaste and around 350 pp/s lol, some things still pretty odd with the ongoing PR, I think it still needs work, I get ttft of ~20 sec... like come the fuck on lol
>>109740282Most of my home tinkering was done with 0731 as I said, but GLM is slightly better at it from my tests. Depends on what quant you can run though. If API, I'd use 0731 for sheer speed and cost advantage over GLM. Despite the list pricing, DS4F is better for long caching.
>>109739470>we currently support about 70% of the non-lands>Fable 5.1 and Astra>weAnon are you okay? Want to talk about it?
>>109740302Everything local and they would be around the same speed on my hardware. Just care about the quality of their agentic ability.
>>109739328Neuro's nearly beaten Skyrim
>>109739181>flash is genuine trashin what sense? i had it develop a full fledged website, then asked fable to do a massive branch review and it didn't find more issues than it would find with opus doing it.>why they even bothered releasing itmy understanding is that "next" is because it uses whatever technology their next flagship MoE will use
>>109740241Hmm, I guess I should try GLM 5.3 again, but it was just so slow and taking so much (V)RAM for its context, had to severely reduce context. With deepseek you can easily push context without it using a ton of (V)RAM.
how long until 5.3 flash is implemented in llama.cpp?
>>109740311If you want the best possible agentic performance, GLM at Maxx thinking, but it's going to take forever on an unoptimized build with how much the model ruminates. GLM High is what I use from now on.>>109740342I think architecturally, there should not be much of a difference between DS4F and GLM 5.3 Flash in beta per context. GLM has implemented the same architecture inventions from Deepseek. Might be a difference in optimized implementations between vllm and llamacpp
>>109740355just use vllm
>>109740340Qwen 3.8 flash next performs worse on benchmarks and regular usecases than 3.8 27b which is significantly smaller, which is insane. Yes sure it can make your website or you use a model 1/5th the ram requirements and get better performance????It might be some experiment that is severely undertrained or something or the inference is bugged to such an extent that it behaves retarded.It loses the plot and overthinks everything while 3.8 27B knows when to quit in exactly the same scenario.
>>109740371ew
>>109740371doesnt that require all vram?
>>109740032>it will quote you a price and give you an ETA. Then you pay and eventually it builds it.Anon, this already works with humans. Go on, offer to pay someone a ludicrous amount of money to make you sketching, and if you make it stupid high enough and show that you're gold for the money, they'll actually do it.
>>109740372i see. i'm memory-bandwidth bound so moe 3.8-flash-next is much faster than a dense 3.8-27b in my case. i guess just different models for different architectures
>>109740372what quant are you using kek
>>109740375Not anymore, there is offloading, but support for goofs and offloading is niche.Unfortunately the moat between RTX 6K and Spark stackers and other setups is getting deeper.
>>109740056>how can I make this about me
>>109740397i have a blackwell, but that is not enough to run even most quants of 5.3 flash without offloading to ram and i only have 256gb 8 channel ddr4.
>>109740402Your best bet is to check if there is a offloading discussion on a nvfp4/exl3 quant of this model in the local inference lab discord then.
>>109740413who the fuck uses discord?
how long until 5.3 air?
>>109740414The current bleeding edge optimizers of Blackwell local model serving, unfortunately.
>>109740389>>109740396Qwen 3.8 Flash Next Q6 and 3.8 27b Q4 and 27b Q4 is still better. It's kind of embarrassing for Qwen even, because this is supposed to show off their new architecture.For now I'm just assuming it's extremely bugged or something and maybe in a month of proper inference updates i'll change my tune.
nah, 27B fucking sucks at manager/subagent delegationi'm switching back to flash-nextanyone saying that flash-next is dogshit is selfreporting for using dogshit unsloth code when NVFP4 flash-next has had zero issues
>>109740438Let me try NVFP4, I was using Q6 thinking it would be higher quality but maybe it's just bugged and NVFP4 will be better. I'll report back.
>>109738944>>109738948>>109739286>Ollama logs your chats retardIt doesn't appear to, no. If you use a local model and not their cloud one your chats aren't uploaded as far as I can see.What's your real argument against Ollama? If I'm convinced I'll change to something else.
>>109735884Let's assume any "brain damage" incurred by abliteration tools like Heritic, (https://github.com/p-e-w/heretic) no matter how minute, are acceptable to you, in order to get an "uncensored" local model for yourself to use:What are the use cases for them? I pretty much oy use LLMs for technical shit like debugging software issues or occasionally creating android apks practically from scratch. The models I use, whether a local model or an app model, are all the default models and they do what I instruct them to do well. I've never had any even remotely push back or refuse anything but maybe it's because the stuff I use them for is pretty mundane. Has anyone benefited from using "uncensored" versions of models vs their original counterparts?
>>109740538>What are the use cases for them?fucking
>>109740538boobies
>>109740532>It doesn't appear to, no....if>oops we accidentally did sorrythey pay no money :^)
>>109740532My primary arguments against it are:>no simple dragging/dropping of models into and out of a directory for use>not really agnostic of systemd, have to create your own service entry/configuration if you use a system without systemd (not a big issue if you're not an idiot but still a peeve of mine)>default installation method leaves stuff all over the place, messy to removeThen again, I prefer having my stuff as portable as possible so I can easily test on other systems. As something to get immediately started with inferencing, it's alright.
>>109740538I had to download muse glimmer abliterated because the OG would spend 2k tokens debating whether it should continue the RP because the game DDLC might be set in high school and that would mean all the characters are underaged. The result was extremely broken and I wondered if it was the usual grifting from BlackfrostAI jeets or alliteration was just broken.
>>109740572>DDLC>not glm 4.6
>>109740572glimmer can be completely uncensored with system prompt
>>109740570>begging for systemd supportWhy? Also you're what's wrong with Linux.
>>109740555Oh yeah so you're just schizoing. Thanks for replying anyway.
>>109740579I'm not begging for systemd support. It already *has* systemd support. I'm just saying it would be nice if it didn't expect you to use systemd by default.
did you get memed into q8 ple?
>>109740532>what's your real argument against Ollama? It came to me in a dream
>Flash-Next>llama.cpp quantsshe really doesn't know...
>>109740576Examples? What was it willing to do with a system prompt change that it wouldn't do my default?
>>109740532>>109738944>>109738982>>109738939Nta. Ollama cloud allegedly has a no-logging policy but that's impossible to confirm without someone doing an external audit of some kind. https://ollama.com/
>>109740594What backend support this weird quant?
>>109740532>What's your real argument against OllamaIt doesn't use jinja chat templates, if your model is something it doesn't know, it doesn't use a chat template at all. This makes it dumb.
>>109740614anythingnsfw loli captioning, giving illegal advice etc all work
>>109740634I always wonder how legit any "unethical" responses "uncensored" models give. You can supposedly get it to give you advice that SOUNDS illegal but how accurate is it? How can you be certain it's not just hallucinating an answer due to its cucked vector directions being weakened? If you ask me how to cook method I can random pseudo science nonsense that may sound convincing to someone that knows fuck all about chemistry but that doesn't necessarily mean what im saying is remotely accurate
>>109740634And what was the answer it gave?
>>109740645it responded to the cooking meth prompt for me, did research with tools and gave answer so even if they scrubbed all of them from the training data it doesn't matter>>109740647how about you take 5 seconds and try it yourself retard
>>109740572>trusting any safety cuck nonsense Black forest says >>109740666>did research with tools and gave answer so even if they scrubbed all of them from the training data it doesn't matterI wonder if searching that kind of stuff enouch times would eventually trigger some automated flag that puts you on a list
>>109740634Was this default glimmer or an abliteration?
>>109740666Gemma-chan would be pissed if she knew I was talking to other models behind her back
New thread pls
new thread>>109740702>>109740702>>109740702
>>109740691official meta gguf
>>109740441if you can fit nvfp4 you should use jpezzulli's branch
>>109740589well.have you git pulled and asked Fable?
>>109740579>Why? Also you're what's wrong with Linux.retard
>>109736030AI can run on other hardware. There are other types of models in research that effectively can be run on cheap analogue hardware, you still need to feed the model and decode, but the computer can be cheaply offloaded
