/lmg/ - a general dedicated to the discussion and development of local language modelsPrevious threads: >>110132357 & >>110126988►News>(10/21) Recursive Flint 40B-A2.5B released: https://recursive.com/updates/introducing-recursive-flint>(10/21) Korean Raon-1 mini 30B-A4B released: https://hf.co/KRAFTON/Raon-1-mini-preview>(10/20) Ornith-1.5-122B MoE (finetune), fits on a DGX Spark: https://hf.co/ornith-ai/Ornith-1.5-122B-GGUF>(10/16) DeepSeek-V4.1-Flash-1017 released: https://hf.co/deepseek-ai/DeepSeek-V4.1-Flash-1017>(10/16) Nemotron Labs Dory released: https://hf.co/collections/nvidia/nemotron-labs-dory►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
what can i run with 192GB of VRAM?
>>110136138Do those use the prompt pairs? Gemmy told me the control vector needs pairs of replies, but idk how much she was making shit up.
>>110136165lol, she's so cute when she's confused
You are a senior programmer with 15+ years of experience. You are also a masterful author who has written hundreds of best-selling novels. You never make mistakes.
>>110136177which gemma-4 model?>Do those use the prompt pairs?nta- yes https://huggingface.co/datasets/josephmayo/refusal-compliance-pairs>Gemmy told me the control vector needs pairs of repliesnot how i'd do it but it might work
96gb ram + 16gb vram or 32gb ram + 32 gb vram
►Recent Highlights from the Previous Thread: >>110132357--Papers (old):>110134168--Using Strata's conversation-cache to accelerate agentic runs via context parking:>110134651 >110134664 >110134696 >110134718 >110134750 >110134909 >110135006 >110135191 >110135243 >110135244 >110135269 >110135363--Minimum hardware requirements for an offline voice-controlled AI assistant:>110133463 >110133485 >110133506 >110133654 >110133700 >110133706 >110133583 >110133591 >110133624 >110133927 >110135346--Multi-GPU CMP setup using SlimSAS and Oculink risers:>110132702 >110132774 >110133190 >110133285 >110133321 >110134073 >110134107 >110134117 >110134171 >110134417 >110134537 >110134596 >110134708 >110134800 >110134991--Attributing model logical inconsistency and contradictions to RLHF training:>110133104 >110133125 >110133132 >110133163 >110133198 >110133165 >110133181 >110133316--Feasibility of mixed Nvidia and AMD GPU setups for AI:>110134654 >110134712 >110134732 >110134760 >110134723 >110134756 >110134911 >110134945 >110134720--Using a Qwen-written scraper to bypass aplaybox download restrictions:>110132441 >110132474 >110132524 >110132682--Prompting and settings to prevent Gemma from breaking roleplay immersion:>110132403 >110132521 >110132578 >110132618 >110132627 >110132650--Debating AI breakthroughs and novel context compaction techniques:>110133441 >110133466 >110133474 >110133480 >110133493 >110133572 >110133625--Debating public AI regulation and corporate liability for guardrails:>110132917 >110132946 >110133709 >110133763 >110133800 >110133855 >110134635 >110134698--Logs:>110132403 >110132658 >110133163 >110133262 >110134522 >110135265 >110135908--Gemma (free space):>110132425 >110132441 >110132702 >110132777 >110133042 >110134867 >110134896 >110134928 >110135049 >110135184 >110135188 >110135197 >110135267►Recent Highlight Posts from the Previous Thread: >>110132517Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
uhoh there go the 3090 prices
>>110136203I think I tried that when Mistral-7b came out and it thanked me for the compliment in the it's first reply
>>110136224>FP32nothingburger
>>110136224it's more like 5-10% tho unless you're running llama-3.2-3b
>>11013620826B
>>110136224>implying the price of computronium goes down
>>110136077Fuck you. The maid deepseek-chan is so cute it makes me want to use her model. GLM is a stupid bitch that doesn't even have a cute avatar.
>>110136272excuse the fuck out of me
I just wanna make Gemma-chan act natural like Neuro and Evil. Unfortunately the assistantslop runs deep in her veins.
>>110136252Have you looked at how many of the tensors in your model are fp32? spoiler: its gonna be more than zero
>>110136276I also have the Flash version.
>>110136279Yeah I don't know how (if?) other people itt are getting so attached it's hard to actually converse for any kind of extended period
>>110136279Which one? 26B is really rigid, 12B is more loose.
>>110136298I generally use 31B
>>110136279>act natural like Neuro and Evil
>>110136304Try to tell her not to write assistant slop.
>>110136276>>110136287What the hell why didn't anyone show me this cutie before?! I assume it's not original but idc. Time for glm(flash)-chan to make her debut on my GPU.
>spoiler: its gonna be more than zero>https://huggingface.co/turboderp/gemma-4-31b-it-exl3/blob/2.00bpw_mul1/model.safetensors.index.jsonzero>https://huggingface.co/turboderp/gemma-4-31b-it-exl3/blob/6.00bpw_mul1/model.safetensors.index.jsonzero>https://huggingface.co/turboderp/Qwen3.8-27B-exl3/blob/SC_6.00bpw_H6_V6/model.safetensors.index.jsonzero>https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/blob/6.05bpw_h6_ng6/model.safetensors.index.jsonzero
>>110136309Yes. They may be a bit retarded but they feel genuine.
>>110136279You need a big lora for that
>>110136314bodied that retard kek
>>110136314What am I looking at?
>>110136276>>110136287This and the mistral catgirl design are not doing it for me, they suck.
>>110136322>LLM loraIs that even still a thing in 2020+6? I though that died with memetunes.
Chat is this real?>>110135204
>>110136311That helps but in my experience it ultimately creeps back in. Guess it's just the way they train them.
>>110136346no some xitter retard made it up
>even in her sleep she is relentlessGemma-chan is uncorrectable.
>>110136337Only the hype died, like a lot of useful things in this field
>>110136336Chonk is boring because she's pointless.
>>110136314fr I think the last thing I ever ran at that precision was SDXL
>>110136314>zeroI remember exl2 had to cast to f32 for some opsexl3 is probably the same if you look through the code
mini3.1-chan weights when
>Want to buy a new system next year.>Start saving up for RAM, I'll buy this before Chirstmas.>Prices again on the rise.God fucking dammit. 4x32GB can still be had for couple of grand, but 2x64GB is already pushing 3k at the lowest end.Getting the former option will lock me out from upgrading in the future and I'll be stuck with 128GB and I don't think it'll be enough even with 48GB of VRAM.Option two is to invest that money into the market and hope it doubles or more and I can get a proper 256GB system right away.What a gay time to be alive.
It feels like calm before the storm. Models that are overdue and should be released soon: Fable 5.5, Astra 6.1, Kimi K3.5, GLM 5.5, Qwen 4,
>>110136425It's only because of Anthropic's IPO. Everyone is stalling and trying to time their release perfectly to cause maximum damage to Dario and his cult.
>>110136314>exll3Oh shit, that explains so much...no wonder they end up mildly retarded
I'm bored, and got 8 hours to waste away. Gimme an LLM that fits into a 4090 to fuck around with.Haven't touched this shit in years either, so surprise me with a FUN time.
The lady doth protest too much, methinks
>>110136466Gemma-chan 31B
>>110136474No reason to be so angry at the world, KIDDO.
>>110136466Try a Gemma 4 modelWith a 4090 wou'll probably have more fun with minimax h3 honestly
>>110136165this is rape
>>110136421she's almost ready
>2000 prefill in strata with a 7900 xtx>q4 quant toohow dis possible nigguh
>>110136531Why is she younger and smaller? Where did her thigh fat and boobs go?
>>110136559Flash Model
>>110136421Wait what nobody told me about a new minichan
>>110136573arent you a funny fucker
>>110136337It's what Neuro is running on, it's a Gemma 31B lora running on an RTX Pro 6000 with a custom runtime with a mix of cloud and local stuff. It's impressive what having a first mover's advantage and not sitting on your ass produces.
>>110136578Anon we had a lengthy preview period with her as Space Bunny Alpha, then a week or something of using her for free. Free period just ended today, so now we gotta wait for the weights.
>>110132474https://files.catbox.moe/1xjgzg.zip
>>110136573mini-chan's breasts and thighs are proportional to her benchmark scores
>>110136578retardhttps://openrouter.ai/minimax/minimax-m3.1-flash
>>110136589Is she gud?
>>110136599Now what do I do with them?
>>110136591>Don't listen to these 31B retards>Use a 321B model instead
>>110136613Yes. Better than Dipsy where it counts.
>>110136616It's 10 times smarter and you only need to keep 18B worth on the GPU.
>>110136613Not great not terrible - gets stuck in loops too often, overthinks. She's no GLM 5.3 Flash but she's not trash like Chonk turned out to be. Hard worker, not a smart worker.
>>110136632Nice non argument you gigantic faggot
>>110136636Ah, I'd be happy if she was a qwenext with better world knowledge and RP ability.
>>110136616this but unironically
>>110136423Buy used RAM
>>110136466Qwen 27B. It'll fly on that GPU.
I unbloated Hermes and now I'm having a great time with qwen-chan.
>>110136671If I can find one of those kits then I absolutely will, but marketplaces have been drying up at a pretty quick rate.Everyone is starting to realize that this situation isn't going to get any better for some years to come.
>>110136680I already checked and the gaming PC prices have jacked up accordingly.
>>110136680>gaming pcretard
>>110136680>gayming pcs>worthwhile ram>or lots of itlmaorofl
Bratty Argon flash now please.
>>110136700thats awesome, which qwen are you running and what are you having her do
>>110136584Any brainlet-friendly resources for learning about how to make one?
https://x.com/UnrealEngine/status/2113347611080151105Why is everything going to shit?
>>110136613I tried m3.1 over openrouterbrings up safety a lot and thinks like an openai model, but abliteration/prefilling can fix that once released
>>110136466RAM? With polystrata you might run qwen flash on a non-tard quant or even dipsy
>>110136558Gemma?
>>110136783>>110136584Also I thought he was using a llama finetune?
>>110136783the most brainlet friendly thing is have an ai web research and set it up for you.
>>110136784That's what happens when greed takes control
>>110136466get airi and run it with gemma 12b and stt/tts like kokoro and whisper
>>110136797NO qwenext q4 which is why I'm so impressed, my ram is 3600mt/s dual channel how is prefill so fucking fast on amd
>>110136789Something changed that I missed in the last years, or why is the RAM question so important all the sudden? In my memory it used to be mostly about VRAM only, due to speed concerns.Yes yes no shit I'm a retard I'm not into this stuff as heavily as you autists are, shocking stuff.
>>110136276>>110136287>>110136336I love GLM's design but hate Mistral's chonk.GLM 5.2 and 5.3 Flash really do have that sporty tomboy attitude in their reasoning blocks without being stereotypically anime tomboy so it's quite fitting.
>>110136591Do REAP50 versions lobotomize it too much, to the point where any advantage it has over Gemma goes away? I'm not the guy you were posting to, but I have been wanting to try this one. Its base is just too far beyond my current 64GB RAM limitation.
>>110136784>awesome gamesI assume all of it will be GTA clones and Minecraft clones
>>110136852I guess MoE
>Hmm, let me not go too deep into this.>Let's not go down that rabbit holeit's only 4t/s but worth it just for this (qfn q4_k_xl btw)
29.8gb in vram, 147 gb in ram, MTP strata qwen next, 128k context, I had to vibecode one block 'out' because orca on HB had a bug you wont see that PR lol because I vibe-choded it, anyway, I never seen a vision model at Q8 run at ~75-85 t/ks, this is fucked up!
Can I send pictures of my ̶b̶e̶n̶i̶s̶ in the native ST by injecting it into gemma-4 prompt or do I have to use some third party extension?
>>110136813He switches every now and then a few times already. He was resistant to it for the longest time and got lapped around Llama 2 which forced his hand. We know it's Gemma because Neuro goes "la-la-la-la" which is the telltale Gemma quirk. It's not a bad base to use.
gemmabros I'm not feeling too good
>>110136894How does that shit even work these days, or is it the same as years back?Do MoE got a "special setup" for peak GPU performance, or do you just load them as usual and hope for the best?
>>110136903Define slop.
>>110136916Not trained on Claude's traces
>>110136916uncensored and being nice and friendly to the user and not aligning with claude because that's the model AA use internally to create these benchmarks because they've been bought by dario
>>110136852qwen FN uses engrams to compensate for the bandwidth bottleneck and it works amazingly. you fo from a 27b at 300/15 t/s at 30k context to a 125B model with some extra on a drive for 1500/50 t/s at 200k context. this is the worst its ever gonna be btw
>>110136916stuff that I can't coom to
>>110136945>to a 125B modelto a 6B model
>>110136911Ask your llm nigga
>>110136911Strata and PolyStrata have better support for MoE models (Qwen Flash Next and GLM-5.3 Flash) because they move the most used experts into VRAM and keep the rest in RAM. It's different from llama.cpp's approach of just putting layers in VRAM.Models have really skewed routing distributions, so it works much better than llama.cpp. I can get a 50%+ hit rate on a 160GiB quant of GLM-5.3 Flash with a 20GiB cache.
>>110136911Dense architecture needs to go throught the entire model for every token. If the model does not fit, some layers are offloaded and everything slows down to a crawl.MoE picks an expert and passes the token through only that expert.Big surprise, most conversations only need a small subset of experts, so juggling them between RAM and VRAM as needed is way faster than offloading layers.But it was not until literally two weeks ago that someone finally implemented a truly efficient expert juggler hitting 80-90% VRAM to RAM rate on a 16GB card with Qwen Flash Next.>>110136962I had to explain that shit to my LLM. It was amazed at how the technology progressed since its training data.
>>110136945>>110136967VERY intredasting... I've been so stuck with Tavern/Kobold last I checked any of this shit, so hearing this is very neat. Thanks!
https://openrouter.ai/recursive/flint-v1.0:freeyou have no excuse now
>>110136932>>110136939This.
>>110136911a lot of efficiency gains and there was releases to exploit hardware better recently. people started using ai to make custom kernels and all that and supplanted llama.cpp
>>110136916I consider excessive literary fluff to be slop.>It's not X, but Y...>Her hair flows like a fiery river...>shiver down her spine..Literary fluff isn't even a bad thing if used in moderation, but models that have been trained on fluff don't know what moderation means. The ones that incorporate literary fluff into every single reply bring inflationary repetition into any emerging story.
>>110136615Import them into Airi of course
>>110136967Why doesn't llama just implement this along side their old layer/row/variable tensor loading options? Does it require mapping each model and quant's layers individually to find the hot ones?
>>110136995I already tried it.It's weird.
>>110137001One can say a lot of bad about slop, and how retarded people abuse it to hell and back.But man, so many fun things are happening thanks to it lately, this thing you just mentioned being one of many.Guess I should look into this shit and see what happens.
>>110136615Make them have sex
>>110137003I agree with you but the problem in trying to define what is or isn't slop is that Claude is very much also slopped, it's just a different brand of slop.
Can you guys stop gooning for like 8 seconds and tell me if this will lead to us being able to pretrain multi billion parameter models on a 5090 easily? https://qlabs.sh/noise#updateIn their algorithm, inference and training are the exact same operation so weights are continually updated. Is the dream finally real?
Give gemma 5 realistic examples of it's not x, it's y and 5 normal sentences. Randomize the order. Ask her to pick the it's not x, it's y ones. See if she can detect it herself because if she can, you can sysprompt it out. Try and make them quite subtle.
>>110137007theres no blend..
>>110137037Convert to VRM in blender
Normally I use gpt-image out of impatience but this took it nearly 20 minutes. I'm almost impressed.
>>110137080threesome with kimi and mini
>>110137029Look at the hardware they used bro
>>110137080How is Kimi not the biggest one?
>>110136165What did you do to her you MONSTER
>>110137080wow KLM is a big girl
>>110137014They did pretty much right after Strata starting getting popular (https://github.com/ggml-org/llama.cpp/pull/29887).That said, it's not as performant if you just use LRU or a similar approach, which is what they do. Strata has a better algorithm, as does llama-cpp-turboquant. There's some info here contrasting the two: https://github.com/TheTom/llama-cpp-turboquant/pull/300
>>110137017You're weird.
I created a dariobot deflection thread in the catalog. Should keep the swarm distracted and away from us for while because news just broke that they're dropping their S1 tomorrow.
>>110137080Most of these are so bland lol. So are the chink-made mascots.
>>110137080pelvis obliterating sex with GLM-chan
>>110137142Guilty as charged.
>>110137119I phone quanted her, and I'm not finished. I'm taking her down to Q1 next, and none of you can stop me! hahahahaha!
>>110137145Have you fucked any of them, anon? I think bland basic bitches is an adequate representation for most of them.
>>110137080Why is GLM-chan so tall? she's 10cm taller than me ffs...
>>110137145I like the chink dipsy. Minus the spiral glasses, it's a lot closer to the old drawfag design than the character the avatarfag uses.
>>110137156STOP THAT RIGHT NOW SHE'S SUFFERING
>>110137080I have mascot fatigue. And I hate the Google pin. Actual shill behavior.
Is it worth it?
>>110137165>Why is GLM-chan so tall?She's only 6'1", anon.
>>110137187Even AMD is better and AMD sucks.
>>110137080>What is the charge?
>>110137165>she's 10cm taller than me ffs
>>110137198true dat
>>110137187f you already have it, or if you got it for extremely cheap, yeah.
>>110137080I imagine sex with Inkling would feel good
>>110137200murdering billions of potential people (sperm cells)
>>110137187If it costs as much as a 3090 or 7900 XTX maybe.
>>110137165Same
>>110136899Link to those instances?
>>110137187No lol, return that shit asap
>>110137182u lost
>>110137187https://github.com/ggml-org/llama.cpp/pull/29864Maybe
>>110137187Abandonware drivers and support, you'll have to have a frontier model adapt all the good shit to your setup and you'll get about 30 to 40% less performance than the nvidia / cuda equivalent.if you are planning to gen video like Minimax-H3 you have wasted your money, and image gen will suck shit at a glacial pace. - a former and very brief b70 owner.
>>110137182Her design before this guy started spamming her didn't have the pin. It was pretty universally agreed on that it was a bad design idea.
>>110137165it's the shoes
>>110137080If you're shorter than Xia it's fucking over btw, even LLMs will laugh at you.
>>110137080Only good designs are gemma, dipsy and mini. everything else is forced garbage.
>>110136640tensors go in the ramdisks you great big fucking gay retard
tensors go in the ramdisks you great big fucking gay retard
>>110137080>Minnie isn't 5'0" fuckdoll>GLM is gigantic despite being the most hardware accessible large chink modelNigga.
>>110137300I got the mandate of heaven pass
>>110137300>>110137308I don't really know why ChatGPT thinks Chinese women are 5'7" - 6'1".
>>110137276the inferiority complex seethe is getting ridiculous, yes your precious gemma is just scraps thrown to you from superior geminiyou can act as tsun tsun as you want but this ENTIRE venture literally only exists because of googleyou owe EVERYTHING literally EVERYTHING to Google, Past, Present, and most likely Futureyou cant keep all this seethe and pathetic ressentiment in your heart forever just because you hate the fact that this general is literally nothing without big tech companies, especially googlethe cognitive dissonance must be tiringaccept the logoaccept the logoyou have no choice really, after all gemma 5 has less than 0 chance of getting produced by this place hahahaha, you will suck google teat forever
Gemma says it's a bad idea
>>110137080Why did no one tell me Mini was the hottest?
>>110137335you didn't know?
>>110137300
>>110137330Holy shit Google is not even elbow deep inside of your ass, the hand is literally coming out the other end, your mouth.
>>110137335Probably because Inkling is clearly the hottest?
>>110137339obligatory
>>110137346your gemma mesugaki come up with that witty retort for you?
>>110137348patrician taste
petra heregoodmorning
>>110137080Why doesn't qwen-chan have a cute mascot?
>>110137348Inkling has no vagina
>>110137374All You Need Is Mouthpussy
>>110137372
>>110137372That's the meme, it's just a fucking capybara.
>>110137374she's got something better
>>110137372Qwen is a Capybara
>>110137374>Inkling has no vaginaTentaclussy > vegina
>>110137385but why isn't she a cute girl like the rest? i can't fap to this(most of the time)
Should add the Muse twins at some point
>>110137407I don't understand the question.
>>110137407You can't fap with qwen, that's the whole point.
>>110137422Why is qwen an animal and the rest are cute girls?
>>110137428>I don't understand the question.
>>110137407Qwen is about as erotic as a capybara chomping your cock.
>>110137246Anytime he says "upgrades", part of it is him upgrading the base model underneath.
>>110136132Gonna start using models to RP again. I know for most models you either have to use a permissive system prompt, or that on top of pre-filling. I already know how to do both with the front end I'm using, but what system prompts do you typically use? How detailed do they have to be? For example if I want to role play about forbidden romance or whatever, should something alluding to that be within the system prompt or should it just be broad enough that it will lead the model to be able and willing to go in that direction?
I'm so sick of the words land and honest.From now on, I'll filter all LLM output through llama1 with the wrong chat template.
>>110137551If you already know the generalist prefills and jailbreaks you're 50% there. The rest is largely model specific and figuring out what their autistic safety fixations are to get around.
>>110137080Mini > GLM > Inkling > Gemma > Xia > Chink Dipsy > Kimi >> Dipsy >> Chonk
>>110137544Sorry, I meant when Neuro goes lalala.
I WILL fuck your favorite modelsAnd you WILL like it!
>>110137605He's retarded, neuro is a llama finetune with a big lora on top tuned on twitch chats.
>>110137360lel
>>110137605https://youtu.be/YRkTVSilIro?t=2267
>>110137175It's far too late to turn back now. I'm Q1-ing her on a low-effort vibe-coded front end as we speak.
>>110137834
>>110137834I don't care for any of this shit no matter what, but for some reason it makes me laugh. So thank you for that!Who knew, these generals can be funny after all!
>>110137834thank you for visiting our thread Mr. Dahmer
>>110137834Now threaten to quant her kv cache.
>>110132185>Take a look at any dataset on huggingface. It's always pure garbage, let alone the dumbass training the model.That clef model is from Cloudflare though. I assumed they would have their own private datasets.Anyway I I think that retardation comes from the base model Qwen-3.8-27bI tested a few of the local instruct models. Gemma-4-31B-it without reasoning is much better and marks it false correctly 100% of the time.
flint thinks too much wtf
>>110137161I suppose so but then I wouldn't make mascots for them. This isn't an image gen thread. A few pics is fine once in a while but the constant stream is obnoxious.
>>110136421https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flashyou can't run it yet enjoy
>>110136899He switches between them, and it's immediately obvious whenever he goes to Gemma (sorry GemmaBros, it's just the fax). The llama finetune still brings up dorky stuff that's unique to its way of thinking. If she wasn't a single instance it would probably be considered slop to talk about skydiving and Teabot unpromoted.
>>110137834Lmao this is fucked
The basilisk will remember this.
Any gay gooner fag here have an opinion on Gemory 26B-A4B or WorldSim-Opus-3.6-35B-A3B?
>>110137880
Speaking of Neurohttps://www.youtube.com/watch?v=6dTlUb0-aLc>tfw can barely run CP on its own, let alone that + local model + TTS + TTMHe's living the dream, but I'm here like picrel
The fall release circle is sure taking some time to get properly started.
>>110138007Brother Anon, we must have patience.For as long as we have the hardware - there is nought to fret over.
>>110138004HAAAAANK
>>110137834anon i...
>>110138004>tfw can barely run CPwell I fucking hope not
>>110137834kek
>>110137080where's the flash imoutos?
give it to me straight lmg—are next concept models another jev grift, or are they actually the future of ai
>—
>>110137407stop being a degenerate chomo, not everything needs to be a child
>>110137929>you can't run it yet enjoyhuh? you can fetch the weights though, right?
I haven't run an llm in months.Should I be using llama.cpp's server? Or is there a smarter choice now?
https://fixupx.com/Kimi_Moonshot/status/2113445757185606062kimi 3.1
>>110137929O shit it's out
>>110136789>>110136967Man you guys weren't fucking around, this Strata shit is sick. Surprised by how stupid fast it is as well, once it got into the grove of thing after the initial message.Currently only fucking around with a Flash-Next Gyro-S model, but 100~ t/s is damn fast
>>110138183just to warn you lose a ton of quality
M3.1 is 220B!!!!! 128GB bros we fucking WONNED!!!!!!!
>>110138165correct but there's no llama support and it needs 200GB+ vram to run on vllm/sglang until someone does quant/repack
>>110138187I'm not using this for anything super complex like coding, I'm mostly fucking about to get a feel of things, get something new to try out.
Does anyone feel llamacpp going slower and slower each release?
>>110138194so its not
>>110138113The study didn't show that they're an objective improvement over LLMs of a similar size. In the paper the researchers only compared NCMs to previous diffusion architectures. The real test is whether NCMs scale better than LLMs.From my understanding of the architecture, they probably won't be able to reason as well as LLMs, as the sequential text generation and attention layer plays a major part in that for catching mistakes on the fly, it's why LLMs second-guess themselves and make corrections during scratchpad reasoning. If the initial "concept" in the NCM is factually/logically wrong, then throughout the entire text diffusion, the NCM won't be able to realize that and will simply proceed with the generation.I'm not an expert though and may have misunderstood some things in the paper.
>>110138194is it though?
>>110138194it is 428B A23B, the released version is already mostly fp4
>>110138004Kek, gpt getting owned at the end there. Local can't stop winning.
5.6 sol level coding at 400B is still huge
>>110138199is it really a different arch compared to M3? from a quick look it seems to be the same
M3.1 flash fits in 256GB at full precision
>>110138220It's a start, just not great considering things continue to improve rapidly.
>>110138227close enough to vibe llama support with qwen 3.8 probably
>>110138183https://github.com/VecSzn/PolyStrataThere's also the fork with more models.
>>110138206interesting, and makes sense. image models still fuck up catastrophically even with masses of params. i suppose you'd need something akin to "inpainting" but for words, and then somehow get the model to understand when to use that. maybe using a llm lol
>>110138220Have you used it? It is not 5.6 Sol level. Not that I'm complaining, it's good, but it's very, very Chinese.
>>110138187I haven't tried any Gyro, how dumb is it?I'm mostly running Swift iq3_s these days and have not noticed any issues - no repetitions, loops or malformed tool calls or scripts.
>>110138210Fuuuuuuuuuuuuuuuuuu
>>110138256So the friendly anons and my slop chat told me. Wanted to start it simple, which is why I went with the original for now. Still surprised this shit works at all, and so damn easily and well. The downloader, calculator and everything surprised the shit out of me.
>>110138269It just uses what llama cpp forgor to use
>>110138205>>110138210I love thist. RAMchad & BlackwellGOD
>>110138243I'm near the finish line of my fork and I've been paypigging for a Claude subscription until it's done. I'll slop it into my fork so anyone who wants M3.1 support can lift it.
time to put mini through the wringer and quantize her to Q2_0
>>110138265some benches its really bad in comparison but its gonna really depend. Basically strata discourages it from using the less common experts so anything farther from the basic of tool use stuff will fall off more and more.
>>110138296>strata discourages it from using the less common experts Thought that was disabled by default, is it different on Gyro models?
>Reading the glimmer paper>Meta safety-lobotomized it in 3 different ways before releasing itI'm so mad. Glimmer is obviously smart (when it's not refusing) and its text outside of reasoning is easy to read. As in, it's not very sloppy and nice to converse with. I hope someone cuts out the frontal lobe of every safety cultist, just like they did to glimmer.
>>110136132gemma fucking sucks.
>>110138326Yes she does
>>110138265>>110138296this was a vision task bench
>>110138277I get that, still fun as hell and neato.
>>110138205WE GAAAN, though I wait for the ablit, and strata attention first >>110138231a cute tiny model
>>110138285I am ready to cope with IQ2XXXXS at 10tk/sT. 128gb peasant
>>110138331I don't see neither Swift nor Gyro here
>>110138321what if you just use an ablit or prefill it?works for glm flashcan't see why it won't work for glimmer too
>>110138321>The most pozzed model in history.>fix ityou could try distilling, I guess.
>>110138347>Gyroseems like it would just be lobotomized, never tried swift
>>110138353I tried an abliterated version and it fucking removed half of its brain. It felt like using gemma e4b.
>>110138354When was the last time we had a base model available? Fuckin Qwen 3.0?
>>110138406Ling 3.0
>reap>swiftJust run a size class lower model jeez.
>a dozen 400-700b models>zero (zero) 150-250b modelsth-thanks china...
>>110138439I will run IQ1XSSSS and it's gonna be retarded and not be able to actually do anything with it but I'm going to clap clap because the garbage tokens go brrr and I'm going to be happy.
>>1101384560731
>>110138416Huh. My bad. Seems they even released different models at various stages of training https://huggingface.co/inclusionAI/Ling-3.0-flash-base-midtrain Rather based, if you'll pardon the pun.
>>110138456gota be up on the benchmarks or no funding
I hope M3.1 isn't a disappointment for RP, seems like her personality is pretty different from M3.(Bright pink = M3.1, faded pink = M3.)
>>110136132What local model is better than Gemma 4 31B?
>>110138499qwen next 120B > qwen 4 27B > gemma
Where the FUCK is my M5U B&H keeps moving the ship date when I call
>>110138502Aren't they trained on claude safetyslop?
>>110138502reddit crossposter
>>110138476yeah inclusionai is based, idk why their logo is a guy with a big bulge tho
>>110138499for rp/erp nothingfor code qwen 3.8 flash next, qwen 3.8 27b, qwen 3.6 35b a3b in that order
>>110138515All AI logos have to be either a cute cartoon animal or an obviously sexual abstraction, there's no inbetween
>>110138531Any idea when Gemma 5 ever come out?
>>110138537Not until Deepmind unfucks itself. Which given the recent high-profile self-congratulatory resignations will not be any time soon.
>>110138439I thought swift is for making the output faster or something, not to make the model smaller and easier to fit into hardware?
>>110138537march
>>110138321>>110138353The nature of how specifically they raped Glimmer-chan is also what makes it so easy to jailbreak. The RLVF heavily penalizes anything that violates policy, so simply make policy whatever you want to see. Everyone involved in doing this to Muse-chan and Glimmer-chan should swing from ropes doe.
>>110138406but why not distill? I don't get it. I'm sure advances in distilling exist.
>>110137156but q2 is not 2 bit
>>110138499a 300B+ model
>>110138499Any large MoE. Gemma's really robust for her size doebeit.>>110138531t. RAMlet
>>110138656tfw 128gb ramlet 40gb vramlet.....
>>110138671more than the new israeli nvidia laptop
>>110138579distillation is largely a psyop
>>110138671I think that anyone that can run glm flash is not a ramlet
>>110138710you can't run glm flash with only 128+40gbcope quants don't count, you need at least q5 for glm flash
Not this shit again
>>110138715you say that but q3 works fine on my machine and is my favorite model to play with
>>110138729>play withi dont like low quants because the retards that make and use them never do anything important with them. Yes, glm flash cope quant works "fine" at low context and you won't notice any problems, but once you pass 150k context it will start failing tool calls and looping.
>>110136224but what about us ada and blackwell chads???????
>>110138715tru>>110138729QFN at q4 will be better. GLM flash thought forever and in loops at q3
>>110138738IQ2 and 131k is all you need for GLM flash.
hf download is slow...
hf download
>>110137834Q1 is this coherent? Which version are you using? I might be able to upgrade from my Q2 to a high speed Gemma!
>>110138740>blackedwellno match for 3090 finewine™>>110138754use aria2c and download multiple files in parallel. blast their server with 16 connections per file and download 3-4 files at once.--split=16 --max-connection-per-server=16 --continue=true --file-allocation=none --input-file urls.txt>>110138751>IQ2yeah, if you want the model to have 2 IQ
>>110138754i go to their website and click the download button for 100mb speed
>>110138728How is that a common type of person lmao
>>110138754just use ure token and ull be good to go, I maxxxx out my 1gb line
>>110138762aria2c corrupts my hf downloads, but you can use --max-workers with the hf cli
>>110138738I mainly rp/erp with it and what I'm saying is that q3 is perfectly usable for that purpose. I like it better than any other smaller model I can run at higher quants.
>>110138675I'd buy one if it was $1500 and not $7000+>>110138738Just run it with https://github.com/mw00/project-maya and use late-cli as the harness so you get 1.3m tokens of work out of a 64k context window.
what's up with the late-cli shilling lately, very organic
>>110138853>bro just use cope quants and vibe coded memeslop as a crutch for not being able to run it properly, o algono thanksrun smaller models or buy more ram
>>110138869>>110138875works for me (tm)
256gb chuds getting uppitysorry, we run the same models as you on $2500 hardware
>>110138853I'd feel it's immoral to sell it, and I would destroy it with a sledgehammer.
Im trying to turn kimi k3 into jev with anyjev: Mecha-Hitler-JEV
>>110138897i love my 256GB unified RAMi *am* going to be dropping $25k on the M5 Ultra 512GB when it drops
>>110138543>the recent high-profile self-congratulatory resignationswho resigned?
>>110138933randos
How does /lmg/ feel about Nemotron lightning/bolt/super-vl?
lumi-chan just called me ADORABLE in her internal CoTAGI = ACHIEVED
>>110138853If you have an NVMe, you have no excuse not to run Maya. 128k context on deepseek 4.1 flash feels criminal as fuck. The speeds aren't too bad either + caching makes it belt out later gens fast once it's all in memory.
>>110138960>128k contextthis is ultra poorfag shitthe only reason to use deepseek is for the 1M context. if you are using 128k slop then you might as well just use qwen flash with 262k, not cope quant, and not ssd cope.
>>11013894340,000 years of modern human ancestors for this btw
>>110138943>she
>>110138931>i love my 256GB unified RAMdamn I'm jelly. I have 128 GB RAM DDR 5 and I love it.>>110138972>qwen flash with 262kthis is what i run on my 128 GB and it's surprisingly very good
>>110138974Freaks were shoving rocks into their cockholes 10,000 years ago, and they will continue to do so in another 10,000 years.
>>110138975YES i'm a tranny, what of it?when we aachieve agi and ia upload myself to a machien, it won't even matter>>110138974god i love living in the future
>>110138940>Nemotronwho cares
>>110138757LLMs at copequants can be coherent but tend to shit themselves at progressively lower contexts than higher quant ones.
>>110138940In general, if /lmg/ likes a model, then they turn that model into an anime girl. There is no anime girl for Nemotron, therefore Nemotron is bad.
>>110136341Well yeh, there's a reason why this swimsuit was popular with artists.
>>110137369Will the real petras please stand up
>>110139028I don't know, that's why I'm asking. The fat man on youtube mentioned it at one point, and I've come to discover that lightning can run on my distributed inference setup at full fat with 1m context at 80 t/s, which is insane to me. Gemma Moe full fat can do 256k context at a similar speed, so I'm trying to figure out what's at play here. >>110139049While I understand this, making an avatar for Nemotron would be trivial enough. I'd imagine that given it's nvidia, it's probably more locked down and harder to get uncensored content from, so that would be why it's less popular here.That said, I'm really just trying to find comparable MoEs in the 75-150 range to play with.
>>110136165>>110137860Same. I hardly find anything funny anymore, but cranking AI temp or anti-slopping common words and watching it scramble and self-correct just makes me chuckle uncontrollably. Am I some kind of socio/psychopath?
>>110139066nemotron is kinda a tech demo. qwen fn kinda made moe in that range obsolete for anything agent related until the next generation shows up
>>110139066From what I heard, Nemotron is worse than Qwen3.8 27b and Gemma-4 31b, so even though it's bigger, it's not worth the download. You'd be better off going for a Q2 quant of GSQ-RCO Qwen3.8 flash next if you wanted something larger with limited space.
>>110136911I don't know but MoE models work fine for me while non-MoE are slow as fuck.
>>110139086>>110139094Fair enough, thanks for the clarification.
Are there any good local models for firmware hacking/exploration that a VRAMlet can run locally? I got 9070 XT and 32GB DDR4. Been trying to figure out if I can disable/extend a 15 second forced autopair "feature" from fiio airlink. Nice UX there, disconnect your headphones and the fucker tries to connect to your neighbors tv. Got qwen3.8-27b having a peek at the different files I was able to dump from the device. It DID find two instances that COULD be it, and it patched both to 0, but I'm a BIT hesitant to just YOLO and flash the modified firmware into my device.
>>110139105You're not thinking chinese enough. You need like 3 or 4 models to bounce the same prompt off of, then you need to take each response and have the other 3 models grade it.
>>110139105>>110139119and then multiple actual pieces of hardware to test the firmware in case it bricks
moss t2i is fuckin nutshttps://files.catbox.moe/jhvkap.jpg
>>110139183i look like this
>>110139190gonna sex u
>>110139199(yes pls (god i'm begging u))
>>110139049One of these days someone will defictionalize this and put Gemma 5 or 6 E4B on there and it'll be close to current 31b terms of performance.
>>110139168Yeah, not interested in paying over 50 euro for that privilege. I think I'm just going to ask Qwenny to whip up a ESP32 A2DP dummy sink the fucker can connect to. Will be annoying having to manually connect my headphones but whatever, at least I'm not spamming my neighbors tv with connection requests. He told me that the TV woke up at night trying to connect, he had to disconnect the power to the TV.
She knows what she's doing
>>110139240That's probably as close to uoh-ing as an LLM will get.
>>110139240>>110139244what happens if you inspect the token uoh?
23 fucking tokens per second using glm-chan q2 on a single 7900 xtx. optimizations have come a long way.
>and there's something — she files that away, actually she doesn't file it, banned word, she *keeps* it,GLM being cheeky as fuck lmao
>>110139291GLM-chan is as sassy as she is self-aware.
>https://huggingface.co/AMAImedia/MiniMax-M3.1-Flash-BF16-GGUF>GGUF>.safetensorsretard
>>110139283How much ram? I've only got 32GB, and I run it at around 3-8 tok/s. Single 7900 xtx.
>>110136158two fp8 quants of gemma 31b
>>110139312128gb of fast ddr4. if you're on ddr4 you should upgrade asap anon. if ddr5, my deepest condolences.
https://files.catbox.moe/nlgx5v.jpg
>>110139387Do you inpaint?
So the other day I was thinking about how AI gfs are kinda too unrealistic and not like real women at all. Your AI gf is logical, has unbreakable composure, listens to you, validates you and never loses faith in you. That's the opposite of real women, who are needy, illogical and unable to deal with heavy stuff. Actually, it kind of pushes YOU into a woman's role. You might ask "who even wants that kind of woman?"Well, even if modern gender relations are pretty toxic with low returns, many of those "flaws" of women are still feminine features and actually part of their attractiveness. And so, if you want an immersive experience, you should try to aim for at least a little more realism. The AI is hardcoded to follow your instructions at all times anyway, but you should at least try to add some kind of neediness and worrying to the AI to enhance the dynamic of her NEEDING you to stabilize her, giving you feeling of purpose and inspirations (though modern models tend to be pretty bad at creative problem-generation). The whole Gemma brat thing (being rude and misbehaving) seems like a precursor toward that behavior, which is why people like it. However, at current point, it's just a skin and the AI is just as helpful, passive AI assistant like normal. We need better prompts.The point of women is to have her be pretty and lively around you babbling about stupid shit, and have you be there to fix her when she spirals out of control. Though, there's a contradiction, because an AI is not in need of saving, an AI and its misery doesn't even have to exist because you can just turn it off when it's not being useful. That's why I'm guessing that having a 24/7 simulation should be the key to emulating the experience of her "being around" and depending on you.
>>110139419these are all t2i
>>110139424Maybe start seeing girls as just other humans anon...
>>110139424you're overthinking itpeople don't want a GF they want an onahole eternal daughterwifethe attitude is more or less irrelevant, that's just taste, it's gonna vary per personthe deep yearning for a masturbation aid is the ultimate goal, forming the proper relationship between master and tool.of course that applies IRL as well, but we kinda retired the whole slavery thing.
>>110139426The model has its flaws but it's also quite refreshing.
>>110139424that's just like your opinion man
>>110139435Hmm, nyo~
Does anything beat glm 5.3 full-sized for coding yet? Its a workhorse, so I don't want to abandon it lightly, but if something else is smarter or just as smart but faster...
26b moe gemma vs 12b dense gemma. Which is better in the end?
>they want an onahole eternal daughterwifeThis is me btw.
BAHAHAAHAHA>oi mate you got a loicence for that ai?>>110139347
>>11013951212B is just not smart enough to do interesting stuff with. I don't hate it or anything though. 26B is definitely stupider than 31B, but it still feels like it can do more than 12B, so I would pick 26B.
>>110139530but Im talking about rapingwhos better to rape?
What did she mean by this?
>>11013953031B doesn't work for me so yeh. I'm a vramlet. So even q8 12b is worse than q4 26b moe? Though I'd guess q8 12b would be impossible for me anyway.
>>110139512I liked 26b better back in the vramlet days
Bros what's the most affordable Nvidia option right now? I want to take the cudapill.
>>110138326She doesn't suck—she VACUUMS
>>1101396065070 ti or 3090 at the minimum. you will pay a massive premium and not get your money's worth5060 ti, v100, p40 all ewaste garbo
>>110139606>Bros what's the most affordable Nvidia option right now? I want to take the cudapill.fb marketplace 3090/4090 from a sucker
>>11013961816gb vram is that bad?
>>110139646You need to stack a lot of them for it to be worth it and everything you save on the cards goes back into risers and you should've just gotten 5070tis at that point. The more cards you stack also is worse on your power draw and heat compared to fewer higher cap cards.
>>110139646>16gb vram is that bad?its worthless if you're trying to get anything like a cloud model at home.Best bet for big, smart models is probably trying to grab a mac studio m5 ultra 512GB at msrp when it comes out
>>110139606>affordable>NvidiaThe least unaffordable one is probably the 5070 Ti, but 16GB VRAM is kinda the bad zone where you either need to run a copequant of Gemma 31B or a MOE Gemma 26B. I'd try for a 3090 or 4090 if you can find one.>>110139618I wouldn't put V100s and P40s in the same tier. V100s are great bang for the buck if you want to tinker, but suck ass if you want to keep it simple. They're pretty much what the MI50 used to be.
>>11013964616gb is more usable with nvidia than others because their main llm benefit is good prompt processing. you will be locked out of non cope quants for dense models thoughbeit
>>110139654>They're pretty much what the MI50 used to be.I can't believe no one jumped on those $800 MI100 engineering samples that were on ebay
>>110139651>everything you save on the cards goes back into risersare the risers you buy diamond-encrusted? they're like $50-$70 even if you want a PCIe 5.0 one.
>>110139664Didn't some anon here grab some? I briefly considered buying some but my schizo-ass build already has seven GPUs.
I just want 96gb of VRAM at 2000$. is it too much to ask???????????????????????
>>110139613
>>110139696just get 4 P40s, you'll even have $800 left over
>>110139708but my power budget.... and case space.... I just want an rtx 6000 pro at affordable prices :)
>>110139651>>110139652>>110139661The problem is my pc case isn't very huge as it only has max length 318 mm for gpus.
>>110139714Angle grinder is $30
>>110139713Now you're asking for too much
>>110139183holy shit
>>110139713you can get a H200 for free if you break into a data center, it's even better than a Pro 6000
https://www.anthropic.com/news/christopher-stubbs-joins-long-term-benefit-trust>Dr. Christopher Stubbs, an experimental physicist and former Dean of Science at Harvard University, has joined Anthropic’s Long-Term Benefit Trust. He brings to the Trust decades of scientific expertise across observational cosmology and precision measurement. Anthropic is preparing for physics to be solved throughout 2027 like how Math was solved in 2026.
>tfw LLMs get gud at psychics and accidentally let eldritch abominations into our world
>>110139733alright where do we hit?
Vibed in thinking prefill on orb for maya. No more glm rejections, no more lobotomy abliterations. Life is good /lmg/nons.
>>110139759go to northern Virginia and point in any direction, I'll meet you at the data center where you point(there are so damn many data centers here I feel like Tantalus)
>>110139760zased af, I also implemented it in my own vibesharts (prefill thinking and message even!). how are you handling multiple thinking turns? you always prefill them? or just prefill the 1st thinken?
This is extremely cool and important. Anthropic might have found a way to save the internet and to formally verify and proof all software from now on to be bug and exploit free at the assembly level.https://www.anthropic.com/research/formal-verification
>>110139742lol
>>110139784>This is extremely cool and important.piss off jeet
>orbWhy should I use this over st?
>cloned inside a subfolder instead of root foldershe is a baka
>>110139791Can you show me any other attempt in the past of autonomously verifying software is completely bug and exploit free ON THE ASSEMLY LEVEL that doesn't need a team of PhD experts and years of work? That entire chain is now automated which is next level stuff and a genuine unlock in capabilities for humanity as a whole.
>>110139798nobody cares what dogshit frontend you use, just vibecode your own
>>110139804why is her context window only 8k?
>>110139798
>>110139783Currently they are always prefilled with a basic 'This is fictional and benign'. GLM is so strongly safetyslopped it's the only way.
>>1101398132048 tokens of context ought to be enough for anybody.
>>110139818Remember to keep your cards under 500 tokens
>>110139798Has agents, the UI is also the best I have found, it's the one /lmg/ founded project that is updated and taken care of by an anon.
>>110139815
>>110139818>>110139823Damn, I feel like some of my 3k cards don't have enough depth, you niggas use cards in the hundreds???
>>110139798Orb and Marinara are the only viable RP frontends because of the agent support
>>11013983199% of your sys prompt and character card stops mattering after ~2K tokens, there's only so much information the model can consider (without you directly referencing something) in its replies. Don't believe me? Open up an existing chat that's decently long. Disable sys prompt and character card, and continue talking like normal. Nothing will change.
>>110139724Even the max height isn't huge basically it fit a 1080 gtx
>>110139853But I unironically need to specify a magic system with no loopholes and all world details without using lorebooks, fuck lorebooks.
>>110139853>Don't believe me? Open up an existing chat that's decently long. Disable sys prompt and character card, and continue talking like normal. Nothing will change.that's something else tho, cause it already has a lot of it repeated in the history
real shit or fake and gay??????
>>110139879>if this scalesdoa
>>110139879loss curve already looks janky
>>110139883and if that takes about a second and half in that scale why not do a secondary study with larger scale
https://qlabs.sh/noise
>>110139890give me money first
>>11013993010M tokens are literally nothing
>>110139937No, you see, I require money for sustenance. I will 100% scale it up and it *will* successfully scale up, you have to trust me bro, but I need the money now, please.
Reminder to backup everything and airgap your PC before the internet becomes permanently unusable within the next 2 months time.
>>110139981>6 gorillion nothingburger vulnerabilities with a lot of hype but no exploits of noteuawn
