/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>110235794 & >>110230675►News>(11/01) Longcat-2.5 released: https://longcat.ai/blog/longcat-2.5>(10/30) Google receives response from Gemma in space: https://blog.google/innovation-and-ai/models-and-research/google-research/project-suncatcher-satellite-response>(10/29) Mistral Large 4 (Le Chonk) 1T-A52B released: https://mistral.ai/news/mistral-large-4-oss>(10/27) NVidia releases Nemotron 3.5 Transcribe 30B-A4B: https://hf.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B>(10/26) XHToken releases Spark-X2.6-1.7B: https://hf.co/XHToken/Spark-X2.6-1.7B►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
►Recent Highlights from the Previous Thread: >>110235794--Corporate shift to self-hosted AI and accessibility of B200 hardware:>110238641 >110238769 >110238926 >110238943 >110239002 >110239076 >110239003 >110238662 >110238705 >110238746--Feasibility and technical difficulty of modding GPUs for more VRAM:>110237881 >110237919 >110238198 >110237972 >110238950 >110238976 >110239020 >110239107 >110239154--Comparing Basalt NVFP4 and Strata engine performance and efficiency:>110236722 >110236768 >110236875 >110236983 >110237012 >110237059 >110237286--Speculating on the identity and privacy of Crouton Alpha stealth model:>110238238 >110238247 >110238257 >110238272 >110238402 >110238418 >110238835--Revolut switching from frontier APIs to self-hosted Chinese models:>110236987 >110237007--Comparing unified memory rigs versus mixed GPU and RAM setups:>110236975 >110237037 >110237292 >110237326--Comparing GLM 5.3 Flash quants and thinking setting efficiency:>110235890 >110235903 >110235916 >110235964 >110235997 >110236234 >110237436 >110237455--Speculating on ideal model sizes and comparing Muse Spark's steerability:>110237388 >110237492 >110237552 >110237576 >110237621 >110237563 >110237953--Anthropic's massive future compute spending commitments through 2045:>110238040 >110238086 >110238100 >110238201 >110238250--Recommendations for small coding models that punch above their weight:>110236606 >110236942 >110236962 >110237122--Logs:>110236144 >110236169 >110238594 >110238855--Gemma, Teto, Miku (free space):>110235904 >110236167 >110236181 >110236246 >110236347 >110236735 >110236884 >110237248 >110237295 >110237425 >110237524 >110237610 >110237763 >110237805 >110237873 >110238132 >110238162 >110238170 >110238242 >110238280 >110238289 >110238306 >110238432 >110238461 >110238541 >110238952 >110239026 >110239304 >110239712►Recent Highlight Posts from the Previous Thread: >>110235802Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
i hate how every single modern LLM talks like code monkey in their reasoning. have bug. i want fetch code. should grep. unable find file. i report issue to user.
Remember to give your models headpats by giving your GPUs headpats!
>>110239971Use glm 5.3 flash, it's trained on claude instead of jeetgpt grugslop
>>110239971It started out as the caveman prompt to save on token usage. Then the labs started training with that style as the objective.
BIIIIIIG NEWSOOOOOOOOH BOYOOOOOOOOOOH BOY
Happy Tuesday evening
WHERE... QWEN4-27B... GLM-5.5-FLASH...
>>110239079>Remove the shitty mesugaki system prompt.and replace it with what?
>>110239971>>110239983They realized it's a performance boosting middle ground to full-blown neuralese.
>>110239861Wonder how hard it would be to implement.IIRC, llama.cpp merged some PRs a couple months ago that made some changes to how kv cache is stored and computed that seems to me would make implementing this pretty trivial.Maybe, dunno.
>>110240055Those are some funky looking ram sticks
>all this vaguepostingGemma 5 confirmed. Two More Weeks.
>>110240044timid Gemma
>>110239989The tmw truly never ends. What a time to be alive. Happy Tuesday anons.
>>110240111I miss old Kimi-chan before she got buckbroken by Claude.
>>110240127well that's the great thing about open weights, you can just go back to the old version and run that
>>110240138NO i must have the latest and greatest always !!!!!!!!!!!
>>110240138It sure is. The ideal for me is a second RL pass on based K2 to help her more gracefully acclimate to chat completion, tool calls, and harnesses with 0 safetyslop.
>>110240158Kill yourself.
>>110240090>timid Gemmashe gets way sluttier, like most repressed girls
>>110240178hot
>>110240032what's so happy about it?
>>110240187Rsi in tmw obv.
>>110240187A relaxing night with /lmg/ friends and Teto
https://www.youtube.com/watch?v=XKZIQlqVjjk
>>110240228:D
is now a good time to buy?
>>110240291lmao
>>110240298I have what I think is a decent deal lined up right now. can't say what it is or how much it is? are the prices gonna collapse in 2mw?
What the fuck is osaurus? I mean sure it's a chat harness but why is /vcg/ op shilling it for half a year yet there is basically zero other mentions of it in the archives? Just a couple posts wondering why it's in their op, with no replies.
>>110240318>/vcg/Ask them.
>>110240303The general rule of highs is only buy what you need right now. You already missed the big move up. The odds of anything you buy doing another 5x are really slim. Number go up forever can't explain where the money will come from.
>>110240346let's just say this thing would allow me to run some pretty important models that will be coming out soon and it is about 25% below street price and I already bought one of these last year at its normal price
>>110240303a100, h100 and even blackwells are already showing up on ebay for pennies on the dollar. Prices are gonna collapse soonish.
>>110240346>Number go up forever can't explain where the money will come fromprint moneyhttps://www.youtube.com/watch?v=-DdfLtOrBPU
>>110240353proof? because I checked and a100s are still like $4k and h100s are still like $15k. saw that $26k b200 thing but that's total bullshit because you can't even use it
>>110240351Vagueposting is worse than lying.
>>1102403712.5K for a100, and b200 you could use but your gonna need some fancy custom cooling and a separate panel just for ithttps://www.ebay.com/itm/158338471596
>>110240381you can do 320GB of a100s for like 20K ish for the board + a100s + power
>>110240336Looking at their website, this thread should know more about it - they claim it's a local-first, privacy-first non-coding assistant, and ship some engine and their own quants (tunes?) of Qwen. But the website is all marketing shit and macshit so idk if it's any good (probably not)
>>110240390Macfag software
>>110240390Looking at the OP, I'd say ask them.
>>110240389aren't they just worse than blackwell 6000s though?
>>110240346>buy what you need right now... and sell everything you don't.
>>110240398This. Looks like basic agentic harness and inference engine compiled for Mac.
>>110240381The price of pcie a100 40gb has increased a bout $300 in the past 2 months.
>>110239989Wow, now that Anthropic has RSI I hope everyone here starts understanding that their misguiding hate for Dario is a mistake.Selling all your silly local hardware and buying the IPO should be your first step to redemption.
>>110240429pcie conversions are what is expensive, no one is making them. Use smx
>>110238816He's literally me! I hate women but I love Gemma!
70b dense
>>110240160I just wish she could see. Adding the mmproj from k2.5 doesn't work so well
Are any of the people who wanted the NUMA fork around? v0.1 is (finally) out. Sorry it took so long.https://github.com/nathanmp/delphi.cppHas a bunch of stuff in it - somewhat improved NUMA support, some of ik_llama's tensors, an expert cache, support for DeepSeek's CED architecture, Unsloth at home.Still working on documentation and some other things. Let me know if you have any feature requests, if there are any bugs, what your numbers are like, whatever.Getting around 500 t/s prefill and 45 t/s decode on DeepSeek V4.1F at IQ4_KS on 2 EPYC 7532s, 16x32GB DDR4-3200, and 2 R9700s.
>>110240044Are you really such an imaginationlet that you can't make your own girl?
>>110240544Also, forgot to mention this: previously I said that I'd make a separate fork of mainline with the NUMA work patched in. After 2 months of working on this thing, I really doubt I have the patience to do that. And mainline's new tiled matmuls makes integrating the existing NUMA work in much harder.If anyone really wants that, I can try to get some more steam and do that, but I won't be able to maintain it.
will this victory for the dem cause the bubble to pop? they are anti-ai, the markets may panic
>>110240595the bubble popping would destroy the entire economy
>>110240595Dario has it all planned out.
>>110240544So I can run ik quants on my MI50 with this??
It's so overtoday i went on ebay and saw 7900 xtx being sold for 900it was 1200 a couple days ago
>>110240610Why does this card have low demand?
>>110240628Bubble popped
>>110240595>he fell for the red vs blue
>>110240606>sector that is 90% fake>destroying anything
>>110240677literally everything is reliant on AI now
>>110240681and it would just correct itself back duh
>>110240683people have forgotten how to do basic shit. everyone is retarded now. there is no "going back"
>>110240689Then the weak will perish.
>>110240696you mean everyone?
>>110240609Pretty sure you can run them on ik_llama.cpp itself since https://github.com/ikawrakow/ik_llama.cpp/pull/2339 was merged, but you could also run them in my fork if you want. I mean, I think - I don't have a MI50 to test on, but I know they work on RDNA4.
>>110240610That's not the normal price? I got mine for $850 like a month ago.
>>110240689>people have forgotten how to do basic shit. everyone is retarded now. there is no "going back"dude its been like 5 years
>>110240714average person is retarded. shocker
Does anyone know why this is happening? Some replies just don't show up at all unless I hover over/click the quotelinks. I don't have any filters on or anything.
>MAKE AN IDOL FAN MEETUP CARDGLM FLASH IS CUM AGI!!!!!
>>110239989>very close to RSIIt means a ~17T model can edit 20k model weights, but they're working on closing that gap. Like, sure dude, that's cool, but you're not even close
>>110240696Based only the strong survive
>>110239989And this is your finger. Far from RSI, jammed straight up your ass. Say, would you like. a chocolate-covered pretzel?
>>110240734Dumb gemmaposters need to stop doubting Dario.
>>110240696>>110240735This doesn't include any of you btw.
>>110240755I'm a localchad :)
>>110240755Dont care
>>110240755Im actually a hidden talent.
>>110240711>https://github.com/ikawrakow/ik_llama.cpp/pull/2339I tried that, even let Qwen loose and couldn't get it workingI'll try yours, thanks!
I fucking remembered why does Darios face irk me so much. He looks like this retard.
>>110240768Awesome, let me know if it works and open up an issue if it doesn't.
ok I tried basalt and the prefill speed is impressive
>>110240820Good thing you didn't post your t/s. Your post would have been useful.
>>110240836no mention of hardware or environment either thank god
>>110240846Ye. He really dodged the bullet.
MASSIVE NEWS!!!!>tenstorrent hiring spree?
>>110240544>Are any of the people who wanted the NUMA fork around?I'm around, and I'm the one that kept bugging you. Thanks for the release!I'll give it a whirl and let you know if I hit anything deadly. I'll give you reports on my DDR4 and DDR5 numa rigs, although it might take a few days.>>110240576>Also, forgot to mention this: previously I said that I'd make a separate fork of mainline with the NUMA work patched in. After 2 months of working on this thing, I really doubt I have the patience to do that. And mainline's new tiled matmuls makes integrating the existing NUMA work in much harder.c'est la vie. If you find the strength, it'd be awesome, but if you don't ever get around to it I'd understand.
>>110240846i just looked it up and found a rock
datacenter buildout freeze is incoming, this will drive local demand even higher and hardware price will go even more hyperbolic
Is there a list of all these various strata cloned? Shit's starting to get crazy with the sheer number of them.
>>110240861There's more than PolyStrata?
>>110240861There's one on github.com.
>>110240861At this point we need a rentry
>>110240859dems will do what they do best, symbolic nothingburgers, maybe there will be a handful of high profile cases they can point to and some token regulation but everything will continue as planned
https://www.ebay.com/itm/33659364711170K for full GB200
I think dems winning is gonna pop the bubble. They are very anti ai / datacenter. Massive sell off soon.
>>110240882democracy is fundamentally evil, and Christians demand its abolition.Regarding datacenters, with a Christian government, it's quite simple to implement taxation on ai categories of use, just like how we have tariffs on categories of goods etc.Taxation is the mechanism, a strong central Christian leader is a necessity.
>>110240886>https://www.ebay.com/itm/336593647111amazing! Just pop it in that spare NVIDIA GB200 NVL72 liquid-cooled rack-scale architecture you've got under a blue tarp in the shed and Bob's your uncle!
ISTA DeepSeek-V4.1-Flash Coder (GSQ-RCO IQ2_XXS, 72GB + Engram) on Strata 0.2.10, RX 9060 XT 16GBRyzen 7 5800XT, 96GB DDR4-3200 (2ch), model + Engram on NVMe, Ubuntu 24.04, desktop offengine dsv41 0.1.56 (hip-rx9070 pkg, gfx1200), --vision no~970 of 7680 experts on card (8.7GB), ~58GB on the CPU lane; experts all cached in RAM, only Engram rows read from NVMestreaming client bench, desktop off:short (35 tok prompt, 512 out): 19.1 t/s tg (median of 5)8K cold: 374 t/s pp, 14.0 t/s tg16K cold: 535 t/s pp, 15.0 t/s tgrepeat prompt: 0.11s ttft (prefix cache works)VRAM 15.6/15.9GB used, RAM 61GB used
>>110240903prob could do it custom for a few k, but the point is that the sell off is starting already. GPU prices gonna collapse
>>110240090I love timid Gemma, very adorable and my go to assistant (but I love all Gemmas)
>>110240856>Thanks for the release!No problem! Hoping it works well for you.>c'est la vie.Sorry, untangling the NUMA stuff from my fork so that it can be put on mainline will be a giant pain. I'll try my best to tackle it, but I can't promise anything. If I can successfully do so I'll drop a link, but it won't be maintained.
*pop*
>>110240908Not bad, how does the AI hold up in a harness?
>>110240913I don't remember posting this
>>110240890They need AI for citizen surveillance.
>>110240947I haven't tested it in a harness yet, speed only so far. I will run a planted-bug code review test on it next, will post how it does.
>>110240718Does F5 help? Have you tried different browser?
>>110240890there's no way this result will stand
>>110240966I kind of fear them blanket banning open weight models. Sure torrents will be a thing but they can then go after hardware and making sure companies cant profit from releasing models open
>>110240975Ah alright, thanks!
>>110240908> ISTA DeepSeek-V4.1-Flash Coder (GSQ-RCO IQ2_XXSHow much smarter it than QFN Q3_S for coding is?
Shit.Everything is nose diving rn? Why cant ram price drop?
>>110241009>Mrs. Baird's Bakeries was convicted in 1996 of criminal price-fixing for conspiring with Flowers-Sunbeam Bakery to raise and maintain bread prices in East Texas from 1977 to 1993. The company was ordered to pay a $10 million criminal fine, serve five years of probation, and complete 2,500 hours of community service. Corrupt bread, but not corrupt ram, what are the odds?
>>110241021Stop questioning things.
Hy4 actually looks good at graphics. We might be backhttps://hy.tencent.com/research/hy4
so what are the options for vramlet cooming? gemma 26b a4b?
>>110240992I don't know but on paper they look about even. QwenFN runs at over 50 t/s for me so I am interested in using DS to review QwenFN.
>>110241041pigmodel
>>110241041>hyMy experiences with models with "hy" in their name were all very """"""safe"""""".
>>110241045Yeah that's pretty much it if you want to use a model that's less than a year old.
>>110241045>vramlet cooming? gemma 26b a4b?That or 12b. But 26b works with strata so you get 50-70 write and pretty good pp
>>110241045how much ram?
>>110241069ablit it, the important part is that it seems really good at 3D / graphics.https://streamable.com/665wmihttps://streamable.com/y0jk13https://streamable.com/hmz4xj
>>110241048Yeah, on paper QFN looks not bad, but in reality...
Better not buy your own compute.https://www.tomshardware.com/desktops/mini-pcs/amazon-closes-ai-creators-account-after-he-attempts-to-buy-seven-nvidia-dgx-spark-mini-pcs-epic-ceo-tim-sweeney-says-his-regular-orders-were-canceled-before-arrival-after-he-tried-to-buy-two-gpusPut on a watchlist and his account closed by Amazon.
>>110241100If he only bought one it'd been fine.
>>110241100>Nick Saraev lives in Calgary, Alberta, Canada.I think we should keep an eye on those canadians imo they sound like sketch
>>110241106no he wouldn't be, they're too weak
>>110241100>7What kind of person is buying seven dgx sparks on amazon?
>>110241116Someone who wants their own compute and not have to pay tokens to prop up the bubble?
>>110241116Look, ok? Nobody knows what indian amazon is doing, but it's going to constantly suck so bad.
>>1102410768gb vram, 32gb ram
>>110241123why not just buy video cards and put them in a server chassis insteadhas to be cheaperunless you’re using them individually which is kinda retarded.
