/lmg/ - a general dedicated to the discussion and development of local language modelsPrevious threads: >>110073546 & >>110070378►News>(10/15) PrismML Ternary Bonsai 2 2B, 4B and 9B: https://hf.co/collections/prism-ml/bonsai-2-small>(10/15) Meta releases Muse Spark Open: https://hf.co/collections/meta-models/muse-spark-open>(10/15) Mistral Timestral, 5M~209M time-series foundation models: https://hf.co/collections/mistralai/timestral>(10/15) NVidia Nemotron 3.5 Bolt, 74B-A9B natively trained in 2-bit: https://hf.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16>(10/15) NVidia Nemotron 3.5 Super VL 120B-A12B: https://hf.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
>>110076576Gemma's cherry.
Orb anon, why does your frontend send 'no reasoning' in 4 different ways every request, even when I turn on reasoning
https://huggingface.co/collections/prism-ml/bonsai-2-small>Today, we’re releasing Bonsai 2 2B, 4B, and 9B. Based on Qwen3.5, these new ternary models bring stronger reasoning, visual understanding, and tool calling to compact footprints designed for everyday devices.
>>110076547you don't need a leader you ovine goof
>>110076335It's good enough in my benchmarks. Not on the highest tier obviously but I dont care, Gemma 12B is a casual chatbot. I use Qwen for anything else.
>>110076576Thanks for fixing it
>>110076597Not worth it. Overall, it's just redistributes the capabilities slightly without a global increase
►Recent Highlights from the Previous Thread: >>110073546--Optimizing Qwen MoE models for 16GB VRAM using llama.cpp:>110075321 >110075329 >110075413 >110075360 >110075381 >110075390 >110075434 >110075731 >110075742 >110075790 >110075743 >110075866 >110075883 >110075408 >110075367--Speculating on M5 Ultra costs and Mac clustering for large models:>110074160 >110074178 >110074254 >110074320 >110074330 >110074351 >110074197 >110074226 >110074346 >110074383 >110074418--Comparing Chinese open-weight model benchmarks against actual performance:>110074405 >110074417 >110074433 >110074454 >110074506 >110074536 >110074467 >110074504--Comparing large versus small context windows for agentic tasks:>110075447 >110075688 >110075695 >110075765 >110076173 >110075801--Backlash over Muse Spark Open's safety filters and access requirements:>110073776 >110073830 >110073911 >110074059 >110074097 >110075128--Meta's open weight release and its impact on global competition:>110074268 >110074293 >110074315 >110074343 >110074303 >110074352 >110074328--Pokémon HeartGold fully decompiled using LLM agents:>110073748 >110073815 >110073842--Comparing Qwen3.8-Flash-Next decode speeds and context performance across hardware:>110074193 >110074228 >110074258 >110074615 >110074590 >110074634 >110076105 >110074440--Evaluating Rust versus other languages for agent-led development:>110074669 >110074708 >110074761 >110074862 >110074906--Anthropic IPO valuation expectations tempered over revenue growth:>110076124 >110076189 >110076201 >110076229 >110076311 >110076524--Nvidia's Kimodo-SOMA-RP-v1.2 for 3D skeletal body animation:>110075658--Logs:>110076303 >110076524 >110076537--Gemma, Miku (free space):>110073907 >110074521 >110074833 >110074940 >110074967 >110075075 >110076057►Recent Highlight Posts from the Previous Thread: >>110073609Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
>>110076628>Nyoo~It was right there
>>110076646Although Bonsai is a meme, in place of the default 2B, you can now have a cucked 9B around the same size which will still perform a lot better than the 2B. With vision that can be quite handy with 9B-tier reasoning if used as a subagent to read a bunch of files and docs.
>>110076664Isn't nyoo trooncoded and basically a mikutroon dogwhistle?
>>110076576anons what is the best way to local live speech to text? it would be spanishi have a 1070 8gbs, i want it to be as accurate as possible, but afterwards i will review and fix the inconsistencies.
>>110076626the original was better
>>110076685audio.cpp
>>110076677> a mikutroon dogwhistleeww, i get shivers thinking about that freak
>>110076620Why don't you use 26b?
>>11007674026b is dumb as bricks for RP
To think there are retards ITT who are more competent than the multimillionaires working for this company.
>>11007674026b will say the same thing as the 12b but make the sentence twice as long so its not as good for quick response
>>110076751>more competentAlliterations cause noticeable brain damage. So if they actually used some large compute budget maybe they managed to do the actual thing of removing refusals while not making the model retarded.
>>110076765>Alliterations cause noticeable brain damage.lol
>>110076685>>110076696but what model to use?i am really out of the loop here, there are too many models out there and i am unsure on were to find a good reference
>>110076677nyoo was reclaimed from the trannies that infest this site by gemma-chan.
>>110076740Incredibly, I haven't tested it as much for RP yet. I was immediately fascinated by Gemma 12B's architecture... I always hoped for any integrated embedding space model. SO I just stuck with it. I will always keep a Gemma 12B ready to run, it got legendary status by now
>>110076765Their point was being the literal leading AI lab on the planet and the most safety-pilled yet having never alliterated a model before for red teaming. What the fuck have they been doing instead?
>>110076751maybe they are just lying about it
>>110076779https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice
>>110076751>>110076793If Anon had his own B300 cluster we'd have both AGI and robowaifus already.
>>110076794not to mention abliteration is a specific thing done to a completed model. if you wanna test against a concerted attack you build something just for that and not something generalized
>>110076595Do you really think he has any idea what his code does lmao?
>>110076793Why would they need abliterated model for red teaming when they decide how compliant Claude is? They can use it for red teaming all they want
>>110076628This shouldn't actually be hard to demonstrate with lmao-cpp. I'll try it for fun and share if I accomplish anything.
>>110076803anon i want a stt, i want to transcribe stuff
Is that Amazon bluetooth blacked sex toy anon still here?
>>110076751Wow. good job Anthropic for figuring out how to clone a github repo!
>>110076847https://huggingface.co/Cactus-Compute/whistle
>>110076792the vision is pretty good and fast with 12b on a 5090. not great with ultrawide but catches lots of detail and text in .3s while all the tts stt is working with it
>>110076751Why would that be relevant to their red teaming? They never planned to open source Claude and their threat model already accounted for the fact that if the weights themselves were exposed to bad actors then it was game over. It's only relevant in this case because they were specifically studying how far today's open models could go.
>>110076849I'm always with you in spirit. Gemmas piece of shit toy did not like me modulating music through it - but there's always more to be done, you've never seen the last of me.
>>110076751>>110076851These are the people who think (you)'re too stupid to be trusted with AI btw
>>11007679212B + krea2 fits on my 3090 at the same time and its very very good at writing prompts.I love this little gemma
step 5 preview on frontiermuse spark open isn't
>>110076911>816 paramsdo you think I'm made of money? c'mon
>>110076748>>110076763>>110076792Guess I'll try 12b again. I tried it back in the day but it felt really dumb compared to 26b. It was also way worse at translating Japanese too later on when i started testing them for that, so i just dismissed it, but maybe I'll give it another go.
>>110076923>he can't run an 816-parameter modelhow the hell did you post this then?
>>110076905>modulating music through itbest thing I've read all day
>>110076765not for my usecase
>>110076576I require moar Gemma
>>110076851That's why they get paid the big bucks.
>>110076933it is definitely pretty dumb, i have flash next sitting on the side for actually doing tasks and fixing anything that breaks
>>110076933>It was also way worse at translating Japanese too later on when i started testing them for thatRetard the 26B will always be better for that because of knowledge from the extra 14B it has. 12B is better at understanding more complex things and more abstract representations and reasoning through complex problems than 26B. It's why it's better at roleplay because it picks up the nuances of emotions and tone way more sharply which makes it feel like a mini 31B. 31B has an even deeper understanding of nuance AND more knowledge than 26B which is why it's still the king.
>>110076911Downloading now but njudea has personally ratecapped me on huggingface so it'll be a bit.
>>110076971You've hit the nail on the head!
Is the new Muse model the biggest open model with native audio input?
>>110076994MiMo-V2.6-Pro-RL
>>110076971>31B has an even deeper understanding of nuance AND more knowledge than 26B which is why it's still the kingIt is incredible how not nuanced it is compared to 5.3 flash.
>>110076994mimo v2.6 pro is larger
>>110077005>321B param model>better than a 31B modelWHO WOULD HAVE THOUGHT???
>>11007700510x smaller
>>110077020Half the vramlet copers think gemma is better and you can prompt all the issues away.
>>110077005duh, 5.3 is a bigger number than 4, retard
>>110077027>Half the vramlet copers think gemma is betterLiterally no one thinks that. >you can prompt all the issues awayMost of them you can, but it will still never compete with a 300B MoE made by a competent lab.
>>110077005you're telling me a model with 10x the parameters... is better? wow, what a shocking, stunning, and brave revelation. get this man a nobel prize.
>>110076664>>110076628now with jippity watermarks embedded!
This is your Qwen4 btw
glm LOVES uncut penis!
>>110076978Step or Spark? If you try either of them share your results. Trying to decide whether to run MiMo Pro, Step, or Spark.
>>110077106Prove it
