/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>109211479 & >>109205164►News>(07/06) Hy3 officially released with 295B-A21B & 3.8B MTP: https://hf.co/tencent/Hy3>(07/04) LongCat-2.0 1.6T-A48B released, trained on AI ASICs: https://hf.co/meituan-longcat/LongCat-2.0>(07/03) Orb Anon releases purple prose classifier and ablater: https://github.com/OrbFrontend/Chartreuse>(07/03) Leanstral-1.5-119B-A6B released: https://hf.co/mistralai/Leanstral-1.5-119B-A6B>(07/01) Nemotron-Labs-TwoTower released: https://hf.co/nvidia/Nemotron-Labs-TwoTower-30B-A3B-Base-BF16►News Archive: https://rentry.org/lmg-news-archive►Glossary: https://rentry.org/lmg-glossary►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsAlpha Calculator: https://desmos.com/calculator/ffngla98ycGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllm
►Recent Highlights from the Previous Thread: >>109211479--Comparing Gemma 31b and Qwen 27b with detailed RP prompts:>109211867 >109211877 >109211896 >109211910 >109211930 >109211988 >109212061 >109212108 >109213365 >109213403 >109213436 >109213498 >109213570 >109213653 >109213759 >109213986 >109214094--GLM-5.2 performance logs and architectural inefficiency discussions:>109214755 >109214969 >109215006 >109215023 >109214992 >109215326--Hardware recommendations for running DeepSeek V4 Flash locally:>109212932 >109212974 >109213215 >109213515 >109213368 >109213380--Debating a chart projecting the lag between frontier and local models:>109213715 >109213866 >109214630 >109215085 >109215148 >109213869 >109213895 >109213908 >109214106 >109214154 >109214188 >109214229 >109214266 >109214357--Implementation of AI character expression systems:>109212766 >109212782 >109212804 >109212799 >109212818 >109213707 >109212903 >109212955 >109213003 >109213119 >109213092--Using Gemma 4 and Qwen 3 for automated eroge translation:>109212174 >109212262 >109212281 >109212331 >109212369 >109212399--Heretic PR and the cost and effectiveness of abliteration:>109211489 >109211500 >109211533 >109211536 >109211549--Using LLMs for robotics, microcontroller programming, and circuit analysis:>109212895 >109213134 >109213186 >109213301 >109213343 >109212917--picolm project for running 1B LLMs on low-resource hardware:>109211517 >109213483 >109213508 >109214000--China's regulations targeting anthropomorphic AI interaction services:>109211717 >109211916 >109211951 >109214159 >109211972 >109212222--Speculation on Meta's leaked Watermelon model catching up to GPT-5.5:>109211964 >109211982 >109211990 >109211999--Logs:>109212108 >109212766 >109214916 >109215011--Miku, Teto (free space):>109211734 >109211806 >109212064 >109212272 >109212963 >109214943►Recent Highlight Posts from the Previous Thread: >>109211481Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
Gemma's good at translation but has anyone tested notoriously difficult stuff like Monogatari?
>>109216038
>>109216018I really didn't make it into the highlights? -_-
90B A5B MoE
So how's Kimmy 2.7? Another anon recommended K2-Instruct for RP to someone a few threads ago, is the older model actually better?
do any of you use these? do they work well?https://huggingface.co/gghfez/gemma-4-31b-it-control-vectors
>>109216128K2.7 is the only one that's even on the table with K2-Instruct for RP due to reduced <think> blocks. K2 is still all I've ever needed in RP and is way less safetycucked than 2.7 and unless you specifically need to solve benchmarks in-character, there's not a lot that 2.7 does that 2 can't.
>>109216164who do you use for quants? ubergarm? I want to keep the ability to use both, so I am hesitant to download his quants
>>109216214Ubergarm is usually a safe pick. Bart if he's not available.
https://huggingface.co/bullerwins/DeepSeek-V4-Flash-GGUFHow usable are these Q3 quants? Seems strange that bartoski and unsloth refuse to provide these quants when they enable using them on a single 128GB unified RAM device.
>>109216162I use his Kimi-Chan and GLM-Chan ones. They're really cool.I didn't know he made them for Gemma-Chan.
>>109216100There's an upper cutoff once a post has too many replies. Betting that cut it. Just re-post it.
>>109216273>tfw I can only run the IQ2_S
>>109216273https://github.com/antirez/ds4there's a lot of people using 2 bit quants. Q3 is practically flaunting.
>>109216273>How usable are these Q3 quants?I haven't tried them but in general Bullerwin does high effort quants Ubergarm/AesSedai style.I've been using the IQ3_XXS of this: https://huggingface.co/tarruda/DeepSeek-V4-Flash-GGUF and it's been fine.
>>109216273If those L39 etc in your graph mean "I used a higher bitrate for Layer 39", then he's indirectly benchmaxxing his perplexity calibration dataset.
>>109216273Is there any reason to ever run the Q4 if the Q8 isn't that much larger?
since gemma's reasoning is steerable (you can even get it to think fully in character for rp) would in theory be possible to instruct it to use a better reasoning algorithm that wastes less tokens and makes it think smarter?
>>1092163601. Yes, probably2. If you figure out how, apply for a job at any major research lab
>>109216360Yes. Just add "Think like a caveman".
>>109216273I'm using IQ3_XXS-L21 and it seems to work fine for megoing below 100gb causes serious lobotomy to the model so no wonder antirez quants felt like shit
>1.5 mb/s download throttleHuggingface has sneeded me for the last time.
>>109216017
>>109216427Where's the bulge?
>>109216429My pants?
i just want a model that doesn't fucking overthink why is this so hard...
>>109216384Yep, solved it. Hire me altman.
What's /lmg/ favorite agentic cli or TUI for ERP?
>>109216427Living in Teto's territory
>>109216466SillyTavern
>>109216451Does it actually make it smarter though? Grug speak seems like a sidegrade at best.
>>109215801>>ready to die>>hasn't even had the chance to impregnate a robololi>quitterheretic-resistant, in character safety cuck
>>109216501Not really, but it's not dumber either which tells me that reasoning is a complete meme.
>>109216533Gruk isn't dumb though. Language is about the concepts behind the words not the words themselves. Prosekeks btfo'd as usual.
>>109216480Always.
>>109216533>>109216544If nothing else you managed to abridge thinking in an afternoon the same way that took Moonshota 4 Kimi revisions to do.
>>109216480>Teto's territoryFucked up so bad.
