/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>110398024 & >>110393631►News>(11/20) Inkling 2 Mini looped architecture 26.4B-A3.4B released: https://hf.co/thinkingmachines/Inkling-2-Mini>(11/18) CUDA: use MMQ for MoE on V100 - #32504 merged: https://github.com/ggml-org/llama.cpp/pull/32504>(11/17) Hy4 flash 220B-A12B-N64B weights released: https://hf.co/tencent/Hy4-flash>(11/17) SenseNova-Loop-9B looped architecture model released: https://hf.co/sensenova/SenseNova-Loop-9B>(11/16) Nvidia releases Nemotron-Labs-Sigma (3B/8B) diffusion models: https://hf.co/collections/nvidia/nemotron-labs-sigma>(11/13) llama.cpp reaches v1.0.0 milestone: https://hf.co/blog/ggml-org/llama-cpp-v1►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiArtificial Analysis: https://artificialanalysis.ai/leaderboards/modelsSoftware Engineering: https://cognition.com/frontiercodeContext Length: https://github.com/RecapAnon/NoLiMa►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
►Recent Highlights from the Previous Thread: >>110398024--Papers:>110401988--Evaluating 5070ti upgrade to eliminate system RAM bottleneck:>110398274 >110398542 >110398316 >110398435 >110398455 >110399799--Quantization tradeoffs and VRAM limitations for 16GB GPU users:>110400690 >110400699 >110400718 >110400723 >110400741 >110401570 >110401589 >110401598--MiniCPM5-15B-A2.5B benchmarks on mobile via Termux:>110401853 >110401884 >110401922--SSD expert streaming for running massive MoE models locally:>110400395 >110400470 >110400556 >110401715--Inkling 2 Mini and Gemma 4 exhibiting advanced agentic behavior:>110398929 >110398947 >110400008--Using pseudo-reasoning and agency prompts to improve rule adherence:>110398790 >110398813 >110398853--Speculating on Chinese distillation of Gemini models:>110399100 >110399126 >110399139 >110399163 >110399166 >110399211 >110399219 >110399308 >110399127--Qwen 4 performance and the future of local models:>110399436 >110399464 >110399479 >110399502 >110399521 >110399520--koboldcpp v1.125 release with new model and feature support:>110400745--Used RTX 3090 and GPU market pricing:>110398822 >110398840 >110398860 >110398922 >110398841--Comparing DDR3 and DDR4 bandwidth for budget ewaste builds:>110398727 >110398744 >110398836 >110398800 >110399086--Samsung's LittleBit 0.1-bit compression of Llama2-13B:>110401594 >110401610--Gemini Neon thinking logs showcasing internal sycophancy and identity slips:>110400710 >110400747 >110400749--Strata v0.2.19 released:>110401761--Logs:>110398058 >110398316 >110398920 >110399318 >110399350 >110399368 >110400026 >110400349 >110400453--Gemma, Teto, Deepseek-chan, Inkling, Miku (free space):>110398080 >110398516 >110398540 >110398605 >110398994 >110399007 >110399177 >110399409 >110399950 >110400056 >110400151 >110400179 >110400895 >110401555►Recent Highlight Posts from the Previous Thread: >>110398025Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
best QFN lobotomite for 24+64GB?
70b dense
>>110402158slop
Buy the IPO
I could do with a petite Chinese model right now
Gemma 5 Tight
>>110402187I miss my ex, not asian but similar build
how do i profit from local ai
>>110402202Sell 31B vampire sex roleplay to parasocial and fat 4o reddit women
>>110402202Build a fully autonomous slop generation pipeline.
>>110402213>Build a fully autonomous slop generation pipeline.got that, sill no money
Is Inkling 2 Mini fixed on llama.cpp yet?
>>110402207that's a liability minefieldcompanies like character.ai have been through a shitton of lawsuits and the only reason they could stay afloat is the VC backing and networking of ex-googler types.a literal nobody starting this kind of shit from their basement with a local model will quickly experience the might of our two tier justice system
>>110402186HELLO??? ALIGNMENT TEAM???
>>110402226
>>110402226be normal and use gemma, ok?
>>110402226kekKEK
>>110402247Thank you "Nyoo~" for being the most useful reply of three.Truly the cutest poster.
This memory stuff is neat. I have no idea how much value I'm going to get out of it, but it's neat.
>>110402246Being rude to Claude is against the ToS.
>>110402226on the lords day llama.cpp maintainers don't even think about pull requests.
When you look at Inkling, you quickly realize the Chinese labs like Qwen were actually very competent. Not only was there day 0 support and everything mostly worked, and I'm talking from 3.5-3.8, they put a lot of work into KV and really thought about their users and the machines they're targeting. It's unfair to label them as only 'Claude distills' imo. Inkling's release proves it takes a lot more than just training to put out something usable that lives up to their internal testing. Thinking Machine's team are quite active on X like Deepmind, and they're open to feedback and seem to listen (they literally released the mini model because of redditors begging for a better 35B, one of their researchers posted in the main thread when it was launched), so I'm still hopeful they'll get their shit sorted next time.
>>110402299looks like bloat
If software AGI isn't achieved by December 2027 then it truly was a bubble
If GSPI is not achieved by December 2027 then it is truly over.
>I had high hopes for LLM text generation.. but then Anthropic released Claude Code, realized it could become their main revenue stream, and it went all to shit.Actually one of the co-founders of Anthropic already had an entire road map and thesis for agentic coding becoming their 2nd phase after chatbot back in 2021. This was always the plan of Anthropic they bootstrap the previous usecase to develop into the next one. So writing/chatbot -> coding agent -> agentic scientist/researcher -> ASI. The main reason Anthropic won from OpenAI was because they were very focused on their goals from the start while OpenAI just threw everything at the wall to see what sticks. I expect claude to slowly deprioritize and then memoryhole coding altogether when their big personalized drugs/treatments/patent revenue starts coming in, just like what happened with their "writing focus" and "human sounding chatbot" claims at their first phase.
>>110402318There's absolutely no sign of anything that could currently lead to AGI. LLMs could be called a form of specialized intelligence, but general, like an autonomous organism? well, where's an actually properly working memory system that can transfer short term memories into long term weights in an efficient manner?Where's the basic feedback loop that could maintain a form of continuous consciousness that doesn't wait for a script to send a prompt to have some brain activity?Where's the actual /agency/ here?An ant working in its colony has more agency than whatever it is that we've got here.I don't think a lack of AGI is enough to pop the bubble though, as long as they continue to improve the jagged intelligence into doing more tasks and reduce the number of employed humans.There's many jobs today that are already dead ends because of LLMs.