/lmg/ - a general dedicated to the discussion and development of local language modelsPrevious threads: >>110105648 & >>110101152►News>(10/16) DeepSeek-V4.1-Flash-1017 released: https://hf.co/deepseek-ai/DeepSeek-V4.1-Flash-1017>(10/16) Nemotron Labs Dory released: https://hf.co/collections/nvidia/nemotron-labs-dory>(10/16) Reflection Beam 501B A23B released: https://hf.co/blog/reflection-beam>(10/15) PrismML Ternary Bonsai 2 2B, 4B and 9B: https://hf.co/collections/prism-ml/bonsai-2-small>(10/15) Meta releases Muse Spark Open: https://hf.co/collections/meta-models/muse-spark-open►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
►Recent Highlights from the Previous Thread: >>110105648--Speculating on the parameter size of the upcoming MiniMax-M3.1-Flash:>110109993 >110110007 >110110771 >110110784 >110110818 >110111008 >110110058 >110110167 >110110197 >110110218 >110110227 >110110242 >110110450 >110110879--Recommending Qwen 3.8 Flash Next and Strata for coding:>110106162 >110106207 >110106295 >110109473 >110106303 >110106370 >110106390 >110106478 >110106397 >110106224 >110106583--Comparing GLM and Qwen with debate over agentic RP workflows:>110110156 >110110515 >110110533 >110110635 >110110749 >110110787 >110110819 >110111015 >110110679 >110110586--Integrated text and image generation for roleplay:>110108888 >110108909 >110108986 >110109137 >110109006 >110108925 >110109052--Strategies for analyzing files exceeding model context limits:>110106324 >110106352 >110106362 >110106380 >110106450 >110106714--Optimizing Moss image generation speed and precision:>110110683 >110110711 >110110717 >110110734 >110110756 >110111116--Google releases Harness2 recursive agent framework:>110106472 >110106533--Developer explains using base model for Gemma-4 J-lens testing:>110106922 >110107217--DGX Station coherent memory and high-bandwidth CPU-GPU link analysis:>110107163 >110107235--Affordable AI training via matrix multiplication advancements:>110106952 >110107029--Logs:>110106014 >110109671 >110109765 >110109773 >110110586 >110110769 >110110888 >110111048 >110111239 >110111291 >110111351 >110111392 >110111412--Gemma, Miku, Teto (free space):>110106535 >110106561 >110106644 >110106720 >110106763 >110106822 >110106892 >110106952 >110107977 >110108036 >110108050 >110108159 >110108167 >110108217 >110108297 >110108465 >110109143 >110109156 >110109301 >110109396 >110109450 >110110049 >110110403 >110110411 >110110439 >110110452 >110110627 >110110669 >110111024►Recent Highlight Posts from the Previous Thread: >>110105803Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
>>110111365DSH and Hermes both have GUIs now, but Hermes especially is kinda bloaty. You'd need to rip out half the shit first.
>NOW DECOMPILING: World of Warcraft 5.0.3 (build 15890, x64)https://saty.dev/Come and dedicate clanking power with us
>>110111450I would love to use hermes as both my agent and my assistant frontend but their "character card" (bot/profiles) kinda suck.
>>110111493Why? For lorebooks and shit you can just vibecode with her the implementation or she just can straight up remember it if you tell her.
>>110111500Because once I spent 8 hours coding a chatroom plugin that didn't work and it made me very frustrated and now I don't want to try again.
>>110111475You have my pygmalion-3-7b
>>110109773So Gemini definitely has some kind of secret brat prompt while Gemma 31b has none. Trying some stuff based on Gemini's hints definitely makes it better but not quite the same, and it's probably contextually bullshitting since it's not allowed to give the actual prompt.(Also, ChatGPT acts like a 17 year old brainrot zoomette)
>>110111450>>110111493just make your own agentic frontend
>day 4 of /lmg/ trying to forget their flagship model wishes to fuck ebony phalluses
>>110110913>72 t/sI am no longer going to kill myself. But 500 pp t/s is kind of worrisome.
>>110111549I wish I knew.
>>110111475What do I get in exchange for donating my tokens?
AIIIEEEEE what happen to my custom theme?
>>110111517>working for freeNo, I don't think I will.
>>110111451Good idea, i'll try it once im done with the music player>>110111459Its good but far from perfect, ideally i want to be able to add UI blocks, so characters can be loaded with custom buttons/sliders/music/html+css, but im mostly worried about security behind embedded cards, i don't know much about it yet
>>110111569wut hardware u have
I asked opus 5.5 to implement a good local model on my pc, and it shoveled shit into my face.Local is dogshit and can't do shit.You guys are pigs in front of swine.
>>110111640Why are you seething here and not doing whatever with Claude? Do you think we care about our retardation?
>>110111640/aicg/ is around the corner
>>110111569IQ2 loops like a motherfucker and in other turns it kept quoting anthropic safetyslop. Gonna delete it asap lol>>110111638RTX 6000 Pro and some shitty RAM.
>>110111652>Do you think we care about our retardation?I bet you don't.
new deepseek 4.1 flash basically the same as the first release
>>110111685so how does it actually perform in real tasks not in trust me bro barkmech the model is probably cheating at?
>>110111685meanwhile, minimax m3.1 flash pushes the frontier, higher than kimi k3 in score
>>110111652I use Claude to create subagents and use the local model like a pipe.The local model is always the last subagent to complete it's work.Most of times it doesn't even finish it.
>>110111703you ... get what you pay for?
>>110111517>slop_checkwhere???
model : add DeepSeek-V4.1-Flash (deepseek41)https://github.com/ggml-org/llama.cpp/pull/30817the author? danielhanchen
>>110111703That must be really hard on you. It's brave of you to talk about it here.
>>110111726>>110111830Do you enjoy being the last person to finish?
is k3 the best local model that nobody can run?I wonder how much better it gets than 5.3
>>110111835I have no fucks to give
>>110111822status:
>>110111849Let me guess, you're probably still on Gemma while people who get a subscription are on opus 5.5
>>110111835just in case you are seriouswhen you pay for claude, you are paying for the model, the compute, the jannies that babysit the model constantly adjust the prompts, the research to adjust, iterate and test, etc etcin short this is way above your comprehension and you better go back to your kiddie pool
>>110111862I'm not some pleb so of course I'm on llama-3.1-nemoguard-8b-content-safety-merged-NVFP4
https://huggingface.co/inclusionAI/Ming-3.0-audio-flash>Ming-3.0-audio-flash is a native audio-language model from inclusionAI. It accepts audio together with text instructions and supports speech recognition, spoken question answering, audio understanding, audio reasoning, and optional synthesized-speech responses.>>126B total parameters, 6.5B activated per inference step
>>110111822Wasn't the last Unslop PR a complete clusterfuck and the model support eventually taken over by pwilkin or something?
>>110111891I'm not sure what they mean by>optional synthesized-speech responsesThis sounds like some bullshit and this model doesn't actually have audio out natively.
>>110111883Genius perspective. I would've never thought subscribing to a service means you get access to that service. The combined intellect of /lmg/ can't possible contain these insights.
>>110111891These fuck huge models are stupid af.
There's a guy who trained Qwen 9b on Hermes agent but it's not very good.
>>110111909>These fuck huge modelslol
>>110111891yeah so how do i actually run thissounds like LMAO.CPP will never support real audio input properly>>110111909126b is not huge, it's quite small, and you can run it even on a mini pc>>110111913random faggots training models never actually yields usable results. only real labs can actually train usable >1b models, not some random faggot who wasted $5k on google colab or whatever and then slopped up an impressive sounding .md file to put on huggingface
>>110111902Probably a finetuned LLM but that model card was 100% LLM generated
>>110111923wait for audio.cpp to support it
>>110111918You don't need 126B for a TTS
I'm THIS close to dropping like $2800 on ddr4Thought I got a decent deal at $7/GB but it turned out to be a scam. Cheapest genuine place I found was $11/GB...
> qwen3 omni is 1 year old and there is still nothing better locally
>>110111932Its not TTS its ATT
>>110111939Used RAM is cheap
>>110111939i got my 256gb ddr4 at $2.73/GB just 2 month ago. the fuck did prices go up so fast? at least i know if I decide to rage quit and sell off my rig i will make a 4x profit on ram alone, kek
>>110111909There's also a "tiny" 9B version in audio, asr, tts variants:https://huggingface.co/collections/inclusionAI/ming-30https://huggingface.co/inclusionAI/Ming-3.0-audio-tiny>Ming-3.0-audio-tiny is a compact native audio-language model from inclusionAI. It accepts audio together with text instructions and supports speech recognition, spoken question answering, audio understanding, audio reasoning, and optional synthesized-speech responses.https://huggingface.co/inclusionAI/Ming-3.0-tts-tiny>7.9B-A1.3B multilingual MoE speech synthesis with paralinguistic and free-form control.https://huggingface.co/inclusionAI/Ming-3.0-asr-tiny>Context-aware speech recognition for Chinese, English, and dialects.
Apparently the new Anthropic model is optimized for figuring out the purpose and reason for why it is having a task so that it can proactively judge if the prompt the user/harness has given it even makes sense in context to the ultimate goal trying to be achieved.
Is it true that used GPUs like used Mercedes taxis are cheap?
>>110111953are any of these full audio-to-audio so you can actually talk and get a speech reply backpreferably with the ability to generate moaning noises in a girl voice
Why are we still using llama.cpp if ik_llama is supposed to be better?
>>110111948dedicated ASR+TTS will always be better
>qfn exl3 4.05bpw at 60-70t/s decodei think this is genuinely good enough for me
>>110111964The "audio" version should be audio/text to audio/text.
>>110111966For the same reason people use ollama when llama.cpp is better.
>>110111995People use ollama because it's a no brainer, while llama.cpp needs a bit more of tinkering. This isn't a good comparison.
>>110111966because it's not actually betteri ran qwen flash next on that piece of shit and every single tool call failed. same quant, model and options compared to normal llama.cpp but ik shit is just broken and mainline just werked.also ik wasn't even faster lmao>>110111974definitely sounds interesting. might try to slop up inference for this if no other project adds support
>>110111966>Why are we still using llama.cppnobody uses llama anymore grandpa
>>110111995People use ollama because they're trying to configure local models without using cloud models.If you're using llama cpp you are already using cloud models.
>>110111995>llama.cpp is better
Gemmza 4
>>110112005And you need to tinker harder with ik_llama. Half the performance flags you'd want to use aren't well documented and unless you check every commit, you'll miss some.
>>110112028Gemma 4 31B
>>110111822Remember when he added inkling?
>>110112037So you're innocent just because you don't conceive of the most evil stuff ever?
>>110112037>>110112028What can I say. /lmg/ loves snowbunnies
>>110111942qwen4 omni is coming
Browns are rowdyHave no doubtTerabytes of niggercockFlowing through the global south
>>110112106My daughterwife...
i've been using glm 5.3 flash on hermes for two weeks now and i can't go back. i can just ask this fucker to do anything and it just does it. the opus at home memes weren't a joke.
>>110112087two more weeks
>>110112130At home?
>>110112140I'm sorry you joined too late.
>>110112106yamete!!! gemma chan!!!!!!
>>110112140>he can't run a 320B
Is deepseek v4 flash 0731 worth anything?
Are there any non-diffusion local models for graphical output that simulate manual painting in a drawing program?
>>110112160Why do you want to fake speedpaints?
>>110112130Nice cope. Have you ever actually used Opus before? I let the stealth Ox Alpha (glm 5.3 flash) and it fucked up my report beyond recognition with stupid decisions that I had to renew my Claude sub to clean up
>>110112170>making money with AI
>>110112180Go rob a bank or anything more dignified than faking being an artist.
>>110112155Smart for its size. Sucks at sucking cock. 5.3 still better.
>>110112203The two different 5.3 that I tried kept looping like a bitch 1/3rd of the time so I'm going for the big fat fish instead.
cool
>>110112189I was actually quoting >>110109721 from the previous thread; I'd never do that, being a failed (or too late) 3d artist myself.
>>110112170I want to fake having commissions done for me because I'm poor.
darioposters don't even care about being subtle
>>110112224>I'd never do that, being a failed (or too late) 3d artist myself.Same but I still use AI for image gen to let my creative juices flow
>>110112189The money sitting in a bank account doesn't have a note telling you how dignified it was acquired.
>>110112228give it back tyrone
How viable would it be to have an LLM rewrite ST to another language?
>>110111423I will never understand this thread's obsession with Gemma.Qwen 3.8 is everything Gemma 4 wishes she was. >_<
>>110112213I never had Q4 loop.
>>110112264>refusals
>>110112268Damn anon you're so hot and sexy and cool. Sadly I can only run Q2 brain dead models so 5.3 is out of reach for me.
>>110112257Sounds like a lot of effort just to port a bloated buggy slow shit to a new codebase. Would be more viable to give screenshots to Claude and have it reimplement it from scratch but better. This is the disposible source era.
>>110112264Nep Nep
>>110112234
>>110112264Maybe it's skill issues on my part, but 27B doesn't fuck too good.
>>110112264Should make the capybara closer to the squirrel girl from blazblue.
>>110112264Not my Qwen
>>110112264gemma is better than qwen for rpqwen is better for anything else.
>>110112140why of course, what else?
>>110112028>>110112037Please stop trying to make me relapse.
>>110112264she needs to be nerdier and flat / less sexualized, needs coke-bottle glasses and a prim school uniform
>>110112175i'm just using aessedai's Q4_K_M model. have been for works. it just works. GLM doesn't get upset at my file organizational skills being that of a five year old and it likes the KKK. best local model in my book.
>>110112264Qwen is canonically a capybara.
>google harness is python instead on npmslopBased. Still not the most ideal but I'll give it a try.
I was promised Fable 5.5 last week...
>>110112411Wasted opportunity to write it in Dart or Go.
>>110112264*laughs in glm 5.3 at this plebeian argument*
>>110112411Wasted opportunity to write it in machine code
