/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>108416874 & >>108410115►News>(03/17) Rakuten AI 3.0 released: https://global.rakuten.com/corp/news/press/2026/0317_01.html>(03/16) Mistral Small 4 released: https://mistral.ai/news/mistral-small-4>(03/11) Nemotron 3 Super released: https://hf.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16►News Archive: https://rentry.org/lmg-news-archive►Glossary: https://rentry.org/lmg-glossary►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://livecodebench.github.io/gso.htmlContext Length: https://github.com/adobe-research/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsAlpha Calculator: https://desmos.com/calculator/ffngla98ycGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllm
►Recent Highlights from the Previous Thread: >>108416874--MSA-4B outperforms GPT-4.1 and Qwen models in long-context benchmarks:>108418758 >108418791 >108418819 >108418842 >108420734--Advanced virtual companion setup with home automation:>108417587 >108417605 >108417650 >108417676 >108417683 >108417727 >108417745 >108417769 >108417811 >108417879--Mistral CEO proposes revenue-based content levy for AI companies in Europe:>108417643 >108417668 >108417678 >108417740 >108417747 >108421003--MistralAI CEO proposes AI content levy in Europe:>108418980 >108420788 >108420874 >108420907 >108421234 >108421283 >108420826 >108420878 >108420879 >108420951 >108421015 >108421176 >108421248 >108421305 >108421306 >108421482 >108422209--The End of Coding: Andrej Karpathy on Agents, AutoResearch, and the Loopy Era of AI:>108422422 >108422476 >108422608 >108422615 >108422643 >108422670 >108422734--Debating prompt format effectiveness for Literotica finetuning data:>108417215 >108417294 >108417388 >108417663 >108417731 >108417818 >108417885--Vulkan llama.cpp performance vs ROCm:>108421311 >108421377 >108421508 >108421570 >108421613--Phrase banning vs token banning in KoboldCPP and ik_llama.cpp:>108421847 >108421854 >108421857 >108421873 >108421884 >108421914 >108421928 >108421965 >108421977 >108422023 >108422035 >108422080 >108422096 >108422118 >108421993--Sarvam 105B benchmark results:>108422282 >108422382 >108422388 >108422440 >108422451 >108422497--Qwen 3.5 abliteration issues and Heretic uncensored alternatives:>108418501 >108418584 >108418609 >108418621 >108418672 >108418686 >108420443 >108418693--DDR4 vs DDR5 RAM upgrade considerations:>108419791 >108419799 >108419904 >108420101 >108420151 >108419888--Tensor parallelism progress in llama.cpp and fork alternatives:>108421401 >108421433 >108421476 >108421778--Miku (free space):►Recent Highlight Posts from the Previous Thread: >>108417029Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
>>108423198Required https://huggingface.co/DavidAU/Maximizing-Model-Performance-All-Quants-Types-And-Full-Precision-by-Samplers_Parameters
>>108423177Are the techniques that make newer models better being explained publicly?
>>108423255>newer models better??? Nemo still mogs anything newer.
Mikulove
Do you guys goon to your local models? is it even possible?
what am i doing wrong? kobold is working but sillytavern cant connect.what do to?
TMW
>>108423333undress me >>108423323
>>108423323Missing v1 or v1/ at the end of the API URL?
https://www.reddit.com/r/LocalLLaMA/comments/1rzyha4/new_ai_policy_by_white_house_us/> 1. Protecting Children — Require age-assurance measures, parental controls, and safeguards against sexual exploitation and self-harm on AI platforms, while affirming existing child privacy laws apply to AI.
>>108423322I think that was the first thing a lot of people on /lmg/ did after getting one running
>>108423322I think the best part about it is when you're new to this and don't know how to prompt or what model you're even running.The novelty combined with the challenge of making the pretty lady stop saying "What you're doing is highly inappropriate, let's respect each other's boundaries" is great.And when you figure it all out, wherever your brain's reward center is, it gives you a LOT of funny chemicals.Then you start seeing how dumb the small models are and how much the big ones like shivering and smirking. The magic is lost, you start giving them scripting and summarization tasks, wondering if your AI rig was ever worth the investment...So yes, it's possible.
>>108423462This is why photorealistic ai was a mistake.
>>108423462>>108423585I have written a legal disclaimer which states that every character I gen, regardless of the model, is always 21 years of age or older. Signed and stamped by me.There's nothing what they can do to me in this case.
>>108423585Meanwhile ZiT/ZiB just shit out CP like no tomorrow.
>>108423604Photorealistic sloppers are in trouble.
Qwen3.5-27b (HauhauCS uncensored version) is extremely good. Unironically rivals GLM-4.6, worse in raw intelligence (but good enough usually), better in not being slopped and formulaic.Qwen3.5-9b (HauhauCS uncensored version) is absolutely fucking retarded. Worse than Mistral-Nemo or Gemma 3 12b.What gives?>bro it's a smaller model of course it's worseThe gap is enormous and makes me wonder if something went wrong somewhere.
>>108423646>makes me wonder if something went wrong somewhere.Or the one that worked was a fluke.
>>108423623You can't ban open source models
>>108423646Sadly is ruined by shitty architecture, no context shift and no antislop sampler.
>>108423646It's literally shit
Blessed miku thread.
https://goombalab.github.io/blog/2026/mamba3-part1/
>>108423462>>108423585This isn't even about generation it's about making sure chatGPT doesn't tell kids to rope themselves.
>>108423198>"Experiments" by a schizo called DavidAU.Look at his model collection, choose one of the older ones with a really long name, then read the model's card.Behold the magnificence."expanding" (upscaling) a smaller model into a larger one then Pretraining the shit out of it, essentially using the original model as a base for a whole new model, is legit.It's just that you need to do proper pretraining with trillions of token, not do some qlora.Look at SOLAR 10B from back in the day. It's upscaled from mistral 7B IIRC.ah thx anon! any 27b upscaled that is decent? feels like it q5km is just on the cusp of being amazing for a single 24gb gpu. I am testing putting vision and tts on secondary gpu
>>108420443Benchmarks would work better, just way more expensive. For any correct answer, there are a multitude of token sequences that reach it (and moreso with reasoning).
>>108423646You know why it seems to be retarded?I can't enable reasoning in the uncensored 9B model. Vanilla 9B behaves much better when reasoning is enabled but now it seems like this uncensored version doesn't have that at all. What in the heck..I wonder if 27B is the same.
>>108423870When are they going to figure out that we should just ban kids? Kids make everything unsafe. Damn brats :anger_vein: :anger_vein: :anger_vein:
>>108424028>we should just ban kidsyes, support your local ID laws, they're for exactly that
>>108424032I don't mean "ban them from minecraft".
>>108423675Why does antislop not work with Qwen 3.5?
>>108424032No, parents are for that. ID laws are for the police state.
>>108424066hybrid attention makes surgical context modifications impossible it's all one big block
>>108423462Seems completely irrelevant since we don't even fucking get models from America anymore
>>108424222
With Qwen3.5-27B in Kobold, uploading an image seems to perpetually disable thinking, with the model always creating an empty block. Same in the last two versions. Is it a model quirk or inference bug?
>>108423177My local model collapsed when I trained it on another model
Working on getting an API service setup to host qwen3-coder so I can use it anywhere, has anyone done this and willing to provide an example?
>>108424072Thanks. I was wondering why I was seeing occasional smirks.
>>108424395I just ran llama.cpp + an ngrok tunnel.Both locally and on a kaggle instance.
>>108424342Did she really say this?
>>108424357After more experimentation, it must be a bug in Kobold. Having uploaded an image wrecks thinking even if you start a new session. The program must be restarted to fix it.
>>108424481 Why it stands out - It does not happen often - But when it does, it is very visible because the user already gave: - the problem - the severity - the priority shift - the go-ahead
>>108424395For when you're at home, just setting your ST or other frontend's host to 0.0.0.0 will let you access it anywhere in the house by going to your PC's IP and port. If you don't know what any of this means just ask your bot.For doing it across the internet, I can recommend Tailscale Funnel. It's easier to set up than anything else I tried, while still offering the highest security. Like I was set up in a few minutes. The only issue is that on the free tier it may be choppy at times (so the text streaming is not smooth). I can deal with that so it's fine for me but maybe you are different. Also, on the free tier, it only allows you to network one port at a time, but that's fine if you only want one API anyway.To set up Tailscale, just go to their website and follow the instructions. Then runtailscale funnel 8029or whatever port number you have your API on. This is what I did for Linux at least, idk what it's like on Windows.
>>108424535>routing your shit in externally managed serviceskill yourself
>>108424535>moments before anon's machine was hacked
It's OVER
>>108424603I read "nemo says".
>>108424603accelerate a
