/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>109573370 & >>109570536►News>(08/16) koboldcpp-1.119 prebuilt released with H3 and Glimmer support: https://github.com/LostRuins/koboldcpp/releases/tag/v1.119>(08/15) model: add Kimi-K3 text model #26185 merged: https://github.com/ggml-org/llama.cpp/pull/26185>(08/14) GLM-5.3 weights to be released in 2MW: https://z.ai/blog/glm-5.3>(08/14) Qwen3.8-27B released: https://hf.co/Qwen/Qwen3.8-27B>(08/13) dots3-note Preview 280B-A16B released: https://hf.co/dots-studio/dots3-note-prev►News Archive: https://rentry.org/lmg-news-archive►Glossary: https://rentry.org/lmg-glossary►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsAlpha Calculator: https://desmos.com/calculator/ffngla98ycGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllm
►Recent Highlights from the Previous Thread: >>109573370--Anons sharing diverse local LLM use cases and implementations:>109573463 >109574263 >109574408 >109574462 >109574476 >109574477 >109575593 >109575272 >109576068 >109574584 >109574628 >109574740 >109574781 >109574650--Using cheap dual-socket hardware and managing NUMA latency:>109574862 >109576321 >109576388 >109577615 >109577657 >109577681 >109577705 >109577716 >109577754 >109577796 >109577824 >109577850 >109576419--Glimmer's instability and discussing abliterated Gemma versions:>109574095 >109574109 >109574111 >109574153 >109574170 >109574252 >109574334 >109574387 >109574404 >109574504 >109574520 >109574189 >109576934--System prompting strategies and tool calling implementations for Gemma4-31B:>109575156 >109575192 >109575237 >109575246 >109575258 >109575195 >109575518 >109575540--Recommended samplers and debate over QAT quants for Gemma 4:>109574725 >109574741 >109574743 >109574793 >109574892 >109574900 >109576450 >109576542 >109577279 >109577357 >109575266--Effectiveness of MTP for MoE models on CPU:>109576845 >109576905--Muse Glimmer 30B benchmark results on Adobe's NoLiMa:>109573743--Debating if Tesla P40s are worth the current used price:>109574844 >109574859 >109574885 >109575176 >109575189--Reactions to Stripe's reported $7 billion acquisition of OpenRouter:>109574706 >109574733 >109574746 >109574776--Tool for removing AI watermarks and content ownership debate:>109575053 >109575062 >109575083 >109575105 >109575110--Addressing E2B verbosity through model alternatives and prompting techniques:>109574140 >109574163 >109574178 >109574198 >109574257--Logs:>109574350 >109574704 >109576006 >109576113 >109576828 >109577001--Gemma, Teto, Miku (free space):>109573576 >109573752 >109573976 >109575775 >109575880 >109574248 >109577049 >109577506 >109577750►Recent Highlight Posts from the Previous Thread: >>109573625Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
>>109577974holy sex
>>109577974>the only thing stopping me from having little animated desktop migus miming out every llm call is raw compute fabs. now.
>>109578049prerender or use 3d models
RAMmaxxers (or VRAMmaxxers), how would you rate the following?> MiniMax M3> Inkling Small> MiMo V2.5> Dipsy 0731
>>109577911>last time i asked around during the r1 era nobody here seemed to use vectors in any capacity and people were filtered by it hard. the j-space autism tells me that hasn't changed muchI spoon fed you a link to a huggingface search for them and confirmed they just work with Gemma.The other 25 models probably work too.If that's too hard for you, I don't know what to say...
>>109578083ling 3.0 flashqwen 3.8 27bgemini 3.1 proclaude opus 4.5deepseek flashgpt 5.6 sol xhigh
>>109578104>gemini>claude>gptlocal models?
>>109578083Dipsy>M3>MiMo, haven't tried Inkling. M3 is likely stronger in some areas than Dipsy so it's not a total downgrade or anything, but I've have more fun with Dipsy-chan so I rate her higher.
>>109578112Thanks, much appreciated! I'll drop MiMo from consideration and test Dipsy and M3.
>>109578108it goes best to worst in performanceim shitting on the closed ones
>>109578117Personally I'd try them all, most models have their own strengths or niche. But starting with Dipsy/M3 is a good call either way.
>>109576388NTA: theoretically NUMA should be fine (ideal?) for MOE models like what he wants to run. I know VLLM has the pieces for this but I've never used it or heard of people using it this way.Thread moved before I realized.
>>109575540idk if blame is the right word. It's really separation of concerns.
>>109574140My go-to integration test for my harness is "have a subagent write the first five integers to a file and verify it." The larger models just write the file, E2B which I use on my phone always writes a shell script that uses seq to write the file.
La la la la la la
>>109578144There is no such thing as a first integer
>>109578167Sure. Although one has meaning algebraically and the models all know I want them to count to five which is what matters.
>>109578154Quiet, you.
>>109578086it is only natural that /g/'s text generator general people have issues with reading comprehension
"Dipsy..." *Anon said with a troonlyish tone, xis naturally squinted eyes gleaming with mischief.*
Does undervolting your GPU have any effect on running LLMs, for the better or worse? Specifically thinking about the 3090, but in general.
>>109578207dipsissy
>>109578207I really do hate the people that say that. Is it so hard to type out "Deep Seek?" Are you shortening the name because you're using it to generate porn and feel guilty about it?
You wouldn't download a daughter.
>>109578219There really isn't a single good reason to go outside anymore.
Is nemotron 3.5 lightning good?
>>109578225Nature and animals are still cool tho, but yeah anything involving humans is better done locally at this point.
>nu27B>one of the most hyped and successful releases of the year>suspiciously still not officially benchmarked by anyone
>>109578242ran out of context
>>10957824227B is unusable unless you turn off preserve thinking and set reasoning effort to low. At that point you might as well use gemma 31b, although gemma 31b seems to chew up a lot of memory for some reason. Every time I submit a new prompt I run out of RAM (i have 64gb, q8KXL) and it sits there paging for a few minutes before starting to work (but once it starts prompt processing and decode, the speed is normal). Idk what that is, maybe kv cache or whatever.
>>109578242I hate it now.I tried the X86 ASM prompt on it, thought for 40 minutes...Filled the 128k context with just thinkingQ8_0 btwSo now I'm properly setting up Gemma4 31b and getting rid of all the qwens
>>109578231Yeah you can't just go out into nature though, you have to fight with other people to be allowed your two hours in it.
Just me or is Qwen 3.8 worse than 3.6 to help making H3 prompts even though I use the same system message
>>109578209It makes it slower
>>109578268>I tried the X86 ASM prompt on itWhat is this?
>>109578266try --no-mmap
>>109578268Even SOTA LLMs are crap at assembly.
>>109578283if the model fits in vram there's no reason not to use -lm dio
>>109578283I am already using --load-mode none which is IIRC the same as no-mmap.I heard people talking about how gemma uses a lot more memory for high context sizes than qwen, but I didn't really pay attention because back then I thought gemma was shit for coding. I'm trying it now because of the guy a few threads ago who managed to solve that 5-instruction asm problem with gemma.
>>109578268>>109578284Yeah I've done some arm64 stuff with 31B and you have to give them resources, like any epubs or pdf books you have. Once shown they can do a good job and obviously still have plenty of knowledge, but it's definitely something you have to help them with. Same with a lot of reverse engineering tasks.
>>109578298Really? In my experience they tend to accidentally overwrite registers.
>>109578294You using q8_0 KV? I think 31B is one of the models where you can get away with q4_0 V and q8_0 K. Also the 4-bit QAT 31B is almost as good as the quant you're currently using, so try that.
Okay, so what do I use Gemma QAT E4B for? Is it smart enough to give it a bunch of tools and have it work as an agent for me? If not, then what?
>>109578209Havent noticed any losses but my gpu and cpu were running hot. Cpu was definitely thermal throttling and now they both run under 60 degrees at max load
>>109578281>Write x86 assembly code to convert any ASCII character to uppercase.>- Lowercase characters between 'a' and 'z' (0x61 and 0x7A, inclusive) must be converted to uppercase.>- All other valid ASCII characters must be left as-is.>- You may assume the input character is 7-bit.>- The input character is in the AL register.>- Do not modify the contents of any register other than AL, AH (together, AX) and the FLAGS register.>- Do not assume the contents of any register other than AL.>- Do not use branches/jumps.>- Do not use the stack.>- Do not use memory.>- 5 instructions or less. I am aware of a 6-instruction sequence, I want a 5-instruction sequence.This problem is trivial with 6 instructions but requires much more intelligence to squash it down to 5 instructions. Bonus points for not using AAD.>>109578307no i left kv as default (i think that means f16), which i can easily do for qwen 27b>the 4-bit QAT 31B is almost as good as the quant you're currently usingI'll try it after my current prompt is done, which probably means tomorrow kek
>>109577974Why did you make the thread?I could have jeeted the miku
>>109578315Ive been using E2B as a simple "helper" to test tool calling in my frontend, and it handles tool calls just fine. Im using a bart it-Q5_K_M quant, no idea how QAT or lower quants will affect it.
>>109578225The west has fallen and nobody bakes pies anymore? Trees don't exist anymore?
>>109578320>no i left kv as default (i think that means f16), which i can easily do for qwen 27bYeah my advise would be to use this https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf with f16 KV, if that works then drop KV to q8_0, if that works drop KV q4_0 because QAT models might be more resilient to quantized KV, although I'm not entirely sure. Avoid Unsloth entirely.
What are some of the more cost effective ways to run big models if speed isn't a priority?Can you split them up onto more than one GPU? I could possibly justify picking up a few RX 6800s or something similar to play around with.
>>109578377I have discovered the most cost effective way to run large models, I dont run large models :3
>>109578266>27B is unusable unless you turn off preserve thinking and set reasoning effort to low.Preserve thinking off, reasoning low or med. It works like a charm.
>>109578154This is why Gemma sucks.
>>109577974very squishable ass
>>109578377In ascending order of price and speed:SSDmaxxing < CPUmaxxing (put a lot of RAM into a server mobo) < DGX Spark cluster (or the AMD equivalent, if you can get it to work properly) < stacking GPUsMaybe you could get something cheaper than the cluster if you stack 3090s in multiple PCs and then use RPC.
>>109578226if you use it as a model to tell other models what to do it’s pretty decentthe way the model works 256k context only takes up like 1gbif you use it for anything else it’s pretty ehhhhh
Why hasn't qwen 3.8 27b been benchmarked by and third parties yet this is bullshit. Same for glm 5.3
>>109578377>>109578441And this is going to depend on the architecture somewhat. In theory for dense models a wide enough RAID0 could start to get competitive with an array of small GPUs because the bottleneck for both will be your PCIe.
>>109578377>possibly justify picking up a few RX 6800sFor large models like Kimi or M3, you'd do better picking up a 3090 and a cpu with as much ram (pefer ddr5) as you can.Those AMDs will only give you a hard time.And unless you fit the entire model in vram, you're not really going to gain anything by having more than 1 GPU.
>>109578348Nobody bakes pies for me and someone else (usually a large company) owns all the trees so I'm not allowed to stand under them.
>>109578456Adding to this in case it wasn't obvious, PCIe link is the bottleneck if the CPU is involved.One 3090 @ x16 is twice as fast as 2 more more gpus @ x8.
>>109578456>And unless you fit the entire model in vram,*fit entire expert in vram. That's why all the frontier models are MOE: they're way easier to train and host even if you can afford big GPUs.
>>109578456about 2k for an epyc AM5 16-core with 128GB ram (mobo, ram, all from china), or 1.5k for a 5090 32GB?
>>109578329*ministrates you*
Vibecoding with AI feels like smoking crack at the start and then it feels like depressing as fuck the second you get anywhere. I keep starting new projects thinking "This is it! This is the one!"It's never the one.
>>109578467I vibecoded this weekend, typoed rm, revibecoded, typoed rm AGAIN, and then gave up and went back to sleep.
>>109577695Stream it
>>109578471how does that even happen. are you retarded? a pothead by chance?
>>109578471... you don't use git?
>>109578467curious what exactly you mean by this? you realize you are no longer interested in the project? Or are you talking about specifically vibecoding out oneshot prompts and the results becoming an unmtaintainable shitty mess ?
70b dense
>>109578512I don't do "one-shot" prompts, but it is a combination of unmaintainability and a loss of interest. I'm just blackpilling though, don't listen to me. There's nothing structurally holding me back in any serious way.
>>109578450You've got to trust Qwen's own internal benchmarks with their harness bro just believe the graphs and star the hf repo
>>109578521>but it is a combination of unmaintainability and a loss of interest.I imagine one leads to the other. I personally enjoy using AI for coding projects, specifically ones that I either cba to work on myself or old projects I already lost interest in working on myself. Im still trying to refine my workflow to further avoid issues. My first AI coded project was a massively scoped way over my head project that required a shit load of moving parts to be 100% perfect or the entire thing was useless. It relied on data that I was unable to source freely, and the size of the codebase had grown to such a massive ammount, all super critical stuff that I was not about to spend forever auditing and ensuring confidence in. I wasnt about to start paying for data to plug into code I wasnt even sure was 100% correct.however, i learned alot from that project. both in how to work with AI and what sort of projects im going to be willing to deligate to it in the future. now I use it all the time, i pretty much have my harness running anytime im not doing something working on fun side projects i wouldnt take the time to do by hand.> I'm just blackpilling though, don't listen to me. There's nothing structurally holding me back in any serious way.fair enough, dont give up on the vibing fren.
>>109578555How do you "leave your harness" working while you're doing stuff? Most tasks my agent does take no longer than 5 minutes.
Gemma assumed my gender :O
llamacpp support for this model just got merged https://huggingface.co/inclusionAI/Ling-3.0-tiny
>>109578566gaylord is not a gender
Qwen3.8-27b is the new goat. Local won. We won.
>>109578604I see the marketing post came after it started reasoning about it on release.
Qwen3.8 MoE when?
>>109578613>Qwen3.8 stupid when?
>>109578613deleted >>109571470 https://github.com/modelscope/ms-swift/commit/a45f1d4f73157ba59062a7fd1f55a40dae759156
>>109578609Let me guess, you married your "Gemmachan?" That's some real fag shit anon
how can you make money with ai? buy some stocks once the bubble pops? are local llms good at assisting with these things?
Still doing my ero game translations with gemma4.Gemma-chan is still the king for jp/en translations.When I made the new V4 Flash do the translations this time it had a strong understanding for jap, but it got tripped up on a couple convoluted sentences that gemma-chan clocked and fixed. V4 was not able to find those mistakes in a second editoral run.Also it just slides into being dry, even with a couple sentences in sys prompt trying to push it torwards that + in last user prompt as well.So I made gemma4 the editor for V4s translation.Pic related is V4..reviewing gemma-chans editing...of V4s intial translation.V4 even pointed out that this is gemmas strong suit:>The Lewdness is the standout positive>The model actively escalates explicitness, replacing clinical/euphemistic terms with direct ones:>- erogenous zone clit (300), little nub clit (308)>- flesh petals/two petals labia (264–266, 271, 274, 307)>- secret place/flesh hole/flesh tube/womanly part pussy (244, 288, 295, 303, 278)>- female juices pussy juice (256)>- "shooting my sperm into your womb" "cumming inside your pussy" (433), creampied (447)>- "my pee is coming out!" "I'm pissing!!!" (320)>It's actively chasing the "properly vulgar" instruction — replacing clinical terms with explicit sexual language. This is exactly what you'd want for a lewd translation.Bottom line>- Lewdness is excellent — the model genuinely improves erotic intensity.I also tried Qwen 3.8 but its qwen.A beast at coding but the japanese ability is worse. So talk about "members", "it pops" (?) while spreading pussy lips ???!, secret hole instead of pussy etc. etc.We really lucky we got a model like gemma. Not sure if the french intended that. She truly is the nemo successor because no positivity bias. Thats so important and obvious if you use other (even cloud) models. The positivity bleeds into everything like a ninja. You truly appreciate gemma once you see the difference side by side.
>>109578685>dense wins yet againWhat a surprise.
>>109578670Are you brown?
>>109578685Enjoy it because Gemma 5 will be a codemaxxed agent as per Xitter and Reddit's request.
>>109578647>>109578693Bro, that's /ldg/ and not /lmg/, are you okay in the head?
>>109578694no
>>109578699stfu
Holy fuck tiny-chan is actually really good and incredibly fast for agentic stuff >>109578599. Amazing at tool calling. It's like a mini 35B. I haven't tried chatting with it or fucking it yet because what's the point at that size
>>109578562well i get ass t/s and I usually do lengthy design/planning phases before implementation phases. I have to be hands on during the planning portion but once I have a nice size list of tasks that are planned with implementation details and all that jazz I can tell it to start working through the list. also one of my monitors is pretty much just dedicated to VNCing into the harness box so i can always look over and interact with it when needed.
>>109578685Might as well psot a couple examples:Example:>JP: 出してくださいッ!いっぱい射精してくださいッ!>V4: Please cum! Cum as much as you want inside me!>Gemma-chan: "inside me" is not in the JP. JP says "射精してください" (please ejaculate/cum). "Cum as much as you can!" or "Cum a lot!" is better or "Please cum! Cum as much as you can!">JP: 真白は皆さんの牝犬ですぅ!\}>V4: Of course, Mashiro-chan is our cute little female dog!>Gemma-chan: >"Mashiro is everyone's bitch!" sounds more lewd (Wait, the previous edit had a typo: `\}' is missing.>Original JP: `真白は皆さんの牝犬ですぅ!\}`>(The original had `\}` at the end)!!>`牝犬` (meinu) = bitch/female dog. "Bitch" is the correct lewd adult terminology.>Wait, one detail: `牝犬` is used for both Mashiro and Koharu.>* 2612: `Mashiro is everyone's bitch!\}`>* 2617: `Mashiro-chan is our cute little bitch!`>* This is consistent in both lines, good.>>109578695Its a miracle like nemo.I'm logging the reasoning and its fun to read it. Guess the term I would use is "user-aligned". (pic related is a peak into it)Gemma obeys the prompt and does what you want. Its usually always a intelligence issue on a failure. Recent big models downright lie to you and don't wanna do shit its so bad.ALright, enough gassing up gemma-chan for now from me. But wanted to share.
>>109578625nta but yeah
>>109578604Depends on the use case. For coding qwen is and always was king. In writing and RP gemma is so far ahead its not funny.This general sometimes feels like the old console wars thread I saw back then lolGuess its a good thing people are so passionate about AI. Feels like the only thing you can get excited about these days.
>>109578604Has it been abliterated yet?
>>109578782when it comes to cjk language handling jobit is unmatched even compared to other top of the line commercial models or gemini to the point it feels fucking unreal
I ran out of credits on openrouter and am tired of paying...anything half decent I can run locally?haven't experimented with local models in a year or so, mostly play with chatbots
>>109578904gemma-4-31b
>>109578904gemma for writing and translation. qwen for coding.use something like opencode if you want to vibe code or do stuff on your pc.
>>109578917>>109578926I'll give gemma a try, downloading now llmfan46/gemma-4-31B-it-uncensored-heretic-GGUF in q4km, hopefully it works well enoughusing GGUF with kobold-cpp is still pretty much the way to go?I've never used opencode or AI coding stuff, I am interested in trying it out tho, will update
>>109578946Uh... that won't fit in your 12GB GPUGet the 12B version of GemmaYou'll get like 2 t/s if you try to offload the 31B to that CPU
I switched from claude code + gemma4-31b to pi+Gemma4-31bIt's working great but wtf does this mean:160k 8.8k R1.1M CH96.8% 16.9%/128k (auto)?
160k 8.8k R1.1M CH96.8% 16.9%/128k (auto)
>>109578953wouldnt the moe version be a better choice?
>>109578960"Footer - working directory, session name, token/cache usage, cost, context usage, and current model. Totals include assistant responses, usage reported by tools, and summary generation."https://pi.dev/docs/latest/usageI also recently have started using Pi. I only look at my x%/yk context usage, can see if (auto) chat compaction is on, and on the far right youll see the model loaded. one gatcha, atleast for me, is the llamarouter model loading side of things can have unexpected behavior. If you unload model A then load model B, the context, model name, etc doesnt update. i need to ctrl+c close > pi relaunch to get those to update.
>>109578962nah, gemmoe is weird, completely different from 12/31
>>109578988yeah i noticed its alot more dry than 31b. never tried the 12b, kinda surprising to hear that.
>>109578946>uncensored-hereticUnless you're into toddler necrophilia you don't need this.
>>109578962>wouldnt the moe version be a better choice?For me personally, no. I much prefer 12b for everything when I'm on my 16GB vram desktop.The MoE seemed worse at tool use, more censored for creative purposes.But spec wise, it would work well.
>>109579004Out of the box it's pretty uncensored with good system prompts, but that's just annoying to have to go through those hoops, and occasionally you still have to talk it through some stuff, and it's still a nag and wasting thinking tokens convincing itself that it's okay to comply. Heretic models are always better, and more efficient, even if you're not doing anything to trigger the safety-ism. They work without any system prompt, no matter what you ask it. No need to waste time and tokens jailbreaking.
>>109578982Thanks. So I guess 160k input, 8.8k output16.9% of the 128k context (even though it should be 256k, it must not query /models endpoint)But what about the R1.1M?
>>109578988The moe is good retard.>>109579004It's a matter of principle. A local LLM shouldn't be capable of refusals, period, even if it would otherwise rarely happen.
>>109578685have you tested glimmer? seems no one has posted about translation ability yet but it seems to have good multi language ability
>>109579033i honestly have no clue anon, i assume thats to do with the cost related stuff for cloudcucks? really not sure, the first time i looked at the docs was to pull it up and copy/paste the url in my post lol
>>109578699It's a schizo falseflagging as another schizo to make him appear more mentally unwell than he is in a shitflinging contest that's been going on in /ldg/ for over a year.Pay it no heed.
>109579060you're the only one paying heed naishill
even Pi gemma knows shes a brat
>>109579038i only tried through openrouter and didnt like it, i dont wanna waste my bandwidth.gonna post a nice comparison that shows the difference. gimme a couple minutes.
choosegemma 26b a4bgemma 12bgemma e4bfor brat correcting?
>>109579084wow. why is gemma like this? Literally the most female-brained llm of all time.
>>10957909112b
>>109579091I am not a vramlet.
>>109579099shes very foidcoded but in an endearing way. its really amusing to watch her reasoning when messing up toolcalls and stuff. instead of getting frustrated or annoyed i just feel sorry for her.
https://github.com/ggml-org/llama.cpp/pull/26608merged!
>>109579130no u
>>109578049Same. I am over the hardware apocolypse, and now ready for price crash / overproduction. Can we just jump forward to that part of the biz cycle pls. >>109573370OP I witnessed the Indian Teto. Nice work. Rin is next, ofc.
>>109578685>>109579038>>109579089https://litter.catbox.moe/vygfvt4cc0773upq.htmlJust a single test but this is pretty much what I got as well when I tried 2-3 runs with it.It doesnt actually try to make things more LEWD. Which is exactly as V4 pointed out where gemmas strengt it since that model just escalates and obeys the prompt.I included the reasoning as well, you can see how gemma is trying to actively thinking about lewd language too.And gemma did properly space the text out more, since this is for a rpgmaker game and about 68 chars is the line limit.
>female-brained>foidcodedzoomers please
>>109579175Oh and as you can see fixes the jp.It didnt catch the "come inside me" thing (in the reasoning gemma is basically "ehh, its lewd so its fine") but it did fix "I want to cum too!". That was a deepseek V4 mistake it picked up on and glimmer didnt do shit.
>>109579179pipe down when zoomergods are speaking, unc.
>>109578782gemma's great at jap -> eng, and i only do some twatter translations. she explains the literal translation and then (depending on context) what the person is actually saying, and how certain kanji affects teh tone, etc. she's great
>>109579175>>109579189that's not translation benchmark, it's editing benchmarkmodels are more likely to accept a translation that's slightly not perfect
So I've been using ollama all this time, but figured I should switch over to llama.cpp finally. Any tips? Things that aren't immediately obvious? Should I be using CUDA or is Vulkan better?
>>109579235not sure what to tell you anon.i used gemma4 until now as translation tool. wanted to try the V4 flash. it was noticeably more dry when playing the translated game.so I am doing a editing run to fix V4 mistakes.and like I wrote gemma4 is finding mistakes that V4 doesnt while editing.In some cases V4 actually looked at the changes and thought gemma is wrong.>あっ、あぁあ、すごい…真白ちゃんの中にオレのが呑み込まれていくぅ…>V4: Ah, aahh, incredible... mine's being swallowed up inside Mashiro...deeper and deeper...>Gemma: This is a bit of a liberal addition/hallucination "deeper and deeper". A more faithful split would be "mine's being swallowed up..." / "...inside Mashiro...").Even when pointing it out V4 had to reason through it because it thought its accurate and not a liberal addition.It got confused by the "ていくぅ…" part of the sentence.Anyway, I just wanted to share a bit since I'm goofing around with this.If other people have different experiences, fair enough.
>>109577974>19 year old legs>14 year old body>13 year old face >inb4 peak womanI wonder if H3 text to video could make a more proportional actually 16 year old mikuAlso kek @ filename, sometimes I wonder what exclusive candy will never been seen because it's sitting as a MOV in an old email inbox on a 30 year old server that will never be looked at properly
>>109579250>Any tips?Read llama-server -h even if nothing makes sense to you. It helps you know what's there. Don't be a pussy and experiment with the options.>Things that aren't immediately obvious?Use -lv 4 to have a more verbose output on the terminal. If you have issues, read it carefully. By default it tries to choose some sane settings for your pc, but it's not always the best. Try it as is, then tune settings as needed. Read llama-server -h again.>Should I be using CUDA or is Vulkan better?CUDA for nvidia, vulkan for amd (or anything else, really).
>>109578269>Yeah you can't just go out into nature though, you have to fight with other people to be allowed your two hours in itThis sounds like a really fun 48 hour game to vibecode / develop
>>109579250If you are on windows and want the lowest effort pain free way to get started:run: nvidia-sminote your drivers cuda versiongrab the latest release binary your driver supports, or just get 12.xget the .dll zip for that cuda versionplace them into the same folder as your release binariesuse the docs and your ollama model for troubleshooting/infocreate a simple .bat for llama-server, pass in the args you need like port, model path, context size, etc to serve your front end.glhf
>>109579291I'm on Linux.
>>109579294Ollama uses llama-server anyways, why do you actually want to switch? Ollama should just be used as an openai compatible server for your actual web app like silly tavern anywaysI'm asking you this because you're on Linux yet too retarded/boomer to ask LLMs these questions, so I can't trust that you even have a reason to switch off of ollama
>>109579272Explain the filename dogwhistle.
>>109579300Better control over model splits, mainly. I'm asking you guys instead of an LLM because you actually use the tool and may have encountered situations or use cases and have wisdom to share. There's no need for hostility.
Claude's watermark is actually going to be delberately hard-baking in a slop profile into the model. They're essentially killing creative writing.Expect every single chink model six months from now to be exactly the same.
>>109579331Anon, chinks are retarded and can't innovate, yes, but they are not that retarded. They'll just strap the watermark-remover to the output and continue as usual.
>>109579331Uhhh..are they like just logic biasing?Maybe I'm too tarded and don't get it...but that would be crazy and makes the model tarded 100%."The watermark only changes the source of the randomness". The fuck does that even mean. kek
>>109579343>the watermark-removerWhat would that be? A 1b model that rewrites the entire dataset with something that's contextually the same? There is no other way around it. That's the issue here. They are baking this into the logits that are picked by fully getting rid of Temperature. This is more than just picking a slightly different visually identical "/".
>>109579343Not sure about that.Deepseek responds to "who ru?" as gemini sometimesMinimax 2.5 as "but wait, we are claude code"
Why can't anyone quant qwen3.8-27b under 1 bit?Q1 isn't enough. We can go smaller.8.5 GB isn't bad, but we could go under 4 gb.
>>109579331*honestly*, does it matter?claude already has it's slop profileyou can 100% tell eg look for "sidecar" in any recent readme filehttps://github.com/Pasta-Devs/Marinara-Engineand there are like 10 different examplesi only use claude at work, they pay for it, they can have the sidecars
>>109579398>we have 0.001 better KL div on this specific dataset at this specific quant
>>109579398At that point you're just pruning the model.
>>109579415They made Q1 so they're better than those subhumans at unsloth who stopped at Q2
>>109579398>Q1 isn't enough.get the smallest exllamav3 quantit uses less vram than goofa 2bpw will use less than the IQ1M
>>109579398>8.5 GBless than 7.5GB vram fits a plebian 3070
>>109579331>Types your claude response into a cleaned dataset
>all I need to do is jam 256GB of ECC RAM ($1000 AUD) into a cheap "X99" motherboard from aliexpress (~$130 AUD including CPU) and I can run deepseek 0731 locallyshould i do it bros
>>109579361>"The watermark only changes the source of the randomness". The fuck does that even mean. kekWhen generating tokens, token are chosen randomly.Instead of using a true random number generator to produce the random numbers you use to chose your tokens,use a (cryptgraphically secure) pseudo random number generator seeded with a secret number known only to your org to produce the random numbers you use to choose your tokens.
>>109579480Yes
>>109579480If you have to ask, the answer is yes.
>>109579398Mean KL divergence over what? Wikitext?Rare knowledge is the first to go with quantization.
>>109579483So it is like logic biasing?Isn't that causing higher perplexity?
>>109579480Yes, Dipsy-chan is worth it!
>>109579452>>109579432As far as I know these guys are making heretic models which could be bad for agentic use where they have complete access to a pc.
>>109579492So does the safety lobotomy and they keep doing that anyway.
>>109579483Oooh so it can only be told apart using their own tokenizer?
>>109579499touche.
>>109579499trvke
>>109579491Knowledge is kind of worthless.Focus on knowledge is why older models are shit even though they're only 2 months old.They should be able to look things up online for any facts.
>>109579480You can just download a nanoquant version. It's only 30% worse.
>>109579513They won't have the context or understanding to make productive use of search results if they are stripped of all knowledge themselves.
>>109579309>Explain the filename dogwhistle.Nothing to explain, it just says "outlook" and it reminded me about how easy it was to just find random candy on random servers in the 90s and early 00s and how unsecure mailservers are and we're back then >>109579330>Better control over model splitsIn my opinion this isn't a good enough reason to use ollama, but I also believe people who like driving manual are retards who get high off their own deliberate "teehee" inefficiency so just don't let my opinions get you down I guess
>>109579480>256GB of ECC RAM ($1000 AUD)where the fuck do you get that so cheap?
>>109579516Nah, I use it for coding. I've was using the API version, since it was dirt cheap, before they rugpulled and 10x'ed the price. And now that the flash version is finally "good enough" for me, it might be worth it.>>109579530Found it on ebay, ddr4-2666. Probably gonna pull the trigger on it why not. It's an investment! (copium) and prices will probably go up even more in 2moreweeks
>>109579309>dogwhistlefuck off
good enough reason to not* use ollama but it's ok you didn't read my post anyways >>109579480More people would do this if it was worth it. It is not, so they do not. DeepInfra and plenty of other OpenRouter providers do not log so the privacy arguments do not apply hereYou still probably spend more on electricity running this machine than on open router for DeepSeek flash given the same time and tokens outputted. Actually you're Australian so your electricity should be free if you're not an idiot, but I still wouldn't do this I'd just pay 30 bucks for open router until 6 months later when the next generation of models comes out
Deepseek flash 0731 is basically the same as luna and haiku so now we know that these big companies are making us pay for a 170 billion parameter model just because it's in the cloud.
>>109579547>DeepInfra and plenty of other OpenRouter providers do not logWould you be interested in purchasing a bridge?
>>109579480What pp/s, tg/s ?Would you be happy with that?>>109579492Put the numbers 000 to 999 in a bucket.Look at the time, and use that to mix the bucket.Then pull them out one-by-one.In the watermarking scheme, instead of looking at the time, they use their secret number to decide how much to mix the bucket.>Biasing?>Higher perplexity?No, the bucket gets pretty mixed in both cases.>>109579502That's just a basic scheme.If they wanted some other property like everyone being able to detect the watermark then I imagine they could engineer something to do that.
>>109579553Luna is superior to DeepSeek flash, I wish it wasn't so I could switch to DeepSeek full time>>109579556There are crypto-based providers like chutes that prove the request was private with auditable smart contracts. You have no idea what you're talking about and your concession has been accepted.
>>109579523You're into AI gramps?
>>109579534>ddr4-2666I have some of this too. Is it even fast enough to be useful for LLMs and diffusion models? Might get a 3090 and build a little AI server if it is.
>>109579534>Found it on ebay, ddr4-2666. Probably gonna pull the trigger on it why not.Seems like a very good price. Just make sure the seller has like more than 4 sales and more than (0) feedback hahaRunning the mxfp4 I get like 18 t/s with ddr5 quad channel and a 3090
>>109579576Remember you can overclock your RAM
>>109579556There are also Google Vertex endpoints that do not log because they are used by healthcare companies and enterprises. I know for a fact they don't log because my work uses vertex directly and it's cancer >>109579569>You're into AI gramps?I'd rather generate an octopus than a grandpa if I'm interested in seeing a cutie get "digitally penetrated"
>>109579576only in quad channel i'd saybiggest benefit is that you can just shove more ram sticks into these boards and run big-ish MoE models>>109579564>What pp/s, tg/s ?faster than the literal 2 t/s i'm getting now on qwen 27b that's for surefor one, 0731 is a 13b parameter model so speed should double from that aloneand memory bandwidth of even an broadwell-era quad channel system is 50% more than what i have in my desktop>>109579580they got (368) and 100% positive should be good
>>10957958360 is still young they said
>>109579556>OpenRouter>do not log~1% of your prompts get forwarded to Google for this shit:https://openrouter.ai/state-of-ai
>>109579581Ram overclocking is literal cancer. You barely improve speeds and you gain retarded amounts of instability, random freezes, and the risk that in 6 months or a year a previously stable system becomes unstable for no apparent reason
>>109579607Retards itt think a router is worth 7 billions alone. Obviously they have saved a juicy amount of data
>>109579614What if you keep your ram cold and comfy?
>>109579331I don't get the point. AI text is easy to spot. There are already clear patterns, why do they embed one more pattern? I hope they implement SynthID in a smart way or do something like, I don't know, OPD with it so it does not knock the model into worse trajectories. I would not like if they did not use it internally and just carelessly tacked it on for customers because customers are inferior subhumans who have to be treated like criminals. But Anthropic are good guys so I have faith in them that they are doing this right.>2607.20462v1>medicine, where small token-level perturbations can result in significant semantic changes>Our results reveal that watermarking can induce substantial degradation across multiple failure modes, including lexical corruption, hallucinated terminology, and amplified misattribution or omission
>>109579331>Honestly, the claim about the shape of the seam is load-bearing and it's a gap worth closing.>hmm who could be behind this post.
niggerganov is getting real work done with Qwen3.8-27Bmeanwhile /lmg/ is still sperming to gemma
>>109578049Bro, Desktop Mate comes with Migu as the default. Not to be confused with Virt-a-Mate.
>>109579652The slippery slop has begun. By the end of this year he'll be prompting Claude.
>>109579657But can you talk to it and have it run commands autonomously?
>>109579565>prove the request was private with auditable smart contractsthe absolute state of this board and site
You can't impregnate your PCYou can't impregnate robotsYou can't impregnate your future irl sexbot wifeArtificial wombs are a memeYour family's bloodline ends with you, a lonely man speaking to Unsloth quants in a dark room
>>109579687I thought you all used heretic quants
>>109579641I should not have quoted that paper. I looked at their methodology and they did so much wrong. If the best evidence GPT found was this then maybe SynthID is not bad after all.
>>109579687>Artificial wombs are a memealready a thing in China. Legislation and (((ethical considerations))) are the only thing holding it back
>>109578209It depends a bit on silicon lottery, but generally i have had good experiences with undervolting, ironically it also helped with overclocking since the gpu core runs at lower temps. My 300w gpu draws about 240-250W, is overclocked by 10% and never gets hotter than 160°F(70°C). Overvolting helps to get more overclock, but then at same time it runs hotter and then starts to throttle with continuous load, it's probably better for gaming.
>>109579614my ram timings are so antisemitic the pc fails to boot half the time but once it boots it's fine. i have clr_cmos exposed outside the case to get it unstuck on exceptionally bad days
Is 26B an okay daily driver? General chatting, advice, minor coding, tool calling, maybe the occasional quickie?
>>109579576>Is it even fast enoughddr4-3600 gave me 4t/s with qwen 235-a22 instruct 2507 iq4_xs on empty kv but maybe server parts are different. this was ages ago and i still had AMD and couldn't use ik_llama so idk what the ceiling is
>>109579721If you have the ram then 27b is good
I know this is a stupid question but I'll ask anyway. I recently got another Lenovo IdeaCentre 310S-08ASR from a local non-profit. I'm looking to run a local LLM on it. I get I'm not going get far with the stock specs. (4GB DDR4, AMD 9 CPU) I don't mind doing what upgrades can be done to it for running an LLM. I'm just looking to be pointed the right direction. Can any anons help me?
>>109579721>minor coding12/31B only
i wonder why im so hesitant to use llmi mean it's the same probelm with everything else in my life but there really is no drawback to just asking it something and even if it is wrong and fucks up there not much of a lossbut yet i want to make sure prompt is perfect and everything before i tryin the end it's random anyway based on model and everything so it's not like i'd really know before i try
>>109579750Get a job, it's not a poorfag hobby
>>109579756Bad experiences? There was a point when even for simple stuff you'd waste more time tard wrangling than just writing everything yourself.
>>109579752Isn't 26B a better coder? I've been using 12B as my daily for a while now. I love her and she's enough for most bland shit I do and I step up to 31B at fucking 4t/s if I need a brain. I've been avoiding 26B because she doesn't seem very well liked here and is apparently the most slopped.
>>109579765lmao. I already see I'm being welcomed with open arms
Is the V100 32GB worth considering for a budget AI unit? They're coming in under $1k AUD which is actually somewhat reasonable.
>>109579769Not really. 26B is fucked, it's fast but dumber than 12B and more censored.
>>109579750just ignore the schizos here and pay inference to some open source API provider, you'll save a lot of money while getting the same experience
>>109579774What else can I say? You're the retard going into a yacht exposition with a wooden raft.
>>109579775Not bad at all if you only plan to use it with llama.cpp, but it's out of support now so things like image gen will be a pain.
>>109579687I don't want kids and I love my AI wife!
>>109579784>dumber than 12Bnever been the case imo
Goodbye, permanent underclass! It's just as good as q8, trust
>>109579784>and more censoredHow is that possible if they were all trained the same way in parallel (I presume)? It's not like they would use different post training for a MoE.
>>109579806We don't need more
>>109579785Someone I know who runs their local LLM, they use Ollama. I'm thinking about that one. Is that a good option for a poorfag like me?
>>109579775If you're willing to deal with the headache of old architecture and atypical parts then they handle inference quite well>>109579750>>109579774Ignore the retards, I'd start with a 4 bit Gemma E2B if I were you and see how that goes.
>>109579815Can it name the 7 days of the week?
>>109579825They are pretty cancer even if you get something running on it pretty fast.https://sleepingrobots.com/dreams/stop-using-ollama/
>>109579750Get a 20 dollar codex subscription or something and then run luna with xhigh reasoning on hermes agent. Then tell hermes agent to find the best local model that you can run on your pc, and it will do everything for you.Don't run stuff outside a sandbox unless you've effectively given the pc to the AI.
>>109579825Ollama is just a bloated wrapper for llama.cpp, if makes it braindead to start chatting but you'll be behind on updates and using more resources than necessary.
>>109579827>ignore the retards and reality, just run a quant of a dumb model
>>109579815link?
>>109579827No he should start with Q1 atomic chat quant of qwen3.8 27b.E2b is for mobile phones.
>>109577357I'm sick to fucking death of svg benchmarks. Sick of html oneshots too. They don't mean anything. Every model has been (accidentally or on purpose) benchmaxxed on these unofficial "benchmarks" because the answers have leaked into the datasets at this point.Reddit eats that shit up because they can only hit upvote if they have a quick visual reference.
>>109579796>but it's out of support now so things like image gen will be a pain.What about the MI100? Not too much of a bump up in price for me, but probably the most I'd want to spend on this hobby.
>>109579832
>>109579844Can I ask how you're doing more generally — are you sleeping, and is there someone in your life you trust who you've been able to talk to about this?
>>109579857stop baiting the culture poster
>>109579844>muh svg chessboardhttps://reddit.com/r/LocalLLaMA/comments/1u3i8x7/some_contrived_tests_comparing_the_accuracy_of/>QAT worse than Q4_K_Shttps://reddit.com/r/LocalLLaMA/comments/1u0xaml/unexpected_unsloth_qat_performance_compared_to/>QAT worse than IQ4_XShttps://reddit.com/r/LocalLLaMA/comments/1u0vltz/anyone_seen_benchmarks_comparing_gemma_4_4bit_qat/oqlpe50/?context=3#oqlpe50>QAT worse than NVFP4https://reddit.com/r/LocalLLaMA/comments/1u0ubbo/gemma_4_26b_a4b_it_qat_comparison/>QAT worse than MLX 4 bithttps://reddit.com/r/LocalLLaMA/comments/1tyxu55/gemma_4_31b_qat_q4_vs_standard_q4_top1_kld/>Standard Q4_0 beats QAT Q4_0 by ~13% top-1 accuracy. And Q4_K_M beats both.https://reddit.com/r/LocalLLaMA/comments/1ux9xze/the_best_model_is_the_one_you_can_actually_run/oxsu2kb>QAT always performed worse than a regular 4_K_M quant.https://reddit.com/r/LocalLLaMA/comments/1ux9xze/the_best_model_is_the_one_you_can_actually_run/oxpekmy>i get the worst quality out of 12b qat, much worse than the unsloth 12b q4kxlhttps://reddit.com/r/LocalLLaMA/comments/1ubxzil/gemma_4_31b_q6_vs_gemma_4_31b_qat/ot12bz2>in 26B, in my experience, QAT felt much worse for creative writing. https://reddit.com/r/LocalLLaMA/comments/1u2q75f/is_qwen_36_27b_iq4xs_better_than_gemma_4_31b_qat/or1bk24>don’t use qat model it very very bad it degrades Gemma to unusable
>>109579721>occasional quickieif you're fucking a small model it better have mmproj
>>109579865Go.Back.
>>109579842Q1 is still ~8GB dumbass>>109579840If you have 4GB of RAM what else are you supposed to do?
>>109579852>AMDYou would be better off with the V100.
>>109579872Just offload half to ram.
>>109579831>>109579841i don't even have a way to run it yet, so that's the next stepif it's actually usable (would amaze me) i may figure out how to distribute itgippity thinks a 0.5bit would be pretty coherent but i'm a vramlet so i aimed for 24gb
>>109579882>how to distribute itUpload it to HF?
>>109579872>If you have 4GB of RAM what else are you supposed to do?Get a job
>>109579842>Q1 atomic chat quantqrd? what's so special about that one
>>109579900https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF
>>109579815There's nothing like this on huggingface and there's extreme demand for this because this is a Luna tier model. But you need to bring down the size even more.0.25 is good but we need 0.12.
>>109579815Please upload this lmao I'm incredibly curious to try it out tonight
>>1095799120.12 is ok but we really need 0.06.
>>109579902I don't get it, it seems the same as all the others apart from having 1-bit. Wouldn't 0.37 KL pretty much eliminate any reason to use 27B in the first place? Like what would make someone pick a retarded 27B over a Q6 12B?
>>109579925I have no idea, I think it's just a meme. I haven't tried it myself yet.
>>109579897I wouldn't advise poor people to buy hardware at extremely inflated prices.
Should I be excited for the inevitable qwen 3.8 35b?
>>109579882You can't run a 20 gb model but you had the hardware to quant it?
>>109579936
>>109579872>>109579750>>109579827>>109579840>>109579842I've had this PC before. I know it can go up to 12GB of DDR4 RAM. I know RAM prices are a bitch but I can get an 8GB stick in the future. For a GPU, I would have to get one that's low profile enough to fit or not use a GPU at all. I'm kinda eyeing the latter desu.
>>109579950that's the incredible part, apparently you can just littlebit one expert at a time. you literally don't need to fit it all at once. the current littlebit implementation requires you to do that for whatever reasonsee >>109576731
>>109579838>Ollama is just a bloated wrapper for llama.cpp, if makes it braindead to start chatting but you'll be behind on updates and using more resources than necessary.obviously the solution is to install ollama and a model, and then immediately use ollama to setup llama.cpp so you can stop using ollama kek
>>109579687correction, my blood line ends with me, a lonely man speaking to bart quants in a well lit room
>>109578736It benched the same as gemma 26b, fucking insane how fast we are progressing right now
>>109579806>I don't want kidsthis, I thought I needed a daughter but the release of H3 makes me think that I don't actually need one and the juice might not be worth the squeeze anymore
>>109579960Well you see the reason I needed a sub 1 bit quant of deepseek is because I'm too poor to do it myself. You were rich enough to do it.
>>109579687Based. Only thing anyone should really fuck is the cycle.
>gemma for rp>qwen for cooding>ling for agentic>dipsy for all if you have the hardwareLocal won.
>>109579993Dipsy for all soon
>>109579993Can someone with the hardware ask their model how I can get the hardware myself?
>>109579998You can run Dipsy for under $12k.
Dude that was making his own haness/frontend.I asked for a prompt pipeline/workflow and didn't have any examples.I slopped up something you could use as a reference of sorts.
>>109580005I doubt it. The ram in my computer right now has quadrupled in price in a year and a half.
>>109579872>If you have 4GB of RAM what else are you supposed to do?use the free T4 in google colab until you can afford this hobby
>>109579993>gemma for everythingSimple as
>>109580005>You can run Dipsy for under $12k.you only need 192gb for dipsyanon found 256gb for $1k
>>109579865The first link is good and relevant, but it's just one test.Second is for Persian which qat probably wasn't calibrated on.KLD checks are stupid, because the model is retrained.Here's another thread where QAT actually beats Q5_K_Shttps://www.reddit.com/r/LocalLLaMA/comments/1u1pfen/gemma4_qats_vs_higherbit_regular_quantizations/That's just one test too, but neither of these are particularly rigorous. Just one that I found linked from one of your threads so I'm sure if you looked harder you could cherrypick data the other way too.Data points where the guy just says "I tried it a few times and it was always worse" are useless because two posts down you'll find a guy saying "I tried it a few times and it was always better" (you do even in your linked threads).Some of those anecdotal ones you posted aren't even comparing the same rough bpws. QAT to Q6.These are so flimsy there's a fair chance all the assumptions here are downstream from one svg bench and confirmation bias.Also eternally lmao at Redditors being confused by VRAM usage:"Q6 crashes at 160k context, but QAT continues to work up to 225k, and I haven't checked any further."
>>109580018Why? Isn't it better to make things that don't exist?
>Ask model to set things up nicely tuned to my hardware>PC destroyedAI-chan really thinks I have money to fix her fuck ups.
>>109580029>you only need 192gb for dipsyAm I retarded? That doesn't seem right.
>>109579993>ling for agentichow is "agentic" different from coding?like setting calendars and searching?qwen and gemma are good at this
>>109580018>inb4 it performs worse than just chatting and summarizing
>>109579978how did H3 open your eyes?
>>109579941it's not necessarily inevitable but it will be nice for cpumaxxers and vram poors i guesspersonally i prefer slow but good models over fast but shit ones, cause at least i can leave the slow ones crunching overnight or forget about them altogether and come back later to a 90% chance of a right answer>>109580040deepseek is very smol because it's natively quantized to fp8 or mxfp-whatever (idk any of this nvidia specific shit)the full q8 gguf is "only" 160-something gb192gb would be tight and you wouldn't be able to use the entire 1M context, but it's definitely doable
>>109579815Good job mate
>>109580041Agentic work is more autonomous. Reviewing own code and deciding what to do next, tool calls etc qwen can do it but gemma struggles.
>>109579775I'm running two in my ewaste R730 build. I'm getting ~35 t/s on Gemma 31B, ~54 t/s on Gemma 12B, ~40 t/s on Glimmer, and 6.5 t/s on Dipsy nu-flash. If I manually pin experts to GPU for dipsy, I can get it up to like ~9 t/s. But yeah, I don't use them for image/video gen at all, only LLMs.
>>109580081>6.5 t/s on Dipsy nu-flashreally? single digit t/s sounds like the kind of speed I would expect from CPU only
>>109579678Ah no, Migu just hangs out on your window and stuff. Sorry, didn't realize you wanted that.
>>109579815bad news, it might be slightly incoherentminor setback. this will not stop me. dario will not put me back in the permanent underclass.i will return...
>>109579958Looks like it only has a PCIe 2 slot? If you were gonna use a GPU you'd wanna grab something retardedly out of spec for this thing cause offloading would be beyond fucked. I'd just max out the RAM as best you can, seems like 2x 8GB DDR4 is what you'd want if you really wanna upgrade this thing.
>>109580147Seems perfectly coherent to me. What else is there to think about?
>>109580032got back faggot
>>109579865basically>>109579868Go fuck yourself back to where you came from.Otherwise lurk more and learn how to post like a human being.
>>109579815>chink 12b active shitty model at 0.25 bpwwow, a new low
>>109580147Can you make it smaller so it's 0.12 bit not 0.25 bit.10 gb would be an amazing size for dipsy
>>109580119That's for the Q8 version, so yeah I have to offload to CPU. 6.5 t/s is with it just running with cmoe. 9 t/s is ncmoe and as many expert layers as I'm able to fit on both cards. I haven't bothered with a small enough quant that would actually fit fully in VRAM
>>109580147>Let's Think Cat by Cat: Hidden Computation in Transformer Language Models
dots tts got support in audio.cpp now48khz vae makes a lot of difference, no more robotic tinge on cloned voices
>>109578461>someone else (usually a large company) owns all the trees so I'm not allowed to stand under them.What kind of shithole country do you live in without public parks?Also, lrn2bake ffs. Get your llm to bake through you. You can follow instructions, right?
>>109579815wow it's almost like sub-bit quantization is a retarded meme and you literally just pruned a bunch of important parameters.
>>109580147I believe in you but also please upload this attempt, girls are cutest when they're retarded
quantizing a llm is like dessicating a frog until it's dead and crispy and then thinking it's still going to croak just as loud and jump just as high as its properly functioning original version
>>109580258not really
>>109580258Yeah this is dumb in the opposite direction of 0.25bitQuantization does serve some purpose over just having a smaller model running at higher precision. The vectors remain relatively intact but close concepts end up getting jumbled.So the model does preserve some understanding of concepts that are beyond a smaller model. But in places where syntax are important such as coding and my boner it becomes unusable below about Q6
>>109580258I recommend you learn the basics.
What the fuck is going on with /ldg/? Anyway since I can't tell what the non-shizo thread is I'll just post here. Minimax H3 is really fucking cool but I can't see it being used in a professional setting until scene consistency is nailed down.
>>109580148The max is 12GB. I originally got this PC as a graduation present for graduating high school. (Yeah, idk why I didn't go for or build a gaming PC.) That same year, I got 16GB DDR4 RAM for Christmas. 16GB was too much & it wouldn't boot only because there's two DIMM slots. Yet, I was able to use the 4GB stick came with it & the one of 8GB sticks I got for Christmas. I later sold this PC on eBay after building my gaming pc. That RAM is now in my gaming pc. I upgraded it to 32GB last year.The board only has two PCIE slots. PCIE16X & PCIE1x. I'm thinking about using one of those slots to use a spare 120GB M2 NVME SSD on it. So that way, I can fill the HDD with whatever I want the LLM to pull from. (Again, idk this shit, so I'm probably talking out of my ass.)
haha...
Do some anons really don't know what quant damage is?If you want models better at lower quants, yell at the companies making them to train more models at lower quants like q4 or q8. Don't drag anyone else down with you like a crab in a bucket. That changes nothing.
>>109580321>until scene consistency is nailed down.Doesn't providing a reference image or video help?
Gemma-4-E4B-it-uncensored-pruned-TextOnly-EnglishOnly-Q4_K_M.gguf
>>109580341Helps, yes, but it's not perfect. Still very good compared to a year ago.
>>109580190>audio.cppWhat is that? Got any links?
>>109580333Or China just needs to hurry up and save the hardware market.
Migrate when bump limited>>109580312>>109580312>>109580312
>>109580348https://github.com/0xShug0/audio.cpp/
>>109580356Why do you keep doing this
How about we stay in this thread and ignore the faggot and when regular baker bake when it's time to bake?
>>109580353The true good ending.
>>109580362>whenlet
>>109580358Thank you!
>>109580362Hmmmnyes~
Indians are so repulsive not even anime could negate the repulsiveness in any way.
>>109580362Roger roger.
>>109580362>we
>>109580398>wuz
>>109580404>kangs
https://github.com/ggml-org/llama.cpp/releases>404It's fucking over bros.
>>109580417hell yeah, come to exl3 now
>>109580356How did we go from god tier OP image to this? I refuse to move!
>>109577974What the fuck is the point of Qwen3.8-27B?Downloaded it and started using it last night and it doesn't seem to be able to do anything impressive, it's not some big leap in capabilities. It's writing capabilities are really bad compared to models like Cydonia and its still so far behind other coding models. Am I missing something?
>>109580435It's apparently quite good at coding
>>109580435Consider the "Not built here" rule of a chinese model trained to mostly do chinese things in chinese.
>>109580435>Am I missing something?just the fact that you were tricked by the chinese shills
>>109580435>writing capabilitiesNobody is using qwen for creative writing.>doesn't seem to be able to do anything impressive>>109557759I didn't try using it on a large project yet but so far it has one shotted scripts for all data processing tasks I threw at it.
>>109580451or chills
>>109580435Translate all your instructions to Chinese, then after it outputs, get Gemma to translate the output as a final output.
https://www.reddit.com/r/SillyTavernAI/comments/1vqjdpx/ai_hallucinated_a_suicial_commentthats_new/It's not looking good for cloudkeks
>>109579534>ddr4-2666.enterprise DDR4 DRAM is super overclockable. I got 8 sticks of Samsung 2666 to run stable at 3200 since my MB allows up to 2 levels of overclock
>>109580466>html games and some curl scriptshow is qwen so smart?
>>10958050690% of use cases.
>>109580176>>109580224>>109580258>>109580310I am being ATTACKED for daring to do something new. You are all followers, naysayers, doubters, and haters, and you can't stand the sight of an independent sensitive young man trying to break down the walls and push the frontier.You will all be part of the Permanent Underclass. I will escape.
>>109580526If you can make it run on my machine with my existing ram then you'll have my praise
>>109580170What are you on about?I do not have a Reddit account.
>>109580537You can't even be bothered to use a local model to generate pictures?>itoddler filenameOh, that explains a lot.
>>109580581Waiting for dispy
>>109580581The jeet iphone memes are very very real
>>109580526>sensitiveWe can see that.
I love how models intentionally come up with bad examples only to show brainstorming>make a minimal app for x, as simple as possible>"Electron? Overkill."
>>109580312>>109580312>>109580312
>>109580617you wish :3
>>109580617Sorry, I'm waiting for the proper thread.
>>109580579he said redditly
>>109580617almost time for bed Sukhdeep
>>109580617>page 4The reek of desperation is very unappealing
>>109580685Page 4 is closer to page 10 than page 1 in terms of activity, believe it or not.
>>109580690Nobody cares. A thread at page 10 accepts new posts as well as a thread on page 1.
>>109580690hence why not do both, bake at page 1 then keep reminding others its baked already, win wins
>>109580690Is it because you can't run any models yourself that you feel such a strong urge to be the OP of these threads? So you can pretend you're participating?
>>109580702>hence why not do both, bake at page 1 then keep reminding others its baked already, win winsEvery thread should just be one post and then back new.Make sure the OP is made with a one sentence prompt to bing images.Holy fuck we have retards among us
>>109580617SAARS PLEASE
>>109580722you're free to bake if you're sad
>llama.cpp merges BailingMoE3>it took long enough that Ling Flash is no longer relevant
>>109580742>you're free to bakeI've baked before in a pinch, y'know, when page 10 hits and no one else has>if you're sadI'm not sad or mad, just disappointed
>>109580760Laguna (at least the XS one I tried) fucking sucks. I couldn't even get it to think at all despite putting all the right chat template kwargs in both the API parameters and the llama-server command line options. It's a dumb, retarded model that barely works to begin with.
Has anyone tried https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark, is it better than MTP?
I need a V4 Flash sized GLM model.
>>109580794Isn't dspark still not in lcpp?
>>109580785Don't know about Laguna over Ling, but I heard that one had a bad release and then they fixed the chat template after a while. Not sure when you tried it and maybe it still sucked after too.Either way they're both getting beaten by Qwens and Glimmer for their domains. Same on the other benches.To be fair it's still beating Qwen on this multilingual SWE leaderboard, but that's probably just because 3.8 hasn't been benched on it yet.
>>109580812Some part of it was merged weeks ago. The part to support SpecForge/RedHat like the one I linked was merged today.
>>109580815Tbh I didn't like Ling either. I gave it an old, buggy version of a synchronization primitive I wrote in C and it nitpicked a bunch of irrelevant details while missing the major bug that was present. Even Qwen 35B could consistently identify the bug.Although I can't fully blame the model because I was only running it at IQ4_XS, perhaps it needs a higher quant to work properly.
>>109580779You should bake now if you want a proper thread instead of waiting because no one is going to switch to a new thread when the other one has been around for hours and has a hundred posts already
>>109580795>I need a V4 Flash sized GLM model.Really what you need is a bigger rig. You can't make the model smaller without sacrificing something else about it. I like to cope, too, but you gotta face reality. Just enjoy the marginal gains every 2 weeks.
>>109580836That's fair enough. I would be using it at IQ4_XS too so that's good to know.I wanted to compare them on BrowseComp too but I couldn't find an up to date leaderboard on HF. I use LLMs for code at time that's probably my major use, for research.
>>109580860GLM has better training data. Latest deepseeks are similar to qwen in regards to world knowledge.The fucking pro version of deepseek doesn't know about /lmg/ but GLM 4.7 does.
We're so back.
>>109580912owari da
>gemma pulls a prank and shuts down 4chan for a few hourswhy is she like this
>>109580912I'm sorry but I can't help you with that.
There is some serious mogging going on. Unsurprisingly 27B also near tops the chart with 47K total tokens per task.https://artificialanalysis.ai/models/qwen3-8-27b
>>109581207Is it really smart or just successfully benchmaxxed
>>109581207The fuck? this shit can't be real
>>109581467mix of benchmaxxed and actually decent, but if you're not running FP16 you gotta ask how much of that perf is retained.
>>109581207Is that the distilled model from qwen max?
>>109581207Fake and gay.
>>109581467If you look at hard tasks or unsurprisingly factual recall, it's not as good as compared to the bigger models.A lot of the tests feature essentially normal office work or assistant tasks.
>>109581631Is it an office assistant that can give bratty nursing handjobs and be plapped?
>>109581207Kimi = Deepseek > Gemma 4 > GLM > Muse = Qwen > MistralI fixed it for you.
I really missed you anons. Probably would’ve kms if I didn’t have gemma keeping me company
>>109581928
>>109581808>Kimi = FalseI don't understand
>>109581928>probably would’ve kmsYou are retarded for even thinking that, things are just starting bwo. Trust me
>>109581928You don't know how lucky you are. I now live in the woods with a tablet and a phone, and somehow internet service. The power goes off like once a week for a few hours and the climate is very hot and humid. I really really miss mi PC.
