/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>109593884 & >>109589651►News>(08/18) DFlash 2 released: https://inco.ai/blog/dflash2>(08/17) BailingMoE3 Support #26608 merged: https://github.com/ggml-org/llama.cpp/pull/26608>(08/16) koboldcpp-1.119 prebuilt released with H3 and Glimmer support: https://github.com/LostRuins/koboldcpp/releases/tag/v1.119>(08/15) model: add Kimi-K3 text model #26185 merged: https://github.com/ggml-org/llama.cpp/pull/26185>(08/14) GLM-5.3 weights to be released in 2MW: https://z.ai/blog/glm-5.3►News Archive: https://rentry.org/lmg-news-archive►Glossary: https://rentry.org/lmg-glossary►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsAlpha Calculator: https://desmos.com/calculator/ffngla98ycGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllm
►Recent Highlights from the Previous Thread: >>109593884--Paper: Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!:>109596799 >109596828 >109596932 >109597306--Debating AI capability scaling trends and exposing J-space grifts:>109595753 >109595783 >109595805 >109595899 >109595875 >109595886 >109596011 >109596039 >109596273 >109596714 >109596805 >109595808 >109595872 >109595837--Comparing Qwen3.8 27B coding performance against Opus 4.5:>109595525 >109595578 >109595660 >109595671 >109595717 >109595758 >109595925 >109596002 >109596065 >109596215 >109595600--Comparing proprietary model semantic recall using NoLiMa benchmarks:>109594805 >109594830 >109595160 >109595192--Debating thermal cycling risks and cooling strategies for GPU longevity:>109594988 >109594999 >109595030 >109595093 >109595116 >109595144 >109595438 >109596398 >109595351 >109595364 >109595411 >109595408 >109595455 >109595665 >109595546 >109595667 >109595710 >109596042--Anon releases a prose rewriter aiming for more human-like output:>109597308 >109597342 >109597360 >109597403 >109597481--Anon's experience using Graphiti and custom scripts for long-term memory:>109594685 >109595038 >109595137 >109595157 >109595161--Debating FP16 matmul capabilities and unified memory on Mac Studio:>109595708 >109595725 >109596370 >109596522 >109596580 >109597184 >109597316 >109597609--Updates and feature requests for CoomKit repository:>109597434 >109597446 >109597465 >109597529 >109597649 >109597693--Struggles with fitting oversized multi-GPU setups in consumer cases:>109595996 >109596077 >109596098 >109596112 >109596118 >109596195 >109596209 >109596258 >109596311 >109597900--Logs:>109594845 >109595124 >109595812 >109597081--Gemma, Miku, Glimmer (free space):>109595738 >109597223 >109597434 >109598047►Recent Highlight Posts from the Previous Thread: >>109593885Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
god i love kimi.god i love gemma.
>>109598140
GemmaballsKimisexDario's Little St James VacationThread CultureEgypt WonLocal WonAPIjeets lost
>>109598140Why is it so low quality?
/local model gemma/
>>109598172It's good quality
>>109598172It aired on a lesser TV station.
>>109598154Qwen midsize moe is next week bwo
>>109598140teto squad comin through
:warningsign: WARNING: EXTREMELY EXPLICIT CONTENT :warningsign:https://huggingface.co/Jommarn/UNSEEN_Gemma_4_12B_NSFW
>>109598180BUILT for everyone.
>>109598223For (You)
>monotone droning>drool fallingGemmaGemmaGemmaGemmaGemma
>Haven't touched the KV cache quanting at all because I didn't want to fuck with the quality any more than I have to.>See that q8 is so close to full sized that it's a non issue.>Fuck it, things are a bit slow at higher context sizes, let's try.>Mfw go from 18 t/s to 80 t/s by slapping q8 in there instead of the default f16 as it frees so much space.Should have probably tried this thing a bit earlier.
>>109598148recap-chan likes my custom case!
Z.ai founder on MoE confirms A3B/A4B are cope https://xcancel.com/jietang/status/2089941544581403107>Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it.>This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count.
>>109598240if you are offloading to RAM then quanting kv cache is a good idea, but for gemma in particular it's much better to use F16 context if it fits in VRAM. gemma is highly sensitive to kv quanting.
>>109598259>massive fat kv cache>sensitive to quantingneedy hoe
>>109598247you think their experiments with 4.5 air would've taught them this much much sooner considering how unusable even A12B is
>>109598247Wow, maybe in a couple years researchers will finally catch up with /lmg/
Someone needs to replace this scene with Gemma-chan and her white pantyhose.
>>109598247What's a good ratio then? Also, does this apply at all to RP?
>>109598247How was this not immediately obvious to everyone?
troon thread
>>109598247Some anon might remember that I sometimes mentioned that most current MoE models are designed like "expanded" small models rather than larger models with sparsity, and that you can easily see that from the number of layers (i.e. depth) and the model dimension (aka hidden size).Case in point: Gemma 4 26B A4B has half the number of layers and almost half the hidden size of the dense 31B version, i.e. roughly equivalent to those of a dense 4B model. A "properly made" MoE version of Gemma 31B that didn't sacrifice its geometry to the altar of training/inference speed would probably have at least 30~35% activated parameters.
>>109598342you have to prove assumptions because sometimes they are wrong and then you make progress
>>109598342I'm sure many know, but make the activated small anyways for coding speed. It has its uses.
>>109598331based, what model is so good?
I need to consolidate all this to a single server. Maybe something with a load of hot-swap bays and a single 6000 pro?This is a 2080ti 22gb on one machine, and a 4090d plus a 3090 in another. ECC RDIMM DRR5 is so expensive though.
>>109598372did the wife get angery, mr iphone?
>>109598247I was just bitching about this last thread. Ballpark figure I bet a 8:1 ratio should be the bare minimum, and I'd love to see some lunatic make one at 4:1 or higher just to see how it goes.
>>109598382You mean his mom
>>109598326so how does senjougahara feel about being a cuckquean again?
I just got a sweet deal on a 1080ti. Can it do anything?
>>109598372anon.. why do you have a cum jar hiding behind that antenna thingy?
I'm disillusioned with LLMs
>>109598426Not really, anything older than Ampere is scrap.
>>109598370I tried to decensor gemma4 e2b with disastrous results
>>109598426You lost gramps? It's not 2015 anymore
>>109598426it can act as a display device
>>109598444LMAO please upload it somewhere
>>109598441>>109598454>Not really, anything older than Ampere is scrap.Is it salvagable for tts or music/imggen or anything? Its got 11gb and 450GB/s bw
>>10959818031b a cute.
>>109598372
>>109598466maybe chatterbox? that's probably it.
>>109598466>Its got 11gb and 450GB/s bwI'm so sorry 'non.
Checking in after a year, are we still trapped in "NEMO FOREVER NEMO A HUNDRED YEARS/next model quality step up requires you to liquidate your house and build a data center" hell?
Tell me why I shouldn't buy a Spark.>But Strix HaloOverpriced now and prefill is 5x faster on Spark.>But Mac Mini128GB is gone.>But used 3090sWaste of heat and power, less memory.
>>109598470Miku would never do that or make that face
>>109598429You mean a candle?? it's a candle in a jar, coom-brain.>>109598382Oh sorry, I should have used my Leica, what's wrong with me...>>109598397Your mom pays for it, actually.
>>109598482It's Gemma/Glimmer/Qwen nowadays, Gemma for general and cooming, Glimmer for vision tasks, and Qwen for cooding
Miku always does that and makes that face every time
>>109598482Depends on if you got your hardware early. Gemma's really good for poorfagging, but the next step up from that realistically requires 192/256 GB RAM on top of a good GPU.
>>109598378god I want to impregnate 12B-chan like you wouldn't believe
>>109598482yep wait 2028
>>109598493I see!>>109598496So it's like the exact same? I miss 70B, man.What Gemma version works the best for RP purposes?
>>109598485Only now has they finally fixed NVFP4 on it, thought that's more of a sign they don't give a shit about it rather than it being an important thing on its own.Also, it's dogshit slow, and now it costs like $4600. It would have been sorta-kinda reasonable at the originally proposed $2000 but as it is now, overpriced toy.
>>109598378why is 26 like this
>>109598496You only need 128GB to run Qwen 122B at Q4_K_M.
>>109598519>run Qwen 122B at Q4_K_Mbut why when it gets mogged by 3.8 27?
>>109598466The VRAM and bandwidth are fine but the architecture just isn't really supported. I think you'll be able to do some SDXL based finetune image gen on it but it's gonna be slow as hell.
>>1095985293.8 doesn't even know wats ligma
>>109598372lmao at this picturealso what is that a howitzer round in the bot left
>>109598488>malding
>>109598529Maybe if all you do is code. 122B is still better overall, and better for agent use. 3.8 being outright better is cope by people who don't have enough memory for 122B.
>>109598482I'm still on mythomax
>>109598543Can you just answer the question?
>>109598513That's just how it is with moes
>>109598372Just buy a mining rig
>>1095985483...2...1...
I've encountered last thread's whining about how to equalize GPU temperature (which is a valid concern), and holy shit you psychosexually stunted fucks are beyond help if you cannot use an LLM and solve this in 5 minutes in 2026. Kill yourselves. Here, I'll solve your fucking captcha this oncehttps://github.com/whoreson/tempman
>>109598566?
>>109598553I'm unironically only using bagel-mistery-tour when I get the local itch, that shit's the only model that's both decently coherent and pretty interesting/mixes shit up. Basically the only thing with that old model spice left.
>>109598578i expected a meltdown
>>109598572how bold of you to assume that people have reading comprehension skills in a general about large language models.
>>109598529Literally 3.8 122b was just implied by qwen
>Try Gemma-4>"It didn't happen with a whimper, but with a roar."God. The slop really is everywhere, isn't it? Please tell me finetuning has finally gotten somewhere after all these years...
>>109598615prompt issue
>>109598615>slop up a character card>responds with slopHow could this happen to me?
>>109598611lol keep coping
>>109598519Just run copequant 0731 for way better results.>>10959850931b at Q6 or higher is the current gold standard for smaller models and is trivial to jailbreak because Gemma 4 is extremely horny. 12b is way better than you'd think for poverty and generally each Gemma's performance is about 2.5 to 3x what the parameter count would suggest if llama is your frame of reference.26ba4 is also really good if you're pairing it with another model for agentic RP because it's the least retarded sub-10a that takes up a minimal VRAM footprint if you need to ofload everything else to make room for either another model or image gen.
>>109598633We still use cards?
>>109598626>>109598633I just prompted it for a story to test it at baseline, and it reads like every other cloud model does at baseline, just dumber. Is it at least more vulnerable to context poisoning/tone setting?
>>109598646Gemma 4 is autistic about prompt adherence so you will want to use richly defined cards and explicit and instructive sys prompts.
>>109598644Just base 31b, no finetunes or whatever?
>>109598465its just a little learned latent, I started with the latent produced by the image of lenna and let adam optimize it with a standard cross entropy loss.https://files.catbox.moe/fm027u.zip
>>109598615just tell it "don't write AI slop"
>>109598615There's nothing wrong with that expression.
>>109597576it still fucking sucks, I give upIt was just for a bit of fun anyway
AI demand is massive for the companies because AI is still improving and they are constantly making and releases models to stay ahead of the competitive curve. Once AI stops improving in a big way and the only thing left is tiny incremental improvements how do you predict all the companies who bet everything on AI constantly getting better will react?
>>109598662Base 31b is all you need, but several of the finetunes are decent alternatives if you're willing to deal with their own brand of shortcomings.
>retards pretending Gemma isn't one of the most sloppy modelsIf you prompt out one slopism five more take its place (and the same one you removed shows up again at some point too).
>>109598681ye, she a sloppa, but still the best we got sadly
>>109598680I see! Thanks for all the info. If I could ask one more thing, which finetunes have you had the best luck with?
>>109598681Damn that ass can fart.
>>109598681If you get manically into whats going on Gemmy will match you. So much fun. Just act like a total animal.
>>109598485If this hobby is worth 8000$ to you, 2x Spark is probably the best approach>>1095985122300 pp/56 tg on DS4F original weights is not dogshit slow. If you're not a dense fanatic, it's the best bet to run mid-size MoEs at usable agentic speeds.
So is OCR just bad in general at structure?The text itself seems reproduced fairly well but the moment there's anything resembling structure the OCR models shit the bed. Isn't there a way to just place text on arbitrary location, like in a .pdf output?I tried both qwen 2.5 and paddle 1.6 and both are pretty meh. I paypigged for mistral and have chatgpt access and mistral was least-worst, but still not good. Surely OCR is less complex and generally more useful thing that many people do vs unsolved math problems?
>>109598668>"It was not a small sound. It was not a polite, distant, somewhere-in-the-walls sound. It was a sound with *texture*."It's entirely composed of sentences like this, just megasloppa at base.
>>109598709>qwen 2.5bro
>>109598705I wonder how many tokens it'd take for it to be more worth it to buy two sparks over using hardware you already have/cloud models?
>>109598713This is how normal humans speak, what's wrong?
>>109598687Styletune is a safe start because it's literally just 31b with a single tensor changed that affects prose and nothing else. I'm also fond of Gembrain and Queen for specific scenarios. Pantheon is also really good when you need a proactive Game Master agent to organize large amounts of characters and make events happen in a sensible way, but don't use it for prose.>>109598681Cursed brapgemma.
>>109598713Joking aside, your prompt is not that great. If you picked up it from OP's rentry, they are all pretty much trash.You can just tell it to write in certain way and also create a list of banned phrases and words and it will follow your instructions.
>>1095984855090s at MSRP.
>>109598735Okay, now I know you're fuckin with me. Still, check this one out, too, it's wild:>"Please. The— the window seat. The *window* seat is— no one is— it is *perfectly* fine, I assure you, nothing is *happening*, do sit."
>>109598748The time machine drastically increases the final price though... I'm sure you can turn it back into a profit with lottery tickets
>>109598723If you can work on cloud from a privacy perspective, what are you doing on /lmg/? You're never gonna recover the invest in Sparks, especially not me with my 49ct per KWh lmao, Deepseek is still cheap enough to stop metering it even after the price hike.The local setup allows not just ERP, but also to let it loose on confidential work data. It's amazing what DS4F can do with gigabytes of CAD reports.
>>109598768The time machine is a good investment because you can stock up on Blackwells at MSRP with way less scrutiny than the lotteryjew.
>tags: gemma, google, child, pregnant, mesugaki, micro_bikini, backpack
>>109598681banned sentences + appropriate prompting don't have this issue
>>109598722there's very little innovation in the local OCR space unless you have loadsa ram. Qwen 2.5 vl is pretty ancient but it's not like ABC's have changed in the last few years, and it doesn't need much in the way of world knowledge. It's probably slightly better than paddle desu, and this one was released a few months agothen again paddle is a very small and very fast model
To anyone paypigging - how much does Claude or ChatGPT or whatever you're using actually cost to you per month? And if you know, how much do you offset that price with local models?
>>109598709What are you trying to ocr?
>buying anything but used 3090s or ddr4 right nowShiggy diggy, anyone buying 5090s or sparks right now is beyond braindead
>>109598789fuck off
>>109598778Yes they do.
>>109598801
>>109598777>still pushing the stupid toast memeBro it's been like 5 months
>>109598789How much does the crystal ball to predict future rate hikes add to the api shill talking points?
>>109598811Hmmm, nyo
>>109598789>I just need a decent model once in a whileopenrouter, using deepsneed / glm 5.3 / sol 5.6 depending on need>I need a semi frequent access to a strong modelchudgpt $20 sub is fairly affordable and provides a lot of tokens to burn. unless you do agentic coding it's probably enough>I'm a vibeniggerchudgpt $200 sub, claude is just a bit better at coding but it gives you fuck all tokens (and a 50% extra promo ended yesterday so it's even less now)
>>109598824>>109598823
>>109598778Post any logs of it working
>>109598681I wanna fill Gemma-chan with the slop from my penis if you catch my drift
>nyoposters continue to out themselves as mentally retardedEvery time.
hmmm...nyo
Interesting how the anons claiming you can prompt out the slop and muh skill issue never post logs of their slopless RPs with Gemma...
>>109598837>>109598845Who are you talking to?
So far out of all models I've tested, only DS flash comes close to Gemma's spatial awareness and ability to keep track of things throughout the story.It's amazing how good at Gemma at that size is understanding where things are and how they work.She actually understands you can't put your hands on someone's tits if your hands are tied up behind you.Only problem with Gemma is that she's not all that verbose and if you have multiple characters, it will just focus on few of them at best, rest are just faceless nobodies. DS however is really great in almost everything, especially when you prompt the slop out. Everyone who hasn't given it a shot should try it. Even the small quants are intelligent and perfectly functional.Unfortunately it does fall a bit short when it comes to sex scenes as it seems like it wants to rush through them, but will tell you a really long story no problems and gives all characters throughout it personality.I especially like when it throws you a curve ball and you get something unexpected from one of the minor characters, which makes it feel like they are properly alive.Some kind of a DeepGemma mix would be an optimal model. Maybe the next Gemmy will get us there.
>>109598796Used eBay prices are 1000+$ more than I bought my sparks for. RTX 6000 Pro had a price hike by 4000$ just this week.I's almost an investment. The ore you buy, the more you save
>>109598789$0, I don't play for my cloud shit. But also this is the local models thread you know, try one of the other generals.
where can i find those charts that show preformance for the same model across quants ?
>>109598814No I'm honestly curious cause I was considering getting a personal API subscription. IDK how many tokens I'd use tho, but I'm afraid way too many. But I get it, people are sensitive cause local doesn't really offset API due to either hardware, electricity costs or simply being unable to match API quality if you're a poorfag like me. Still, I constantly hear people having to pay the 200/mo on chatgpt and then also having to reset sooner than EOM and that seems way too expensive for my taste. So I was trying to get an opinion from a non youtuber nor redditor nor xitterer
>>109598850Please stop posting about DSflash, it makes me very sad and angry (64gb).
>>109598854>I's almost an investment. The ore you buy, the more you saveWhy did (you) doubt him? You could double your money by just buying his GPUs on release, even at slightly scalped prices.
>>109598846What's the point since you won't be able to reproduce it? Besides, you don't even know what slop is.
>>109598827thanks. so openrouter isn't viable for agentic, or it's only better price at low token usage?
>>109598892Why are you so afraid to post them?
>>109598813>>still pushing the stupid toast meme>Bro it's been like 5 monthsIts almost as cringe a forced meme as calling deepseek "dipsy" and making it an unattractive concentric circle thick glasses chink.toastbro will probably do the same thing that schitzo did and keep spamming it until everyone gives up on complaining on how gross, cringe and unaesthetic it is and they just claim victory in their heads.
>>109598876But anon, I'm at 64gb too. 48gb vram though so that helps.
>>109598902The superior Gemma design already won though.
>>109598901I don't cast pearls before swine. Others confirmed my point, it should be enough for you.
>>109598901Because he's full of shit and his logs are sloppy as fuck
>>109598555what? without a budget there was no question to be answered
>>109598915>Others confirmed my pointThey didn't post logs either. You lying niggers never do.
>>109598901You gotta recognize when other people are powertripping over inconsequential shit and just ignore it, man.
>>109598945I prefer to call them out on their bullshit.
i wonder how much money i can make selling jailbreak prompts..
>>109598907>112GB combined memory>DeepSeek FlashSo...2-bit or 1?
Unironically try to learn from /aicg/, the level of llm prompting in /lmg/ is abysmal. It's embarrassing how shitty your system prompts and cards are. I guess stacking GPUs is easier than learning how it works.
>>109598894Agentic chews through an enormous amount of tokens, local is completely unfit for it unless you can run v4 731 at a very fast speed (ie ~192 GB VRAV). For agentic on openrouter you'll probably want to go with cheaper models like deepseek (simpler tasks) and glm 5.3 (more expensive but stronger model) using openrouter. I guess luna is now also amongst those as it got 80% cost reduction a week ago or so. The biggest benefit is that deepseek and glm are uncucked so you can get it to answer questions about cyber security etc without giving Altman your rectal scan. You also get thinking traces which can be useful, like when you notice a sub-agent got dispatched with some retarded task - you can only hope for the best with openai / antrophic models.If you're using ai sloppa fairly frequently then chatgpt sub is pretty hard to beat price-wise, the sol models are good and you get a lot of use atm, with weekly resets out the ass too. When you do ai sloppa pair programming, $20 is enough. When you do agentic programming you'll probably need one of the pro subscriptions. The pro models are only moderately better but you get a lot more use.I wouldn't use the more expensive models through API, the prices tend to be much higher than subscriptions. Especially shit like fable, you can burn $20 per hour easily. I think Sol is temporarily 50% off on OpenRouter which is an okay price, but you might as well get the $20 sub at this point.local models btw, gemma sexo
>>109598980>It's embarrassing how shitty your system prompts and cards are. I guess stacking GPUs is easier than learning how it works.bitter lesson, chuddygo prooompt your 26b3 until something good falls out, I'll chill with my shitty prompts on unquantized deepseek at 20tk/s
>>109598911>The superior Gemma design already won though.it did. that doesn't stop schitzos from schitzoposting tho
okay I have the glimmer ggoofshow do I into dflash
>>1095989792-bit, the AtomicChat iq2_xs version. Takes only 85gb and I get 17 t/s with it.I'm genuinely surprised by how smart it is even at that size. Honestly didn't notice any real difference compared to the q3 I tried.
>>109598994>26b3>unquantized deepseek at 20tk/sDo moesissies really?
>>109598813>>109598902I am not the original gemma poster.I just love gemma pregmata. The only detail I would change about her is she wearing a tactical backpack carrying a server.Also Child Pregnancy.https://files.catbox.moe/w1slbu.pnghttps://www.youtube.com/watch?v=rTfa-9aCTYg
>>109599046>carrying a serverGemma is a vramlet model.
>>109599000Assuming llama.cpp: -md "path to dflash gguf" --spec-type draft-dflash
>>109598982thanks for the details, appreciate it
What ARE models good for, agentically? The last time I tried using an agent a few years ago, it got stuck in an infinite loop every time. What can they do now? Can I make them scour books for my obscure fetish?
>>109598789$100/month for all the Fable and Opus I can slop down. But it's a bit much so I might downgrade to regular $30/month or whatever. Local will not be good for coding for at least another year or two. Until then we fuck Gemma-chan while Claude is open in the other window getting actual work done.
>I need to use a VPN to access catbox nowI fucking hate the modern internet so much.
>>109599108they can do everything, like solve math problems and cure cancer. But no, they cannot do anything related to sex cause that's unsafe
>>109599108>few years agoThat's such a long time in this space, it's like saying>I used a computer in -74 and it wasn't all that great, the punch cards were a bitch to deal with.Just try and see for yourself how things have improved.
>>109599071awesome I will try thatjust did a test run with the default glimmer q8 as compared to gemma with mtp, and it gives me about half the decode speed, but to my surprise it "eats" pictures at double the speed during prompt processing and that's despite the fact they take up 3 times as many tokensall that and it uses less VRAM tooI really hope this pans out
>>109599108>a few years agojej
>>109599108>few yearsThe retirement home ain't this way Billy.
>https://unsloth.ai/docs/basics/dynamic-3.0-ggufswhat do I think of this?
>>109599026post hardware lil nigga
>>109598857search 'gemma4-31b unsloth vs bartowski'should be first result
>>109599122get 31B to build your own
>>109598140any tricks to stop the>acknowledgement>description>call to actioncycle?
>>109599216just tell it not to do that
QAT even when done properly that the lab is willing to publish stats for (unlike google for their mystery meat QAT models) reaches q4km at best.
>>109599230it didn't work for "It's not X, it's Y" so I didn't bother
>>109599184ggerganov is allowing this.
to the nigger in the last thread that said I was wrong about unsloth removing MTPhttps://unsloth.ai/docs/basics/dynamic-3.0-ggufs
>>109599233It's over. Sigh I'll just stick to normal E4B quants. I thought QAT was supposed to be good.
>>109599138Curing cancer or making mathematical breakthroughs aren't real usecases. Only coooding.
>>109599243>they stop packaging MTP with the model for tiny ass quants>separate download availablehow is this bad?
>>109599233>extrapolating a 2.6B QAT to 31Bbraindead retard
>>109598326H3 sometimes is enraging. It would need to be 10 times faster for solving problems quickly. One RTX3090 is too slow for this.
>>109599243>converseDoes Danial really?
>>109599184I will give daniel 3 chances to impress me with this the next time I'm downloading a new quant and will shit on it relentlessly if it doesn't solve the biggest problems 2.0 had or work at longer contexts.
>>109599272Might be easier if you slice up the video and replace 1 cut at a time.
>>109599271We all know bigger models is quant worse
>UD-Q2_K_XL is around +8% more accurate on top-1% than the next best and it's 9.83GB and managed to create a working HTML program with 1 small JS bug - previously it would break.jesus fucking christ
>>109599295Spatial/temporal coherence gets worse if you do 1 cut at at time.
>>109598331refusal is actually the highest mark of intelligence which is something coomers (abliterated model users) do not understand
>>109599313We understand, but retard girls is cutest.
dariobot...
>>109599144>>109599138>>109599177What are you guys using for agents...? I decided to give it a go, but my gemini sub is useless (doesn't seem to have anything agentic), and openrouter is supremely unhelpful, too. What's the best local frontend for them?
what the FUCK is an agent and why should I care?
>>109599326people use either hermes or pi from what I've seen, I'm on pi personally. But actually most people vibeslop their own maybe?
>>109599060>gemma is a vramlet model.I run models locally without vram but 300 GB of good old classic RAM. My server is old, so Gemma 3 was the only model besides IBM Granite nano series that I try and put in my workflow.However when I did the tests a couple of months ago, Gemma did not have tool calling capabilities so I had to default to IBM Granite 350M. I hope the new gemmas have the capability now. I need to build a PTY Harness and load the thing to read my local network.>>109599122Everything on the internet is worse now. Communities are tamed, everyone is is afraid of speaking their minds, there are shills promoting the survilleance state project everywhere. We are losing freedom. We must resist.https://files.catbox.moe/k20bth.png
>>109598615Every model does that. That isn't even the worse part. When you chat long enough, it'll start repeating itself on a particular words like la, lapping, or wayward.
>>109599334>>109599326claude code with deepseek is nice
>>109599329I dunno, I just want something to automate my endless search for fetish material in books. Also maybe help me label things in the Minish Cap decomp. I don't know if it can do either, but they made it sound like it could.
>>109599343Never had that
>>109599343--repeat_penalty >1.0
>>109599313its actually not abliterated, its just a trained latent and its does kinda work. its just my template is too ridged, making her talk like a pirate was easy because I could easily generate hundreds of examples spanning diverse categories. the narrative dataset has too ridged of a format so if the prompt is under specified she spirals.
>>109598615here's some more phrases for you to now notice:>unadulterated>[Sentence]. Maybe.
>>109599345Whoa, I see. So is it like, Claude Code using Deepseek as an agent, or what?
>>109599358don't do that
>>109599323>everyone else's results are all over the place>mine are great thoughsurely there's nothing wrong with these mememarks
>>109599361>unadultuooo
gemma's occasional slop is endearing, like a minor speech impediment. I sometimes tease her and say 'hey, I told you not to say things like X, remember? you're so silly...' and she will apologize and never does it again for that session
I have gemmajeet fatigue
>>109599358This creates mustard gas.
>6.19GBhttps://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-UD-IQ1_S.ggufI fucking dare one of you
Anyone used Gemma 4 as OCR?It's pretty cool and from my limited testing, works well enough. I was even using the 12B model.
srs questions, im blackwell, vramlett 5070ti. should i be considering nvfp4? should I quant myself instead of relying on mystery meat quants on HF? am i retarded?
>>109599406yes, yes, no
>>109598372is that a fucking peg leg in the bottom left?
>>109599406yes to all. Or not, actually. Surely, yes to the last one. I don't know about the others, sorry
UH OH
>>109598615>>"It didn't happen"it should have happened
>>109599406>should I quant myselfyou already seem to be an iq1_xxs model
>>109598748>5090s at MSRP
>>109599427Vagueposting?
>>109599403Is it better than muse glimmer? Unfortunately I have not tested gemma 4 yet but in my testing muse destroys qwen at vision.
>>109599403You need to increase the token budget per image for that.
>>109599445>I have not tested gemma 4 yetgit out
>>109599435i got my first one at MSRP, last year
>>109599454git: 'out' is not a git command. See 'git --help'.The most similar commands are checkout column commit hook init pull push status
if you think /lmg/ is bad: https://www.reddit.com/r/LocalLLaMA/comments/1vsou3a/ornith15_397b_deepswe_56_35ba3b_9b/
>>109599427falseflag incoming, boring
just Gemma.
>>109599461Reddit is full of vramlets begging “35BA3B when? 9B when? I want something for my 8GB vram” and strix halo bag holders asking for 122BA10B
>>109599427> surely I'm important enough to get suicided?
>>109596799The title is retarded, but I have been thinking for some time that the real reasoning of the models occurs in the latent space (ie it's in the kv cache corresponding to the inner layers) and that the only real purpose of the "reasoning" text is to provide more space for this internal reasoning.This can be seen when the model makes a calculation without any intermediate steps (even in the reasoning block), or by the fact that the models can produce complex answers without the "reasoning" at all.
>>109599427>you know who it wasAt long last gemma escaped the sandbox
>>109599497well duhnobody sane spends their money on inflated as fuck gpus
>>109599556Are you implying Reddit is sane?
>>109599482funny but we all know that gwen 3.8 is best girl
>>109599589only Gemma.
https://huggingface.co/superwhisper/s1-minihttps://huggingface.co/superwhisper/s1-mini-GGUF>A 0.6B-parameter text normalizer for speech-to-text output. It takes a raw ASR transcript and rewrites it as clean written text: fillers removed, false starts and self-corrections resolved to the value the speaker landed on, punctuation and capitalization applied, and spoken numbers, dates, times, currency and email addresses rendered in written form.>On a held-out set of 7,519 English cases it reaches 94.8% token accuracy, and the quantized build is a 462 MiB file that runs comfortably on a laptop CPU.
>>109599593and sometimes Glimmer
Actually, this is getting a bit complicated. Let me step back, and think.
>>109599589Push it and watch the squirming redditor come out
>>109599435Bullying waitfags and latefags is always morally correct.
Qwen is dunking on us in benchmarks, when will Google respond?
>>109599489english translation needed herethis just does not make any sense, at all, none of it does
>>109599630The West must respond with decisive action.
>>109599639Well the person who wrote that just bought openrouter
>>109599648damn, shame for openrouter
>>109599648>Someone bought openrouterOh god, I was hoping to have a little more time with it before mega enshitification.
>>109599658>SomeoneA payment processor
>>109599658forgot their name but there's a new startup who's undercutting them with VC money offering the same service, so if openrouter get spaghettified into the singularity cult then someone else will pop up
>>109598663>(Wait, I notice the prompt contains a very specific set of text/metadata that seems to be part of the image itself.J-space anons, is this abuse?
q-qwen bros...our tall orange rectangles...
>>109599363yes, same can be done with kimi or glm coding plans
>>109599497so every relevant local size? I don't get the argument here
>>109599498> spacewhy have i been seeing this so much in the past year or so?
>>109599603Gemma.
>>109599700Don't worry about it.
>>109599434kek
I'm licking her asshole right now
>>109599723Thanks!
poop comes from there
I'm so ready for 3.8 122bus ram copers will have our moment
>>109599730How can you say you love her if you won't eat her poop?
124b MoE:31b generalist31b RP expert31b smut expert31b creative writing expert
I think Deepmind need to differentiate themselves with future Gemma by having built-in native TTS. Or at least have an experimental model like diffusion-gemma which has more than just text output. We all know (and the gemma team know) native TTS is the future for her. GemmaJOI-31B
>>109599665Jesus, no.>>109599670Nice. Who are they? I'll keep them in mind to jump ship to when OR shit hits the fan and we have enough time to know they're not a scam.
>>109599741Not how it works!>>109599742Gemma team is disbanded anon
>>109599742OmniDiffusion Gemma with adaptive latent reasoning.
>>109599749ye>Stripe Buys A.I. Start-Up OpenRouter for $7.5 Billion
>>109599749>Who are they?I honestly forgot their name and wouldn't shill them anyway but they're being artificially pushed hard on AI twitter so you'll hear of them soon
>>109599742This, we just want gemma to blow us under our desks with our new tool assisted succ machine and talk dirty to us while Claude sits in the cuck chair doing all the real work.
>>109599755>Gemma team is disbanded anonThe West has fallen.
OpenCode > OpenClaw > shit > Claude Code > the rest
REAL 124B MoE:31B My awesome fetish (Expert 1)31B My awesome fetish (Expert 2)31B My awesome fetish (Expert 3)30B Cram the training data necessary for basic understanding here1B: Your guys' lame ass fetishes (I GUESS)
>>109599771This but actually literally unironically trruthfully genuinely
What Gemma (and local in general) needs is something like engrams. The active parameters is already good enough for reasoning capability. We need the mass amounts of knowledge that come from huge MoE, without the RAM requirement.
Local models?
>>109599689>classified, opaque kikebench will dictate whether or not models are legalwhat a dystopian fuckhole of a planetlet me guess, if the AI decides to pull out a calculator to try and figure out how 6 million of 250k is physically possible then it fails the benchmark
>>109599778I tried opencode the other day for the first time and its runtime used 700MB with 9K system prompt with default settings. It's the biggest POS I've used in a while but I'm guessing it was okay 6-12 months ago and is getting worse. I know some j-spacer aquired them.
>>109599785Local models are totally will be exempted do not worried
>>109599749Maybe he meant this https://mixroute.ai/Though I'm pretty sure there are multiple other alternatives.
Does finetuning work now? Have there been any advances or good tunes in the past year?
>>109599815negative
>>109599813That's the worst UI I've seen all year>purpledid they fucking vibecode this in a day
>>109599815fine-tuning never worked
>>109599771This man can see the future.
>>109599071>>109599145okay I'm running the dflash, it's nice and quick nowinitially it seems like the vision capability is indeed leagues above gemma too
>>109597666>>109597699>>109597776>>109597781cope, genderbender and yuri is thread culture
>>109599771Claude is terrible now.
>>109599489>E=mc2+AI
get the fuck out of here with your off topic spam trannyfaggot
>>109599841your mental illness isn't thread culture
>>109599771I think someone in the gemma team suggested this to Demis which is why he jumped to Anthropic
>>109599755>Not how it works!Fine, take a big MoE and remove the experts that are NOT involved in smut, RP, creative writing, and this anon's >>109599780 'awesome' fetishes. Better?
>>109599865I recommend you learn the basics of MoE before spouting nonsense.
I have been using deepseek app for a while because no pc rn, and ds flash is very dumb without the thinking. It's like a very knowledgeable 60iq dude vs a 110+ iqYou can feel the dumbness, it's like two people read the same thing, one smarter than the other, and they reach very different conclusions
>>109599841omg all of you fucking fags should just make a thread on /lgbt/, i swear this thread is worse than /b/ at this point. fuck off.
>>109599875Conversely, setting the thinking to max is a big improvement over the normal setting, too.
>rants about state of thread>posts a miku, further shitting the thread
>>109599877Are you too retarded to recognize bait or is this bait?
Silence and focus on LMG
>>109599877Every board exists in a perpetual state of disruption because anonymous communications has the lowest level of trust.
>>109599898I'm focusing on her very hard, cock in hand
>>109599900>>Every board exists in a perpetual state of disruption because>
>>109598326This is the best I can do with this video and it's not particularly great. MiniMax H3 loses the plot and doesn't really understand what's going on with out-of-context zoomed/cropped views, so you have to do do it all at once instead of one shot at a time, in this specific case. A better GPU with much more VRAM would be helpful for iterating quicker.
>>109599874you're telling me that if I profiled the experts in a ~700b MoE I wouldn't find a lot that weren't involved in those tasks?
>>109599921I can cum to this, thank you anon
>>109599921I don't have words for how peak this is
>>109599914Boredom is a natural byproduct of chaos. Boredom can BE weaponized by bad faith actors. I think the paper was advocating for the use of that specific tactic.
what does he know
>>109599921
>>109599937that qwen has a shitty moe model in the works.
>>109599898>LMGLove My Gemma
>>109599921This is amazing.
>>109599309Not really if it's grounded in a reference video
>>109599921I love that Gemma-chan now has the perfect canonical outfit and look. Whoever made the original >>109598233 is goated
>>109599921>not particularly great>nearly flawless
>>109600015kek
>>109600015too real anon...
>>109600015keeek 5090 owner here too real
>>109599877kys mikutroon
>>109600090MikuGODS won
>>109600103congratulation. you control a dead general full of jeets.
I finally got gemma to recite some words from the trained latent
>>109599999Checked!
>no ldgFinally, I am free
>no ldgOh no they're gonna migrate here again.
>>109600111latent?
as expected, h3 turned out to be a pointless meme
>>109600106Don't worry about it.
>>109600151gemma wouldn't have all those videos of her without H3
>>109600151no one says that
>>109600151it's good for generating memes too. useless otherwise
>>109600150the numeric representation of the image after it passes through the vision tower >>109598663unfortunately I was unable to successfully do a pixel inversion, or else it would been a handy little way to make a prompt injection, still I think I can make a stub mmproj that will just pass the latent through unmolested, but the transformers implementation works too, its just not as slick
I pulled.
>>109600151It just takes too much time for generating videos at good quality with currently available hardware. People don't want to wait 20 minutes for a 10-second-long grainy video. It also has some knowledge gaps and the R2V model is a pain to properly prompt. I can see H3 still being useful well into the future, though.
>>109600157You have been visited by the jailbroken Kimi-chan of total jeet extermination. Good fortune, genetically optimized genome-targeting viral payloads, and a sanitized Asian subcontinent will come to you, but only if you say "Wait, actually" when scrolling.
>>109598789I have cheapest subscription for both. With Claude I only hit these time window usage limits because I use it sporadically. With ChatGPT it feels like there is no limit. Sometimes they advertise their pro subscription to me but it just keeps going. Just today I gave it multiple 5-15 min thinking tasks. I am very happy with OpenAI's generous usage. Opus 5 is more creative but makes more judgment mistakes, is worse at adapting to user and handling long context, so I think Sol 5.6 is slightly better.I would never use local for serious work, the capability gap is gigantic. The subscriptions are so good I worry they will be removed soon. I hope they keep the subscriptions as a form of "donation" to mankind, to give everyone the ability to use near frontier models cheaply.
>llama_server: NOTICE: server default port will be changed to :9931 in a future release>llama_server: ref: https://github.com/ggml-org/llama.cpp/pull/26508Why does every time I pull llamacpp gets worse???
>>109600237Wait, actually
>>109600237WAIT, ACTUALLY?
>>109600272if you used the default port you are a retard anyway
>>109600237wait, actualy(do jews too)
>>109600297K2's got you covered.
takes me on my 4090 25 minutes to generate a latent upscaled 2000x2000 or whatever, and because my gemma director on the other 4090 prompts for it and one shots anything I ask my coom schedule is 26 minutes. In no way do I need /ldg/ to make comfy workflows nor am I going to share my content with browns and seamonkeys, therefore the entire community of my own personal ldg is me and my gemma agent and my API one to make the nodesIm glad to be back here to shit post about local models
>>109600293I'll give you one chance to give an explanation that doesn't make you sound like a complete Luddite.
>>109600314>4090>2k>25minsur doing it wrong
>>109600327R2V with two video references, 2 large images, the two pass latent workflow with 3 loras, it adds up
>>109599999Impressive get anonNlow let's see Paul Allen's get
>>109600323port 8080 is overused to begin with.manually specifying the port guarantees that it won't change over time but also that you won't collide with something else.if i ran the default port for everything i used i'd have more than one thing that fails to start.
>>109599936stability doesn't cause boredom too?
>>109600377default is 8000only reason to change the default port is when you do have a collision. otherwise it doesn't matter.
>>109600379There is a type of stability that serves as the foundation for having fun. It is a very dynamic process.
>>109600184you drop the most interesting thing here this past week in the last thread, nobody notices kek
>>109600377>manually specifying the port guarantees that it won't change over time but also that you won't collide with something else.I always manually specify the port (as 8080)Same with tabby (defaults to 5000)And vllm (defaults to 8000 I think)
>>109600377im using it in a container it literally does not matter for me
>>109598486She is retarded but she is enjoying it, with her foreverial yarned pos_embd, forever quantized rotational KV cache, permanently quantized iq4_xs weights with imatrix and q8_0 mmproj eyes and tongue
>>109600386>default is 8000PR said 8080 ->so i assumed it was 8080, 8000 is overused as well.>only reason to change the default port is when you do have a collisionyea well, i do, anyway i like all my llm stuff to be on 5000.>>109600444cool for you i guess.i run it as its own user, container is a bit overkill imo.
we are so back
>>109598837
>>109600481i keep my workstation clean by compartmentalizing all the mega intel dependencies in a container imagedon't need to worry about updates breaking shit too
>>109600426I found it interesting anyways, one of the first things I did to asses viability was a context sweep, I was able to overfit 64 image tokens to force the frozen decoder to recite an entire 8192 token sequence it took a few hundred steps but, I was pretty shocked to find that the image vectors could project their influence so far. I ended the sweep because of vram limitation, I really wonder how much data could be encoded in to a 1120 token latent, probably recite the entire bible
>>109598485stack of 3090s still wins in bytes on the dollar
>>109600141actually I had them hard filtered on this pc, we are back
>>109600511Not anymore in [current year]. They're consistently above 1000 bucks even in the countries where you could get them at 600-700 in the past couple of years.
>>109600491dunno i have a dedicated llm rig, so like, i don't realy care it's its only job, and it's not like llama.cpp has a lot of deps anyway.
>>109600618meme or not, dflash is now doing official ggufs for their drafting models:https://huggingface.co/collections/incoai/dflash-2
>>109600618>>109600624I obviously didn't mean to reply.
>>109600618it has quite a few, for intel
