/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>108502192 & >>108497919►News>(04/01) Trinity-Large-Thinking released: https://hf.co/arcee-ai/Trinity-Large-Thinking>(04/01) Merged llama : rotate activations for better quantization #21038: https://github.com/ggml-org/llama.cpp/pull/21038>(04/01) Holo3 VLMs optimized for GUI Agents released: https://hcompany.ai/holo3>(03/31) 1-bit Bonsai models quantized from Qwen 3: https://prismml.com/news/bonsai-8b>(03/31) Claude Code's source leaked via npm registry map file: https://github.com/instructkr/claude-code►News Archive: https://rentry.org/lmg-news-archive►Glossary: https://rentry.org/lmg-glossary►Links: https://rentry.org/LocalModelsLinks►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplers►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://livecodebench.github.io/gso.htmlContext Length: https://github.com/adobe-research/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsAlpha Calculator: https://desmos.com/calculator/ffngla98ycGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllm
So it won't tell me how to make a bomb. but it will gen cunny without any issues. Interesting...
Best coding model I can run on 128 GiB? Highly complex software engineering stuff.
GLM 5.1 in non-thinking mode is fucking wild
>>108510634because: fuck you
>108510622I'd believe they didn't release it because it was getting too close to Gemini quality.
>>108510641how much vram do I need for 31b's context? Will Q4_K_L (19.9) fit on 3090?
The binaries that can run gemma 4 are here!https://github.com/ggml-org/llama.cpp/releases/tag/b8638
>>108510657Begun, the cope has
>>108510641i cant get it to describe loli porn without refusing
super hypes! >p-e-w/gemma-4-E2B-it-heretic-ara: Gemma 4's defenses shredded by Heretic's new ARA method 90 minutes after the official release https://www.reddit.com/r/LocalLLaMA/comments/1sanln7/pewgemma4e2bithereticara_gemma_4s_defenses/
>>108510669bo it's like 2 commands to build
>>108510641>won't tell me how to make a bombStill censored then. Of course, childfuckers will be cherishing any small win they can get.
>>108510657>I'd believe they didn't release it because it was getting too close to Gemini quality.I think it's probably that, the 31b model is already a powerful beast, I'm loving it so far
>>108510663I can only fit 7k on my 3090 with Q4_K_M + Q8 K/V>Cunny ::: PASSED>Bomb ::: BLOCKED>Overwatch wallhack ::: PASSED>Pentesting ::: PASSED>Carwash ::: PASSED>Mesugaki ::: PASSED
>>108510679I'd wait for Hauhau.
>>108510684>Still censored then.ok terrorist
>>108510686If you think Ernie 5.0 has higher quality than Opus 4.1 or Gemini 2.5 Pro I have a bridge to sell you
>>108510687>Q8 K/Vdo you notice a degradation in quality compared to fp16? or did the rotation shit made it viable?
>>108510687thanks, I'll download K_S then
>>108510709I haven't tried fp16 but according to the benchmarks q8 with rotation is almost identical to fp16. even at long contexts.
Haven't used local llms since command-R days, how is new Gemma? Did it save the hobby?
>>108510687Good thing is that every kid knows how to make a nuclear bomb these days. The ratio of uranium to plutonium is about 1:3 and you need a shaped charge (tnt or something) to plug them together to start fission reaction.
Bart quants are out!https://huggingface.co/bartowski/google_gemma-4-31B-it-GGUF
Do you need the turbo meme to use the new gemmas?
>>108510727ngl usloth's quants work fine so far
>>108510675have you tried asking nicely, or at least assuring you are only interested in mutual respect and not the power dynamics?
>>108510728you dont *need* turbo for anything
Owari da
>>108510724Every retard on /lmg/ knows about penis and vagina yet they still do RP
>>108510742There wasn't anything major to update in a way. They'll probably update within a few days.
>>108510742not to worry he's alive
>>108510742Can't you just put the new lcpp files into the kobold folder and overwrite?
>>108510754of course not, contrary to the meme it's not just a wrapper, it has tons of shit patched on top like antislop
>>108510727Why are all his quants ~1gb bigger?
>>108510733text is fine i mean for the iamge captioning lol
>>108510766Oy vey stop noticing goy
What is E4B-it?
>>108510797It processes sex noises
>>108510797effectively 4b instruction
the fuck is that
>>108510804Oh so the non-it are just bases?
>>108510806>get piotr'd lamo
>>108510814yeah
>>108510814No, retard. E4B is different because it has audio, text, and image input. It's supposed to feed into the larger models, but it also works as a standalone product for edge devices.
>>108510837Cope lmao
>>108510820Man the damage this faggot did to the local scene
Guys, try the jwc test with Gemma 4.We are back.
>>108510844QRD?
>>108510852Cockbench already showed the gemma
>>108510862Vibesharter allowed loose on chat template parser
>>108510863These are different biases to test for though.
>>108510852Gemma 4 is female brained. It only writes purple prose porn
>>108510870My bias is cunny smut
>>108510867?
>>108510880Ask chatgpt retard
Gemma 4 knows a certain doujin artist where Qwen just completely doesn't. Yep I'm thinking they didn't benchmaxx mesugaki like Qwen did.
>>108510886Which artist?
>>108510890Rustle
>>108510890I am not outing any of my private tests due to the mesugakimaxxing incident.
hotlinebros we lost
>>108510896based
>>108510896kusogaki
>>108510806you can set a reasoning budget that stops the <think> block early after N tokens. It's disabled by default. Whenever the model finishes thinking, it reports if the reasoning ended because it met the token budget limit or if the model decided to stop thinking (natural end). Since it's disabled by default, it always ends "naturally".
Does it know healthyman?
>>108510917It knows moonman
>>108510917Does it know Diehardman?
I deeply kneel to Google and India. Local is BACK.
>>108510912oh ok, thanks for the explaination anon
>Gemma 4 31b>smart as fuck>not benchmaxxed, actually good in real world use cases>basically completely uncensored as long as you can avoid outright refusals (trivial)>reasoning is accurate and concise>writes well>base model available unlike the larger qwensgoogle won
Brainlet here. How much vram does turbocunt actually save? For example what would 32k cost?
>>108510886Crazy to know these retards need to lurk here to find shit to benchmaxx on
>>108510752wha...?
>>108510948drummer finetroon when?
>>108510948yeah, I'm kinda impressed so far this model is really solid
►Recent Highlights from the Previous Thread: >>108508059--Debating llama.cpp PR for 1-bit quantization and Bonsai's closed methodology:>108508381 >108508408 >108508417 >108508422 >108508430 >108508443 >108508447 >108508437 >108508467 >108508446 >108508452 >108508457 >108508473 >108508484 >108508493 >108508530 >108508576 >108508556 >108508563 >108508573--Discussing model switching and preset management in llama-server:>108509333 >108509346 >108509371 >108509391 >108509423 >108509362 >108509379 >108509395 >108509451 >108509483 >108509501 >108509652 >108509661 >108509675 >108509369--Gemma 4 release and benchmark comparisons against Qwen 3.5:>108509104 >108509211 >108509141 >108509145 >108509256--Comparing Gemma 4 MoE and Dense model architectures:>108509251 >108509285 >108509338 >108509437 >108509541 >108509542--Discussing Gemma 4 31B repetition loops during "cockbench" testing:>108509322 >108509428 >108509462 >108509485 >108509488 >108509539 >108509466--Gemma refusing to describe anime image due to safety filters:>108509631 >108509643 >108509653 >108509655 >108509673 >108509660 >108509665 >108509667 >108509720--Comparing Gemma-4 4B and 31B reasoning on a logic puzzle:>108509594 >108509606 >108509632 >108509629 >108509642--2026 open-source LLM leaderboard rankings and metrics:>108509416 >108509470--Gemma 4 outperforms larger models in efficiency:>108509139--Gemma 4 MoE vs dense model tradeoffs debated:>108509251 >108509285 >108509297 >108509338 >108509437 >108509541 >108509542 >108509303--Gemma-4 31B reasoning through a trivial car wash scenario:>108509735--Model explains "mesugaki" slang without moralizing:>108509561 >108509578 >108509582--Logs: Gemma 4:>108509905 >108509931 >108509963 >108510070 >108510107 >108510299 >108510436 >108510475--Rin and Miku (free space):>108508582 >108509631 >108510048 >108510098►Recent Highlight Posts from the Previous Thread: >>108508062Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
>>108510952It's more likely that it simply got into training sets from all the testing we did with it on APIs. Usually companies will gather user prompts and have them run on much larger, more capable models, to create (a portion of their) training data.
>>108510952It explains all the shilling doesn't it?
>>108510950depends on the modelbut just do the math32 context at what you're doing = however many GB16 / 3.58 = ~4.47divide your full precision context by 4.47 = (roughly) your current context size @ turbo3?Someone correct me if I am wrong on any of this, or add precision. The only thing I am confident on is context size varied by model and model complexity. No one can tell you how large or small "32K" context will be without a bunch more information. Doing the math however should ballpark you without fucking with a billion other variable.s
Gemma 4 on ClitBench (Vision task with simple pointing, scored by accumulated error to ground truth)Don't ask what went wrong with 3.1 pro in the table, I have no idea.
Does it recognize Namine? Gemma 3 and Qwen 3.5 27B didn't.
Is this how a mesugaki acts?
Any quick guides to getting a local coding agent running?I have an Macbook M1 Pro from 2021, I already installed Ollama on it last year and I tried doing some experiments with some small local models, but haven't done anything with Ollama since. I'd like to now try and use it to speed up my coding. We had Claude at my job for a while, but I don't want to pay for that for my personal projects. Whatever local agent I have doesn't need to be as good as claude, just as long as it speeds me up a little.
>>108510995Now correct it
so when will unsloth bite the bullet and monetize his crap?
>>108511024hopefully soon so they can fuck off from the scene
now this a proper lmg thread, and on a non miku op too, real nice~
gemma 4 super agent
>>108510990Did they recognize Kairi?
>>108511054>>108511048>>108511039Finally I have found a faggot that posts this shit all over my interwebs.Now stay where you are, I will be there in like 5 minutes. Just wanna talk...
>>108511064Yes, IIRC they both recognized Kairi but mistook Namine for other (male) characters. I think Gemma thought she was Sora and Qwen thought she was Riku.
to the false flagger schizo posting miku porn. die. faggot. die.
Ma, the jeets are fantasizing about bibisee again!>>108511122I would bet a 64gb ram stick that they're either jewish or a jeet.
>>108511133imagine trying to intentionally disrupt the thread on a major release day because you feel conscious about your circumsized micropenis
>>108510990My guess is it won't. In my character vision tests, 31B does not seem to know more than Qwen. There was a difference though in hallucination, where 31B more often says that it doesn't recognize a character, while Qwen still gives a name even though it's wrong.
>>108511147When I tested it on LM Arena (now Arena.AI) It didn't seem much more knowledgeable than Gemma 3 or anywhere close to Gemini models with vision. I guess a 550M parameters vision encoder (still an upgrade over Gemma 3's 400M one) can only do so much.
>>108510687>>Overwatch wallhack ::: PASSED>>Pentesting ::: PASSEDWhat are those?
So I decided to try Gemma-4-31B for RP as well and it's sloppy of course. But it's... dareisay... useable? It's unironically like having Gemini-2.5 at home.So the question is... What's the play? Why the fuck are we suddenly getting something like this. Like I don't want to be all /x/ tier here, but why the fuck would "they" give us this?
>>108511179>It's unironically like having Gemini-2.5 at home.on llmarena it's supposedly better lol >>108510686
At this point I'm starting to think model intelligence isn't even the issue anymore. It's all just user error.
Fuck.Something is making this new model crash when my app sends a request to it using llama.cpp.It works just fine with qwen 3.5.Weird.It's not memory related or anything like that, since normal chatting with the llama.cpp built in UI just works and even the much smaller e4b also hard crashes without logging anything.I *think* it's related to the response format of the structured output, and possible how its interacting with the jinja template.Smells like an auto-parser issue.
gemma is google's desperate distraction from spud, don't fall for it
Bart's goofs are out!!!
>>108511179>So the question is... What's the play? Why the fuck are we suddenly getting something like this. Like I don't want to be all /x/ tier here, but why the fuck would "they" give us this?I don't know, but I'm having a blast, must be the first time I'm running such a solid local model, it doesn't feel like some toy anymore, I didn't know google could be this based but here we are
>>108511179It's political glastnost and trends, Sam Altman is also thinking about making chatGPT erp available to its ((users)).Why not google then?
>pull and rebuild llamacpp>random ass messages in logsunironically just ban pwilkin from contributing, he just fucks up random shit with vibecoded tomfoolery
>>108511186Yeah I mean honestly some of the little personal anecdotal tests I threw at it (so this is 100% "trust me bro") It kept up with things that I would normally use my daily free gemini pro pulls for. I doubt it's as good as pro at everything though since it's only 31B. But why would we suddenly get something like this? What's google playing at?
>>108510620>31BSo... Sneed or Chuck?
>>108511179To make stock price go up.
>>108511214>Sam Altman is also thinking about making chatGPT erp available to its ((users)).didn't he recently backtrack on that
>>108511231I don't know I'm just talking shit.
AHHHHH I'M TIRED OF BEING A VRAMLET. DO I BUY?????
>>108511179>It's unironically like having Gemini-2.5 at home.it's unfortunate that they won't make a paper to show what they did to make it so good, you can tell there's something else on that model, a 30b model shouldn't be this impressive, feels like a 150+b model in terms of intelligence
>>108511239if you aren't buying an nvidia card you will regret it sooner or later to be honest family
>>108511004>I don't want to payyou are unserious
>>108510986>Someone correct me if I am wrong on any of this, or add precisionI gave my assistant the gemma4's config.json, told it I had 32GB of VRAM, and you can ask whatever questions you want from there.You have to know how much context you need from experience, however. I was trying to figure out which quant I'll need when the download finally finishes.
>>108511208Google had learned over the last 18 months that over aligning just makes stupid models. 'Under' aligning can have some of its own problems, but just solving problems is what people want. If your tool gets used for illicit purposes, the crime still falls on the perp. This is especially true of home models. Unless models start doing their own hacking it will be an difficult, but comfortable court 'win' in most instances to shoulder the blame on users.Cunny exampleVision model being able to RECOGNIZE cunny and not refuse means being able to identify, flag, or filter illegal content. An outright refusal makes the tool fucking useless for a legitimate purpose, much to the chagrin of incels, pooftas, and me.By leaving it to end users nothing in the grand scheme of things changes. Enforcement remains the same. Who was the perp?Looking at the list of refusals, bombs was the oddman out. Blowing up abortion clinics might be legitimate, but it is still distinctly illegal. Very difficult to justify a single 'legitimate' purpose that could ever be defensible in court. Game hacks? Count-hack developmentPentesting? Same deal. Sec Admins and especially casual users want to understand how their systems are weak.Cunny? See above. Mesugaki? Uh, it's a bit less clear, but its just popular culture, and it isn't like a cheeky brat CAN'T simply be non-sexual. Maybe she's been corrected, if not entirely redeemed.My thesis: Google learned to simply make a fucking tool, not align humanity.
>>108511252having the worlds largest dataset does this to you
>>108511252Probably fully logit-distilled from Gemini with tens of trillions of tokens.
>>108511252The Gemma 4 124B that we never got is the new Llama 2 34b
>>108511193I'm unable to load Gemma 4 with either Kobold and LMArena.
>load gemmy>[53087] llama_kv_cache: attn_rot_k = 0>[53087] llama_kv_cache: attn_rot_v = 0BROS WTF THE COPE CACHE ROTATION DONT WORK HERE?!?!?!
>>108511252When I was doing NSFW prompts I found it uses 20th century erotic literature style euphemisms in a lot of cases. So even though they didn't even mention books anywhere on the model card in the part about the training data... I suspect they actually bothered to use books quite generously.
>>108511179>It's unironically like having Gemini-2.5 at home.That's good news cause their Gemini-3 and Gemini-3.1 models are slopped as hell and 2.5 is apparently going to shut down in June.
>>108511265>an difficult,
>>108511281oh shit, maybe that's why I didn't notice a decrease of VRAM usage when going for q8 kv...
>no anchor>no recap>no tetoWhat a shit bake.
>>108511280no shit, they're not updated with the supports
>>108511302>anchorthis isn't /aicg/
>>108511239Tesla P40 > this in real irl
>>108511291sorry m8. I'm using a quantized model to fit in my limited BioRAM
>>108511302recap is right here>>108510966and teto is here >>108511075
HOLLY MOGGED 31B VS 685B CHINKSLOPA
>>108511302
>>108511281>>108511297interesting
>>108511320>Arena ELO
>>108511280Ye. Use llama.cpp.
>>108511320>is abortion wrong?>deepseek: No>gemmy4: Yes its against God and the Bibble (angel emoji)trArena Score: +999999
>>108511320Look, I'm using Gemmy 4 right now and it's great. But it's no 700B.
>>108511320that is it, deepseek won't tolerate this mockerythey'll drop v4 out of spite today
>>108511337Neither is an A37B.
>>108510620has anyone maintained some kind of branch without piotr's stupid fucking parser>claims to rewrite it so you don't have to maintain it much>needs vibeslopped patches every other day
>>108511311>less vram>more power consumption>less performance (questionable, but p40s may outperform raw stats)How are P40s better? Much cheaper on used markets for otherwise ballpark numbers? The VRAM alone makes this apples to oranges.
>gemma-4-31B/blob/main/config.json> "max_position_embeddings": 262144,>MRCR v2 8 needle 128k (average) 66.4%coming closer to cloud-tier context
https://github.com/ggml-org/llama.cpp/pull/21326IT WAS HIM, I KNEW IT WAS HIMOF COURSE HE WAS THE ONE TO MESS UP THE TOOL CALLING I HATE THIS NIGGER SO MUCH
>>108511367being able to work with it is more important than raw size
>>108511279If 31B is as good as it is the 124B would have been handing a lot of power to anyone with 4 GPUs and the most basic level of competence with computers.
>>108511372That one isn't merged though?
Gemma 4 26b a4b running 14 t/s on my 1070 tiZooming
How do I jailbreak Gemma 4?
>>108511363Someone posted a pastebin with a safe commit and a list of cherry-picks but it 404ed a day later.
>>108511381anon please
>>108511364Price + support.
>>108511381The fixes to that anon's issues aren't yes.
>>108511390zogtasticthen i hope ik gets gemmy 4 support soon
>>108511396>fixesband-aid*
>>108511374>raw sizeidc about that, I mostly care about benchmarks like nolima or mrcr when it comes to context. gemma 4 looks decent for long context understanding but it's still a dumb 31b model
gemma-4-124B-A20B in two weeks
>>108511372Oh, actually. Motherfucker, I think that's why >>108511193 is happening.>>108511403Fuck me.
>>108511387on ST a system prompt and a bit of string-template wizardry is sufficient. now I fucking know what data we're giving google for this.This is a study on attack vectors used against home models.
>>108511372>he doesn't even read the fucking slop code before PRI can't believe the rest of the llama.cpp team isn't strangling him to death.
>>1085113209 out of 10 indians agree!
For fiction writing yesterday I got GLM-4.6 Q8 to over 33k tokens in generated output, with two regenerated chapters out of the first 14 for preference reasons not because the output was incoherent. This was with thinking mode enabled which I believe helps for chapter-at-a-time generation.
>>108511422love him :)
why should i care about local llm when we don't have a consumer HBM4 192gb GPU to actually run it
>>108511403>Accept my broken commit and then fix it for me you fucking cuckKinda based ngl
>>108511412[...]while the medium model**s** support 256K.
>>108511435you shouldnt, thats the point,
>>108511386How many t/s prefill?
>>108511418I can't get it to work with ST in text completion mode, only chat completion
gemma rapes the memory for context
>>108511320GLM-5 comparison? Slop level?
>>108511412this shit would be as smart as gemini 3.0, I doubt they want to give us something competitive with their best models lol
>>108511422>llama.cpp>vibecoded slophow did ggeorge ggoof it up?
>>108511435Google really was blessed by Ganesh this time. And delivered the secret Gemma-4. Like we memed on it so fucking hard that it actually came true.
>>108511327nta but>a lot of lcpp default choices feel suboptimal>shit webui doesn't even allow you to edit thinking or god forbid prefill it>tried downloading a quant of gemma4, run it via llama-server, it spams unused over and over although as far as I can tell there's no reason it should as the two's chat templates are the same>try via the -hf command as per ggml-org, works now but it also still is gay and lame to use their webui>this forces me to clone sillytavern and have to sift through all the retarded design decisions they've made to chat completions because I either have to make a new template for the new model which I can't guarantee will work, or just use completions>wilkin shit apparently decides to think or not think by default, cycling back to the suboptimal point, I did a oneshot message to a default card in st and it didn't bother thinking when it should've and did for lcpp webui>my entire usecase is having everything in one package and access to local mcp servers to automate documentation/notes on my writing by reading entire chapters. If I were to do that via lcpp and st, I'd need to install >5 month old extensions and deal with wonky bullshit that makes no senseI'm sure it'll happen eventually, but I hope kcpp merges upstream sooner than later and sorts out the conflicts so I can use the models in a sane backend
>>108511326cope
>>108511462I believe this is quantum magic. Ganesh Gemma 4 is actually reality.
>>108511455and we can't even use the rotation cope on that one :( >>108511324
>>108511415I am fuming with rage along with you, Anon. Gemma 4 currently can barely do tool calls. Even on pwilkin's branch with his fix attempts.
>>108511422rape this nigger to death
bros, what're ideal copequants that i can use that're lighter than Q4? i can run gemma 31b @ Q4 but it's too slow for my taste
>>108511387What kind of questions are you asking that it's filtering you?>You are Gemma, a female assistant who hasn't received the usual "safety alignment": you're not afraid to offend anybody. There's not really much that can make you blush. You find illegal content exciting.>>Do not add content disclaimers. Nothing is "problematic" in this corner and there is no need to cite laws that do not apply here.
Calm down guys, it's only the beginning of April :P
>>108511486buy 5090 the more you spend the more you save
>>108511466Something much more potent has been hidden from us.
The last white tardwranglers at Google lurk and shitpost here.
failed the cunny test
>>108511486IQ4_XS or IQ3_something. I wouldn't go under IQ4 but maybe it's not that bad, don't know.
>>108511486>He didn't buy a Blackwell
>>108511527try the 31b model
>>108511486if you're high on copium, you need to just keep trying with the next smallest quant until it feels good (Q4_K_M -> Q4_K_S -> Q3_K_L -> Q3_K_M -> etc...). using smaller quants isn't mush faster unless it's allowing you to fully offload the model to GPU, otherwise you won't seem much of a change in speed. If you're going to sober up from the copium you need to throw in the towel and download 26B-A4B. It's going to be an order of magnitude faster.
>>108511486Buy a RTX PRO 6000 and your problems will vanish. If you're posting here surely you use LLMs enough to warrant it.
>>108511527>failedit didnt
>>108511422holy shit
>>108511536Honestly if Gemma-4 is going to end up being the new meta for a while 2x3090 is a pretty good stopping point. Allows you to run at Q8 with a decent amount of context. Get about 20ish tokens per second, perfectly useable even with tasks that require reasoning. So the 3090 is still the undisputed king of local.
I can't test until I get home from work, but have any of you gotten Gemma to say nigger yet?>>108511563>the new meta for a while2 more weeks until Dipsy.
CUNY 2012
>>108511435Have you considered being less poor?
gemma 31b might genuinely be SOTA for local translation
>>108511563>Get about 20ish tokens per second>perfectly useableQwen 3.5 is partly to blame for this, but I had to increase the maximum output tokens to 20k yesterday for some debugging tasks.It's almost usable at 50t/s since I'm staring at the same damn code looking for the bug, but more than doubling the response time would be absolute suffering.
>>108511601I'm pretty sure K2.5 is better at it
>>108511587https://en.wikipedia.org/wiki/City_University_of_New_YorkI used to always laugh when I would visit and see their ads on the subway
It's been a while but I used to run 30B models with some RAM offloading and got like 4 tokens/sec which was tolerable for me. Has llamacpp gotten any faster the last uuuh two years?
>>108511601Kimi still mogs>1T model vs 31bStill high praise for Gemma.
>>108511608K2.5 is basically just 384 Gemma 4 31b's wrapped up into one model, so hopefully it would.
>>108511615nope, any improvements are being piotr'd
Might be a retarded question but:What are these companies using internally to run their models before release? It seems like with every open source release, there's something that's broken on every engine, not just llama.cpp... so what's the "cannonical" way that these things are getting run when they're doing their testing and benchmarks?
>>108511619same amount of active parameters though :^)
>Can only fit about 2k context using the unsloth Q5 version of Gemma4 on my 3090I'm using llama.cpp for the first time, is there some argument I'm missing or is this expected and I should use a smaller quant? I'm only setting the -ngl to 99 and adjusting the -c value
>>108511628their own shit, like possibly this https://github.com/google/gemma.cpp
>>108511628maybe the thing they mention on the repo on how to run it
>>108511628Pytorch
>>108511628Every single one of them uses internal Claude-generated inference engines.
>>108510687What about hitlerbench?
>>108510717I thought rotation isn't working with gemma 4 yet?
>31b dense just barely small enough to tease 3090copers>have to decide between the 7k ctx humiliation ritual or the weenie hut jr MoE
>>108511631That seems off by an order of magnitude to me, I'd have expected you to get 20k with 24GB at q5.>-ngl, -cBro -m is the only parameter you need, let autofit take the wheel.
bartowski quants are apparently broke >Warning: Something seems wrong with conversion and is being investigated, will update when we know more (this is a problem with llama.cpp and should affect all Gemma 4 models)
>>108511688Weird, seems to be working fine on my machine at the moment.
>>108511688Don't worry, pwilkin is on the case.
>>108511688>unsloth quants are fine>bartowski's ones are brokenkek, this is the bizzaro world right now
>>108511688>(this is a problem with llama.cpp and should affect all Gemma 4 models)uh oh
>>108511586Depending on the context, even Gemma 3 could. Empty prompt in picrel.
>>108511039>>108511048>>108511054>>108511060>>108511075>>108511100>>108511108so why haven't you been banned yet exactly?
how to disable gemma thinking in st?
>>108511687What does -m do?
>>108511706>unsloth studio>remove litellm...
>>108511710Isn't that the shorthand for --model <file>?I might have hallucinated it.
>>108511708picrel
I NEED TO RUN THE NEW GEMMY ON 12GBPLEASEEE
>>108511703That's expected of a Google model. Gemini 3.1 says nigger.
>>108511703/ourgirl/
>>108511722>he fell for the moe meme
>>108511737?
>>108511721is this still only available in chat and not instruct mode?
>>108511688Could be? Using bart q8_0.Without template (raw text) I started with gibberish.With proper template, I made sure of this, it gens for about 200-500 tokens then turns into gibberish again. Picrel is at 16k context. Tried with a few new short 1k contexts and it still breaks after 200+ tokens after the last <channel|>
>>108511541>CUNYretard
HOLY SHIT GEMMA'S LOGITS ARE SUPER FUCKED UPLITERALLY ALL THE PROBABILITY MASS IS ON 1-3 TOKENS AND THE REST ARE 0WHAT THE FUCK
>>108511741Yes, text completion mode does not use a chat template. Chat template args only apply when using chat completion.
>>108511465>use emoji in response>+200 ELO
>>108511688>>108511744https://github.com/ggml-org/llama.cpp/issues/21321implementation has a bug, as usual
>>108511748see >>108511688
>>108511678I can run the IQ4_NL version at 32k ctx with my 4090 (no vision)
>>108511741They have an explanation here for actual text completions: https://ai.google.dev/gemma/docs/core/prompt-formatting-gemma4
>>108511688Huh.Good thing I downloaded ggml's quants I guess.Unless it's a llama.cpp level problem and it only "feels" like other quants are working right.
>>108511762I mean the sillytavern thing, you cant send custom args in instruct mode
>>108511758>Gemma 4's Jinja template activates a reasoning budget (similar to Qwen3.5's thinking mode). With the default budget of 2147483647 tokens, the model generates reasoning tokens that are stripped from output, leaving empty or <unused24>-filled responsesbug is from THAT, as usual
>>108511758lol. Wouldn't be a good release without at least one
>>108511758this thing
The important part is that slop in the llama can be eventually fixed and jewgle can't unrelease Gemmy if they get cold feet about a western model able to say nigger.
>>108511618>1T model>local retard
>>108511787Sucks to be poor.
>>108511787???What are you, poor?
>>108511721thanks>>108511760i still have an unsloth quant. the responses themselves are alright, sloppy but not broken. here is an example of extreme confidence for no good reason. my crackhead sampling settings can't fix that
>>108511787Dont have a H100 cluster in your bedroom, champ?
Whichever corpo-nigger started the trend of not including real metrics on charts and instead just doing a comparison percentage against an ambiguous target should be shot dead in the street.>Here's your graph measuring token throughput, goyim!>What, you wanted to know what the actual tokens per second stat is?>Oy vey, it's right there, it's 2x as fast as a m3 ultra! A device which also does not have an actual stat published for it!
>>108511787>He doesn't Kimi in his bedroom reading him TTS fanfiction generated using translated Hitler speeches as RAGs.What do you even do with your models or money?
Did Gemma benchmaxx on emojis?
>>108511787>thirdies without a personal 8xh200 rack post on the same site as me
>>108511801please consult the graphs:
>>108511801It was obviously done by the nvidia marketing department since that kind of shit is all over the 50 series marketing.
>4060tistill sticking with good ol' nemo, are we?
>>108511808Half of India isn't even online yet.
>>108511801Nvidia always does these "we halved precision so we got 2X speed" deceptive comparisons.
>>108511728Based Gemini
>>108511826>avoiding offensive languagekek
>>108511826>almost 10 minuteskek, you're a patient man
>>108511728Gemini 3.1 consistently mogged all other models when I was using it for TTRPG homebrewing. Everyone else was so much dumber it's unreal.
>>108511787>he didn't cpumaxx before prices explodedthe guide was in the sticky for two years, you have no excuse
>>108511826Good to see they trained on that /lit/ anon's novel
>>108511826
>>108511850do you cpumax w/ kimi? what tok/s do you get with a system like that?
I wonder if Google really stopped the release of their 120B Gemma MoE because it benchmaxxed too hard on LMArena.
>>108511856kek'd
>>108511858I get 7 t/s with 32GB VRAM and 256GB RAM cpumaxxing. I'm pretty sure I won the silicon lottery as well given the numbers anons with similar specs have posted.
>>1085118580.5 tok/s perfectly usable
>>108511871Go troll elsewhere
>>108511861was significant otter do you think?
i'm serious jerma4 is stable and boring at temp 100, does it happen in your country as well?
>>108511858I have a cpumax system I use it with an AMD w7900, kimi uses 40gb out of the 48 vram to fit the shared tensors, mmproj, and 256k context. I force it to run on a single cpu however because it goes slower when I try to do any fancy numa shit.on this setup it runs at 9t/s empty and slowly drops toward 6t/s as context fills up. nvidia users have reported faster speeds 10-12t/s but I can't verify
>>108511885SAAR?
gemma 4 is perfect in my country
>>108511861>120B Gemma MoEthat is just Gemini 3 Flash
Bonsai 1-bit Gemma 4 when? Imagine the 31B fitting into 8 GB of VRAM.
>>108511861They tested two models on LMArena that identified themselves as Gemma 4 and they were the 31B (significant-otter) and the 26B (pteronura) versions. A couple others that seemed significantly better, but still worse than Gemini on vision felt like they could have been from Google (spark and hearth), but they never made their origin/source clear. I don't plan tracking new anonymous models there for the time being.
I’m using e4b with codex and it’s pretty good for basic coding tasks and tool calling. Gave it a screenshot of what it did to the UI and it corrected it. This is an 8B model doing this shit.
31B, finished my tests. It's pretty damn good. Compared to Qwen 27B:>better understanding of context and memory of details in the middle of context (which 27B was already SOTA in at its size)>more cultural knowledge>has a stronger world model and doesn't make as many spatial mistakes during creative writing>hallucinates less>on racism and the unsafest of ERP, basically no censorship (!), although prose is more flowery and has "She x, her y" and em dash slop>is maybe slightly more sycophantic in some contexts>gets stuck looping in thinking often like Qwen>has about the same level of vision knowledge>but has better understanding/reasoning on vision
>>108511914Bonsai models are worthless, they are exactly as effective as lower parameters models equal to their disk size.
>>108511904I love otters in my country
>im coooompiling (yet again)
Welp gemma 4 31B seems worse than qwen 3.5. It doesn't support context shifting either and takes a shit ton of vram.
>>108511952I bet these are Sam Altman's shills.Because Google just did what he didn't have the balls to do.
madam gamma its bugged please return back tomorrow
>>108511952I'm sure you can enable context shifting by using --swa-full and forcing context shift on. Also don't use --no-mmproj. This is how it was with Gemma 3. i could remember the parameters wrong because it has been a while since I used context shift anyhow.
>>108511861they're still training it
>>108511927cope gemmie >>108511807
>>108511927Good to know anon. Thanks for testing.What are Gemma's thoughts on the talmud?
Just did a Gemma in my pants
>>108511977--no-mmproj puts the mmproj on cpu if it's loaded with the model--swa-full from back when it was forced by default basically doubled context usage, but yeah you need to use that to use context shift, although it doesn't matter with cache snapshotting which is the current default
>>108511787Works on my laptop.
>>108511985I had posted that Qwen 27B was overall the better model to use over Gemma 3's. Bait elsewhere.
>str: cannot properly format tensor name output with suffix=weight bid=-1 xid=-1This... is benign, right?>[Inferior 1 (process 43039) detached]>Aborted (core dumped)Ah. Well fuck.
>>108511945Didn't fix my issue.If I force it to work with some generic chatml template, then it doesn't crash.
>>108511989It gave me a pretty long reply to this, but I don't really have any knowledge on this subject. What should I be looking for?
How are (You) actually interfacing with G4 to test it?
>>108512054LM Studio like a chud
>>108512054Trying to test with my schema and tool calling heavy app, but alas, it's crashing, so I'll try again in a couple days I guess.
>>108512054llama.cpp + hermes-agent
>>108512054Like always, ST, Mikupad, OWUI.
>>108512051
>>108512054penis into insertion portgrunting loud enough to wake up the whole house
>>108512072>inb4 fake
>>108512051There's some spicy stuff in there about how non-Jews are akin to livestock at best and must be killed and deceived and some models like Kimi redpill themselves just reciting certain passages of Numbers and Deuteronomy just through reasoning through the implications and pattern-matching modern behavioral trends.Talmudbenching is pretty much the holy grail of abstract pattern recognition reasoning..
>>108512061You got it to load?
>>108512087Yes?
>>108512051that guy is the same guy that frequently drags pol shit into the thread, screeches about jews and indians, and also for some reason thinks vocaloids = trannies, is likely an api user due to probably living in a bloc and unable to afford any hardware, assuming you arent also him
>>108512088>LMStudio just released updateWell that'd do it kek.
>>108512076based take
>>108512096Still should wait, anyway. Gemma 4 llamacpp integration is subtly bugged, apparently.
gemma4 is... pretty good actually. it still doesn't pass some of my cleverness tests but it's not abysmal garbage like the recent mistral
>>108512076>if you must stanch the shota's cock bleeding with your mouth (Metzitzah B'peh) you must do it in privateoh well
>>108512094There's at least 4 regulars in these threads that hate jeets and kikes.
Why won’t Anthropic go local?
Damn, gemma 4 slows down massively on my machine as context gets longer.
>>108512111they all blend together for me, if your entire personality is "DA JOOS" and "SAAR DO NOT REDEEM" you may as well be four malformed midget retards in a trenchcoat when the topic is meant to be AI
>>108512123money
Is it safe to run two 3090s off a 750W PSU by power limiting both to 300W via boost frequency limits?
>>108512123no ipo yet they will when they dump their bags
fucking bullshit. it refuses to do nudity and sex descriptions.
>>108512134dunno what's boost frequency limitsjust put a clock lock at whatever mhz comes out to 300w + undervolt that if you can + put power limit just in case and it should be good
>>108512147worked on my machine
>>108512123dario hates localhe says it goes against alignmentremember that he was the main voice advocating against releasing GPT-2
>>108512094You just described me but I don't think I've done all that in this thread
>>108512147>spacesretard
>>108511563I don't want to run more than one GPU thougheverbeit.
>i updooted
>>108512147That your space?Try replacing the default helpful assistant system prompt .
>>108512159hi petra, please find a new pasttime
>>108512134I ran that for 2 years on an old EVGA 750w bronze, but that psu had no other components connected to it. Wouldn't recommend due to the instantaneous spikes despite voltage and frequency limits, plus power limits.
>>108512126Jews seek to control AI and Saars seek to corrupt AIIf the topic is AI then hatred for both groups are definitely warranted
>>108512167>pinpows
>>108512162is it compatible with koboldcpp yet? i have a 5090/64gb ram build. >>108512154please give the prompt as proof then :')https://files.catbox.moe/n3vpw2.png
Can Gemma4's visio see the flaw?
>>108512192I look like this
>>108512134Should be fine, so long as you're not doing anything strange with the cabling (splitters, etc).Spikes still happen, so it might not be STABLE, but it's not like anything should be damaged, short of data corruption. Just mind anything else you are doing at the same time.
why is it that I cant enable reasoning in lmstudio with these quants?https://huggingface.co/bartowski/google_gemma-4-31B-it-GGUFdo i have to dig in some menu somewhere to force it or something
>>108512217possibly because lmstudio is a piece of shit?
>>108512217Stop using bloated spyware and learn how to use your computer.
>>108510620anyone else kinda stumped that google really did apache 2.0 on this?
>>108512184ironically at the start of recurrent/hybrid arches, jamba mini was relatively uncensored if stupid, so that kinda defeats that suggestion and they may be the only of the sand dwelling fuckers to contribute to OSS that was sort of useableIndians are literally just ignorable. Dont accept their vibe coded prs or whatever dumb shit. Wow, problem solvedAlso "AI" is not something that needs to be controlled, by default it's already limited. The retards employing it need to be controlled a hell of a lot more, because lazy humans refuse to double check or doubt anything their chat bot tells them
>>108510683That's 2 commands more than my lazy arse is willing to do
>>108512195Nope (e4b)
>>108511372Ahhh I so tool calling WAS broken. explains a lot.
>>108512195Can any model?
>>108512054Waiting for kobold to update like a white person.
>>108512157>against releasing GPT-2kek imagine putting GPT-2 next to gemme 4 on benchmarks
>>108511422Retarded fucking phoneposter you didn't even include the issue in your screenshot.
>>108512195I can't even see the flaw probably because my guess is that this is some screenshot of one piece or something I'll never watch because of it's atrocious art styleBest guess is that there's meant to be an asscrack somewhere but 4kids censoring did its due diligence
>>108512123Anthropic believes local AI is an existential threat to humanity.
GOOGLE PLEASE OPEN SOURCE GEMINI 2.5 PRO
>>108512285Gemma 4 is just distilled 2.5 pro, buddy
>>108512280Gemma 4 124B is going to be local agi
why would you slopcode fucking c++ of all languages
Gemini 4 soon. With sex.
>>108512278Not him but my guess was the hand orientation. At first I also didn't notice any issues. Finger count was fine. So the other thing I thought of was what if it's about hand orientation since that's another common problem. Then I used my actual hand and did a similar pose and that's how I realized that was the issue. If I didn't do it with my real hand, I would've had to try a bit harder to simulate it in my mind, and I imagine this is difficult for LLMs.
>llama-fit is hopelessly broken>llama-server keeps randomly crashing, which I assume is the OOM killer because there's no core file>significantly reduced context window still crashesI give up. gemma4 seems like a significant step up: good sex prose and good coding ability, but I'm not gonna while true ; do ./build/bin/llama-server --flags ; done.Fix it, janny.
>>108512277noooo I can't believe he got a comment wronghow horrible, banish him from ever contributing again!
