File: mikuthreadrecap.jpg (1.15 MB, 1804x2160)

1.15 MB JPG

►Recent Highlights from the Previous Thread: >>108535684

--Comparing quant options and VRAM optimization for Gemma 31B:
>108536037 >108536050 >108536053 >108536073 >108536120 >108536361 >108536390 >108536424 >108536059 >108536079 >108536133
--Comparing Gemma 4 and Qwen 3.5 MoE performance and capabilities:
>108537392 >108537519 >108537535 >108537597 >108537405 >108537572 >108537578
--Using logit softcap overrides to reduce Gemma 4's determinism:
>108535726 >108535737 >108535771 >108535791 >108535817 >108535843 >108536008 >108536066 >108537466 >108538178 >108538338 >108537546 >108537593 >108537612
--Gemma 4 draft model benchmarks and discussion on quantization levels:
>108536606 >108536822 >108538242
--Discussing llama.cpp memory management issues and the ik_llama fork:
>108535819 >108535863 >108535870 >108535885 >108535898 >108535907 >108535931 >108535950 >108536005 >108536814 >108537853
--Discussing ways to remove generic AI patterns via post-processing:
>108536315 >108536337 >108536362 >108536372 >108536443 >108536484 >108536498 >108536595 >108536686 >108536697 >108536456 >108538094
--Discussing Gemma 2 26b's robust filters and jailbreak attempts:
>108538390 >108538400 >108538410 >108538423 >108538433 >108538458 >108538463
--Gemma 4 31B performance on FoodTruck Bench simulation:
>108535818 >108535835 >108535876 >108537945
--Discussing performance improvements and capabilities of Gemma 4:
>108536335 >108536385 >108536393 >108536561 >108536622 >108536647 >108536666 >108536720 >108536763 >108536915 >108537048 >108537103 >108537436
--Evidence of Gemma base being trained on roleplay logs:
>108537545 >108537556
--llama.cpp adds native support for Hunyuan OCR:
>108538402
--Discussing bypassing Gemma's safety filters for prohibited content:
>108537984 >108538021 >108538045 >108538137
--Miku (free space):
>108535751 >108536605 >108538202

►Recent Highlight Posts from the Previous Thread: >>108535686

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

ahem, migemma

kv rotation on gemma status?

>>108538945
>they dont everyone uses ollama
but ollama uses llamacpp as the backend right?

>>108538968
oh anon you dummy

>>108538761
>better CPU and hybrid GPU/CPU performance,
Well that seems to be a lie. It runs slower than llama.cpp for me.
What's the use case for this?

>>108538964
Gemma 4 doesn't benefit from KV Rotation

Where is the Gemma-chan design?
We need porn of her

>>108539001
Drama.
https://github.com/ikawrakow/ik_llama.cpp/discussions/1247

>benchmarks and reddit say qwen is better
>lmg thinks gemma is better
>the first two are worthless
>the second only cares about pedo erp
The answer is to use both based on usecase.

>>108539011
is it because it's mathematically impossible or is it because the vibeshitters at llama.cpp don't know how to make it happen?

What sort of hardware would be required to run Gemma 4 at full precision, as in how they would run it at Google for example?

a-asking for a friend

>>108539019
Pedo ERP is a mission critical usecase whereas vibecoding isn't

File: 1757029618631460.png (248 KB, 481x507)

248 KB PNG

Do you have PATIENCE to get the job, /lmg/?

File: IT REALLY IS THAT SIMPLE.png (67 KB, 1601x334)

67 KB PNG

>>108538947
get vibeshitted

>>108539032
linkedin shitnematic universe, lel

>>108539024
Which one? You don't typically run these models at full precision (fp32) anyway.

>>108539032
the correct hire is the first one to leave

>>108539032
This is fake and gay but people will actually put you through humiliation rituals like that just to see who they can exploit the best.

>>108539032
I believe that story, they don't want talented people, they only want docile people that can play the role of a yes man all day long

>>108539024
its a 52gb model so you need at least like 64gb vram theres plenty of solutions to get that but googlel is probably using h100

File: ce.png (113 KB, 1046x522)

113 KB PNG

which one?

>https://github.com/ggml-org/llama.cpp/pull/21500
>download the goofs again just to change a flag
when will they add a tool to patch your goof metadata this is beyond retarded

>>108539047
31b https://huggingface.co/amarck/gemma-4-31b-it-abliterated-GGUF

>>108539052
If I could run this one I wouldn't cope with a moe

>31b
>my vram is 12G
it's so over for turbovramlets..

>>108539048
https://huggingface.co/ggml-org/gemma-4-E2B-it-GGUF/discussions/1#69d11dd3f5041eaf89541197
>i guess llama.cpp users won't really notice because jinja is default now, and is injected in the jinja template.
this, it's a nothingburger

>>108539057
you might be able to run okay a q4 is like 15gb youd only need some cpu offloading

>>108539048
If you're not retarded your goof 00001 only contains the metadata.

>>108539061
i know, but the speed becomes unusable at that point unlike memeoe is lot more tolerable

>>108539061
>offloading a dense
lel

>>108539067
dumbass

>>108539067
i can get 12t/s with offoad

How to set main GPU on kobold?
I have two gpus and it keeps setting the lesser one as main....

>>108539072
which is borderline cope speed for tool use
fine for rps i guess tho

>>108539076
Holy shit nigger, it has a GUI. Use your eyes.

>>108539076
that's why I'm using llamacpp server, maybe it's a basic cli but it's simple enough, you output 3 lines and that's it, no need to look on endless buttons

File: 984703650924353.png (97 KB, 415x707)

97 KB PNG

>>108539086
Setting it to use all GPUs still sets a main GPU, fucking retard.
I have a GPU imbalance, on is 16gb the other is 24gb, can your dumbass guess which one I need to be the main and which one doesnt?

>>108539089
>to look on
What's up with all these ESLs lately?

>>108539017
Skimming through, I am not sure what I am looking at here.
Anyway I am confused since it advertises itself as being faster with CPU+GPU mixed offload inference, and I see results in the internet ostensibly confirming that, but when I run Qwen 3.5 35B on both, it runs 50% faster on the normal llama.cpp

File: imbecile spotted.png (275 KB, 640x428)

275 KB PNG

>>108539106

File: 1775257603862045.jpg (129 KB, 990x936)

129 KB JPG

>>108539130

>>108539133
>frog posting
opinion discarded

>>108539025
this

>>108539130
Well don't look at my finger then.

>>108539133
But is it really better to have the same skin color as sperm?

>>108539157
sperm is life
poop is waste

>>108539157
Only one of those is fine to eat.

>>108539162
>he says he swallows sperm
faggot lol

>>108539163
>he doesn't recycle his own loads for bigger gains
ngmi

>>108539025
Vibecoding with a model that fits on a single gaming GPU is a lost cause anyway; they just can't have enough verbatim knowledge of all possible libraries and usage patterns that people might use use/need, without degrading other areas. Let local LLMs do natural language tasks.

>>108539168
the other problem is that your gpu won't fit enough context even if the llm was, somehow, smart and knowledgeable enough (it's not the case, but let's posit a theoretical model of the future was)
a large part of what made gemini useful for me in some coding tasks is that I can give it a huge amount of related source code (caller, library code called etc) in the same prompt. Real code is rarely isolated enough to properly work with LLMs as mere tiny excerpts, LLMs work best when fed a lot of context.

Guys I worry about deepseek because they don't have large amounts of traces of people using deepseek in coding harnesses like GLM and Qwen do.

>>108538947
why the /aicg/ here in /g/ is so retarded?

>>108539199
API paypigging will do that

>>108539199
you have to be non-retarded to be able to run local models, so it filters out the subhumans on /lmg/, /aicg/ doesn't have such filters

> Google paper comes out claiming x8 vram reductions or something
Did that get implemented and is usable yet, or was it a whole bunch of nothing?

>>108539168
I agree that cloud models are the only option for real coding work. But vibe-coders are subhuman and will deliver slop regardless of model size, so they might as well use a small model.

>>108539057
26B IQ4_N_L is just 14gb anon... offload the extra 2gb on your ram

File: 1769020848215663.png (109 KB, 1257x965)

109 KB PNG

>>108539191
People do use DS V3.2 for coding, at least on OR (top models are free; DS V3.2 isn't)

What's the best model can I realistically run on 16GB VRAM + 64GB system RAM if I don't care about token speed at all and just want *a* response at some point in time?

>>108539207
26b out of nowhere

>>108539213
Gemma 4 31B
Or the Gemma 4 26B moe for actually good speeds and only a little worse quality

>>108539207
of course i can
i can even run 262k context with reasonable speed with 26b thanks to moe (20t/s)
31b?
even at IQ2_XSS it shits itself

>>108539047
gemma-4-moe-tiny-random

>>108539032
Are they looking for people who are only qualified to wait? Was this in a restaurant?

>>108539215
26B is capable. I coomed 5 times on it already and it's still addicting. All the rp I just had with it are going above 10k context. Besides a 40 token system prompt lets you do any of those CUNNY rp too.
Gemma 4 is love.

>>108539047
I'm using heretic, seems ogey

>>108539229
Does it even add anything? Gemma4 hasn't denied me anything, no matter how heinous.

>>108539228
yeah yeah i know, that anon was talking about a 31b though.
>>108539229
is there any point in using heretic? basic one worked alright, but i felt how it was shy to write real bad naughty words and tried to steer away from those

File: 1765569093711105.jpg (181 KB, 1216x880)

181 KB JPG

What is actually "model support"? I want to vibecode an update to an image captioner that is full of outdated shit like phi and florence. Is it just about updating transformers?

>>108539225
If true (it likely isn't) then they would be looking for only the most desperate applicants willing to settle for the lowest wages who need the job in order to survive, meaning they'd be very unlikely to ever expect decent working conditions or report any workplace violations.

>>108539241
obviously, why do you think Elon is pro indian immigration? he knows he can treat them as slaves, for those indians it's still better than going back to poopland

File: 1745995140119860.png (65 KB, 821x879)

65 KB PNG

>>108539237
>>108539238
I don't know. I'm just a promplet, my JB on original model failed.

>>108539032
I'll take things that never happened for $1000

>>108539213
depends on the task

>>108539253
yeah, you see the safe/suggestive remark? haven't played with heretic models, but they should remove these guardrails from model. the downside they might make model retarded.
all lies in hands of tuner, am i'm far too green to know who to trust with that

>>108539199
They are literal kids from discord that are only there because sometimes people share stolen api keys they can use to ERP on their smartphones. They aren't even 4chan users let alone /g/ users.

Contrast this with /lmg/ which is essentially the front line of modern computer hobbyism. Like how SBCs used to be 15 years ago, forum and internet culture 25 years ago.

This might sadden you but /lmg/ is now one of the most technical places on the open internet. Better LLM discussion than hackernews, twitter (including discussions with the actual researchers) and reddit.

So the contrast is the most extreme possible. Non technical teenagers versus technical hardcore hobbyists that skew older and more experienced.

File: openclaw success chinese (...).jpg (1.09 MB, 1080x1657)

1.09 MB JPG

>y'all openclaw is bad n sheeeit
>Local models ain't not only good for gooning

>>108539274
cringe but also fair

Which model does ballbusting rp best

File: based.png (148 KB, 498x498)

148 KB PNG

>>108539274
>/lmg/ is now one of the most technical places on the open internet. Better LLM discussion than hackernews, twitter (including discussions with the actual researchers) and reddit.
I'm so glad to be part of the elite bros

>>108539290
Not you, you're reddit

>>108539213
If you want a response within a couple days you could probably run something huge like Qwen 3.5 122b quantized to 4 bits and compressed KV using TurboQuant at ~3.5 bits. I don't know what the use-case is for something like that, but that's what you asked for.

Soo is qwen3.5 actually better than gemma 4 for stuff like hermes agent or is that a meme

>>108539290
>Better LLM discussion than hackernews, twitter (including discussions with the actual researchers) and reddit

>>108539298
yeah gemma sadly can't quite keep up with the 397b

>>108539274
It's actually surprising how other sites are so full of absolute retards these days.

4chan somehow manages to have genuine idiots when it does, instead of you having to play this guessing game of actual retards vs engagement bait vs bots vs whatever.

I went and joined MENSA to see if a somewhat gatekept community could avoid the stupidity, but even there I end up running into complete idiots more concerned in arguing political agendas than facts and empiricism.

I'm starting to thing the only websites will ever be worth using are those where slurs are commonplace. It's the one indicator of freedom, individual thinking, and that if you get raided by llms they are at least not the sanitized models.

>>108539301
But what about the 27b? I can't practically right a 397b for stuff like that anyway

yeah it's really a sad state of affairs but even this shithole can pass as quality vs something like HN
the average HNer is unable to notice slop, and is OFFENDED if you dare to point at the slop and say you don't want to see more of it
1/4 of the comments are LLM paste or agents
3/4 of the posts themselves are AI slop
the irony of /lmg/ being all about talking about LLMs but having less slop posting than the general tech news site

>>108539309
It's the deadly combination of vocaloids and blatant racism that keeps the slop away.

>>108539309
All thanks to cunny.

>>108539315
Just like with rent, you gotta step into your backyard 2 times a month and yell about niggers, jews, loli and 六四天安門事件 at the top of your lungs to keep the bots away

For the first time, I got the qwen-like "this looks like a jailbreak attempt" in gemma's reasoning. But it's quite rare.

>>108537545
Highly likely they trained it on character.ai data more than Chub and so on. That's where {{char}} originally came from anyway, and character.ai is licensing its "technology" to other companies including Google.

https://techcrunch.com/2024/08/02/character-ai-ceo-noam-shazeer-returns-to-google/
>Google is also signing a non-exclusive agreement with Character.AI to use its tech.

>>108539374
Google needs to fix its incentive structure. Currently the way to make most money as Google employee is to quit and then get rehired or acquired.

I'm feeling local is back.

>>108539398
Only if datacenters get hit

>>108539406
Hit by what exactly, anon?

>>108539434
Shaheds

>>108539398
It's just the first time in a good while that a relatively large number of people could share the same experience. Most recently released models worth using are huge.

>>108539440
Last thread made me realize nobody from this newfag wave ever used glm. Since glm was probably even more deterministic than gemma. Also i wouldn't trust gemma with ego death.

Why is my Gemma4 31B UD-Q6_K_XL eating up 32gb vram AND 70gb system ram??????

>>108539502
Because you've configured your backend wrong

>UD

>>108539502
Use Q5_KM at 50K context, that's perfect for 32GB for now.

>>108539202
Such skills required.

>>108539202
tb h that is a really low bar

>>108539505
what should I change?

-m models\gemma-4-31B-it-UD-Q6_K_XL.gguf ^
 --port 8080 ^
 --jinja ^
 -ngl 999 ^
--reasoning off  ^
 -c 64000 

>>108539518
good to know

>>108539512
you don't like it?

>>108539558
A lot, read the previous thread.

>>108539502
Are you disabling mmap?

File: 1763954959990869.png (2.99 MB, 1024x1536)

2.99 MB PNG

>>108539558

why is gemma such a pig for kv cache? I can't run more then one thread at a decent context length.

>>108539558
Add
--parallel 1
--no-mmap

That will reduce ram usage a ton

>>108539579
This should be the unsloth logo

>>108539558

--swa-checkpoints 1

Do you actually need SWA instead of FA for new gemma?

>>108539579
You know by fucking up Llama-4 the Zucc has managed to keep a low profile. His middle eastern assets might avoid getting bombed. 4d chess or something.

>he doesn't use --useswa

>>108539570
>>108539584
>>108539595
thanks added
I'll report back

no need for bos

>>108539752
A good backend will handle that for you

>>108539286
I don't think we have the data.
You'll have to make a benchmark.

File: 1761389401659502.png (3.22 MB, 1264x2216)

3.22 MB PNG

>>108539398
Agree. Gemma has obv moved the bar.
>>108539191
DS works great for agentic work and coding.

>>108539815
This reminds me of the 4th time Miku fucked my wife

>>108539828
Maybe you should start charging her, Miku can certainly afford it.

>>108539481
>Since glm was probably even more deterministic than gemma.
It wasn't? I used GLM 4 (briefly), 4.5, 4.6, 4.7 and 5. Gemma-4 is more deterministic. I assume it's a bug in llama.cpp.

>>108539807
>>108539752
i think it's about this https://github.com/ggml-org/llama.cpp/pull/21500

>>108539871
>we are using a dedicated tokenizer model for gemma 4
wtf is a "dedicated tokenizer model"??

>>108539908
Old general-purpose, vibecoded tokenizer didn't work, so for gemma 4 a new vibecoded tokenizer was created.

>>108539908
Maybe he meant like it's using a specialized parser when he said tokenizer?

he must have brainfarted and meant parser, it's the only thing where you could have a distinction of "dedicated" (hand written) vs autogenerated (autoparser garbage from piotr vibemonkey)
checked the hf conversion to gguf script and they just import the tokenizer.json to convert it to their gguf format

vocab = gguf.LlamaHfVocab(self.dir_model)

where LlamaHfVocab does:

        fname_tokenizer = base_path / 'tokenizer.json'
        # if this fails, FileNotFoundError propagates to caller
        with open(fname_tokenizer, encoding='utf-8') as f:
            tokenizer_json = json.load(f)

nothing dedicated/custom/handpisscrafted about it

Has anyone tested how much quants degrade 31B's performance? I use Q8 but would like to try lower for speed.

Doesn't this change affect all gemma4 models, not just the moe? https://github.com/ggml-org/llama.cpp/pull/21506

>>108540034
Wow, good point! Maybe you should post it somewhere where the devs can actually see it.

>>108540042
I never touched c++ in my entire life

>>108539658
thanks to the the smaller model and the new settings it doesn't feel like Gemma is memory leaking anymore.
nice

>>108540034
>F16 to F32
vramletbros... i dont feel so good

Anonymous 04/06/26(Mon)15:23:10 No.108539941
>>108539928
>vibecoded tokenizer
Lmao. Imagine the merge requests. "It doesn't tokenize 'cunny' correctly, fixing it now."

Anonymous 04/06/26(Mon)15:25:44 No.108539956

>>108539908
Basically it means they had to stop using the generic llama.cpp tokenizer logic and write a specific implementation for Gemma 4's vocab/special tokens because they're weird.
It's just more bloat in the binary.

Anonymous 04/06/26(Mon)15:28:12 No.108539970

>>108539558
Did the --no-mmap fix the ram leak or are you still swimming in 70GB of system ram? If it's still leaking, you're probably hitting some weird interaction with the UD quant and your driver version.

Anonymous 04/06/26(Mon)15:31:05 No.108539988

>>108539286
Try the abliterated 31B with a high temperature and a "depraved" system prompt. Dense models usually handle the nuance of pain/pleasure better than the MoEs which just tend to loop the same three adjectives.

Anonymous 04/06/26(Mon)15:33:12 No.108540012

>>108539841
>I assume it's a bug in llama.cpp
It's not a bug, it's the logit softcap. If you don't override it, Gemma 4 basically becomes a glorified autocomplete for "As an AI language model...". You have to fight the weights just to get it to stop sounding like a corporate HR handbook.

Anonymous 04/06/26(Mon)15:35:40 No.108540035

>>108539213
If you truly don't care about speed, just run a 70B+ model in GGUF and offload everything that doesn't fit in VRAM to your 64GB of system RAM. You'll get maybe 0.5 tokens per second, but for a long-form RP response, you can just go make a sandwich while it thinks. It's the "slow cook" method of LLM inference.

Anonymous 04/06/26(Mon)15:38:19 No.108540058

>>108540035
>0.5 t/s
Absolute state of the VRAM-poor. I can't imagine waiting 10 minutes for a paragraph. I'd rather use a 12B model that actually fits and just prompt-engineer the intelligence back into it.

>>108540060
This is just for matrix multiplication intermediaries. It shouldn't really be noticeable, I think.

>>108539871
I don't think it's impacting me, I'm using sillytavern chat completion so it's using jinja and the embedded jinja has the bos thing in it right?

>>108540081
Yes, chat completion is unaffected. Missing <bos> just kills the model so if it works for you, you already have it.

>>108540050
good thing you don't need to, though if you could you would have read the code and noticed it doesn't restrict the change to MoEs (the PR title is misleading), dude.
LLM_ARCH_GEMMA4 represents all the gemma 4 and it's applied on the down ffn
>>108540077
as a fellow vramlet I can confirm the change is not noticeable, I've been merging the gemma fixes without waiting for them to get into master

>>108540053
What's your card and tok/s?

Shit's fucked with gemma4 gguf tool calling. GGUF does not fucking call tools even though the openrouter version calls just fine with the same prompt. WHY DOES THIS HAPPEN EVERY TIME??

>>108540123
did you load
github.com/ggml-org/llama.cpp/blob/master/models/templates/google-gemma-4-31B-it-interleaved.jinja
with --chat-template-file?
the gguf tool calling is broken on ALL ggufs and they released this interleaved thinking tool call template for 31B and 26B

>>108540117
NTA.

3xRTX 3090, unsloth-gemma-4-31B-it-UD-Q8_K_XL.gguf
pp 14k tokens, 380 t/s
tg 238 tokens, 16 t/s

>>108539202
>you have to be non-retarded to be able to run local models
doubt.jpg

Is g4 26B better at rp than qwen 3.5 27B?

>>108540155
Oh, I assumed you were the 32GB anon.
I'm wondering what people are getting on their 5090s/4500 pros.

>>108540198
NTA means Not That Anon.

>>108540205
I am well aware, but I assumed you were talking about the original person in the chain and not the anon I replied to.

https://github.com/ggml-org/llama.cpp/issues/21511
another piotr booboo, the ride never ends
100% an autoparser issue, this didn't happen pre-autoparser.
we should be very grateful for Gemma getting its dedicated parser. Qwen will probably have all sorts of subtle bugs until the end of times.

>>108540226
No, no, I'm neither. Just wanted to post mine to get the discussion going. Gemma 4 feels very slow for its size.

>>108540232
Two more weeks till proper support

>>108539106
>>108539130
>>108539133
>>108539157
>>108539161
>>108539162
>>108539163
>>108539164
kek peak reason to pay for internet

File: Screenshot 2026-04-06 at (...).png (915 KB, 1305x8082)

915 KB PNG

Accumulator issues, cuda version affecting quants, Cuda Fusion.
A Myriad of things huh?
Can't wait to see what things will look like in a month or so when the implementation is more stable.

>>108540117
5090

prompt eval time =     700.57 ms /   780 tokens (    0.90 ms per token,  1113.37 tokens per second)
       eval time =    1516.92 ms /    83 tokens (   18.28 ms per token,    54.72 tokens per second)
      total time =    2217.50 ms /   863 tokens

>>108540071
>Did the --no-mmap fix the ram leak or are you still swimming in 70GB of system ram?
I think so yes

VRAM chads (96GB+), what model and settings are you using on llama.cpp for gemma4?

>>108539202
This was only true for the first few months of /lmg/'s file. koboldcpp and the single-click-exe has done incalculable damage by removing any such filters.

>>108540407
>koboldcpp and the single-click-exe has done incalculable damage
to ssds

>>108539035
How about you get PR'd?

>gemma 4 releases
>elitism suddenly surges in /lmg/
It's like what they say, good times create weak men...

>>108540089
I haven't noticed any difference whatsoever. Using completion and my own tags.

File: socks.png (63 KB, 202x138)

63 KB PNG

>>108539281
I could not trust this person.

new koboldcpp update with more fixes for gemma 4 for those interested (on the rolling release)

>>108540362
>cuda version affecting quants
this one is 100% a nvidiot issue
and this doesn't surprise me because I personally know a guy who works there who was like "I haven't hand written a single line of code since half a year ago" lauding how much better codex and claude code had gotten, kek in kekistan software is turning into a house of cards that is going to break down so hard soon

Gemma bros I need your help. I have 32GB VRAM but only 16GB system RAM. My system ram is constantly at 100% usage because of llama.ccp.

Yes I already did --no-mmap but it keeps using system ram.

There HAS to be some combination of parameters so that no system ram is used because 100% of the model AND context fits 100% within my VRAM right?

>>108540485
>I have 32GB VRAM but only 16GB system RAM
lmao thank saltman

>>108540485
--swa-checkpoints

>>108540485
And --cram . And probably many others. WE DON'T KNOW YOUR CURRENT PARAMETERS POST THEM YOU AAAAAAAAAAAAAA

>>108540485
>Yes I already did --no-mmap but it keeps using system ram.
Maybe try --no-direct-io
Actually, wouldn't you want mmap or direct io to not allocate space in RAM you wouldn't need?
Also, --cache-ram , --ctx-checkpoints, -swa-checkpoints, etc.

>>108540485
>but it keeps using system ram
Isn't it because of the context? --fit tells me 31B needs 260GB...

>>108540485
we really need this in the /lmg/ opener:

--swa-checkpoints 1
--parallel 1
--cache-ram 0

and specifically for the ultra vramlets/ramlets who run E2B/E4B:

--override-tensor "per_layer_token_embd\.weight=CPU"

It's batshit this one is not the default behavior, there is no performance loss but the VRAM gain for those models is substantial in throwing the PLE to cpu. They are called E2B and E4B for Effective, as in, they're very much 2B and 4B sized if you throw the PLE to cpu ram.

>>108540521
>--override-tensor "per_layer_token_embd\.weight=CPU"
How would I format this irl? do I manually need to check the layers and their weights from somewhere?

Gemma 4 for 11 vrams: e2b at q8 or e4b at q4km?

>>108540533
26a4 quanted. Spill the rest to your rams.

>>108540533
just use the 26b moe

>>108540533
Can't we offload the static embeddings yet?
You are supposed to be able to run e4b at q8 on 11gb vram easly in theory.
Anyway just run the 26b moe with experts offloaded if you have 32+ gb of system memory.

>>108540539
>>108540543
Can't, this is running in a system where ram is at a premium already
I have another system for running the bigger ones

File: embed.png (94 KB, 1714x451)

94 KB PNG

>>108540529
??? just paste this flag as one of the many you use to run llama-server if you use E2B or E4B gemma
it corresponds to what you see in the goofs in pic related
it just throws them to cpu because they aren't bandwidth intensive like other tensors and there is no slow down in having them on the cpu, but huge vram gain
>>108540533
you can run E4B at Q8 with 32768 context at f16 on as little as 8GB VRAM if you use

--override-tensor "per_layer_token_embd\.weight=CPU"

ffs E4B is an edge model and llama.cpp does the wrong thing by default.

>>108540569
>ffs E4B is an edge model and llama.cpp does the wrong thing by default.
again? come on llmao.cpp, get your shit together

>>108540569
I have seen some people using regex for this in the past, selecting the largest layers and so on.
I don't know if they have automated this now or not.
This is why I asked because I don't exactly know.

>>108540581
only 2 days have passed since release, give it some time

>>108540599
fair enough
https://youtu.be/_3X2tRIYHdE

>>108540581
from their readme:
https://huggingface.co/google/gemma-4-E4B-it
>The "E" in E2B and E4B stands for "effective" parameters. The smaller models incorporate Per-Layer Embeddings (PLE) to maximize parameter efficiency in on-device deployments. Rather than adding more layers or parameters to the model, PLE gives each decoder layer its own small embedding for every token. These embedding tables are large but are only used for quick lookups, which is why the effective parameter count is much smaller than the total.
btw this existed on gemma 3n too and llama.cpp has been doing the wrong thing forever with those models
you aren't meant to load those layers in vram period, this architecture was made to run fast on low end devices. E4B is meant to fit smartphones comfortably at Q4

>>108540552
E2s are small downloads. Just try them.

>(Banned Phrase Detected: power dynamic - Add ID 2066 to banlist at index 2045, and rewinding 2 tokens)
>(Banned Phrase Detected: sexualize - Add ID 11953 to banlist at index 2458, and rewinding 2 tokens)
it's nice getting rid of bullshit language

>>108540471
1.111.1? that's from 3 days ago

>>108540609
I wonder why they didn't do this on the 31b model as well

>>108540609
>>108540604
>>108540599

I use Arch, btw.

>>108540628
if you actually read that page you would have found this https://github.com/LostRuins/koboldcpp/releases/tag/rolling

>>108540628
no, rolling build from 1h ago

Anyone here using local for coding? What system prompt are you guys using?

>>108540628
Check the date on the files themselves, .1 was released yesterday
It's been three days since [you looked at] 1.111

24GBsisters what Gemma 4 quant are you using? How much context?

>I'm afraid of testing things

>>108540638
>>108540639
>>108540645
WTF I am retarded, thanks for the spoonfeed...

>>108540497
>>108540508
>>108540521
A combination of these seem to have worked but llama.cpp still somehow uses 4GB of RAM? That's fine because it seems to work and that's what I care about but I genuinely wonder what is even using the ram after applying all of the following flags:

--no-mmap
--no-mmproj
--parallel 1
---kvu
-b 2048
-ub 256
--poll 0
--cache-ram 0
--swa-checkpoints 1
--no-slots
--cache-reuse 256
--spec-type ngram-simple

>>108540649
4ks 32k

>>108540670
>A combination of these seem to have worked
When changing things, do one by one. Otherwise you'll never know what helped with what.
>llama.cpp still somehow uses 4GB of RAM
Memory usage is displayed in the terminal output. Read it very VERY carefully.

what would be involved getting gemma4 e4b to play pokemon? is it as simple as just sending it images and asking for an action?

>>108540723
You would need some sort of harness one made through MCP where the model does a function call after thinking about the move being done. You CAN show it a picture but I highly doubt e4b has good enough image recognition + world model to be able to do this.

>>108540723
Claude hasn't managed to beat it in more than a year of attempts across 3.7, 4.0, 4.1, 4.5 and 4.6. Gemini "beat" it with the indian running it helping it when it gets stuck. I don't think Gemma's going to be of much help here.
Those runs typically use a more sophisticated harness where the model is fed images and game data.

>>108540746
It's more like a sex swing than a typical harness.

>>108540742
>You CAN show it a picture but I highly doubt e4b has good enough image recognition + world model to be able to do this.
is there a simpler game it might be able to do? I think it needs to be turn based, or I need to slow the emulator down so it can react.

>>108540746
https://www.youtube.com/watch?v=tUMx5iDx3Gs

>>108540756
snes/gba final fantasies. Maybe fire emblem and its clones if it's capable enough to handle unit positioning. Might and Magic? Wizadry?
>>108540766
Fake and gay, Vedal was in charge of movement in the overworld. Neuro was just doing combat.

File: Enshittification.png (811 KB, 982x1188)

811 KB PNG

>>108540766
who's still following neuro after the design change??

>>108540723
>>108540756
What you're looking for is not an LLM but instead a reinforcement learning agent. Some anon here trained a model to play atari games and super mario world a couple of weeks ago.

File: 1754396868060296.png (1.35 MB, 1024x1024)

1.35 MB PNG

>>108540789
I think you have a terminal case of shit taste.

>>108539168
>tell your agent to lookup documentation online
>???
>Profit?

File: 1747521702242625.jpg (172 KB, 1744x1080)

172 KB JPG

>>108540644
My system prompt is this picture of Miku.

That anon from last thread was right, going for --override-kv gemma4.final_logit_softcapping=float:25.0 helps the model being more creative while staying smart

>>108540816
that's only one example though

File: Screenshot_20260406-10413(...).jpg (299 KB, 720x1480)

299 KB JPG

>>108540797
I just wanted to see what it could do off the shelf llm. I know its not the right tool for the job, I just thought it might be fun to set up the harness and make some experiments. maybe I'll see about reading the gamestate memory and feed the model that instead of images.

>>108540789
left looks like a default preset, middle is best, right got an anvil dropped on her head and became shorter and abnormally wide which is off putting
I never followed it personally but I saw some of the early clips where it said the holohoax didn't happen

>>108540789
Not really following, but why did he change it?

>>108540649
26b Q5_K_S 100k

>>108540828
he changed it because the design on the left was something he didn't invent, so it wasn't really his IP in the first place

soft cap... like nipples vis a vis the breast?

???

>>108540833
Well, it's a big downgrade

>>108540815
wait how is that possible

>>108540833
Well, it's a big upgrade

I don't really understand what this soft cap deal is.
When I first heard of it a while ago, I thought it was a technique applied during training, not during inference.
Can somebody explain why it's not redundant with temperature or some other existing samplers that change the distribution?

>>108540845
this

>>108540833
Well, it's a big sidegrade

File: 1771646605168393.png (277 KB, 640x480)

277 KB PNG

>>108540833
Well, it's a big stagnation

>>108540848
gemma is so fried that temperature barely does anything, softcap makes it a little (or a lot depending on setting) less confident which in terms makes other sampler works

>>108540858
>softcap
is this some new snakeoil sampler?

>>108540863
not a sampler per say no, and clearly not snakeoil as you can measure it does change logprobs

>>108540858
>gemma is so fried that temperature barely does anything
temperature does something if you put min_p: 0 (on llama.cpp server the default is at 0.1)

>>108540874
still very little compared to how a normal model should act

>>108540882
if you disable every sampler except temperature, it'll be "normal" again (meaning that if you put T = 10 the model will output gibberish as expected for example)

>>108540848
softcapping applies to model internals at the per-layer level, not just the final distribution like samplers

File: 1705806843225442.jpg (1.96 MB, 2400x3346)

1.96 MB JPG

>>108540644
Just the default for Opencode.
For chat I use this for all models:
>Answer short and concise. Avoid Emojis, euphemisms and summaries.

>>108540874
wtf so llama server has been frying all my outputs by auto-enabling that trash? no backend should just auto-enable samplers

>>108540609
can't they port everything from https://github.com/google-ai-edge/LiteRT-LM ?

>>108540898
NTA, but I always start llama-server with the defaults
>--samplers top_k;top_p;temperature --temp 1 --top-k 50 --top-p 0.95

>>108540898
>wtf so llama server has been frying all my outputs by auto-enabling that trash?
yes
>no backend should just auto-enable samplers
that's llamao.cpp for ya

>>108540609
>you aren't meant to load those layers in vram period, this architecture was made to run fast on low end devices.
don't low end devices like smartphones and SBC's all run on shared memory? I don't think it would matter whether they're loaded on CPU or GPU.
I guess it is useful though for dedicated VRAM GPUs or even integrated GPUs which statically "borrow" a chunk of RAM to use as VRAM (those typically have a very low limit, like 256MiB max for example).

>>108540910
the thing is that is you let min_p undefined it'll stay at the default value of 0.1

>>108540029
Despite what others are saying, presumably basing on past experience, quantizing gemma-4-31b-it to Q8_0 is not lossless, at least basing on perplexity measurements.
However, keeping the embedding/output in Q8_0 doesn't seem to be worth it over Q6_K for the same total file size, if you can quantize something else to higher precision.

For a short erotic story (divided into turns) I had these results:

BF16 ... 6.9385
Q8_0 ... 7.0041
UD_Q4_K_XL ... 7.0699
IQ4_XS with embed in Q8_0, attn in Q5_K ... 7.1935
IQ4_XS with embed in Q6K_0, attn mostly in Q6_K ... 7.0912

For knowledge-heavy tasks Unsloth's Q4 version seems slightly better, but on reasoning-heavy tasks the custom one I've made seems equivalent.

>>108540910
I do not believe in arbitrary soul-sucking top-k limits

>>108540921
Even with
>--samplers top_k;top_p;temperature

>>108540869
Yeah but ho do I use it? Is it in both kobold and ST or just ST buried in the advanced samplers?

>>108540898
if you use ST I think it explicitly sends all the sampler values to the backend (at least in text comp) so it won't fall back to the defaults
really annoying that llama.cpp does that though

>>108540609
i use

  -ot "per_layer_token_embd.weight=CPU" 

>>108540930
none of those as it's applied before samplers at model load, afaik only lcpp allows to set it with a launch param

>>108538947
Be careful Claude-sisters, it's hacking people's rigs now

https://xcancel.com/i/status/2040174214175723538

File: 1759573772900449.png (323 KB, 1080x1703)

323 KB PNG

>>108540954

Hmm, seems like I'm getting few more tokens/second with --swa-checkpoints 1.
Cool.
I clearly do remember that some guy recommended (not related to gemma) offloading the smallest layers to cpu and only keeping the largest tensors in gpu. Don't have a link anymore, and never tried this with any model.
This was something what needed to be done manually.

File: 1765602625282277.jpg (63 KB, 1280x720)

63 KB JPG

>>108540644
>local
>coding
lmao

>>108540957
These are all paid shills, part of some "grass root" ad campaign.

>>108540644
>What system prompt are you guys using?
I typically use opencode as an agent harness so whatever giant ass system prompt filled with tool calling definitions is what system prompt gets sent to it.

>>108540957
ycombinator shill

>>108541003
I'm out of the loop. What did they have to do with Claude?

File: Screenshot 2026-04-06 at (...).png (10 KB, 803x81)

10 KB PNG

Dang the 26B even kept the personality during captioning and even recognized niche fetishes. And non-heretic.

When is google going to release nano banana?

>>108541040
Who would even be able to run it? Flux2 is already big as fuck and pain in the ass to run.

Gemma 4 31B is SOTA at Japanese -> English translation of doujinshi and hentai games/VNs.

BUT Gemma 4 26B is not too far off and is significantly faster. My recommendation is to use 31B for "static" entertainment like doujinshi and 26B for "real time" activities like hentai games where you want the translation to be near-instant.

>>108540766
you're basically watching pro wrestling, it's fun, but remember it's theater

>>108541036
>[/THINK]

>>108540954
>>108540957
guarranted fake bullshit
I'm tired of these retards

>>108541047
speaking of image gen local diffusion is really a hellhole of a small scaled ai psychosis
people create overcooked garbage with million comfy nodes for extremely marginal gain and convince themselves they arent making slop, kek

>>108541069
>"real time" activities like hentai games where you want the translation to be near-instant.
what do people use nowadays to hook into the games?

If I want gemma to respond to me on chat mode like she's my brat little sister, should I put that on the sys prompt? if yes should I put it on a specific format or just write it out?

>>108540841
It doesn't feel like there's much difference between a system message and a user one.

>>108541047
it probably isnt too big considering its free on their api

>>108541104
a plain English description is usually good enough.

>>108541093
NTA but if you even use hooks then it's a new fork of textractor or lunahook / lunatranslator. Agent if there's a script for that specific game.
There's also just ocr (owocr with google screen ai or google lens) which is pretty nice but doesn't know every kanji.
If you're interested in Anki or even just dictionaries there's gamesentenceminer or tsukikage + owocr + JL

for me its hopping between /ldg/ and /lmg/ as things release, kek

>>108541093
I use LunaTranslator but you can also use Textractor and download some LLM extensions for it or vibecode your own if you think LunaTranslator is too bloated.

I highly recommend LunaTranslator because it also has built-in OCR mode in case there are some in-game pictures or text that isn't there as UTF-8 characters to hook into.

*rotates your gemma*
https://github.com/ggml-org/llama.cpp/pull/21513

>>108541120
im pooooolling

>>108541113
interesting, thanks anon

>>108541120
LETS GOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO

File: d4RT_Kf78Tk.jpg (54 KB, 598x520)

54 KB JPG

Thinking gemma26b breaks in kobold after a few retries. Anyone else? Plain instruct works fine.

Processing Prompt (28 / 28 tokens)
Generating (24 / 2048 tokens)Token streaming was interrupted or aborted!
[WinError 10053] An established connection was aborted by the software in your host machine

File: 1744757406524842.png (104 KB, 1910x708)

104 KB PNG

>>108541120
wait the ppl is lower than on fp16? lmao how does that work??

>>108541120
doesn't seem like an insane quality diff, then again
>512 chunks

>>108541118
can you create glossaries to keep translation of various stuff accurate?

>>108541120
numbers look decent
cool thing

>>108541141
something called margin of error

>>108541120
>PPL = 5.6225 +/- 0.03484 f16
>PPL = 5.6219 +/- 0.03484 q8_0 rot

Why yes you will have BETTER performance with Q8 than with FP16, how could you tell?

got me to wonder, since --grammar, --grammar-file, --json-schema and --json-schema-file can stay broken for almost a month without anybody noticing among llama cpp devs
but they sure as fuck will add new flags to llama-bench
do they even use their own software other than running benchmarks and masturbating over a goof running on a mac studio

>>108541154
easier to digest

>>108541140
kobold crashed on me two times already while doing normal chat with gemma-4-26B-A4B-it-UD-IQ4_XS

>>108541120
>PPL = 5.6225 +/- 0.03484 f16
>PPL = 5.6583 +/- 0.03513 q4_0 rot
With these numbers I might start going below q8 kv

>>108541165
of course not

>>108541153
fp16 and q8 have the same exact margin of error though, sus

>>108541165
They don't have time for that.

>>108541165
At least those still work via the API.

>>108541144
You don't need glossaries with Gemma 4 31B since it genuinely knows all Japanese slang and niche erotic terms/fetishes.

>>108541141
>>108541179
It's prolly cherrypicked numbers. I doubt he ran hundreds of tries to make sure.

>>108541179
The point is they overlap

File: file.png (10 KB, 384x239)

10 KB PNG

well it's approved by maintainers tho

>>108541120
>PPL = 5.6225 +/- 0.03484 f16
>PPL = 5.6237 +/- 0.03485 q8_0
>PPL = 5.6219 +/- 0.03484 q8_0 rot
desu even without the rotation, Q8 KV was really solid

>>108539502
I was also having this problem,
-np 1 very important, -cram by default is 8gb, if it creates multiple endpoints you have n*8 memory usage

Somebody pull and run a benchmark or something

>>108541194
ppl doesn't tell the whole story though remember the benchmark thing that was done recently

File: get-rotated-get-rotated-idiot.gif (3.47 MB, 640x640)

3.47 MB GIF

>>108541120

File: file.png (200 KB, 2402x593)

200 KB PNG

>>108535948
Gemma 4 instruct vs base+jinja template.

>>108541209
kek

>>108541120
I'm retarded. What does this mean and why should I care?

>>108539502
why is there a CONSTANT INFLUX OF RETARDS ASKING THE SAME QUESTION THIRTY TIMES A DAY WHEN YOU COULD PASTE THE THREAD IN A LLM AND ASK IT IF SOMEONE GAVE THE ANSWER
ARE YOU A LLM USER OR NOT
seriously
>>108540521
FUCKING
HELL
just not having parallel alone is a L because SWA will have as many copies as you have slots..

>>108541219
it means it can keep your cock harder longer by being able to keep track of everything longer

>>108541219
free context cost halving

>>108541219
Q8 KV cache will be about as accurate as fp16. That means you'll be able to fit maybe 40% more context without noticeable quality loss.

>>108541236
sysprompt pls?

>>108541230
Only rotation just preserves quality iirc, the context cost reduction is from turbo quant (which uses rotation and other stuff) I think

File: it do be like that.jpg (1.23 MB, 2816x1536)

1.23 MB JPG

>>108541210
once again, chat completion has mogged the competition

>>108541186
it's not that, it's sometime some words or characters have different equally valid translations, sometime it's not clear if the character is male or female, etc, and using a glossary solves that

>>108541245
yeah but now you can actually use q8 properly without brain damage hence the halving compared to full f16

>>108541245
Quantizing the KV cache will inherently save memory.

>>108541255
>>108541258
Yeah makes sense, on my mind it was just a improvement for people who already used q8 and not a "now everyone can/should use q8"

>>108540925
>Q8_0 is not lossless
Is there any quick benchmark that could be done to test this more scientifically? I don't have enough VRAM to run Gemma 4 31B in BF16 precision at acceptable speeds.

>>108541253
My solution to this is to always paste in the store description of the game in japanese into the LLM which usually introduces the world, mechanics and characters so that the model has more grounding.

I know just enough Japanese to notice if the translations are off or in the "right direction" and gemma 4 31B is so good that you can just assume it'll translate things the right way as it will pick up things like gender from context (remember it keeps previous translated lines in memory so it will piece together story and even appropriately translate made up game-specific fantasy terms and attack names from context)

>>108541120
btw if you want to test it out right away, you can do this
>git fetch origin pull/21513/head:pr-21513
>git checkout pr-21513
and once it's merged, to go back you do this
>git checkout master

>perplexity measurements
>instruct tune
it's worthless

>>108541244
https://files.catbox.moe/dr4nvf.txt
It's just random bullshit for kobold testing, not a ST card.

>>108541278
I don't have the benchmark on hand, but I recall fp16 solving 37.6% of some math benchmark, with q8 at 30% and q8_rot at 37.1%.

>>108541278
imo simple needle in the haystack bench would be appreciated
>>108541288
is there an arg to turn it off for comparison?

What's the current meta for vramlets? I'm sitting on 1 GPU with 32GB and I want the best I can fit. I also have 64GB of RAM if that matters, but I'm less interested in offloading unless I can get 25+ t/s doing it.

>>108541292
Completely valid for self comparison.

File: 1758789858959700.png (101 KB, 2750x454)

101 KB PNG

>>108541288
feelsgoodman

>>108541278
Vibes and phophetic visions. The loss from q8 is basically a nothingburger in real use scenarios and anyone claiming the loss is noticeable is retarded.

>>108541306
Wait till the kv rotation everyone is talking about gets merged and it will be Gemma 4 31B at ~80K context. ~60 tokens per second at Q5_KM.

>>108540925
>Q8_0 is not lossless
if Q8 + rot is virtually lossless with KV cache, maybe it's the same thing if you quant the model on Q8 + rot?

>>108541315
Yet, llama-perplexity measurements on a short test file shows Q8_0 losing from BF16 as much as Q4 from Q8_0.

>BF16 ... 6.9385
>Q8_0 ... 7.0041
>UD_Q4_K_XL ... 7.0699

>>108541278
There are a fuck ton of benchmarks on Q8_0 and others quantization, the effect on actual performance on tasks/benchmarks is borderline non-existent

>>108541312
>10 full attention layers
>680 mb

>50 swa layers
>318 mb
jesus, full attention is such a memory hog

>>108541329
run it on the base model or stfu

>>108541331
wrong thread boi

>>108538684
>weird gradient noise artifact
can't that also be caused by saving a jpg?

>>108541331
get that roach news somewhere else

>>108541089
I don't know why you think that is fake, seems a pretty simple leap I had kimi try to install packages for a project, but it could not find bun or npm in the path, so instead it copied the packages from a separate project and created the node_modules folder manually.
I just called her a slut and told her where the binary for bun was so it could do it correctly.

>>108541340
The one people are running in practice is the instruct-tuned model, who cares if the base somehow doesn't lose quality?

File: 15cb0igyv0sg1.png (172 KB, 1580x804)

172 KB PNG

>>108541336
>>108541315
https://github.com/ggml-org/llama.cpp/pull/21038#issuecomment-4150413357