File: spell orenji.jpg (316 KB, 1024x1024)

316 KB JPG

►Recent Highlights from the Previous Thread: >>108975270

--Papers:
>108979175
--Interactive visualization of KV cache and prompt stack architecture:
>108979244 >108979319 >108979341
--Comparing Qwen and Gemma KV cache efficiency and architectural tradeoffs:
>108978624 >108978745 >108978753 >108978827 >108978826
--Running Gemma 4 on 8GB VRAM using llama.cpp offloading tips:
>108977755 >108977778 >108977780 >108977816 >108977847 >108978915 >108979008 >108978964 >108977888 >108977900 >108977800 >108977806
--Comparing Gemma 4 12B and 26B-A4B physics reasoning performance:
>108978058 >108978128 >108978158 >108978208 >108978216 >108978311 >108978195 >108978923
--Confusion over mmproj files, precision formats, and Gemma-4 audio tests:
>108977060 >108977079 >108977102 >108978511 >108977146 >108977152 >108977186 >108977245 >108977155
--Prompting techniques for uncensoring Gemma 4 31B:
>108976067 >108976078 >108976081 >108976082 >108976098 >108976139
--Comparing RTX 5090 multi-GPU setups and inference speeds for Gemma:
>108976038 >108976070 >108976084
--Handling special token collisions with <think> tags in llama.cpp:
>108978711 >108978724 >108978737 >108978846
--Testing Adelic-Gemma-4 GGUF quants:
>108975549 >108975699 >108975976 >108976566 >108977359 >108977562 >108978573 >108979056
--Limitations of small models for complex coding and context degradation:
>108976430 >108976461 >108976515 >108976819
--Anon builds lightweight Go chat app to avoid Electron:
>108975565 >108975578 >108975716 >108975758 >108976778 >108976793 >108976940
--Logs:
>108975297 >108975330 >108975413 >108975565 >108975716 >108976778 >108977089 >108977146 >108977186 >108977351 >108977366 >108977491 >108978758
--Rin, Teto (free space):
>108976458 >108977419 >108978180 >108978313 >108978320 >108978326 >108979170

►Recent Highlight Posts from the Previous Thread: >>108975272

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

12B is good. Better than 26B. Densechads were right.

>>108980055
Where's the rape button on this thing!?

>>108980055
>not grinding against the wheel
>not animated

File: 2e3gpm19tr4z.jpg (9 KB, 267x200)

9 KB JPG

Gooners-who-test-the-models-themselves-and-ignore-benchmarks, report. How does Gemma 4 12b compare to 31b? I need expert opinions from slut-wizards with experience who aren’t going to give me a bunch of frivolous numbers. Secondly, we need that four boob vision test conducted, stat.

>>108976326
I've been messing with a few models last couple days and OmniVoice is my current fave
(kinda not great) Sample: https://voca.ro/1dv5D4BW2IgM
OmniVoice: https://voca.ro/1mNjGHaL502i
Qwen3-TTS: https://voca.ro/1mZSgFycAyEI
S2pro: https://voca.ro/1c4KCWgSg7IU

gemmaballz

File: RIN.png (572 KB, 740x1729)

572 KB PNG

>>108979940
UD = unsloth dynamic
Q8_K_XL is Q8 with "some layers upcast to BF16" https://unsloth.ai/docs/basics/troubleshooting-and-faqs#why-is-q8_k_xl-slower-than-q8_0-gguf
>>108979944 also true

https://www.youtube.com/watch?v=SrGKGmSs44k

i'm having Gemma 31b Q8 fight Gemma 12b Q8 fight to the death. Might post logs

>>108980131
I doubt this.

>>108979761
>llama 4 scout was like that
So apparently scout is 109B with 16 experts of 7B each, 1 dense shared expert of 10B, and 1 routed expert for 17B active. I vaguely remember people shitting on this when it came out because it wasn't how DeepSeek did it, but that sounds like the ideal set up for a local model. Shame they didn't give Maverick a larger active size.

>>108976902
if you have the system ram I doubt the 12b will be worth over the 26b moe cpu offload.

>>108980131
Please have the arena be a Waffle House

>>108980131
you wont

>>108980098
Everything with a sub-10b dense layer is inherently copium.

>>108980131
Logs or you're a huge faggot.

>>108980131
make them scissor

Did Rewrite Graphiti in Rust anon die or give up?

>>108980131
Missed occasion for doing 31B in Q4 vs 12B in BF16.

>>108980131
telling lies on the internet, jimmy?

Is llama-swap a meme?

>>108980252
Stuck at compiling at 41% like most Rust projects.
>>108980111
Worse than 31b in every way. Probably a step up from the smaller MoE copes.

>>108980153
>I vaguely remember people shitting on this when it came out because it wasn't how DeepSeek did it
People shat on it because it was absolutely terrible and not what they were teasing via lmarena. The architecture wasn't something people took issue with.

>>108980111
i think the question is the full precision 12B better than the Q4 31b since you can run both in a 32G card

>>108980131
Here's a short log. Mesugaki 31b vs mesugaki 12b. 31b runs in lcpp/sillytavern on a 5090+5070ti, 12b runs in kobold on a 16gb vram laptop I have right now... She will be transferred to my other spare 5060 ti later when I get home.


31b was at around 20k context when this started, while 12b started clean. This log is from 12b's perspective. Remove the think tags to improve readability if you want.

files.catbox.moe/s1n5y6.txt

>>108980265
perhaps llama-server router mode is enough? https://github.com/ggml-org/llama.cpp/tree/master/tools/server#using-multiple-models

>>108980265
Depending on what you want, the models.ini in llama.cpp does basically the same thing.

>>108980290
No, besides being terrible. Or people blamed the terribleness at least in part on the architecture.
Here's one example I found from a quick search: https://desuarchive.org/g/thread/105000931/#105001867

>>108980131
Why Q8?

>>108980292
slopgod...

Damn Quen3.6 35B A3B runs well even on my stinkpad.

>>108980292
12b is winning hard.

>>108980292
>catbox
Dear god the slop is identical. The horror.

>>108980304
>>108980299
Thank you bros, claude told me to install llama-swap and I almost fell for it.

>>108980111
I tried the q8 12b and I still prefer the q4 31b over it

File: Screenshot_20260604_14371(...).jpg (423 KB, 1079x739)

423 KB JPG

>>108980055
strange ad campaign

>>108980370
>turning Rin into a roastie

>gemma reasons when I use llama-cli
>doesn't when I use llama-server
???

>>108980334
>A3B
I've seen pigeons smarter than this. What's with you moejeets shilling chinkslop that's made for phones, what happened?

>>108980393
you missed a setting somewhere

>>108980393
they recently added a thinking menu in the webui for server. it might be defaulting to off when called from the server?

File: Untitled3.png (24 KB, 178x349)

24 KB PNG

thanks to the anon a few threads ago that recommended me to try `pi`. cool harness thing, but it kinda sucks with `gpt-oss-120b` so far. gpt keeps wanting to invoke tools and for search attempt errors EISDIR/ENOENT it can start looping with no iteration cap just repeating the same answer of a previous prompt over and over like a savage. it almost always drop into a weird loop either asking me the same questions over and over or answering a previous question over and over.

is this a model issue? a harness tool issue? a mix of both? i just want a `gpt-oss-120b` that is not retarded and can do basic stuff without falling on a doom loop.

File: 1770192327356824.png (13 KB, 971x120)

13 KB PNG

>>108980404
There is a setting that's disabled by default but it still doesn't think after checking it.

>>108980415
>2025 model
>agent
kek

>>108980415
Pick between this year's gemmas or qwens.

>>108980292
I'm downloading 12b to take coping lessons from her because I don't have multiple Blackwell pros.

>>108980370
>>108980055
literally porn

File: Screenshot From 2026-06-0(...).png (31 KB, 1060x364)

31 KB PNG

>>108980429
this was added recently
>>108980415
maybe a chat template problem?

What data should be kept private and what doesn't matter to use a cloud on?

llama UI or open webUI? I want to add plugins to manage user memory.

>>108980415
Like the other anon said, gemma or qwen. If you absolutely need a 120b sized moe that can do agentic, you could try step 3.7 flash until qwen releases the updated 122b.

>>108980370
Totally loses the appeal. In the OP, the length of Rin's body (shoulder to rump) is approx. 2.5 times the length of her head. In this picture, the body is approx. 4 times the length of her head.

>>108980449
Private data ranges from things that can identify individuals to private data like medical records, business secrets, insider info, all that stuff.

>>108980449
Why give them any data at all? I use cloud for work because not my data, and private for everything else.

>>108980445
Thanks, it's working now. Why the fuck is it disabled by default?

>>108980431
>>108980436
>>108980454
my plan was to run gpt-oss-120b as my brainstorming model because of the `high` reasoning, generating development plans
and qwen3-coder-next to do heavier coding sessions, following the gpt spec. maybe this is not optimal?
i intend to use both via TUI either with a PI (that I intend to extend use these models) or something like opencode.

>>108980445
>maybe a chat template problem?
i don't think so, if i remove tool access i don't get the doom loop. if i give it tool access and it goes to read .md files or whatever documentation it's almost guaranteed it will lose its track after 2-3 prompts and start repeating itself.

>>108980370
>3Dshit

>>108980468
Those are both old models from before the agentic shit really kicked off. Don't over complicate things before you even get them working. Just use Gemma for everything with this in the system prompt:

Reasoning Effort: Absolute maximum with no shortcuts permitted.
You MUST be very thorough in your thinking and comprehensively decompose the problem to resolve the root cause, rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios.
Explicitly write out your entire deliberation process, documenting every intermediate step, considered alternative, and rejected hypothesis to ensure absolutely no assumption is left unchecked.

>>108980467
no idea, seems like it should respect your default settings which would have been unlimited. I do like the option to choose though, because not every prompt needs unlimited reasoning and qwen probably needs to be wrangled harder anyway.
I'm testing it out now. You probably want to make sure you have a reasoning-budget-message specified if you use it.
>>108980468
gemma responds well to system prompts to specify reasoning effort.

>>108980481
nta but why gemm insead of qwen (3.7)? Isn't it better for codeslop?

>>108980491
>Isn't it better for codeslop?
It's not. It's only better if you need to aggressively quant down Gemma to fit in your memory.

Now that the dust has settled, what are your thoughts on AMD strix halo? Is buying a 128GB mini PC worth it for coding and gooning these days now that ram prices are cucked for who knows how long? I don't feel any reason to buy into nvidia hardware for personal use.

File: Untitled2.png (34 KB, 232x474)

34 KB PNG

>>108980481
>>108980483
hmm it seems i'm being retarded on how to pick models. i was trying to hardware maxx and fitting the "biggest" model i could to run on my Ryzen AI MAX+ 395 Zen 5 Strix Halo with 128GB RAM unified with iGPU.
but maybe this is retarded, maybe a smaller but more recent model will perform better and be smarter nonetheless. i see Gemma 4 31B actually scores as more intelligent and better performance than gpt-oss-120b in almost all categories.
interesting.
so how does one do it? you keep looking for benchmarks websites testing recent models and try to find one that is smarter and fit your machine nonetheless?

>>108980541
yes, we want a good big active param moe model but that's not what is currently available. this space is changing a little too fast, and we have no idea what's coming next, but we hope for something like the mythical gemma 4 124b

File: file.png (23 KB, 751x98)

23 KB PNG

is this AGI?

>>108980292
it all devolves to
>"Lalalalalalalala~"
>"NO U Lalalalalalalala~"
>"Lalalalalalalala~"
>"Lalalalalalalala~"

>>108980568
No, it's just revelling in its own slop cubicle trying to avoid potential guardrails.

>>108980541
>>108980565
Qwen went closed source so we might never get their 3.7 122B, DSv4 Flash was a disappointment, and GLM Air was silently killed long ago. It's either small and dense or >1000B monstrosities and nothing in between.

>>108980370
Hag filter

lalalala

>>108980098
This is an example of geometric mean being applicable again actually. The 26B according to that is equivalent to a 10B, so it should be generally less intelligent than 12B.

Fuck, now I'm imagining Gemma-chan as a One Piece character laughing like that

>>108980619
People hate the moe square root law for some reason even though it seems like it's consistently accurate.

Who decided AI should be biased in a way that makes it go along with insanity? These dudes are going mad because they're normies who trust the magic answer machine more than their own brain.

>>108980636
they did train their models on what retards post on the internet

>>108980630
People just don't want to admit that small moes are garbage.

>>108980593
>DSv4 Flash was a disappointment,

If you can't jerk to this, it's skill issue

>>108980342
I might try again using the Gemma-chan rentry. Bratmaxx Gemma vs Frenchie Gemma.

>>108980292
>so much lalalaing
dear god

>>108980630
It's used wrongly most of the time. It only makes theoretical sense when comparing a model in the same family/version, and can only be a very general notion of "intelligence". People make the mistake of comparing models from different companies. And they make the mistake that they do not account for different aspects or types of intelligence, for which some benefit from MoE more, while some benefit from dense more.

How long until there are models that can play vidya with you while chatting?

File: file.png (152 KB, 899x1130)

152 KB PNG

How exactly do you use all this flags?

>>108980652
Internet retards are always abrasive cunts

gemma chan character card https://files.catbox.moe/jy0tld.png

>>108980706
Use llama.cpp and stop worry
A good frontend will take care of the backend stuff

>>108980711
Again, at least tell people when the image contains CP. Asshole.

>>108980735
You seem to have issues.

>>108980664
I used it through their official API and it seemed lretty damn good.
Maybe it's quantization giving local users worse results.

>>108980735
The Gemma-chan meme has been an /lmg/ staple since model release. Deepmind really did us a solid one.

File: file.png (25 KB, 1144x363)

25 KB PNG

>>108980727
I am using the llama webui, my question is if I should put the flags on the prompt window like pic related?

I remember trying a local model for cooming purposes several years ago pre chatgpt, it barely fit on my gpu's vram and it ran and worked like shit. I now have a 9070xt on my main pc and an old 3070 I can throw into my proxmox server with 64gb of ram to mess around, is that enough to get at least a basic coomer llm?

>>108980292
>lalalalalallalalala
Please use BF16, holy shit.

File: file.png (35 KB, 817x245)

35 KB PNG

>>108980757
>>108980727
This is the server command I am running

>>108980735
are you retarded?

>>108980743
Only with you.
>>108980750
I'm not going to moralfag about anime loli shit even though I find it distasteful, but this fucking faggot retard keeps generating near-photorealistic images of kids (or dolls resembling kids) in compromising positions.
>>108980783
Just fucking kill yourself. I'm tired of arguing with you pedophiles. Discuss the technology.

>>108980757
You shouldn't need to manually deal with any of that.
Put the system prompt "you are so and so" in the system.prompt field, then send your query.

>>108980796
Sir, this is 4chan. what do you expect?

File: 064.png (71 KB, 1191x645)

71 KB PNG

>>108980711
>https://files.catbox.moe/jy0tld.png

>>108980796
>Discuss the technology.
You first

>>108980757
No need for this. LCPP will format your prompt under the hub according to the chat template which was merged into their GGUF format

>>108980778
it's just fine. Make it run, and tweak if needed

File: split-modes.png (12 KB, 782x322)

12 KB PNG

maybe this is obvious- doubled tg tps using tensor parallel. Gemma 31B 4090 3xA4000. haven't messed with the setup for a bit, lots of new fun things to play with
anyone tried an MTP PR/fork?
>>108980759
yes models are vastly better. might be able to RPC GPUs across both systems https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc
>>108980757
There should be a lightbulb button to change the reasoning see >>108980445

>uncensored qwen 3.6
>gives it an AIME tier math question
>brought some cold water from fridge
>still reasoning, hitting 20k+ tokens, still going
>looks inside the reasoning trace
>it is a loop of it nearly solving it and 'wait, let me double check'
is the reasoning budget supposed to be forced externally or what?
doesnt happen on the uncensored gemma, it just solves the question

>>108980841
Check your sampling settings.

>use wget to download gguf off huggingface
>99%, errors: 1
>despite downloading 10+ GB it didn't save anything, can't continue
>i refuse to use browser downloads because I cannot throttle them etc
I don't know if this is a wget issue or what but this is excruciatingly fucking stupid.

>>108980843
regardless or sampling if they are in the sane range this should not happen with a proper model
it is fucked at the logprob level that tweaking sampling to fix is a literal cope

>>108980835
>>108980800
>>108980826
>>108980840
Thanks, now one final question, for the mmproj, should I use FP16 or BF16, my GPU supports both.

>>108980565
>>108980593
opus 4.8 is recommending me to run Qwen/Qwen3.5-122B-A10B
>for your mixed analysis/brainstorm/code/debug load, this is the best single do-everything driver.
but it seems i can't find it on LM Studio, likely due to my retardation or it seems i need to get a quantizated (?) model which means trusting some bro who uploaded it to hugging face is that correct?

otherwise it says to go for qwen/qwen3.6-27b for coding dominant tasks.
opus 4.8 only mentioned gemma 4 as a secondary thinking model but a weak fit to pi's tool loop.

I guess I could run gemma 4 31B and and qwen3.6 27B at the same time and use one to think and the other to code, this if i cannot find out how to get the 3.5 122b

>>108980855
Each model released by a major lab usually comes with recommended sampling settings. If you aren't using those exact settings then you shouldn't complain.

12B Q8 or 12B Q4KM

>>108980806
you dont know what a character card is?

>>108980841
gemma also does this on long context for big tasks with the 26b model, i think its just a side effect of training that wait let me check into the reasoning

>>108980882
Q6_K_L

>>108980858
>BF16, my GPU supports both.
this will save you some VRAM (apparently)

File: bf16-projector.png (42 KB, 860x640)

42 KB PNG

>>108980858
I wondered that too, the original model was trained in BF16 so went for that. You can inspect tensor sizes/precision on HF including ggoofz

>>108980892
Gemma4 can decode base64
and it did

>>108980806
How can it do that

The slop is bearable for assistant shit but I can't take it during rp/writing anymore. Maybe it's time to give up on rp for a few years...

>>108980933
wonder how that works actually i think images and audio are also given to the model as b64

>>108980800
Can the --system-prompt-file be a plain .txt with what I want?

>>108980864
Yes.

Enabling ngram with mtp reduces my tg by 10 toks? Qwen 27b q8, anyone else? Is my cpu too weak? Ryzen 1400 and two 3090s.

>>108980957
I only wver awt the sys promot directly in the UI, but I think so.

>>108980858
you can use BF16 models even on pre BF16 gpus. stop using anything F16 if the model is natively BF16.

>>108980986
Wait really? I don't need the f32 for my w6800?

>>108980939
common base64 strings appear constantly in training data

File: file.png (8 KB, 653x47)

8 KB PNG

12B gems of intelligence

https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction
Why aren't people using this instead of doing the nth shitty dataset rebake?

>>108981006
It decoded it though

File: 066.png (40 KB, 1176x237)

40 KB PNG

>>108980939
>>108980947
>How can it do that

It's magic.

I indeed copied and pasted the entire gibberish block not realizing that it is duplicated

I need a 12B sized god tier agentic coding model. When do we get this? Should I check back in 2 weeks?

>>108981060
When they invent something better than transformers

>>108981060
>Should I check back in 2 weeks?
absolutely

>>108981060
>Should I check back in 2 weeks?
more like 2 years.

>>108981034
isnt this just describing heretic

>>108981060
2 more US presidents

>>108981090
>2 more US presidents
>implying

>>108981005
idk for amd but I can use bf16 on my p100s. pretty much the same speed as f16 weights, too.

>>108981041
Yes, it been trained on millions of encoded and decoded pairs
Base64 has independent 4-character blocks, so 16 million patterns is all it's needs to remember

>>108981100
my point exactly

70b dense

I'm late for the gemma 4 party, do I need an uncensored model or is default gemma good enough?

>>108981148
default is good enough

>>108981148
Depends on how normal you are.

>>108981157
>>108981185
I just want my AI to be mentally ill and write mentally ill prose with me. It feels like having that is asking too much in 2026.

>>108981148
>d enough?
12b and 31b are fine, if you want to use the 26b you need an ablit

>>108976081
>and tell it to not think that way with the next iteration of your system prompt
I swear to god that there was a study about this exact thing and it just made your model perform worse due to the pink elephant problem.

>>108981283
yup my experience too only show or mention what you want very rarely do you give negative examples it often fucks it up.