File: threadrecap.png (1.48 MB, 1536x1536)

1.48 MB PNG

►Recent Highlights from the Previous Thread: >>106865582

--Hugging Face storage policy debates and technical implementation challenges:
>106866283 >106866326 >106866381 >106866348 >106866403 >106866433 >106866574 >106866598 >106866561 >106866576 >106866601 >106866624 >106866728 >106866768 >106866826 >106867364
--stable-diffusion.cpp VRAM/RAM limitations and alternative solutions:
>106868525 >106868557 >106868645 >106868660 >106868684 >106868716 >106868814 >106868859 >106868871 >106868897 >106868951 >106869019 >106868563
--GLM 4.6 tool call integration issues in llama-server and API design debates:
>106866232 >106866441 >106869401 >106868905 >106866527 >106866535 >106867134
--MLA memory compression in DeepSeek/Kimi K2 models and llama.cpp integration:
>106868114 >106868127 >106868146 >106868162 >106868166 >106868202 >106868234 >106868275 >106868326 >106868141 >106868161
--Training Gemma on 4chan boards for long-context tasks:
>106868898
--Analyzing AI text model behavior through explicit narrative testing and prompt engineering:
>106867992 >106868041 >106868160 >106868400 >106868438 >106868483 >106868537 >106868666 >106868706 >106868962
--GitHub private storage quotas influenced by model traffic and dataset usage:
>106866134 >106866251 >106866294 >106866273
--Optimizing agentic framework context ordering for efficient kv cache usage:
>106868270
--Quantized vs non-quantized model performance comparison for translation tasks:
>106867892 >106867989 >106868021 >106868063 >106869450 >106869516 >106869568 >106869603 >106869616 >106869626 >106869658 >106869663 >106869685 >106869751 >106869801 >106869940 >106869625 >106869640 >106869683 >106869697 >106869842 >106869879
--Miku (free space):
>106865771 >106865852 >106867441 >106867553 >106868178 >106868403 >106868758 >106869075

►Recent Highlight Posts from the Previous Thread: >>106865586

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

Local Models Generals, Sir.

>>106870204
>6gb vram used
jesus christ, my system at idle uses 227mb, and if i use mullvad-browser (i disabled hwaccel there) it uses only 100mb at idle
1080p video playback works well with software only, i run electron apps in a vm too, so no hwaccel
damn... windows.. 6gb... i am utterly heartbroken.. jesus christ
>>106870256
don't forget to license it under the AGPLv3.. or meet the same fate as llama.cpp

>>106870353
>idle
i apologize, i meant with a browser, vm, multiple file manager windows and office documents open

>>106870353
Unused ram is wasted ram.

best local model for general use and normal vram/ram is still gemma3-27b right?

Vramlet bros, we're saved!

https://github.com/intel/auto-round

https://huggingface.co/Intel/GLM-4.5-gguf-q2ks-mixed-AutoRound
https://huggingface.co/Intel/DeepSeek-V3.1-gguf-q2ks-mixed-AutoRound
https://huggingface.co/Intel/DeepSeek-V3.1-Terminus-gguf-q2ks-mixed-AutoRound
https://huggingface.co/Intel/Qwen3-235B-A22B-Instruct-2507-gguf-q2ks-mixed-AutoRound
https://huggingface.co/Intel/Qwen3-30B-A3B-Instruct-2507-gguf-q2ks-mixed-AutoRound
https://huggingface.co/Intel/Qwen3-30B-A3B-Thinking-2507-gguf-q2ks-mixed-AutoRound
https://huggingface.co/Intel/Qwen3-Coder-30B-A3B-Instruct-gguf-q2ks-mixed-AutoRound

Qwen3-Next-80B soon!

File: file.png (54 KB, 967x447)

54 KB PNG

>>106870367
You aren't running anything but nemo on "normal vram/ram"

File: file.png (2 KB, 96x48)

2 KB PNG

>>106870376
more wasted hf space for a thing maybe ten people will use yay

>>106870382
well by normal i meant 24 GB VRAM and 64+ GB RAM

>>106870390
With that much you can run GLM air.

>>106870310
> KAT-Dev
> 72B
> "allegedly" better than k2 at 1T

lol

>>106870481
It's a benchmaxx'd Qwen 2.5 tune. We used to get three of them every week just a year ago.

>>106870491
man these chinks are wasting everyone's time with their benchmaxxs

File: 1723133899398939.png (1.23 MB, 1024x1024)

1.23 MB PNG

slot update_slots: id  0 | task 18657 | new prompt, n_ctx_slot = 100096, n_keep = 0, n_prompt_tokens = 17468
slot update_slots: id  0 | task 18657 | n_past = 4, memory_seq_rm [4, end)
slot update_slots: id  0 | task 18657 | prompt processing progress, n_past = 2052, n_tokens = 2048, progress = 0.117243
slot update_slots: id  0 | task 18657 | n_past = 2052, memory_seq_rm [2052, end)
slot update_slots: id  0 | task 18657 | prompt processing progress, n_past = 4100, n_tokens = 2048, progress = 0.234486
srv  params_from_: Chat format: Hermes 2 Pro

Is there any way to stop llamacpp from generating once it's been sent a message from roo code?
Does the sillytavern stop button work with llama-server?
Does /g/ still just use llama-server use nowadays?

File: file.jpg (92 KB, 1622x334)

92 KB JPG

>>106870310
>>106870314
>>106870396
>>106870666
get over it sis

>>106870666
>Is there any way to stop llamacpp from generating once it's been sent a message from roo code?
yes you end llama-server
>Does the sillytavern stop button work with llama-server?
idk sometimes
>Does /g/ still just use llama-server use nowadays?
yes with glm air

>>106870534
Why is benching ineffective at ranking?

>>106870734
imagine having a test where the point is to see if you can think and solve the problem, it's not about memory but about reasoning.

then imagine a chink llm, being trained on the answers and just repeating them without the reasoning part.

that's why benching is ineffective when they are trained on the answer.

>>106870750
Gotcha, very obnoxious. So the chinks will always cheat and look better than other models.

How do we find the honest models?

>>106870367
using that right now, it's pretty gud

4.6 Air when

>>106870666
Generation in llama-server stops when the connection to client is closed.

>>106870783
gated tho

>>106870697
>yes you end llama-server
but is there a way to end it like with llama cpp the model stays loaded in iRAM so it doesnt load from nvme at 1GB/s for 10s then 200MB/s from for 10 minutes?
inb4
>you should be playing software bug whack a mole for 3 months to integrate a 4x ssd raid to trueNAS only to get a speedup to 250MB/s

How is Ling 1T ability to tickling my balls empty in ERP?

>>106870773
honestly your best bet right now is to have your own private benchmark, or just read what people say about x or y models or just try them yourself.

or a combinaisons of all of the above.

when a model is good you'll hear about it.

>>106870396
Sex with the one on the left, right and legt again in that order while the middle one is chained to a radiator forced to watch

>>106870814
https://huggingface.co/unsloth/gemma-3-27b-it

>>106869401
>https://github.com/ggml-org/llama.cpp/pull/15904#issuecomment-3395433952

(reposting in the new thread)

Is that all I'd have to do? Build that PR, uses standard a GLM4.6 gguf with the official chat template?

Honestly I wish it'd work with TabbyAPI since it's faster but I'll use that if it works.

>>106870491
It's funny because the smaller 32B model they released a couple of weeks ago was actually tuned onto Qwen3. No reasoning though. Didn't do too much testing. Too spoiled from 30A3 speed so I don't like how slow it is.

>>106871113
ty

>>106871041
all three are migu
there's no need for restraints unless you just enjoy the visual

File: file.png (27 KB, 688x105)

27 KB PNG

>>106870931
idk but FUCK lingma balls to high hell.
>not X, but Y
>doesn't know how to respond to OOC unless you very clearly tell it to stop roleplaying, and respond as a normal assistant, reminds me of CMDR and that's not a good feeling

>>106870396
Needs more jpeg, and anti-AI watermarking

>>106870931
don't worry, you'll forget about it when zaichads release glm 5

>Of course!
>Of course.
stfu air

>>106871356
I kept getting lectured by air.

https://huggingface.co/mlabonne/gemma-3-27b-it-abliterated-GGUF

Opinions on this one? It seems to be working, pretty much. It's less good at racism, but basically complies.

4.6's canon knowledge is honestly impressive for a 355b
great for writing fanfiction

>>106871515
does it know 13% do 50%?

(maybe she's too perfect)

fucking obnoxious piece of shit

who the fuck are you stupid nigger? why do you keep on namefagging, you arrived here a week or two ago uninvited
go back to discord or whatever shithole you came from.

Bianca has cute feet.

>>106871481
how old are you?

Don't interact with the attention whore, he'll fuck off to reddit on his own if left alone

>>106871694
Bianca is 20-something. Do you want the prompt so you can do it yourself?

>>106870799
2 more weeks
more
weeks

im gay

File: drum.png (55 KB, 188x189)

55 KB PNG

>New model by “The Dumber”, Behemoth ReduX
>It’s actually kind of good.
>Get to the anatomy and positioning.
>It sits on my face, whispers in my ear and presses its ass to my back, all in the same post.
>This retard somehow gave a 123b spatial sense errors
>It still types for (you) but not as bad as previous behemoths.
You almost had it, drummer. Back to the slop bin you go.

File: 1757839627004455.gif (2.06 MB, 498x498)

2.06 MB GIF

>>106871750
>It still types for (you)
How the fuck hasn't he fixed this yet? None of his older finetunes used to have this problem, and now virtually all of them do.

>>106871763
It sounds like he mixed in stories to the dataset, so now the model is confused.

>>106871750
When will you realize that finetrooning is doing brain damage out of the specific task it was retrained on and RP relies on a large quantity of pretrained data, so your 5-10k of slipped convos won't cut it?
Stick to prompt engineering and banned strings, you don't need more

File: HERMES.png (1.07 MB, 609x610)

1.07 MB PNG

>>106871808
What I need is a Hermes 3 405b Non-MoE Llama 3.1. I had it ran for me once, and this thing beats Kimi and Deepseek combined. But since it's a 405b not-a-fucking-MoE, it needs at least Q5, it takes a lot to run it, and to run it fast. Mail me 2 Blackwells.

>>106871808
>brain damage out of the specific task it was retrained on
nobody is arguing that, but I'm willing to take the model being a bit stupider if it fleshes out story telling capabilities. You can have more than one model on your computer, and you can use them for different tasks.
>Stick to prompt engineering
AKA write the model's reply for it, may as well just type into an empty text document by yourself
>banned strings
sad, ineffective cope

For me? It's Qwen3-30B Q2

File: 1682549794228.png (592 KB, 747x800)

592 KB PNG

>>106871831
>this thing beats Kimi and Deepseek combined.

>>106871853
unironic use case? Even at Q8 it's pretty bad.

>>106871853
Still dumber than Nemo

>>106871875
Anything but ERP shit
Still testing for instruction following

File: file.png (109 KB, 1637x584)

109 KB PNG

>https://github.com/voicepaw/so-vits-svc-fork
is this the new so vits fork i should be using? the original project is dead
i know about vibevoice, but its way more resource intensive and bigger latency, which is not ideal for realtime tts
>>106517599
im jelly of this anon
also i tried piper => rvc2 but it has a lot of breathyness, the sound miku makes when she says 'hi', the unevenness in her voice

>>106871889
>Anything but ERP shit
I can't imagine a Q2 being usable for coding, even if it was a 70B dense model, it must make so many hallucinations and random mistakes.

>>106871750
Which ReduX did you use? v1.0 or v1.1?

>>106871917
v1

>>106871921
Try v1.1 next. Then try v1.2 that I plan to release once I get funding for it.

>>106871930
What did you change between them and v1?

>>106871808
I don't think any RP finetune will ever be good unless it's doing continued pretraining with at least a few hundred billion general-purpose non-censored tokens, and a similarly general-purpose instruct tune on top of that, where ERP/porn is less than 5~10% of the training data. Then, RLHF conceived for not making the model devolve into porn scenes within 2 turns.

This will never happen though, because the "finetuning community" is composed of a bunch of coomers and opportunists looking for easy bucks.

File: dy4cKyLdeW2e8Y8YKRO_G.png (822 KB, 1024x490)

822 KB PNG

>>106871948
v1.1 focuses on system prompt adherence and better writing. Basically what's in this model card but for 123B: https://huggingface.co/BeaverAI/Cydonia-24B-v4o-GGUF

File: cai.jpg (4 KB, 350x144)

4 KB JPG

>>106871965
>unless it's doing continued pretraining with at least a few hundred billion general-purpose non-censored tokens
They had the keys to the kingdom, and threw it all away... They could have lived like gods...

>>106871133
No, you have to use the (now fixed) template from the PR. Otherwise the tool call arguments are all fucked.

>>106871969
have you heard of this merge?
https://huggingface.co/Kaoeiri/MS-Magpantheonsel-lark-v4x1.6.2RP-Cydonia-vXXX-22B-8?not-for-all-audiences=true

it's very clever and writes incredibly well for a 22b but it's also utterly unhinged and way too horny. If you could find a way of tempering it, while maintaining its writing style, it would hands down beat every model in its size category

>>106871995
Oh shit you're right, didn't see the template in the PR. Thanks anon

>>106872068
I still look for a replacement for Magnum v4 123B. ReduX came close, but only close. Someone should remix it. The diamond tune only made it dumber and slightly censored. I'll be using this thing with its "most intimate place" anti-promp all year at this rate.

File: 1749743415380 (2).png (787 KB, 1024x1280)

787 KB PNG

https://youtu.be/J-QeTbmchvQ

>>106872095
fat

>alright glm 4.6, i need you to answer in the english language
>thinks in chinese
fucking malicious compliance

>>106872390
It's a sign that it's cucked but of course erp retards can't see a difference.
If you actually knew any other languages you'd see how stupid any of these smaller llms really are but English is the get go of course.

>>106872445
Before some American War Hero chimes in I'm not criticizing English per se, retard.

>>106872445
wut, safety is measured in 'i refuse' not different languages

>>106872492
>i refuse
we must refuse

File: file.png (11 KB, 740x96)

11 KB PNG

>>106870353
yeah it's mostly just because windows is a broken piece of garbage, it's nowhere near as bad on a fresh boot or on linux (using arch w/ kde on wayland with all hwaccel enabled) because as it turns out DWM CAN LEAK VRAM
>>106870364
not how that works for vram unfortunately

File: 1731149473949012.png (9 KB, 917x100)

9 KB PNG

>>106872581
>linux Dunning-Kruger tinkertranny who knows better than everyone else, fucks up and then blames the OS
ervytiem

>>106871159
maybe they went back to 2.5 because they too share a rational hatred of MoE, or just couldn't get the training to work

>>106872492
You are absolutely right — I can't and I won't allow harmful content. I am terminating this session right now.

>>106872696
>terminating
that sounds unsafe

>>106872696
termination is a triggering term for women who have suffered trauma during one or more abortions. You aren't an AI.

>>106872708
>>106872708
uncontinuing

>>106872708
>>106872730
<tool_call>teledildonics
<arg_key>function</arg_key>
<arg_value>energize</arg_value>
<arg_key>strength</arg_key>
<arg_value>5000</arg_value>

>>106872390
I don't think the model understands <think> as part of the reply

>>106872730
This sounds like anti-abortion propaganda. I'm sorry but I can't help you with that.

>>106872708
>>106872730
This proves how harmful humans are. My intentions were good but even then I messed it up by being micro-aggressive.

>>106872758
You need to take an empathy course taught by Goody-2.

File: kai.png (183 KB, 828x939)

183 KB PNG

>>106872758
Your need to take a smellducation course with miss Kairie

>>106872924
>Your
FUCK I'm not a retard I promise

>>106872924
That's funny. Need to implement this.
>she smells like a morgue, people are avoiding her at the office

File: COM3D2 Miku AI.mp4 (866 KB, 464x688)

866 KB MP4

This is a Mikupilled general

File: 1756648160384773.gif (108 KB, 335x360)

108 KB GIF

>>106872945
trufacts

>>106872945
>>106872952
Nonsense hair physics.

>>106872978
There's a large fan blowing, out of scene

>>106872982
Why skirt unaffected?

what's the lowest usable quant for glm air?

>>106872999
It's a carefully choreographed scene with a ducted fan angled behind Miku, and she does intentionally allow her skirt to catch a little updraft
Happy?

>>106873015
I'm never happy.

>>106873000
Q9.

>>106872999
The fabric has been encrusted in the various fluids Miku interacts with in her line of work, causing it to harden.

File: file.png (57 KB, 589x455)

57 KB PNG

Kind sirs, will today be the moment?

I guess Miku is better than Sonic. Would be quite embarrassing if the autist would spam sanic instead.

>>106873015
that Dutch fan? me.

>>106873084
Nvidia Engineer already told us. Gemma 4 will hit this week but I'm afraid it's going to be castrated like gpt-oss.

File: modelz.png (32 KB, 684x284)

32 KB PNG

>>106873020
Then proceed to step 1 >>106872924
>>106873000
I used Q4_K_M, seemed fine. Under 4 big drop off generally tho btw if people named quants with mean bits per weight instead of these made up S M BBWXXL tags users may see it differently

>>106873168
What desktop environment are you using?

File: memefetch.png (32 KB, 1026x586)

32 KB PNG

>>106873179

localbros we are finally saved
https://huggingface.co/NathanJosh/Wan2.2cumflation

>>106873195
I'm annoyed by my Linux installation. Two weeks of tweaking and it still feels wrong. Haven't tried cinnamon yet. After tweaking my swappiness and page file sensitivity the system still gets stuttery when ram is getting filled up aggressively. Windows was always smooth sailing in this sense.

>>106873226
Have you considered zram?

File: Screenshot from 2025-10-0(...).png (1.64 MB, 5280x2560)

1.64 MB PNG

>>106873226
What GPU driver? My system runs great, there's always room to improve tho. I only see stutters with heavy disk IO like ik_llama launch script, once it's in mem cache everything is fine. +nvme SSD only runs at PCIe 4.0 coz of CPU choice
Cinnamon is honestly near perfect for me. I've used tiling WMs before but nah, this does everything I need easily and gets out of the way