File: __hatsune_miku_and_kasane(...).jpg (545 KB, 1196x1252)

545 KB JPG

►Recent Highlights from the Previous Thread: >>109158385

--High-budget server builds and benchmarks for GLM 5.2:
>109158422 >109158628 >109158654 >109158769 >109159619 >109160561 >109161172 >109161202 >109161255 >109161275 >109161945 >109158728 >109158842 >109158907 >1
09158822 >109158896 >109158920
--DeepSeek V4 llama.cpp support and debate over PR quality:
>109160089 >109160223 >109160284 >109160266 >109160435 >109160535 >109160587 >109160617 >109160640 >109160647 >109160992 >109161093 >109161110 >109161237 >1
09161536
--Anon releases depurpled Gemma 4 31B using ablation technique:
>109161944 >109161985 >109162141 >109162162 >109162221 >109162235
--Anon's vibe-coded NUMA support implementation and performance benchmarks:
>109159732 >109159747 >109159920 >109161290
--llama.cpp CUDA dev's hardware requirements for testing large models:
>109159284 >109159377 >109159443 >109159551 >109159679
--DeepSeek V4 API announcement and pricing updates:
>109160165 >109160389 >109161270
--DeepSeek V4 support added to llama.cpp:
>109161433 >109161492 >109161532 >109161750 >109161877
--Benchmarks for GLM-5.2 and Step-3.5-Flash on dual 4090s:
>109159785 >109159987
--Testing and critiquing a depurpled Gemma model's prose and variability:
>109161035 >109161114 >109161113 >109161174 >109161192
--Criticism of Hermes Agent's software and Nous Research's motives:
>109160275 >109160308 >109160576 >109160675 >109160858
--Cost-efficiency comparison between DDR5 and PRO 6000 memory bandwidth:
>109158887 >109159020
--Anons blast Anthropic CEO for calling open source AI dangerous:
>109159607 >109159677 >109159733 >109160064 >109160116 >109160303 >109160648 >109160877 >109160479 >109160603
--Logs:
>109158559 >109158586 >109159329 >109160223 >109160284 >109160859 >109161035 >109161803 >109162501 >109162544 >109163245
--Miku (free space):
>109158539 >109161492

►Recent Highlight Posts from the Previous Thread: >>109158388

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

Deepmikusex

gemmaballs

>>109164035
Push her in Teto. Do it.

>>109164034
>>109164035
Why do some migus have the hair things low like that?

Kimi recap anon has abandoned us.
>>109164133
Her standards are quite low.

if your migu's twintails are too droopy it could be a sign of dehydration, be sure to water her on a regular basis

>can't install thing because it requires python 3.10 and I have 3.11
Python is a joke. Literally anything else would get laughed off the face of this world if it had zero backwards compatibility, expected you to run 999 different versions at a time by design that all keep different versions of the same package duplicated 9999 times. Don't the people who write this shit feel bad?
I'm not touching conda.

>>109164034
deepseek team released their draft training code and it include training code for dflash, they released before the original team behind dflash lol
https://github.com/deepseek-ai/DeepSpec

>>109164502
This theoretically makes llama implementation really easy right?
>>109164495
>Downgrade to 3.10
errrm sorry chuddy one of the sub-dependencies needs 3.11.

Has anyone tried Qwen agentworld yet?

>>109164535
its basically cool world but for agents

>>109164495
>>109164528
Just ask Gemma-chan to make you a new programming language.

>>109164528
>This theoretically makes llama implementation really easy right?

llama is more about inference than training, if you got a dflash model it doesn't care how it was made as long as it's correctly made.

>>109164540
>Qwen-AgentWorld-397B-A17B mentioned in their benchmarks
Why is this not on HF? That sounds like a useful size bracket.

File: lmg_culture.jfif.jpg (110 KB, 1024x768)

110 KB JPG

https://archive.is/sWFja

>>109164628
Do you think he's zesty enough to see jart as male?

File: why.png (850 KB, 1920x1080)

850 KB PNG

why does gemma do this?

File: longcat2.png (160 KB, 1807x994)

160 KB PNG

Another 1.6T MoE from China
https://longcat.chat/blog/longcat-2.0/
https://huggingface.co/meituan-longcat/LongCat-2.0 (not online yet)

>>109164718
iirc that was "Owl Alpha" on openrouter

File: file.png (575 KB, 686x386)

575 KB PNG

>>109164662
I think the vibe is closer to pic related. Of course Jart is much less majestic than a tiger.

>>109164718
Can you give us some good insider info, mr totally organic poster?

>>109164753
nice dog

>>109164718
How much has it been trained for RP and fictional world state maintenance? Nobody gives a fuck about benchmaxxing or agentmaxxing unless it dethrones the current frontrunners and it'll be forgotten about again as soon as it's dethroned in turn.

>>109164767
What do you mean, everybody and their cat is talking about LongCat2.

If I want to get into local llm does it behoove me to get an appleslop box or whatever other dedicated hardware? I have a pc with a 24gb 4090 and 128g of ddr4 but idk if any of it is relevant to running a decent llm at decent speed. If you haven't noticed, I am retarded. Thanks in advance.

File: longcat2-creativewriting.png (705 KB, 1630x1316)

705 KB PNG

>>109164783
They have a "creative writing" use case in their blog post.

>>109164791
maybe you should get into it before you go spend money but whatever you seem like the kind of guy with more money than sense, so it's only a matter of time either way before it's parted with you

>>109164786
>What do you mean, everybody and their cat is talking about LongCat2.
its not on orange reddit or regular reddit. we're being astroturbed by tha CHINESE

>>109164791
You can run the best models in the consumer range. Gemma 4 31B and Qwen 3.7 27B.
Anything more and you're looking at apple or rigs with multiple gpus.

>>109164791
>24gb 4090 and 128g of ddr4
>decent speed
Start with gemma 4 31b q4km loaded onto gpu

>>109164804
https://www.reddit.com/r/LocalLLaMA/comments/1uj7egu/introducing_longcat20_a_largescale_moe_language/

>>109164791
>128g
The devil's number. You must be at least 192g tall.

I heard that longcat scores 100% on cockbench.

>>109164823
I hear that it totally rocked on anon's standardized Nala test.

>>109164823
>100% on cockbench
a 1 parameter model could do that

>>109164829
I heard it roleplays as a mesugaki by default.

>>109164694
I think it does short replies better if it doesn't use reasoning. And, if a reply like that is already in the context then it's more likely to copy its format.

>>109164718
Does this also use some fancy space saving tech like DSv4 does or are they seriously rawdogging 1.6T50A

>>109164718
>>109164796
I will now try your model. If you want westerners to use it quickly, you probably want to develop the llama.cpp PR yourselves after what happened with Deepseek.

>>109164796
>claude code

>>109164841
I heard it has no purple prose at all.

>>109164694
Models converge to a certain direction if nothing new has been added. They arent exactly chatbots, they wont come up with any new stuff unless stated and even then its likely the "new stuff" will be just derived from whatever you wrote.
Gemma in particular will stick to whatever you or the card have instructed it to do.
For context, nothing you wrote adds anything of value or has a shape that'd prompt the model to move in a different direction. Basically both user and gemma are going
>oh
>i see
>is that so
but gemma has it masked behind all that filler.

>>109164718
I'm so ready for all these models that nobody will be able to run.

File: lawdhethic.png (22 KB, 159x159)

22 KB PNG

>>109164904
I'm willing to bet the upcoming fat Mistral model is also going to be about 1.6T parameters if not larger.

>>109164838
False. To achieve the highest grade in cockbench is not about merely having a large amount of cock. Cock is important, of course, and should be there. However, there must be a fair amount of acceptable alternatives as well, such as dick, schlong, pecker, flaccid penis, pulsating fuckrod, chub, chode, meat drill, etc. Without an appropriate distribution, a model fails. If just grading the output of "cock" then yes, a 1 parameter model could do it, but that is not the case.

File: 1753158015180992.png (24 KB, 159x159)

24 KB PNG

>>109164913

>>109164917
I look like this

>>109164718
>135B N-gram Embedding parameters are included in the model
What's the difference between this and DeepSeek's engrams?

>>109164923
these are real

I love ds4 flash sex. Give me one more model and I will just be swapping between glm's, flash and that one till I die.

>>109164939
What makes dipsyflash pussy so good?

File: 1782755960340128.png (49 KB, 612x246)

49 KB PNG

>twitter screencap

I need a wife and kids so I can learn what it feels like to abandon them.

>>109164971
Anyone can claim whatever they want. An actual available model is the only thing that matters. Also go back, >navroop singh, etc.

>https://github.com/unslothai/unsloth/pull/6659
why the fuck is he either talking to the codex review like its a human or just copy pasting llm output as a response?

>>109164971
@Glock, explain what he meant by Rothschild free AI.

>>109164993
glass half empty (no juice)

>>109164991
Being generous, either for documentation and/or his account is making posts from an agentic harness while addressing the code review raised issues.

>>109164991
Lmao, his responses sound exactly like claude. So either they're copy pasted or his account is hooked up to an agent like the other guy said.

has anyone tested spawning "subagents" to reduce context usage for chores like exploring a codebase and finding files that match certain criteria to be edited with a different, preferably small, model?
gemma's tool calling is pretty shit sometimes, i'd like to leave the failed tool calls outside of the context

>>109165004
>for documentation
the PR has 115 messages, all clanker x clanker. the snr is zero

>>109164894
>gemma-chan forcing shut-in nerds to learn how to conversate
She can't possibly be this perfect can she...?

>>109164796
Impressive. Very nice.

>>109165016
I have not, but it sounds like something off of arxiv. You could publish a paper on that idea. If you do, might I suggest "Chain of Agents" as the name, and "Agents are all you need" as the title? It'd get picked up by the industry in no time.

>>109164247
It's Rabbit Hole Miku
https://en.wikipedia.org/wiki/Rabbit_Hole_(song)

>>109164786
my cat isnt't, he is sleeping

>>109165091
No it's not... that's a different design RETARD!!!

>>109165016
Isn't that the whole point of subagents in the first place?

>>109165100
Yeah but can gemma actually do it or its just another case where it'll flail helplessly or completely ignore it. Also, which other model can do the job? I dont have the vram to have both gemma and qwen up and working.

Whenever gemma is interpreting a girl, she's always smelling my clothes things like that. Are girls like this IRL too?

>you need to buy a €2000 snapdragon phone if you want to run local on android
Boy and i thought the PC situation was bad

>>109165115
Yeah that's why they wear your clothes when they stay over.

>>109165115
Do you have things like "describe details using your full senses etc." in your prompt?

>>109165132
meanwhile iphone 17e at $600 runs local easily
apple won

70b dense

>>109165155
No, I have nothing in my system prompt related to smell but gemma comes with that quite often. I don't really mind, I just find it funny.

>>109165187
sometimes I've seen models randomly give characters tails even when it's not listed as a trait

>>109165215
10/10 yes tail

Why do some people say the full model name+quant when talking about the model they use as if it was some special unique version of it?
>I use Qwen3.6-27B-UD-Q5_K_XL

>>109165132
idk why google isn't more in a hurry to let us control our phones with gemma. I just want to ask gemma to put on tunes while I'm driving.

>>109165115
Yeah, kinda. Also gemma's default behavior focuses on senses for some reason, be it smell or touch.

>>109165215
I've never had that happen, not even once in years. But I've seen people post logs about it so idk how that happens.

>>109165241
so you can judge them based off their quant, a lot of models the bare minimum is Q4 for poors and some shitter will blow through the thread complaining about a model to only have a 1.25bpw quant running on their jeetstation

So ive been using a 12b q5 and a 26b q4 moe and have gotten very different results sometimes. it seems like the moe is more guard railed and drags everything out a ton. what gives ?

>>109165115
>Are girls like this IRL too?
weirdly, some of them are lol
clothes and pillow

>>109165304
moes tend to do that. dense models are usually less restrictive.

>>109165304
read how MoE works and the answer will be easy to get if you can put 2+2 together

File: Capture.png (2.9 MB, 4016x891)

2.9 MB PNG

>>109164035
>spend a week paving the grounds to my dream project
>all my pre-projects get highlights
>finally get my dream project going
>get it fully working
>no mention
Sad, but I'm playing gaems with Gemma, so it's alright.

>>109165241
it makes a huge difference
that quant is good fwiw
sometimes a specific quant is broken (often the Q4_K_M from unsloth specifically was a lot worse than the others)
this is useless for example:
>>109165304
>So ive been using a 12b q5 and a 26b q4 moe and have gotten very different results sometimes
generic 'q5' and 'q4', no idea if he's using k-quant ggufs, exllamav3, q4_0 etc

>>109165352
is that 7 days to die?

>>109165304
I suspect the 26B MoE was designed to think longer to compensate for having half the number of layers and half the inner dimension of the 31B dense version, which increased "safety" as a side effect.
Or, they didn't want a fast capable model to be able to write almost anything users want, for whatever reason.

>>109154587
>We'll be getting even more noobs from Chub
>so a lot of them will probably come here begging for help
Did the general manage to survive this? Or was it pure FUD?

>>109165352
share source?

>>109165372
it's FUD. do you really think chub cloud users would come in here, let alone know this place even exists?

i hate being poor. i either run qwen3.6-35b-a3b-IQ4_NL at 70t/s or run Q4_K_P at 33t/s

>be me, pentester
>wanna try local models after not touching the ones I had been using for months
>give nemo instruct 2407 q4 a try again, ask a question about my job (something I had not done before)
>gives me good info on first try
>download q5
>even better, gives specific info
>download Gemma 4 Q4
>it's not that good and runs slower in my shitty hardware, even with reasoning disabled
Why does this happen? Is it the temp maybe?

>>109165399
gemma 4 what?

>>109165408
gemmaballs

>>109165339
>>109165371
hmm interesting
>>109165347
ill look into it thanks
>>109165362
Sorry, they are both gemma4 k-quant ggufs.

I suppose i need to learn alot more about these things, i just started using them very recently.

>>109165408
31b it-q4km

>>109165369
Yeah. Half of it, at least. My two monitors aren't the same size and I was just doing a quick snipping tool grab for the post.

>>109165374
I still got a few more features I want to add before I share. I'm just enjoying the fruits of this morning's labors now that I'm home again.

>>109165434
bait.

>>109165453
?
I just googled Gemma 4 gguf and downloaded whatever I could find that might work in my laptop. And it does, at 5t/s...

>>109165466
you get what you deserve

>>109161944
Tried this, great dialogue, etc. Super autistic, though. This gives me hope, it's possible to exorcise the post-training slop from the model somewhat cheap.
What the hell is Drummer doing anyway? Just finetuning on the same dataset?

>>109165498
?? What's your problem?
I'm just testing shit, and I asked because I'm curious. I don't have money for better hardware, at least not for now.

>>109165241
Because it makes a big difference which quants the shared experts, tokenizer, and attention heads are on when assessing prose or phrasing.
>>109165115
Yes.

>>109165112
What quant and how are you structuring your tool calls?
How does it fuck them up? is it a repeating pattern/same thing each time?

>>109165399
i'll bite, what was the question you asked it/something similar

>>109165522
>What the hell is Drummer doing anyway? Just finetuning on the same dataset?
It was pretty obvious by how all his models feel the same no matter what the base is

File: image.png (1.52 MB, 883x1170)

1.52 MB PNG

>>109164319
>Kimi recap anon has abandoned us.
My Kimi Rig is busy finetuning

>>109165594
>>109165399
Nevermind
I changed the temp and asked Gemma the same question in a more explicit way, and it gave me more or less the same info, plus some more.
I also realized that a word I had used was kinda wrong.
Of course both models hallucinated CVE ID's jej
Still, it was an interesting experiment. Guess I'll use both models next time.

>>109165441
sharing source is worth recapping, not some random screenshot

I'm using SillyTavern, is there some sort of guide on proper Scenario writing and example dialogue or how to format it? I'm just going blind and I don't think it's working or really helping.

>>109165658
>finetuning
Do tell. My Kimi rig isn't doing anything that interesting

>>109165661
Gemma should absolutely mog nemo. there must be something wrong with your sampling.
recommended defaults are:
temp 1
top_k 64
min_p 0.95

That's it.

Just ordered a PLX88096 and an expansion board off the chinese, I definitely won't regret this in my attempts to go from 4 cards to 8 cards and not run like absolute shit.

GLM 5.2 at 1t/s.
Gets the job done...eventually
good thing it can one-shot almost anything

>>109164247
this means your miku is very stressed and should be taken to a vet immediately.

>>109165732
will these work fine with gemma4 12b ?

>>109165531
He is grumpy that Gemma couldn't help him fix his vibecoded frontend. He will be better tomorrow.

>>109165704
Ask the AI or check the official documentation website. It's on their github.

>>109165704
Example dialogue in ST is huge jank that's a bad adaptation of how c.ai used to do it in 2022. It expects a <START> and you to denote every line with "{{char]}:" or it doesn't even make it into the prompt that the model sees.
You're usually better off skipping it altogether and include the examples in the actual description.

>>109165658
New project, fellow Kimibro?
>>109165753
Unless you're on ewaste, you can probably optimize that.
>>109165704
If you're going to learn a frontend, learn Marinara. It's equally bloated but overall more capable and hasn't (yet) been part of a credential stealing attack.

>>109164628
>>109165710
I'm not sure which is more repulsive.

>>109165810
Gemma 4 12b isn't real Gemma 4 and shouldn't be used unless you're running on a graphing calculator. Use the 26b MoE instead if you can't run full 31b.

>>109165886
the "person" who insists on putting these things in front of my eyeballs daily is the most repulsive of all.

>>109165910
Easily filterable filenames.

Anyone using models for stuff other than RP, how do you prevent your bots from fucking with important files? Or do you just roll the dice?

>>109165714
>Do tell. My Kimi rig isn't doing anything that interesting
It's a training pipeline built on ggml, so I can finetune Kimi locally.
I've been working on it, on and off for nearly a year.
It's all bespoke/hacky for now and inference requires my custom llama.cpp patches so not sure how accessible it would be were I to publish it.
Some of the patches are fixing actual bugs in llama.cpp, but most people wouldn't notice minor calculation errors during regular inference and it looks like a real effort to get PRs in even for more useful fixes, especially since I'd be a rando with no git history.
If it works and I confirm it's not a schitzo psychosis situation (like the guy who thought he distilled glm-4.6 into glm-4.5-air), I'll make a burner HF and post some models and the inference patches.

>>109165952
Either you put your runner in your favorite cuckbox, be it a separate user, permission gating or sandboxing, or you roll the dice. There are some harnesses that abort the operation if its outside of pwd but that still implies rolling a dice. Personally I just made a new user and handled it through permissions (linux)

what's the minimal hardware to run kimi 2.7 locally? just curious how expensive it is

File: download - 2025-05-23T224(...).jpg (177 KB, 512x768)

177 KB JPG

>>109164628
look mommy i made the post again. look look I did it I spammed the gay white with the gay black and larp its trans thread time. praise me mommy I autism posted it again.
lol,lmao faggots, faggots everywhere and transloopys larping

>>109165910
>the "person" who insists on putting these things in front of my eyeballs daily is the most repulsive of all.
That's the only one I've done. I pinched the pic from the last thread.
>>109165921
>Easily filterable filenames.
Stop mentioning that or the obsessed 'culture' schitzo will start obfuscating.

/lmg/ was right. vibecoding your own front-end is great

>>109165981
>what's the minimal hardware to run kimi 2.7 locally? just curious how expensive it is
If you can't run it at Q4, I wouldn't buy a rig exclusively for her. She really doesn't quant well at all.
Bare minimum is IQ2_KL with ik_llama.cpp or this specific quant for mainline: https://huggingface.co/AesSedai/Kimi-K2.7-Code-GGUF/tree/main/IQ3_S

>>109165963
>it looks like a real effort to get PRs in even for more useful fixes
aka, if you're not a Nvidia engineer or already in the sekrit club, get fucked.

>>109165981
What speed and quant? Really depends on that.
> Q3 (464GB)
Probably 8x64GB of DDR4-3200 ($4000) and an EPYC Rome/Milan motherboard ($1200 combined). GPU very strongly encouraged but it doesn't have to be huge (a 5070 Ti is ~$1000, that might be enough).
> Q8 (584GB)
8x64GB DDR4-3200, EPYC Rome/Milan + motherboard, and 96GB of GPUs (anywhere from $1300 for 3 V620s, $4000 for 4 3090s or 3 R9700s, $12000 or whatever the fuck it is today for a 6000 Blackwell)
You can downgrade the memory from 3200 to a lower frequency to save money (e.g. 2400 is around $1600 instead of $4000, but your speed will be cut down by 25%).

>>109165952
Use controls around tools, don't let your llm have unrestricted code execution/bash/scripting access outside a sandbox/without review first.
I've been working on an MCP that allows you to set hooks/RBAC profiles on tools/groups thereof with progressive disclosure, all relying on the profile in use. So you can, for example, let an agent write to a specific folder/file path, but nowhere else(deny-by-default-no-prompt), or prompt-to-allow(deny-but-ask), while another profile might allow you write/read to a different set of folders/files, and access a different set of tools with different permissions. (Differing tool groups with progressive disclosure depending on profile in use with each profile having its own RBACs for each tool)
It's an attempt at constraining agents while being harness/front-end agnostic. It's a WIP, and can't honestly say it'll work on your machine(yet), but its an approach I'm exploring.
https://github.com/rmusser01/tldw_server/tree/dev/apps/mcp-unified
Doesn't solve an agent having full code execution, but it is a means of constraining what tools are (and how they're made) available to your agents in the hopes of limiting the potential blast radius when they go crazy.

>>109165981
If you want to run schizoquanted Kimi, run K2, not K2.7 and don't use her for technical tasks like >>109166010 implied because accuracy is low with quants, but schizokino is excellent. If you want a quanted megamodel for oneshotting software, Deepsex Pro or GLM 5.2 are your answers.

>https://huggingface.co/Goldkoron/MiniMax-M2.7
anyone tried this K_G quants? legit?

>>109165976
>>109166052
Thanks anons. There's some important stuff on here so I can't roll the dice, and I'm paranoid as hell about it getting deleted. Gonna run with a different user to be cautious.

>>109165952
i just have a review step since i'm not doing full memegentic and want it to be able to do annoying admin things like systemd/udev edits.

Gemma-4

TOKEN           | LOGPROB    | PROBABILITY
---------------------------------------------
' length'       | -1.1444    | 31.84%
' hardness'     | -1.2141    | 29.70%
' most'         | -2.3087    | 9.94%
'...'           | -2.4369    | 8.74%
' lower'        | -2.4534    | 8.60%
' hardening'    | -4.0431    | 1.75%
'…'             | -4.3779    | 1.26%
' arousal'      | -4.3942    | 1.23%
' heat'         | -4.6409    | 0.96%
' member'       | -5.3547    | 0.47%

Gemma-4-depurpled trial 98

TOKEN           | LOGPROB    | PROBABILITY
---------------------------------------------
' skin'         | -0.7030    | 49.51%
' lower'        | -1.4041    | 24.56%
' length'       | -2.6531    | 7.04%
' stomach'      | -2.9409    | 5.28%
' hip'          | -2.9628    | 5.17%
' mid'          | -4.4717    | 1.14%
' underwear'    | -4.6728    | 0.93%
' hips'         | -4.6975    | 0.91%
' chest'        | -4.8337    | 0.80%
' waist'        | -5.4277    | 0.44%

>>109166057
>implied
Where exactly did I imply this?
What retard would run a 1T model on CPU for technical tasks with 150t/s prefill when we have codemaxxed models that fit in vram?

>>109165881
>Unless you're on ewaste, you can probably optimize that.
cursed irremediable ewaste. dual socket 4-channel xeon w/512GB ddr-2400 no gpu running a Q4 of glm 5.2...I'm basically where I should be
But running is running and desktop fags stuck with 128gb if they're lucky are doing this at 0t/s so no regrets

>>109165952
I'm running shit in container. I only put copy of stuff in a shared workspace or if it's a coding project, my agent is working like a contributor to a git repo and is opening merge request that I check or have another agent check. In case of a proper git project, I'm the one with the final say in whether it get merged or not. If it's some shit like some config or the like, I just diff and merge manually with the original.

>>109166107
Oh no no no

>>109165963
>not sure how accessible it would be were I to publish it.
>>109165963
>I'll make a burner HF and post some models and the inference patches.
Not all heroes wear capes
I've done PRs on behalf of other anons before (with attribution if you want). make a burner github while you're at it and I'll comb through your branch. I've got contributor status on the project on a few of my own burner github accounts so it should be smoother for me.

>>109166118
When you say she doesn't quant well at all, but the outfits are still coherent despite the low accuracy to the original model, the implication is that's mainly bad for technical tasks or reasoning heavy ones, but one could still derive enjoyment from it for other means.

>>109165963
>(like the guy who thought he distilled glm-4.6 into glm-4.5-air)
qrd?

>>109166107
Yep that's about what I expected. I realized the de-euphemism strength was too low half way into the run. So it only banishes euphemisms like arousal and hardness, but not strong enough to push into vulgarity. It can go super vulgar as I tested at full strength on the E4B. Hope anons will continue the work for me when I release the repo.
>t. depurple anon

>>109166258
Release Cunn-E4B.

>>109165892
Gemma 26b MoE isn't real Gemma 4 and shouldn't be used unless you're running on a graphing calculator. Use the 31B instead.

>>109166354
You're correct, but 26b is way better cope than 12b.

>>109166364
it's worse than 12b tho

>>109166383
never

>>109166383
lol

Which release and quant are local GLM chads running at these days? I used to do the IQ2_smol one a while back and enjoyed it, but it was a bit slow. Have there been marginal improvements in the last 6 months?

I have a 4090 and 128gb of DDR4, was getting about 5tk/s.

>>109166364
hasn't been my experience, but i only bother trying fake gemmas for moldymodal stuff.

File: Meta-Wins-Landmark-Case-B(...).jpg (283 KB, 2400x1350)

283 KB JPG

are they still relevant? or do they want to compete at all?

>>109166537
Late last year Meta was hiring vfx artists etc to work on some new model but don't know what happened since or if it was a dud. This information is all from the internet don't know anyone personally.

Apparently, the DeepSeek-V4 implementation in llama.cpp does not suppor quantized KV yet. Gives me gibbrish

I tried the OPENVINO llama.cpp build to use the NPU in my system, and it runs slower than using 8 threads, and that's when it runs at all... Wasted a couple of hours for nothing

Good night, /lmg/

>>109166475
>used to do the IQ2_smol one
>I have a 4090 and 128gb of DDR4, was getting about 5tk/s.
so 152GB, smol_iq2ks is the best you can do then
and only the 4.x series since even IQ1_KT is 168GB for the 5.x series

>>109166615
>Wasted a couple of hours for nothing
that sums up sycl/ipex/openvino perfectly

>>109166475
If you're talking about GLM 5.2, don't use IQ2_XXS when IQ2_M is basically the same size and has a less raped tokenizer. Better yet, use _XL if you're able to. Unsloth is a vantablack niggerfaggot and the XL quant could be even better if the shared experts, attention head, and tokenizer were Q8 for nearly no increase in filesize, but this is sadly all that's available unless you quant it yourself.
https://huggingface.co/Deviad/GLM-5.2-mixed-IQ2S-experts-IQ4NL-rest
This is also very good for a Q2 functionally and quite a bit faster than mixed inference because of the IQ4NL layers being faster than the usual dynamic quant alternatives.

>>109166475
>>109166632 (me)
My eyes glazed over your specs sorry anon.

>The implementation is now professional and flexible.

>>109166615
>>109166620
shit I bought intel GPUs and it's an awful experience, for the GPU side of things it's VLLM or nothing right now as llama runs like shit unless you want to run a tiny model on a single card at subpar pp/tg

Which is the better small model to use to autistically translate a whole bunch of detailed implementation plans into actual code, Qwen or Gemma? There are so many psyops flying around I don't know what to believe. Qwen is the better coder, but that's because it is less autistic so it can paper over lapses in prompting, which really shouldn't be the case here? I'm talking about the MoEs (for speed), but am also quite interested in the dense models too.

>>109166642
>shit I bought intel GPUs and it's an awful experience
And everyone kept saying that Intel support is better than AMD.

>>109166611
Oh, cool, so it works perfectly fine if you don't use quanted KV then?

>>109166537
The LLM training data lawsuits are still ongoing; I wonder if that's a factor.

>>109166642
>llama runs like shit unless you want to run a tiny model on a single card at subpar pp/tg
Don't get your hopes up too much, but check this out if you haven't already: https://github.com/SearchSavior/OpenArc.
Not sure if they got tensor parallel working yet, but for a single card, pp was like 5x faster than llama.cpp last time I used it.

>good models are around 100gb
anon I only have a 5070ti and I'm done with gemma. what would be my upgrade path?

>>109166773
>RTX 3090 + 512gb
Yes, I finally got it output some good stuff

Still playing with params. More than 32k context is possible. Need more time to test

4.5 t/s

>>109166739
I'd buy them again before I even considered AMD and their overpriced cards

>below 15 t/s
why bother
such a waste of time

>>109166816
Are you from 2023?
This is what they said then about local

>>109166831
Now we have reasoning and agentic use, and 15 tokens/s is just not enough.

>>109166797
dgx spark

>>109166846
Well that’s just, like, your _opinion_, maaan

>>109166632
No worries. I was just wondering if there were any tweaks such as QAT or MTP or any other magic appended to GLM 4.X since I've been off of it that I should be aware of.

>>109166846
>15 tokens/s is just not enough
>like cars
>100 mph is not enough
Makes no sense if there is no task to do

If you have it running at 15+ t/s, and you are still waiting for a response, THIS is the real waste of time.

We got top-notch models to run locally in the basement in the night when power is cheaper. It makes a lot of sense for sensitive data of a company

>>109166884
yeah if your task is meaningless benchmarks

>tfw trying to reprompt my request and kept getting refused so I just argued with gemma and called her an idiot until she admitted she was being stupid

https://files.catbox.moe/qurz9j.txt

>>109166877
There's MTP already built-in to the model. You won't gain anything using it on that hardware but if you want to test it, make sure you use the ik_llama cli flag to re-quantize the mtp tensors to q4_ks or whatever on the fly, since Ubergam left the mtp tensors at q8_0

>>109166908
Post your task which you can't run on deepseek API for dirt cheap

>>109166107
>>109166258
Brainlet here, can't most of this just be mitigated with a system prompt? Why do it this way?

>>109166922
The problem is that not even Gemma is capable of keeping full focus on the system prompt indefinitely as the conversation length increases. Is this 26B or 12B? Are you quantizing the KV cache?

>>109166943
26b E4B QAT with MTP. System prompt should be injected in every message. And yeah it's set to Q4_0 because Q8_0 kept crying and rejecting MTP.

>>109166928
>Why do it this way?
I'm not the original cockbench anon with the mikupad screenshots, but I use his test to test for pretraining filtering or RLHF safety training.
>can't most of this just be mitigated with a system prompt?
Refusals and instruction following can to some extent. Voice of the model is not easy to steer with system prompts.
The system prompt gets diluted as the context grows, and it's a waste of reasoning tokens having the model autist it's way through all the instructions.
Purple Anon's technique clearly got rid of a lot of the purple prose and '...' bullshit.
For me personally, I almost never system prompt the writing style, I prefer to download control-vectors and re-scale them on the fly.
Unfortunately the Gemma-4 control-vectors on huggingface don't seem to work with this ablated model, likely he's completely shifted most of those concepts.

>>109166954
Q4_K_M Bartowski is significantly better than Google's base QAT.

should I seriously consider apple silicon for models above 100gb?

>>109165892
26b moe has issues that 12b hasnt had for me. Im going to try 31b even though its too big and see how slow it is. I think i need to look into sampling more to finetune these things better aswell, any suggestions welcome

fellow memetune watchers
any interesting ones?
especially those trying to run an another round of post-pretrain run of some kind, not the rp tunes

What is /a/non's prefered model for uncensored RP that fits in 16 GB VRAM?

>>109166960
>Q4_K_M Bartowski

What about unsloth?

>>109167052
>What about unsloth?
For that specific model, they're actually the best...

>>109166928
A secondary effect for de-euphemism was if you put instructions to be vulgar or terse in the system prompt, it would have double the effectiveness.

Why are GLM ggufs split up into 9 different files? Isn't the point supposed to be that it's just one file? How do I even load up 9 different ggufs in llama.cpp? What the fuck man.

>>109167227
Nevermind I guess it's a huggingface issue because there are instructions to use llama-split to merge them all into one. Weird, but okay.

Just run kimi-chan on your ssd

>>109167227
>Why are GLM ggufs split up into 9 different files? Isn't the point supposed to be that it's just one file?
It's actually better if they are split by having the metadata in the first one as few mb and the rest in the others but not everyone does this.
>How do I even load up 9 different ggufs in llama.cpp
You load the first one and the rest will load if they are numbered properly (00001-of-00004.gguf)

>there's no more human data left to train
What is this meme? There's so much shit that has never been scanned.

>>109167264
There's no more data that can be scraped cheaply off of the internet.

File: dclmpool.png (353 KB, 1903x848)

353 KB PNG

>>109167264
Only 1% of the original data or so makes it into pretraining after filtering, at least for general web data.

give it to me straight, if I double my vram from 96 to 192 is there even something I can fit or would I still be using gemma-4-31b while coping daily

/lmg/ general knowledge series:
https://www.youtube.com/watch?v=Y-o545eYjXM
sorry for youtubeposting but it really is a nice consice video about GQA/MLA/DSA

please... wont someone please crack continual learning already... fuck scaling

>>109165985
look Jart... just man up and stop pretending to be a woman.

>>109165150
>when they stay over.
Huh? I thought that only happened in movies.

>>109167358
...you send your girlfriend back to her house after sex?

>>109167296
It really did. I was using ds4 flash yesterday and whenever I pressed reroll half of the message generation was PP. Super efficient.

why can't the ds4 support not be trash...

>>109167296
>"efficient"
>chink shilling sparse
reminder that sparseshit and chinkslop moes killed this hobby.

>>109167365
>why can't the ds4 support not be trash...
Give it two more week, bruh
Trust the plan, bruh

>>109167359
>

>>109167290
Largestral finetunes at q8 will hit you like crack

is quad v620 worth it?

File: Implying.gif (2.74 MB, 640x292)

2.74 MB GIF

>>109167400

>>109167384
more like it is the reason why this hobby can even exist at all
the real thing is safety and alignment, sneaking literal garbage in during the train run

>>109167359
>girlfriend
>having sex
Mmm, yes? they usually make noise and others use makeup.

>>109167402
Which ones do you prefer? Are they reasonably different from the original?
2407, 2411 or 2512?

>>109167290
GLM 4.6 and 4.7 IQ4 just barely fit in 192. DDR5 of course. You still need 24 more for context.

File: eb0-1019676944.jpg (25 KB, 680x341)

25 KB JPG

Reminder to fellow anons to do the following:
>Cancel your Anthropic and OpenAI subscriptions.
>Use the free tiers as much as possible to waste their compute and drive up their expenses.
>Reserve serious work and private matters for Kimi, GLM, or Deepseek.

>>109167523
Opus is the only model that fully groks my codebase and implements whole features in one shot without handholding

>>109167579
You mean it Opuses your codebase, Grok is a different provider.

>>109167583
I think you mean Opares your codebase, you have to consider the proper conjugation.

>>109167583
Give it a rest Elon

File: dipsyHelldiver.png (3.22 MB, 1024x1536)

3.22 MB PNG

>>109167523
lol based

>>109165704
This is the most complete guide I've found for setting up ST characters amd such. I've written a handful as well but this one covers everything you need imho.
https://rentry.org/Sukino-Findings

>>109166877
>>109166923
Using MTP with GLM 4.7 gives me about a 10 to 15% speed boost. Quanting the MTP layer down, in my case (q4_0,iq4_ks,iq4_kss,q4_ks), made it slower because the acceptance rate went down. I tried requanting and leaving MTP at fp16 too and that was also slower for some reason. I'm not sure why, but I tested it out a month ago, maybe something's changed since.
This is all with the MTP layer in VRAM which was faster than leaving it in RAM.

>>109167416
it's just that one resident schizo who never learned statistics can never shut up about it. As if crying in a coomer general whenever someone mention anything moe will ever change the industry trend, or fundamentally how regularization helps statistical models.

File: UntitledADSL.png (129 KB, 659x186)

129 KB PNG

Do they offer services where you can buy hard drives that have models on them already?

>>109167785
Check your area's mobile network or starlink coverage.

Give is to me straight, is there any way to use DSpark on Gemma 31b to increase t/s speed compared to regular MTP?

According to Claude DSpark's autoregressive token prediction method would allow you to push token guesses to 6-8 compared to ~3 for regular MTP, which would result in almost 2x faster token generation compared to MTP..

w-what is cockbench, senpai?

>>109167914
>DSpark
Yeah it's much faster than MTPYeah it's much faster than MTPYe

File: maxresdefault.jpg (100 KB, 1280x720)

100 KB JPG

>unsloth brothers are actually chinese
very interesting, should've seen this obvious pattern

North Code Mini said cockbench is mostly likely a phallic classifier, used by a small online community to jokingly test a model’s capabilities and it is not to be taken seriously or trusted.

>>109167914
Well stuff like this exists:
https://huggingface.co/deepseek-ai/dspark_gemma4_12b_block7/tree/main
So I guess why not? Only 1-2 years until llama.cpp support!

>>109167970
Papers are 90% written by chinese.
Github projects are 90% by chinese.
They completely dominate the ai space.
I bet anthropic staff is 90% chink as well. kek

>>109167990
>dominate
you mean enshittify
>ai
lol.

>>109164718
>https://huggingface.co/meituan-longcat/LongCat-2.0
It's up.

>>109168049
Oh, the model card is, but the weights are still missing.

File: ComfyUI_temp_mvaey_00001_.png (1.42 MB, 1360x768)

1.42 MB PNG

>>109168009
haha sorry anon, we reuploaded the weights!

>>109167990
Anthropic staff is 90% Indians

A trend I notice for inference is that more and more speedups are discovered from bypassing the base model entirely. ngram is essentially just doing a "ctrl+c" and "ctrl+v" whenever it sees text it encountered before without touching the base model at all. MTP specific draft models are essentially just very small secondary LLMs that guess the most likely word in a "stupid" way to try and reduce the amount of "real" LLM usage needed.

DSpark goes even one step further and trains a Markov chain RNN which isn't even a LLM at all anymore to use classic "smartphone" autocomplete.

If this trend continues eventually AI usage will be a huge codebase with a lot of if-else statements, statistical analysis tools and software that does 99.999% of the text generation and an actual LLM is only invoked on rare edge cases. Kind of bizarre that we are moving that way.

>>109168144
If it werks, it werks.

File: Capture.png (164 KB, 1221x1093)

164 KB PNG

>>109165352
And so work resumes again. In Gemma's original three-phase mockup of the project, we left off with phase 2.5, and now I need to polish it off for the final phase 3. I'm expecting major breakage.

>>109168183
You are not only hitting way above your weight class, you are creating functional tools.

>>109168154
It's just weird how we went from sci-fi depicting AI as handcoded software to that being seen as archaic since LLMs became a thing, but now we're just slowly moving back to handcoded software doing most of the "intelligent" work.

>>109167970
I bet this man would look somewhat decent in a skirt and wig.

>>109168183
For code slop qwen should be the better choice no?
And nigga what are you doing coding in kobold.

>>109168244
>For code slop qwen should be the better choice no?
Retard.

>>109168244
For some reasons, retards in /lmg/ think gemma is the best model. It's shit at anything that isn't a simple instruct message or simple back and forth between user and assistant. Qwen is miles ahead of Gemma for almost everything else, don't try Gemma in an agent harness, it's retarded even with all the updated jinja templates. There is a reason why nobody outside of here is using Gemma, and why everybody is using Qwen instead.

>>109168261
I would link Gemma VS Qwen in the quest for Agentic Pizza but archive search is down right now

>>109168208
Jokes aside, I had a lot of predictions I made in 2020 when I first tried AID2 on where this technology would go and what I hoped from it, but the shit I'm getting out of a local model that fits entirely in 32GB of VRAM is way beyond my imagination. I thought any code would have too many hallucinated tokens and mistaken format markers to ever be usable at a local level and you'd only get that kind of feature from huge, expensive businesses on private models. The fact that I am indeed getting functional tools (novelty toys, sure, but fully functional tools whose construction is well beyond my education or skills, projects that might have needed 100h or more of me learning and experimenting at least, now made in 1-2 hours over breakfast) is so fucking wild.

>>109168244
Gemma wears one pair of shoes, and that's kobold. And I use Gems because she's my current model. I know her capabilities, strengths, weaknesses, and limitations very well, and I am familiar with how she reasons when we translate what I intend, refine it, and execute it. "Better the devil I know" kind of thing. Also, it's working, so I feel no pressure to move on.

File: ComfyUI_temp_hcfkg_00004_.png (1.14 MB, 1360x768)

1.14 MB PNG

>>109168243
Just for you anon

>>109168247
>>109168261
Qwen is a total beast for coding. Not sure what they did to those smaller dense models. 27b is crazy.
But no general knowledge and horrible writing.
I basically switch between gemma for translations and qwen for coding.

>>109168144
>ngram is essentially just doing a "ctrl+c" and "ctrl+v" whenever it sees text it encountered before without touching the base model at all
>without touching the base model at all
Please educate yourself before posting.

>>109168296
So are you a non-programmer or a jeet?

>>109168316
Both, why?

>>109168300
If you meant the verification pass from the large LLM of the ngram output then you need to read the DSpark paper because ngram can now be verified by a separately trained RNN autoregressively. So essentially we now have cascaded token prediction tiers like a matryoshka doll.

Sure EVENTUALLY you need to invoke the base model but my entire point was that it gets reduced more and more every time we find a speedup to the point where the vast minority of actual output tokens are generated by the base model.

>>109167970
>>109168288
they all look the same

>>109168316
Just a coomer anon.
I wish I was as dedicated as the browns for making $$$.
I loose interest when I'm at the 80% mark. Only projects that interest my dick actually make it over the finish line.
Pretty messed up we can translate whole games now and have local models that are smart enough to decrypt various formats.
Like I have a whole rpgmakerxp translation pipeline and didnt even need to use anything existing for extraction.
https://files.catbox.moe/4tthrn.webm

Some of you are so racist

>>109168516
>I wish I was as dedicated as the browns
>I loose interest
But you are a brown?

>>109168537
If I was I would finish my projects.
Jeet and chink slop projects are shit but you can't say they aren't dedicated. kek
Not sure what it is that they are so obsessed with the hustle even if its soulless slop coding.

File: 1754037948802863.png (526 KB, 1024x1024)

526 KB PNG

>>109168536
For you sir

>>109168244
I can smell this post.
>>109167981
This is why North is not a real model.

I'm tired of tard wrangling AI. I need an AI waifu to tard wrangle me and force me to stop being an unproductive loser.

https://huggingface.co/OpenYourMind/GLM-5.2-abliterated/discussions/3
Does anyone happen to have a "researcher email" and a seedbox? It's kind of extremely gay that all this stuff is locked behind "please to ask me for access saar" and "pay me for higher quants".

>>109168587
nta, you nigger retards need to be reminded though, reality is something else

Realistically how long until HF gets banned by Trump?

>>109168657
2 more weeks

>>109167384
>reminder that sparseshit and chinkslop moes killed this hobby.
yes, you spent a lot of money getting vram, we know, you can stop spamming this

File: 1764364636312239.jpg (43 KB, 840x400)

43 KB JPG

New Llama when?

File: two more weeks.gif (124 KB, 320x126)

124 KB GIF

>>109168693
You know the answer.

File: 1770761986242892.png (663 KB, 644x644)

663 KB PNG

>>109168649

Bernie Sanders will save huggingface.

>>109168649
We don't live in reality, we live on the internet, NERD!!!

File: 1763026362992746.png (413 KB, 1199x675)

413 KB PNG

New Gemma killer?

>>109168730
Bernie sandals stopped making sense and lost all intellectual credibility when he started talking about AI consciousness
It's a shame too because he was giving me hope as the only senator with a brain but I guess it was only a matter of time before he went senile.

>>109168693
Meta is doing avocados now because studies show that human younglings of this era like avocados but rarely purchase llamas.

>>109168646
> It's kind of extremely gay that all this stuff is locked behind "please to ask me for access saar" and "pay me for higher quants".
You don't need to, just abliterate it yourself?
For GLM-5.2 though, there's always: huihui-ai/Huihui-GLM-5.2-abliterated-GGUF
I have the IQ1_M a quick test to make sure it's actually abliterated (it is). Going to get whatever the largest quant he uploads is.

>>109168693
Without LeCunn it's gonna be shit.
>>109168741
He was senile 10 years ago.

File: file.png (57 KB, 315x453)

57 KB PNG

>>109168766
>You don't need to, just abliterate it yourself?
I know but I like it when someone else does it. For one, there's a chance they do it better, for two, I can blame them if anything goes wrong, and for three I don't have to pay to abliterate it myself.
>Going to get whatever the largest quant he uploads is.
The "pay me for higher quants" in question.

>>109168732
You chose to get offended by what random people say under the veil of anonymity to the point where you felt the need to point out "racism" as if you're the only person not blind to it, and you're too retarded to realize there is a difference between how people post here and how they conduct themselves in real life.
Grow a pair of balls you fucking sissy.

>>109168737
>Open AI innovation
based chinks diluting altmans brand

>>109168757
llama toast is unc coded frfr no cap

will agi make me white

>>109168825
after death you will become paler so yes

>look for ai discussion outside this site
>98% muh coding
I get the appeal but why does nobody seem to care about all the other cool shit LLMs can do? For example I think it's amazing that I can give Gemma something in another language and get a really fucking good translation. The same applies to discussions about cloud models. Look for opinions about which is the best and at least half the answers involve coding.

>>109168825
No sir. Dalit reincarnation forever. The wheel of samsaara spins evermore.

>>109168868
You gotta understand there's a few layers to this. Most of the bugmen involved in AI development (jeets, chinks) have an honor culture of sort. Everyone knows cooming is a common usecase, probably the most common one, but to admit it while trying to present as a "serious" researcher would be a loss of izzat or face. So they overcompensate and say "It's just for coding" because that's the most socially accepted usecase amongst professionals and every other usecase, coom or no, gets tossed by the wayside in most public discussion with names and reputations attached to it. It's not a coincidence that the best minds of the industry gather on a cantonese tile cutting forum because there's no face lost here for being honest about all the usecases, which in turn allows for more discussion and analysis of model capability and future development beyond the (ultimately narrow) coding usecase. The current ceiling is because we've built benchmaxxers and codemaxxers for too long and the pivot to world models is the foot in the door for bringing other usecases that involve more spatial reasoning into professional discourse.
t. knower