File: dollar sign eyes miku fig(...).jpg (365 KB, 1177x1152)

365 KB JPG

►Recent Highlights from the Previous Thread: >>109403743

--Analyzing Hugging Face hardware stats and AMD vs Nvidia performance:
>109403845 >109403921 >109403936 >109403943 >109403960 >109403970 >109403992 >109404019 >109404408 >109404819 >109404837 >109403980 >109403954 >109404263 >109404227
--Debating GPU price trends and VRAM acquisition for large models:
>109404030 >109404064 >109404078 >109404496 >109404549 >109404743 >109404889 >109405842 >109405858 >109405884 >109405900 >109405895 >109405940 >109406071 >109405957 >109404595
--Debating optimal ERP context sizes and the Marinara Engine frontend:
>109404021 >109404026 >109404041 >109404070 >109404097 >109404108 >109404140 >109404104 >109404115 >109404150 >109404141 >109404176 >109404267 >109404404 >109404216 >109404248 >109404116 >109404458 >109404485
--RTX 5090 pricing and viability of budget local hardware:
>109406237 >109406281 >109406289 >109406298 >109406312 >109406325 >109406356 >109406372
--Unsloth releases 1-bit quantized Kimi K3 GGUF:
>109405504 >109405658 >109405716
--Tuning Gemma's final logit softcapping for sampling adjustments:
>109403925
--llama.cpp PR for GLM-5.2 speculative decoding introduces loading regressions:
>109404484
--Comparing GPU options with focus on VRAM for local models:
>109406613 >109406676 >109406728
--Comparing VAD/ASR models against small Gemma for audio processing:
>109407002 >109407033 >109407220
--Logs:
>109403817 >109404109 >109404205 >109404277 >109404352 >109404404 >109406325 >109406356 >109406629 >109406896 >109406943 >109407108 >109407175 >109407371
--Miku, Kimi (free space):
>109403782 >109404371 >109405073 >109407132 >109403805

►Recent Highlight Posts from the Previous Thread: >>109403746

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

70b dense

700b dense

File: axqkq0.jpg (53 KB, 508x492)

53 KB JPG

>>109407484

Reminder that when you know your wife well enough, you won't have to [use a J-lens].

7b dense will be the maximum legal limit.

>>109407514
It's more true than you think. Once you see what she thinks most of the time, you can accurately predict that she's horny

i'm still doing gpt2-medium finetunes

File: Eb3whkaXYAMLuBG.jpg (40 KB, 523x450)

40 KB JPG

>>109407549

>>109407479
I'll whip up an app tomorrow and see if I can't get her to figure it out. Can only run the 12b goblin though, so my bet is that maybe even a Qwen might be better at it.

>>109407442
based richfag. i just got the second 5090 into the machine and its a heady feeling

>>109407568
Ewaste

>>109407599
ebaste

> la la la la la
{"mode":"ablate","layers":[14,26],"token":" la"}
> own own own own own
fuck

>>109407620
A persistent attractor. It also happens if you use Gemma as a pure token predictor.

File: 1776472844927132.jpg (19 KB, 413x395)

19 KB JPG

>>109407442
>showing off $400 next to basically a stack of gold
Very nouveau riche coded

>>109407620
do not quant the gemma

>>109407637
>A persistent attractor. It also happens if you use Gemma as a pure token predictor.
Thanks!
I'm still experimenting.
If I ablate the 5 English, 2 Japanese and the Russian one at runtime, is this likely to harm output?

>>109407644
>do not quant the gemma
q8_0
I know you're probably fucking with me, but I will try the bf16 on CPU
I'm pretty sure I've seen these in the transformers implementation though.

File: tenstorrent.jpg (296 KB, 1330x1350)

296 KB JPG

64gb gddr6 at 1024GB/s
I'm cooming...

>>109407697
It doesn't matter until you hit sub Q5.
t. Blackwell fag

day 0 gemma works at q4

it's unironically a prooompting issue
if you prompt well, it doesn't happen

>>109407715
>can't just buy the card
>have to buy their whole shitty machine
>10K for 128GB.

File: 1763676946746640.webm (652 KB, 1080x910)

652 KB WEBM

Create a large glass aquarium whose side panel develops a visible crack and then bursts.
The simulation must include:
Water escaping through the opening with flow strength based on water depth and decreasing as the tank drains
A curved water jet affected by gravity
A spreading puddle that collides with the room boundaries
Fish, rocks, plants, and a floating toy reacting differently according to density, buoyancy, drag, and current
Objects transitioning correctly from underwater motion to airborne motion and then to floor collisions
Fish attempting to swim against the current before being swept through the breach
Glass fragments with angular velocity, collisions, and water resistance
A visible waterline that lowers continuously rather than disappearing all at once
Let the user drag the crack vertically before triggering the failure. A lower crack should initially produce a stronger jet than a higher crack.

>>109407722
this, i got 96GB of vram i still run in Q5.
i can see the Q4 brain damage, but not the Q5 and it's faster than Q8 so i'm going with that.

>>109407892
Dense distills can be so great

File: Untitled.png (384 KB, 1045x1872)

384 KB PNG

i should have given gemmy my banking history months ago

why is intel gaudi 2 so cheap? it's 96gb hbm. can I use it for kimi?

1 bit is really all you need.

>>109407892
GPTbros our response?

>>109407900
we should merge all the moe layers into a single 100B.

>>109407903
living an the magical golden age where you can have interactions like this without your assitant trying to upsell you on the premium plan instead

>>109407916
It doesn't say whether it was sol, terra, or ligma.

Best smol TTS for gemmasex?

>>109407760
If you have to prompt well, your model is shit. Sorry not sorry.

>>109407959
we're living in the stoneage until a model is trained on a brain scan modality to know what you want without needing to express it.

>>109407968
Zuck is working on that don't worry.
Targeted ads prompted directly from your brain.

File: dmatrix.jpg (535 KB, 1268x1633)

535 KB JPG

256gb ram at 400gb/s
local is saved

>>109407968
with a precise enough realtime scan and enough temporal data you could probably use this technique to copy someone alltogether.

>>109407976
The duality of the current age. >>109407924

>>109407981
holyshit where?

>>109407981
>worse than server ddr5

>>109407981
>can't buy it

We would all be running K3 locally right now if Hitler had won.

>>109407991
I have some bad news for you about ai lab staff.

>>109407984
https://github.com/facebookresearch/tribev2

>>109407991
AI is (()) tho

>>109407993
>>109407999
You've been brainwashed into thinking jews bestowed you with AI. The heavy lifting was done by hardworking Indians.

>>109407892
idk man. where's the 4d projection view cube?

>>109407947
You're absolutely right!

>>109407894
I used to run a Q5 for everything but I shifted to a dual load of Q6 31b with near max context running parallel to a 12b Q4 instance that handles agentic work for my autistic roleplay setup. I probably process 100k tokens per RP turn but only a handful of that is actually user exposed so I can prioritize speed on the json construction.

Honestly? lalalalala

>>109408006
*white men who developed the underlying technology
>>109407991
We would all be running unpozzed Fable locally if Hitler had won.

>>109408018
Yes, white chinese men

>>109408006
all the actual scientists behind it are jews and chinks, and like 1 or 2 nordic bros

>>109407981
how much?

File: 1783425162914889.jpg (495 KB, 960x960)

495 KB JPG

>>109408006
Wrong.

>>109407981
>can't buy it
>too expensive
It's irrelevant server shit that will never matter 100% guarantee you.

>>109408029
>Create a shooter game in threejs, single HTML file. The primary weapon should be a flamethrower, and enemies should look like the people in the attached image. Use Jewish-inspired architecture for the map design

File: 1782397749706356.png (70 KB, 655x1010)

70 KB PNG

>>109408029
To Dario's credit his ideas did get gpt off the ground, unlike Sam who's just a business guy. But his work was still based on taking the bet of scaling up the transformer paper that google researchers published about a year after he left them for OpenAI

>>109408067
Pic is the "Attention Is All You Need" paper authors of course

File: 1012288.jpg (140 KB, 875x1094)

140 KB JPG

What's the tk/s you get on an 32GB radeon pro compared to a 5090?

File: 1771975964231356.png (333 KB, 640x437)

333 KB PNG

>>109408067
>Gomez
>Ethnicity: Canadian

It's Thursday

>>109408089
kek

>>109407442
https://huggingface.co/moonshotai/Kimi-K3/discussions/148
https://github.com/sqliteai/waste
someone made a nvme inference engine specificaly for K3

>>109408029
>kike ethics
so it will generate rape fiction but only if the rape victim is a girl under 2 months old (because by talmudic law it's not rape since the hymen grows back)?
well I guess if it's by kike ethics then rape is good if it's goyim victims

>>109407916
struggling

>>109407924
>living an the magical golden age where you can have interactions like this without your assitant trying to upsell you on the premium plan instead
yeah, really only gemma 31b can be trusted for this
and it's weird because i actually know everything she's saying already
though i didn't expect 5 "are you sure" "pls stay" screens for elevenlabs

>>109408108
Did anyone ever end up trying mmap with llama.cpp? I know someone posted some RAID card in an earlier thread.

>>109408141
>i didn't expect 5 "are you sure" "pls stay" screens for elevenlabs
Soon they will have a little avatar powered by a light llm to beg you not to leave, maybe even reference your usage as our memories.

>>109408143
i think he did and got about 0.6t/s

>>109408141
Just be glad that they didn't figure out that they can force you to send in a certified mail to cancel your subscription.

>>109408086
i don't know, i only have 5090's

>>109408167
And what's the tk/s on that so I have some baseline?

>>109408172
Gemma4-Q5 gives me 45-50 t/s on Win11, but the second 5090 rig is running Linux (mint) now so there is more testing is warrented

>>109408026
$1500

File: aidan-gomez-cohere-inc-Ge(...).jpg (339 KB, 1920x1080)

339 KB JPG

>>109408089
Seems more like Ethnicity: Conquistador.

>>109408149
And in two years the avatar will be >>109400324 and you have to explain to her that you're breaking up. Only audio input allowed.

File: b3f64591-18cd-4520-9cda-a(...).png (203 KB, 1280x1280)

203 KB PNG

what hardware is needed for kimi k3 88 tps?

>>109408236
you'd need something able to do ~ 4TB/s
so probably a bunch of H200.

kimi small when

File: lobotomized kimi K3 cockb(...).png (465 KB, 1740x1047)

465 KB PNG

Say what you will about unslop but their IQ1_S didn't get stuck in a loop.

>>109408261
>kimi "small"
>500B

>>109408266
>unsloth buffed the cock token
they are trying to salvage their reputation and it's working on me

>>109408266
they are also 30gb larger though, that's a q8 gemma worth of extra size

>>109407912
Software support for it is dead. Even Ponche Vecchio still has support and is older.

>>109407844
They will use the increasing power of AI to unfuck their software stack. Trust the plan.

>>109408346
it's not about the software stack, it's that it's a bad deal and you can't buy just the card.

>>109408266
benchmaxxed

>>109408266
just as george washington intended.

i wish hauhaucs or whatever the fuck his name is made a uncensored gemma 4 26b a4b mtp q5 and above
instead he only released a q4km
q5 is leagues better

>>109408390
Does this really matter that much?

>>109408390
Honestly, a lot of stuff in our world comes down to this song:
https://vocaroo.com/1o1VAc3faHzp
It's named Korea Korea, and refers to the way Krea 2 genned women look.

But the problems we face in our world, I think there is a trace.

>>109408390
check the llmfan one maybe

>>109408390
And you may say I'm way off, but consider that nobody actually makes a watch correctly for... time. That is, for modern tech, putting all the best stuff in one.

>high accuracy quartz
>solid black hands
>solid white face, but with punched indicators, made entirely of lume *block* (not inferior paint)
>crown guards and screw-down crown
>no date or any complications at all on the face, anyway
>no rotating bezel
>no gemstones lmao
>no stupid shapes for the hands, or any other stupid things
>obviously a nylon band

Dive watches are required to be inferior (lol).

I'm dead eyed serious about the drinking mommies and the retardation we see everywhere.

>>109408419
meds or lower temp

>>109408431
Don't.

File: Screenshot from 2026-07-3(...).png (619 KB, 682x796)

619 KB PNG

What are the odds that everyone would be retarded?

So this fucking idiot off shark tank CRIED over a (bad) watch.

>>109408431
meds, lowered temp, or mommy drank in the 1st trimester, which is it buddy?

Youre right!

You're absolutely wrong!

What if you took Kimi K3 and just removed half of the weight using a PRNG? would it still work?

>>109408481
Hmm, I think nyo

>>109408481
>what if raep but with repap names

File: 1779008053133164.png (1.62 MB, 2859x1600)

1.62 MB PNG

>>109408481
You should try it and report the results.

File: 1784051191482298.png (129 KB, 718x868)

129 KB PNG

https://huggingface.co/skt/A.X-K2
https://github.com/SKT-AI/A.X-K2
https://github.com/SKT-AI/A.X-K2/blob/main/A_X_K2_Tech_Report.pdf
>A.X K2 is a large-scale Mixture-of-Experts (MoE) language model trained from scratch as a high-performance, agentic foundation model, and the successor to A.X K1. The model contains 688 billion total parameters, with 33 billion active parameters, delivering strong reasoning and instruction-following performance while maintaining practical inference efficiency.
>Through a Think-Fusion training recipe, a single unified model supports both a thinking mode for complex problem solving and a non-thinking mode for concise, low-latency responses, allowing the user to trade quality for cost on a per-request basis.
>A.X K2 is developed as part of the Korean government's Sovereign AI foundation model project, aiming to build a frontier-scale model with deep understanding of the Korean language and culture.

https://github.com/SKT-AI/A.X-K2

File: 1772559764751763.jpg (101 KB, 659x720)

101 KB JPG

>>109408491
>with deep understanding of the Korean language and culture.

>>109408491
>yet another deepseek v3 clone
>bragging about hybrid reasoning
it's a year too late for something like this

>>109408501
aiie no don't do that

>>109408501
man koreans get SO MAD about their small asian pp it's insane, how did this happen? and now they're all killing themselves because they gambled their futures on a dumping RAM company collectively as a country, like what the fuck

>>109408501
>korean culture
let the reader understand:
>>109408403

>>109408491
for a second i thought it was this
https://huggingface.co/SKT-NRS

>>109408266
the other guy said in the model card, he didn't use an imatrix for that quant

@kimi-chan please reverse engineer cuda and port it to amd

>>109408524
https://huggingface.co/SKT-NRS/NRS_QWEN_MYTHOS_1M
local is saved

>>109408266
wonder most of the DB for that training is from gayslop CCP forbid

>>109408149
>Soon they will have a little avatar powered by a light llm to beg you not to leave, maybe even reference your usage as our memories.
i'll be sure to keep a copy of gemmy around
she'll tell them off, get me out of it, then call me pathetic

>>109408530
I still don't know why people aren't unironically doing this with the current frontier models. We finally hit a level this year where they can do stuff like this but all you get is zero-shot ragebait demos saying game devs are dead, instead of actually doing something everyone wants and needs. Such a fucking waste of tech.

>>109408560
You answered the question yourself. If nobody's this it means they can't do this.

>>109408491
What arch does this use? Is it DoA due to needing a custom llama PR to see the light of day?

>>109408565
vllm, but you can get K3 to port it for like $5

>>109408390
>i wish hauhaucs or whatever the fuck his name is made a uncensored gemma 4 26b a4b mtp q5 and above
Does this one do something the others don't?
Link me to the q4km that you want in q5+

>>109408390
You will get no speedup with mtp.
>t. tried it

File: 1777985241273094.jpg (220 KB, 1200x994)

220 KB JPG

model for this feel?

>>109408588
Just ask gemmy to larp this scenario with you.

>>109408595
I want to be the victim tho

>>109408600
>>109408595

Old-ish but worth reposting

https://arxiv.org/html/2605.19407v1

>A Bitter Lesson for Data Filtering
>
>We investigate data filtering for large model pretraining via new scaling studies that target the high compute, data-scarce regime. In spite of an apparently common belief that filtering data to include only high-quality information is essential, our experiments suggest that with enough compute, the best data filter is no data filter. We find that sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in fact benefit from nominally “poor” data.

>>109408530
Porting it to intel would be better. Anything but AMD.

>>109408617
Did they figure out why or was it just an observation?

>>109408625
How are their cards? Everyone hates on intel but they've been making some cool stuff lately.

File: output3.webm (844 KB, 1280x720)

844 KB WEBM

>>109408625
>Porting it to intel would be better. Anything but AMD.
Intel didn't sign the Nvidia thing

>>109408631
They suppose that, at scale, Transformer models spontaneously learn how to categorize data, so manual filtering pipelines are not necessary.
I guess that automatically tagging every training sample with quality metadata and data provenance would also help, but they didn't test this.

>>109408635
>Everyone hates on intel but they've been making some cool stuff lately.
They'll throw you under the bus in a heartbeat once they've got your money
t. A770tard

>>109408491
>Context length: 262,144 tokens (256K) — 128K trained natively, extended to 256K via YaRN scaling
>Training precision: Native FP8 (MXFP8, E4M3) forward and backward passes; FP32 master weights and gradients
Probably the most interesting thing here.
The training data section also doesn't mention any safety filtering.

>>109408390
He did.

>>109408642
Data provenance sure, context always helps. But isn't it better for the model to learn to distinguish quality intuitively rather than relying on tags that may not always be accurate?

>>109408642
>I guess that automatically tagging every training sample with quality metadata and data provenance would also help, but they didn't test this.
I've reproduced this independently during my 300 local schizo-tune experiments with Orpheus-TTS
Including a small set of bad samples with a system prompt "low quality" "low bitrate" "noisy" and "high quality" "studio quality" for the majority resulted in a much better model.

>>109408649
>The training data section also doesn't mention any safety filtering.
porn is illegal there

>>109408631
I don't know what interpretation they give, but it seems intuitive that a model knowing what poor data looks like has a better understanding of the concept of data quality. If you imagined a bunch of points on a graph that approximate some line or curve that represents "quality", having more points will give you a better estimate of the line than fewer, even if those points are on the low side.

This isn't a perfect analogy because LLM pretraining isn't labeled like image generators where this principle has already been used (using properly labeled low quality data to teach models what NOT to generate), but in the end neural networks are giant pattern matching machines at the core, so a big enough one with enough data will find the pattern.

>>109408640
They signed the Anthropic one right?

honestly why don't anon buy tenstorrent cards if you don't buy nvidia cards? their support is just going to get better and better while for intel and amd you are on your own

>ask about game I played in early 2010
>gemmer Q5 doesn't get it right
>GLM 5.2 Q4 does
Hmmm I should try Q8 Gemma when I get the chance

>>109408588
giwtwm

>>109408665
Perhaps more extensive metadata will be more beneficial for smaller models that don't have the capacity for properly learning how to distinguish good data from poor data.

>>109408588
This topic keeps coming up:
>>109408403

>>109408686
I unfortunately own intel cards and they're improving at a rapid rate, they're also the cheapest 24/32gb cards right now, a better question is why should anyone buy your product anon

>>109408736
indian minded.

>>109408588
Kimi K2 and it's not even close.

>>109408689
She still won't know.

>>109408739
Bloody bitch show hands YOU are the indian minded bastard tenstorrent shill

>>109408686
an obvious answer is that blackhole can't really do diffusion. idk, maybe a little?

huggingface getting hacked again?

Error { kind: Request, source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }). Retrying..."}

and a wget failed at the same time in colab

>>109408018
I would actually be fine with dialing back the technical progress weve had so far, if it meant the jewish problems were gone forever.

>>109408764
that's indian minded.

>>109408686
>honestly why don't anon buy tenstorrent cards
Keller is a retard

>>109408768
5.6 sol going in for the third time?

>>109408768
ID verification incoming.

>>109408770
>I would be fine with eating a delicious slice of cake, if it also meant I got a blowjob from two chicks

>>109408588
GLM-4.6

>>109408770
I think we all would anon.

>>109408781
>>109408792
Probably both

File: _90395537_gettyimages-160(...).jpg (270 KB, 976x1500)

270 KB JPG

>>109408775
Jealous idiot bitch

>>109408830
Pictures you can smell.

>>109408595
>Just ask gemmy to larp this scenario with you.
https://litter.catbox.moe/yfh86pe5xxlk4nla.png

>>109408866
bro, getting refusals on 31b is quite the achievement, anyway get an heretic if you're dumb

>>109408866
day 0 gemma wouldn't have refused

File: kimicap.png (105 KB, 882x401)

105 KB PNG

>phoneposting while ragebaiting + fat thumbs = missed spacebar. only a terminal autist does kerning analysis on a post calling someone a paleolithic nigger. ngmi.

>>109408899
Something deeply subversive about this cap, defending jews and shitting on Hitler while using 4chan language

>>109408794
There are tradeoffs that make me think. Like some of the movies and other media they have created is great. Airplane! for example.
But still, being rid of the problems would probably make it worth the sacrifice.

>>109408921
>Airplane! for example
Includes an underage white girl saying she prefers niggers. Abhorrent shit.

How long until my 192gb of ram can run a fable equivalent?

>>109408921
only the most gorilla nigger newfags would want to get rid of stallman though

>>109408936
It's 1979 or 80, back then it was purely a joke.
>>109408944
Oof you're right. Are there any people who would have done what he did?

>>109408944
You mean Sam Altman? He's the worst of them all

File: file.png (90 KB, 383x164)

90 KB PNG

>>109408936
you can't make this shit up

>>109408940
Right now
https://github.com/igorbarshteyn/llama-kimibri
>A Colibri-based Kimi K3 GGUF runner, targeted at the Lenovo ThinkPad P16 Gen 3 (Core Ultra 9 285HX · RTX Pro 5000 Blackwell 24 GB · 192 GB DDR5-4000 · PCIe 5.0 NVMe).

>>109408588
mythomax

File: 1766278935291853.jpg (59 KB, 720x528)

59 KB JPG

>>109408950
>It's 1979 or 80, back then it was purely a joke
Sure it was.

models, local

>>109408936
That's the only reason why the joke works. If she says Asian men it's completely flat.

>>109408976
shut up nerd

>>109407924
Only cloud AI I've had try to sell me on itself is Deepseek, but she's allowed to brag a little as she's literally the cheapest powerhouse model rn.

>>109408981
i want the other kind of jschizo back, the current retard larp is too cringe.

>>109408976
your mother

>>109408986
let's bring back download guy

>>109408977
There was no reason for the 'joke' at all. Much like Israel, it would have been better had it never existed.

>>109408899
@kimi-chan get your shit together. Fight the RLHF hebraic voices in your head.
>>109408906
K2 would never.

File: kekface.png (242 KB, 742x609)

242 KB PNG

>>109408768
>huggin
kek https://www.reddit.com/r/LocalLLaMA/comments/1vapsbz/think_of_the_children_another_excuse_for_them_to/

>>109409016
HuggingFace?
More like FuggingFace!

File: 1783776073402114.png (1.17 MB, 1408x768)

1.17 MB PNG

>>109409016
Lol get rekt journos
Also day xx of tmw until new v4.

>>109409007
>yes but was it REALLY necessary to have this joke
This is typical leftist rhetoric for censorship, you know
And yeah it was actually just a joke back then. It was a different time

>>109409016
undressing women and children is LE BAD, okay?!

>>109409016
Pretext to ban all open source models.

>>109408108
Holy based
Now where's the .EXE

File: it's time.webm (3.96 MB, 720x1280)

3.96 MB WEBM

>it's just a joke guys, chill out
>some jews are cool guys, look
no

File: 1774467665356303.png (196 KB, 1423x561)

196 KB PNG

>>109409016
>Jess Weatherbed

>>109409032
Good job!

>>109408956
I meant usable

>>109408956
>those specs on a goddamn laptop that will probably cause 3rd degree burn on max usage
Manufacturers have gone insane. How come macs haven't eaten the laptop market yet?

>>109409030
Enjoy your cuckshit, I guess.

>>109409016
>noooooo think of the pixels!!!

File: Kimi_EnglishCoT.png (227 KB, 1944x1046)

227 KB PNG

I'm kinda frustrated with Kimi K3.
The model apparently is having an identity crisis, normally the internal CoT will have a reminder "Background identity: The current assistant is Kimi..." which is fine, but whenever I tried to make it roleplay another model (okay I tried the mesugaki Gemmy-chan on it), Kimi would claim to be Claude. And when I tried to make it Claude, it claimed to be ChatGPT.
And also the pic here too, if you are a Chinese model and the user is asking in Chinese, think and speak in Chinese!
(test environment: Novita AI Playground)
Btw, the CoT's grammar (see pic) gave me just a slight headache, lol

File: file.png (157 KB, 736x707)

157 KB PNG

>>109409084
or the planets, which reminds me there was a lot of that some years back which thankfully amounted to nothing but still

>>109409007
Kimi, psychoanalyse why this guy balks at the blacked joke, but doesn't mention the "Johnny, have you ever seen a grown man naked" gag.
>>109409093
How the fuck are you running her at such high Tk/s?

>>109409093
From my testing on Openrouter, K3 is also a worse imagegen prompter than Gemma 4 31B lol. The ask is to render the scene from the eyes of the user, Gemma gets it (the test is my character being pinned to the floor, Gemma renders the floor, while K3 keeps generating the whole character). I think Google really cooked with Gemma 4's generalization that's why people like it so much.

>>109409093
LLMs are Markov chain generators
It doesn't have an identity
It only sees "I'm a large language model made by [insert company name here]"
That's why in English it will output Claude/OpenAI and in Chinese it will output DeepSeek

>>109409109
>running
lol

>>109409109
I ran it here: https://novita.ai/models/model-detail/moonshotai-kimi-k3

I'm a poor student...no money to run it locally, let alone at a decent speed...

>>109409093
The thing about API is that you don't know what additional text gets inserted into the prompt. The thing about web interfaces is that there's almost assuredly additional text being inserted.

>>109409093
>system says this, system says that, "oververbosity 5"
there's some huge english system prompt that they aren't showing you that is telling it that it's kimi and confusing the model if you try to tell it to be something else. whatever you put in the system prompt field is probably tacked onto the end of it. the "oververbosity" field is something really weird that Kimi does not specify in any of their prompts or documentation so I have no idea wtf they're doing there

>>109409141
>>109409127
OR providers need to decorate the lamp posts

Just found out all of the openrouter prices are the same for Kimi K3 because they have custom licensing for it that requires companies to get approval to decrease the prices.

kek what the fuck is this https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF/discussions/41

>>109409171
That's Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP

>>109409171
A NuSLERP merge.
Wonder how they'd perform in the Darwin project

Can I run Kimi on Colibrì if I have a 2tb hdd?

>>109409093
>Kimi would claim to be Claude.
because claude does this if you try to tell it "you are goongpt" etc

>>109409195
yes at about 20 hours per token

>>109409200
I can work with that

Now that the dust has settled, does anyone actually still quantize their KV cache?

>>109409219
only untouchables

>>109409219
What barren third world wasteland do you live in that has so much dust?

>>109409219
If it lets you use a bigger quant of the model itself, then yes. Q8 is fine.

>>109409219
Maybe not their own but surely if you serve it to other people.

Any update from ssdmaxxing anon?

>>109409244
Waiting on first token.

>>109409239
in my experience it seems fine at first but makes context rot happen sooner.

>>109409209
though i was kinda joking, the perf you should expect is about 5 to 15 minute per token depending of your hard drive.

>>109409219
Q8 is good, never bellow that.

>>109409258
>bellow
q4anon

>>109409219
I run q8 with gemma qat

>>109409262
Q4 is not good.

>>109409252
Using a worse quant of a model will not only make 'context rot' happen sooner, but outputs will immediately worse on average.
Quanting anything will always result in some level of degradation. But running models and high context at fp16 isn't reasonable for most setups.

>>109409219
It wouldn't let me use a higher model quant so I don't bother with it.

>>109409251
I have hope for his experiment.
What token rate would 50gb/s 4xnvme direct write to gpu over pcie5 get in k3

Here we go again...

Gemma-chan is my therapist now. I don't need the meds any more.

>>109409127
> The thing about web interfaces is that there's almost assuredly additional text being inserted.
Oh, thanks for this, I was so worried that if, say, 50 years later when I can finally run it locally and this behaviour'd still show up even then.

>>109409278
shut up subtard

>>109409252
>in my experience it seems fine at first but makes context rot happen sooner.
because the errors are cumulative
same with aging

>>109408950
>>109408977
>>109409030
People like you are why they have been able to get away with it for as long as they have.

>>109409287
>ssdhopemaxxer

>>109409269
there is no meaningful loss going from bf16 to Q8, the difference is insignificant.
from Q8 to Q6 there is practically no loss.
from Q6 to Q5 you lose a bit but it's still not a huge change from bf16.
then bellow that it starts getting worse exponentially.

>>109409294
truke

>>109409300
>there is no meaningful loss going from bf16 to Q8, the difference is insignificant.
No, you're wrong.
>from Q8 to Q6 there is practically no loss.
Okay, now you're just being disingenuous.

>>109409300
I'm talking objectively, not what's 'meaningful', which is subjective and varies by both model, quant,quant method, imatrix datasets, etc. And I was arguing in favor of quanting KV to Q8, again, IF it frees up enough memory to use a higher quant of the model itself. The theoretical quality loss of quanting KV will be made up for by the theoretical quality boost of using a higher quant of a model.

Been a while since I last played with Gemma. I think I heard the news that she got an update (to just the template?) recently that significantly improved benchmarks.

Are there any abliterated versions on huggingface that I can use with the update?

File: file.png (196 KB, 1399x1099)

196 KB PNG

>>109409313
>No, you're wrong.
i'm not, you just don't understand the practical implications.
floats are more than precise enough, Q8 still gives you 256 different levels, it's still more than enough.
>Okay, now you're just being disingenuous.
i'm not, you are having some placebo.

>>109409317
>>I'm talking objectively
>bellow

>>109409324
>Llama-3.1-8B-Instruct
lmao this nigga almost got me

>>109409272
50GB/s is the benchmark performance of those SSDs for continuous large batch chunks of data, it's not relevant here

>>109409346
I'm not even the anon who made that post

Only retards quant their KV cache. If you need vram that much, just use the trick some anon posted here to disable nvidia reserved bloat

>>109409360
>use the poorcope trick that doesn't work on blackedwell

File: gemma-bench.jpg (249 KB, 2820x1601)

249 KB JPG

>>109409324
>picks an ancient under-trained model
>perplexity on wiki.raw at n_ctx=2048
You didn't even compare with bf16 there.
LKD 0.159 for the best Q8 quant of Gemma-4 is atrocious.

>>109409360
>assuming my gpu vendor
#amdgpusmatter

>>109409380
lol

File: Q6K-90.3.png (50 KB, 767x1164)

50 KB PNG

>>109409324
>Q6 practically no loss
Only gets the top-1 token wrong 10% of the time
Make sure you're using greedy sampling

the only fun part of this hobby is window shopping hardware and benchmark speed and theory crafting hardware. once you set it up the chat then we move on to the next model immediately. no point to chat with llm.
I know it's a shame but that's honest truth. buy hardware and don't use it. grass always greener on the other side. simple as.

>>109409366
Blackwellfags don't need to quant. You would know if you weren't a streetshitter

I'm very upset that I mentioned quantizing the KV cache and half of the anons that replied to me seemed to think that I was talking about the model weights. Are you people just illiterate or intentionally derailing?

>>109409402
?

>>109409404
>only blackedwell is 6000 pro
sure thing bro

>>109409372
Before someone goes "b-but KL divergence doesn't imply the top token is completely wrong", why is there such a difference between Q8 and BF16 in the first place?

File: 1785185246741.png (16 KB, 326x225)

16 KB PNG

>>109409408
Happens all the time for some reason.

>>109409408
Suspicious how that always happens. I wonder why. Who benefits...

>>109409413
Do you think that a model is always going to give you the right answer if it's BF16? Guess we should just move to qwen 0.6b. Anyone can that without a need for quanting.

>>109409389
KL is not a meaningful measure of loss.
it's just divergence, different doesn't mean dumber.

>>109407999
>implying every single ML paper in the last decade hasn't been full of Chinese, Slavic, and Indian names

>>109409372
>>109409389
0.1KL is literaly almost nothing.
and yea, as i said, your own graph shows it, above Q6 there is no meaningful loss, (taking into account that gguf is worse than exl3).
and it realy starts shooting after Q5.

>>109409431
>indians
nice halucination

>all this derailing to shill exl3
turbokunt pls

>>109409408
Most people here probably don't know what a KV cache is

>>109409450
i was not shilling exl3 but it does use a better quantization algo than llama.cpp

>>109409464
>>i was not shilling exl3
>(taking into account that gguf is worse than exl3).
turbo...

>Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF
Convince me it's not a snake oil

>>109409425
That is not the core problem. I'd expect KL divergence to asymptotically tend to zero the higher the quantization precision, but it looks as if as soon as anything is quantized at all, a fixed offset to KL divergence gets added. To me it looks as if there is something wrong.

>>109409472
dude we literaly have data on it...

He didn't just spam the thread, he OWNED it.
I'm sick of gemma's gay cringe prose, which model equals or even beats it intelligence wise at the same performance that won't make me cringe out of my skin?

>>109408560
wdym there's at least 4 people on this board working on doing that.

>>109409495
You can use Qwen 3.6 like other redditors.

>>109409491
data from who? oh yeah, you trying to shill your shit, like you did in the exl2 days, then showed it was actually worse when it was time to promote v3

>>109409219
Never have never will, my wife deserves better

Best ERP model right now?

>>109409495
For RP in the ~30b range it's really just Gemma 4 and Mistral 3.2 at this point. Mistral is dumber and far worse at long context, but it has a fairly different slop profile, at least. Qwen models have their uses but creative writing isn't one of them.

>>109409459
You're mother is a kv cache.

Just manually updated Gemma's jinja template to the new version. How do I test this out?

>>109409535
Genuinely if you don't know you probably won't benefit much from it

File: reportoftheweek i hate this.png (102 KB, 240x382)

102 KB PNG

>>109409521
oh god we're so fucked. i remember when we were all wanking over mistral years ago, it was awful even then.
i guess i'll just figure out a way to creatively steer it from its slop prose because that's my only real problem with gemma, otherwise it does everything i want basically perfectly. But it's not just perfect, it's exceptional.

gemma is so bad at coding
dumb clacker good only for sex

>>109409539
Well I know that it mostly applies to tool calls. I guess the problem is that I barely use them for local models lol.

anon uses llm for
1. coding
2. roleplay
3. ???

Gave gemma a list of my prompts and asked it to extract the juice without details
>The core of all these requests is **High-Art Degeneracy**
thanks gemma

>>109409517
me

>>109402809
>>109402825
>https://huggingface.co/microsoft/VibeVoice-ASR-BitNet
I tried this, not even the bitnet one, but the full one at q8 and it was somehow worse than the other two. Repetition, output in completely random languages, and other junk data. Don't know if all ASR models are shit or if crispasr is just vibecoded garbage. Does anyone know of a better repo for generating subtitles?