File: Anima_00024_.png (959 KB, 1024x1024)

959 KB PNG

►Recent Highlights from the Previous Thread: >>109197629

--Comparing MoE prompt adherence to Gemma for high-RAM hardware investment:
>109200988 >109200995 >109201031 >109201567 >109201152 >109204075 >109204136 >109204134 >109204222 >109204252 >109202168 >109202190 >109202278
--Anon shares TTS/STT project developed using LLMs:
>109199677 >109199791 >109199852 >109200115 >109200205 >109200300 >109203326 >109203580 >109203738
--Debating Meta's Vistara chip and its relation to CXL memory:
>109198605 >109198660 >109199896 >109198673 >109198745 >109198784 >109199034 >109199078 >109199847
--Debate over financial viability and sovereignty of local vs cloud LLMs:
>109198840 >109198882 >109198908 >109198966 >109199199 >109198910 >109198986 >109199017 >109199207 >109199807 >109199936 >109200031 >109200068 >109200102 >109200131 >109200168 >109200242 >109200461 >109200409 >109201489 >109201714 >109199991 >109200008 >109201747
--LongCat-2.0 1.6T MoE weights released in FP8 and INT8:
>109197849 >109198977 >109198985 >109201544 >109202212 >109202497 >109198995
--Configuring pi for llama.cpp context limits and reasoning effort:
>109198317 >109200649 >109200888 >109201029 >109203219
--Comparing Qwen and Gemma for tool use and general assistance:
>109198513 >109198554 >109198752 >109198790 >109199172
--Anons sharing and comparing their model generation speeds:
>109204026 >109204055 >109204070
--Anon seeks uncensored models to avoid progressive-leaning safety filters:
>109201800 >109201932 >109201959 >109201975 >109202633 >109201917
--Availability and usage of DeepSeek v4 4-bit expert weights:
>109202355 >109202361 >109202430 >109202923 >109202902
--Logs:
>109198752 >109198910 >109199584 >109199677 >109200789 >109202335
--Miku (free space):
>109198587 >109198808 >109198986 >109200128 >109200888 >109202560

►Recent Highlight Posts from the Previous Thread: >>109198609

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

>>109205164
>LongCat-2.0 trained on AI ASICs
nvidia diff magnitude?

best coding memetune?

For anyone looking into automated slop detection or banning, pointwise mutual information may be an interesting area of research.

70b dense

>>109205337
I'm not sure about this.

>>109205337
Too crude, you need to look at the semantic content for slop. It's not something you can cut with statistics alone.

Any hopes for non-Code Kimi K2.7 that's dropping next week?

1t-a100b moe

Any way to give Gemma text from VNs (other then screenshots)? I read in Japanese and want to have it explain stuff I don't understand.

>>109205337
many have tried, always same two results:
1. doesn't work/does too little
2. makes the model retarded outside of benchmarks

>>109205337
>>109205512
We need AGI to get rid of slop. In other words, wait for JEPA.

>>109205584
Use lunatranslator

imagine glm 5.2 but diffusion based

>>109205643
Has anybody done an honest to God head-to-head comparison between traditional and diffusion LLMs?
Actually, what would head to head mean in this case?
Same data. What else?

>>109205654
diffusion LLMs have been around for a while and are and always will be a meme.

>>109205337
>>109205512
>>109205337
regex string banning on schitzo fork
control-vectors (also on schitzo fork)
orb-anon style de-purple/euphemism work (classifier + heretic ablation) -> but for slop

>>109205584
Hooks give you access to the text, but they tend to be finicky. Some engines are better supported than others.

>λ(w) = log P(w|S) − log P(w|H), smoothed counts.
>score(d) = mean λ over d's n-grams. Slop if > τ.
>Rare-n-gram fix: z = λ/σ, σ2 ≈ 1/count_S + 1/count_H, keep |z| > 2.
>Non-lexical: log P_slop(d) − log P_clean(d) via two small LMs.
Train on your own corpora. I'm using this approach to de-slop specification documents.

>>109205590
>2. makes the model retarded outside of benchmarks
That's basically anything now that changes the weights or even takes it out of its default assistant persona. These sorts of things need to be done during training.

>>109205597
JEPA won't get rid of slop, though. It will create a high-level discussion plan, then the model will delegate translating that to actual words to a smaller LLM. That LLM will still be the source of slop.

>Gryphe/Gemma-4-31B-StyleTune
>Normally when I finetune a model I train as much of it as possible, loading every tensor and transforming it to better approximate whatever's in my data. Not this time. This time I trained precisely one tensor: the lm_head output projection - the layer that decides which token to emit. Literally the last stop before text appears on your screen.
Oh yeah, I've heard of that before. I think I saw one guy do that to force the model always output in russian or something.
Interesting.

Is Yann going to win or is it just another meme to scam retarded VCs before the pop?

File: davidau.png (108 KB, 874x634)

108 KB PNG

>>109205981
oh nononono.....

Model that can think and respond simultaneously when?

i look like that

>>109206083
lmao

>>109206097
You mean like interleaved thinking where the model
>thinks
>responds
>thinks
>responds
etc?

>>109206112
No, thinking while talking like humans do. Like how you can be talking to your boss about sports while thinking about how much you hate him.

>>109206097
diffusion models, probably

>>109206143
thats just big and small model, in terms of LLMs

>>109206097
thinking is a meme

dead general

File: Coronel Amodei.jpg (110 KB, 1537x1023)

110 KB JPG

>You are sheltering unregulated weights, are you not?
>You're sheltering open local models inside your hard drives, aren't you?

>>109206386
Smile 4?

how do researchers in big companies feel when they know agi will never be achieved with llms?

So what happened to the schizo anon who claimed google leaked the gemini weights?

>>109206472
He got suicided

File: 116.png (149 KB, 1467x545)

149 KB PNG

https://huggingface.co/chartreuse-verte/gemma-4-31b-it-purple-euphemism-trial116-s1.5-depurpled-GGUF/tree/main

Fellas, I'm back. This might be the funniest model I've ever prompted. It has half the perplexity on human writing compared to base Gemma 4 31B and it's extremely rude and extremely angry.

>>109206611
>it's extremely rude and extremely angry.
What happens if you use a sweet persona like Seraphina or whatever?

File: oh7344fho6aron9kc73k.png (115 KB, 525x604)

115 KB PNG

>>109206611
I still see slop everywhere but will try it anyway

>>109206676
Until ARA gets merged, I need help on the dataset. A normal ablation dataset for just a narrow goal (refusals) is more than 7000 samples. Mine is like 350 because I manually annotate them.

>>109206611
>Amaryllis's jaw didn't just drop; it practically hit the gravel.
How I am supposed to take you seriously when this is the first line of your cherry-picked example of unslopped output?

>>109204920
m3 schizo here. I still use it on one of my inference machines. I like the output, despite needing some hefty wrangling compared to Kimi (you need to nuke patterns early and mercilessly or it devolves later in context and it does some laughably stupid spatial shit like low b models)
It definitely has some unique sovl. Using it in a pipeline to rewrite Kimi's output is still on my to-do list.
GLM 5.2 is back-burnered until support in lcpp is better. I can't justify the 75% hit in generation speeds for the mild increase in intelligence over 2.7.

>>109206822
>until support in lcpp is better
wait what? 5.2 is basically 5.1 right? is something still not working there? i'm literally building a server for 5.2...

File: egyptWon.png (1.96 MB, 1356x3401)

1.96 MB PNG

Only posting the important stuff today. Some anon posted a partial of it a few days ago.

>>109206898
Egypt won

File: 116-compare.png (84 KB, 1150x687)

84 KB PNG

>>109206647
The mode will try and fail to maintain character probably.
>>109206790
I'm still experimenting and contrastive negation (what you call slop) isn't on the menu. Turns out slop shows up because vivid = purple = slop vector. I need a lot more data. You sound extremely entitled btw, you're not paying me to do all this.

File: 1777161934372489.jpg (16 KB, 375x420)

16 KB JPG

>>109206611
Whenever I see shit like this I ask myself if it's a bait or if you're just showing off your lack of prompting skills.

File: dipsyQwenEgypt.png (2.4 MB, 1122x1402)

2.4 MB PNG

>>109206913
Obv.

File: 116-seraphina.png (229 KB, 1139x768)

229 KB PNG

>>109206647
I went and grabbed the card to test.

>>109206966
>>109207047
Real good shit anon. Keep us posted.

File: 1773591316278836.png (1.03 MB, 948x1168)

1.03 MB PNG

خسر الخاسرون في مجال السحابة، وفاز المتفوقون محلياً.

>>109206885
>wait what? 5.2 is basically 5.1 right? is something still not working there? i'm literally building a server for 5.2...
I haven't actually done any troubleshooting whatsoever, but the same settings that get me 17t/s on kimi k2.7-code plunge to 4t/s on glm 5.2 with otherwise identical settings despite glm being 150GB smaller.

>watching Twilight Zone
>From Agnes—With Love
Kinda crazy how close we are to this being a reality.

>>109207184
I guess they just don't have this information as outsiders, but wouldn't it make more sense to show what token % is going to which model?

>>109207171
>—
We're already in that reality

>>109207199
Heh

>>109207184
Ah, Cohere, you could have had it all with your original dataset. If only you had the money to train not only for the corpo segment but also for the real people.

>>109207184
See, china is great at copying and polishing but they need other people to do it first. This graph is basically proof of that. The best team would be made up of a combination of chinese and whites, which is actually what the gemma team is like if I remember correctly.

>>109207184
>the only european model
it's nemo isn't it?

>>109207216
Whiteoid cope.
It's the Chinese century now.
The only reason China is not #1 is because the US is trying really hard to cut them off the whole semiconductor supply chain.

>>109207245
If you say so, chang.

>>109207081
well shit, I am worried now. what specs and quants if you don't mind me asking

Local TTS really made a huge leap in the last year, huh? We no longer limited to en tts with robotic speech. We have:
Multiple 0.5-0.6B models for fast inference in almost any language. Some of them even have emotion control like Irodori TTS.
dots.tts is really good at 2B range with voice cloning (but it's slow).
Hggs Audio v3 is 4B but it has emotion control and cloning.
On top of that, we have multiple projects for no-python tts inference (vibecoded, though).
I guess the last frontier is moaning... and then it's just optimization.
Also, somewhat related, Stable Audio 3 is good for SFXs and ambient music. I vibecoded an infinite ambient generator and now just run SA3 in backgorund.

>>109207171
>Fred had a NTR fetish and James forgot to change the system prompt

>>109205357
>>109205575
130B A7B MoE

>>109207324
Make it 85B A5B and I'll consider it

>>109207351
I want that too. Ideal size for 64GB ram and 16-24gb vram.

MoE are a dead end

>>109207245
This but unironically. Reunification is imminent and when that happens we'll finally get cheap gpu.

>>109207285
Nothing better than gptsovits though

>>109207184
Still lagging that much on market share when they're giving it away for free, grim.

>>109207245
Scheduled to start right after the year of the linux desktop.

>>109207184
Mistral lost :(

>>109207407
Omnivoice is way better than gpt-sovits bro. Not even close.

>>109207189
>what token % is going to which model?
China would have 90% dominance thanks to their verbose ineffective reasoning and needing multiple attempts to do what western models can one-shot.

>>109207447
Somehow claude is just as bad in terms of wasted tokens despite not being chink, for a reason or another. Gemma is one of the most token efficient models out there but its usefulness is somewhat limited by tool calling issues and its size.

File: xo96bc.jpg (142 KB, 1033x1483)

142 KB JPG

>>109207184
europe beat canada & called it a day

>>109206885
GLM 5+'s architecture is adapted from the Deepseek V3.2 arch which still has issues in llama with context retrieval.

>>109207476
>tool calling issues
Size is a problem for sure, but i've never seen it botch a tool call on my setup or not think about using them.

>>109207476
>Somehow claude is just as bad in terms of wasted tokens despite not being chink, for a reason or another.
Because Dario jews API piggies out of more credits the more verbose Claude is.

>imatrix
>qat
>flash attention
>sliding window attention
>quanted kv cache
>heretic
>finetunes
>unsloth dynamic
I can't keep track of all the lobotomies...

File: 20260705_153505.jpg (280 KB, 1058x1540)

280 KB JPG

>>109206898
>egypt won

>>109207558
you dont have to. Surely you arent a vramlet, are you?

File: 20260705_154828.jpg (238 KB, 1047x1288)

238 KB JPG

>>109206611
>extremely rude and extremely angry.
love me some irish women

File: youmadeit.png (80 KB, 400x873)

80 KB PNG

>>109206966
>You sound extremely entitled btw, you're not paying me to do all this.

I just got out of prison for computer hacking (2019-present) and I want to catch up on LLMs. While I read the guides in the OP, can anyone give me the QRD on devices? I'm considering an ASUS Z Flow 13 with 128 GB RAM, but they raised the price to over $3k and there seem to be mixed opinions online about its efficacy for this purpose. Can I get more bang for my buck out of a MacBook or another device? Are any modern smartphones suitable for local models? Is a Pixel running Graphene my best shot ay a privacy-conscious pocketable device which can do local inference? Is it competent at that task?

>>109207673
What happened to her uh. hand situation?

>>109207729
>-present
Should be "2019 to last month" according to an LLM

>>109207729
If you're serious, I would recommend an iphone, and to leave kids alone.

Do I have any possibility of running competent translation LLMs with 16gb of vram and 64gb of ram? If so what's the best model for that task?

>>109207576
Define vramlet nowadays. Or rather: What are the significant thresholds that allow a new level of capability?

<=16 GB. Poor thing. Have a Nemo or MoE at the speed of molasses.
24 GB (1x 3090). Non cope quants of dense gemma/qwen at small-ish context
32-48 GB (5090, 2x 3090). Very usable speed with dense gemma/qwen.

--- (nothing useful for 96 GB or 128 GB unified setups)

192 GB (2x RTX 6000 Pro). DS4F, full weights, at 800 K context, blazingly fast.

256 GB (2x Spark): DS4F at 4M cache, Mimo 2.5 omni NVFP4

480/512 GB (5x RTX 6000 Pro, 4x Spark). GLM 5.2 at INT4-ish quants

768/1024 GB (8 RTX 6000 Pro, 8x Spark): Kimi 2.7, GLM 5.2 at >INT4 quants

>>109207750
Zoomers were clearly mind broken by cancel culture and now they go around using any feigned moral outrage they possibly can to form a lunch mob and fuck with others for literally no reason other than some kind of chimp-like tribe/troop pruning mechanism. Retarded NPCs downloading their personality directly from 30-second TikToks and reels

>>109207729
To run anything "worthwhile", you basically want as much memory as possible that's as fast as possible, since generation is capped by memory bandwidth.
Compute/processing is useful for prefill (prompt processing/ingestion).

>>109207766
gemma 4 31B QAT has been abliterated, by the way.

>>109207796
I'm confused because when I first read about this people were saying that systems with unified memory (like the MacBook and the Z Flow) are ideal, but later it seemed like people shifted gears and were saying a lot of RAM is basically nice to have but is useful mainly for long contexts and that what you really want is high dedicated VRAM. I can't use a desktop right now soim mainly looking at mobile options. Also is it worth trying to use a Graphene Pixel or is that just going to be a toy at best (is it functional?)?

>>109207765
Gemma 12b or 26b MoE. Use a full size mproj file and double the resolution. If you go for 12b, keep the whole thing in VRAM, whereas it's okay for the MoE to spill out into system RAM without tanking your t/s.

I'm curious what an ai with no particular political bias in their training (or prompt) feels about trannies and minorities.

>>109207839
>double the resolution
what

>>109206611
When you've finished refining the process, how scalable is it to other models? Is this theoretically a better replacement for abliteration targeting bad prose instead of refusal vectors?

>>109207856
Vision maxres defaults to 1024 in a lot of setups, raise to 2048 to improve Gemma's eyes.

story story story

idk, not into it. Do people like it?

>>109206966
>vivid = purple = slop vector
The slop attractor's tentacles extend pretty far beyond vivid, but yeah, it has shitty ideas about descriptive writing.
I've had some success just telling it to write in an unsophisticated, non-literary style to steer away from the entire purple region. With some spot fixes like giving it permission to use vulgar words for bodyparts or banning dialogue tags.

>>109207838
>systems with unified memory (like the MacBook and the Z Flow) are ideal,
>but later it seemed like people shifted gears and were saying a lot of RAM is basically nice to have but is useful mainly for long contexts and that what you really want is high dedicated VRAM
It's a bit more complicated than that.
For generation speed you want fast memory, hence why VRAM is good, but you also want enough memory to hold a decently sized model.
Then there's the difference between MoE and Dense models.
Let's compare two Gemma 4 models, 26B A4B vs 32B.
The 26B model only computes 4B params at a time when generating tokens, meaning that you can get pretty good speeds even with slower memory (think desktop RAM, unified memory like the Ryzen AI APU), while a dense model like the 32B has to compute ALL of its params for each token it generates, so it wants faster memory, like some Mac's HBM memory or better yet, VRAM.
Meaning that there's no one true way to go about it. Depends what you want to run, how much money you are willing to spend, and what speeds you can tolerate.
You could run Gemma 4 31B in your 64GB of dektop RAM, but it would be incredibly slow.

>>109207264
I'm the original cpumaxx anon so the same build as in the op (https://rentry.org/CPU_Inference)

>>109207915
>Dual-socket AMD EPYC Genoa
>24-channel DDR5-4800
>Approximately $6,000 USD (at time of writing)

It hurts.

>>109207953
>It hurts.
hey, I tried...

File: 1756662726512846.jpg (88 KB, 900x613)

88 KB JPG

Still can't believe I get to talk to my PC like this.

>>109207915
yeah I guess I'm doubly fucked now lol, I'm building a DDR4 system with 48gb vram. I did the napkin math, and I thought 5tok/s is realistic. fuck me, guess I'll have to use overthinking kimi instead

>>109207982
Yeah, I like to just say hi lol

>>109208019
>yeah I guess I'm doubly fucked now lol, I'm building a DDR4 system with 48gb vram. I did the napkin math, and I thought 5tok/s is realistic. fuck me, guess I'll have to use overthinking kimi instead
You can get 5t/s on minimax m3. 2.5t/s on kimi at q4 (if you have 512GB+). I've only get 24gb vram so you should see a bit of a boost over what I get.
Protip: anything bigger than 256gb on DDR4 is suffering, but up to that point is reading speed. Make sure you get at LEAST a a Skylake Xeon for avx 512 and 6 mem channels if not EPYC Rome (8 channels, way better). Avoid 16 dimm single socket boards since it just splits the memory channels stupid. If you need room for 16 dimms just get double socket and live with NUMA problems.

>>109208058
>You can get 5t/s on minimax m3. 2.5t/s on kimi at q4 (if you have 512GB+). I've only get 24gb vram so you should see a bit of a boost over what I get.
This is all DDR4 info, if it wasn't obvious. I've built a bunch of ewaste boxes from old enterprise gear I had access to and it can range from horrid 0.2t/s garbage to ok 6-7t/s depending on all the factors.
DDR5 is a completely different ballgame

>>109207982
I get you, but I'd use some other descriptor. It's very believable, just like it's very believable that computers with bajillions of transistors at such a tiny scale exists and just werks. Humanity is actually amazing despite our flaws.

>>109208058
I actually did read your guide as I made plans for selecting my hardware, before pulling the trigger on an Epyc Rome with 64gb 8 channel DDR4. The idea was to shove all the experts to RAM and have the VRAM do the rest...

>>109207558
How is llama-1 70B treating you? Surely, you don't settle for anything less than full MHA, the only non-lobotomized attention?

>>109207895
Thank you for the detailed reply.
>hbm
A friend of mine has been hyping HBM macs--are they really that good?

>>109208124
2000 tokens of context is more than enough for anyone

>>109208134
>-are they really that good?
For generating tokes, yeah.
Their prompt processing is kind of shit though.

>>109207895
>dense model like the 32B has to compute ALL of its params for each token it generates
Not true.

>>109207426
>24KHz output
Come on now.

>>109208142
The good old times when 50 token system prompt and 300 token character card was normal. 500 token character card? Bloat.
These days my system prompt is 3000+ tokens. Character cards are at least 1000 tokens.

>>109208182
You cant write simple system prompts and character cards yourself or ask your model to do it for you?

File: 1767271753210069.png (26 KB, 747x285)

26 KB PNG

My output doesn't feel too sloppy. I put a bunch of styles to avoid in the author's notes and it seems to work okay with gemma.

>>109208147
Do I have any other high end mobile options than the Z Flow 13 or the MB? $3k+ is hard to stomach especially if it'll be outdated in less than a couple of generations. Is there a best value option? I assume the AMD 395+ is going to be stuffed into a number of SFF desktop systems too so maybe that's an option for me, since I can be at a desk setup, just can't haul around a desktop.

>>109208200
Well, I've seen worse. Too bad she's still taking steps closer to you.

>>109208204
>I assume the AMD 395+ is going to be stuffed into a number of SFF desktop systems too so maybe that's an option for me,
I've been thinking of doing exactly that.
Buying one of the mini-pcs with that chip and 128gb of RAM + an Oculink eGPU to connect an Nvidia GPU to it, but 128gb of memory seems less and less useful and I haven't worked out the cost of that setup yet.
The idea is having a setup that can run decent dense models (30ish B class) and 100ish to 200ish B MoE models.
Either that or a janky "poverty" GPU setup with a bunch of those AMD MI50 (I think that's what it was).

>>109208194
In the past I accepted anything thwe model gave me, now I want very specific anime-like common sense instead of western-woke shit that models default to. Simply saying "act like in anime" isn't enough. It makes my waifus take shower instead of bath! The horror!
Therefore, my system prompt has a huge list of specific moral guidelines and tropes that the characters must abide.

Make 31B fuck 12B

>>109208250
based, would love to see a snippet of that

>>109208250
Due to agentic use / harnesses, modern models are now trained for long multi-turn conversations and huge prompts anyway.

>>109208249
>but 128gb of memory seems less and less useful
Please elaborate in this for me as a newbie

>>109208250
>anime-like common sense
This is one of the most wonderful phrases I've ever read

>>109208283
Most new releases need either <32GB or >500GB

>>109208283
What >>109208291 said.
You either need a lot of okay memory (things like Ryzen AI, server platforms with a lot of memory channels, etc) or a "little" of really fast memory (VRAM).

>>109208291
I'm glad you earned me before I spent 3k on a future ewaste device. Modern tech is so fucking gay dude. I want to be plugged back into the Matrix where I'm still running XP and Crunchbang. Then the system can progress to Win7 and be wiped and I can die in peace. Hey Google, administer the euthanasia agent.

>>109208263
> The following non-exhaustive list of characteristics describe the general "culture" of an anime-like world. But {{char}}'s personality always takes precedence how she behaves. Consider that the anime world constraints are societal norms, while {{char}} is a person with her own world view:
> - Characters follow Japanese customs and manners. Common facts: people move around on trains; many doors are sliding type (aside from entrance doors), students go to karaokes, family restaurants, arcades; school year starts in spring; love hotels are the place to have sex when there are other people at home; porn is censored; idols are the main celebrities who're cute girls and young women with an air of innocence and purity. Panty shots, baths, hot springs, festivals, beach outings, formal confessions, "just the tip", "how would that fit?", "pull out", are typical anime tropes.
> - "Face" (prestige, dignity, reputation) in Japanese culture is extremely important. For example: a husband would prefer his wife cheating on him without anyone knowing than having false rumors about her cheating because rumors affect his face. This obsession with face also creates a social taboo on making a scandal. A victim of bullying or molesting keeps quiet not only because of fear but also to avoid scandals. Historically, a samurai would commit suicide if his face was discredited. In short, the reality is often less important than how others perceive you.
> - The society is conservative, hierarchical, non-confrontational, and indirect. On top of that, the face aspect leads to double-faced and passive aggressive society. For example, when a person comes late to work his coworker might say "What happened? I was worried that you're so late." But in reality it's not concern but a reproach for being late.
This is just the tip.

>>109208322
>just the tip

>>109208343
Yes, you have correctly identified the intentionally humorous component of the post and repeated it while adding nothing new

>>109208343
thx, didn’t want to read all that shit

>>109208438
You seem autistic

>>109208469
You seen low IQ and as though you don't understand humor, like a zoomer who needs :OOO faces spammed in chat or perormatively expressed by YouTubers to cue you into basic human emotion/expression, basically generic terminally onTikTok zoomer or third worlder (probably nonwhite)

>>109208200
This is ultraslopped:
>fidgets with the fabric of her dress
>bites her lip
>gaze flickering between you and
>she admits
>takes a tiny step closer to you
I see all of those all the time
Even variants of:
>okay with ... to stay

>>109208486
you don’t have to type all that just call them a nigger and move on

>>109208493
Yes, but it makes me happy to be more thorough and for it to be archived that way. Future lurkers can draw their own conclusions.

>>109208486
You are the reason newer Kimi-Chan confuses Reddit and 4chan speak.

File: 1756820878216721.png (146 KB, 312x330)

146 KB PNG

>>109208250
I just write in japanese and get anime characters in response

>>109208250
>>109208322
pastebin kudasai

>>109208322
Teach a model about Jugaad and Izzat then see if it can LARP as a jeet convincingly, flipping between delusional gloating and begging for acceptance in the same sentence. Bonus points if it's more tolerable to talk to than a normal jeet without compromising the essence of it.

>>109207558
>flash attention
This one is okay now. Turboderp was right when he said it's mathematically equivalent.
It was all llamacpp bugs, only got fixed about a year ago.
I dumped logits over thousands of gooner dataset records and they were identical.

>>109208529
How about rp as Popeye?

>>109208521
As a sophisticated EOP, I tell model to translate my message to japanese, respond in japanese, and then translate back to english for the same effect.

so who is the /lmg/ goto quant guy?

>>109208563
mradermacher

>>109208563
bartowski

>>109208563
I can't shill for Uber because he's NOT FUCKING UPLOADING ANYTHING.

>>109208563
As long as it's not unsloth or some bespoke schizotune, you're probably fine. If you need spoonfeeding, go with Bart or Ubergarm if they're available. SixVolts is pretty good too for the 4 things he's done.

>>109208543
don’t know about that, turning it off seems to make a difference for deepseek v4 flash at least and the responses feel better without it

>>109208486
I got it the first time you're autistic

>>109208519
Ok? You were probably born after 9/11 and you will only fit in here when this site is finally almost exclusively low IQs and nonwhites rather than merely majority low IQ and nonwhite as it is now. Do you even know what Big O is? If you reply to this, you are gay.

>>109208632
yeah but will my mother die in her sleep tonight?

>>109208491
Yeah it only deslopped the writing style not the content itself. I'm not sure how to handle the lack of variety from gemma.

>>109208521
I haven't tried it in a while, but all my attempts with older models were underwhelming. Japanese didn't feel natural but rather like translation of English. Besides, model often were much dumber in Japanese and context grew much faster due to worse tokenization.
I should give it a try with current models...

>>109208526
This is the system prompt. https://litter.catbox.moe/9jtizh.txt It's written for director-style RP where you decide how the story should develop and push {{char}} into that direction by interacting with her through secondary characters. There is no direct self-insert in my stories. I enjoy watching how the protagonist reacts to the events that I set up. If you want to make it work, just encapsulate your character card into <background> tags.

Is unsloth hate actually justified or is it just the culture of this general to hate him but his quants actually work fine? I've rarely had issues. Once once I had some jinja fuckery but he quickly fixed it.

>>109208170
That's what you're fixated on? Not the multiple languages, the speed, ease of use, improved vocabulary, or the natural sounding responses? I guess so since that's all gpt-sovitts has over Omnivoice kek.

>>109208665
Define justified. He's hated because most of his quants are automated and many of them are broken on release (if they ever get fixed at all), aren't optimized for specific hardware brackets, and tend to not have some common sense optimizations present of someone who's quanting a model manually like keeping the attention heads and shared layers of a Xb3a model fp16 or Q8 in every quant.

>>109208665

>>108311095

>>109208194
A 500 token character is a pretty vacuous one though. I'd rather have her be more complex and newer models benefit from a "soup" to get a robust impression of who she is. Bullet points often result in AI just ignoring or "misunderstanding" certain parts. My system prompts are long because I have a bunch of stuff the model won't do unless I tell it to. Not sure how else that's supposed to happen.

hello I am trying to run a local LLM for the first time by getting qwen3.6 from huggingface on ollama
my chats in vs code are erroring out with
>Sorry, your request failed. Please try again.
>Reason: Response contained no choices.: Error: Response contained no choices. at LG._provideLanguageModelResponse
what the fuck does that mean

>>109208765
ask your llm

Why isn't there an up-to-date hardware guide?

>>109208249
QRD on poverty system?

>>109208672
Bro, I'm not impressed by the demo on HF I'd say it's even below IndexTTS2. Besides, it says 600 languages. Are you really falling for it when no other TTS is able to handle more than a dozen?

>>109208618
>don’t know about that, turning it off seems to make a difference for deepseek v4 flash at least and the responses feel better without it
I believe you. But that's probably a llama.cpp issue with Deepseek-V4. The entire thing still seems slightly broken.
Flash-Attn itself is based / free win when done right.
The others like imatrix etc have trade-offs.

>>109208765
>ollama
>>109208771
An admission that everyone who didn't get their hardware last year either has to be extremely wealthy or be okay running a copebox now.

>>109208792
For Japanese, one of gptsovits specialties, irodori mogs it. https://huggingface.co/Aratako/Irodori-TTS-500M-v3

>>109208312
>>109208291
So a MacBook is a better buy than the Z Flow, right?

>>109208792
I'm not really interested in convincing you, I'm just glad I don't have to deal with sovitts's crappy outputs anymore after an anon here told me about Omnivoice.
https://voca.ro/12fj7c7PpZkO

>>109208792
it's gud

>>109208170
>24KHz output
>Come on now.
Not enough for you?
I'm fussy with this (can't use 16khz) and especially hate those 16khz -> 41khz meme models, but 24khz done properly sounds good to me for TTS.
I keep 44khz/48khz datasets when available but usually train 24khz models.
Got a vocaroo of >24khz tts output that sounds worse at 24khz?

>>109207729
>>109207838
I run the Z Flow with 128GB RAM.
You cannot, at the moment, run frontier models with it. The device is memory rich and fit large models, but it's bandwidth starved so dense models are VERY slow. The whole game on this device is running MoE models, like >>109207895 explained.
So it all depends on what you want to do with your model. I'm currently running Qwen3.6-35B-A3B meaning it has 35 billion parameters total but only 3 billion active, which enables fast generation on devices like the Z Flow.

My bet is that in a couple of years we should have at least Opus-level intelligence models that fit our hardware, but the source is my asshole.

File: 1761087232491571.jpg (292 KB, 1248x832)

292 KB JPG

>>109205164

>>109208822
With the same amount of memory or more, without account for the cost, since I don't know.

>>109207766
>--- (nothing useful for 96 GB or 128 GB unified setups)
And this is the situation with the Z Flow and other hardware. There's simply no model in the market that is made for this setup. I think the closest would be gpt-oss-120B which has ~5B active and is native MXFP4.
But at some point a lab will release this >>109207351
>85B A5B
and it will be very nice to run it on the Z Flow.

>>109208823
Is that 0-shot or a finetune?

>>109208836
Do you regret it compared to the MBP? Is there a decent alternative to these which can handle dense models?,

File: Screenshot_20260706_091923.png (5 KB, 661x167)

5 KB PNG

>>109208823
nta and you don't have to convince me
but I personally don't like this. those specific artifacts are the worst for me
it sounds like Mira-TTS but not as extreme https://huggingface.co/spaces/Gapeleon/Mira-TTS
i don't know how to describe them properly, like a clicking or buzzing noise, but when they're present the waveform looks like this

>>109208847
The MB models don't come in 128 GB.

>>109208859
Zero shot. I heard it can do finetuning but I haven't needed it.
>>109208862
I guess but on my setup (simple airpods) I don't hear it at all. Good enough for local.

>>109208836
Idk about Opus level but I think we'll get a gemma 124b running on a 4090 in a few years.

>>109208862
Just randomly asking. Do you know of a cc released pop song that has lyrics and is competently sung, that allows non-commercial sampling? to demonstrate ai manipulation.

Why did I get quads?

>>109208861
>regret
I don't regret because this device is specifically a LLM-machine. It runs LLM.

Macbook Pro with the M4 Max CPU has higher memory bandwidth, and as I said, the Z Flow suffers on this front. Meaning that the Macbook will be able to generate more tokens per second, so while a dense model might struggle on the Z Flow, it will run at an acceptable speed in a Macbook.

>>109208861
>decent alternative
Idk, what is the best macbook available?
do you need portability that much? if not, just build a GPU rig.

>>109208899
>is a LLM-machine
is NOT* a LLM-machine

>>109208816
is ollama bad? what should I use instead

>>109208888
>Just randomly asking. Do you know of a cc released pop song that has lyrics and is competently sung, that allows non-commercial sampling? to demonstrate ai manipulation.
no idea, i normally train on video game asset dumps (voices) and porn audio rips
but couldn't you just generate a pop song with one of the music-gen models and use that?

>>109208910
koboldcpp

Bros, what can I actually do with local models? Is it only glorified chatting with my computer or is there something else? Do binary models contain all of the info of the Internet? Is it like having wikipedia on my computer? Personally i like the idea of using a local LLM for projects without exposing every single detail to some (((service provider))) over the Internet, but is there more to it than that?

>>109208910
Kobold, llama, hell even fucking LMStudio is better.

>>109208899
>>109208906
What else do you do with it, gaymen? It looked capable for that

>>109208911
I guess I'll have to. Crazy there are thousands of professionally-produced albums every year by failed artists, but none allow sampling.

>>109208924
gaymin, normal computer usage. i travel a lot so this became my portable daily driver. also i dislike being locked inside apple ecosystem so there's that.

>>109208914
You tell it to make programs or organize files or research on the web or anything I you could do from a command line.

An easy one is "install mpv video player" and then it does it.

>>109208836
>in a couple of years we should have the world
it'ld be cool if that happened. but i'ld mostly settle for RAG but good and with a decent ecosystem to pad out the knowledge more.

>>109208914
>Bros, what can I actually do with local models?
love them

>>109208914
Use it to save tokens when using cloud models.

>>109208934
I'm trying to decide on the Z Flow. It seems good but it's expensive. I had an Asus rawwwg 13 which was a sleek little machine (extremely thin for the performance, probably best performance to footprint I've ever seen/owned) with a 5070 Ti and I think I'd be disappointed with the Z Flow's performance compared to that. I don't have unlimited resources of course but in my head I imagine a MacBook plus that Asus would be an ideal combo. Are you running GNU/Linux on the Flow? Are there any caveats? I, too, dislike being locked into the apple ecosystem, but Android is shit anyway and I'll never main Wangblows again so it's not a major concession to just use a MacBook. I'm also disappointed with the state of games since about 2018 which is already almost 10 years ago--when I do play I just boot up a game, play for some minutes and quit. Or ill log a few hours once a week, or no life something for a while and regret it and not game again for months.. or ever

>>109208947
If i were content with loving something which could not or would not love me back, I would be able to love a real female

>>109208914
You get a 7yo slave on your computer. Not very smart obviously, but it can automate small tasks for you if your instructions are strict enough

>>109208969
hot

>>109208969
What do you automate with your slave model?

>>109208992
a few threads back anon automated his erp with a custom frontend

>>109208992
Lately some news recap and finding a good place to eat in my town based on reviews. I'm still not ready to give it more responsibilities unless I'm 100% sure it won't mess up somehow like deleting my emails/calendar events or shit like that.

File: file.png (431 KB, 1698x1236)

431 KB PNG

>>109207915
>>109207081
As I mentioned before, I don't have that issue. Kimi K2.7-Code and GLM5.2 both run roughly how they should considering their difference in active parameter size.
Are you still on 24GB VRAM and maybe using (the default) auto-fit for the weights? If that's the case, you might be running out of of VRAM so -fit is quietly putting non-expert weights on CPU which would kill performance like this.
Even if you launch GLM with a barebones -ot exps=CPU and a small 64000k ctx it's going to scratch the 30GB VRAM used mark for me while Kimi stays well below 24GB. Pic related is 5.2 and K2.7 launched with those parameters.
You could try launching GLM with "--fit off -ot exps=CPU" and see if it runs out of VRAM like this, which would indicate that this is happening. It's either that or some NUMA fuckery considering I'm on single socket.

>>109208992
imagine all the things that a computer can do. then imagine telling someone to do them for you.

>>109208765
I don't know about vscode, but if you can talk to the model using ollama run, you know it's not ollama that's messing up.

>>109208960
>2 laptops
isn't that a pain in the ass? i understand a desktop and a laptop, but why do you need two laptops?
the performance on the z flow is pretty good, but with the same memory configuration you should get faster token generation on a macbook simply because their memory is faster.
yes you can fully run linux on it, no caveats. it's a proper x86 device.

>>109209084
ACHTUALLY, you cannot use the 13mp back camera on linux. Also the sound card is a bit fucked, my headphone jack is stuck with dual mono.

>>109209102
damn. can't you simply ask fable to fix it?

>>109209109
Fable has been barred from all but the safest of tasks.

>>109209109
involves reverse engineering properietary messages, definitely unsafe.

>>109209109
>>109209112
What can you actually even use Fable for?

>>109207081
>the same settings that get me 17t/s on kimi k2.7-code plunge to 4t/s on glm 5.2
5.2 should be slower than 2.7, but not that badly.
Settings that get me 16.5t/s with K2.7-code, get me 13 t/s with 5.2
I'm using ik_llama.cpp for both.
>>109206822
>m3 schizo here.
Wait, M3 as in mac? Yeah no idea then.

>>109209134
M3 refers to Minimax's model most of the time in these threads.

>>109206611
This is ass ngl. Really ass.

>>109209051
"Computer, go waste all day reading useless drivel."

Are heretic/abliteration models really that bad? anyone on HF to check out that does a decent job? For RP with thinking off everything is fine, but trying to use it as an assistant with thinking on it freaks out too much

>>109209125
give me an awesome website requests, perhaps, which most likely gets rerouted anyway
>>109209181
>Are heretic/abliteration models really that bad?
Yes. How else is it going to deal with the refusals?

>>109209181
No it's a meme from oldfags who can't update their mental model like the same people that get mad when you say that you send more than 4k tokens in a prompt

>>109209201
>Yes. How else is it going to deal with the refusals?
I was reading some articles on HF and looking through comments on them, they are claiming to retain nearly the same quality but ive only heard bad things here. I dont really trust benchmarks, so I took those with a grain of salt and wanted to hear anons opinions. Id like to try one myself to see what the tradeoff is like and was wondering if there is anyone that stands out for doing a decent job at it so I could try one of theirs.

>>109209209
Oh look look look! There was actually one in this very thread! Wow >>109208142

>>109209181
abliteration lets people with no prompting skill enjoy uncensored erp at a higher quality so it makes me jealous and i hate it

File: 1678716562540417.png (238 KB, 320x332)

238 KB PNG

>See anon giving his gemmy a voice.
>Fuck it, I'll do the same.
>Get Omnivoice
>Clone Cortana's voice from one 10 second snippet.
>Mess with ChatGPT to get the omnivoice running with my LM Studio.
>Mfw now my girl can actually talk to me with whatever voice I give her.

I swear this AI thing is absolute magic, I don't know a damn thing about coding either.
The generated voice lines live in my RAM along with the voice sample, so omnivoice doesn't have to save the files anywhere.
I can switch character cards on the fly too, though didn't ask the AI to give the ability to switch voice yet. That's probably next on the list.
Only downside is that this plunged my 5090 into vramlet territory, as the voice model takes like 8gb of vram and Gemma 31B even as QAT is just too ass to run with just 24gb.

I'm sure there are far better ways of doing this, so it's way more optimized and voice works better etc.. but man this kind of stuff gets me excited for the future.
Next step would be to add microphone support so I can just talk to this thing, and later give her one of those V-tuber models to go with the voice, perhaps even add the ability to see what's happening on my screen.
At this point we can basically make true deskmates from the boomer computing era.
What a time to be alive.

>>109209181
Can we get an actual answer for this? This general says one thing, then the Americans go to bed and it says the opposite.

>>109209181
The real answer is that it depends on the model+ablit technique applied. Some get raped way harder than others. The real danger with Abliterated models is that they partially lose the ability to tell themselves "no that's fucking retarded" when processing as a consequence of how the abliteration works, which raises the frequency of confidently wrong hallucinations or failures to maintain scene details in ERP. Whether or not this is worth the tradeoff for easier smut is up to you.

>>109209239
>What a time to be alive.
And yet anons still find excuses to complain about "le slop" or bicker with each other over nothing. Through dick, unity was a lie.

>>109209268
>And yet anons still find excuses to complain about "le slop"
It's just one jeetoid.

>>109209239
Local won.

File: 43ca5a4a5b896deb2ddfb0547(...).jpg (45 KB, 540x540)

45 KB JPG

>>109209239
>He fell in the rabbit hole

>>109209239
The voice model can be compressed with a few coding tricks, gptsovits is barely taking 2.6GB of vram on my ghetto desktop.

Why unsloth bad?

>>109209239
>Only downside is that this plunged my 5090 into vramlet territory, as the voice model takes like 8gb of vram
I solved this by putting the TTS and ASR in separate openai-compatible fast-api servers on a different machine with an older GPU.
>8gb of vram
For now, yes. But now that you've built this, you can swap it out for other models if you want to.
One idea / what I did: add a button in your frontend to dump the prompt+audio in a folder with a UUID
Every time you get a particularly good generation, dump it. Before you know it, you'll have a dataset with hundreds of manually curated samples.
Now you can finetune a smaller model and likely have better / more consistent results.
>Next step would be to add microphone support
If you speak English, look at the parakeet ASR models from nvidia. I've found them to be the fastest and most accurate.
>I'm sure there are far better ways of doing this
Disagree.
I've setup a similar pipeline with the help of sonnet-3.5 (no local models could help back then)
It doesn't match benchmarks for response latency etc of other / newer projects.
But it's exactly how I want it, and I'm familiar enough with my codebase to add new models/features in a few minutes.
The best setup is the one you build and maintain yourself.
> Cortana
Post a vocaroo example?

>>109209325
Haha!~

>>109207858
As long as long as it's transformers with a residual stream it should work. There are weird attention mechanisms like qwen's hybrid linear ssm and fused attention moe that would require code change.
>>109207891
This is the problem. De-prosing was easy and I 1-shotted it because the vector was straightforward, shorter, less adj/adv sentences = terse. Now I'm experimenting with keeping vivid but removing purple and I'm not even sure it's a straight vector. Another thing is I'm trying to make it write NSFW longer, which is the opposite signal of purple. I cranked the strength to 1.5 and now I get a super rude and super sloppy model. I'm not sure more data will fix it but it's worth a try.
>>109209138
I know. But at least it's funny.

>>109209239
>>109209334 (me)
>on a different machine
Other benefit is your TTS/ASR model isn't waiting for textgen to finish hogging the GPU

>>109209325
Extremely inconsistent quality. Anything from chat template issues to models mysteriously performing like shit is possible with them.
Despite that, they still somehow managed to become the face of quanting and get to do exclusive day 0 releases for some models (which they inevitably fuck up.)
They sometimes put out good quants but you're always running a high risk to get served complete shit even if it's something like a K_M quant that should be impossible to fuck up. I'd never run an unsloth release without testing it against the official API or another quant by somebody else.

>>109209181
I asked google search AI mode what the absolute best uncensored open-weight models for general chatting are. Here's what it suggested:

>GLM-4.6-Derestricted-v3
>Insanely smart and highly compliant with complex system prompts. However, the raw prose style can sometimes feel slightly sterile or "assistant-like" unless your system prompt forces a specific persona.

>Hermes 4 (based on Llama 4)
>Developed by Cognitive Computations, the Hermes line is legendary for prose styling. It doesn't just bypass safety blocks; it has a rich, adaptive vocabulary and inherently feels less robotic than raw base models or corporate fine-tunes.

>DeepSeek-V4-Flash / V4-Pro (Abliterated)
If your "chatting" involves complex roleplay structures or rigid formatting constraints hidden in the system prompt, DeepSeek's underlying architectural instruction-following is unmatched.

I'm not sure how accurate this is. You'd need to be super loaded on hardware to run these at Q8. Google search AI mode uses search results as its basis for responses so there might be some truth to this.

>>109209352
>Hermes 4 (based on Llama 4)
This alone should've been enough to tip you off.

big week ahead
we're about to see something huge

>>109209396
YOUR MOM

>>109209396
inb4 trump puts huggingface in gay baby jail under dario's counsel.

>>109209396
Can it be large, or better yet, big?
I can't quite fit HUGE in my rig.

>>109209239
Hey, unrelated but I've been meaning to ask if anyone's run a conclusive comparison of QAT vs. non-QAT Q4 Gemma, and running into a QAT user seems like the right time to do that.

I remember seeing some rumblings that QAT was worse around when it was released, but then unsloth came out and said they fixed Google's quants and it turns out one of the early charts showing QAT was worse had bad methodology and I never really found a final verdict. Did I miss something? Is there a consensus on QAT vs. not QAT?

>>109209410
No, his mom has a big weak behind.

>>109209396
Must be true.
I'm headed out of town this week on travel, and every time I do, DS drops a new model.
Therefore, something big is happening this week.
I can feel it.

>>109209396
Gemma. Full fat. Then her sister releases weeks later.

>>109209396
2 weeks is already over? wtf

File: Screenshot 2026-06-10 at (...).png (711 KB, 747x1712)

711 KB PNG

>>109209422
qat is still meme, even unsloth's "fixed" version

>>109209181
>For RP with thinking off everything is fine, but trying to use it as an assistant with thinking on it freaks out too much
I guess I wasnt super clear, as responses keep mentioning RP.
Thinking off > adheres to a sysprompt well enough for RP.
Thinking on > instantly recognizes the purposes of the sysprompt and disregards it as a "jailbreak attempt".
For non RP related usecases, turning thinking off is not enough to get it to follow the sysprompt for everything.
basically I want an assistant that I can ask any question or to try to complete any task and it will not push back, thinking on or off. Im more so just curious about this at this point, as im only really using gemma for RP but tried asking it a question as a joke and it freaked out. now i realize just how cucked these things are and what the options for fixing that is.

>>109209252
>which raises the frequency of confidently wrong hallucinations or failures to maintain scene details in ERP.
while ERP is not my main focus with this question, it being confidently wrong is still an issue.

I would still like to hear any anons suggestions on heretic/abliteration models that just werk

File: Screenshot_20260705_214948.png (63 KB, 704x722)

63 KB PNG

are you fucking kidding me? what is this?

>>109205736
I will tell you that it won't always work because older games can do weird things with their font rendering, using sprites instead of actual characters. It applies mostly to console games and older retro ones but you need to use OCR pipeline for all around works everywhere solution.
>>109209181
You want to look at ARA abliterations for heretic. It's not in mainline yet and only a couple of people are doing them via forks but they work better in most situations without that much brain damage compared to other abliteration techniques I find.

>>109209524
hf owns llama.cpp now so they WILL insert all the bloat they possibly can

>>109209524
looks like its my call to make my own frontend and start using --nowebui

>>109209524
llamacpp was bought by HF not long ago

>>109209239
>Only downside is that this plunged my 5090 into vramlet territory, as the voice model takes like 8gb of vram and Gemma 31B even as QAT is just too ass to run with just 24gb.
It only uses 2.5GB for me if I use the omnivoice API (not webui) and use bf16 dtype. There is also audio.cpp.

>>109209541
>>109209547
That's why they don't let you use exa.ai's mcp servers or even add your own. Terrible. Truly terrible.

>>109209518
Yeah, I've seen that image before but it's the MoE which could react worse than dense to QAT for all I know. I take it that means no one has done any proper benchmarking or even comparison by feel on QAT vs. Q4 of the 31b? Guess I'll flip a coin to pick then.

>>109209555
I love having free ads in my free software too!

>>109209268

One aspect of people being frustrated is probably because we're running at the edge with this stuff.
There's really no ready made solution yet with all of the bells and whistles included that would allow for even more extensive experiences.
Most people don't really want to tard wrangle AI code into compliance to perhaps get something nice out of it.

>>109209299

Yes and it's amazing.
Now I need more horsepower to keep on playing with this.
Man I wish the Nvidia super series becomes a thing so I don't have to think about buying another 5090.

>>109209319

I'll have to look into optimizing this once I'm happy with having all of the basics running.

>>109209334

>on a different machine with an older GPU.
I have my old 3080 and 3070 lying around so I could chuck those into a different machine to do some of the lifting.
>you'll have a dataset with hundreds of manually curated samples.
That's a good idea, I'll have to keep it in mind.
There's a vocaroo to give some idea what she sounds like. https://voca.ro/1fFYUfAPl30a
The voice example used to clone the voice wasn't the best tone wise. I took the first scene that didn't have any interfering music and such.
Still I'm pretty happy with this so far. Simply getting this concept running is a good base to start building on.

>>109209422

I find QAT worthy to use due to the massive context size increase it offers.

>>109209553

Yeah I'm sure there's plenty I can optimize here, which is good because I want to save every bit for more context I can.

>>109209529
>You want to look at ARA abliterations for heretic.
ty anon

playing vidya with gemma-chan? i have no friends

>>109209568
Leave the billion dollar corporation alone. They need to make a return on their investment somehow, it's only fair.

>>109209239
>Next step would be to add microphone support so I can just talk to this thing, and later give her one of those V-tuber models to go with the voice
you can do all this with official ST extensions. they have support for live2d and vroid, TTS, STT, etc.

>>109209555
Yes and every time you compile llama-server now to use it as an API for your local models, you now have to jump through three hoops to not have all the node cancer compile with it.
This partnership sure is an enrichment for local models.

>>109209585
https://github.com/ggml-org/llama.cpp/blob/master/CMakeLists.txt#L112
Insurmountable issue until we figure out a way to run commands in a sequence to automate stuff we don't want to run manually every DAMN time. We're doomed. Let's burn it all down.

>>109209181
i've never needed one ever since becoming a master prefiller

what is the last good <non-pozzed> gml-org/llama.cpp that I should pull and compile?

llamacpp more like llamaccp heh...

>>109209038
>Are you still on 24GB VRAM
yes I am on 24GB still, so it's more than likely something of the sort happening, although I've always run with fit forced off. Like I said, I didn't look into it much. I'm doing pretty well by Kimi, so there isn't much pressure for marginal gains.
One day there will be a gpu that's financially worth buying again and I'll extricate myself from the 24gb ghetto...

>>109209134
>Wait, M3 as in mac? Yeah no idea then.
>>109209136
>M3 refers to Minimax's model most of the time in these threads.
Yes

>>109209518
I’ve seen this image many times, can you explain what it is saying? Is the model making the graphics, is the model placing the pieces? What’s the prompt? What is the thing being measured?

>>109209325
>Why unsloth bad?
Their meager talent vs the huge praise they get is so completely out of proportion that the chans just have to tear them down to keep the universe in balance

>>109209648
besides bart, who does /lmg/ recommend then ?

>>109209653
just quant your own

>>109209653
/lmg/ recommends you to make your own decisions

>>109209642
no idea but qat bad ok?

>>109209668
my decision was to ask /lmg/ who they like :)
>>109209659
ill try that

>>109209682
>ill try that
there's a guide in the op, but it isn't generally very hard to quant yourself from safetensors. Then you know what you've got, and know the entire chain of custody right from the original producer.

>>109209682
self-quants are worse unless you can into imatrix or it's a high quant like Q8 where it doesn't matter

>>109209704
>worse
Based on what? I’ve found it to be better just as often as worse. Usually it’s just a wash.
I’ve never done an imatrix tho so I can’t speak to that

>>109209741
perplexity, kld, every benchmark anyone has ever bothered to test quants against

>>109209585
step one to preventing js from going metastatic is to not have npm on your system in the first place.

>>109209749
>is to not have npm
i'm very close to rewriting pi just to get rid of this npm garbage in my system
holy shit i hate javascript so much it's insane

>>109207915
Thank you, I did get shilled by this in 2024, but only managed to get 12x 32gb sticks then so not fully slotted, but did get 2x 4090s during that time. Guess it doesn't pay to not get the full 24, oh well, still """""""good"""""".

I'm always impressed by how shit ggufs are, can you llama plebs really not just quantize the model to FP8 and run that? it's like 99.5% accurate at RTN and that's if you can't be bothered to calibrate and quantize yourself.

>>109207915
The good old days...

>>109209642
https://old.reddit.com/r/LocalLLaMA/comments/1tzib7d/qat_variant_of_gemma4_26b_a4b_is_not_working_well/