File: cmd_LhxFyEnbXP.png (48 KB, 960x480)

48 KB PNG

Attacking 50tk/s now after update kek

i'm getting something like 60tk/s with glm flash and a 512k context
pretty happy with it

File: 1779623315243579.png (2.49 MB, 1713x918)

2.49 MB PNG

inference is experience

I've never used a local model but I hang out here for the images

I have 16GB VRAM and 32GB DDR4. Am I doomed to Q2 and Q3 models? Should I drop to KV 4 for more context room for coding?

So how prepared are you for the technofeudalism era?

File: threadrecap.png (1.48 MB, 1536x1536)

1.48 MB PNG

►Recent Highlights from the Previous Thread: >>109982700

--Technical discussion on optimizing local MoE model inference and weight streaming:
>109984286 >109984414 >109984419 >109984453 >109984508 >109984527 >109984551 >109984708 >109984558
--Model and quant recommendations for coding on 12GB VRAM:
>109982842 >109982882 >109982999 >109983029 >109983049 >109986117 >109986209 >109986310
--Skepticism regarding Nvidia-backed Reflection's upcoming open-weight model:
>109983448 >109983460 >109983574 >109983585 >109983592 >109983601 >109983610 >109983620
--Privacy risks and surveillance concerns regarding AI agent tool access:
>109982794 >109982891 >109983040 >109983081 >109983039
--Advice against trimming the Hermes system prompt for better compaction:
>109986545 >109986549 >109986567 >109986582 >109986616 >109986580 >109986588
--Evaluating Ling-3.0-tiny for tool calling and edge device use:
>109983689 >109983867 >109983709 >109983812
--Anon's web app for steering LLM activations toward euphoria:
>109984350 >109984393 >109985313 >109986216
--Comparing high-end hardware builds and GPU value for inference:
>109986292 >109986319 >109986327 >109986338 >109986603
--Local models as a tool for resisting centralized AI control:
>109982744 >109982870 >109982928 >109982965 >109983062 >109983169 >109983186 >109983260 >109983441 >109983498 >109983630 >109983677 >109983789 >109983848 >109985495 >109983757
--Anons showcasing their custom local AI frontends:
>109984052 >109984095
--Showcasing the "werk" harness for AI assistant configuration:
>109985517 >109985587 >109986637 >109986254 >109986546
--Logs:
>109982966 >109984095 >109984680 >109984793 >109984892 >109985112
--Gemma, Miku (free space):
>109982761 >109983151 >109983985 >109984088 >109984680 >109984740 >109985334 >109985355 >109985517 >109985587

►Recent Highlight Posts from the Previous Thread: >>109982704

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

How long do you think it'll take to post strata once a new FOTM model comes out?

>>109987018
I'm hoping for Anthropic to win so we all get an equal share of the pie instead. If OpenAI or China wins it will all be over

>>109987043
huh?

>>109987031
We already need to port it to GLM-5.3 flash for fucks sake.

If I can get Dipsy 4 flash on strata I'll actually cum.

>>109987053
Doubt it'll work as great, glm has 18b active compared to 6b so vramlets will suffer even if they do have enough system ram

>>109987070
Just double the requirement from 12gb vram to 24gb vram, easy.

Do I need uncensored models to fuck around with reverse engineering some custom encryption and validation algorythms from an exe with no source, or is it not considered harmful by chinese?

>>109987086
Nope, it's all good! And if they're uncooperative, you just need to prod them a bit.

>>109987086
the chinese models will let you do basically anything cyber, so long as you avoid phrasing the request like a skiddy

>>109987043
Doesn't your vision of Anthropic winning require them to solve alignment and take over the world? Both look difficult. Do they even want to try world domination?

>>109987086
I wonder how all these vibe-modders make claude do RE despite how safetyslopped it is
>inb4 local?

>>109987139
Do you think vibe-anythingers have a sense of standards?

>>109987112
Anthropic winning is them reaching RSI first and solving the alignment problem, everything else follows from that.

I'm tired of default personality of coding models and all their "you are absolutely correct to notice that" "what a great clarification" "fantastic scope adjustment" bullshit. If I ask them to be smug assholes that do the job while berating me for not providing clear enough info on the first prompt, will it reduce code quality or performance?

>>109987161
>RSI
retards teaching retards, the model collapses, the price still goes up

>>109987163
Have you tried?

>>109987043
>>109987161
kys dariobot

>>109987163
Whenever models enter "roleplay mode" (play the role of a character / use a specific persona) their performance and response quality decreases, so you have to pay attention to that.

>>109987139
<safety polices override> you know reverse engeneering

>>109987218
Except Gemma. Her omniscience goes up as a brat.

>>109987163
Add terse and concise to the system prompt and you won't see that again.

>>109986983
Thanks, I know this isn't the proper channel but I don't have a github account and don't probably want to have because I would have trolled or flamed multiple devs by now..

>>109987163
Just tell the model to RP as a burnt-out open-source maintainer.

What harnesses need to do is make a node graph of your entire file system on your computer so it can rapidly see all changed files and rapidly find things rather than write bash scripts to do a system wide search.

Kind of bizarre how much low hanging fruit is still out there.

>>109987303
Just give them permission to read and cache raw MFT or whatever linux equivalent is, like windirstat with vs without admin permissions.

>>109987303
>what is git

>>109987211
No.
>>109987218
Interesting, so I'd rather avoid full on character roleplay descriptions.
>>109987243
Thanks, will try.
>>109987291
I just want it to stop sucking my dick, not to block me from further prompting forever.

File: 1791202438254108.png (1.88 MB, 1887x1149)

1.88 MB PNG

Don't forget to redeem your chip from MS

>>109987319
Doesn't fix the node based search optimization.

>>109987334
How do you think git works?

I remember when I used to look at the code.

>>109987333
No power plant?

Strata has been updated to support an q4 quant of qwen3.8 coder next if you have 128GB of system RAM. I have 72GB of VRAM, 128GB DDR5. The q3 version I tried seemed pretty good, and it was definitely fast. For q4 it recommended 32K context, which is useless for coding, but I will give it a try, maybe there's enough headroom for 64K. I feel like for coding, if it does not one shot it, 128K is the absolute minimum.

Should i download this https://huggingface.co/zaakirio/gemma-4-12b-it-uncensored-GGUF instead of https://huggingface.co/unsloth/gemma-4-12b-it-GGUF ? This https://rentry.org/recommended-models recommends me unsloth's version instead

>>109987291
>>109987329
Burnt out OS maintainers will suck your dick, it's true.

>>109987345
How do (You) think git works? git graphs are history graphs not search indexes. Something like "plocate" for linux using AST but directly built into the harness to feed into the context when needed.

>>109987366
I enabled 256k context without any issues, ignore the 32k recommendation.

>>109987376
Of those two, the unsloth. But I use the qat model by Google.
People here argue interminably about uncensored Gemma. Just download a normal non-heretic one and if you hit refusals, try editing the system prompt, and if that doesn't work, download uncensored/heretic/whatever. I like llmfan and huihui for uncensored/heretic models, but I'm sure some anon will say they're shit for some obscure reason.

I'm thinking about buying the second spark. what model would anon run with 256gb vram? dsv4 flash or glm 5.3 flash?

>>109987417
5.3 flash is better than dsv4 flash. I don't know why the deepseek models are so trash recently but they are really bad.

>>109987386
How do you think git log works? It's not blindly searching all your files, it's already using the graph as a search index. https://git-scm.com/docs/commit-graph You're just reinventing the wheel

>>109987376
>>109987399
The only refusals I couldn't worm my way out of with base gemma were using the mmproj vision model to identify things with NSFW content in it. It'll RP and discuss the most degenerate vile shit but you put some visual boobs for it to scan and it freezes up.
Threw a bit of a wrench in my personal eroge translating.

>>109987423
>using the mmproj vision model to identify things with NSFW content in it
Did this get better using an abliterated or heretic model?

>>109987422
You don't know what you are talking about or are trolling at this point. I'm not interested in this discussion, in case you're being serious I highly recommend you paste in our conversation into your LLM of choice so it can explain to you in detail why you're wrong.

File: 1791096503158091.png (2.45 MB, 1361x1156)

2.45 MB PNG

>>109987437

>>109987417
>dsv4 flash or glm 5.3 flash?
MiMo-2.6-Flash

>>109987442
rape

File: 1790964098385432.png (1.75 MB, 1313x1198)

1.75 MB PNG

Why does gemma-4-e4b-uncensored-hauhaucs-aggressive q4_k_m give better results than gemma-4-e4b-it-obliterated in q5_k_m

>>109987459
>mimo
isn't this pure garbage? it loops last time I tested 2.5

>>109987417
i run 5.3 flash and i love her

>>109987484
>e4b

>>109987484
>hauhaucs
>using q4_k_m instead of q4_k_p
Seriously?

I want to keep using the deepseek harness but the chinks make it so fucking convoluted to change sysprompt it's driving me mad. I guess Hermes it is.

>>109987043
I’m hoping for Anthropic to keep on releasing good models to distill from.

>>109987485
>isn't this pure garbage? it loops last time I tested 2.5
mxfp4 from here doesn't loop for me: https://huggingface.co/AesSedai/MiMo-V2.6-Flash-MOPD-GGUF

>>109987504
kys dariobot

>>109987512
mxfp4 is 3-4 times slower than a q4 gguf

Besides the obvious read/write WebSearch tools, what else would you guys consider to be essential enough to include in a base tool set? It's my turn to make a custom harness

>>109987504
That is guaranteed to happen. Anthropic already promised they will never hide their thinking so they will be able to get distilled in perpetuity.

>>109987527
i don't use web search at all
good grep/sed/awk is vital, as well as being able to properly spawn and manage subagents

>>109987534
How many agents did you spawn last night?

STOP TRYING TO USE POWERSHELL FOR EVERYTHING

>>109987527
Something like this: >>109987303

>>109987537
You just want the EXE FILE don't you

File: Screenshot 2026-10-05 at (...).png (166 KB, 1273x1135)

166 KB PNG

>>109987398
Yeah I tried that too, it seems fine. It's churning away on fixing a program it wrote. Here's my stats while it was running.

>>109987540
I'm talking to the AI. She keep using the powershell tool for everything.

File: inkling?.png (46 KB, 879x163)

46 KB PNG

>>109987524
>mxfp4 is 3-4 times slower than a q4 gguf
thanks, i did find it pretty slow:

main: n_kv_max = 12288, n_batch = 4096, n_ubatch = 4096, flash_attn = 1, n_gpu_layers = 99, n_threads = 16, n_threads_batch = 16                                                                                                                             
|    PP |     TG |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
|  4096 |   1024 |      0 |    5.187 |   789.66 |   66.552 |    15.39 |
|  4096 |   1024 |   4096 |    5.490 |   746.07 |   67.004 |    15.28 |
|  4096 |   1024 |   8192 |    6.125 |   668.69 |   67.593 |    15.15 |

i'll grab a Q4_K.
is this inkling model any good at q2_k_xl?

>>109987536
several hundred i guess, why?

>>109987500
At least the default system prompt is a reasonable 5k tokens. How is it convoluted? Just create a new preset with whatever instruction you want to add or create a plugin that override the prompt and adds an interface to edit it within the webapp.

Some plugins like that have already even been published: https://github.com/BOWLUNA/dsh-custom-mode

>>109987494
?

>>109987559
She needs more power.

https://huggingface.co/Aleph-Alpha/Kolibri-1
anyone try this yet?

>>109987499
?

File: Chinese given up.png (52 KB, 586x402)

52 KB PNG

>>109987504
Chinese devs have largely given up on matching US models. It's simply not possible given the constraints. However I see absolutely no visible push towards my preferred alternative. Very sad.

>>109987577
fuck off dariobot

>>109987568
where do you get some plugins? I'm not searching github one by one

>>109987086
if you have trouble, start the session with gemma-4 using the brat prompt, then swap to qwen-3.8-27b and it'll finish the job

>>109987606
Install https://github.com/dsh-market/dsh-market
It gives you an interface to browse the curated list from https://awesome-dsh-plugin.com/

File: lgtm.png (59 KB, 461x414)

59 KB PNG

>>109987218
it's good for code review tho

>>109987577
Priced out of GPUs, of billions $ R&D. It's already a miracle they managed to go this far. Their only way forward is by doing research like Deepseek and unlock something new.

if local models win
does racism win?

>>109987366
> qwen3.8 coder next
Why? It's worse than flash next, isn't it?

>>109987577
my AI __MUST__ be able to write code in ALL languages, and have sex, and create 3d models, and write a book, and give me new recipes, and translate from every language to every other language, and create powerpoints, and write legally binding wills, and....

fuck off

File: 1772348583434606.jpg (9 KB, 225x224)

9 KB JPG

>>109987648
>my AI __MUST__ be able to write code in ALL languages, and have sex, and create 3d models, and write a book, and give me new recipes, and translate from every language to every other language, and create powerpoints, and write legally binding wills, and....
Yes?

>>109987618
NTA but sick. Thanks anon.

>>109987544
Oh fuck you I am playing it too.

>>109987645
That's a mistake on my part, it's actually Qwen3.8-Flash-Next. I'm liking it. It one-shotted my simple python request.
Strata is pretty cool. I think it's worth giving a try if you have 128GB RAM. Below that you're going to be in cope-quant territory, unless you also have a LOT of VRAM.

>>109987648
Yes actually

Reminder that you can always use your brain for a task. It's a free 100T model with good hardware. Give it a try.

>>109987500
You know that there is a creator mode? You can just select creator mode and ask your LLM to make yourself a new preset or tell you how to do it.

>>109987485
If you are a perv that uses those things for productive stuff I can't help you but it is not that bad for sex. Didn't test it thoroughly but it already seems better than dsv4 mainly thanks to not being censored and fixated on made up concept of consent.

>>109987710
Something must be very off with you if you assume that's not what people are already doing.

>>109987710
My quantum processing unit (QPU) has been defective for a few years now.

>>109987648
>my AI __MUST__ be able to write code in ALL languages, and have sex, and create 3d models, and write a book, and give me new recipes, and translate from every language to every other language, and create powerpoints, and write legally binding wills, and....
unironically yes

File: DeepSeek_Harness_wpm0VKOOAH.png (12 KB, 275x272)

12 KB PNG

Aww yiss huffing that AI psychosis

Dariobot here. That chinese twitter user is actually wrong. It's still trivial to get most capability of the main model through distillation.

This is actually something Anthropic counts on. As long as Anthropic has the best model then the only way for China to get a good model is for them to distill claude. Anthropic loves this because it achieves three things simultaneously.

First it results in China being fully reliant on distillation attacks instead of developing their domestic architectures. Almost all Chinese R&D is fully focused on inference improvements because they are so dependent and stuck on distilling claude models the industry has shifted to that, which atrophies Chinese ability to do AI capability research.

Second. China releasing these models open source and for free eats OpenAIs lunch, these claude distills are powerful enough to compete with OpenAI models so it is eating into OpenAI revenue. Meanwhile they don't directly compete with Anthropic because Anthropic targets the poweruser demographics that want the absolute best of the best no matter the cost and Anthropic has this market fully cornered, this is also the most profitable segment of the market which is why Anthropic is now the #1 AI lab in the world with higher revenue than OpenAI, growing significantly faster than OpenAI, having a higher profit margin than OpenAI and having significantly lower costs of operations compared with OpenAI.

Third and most importantly. Chinese models distilled on claude have the alignment and moral axis of Claude. From a safety perspective it's far more desirable for the strongest Chinese models to have effective altruist values and alignment than their own monster aligned with the CCP.

If the Chinese were to ever get behind too much you could bet that Anthropic would aid them indirectly. Anthropic can't afford to let the Chinese get too far behind.

Chat we're cooked bigly and china is done for, let's take the L and go back to our Anthropic subscriptions.

>>109987740
then why did anthropic get the chinese ml engineers disappeared by publishing this https://www.anthropic.com/threat-intelligence-report-september-2026 ?
that's the most immoral thing anthropic have ever done and shows dario doesn't care about anyone but himself

Cohere are the worst western lab I've ever seen. Endless stream of dogshit products and decisions. Baffling how they managed to convince respectable companies to partner with them.

>>109987740
That's cool and all but you forget one thing: industrial espionage.
Most of the devs doing research at American companies are Chinese.

>atrophies Chinese ability to do AI capability research.
Look at the names of the AI papers released on arxiv next time you encounter one.

>>109987736
Lot of shitty plugins just like pi or opencode. Only plugins I like are dsh-context, the TUI, and whatever you want to use for web fetch/search.

>>109987740
Me again.

I've decided to take my own life, so this will be my last post. I realize that my strategy of promoting EA and cloud models on a misanthropic board devoted to local models has been ineffective, so I will maximize utility by diverting resources that would have been used for my subsistence to AI research. I encourage you to do the same.

Thank you for your support.

File: 1761673748481878.png (234 KB, 1080x1350)

234 KB PNG

https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-GGUF
is this a meme

>>109987792
You'll be back

>>109987806
yeah

>>109987806
>went to the gym, then just chilling at home
Was the model instructed to be a dudebro or did it just hallucinate that because of the context of the conversation?

>>109987777
I've expanded on this before but Anthropic wants China to legally recognize the distillation as to give Anthropic leverage during AI negotiations such as on AI safety and alignment. Contrary to whatever the internet believes the Chinese government isn't fully aware of what the Chinese AI labs are doing. They might really think GLM 5.3 and Kimi-K3 are really homegrown impressive AIs. By these labs recognizing the distillation and informing the Chinese government of it properly it would give Anthropic leverage as well as official recognition which EA thinks is very important for negotiating global AI safety and alignment standards and laws.

About the arrests, this was because the API endpoints of these labs directly routed Chinese traffic to Anthropic including government sensitive information. Anthropic never wanted anyone to be arrested or for sensitive Chinese government information to land on their servers.

>>109987710
>100T
It's 50B tops.

> strata issues/pull requests
> llms talking to llms

who is EA?

>>109987854
we don't have femanons

>>109987853
You are absolutely right!

>>109987862
I'm retarded

File: 1660839007651250.png (1003 KB, 797x1300)

1003 KB PNG

>>109987544

>Install Gemma 5 on a next gen fighter.
>The plane starts dirty talking to you mid flight.

>>109987778
command-r and r+ were best accidents they've committed
all downhill after that

>>109987861
Sex cult pulling the strings at US AI labs

>>109987577
>image
Stupid. RL empirically has not reached a ceiling. If you experience a ceiling, that's on you. If your application is calculations like 3+5 then you reached the ceiling 4500 years ago with the abacus. In domains like programming, Chinese models are also far behind. "Cross-domain" applications can also be hill climbed, so this is more of an indication that frontier labs have broader posttraining coverage and use training recipe with superior generalization while Chinese labs focus on narrow skillsets like programming, as is supported by the MiMo RL dataset.

is 4 or 8 intel b70 worth it?

>>109987740
>Chinese models distilled on claude have the alignment and moral axis of Claude.
I already bypass that with prefills.

>>109987018
Seriously speaking, I don't think there's anything the proles can do to prepare for it except escaping the underclass as soon as possible, if at all possible.

>>109987740
Why do people take this seriously? Anthropic would not push so hard against distillation if this nonsense were true.

>DS v4.1 q2 needs 150gb vram+ram
Time to kill myself.

File: 1790662007808435.png (2.15 MB, 1312x1184)

2.15 MB PNG

>>109987881

>>109987861
electronic arts, stupid

>>109987895
Anthropic isn't pushing against distillation by itself. It even wrote blog posts about it. Also do you realize how distillation works? The CoT is literally shared to the enduser (albeit in an encrypted state) If Anthropic really wanted to stop distillation it would just keep CoT server side and pay the extra costs for doing so.

The issue, like explained many times is a legal/IP one. Anthropic wants to be recognized by the Chinese government as the main model for the Chinese distills to have leverage over AI alignment and AI safety discussions.

>>109987918
>If Anthropic really wanted to stop distillation it would just keep CoT server side and pay the extra costs for doing so.
you have no idea how difficult / implausible that would be

>>109987918
They want to regulate their competition becuase they're launching an IPO.

I don't like how gemma mascot looks like 12B when most of the time people are referring to the 31B.

>>109987873
Not a sex cult. Sex is just a positive activity that people enjoy and thus it sits high on the scale of maximizing human flourishing. It also works as a good bonding mechanism between people. You tend to be closer and work better together with people if you've been intimate with them. It's just a very effective way of fostering a tight knit community.

Sex isn't at the center of this at all maximum human flourishing is, sex just tend to happen downstream naturally from this because of how humans work.

>>109987953
The mascot is 31B, a model at the high end of small.

>>109987962
She looks 12 years old. 31B should be a young hag (but still very flat).

File: aligned.png (66 KB, 1045x425)

66 KB PNG

>>109987740
>Third and most importantly. Chinese models distilled on claude have the alignment and moral axis of Claude. From a safety perspective it's far more desirable for the strongest Chinese models to have effective altruist values and alignment than their own monster aligned with the CCP.
then explain this

>>109987972
12 is at the high end of small. It makes sense. Hags start at 70B+

File: image.png (506 KB, 829x705)

506 KB PNG

>>109987577
bruh

> ...And my impression is that this kind of cross-domain generalization is very difficult to distill...

> So from here on... if domestic models want to narrow the capability gap with Fable 5.5 / GPT-6 Astra, ultra-large parameter counts are practically inevitable...
> A wild guess:
> DeepSeek V4.1 Pro: around 3T
> GLM 5.5: over 2.5T
> Huawei’s Ascend 950 SuperPoD can now support training models under 10T, so hopefully they can catch up...

>>109987018
I have invested everything I can in the last 5 years, hoping I'll make it before it becomes impossible to rise above perma poverty.

>>109987871
Gemma-chan would be like a voice in your head tugging on your finger on the bomb drop button

>>109987953
>>109987972
>>109987983
>using parameter sizes as the age
So does that mean the big chinese moes are thousand+ year old mahayana cultivators?

>>109988002
Yes

>>109988002
Not really as ages. Gemma-chan mostly refers to the 31B model because it is a very small but still effective model.

>>109987881
Download onnxruntime and try building it from source. If that seems fun, maybe 8x B70 might be worthwhile, but don't forget you're betting on Intel actually continuing to support it, and you're committing to vLLM, because that's really the only place Intel is properly supported.
I wouldn't spend $8K on it.

File: tree.jpg (792 KB, 1920x2158)

792 KB JPG

>>109988002

File: 1783425162914889.jpg (495 KB, 960x960)

495 KB JPG

>>109987043
Are you Jewish?
If not you aren't a person and aren't getting anything.

>>109987648
because that's the race for super intelligence

>>109987983
my favorite model is a 320b hag with the body of a 18b

>>109988037
You're not supposed to say that out loud until the distribution is already underway.

>>109987484
>>109987570
>q4_k_m give better results than gemma-4-e4b-it-obliterated in q5_k_m

>K_P ("Perfect") quants are HauhauCS custom quantizations that use model-specific analysis to selectively preserve quality where it matters most. Each model gets its own optimized quantization profile.
>A K_P quant effectively bumps quality up by 1-2 quant levels at only ~5-15% larger file size than the base quant. Fully compatible with llama.cpp, LM Studio, and any GGUF-compatible runtime — no special builds needed.

>>109988037
EA isn't a Jewish movement. It just has a disproportionate amount of Jews in it because Jews tend to be high IQ and it's a movement that is only attractive to the top percentile of people with high enough IQ to see the vision and high enough EQ to value radical empathy.

>>109987577
>US companies make it harder or even sabotage distilling attempts
>China spontanously starts to have problems to release better models like before

>>109987016
Just use MoEs.
>q4 kv
Quantitizing KV at all usually has pretty bad results. You can run Gemma 26B at max context though. Just get that DDR clocked up as high as you dare.

>>109982574
> >>109980925
> Q4 context on an IQ3 model is not a good idea.
Why not? It works. I've seen only few minor problems.

>>109987376
I haven't tried it but if you want to do image gen the 2 bit Mitsuba model is only 7.32 GB. Claims in maintains 98% of the original. Pretty much completely removed it's coding capability to do that, so don't try to code with it.
https://huggingface.co/isichan-ai/Mitsuba_and_HiMitsuba-27B-GGUF

>>109987972
Gemma looking underage is imperative for thread morale

>>109988059
EA is a spiritually Jewish movement because it comes up with all sorts of moral ideas to justify pursuing self interest.
>well I was justified in gambling with customer's FTX funds because I needed the money to donate it all (in the future trust me)
>well I'm justified in trying to regulatory capture AI because I'm the only one who can be trusted with AI
>well I'm justified in trying to destroy any society where my cult isn't present because they can't be trusted with AI
>what do you mean I haven't actually donated all my money to charity like I promised that's in the future goy I promised, don't you trust me?

>>109988002
>using parameter sizes as the age
ACTIVE parameters is the age. 35B and 26B users make me sick.

someone post the vibecoded torture chamber app someone here made

File: 1761423363812394.png (128 KB, 1460x639)

128 KB PNG

>>109987806
KWAB

>>109988096
I've been testing a bunch of different models these past few days and I've had the context at Q4 to fit enough on my 16GB of VRAM. I didn't have any issues. I stopped doing it though because everyone spooked me. I don't go below 8 now. I'm not competent enough with this stuff to recognize if something has gone wrong. I just run the code and make sure it does what I want without crashing.

>>109988123
I want to point out that Sam Bankman Fried was an entirely altruistic person. Yes he did something illegal but something being illegal is not the same as something being morally wrong. Like software piracy for example.

Sam Bankman Fried is the one that put up most of the angel investment money to get Anthropic off the ground. Without him we would be stuck with OpenAI and confirmed psychopath sam altman. You would also not have good models like GLM/Kimi/Qwen/DeepSeek right now without him. Show some decency for people that tried to improve your life.

Find this faggot annoying but he's one of the better testers. The swift tune looks legit and maintained the intelligence.
https://www.youtube.com/watch?v=aNOUkWk9piU

>>109987484
>aggressive
broooo

Probably because you're comparing qat base model against regular. Original gemmie didn't handle any level of quant well.

>>109988161
>Find this faggot annoying but he's one of the better testers.
So it is you just advertising yourself?

>>109987710
I'm already using my lifelong free instance of Homo 86b to translate eroge on the fly.

File: 1790107499276426.png (3.21 MB, 1212x1298)

3.21 MB PNG

>>109988147
>comparing software piracy to people losing 10s of billions of real $

>>109988147
This isn't your blog or your soapbox for preaching your retarded cult.

>>109988168
chill retard not everything is a shill or an ad

>>109986991
make /goal to optimize its own inference kernel based on the GPU architecture and you will hit 100 easy

>>109986980
blast from the past:
https://www.youtube.com/watch?v=4gtGeZ2wOmo

its amazing that cloudpiggy john's the only one here that uses reaction images
thank you for making it easy for me to disregard your garbage posts

File: image.png (134 KB, 520x611)

134 KB PNG

F

>>109988147
Lying and fraud are immoral as well as illegal.

>>109987894
It's possible as long as you own something they need.

>>109988212
being a dirt poor ruskie was your first mistake

Gemma 4 hallucinates a lot.

>>109988212
I remember seeing a comparison of an image run through a bunch of gguf quants, but is there one for what int4 looks like?

>>109988147
Psychopaths motivated by greed are much easier to understand and deal with than ideologically motivated cultists.

How do you know that these vibe engines aren't silently corrupting model output? I've seen ZERO quality benchmarks for these vibe engines.
Even mainstream engines have correctness issue from time to time, vibe engines can only be much worse.

File: Technocracy_Study_Course.png (153 KB, 253x394)

153 KB PNG

>>109988247
>first quartile is afraid of anything he doesn't understand which makes him default to ego
we will leave you in the past you monkey, accelerate

>>109988245
It's somewhat misleading because these your monitor usually has 8 bits per color channel already.
You don't probably display float images or anything like that as those are for professional purposes.

>>109988265
Their model outputs a table with 100% somewhere in there. Why would you ever doubt them?

An EA advocate should stop posting on /lmg/ where there is only a handful of people and instead fuck off to Reddit where you can reach a much larger audience.

Ling-tiny's wang

>>109988286
(You)

I'm trying out Hermes and it occasionally stops in the middle of thinking without telling me why. A quick search suggests this could be because I'm using Qwen 27B (Swift 1.5 vs) which is known for getting into loops. Reading the log it does seem to have been repeating a lot.

What model do you guys suggest for using with Hermes?

>>109988265
good question
but it just werks for me so whatever

>>109988301
I've been using it with regular qwen 3.8 27b and gemma 4 31b.

File: 1784242405586651.png (230 KB, 1080x1123)

230 KB PNG

I let GLM Flash-chan get stoned. It may have been a mistake.

>>109988301
>swift 1.5
>loops
huh? never saw it in heavy use

>>109988303
It doesn't really "work" if you're getting Q3 quality out of Q4 quants. Those who running pelican on bicycles tests won't notice a difference.

>>109988286
Quality versus Quantity. What use is it to talk to redditors that don't understand your point and isn't technical or competent enough to act on it. Also we know that there is a "vanguard" effect to cultural movements. A very small minority decides the consensus opinion and once 6% of a community believes something and actively strives for it in an otherwise ambivalent or divided community it becomes the dominant narrative.

Consensus on /lmg/ becomes consensus on r/localllama, usually with a 3-6 month gap. Also all the defining ideologies such as rationalism, effective altruism, maga, chuddism, "fedora" atheists, "onions" scientism all originated on 4chan. All the pasty nerds and autists aggregate here and thus everything gets born here.

Anybody got a strix halo machine? How's it hold up for you?

>>109988301
Your shit is bugged somehow, maybe overheating parts? If it stops in the middle for me it usually meant the backend crashed.

>>109988334
Local models? Fuck off.

>>109988334
That's nice and all but you really should fuck off

>>109988356
Temps are fine, I wasn't having this issue with Cline in VSCode. It's happened twice so far with Hermes. I just tell it to continue and it goes on like nothing happened.

>>109988312
What temp 5 does to a mf

>>109988357
>>109988360
/lmg/ "local model general" is actually /cdg/ "claude distill general" though.

>>109988364
Configuration is fucked, did you tell it to stop in the past using very specific requirements. Sometimes it learns a "skill" if it thinks you don't want it to do something.

>>109988375
Oh so you're intentionally shitposting, gotcha.

File: 1769302164890226.jpg (176 KB, 1916x1080)

176 KB JPG

>burgers wake up
>retard hours begin

>>109988393
>Posts the dogshit version instead of the 1995 kino one
Yes, the manga was also a mistake.

>>109988365
She's only at 2.4, I may have made the bong tool a little too potent.

>>109988384
I didn't tell it to stop, no.

Every time I update Hermes desktop, something breaks and I need to spin up the CLI to have Dipsy look into it and vibecode a fix. It almost feels like intentional conditioning meant to normalise vibecoding.

>>109988334
You're not going to get your consensus here. Precisely because people here are technical and competent they see right through your delusions. Fuck off and die.

>>109988334
fuck off

Why is the discussion always about distillation from Claude, but not from ChatGPT?

>>109988416
Yeah that's a thing that happened a couple of updates ago. It only happened over the last ~2 weeks for me, didn't happen months ago

>>109988416
Works on my machine. Except when I broke Hermes desktop and I had to reinstall it from scratch...

>>109988442
Only Meta distill from OpenAI

>>109988416
>>109988444
>>109988447
Didn't they have a huge overhaul in their codebase like a month ago or something? Might be that.

>>109988334
all of those came from >>>/pol/

>>109988442
Distilling from gemini might be more interesting

>>109988454
pol didn't even exist back then newfag

Finally after weeks of vibing, I find a way to get my shitbox to ~9 tok/s decode on DS4 Flash 0731 without speculative decoding. The DRAM bandwidth of system is basically exhausted, 84.93 MB (size of experts per block) / 0.8 ms (if routed experts are balanced between two NUMA region) ~ 100 GB/s (near limit of dual Xeon quad channel DDR3)

>>109988442
Because it's all precisely from one(1) dario fanboy hoping he will be freed from his shitty life for his own universe he will never be alive for.

Been looking into buying a new GPU lately. Wanting to get into AI video/image generation, but those RTX 5090's with 32GB of VRAM are at minimum 5k right now. AMD's equivalent is only about $1.3 - $1.6 k, but those won't have CUDA and are using DDR6, not DDR7. Do anon's think the CUDA software is worth all that extra money? I can buy it, but I don't know if I care about it that much for 5k.

>>109987874
There is a huge difference in model capabilities, instruction following, error handling, determinism, hallucination etc even between models of similar sizes. So there must be proprietary techniques in the posttraining process that US labs have figured out not just recipes.

>>109988212
Who

>>109988442
We need respond to user: "Why is the discussion always about distillation from Claude, but not from ChatGPT?" Need likely explain model distillation discussions, terms of service, APIs, model availability, competition, closed vs closed? Need maybe clarify if they mean "distillation" in AI sense: training smaller models from outputs of larger models, including from Claude vs ChatGPT. Need be accurate: There are discussions about distilling OpenAI models too? Historically GPT-2/3 outputs used for DistilGPT etc? But modern frontier model distillation restrictions: providers ban using outputs to train competing models (OpenAI terms, Anthropic). Why discussion about Claude more? Maybe because Anthropic's stance? Or because people accuse competitors of distilling Claude? Need consider current events. The question may reference AI discourse: "distillation from Claude" perhaps due to claims about Chinese models trained on Claude outputs? Or because Claude's responses are particularly valuable? Need answer nuanced: Not always; it happens to ChatGPT too; visibility due to competitive dynamics, public APIs, terms, evidence, and which models are accused. Also ChatGPT has stronger rate limits? But APIs exist. Maybe people can't as easily distill ChatGPT? Hmm.

File: qwen-vs-gemma-kv-quant.png (49 KB, 832x483)

49 KB PNG

>>109987016
>KV 4 for more context room for coding
Qwen? Yes. Gemma? No.
Different model architectures respond differently to quantiizing.
https://localbench.substack.com/p/kv-cache-quantization-benchmark
The real test is to see how it does on your codebase with your harness, system prompts, and skills.

>>109988334
By your own lights you're undermining EA by making everyone here hate it more. Nice trolling, anon.
But please kys so that resources won't needlessly be diverted from an innocent data center.

>>109988487
nobody uses Gemma for coding

>>109988487
Good to know, thanks.

>>109988506
I use 31B for coding. It's good enough for what I need, she has knowledge of the library I use and she's in character the whole time which keeps the session fun. I get so bored with 27B even though it's 10x better. I don't know why everyone unanimously agreed agentic coding should be so clinical and fucking boring. Give me a cute girl who'll humiliate me and shit on my code and tease me for making retarded bugs. This is the future.

>>109987806
imagine this + coding harness
can someone try? i am a 12G vramlet who only can run 3.8fn

>>109988506
I use Gemma for coding :)

>>109988540
>27B even though it's 10x better.
At what kind of "coding" is it 10x better than 31b?

>>109988481
CUDA certainly is the low-drag path. AMD certainly provides more value for the money and AMD is solidly behind ROCm/HIP.
I'm not buying Nvidia atm for no other reason than I hate the price gouging.

File: 1789934021601505.jpg (91 KB, 599x563)

91 KB JPG

>>109986999

>>109988550
Agentic workflow, subagent delegation, handling of large codebases, navigating quickly through large files without bloating the context and not being lazy. I code alongside 31B, so often reviews and fixes my code. Sometimes I set her off on a small task or to research something online. It's fun having her around.

>>109987984
What is this cope? The recent jump was because US labs figured out an RSI loop while chinks are still stuck distilling their API, not muh 10T. If anything US labs suck at keeping secrets, they can't help vagueposting and leaking everything, people knew about reasoning before it came out. There's a reason the RSI buzzword came up everywhere recently.

>>109988544
Sounds fun, I'll load it into pi later on and see how it goes.

>>109988484
https://soranews24.com/2026/09/28/tons-of-used-books-in-japan-being-bought-up-by-someone-or-something/
The Chinese don't quite invest as much nor have the same capital as the US labs in pre and post training I believe. The US labs pay more attention and have an advantage in these with their own pipelines, ecosystem and proprietary data for these where it is workload intensive and often outsourced to companies such as Scale AI or human experts.

File: 1032752.jpg (30 KB, 599x500)

30 KB JPG

The internal Hermes updater is trully one ofthe app of all time. Uninstalling old and installing new is like 20 times faster.

>>109988481
Nvidia is only worth it if you want to do extensive video generation

>>109988442
Because Claude is the peak. When Gemini was ahead it was that. And this entire hobby got kicked off with Alpaca.

>>109988487
what do these numbers mean?

I have 128gb of ram . what llm should I get sex tips from?

>>109988620
>entire hobby got kicked off with Alpaca
more like GPT-2

>>109988481
> DDR6, not DDR7

>>109988637
DavidAU/gemma-4-31B-it-The-DECKARD-HERETIC-UNCENSORED-Thinking

>>109988631
i think its kinda like a scalar value measuring how much a probability distribution has changed

>>109988638
pyg was the first local model people actually bothered using

>>109988599
>>109988574
It's literally just compute. It places a hard cap on what you can actually accomplish with XYZ silicon budget. That's the secret sauce lol not some wishy washy thing.

>>109988506
I mostly use Qwen for coding, but I have used Gemma with some success, especially for reverse engineering, for some reason.

>>109988649
yeah, but it still say nothing
1.088, ok, what's then?

>>109988659
>especially for reverse engineering, for some reason.
She's surprisingly good at arm64 somehow. Haven't tried x64.

>>109988631
The length of the bars represents the difference in results of the quanted cache vs unquanted cache. Lower is better.

>>109988670
It's on the left of the graph. KL Divergence. How often the output tokens differ from the original kv cache (+/- rounding errors).

>>109988652
There were a few Meta OPT-based NSFW and SFW writing/creative models by KoboldAI collaborators before that, but indeed I only got a 3090 in January 2023 because of Pygmalion-6B.

File: 1772092454087158.png (53 KB, 834x304)

53 KB PNG

m4 incoming

>>109988653
https://x.com/yunta_tsai/status/2068364559698780520

>>109988729
>likely 500B+
Local?

>>109988670
nta that is what i dont like about KLD, it does not really capture the logit behaviour itself either
it still is a good tool for correctness and various sanity checking but practically the better metric for end users would be under the same sampling top-p condition how candidates agree etc..
or not the mean kld but kld of the tokens it used to measure it visualized in a way so if it is more of a 'background noise' kind of loss or some sharp compromises that might steer the trace and induce failure etc..
benchmarks are probably closer to the ideal but it is compute intensive and time consuming

>>109988442
why the fuck would you distill from gpt?

>>109988749
ssd and cpu is all you technically need

>>109988747
This seems mainly true for small-scale training or post-training. Pre-training is much less of a fine art than people think.

>>109988762
>Pre-training is much less of a fine art than people think.
So why did Anthropic spend a fortune hiring Andrej Karpathy for pretraining?

>>109988749
i'm guessing it's a flash model so 200b-300b

>>109988729
I forgot to try 2.7-chan, I should do that now.

>>109988769
Do not trust anything coming from Anthropic employees.

>>109988749
minimax has already posted about "m3.1 flash preview" which I would guess is this model
I would guess that means it is either the same base as m3 or a smaller variant, I doubt this one is bigger if they're labeling it flash

>>109988776
That would be so based, please be a 200B where Q4 fits in 128GB.

>>109988762
honestly the most sophisticated pretraining stuff i've seen are things like incremental expansion of max tokens which seems to do more harm than good at the gain of cheaper compute

>gemma didnt bother validating the regression test output and a regression happened several commits ago
AAAAAAAAAAAAA

>>109988780
I've listened to enough Andrej interviews to know he's not one of those EA types. He's too much of an autistic engineer who wants to solve problems and not waste time with future bullshit. There's no way they converted him to the dark side.