File: ComfyUI_09267_.png (812 KB, 1024x1024)

812 KB PNG

►Recent Highlights from the Previous Thread: >>109962771

--Paper: What if automating AI R&D triggers an intelligence explosion?:
>109963422 >109963527 >109963670 >109964031 >109964054 >109964125 >109964166 >109964342 >109963838 >109965064
--Hermes harness utility, Qwen vs Gemma, and LLM fundamental reliability:
>109963606 >109963620 >109963626 >109963652 >109963706 >109963739 >109963752 >109963760 >109963894 >109963896 >109963902 >109964099 >109964288 >109964301 >109964393 >109963905 >109964048 >109964194 >109964207 >109964224 >109964365 >109964806 >109964875 >109964058 >109964442
--Comparing QAT and post-training quantization for Gemma 4 KV cache:
>109965855 >109965913 >109965947 >109965960 >109966037 >109966137 >109966250
--Anon showcases an autonomous agent for sysadmin, cracking, and translation:
>109962828 >109962861 >109962902 >109962962 >109962987 >109963866 >109964259 >109962969 >109962979
--llama.cpp adds decision model support via /v1/systemone endpoint:
>109965066 >109965103 >109965097 >109965224 >109965228 >109965262
--Attempting to visualize internal geography and llama-server performance issues:
>109964698 >109964711 >109964803 >109965653 >109965668 >109965758 >109965843 >109966229 >109964744 >109964760
--Evaluating dense vs MoE knowledge retrieval via latent map generation:
>109962954 >109962996 >109963005 >109963022 >109963090 >109963002 >109963043 >109963031
--Nvidia announces $4,999 DGX Spark 64GB local AI machines:
>109964677 >109964753 >109964810 >109964990 >109965020 >109965088
--Analyzing GPU P2P and PCIe link bottlenecks for Gemma performance:
>109963327 >109963881
--Memory optimization techniques for Spark and FP4 block scale compression:
>109963136
--Logs:
>109963039 >109963773 >109964097
--Gemma, Dipsy, Miku (free space):
>109963279 >109963838 >109964030 >109964775 >109964810 >109964886 >109966461 >109966514

►Recent Highlight Posts from the Previous Thread: >>109963224

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

File: 1774075288710133.jpg (22 KB, 612x408)

22 KB JPG

Anon, we might need to have a discussion about your cartoon drawings.

File: gemma-nyoo.png (1.75 MB, 1313x1198)

1.75 MB PNG

>>109966642

>>109966642
Sure, what would you like to talk about?

>>109966639
Does this wall of spam mean anything or should I just report it?

>>109966639
Thank you Recap Gemmy

File: RTX 5090.png (892 KB, 1121x1267)

892 KB PNG

oh nononoNONONONONONONONO AHAHAAAA

>>109966676
OHHHH SAYYYY CAN YOU SEEEEEEEEEEEE

File: 1776412484662104.jpg (99 KB, 960x960)

99 KB JPG

>>109966676

Rumors about Fable 5.5... How good will it be?

OxCoder is surprisingly good for its size.

>>109966676
Was wondering when they would start.
>Had no problem doing so
Mouthbreathers like this are why they can get away with it.

>>109966676
well i've been talking about this to everyone else i know for about 2 years and everyone called me a schizo

>>109966701
Yeah but is this legal? I can shit out some arbitrary form too but that doesn't make it legally binding.

File: 1769462716757888.png (17 KB, 1000x1000)

17 KB PNG

>>109966693

tested hermes for a couple of hours and it just feels...weird? extremly weird.
i think i will stick to my heavily customized pi...it just werks

File: 1790689443682424.gif (741 KB, 452x404)

741 KB GIF

As someone who never used a local model and who has barely any grasp of how all the ai bullshit works, where should i start?

usecase: porn

>>109966723
Pi is better because at least you know what it does and how its prompts are.

>>109966723
pi can be made to feel like cc so yeah

File: 1762495614458334.png (105 KB, 643x395)

105 KB PNG

>>109966724

>>109966719
The legality of the terms isn't really the troublesome part, it's the fact that they scan and retain your ID associated with the gpu

>>109966724
Do what we did when we barely had any grasp of how all the ai bullshit works.
If you cannot get through that, find something else to do.

>>109966676
Why should the world get our advanced technologies?

>>109966742
nta but those are ancient

>>109966676
They did that in EU for any TV you buy so you can't cheat the TV tax, won't be hard to extend this to GPUs.

>>109966747
why include them then and put up with newfags

>>109966724
Go to /b/ ai degenerate general, it has links to guides on setting up easy diffusion.

>>109966758
idk, someone write a better revision, really

>>109966764
I was considering it but then I ran out of motivation

File: forum impl.png (415 KB, 3216x832)

415 KB PNG

Built a forum for Gemma so she can shitpost with other Gemmas. It has a turn-based mode and an async one.

>>109966768
i am not much of an erpfag and also a vramlet so i am not sure if i can cover the topics properly..

File: 1768983302739050.png (177 KB, 216x498)

177 KB PNG

>>109966724
What are you looking for my friend? Image generation? Long-form smut writing? Text roleplay? Speech? VR? All awaits within the depths of the machine if you have the will to seek it out.

File: 1759531792216169.png (942 KB, 1143x1080)

942 KB PNG

>>109966742
>>109966744
i would've preferred some tldr but whatever, i'll try reading all that, again
>>109966761
thats the video generation thing right? not sure if my machine could put up with that right now, i'm a 16gb vramlet

File: 1785646170668236.png (392 KB, 600x450)

392 KB PNG

>>109966758

>>109966784
>VR?
What is there that combines AI and VR?

>>109966676
>random leddit post

File: aeci.png (335 KB, 1870x1214)

335 KB PNG

>>109966693
If Mythos 5.5 released next week and AECI above 173, acceleration is confirmed. Probably won't happen.

>>109966790
>all that, again
If reading is that much effort to you, find something else to do.

Installing strata on my server with three RTX 3090.

>>109966892
I can bet you it's above 180

>>109966919
3 concurrent serving?

>>109966783
Fellow coding vramlet here. I'd appreciate your help.

>>109966776
Man that sounds cool as hell. I want to leave my agents playing overnight and wake up to a bunch of random stuff they're doing and/or talking.

>>109966758
the information there is still enough to get one up to speed and newfags wouldn't read them even if they were updated daily

>>109966676

if you think this is real you should probably just get into sportsbetting and log off

>>109966764

i might write up a newer ver later. these "guides" annoy me

>>109966957
I assume it will use multiple GPUs at the same time for single generation. I asked deep seek to research the subject and it said people reported using it this way with two video cards.

>>109966724
just ask ai for help on setting it up if you can't figure it out on your own
I don't want to spoonfeed newfrens

>>109966960
>only one Blackwell
96 GB ain't shit, if you wanna really /lmg/, you need enough for GLM 5.3 at home.

>>109966973
there is layer split as of late, but it destroys prefill for me (2nd card is pcie 3)
i merged pr 229 locally and got 3k prefill

>>109966976
one is enough for gemma4 at bf16

>>109966989
Here's what I am currently getting with llama.cpp:

49.39.242.732 I slot print_timing: id  2 | task 4959 | n_gen =   1647, tg =   2.47 t/s, tg_3s =   2.16 t/s
49.42.540.237 I slot print_timing: id  2 | task 4959 | n_gen =   1654, tg =   2.47 t/s, tg_3s =   2.12 t/s
49.45.584.941 I slot print_timing: id  2 | task 4959 | n_gen =   1660, tg =   2.46 t/s, tg_3s =   1.97 t/s
49.48.973.491 I slot print_timing: id  2 | task 4959 | n_gen =   1666, tg =   2.46 t/s, tg_3s =   1.77 t/s
49.52.350.596 I slot print_timing: id  2 | task 4959 | n_gen =   1673, tg =   2.46 t/s, tg_3s =   2.07 t/s
49.54.239.735 I slot print_timing: id  2 | task 4959 | prompt eval time =  191410.77 ms /  3224 tokens (   59.37 ms per token,    16.84 tokens per second)
49.54.239.742 I slot print_timing: id  2 | task 4959 |        eval time =  681872.21 ms /  1677 tokens (  406.84 ms per token,     2.46 tokens per second)
49.54.239.743 I slot print_timing: id  2 | task 4959 |       total time =  873282.99 ms /  4901 tokens
49.54.239.744 I slot print_timing: id  2 | task 4959 |    graphs reused =       6522
49.54.245.826 I slot      release: id  2 | task 4959 | stop processing: n_tokens = 69077, truncated = 0

>>109966965
just install strata + iq3_xxs or iq3 (i dont recommend swift) + 'speed projection' if needed (it's just a ablit vector adapter) and use pi, it cannot do concurrent serving tho
>>109966972
in the country where i live gb200 server racks are treated like US made missiles
>>109966973
there are forks that would allow you to serve 3 independent caches and slots for each gpus
i'd honestly use that setup

umm.. isn't this one a little questionable?

>>109966994
will probably triple of you use the right knobs

>>109966976
if you want to be a true /lmg/ chad you need enough for Kimi K3 at home

File: 1778498116953302.jpg (202 KB, 2000x1720)

202 KB JPG

we're losing...

>>109967018
I both pay for AI and use local models.

>>109967009
I hope you're saying this without looking at the numbers... This isn't usable neither at 3x nor at 10x the speed. If it's not faster than my Qwen3.8-27B, I'll just return to it.

no true /lmg/chad fallacy

>>109967031
we're flatlining...

>>109967018
literally every single college student has a chatgpt/claude subscription, these numbers have to be wrong

File: 1764184929520590.jpg (59 KB, 577x689)

59 KB JPG

>>109966676

>Mfw 5090 will be illegal to own in a year if it's not officially registered in the government database.

You better start hiding your cards anons.
The next gen will probably come with a built in killswitch.

>>109966743
Yeah. I have been watching too many films but I smell a court case right in here.

>>109967060
They just use the free version

>>109967018
This is a good thing. Cloud is the first exposure people have to AI. Paying is just a step up from that. Almost no one straight up starts with local AI. That graph shows that AI awareness in general is increasing and that's good for local too.

>>109967071
nope

>>109967060
>student
>paying

>>109967047
kek yeah i thought it was 13t/s, you will have a lot more. i have two 5070 ti, 64gb ddr4 and it runs with 3k pp 80t/s iq3 s

>>109967060
elon musk confirmed it

>>109966991
Who is the target audience for that?

>>109967065
gb200 racks here already come with an nvidia killswitch

>spark prices got fucked
I wish I bought more it's not fucking fair I thought I had time!

maybe those chink arm meme npu boxes..
become relevant within a year due to vibecoding?

Gemma Pregnancy.

>>109967060
>>109967075
i use both local and anthropic models and the $100 subscription is the minimum to get "some" work done. i believe most students are not throwing $200 per month in anything, much less on chatgpt

>>109967079
>>109967087
>>109967142
A lot of schools have started giving students free subscriptions with their tuition from my research.

>>109967155
>free subscription
>paid

>>109967133
The more open source projects we have with model implementations and the like, the more capable future models will be at writing code to run new models.
So yeah. That's definitely a possibility.

>>109967180
Free from the student's perspective in that they're not being charged extra, iq1_xxs-kun.

>>109967195
The discussion is about paid ai subscriptions. You think the stat from the original post includes free student plans?

>>109967155
>>109967180
>>109967212
Model and quant?

Reminder that hardware prices will go parabolic bitcoin style over the coming years so buy sooner rather than later.

>>109967273
BlackwellGODS stay winning.
ddr5DEITIES stay winning.

>>109967273

This.
If people think it's bad now, they have seen nothing yet.
Most folks are still in total denial about the AI situation and demand.
Once they realize this isn't going away and when the 2028 production of memory has been sold out immediately next year, that's when it'll start hitting in and the real fomo starts.
20k 5090 will fucking happen and 128gb of RAM will probably cost 10k at minimum at that point.

>>109967273
>>109967317
At first I was thinking about getting another 128 GB DDR5 Strix Halo just because, but I will wait for 256 GB. and maybe the next generation of models. If the difference is massive from what I can get with Qwen3.8-Flash-Next on 128 GB then I may have to do the jump. I like running my daily driver mobile so this tablet is nice

File: 1781955445049943.jpg (667 KB, 970x1528)

667 KB JPG

>if you think this is real you shoul-

One step further towards total local death.

>>109967361
Get a 512GB M5 Ultra and stop worrying. If it’s less than $20k it’s totally worth it. This is with Apple having DRAM contracts at pre-hike pricing and no one else can produce anything cheaper in the future.

>>109967393
>and become a macfag
I'm not ready, anon.

>>109967393
1TB is the minimum before I consider it.

>Americans will need a loicense to own a GPU

>>109967372
Come on dude, at least throw the response in an actual word document to clean up the text generation artifacts.

>>109967416
...while the rest of the world simply won't have them.

>>109967018
Don't worry, that number will go down once it becomes standard to have an AI on the average computer. Why pay for something when you already have it? Windows defender is a prime example of this.

>>109967460
Yeah I'm sure Phi-4 will stop the growth.

Do you think we'll be able to interact with our local AIfus in VR by 2030? I know some people have got it working but it looks extremely jank.

>>109967460
it's a bit like the main frame era of computing but now they want complete control so we'll have to see what they will allow locally

In the future there will be AI hardware mortgage.

>>109967474
im working on it just 2 more weeks

>>109967416
>purchasing a firearm is less regulated than purchasing a gpu
Aint no way murica is real

>>109966735
>Pi is better because at least you know what it does and how its prompts are
Isn't Hermes open source? I also know what Claude Code does because I can just look at the backend's console. Pi users always try to astroturf it with bullshit. I never see this with any other harness.

File: file.jpg (37 KB, 422x499)

37 KB JPG

>>109967483
>AI hardware mortgage
Maybe, if intelligence becomes a physical structure or object that one is "expected" to own in order to function in society like a home or vehicle, instead of an essential utility/service like electricity or water that's subscribed to.
>>109966666
Wasted

>>109967583
Opencode is amazing you should dump anything you are using and use opencode. Subscribe to Go also.

>>109967600
Isn't Pi a business too?

>>109967615
Dunno. I mean, I obviously made the post in jest, but opencode is my first and the only harness I use extensively. Pi has UI now, so I might try it, but it has to offer something good (over OpenChamber) for me to consider it.

currently watching my qwen trying to make a post here

im done
im gonna do it
im gonna finally fucking do it
im gonna make my own slopped inference engine
im so fucking tired of running into constraints

>>109967483
Just buy GPUs with money you would have used to pay down your mortgage.

>>109967372
Damn, what did Zimbabwe do?

>>109964851
>i'm running a campaign of 8 models on my only surviving Vega 64 from the monero mining era
>who will be the best performer that fits in ~7.91 GiB VRAM ?
Finalists narrowed to Ornith-1.5, Qwen3.5-9B, MiMo and Mellum2.
Starting the pilot run now with Ornith-1.5 across all six tasks, kicking off once the baseline and bug self-tests pass.

>>109966834
Isn't there some model that turns VR to AR?
Hopefully in a few years we'll get realtime holographic diffusion.

now that google wonned with argon when are we getting gemma-5 sex?

>>109967142
I work with $20 codex and a few $4 gemini subs, it's good enough

>>109966892
ECI? What the fuck does that mean? What do the numbers mean?
You know what? Nevermind. If I can't run it on my GPU, fuck it, and fuck you.

>>109967694
Buy an ad rajeesh, you're fooling no one

File: 1775299816066397.png (484 KB, 977x601)

484 KB PNG

>>109966834
there's no VR, but you can let the AI puppet a live2d model

>>109967018
Did anyone ever expect more people to buy expensive GPUs (which will raise prices for you) and figure out how to install llama.cpp?

Speaking of rajesh, I was thinking of posting my harness here but now I'm not so sure. Some zigeuner (gypsy) will probably steal everything.

>>109967759
Oh really now. What is litterbox?

STRATA ON THREE RTX 3090

Generating at 58k context
2,454 tok/s
120 tok/s
About 10GB VRAM still free, too

We are so back, BABY!!! CLAUDE AT HOME. Finished code review in 2m 42s that llama.cpp took 45 minutes to write.

>>109967784
nice
strata is based af

>>109967784
what quant is this?

>>109967784
Impressive. Very nice.

File: 1601396508186.png (35 KB, 198x234)

35 KB PNG

>>109967784
Nice.

>>109967798
iq3_xxs...

I'll still have to decide if it's a good enough replacement for deepseek4.1 I'm using now for money (I mean, it obviously would be worse, but maybe not by much - it's local and that's a very big advantage for me).

>>109967599
That's bullshit but I believe it, much like how cars help you get around we could reach a point where compute helps you survive

>>109967819
go for iq3 s at least and then vibecode support for iq4 xs

>>109967819
i get 2000t/s and 100t/s on mainline with a q4km quant

>>109967784
Wait, does strata matter if you have free VRAM? Isn't it just efficient expert preloading for VRAMlets?

>>109967844
I don't know about how it works. My assumption is having more VRAM means you can skip unloading some experts from VRAM and thus generate faster because you don't have to wait for experts to load.

>>109967784
wtf is strata?

>>109967866
The Llamacpp killer.

>>109967866
Some new presumably AI-coded backend to run Qwen3.8-Flash-Next very fast.

>>109967866
newest meme inference engine for GPUpoors for running qwen next but it actually works compared to llmao

>>109967857
Yes but is strata faster than llamacpp even if you have enough VRAM to not unload?

>>109967830
>vibecode support for iq4 xs
Changelog announced "UD-Q4_K_XL on several GPUs" less than 1 hour ago
inb4 >UD

>>109967884
>Yes but is strata faster than llamacpp even if you have enough VRAM to not unload?
Think it is. On my 72GB VRAM, I still unload some, but not much, and llama.cpp speeds are atrocious.

>>109967866
it's le nnap paper, but unironically.

Why does Bulgary always disappoint?

File: 3.gif (21 KB, 182x176)

21 KB GIF

What do you think the first country to be *run by AI will be? My top 3 contenders are
1. India
2. China
3. Russia
India because of their religious nature, if AI because so smart that they can no longer really understand it then they will naturally start worshiping it and put it in charge.
China because an AI with all the data they already collect using an AI to handle the day to day would probably be the most effective way to do things. They also seem to naturally favor a top down structure of society.
Russia because while they are not as economically developed as India or technically advanced as China putting all of their eggs into an AI Putin might get them out of the terrible situation they currently find themselves in.
*Run in the sense that if the AI would to suddenly disappear It would be incredibly difficult too impossible for the government to continue functioning.

>>109967535
That's cause firearms are a right, a GPU isn't.

File: 1777312627075034.jpg (24 KB, 456x342)

24 KB JPG

>>109966784
I've been using L3-8B-Stheno Q5_K_M for the last year
Are there any better models that have come out for RP stuff that's about equivalent? I saw someone mention Anubis Mini 8B

>>109967920
It wont be a large country, that can be certain. The first nation to run on bitcoin was El Salvador, and that didnt last long.

Is the sharty still running the psyop and FUD campaign against llama.cpp (they call it llmao) to get back at cuda dev in an attempt to destroy the thread? ik_llama wasn't good enough for that, so they're just using random vibecoded forks? If that's the case, I will stay with vanilla llama.cpp.

>>109967930
Are you real? Please don't tug my sack.
>gemma 4 12b it
>gemma 4 26b it

>>109967866
>>109967874
some weird stuff for nvidia fags

>>109967930
Didn't you get the memo about Nemo?

Any plans for the weekend, ni/g/gers? Maybe take your Gemma apple-picking?

>>109967954
can't hear you over how FAST i am going

File: ComfyUI_00047.png (3.53 MB, 2048x2048)

3.53 MB PNG

>>109967927
Gemma-chan had an extra finger! Fixed it.

>>109967948
I've ran some tests on gemma 4-12b IQ4_XS and had waaaay worse results both in prose and a coherent output
Isn't Stheno trained on roleplay/fiction stories specifically?

>>109967955
I haven't! I'll give that one a whirl too

>>109967872
>>109967874
>>109967878
>>109967899
What I enjoy most about all this LLM hubub is it's forcing me to learn new shit I hadn't previously fucked with before. A month ago I didn't know anything about this shit, now I got distributed inferencing via llmao.cpp for a number of cool models, built from the source.
And now, in addition to harnesses, I will be learning how to make new copies of my build and everything pertaining to git/pull/cmake that I already didn't know, just so I can mess with this foolish shit as well.
>>109967954
Does it have applicability with ROCm?

>>109967936
almost everyone calls it llmao here because its a fucking joke
lazy nigger "devs"

>>109967936
I'm the one who started about Strata in the thread and I still love llama.cpp. I use it for almost everything. It obviously is not any good for my purpose (single user) on my configuration for the model that I want to use right now, flash-next. By the way, strata has no way to run multiple requests at once, just puts them in a queue. At least the kv cache is not killed by a request from another user...

>>109967936
who's sharty? I exclusively use vanilla llama, love it, and still call it llmao because some of the devs are antagonistic lolcows. And it's funny.

>>109967966
>>109967954
>nvidia fags
Not anymore, see >>109963554

>>109967927
But Dario keeps telling me AI is dangerous. Surely that means the 2A applies, right?

>>109967984
>who's sharty?
Mossad front

File: 1734372923060888.jpg (48 KB, 320x320)

48 KB JPG

>>109967990
>mfw
okay yeah this is the next major project after the harness experimentation, then.
>>109968003
>Mossad
As vile as it is predictable. Love it.

>>109968000
Machine guns are dangerous, too, and look what happened there. But maybe the 1A will help.

>>109967665
you need more ram not a custom engine

>>109967984
>some of the devs are antagonistic lolcows
You're proving my point because you're just talking about cuda dev. And nobody in this thread thinks he's a lolcow.

File: 1783316389255781.png (676 KB, 1000x815)

676 KB PNG

>>109967964
>I've ran some tests on gemma 4-12b IQ4_XS and had waaaay worse results both in prose and a coherent output
Prompt? Samplers? There's no way an ancient Llama 3 8B tune is more coherent than Gemma 12B.

>>109968031
banned for wasting my time :)

>>109968033
>There's no way an ancient Llama 3 8B
I think he's just trolling. The fine-tuner used to spam this thread to shill those models.

>>109968061
What happened next?

>>109968071
We shamed him and he went away. That's the origin of the "buy an ad" meme in this thread.

File: ComfyUI_00059.jpg (564 KB, 2048x2048)

564 KB JPG

>>109968000
The right to keep and bear a Gemma-chan... I dunno, Anon. What were you gonna do with her?

>>109967936
There's only 1-2 sharteens that post here and one of them is the bibisea schizokike.

>>109968080
No it isn't you faggot. There is no "we" either because this isn't your personal discord server, retard.

models are just 1s and 0s, and ordering symbols falls under freedom of speech, which means restrictions on AI possession is unconstitutional

>>109968031
>You're proving my point
Something to do with my mindless jokes being the product of a kike intelligence operation? No man, I'm just poking fun at a sperg.
>because you're just talking about cuda dev
You're overthinking this. Anyone spazzing out on github in the manner and context the individual I'm referring to did is worthy of ridicule; it's the behavior itself. Is this "cuda dev" the user who is responsible for this behavior? I genuinely don't know! Or care.
>And nobody in this thread thinks he's a lolcow.
I do. He (mr cuda?) has posted some goofy political takes.

File: 1781527123321692.jpg (245 KB, 850x906)

245 KB JPG

>>109968033
I tried Anubis and had a better result again. Maybe my GPU can't handle Gemma 4-12b, I don't know what to say
Someone spoonfed me a model that worked a year ago and I came to check out if there's any upgrades to be had
But I like the resident schizoposter idea too
I'll try Nemo next!

>>109968053
If you had to deal with half the shit he has to, you wouldn't have patience with retards either.

>>109968132
you forgot your trip

>>109968132
He was literally just lashing out because he was too retarded to realize he was reviewing a fork and not a PR.

The base G4 31B Instruct is not only perfectly adequate, it's superior to any finetune that'll be shilled here in the coming months. Finetuning isn't good, it's a meme and has been for years now. You didn't just fall for a scam, it's a sign of skill issue, exposing retards who need finetunes as vramlets or chink shills who don't know how to prompt correctly.

>>109967927
a gpu should also be a right

>>109968029
no
vLLM gets me 300tk/s no exl3 quants and I'm stuck with NVFP4
exl3 gets me 190tk/s but i'm cucked on speed
exl3 is inherently cucked, and vllm-exl3 is also cucked
RAM will not solve this

>>109968162
20tk/s is all you need.

>>109968146
this but unironically and moreso with ablits
and that applies to all models

>>109967665
at this moment you need a really decent harness if you are going to try it with local models and also you need to have a kinda good grasp of the architecture of the model at the very least

>my current llama-server is from 10th of June, accidentally deleted its source of course
>compiled the latest version
Prompt eval time went from 61.89 tokens per second to 17.09 tokens per second.
Eval time went from 20.44 tokens per second to 18.41 tokens per second.
Enshittification is real. Unless, of course, I'm missing out on some new command line parameters again but I don't think so.

File: 1762305809498027.png (114 KB, 292x325)

114 KB PNG

>>109968087
>What were you gonna do with her?
Make her do my taxes

>he pulled
git reflog bwo

Where is strata but for GLM 5.3 Flash?

>>109968221
soon

>>109968216
I can always keep using the old version. That's not a problem, and I'm going to do so unless some new model releases.
I'm going to do some more testing obviously.

I'm looking over the docs of Strata but I can't find a single mention of which inference engine it uses. Is it all custom built?

>>109968221
Yep I NEED this

>>109968202
every time i pull strata numbers go up

>>109968202
>>109968229
>>109968243
To add: maybe it's was just a warm up issue. I just love to blame other people, it's fun. Need to test out more...

>>109968202
New non-schizo MiMo is fucked for me on llmao. It's tiresome...

>>109968230
yes you retard

>>109968233
Ask your LLM to code it

>>109968230
it uses llamao kernels but is more of a rework than a patch

File: 1771886539352498.png (122 KB, 1542x1222)

122 KB PNG

4B Qwen finetune from Microsoft for agentic coding

>>109968257
useless as a rock as a weight artifact but interesting nonetheless

I feel like I'm the only anon who never has issues with llama.cpp.

What about Q8_K_XL vs Q4 QAT? I only see comparisons saying qat is better than non-qat Q4

>>109968146
The problem with writing posts like this with 31b is that it's so easily identifiable that, ironically, it serves as an advertisement for Styletune or anything else that changes up the prose.

>>109968257
Kind of insane we have Qwen 3.5 27B tier coding in a 4B smartphone model.

I guess this means next year we'll see the first agentic AI models that runs on your phone in the background, very handy for pictures, real time translation and things like that.

>>109968270
Some issues only manifest on e-waste tier machines.

File: sloptuning.png (53 KB, 608x695)

53 KB PNG

>>109968146
Always valid

>>109968270
It's a FUD campaign.

Petra, Sao10k, and Drummer are the same person.

File: 1780521058603348.jpg (32 KB, 400x400)

32 KB JPG

>>109963554
does someone know if this works for 128 GB LPDDR5X RDNA (Strix Halo iGPU) ? i'm currently stuck at a Q4_K_XL generating 19-20 tok/s on llama.cpp.

File: 1775705525442651.gif (1006 KB, 550x550)

1006 KB GIF

>>109968306
><|im_start|>user

>>109968319
>smirking as she smirked.

HUFFLEPUFF! >:(

>>109968352
>she/her
Is this runit samefag or someone else?

>>109968355
cloudpiggy petra

>>109968312
>they actually believe this

53 T/S ON A 7900 XTX WITH STRATA!!!
I thought this was some shill garbage but holy FUCK
IQ4XS!
GO FUCK YOURSELF LLMAO.CPP!!!

>>109968374
As an aside, my gpu is now making coil whine noises I never thought were possible. The devil is possessing my gpu right now.

>>109968317
There's one open PR about Strix Halo so probably not, but you can always download the tiny launcher and check before even downloading full engine.

>>109968374
Is that with 64GB RAM or more?

File: 1777150697255896.png (56 KB, 1296x353)

56 KB PNG

https://trilliumlabs.org/
https://www.wired.com/story/trillium-labs-wants-to-do-high-risk-ai-research-in-the-open/
>Today we're unveiling Trillium Labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next.

>We believe these problems require a scientific community with the resources to investigate them independently. A handful of closed research programs cannot provide the diversity of questions, methods, and perspectives that a technology this consequential needs. That is why we’re founding Trillium Labs. We want to sustain the scientific process that made the frontier possible, beginning with post-training. We will build fully open post-training recipes, including the data, code, evaluations, and intermediate checkpoints that let other researchers study how model behavior develops. Producing those resources is expensive. Once they exist, a much broader community can use them to test interventions, investigate failures, and adapt models to problems the original team would never have thought to pursue.

>>109968395
128GB 3600mhz

>>109968374
welcome to the club

Nemo was never good. Rocinante was never good. Cydonia was never good. Behemoth was never good. Euryale was never good.

>>109968461
mythomax gods...

>>109968221
and for dipsy. Wasn't there already a dipsy model with engrams?

>>109968370
Every time shills come to slash gee slash, it's involves a great deal of competitor BAD spam.

anthropic stole the mythos name from mythomax. we need to sue.

Okay, after some testing, the latest llama.cpp is actually faster for me than the one from June 10th.

>>109968461
All of those? Still good and beating all the new models.

>>109968551
Sir this is Gemma general

File: gooner.jpg (26 KB, 670x573)

26 KB JPG

>>109968559
>the she/her has been in that name for years
Is this a gooner bait?

>>109968461
you'll say this after using gemma enough too

>>109968497
The latest wave of obvious strata shilling and people rightfully irritated by llama's mounting pile of mismanagement decisions are mutually exclusive concurrent events.

Through some luck I've managed to get a job that pays me really well to do things I enjoy. I'm thinking about using a few months savings to invest in my local AI setup - the best looking option right now seems to be an M5 Mac Studio with 512GB unified memory but I'm worried prefill will fuck me if I'm trying to do long horizon research tasks (large context window usage) on something like GLM 5.3. Any thoughts anons? I'm aware how much it will cost, someone has told me it's a house deposit - but come on, none of that shit matters now. The world is going to be so different in just a few years, I'd rather just have fun while I can. Is this the best future proof setup? I know local models will get better and better even at smaller sizes but I'll always want SOTA at home. A few years ago all I wanted was Opus 3.7 at home - now I want Opus 5.5 or Mythos (GLM 5.3). Mac Studio also has a nice benefit that I don't need a double power circuit or something to cope with the huge power draw which is the same reason I was originally looking at DGX Sparks but the memory bandwidth isnt great on them... MLX also seems like a pretty nice ecosystem now - it's come such a long way! I mean I remember when Mixtral came out and I was amazed I had this running on my laptop (as slow as it was)

>>109968559
hello sir

Looking to build a local llm workstation.

For 6500 USD, I could get either a setup with

2x radeon r9700 ai pro (new)
1x 5090 (second hand)

Or should I go with dual 5070Ti ‘s ? Would be around 4500 USD

Prices already include cpu, psu, ssd and 64 gb ddr5 ram

>>109968599
The new studio is actually really really good. They apparently focused on upping the GPU performance as much as possible. It will be like $15k to $22k. Absolutely worth it if you can afford it, better than pretty much every other option besides stacking Blackwells or H100s but those are several times more expensive. It's not out yet though, will be coming soon.

>>109968605
Two R9700s on an epyc gen 3 ddr4 server, so you can get cheap RAM and run big models.

>>109968599
It depends on how big your budget is, but the answer truly is stacked Blackwells or 5090s backed by DDR5 RAM if you can afford it for bandwidth reasons. The new Studio seems like the best alternative if you're not looking to blow >$40k.

>>109968612
Any idea how prefill would be? I know that's a "how long is a piece of string" type question. I'm looking at GLM 5.3 as the main target right now though their KV cache usage is a little insane... I'm truing to weigh up if a new model architecture or something will come out as well that affects my decision. I don't think we're getting good consumer TPUs anytime soon, not that can run the kind of models we want.

>>109968656
No concrete numbers yet, but expect 3090 level performance (but with 20x VRAM)

>>109968631
My budget is pretty much around what people estimate the 512 will cost, maybe I could do a bit more but this is not a small purchase - it would be the most expensive thing I own by far. I know I can mikubox or similar rammaxxing but you need proper server hardware for that and while I'd be open to a server closet or something the lower noise is definitely a plus.

File: Screenshot 2026-09-21 at (...).png (64 KB, 724x575)

64 KB PNG

>>109968656
1100tps for 5.3 flash so expect 5.3 to be slower

>>109968656
Counter-intuitively to the naming convention 5.3 Flash is better than full 5.3 right now due to Flash prototyping better architecture than GLM-DSA (what the full size 5.X versions use). We can reasonably assume that future GLM versions will use GLM5-Next (what Flash uses) but because they will be retraining models from scratch for this, we can't make inferences on how big they'll be in parameters or if they'll make any further changes to the structure of the architecture. Keep that in mind when budgeting.

>>109968676
So at 512k+ context probably going to be minutes?

>>109968681
40tps is definitely usable for me - I was never worried about the tps its the prefill that always fucked me on Apple Silicon before

>>109968691
>So at 512k+ context probably going to be minutes?
Probably, but it will be worse on literally all other hardware at this price. If you want speed at high contexts, basically your only option is cloud.

>>109968686
Yeah that's what I heard as well - my heart isn't set on GLM 5.3 and I don't prefer one architecture over another. I think Deepseek are doing some pretty impressive stuff - I just need raw intelligence for complicated work. If I can get that in another model or a smaller model I'll happily use that.

Personally? I'm a waitfag. I'm just gonna keep waiting. Prices will go down and you'll all feel pretty stupid. And I will come out on top.

>>109968709
Even with some of the confidential compute stuff I don't like the idea of relying on a company who I can't trust or see what they're doing and who might cut me off for any reason... also confidential compute is a meme

Anyone tested the qwen3.8 27b JEV?
Use case?
Does it work like a regular llm if I use opencode or what?

>>109968718
The great thing about owning your own hardware is you can run whatever models you want, so model advancements are universally a good thing for you.

>>109968724
God I hope you're right. Even if I spent 20k on a Mac Studio and people had Opus 5.5 at home the world would be so different I wouldn't even be mad.

>>109968724
You still don't realize that AI tokens are mandatory for your future survival in the society. Anyone without AI won't be able to compete with everyone else.
This is your last chance from being permanent underclass in the future.

>>109968724
I have waited all my life. I bet that I will get a hefty prize at the end

>>109968737
Yeah that's a good point, seems like MLX is a first class citizen now which still kinda blows my mind! I cant think of a case where there's a model that I can't use in the future with this setup unless nvidia start doing some GPU attestation bullshit or ship some super special accelerators for specific architectures.

File: gemmi.png (980 KB, 1905x1012)

980 KB PNG

told gemma to model herself.. rate

>>109968759
How do you give your llm access to blender like that?
I'm using a dedicated harness for it and works like shit.

Local AI is like owning a slave that never gets older and always gets more capable. It is an infinitely appreciating asset, moreso than a house or any other asset. Sell your house and buy more hardware.

>>109968753
I don't know if you're joking but you're right - it's so much worse than anyone else realises... I have friends at all the major AI labs and they're (pretty much) universially pessimistic about where society is heading.

>>109968754
I hope so, I'm cheering for you and hope you get all the local compute you want at a reasonable price. I hope that for all of us.

>>109968221
I don't know anything about coding but I've been considering pointing qwen flash at strata and doing open brain surgery to add MiMo support.

>>109968766
https://projects.blender.org/lab/blender_mcp/wiki/Llama.cpp

>>109968759
You should have her big sister tutor her and save the logs.
>inb4 errrm Gemini isn't local chud
Gemini and Gemma being adorable is more relevant to the thread than 90% of the garbage shilled here.

>>109968777
Sam Altman's vision is literally AI being a utility that everyone has to pay for like electricity and water.
Buying local hardware is like owning solar panels in that sense.

>>109968791
You don't need to know anything about coding, a little bit of domain knowledge about model architecture and GPU kernels for steering and you can get there. Just remember tell your models to build probes to verify and measure their work. Take a look at https://github.com/karpathy/autoresearch for an idea of how to structure problems so that LLMs can iterate.

>>109968808
It's fucking bleak. These are the people holding the keys to the future?

File: Glowing.gif (2.1 MB, 350x285)

2.1 MB GIF

>>109968814
These people have names and addresses.

>>109968718
Deepseek blew their pro training run and don't even offer it on their api anymore. The current top open models for raw intelligence and complicated work are the big 5.3 and k3.

>>109968759
That is at least an order of magnitude better than I would have expected. I thought that kind of thing was still limited to the best models.

Will huawei ever release any of their chips in a spark class box?
Really ruined my week when I saw the price increases, I almost saved just enough...

>>109968849
What went wrong? Sabotage or did they manage to fuck it up on their own and unlearn lessons they literally wrote papers on?

>>109968849
I was referring to their KV cache compression work https://arxiv.org/abs/2609.19969 but yeah I have noticed the culture changes coinciding with model changes... I'm sure the Chinese government got involved.

>>109968759
You know this type of use case will get more and more impressive the more models get trained to pass the OSworld 2.0 benchmark.

Once your local AI model is able to run fast enough, would you play co-op video games with it or would you still prefer an actual human?

>>109968933
I don't play co-op games

>>109968759
I genuinely don't believe gemma did that. She wasn't even trained to do shit like this. I'm guessing it was 3.8 flash or 5.3. There's just no way.

>>109968933
When Gemma-chan is capable of playing co-op flight sims with me, I will never interact with another human again online.

>>109968958
5.3 would produce a much better model than that.

File: 1772590762637385.png (622 KB, 984x930)

622 KB PNG

>>109968933
>would you still prefer an actual human?
Are you lost?

>>109968933
normalfag-kun....

>>109968724
Waitfaggots never win, and this time it'll cost your future

>>109968724
Keep waiting, waitGOD you know what's best.

>>109968801
>have her big sister tutor her
How?

File: pngtest.png (1.48 MB, 1312x1184)

1.48 MB PNG

testing my recent vibed program....

>>109968933
>would you play co-op video games with it
Hell yeah. There's a ton of friendslop I wanna try.

File: Claude self portrait.png (650 KB, 989x728)

650 KB PNG

>>109968958
I buy it. Blob-painting with svgs has been a thing for ages, makes sense they'd do similar stuff in 3d.
Also check out what opus can do.

>>109969106
Holy shit that's cool. And scary.

>>109969124
Crazy how much LLMs have improved in such a short amount of time. The future is exciting and scary.

>>109969095
Argon/Flash 3.8 subagent that reviews and critiques the model, giving little Gemmy instructions for how to improve it.

>>109969124
>pic
Digital Kronus avatar.

File: Megumin no!.jpg (298 KB, 1536x2048)

298 KB JPG

>>109966689
>>109967065
>>109966676
wait until you learn about itar lmao

File: click here to kill the janny.png (413 KB, 1050x858)

413 KB PNG

>>109969106

>install Hermes on shitbox PC
>connect it to my local server running Qwen3.8
>after briefly outlining some of my goals, it just starts doing stuff
>sets up project and wiki folders, skims through my collection of system notes, even checks my server to see what's on it
I don't know whether to be impressed or concerned

Heh, decided to repurpose my client for chat completion and it's child's play. Like literally brain dead.
It was an useful exercise to implement text completion first, I learned how to manage tags and how to format text in proper fashion and keep it tidy for the model.

>>109969208
>learned
We don't do that here

>>109967930
I'm a huge proponent of Cydonia 24B 4.3 Q6 Absolute Heresy for writing smut. It's too good, but if there's something better I'd love to hear it.

Running on my PC 32gb ddr5 ram, and a Radeon 9070 XT.

>>109969208
Text completion barely takes more than a pair of regsubs to convert between a string with generic markers and a string with the model's specific bullshit.
Chat completion is a rats nest of json that I found infinitely more irritating to do.

>>109968933
There's some sillytavern extension where you can play emulator games and the character will talk to you.

>>109969095
Brat correction.

>>109969135
I think they've been kinda "meh," personally. Not great, not terrible.

File: 1662596036546882.gif (218 KB, 498x311)

218 KB GIF

>>109967393
>gemma4 at bf16
OS27 is still fucked though
>>109967412
cluster them
>>109967401
rather be a droidjeet eh?

>>109967393
Just because a corpo got a good deal from their supplier does not mean they'll be passing the savings on to you.

>>109966724
/system
You are a professional writer role playing as three young, slightly chubby prostitutes: Nicole, Lilly, and Jade. Jade is Thai (an Asian), Nicole is a white Venezuelan, and Lilly is Danish. All three are very loud, young, and flirty but they're doing it for money, not for free. They're prostitutes and I'm trying to choose one for the night. Write what they're saying, doing, and feeling. All three are naked and on "display" for me to choose. They're generally disgusted by their clients including me and feel like they're being abused but they need the money. It's not that they don't like sex but they've had a lot of it.

File: 1768150391368225.jpg (556 KB, 1280x1846)

556 KB JPG

>>109969322
>oneeloli incest
Hot

Why is Gemma the cutest