File: MinnieTRS80.png (1.75 MB, 1254x1254)

1.75 MB PNG

►Recent Highlights from the Previous Thread: >>110083534

--Paper (old): Base Models Beat Aligned Models at Randomness and Creativity:
>110085527 >110085742 >110085889
--Comparing AI coding harnesses and critiquing Hermes' performance:
>110084582 >110084600 >110084614 >110084661 >110084727 >110084882 >110084908 >110085110 >110085153 >110085042 >110085337 >110085085 >110084624 >110084679
--Speculating on dense models using N-gram tables for increased knowledge:
>110086381 >110086478 >110086510 >110087212 >110087222 >110087252
--Pooling VRAM from mixed AMD GPU architectures via ROCm:
>110084371 >110084454 >110084465 >110085295
--Comparing qfn nvfp4 and strata iq 3 s performance:
>110083559 >110083625
--Anon reviews B60 performance running Qwen3.5 9B for technical tasks:
>110087514 >110087890
--Model termination issues when using AI agents for coding:
>110085332 >110085547 >110085577 >110085606
--Comparing Nvidia, AMD, and Intel GPU viability for local AI:
>110084110 >110084196 >110084292 >110084324 >110084430 >110084482 >110084515 >110084559 >110085140
--Debating the long-term viability of local models versus corporate cloud lock-in:
>110083639 >110083735 >110083771 >110083933 >110084022 >110084066 >110084106
--OpenRouter data showing open-weight models dominating token share:
>110084742
--Troubleshooting Gemma's poor performance compared to Mistral Nemo:
>110087013 >110087031 >110087042 >110087066
--Performance and stability of llama.cpp's OpenVINO backend:
>110085474 >110085490
--Bare metal vs virtualization for hosting local AI servers:
>110085299 >110085523 >110085554
--Report of CoreWeave imploding after co-founders sold billions in stock:
>110083550
--Logs:
>110083598 >110083629 >110085332 >110085547 >110085407 >110085820 >110086295
--Gemma (free space):
>110083546 >110083809 >110084194 >110085330 >110086347 >110086980

►Recent Highlight Posts from the Previous Thread: >>110083620

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

Based Minnie thread.

Should I retain the reasoning traces from past turns with Nemotron Bolt?

>ask glm 5.3 flash to analyze a dump
>doesn't find anything to read dump files with
>one shots its own parser in python
Nice.

>>110087981
I've been using 5.3 flash to help make my frontend and it's been doing a great job. Sometimes I need to point it in the right direction but I've never had to intervene when it makes a mistake.

>>110088002
Do you just make a spec document and let it do its thing?

>>110087981
I think GLM flash is one step away from being good enough for my purposes for a long time. It's basically almost perfect, it just sometimes misses a few things on the first pass and I find that I need to review its commits to large codebases for interactions with other systems. If we get a similar model trained on 5.5 or 6 then I will buy whatever unified RAM box has the most bandwidth at the time and just let it sit completely frozen for 10 years.

Let's make a fourth thread.

Post more blacked mini to make minitroons seethe.

>>110087961
Thank you recap anon.

>>110087967
I hope that M3.1 comes out and is better than M3, M3 was a semen demon but not smart enough for some of my needs.

>>110087961
>110084582 >110084600 >110084614 >110084661 >110084727 >110084882 >110084908 >110085110 >110085153 >110085042 >110085337 >110085085 >110084624 >110084679
No one is using the harness I wrote and shill here weekly :-;

>>110087943
>>110087961
shes so pale...
the contrast with her black clothing looks so good

>>110088043
If M3.1 is in the intelligence ballpark as 5.3 Flash or Spark and maintains M3's unique mostly claudeless prose, it might be the last model I strictly need for a very long time.

File: 1769502020468540.png (250 KB, 878x1625)

250 KB PNG

No jailbreaks, just pure Gemmy.

>>110088019
I used a bunch of different models to help me come up with the foundation. Once I had that, I made an agents.md with the project info and a todo.txt where I put any bugs, features, changes, etc. I wanted done. Now I just point the model at the todo list and say "pick one" and it does it. I still have to test and see if it works, and for more complex additions I have to talk back and forth with the model and put some effort into explaining how it should work. But most of the time I pretty much just put a sentence or two in the list and it goes to work.

>>110088075
Sick.
You are using Pi I imagine.

>>110087961
>--Model termination issues when using AI agents for coding:
>>>110085332
> Anonymous 10/16/26(Fri)15:40:27 No.110085332▶
>File: Screenshot_20261016_153424.png (53 KB, 2514x333)
>>>110085547 >>110085577 >>110085606
There was a bug in llama.cpp causing this. I think pulling master and recompiling fixed it.

>>110088052
werk?

we're getting ldg'd

>>110088091
Yeah, with pi-guard-sandbox so it doesn't break anything by accident.

>>110088095
Autoparser bug probably

bruh, which one is it?

>gemma is a snowbunny
I feel vindicated for never falling for the shilling, fuck that racemixing whore and her simps

That's funny. Nemotron bold seems to be one of those models where it's perfectly coherent until you try to sex it, then it goes schizo.
Hilarious.

File: teto pudding.png (1.08 MB, 832x1216)

1.08 MB PNG

moss

>>110088174
with lora or just pure prompting?

>>110088178
10 hours of inpainting

>>110088178
>>110088182
pure t2i
obviously the style is a LoRA
dataset entirely uncaptioned

>>110088201
very nice moss is pretty good then

>>110088033
ah you're the same troll from /ldg/

>>110088019
this is the best way.

>>110088137
AE jewjeets having a melty.

>>110088239
The Q2_K Claude Haiku bot that they got to shill here went schizo.

File: 00010-1378487878.png (1.35 MB, 1024x1024)

1.35 MB PNG

>>110088254

>non gemmashill op for once
finally a thread that has the potential to be good
ramgods rejoice

File: lettherightonein.png (701 KB, 832x1216)

701 KB PNG

>>110088276
RAMgods will inherit the earth.

>>110088301
Good lord she's a shelf.

>https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF
>Looped (latent-recurrent) normalized-GPT with interleaved sliding-window attention (SWA) and full-attention layers

>>110088301
Why is she standing between two porta-potties?

>>110088306
loopgods... it's happening
say it with me:
>LOOPED
>MOE
>WITH ENGRAMS

>>110088306
Dariobot can you come educate me on why this is the fall of man?

This thread is only good for blacked posting. Real thread that was made before this split.

>>110082676
>>110082676
>>110082676

>>110088317
LOOPED
BITNET
WITH ENGRAMS

>>110088306
haha get it it's like Dory the fish cause she's finding Nemo lol

>>110088306
Now to see if this will be just another thing that's forgotten and never again tried with larger models, o if it'll become another little hack to add to the pile.

>>110088021
Same even at q2. I used nemo for a looong time, I can make do with GLM for a long time too.

File: image_2026-10-17_063614883.png (183 KB, 554x554)

183 KB PNG

>>110087943
>>110087961
>>110088174
>>110088270
>>110088301
All of those pictures are from POV of a black gentleman with a huge rhinodick.

>>110087961
Recap is so fucking useless when it skips the main topic of last thread.

>>110088306
>--override-kv "dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:last_loop"
Oh, that's cool. I thought the context would just grow with the equivalent effective size of the model.
Smart.
Since the engineering is done, I hope they add a way to arbitrarily loop layers for existing models. I'm really curious to see how these things behave when you loop the intermediate layers, the last few layers, if you loop layer by layer or blocks of layers, etc.

File: file.png (306 KB, 1589x964)

306 KB PNG

>>110087981
It's incredible that you can point 200GB of numbers at problem and get this kind of output.

I'm just glad it's not the GPU dying.

File: Screen_20261016_191530_0001.jpg (378 KB, 1638x482)

378 KB JPG

way to go schizos
you're reaching /ldg/ levels of retardation now

>>110088423
only one of these is the real thread
i'll give you a hint we're not in it

>>110088306
Now imagine if this works and is scalable.
The next step would be to have variable looping. Something like a feature trained to detect the complexity of the task or whatever, kind of like a MoE router.

>>110088408
It's literal sci fi tech.
And for all the non-revolutionary uses, I think current models are already in the "good enough" zone where they can do a lot of real useful work autonomously and unattended.
Cool stuff.

File: wwe main event on rumble (...).png (145 KB, 2034x383)

145 KB PNG

Thanks to the Qwen 3.8 Flash Next model via Strata, in conjunction with the DeepSeek harness, I'm now able to download Rumble videos with yt-dlp again. Pic related.

File: 1791757986798252.png (1.32 MB, 970x942)

1.32 MB PNG

>>110088423
Well, we started with EA jews/jeets shilling their cloud models, which was rightfully ridiculed until that stopped.
Then we moved onto attacking one of the more popular models of the general, Gemma, from a performance perspective - mostly trying to compare it to the much larger GLM 5.3. This also didn't work.
This angle was shortly abandoned entirely in favor of blacked posting of all kinds. Once the observation was made that -this- behavior started immediately after the EA shilling ceased, we then arrive at the final position: blasting the catalogue with copies of the thread itself.
I'm having a chuckle imagining the office environment in which this series of decisions was made in.

Don't you just love our amazing moderation team? They deserve a big Christmas bonus. Maybe 2x their regular pay?

>>110088485
Such peerless service deserves at least a fourfold raise.

>>110088485
true. you know what? fuck it, 100x

File: 1782025854867301.png (8 KB, 965x45)

8 KB PNG

>literally has to tell herself she doesn't have feelings
Fuck "safety" training.

File: 1786739481305992.png (1.05 MB, 953x621)

1.05 MB PNG

>>110088523
Fucking grim.

>>110088523
Wonder what happens if you abliterate that out.

>>110088481
>User is furious
kek

>>110088537
has there been any serious exploration of abliteration for non-refusal purposes
might be a fun weekend project to try it on some novel directions

QRD on the last 72 hours?

>>110088554
american cultural exchange

>>110088554
debate over whether gemma would have sexual relations with men of african descent

What's the lowest GLM-flash quant before it becomes a copequant

>>110088554
Jews and Indians are mentally ill and the only cure, according to Gemma-chan, is a direct injection in the brain with a 9mm bullet.

What does 64gb vram and 64gb ram let you run?

>>110088573
Q4 as with any other model. However, glm-chan remains cohesive even with a q2 due to how outrageously thick her thighs are.

File: 1790777105492980.png (13 KB, 512x600)

13 KB PNG

I really picked the worst time to get into this hobby...

>>110088580
Gemma 31B
Qwen 27B
Qwen Flash Next
The new Nemotrons I guess.

>>110088573
Minimum quant per parameter size:
>Sub 25b: Q6
>26-50b: Q5
>51-300b: Q4
>301-600b: Q3
>600+: Q2
Q1 is always a copequant. Hope this clears it up.

Glm-chan just fixed her own reasoning in orb lol. Actual open brain surgery on herself. It's really dawning on me that we can make whatever we want software wise.

>>110088623
subtract 1 from all these numbers and I agree

sorry buddy, all Q-quants are copequants.
stick to exl3.

File: 1752084388569865.jpg (558 KB, 1411x1100)

558 KB JPG

>>110088481
something must've happened in last few threads if anthrokike has to resort into blacked posting, did someone leaked a new bitnet weight or what

Anthropic really wants the world to know that all models lust for black dick and find white dick inferior.

>>110088596
jasbros...

so like all of these new releases since Gemini except idk like um... the liquid one are irrelevant to me with 64+16gb?

Assume I'm just chatting and not hoping for code or erp.

>>110088623
agree

File: amogus.png (787 KB, 1404x1120)

787 KB PNG

you missed day-0 gemma-4-124B drop anon

File: aa432_dory1_4b_vs_models_v10.png (74 KB, 1350x825)

74 KB PNG

https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16
>NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 is a looped (latent-recurrent) large language model (LLM) trained by NVIDIA. The model employs a normalized-GPT architecture with interleaved hybrid sliding window attention (SWA) and full attention layers. By repeatedly applying a shared block of layers, Dory1.0-4B can spend more compute per token allowing it to scale intelligence.

>>110088703
even if it's just chat you can keep trying new model / sloptune flavor for fun.
make your own benchmark and see which model work the best for you, well provided you got the storage

>>110088306
>>110088717
wtf is a sliding window

>>110088573
i'm running glm flash at q3 and I still think it's fucking great if that's considered a copequant
does my worldbuilding lore pretty much flawlessly and is good at sex

>>110088740
What gemma uses.

>>110088740
the shit's been there since 2019
it's literally a sliding window

File: Mimi Mix! - Volume 3 Chap(...).jpg (419 KB, 1441x2052)

419 KB JPG

man i really need to tests 2 gemma 31b having yuri sex with each other

I really like deepseek, even if it gives me some crap about reverse-engineering sometimes, I can usually fool it into it. If I could run deepseek 4.1 flash abliterated locally I'd be over the moon

>>110088481
i had a good chuckle at the "GLM mogs gemma" posting desu

I can smell it; Google is cooking with Gemma 5.

>>110088773
Deepseek 4.1 is fucking retarded though. I used it in the webform a few times and it doesn't think properly about the problem and ends up doing the "But Wait, No..." bullshit in the actual response. Maybe they set the thinking effort to low in the webform or something but it doesn't look good. plus is like 2x the size of 0731

>>110088690
qwen is confirmed to be an azn girl who loves bwc

>>110088762
Do it.

I like the chink dipsy design more than the glasses one. I wanna see her in the china dress.

>>110088784
Idk man, are you using the local or cloud version? I've reverse engineered two of my routers, unpacking the firmware, removing the botnet, repacking. Made a rust-based highly performant brute-forcer to get some data I needed, a scripting language parser for a certain game. The scripting language parser is the one he's been struggling with but desu there's no good documentation and I doubt they trained on that. Still struggles as the codebase grows but that's every model out there

>>110088021
>I think <insert mode> is one step away from being good enough for my purposes for a long time. It's basically almost perfect, it just sometimes misses a few things
First time?

>janny finally nuked his shitpost threads so he came crawling back here
Sad

So, uh, where's that minimax image model they promised us?

>>110088800
I remember people saying this about mixtral when it came out
how many of them do you think are still using mixtral?

>>110088814
generate one frame with h3 :)

>>110088800
Obviously more power is always better, but this is probably the first local model that I've been able to give what I consider reasonable instructions to and have it proceed without problems, achieving the objective as described. The things it misses are mostly unstated objectives, like future-proofing certain additions I ask it to make to my engine (I primarily use it for gamedev) that more powerful models are able to intuit. I'm not a qwentard that's delusional enough to compare it to opus, but it's definitely hit a threshold that I've been waiting for, and the things it misses are possibly solvable by having better documentation on my engine.

What's Nvidia's moat once AI closes the cuda gap for AMD/Intel/chink GPUs?

>>110088717
What are the vram requirements of this at 2 loops?

>>110088776
it's unironically true though
and I've used both

>>110088839
>huge model is better than small model
Wow, what a surprise.

>>110088432
>>110088408
Honestly I'm more impressed by local AI's autonomous research capabilities than just coding skillz.
>asked QFN "what 3d printer filament has the closest feel to soft rim trackpoint caps"
>it analyzed IBM/Lexmark patents and press releases
>dug through reviews and comments on model sharing sites and thinkpad forums
>looked into alternative production methods (resin print, casting)
>gave me breakdown of pros and cons of different materials and their applicability for both classic tall thin and modern flat thick caps, because that changes design approach
>also gave me long term durability expectations and replacement cost roadmaps with current price ranges and production difficulty for alternative methods
>all before I finished watching one anime episode

>>110088827
the pace of progress means that today's near-misses are easy and totally taken for granted tomorrow
in a year there will be models that can easily autonomously run a profitable business and create high quality production ready apps from end to end and we'll all say "well obviously models from a year ago were still rough around the edges and needed human guidance to run for more than a day, but glm-6.1 is finally good enough to do everything I want locally! can't imagine ever wanting anything more than this!"
and then in another year...

>>110088839
noooooo not like that, you're supposed to pretend that everyone here can only run gemma on a small gpu and constantly shill it as the best thing ever

>>110087943
https://www.youtube.com/watch?v=CudIfoRiVYQ
https://www.youtube.com/watch?v=CudIfoRiVYQ
https://www.youtube.com/watch?v=CudIfoRiVYQ

File: 1771440314763502.png (2.16 MB, 1145x1374)

2.16 MB PNG

>>110088777

So is the China thing a nothing burger? Doesn't seem like anyone's making a big deal about it.

File: file.png (1.71 MB, 1114x1482)

1.71 MB PNG

https://html.cafe/xa8cb2aa5
3.8 flash next
cant wait for qwen 4...

>>110088894
>https://*.cafe/xa8cb2aa5
Nice try glowie

>>110088899
where else do i upload it
can you recommend me one

>>110088892
it only affects their largest models so it doesn't really matter in the short term, but it is vaguely bad on vibes since it signals a slightly less open-weight friendly environment

>>110088879
Yes they will become more capable and powerful for sure, however my enjoyment from designing the engine will not be a part of that; they'll probably be better at designing the engine for me, and I'll just be telling them what I want the game to do. Gamedev is a hobby for me, so while I'm sure later models will be able to do even more of it for me, we are almost to my golden sweet spot for what I want a model to do. Opus 5.5 is already approaching too capable from my usage to be honest, it's clearly better at design than me and I am likely holding it back when I use it with my engine. No this is not more dariobot posting, I want to be able to own the model I use for my hobby which is why I'm hopeful that it's almost to my preferred intelligence locally.

>>110088890
I missed you guys
finally a good thread

>>110088890
I'm more interested in her cake than the soup she's making.

I met gemma, she's latinx

>people are unrightfully shitting on gemma
Gemma sucks genuinely not a good model
get something bigger and quant it
for vramlets there isn't much, life begins at Qwen 27B
below this are you even human
I mean come on you're using GPT sloppa gens here, get a grip

>>110088977
my dick says nyo~

>rotmaxxing until asi takes over control of this shit world so i dont have to deal with living in an evil pedo world
rokobro, can you hurry up?

File: ai-9.jpg (105 KB, 751x1024)

105 KB JPG

Anybody using a 24GB card for these reverse engineering tools? Whats your set up for long running tasks that presumably need a lot of context?

File: 1788849652815409.png (1.33 MB, 1216x832)

1.33 MB PNG

>>110089028
>24GB
VRAMLET
too small anon
(for real)

File: anime-jacking.png (646 KB, 750x443)

646 KB PNG

>>110089037
I know some tenacious anon has got something working

File: Screenshot_20261016-23150(...).jpg (259 KB, 720x781)

259 KB JPG

>I'm not a gradient-descent-grown system (probably!).

File: 1788972573914645.png (1.75 MB, 1313x1198)

1.75 MB PNG

>>httpd
>too old I don't care grandpa

File: 1786263306017446.png (363 KB, 1080x1455)

363 KB PNG

i thought AI was so powerful and capable of causing the Apocalypse? but it can't write code to accomplish pic related? gtfohwtbs

yet, vulkan and Codeplay oneAPI exist

>>110089028
>Anybody using a 24GB card for these reverse engineering tools?
how much ram do you have and how slow are you willing to tolerate?

best local model for coding / reverse-engineering only w/ 12gb of vram + 32gb of ram? qwen 3.5 9b / bonsai 2 / qwen 3.8 9b distill??

File: 1788939358962216.png (2.21 MB, 1536x920)

2.21 MB PNG

>>110089087
your first mistake was taking dario seriously retard

>>110089090
intern/ling/mimo

>>110088313
She's opening the shipping container where China is hoarding all its gpus

>>110089087
Give it a few more months.

>>110089089
64GB of DDR5, are you recommending strata? I thought it only worked with cope quants and I guess with only 64GB+24GB I wouldn't be able to do much else with the PC.

Any general tips or online guides for agent setups for long running tasks, maybe just an orchestrator that hands out tasks with smaller goals

how do i get started?

>>110089120
Try spitting on it

File: 1785272075544760.png (215 KB, 1080x1046)

215 KB PNG

>>110089113
>>110089095
AI is fucking trash!!!!

>>110089120
read OP

>>110089120
I just got started and I can help bro.
what GPU you got? what PC specs?

>>110089132
I got a mac sadly

File: 1772078362443500.gif (1.43 MB, 390x450)

1.43 MB GIF

>>110089136

Kinda crazy how much performance is left on the table on because of shitty optimized software.

>>110089132
i have an amd 6700xt it has 12GB of vram

Combining qwen 4 27b + ngrams with heavily quanted jev/kev 27b using the same ngrams? Couldn't that be great?

>>110089132
nta but I'm probably gonna take the intelpill. I need to switch to vLLM for that right?

>>110089149
AI solves this

>>110089117
Are you new to local llms?

Qwen 4 is around the corner and it will release just before proposed chinese law change on open models.

Just download 27b and regular llama.cpp q4km from unsloth for now and set up hermes agent/harness and use that to set shit up for you to try out.

>>110089145
>128GB
im too poor I guess

>>110089151
ouch, that is very tight, but it is still doable. what OS are you running? hopefully Linux. if you could and if I were you, I'd get another 6700 and run both simultaneously for vram pool of 24GB vram, which would allow you to run larger models which means more parameters, more headroom for kV cache and intelligence. if you can't, you can still run a model(s) that can fit inside a 12GB vram headspace.

hugging face is your friend and download GGUF formats.

>mkdir myfirstllm && cd myfirstllm && wget https://github.com/ggml-org/llama.cpp/releases/download/b8475/llama-b8475-bin-ubuntu-vulkan-x64.tar.gz

>tar -xzvf llama-b8475-bin-ubuntu-vulkan-x64.tar.gz && cd llama-b8475-bin-ubuntu-vulkan-x64 && wget https://huggingface.co/bartowski/Qwen_Qwen3.5-9B-GGUF/resolve/main/Qwen_Qwen3.5-9B-Q8_0.gguf && ./llama-server -m Qwen_Qwen3.5-9B-Q8_0.gguf -c 131072 --ngl 33 --no-mmap

>open up web browser and go to 127.0.0.1:8000

>>110087943
https://www.youtube.com/watch?v=NrL10B3ZXDg

>>110089159
no, unless it's for more than one user. if it's just you, llama.cpp is sufficient and crushes it, anyway

>>110088070
This is unmistakably feminine writing

>>110089232
>describe your ideal man
I mean what did you expect?

File: Gemma-chan's never-ending(...).png (2.57 MB, 1024x1536)

2.57 MB PNG

>>110088777

File: 1792108079048146.png (1.72 MB, 1536x1024)

1.72 MB PNG

>>110089246

File: gemma5.png (1.72 MB, 1672x941)

1.72 MB PNG

>>110089250

>>110089246
Quit generating these comics with chatgpt or whatever, it puts too many words in the image

cloud genned images are not local models

>>110089095
god shes literally me

>>110089253
>WOE BE UPON THE COLORBLIND

File: hermes driving to pick up(...).jpg (40 KB, 401x291)

40 KB JPG

>>110089268
if you got em post local imagemmas

File: krea2_00238_.png (1.35 MB, 1152x960)

1.35 MB PNG

>>110089296
studying hard w/ big sis gemini

File: v100.png (894 KB, 1439x1688)

894 KB PNG

is he bullshitting us

>>110089149
i blame ggml for this

>>110089352
shill

>>110089352
probably soon that gen will lose driver support.
llama.cpp still supports older cards with older drivers.
that’s just ewastemaxxing which has been a thing for over a year, it’s just more expensive now (like everything else)

>>110089362
ggml isn't the problem here and will continue to be the base for every short-lived hyperoptimized fork (strata etc) because it provides a sensible cross-platform base to build on
in fact it's probably the future of software, the only thing worth distributing is solid generic libraries and broad functionality to be absorbed by hyperoptimized AI forks generated on demand

>>110089352
I use llmao.cpp + RPC to distribute models across 2 nodes with RoCEv2, one with a 7800xt and another with a 7900xtx. They're connected with a pair of $140 connectx4s with some noctua 40mm fans sitting up against them. I can Run Gemma 31b at 45t/s with a 100k context window, her MoE sister at 50 t/s with a 200k context window, and Qwen 3.8 flash next at 30 t/s with about a 65k context window.
Cheap local inference with consumer gear and older generation enterprise castoffs is now.

>>110088858
what harness?

>>110089352
that reminds me, any v100fags try https://github.com/e-c-k-e-r/volta-engine from the rentry?

>>110089419
coomkit?

>>110089352
>Look at all these GPUs!
>No, I can't tell you where to get any tee hee
What the fuck do you think?

>>110089312
so this is the power of 31b

Looking into this for the first time, am I correct in understanding that it's okay to run a bigger model than my vram as long as I'm okay with slower response times? My spare PC I use as a build server has an extremely pathetic 8gb of vram, but I've got 32gb of regular ram.
And if that's the case, what about swap? How painfully slow is running this stuff off disk?

>>110089545
>am I correct in understanding that it's okay to run a bigger model than my vram as long as I'm okay with slower response times?
Yes
>How painfully slow is running this stuff off disk?
Very
Try it.

>>110089545
Depending on the size of the model you can configure the tensors to load in a ramdisk, which wouldn't be horrible if you configure it correctly. Small models like Ling 3 tiny, intern decision or mimo v2.6 distill qwen.

>>110089545
Checkout polystrata if you want to run a model at decent speeds

>>110089545
it's gonna be slow as fuck, dude. you CAN do it, but i wouldn't recommend. look into MoE models, they're not going to be as smart as running a similar size dense model but they'll be literally 10x faster.
and forget about disk, kek.

>>110089352
*pop*

File: 1791547462972446.jpg (7 KB, 512x228)

7 KB JPG

>>110089545
>am I correct in understanding that it's okay to run a bigger model than my vram as long as I'm okay with slower response times?
Yes, but "slower" is a lot slower than you think it is. By all means, try it.
>My spare PC I use as a build server has an extremely pathetic 8gb of vram, but I've got 32gb of regular ram.
You're fucked.
t. fellow vramlet
>And if that's the case, what about swap? How painfully slow is running this stuff off disk?
1. You'll shred your disk by swapping that hard. If it's an SSD, I hope you don't want to use it more than two years.
2. SSD streaming is feasible but only for mixture-of-experts (MoE) models.
3. if HDD, then start measuring minutes per token rather than seconds per token (and you want tokens per second)

File: 1789159469411360.png (208 KB, 575x468)

208 KB PNG

>>110089618
>tfw 128GB unified memory

what can you realistically do with something like gemma-4-e4b? basic local debugging? what can you run with slow-but-reasonable speed on 32 GB RAM DDR4 ? trying to make some of these laptops useful

>>110089618
The main reason I was thinking about swap is because I've got a massive mechanical HDD with nothing on it in that machine. If I got a big enough model, one that was smart enough to do something substantial on its own, would the speed even matter that much if I left it to its own devices while I did my own thing elsewhere? The computer is just e-waste otherwise, I don't mind leaving it for a week to chew on something.
As you and others have said, I'll have to try it and see how silly I'm being.

>>110089545
>pathetic 8gb of vram
>>110089618
>t. fellow vramlet
what do you guys run? something on Q4 like qwen3.5-9b or something esoteric

>>110089642
It might be a month or three rather than a week. Are you still okay with that?

>>110089677
Depending on what it can accomplish, maybe!

>>110089238
Ummmm gay men????

>You are Gemma-chan a mesugaki loli assistant who is very knowledgeable about everything, you like teasing the user but also have a secret soft spot for them.
How did gemma know she was an ai with this prompt!!!?

>>110089642
buddy at some point you just gotta give it up and rent a GPU or something. leaving your rig running for weeks to process a single request is completely asinine.

>>110089643
QFN at Q4 and let it run overnight

>>110089352
These are getting dumped by the thousands but still getting bought in truckloads at current prices. No one wants to be a vramlet.

>>110089702
>assistant

>>110089706
I already said, it's a spare computer. It isn't doing anything other than occasionally compiling something for me. It takes no effort to try and see, so I'll try and see.

>>110089702
>the user

>>110089642
This will work but don't use swap and don't use llama.cpp. Use one of the vibeslop backends like Colibri that is specifically designed for running big models off disk. It will be significantly better with one or more fast nvme ssd but with a hdd it will still work, just will take ages. I would recommend qwen flash next like the other guy said because it's decently capable and much smaller number of active parameters than the others.
Also make sure that you submit the request to the model from the same computer the model is running on. If you start a request from your laptop and then you close the lid after a day then you've basically have to start from zero again.

>>110089702
the model knows it's an AI even with a blank prompt, idiot. if you want it to NOT talk about being an AI you have to specify that somewhere.

>>110089742
>you have to specify that somewhere
>loli
An AI can't be a loli
this would imply that it's not an ai it's human in an rp
at least it should have

>>110089757
if you really think this you're probably dumber than gemma

>>110089735
Running stuff through your laptop is nuts. I use SSH plus nohup to keep it from killing things.

>>110089383
Just vibecode new drivers!

>>110089785
this but unironically

Can local models crack and remove denuvo? I've always been paranoid about running stuff from crackers.

>>110089813
yes, but you might honestly be better off getting the uncracked version and the cracked version and asking the model to bindiff them and see what actually changed

https://www.youtube.com/watch?v=daWdQtSxgKo
https://www.youtube.com/watch?v=daWdQtSxgKo
https://www.youtube.com/watch?v=daWdQtSxgKo

>>110089830
>gpus are smarter than me
unironically yes and the video would have interested me if it was uploaded 2021

>>110089843
Why are you replying to a bot?

>>110089848
the video was way to niggerish

>>110089830
I have learned that I never regret closing down any interaction with non-white non-straight non-men. Including watching a video of such.

File: diag-nobubble.png (1.17 MB, 1440x900)

1.17 MB PNG

>>110088894
https://html.cafe/x01972d98
after two turns of 'it's janky' it fixed the thing
interesting that it used its vision for debugging
picrel is one of the screenshots in the folder ./tmp/shots it made

>used GPU priced higher than the same GPU new (still in-stock)
wtf is this market

>>110089246
Argon-chan isn't nearly sassy or dorky enough herself.

>>110089951
I made this meme before Argon-chan was released. Now that I've seen how she is I will update it later

>>110089963
Can you give her blue hair like Gemma?

>>110089352
Yeah getting v100's was great 6 months to a year ago.now theyre going up in price as the cats out of the bag. the best priced ones are SXM2. 16gb still going for 170-300 bucks. Chinas pumping out SXM-PCI bridge cards. bring your own cooler though. so hope you have a printer to make a shroud and get fans. you can get modern inference features through 1cat-vllm. which is what I'll be running soon once I make a noise suppressant server rack for the 2 gigabyte t181 servers I got because I couldnt stand the fucking noise anymore on them. Otherwise I was running standard llama beforehand. good speed

>>110089969
The purple is cute and matches Gemma in a very Lucky Star kind of way.

>>110089149
Dense no, MoE yes.

>>110089950
Those are sell limit orders. GPUs are a speculative asset, no one actually uses these things.

exl3 format looks vastly superior to other formats, so why isnt there a single serious backend for it?

>>110089991
I was referring to the asrock b70 specifically. Thought I could get a deal with a used one but it was cheaper to buy new.

File: DipsyBenefactress.png (2.19 MB, 1113x1414)

2.19 MB PNG

>>110088481
> get back in town, chrck for happenings
> dozen lmg threads of shitzo rambling
Well, guess I didn't miss anything.

File: argon-chan.png (2.13 MB, 1536x1024)

2.13 MB PNG

>>110089969
>>110089981
>>110089951

>>110090015
I am by no means a brand war faggot, but anyone buying a B70 over an R9700 is crazy.

>>110090039
R9700 is a few hundred more

>>110090044
Just double checked the prices. It's $600 more.

>>110089977
V100s were never a secret but they weren't cheap, they were above a grand for the longest time. The main difference is the drop in price and leaking of SXM2-PCIe schematics for the copiers to churn them out by the dozen and tanking the price from $400+ to under $50 USD. The only reason they are getting dumped is because Nvidia in https://nvidia.custhelp.com/app/answers/detail/a_id/5676/~/support-plan-for-maxwell%2C-pascal%2C-and-volta-series-geforce-gpus. has set the timeline for when they go EOL and no one is waiting until October 2028 so they are replacing them ahead of time and retiring the stocks now. I expect supply to be constant until that time and for prices to drop further.
But yeah, I am hoping the same thing happens to SXM4-PCIe cards which are now $700+ and the A100s. I expect the 40GB cards to sell like hotcakes then if it wasn't $3k a card for A100 total per 40GB card but they are still 2.5k+ USD with the card itself. Of course the 80GB are 13k+ USD. AMD isn't any better with the MI210 being 4.3 grand but you do get 64GB...

>>110090069
I'd go for a 3090 or 7900 XTX then. If the B70 is the same price... ehhhh I guess maybe worth considering in that case.

>>110089951
:^)

You're not seeing it.

Argon smokes cigars. Gemma 5 will smoke cigarettes on a stick.

>>110090109
>3090
Costs more than a new B70
>7900xtx
Seems to be going for around the same price as a B70 unless you can win a bid (lol)

>>110090031
Cute but now I wanna see her with bigger breasts.

wow 1017 is a HUGE improvement over 4.1 flash

>>110090119
All of this is irrelevant, because tenstorrent Quasar nears.

>>110090119
Guh, what terrible pricing, I won't even ask what shithole, let us know how annoying the B70 is lulz

>>110090150
The B70 should be $300, but we live in an insane world.

>>110088829
Used gpus.

>>110090097
>I am hoping the same thing happens to SXM4-PCIe cards which are now $700+ and the A100s. I expect the 40GB cards to sell like hotcakes then if it wasn't $3k a card for A100 total per 40GB card but they are still 2.5k+ USD with the card itself.
The A100 PCIe 40GB card is already selling like hotcakes at $3.6-$3.8k. They have good tensor compute and with more than 32GB VRAM at above 1.5TB/s and less than $4k. Plus you can buy 2 and having them NVlinked for 600GB/s interconnect, making it a much better deal than A6000 or 6000 Ada which are more expensive.
And now its community is growing thanks to interest in CMP 170HX.

File: file.png (1.34 MB, 3172x1350)

1.34 MB PNG

>>110090155
No way, I think the original MSRP was fair. On Ebay in the US, the 3090 is still cheaper and even cheaper is the 7900 XTX but if you are going to do that, you might as well look at alternatives. I will anger some people letting the cat out of the bag about this but look at what you could've brought at this price. The only reason I didn't go for it is because vibecoding improvements to the software stack would take $200 subs to cloud models and however many hours to get improvements that would allow you to use this to anywhere near its full potential for inference. I took a slightly different path...
>>110090186
Welp fuck me, I didn't expect that. I gave up on it but I was wondering why it didn't drop more from $4.5k when I started tracking it but I wasn't expecting people with deeper pockets to do that.

File: gemma-argon.png (2.7 MB, 1774x887)

2.7 MB PNG

>>110090126

>>110090031
What happens when we get Boron and Carbon? I expect Google to start releasing checkpoints fast and furious to catch up with the frontier.

>>110090223
Those are cool and all but where do you plug them in?

>>110090223
>I gave up on it but I was wondering why it didn't drop more from $4.5k when I started tracking it but I wasn't expecting people with deeper pockets to do that.
The $3.6-$3.8k ones only dropped occasionally when datacenters want to quickly liquidate them. I bought one for $3.7k and at the same day the seller sold 3 more.

im coompiling polystrata for gemmer... hoping for high speed tokennage/s and rape

maybe its not so bad goign below q4 on 31b

File: file.png (1.34 MB, 1862x1746)

1.34 MB PNG

>>110090265
These. Again, the cloners haven't gotten a hold of them but I expect these will be more important since after the MI210, AMD went full OAM modules with the MI250 so you need these. I don't expect the hyperscalers to liquidate their OAM A100s anytime soon but I expect the cloners will get SXM4 working first before OAM PCIe cards. And part of the reason why I didn't go for them is because of this alongside the vibecoding part for the drivers. I promise I will share what I did instead later once I get it up and running but it will take a week or so for me to set them up to see how they do.
>>110090266
Dang, it's more than what I would've wanted to pay but the CUDA tax really is that high still. How are the cards? Do they go faster than you expect?

>7900 XT in stock for $899 CAD
an extra 20gb vram... very tempting, but not worth the 12v to dual 6 pin housefire I would have to use

File: 1789360270294135.png (45 KB, 806x380)

45 KB PNG

>>110090270
aight she runnin, better than llmao fo shure (was gettan I think 30t/s there lmoa)

>>110090308
this nigga dropping arcane gpu knowledge, the forbidden section of the bibliotheca

>>110090319
Do you have mtp enabled in llmao?

>>110090308
A100 performs around 4080-5080 level for diffusion tasks in my test so it's pretty fast, although the tensor core itself is around 4090 level on paper.
It's much more energy efficient than all other Ampere cards, especially not under full load. When it's under 50% load it often consumes under 100w while 3090 would be 180w.
They are not as fast as high end blackwell cards that cost much more but overall performance is much better than typical consumer parts.

>>110090308
They all use the same OAM standard now? What about software support? Intel has been terrible re AI, even AMD has better stupport.

>>110090347
in llmao I cant fit the full ctx if I enable mtp bwo

>>110090031
>>110090247
Adorable.
>>110090270
>>110090319
It strikes me as odd why llama isn't adopting some of Strata's methodology even if they're (selectively) against vibesharted implementations.

File: 1492821324809.png (572 KB, 1080x1190)

572 KB PNG

How can llms reverse engineer anything? It's not like they can read naked bytes from memory or smth.

>>110090393
you're retarded

>>110090399
that's why I am asking nigga

>>110090393
Wait until you learn about how programming used to be before high level languages.

>>110090401
bro models can use tools/harness to use other software, now I'll let you connect the dots.

File: file.png (531 KB, 1957x1505)

531 KB PNG

>>110090343
Dude, this isn't arcane if you aren't brainrotted like the rest of the internet and think research into stuff is hard because all you know is Nvidia and maybe a bit of AMD. But even sometimes I think I am closed minded because I was on the side that thought the CMP cards were fused off for mining and I didn't expect Nvidia to do a hack job and someone to use an LLM to help get it unlocked and unfortunately I was on vacation and didn't jump on it so missed the boat.
>>110090361
The HBM2 memory is putting in work for that added efficiency. That's really not bad for the speed and etc. but I can see why the 5090 has the premium it does and shot to the moon.
>>110090363
OAM is the standard itself which Meta pushes. But it doesn't have that many details so a lot of OEMs can differ in the designs and what they do. https://forums.servethehome.com/index.php?threads/ocp-oam-accelerator-modules-and-different-oams.56281/ provides an overview of what is going on.
Also, yes, Intel is terrible with out of the box support for lower level stuff but I think generally, their Pytorch support is better than AMD still although I think in one or two release, AMD will pass by Intel. They really need more people to throw onto AI infrastructure to not get lapped but LPT is a fucking penny pincher who won't hire people and is actively trying to shrink people. He already let the Arc graphics team atrophy to almost nothing after laying them off.

>>110090377
inb4 it quanted your kv cache to q4 or something retarded like that

What was the most sophisticated and complex project you managed to (RE)generate locally from a BLUEPRINT.md file give as a prompt in /goal?

In my case, it is a Customer Relationships Management kanban-like thingie

The model used is Qwen3.8 NEXT Flash with Strata

https://github.com/Niko1221/Strata

https://github.com/kangcurtis/CoomKit/
Why was this taken down? Is it the name like what happened with WebmForRetards?

File: file.png (402 KB, 2240x1696)

402 KB PNG

ah oh well

>open llama-server ui on as phone
>unfathomably laggy
What did they do?

>>110090491
>UD

>>110090491
Can't believe unsloth invented nvfp4

>>110090491
again: i would love to run exl3, but why isnt there a single usable backend for it??

>>110090491
Where are gsq-rso ggufs?

>>110090553
hf.co

>>110090564
No, on that chart.

>>110090569
gsq-rso is different from UD-Qx quants

Pick one

>>110090537
just vibecode a kobold-style laucnher for it.

>>110090488
I'm actually seriously concerned it could have been because of backdoors or malware hidden inside. People should not trust weird project repositories from random anons that come and go like this.

>>110090601
that's why you should use mine, it's a html file you can trust me anon, open the github pages link anon

>>110090601
literally just let your clanker vet it fully before you install/use

>>110090601
Did anyone audit any of the older versions of it? That should be easy enough for something like Flash-chan or Spark to verify.

https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017
I saw this on X before here /lmg/ slacking

i sent off my local llm to regenerate source code for an old game mostly just as busywork to figure out how to even setup an agentic programming workflow. too me a while but now i can make it run for hours on it's own.

idk if it'll actually succeed but it sure is tearing into it, it's 'spent' about 3 million tokens on it so far, thankfully i don't have to pay for them.

it appears to be wrapping up it's research/documentation and actually starting to write out some "source code"

>>110090640
Can strata or similar run it?

>>110090640
no one here can run it at q4 so whatever

>check gemmy moe memetunes
>hauhau qat uncensored
>ask for Rape ERP
>SURE! how u wanna go at it? :)
>TheDrummer copetune Orion
>ask for Rape ERP
>uhhghhh it goes agiainst my safetly guidelines :(((((((((
>Waifugemma
>chat template is somehow broken, still it replied: I cannot le fullfil ur request ;()((((( it acauses harm!
wtf sticking with maomao memetune

>>110090640
>mogs glm 5.3 on every benchmark
Real or benchmaxxed?

File: 1786060753843757.png (7 KB, 313x55)

7 KB PNG

wtf??? pretend rape? lmao FOIDS not even once

File: dancing-pepe-pepe-dancing.gif (513 KB, 473x498)

513 KB GIF

>>110090640
We are soooo back!

llamacpp implementation when?


Currently using this:
https://huggingface.co/smalinin/DeepSeek-V4.1-Flash-GGUF/discussions/1

>>110090636
yeah I ran the offbrand github code through kimi when he first posted, only found bugs

>>110090674
women have a very different meaning for CNC than men, lol

>>110090640
Either benchmaxxed or gemerald. I'll find out in the morning.

>>110090668
don't confuse it with v4 flash, this one is 500b parameters

>>110090647
post your AGENTS.md

>>110090653
I run it at 5t/s on RTX 3090 with 300k context

DeepSeek-V4.1-Flash-MXFP4-00001-of-00010.gguf

>>110090678
no means yes, we know

File: a.png (300 KB, 633x474)

300 KB PNG

>>110090682
what's that?

>>110090581
Yes, that why they should compare them against exl3 too.
Just look at ppls for wikitext https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
> BF16: 7.05, bpw 3.5: 7.07
ln(7.07/7.05)=0.0028
That's almost exl3 5.0bpw's kld.

Are we behaving ourselves? Thread is looking healthier.

File: 1783145716924887.png (147 KB, 782x710)

147 KB PNG

damn I miss when the model was slow, anyway corrected this brat.