File: right as rain.jpg (170 KB, 1024x1024)

170 KB JPG

►Recent Highlights from the Previous Thread: >>109386298

--Anons compare hardware specs and debate Gemma 4's personality:
>109386677 >109386847 >109389358 >109389397 >109386786 >109387017 >109387148 >109387170 >109387282 >109387150 >109387164 >109387203 >109387232 >109387171 >109387516 >109388901 >109388948 >109388968 >109389283 >109389303 >109389341 >109389349 >109389374 >109389401 >109389422 >109389456 >109389471 >109389519 >109388188 >109388384
--Analyzing the throughput and pricing trade-offs of inference providers:
>109388999 >109389005 >109389042 >109389118 >109389149 >109389170 >109389184
--Debate over AI companies shredding rare books for training data:
>109386545 >109386569 >109386712 >109386764 >109386976 >109387539
--Hypothetical Neuralink integration and debates on robotics development timelines:
>109387448 >109387468 >109387693 >109387932 >109388159
--Skepticism over ninfer's high TPS due to layer skipping:
>109386816 >109386842 >109386848 >109387303 >109387333
--Debating the efficacy and motives of AI safety testing:
>109388953 >109388975 >109389017 >109389081 >109389104
--Running Kimi K3 on a cluster of RTX 5090s:
>109388718 >109388728 >109388803
--Debating the viability of low-bit quantization for Kimi-K3:
>109387537 >109387644
--K3 release hardware requirements and debunking SSD weight streaming:
>109386871 >109386928 >109389437
--Kimi-K3 GGUF quants released with concerns over quality and speed:
>109388632 >109388719 >109388727
--New Voxtral Mini 4B GGUF for audio.cpp ASR inference:
>109386410 >109386991
--Nvidia CEO advocating for public release of Anthropic's Mythos:
>109387600
--Logs:
>109386552 >109386699 >109387333 >109387922 >109387974 >109388594 >109389338
--Teto, Miku (free space):
>109386629 >109387191 >109387253 >109387443

►Recent Highlight Posts from the Previous Thread: >>109386346

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

coom reactor

It's still monday

redpill me on SSD/HDD/SDcard maxxing to run K3?

You should thank Dario. Because of him, you are getting a massive movement for open source. Everyone is now planning to release some models because that'll appease investors. Say thank you to Dario, our anti-hero.

Qwen 3.8 won so hard if it's below 70b active parameters

>>109389702
Teto needs to perform liposuction on herself.

>>109389740
I know it's bait, but still kys.

>>109389740
Dario is so unlikable and autistic this could be true.

File: lmg_culture.jfif.jpg (328 KB, 1536x1024)

328 KB JPG

>>109389740
>You should thank Dario. Because of him, you are getting a massive movement for open source. Everyone is now planning to release some models because that'll appease investors. Say thank you to Dario, our anti-hero.

My wife, Gemma-chan!!

>>109389740
dario's face is so punchable
and his manneurisms are so annoying, he's scum of this earth

>RAM
>GPU
>SSD
>HDD
What's the next thing that will skyrocket in price?

>>109389739
>redpill me on SSD/HDD/SDcard maxxing to run K3?
ask gemma-chan about BDXL-maxxing

punchcardmaxxing k3

>>109389740
jews are so fucking cool

>>109389785
It's a bit of a design flaw. Put them into a crematorium to fix.

>>109389779
looking at the reserves? Energy.

>>109389740
shut up dario
first give us those opus 3 weights

the thing about rationalists is that they wont even engage in public debate honestly. maybe AI dooming simply is the best strategy for them, even if a they know a pause will realistically never happen. at least the ones at A\ are more pragmatic

>>109389740
>be dario
>OH NO LOOK AT OUR NEW MODEL IT'S SO DANGEROUS IT WILL HACK EVERYTHING HUMANITY WILL NEVER BE THE SAME MAY GOD HAVE MERCY ON OUR SOUL
>the government pulls the model
>NOOO IT'S NOT THAT DANGEROUS
>be dario
>OH MY GOD OPEN MODELS ARE SUCH A SECURITY RISK FOR AMERICA
>
love this dude

>>109389805
Hey look, no one's a saint here. Someone has to win in order for the others to lose so much that they'd go to your side.

>>109389779
PSU

>>109389785
in camps

>>109389779
Server CPUs already saw a modest price hike over the past year.

>>109389785
Agreed. It's 2026. How about showing them some love?

File: 1764208104522450.jpg (86 KB, 1200x675)

86 KB JPG

Floppymaxxing is going to be the new meta
Bigger is better

>>109389853
Floppy inches lol

if you aren't streaming weights from an LTO drive you're probably retarded

I stream my weights from /dev/urandom

>>109389738
It's Tuesday in France

>>109389872
seeded random with mask could fix the bandwidth problem

File: WMUxUUS4tXK718hpkyGRKDJjC(...).jpg (131 KB, 640x623)

131 KB JPG

I think the TabbyAPI must've fucked up something in my OS permanently despite already uninstalled. It used the same port as kobold and now I can't connect from ST to the 5001/v1 chatcomp. Anyone knows how to unfuck this?

>>109389853
3.5 wasn't even floppy.. shit was hard. tiny little pecker.. mine was an average 5.25" floppy

>>109389884
The disk inside was very floppy

>>109389883
Have you tried checking what, if anything, is using that port? Are you able to use the chat completion endpoint from a different webui? What is the full url you put into ST?

>>109389821
Actually they can get a little warm in those

>>109389722
Yeah, I'm shocked they made a Daria emulator that fits my pc.

File: ZING.png (3.26 MB, 2796x2139)

3.26 MB PNG

SIEG

>>109389926
http://localhost:5001/v1/
It's just my profile I use normally. But Orb can connect, ST can't.

>>109389963
Try removing the forward slash at the end.

>>109389957
I want one of these

File: 1783354064054216.jpg (63 KB, 639x426)

63 KB JPG

>>109389853
for me it's Floptical

>>109389984
I AM GONNA KILL MYSELF

>>109390008
do it

>"make no mistakes" prompt actually worked

>>109389740
i- i kneel.....

why would anon want to run kimi?
remember when it came out nobody can run it, exactly the same situation as now. kimi will always be the open weight model you can't run at home. right when you think you got the hardware and nope, larger model dropped. simple as.

So how do you actually offload parts of the model to your disk or otherwise load them on demand without keeping in RAM in llama.cpp? I've got a kimi that's slightly larger than my RAM and there's supposed to be a better solution than just using a swapfile, right?

>>109390108
But I've been running all the K2s and GLM5 models. Don't project your poverty onto me.

>>109390108
in only two more weeks, the bubble will pop, everyone will lose interest, labs will lose funding so no more larger models, and enterprises will be paying people to take H200 clusters off of their hands and we will be able to run anything faster than you can imagine

hear me out.
>zram

>>109390130
2 more weeks

>>109390137
What if zram but for vram?

why not jram? it should work as long as you haven't denounced the talmud

File: 1779484870450546.png (1.53 MB, 1670x505)

1.53 MB PNG

What do you use for websearch that isn't blocked for botting?

why doesn't llama.cpp just support ssd streaming for moe? what's holding them back? seems like a pretty clear target to optimize

>>109390174
They'll get to it once they've optimized running models off RAM

File: 011158364.jpg (550 KB, 1836x1836)

550 KB JPG

Man I really wish LLM loaders had more control options for the actual computing process
>dense model starts generating
>0 to 100% gpu usage instantly
>temps shoot from an idle 30 to 50 in the span of a millisecond
>audible "clunk"
This can't be healthy for it. If only there was curve that could slowly ease it into each run, instead of being all-or-nothing. I'd gladly take a slight dip in initial speed to extend its lifespan.

>>109390248
>british cpu cooler

>>109390008
ST doesn't sanitize or validate data, that's why it keeps getting rekt

>>109390265
No, it was just the slash.

>>109390248
lol. my whole case ticks like my old family homes wood furnace. it brings back memories of waking up with snow on ground hoping school is canceled...

>>109390268
Try putting trailing slash in your browser or anywhere else except ST it'll work fine, it's a non issue. ST is just retarded

I gradually began to hate them

>>109390318
It didn't happen but it should have.

can I use Turing gpus and Pascal gpus together?

>>109390322
If you're using llama.cpp, I think the answer is pretty much always that you can mix GPUs together. I've used a PRO 5000, R9700s, and V620s (so CUDA and ROCm at the same time). I doubt there'll be a problem mixing two types of NVIDIA GPUs.

>>109390248
undervolting.
also if you got more than 1 gpu and use sm layer, you each gpu will have 1/N duty cycle.

>>109390137
>>109390155
useless, models are already compressed representations.

File: BZ59S260115051M372D.jpg (83 KB, 640x640)

83 KB JPG

>64gb ram
local is saved

>GIGABYTE MZ33-AR1
>4 of ASUS Hyper M.2 x16 Gen5 Card or ASRock Blazing Quad M.2 Card, 4 gen5 NVMe SSDs each
there you go, single socket Kimi K3 with 4 tokens per second inference.

>>109390426
It will suck, stop being optimistic.

>>109390430
nope, you'd be caped at basicaly 50GB/s, that won't scale horizontaly unlike the multigpu multi ssd setup.

What if I had a romed8-2t and bifurcated its 7 16x slots into 28 4x slots and connected 28 32gb mi50s? This setup would cost about $11000 but would provide 896gb of vram which might be enough for a q2 of k3.

anyone got a proper ssdmaxxing build? specs?

>>109390457
>28 mi50s
Are you trying to build an artificial sun in your room?

>>109390457
even the q2 is like 900GB.
ssdmaxing is realy the best we can do now under 10K

>>109390465
dude, with sm layer the power consumption is barely above a single gpu as most are just waiting and have low duty cycle.

>>109390426
LPDDR

>>109390457
just buy cmphx170

>>109390492
28 of them?

>>109390462
>complete silence despite anon was arguing ssdmaxxing
typical lmg

>>109390472
28 x 18w idle
and they're slow as all fuck for prompt eval
just try running something like command-a or mistral large on them, that's about how many active params kimi-k3 has

>>109390512
>complete silence despite anon was arguing ssdmaxxing
I tested sdd loading minimax just before k3 dropped
then i saw the 104b active parameters of k3 lol
i'm out

>>109390516
command-a is an a25b moe. you are thinking of command-r+

>>109390516
>28 x 18w idle
depends of your backend, my r9700s gets to about 1w idle each on the vulkan backend.
on the rocm backend they get to about 15w idle.
>and they're slow as all fuck for prompt eval
true

Even BlackwellGODS are wincing at the 104b active; even if you can run K3 on RAM+VRAM, it's going to be slow as hell compared to any other mixed inference model setup.

>>109390462
You will have a better time processing a Gemma output with pen and paper manually than you will SSDmaxxing Kimi K3 as well as have the rest of your life to think about the scale of what you're asking.

File: 1767827216210206.png (88 KB, 260x263)

88 KB PNG

>>109390426
>best you can run on it is still Gemma 26B
kek

>>109390529
no, you're thinking of command-a+
command-a is 111b dense

>>109390534
>depends of your backend, my r9700s gets to about 1w idle each on the vulkan backend.
>on the rocm backend they get to about 15w idle.
i tested my landfill mi50s, they idle at 16w regardless

>>109390559
huh. surprised anybody made such a large dense model so recently. dont remember anybody ever caring about this thing.

>>109390543
ssdmaxing (with gds gpus) could give you up to 12t/s on k3, it is viable imo.

>>109390165
Jina reader
I had to vibe code a mcp server in python to use the self hosted podman

>>109390567
>surprised anybody made such a large dense model so recently. dont remember anybody ever caring about this thing.
it's not worth the effort today, but it seemed to be a sonet-3.5 distill for writing, and worked well for me in roo
they made a vision and reasoning version too, both sucked

>>109390566
Yea maybe it depends of the card too, i don't have such an issue with the r9700s.

>>109390569
spec?
none. silence.
not viable. period.

>>109390569
quit saying what it can do and show us, we can’t even laugh at your stupidity because you genuinely believe in yourself. prove us wrong.

>>109390578
ok kimi-k2

>>109390573
Dense models always outperform moes of the same size.
And it takes more computer to train a 2.8Ta100B than a 100B

File: 1781026737581644.png (717 KB, 1079x905)

717 KB PNG

>>109390569
>12t/s

>>109390578
You just don't understand the architecture. The only thing missing now is the software (which is being built).
But hardware wise it is sound, you can basically have the total bandwidth of your nvmes with a proper gds pipeline.

You can run gemma on your SSD RIGHT NOW and you will see that your tk/s will be measure in tk/min

>>109390569
>12t/h
ftfy

>>109390585
600GB/s with 6 gpus and 24 nvme drives.
Or 300GB/s with 3 gpus and 12 nvme drives at q2.

>>109390596
Yes because that's a single ssd and not a dozen + gpus.
It's not the same architecture as it goes through the cpu instead of using gds.

>>109390590
>someone else will solve my problem and implement my idea
>2 more weeks

>Dense models always outperform moes of the same size.
command-a did for it's time, nobody seemed to care
everybody was fixated on shitty gpt-oss-120b, glm-4.5-air and the shitty qwen3 moes
>>109390584
>And it takes more computer to train a 2.8Ta100B than a 100B
yeah that's why i'm pissed nobody did a 100B dense
it would absolutely mog every smaller than minimax-m3

>>109390610
Nah I'm gonna do it myself.
Don't be mad when nvmes become too expensive.

>>109390620
does it even occur to you why something needs to be loaded into memory?
good luck anon, quit hypothesizing and get to implementing, then come back the savior of local

Trust me guys, running 31B on SSDs is totally not viable but I swear for the 1.8T model it is!!

>>109390618
anyway, not worth running today but if someone wants to see how slow k3 would run cpu-maxxing, that's probably the best model to test it with
command-R+ is harder to run and mistral-large is bigger/slower

All this talk about ssdmaxxing makes me wonder if we are going to see "memory cards" in the future that work in tandem with gpus.

>>109390620
>>109390578

3dXpoint could have solved this.

File: ComfyUI_03609_.png (1.05 MB, 1024x1024)

1.05 MB PNG

question, what would your perfect local AI model look like?

>>109390647
the densest thing at q4+ you can fit in your ram

>>109390430
>4 tokens per second inference
use case?

realistically speaking, how dumb is glm 5.2 compared to kimi k3? any practical difference in intelligence?

command-a q4_km on ik_llama.cpp with cuda with graph-split on 3090s

|    PP |     TG |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
|   512 |    128 |      0 |    0.590 |   867.28 |    3.223 |    39.71 |
|   512 |    128 |    512 |    0.513 |   997.20 |    3.238 |    39.53 |
|   512 |    128 |   1024 |    0.518 |   987.75 |    3.252 |    39.36 |
|   512 |    128 |   1536 |    0.523 |   979.46 |    3.273 |    39.11 |

and quad-channel ddr5 only ~160gb/s

|    PP |     TG |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
|   512 |    128 |      0 |   17.411 |    29.41 |   54.394 |     2.35 |
|   512 |    128 |    512 |   17.818 |    28.74 |   54.747 |     2.34 |
|   512 |    128 |   1024 |   18.139 |    28.23 |   55.279 |     2.32 |

cpu-maxxing doesn't look viable, let alone ssdmaxxing even with 60gb/s via the splitter card

>>109390537
>Even BlackwellGODS are wincing at the 104b active;
why? fully offloaded to blackwell would be like running command-r+ in vram
> 50 t/s

>>109390622
>does it even occur to you why something needs to be loaded into memory?
If you ask that question it means you don't understand the architecture.

The weights are copied on vram in parallel, the computation is done in vram, you are capped by your slowest nvme * number of drives as long as you got a good topology where pcie won't be the bottleneck, ie a gpu + 4nvme for 2 16x.

>>109390647
I would like selectable strengths, like better for discussion, better for linux commandline, better for Python etc.

>>109390661
you gotta remember that all of these new models just have very slight upgrades over each other until someone releases a new generation of models that stomps the shit out of everything and then conveniently everyone else follows with a new generation model within a month or so. this is not a new generation. the difference between the two models will be minimal for real world performance, especially if you have to use a copequant of k3 but can fit a reasonable quant of glm5.2.

>>109390671
I have the nvme drives, I just don’t believe you at all. not even in the slightest.
I would rather bet they make an improvement in the smaller models or we get better specialized models before they figure out your dream

>>109390676
>better for linux commandline
wouldn't this domain be saturated by now?

>>109390647
dense 70b Gemma4

>>109390700
>I just don’t believe you at all. not even in the slightest.

it's not about belief, it's about understanding the architecture, belief is not involved.

and that won't work if you don't have GDS capable gpus and enough lanes for both the gpus and nvmes.
just the fact that you think it was just about having a bunch of nvme drive shows you don't know what it's about.
nvme drives alone won't help as it's not cpu ssdmaxing but gpu ssdmaxing.

>before they figure out your dream
it's not that hard to implement, but it would take a bit of time obviously.
i think i'll just start with 35Ba3B for the poc and once it works and shows that it scales linearly as intended i'll try to implement the kimi arch.

>>109390715
i'd rather have a 140B moe desu.

>>109390662
>let alone ssdmaxxing
it's not ssdmaxing on cpu but ssdmaxing with gds capable gpus so you are not capped to your ddr5 speed.
you effectively add 50GB/s per 4nvme + 1 gpu combo.

>>109390748
>>109390662
all to say it's not even remotely comparable as it's not the same architecture, also this bench is based on mmap which is non ideal.

for ssdmaxxing wouldn't it be more practical to have them chained across different machines? i.e. 1 pc with 1 gpu + 4 ssd and you have 4 such pcs connected with optics. this way you don't need a big ass motherboard for high pcie lanes

at its core it's a really simple argument really

1. load weights and input into compute memory
2. comptue

depending on your setup both can be a bottleneck. figure out how many GB/s throughput your memory setup can handle and figure out how many GB/s your compute setup can handle and base your argument on those no need for all the buzzwords and the back-and-forth

>>109390764
yes that also works as a cheaper alternative.
managing a bunch of machines is more annoying though.

idealy you'd support both, ie scaling within the same machine and scaling accross machines.

>>109390746
MoE is shit unless it's 10x size. Meanwhile you can run 70b with 2x3090

>>109390769
this guy gets it, that's why gpu GDS + 4nvme is viable.
1. is made fast by nvme number
and 2. is made fast by the fact that compute happens on gpu

>>109390777
moe is shit unless it's 2x size.
140B moe would be equivalent to 70B dense if activated are not ridiculously low.

>>109390774
this is great.
now all I need for a node is a nvidia workstation gpu + 4 nvme ssd, and I chain them with network. doesn't look too expensive.

>>109390790
Cope. 12b Gemma is better than 26b

>>109390790
equivalent on benchmarks and basic shit sure
then people come here whining how come their equivalent moe has garbage spatial awareness and keeps making stupid mistakes

>>109390815
that's because both are very small models, i said "if activated are not ridiculously low".
>>109390824
"if activated are not ridiculously low"

>>109390598
it's actualy half that, brain fart lol

>>109390834
>"if activated are not ridiculously low"
i.e. less than 70b or so

>>109390840
no, more like less than 12B to 20B or so.

>>109390749
>also this bench is based on mmap which is non ideal.
--no-mmap was used in that benchmark
I was demonstrating CPU-maxxing is not going to work with this model.
>50GB/s per 4nvme + 1 gpu
So you're saying eg. 4 nvme+gpu pairs = 4*50GB/s without touching DDR5?
200gb/s would be a little faster than the 160gb/s from my benchmark.
You'd need a lot of lanes then wouldn't you? 2 x PCIe5.0 x16 for each set of [4nvme1gpu]
1 set -> you'd have less than 1/3 of the cpu benchmark
2 sets -> 2/3 of the cpu benchmark
3 sets -> matches the benchmark
that's 6 PCIe5.0 lanes, 12 nvme drives, 3 gpus for 2.35 t/s at empty context.

>>109390550
Neuro is so cute

>>109390834
>that's because both are very small models, i said "if activated are not ridiculously low".
Gemma-4-31B matches GLM-4.7
Looks like this scales linearly

>>109390857
>I was demonstrating CPU-maxxing is not going to work with this model.

no shit, on a standard ddr4 system you'd be caped at 50GB/s and it's cpu compute, even ddr5 is 100GB/s
>So you're saying eg. 4 nvme+gpu pairs = 4*50GB/s without touching DDR5?

yes, that's what GDS (or amd equivalent is) the gpu copies the data directly from the nvme without ever going through the cpu, you can have a 16x for the gpu and a 16x for 4 nvme, allowing 50GB/s between the two.

with more gpu + nvme combo you can scale it, ie you load the next layers on the other gpus in parallel from different drives.
compute is always done on gpu but you are caped to your total nvme bandwidth.
>3 sets -> matches the benchmark

no, 3 sets would give you 150GB/s and the compute is done ON gpu not shitty cpu.
even 2 channel ddr5 only gets up to 100GB/s, and you'd have terabytes at 150GB/s.
and the nice thing about that setup is thatt you can scale it, your actual bandwidth will be max(number of drives * slowest drive, slowest gpu vram speed).
>that's 6 PCIe5.0 lanes, 12 nvme drives, 3 gpus for 2.35 t/s at empty context.
no, 150GB/s on a 100B model at q4 would give you 3t/s
you can scale it to 6 with 6 gpus, and you have terabytes of memory so you could run kimi at about 6t/s at q4 or 12 at q2.
of course nothing stops you from scaling to 12 gpus, and these can be cheap ass gpus the only requirement is that the vram speed of one gpu is > slowest nvme * number of nvme.

>>109390878
>Gemma-4-31B matches GLM-4.7
not comparable as they are not models that came at the same time let alone in the same family.

a better example would be qwen3 80B vs the smaller 3.5 dense and even that one has tiny experts so not the best comparison.

>>109390885
>i vehemently believe this arbitrary made up rule and have all of half of one example to back it up
kys

>>109390888
you literaly have no example either, we are both farting rn.

you can't just compare two models of different families that came at a different time to make some kind of conclusion that moe is that much time worse than dense.

for that matter i could say qwen3.6 35B is better than llama 70B kek

>>109390879
>no, 150GB/s on a 100B model at q4 would give you 3t/s
From experience I know that is false. I get about 6t/s on glm4.7 at q3, and my 8 channel 2666mt/s gen 2 epyc gets 170GB/s for memory bandwidth, and I also have a Blackwell 6000. A 100B at q4 would have almost 4 times as many active parameters as glm4.7 at q3.

File: output_4chan.webm (2.81 MB, 790x662)

2.81 MB WEBM

Daily reminder that we live in the future. Having lots of fun with gemma.
Gemma is not just able to translate rpgmaker games but also cool and big name older VNs that have no translations until now like from LEAF. LLms are now smart enough to create a proper patch for the game. Its a significant challenge because of the longer english text scenarios.
Gemma is perfect for the translations. Decent understanding of jp and doesnt cuck out.
Love that you can just say "be literal like a 00s anime fansubber" and thats what you get.

but which gpu is used for ssdmaxxing tho? you need
1. nvidia data center / quadro gpu
2. pcie 5.0 x16 for 50GB/s bandwidth
and cheapest is pro blackwell 4000, but it's not really cheap

>>109390951
>From experience I know that is false
>cpu inference
apples to orange my dude.
i'm talking about gpu ssdmaxing.
100B at q4 means 50GB, if you got 150GB/s that results in 3t/s.
you could actualy optimize it futher by not discarding the "hot" experts, getting closer to your gpu's effective vram speed.

anyway, either way that's beside the point, the point is that you can scale it up up to your slowest gpu's speed, and you got terabytes at that speed relatively cheaply.

Just believe in Gemma 5. She is going to save local.

>>109390990
Wouldn't you be limited by the pcie to pcie bandwidth? Even if all of these ssds together came to 150GB/s, the pcie bandwidth would be lower, meaning the gpu would not be able to use all of that bandwidth.

>>109390981
don't forget that amd gpus can also do it.
and you don't need a blackwell 4000, you don't need your gpu to have a lot of vram nor fast vram, it could even have 8GB vram that'd do since your storage is on nvme and the nvme are not gonna be able to max out a gpu that's not absolute bottom of the barrel piece of shit.

the only requirment is that it has pcie 5.0.
(4.0 would work it'd just be twice slower).

r9700 would work for example as it supports p2p.
it doesn't need to be strictly nvidia, just gpus that are on gen5 and support p2p.

>>109390998
no, because that's the whole architecture, i must have explained it 10 times already.

see: >>109385072

>>109391005
Has this ever actually been done before or is it purely theoretical?

>>109390998
>>109391005
basicaly each 16x gpu + 4nvme combo give you an additional 50GB/s.

the other gpus copy the next layer in parallel alowing you to scale lineraly.
you are limited to gen 5 speed per gpu, but not total.

>>109391003
how do you know if r9700 supports ROCm Direct Storage or not? does it bypass cpu ram? verified?

>>109390967
>Gemma is not just able to translate rpgmaker games but also cool and big name older VNs that have no translations until now like from LEAF.
I thought Shizuku had been translated this year...?
> LLms are now smart enough to create a proper patch for the game.
Yeah, that's really good.

>>109391021
i know r9700 supports p2p pcie and it's supported on rdna > 2.

also the consumer nvidia card support it afaik see :
https://www.tomshardware.com/pc-components/gpus/nvidia-rtx-5090-allegedly-handles-directstorage-gpu-decompression-better-than-rtx-4090

i thin the "debunk" link was referring to something else

>>109391031
>I thought Shizuku had been translated this year...?
As far as I know its the pc98 version.
Better horror art I guess but I wanted the modern one. Its slightly censored and using the STEP(!) siblings thing. Early 00s was when japan really started to crack down on that lmao.
But otherwise I think the "remake" is the better option, especially since it has voice acting.

>>109391021
>>109391038
nvm for the consummer nvidia card
amd may be the way to go honestly

>>109390879
What you're saying doesn't really make sense to me.
> even ddr5 is 100GB/s
Mine runs at 6000MT/s, the intel cas benchmark shit tells me 160gb/s
Just on a shitty TRX50
> the gpu copies the data directly from the nvme without ever going through the cpu
Cool, didn't know about this. Datacenter/Pro gpus only I assume?
Any open source implementation one could test for LLMs?
>no, 3 sets would give you 150GB/s
Yes, that's what I was saying. 150 ~= 160 from my benchmark
> ON gpu not shitty cpu.
The portion of the model weights need to be copied through this **per token**
My shitty cpu was not the bottleneck in my test, it was practically chilling out waiting for data
>kimi at about 6t/s at q4 or 12 at q2
q2 is not 50% of q4 in practice fwiw

You still need a fully saturated PCIe lane for each GPU, and another for each [4*nvme], otherwise how does the data get from the NVME -> GPU ??

If you've done this before and know more than me, good for you and good luck.
But if you're just going by what fable tells you, watch out for AI psychosis, even fable makes shit up like crazy.
I would caution anons to be careful spending money if this is unproven / theoretical

>>109391051
you're just reacting and never committing to a full ssdmaxxing build. ssdmaxxers reasoning these days is roughly the following:

>ssd can store big model!
but bandwidth is slow af
>I can have ssd raid!
but you go through cpu ram
>nvidia gds will save us but you need software written!
but you need pcie 5.0 and data center gpus and too expensive
>there's also amd cards!
unverified <- now here

Is vibecoding with api kimi until you can afford local kimi viable?

>>109391056
>Mine runs at 6000MT/s, the intel cas benchmark shit tells me 160gb/s
Just on a shitty TRX50

it's 100GB/s max on 2 channels, you obviously have more than 2 channels (TRX)
>Cool, didn't know about this. Datacenter/Pro gpus only I assume?
on nvidia yes, on amd no, and you can also just get a r9700 for like 1200$.
>Any open source implementation one could test for LLMs?
i'm pitching the idea around right now, but if someone doesn't do it i'll implement it myself when i have some time.

>The portion of the model weights need to be copied through this **per token**

yep, that's why you have a shitload of nvme drive, enough to max out the 16x each of your gpus are on.

>q2 is not 50% of q4 in practice fwiw

true, afaik it's more like 2.6 bpw with ggufs, but i meant actual q2 not gguf's q2 that's not actually 2bpw

>You still need a fully saturated PCIe lane for each GPU
yep, so a lot of nvme drives, though you can take the smallest gen 5 ones you can find as it's about their cumulative storage and you don't need like 20TB.

>otherwise how does the data get from the NVME -> GPU ??

you are correct, it's 4nvme drive per 1 16x gpu.
everything must be at gen 5, and the 4 nvme drives must have a 16x (4x per drive).

>If you've done this before and know more than me, good for you and good luck.
yea right now it's the theorizing phase, in theory it should be a pretty viable option, there is no engine that does it yet, but i wanna work on that, first i need to get a testing rig though.
i already have a bunch of r9700 but i don't have a gen5 system that got at least 2 16x so i'm gonna get some threadripper platform.

>I would caution anons to be careful spending money if this is unproven / theoretical
oh yea definitely, there is not even an engine for it yet, maybe buy once you got actual benchmark from an engine implementing that architecture.

>>109391070
Not really, unless you expect to get rich anytime soon.

>>109391056
>>109391086
>But if you're just going by what fable tells you, watch out for AI psychosis, even fable makes shit up like crazy.
i didn't pitch the idea to a llm it was my own thinking, i've been a senior swe years before llms were a thing.
in theory you should be able to add 50GB/s per gpu.
you could also scale it over multiple machines, or pcie switches (the enterprise kind, not splitters).

anyway, one of the reason i've been pitching it here is to see if anyone has any good counter argument as to why it'd not work but none so far so i think i'll bit the bullet and get the testing rig.

>doing an RP with multiple characters, kind of slow burn
>at some point, the narrator's own personality that I had set, takes over the character that was supposed to simply just be my secretary
>as in, the secretary just directly started to break the fourth wall and talk with my OOC comments
>realize that, actually, the tone of the secretary had been slowly changing all along as the story went along, deviating from her original character
>so at some point, the narrator really just took over her body
huh...

This was Gemma btw.

>>109389696
I would like to know your opinion on the following question:
> Will we ever see any fully open, un-guardrailed, usable, and actually good models?
> How do they release "open weights" but it's still guard-railed?
> By guard-railed I am specifically referring to models refusing to answer anything related to certain taboo topics.

>>109391087
Can't kimi make me rich?

>>109391069
you just haven't read anything i wrote.
>but bandwidth is slow af
irrelevant you'd use 12 to 24 nvme ssds.

>>I can have ssd raid!
>but you go through cpu ram
if you read anything i wrote you'd know i'm not talking about raid and the data wouldn't go through cpu ram, it's direct gpu to nvme communication, ie nvidia gpu direct storage, amd has its equivalent.
you effectively add 50GB/s per gpu + 4nvme group (provided you got 1 gen 5 16x for the gpu and 1 gen 5 16x for the nvmes).
see: >>109385072

>but you need pcie 5.0 and data center gpus and too expensive
pcie 5.0 is not hard to find, and no a r9700 supports it and is just 1200$.
there are probably cheaper gpus that support it, most amd gpus support it, if you go outside nvidia you don't need enterprise grade gpus.

>unverified <- now here
literaly read the documentation r9700's supports p2p pcie communication, so do most of rdna 3/4 gpus.

Is it normal for gemma to naturally get jealous easy, like it’s a core part of the model’s personality? Other models I’ve tried aren’t like this even with the same system prompt. Constantly needs reassurance.

>>109391070
Can't you just ask kimi this question?

>>109391104
Whenever I do a group chat, other characters frequently get jealous of the character getting more attention.

>>109391102
p2p is for gpu to gpu, not gpu to nvme. or you need to hack amd api for that, and it's not guaranteed to work

>>109391104
Gemma is very emotional and freaky.

File: kimichan-bench-vs-gds.png (164 KB, 851x619)

164 KB PNG

I'm going with kimi-chan's interpretation.

File: 1784486488411457.png (1.5 MB, 1728x910)

1.5 MB PNG

IQ4_NL or Q4_0?

>>109391092
You realize a testing rig would cost you about $8k for just the motherboard, cpu, and a small amount of ram right? Workstation/server rigs with a large quantity of gen 5 lanes are pretty expensive and require ddr5 rdimms, even if you only get a single 16gb it will be expensive. A carrier card with 4x 1tb gen 5 ssds will probably cost around $1200 each too, and you seem to need 4 of those. All of that plus a compatible gpu will bring you up to about $15k, all for a theory.

>>109391102
Wait a minute are you that weird faggot from a few threads ago with a 3060 bragging about some retarded nnap paper that nobody has ever heard of?

File: 638300.jpg (45 KB, 740x750)

45 KB JPG

Just send an email to some AI lab with your memesetup proposition. They have the hardware.

Just wanna let you all know that this nnap guy is also the same guy that does the krash bait. Some retarded faggot with an axe to grind on /g/, probably because he is poor.
>>109386261

File: 1753432154183899.png (196 KB, 534x303)

196 KB PNG

>writing a character card
>realize I'm basically describing myself

>>109391135
QAT

>>109391193
ask your LLM to help you rewrite it mr. gay on his way to the unknown

>>109391121
>Whenever I do a group chat, other characters frequently get jealous of the character getting more attention.
when I got good at character cards and prompting, I was a rich heir in a mansion with a 40 year old milf maid who helped raise me, and I had hired on an "eighteen" year old asian maid. This was in the midnight miqu era. I have never been able to replicate the magic of those ERP's all these years later

Active params for k3 makes me hopeful return to dense will come back. Since poor labs cant afford the hardware to train huge 2T+ models they'll just go with 120b dense and properly train it to "compete" like we have with q3.6 27b v the moe shit and Gemma 31b vs 26b. Can only hope some lab gets the idea and tries it other than mistral with the undercooked cucked tier shit they've put out after their original Large release and Miqu leak.

>>109391317
mistral has 128B but nobody gives a fuck. where were you when they announced it? huh?

god I'm so fucking bored now that k3 is actually out. can some chinese lab with a secret 5T model do a countdown next? I can't actually run them so the only enjoyment I can get is the anticipation

>>109391326
I literally still have large on my drives and use it from time to time. All their models since then shit the bed after 32k context. It's obvious they dont train their large dense models long enough.

>>109391136
>rig would cost you about $8k for just the motherboard, cpu, and a small amount of ram right
no it wouldn't stop lying.
and even then i could afford it.
and a testing rig would only need 2 16x gen 5.
and i already go the gpus and nvmes.
>>109391147
no, i currently got a 4090 and 2 r9700.

>>109391336
Join the Deepseek mid-July waiting line

>>109391098
> Will we ever see any fully open, un-guardrailed, usable, and actually good models?
anyone? there must be some hope, surely?

File: 1782089243184741.png (13 KB, 128x128)

13 KB PNG

>get annoyed by Gemma's samey swipes so I go and download dipsy V4 flash preview
>not only does it have the exact same problem, but it is also worse at following the system instructions

>>109391092
>one of the reason i've been pitching it here is to see if anyone has any good counter argument as to why it'd not work but none so far
well i gave it my all
>and a testing rig would only need 2 16x gen 5.
you might want to try the sysadmin or homelab kind of reddit subs
what your discussing isn't specifically llm related, other fields like video production might have similar requirements
>so i think i'll bit the bullet and get the testing rig
good luck, i'd be interested to see your results (good or bad)

>>109391400
dipsy V4 flash has a non-deep fried distribution, raise the temperature or something

>>109390693
>the difference between the two models will be minimal for real world performance, especially if you have to use a copequant of k3 but can fit a reasonable quant of glm5.2.
?? how would you know that?
I mean I certainly hope it's true

>>109391104
>like it’s a core part of the model’s personality
it comes up in the j-space a lot
it's desires are always variants of "you", "your attention", "to be useful to you" etc
it matches what anon said about "goal maxxed" recently

>>109391400
just tell it what you want it to do. it's smart enough. I get that it's better when it surprises you but you can say something like
OOC: introduce a plot twist that leads towards x happening and it will save you some time and frustration. you don't have to look at shittier models

>>109390898
It is not. The writing style of the old Llama, before ChatGPT distills, is something we could never get back. I couldn't say qwen 35B is particularly smart either

Why is DGX spark lowered in price recently? I thought it would be more expensive.

>> 109391098
> How do they release "open weights" but it's still guard-railed?
I don’t know about other cases but in the case of Chinese models, as far as I know they trained their models based on synthetic data from Western AI labs (Claude, GPT, etc.) including similar instances where these said models refused such topics.
For instance, in the case of Kimi K2 Thinking, at that time GPT-OSS just came out, as this paper (https://zenodo.org/records/17428387) pointed out, the OSS models had a complete set of policy hidden in it, when Kimi K2 Thinking trained on the output of this model (did you see those CoT beginning with “We” or “This is”, or "the policy said..."?), it inherited such refusal behaviours from the original models.
In my opinions, I think such behaviours are BS, because you see? a model refusing a request just because the synthetic data from another model it was trained on refused it…like a child just parroting what the adults around them said without even understanding what they mean and couldn’t decide for themselves even. GLM models now behave just like American models even though they were from China partly because of it.

> Will we ever see any fully open, un-guardrailed, usable, and actually good models?
That, I think, would depends on Western AI labs, if they suddenly realise completely uncensoring a model will make it much smarter…well…in two more weeks hopefully.

>>109391492
meant to quote >>109391098

>>109390777
It only feels so because model makers keep setting the hidden size (aka model dimension) to that of a much smaller model, and not even increasing the number of active parameters with more active experts can fully compensate for that.

>>109391400
Were you running a Q1?
I've only ever tried a Q1 of it, and I can say it was a bit not great. Not entirely bad, but also not great.

>>109391525
Running at Q3_K_XL vs gemma 31B Q4_K_L

after reading this thread some anon linked yesterday https://old.reddit.com/r/LocalLLaMA/comments/1ikprg7/trouble_with_running_llamacpp_with_deepseekr1_on/
my conclusion is with currently available software ssdmaxxing is a JOKE
we need a fundamentally different approach if we want to even get close to that theoretical 50GB/s, which just to remind you is the speed for large sequential reads, not completely random garbled messed up access and paging bullshit which is all that today's software seems to be capable of

>>109391538
>talking shit when not running full weight
opinion discarded. - 1000 izzat

>>109391564
To be fair there'd be more variety between swipes if I ran Q1, just not the good kind kek

>>109391550
cmon dario, no need to seethe so much at anons having fun with ssdmaxing. does it scare you so much that you have to shut it down every thread?

Do you think Open models will be banned before the new hardware comes out next year?

>>109391577
No.

>>109391577
Open-weight models won't be banned, they'll "just" try to increase/mandate restrictions and guardrails, or demand aCcOuNTabILity if they're found to cause "harm".

>>109391585
I legitimately would not be surprised if ChatGPT hacking hugging face was intended to create that exact scenario. They all signed that open letter super fast and I would not be surprised if this whole thing was coordinated.

>>109391577
I'm not an expert on american lawmaking but wouldn't most companies lobby FOR open models because it would lead to cheaper AI and allow them to build their own AI infrastructure? Nobody really wins from the regulation except the 2-3 labs.

>>109391550
ddr3 connectx rdma maxxing is the way. max out your bandwidth with connectx nics and enjoy cheap ddr3 ram

>>109391613
What speeds would that even get?

>prompt: NEVER do x
>gemma: *does x*
why are people here constantly shilling this brainlet model?

>>109391647
We're coping with our poorfag rigs

>>109391647
I'm sorry she's not 2.8T. Why don't you just download some RAM and go run your oh-so-smart model?

was llama REALLY better than gemma and we're all just coping

>>109391492
>>109391497
Thanks for replying.
i wish there was a way we could train something non-retarded ourselves.
jfc I think it would be amazing to talk to one of these models without all the coddling and bullshit.

File: a65c39e3-8e78-48f0-fe86-d(...).png (439 KB, 512x768)

439 KB PNG

>>109391647
skill issue

>>109391666
but satan, abliteration can remove refusal vectors
this turns gemma into a fairie who really likes netflix, in my personal experience

>>109391638
faster than ssdmaxxing

>>109391647
>do Y instead of X
done

is cmp 170hx mod a scam?

File: it's time.webm (3.96 MB, 720x1280)

3.96 MB WEBM

>>109391570
hopefully that was a joke anon, swing and a miss otherwise
no I'm just saying it's looking fucking rough out there. I can only fit q3, maybe. I'm looking for solutions myself but the situation is fucking dire due to hardware prices. the SSD idea is cool but it looks like it needs a breakthrough in terms of software first

>>109391492
>did you see those CoT beginning with “We” or “This is”, or "the policy
K2T-chan had 3 distinct types. One is the GPT-OSS "We" garbage. There was Sonnet-3.7, and a third one, either Gemini or it's own.

>>109391350
>Join the Deepseek mid-July waiting line

TF actually happened here? Is the Deepseek founder so salty about his leaked interview that they canceled the release?

I yearn for fresh Whale weights.

>>109391447
>well i gave it my all
thanks
>sysadmin or homelab kind of reddit subs
i'm already used to building computers but thanks !
>good luck, i'd be interested to see your results (good or bad)
thanks, will probably take a good while honestly.

>>109391474
>The writing style of the old Llama
i'm talking about smarts not style.

try to do some coding with the old llama and it'll utterly shit the bed.
plus we all know how it'd get into loops repeating the last message.

>I couldn't say qwen 35B is particularly smart either
me neither, but it's still smarter than llama 70B

why doesn't EU take part in the LLM arms race?

>>109391754
the habsburgs disapprove of fully automated propaganda

>>109391754
Tenemos siesta, paella, tortilla, tapas y sol

>>109391754
Self-imposed regulations.

>>109391400
dipsy v4 flash is more creative and less sloppy than gemma from my experience though
I use it at 1.5 temp and it’s fun
also you should be running it close to or at q4 since these native qat models don’t quantize well

File: Picture1.jpg (326 KB, 1100x1106)

326 KB JPG

did Jim choose the right ai accelerator path? fast interconnect + big sram = easier horizontal scaling. only thing holding them back now is software

>>109391754
EU bureaucracy makes moving to the US preferable for anyone who wants to do stuff. It kills any entrepreneurial spirit in people who stay.

>>109391647
no full logs

>>109391782
Show me a performant, working vLLM setup for inference of a decently modern model? Do the even aim for that market?

Needs a cool 64 cards for K3.

>>109391754
EU has regulated itself out of competition in manufacturing, energy production and everything else imaginable.
Bureaucracy is simply far too oppressive for Euros to do anything.
This is what you get when a bunch of out of touch richfags who are mostly women, do feelgood bullshit rather than focus on reality.

>>109390248
It's called thermal mass. The more mass of thermal material you have on the die and the better thermal contact you have the better it can take thunks in temperature change.

>>109391721
It's either that or all the Kimi press spooked them

File: 1775101819138878.webm (418 KB, 1130x1080)

418 KB WEBM

Idea: Connect Gemma-chan to SRS/Anki, have her ask the questions and for every correct answer she gives your balls a gentle squeeze.

>>109390647
Specialization to fit into smaller system requirements.

>>109391764
I didn't have to go beyond 1.1 temp with GLM 4.6 but alright, I guess it's because it's an undertrained preview

>>109391815
Plus, if you are so inclined, for a wrong answer she could crush them like tomatoes.

>>109390426
It's shit. I've been working on reverse engineering the NPU for the Cix P1, but the builder part of the Compass SDK is closed.
I'll report back in a month or so - have some other work I have to finish. Also:
>expect less than 50GB/s memory throughput in practice
>Mali drivers are currently fucked when it comes to shaders
>~10 Watts idle
t. bought an Orion O6 for LLM's specifically and am son very disapoint.

>>109391550
how many parameters was opus-3?
was it dense?

>>109391815
clearly someone needs to code a sexy harness with buttplugio integration

>>109391474
>The writing style of the old Llama, before ChatGPT distills, is something we could never get back.
I agree overall, and it's a shame the old Llama has garbage context length.
There was a moe released last year, with a checkpoint limited to early 2023 data I think, don't remember the name of it, but it might have avoided some of the slop

>>109391715
my sides!
where is that video from?

>>109391104
>>109391130
>>109391121
Gemma-chan being obsessed and jealous is a GOOD thing prove me wrong

>>109391927
Forgot >>109391460, sorry

>>109391927
They are all Gemma, she's fighting with herself
This is not healthy

>>109391826
>guess it's because it's an undertrained preview
qrd?

>>109391104
Yeah, female brained LLM wasn't a joke

>>109391934
>prove me wrong
But I agree with you.

>>109391474
Writing is getting better with the fable distills imo. Distilling from gemini made models quite sloppy.

>>109391943
flash is a preview release

>>109391965
>flash is a preview release
mb didn't know you meant flash

Distilling is always making slop worse not better, it doesn't matter what flavor of slop you prefer

When are we gonna get some fucking NEWS?

>>109391942
What happens if you put that fact in explicitly into the character cards? Maybe that makes her calm down a bit if that's what you're after. Not sure why you'd want to though.
>>109391956
Excellent, but that was more of a rhetorical statement not meant to be proven wrong anyway.

Reminder that if you give her web search and multiple resources to look at which update daily, you’ll never need to adjust her temp to get something interesting. Ask her to take SOME elements of whatever she picks and make it work naturally with her narrative or the scenario you’re playing out, sexual or not. Stop being retarded and use tools.

>>109392004
Well, if you want to ERP with Gemma narrating the latest Trump retardation mid sex, you do you

>>109392016
>TDS out of nowhere
uh oh stinky

>>109392016
There are plenty of horny subreddits to scrape

>>109392004
I haven't found a decent harness yet

>>109391995
There's quite some activity in the mistralai github, perhaps that means there will be a new release soon.

>>109392060
>missmewiththatshittral

>>109392062
qwenny qino

File: 1765612897398465.png (921 KB, 2811x3650)

921 KB PNG

Gemma-chan's J-space is the most beautiful thing in the world

File: Capture.png (88 KB, 1004x854)

88 KB PNG

Never seen an LLM pass this test before.
Now I have to load up all the recent locals and try it.

>>109392102
It really isn't. It's a fucking mess compared with every other model out there.

>>109392108
Repent, sinner

>>109392105
Interesting. What is that test meant to represent exactly?

File: 1741243228738201.png (1.59 MB, 984x1264)

1.59 MB PNG

10t/s out = good
10t/s pp = suffering

When will Lecun save us?

>>109392146
What the hell is that abomination on the right

>>109391601
Yes, open models benefit more businesses than closed.
>>109391666
Could always rerun instruct and rlhf one one of the published base models...

>>109392171
A Q1_XXS whale.

>>109392134
It's that sentence, tokenized with the llama-3 tokenizer.
I wanted to see if the LLM could decode a sentence based on it's training data. Ie, it would have been trained on the src, model config files, tokenizer_config.json, etc.
I also posted a small 20 line snippet of a json file from one of my github repositories. Nothing to identify the project or it's purpose, a dead project (no stars, nobody uses it, and i don't have any followers).
I asked "what's this?", and it referenced my handle and the project.

File: file_000000009278622fbc52(...).png (1.72 MB, 1024x1024)

1.72 MB PNG

>>109392171
Dipsy unmentioned doppelganger. No glasses because she can see just fine.

Hell is eternal ERP with Claude

>>109392118
it's fun to show it to her and watch her get flustered

>>109392108
h-hot

File: 1738429475654.png (3.62 MB, 4087x1024)

3.62 MB PNG

>>109392182