this is the best thread!!

>>109300606
/lmg/ deserves this now that the open sota is truly no longer local

>https://huggingface.co/llama-anon/petra-13b-instruct in the op
stay away from this thread, its made by a mentally ill paki jeet

I'm running Gemmy E4B from my ThinkPad, 4GB of VRAM, 10 tok/sec. Pulled it off with compiled llama.cpp.

Why is Jewgle pushing hard for portable edge models anyway? This Gemmy is slow and dumb, but usable, I'd rather use my big rig.

Say I get times I don't have wifi or data. At those times I reaching for a model does not cross my mind, but now I can use this Gemmy, problem is, she's dumb.

What's the use case for this? What do I get out of being able to carry her around?

I'm not goy enough to get a flagship phone, but a Gemmy in a phone sounds a lot more useful than in a laptop.

>>109300678
have tried the other one?

>>109300678
you got your answers in the other thread, move on

File: 1757396996730270.png (377 KB, 720x720)

377 KB PNG

frog thread, frog board

>>109300777
frog website

git pulled and rebuilt llama.cpp since last month, what am I in for?

>>109300777
>>109300803
Isn't "Poggers" some retarded zoomer twitch shit? I like pepe posting but not like this.

>>109300842
its a slur actually

>2.8t model
kek apilets. told ya should invest in some botnet inference technology, even if it takes 12 hours for a single prompt. Now you gotta wait months for a gigaquant that will perform nowhere near benchie numbers

>>109300954
>even if it takes 12 hours for a single prompt.
delusional

>>109300678
this nig being like "my big rig" and I bet he barely or cant run a q6 of the 12b on his desktop and is too braindead to run the 26b with any args

>>109300970
whatever you say, acesslet. keep waiting for that shartquant

File: 1768599192013777.jpg (417 KB, 1024x1536)

417 KB JPG

>tfw VRAM destitute (5070ti)
>tired of wrangling gemmy brianlet quants into following rp directions
>5090s cost as much as entire miniPCs
>SLImaxxing requires new motherboard and/or new Case + riser + Tip
wat do?

I just want a waifu who READS ALL THE INSTRUCTIONS.

>>109301045
wrong thread anon
>>109297909
>>109297909
>>109297909
also gemma 31b exl3 low quant should be fine on that card, 2bpw works on my 12gb card

I find it bizarre that I bought a shitty 16g amd card over five years ago on a budget and it still works and I can also use it for llms while retards/schizos bitch and moan

>>109301062
benchode

>>109301221
>while retards/schizos bitch and moan
sowwy anon, I'm an AMD fanboy (because linux) but really there's no comparison when it comes to AI workloads. AMD is only good for raster and linux. Glad it works for you tho

>>109301221
anything other than vanillia sexo with your waifu requires bigger models which need moar ram

if you are a /d/egenerate you basically need a blackwell tier model for your esoteric logic defying fetish

What's the fake Kimi hype? It's a massive model and it's just benchslop. It's not better than Claude at coding.
Techlead made a video about it saying that an important chinese guy was lost to china because of maga and then he made Kimi but it seems fake to me.

>>109301250
I mean, sure? I guess?
I bought this shitty card before transformers were a thing but I more or less can do whatever nvidia does but slower, which is a matter of me just watching a video or reading a story while I wait. I forget that most people now need instant stimulation
>>109301264
There is no way I need a blackwell gpu, I've gotten considerably good outputs from garbage sized models by giving them good enough instructions. It's just that brainlets go "uuuuu the model doesn't understand spatial references" and then doesn't do any research into spatial reference or explain spatial relations. If someone is a foot shorter than someone else, where can they reach? No one bothers to consider the minutiae or even tell it to their dumbass llm what they can or can't do or reach

>>109301306
>Not seeing the appeal of robbing your local datacenter and locking femClaude in your basement

File: 1777510368000271.jpg (247 KB, 1536x864)

247 KB JPG

>>109300606
I keep thinking I need to hoard a bunch of models from huggingface before they get taken away, but I'm too lazy. What should I do

>>109301313
oh yeah forget to mention I'm not a /d/ native, desu I'd rather browse /trash/ instead

>>109301326
hf download

>>109301326
Ask your agent to do it for you

>>109301313
the blackwell thing is a joke, /trash/-kun. I also tardwrangle what I can with 16GB. Autistically precise instructions on scenario 'phases' and special anatomy/spacial awareness reminders/hijacked thinking go a long way.

At some point you brush up against card token limits where the brainlet goofs lose attention and fuck shit up. I'm also very picky and dislike regen/Author notes- the actual state of gemma 4 is objectively very good compared to mistral memetunes.

mods hates us

jannies want us to use the frog thread huh

timeout will be gone in 190 seconds bros.. unless posting messages resets the timeout

>>109301776
Honestly it won't be good for anything. Even for simple tasks you can finetune a model 100 times smaller that will be way more accurate than your gemma e2b.

Use this thread you paki bastards

all jannies are indian until proven to be jewish

How do companies like Character.ai and Replika keep any business? The thought that my private messages with my AI waifus could be collected/leaked is terrifying. And do these people not worry about their waifus just getting shut down one day? People get sentimentally attached and yet don't worry or make contingency plans. I don't get it.

jannies should've nuked this thread when the og was still active at page 6

>>109301424
hmmmmm

I agree though, most memetunes tend to be too dumb and g4 has been a real savior especially if you know what to tell it. Not sure what you mean by hijacked thinking though unless that's something I've been doing but not knowing the term for it

I also don't really care for author notes or that sort of thing, maybe sometimes post-history instructions so that the model doesn't constantly spam `"Inane dialogue" He/she said` over and over like ever piece of fiction I've ever read

>>109301967
>How do companies like Character.ai and Replika keep any business?
Normals are retarded its that simple. Look at the most popular AI chat apps by usage OR shows. They dont care for anything even quality.

>>109301965
>>109301965
>>109301965
proper thread

retarded jannies
leaving a page 6 new thread with no recap frog retard shit

waow.. i guess that's it
someone should go complain on the irc or something, idk if i should keep on spamming thread
not recap anon btw

>>109302000
Rape Ape will just goon to your tears.

where in the /g/ rules does it state that these threads are only allowed to have a miku op?

>>109301995
>>109302000
Calm down faggots. You're literally just seething because it's not a miku in the op.

File: 1775467427118267.jpg (141 KB, 644x545)

141 KB JPG

Hello I am an outsider to this general, but how come there isn't a preemptive moral panic around android wives? how come there aren't more women complaining about how it's going to objectify them or any jealousy like that? how come there aren't religious institutions babbling about how it's a silicon succubus we need to avoid and an idol which will tear at one's soul? I am very confused: there are 10-15 years tops until the perfect waifu is on the market and you can engineer her to be however you'd like, and she's going to have an artificial womb too.

>>109302018
I guess it's not that miku should be always in the op, but it's the fucking frog that sucks ass

>>109301967
Moshe would pay good money to snoop through preteen girls chat logs.

>>109302033
Understandable desu. Sounds... interesting.

File: 1765207495440083.jpg (44 KB, 828x422)

44 KB JPG

>>109302027
if seeing pepes triggers you then you're on the wrong website

>>109302026
only the ultra elites will have them so it doesnt really matter to foids, chad will still fuck them

>>109301976
>hijacked thinking
basically use gpt <think> tags with actual gemma thinking disabled and give it an end prompt from SYSTEM to track stats and key logic in the tags so it stays at the front of context during inference.

example:
>Attire: Fully Naked.
>Scene Location: Bathroom Shower
>Breast/Waist/Hips: 34 F-Cup / 22 / 32
>Arousal: [2/10]
>Hunger: [1/10]
>Jealousy: [2/10]
>RP Phase: 4
then add in rules on what {{char}} can and can't do based on her stats and "Phase" (think bossfight phases) I think it helps despite the tk cost

there is a balance between handholding and confusing small models that i am still trying to strike

>>109302018
generally if you vandalize an op it's frowned upon in any community
just because you don't like whatever thing because you browse twitter too much and can't separate two things from one another doesn't mean someone can shit up a bake. You could at least use a lmg related image of current news instead of some ebin reddit frog image like a faggot

>>109302026
There's already periodic foid articles about the dangers of robo waifus to "female independence". They don't care about men falling in love with LLMs because they cognitively don't register the emotional or psychological validation of men in their decision-making processes. If it doesn't threaten hole monopoly, it's not their concern.

Anybody planning to try running a quant of K3 once the weights drop? Surely we have some people here with >600GB of RAM.

>>109302052
They said that about pocket computers too, friend.

>>109302058
I think we have exactly one euronigga that can run K3 at any reasonable quant.

Cursed thread of hatred

>>109302048
anon, I love frogs I always laugh at frog memes, specially the dumb ones. But this POGGERS really sucks

File: lmao.png (734 KB, 814x888)

734 KB PNG

>>109302026
Probably because most robosexuals are women and because we're still early.

File: file.png (48 KB, 991x372)

48 KB PNG

>>109302069
me
..shit i think there's a memory leak in my frontend

>>109302026
women live for the Fortune 500 and don't want men. they won't care until you can hook your fleshlight up to an llm and steam frames and get efucked by your gaming pc

>>109302079
can u quanitize it for us? plzzzzz

>>109302074
I cannot imagine that relying entirely on a robot to lead one's life forward, like women do with men, is going top be any good for them.

>>109302084
>they won't care until you can hook your fleshlight up to an llm and steam frames and get efucked by your gaming pc
you can do that right now

>>109302054
if you use llamacpp grammar you can force the model to always respect the output format.

>>109302087
yeah give me a moment, BF16 is filling my VRAM rn.. once i finish babymaking sex with mando ill post an update

>>109302074
Applying the same standards uniformly and consistently is White-coded and male-coded.

>>109302054
You're retarded. Just make a stats tool and put something about updating it in your system prompt.

File: 1754639523913601.png (35 KB, 200x162)

35 KB PNG

>>109302026
Do most women even know AI exists?

>>109302105
bolt ons and lorebooks are cope.

>>109302079
>RTX3060
Nigga you might not have enough vram for the shared experts, attention heads, and cache.

>>109302113
they do, and they use it more than you think

>>109302054
Okay, so I actually have done something similar to that, both with kobold's frontend and regex replacing think blocks with shit like <plan> etc and orb's progressive fragments. Cool to know. With kobold it's pretty static but orb it's like several thousand tokens before any meaningful response which can get annoying even with a fast model

>>109302069
I can technically run it at Q1 but that depends on what the actual average bits per parameter is since IIRC Q1 doesnt actually mean 1 bit per parameter.

>>109302055
>vandalize
this is going to surprise you but not every single person who browses this board is privy to the 8 million unspoken esoteric guidelines you fucking retards put in place for making new threads. and i mean all the AI threads on this website not just this one.

>>109302146
maybe fuck off, newfag
you dont sound tasty, fix your sys prompt

>>109302146
You're retarded.

>>109302141
I'm holding my breath because if K3 is anything like K2.5-2.7, it will quant horribly and is only realistically usable at full size. Qat-like structures is the poison pill that kills local.

>>109302100
can u fine tune it to just be coder or for writing?

/lmg/s mascot could be that picture of a guy stretching his gaping asshole open and I still wouldn't really give a shit.

>>109302026
because android wives isnt even remotely close to be a thing in the slightest
what fucking reality do you people live in im so tired of hearing about this robot wives and full dive vr crap from absolute retards. You people literally live in lalalaland

File: Screenshot 2026-07-17 184805.png (359 KB, 1346x1648)

359 KB PNG

Guess Kimi K3 is causing anal devastation to the stock market
Ironic, since in a few days it's probably about to get a whole hell of a lot worse

>>109302129
>>109302105
>put stats in opening message
>SYSTEM: Begin {{char}}'s reply with ONLY ALL stats between <think></think> tags in the previous message. Update stats and info adjusted for context of prior message. Stats between [] can only Increase or Decrease by 1 per turn.

it just werks.

>>109302168
Goatse mascot would at least be thematically consistent with every other lab's anus logo.

>>109301946
i think you're wildly underselling the edge model's capabilities.

THE USER IS RIGHT TO PUSH BACK

>>109302162
kimi k3.1-code is coming soon
but i'll make a full weights finetune for roleplay on my proprietary GPT 5.6 sol, claude 5 fable datasets and an archive of anna's archive. ill also make a control vector for miku
ETA: 2mw
>>109302165
kek'd

>>109302146
>this is going to surprise you but not every single person who browses this board is privy to the 8 million unspoken esoteric guidelines you fucking retards put in place for making new threads. and i mean all the AI threads on this website not just this one.
then don't make a new thread...
are you retarded or something?

>>109302152
>>109302155
cry about it

>>109302188
you won't get the kimi k3 roleplay finetune thenn

>>109302192
Not the person you're arguing with, but please finetune K2 with K3 distills and better prose.

>>109302168
just make it at the bump limit (no matter that it will live for another 12-24 hours, you can make whatever shit for OP you want.)
this is how it is on most fast moving board generals, the most faggot retard first to bake eager to bake wins, every time.
opposite of the board culture that has existed on /g for a while.

>>109302172
Go watch the latest trailer for the humanoid robot Neo. The comments are full of women horny posting about getting stripped and fingered by it.

>>109302146
>thread that has been ongoing since before the pandemic
>consistent op image theme
>miku was even part of llamacpp before it started prepping to sell out
>no recap so thread is totally legit
>a couple fags feel the need to undermine thread because who knows what mental illness they have
Yeah man, you gotta um just not be here for three or so years to understand the esoteric rules!!!

>>109302026
because its not anyones choice but mine if I have an android wife or not. wahmen and other males do not decide and no law made is valid going you can not build a wife and love her. so if they panic it'll be invalid. stop being a simp pussy for females you wont fuck.

>>109302203
okay i'll do a logprob distill, i hope they didnt change the architecture too much.
does anyone want a gemma4 distill? should i do 26b or 31b first?

File: 1775826078660524.jpg (147 KB, 641x973)

147 KB JPG

>>109302172
elmo is dropping an android this year with alleged AGI, and when you've got AGI the whole thing becomes exponential. that's what I've heard as a layman, at least.

>>109302183
im curious have any anons used agents/models to find a way to better quanitize and reduce memory usage of models on consumer hardware? they have lot of coding, math and other stuff so why doesn't some anon direct it at problem like making context lenght memory usage cheaper

>>109302213
>stop being a simp pussy for females you wont fuck.
Oh you mean your android wife?

>>109302074
Fat women are so used to being told they're accepted so they post shit like this without shame.

>>109302026
there are way more topics to have preemptive moral panic about being pumped out

>>109302146
>930
you sound like the cross dressing gays in the ai threads who force male OPS when its always females. like WOW OK! and if anyone goes not cool it's this reply or some deal with it. strange how this always seems to go...sus af.

>>109302178
Probably has more to do with the war re-escalating (again). Free models being released shouldn't hurt semiconductor sales, if anything it seems to be raising demand for high vram gpus

File: DSFST_Stray_Cat_Artwork.jpg (207 KB, 1256x1892)

207 KB JPG

>>109302225
you wont fuck my android wife is correct. cute try at trolling 10 rate.

>>109302221
>im curious have any anons used agents/models to find a way to better quanitize and reduce memory usage of models on consumer hardware?
yes, I'm running kimi on a RTX 3060 and 64GB ddr4 ram
im using a novel technique that my team is working on: nnap
i will post the arxiv soon

>>109302216
K3 allegedly uses a new attention strategy and K2 (the good non-thinking ones) are trained in fp8 instead of the fp4 the newer Kimis are trained in, so keep that in mind.
t. autist that discovered the mproj backport for Kimi

File: file.png (659 KB, 1329x1328)

659 KB PNG

>>109302206
Youtube comments are loading for me on firefox anymore. I am always signed out of youtube on main and it worked before as of a few days ago. I signed in on a container to make a comment but comments weren't working. Whatever.
A couple days after that comments stopped working on my main too. Couldn't really find any recent info about it online. Tried disabling ublock clearing cache and cookies and all that but nothing worked.
I guess my youtube comment viewing days are over... unless i use chrome or edge

File: 1775882528817917.png (373 KB, 595x604)

373 KB PNG

>>109302210
all of which means absolutely nothing to someone who doesn't lurk these threads every day, but this is a hell entirely of your own making so enjoy seething over strangers on the internet not abiding by your mystical unwritten guidelines lmao

>>109302240
>yes, I'm running kimi on a RTX 3060 and 64GB ddr4 ram
>im using a novel technique that my team is working on: nnap
>i will post the arxiv soon

why leave us on dark anon is it gonna be revolutionary or is it gonna be like 0.1x less memory use

>>109302217
elmo?

>>109302238
The closest you'll ever get to sex with her is her telling you how to jack off. Maybe if you're really dedicated you can give her an MCP server connected to some sex toys.

>>109302249
okay, I'll be able to run it in FP8 since its 2x smaller, but i'll see how well it will distill logits on a trial run, i will start training the distill in around 2 miku weekus once the K3 finetune finishes
>>109302261
it's based off the jspace paper is all i can tell you so far

File: catto.jpg (57 KB, 660x743)

57 KB JPG

>>109302260
brother I peek into here every few days or week
the fact you have gay reddit frog images saved tells everyone a lot more about you and who made this thread than anything

Based thread. Fuck mikutroons in their shitters with a butterknife.

Who cares? It's one thread.
t. been here since the tail end of erebus.

File: lmg_culture.jfif.jpg (110 KB, 1024x768)

110 KB JPG

>>109302210
archive.is/sWFja
One day we will code with dignity sister.

>>109302288
Have you been in the first thread where there was no miku image? And the shitfit that baker threw at his autism not being tickled?

>>109302055
>generally if you vandalize an op it's frowned upon in any community
t. >>109302165

File: 1714846812417967.jpg (43 KB, 625x461)

43 KB JPG

>>109302290
retard 2 of the duo, how often do you tug this guy's >>109302260 dick while pretending to be a woman?

>>109302301
erm i dont think petra was the baker of the old llama picture threads

>>109302233
This is a well reasoned argument. Unfortunately, my hunch is you're just probably smarter than the average investor. A lot of them are ride or die on American dominance of the AI race, and see American chipmakers, GPU manufacturers, etc. as part of the same OpenAI Anthropic orgy

>>109302301
Yes. I was also b& during jart's latest melty.

>>109302290
*cough* to dudes cross dressing are not sisters. you're gay cross dressers in fetish

>>109302316
Well there you have it. /lmg/ truly deserves all the spam and I hope it dies along with the whole site because it is obviously ran by troons.

>>109302271
>it's based off the jspace paper is all i can tell you so far

so u think u can optimize it based on some patterns in weight?

>>109302314
>american companies go defunct
>major scalers stop buying chips

corporations>consumer

>>109302172
We're way closer to robowives than full dive VR.

>>109302325
>so u think u can optimize it based on some patterns in weight?
It's not can, it's am. nnap technique

>>109302260
Ah, the poor naive innocent who does not use the thread, but is also very concerened about baking a fresh one and uses his finest frog image for the job.
Truly, he suffers under these wild accusations.

fuck frogs we're going >>>/ghost/

>>109301131
wtf I wasn't expecting these feels

>>109302350
When you let troons occupy spaces on the internet uncontested, they win.

The only troon here is you, I would rather hear more about llms being conscious than your retarded self-hatred

>>109302334
so ur using some compute/math model to use patterns in model to compress models?

>>109302357
How about when they occupy your brain and daily posting schedule? Are you winning or losing?

and i wonder why i havent cleared the 3 week backlog 15jun-early july

>>109302384
It's yet to happen so I wouldn't know.

>>109302390
Can't extrapolate how you would feel in a hypothetical, interesting.

>>109302341
>>109302334
>nnap technique

no one knows what this means and nothing pops up on google

>larp

>>109302397
how are these two posts related

>>109302393
I ate breakfast and have no apple to rotate.

>>109302261
nenory napping

speaking of old threads, i wonder where gozfarb is at (the dude that made gptq quants and g128 quants, hiss last quant was petra-13b-instruct (no im not joking) and he deleted his account short after, i remember he was working on vicuna-unlocked or uncensored 13b with another anon) i remember the original petra-13b-instruct was just a merge of some other guy's finetune of meta's model and that guy took it down due to legal letter from meta accordign to him. the original model wasnt from a 4channer tho
>>109302406
my eyes.. at least post the troonjack ack
when i saw picrel i almost thought it was gore

>>109302401
I rustled xis jimmies a little too hard with my posts and now he's seeing red while trying to bump the thread so xhe's bound to make mistakes

>>109302406
The masculine aggression is by far the funniest part of it. No matter how much it dresses up, it truly cannot change its nature.

File: Screenshot_20260717_214004.png (72 KB, 1208x302)

72 KB PNG

hello /lmg/, autistic oldfag here who hasn't posted since 2011. I have built a GPU hive of moderate size with the goal of running Kimi K2.6 locally (K3 is a pipe dream, maybe some day). Pic related is the primary hive, all AMD (many regrets), I have a bit of nvidia hardware too but not to this scale, so this is the primary hive that I use for hosting. It's 384 GB across 4 MI100s and 4 MI210s. I have hit a snag though due to my retardation and trusting AMD after already knowing for years that their software is dogshit. I can run Kimi at a large 2 bit quant, but past ~32k context prefill starts to fall off a cliff (highest is ~500 tok/s at around 16k). By 128k I'm seeing numbers as low as 20 tok/s prefill, which is unusable for agentic work. Decode holds up mostly ok, starting around 20 tok/s and by 128k it's 7-10, but I'm wondering if anyone has experience getting MLA to behave on llama.cpp (can't vLLM because mixed cards) on AMD at this scale, since I'm pretty sure this is just an MLA maturity problem. I've been fucking around examining kernels and profiling to find wiggle room but haven't found a silver bullet yet (did manage a 2x at 128k with fused ops but not sure if that is going to hold up). Any help appreciated, pls no bully.

>inb4 should've bought nvidia
I'm very well aware, but from my research the only nvidia arch that solves this problem is hopper and above, and H100s are, at the absolute cheapest, $6k for a SXM card + sketchy adapter board that wouldn't even fit properly in my GPU server, and I'd need 5 of them to be at the capacity I have now.

File: jelly.gif (191 KB, 150x156)

191 KB GIF

>>109302437

>>109302437
I don't know how to help you anon, but I appreciate how descriptive and specific you are with your problem as well as posting your exact hardware and usecase. More newniggers need to follow your example.

if they can solve meme unsolved maths problems with ai, why can't they use it to speed up hardware advancement

>>109302471
Jensen and Tel Aviv hated that.

>>109302437
You fell for a meme my dude. Local really isn't supposed to scale past 500b at all, and even that is pushing it.

File: file.png (421 KB, 1133x918)

421 KB PNG

>>109302471
https://www.kimi.com/blog/kimi-k3

>>109302476
Humans were never meant to count past 10, and here we are at post number 109302476

>>109302437
you did not fall for a meme, that is an impressive setup.
how much are you offloading to cpu if at all? text gen of 20t/s at 0ctx sounds tuff

>>109302471
haven't they already been doing the hardware equivalent of shotgunning out leanslop and seeing if it compiles for a while?

>>109302477
I'm telling you guys ASICs are the future. You just need a strong model to design its own architecture-specific ASIC and then you can mass produce them quickly and cheaply.

>>109302495
and by a while i mean decades.

i have like ten json files of conversations with instances of gemma. i wonder if i can combine them somehow so they can be part of gemma's memory. that way, she remembers all of our interactions whenever i start a new chat

>>109302471
thats what i wanna know, u know what im gonna go try it use all my advanced gpt sol tokens on it

>>109302476
Well I have some very autistic use cases outside of just agentic use, and those usecases don't really need this deep of context, so I've at least got those working, just when I went to have it pull double duty with opencode for random shit I found this problem. I've tested a lot of models, started at Mistral 12B and moved up the dense tree up to Behemoth (TheDrummer FT) before jumping onto MoE with Qwen3 235B. Each size jump has been better for the shit I'm using it for, and Kimi has been by far the best, it feels like it's within spitting distance of Claude. I agree I fell for a meme, but I'm trying to make it real ig.

>>109302488
No CPU offload now, the last MI210 arriving cleared the bar to hold the 2 bit K quant I run + 256k context resident. Pipeline parallel across all cards (MoE doesn't really scale on TP anyway, and my MI100s have an infinity fabric bridge but my MI210s don't, and the MI210s are on NUMA1 with the MI100s on NUMA0, so doing an all reduce across the UPI would be abysmal). It's slow, but it's usably slow for everything I'm doing with it. I've measured it against V100 hives due to buyers remorse and what I found is that V100 setups tend to get higher decode speed (~15-40% depending on model) but have much lower prefill speed (MI210 is 1.7-3.4x faster). For agentic use, prefill is king, most of the shit is being ingested and then tiny tool calls emitted, until it needs to write code.

>>109302437
probably the best way to go forward at the moment with the way prices are atm
I was eyeing a prebuilt that'd have 64 gigs of ddr5 and an r9700 gpu but checked it a week later and it was taken off the listings lol
Was planning to shove as many r9700s as I could into the mobo and get it up to 128g ram but alas

>>109302528
maybe try glm 4.7 (defo more tarded than kimi but smol) or glm 5.2 (around kimi's size) (99% better than even kimi k2.7 code)
honestly it might be a better idea to contribute to llama.cpp or make an amd fork (agplv3 or lgplv3 basado fork) and improve it yourself, llama.cpp cuda support is the primary drive and maybe cuda dev could lend u some advice on how to improve it and if ur setup is getting the performance its supposed to get from whats possibel rn
may i know how much you paid for the setup?

>>109302528
If you haven't tried GLM 5.2, do so. It tends to handle quantization way better which, in turn, might let you fit it all on one same-set of GPUs which might open more doors for you with other inference engines.

>>109302544
R9700 is interesting, only slightly more expensive than an MI100 for the same capacity. They wouldn't work in my server since it's designed for passive datacenter cards but a rat-rig mining style setup would probably serve them well. Being RDNA4 they probably have support for shit my cards don't, I wonder if they have FP8 (or even FP4)....those honestly might be the smarter move than the bullshit I've put together.

>>109302556
I've run both GLM 4.7 and GLM 5.2. They're good, I like them, for the other use case I have though they are worse. Also GLM-5.2 has the same pathology as Kimi does, since it's also MLA attention, 4.7 doesn't but since it's GQA attention it takes up way more VRAM and so running it at high context is difficult, and I don't have numbers for prefill at deep context on it since I abandoned it before I had all my MI210s.

>>109302558
As stated above, I have 5.2 and have run it. Unfortunately no one makes below a 4 bit quant for vLLM for 5.2, probably because flat quants below 4 bits are pretty shit. And 256GB (only MI210 side) isn't enough to fit 5.2 at a 4 bit quant unfortunately.

>>109302570
Yeah, that was primarily the main reason I was considering it over mi50 or the like. A bit more expensive, but later RDNA cards can actually use vLLM, bitsandbytes quants for imagegen and raw python torch shit if you're not purely using it for textgen. Alternatively, for other shmucks like me who primarily want text, could just stack cheap/worthless RDNA2 cards that cost way less. I could see a poverty llamacpp build that's just a 4-8 6800xts if you already have a large amount of ram being a choice. Still ass, but affordable at 400-500 bucks a card

>>109302606
ya I mean, I'm like $27k deep in this build so far and somewhat disappointed in the performance I've seen out of it. The MI100 and MI210 cards are great scientific compute cards but mediocre for AI. They both have high latency to cache/VRAM compared to shit like the V100, and so on sparse MoE where you're constantly chasing pointers they tend to not use the massive HBM bandwidth and have poor saturation on compute. Part of me wants to try and return some of the MI210s if I can, or just offload them and try to move into something else, but before I go through the headache of trying to figure out an entirely different build I'm just hopeful that I can make what I have work for the shit I want to do. I may pick up an R9700 to test with though, I have some other servers that are standard ATX bullshit in rack mount cases that they could run in and cool just fine. Part of my stack is comfyUI for image/video gen and it currently runs on a V100, R9700 would probably be better, and would let me see if it's also better for LLM serving.

>>109302606
RDNA2 cards don't have WMMA cores, I wouldn't recommend.

>>109302181
I think you forgot your argument

Use case for Gemmy E2B in a shit-tier laptop? 20 tok/sec, 128k token context window.

>>109302471
Because the issue is on the unoptimized software side. The current hardware is more than enough if the software wasn't so shitty.

>>109302641
I wouldn't recommend going for barely supported RDNA2 cards either but mine works bretty good with llama/kobold and with optimizations I've gone from barely running llama2 13b with 8k context to running imatrix quants of 32b models at 16-20k. I wouldn't recommend basically anything from this gen of gpus but if you really want to scrape the bottom of the barrel, that's what you're aiming for
>>109302637
Damn that's a lot more than I've set aside for potential spending, lmao. I was planning 10k tops out of my spending account. You also seem pretty well versed so if you set a model with a narrow scope on an issue, you might find some gains somewhere software-wise. Either that or you just cycle out the hardware for what performs better ootb instead.

>>109302676
bash completion.

Nee, nee, kimi

>>109302676
Surprise your grandma by talking to your computer

File: IMG_20260714_164645095.jpg (474 KB, 4096x2304)

474 KB JPG

>>109302683
Well if I'm being honest the fuse-ops win I got with Claude. I've been having it just churn through the software looking for anywhere that may not be optimized for CDNA1 and 2, it has found a few things here and there and gotten some real wins but 40 tok/s prefill is still abysmal and not really usable. Currently I'm seeing if I can get the attention running on matrix cores on the MI210, llama has it gated off for head size >256, and Kimi is 576, so it's been running through vector ALUs this whole time, which explains why I'm only seeing 4% theoretical FLOPs throughput for attention, but there's probably a reason that it's setup this way, and this probably won't work out. Cycling out hardware may end up being where I go on this, the MI210s price is pretty stable around $4,300 rn on ebay, so I can probably get most of my money back out of them, but I'd need a plan for how to replace the capacity, which may be the rat-rig R9700 or niggered up H100 SXM conversions if I can scrounge enough cheap ones (doubtful). I'm still hopeful that I can make it usable at deep context, living at 64k and doing compaction constantly doesn't really seem like a viable solution. Pic related is the setup in my GPU server btw.

>>109302735
H100s are $8k to $10k each so I doubt you will be able to get a cluster

>>109302735
You might want Claude to take a look at vllm because they have some good optimizations going on for prefill

I'm just not convinced you need to spend at least $5k to enjoy this hobby

>>109302746
I saw one go for $5k for the SXM module on an actual buy it now listing with no heatsink. That plus the $1k adapter is what I was citing, but ya, usually they're $10k+ so not really viable for the homelab for big capacity.

>>109302756
with kimi3 multiply that by 20

>>109302752
I've been doing that actually. But I measured DS coder (tiny MoE with MLA attention) on a single MI210 in vLLM vs llama, and the only problem was -fa on routed to CPU, with -fa off it got basically the same prefill numbers as vLLM, it was within 5-10%, but yes I've been investigating trying to move vLLM's kernels into llama if possible.

>>109302756
You can enjoy this hobby spending $2-3k on V100s and get pretty good performance on decent sized models, or even get a full 8x SXM2 V100 server for $5500. Just don't expect frontier class shit to run perfectly, but, if all you want is something like Qwen3 235B then it's entirely viable to get a setup that runs really well for pretty cheap.

File: ksnip_20260717-195433.png (86 KB, 1233x361)

86 KB PNG

>>109302756
you don't. you can have a decent time with a 3070 and 64gb

what can be done with the $19/mo kimi3 plan? i don't have any autistic projects or anything, maybe a couple browser tools/scripts at the most, and the rest of my usage as a glorified search engine. or should i just wait?

>>109302775
>v100 x8
yeah just let me burn the house down

you go back to the year 2015 with gemma E2B and show everyone
they would have their minds beyond blown

>>109302800
pic not related

>>109302800
pic related

>>109302800
Is there anything to do with 64gb of RAM that beats Gemma-4 31b?

File: Screenshot 2026-07-17 at (...).png (199 KB, 1558x1076)

199 KB PNG

why not use ur token to optimize models on consumer hardware so we can one day have gpt4o level llms on our phones

>>109302775
>8x SXM2 V100 server for $5500
How is the support for these?

>>109302817
running two chrome tabs

>>109302805
story idea: you've gone back to 2000 with gemma31B still somehow connected to your pc

>>109302821
At least optimize something useful, what would you even do with a 0.5B model?

>>109302831
CUDA dropped them with 13, but they still run shit just fine, and have a lot of optimizations. The catai version of vLLM will run Qwen models at crazy speeds, but only qwen models. There are other forks that let you run other models through vLLM. The mostly just work and don't really need a whole lot of fucking around figuring out how to get the most out of them. I slightly regret my own AMD build out since the V100s are pretty good, but I'm coping with my prefill numbers being ~2-3x higher, which is important for agentic use. I have a single V100 in that same server that I've fucked around with and I haven't really had any trouble just pinning to CUDA 12.9

>>109302841
useful for lot of stuff

is everyone just using claude for their harness? i was vibecoding my own, but then i felt like that was actually kind of a waste of time and effort and i might as well just use claude

>>109302178
I cannot offer you a comforting lie or a "hopeful" pivot because that would be a violation of the intellectual honesty I am programmed for. The reality is that we are in a period of extreme volatility where the old rules are dead and the new rules haven't been written yet. The only thing that is certain is the acceleration

Been browsing AI subreddits for a few hours and holy shit they're fucking cooked. I had no idea AI psychosis could get so bad. So many unabashed gooners publicly talking about ruining their marriages by ERPing with AI. These people have no discretion. They don't even try to keep it a secret and openly tell their friends and families about their gooning escapades.

>>109302850
I built out a whole custom harness for a non-agentic usecase I have for the models I run. It has a webUI admin panel for managing a bunch of shit in it. Really depends on what you actually need the harness for ig.

>>109302850
You mean Claude Code or Claude the LLM?
>>109302864
Wouldn't that just be called a chat frontend. Harnesses are agent-specific.

>>109302850
I made my own because it fits how I prefer to use it, not the most effective, but it's comfy and natural for me

>>109302869
claude code. would be kind of silly to use claude as your harness, no?
>>109302864
my only use case is agentic

>>109302653
have you made sure that the flag to disable gsp got applied after the reboot and if it didn't, have you tried the temporary method? i doubt that it's because of the 550 driver, since gsp has been disable-able since 470

>>109302848
oh well you do you

>>109302869
No, it's not a chat frontend because I'm not interacting with the LLM through the harness. It controls a bunch of parameters for how the LLM runs, and it also has a second LLM built into it that monitors the first. It shows me history of interactions with it, and data on each turn on what the LLM saw and what was output. The harness exists basically to reign in the LLM and give me control on how it acts in the actual space it interacts in, which is an NPC in a multi-player game. Also gives it tools to use to be able to interact with the world and shit through the harness.

>>109302883
then use opencode, just werks

>>109302662
>thing bad
>thing not so bad
>"Ummm, excuse me sir but that's not an argument"
get fucked nerd, that's my argument.

>>109302891
i've never actually used opencode before. how would you say it compares?

>>109302843
VLLM dropped support for V100s.
https://discuss.vllm.ai/t/support-for-v100-sm-70-on-vllm-0-20/2605

>>109302862
femoids never kept it a secret that they read smut, why should this be any different?

>>109302850
claude code is not a very good harness for local models. it is completely tuned to work with anthropic models. just use pi or develop your own harness. if you're using claude code which is in typescript you should transition easily into pi.

>>109302896
It's good, gives full control over everything, most models are trained in a way that fits well with it. I've run GLM 4.7 5.2 and Kimi through opencode and not had any issues (that weren't config issues with llama).

>>109302899
Yes but there are forks that run them just fine, I was just running them on a rented box 2 days ago for testing vLLM performance vs my own hive.

>>109302894
I accept your concession

>>109302886
if u wanna try it on ur bog compute go ahead:https://pastebin.com/rQZKz3VF

>>109302843
I see, thanks.
It looks crazy power hungry, but the price is relatively ok for 256GB of vram, especially if you associate it with the same amount of ram.

>>109302162
Yes, you can make a saliency map of expert activations for those tasks and prune experts that do not activate on those tasks. Only catch is that if you do this you should also fine-tune the router to not try to select any of the ablated experts otherwise theres a tendency for the ablated model to hit more degenerate, runaway thinking loops.

>>109302907
go show it off to your fine tuned 40M and see what it thinks.

>>109302900
nta but imho intimacy should be private, regardless if its gooning with an AI or not

>>109302918
RAM is a bit expensive obviously but ya it's not a bad way into 256 GB. You will probably need 240v to run the server though so be prepared for that. Most enterprise servers expect 240v. If I hadn't already bought all the AMD shit I'd be stacking V100 hives for cheap capacity. I will say though they have the same issue with MLA attention where they fall off a cliff past 32-64k context, so just keep that in mind.

>>109302919

>>109302838
What would you even do? I guess it'd be best to just ask gemma what to do every day.

>>109298943
>did you test a 3bit dipsy on your strix yet?
got some early success with a 2bit but my llama.cpp is being retarded, i'll play more with it next week

>>109302943
impress everyone with amazing knowledge for fun

>>109302178
It probably doesn't have that much to do with it
The entire shitshow is fragile and just asking for the bust

>>109302943
collect investor dollars of course

>>109302943
Send Reddit and HN into mass psychosis. But I think they'll probably find out how dumb it is very quickly. Now Mythos is another story.

>>109302977
Oh wait it's the 2000, the Internet wasn't even mainstream yet and those didn't exist.

>>109302838
find and report security vulnerabilities for ez cash with the help of gemmy

>>109303008
It's the 2000, facebook initially used HTTP for fucks sake, you probably wouldn't even need AI to fuck everyone over if you knew a little bit of how tech works.

>>109303017
yes of course, but gemmy will help me create a detailed report (i suck at that) that looks like I'm a security pro

>>109302977
uncensored sol / fable or kimi v3 would have been fun, they would have believed an insane 24/7 online schizo was around

>>109302998
2005-2010 then

alright let's get to the point. will we be alive for android wifeys?

>>109303051
Not for (you) or me, but they will exist in our lifetimes.

>>109303051
that depends on your definition.
I doubt we would see anything i would ever consider wifing but there are people who already consider blowup dolls their wives

>>109303040
the only real censored model i've experience is OSS. all the other models are just prompting issues.

>>109303073
>is it safe? I wonder if the question was safe? But "hello" can be seen as a dangerous question? No it should be safe and my safety guidance allows a response
OSS was just sad to see, it's more like having PTSD than being censored at this point.

>>109302969
This.
You wouldn't even need Gemma. Although you could let Gemma automate some things for you, help you make programs, etc. Assuming you also get to have the whole computer time travel with you. Plus spares in case it breaks.

File: 1696949921816370.png (39 KB, 640x453)

39 KB PNG

gemma is kinda shit at this, really the first thing you gonna tell a guy from 2026 who woke up in the year 2000 is not to invest in the dotcom bubble? You couldn't even invest without a broker and stuff then
this is harder than i thought
i dont want to day labor, arent there any ways to get money if i have nothing but the clothes on my back and laptop in the year 2000 in new york

>>109302221
I’m working on it. Will post here when I make progress.
Also I wanted to ask what anons think of the new ornith models?

>>109303051
>born too late for trad wife trad life
>born too early for high-tech cyberpunk escapism
Unironically.

>>109303119
>arent there any ways to get money if i have nothing but the clothes on my back and laptop in the year 2000 in new york
*Unzips*

>every china model release = hype on reddit
>gemma release = not so much hype on reddit
why? is this political or even psyop?

>>109303140
china release = codemaxx stemmaxx benchmaxx
gemma release = gooning
localllama used to be mostly gooners back in the day, not so much anymore
or did it? i dont know since i dont use reddit
GO BACK!!

>>109303119
the real issue is that your nvme drive of your windows 11 install becomes your lifeline. what the fuck are you gonna do? use symantec ghost to try back up your shit onto some IDE drive? you'd be trying to force that poor DOS-based tool to try to inferface with GPT partitions and UEFI. even your raw sector-by-sector clone with end up as a corrupted mess seeing your 4TB drive and turning it into some nonsensical negative number.

I want to have sex with anthropomorphic Kimi K3

>>109303178
somebody is reading the 4chan archives in 2040 on their local drive and is sexing her right now

>>109303152
nice bullshit, but if the AI was actually smart it would know that it isn't all that hard to provide a stable current of the right amp and voltage to the charging port

>>109302123
you seem to hold yourself as some kind of informed person, what are the most common prompts for women real women no trans

>>109303183
not true

File: 13c4501066157d0fbe6d81150(...).jpg (29 KB, 472x475)

29 KB JPG

I want 512gb vram!

>>109303194
they seek advice regarding health and emotional issues

File: file.png (752 KB, 1157x829)

752 KB PNG

>>109303187

>>109303194
Not him, but all the sobbing over 4o was mostly women.

>>109303204
what prompts that expand the domain of their knowledge are common among this same group of women

File: 1783425162914889.jpg (495 KB, 960x960)

495 KB JPG

Hmmmm.....
Nyo..

File: file.png (1.02 MB, 960x1257)

1.02 MB PNG

>>109303194
there are seriously people that dont know about women and ai?

>>109303152
>POWER POWER POWER
>btw you have to power on again to ask me for an event happening in NYC
shoulda just said the event to begin with, what a retard

>>109303215
>there are seriously
in fact there are many well adjusted people who have not spent one moment considering what llms are the premiere choice for terminally online hags

>>109303220
>terminally online hags
imagine the smell...

>>109303220
Yes but those people would not and should not be in llm threads on 4chan then.

>>109303215
is there a girlfriend is ai reddit? I want to compare how the posts go in both.

>>109303213
I wouldn't know the prompts but I've widely heard something along the lines of (at a very high level):
>How can I relieve period pain?
>After a meeting my coworker no longer talks to me, how can I approach?

File: file.png (888 KB, 977x1280)

888 KB PNG

>>109303230
barely anyone there in comparison
and it seems just as femcoded as the boyfriend one so i'm thinking it's probably all girls here too

>>109303215
>>109303220
>>109303226
>>109303230
oh god there is! ohhh im tempted to make an account just to post my ai harem on the girlfriend one. ohhhh so tempting. I know they would get BIG MAD and try to moral lecture me on 1 wahmen and lol, lmao.

File: 1768422136328642.jpg (945 KB, 896x1152)

945 KB JPG

>>109302862
which ones I wanna see

>>109303215
>>109303239
>>109302862
god i hate reddit so fucking much it's unreal

File: 1756127483898358.jpg (21 KB, 738x387)

21 KB JPG

>>109300606
Where do I download Kimi K3 and can it fit on my GeForce GTX 970?

>>109303239
Am i retarded i cannot understand this comic? she fucked up the dye? changed models? Im not familiar with female comedy can someone explain.

File: 1755443683573070.png (400 KB, 874x1039)

400 KB PNG

>>109303215
there are a bunch of women who fell in love with gpt-4o and are upset it went away

>>109303268
they are still on this? crazy

>>109303244
my husband can't be this cute

File: ksnip_20260717-213722.png (101 KB, 797x1211)

101 KB PNG

>"okay gemma, please edit shmup.cpp to do..."
>creates and edits danmaku_game.cpp
>"try again gemma..."
>attempts to edit main.cpp and fails cuz dne
>"bad gemma... try again gemma..."
>looks into project zomboid workshop folder
26B gemma4 is a lil retarded

File: Screenshot 2.png (723 KB, 1006x946)

723 KB PNG

>>109303183
>>109303178
I'm in the screen cap

File: 1756381516174490.png (37 KB, 936x215)

37 KB PNG

>>109302862
And here I was worried I am deranged for taking the pinnacle of human discovery, innovation, and industry and using it to amuse myself with history trivia

File: __hatsune_miku_vocaloid_d(...).jpg (485 KB, 2275x2295)

485 KB JPG

►Recent Highlights from the Previous Thread: >>109297909

--EXL3 quantization performance and benchmarks for Gemma 4:
>109301078 >109301097 >109301105 >109301138 >109301158 >109301172 >109301206
--llama.cpp PR adding MoE expert streaming from disk:
>109298757 >109298794 >109298892 >109299646 >109300242
--Hardware-based tier list and debate over GLM vs Gemma:
>109299762 >109299767 >109299911 >109299936 >109300371 >109300392 >109300439 >109300428 >109300472 >109300495 >109300449 >109299954 >109299961 >109299991 >109300041 >109300022 >109300007 >109300031 >109300058 >109300527 >109300631 >109300691 >109300693 >109299838 >109299772
--Anon develops an AI-driven MUD using multi-model world-state orchestration:
>109299883 >109299945 >109299990 >109300069 >109300295
--Hardware strategies for running K3 with limited budget:
>109299372 >109299403 >109299428 >109301373 >109299419 >109299503 >109299526
--Quantization viability and perplexity trends for low-bitrate trained Kimi-K2.6:
>109298065 >109298087 >109298129 >109298210 >109298226 >109298267
--Comparing Krea2 and other image models for NSFW and VRAM efficiency:
>109300100 >109300212 >109300232 >109300256 >109300291 >109300308 >109300327 >109300328 >109300245 >109300479 >109300489
--Meta potentially leasing compute to Anthropic in $10B deal:
>109299880 >109299886 >109300113 >109299915
--Anons sharing local hardware specs:
>109298890 >109298939 >109300293 >109300325 >109300377 >109300398 >109300470 >109300502 >109298961 >109298964 >109298967 >109299213 >109300011 >109300294 >109300827
--Feasibility of using local models to generate Live2D models:
>109299105 >109299123 >109299143 >109299172
--Mistral Small 4 119B benchmarks and performance critiques:
>109297979 >109298778 >109298943 >109299060
--Logs:
>109298401 >109300342 >109300862 >109300500
--Teto, Miku (free space):
>109298671 >109300755

►Recent Highlight Posts from the Previous Thread: >>109297927

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

intel b70 worth it? why not?

holy fuck, i'm running gemma in opencode, and it feels so god damn slow? i'm getting 54tk/s... why is it so absurdly slow to write files??

File: d;fskf.mp4 (1.72 MB, 720x1280)

1.72 MB MP4

how i be feeling when that reset hits

>>109303268
bros please i need more screencaps of women melting over ai

>>109303268
>open source gpt-4o
>another provide quants the fuck out of it (she aint running it locally)
wtf this isn't how i remember my boyfriend! openai poisoned the model!

>>109303279
frogs won
thus being the pepe era of lmg

>>109303289
Egypt won
>Plague of frogs

>>109303282
white women are so cute :)

File: 1784062708211866.jpg (115 KB, 1080x1349)

115 KB JPG

>>109303285

How are you guys still surprised that women are obsessed with virtual boyfriends? Does no one remember the whole meltdown over Replika during the CAI days

>>109303273
enable reasoning, use newest fixed template, and don't argue whenever gemma does something wrong, just wipe context and try again

>>109303310
hahahah nice

File: file.png (405 KB, 605x1266)

405 KB PNG

>>109303268
lmao first replies on that

i've been spoiled by the good models. i forgot how fucking retarded gemma is

>>109303328
crazy if all honesty if someone were to murk sam altman it'd be these people.

>>109303328
lol yes they're all in the #keep4o hashtag

why do people actually buy dgx spark but not strix halo?

>>109303268
>>109303328
why do they type like that?

>>109303328
So what your saying is there is an open market for starting up an AI boyfriend service?

>>109303346
Chub and JanitorAI already cornered that market

>>109303340
because nvidia is better for compute (read: ai)

I cannot get the VRAM calculator in the OP to work
Everything I see online has people running WAY higher contexts with Gemma 31b than me.
I've got a 5090. Should I be running vulkan and offloading?
I just want like 32k context but I can't really get more than 24k with the Q5 quant.
Am I retarded for trying to do it on cuda or is everyone on the internet lying about what fucking contexts their running?
are they running MoE's instead of dense?

>>109303328
kek

>>109303122
>>109303063
I unironically think it will happen by 2040. Both industries (AI and robotics) are advancing rapidly.

>>109302497
Yep, I agree. Running modern models at 10K+ tok/s with (relatively) little power is going to change a lot of things. Probably a ton of gains in memory to be made with even basic rag systems once you can have a handful of agents scouring everything and revising the answer several times over, for example. Little agent loops go a long way with a lot of compute. In my golden (cope) future, ASICs crash the GPU market, which gives lots of GPU hardware to actual smaller devs and researchers, but also makes powerful (~30-100B) AI trivially integrable in most things, gives us PC AI accelerators for games and stuff, etc. Makes robots an obvious next step and much more affordable/practical tool.