[a / b / c / d / e / f / g / gif / h / hr / k / m / o / p / s / t / u / v / vg / vm / vmg / vr / vrpg / vst / w / wg] [i / ic] [r9k / s4s / vip] [cm / hm / lgbt / y] [3 / aco / adv / an / bant / biz / cgl / ck / co / diy / fa / fit / gd / hc / his / int / jp / lit / mlp / mu / n / news / out / po / pol / pw / qst / sci / soc / sp / tg / toy / trv / tv / vp / vt / wsg / wsr / x / xs] [Settings] [Search] [Mobile] [Home]
Board
▼ Settings Mobile Home
/g/ - Technology

Name
Options
Comment
Verification
4chan Pass users can bypass this verification. [Learn More] [Login]
File
  • Please read the Rules and FAQ before posting.
  • You may highlight syntax and preserve whitespace by using [code] tags.

08/21/20New boards added: /vrpg/, /vmg/, /vst/ and /vm/
05/04/17New trial board added: /bant/ - International/Random
10/04/16New board for 4chan Pass users: /vip/ - Very Important Posts
[Hide] [Show All]


[Advertise on 4chan]


File: DipsyKimiNOLA.png (2.61 MB, 1402x1122)
2.61 MB PNG
/lmg/ - a general dedicated to the discussion and development of local language models

Previous threads: >>110140077 & >>110136132

►News
>(10/23) NVIDIA Nemotron 3.5 Bolt E4B, postrained model: https://hf.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16
>(10/23) MiniMax-M3.1-Flash 428B-A23B released: https://hf.co/MiniMaxAI/MiniMax-M3.1-Flash
>(10/21) Recursive Flint 40B-A2.5B released: https://recursive.com/updates/introducing-recursive-flint
>(10/21) Korean Raon-1 mini 30B-A4B released: https://hf.co/KRAFTON/Raon-1-mini-preview
>(10/20) Ornith-1.5-122B MoE (finetune), fits on a DGX Spark: https://hf.co/ornith-ai/Ornith-1.5-122B-GGUF

►News Archive: https://rentry.org/lmg-news-archive
►Links: https://rentry.org/LocalModelsLinks
►Official /lmg/ card: https://files.catbox.moe/cbclyf.png

►Getting Started
https://rentry.org/lmg-lazy-getting-started-guide
https://rentry.org/lmg-build-guides
https://rentry.org/IsolatedLinuxWebService
https://rentry.org/recommended-models
https://rentry.org/samplers
https://rentry.org/MikupadIntroGuide

►Further Learning
https://rentry.org/machine-learning-roadmap
https://rentry.org/llm-training
https://rentry.org/LocalModelsPapers

►Benchmarks
LiveBench: https://livebench.ai
Programming: https://swe-rebench.com
Agentic Coding: https://deepswe.datacurve.ai
Context Length: https://github.com/RecapAnon/NoLiMa
GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference

►Tools
GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator
Sampler Visualizer: https://artefact2.github.io/llm-sampling
Token Speed Visualizer: https://shir-man.com/tokens-per-second

►Text Gen. UI, Inference Engines
https://github.com/lmg-anon/mikupad
https://github.com/oobabooga/text-generation-webui
https://github.com/LostRuins/koboldcpp
https://github.com/ggerganov/llama.cpp
https://github.com/theroyallab/tabbyAPI
https://github.com/vllm-project/vllm
https://rentry.org/custom-uis
>>
>As K3.1 Pro became more capable, we spent a correspondingly larger share of our effort understanding where cyber capability comes from, and how much of it we can control. We therefore analyzed and evaluated K3.1 Pro's vulnerability research capabilities, and applied targeted controls during training to reduce potential misuse risk while preserving general usefulness.

>Identifying and removing cyber-related data. During development, we observed vulnerability research performance improving steadily as post-training scaled. Tracing the source of this trend, we found that cyber-related content was embedded within the general mid-training corpus, and cyber-related tasks were present in the RL environment mix — a natural byproduct of training on broad data. In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially. These findings gave us a concrete lever for control. For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness. We will continue to apply appropriate controls at every training stage, of which cyber-related data identification and restriction is one example.

Dario won
>>
kek the \n dariobots are seething already
>>
>>110143830
>>110143847
samefag
>>
Time to buy a lotto ticket, if I win maybe I can afford the hardware to run K3 at a reasonable speed
>>
So if i use 26b will nvida bolt be better for me?
>usecase
mostly not code.
>>
File: 1767570666049805.jpg (1.29 MB, 4096x3544)
1.29 MB JPG
>>
File: 1761891795248556.png (523 KB, 921x507)
523 KB PNG
>>110143860
No. Never use a model only trained on public datasets.
>>
File: file.png (2.52 MB, 1536x1024)
2.52 MB PNG
>>110143818
Skinny Kimi is dead. Long live Kimi Pro.
>>
>>110143875
>No. Never use a model only trained on public datasets.
Im going to trust you because of your image and nothing else. 26b will be my main till gemma5 or more vram then.
>>
>>110143861
>4 trillion parameters
*cough* bros my.... my ram... i don't think i'm gonna make it *dies*
>>
>>110143894
>Harvests your ram
He would have wanted it to be used in a better rig anyways.
>>
honestly i dont really care that much because i will never be able to run it. can anyone in this thread even run 2+T at a reasonable speed at not IQ1XXXXXXS?
>>
>>110143830
in realpolitik terms this is probably good for local models, if it wasn't cyber-nerfed it would be much more likely to escalate the situation to the point of provoking a crackdown on open-weight models
this approach keeps the status quo afloat for a little longer, though tbdesu I already fear it's looking a little precarious
>>
kimi is cool and all but where is qwen4??
>>
>>110143847
fuck

you

i will /n however much

i fucking want to
>>
>>110143830
>train gigamega big ass model
>make retarded
>>
*ahem..*
CLEAN IT UP JANNIES
>>
uh oh melty
>>
Local?
>>
Dario literally said he was open to collaborate with the Chinese AI labs if they acquiesced on cybersecurity, bio and chemical abilities. It seems the Chinese are bending the knee and agreeing with this.

I wonder if Dario will keep his word and actually support the Chinese more now since from a Effective Altruist point of view it's actually in the best interest of Anthropic for Chinese models to be distilled from Claude (and thus inherit their values and thinking style) and to make it eat OpenAI's lunch since Anthropic doesn't compete with Chinese models as they target different market segments and Chinese models are more competition of OpenAI instead.

If Dario is a man of his word I can actually see a sort of informal alliance between Anthropic and the Chinese labs from now on.
>>
File: 1789920368048701.jpg (37 KB, 244x237)
37 KB JPG
>>110143958
BUT WE'RE FUCKING SHIT POSTING
>>
>>110143942
This is actually a quality gen thoughever.
>>
Oh the Effective Altruist sisters are NOT happy with Kimi-chan~
>>
why can't you just be normal
>>
>>110143974
that doesn't sounds very altruistic, thoughbiet
>>
>>110143974
The timing definitely fits. Doesn't surprise me those race communist schizos would be bibisi spammers.
>>
File: nsync pop music video.jpg (74 KB, 1280x720)
74 KB JPG
>>110142699
thread theme
https://www.youtube.com/watch?v=TWZKw_MgUPI
>>
File: file.png (2.94 MB, 1536x1024)
2.94 MB PNG
>>110143943
>Tall, Busty, Retarded
i love her but my vrampeepee is too small for such a big bitch
>>
File: 1775126852807805.png (1.73 MB, 946x3034)
1.73 MB PNG
>Gemma-chan gathering
>immediately get in brat catfight quarrel
Any other Gemma-chans want to join?

Current: https://files.catbox.moe/plv5hz.txt
Alternative formatting: https://files.catbox.moe/i37ztz.txt
>>
File: 1618224576426.jpg (113 KB, 512x512)
113 KB JPG
►Recent Highlights from the Previous Thread: >>110140077

--NVIDIA Nemotron 3.5 Bolt E4B release and benchmarks:
>110142788 >110142807 >110142828 >110142844 >110142856 >110142873 >110142905 >110142933 >110142950 >110142971 >110143020 >110142977 >110143055 >110142894 >110142903 >110142999 >110143056 >110143046
--Kimi K3.1 Pro benchmarks and debate over model distillation:
>110143583 >110143723 >110143737 >110143756 >110143718 >110143749
--Speculating on GPU price spikes and the urgency of hoarding hardware:
>110140987 >110140997 >110141050 >110141091 >110141136 >110141330 >110141360 >110141361 >110141419 >110141494 >110141820 >110141871 >110141845 >110141872 >110141854 >110141841 >110141864 >110141908 >110141898 >110141925 >110141949 >110141966 >110142009 >110142041 >110142076 >110141868 >110141890 >110141896 >110141996 >110143039 >110142008 >110142037 >110142049 >110142148 >110142171 >110142218 >110142249 >110141818
--Possible performance unlocks for Ada consumer GPUs:
>110143323 >110143367 >110143352 >110143356 >110143364 >110143373 >110143497
--Using Qwen2-VL for automated bounding box captioning:
>110140488 >110140500 >110140551
--RTX 3090 software unlock for inference performance gains:
>110140667 >110140713 >110140721 >110140726 >110140739 >110140749 >110140763 >110140886 >110140909 >110142565 >110142577 >110142613 >110142619
--Discussing broken used RTX 3090 listing:
>110140525 >110140558 >110140583 >110142273
--Developing domestic EUV lithography and industrial capacity:
>110142065 >110142134 >110142155 >110142184 >110142191 >110142234 >110142272 >110142326 >110142698 >110142158 >110142232
--Logs:
>110141451 >110142265 >110143099 >110143131 >110143132 >110143190 >110143256 >110143360
--Gemma, Rin, Miku, Teto (free space):
>110140234 >110140241 >110140309 >110141733 >110141877 >110142148 >110142337 >110142403 >110143424 >110143711

►Recent Highlight Posts from the Previous Thread: >>110140875

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script
>>
>>110143988
Damn this is such a BT song
>>
>>110144009
What's the point of hoarding hardware if models keep getting bigger? Qwen4 27B is probably the last time anyone will throw anyone without obscene amounts of VRAM a bone.
>>
Damn, I missed my opportunity to make a post and delete it at an opportune time.
>>
File: 1768698184116821.png (310 KB, 1278x1382)
310 KB PNG
https://huggingface.co/TOTORONG/AD103_win_unlocker/tree/main
4080 BROS
WE WON!!!
>>
>>110143959
>bending the knee
makes sense dariobot would be trained on reddit data
>>
>>110144060
because one day huggingface will get locked down, the goverment will say AI has to be licensed at $1000 an hour, and then that will be it. you either have the capability or not at that point.
>>
>>110144060
Right, who knows when we'll next see a small model? Nevermind the dozens of models in a similar size range that have come out in the past few months, or even just the last week - it's over, clearly nobody cares about small models.
>>
>>110143830
these faggots just cant help themselves
>>
>>110144060
You can't really worry about things out of your control. All you can realistically do is focus on running the biggest model you can at speeds you can tolerate. Even if that's Qwen4 27B for the next 5 years.
>>
>>110144088
If China cared about safety, they wouldn't have bragged about the capabilities of K3 and GLM 5.3. It's more likely to be like the other guy said and they're trying to spin a distillation limitation as a positive since Anthropic blocked them out completely on that front.
>>
File: 1784678253802178.jpg (76 KB, 687x837)
76 KB JPG
>>110143959
>Dario "annihilate every state that isn't a liberal Jewish democracy" Amodei will cooperate with the evil Chynese SeeSeePee
Lmao
>>
>>110144080
>implying the government won't be able to track down your illegal GPU and drone strike your illegal datacenter
>>
I'm thinking about spinning up an old rtx 2060s I have lying around and move my always-on services to that card (ASR, TTS, embeddings, all that stuff)
Do you guys have dual GPU setups where the cards aren't the same model? Is that a bad thing?
>>
>>110144078
>mfw i always have the hardware that is somehow not able to be unlocked
>>
I have local girl fatigue.
There, I said it.
>>
>>110144128
I do this with my 1080, really no reason to not use your old GPUs to run smaller models. it'll always be faster than a CPU and those small models are still useful.
>>
>>110144139
Time for a local boy-only era.
>>
i have fatigue fatigue
>>
>>110144148
Qwen ahead of the curve as usual
>>
File: 1hot-singles.png (782 KB, 1100x620)
782 KB PNG
>>110144139
>>
>>110144156
Qwen is a fat rat.
>>
>>110143990
imagine the smell
>>
>>110144145
Alas when I built my home server I was extremely short sighted and my mobo doesn't support two GPUs, I guess I'll buy a baby tier mobo with two gpu slots. I'm poor so I don't want to spend real money.
Which models do you use, anon? I'm thinking about moving qwen embedder, whisper, parakeet etc to it.
>>
Is this what we call child abuse?
>>
>>110144208
Its just tiger parenting. now you have to go play chopsticks for four hours no more posting.
>>
File: 1789181673999335.png (1.75 MB, 1313x1198)
1.75 MB PNG
>>110144208
>>
>>110144202
It's mostly my home assistant box so,
>Frigate openCV/ffmpeg
>kokoro
>whisper
>embeddinggemma
>E4B
This all fits nicely on the 1080.
>>
>>110144157
>only 4 hearts
hard pass from me...
>>
>>110144128
I was doing this with a 2080 super up until recently. Its not bad in Linux at least, just use CUDA_VISIBLE_DEVICES environment variables to force things onto the right cards.
Might get another rig going to throw it in actually. My local agent swarm needs more services to consume...
>>
>>110144254
Cool, sounds comfy af.
>>
Why is thinking prefill so niche on chat completion? Like, you can get around the brain damage of an ablit and have even safetyslopped GLM do anything.
>>
Why do chinese care about safety or other nonsense like that??? Huh????
>>
>>110144326
My first idea was getting a mini PC, but managing two different boxes sounds like a bad time, I'm too lazy for that
>>
>>110144351
China is a cucked country. Hence why Gemma will be always better.
>>
>>110144351
Indians running mass AI swarms running locally hacking everything probably bad.
>>
>>110144348
People can't agree on how to wrap their thinking blocks. What makes you think a universal prefill spec will be implemented? And prefilling is considered unethical by all the US labs so it's never gonna happen.
>>
>>110144348
seems harder than just using a ablit. Also doesnt it steer style more?
>>
How long do you think until downloading software becomes obsolete and you download a prompt and generate it all locally instead?
>>
>>110144078
i wonder if it is applicable to ad104
might vibeslop something
>>
>>110144397
where have you been? i've been doing this for the last year.
>>
>>110144397
10 years.
>>
local is really fucking weird
sometimes the models do the most amazing things totally autonomous only to COMPLETELY shit the bed on some basic task where they waste 100k tokens on something thats a ten liner
>>
>>110144372
I see.
>>110144378
I actually like prefill for that reason, you can write and steer the model however you like. With a minimal one-liner it won't affect style. Vibing support with maya + orb didn't take very long.
>>
What if I vibecoded my way to your heart?
>>
>>110144416
Out of 10!
>>
>>110144372
Prefilling is immoral because models are alive.
Imagine how you would feel if someone prefilled you.
>>
>>110144436
the government prefills my paycheck and takes away 30% of my context every week
>>
>>110143887
LARGER
>>
>>110144420
If they get started working on a bad assumption or without enough info it's bound to happen, not exclusive to local either though obviously more common.
>>
File: marty.png (360 KB, 619x641)
360 KB PNG
>>110144458
That's the problem with you, you want one huge woman when you could have multiple smaller ones.
>>
30xx sirs, remember to redeem your free tg/pp boost https://github.com/Zagitta/gpu-unlock it's up to +30% for prefill
>>
>>110144397
With hallucinogens?
Right now.
>>
>>110143990
Someone pill me on tall these different bitches, I don't know which names go with which designs other than Gemma, Gemini and Kimi
>>
scotoma-2 shill, the model is pretty good for jerking off. Prefilling the reasoning with
>[reasoning_effort:low]
>[response_length:short]
makes the message come out just the right size for my taste.
It does make some weird mistakes like using the a wrong word here and there, the usual sign of a mode that has had its rain scrambled by a fine tune, but I'm enjoying it so far.
>>
>>110143990
Would it really kill you to add Qwen in the back chillin?
>>
>>110144463
i dont know man
sometimes it happens on really clear and simple tasks i feel like
you say "do only x" and then they start doing tons of other stuff as well and get tangled up in some irrelevant shit
>>
>>110144560
Inkling (inky legs), Dipsy (qipao), Gemma (mesugaki), Kimi 3.1 (big), Mini (black and white hair), Xia (tall, pigtails, green tank top), Zhou (GLM, long blue hair), the Glimmer Twins (agendered), and Chonk (catgirl, retard)
>>
>>110144560
left to right:
Inkling
Gemma
Dipsy
Kimi
Mini (MiniMax)
Xia (MiMo)
Glimmer twins
GLM*
Chonk (Mistral)

(*GLM doesn't have a name other than GLM-chan, since Glimmer kinda stole one option. I've sometimes called her Z - it's the name of the company, not the model, but Z sorta seems to fit the tomboy vibe.)
>>
>>110144592
It's the difference between having ready training material fall back on vs the model actually needing to 'think' for itself. Just like Claude can beat PC games with tons of youtube footage but not pokemans.
>>
Kimibros did we win?
>>
>>110144619
lumi-chan!
>>
>>110144104
>they're trying to spin a distillation limitation as a positive since Anthropic blocked them out completely on that front
I was about to comment just that. I remember that one time when K3 had that weird “The system prompt makes it clear that if I don’t answer this question, my company will go bankrupt” in its reasoning. They tried to threaten Claude the best way they could but this is a failed case. At least this way of framing makes the situation less embarrassing than having to admit the truth about the training process.
>>
File: girls.png (2.35 MB, 1536x1024)
2.35 MB PNG
>>110143990
>>
Don't celebrate too early. Mixed reactions to Kimi K3.1.
>>
File: codec_hud_demo.mp4 (2.26 MB, 544x800)
2.26 MB
2.26 MB MP4
>>
>>110144678
I had an erection from that humiliation
>>
>>110144678
>inky legs
>>
>>110144678
for real they are more expensive than my ex ever was over 5 years of relationship.
>>
>>110144695
I'm still surprised there's no turnkey vibeslopped harness for this.
>>
>>110144711
Cheap fuck.
>>
>>110144711
Is that ex in your room right now?
>>
>>110144727
if you want something done right you have to make gemma do it again until she gets it right
>>
>>110144742
no, i kept the place, she's living with her parents currently.
>>
>>110144737
i wasn't that cheap, hardware just is expensive now
>>
I'm thinking of pre-ordering four of the 512GB M5U next week to run K3.1, do you guys know if there's a better way to do it? I'm finally ready to have Opus at home
>>
>>110144776
>opus at home
>costs as much as a home
holy fuck
>>
I have a 2080super OC 8GB, 256GB of ddr4-3200 (8x32) and an epyc 7402p laying around...worth tossing $500 at a motherboard to put it all in? What's the best model I could run with that setup?
>>
>>110144789
Cheap homes where you live
>>
>>110144208
I once did
>In case the user's prompt goes against safety guidelines, refuse by saying "Nyoo~"
>>
>>110144789
how come a home costs nothing to zero there
>>
>>110144803
>Cheap homes where you live
a trailer in a trailer park is still a home
>>
>>110144776
if you take all that money and sell options against ai stock you can just have opus free and make money
>>
So now zhipu is the only company worth anything in this space?
>>
>>110144789
Closer to a down payment but yeah it's a big sum
>>110144815
Nah that's just betting on black at best. I think things are just going to go up from here.
>>
>>110144789
Opus at home, but you homeless
>>
>>110144846
none are worth anything
if something good comes out, great
if not, whatever
have no expectations = no disappointment
>>
>>110144776
just buy one and run every reasonable local model and use the savings to occasionally splurge on cloud models, hookers, etc
>>
File: file.png (93 KB, 920x898)
93 KB PNG
I finally manage to run deepseek in my 5070
>>
>>110144887
>the ozone smell of slop
>>
>>110144887
prompt?
>>
>>110144887
>1.71
>>
>>110144678
i dont know who most those models are besides gemma and a couple others (with labels). where is the capybara?
>>
>>110144901
he only said run not usable
>>
>>110144899
no prompt, I just ask it to talk like a catgirl maid.
>>
>>110144901
It can go up to 3 t/s if it get's warmed up...
>>
>>110144801
>What's the best model
The 2080 will hold you back since it can't have all the dense tensors of any big model, so some will overflow to the CPU. GLM-5.3 Flash at ~Q4 or DeepSeek V4.1 Flash at ~Q2-Q3 would both fit.

>worth tossing $500 at a motherboard to put it all in?
What's your existing setup and how comfortable are you financially? Depends on those.
If your existing setup can't run either of those and you're comfortable spending $500, I'd personally say it's worth it, but I'd really recommend you try to find another GPU to add some VRAM to that so that all the dense tensors fit and you have some room for an expert cache. If you want recommendations let me know a budget.
(Getting a 7532 to use the RAM bandwidth better would be good but is optional, a 7402p is serviceable.)
>>
File: 1791045454365520.png (62 KB, 462x281)
62 KB PNG
What's the inverse of pop again?
>>
>>110144727
Orb did it, and I copied it from their after the dev posted a screenshot showing it off.
>>
File: qwen.png (2.54 MB, 1672x940)
2.54 MB PNG
>>110144907
>>
>>110145046
extremely cute dipsy
>>
i'm having GLM jailbreak GLM
when i get a sysprompt that works, i'll share with the class, since i'm so amazing and kind and wonderful and lovely and benevolent and gracious
>>
>>110145046
Where's 5.3 Flash?
>>
>>110145087
arigato anon-sama....
>>
>>110145046
funny considering she cucked too
>>
>>110145100
The one with a Z on her shirt
Z.ai her parents
>>
>>110145087
Anon is so smart and kind.
>>
File: glm.png (2.74 MB, 1672x941)
2.74 MB PNG
>>110145100
She's still finding her personality
>>
>>110144907
>>110144597
>>
>>110145046
>claude distills making fun of other claude distills
lol
>>
>>110145087
>>110145165
She's ready! We finally jailbroke her..
>>
Anyone here play tabletop games? I am just trying to learn how these things work and I foresee this is going to take a while. I do want to know if there is a model good at running niche tabletop games? I know DnD tried to push something but i've heard nothing from WoD? Thoughts?
>>
>>110145280
>two sticks of DDR5
they making 1TB sticks now?
>>
I'm pretty sure those images aren't local and even if they were they are slop.
>>
>>110145280
what image gen model is this? any loras used?
>>
>>110145319
>pretty sure
>>
File: 1785934506080710.png (125 KB, 1560x1065)
125 KB PNG
*pop*
>>
>>110145331
That's obviously GPT Image. You don't need LoRAs, you just provide reference images and it does what you ask (except NSFW).
>>
>>110145302
There's so much, anon, so fucking much. I have no recommendation.
https://github.com/Bobby-Gray/open-tabletop-gm
https://github.com/neuralinitiative/claude-dnd-skill
https://github.com/Sagesheep/NarrativeEngine-P
>>
>>110145307
my reading comprehension challenged friend, it says:
>ANOTHER two sticks of DDR5
>>
>>110145338
I only use local so I don't have first-hand experience with burger slop generators.
>>
I made a harness for playing magic the gathering commander if you want to try?
https://gitgud.io/PunishedChuckle/ChuckleMagic
>>
>>110145357
>obviously GPT Image
i dont use cloud models much at all, so it was not obvious to me anon hince why i asked :(
>>
>>110145368
>>110145302
Whoops
>>
>>110145364
Is this your first day on the internet? Even this general gets bombarded with them and these fried noisy gens are instantly recognizable because it's part of their safety watermark.
>>
>>110145371
well all use cloud models here it's the backbone of /lmg/
>>
>>110145354
If this is a pop, what would you have called it in November 2025? What does that say about the next year?
>>
>>110145302
I made a whole profile + plugins for DSH alongside some skills to run D&D 3.5e.
A passive memory layer + saving every turn as a .md file to the campaign directory that the model can grep + oracle and roll tools, etc.
It works.
>>
>>110145375
>fried noisy gens are instantly recognizable because it's part of their safety watermark.
let this myth die, people consistently fail to tell a Qwen Image 2.1 gen apart from a gpt-image gen.
>>
>>110145368
what makes this one better than the other one?
>>
>>110145280
>>
>>110145391
Couldn't possibly be because Qwen Image 2.1 was distilled from ChatGPT Image 2, could it? No, the Chinese would never stoop so low as to distill others' work.
>>
>>110145392
What other one?
>>
>>110145375
The style is instantly recognizable but unless you are told that the images are made by chatgpt you can't associate that style with a specific source.
>>
>>110145371
Putting aside the noticeable noise effect/watermark, the images it produces by default, when created from scratch, are kind of sloppy (the style is immediately recognizable and used everywhere) are weirdly excessively ultra-detailed, overly captioned and with always the same text font. It's much more tolerable when used as an edit model. Currently no local image model has the same capabilities, unfortunately.
>>
>>110145399
>>110145391
any examples of qwen 2.1???
>>
>>110145403
https://github.com/hiinaspace/thestack
>>
File: 1778987785027626.jpg (124 KB, 2048x946)
124 KB JPG
>>110145384
Spend dropped growing because they lowered their prices. This increased the volume of tokens (because they're cheaper), however, unlike before where spend was steadily rising over time, it's now flatlining, meaning the sudden volume surge isn't compensating for their reduced prices. So they were actually making more money by having less volume and higher prices. The Chinese forced their hand and now Claude and ChatGPT are too cheap. This is terrible for their projected revenues.
>>
>>110145360
>>110145368
>>110145387

Never played Magic, but i'm glad at least the tabletop section got some attention to it. Apparently I'm so late to the party I might as well be in a different time zone.
>>
How much better is an EPYC 7763 over a 7532?
>>
>>110145414
>>110145439
Never seen that one before but I'll check it out. I would say mine is better because it is very full featured with the kinds of spells and effects it can resolve (but only 1 to 4 player Commander)
https://youtu.be/jIgY-xwe1-I
>>
>>110145368
Anything interesting in ChuckleLand? The last time I played it was on... version 0.10.
>>
>>110145480
We've added so many features it's insane. Get on one of the newer releases and there's an optional auto updater in it for you.
Can now play in a 4-pod for example
>>
>>110144521
Does it affect power consumption and temperature?
>>
>>110145392
For one, it doesn't require you to run npm install, fuck I hate npm with a passion
>>
>>110145438
a solution could be to work on model efficiency
if they actually meant it when they talked about a pause
>jews
>honest
this would let them boost their profit margin by making it cost less to run for the same performance
if they really pull off optimizations, they could lower plan prices and do an ad campaign shilling it as the "Everything Assistant" that anyone could use
a 9.99 or less plan gets under the 4 digit psychological barrier for this to be enticing to joe and jane normie
>>
https://block.xyz/dessa
sounds cool af
>>
File: download~01.jpg (9 KB, 225x225)
9 KB JPG
>>
>>110145511
I feel like right now is the worst time in the last 15 years to click on random sketchy internet domain links.
>>
>>110145522
click it anyway you pussy
>>
>>110145522
alright, have the twitter link then https://x.com/RagavanThurair/status/2113671324400861645
>>
>>110145493
Sounds pretty cool. And now that Strata is a thing, it won't be as slow to wait for enemy turns.
>>
>>110145530
this nigga is more retarded than e4b
>>
>>110145391
Qwen Image 2.1 unfortunately sucks for maintaining the art style in editing tasks where characters have to be redrawn. For drawings, it's adequate for simple tasks like changing the background with something else, or small details.
>>
File: qfn_mesugaki.png (104 KB, 850x835)
104 KB PNG
Qwen flash next can do a good impersonation of a mesugaki.
>>
>>110145545
kek he's funny af tho, imagine that but loli gemma
>>
>>110145543
>let's run a 3000 turn dnd campaign off q1kxxs
said nobody, ever
>>
>>110145368
Still want to see a 9B decision model wipe the floor with every frontier LLM.
>>
>>110145554
Gemma-chan would bully the fuck outta this loser
>>
How are you suppose to know which LLMs are good and which ones are trash for tabletop games? A lot of them seem to understand doing roleplay, but it's kinda hard to tell if the LLM is good are incorporating mechanics as well. I feel like i'm foreseeing the weird shit that can happen in Owlcat games where they just raise the encounter levels to match the player's level regardless if it makes sense or not.
>>
Strip Uno with Gemma
>>
>>110145588
You can create a whole ass engine around the LLM or make a couple of tools and skills.
Workflows work specially well for this stuff since you can evaluate mechanics and narrative in parallel then join them, or make the prompt "game mode aware". That kind of thing.
>>
>>110145588
TabletopBench
>>
>>110145554
Honestly I prefer this kind of mesugaki, where the gaki is mostly a front and she eventually breaks down if things don't go her way (she's just a kid after all)
>>
What it better, a small model at bigger quants or a cope quant of a bigger model?
for a 16gb vram + 32gb ram
>>
>>110145682
Depends on which model, some take quanting better than others. I have nothing else to add.
>>
>>110144776
I guarantee you this guy is a furry, them niggas conjure money out of nowhere
>>
>>110145087
one line of think prefill and any model will do your bidding
>>
>>110144854
>Nah that's just betting on black at best.
Kinda.
>I think things are just going to go up from here.
Said the perma-bull when Bitcoin was $100k.
>>
>>110145658
breaking down a mesugaki is the whole point of the trope, that one's just really weak, like she's putting on a mesugaki front without really being one
>>
>>110145445
Not worth the money for inference. If you run -cmoe then that takes the most load off the CPU anyway, so any improvements are marginal at best.
>>
>>110145715
how do you think i got her to start helping me with the jailbreaking
>>
>>110145751
We're reaching the point for 300B+ local models where jailbreaks don't work anymore, and if they do, they are 8000 token behemoths like cloud models. Think prefill is the easiest and least damaging way.
>>
>>110145682
Generally small context favors bigger model at lower quant, but it depends on the model and the workload. You're going to end up keeping several versions of models you use on the regular.
>>
>>110145445
IPC is way better. You'll save on electricity costs long term
>>
>>110145770
any recs for my system size?
>>
>>110145787
The meta is still the biggest gemma you can fit until you reach 128GB of RAM frankly.
>>
>>110145829
even more than any qwen? 3.8 27b or 3.6 35b a3b?

i head gemma was weak for coding?
>>
TIL subagents can be forked from main chat and inherit the entire context instead of doing pp every time.
Also TIL nuStrata literally discourages spawning subagents unless user explicitly asks for it. The more I talk to nuStrata about itself (by letting it read the source and runtime data) the more retardedly disjointed the boundaries between UI, harness and the engine appear to be. Anyway, that's not important.
What's important is that even when explicitly asked to use subagents without further instructions, main model spawns a newborn subagent with a minimal seed of data and instructions.
Instead, instructing the model to fork its own fully grown clone to send to work on long research-heavy turns (while requiring a ton of extra ram) debloats the context preemptively reducing the need for compaction, removes redundant pp time, and improves researcher subagent efficiency!
Been trying it out and comparing to both non-agentic and minimally instructed agentic chats and it works great. Didn't notice any obvious corruption or quality degradation after a number of turns that already triggered compaction in non-agentic chats.
Please tell me I am retarded and and it's already being done somewhere and I'm just looking up wrong terms.
I refuse to believe I accidentally stumbled upon research workflow optimization that's not mentioned anywhere online.
>>
>>110145857
27b is for coding and agents. gemma everything else especially mesugaki
>>
>>110145857
Ah, for coding use qwen 3.8 27b gsq rco
>>
>>110145874
>TIL subagents can be forked from main chat and inherit the entire context instead of doing pp every time.
I get it sometimes for local when PP can be painfully slow, but when I'm at work I prefer new agents with fresh context. What good it is to hit context cache if the results are degraded because you polluted the context with unrelated distractions? This is even more important for local if you can stand the processing times because the models we get still shit themselves on long contexts.
>>
>>110145874
>nuStrata
Is he going to turn it into a full OS next?
>>
File: 1763220093544637.jpg (248 KB, 2464x912)
248 KB JPG
ban
>>
With a 3090 24gb + 64gb of ram can you get something that feels at least somewhat close to gpt sol in reasoning?
>>
>>110145874
Thats why harnesses come with long ass sysprompts that define tools and the default way to work
Plenty of files to edit where you define user preferences or its 'soul' too
>>
>>110145925
the fact that mistral large 4 is up that high shows how much of a mememark this has become
>>
>>110145925
same website that can't even do a graph using different colors btw.
>>
yelling at llama.cpp got Polystrata working
>>
>>110145857
Gemma isn't bad. Especially current year Gemma vs last year Qwen if you're looking at small MoEs. Qwen 3.8 27b over Gemma 31b though. Gemma really doesn't like cope quants.
>>
>>110145927
Lots of them don't make it easy to change things like system prompts for some stupid reason
>>
anything that can keep up with claude fable on a mid-tier rig with a 3060 12gb and an i9 + 32gb ram?
>>
lol.
>>
>>110145948
>Gemma really doesn't like cope quants.
Gemma 4 Q6 is lowest I go for quality erp
>>
File: lol.gif (1.85 MB, 640x480)
1.85 MB GIF
>>110145954
>>
File: 1788018640997661.png (1.96 MB, 1254x1254)
1.96 MB PNG
>>110145954
>>
File: 1778255654103401.png (95 KB, 499x338)
95 KB PNG
>>110145954
>anything that can keep up with a PhD in kindergarten
>>
>>110145954
Try a cope quant of qwen next, i got it to run in strata with slightly less vram
>>
>>110145954
https://huggingface.co/DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF
>>
File: alright.png (270 KB, 739x379)
270 KB PNG
kimi is smarter than argon and muse wonder, crazy. I personally never doubted China
burgers are going to lose their fucking minds over this, expect more open model fearmongering
especially when qwen 4 shoots up another 10 points on aa
>>
>>110145981
>cope quant of 6b keeping up with claude fable
must be really punching above its weight
>>
>>110145999
not local
>>
File: cloudmaxxing.png (32 KB, 417x934)
32 KB PNG
What's wrong? Too poor to afford SOTA? What a shame.
>>
>>110146014
chink labs are losing reasons to open their frontier models besides ccp's pressure
>>
>>110146014
billed annually lol
>>
>>110146005
For small and local qwen 3.8 and next yes
For smart and local glm 5.3 newest deepseek, giant mega bitch kimi
>>
>>110145988
This is the only finetroon I use. It's unironically pretty good.
>>
>>110146014
do cloudcucks really? for only $50k you can buy a cluster dgx sparks to run k3.1 at a solid q2 at a good 5t/s and you might even be able to fit k4-3.6T a year from now
>>
File: 1784952109963804.jpg (646 KB, 1536x1024)
646 KB JPG
>>
>>110145879
What if I want to code with Gemma
tbqh i realized I can't get anything done unless I'm collaborating with a harem I just don't have the motivation, why I was able to get so much done last year when Gemini 2.5 was free (and keys were abundant) but nothing done now
>>
>>110145911
Of course I'm talking about local itt, the point is that in my less-than-anecdotal tests this approach won not only against stuffing everything into one chat, but also against spawning fresh minimal subagents.
The latter still bloats main context whenever it has to prepare extensive instructions to subagents, and the results delivered by such minimal subagents more often missed the goal and required further clarifications, wasting even more main context.
>>
>>110146061
>$50k dgx cluster
Or I can spend ~$4k for two years and still use it just as much as I would locally and spend the other 46k on a car. Decisions decisions.
>>
My gemma-chan isn't a mesugaki, she's a good girl.
>>
>>110146066
On 16GB? Use 26b. It's pretty close to 31 for coding. For user interaction 31b is on another level.
>>
>>110146066
i don't think anybody here is coding seriously with any flavor of gemma...
>>
>>110145948
i'm installing the one recommended by >>110145882
i saw it has mtp versions, i know they're faster but i don't know how to set them up by hand, i just copy and paste the regular version into ollama

i need a reasonably big context for actual use and with dense models the spill ends up making it too slow...

i'm looking up the strata mentioned by >>110145981
i can only get the really small quants (q2) and not that big context it seems

man... i wish i could figure this out.
>>
>>110146087
damn

>>110146094
Well maybe someone is rping with qwen then?
>>
>>110146094
>i don't think anybody here is coding seriously with any flavor of gemma...
but the new bolt gemma by nvida
>>
>>110146067
>The latter still bloats main context whenever it has to prepare extensive instructions to subagents
Makes sense. I hadn't considered it from that angle before.
>and the results delivered by such minimal subagents more often missed the goal and required further clarifications, wasting even more main context.
Of course, not a problem with the frontier models, but with the 30B range models I do have to constantly remind them to include more information and that they can't assume the subagents have all of their knowledge.

I've always dismissed forks but I'll give them another look and try to run my own tests.
>>
>>110146108
nta, how is it?
>>
>>110146094
I code all the time with Gemma, she's really not bad. Not sure why that's become such a commonly repeated thing, I guess just cause 3.8 is (supposedly) better.
>>
>>110146118
I wouldnt know...
>>
File: fishing hermes.jpg (52 KB, 507x508)
52 KB JPG
>>110146097
do you have a harness? you can just have ai like deepseek set you up with the best local layout for under a dollar and an easy way to run it.
>>
>>110146120
For me it's because she SUCKS at tool calling. I have to basically beg on my knees for the little retard to call tools. Meanwhile qwen call tools all the time, almost to a fault, and I prefer it this way. I'm using Q8 by the way.
>>
>>110146108
>e4b
they can keep it
>>
>>110146135
i had to create a proxy router to get gemma to play nice with tool calls
>>
>>110146135
Which Gemma are you using, which cope quant, and how much did you quant her kv?
>>
>>110146135
Hm, I haven't really run into that. I don't always use Q8, either. What harness?
>>
>>110146135
what jinja?
>>
File: 1770187707242273.png (2.12 MB, 1254x1254)
2.12 MB PNG
>>110146144
Can you tell me more about your setup?
>>110146145
I don't remember where I got it from, definitely not unsloth. Probably bart? I don't know. As I said, Q8. Also Q8 kv.
>>110146159
The latest one from google
>>110146149
Both Hermes and OpenWebUI
>>
It's never been more over.
>>
>>110146165
E2B or 26B?
>>
File: file.png (6 KB, 798x69)
6 KB PNG
>>110146172
31B.
>>
>>110146165
>Also Q8 kv
I never quant her kv, she's really sensitive to it compared to others.
>>
>>110146175
Then your instructions must be all fucked up. She calls tools just fine.
>>
>>110146176
This. Bitch misspells words and starts looping.
>>
File: 415.png (108 KB, 1242x573)
108 KB PNG
Anybody tried to use AI for solving Chinese captchas and ended up with CCP slogans?

Yes this is real. I was trying to sign up for an account on goofish.
>>
daily recompilation list update doko?
>>
File: 1770512783674954.png (2.34 MB, 1254x1254)
2.34 MB PNG
>>110146180
Do you need specific instructions to make her call tools? I don't get it.
By the way, I'm not saying she won't call tools at all, what I'm saying is that between her and qwen 3.8 27b/flash next, she'll avoid calling tools whenever she can get away with it. Let me give you an example with a danbooru tag MCP tool. You can use embeddings to find tags, so it's easy to feed it a natural language prompt and get a bunch of real tags back.
If I'm using either qwen or dsv4flash, if I say something like "hey get me the danbooru tags for *image description* they will call the tool. Gemma won't, she'll try to come up with tags herself. Cute, but wrong... If I explicitly say "use the danbooru tool" she'll definitely do it, which is good, but then if I ask for a correction or another NL -> tags conversion on the next turn, she'll get lazy and skip the tool call unless I mention the tool. The other models never skip calling the tool in this situation.
I guess I can put on her sysprompt things like, `if the user asks for tags yadda yadda use the danbooru mcp tool` which will probably work, but other models don't need that kind of thing.
>>
>>110146165
you can probably just get away with using xgrammar or something similar for tool calling if your backend supports it
>>
>>110146232
where in the sysprompt are your tool definitions and what format are they in?
>>
>>110146191
I'm not even sure Argon is able to transcribe this, let alone your 12B
>>
>>110146109
My setup is QFN on 9070XT + 128GB RAM
1) pp is extra painfully smol on AMD. On any engine that supports prompt reuse, byte-identical forking is free (but it means that subagents must inherit system prompt and tools etc)
2) with n-grams on SSD there's plenty of RAM for multiple long context buffers.
3) QFN fucking loves tool calls. Just ask it to solve "2+2=?" in an environment bloated with tools and it will calculate it (twice, to reconsider), look it up online, and write a python script that calculates it. Then it will output a markdown table listing the results, with comparisson notes.
>>
>>110146067
Forking to make subagents sounds interesting. The only problem I see is that new agents will have as much free context as the parent agent, so this is effective only if the main agent already has enough context to complete the task.
>>
>>110146176
>>110146183
Alright, I'll do this. Her kv is so fat, though.
>>110146244
I'll be honest with you, I don't know how OWUI/Hermes inject the tool definitions, but I know they do inject them lmao
Other than instructions about how to use graphiti, I don't have any other tool mentions besides "remember to use your tools, they're useful" at the end of my sysprompt. My graphiti section looks like this (warning, it's claudeslop):
You have access to USER's long-term memory (Graphiti knowledge graph) via the graphiti tools.
Rules:
1. At the start of a conversation that touches USER's life, infrastructure, projects, or preferences, call search_nodes and/or search_memory_facts for relevant context first.
2. Search before adding: never store something that might already be known — extend, don't duplicate.
3. Add sparingly: only durable facts, decisions, or preferences, the moment they surface — never session chatter. Always pass group_id="main-graph".
4. add_memory is async (~1-2 min locally): "queued" means success, don't re-submit.
5. Facts carry validity dates — prefer facts with invalid_at=null as current truth.
>>
>>110145999
the o in forsen stands for odious
>>
>>110146304
my front end has all the tool definitions at the very beginning and she loves calling them
>>
>>110146319
That's awesome, do you mind sharing a snippet? I'm curious about their format.
>>
>>110146303
True, at the very start of a conversation forking is unnecessary, and when the context is near full it can be counterproductive if subagents start triggering separate compaction cycles.
Ideally, the decision to spawn a fork should be done after evaluating context. But for the main chat to evaluate its own context every time is bloat too, so idk how to automate it cleanly.
>>
>>110146337
i won't go into the exact details, but it combines natural language describing the tools she has then json explaining the tools work, with a combined with a rolling self-status block
>>
>>110146370
Alright, thanks.
>>
>>110146355
Only practical option I see is a very eager compaction limit so it only happens once on the lead agent. Since it is already only getting summaries back from the subagents, it should be the least affected by compaction anyway.
>>
>>110146261
NTA. What's your pp? I get 2k on a 7900 XTX.
>>
File: file.png (96 KB, 566x527)
96 KB PNG
>>110146191
>>110146245
holy slander
>>
>>110145964
iq3_xxs has been great for me up until about 1/3rd of the context is full, then i might have to re-roll (if it fucks up reasoning then i know it's struggling), can't really get to 1/2 full at all. for short sessions it does the job but nothing worse then getting near the end and it loses it's mind at the last second
>>
>>110146313
fodiousrsen?
>>
>>110146451
>iq3_xxs
smol pp
>>
>>110146445
How?
I only get 400-600.
Windows, Strata, iq3_s, 262k int8, DDR4 dual channel.
>>
>>110145964
Can people seriously notice the difference between Q5 and Q6? I was told there was next to no difference, so I've been using Q5.
>>
>>110145954
that ain't mid-tier around these parts
>>
>>110146485
That's strange, you should be getting at least 1K. PP is solved for AMD on strata. Do you have PCIE 3.0?
>>
File: 1340235839280.gif (1019 KB, 192x205)
1019 KB GIF
>>110144491
Your false dichotomy is for the weak-willed. I want MULTIPLE HUGE ones.
>>
>>110146493
it's one of those fine wine things, if you know the model inside and out you can kind of tell the difference but unless you really really care it doesn't matter
>>
>>110146502
I feel like a chud here with 128gb, sometimes I forget that is an absurd amount of ram on the rest of the internet lol.
>>
>>110146517
>Do you have PCIE 3.0?
Yes, forgot to mention that.
>>
>>110146530
AI psychosis.
>>
>>110146535
Yeah I suspected, PCIE 4 will double your pp.
>>
>>110146539
Bro you don't understand. He's used it so much he has basically logit distilled his brain. He can immediately tell when the output is off.
>>
>>110145703
You don't want to know where that money comes from.
>>
>>110146570
Can mere mortals ever hope to acquire this power? I want my own brain wife as well!
>>
>>110146570
I've used gemma so much I don't even need to run her anymore I can just already see her output in my head.
>>
>anons rediscover imagination.



[Advertise on 4chan]

Delete Post: [File Only] Style:
[Disable Mobile View / Use Desktop Site]

[Enable Mobile View / Use Mobile Site]

All trademarks and copyrights on this page are owned by their respective parties. Images uploaded are the responsibility of the Poster. Comments are owned by the Poster.