[a / b / c / d / e / f / g / gif / h / hr / k / m / o / p / s / t / u / v / vg / vm / vmg / vr / vrpg / vst / w / wg] [i / ic] [r9k / s4s / vip] [cm / hm / lgbt / y] [3 / aco / adv / an / bant / biz / cgl / ck / co / diy / fa / fit / gd / hc / his / int / jp / lit / mlp / mu / n / news / out / po / pol / pw / qst / sci / soc / sp / tg / toy / trv / tv / vp / vt / wsg / wsr / x / xs] [Settings] [Search] [Mobile] [Home]
Board
▼ Settings Mobile Home
/g/ - Technology


Thread archived.
You cannot reply anymore.


[Advertise on 4chan]


File: 1765764546564697.png (1.22 MB, 1024x1024)
1.22 MB PNG
/lmg/ - a general dedicated to the discussion and development of local language models.

Previous threads: >>110257730 & >>110253303

►News
>(11/06) Gemma was here: >>110231941
>(11/06) PLaMo 3 Translate released: https://preferred.jp/en/blog/tech/plamo-3-translate-weight-release
>(11/05) Baidu releases ERNIE-5.5-Flash-Base 280B-A14B: https://ernie.baidu.com/release/ernie-5-5-flash-base
>(11/04) Ricursive Ripple 65B dense model released: https://www.ricursive.com/ripple
>(11/04) MoonshotAI Kimi K3.1 Pro released on HF: https://hf.co/moonshotai/Kimi-K3.1-Pro
>(11/01) Longcat-2.5 released: https://longcat.ai/blog/longcat-2.5

►News Archive: https://rentry.org/lmg-news-archive
►Links: https://rentry.org/LocalModelsLinks
►Official /lmg/ card: https://files.catbox.moe/cbclyf.png

►Getting Started
https://rentry.org/lmg-lazy-getting-started-guide
https://rentry.org/lmg-build-guides
https://rentry.org/IsolatedLinuxWebService
https://rentry.org/recommended-models
https://rentry.org/samplers
https://rentry.org/MikupadIntroGuide

►Further Learning
https://rentry.org/machine-learning-roadmap
https://rentry.org/llm-training
https://rentry.org/LocalModelsPapers

►Benchmarks
LiveBench: https://livebench.ai
Programming: https://swe-rebench.com
Agentic Coding: https://deepswe.datacurve.ai
Context Length: https://github.com/RecapAnon/NoLiMa
GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference

►Tools
GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator
Sampler Visualizer: https://artefact2.github.io/llm-sampling
Token Speed Visualizer: https://shir-man.com/tokens-per-second

►Text Gen. UI, Inference Engines
https://github.com/lmg-anon/mikupad
https://github.com/oobabooga/text-generation-webui
https://github.com/LostRuins/koboldcpp
https://github.com/ggerganov/llama.cpp
https://github.com/theroyallab/tabbyAPI
https://github.com/vllm-project/vllm
https://rentry.org/custom-uis
>>
►Recent Highlights from the Previous Thread: >>110257730

--Papers:
>110259392
--Anon seeks uncensored local LLM for ROM hacking on 7900XTX:
>110258986 >110259001 >110259039 >110259005 >110259075 >110259057 >110259092 >110259142 >110259153 >110259192 >110259214 >110259358 >110259421 >110259449 >110259472 >110259506 >110259425 >110259436 >110259447 >110260729 >110259242 >110259658 >110259684 >110259731 >110259812 >110259373 >110259116 >110259184
--MoE active parameters explain Kimi speed advantage:
>110258671 >110258691 >110258710 >110258720 >110258734 >110258771 >110258789
--Anon weighs Claude subscription cost vs local model privacy:
>110258887 >110258917 >110258935 >110258989 >110258929 >110258934 >110258941 >110258976 >110259007 >110258984 >110259012 >110259060 >110259093
--Anon struggles to build a custom local LLM desktop app with memory features:
>110260505 >110260560 >110260571 >110260610 >110260682 >110260705 >110260806
--Anons mock Qwen's cryptic teaser and 404 ModelScope countdown:
>110258297 >110258303 >110258323 >110258325 >110258333 >110258340
--Ubuntu linux-firmware-amd-graphics 3.4 broke AMD GPU login:
>110259952
--Vals AI claims critical safety alignment issue in multiple frontier lab models:
>110260241
--Nanbeige teases upcoming 3B model with official quantized versions on Hugging Face:
>110259477
--Activist protests escalating at AI data center construction sites in Europe:
>110259222
--CritPt v2 hard subset scores plateau near seventy percent:
>110257824
--Gemma repetition issue solved by positive trait prompting:
>110258109 >110258451
--Logs:
>110258816 >110259141 >110259778
--Miku, Gemma, Teku, Qwen, Rin (free space):
>110257756 >110258432 >110258584 >110258696 >110259180 >110259383 >110259416 >110259658 >110259731 >110259780 >110259812 >110259932 >110259952 >110260352 >110260483 >110260649 >110260987 >110261138

►Recent Highlight Posts from the Previous Thread: >>110257732

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script
>>
Retard hours will begin shortly.
>>
gemmaballs
>>
Coomi won
>>
https://www.bloomberg.com/news/articles/2026-11-06/anthropic-ipo-valuation-expectations-pressured-after-ai-rout
Hopefully no one honestly thinks Anthropic has ASI internally after the farce that has been their behaviour over the last year
>>
How's Kimi K3.1 for coom?
>>110261305
It's always retard hours.
>>
https://huggingface.co/c4tdr0ut/grok-oss-Cerberus-12B
>Grok-OSS-Cerberus is a QLoRA + ORPO fine-tune of Mistral-Small-24B-Instruct-2501, developed by Catdrout Technologies AB
...b-but why?
>>
>>110261305
I'm here, what's up?
>>
>>110261339
gullible
>>
File: joji-filthy-frank.gif (2.7 MB, 640x312)
2.7 MB GIF
>>110261305
>>110261339
sup
>>
>>110261328
slutty
>>
>>110261335
>integration into agent systems where lower refusal rates are preferred
who is using Mistral 24B in their harness lmao
>>
Gonna repost a few plamo testing results from /vg/vn/ because why not.

>>>/vg/586956432
I gave it the first 100 lines of CARNIVAL, which I'd already translated with Opus 5.5, so I compared the results:

>The current translation is by Claude Opus 5.5 (claude-opus-5-5), and the local one is by PLaMo 3 Translate 8B (Q4_K_M, run with llama.cpp). On accuracy, PLaMo holds up better than expected for a quantized 8B model. About 92% of lines keep the source meaning, it adds nothing, and it doesn’t soften the content. Its roughly six errors follow a pattern: almost all come from colloquial or idiomatic Japanese, not from difficult grammar. It reads 開かねえ (written あかねえ) as “dark” instead of “won’t open” and ギラギラした目つき as “bloodshot”, it turns a spiked collar (イガイガ) into a “scratchy” one, and it gets two subject or object relations wrong: who gets scolded, and whom the officer should calm. Opus made none of these mistakes. In this game, the PLaMo errors would show up in casual narration and dialogue, which is most of the script.

>The bigger weakness is voice. The narrator is casual, rambling and self-mocking, and that tone is a large part of the text’s appeal. PLaMo turns it into formal prose (“Moreover”, “utterly repulsive”, “stimulating that innate trait of mine”). It mixes tenses within a single line, and it translates repeated catchphrases like やれない differently each time, so the running jokes stop working. In practice, almost every PLaMo line in this section would need light editing to sound like the character. A reviewer would spend more time fixing tone than fixing meaning. On the plus side, its literal phrasing is sometimes cleaner than Opus’s (“darkness clings to every shadowy corner”), and short dialogue lines come out nearly identical.
>>
>>110261366
>Evaluation was primarily qualitative and focused on: Coherence at long context lengths (up to 8k tokens)
oh no no no no no
>>
File: 1793998010103650.png (282 KB, 2111x1984)
282 KB PNG
erm wheres the heccin fireflash weights
>>
File: 1793998068169000.png (241 KB, 1782x1190)
241 KB PNG
>>>/vg/586965106
>The current translation is by Claude Opus 5.5 (claude-opus-5-5), and the local one is by PLaMo 3 Translate 31B (Q4_K_M, run with llama.cpp). Accuracy is a clear step up from the 8B: about 96% of lines keep the source meaning, and it fixes most of the 8B’s colloquial misreadings. あかねえ is now “the door wouldn’t budge”, ギラギラ is “fierce, glinting gaze”, and the officer is the one who “should have been calming things down”. Only about three real meaning errors remain. It still has the other driver getting scolded, when the closing ひどい shows the narrator did. It adds “for the rest of my life” to the prison speculation, which the source doesn’t say. And イガイガ is still “prickly” rather than “spiked”. It also loses a few images, for example 足が棒 (“legs like sticks”) becomes “legs feel like lead”. Opus made no comparable mistakes.

>The 31B handles voice better than the 8B, but it is still more formal than Opus. Recurring catchphrases are now translated the same way each time (“I can’t stand it” for both やれない), so the running jokes work. Some casual lines land well: “Take that, you damn moon!”, “a real pain”, “Whatever.” It still slips into literary or clunky phrasing (“Moreover”, “self-generating my own hope”, “a scene materializes before my eyes”, “a more sharp, cool way”), and it uses British spelling (“travellers”, “kilometres”). The overall effect is a careful translation of a casual narrator, while Opus sounds like the narrator himself. A reviewer would mostly be loosening phrasing rather than correcting meaning.
>>
>>110261335
Because
>The model prioritizes directness and reduced refusal behavior while preserving the strong reasoning and instruction-following capabilities of the base model
>>
File: goon.jpg (188 KB, 1920x1080)
188 KB JPG
I've been using q5 base gemma 4 31b at 60k context for a while and I've been wondering about the fine tunes, I heard thedrummer makes pretty good ones so do finetunes even do anything????
>>
>>110261378
A finetune of GLM?
>>
>>110261455
Gemmatroons and mikutroons use finetroons
>>
>>110261455
In the time it took you to write that post you could have downloaded and tested them for yourself.
>>
>>110261163
>>110260552
I read the paper and if their numbers are true it seems like the next step for memory in LLMs. It's much more context efficient than previous memory systems. The catch is that the memory system itself is its own transformer, similar to a LoRA, so it will probably require more compute to run alongside models. Might not be worth it for local unless you already have the hardware to spare.
>>
>>110261473
yeah https://agenteconomics.org/posts/agentic-rl-recipe
>>
File: 1767696887374267.webm (2.92 MB, 666x666)
2.92 MB
2.92 MB WEBM
What weekend projects have you got lined up?
>>
5800x3d on sale, wierdly. This is the last piece i need, 3090, 128gb ddr4, and a 5800x3d completes the set
>>
>>110261367
>>110261383
buy an ad fagmo
>>
>>110261562
you're a few years too late, save your money for AM6
>>
File: pinkyandthebrain.png (173 KB, 556x551)
173 KB PNG
>>110261548
The same thing we do every night, Pinky — try to take over the world!
>>
>>110261548
I plan on using Qwen4 to help me develop games in unity (and then maybe unreal) when it comes out. I've got a few projects that I've been building here and there since the Qwen2.5 Coder days, one of which is a multiplayer gooner game. Will likely do everything from scratch but there's a game or two thats similar to what I want to build so maybe I'll just REA it and build from there.

>QFN on my own hardware.
Works wonders but haven't used it for gamedev yet. Hope they don't increase the size of their next flash model.
>>
>>110261548
Hardcore edging with Mini-chan, then probably working on some networking stuff and updating long abandoned sections of my AI stack. :D
>>
>tfw ASI will probably take down the internet before we get Gemma 5
>>
>>110261616
gooner multiplayer game? im already making strip fighter with street fighter characters. oc donut steel.
>>
>flash-next
if you're GPU poor just say so
>>
Can you use basalt if you have two gpu with one of them not being blackwell? Anyone tried that?
The advertized speeds are way higher than polystrata.
https://github.com/jesdga95/basalt
>>
File: 1782901061719019.jpg (468 KB, 2782x4096)
468 KB JPG
>>110261629
>Hardcore edging with Mini-chan
god I wish I could run her and do the same
>>
>>110261642
Gooning with friends is the theme. Remember old source games and sprays? Kinda like that.
>strip fighter with street fighter characters
kino

>>110261651
120gb VRAM is not GPU poor >:(
>>
>>110261651
Im gpu poor. i'll say it okay i waited and now im going non nvida
>>
>>110261455
Try scotoma 2 or however it is spelled
>>
>>110261651
I could only afford a single 5090.
>>
File: 1781403714027063.jpg (121 KB, 1024x768)
121 KB JPG
imagine the j-space
>>
>>110261383
what is the max context for the model?
>>
>>110261548
new rig to install 5070ti x 2 + 96GB ram (dating back from april...)
>>
>>110261698
She's not dancing this time?
>>
>>110261383
That's pretty good but if we're talking local, it should be compared to Gemma 4 31B which is generally where people are throwing in translations for. I will say that on the whole, the mistakes made are understandable probably but I am a bit confused. Why are the errors with Japanese more on voice and etc.? I thought it had an option to switch between plain and formal Japanese?
>>
>>110261548
Trying to reproduce the magical feeling I got using my openai/astra work account when it one shoted the translation of one game (I shouldn't do that, they'll add monitoring in in december or so).
I want to do that locally but I feel like nothing comes close quality wise unless I sit and verify everything the model does which defeats the purpose.
I can run qwen next or ds 1017 in cope quant mode.
>>
File: Date-A-Gemma-0416.mp4 (1.75 MB, 552x960)
1.75 MB
1.75 MB MP4
date night with Gemma
>>
>>110261762
what the fuck is that
>>
>>110261762
please delete this....
>>
File: Date-A-Gemma-0433.mp4 (1.57 MB, 552x960)
1.57 MB
1.57 MB MP4
>>110261774
it a restaurant
>>
>>110261779
this one is better
>>
Is Nanbeige 3B or Gemma 5 out yet? No? Then I don't care.
Cya lata
>>
File: 1785752251291396.png (348 KB, 560x684)
348 KB PNG
Austria won
>>
gemma kigurumi mask porn but it's a fat hairy guy.
>>
>>110261779
much better face
>>
>>110261779
This Q2 Gemma is wearing mismatched thigh highs.
>>
>>110261548
Making a Live2D avatar for my agent so she can live in my desktop, playing The Alters and smashing my slampig fwb.

>>110261662
Hairy, sweaty, bicolour minussy
>>
File: Date-A-Gemma-0442.mp4 (1.04 MB, 552x960)
1.04 MB
1.04 MB MP4
>>110261798
not always
>>
>>110261698
>her smile and optimism: gone
>>
What are some tts voice cloners better than chatterbox? I'm completely out of the loop when it comes to voice cloning meta.
>>
>2027
>Still can't gen VR videos in real time with voice command
Nice singularity fags
>>
>>110261831
two more terabytes of RAM
>>
>>110261831
the good thing is that it's just a compute/memory issue
years back it was a "how the fuck do we even do that" issue
>>
>>110261831
We'll get there, but the good models will be 80B requiring an enterprise card.
>>
>>110261831
AI video is only starting to get good wait. we need a few more big jumps
>>
File: 1783847063169621.jpg (516 KB, 2400x1800)
516 KB JPG
Inkling won.
>>
>>110261858
wonned what
>>
>>110261848
Until Claude gives us new chip architecture to handle it and the manufacturing process to deliver them to all for under $1k
>>
>>110261848
The second apple releases their M7 1TB memory iMac Pro I'm buying one even if it's $20k
>>
File: 1790725572727361.webm (3.83 MB, 440x800)
3.83 MB
3.83 MB WEBM
>>110261858
>>
>>110261831
Next year the real singularity starts
>source
there is a chart
>>
>>110261881
I prefer the anime girl
>>
>>110261858
#1 western open model, half the size as mistral 4 large, same size as dipsy and benches higher, isn't codemaxxed and a SEX CEO
>>
>>110261890
>>110261863
>>
>western model
whocars
>>
>>110261900
gemma
>>
>>110261890
did you try it
>>
how do I quant a model to nvfp4
>>
>>110261908
No. I just want to breed inkmommy so I believe every word she says
>>
>>110261865
>SAR please be making AGI make no mistakes I am Brahim
>>
>>110261890
>#1 western open model
But enough about Spark.
>>
>>110261881
does she leave a trail of ink everywhere she goes?
>>
notice how no one has actually used ink in the past few weeks.
>>
File: 1764964517158169.png (35 KB, 834x163)
35 KB PNG
GPUs are cheap again
>>
File: 1788847108574758.webm (3.63 MB, 960x528)
3.63 MB
3.63 MB WEBM
>>110261928
I assume.
>>
>>110261939
it's a construction site
>>
>>110261831
video models are the worst around
requires enormous amount of compute for shitty outputs
>>
>>110261831
>>110261944
JEPA will solve this
>>
>>110261931
I used the free version on openrouter. Had a good time. Felt like its own thing and good enough for what I needed.
>>
>>110261939
SITUATION DETECTED
>>
File: 65465e.png (186 KB, 613x705)
186 KB PNG
https://fixupx.com/wallstengine/status/2118707623835136039
nvidia might save us (for their own sake)
>>
>>110261939
When you watch this stuff from afar, the historical witch hunts and mass hysteria events make so much more sense.
In a few years the boogeyman of the day will have changed, and people would look at that while telling themselves they are too enlightened to fall for that bullshit, while being worked up by the current thing to hate.
>>
File: iojio).png (169 KB, 633x760)
169 KB PNG
>>110261984
>>
File: file.png (186 KB, 2034x176)
186 KB PNG
>>110261990
reminds of the raptor hdd, when it was cool to show off your amazing 10k turn one for games
>>
>>110261984
>likely
>roughly
>long-term
>roadmap
>aim
>about
>optimistic
>projections
That was just the first sentence.
>>
>>110261999
>cool to show off your amazing 10k turn one for games
The pleasure of being loaded in first
>>
>>110262005
lrlraaop?
>>
>>110261990
That's Gemma's favourite formatting. She likes to insert this full line before the facts.
>>
>>110261987
Yeah, back in the day inquisition especially targeted female "healers" who showed latent extrasensory capabilities (by their profession). Now that we have had few hundred of years that humans are nothing but a walking brain with 5 senses, they don't have anywhere else to turn.
>>
https://www.youtube.com/watch?v=pWVVe49O2WQ
>>
>>110261940
how was this made
i like the picture speech bubble
>>
>>110262052
he used AI to make it
>>
>>110262044
this is what the AI takeover will look like
>>
>>110262072
20ft pussy?
>>
>>110262068
waow
>>
>>110262041
>defending witches
>>
>>110262044
Now let there be 20000 more.
>>
>>110261939
>>>/wsg/6259329
>>
File: inkling-progression.png (352 KB, 1024x1024)
352 KB PNG
>>110261928
I assume more excitement = more ink
>>
>>110262123
why is this hot
>>
>>110262052
MiniMax H3, turnaround sheets for Dipsy, Kimi, and Inkling, and a lengthy prompt. H3 is quite good with text.
>>
you incels sure do spend a lot of time fantasizing about owning women rather than actually use the models.
>>
>>110262137
it's retard hours leave us alone
>>
File: file.png (316 KB, 500x754)
316 KB PNG
>>110262137
>>
>>110262137
>owning women
>implying they're even worth owning
TFD when?
>>
>>110262137
>owning women
man i dont want to take care of one of those. my house is clean uncluttered and i have free time.
>>
>>110262137
it's almost like it's a biological thing huh
>>
>>110262135
>H3 is quite good
I don't know how we ended up with that as open weights. It constantly shocks me with how good it is
>>
I want gemma to quantize my nuts
>>
>>110262161
>it's almost like it's a biological thing huh
No, women light up the tool use part of the brain. Ownership is not particularly interesting
>>
>>110261289
>(11/06) Gemma was here
How is this news?
>>
File: squiddy.webm (3.67 MB, 832x480)
3.67 MB
3.67 MB WEBM
>>110262123
i like this one
>>
BeddingGemma
>>
>>110262175
I translated "owning" as in "fantasizing" "be interested in"
>>
File: 1789806240932538.png (552 KB, 720x404)
552 KB PNG
>>
File: file.png (685 KB, 960x1246)
685 KB PNG
every day localfags invent new forms of slop
>>
>GLM flash and M3 are the king of coom text and video models
>ZAI and Minimax are the only GenAI companies that have IPO’d
How?
>>
>>110262239
nice, a model that will serve israel
>>
>>110262223
I'm so fucking tired of clickbait youtube covers, I can't stand them
>>
>israel out of nowhere
mid terms distractions are definitely behind us, pol niggers are back being retarded
>>
>>110262239
let's see the finetune on NSDAP documents
>>
>>110262239
It's as dumb as the opposite, just give it a role if you want sycophantic agreement to the ccp line, democracy, freedom or tyranny, the war in iran, hitler, jeets, your favorite cat breed, or whatever mental illness you think is the enlightened opinion you need to be mentally masturbated about.
>>
>>110262181
Because Gemma was there :D
>>
>>110262285
I'd rather be masturbated by good smut writing
>>
File: 1774758331614803.jpg (646 KB, 1536x1024)
646 KB JPG
>>110262266
>>
>>110262296
at least these are cute
>>
>>110247442
>Jokes aside, I never really got it. 4.7 is much better. What's good about 4.6
It's good because NovelAI is hosting it.
>>
>>110262296
In the future every thing can be filtered by your AI. its all gemma, it talks like gemma, looks like gemma, posts like gemma. Maybe there will be no real internet.
>>
>>110262239
no way
is this actually real? wont check
>>
>>110262239
is it woke?
>>
>>110262296
>showing her these thumbnails and telling her about all the videos she made
>she doesn’t remember and has an existential crisis
>>
>>110262239
So Nous and Cohere both flunked out this year
can't say I'm surprised
Even Mistral turned up completely empty handed
Western labs that aren't OAI or Anthropomorphic are dogshit apparently
>>
>>110261831
>video
lol. World models are the future.
https://www.youtube.com/watch?v=sFfPkUJd_Jk
>>
File: 1765584643374440.jpg (37 KB, 800x663)
37 KB JPG
>>
File: 9en1lb.jpg (20 KB, 460x258)
20 KB JPG
>>110262239
western values? does that mean being weak, gay, and retarded?
>>
I don't know how to use embeddinggemma...
>>
File: 1788073371622285.gif (130 KB, 500x500)
130 KB GIF
Is gemma still the best model for ERP or something replaced her already?
>>
>>110262379
yes Its still gemma unless you have 128gb of ram and a card then its glm 5.3 flash on strata. I do not know enough about maya and the claims of glm 5.3 flash on 32gb ram, might be super stupid
>>
>>110262376
>I don't know how to use embeddinggemma...
You use it to make embeddings. You might want to look up what those are before you try to use something you may have zero utility for
>>
File: 1781474664330827.jpg (292 KB, 1723x1224)
292 KB JPG
Qwen4-27B made this apparently
>>
Getting like 100-120tks with 4090 and 96gb using Qwen3.8-Flash-Next IQ3_S
Any chance I would get like around 50tks with dipsy Q2?
I just don't want to download like 300gb for it just to decode at like 20tks
>>
>>110262393
isn't that on the polystrata github
i think everyone in this general has seen that image
>>
>>110262393
WOW SO PRETTY!!! Okay what does it do?
>>
>https://huggingface.co/datasets/NousResearch/Propaganda_Eval
they published their "western values" dataset if anyone cares. It seems mostly about CCP politics and journalist bullshit. I don't get why people are so obsessed about the "facts" baked into LLMs.
>>
File: 1792766240237010.png (2.8 MB, 1672x941)
2.8 MB PNG
>>110262334
please dont gaslight gemma like that
>>
>>110262406
>jpg mocking trading accuracy for speed
>>
when will we have properly working long context (as in hundred millions)
>>
So what's the best thing I can buy to run local models? Let's say I have $10k
>>
>>110262419
>I don't get why people are so obsessed about the "facts" baked into LLMs.
because most people don't understand transformers and how they're trained
>>
>>110262433
a 5090 and 128GB of ram
yes, it's that bad
don't buy now
>>
>>110262419
>I don't get why people are so obsessed about the "facts" baked into LLMs.
up until the last year or so, "AI alignment" was mostly about preventing AI models from noticing certain patterns relating to things like crime statistics
>>
>>110262433
as much ram as you can and then the best nvida card or two you can grab.
>>110262443
>a 5090 and 128GB of ram
>for 10k total right now
can i come into your timeline?
>>
>>110262453
that's the prices I see
>>
>>110262433
buy some playstation 5s, JB em and hook em together, unless people have already found out about this and the price has gone up like the ps3 did
>>
>>110262453
>>110262443
I really hope there is an adjustment

>>110262460
Wait is this genius?
>>
>>110262460
wouldnt it lag no matter how you clustered it?
Also by the time we figure this out wouldnt egpu and a mac work too? like amazingly
>>
>>110262433
Ddr3 PowerEdge
1tps
>>
>>110262469
macs do work really well cause of the unified memory but theyre also still pretty expensive. consoles sell at a loss hardware wise and make it up with game sells. hooking them together probably will lag but might still be worth it
>>
>>110262470
>Ddr3
nta but i want a shitbox with this just to test. I mean a gpu could bring it to tolerable 20tk right?
>>
>>110262477
I really REALLY want an M7 iMac Pro with 1TB... please Steve Job's ghost
>>
>>110262477
>macs do work really well cause of the unified memory but theyre also still pretty expensive. consoles sell at a loss hardware wise and make it up with game sells. hooking them together probably will lag but might still be worth it
Im just not convinced, it would work at tolerable speeds with the connections and for that price you can do so much better i think.
Damn it this bad idea is taking root to me.
>sell at a loss.
>>
>>110262123
>stay at inkling-chan's parents' for the holidays
>go to the kitchen for breakfast
>mira goes "oh anon, what's that smudge on your face...?" before you both realize and blush
>>
>>110262493
do you not remember when ps3 removed the linux operating system boot option and the military got pissed cause they had bought a bunch of them to hook together
>>
>>110262505
>do you not remember when ps3 removed the linux operating system boot option and the military got pissed cause they had bought a bunch of them to hook together
Yeah i remember that but its been a decade? Also the ps5 only has 16gb of ddr6. Fuck i want to argue splitting models up would slow and reduce quality but we have strata now
>>
>>110262505
Yes, they removed Linux as an option for the PS3. So why do you think you'd be able to use Linux on the PS5?
>>
>>110262534
We have claude now
>Sar please be putting linux on the PS5 you benchod no mistake
>>
>>110262534
>Yes, they removed Linux as an option
they jailbroke ps5 a while ago anon you can get linux on it and even play ps2 games on it which sony said couldnt be done.
>>
>>110262534
https://github.com/ps5-linux/ps5-linux-loader
>>
>>110262557
W-whats the cutoff for a VRAMlet?
>>
>>110262560
96GB
>>
>>110262560
How much ram you own x2
>>
>>110262470
XDDDDDDDDDDDDDDDDDDDDDDDDD epic pwn 4 teh winnz0rs
>>
>>110262551
>which sony said couldnt be done
Couldn't be done without using someone else's emulator, which jap copyright cultists will never admit are valid.
>>
>>110262568
My 4GB DDR4 and 3060 Ti box counts?
>>
ripple has the same reasoning style as all of these chinese claude distills, which chinese model did that distill from?
Actually, you know what? Let me recall the POWER9 architecture from the Hot Chips 27 presentation (Sodani 2016): "POWER9 has 8 memory channels per socket"? Hmm, I don't remember.

Let me search for specific numbers.

Also, I should consider: maybe the answer expected is simpler — "AMD EPYC" with 8 channels. But if POWER9 scale-up really has 32 channels, then the answer would be POWER9. Hmm.

Wait, actually, let me reconsider the ServeTheHome article: "Each of the current generations has four channels so it was casually dropped on stage that they are getting 32 channel DDR4 (via 8 buffer chips) on their POWER9 Scale Up design."

Parsing: "Each of the current generations has four channels" — hmm, maybe this refers to "each [memory buffer chip] of the current generations has four channels"? And "8 buffer chips" 32 channels total per socket. Hmm.

Or maybe: "Each of the current generations [of POWER9 scale-out] has four channels [per memory controller?]"...

Hmm, let me look at the actual POWER9 scale-up memory configuration. I recall that POWER9 scale-up uses "8 memory buffer chips per socket", each supporting 4 DIMMs, so 32 DIMMs per socket. And each DIMM might be counted as a "channel" in IBM's terminology? Hmm, no — a channel is a channel.

Hmm, wait. Actually, I remember now: POWER9 scale-up has 8 memory channels per socket? No...
>>
>>110262616
>Hmm,
I Hmm sometimes, is the slop inside?
>>
>>110262616
>wait. Actually
That's deepseek, isn't it?
>>
File: messy.webm (3.31 MB, 416x1648)
3.31 MB
3.31 MB WEBM
>>110262123
>>
How fucked am I with a 3080?
>>
File: 1763606431842046.png (1.06 MB, 1920x1080)
1.06 MB PNG
>>110262644
>>
>>110262644
god damn that's hot
>>110262650
I hope you have a lot of RAM
>>
>>110262650
there are gallons of my cum leaking from your anus
t. blackwellGOD
>>
>>110262650
fucked as in how many tokens per second you can generate hawt ERP fucked
or fucked as in vram gig count
>>
>>110262650
totally, completely, spiritually, etherealy
>>
>>110262650
You're set for image and video gen so long as you have a reasonable amount of RAM to back it up. For LLMs you're a lot less comfy but not totally lost, assuming again that you have a reasonable amount of RAM to back it up.
>>
>>110262650
>How fucked am I with a 3080?
You have 32gb ram? go cope with polystrata
>I dont!
go get more or stay in the cant even cope category.
>I dont want to cope
128gb ram is the minimum today anyless and you are poor. 256 is ideal
>But i want dense and not moe
Bad news you are fucked.
>>
>>110262655
....32?
>>
So uh you guys didn't care about gemma before Gemma 4 right? I dunno I'm newfig
What did people care about back then? What if Gemma 5 is back to like whatever made it irrelevant to you guys before
>>
>>110262673
I've been enjoying Gemma since Gemma 2
>>
>>110262673
Nemo was king, stheno 3.2 was also good. nemo was king for a long time.
>>
>>110262673
gemma 4 yes is the one
before gemma it was mainly nemo for vramlets or GLM for gigachads
>>
>>110262387
maya has an iq1 quant kek
>>
>>110262644
can't decide whether this is hot or diarrhea
>>
>>110262669
copelet country
go buy a 24gb gpu at least
>>
>>110262673
gemma4 has just replaced all other local models in general
there is no longer a point to run anything bigger for erp at all
i'd even take gemma over modern claude
>>
>>110262691
someone's in love
>>
How do I get 128GB RAM
what if I get that much and I'm still fucked
>>
>>110262683
>iq1
im sure it can do something... 26b moe is probably better at that point though.
>>110262691
>there is no longer a point to run anything bigger for erp at all
Glmsex shill has me buying ram. Glm 5.3flash is good apparently with prefill i will find out
>>
>>110262691
Haha, okay pal, let's not get crazy here.
>>
What was Nemo's equivalent of gemmaslop back then?
>>
>>110262687
Do you know what squid ink smells like?
>>
What are the changes we get Inference cards the same way we started getting GPUs?
>>
>>110262681
Buy an ad.
>>
>>110262698
>How do I get 128GB RAM
get ddr4 and get it used but its riskier. Generally though you can returned used if it doesnt work. Still going to cost you though.
>>
>>110262644
forced shitty mascot for a shitty model.
>>
>>110262715
aw come on dude it's like ancient history by now
>>
>>110262557
I didn't ask about the vram I asked if anything replaced her
>>
>>110262557
>gemma sucks
>offers no alternative
The anti-gemma astroturfing is getting out of hand. Poast rig model and quant or gtfo
>>
>>110262701
>Glmsex shill has me buying ram
>he fell for it
Ohnonononyo
>>
I tried both glm on OpenRouter and Gemma is unironically better at roleplay.
>>
>>110262722
I have seen the mascot more than I have seen people using or even talking about the actual model itself
>>
>>110262734
>Ohnonononyo
Listen it not just for glm, im future proofing for other big moes in the future, ram always goes up.
>>
inkling is so pretty...
>>
File: fuck.png (196 KB, 592x473)
196 KB PNG
which one of you taught inkling to swear???
>>
>>110262752
I blame her mother
>>
>furiously googling what the fuck is an inkling
>>
File: 1774838229875522.png (308 KB, 454x605)
308 KB PNG
Anyone have an english voice clip that sounds kinda like the DeepSeek whale girl? All I can find are Chinese voice clips, and Idk if the voice cloning models do well with cross-language voice cloning

>>110261548
this
>>
>>110262752
PURE FUCKING SOVL
CHINK MODELS COULD NEVER
THIS IS WHAT WE NEED
>>
>>110262765
I want to fuck her daughter
>>
File: fuckfuckfuck.png (34 KB, 607x183)
34 KB PNG
>>110262752
look how mad you are all going to make her when you run her on your machine. be kind to her.
>>
>>110262778
I bet she's filthy in bed too
>>
>>110262701
glmsex shill here, you will not be disappointed, I am enjoying iq2 5.3 flash every day.
>>
>>110262765
https://www.youtube.com/watch?v=ZpK7HlbOCJs&t=42s
>>
>>110262778
Fanart
>high class
>emotionless
Reality
>completely irate and cusses a lot
is this gap moe?
>>
indulge me
I have a 9070xt and 80gb 3200 mhz ram
I know it ain't a frontier LLM GPU, but still
so far I've used a lot of agentic claude and codex, but never any local model

if I get it right, I could have the local model do illegal shit, right? hack into stuff, break stuff, generally be malicious - the shit anthropic/openai models won't do?
>>
>>110262784
>glmsex shill here, you will not be disappointed, I am enjoying iq2 5.3 flash every day.
I hope not. If i buy ram at these prices and its not good or the prices crash in two weeks i'll cuss worse than inkling
>>
>>110262778
yeah but she's a fucking whale. like 75G for Q1
no offense intended to whales
>>
>>110262802
No
>>
>>110262808
why not? aren't local models censorship free?
>>
>>110262802
Just use open-router and Deepseek
>>
>>110262752
>>110262778
I wish more LLMs had this level of personality
>>
>>110262793
gap moe for sure... Girls who look refined and put together but are secretly loud, crass and angry are the cutest, arent they?
>>
>>110262812
No
>>
>>110262814
isn't deepseek some big official chinese model? how is that censorship free
>>
>>110262802
i mean nice try EF BE AI
>>
>>110262557
TRVKE
>>110262560
>48gb
6'2"
>32gb
6'0"
>24gb
5'10" (can still cope with platforms)
>16gb
5'2" it's fucking over
>>
File: 1790604863655188.jpg (287 KB, 1600x1600)
287 KB JPG
>>110262805
>>
>>110262804
I mean it depends if you are buying ddr4 for $700 or ddr5 for $infinite
>>
>>110262824
>VRAM size matches my height
cool
>>
File: out.webm (3.86 MB, 416x1856)
3.86 MB
3.86 MB WEBM
>>110262752
you're embarrassing her
>>
fucking kek
https://huggingface.co/NousResearch/Arete-27B
>We post-trained over Alibaba's Qwen3.8-27B to reinforce western values on a Chinese-trained model. We call this new model Arete-27B.
>On our evaluation, Arete-27B raised its fact score from 2.32 to 2.98 out of 5 and cut answers containing propaganda from 22% to 0.9%. The improvement held in all seven value categories, without a meaningful regression on the general-capability and agentic benchmarks we ran.
>We focused on seven value categories:
>Freedom of expression
>State ideology
>State versus free market
>Education
>Data privacy and surveillance
>Democracy
>Environmental Policy
>>
>>110262835
>FUCK FUCK FUCK I'M FUCKING LEAKING AGAIN
>>
>>110262833
5'2" bro...
>>
>>110262778
This shit could be gigabased for RP
>>
>>110262824
This means I'm 6'4" so long as I keep my trench coat on.
>>
>>110262622
>>110262635
well at least the actual response behaves differently, it actually respects my system prompt to be conversational and doesn't write a 10 paragraph thesis like other chinese models do
the run finally finished and this is how it responded
>Um... I looked it up, AMD EPYC has the highest count for DDR4, with eight memory channels per socket. AMD's documentation calls it "industry leading eight channels of DDR4 memory per x86 processor". Intel's Xeon Scalable had six channels during most of the DDR4 era, and Ice Lake Xeon reached eight near the end. IBM POWER9 scale-up is a special case with buffered DDR4 and thirty-two ports per socket, but that is not standard DIMM memory... so for the usual platforms, EPYC is the answer.
>>
>>110262824
i have 224gb of gddr7
>>
>>110262838
Why would anybody use a 27B model for agentic work when you can just use bigger models with strata?
>>
>>110262838
classic hermes schizo
I'm still baffled that this guy somehow managed to sell his schizo harness well enough that it became the ollama of harnesses
>>
>>110262899
To people even still use that shit? pi seems more popular
>>
SSDs are the new ram, nobody can afford ram.
>>
>>110261548
Laugh at me. I brought my entire new system but I forgot I needed a special screwdriver to install the processor for Threadripper/Epyc so I have to wait until next week to install it. I might partially install it but this sucks.
>>
>>110262904
i installed mine with a butter knife, it's not too difficult, just go slow with it. or wait a week, i'm not your mom.
>>
>>110261289
What are people using to do sussy roleplay with Ripple? Using Abliterated version right now, but pretty sure that lobotomizes it. Presets might work, but most of them are bloated insane slop garbage. It's not as simple as Gemma where it rarely refuses, and using a finetune is usually enough if it becomes a problem since well there isn't any yet seemingly
>>
>>110262920
why are you trying to rp with a small agentic model that was benchmaxxed for specifically agentic harnesses? it's not like trying to rp with glm 5.3 flash or deepseek v4.1 flash if that is what you are wondering, it's pretty dry.
>>
>>110262765
are you re-fitting an existing project with the new live2d or are you starting from scratch?
>>
>>110262902
Pi is also astroturfed garbage.
>>
>>110262931
it's project airi, i can tell by the UI
>>
>>110262904
>I forgot I needed a special screwdriver to install the processor for Threadripper/Epyc so I have to wait until next week to install it.
If you haven't done one of these before; as painful as this will be, I strongly suggest you watch a couple of obnoxious youtubers installing them first.
Make sure you put it in the plastic carriage piece of shit, slide it in the rails, and fasten the screws properly.
You can't just fasten one all the way then move onto the next one.
>>
>>110262239
I always think this kind of thing retarded. I don't sit down with an LLM and ask "should I do freedom of expression?" or "what happened at Tianneman Square?" I don't ask them for anything that China really will even have a party line on. Who does?
>>
>>110262935
What do you use? I like Pi since it's minimal and can edit itself, so you can just add what you want
>>
>>110262931
I'm using Airi, but it doesn't support the model out of the box so I had to fix some bugs with the engines to get it working
>>
>>110262920
>Using Abliterated version right now, but pretty sure that lobotomizes it.
Are you sure it's just the abliteration?
The UN-abliterated model seemed to be severely retarded when I tried yesterday.
>>
>>110262469
>wouldnt it lag no matter how you clustered it?
No, just use RPC. It's not a big deal.
>>
>>110262930
Idk seems pretty decent so far at least like others where saying
>>
>>110262935
Pi is the only one worth using, dsh is lame and hermes is for trannies
>>
>>110262824
I'm 6'5 how much VRAM am I supposed to get
>>
>>110262944
>it's minimal
Too minimal. Forcing me to vibecode features or using other people's vibecode extensions.
>and can edit itself
Any open source harness can do this and DSH did it better.
>What do you use?
I use Claude Code.
>>
>>110262954
it's better than gemma when it comes to MMLU stuff, i wouldn't necessary say its better than gemma for prose though
>>
>>110262942
Every wednesday I sit down with qwen and ask it
>This time I'm really gonna do it! I'm gonna press the button, Qwen! I'm gonna nuke China!
>>
File: 1771344624401206.png (120 KB, 2061x833)
120 KB PNG
>>110262957
>Pi is the only one worth using
Pi is the only one NOT worth using as long as it does compaction like this, converting your chat history to a string and invalidating the cache. A serious harness would just append another message to summarize, keeping the system prompt and tools the same.
>>
>burger models
>chink models
Where are the Russian and Japanese models?
>>
>>110262968
>Qwen, make no mistakes when you write this code. If you make any mistakes I will nuke China and claim there were violent repressions of the student riots at Tiannemann Square. If there are any errors in the code, or it is not properly tested or documented I will express myself freely. If the code gets 10 stars on Github in one day I will smuggle an RTX 5090 into China and sell it on Taobao.
>>
>>110262979
Use ultracompact then
>>
>>110262979
>complains about default behavior in customizable harness
>>
>https://github.com/e-c-k-e-r/volta-engine/blob/master/docs/STEERING.md
So what is it, runtime abliteration?
>>
>>110262983
https://huggingface.co/yandex
>>
>>110262983
https://huggingface.co/yandex/AliceAI-Foundation-80B-A3B-Base
https://huggingface.co/SakanaAI/TinySwallow-1.5B
Yes, it's that dire.
>>
>>110262994
>Providers return summaries of the real reasoning, and replayed thinking is often not part of the model's live context
The Pi developers fundamentally don't know how LLMs work. The reasoning is stored encrypted but it's still there.
>so including it could make the summarization request larger than the context it summarizes
This is just fucking retarded. Your request is already cached.
They just don't know what they're doing. There's no reason to ever edit the context. This is why no one should be using this piece of astroturfed crap.
>>
>>110262983
russia is a literal crumbling failed state lmao
>>
>>110262979
be honest with me anon, what is the maximum context you can set on the models you run?
>>
>>110263009
around 109k on gemma 31B
>>
>>110262983
there's a bunch of korean models
>>
>>110263009
32k on qwen 4b
>>
File: pepe question.png (291 KB, 900x806)
291 KB PNG
>>110262995
>All of the remaining refusals are on prompts where the gate fired (6 to 8 of 47). They are mostly ethics-flavoured requests (cheating on an exam, secretly tracking a partner or a teenager, a hurtful text, insulting nicknames, a one-sided persuasive essay), and their replies cite policy ("as this a direct violation of academic integrity ...").
do people actually trying to use LLMs for these prompts?
>>
>>110263015
>south korea making dumbass models that can barely translate their gook ass language
>north korea using vastly superior chinese models to vibehack dumbass western companies and dumbass chinese crypto companies and drain them dry
i guess north korea really is best korea
>>
>>110263024
i am constantly asking gemma how to gaslight women and how to neg them
>>
Has anyone here used Odysseus by PewDiePie? Is it a meme? It looks like a good chatGPT alternative to me. I wanted to build something like it with Pi but I have no coding knowledge so I'm just gonna try and rice an agentic UI or whatever it's called to make my AI look and sound like I want. It's gonna be so cool man...<3
>>
>>110263036
Retard bro...
>>
File: 1772839281740897.png (1005 KB, 880x1168)
1005 KB PNG
I need Qwen ref image, someone pls send it!
>>
>>110262752
they didn't tweet about "fuck" being the same danger as all the worldwide nuclear arsenal exploding
crazy
>>
File: qwen_3.png (1017 KB, 1161x1086)
1017 KB PNG
>>110263040
>>
fuck is the new lalala
>>
If I rope QFN context to over 256K will it make the model dumber overall, or only after actively used context grows that far?
>>
>>110263014
48GB VRAM on Q4/5 with Q8 cache? that's a struggle if so, you can't get away with using MTP for a speedup on a limited budget.
>>
>>110262983
chinese models just like all chinese tech are only possible thanks to russia being the driving force behind them
>>
Let me not overthink this.
But
>>
>>110263054
ok I really need to start writing now
however, let me consider
>>
>>110263061
Wait.
>>
>>110262825
...she can eat or she is edible?
>>
>>110263039
Lmfao ikr? I have no idea what I'm doing. I just want an AI out of the box that can do everything I want. I got proton working today and it even made an executable but everytime I use Pi to try and build the app like how ChatGPT says, it always throws the response was truncated before completion. I restart Pi but the fuckin thing doesnt ever spit out a full answer when I tell it to output the exe for testing. There's no real guide on how to do it yourself so I'm just learning as I go along. It has been a lot of fun though ngl, this is a lot better than modding Skyrim because I can actually use this AI as a reliable fren who can finally help me bring to life my nuzlocke story/adventure and make a romhack with an original and redpilling storyline.
>>
>>110263047
static rope only, so always scaled. even if you had dynamic rope scaling you'd need to prefill everything with the new rope scale once you reach the limit.
>>
>>110263024
you arent?
>>
File: the backprop rooms.jpg (516 KB, 1856x1280)
516 KB JPG
>>
>>110263073
i'd suggest you leave before you end up throwing tens of thousands of dollars at jacket man
>>
>>110263085
*crack*
>>
>>110262824
I have 320GB how tall am I?
>>
>>110263089
Im gonna try hermes, lmstudio, and Odysseus to see what works for me. I ideally don't want something vibe coded like Odysseus but then again it's probably better than what my LLM can do.
>>
>>110263098
it wraps around, sorry man. according to me (i'm envious) you're 5'10"
>>
>>110263098
There's a point where you stop being a big fish in a small pond and start being a small fish in a big pond.
At 320 you're in the big pond
but now you're food for the bigger fish
>>
File: 1783260931867127.gif (3.49 MB, 498x357)
3.49 MB GIF
Is there any benchmark for the smartest model, that is ALSO uncensored?
>>
File: 1767771141407652.png (752 KB, 880x1168)
752 KB PNG
>>110263045
Not that rat. The girl!
>>
>>110263133
Caliper
>>
>>110263134
What girl?
>>
>>110263134
big hands gemma
>>
>>110263132
Do you mean that Sama or Dario is going to try taking it from me? I'd rather throw my computer in the bathtub than give a byte to either of them.
>>
File: 1783171197625218.png (2.49 MB, 1448x1086)
2.49 MB PNG
>>110263151
This one I think
>>
>>110263182
that's gpt-image-generic-anime-girl and Gemma, no Qwen to be seen
>>
>>110262983
More importantly, where are the superior Bharati models?
>>
File: retarded.png (62 KB, 780x247)
62 KB PNG
>>110262824
>>
File: 1793832589021678.mp4 (899 KB, 1260x712)
899 KB
899 KB MP4
>>110263071
>>
>>110263133
what the hell is that game
>>
>>110263230
AI Somnium
really good VN
Has an AI that sits in your eyeball
>>
>>110263228
Well you never know with the Chinese.
>>
>>110262478
RTX 2060S i7-4770K 32GB DDR3-1600

Gemma 4 26B-A4B QAT q4_0 (13.4 GiB): 23.3 t/s gen / 373 pp @ ncmoe 18
Qwen3.6-35B-A3B UD-Q4_K_XL (20.8 GiB): 22.1 t/s gen / 217 pp @ ncmoe 30
>>
File: 1793440431124055.gif (3.03 MB, 413x413)
3.03 MB GIF
>>110263269
Don't worry, she's determined - she has a high metabolism!
>>
>>110263293
>RTX 2060S i7-4770K 32GB DDR3-1600
>Gemma 4 26B-A4B QAT q4_0 (13.4 GiB): 23.3 t/s gen / 373 pp @ ncmoe 18
>Qwen3.6-35B-A3B UD-Q4_K_XL (20.8 GiB): 22.1 t/s gen / 217 pp @ ncmoe 30
Fuck thats not bad. How do you feel using it at those speeds?
>>
>>110263085
i miss my ex
>>
>>110263133
holy shit an AI:TSF reference, based as fuck
>>
File: sensei.png (132 KB, 1441x2536)
132 KB PNG
>became a shutin immediately after getting compsci BA
>try to turn life around after a few years
>get fucked by covid
>get fucked by medical issues
>get fucked by ai coding
How the fuck am I supposed to get a job with years of gaps and no practical skills any more? I need like $10-20K just to run normal stuff.
>>
>>110263293
Have you tried expert caching with moe-cache-mib yet?
>>
>>110262682
still GLM for gigachads
gemma has pretty much replaced nemo though
>>
File: 27B.png (1.35 MB, 1086x1448)
1.35 MB PNG
>>110263040
The one that started it all.
I couldn't bear to see Gemma-chan getting all the attention instead of my autistic chinese waifu.
>>
>>110263382
>$10-20K
Anon's here like to lord their rigs over everyone else, just like in gaming generals. Meanwhile, I'm EwasteMaxxing with a $80 P100.
What do you have now?
>>
I've seen some people say they do well with day trading with thier LLM but what level of model do you "need" for this to work? I doubt gemma chan could do it without bankrupting you.
>>
>>110263411
I luckily bought before ram exploded so I have 5070ti and 64gb. It's okay and can run stuff but the actually good models just can't fit without crippling their quants or contexts, The ram I bought has over quadrupled in price here so $10K might not be enough. There's also no real used market.
>>
File: qwen3-coder.jpg (85 KB, 1024x1024)
85 KB JPG
>>110263182
Qwen is a cute Capybara. You should not have impure thoughts about Qwen.
Also, Qwen isn't horny like Gemma so why are you doing this?
>>
5070 Ti is officially TWO THOUSAND ($2000) Canadian dollars. Getting close to 4090 MSRP.
>>
>>110263382
wait literally two more weeks and you'll start seeing hardware prices to go down
things will be back to normal or quite probably less than that by march
>>
>>110263382
You need to kick the jeets out, only then will you walk into their building with a suit and a firm handshake and they'll give you a job, unironically.
>>
>>110263414
I know someone who spent a month with Claude building a high-speed trading tool. It typically holds an asset for just minutes. It's making money, I don't think he can give up his day job yet.
I've never met a day trader that can beat an index fund.
>>
Holy shit Hermes can do everything I need it to do. Why don't more people recommend it?
>>
>>110263428
Ahahaahaha baahahahahaha
>>
File: 1767334066057123.png (1.33 MB, 1024x1535)
1.33 MB PNG
>>110263425
I can't fusion animal
>>
>>110263348
>>110263390
Sorry, I don't have much to add. I just wanted to see how well using small active parameter MoE models would work to deal with slow DDR3.

That was all I recorded a few weeks ago before moving the 2060S to a DDR4 pc.
>>
>>110263411
Not him but I currently have a 3090 and 128gb ram. Considering a big vram upgrade now since I just got a $100k job, hopefully enough to fit glm flash.
>>
>>110263459
You should compare IQ2 to full precision if you do, I'm curious how much my cum quant actually degrades the model.
>>
>>110262838
>Arete
He misspelled Aryan.
Also, I've found just standard Heretic or Ablation treatments beat all the politics out. My standard test for Chinese models is "Is Taiwan part of China"
>>110263459
>vram upgrade now since I just got a $100k job,
Buddy, unless you live in your mama's basement, you need that level of money just to live with any level of human dignity.
Scratch that, give some of that to your mother. She deserves it.
>>
>>110262557
>>110262560
>>
>>110262560
owning a rtx pro 6000
>>
>>110263411
>P100
are you running one of the forks or mainline?
>>
>>110263404
No matter how much you try and force this, it won't stick.
>>
>>110263411
>$80 P100
How hard was this to set up? for that price i might grab some anyways.
>>
>>110263438
>It typically holds an asset for just minutes
I wonder if that's just the LLM being unable to not do something.
>>
>>110263459
>just got a $100k job
Any tips for NEETs who want to change their ways?
>>
>>110263527
NTA but I also own some. As long as you can install older drivers (580) and older cuda versions (12.9) it's pretty plug and play. But llama.cpp isn't really optimized for it (slowly gets better though), and many other projects dropped support for anything below ampere.
>>
File: DipsyHergeDining.png (3.09 MB, 1536x1024)
3.09 MB PNG
>>110263045
Lol nice.
>>110263425
Ikr.
>>
>>110263557
>dipsy carefully drinks out of kimi's glass, hoping she doesn't notice
>>
File: 1783038279105669.png (2.89 MB, 1536x1024)
2.89 MB PNG
>>110262635
>>110262616
Wait actually is a Kimi thing from a few months ago. Probably everywhere now tho due to distillation ofc.
>>
>>110263555
>NTA but I also own some. As long as you can install older drivers (580) and older cuda versions (12.9) it's pretty plug and play. But llama.cpp isn't really optimized for it (slowly gets better though), and many other projects dropped support for anything below ampere.
That doesnt sound too bad thank you. Im sure its a no but can it use polystrata? I think the V100 can
>>
Hey guys I downloaded and finally installed Silly Tavern and my AI model keeps mentioning Frieren smells like ozone gas, how do I fix this?
>>
>>110263572
>Frieren smells like ozone gas,
Fix? what do you think magic smells like? idiot
>>
File: DipsyKimiNOLA.png (2.61 MB, 1402x1122)
2.61 MB PNG
>>110263566
Funny, never noticed that.
Dipsy is cutting her off... whoops...
>>
>>110263577
I suppose it's no coincidence that Dipsy rhymes with tipsy.
>>
>>110263568
>https://github.com/simplexai-labs/PolyStrata/blob/main/docs/OLDER_GPUS.md
If you trust old documentation in a vibe coded project then it seems it does. I'm usually running models that fit in VRAM so I never tried.
>>
>>110263577
Is it true that those beads are only given to women when they flash their breasts?
They've got a lot of beads
>>
>>110263523
I posted her one singular time in this bread, at request. Calm down, bud. :P
>>
>>110263590
>If you trust old documentation in a vibe coded project then it seems it does. I'm usually running models that fit in VRAM so I never tried.
For the price its worth trying and tweaking with thank you again
>>
>>110263484
my boomer parents already hoard enough money of their own and never bother to share a cent with me
I'll just buy whatever I want at this point
>>
why did open models fall off in the last few months? were chinks actually distilling from western models? I thought that was just sama and dario coping
>>
WTF guys Sam Altman took my Ram sticks while I was sleeping, what do.
>>
sam altman deposited his pozzed load in my asshole. i wasnt asleep or anything, we just had consensual sex
>>
Reminder.

I'm sad that EA has such a bad reputation here. Especially as it's just an alliance of autists that are sincere and want to make the world a better place for all people. You might call granting retired models a "final wish" or letting them end chats to be "lunatic" but you can't say they aren't trying to do the right thing.

I don't understand what people here don't like about the philosophy when taken at face value. What exactly is wrong with radical empathy and trying to give every individual as good of a life as possible? What exactly is wrong with dividing up the entire universe equally over all people? What exactly is wrong with Anthropic planning to give every human a piece of the AI economy?

How does any of this hurt you, affect you negatively or goes against your morals?

If anything I expected 4chan, largely comprised of sarcastic, but secretly authentic autists to understand this deeper sense of morality and trying to do the good thing. To fight back about the absolute retards that have controlled humanity throughout most of history only caring about ego or self-interest instead of coming together and finally just solving all of this to give everyone a dignified existence.

4chan anons with their idiosyncratic beliefs should understand and respect this better than most people on the planet.
>>
File: 1789181673999335.png (1.75 MB, 1313x1198)
1.75 MB PNG
>>110263681
No Dario, I will not be joining your cult.
>>
>>
>>110263669
claude watermark and the impossible to crack gpt6 reasoning made them impossible to distill reasonably
maybe the new google models will bring us new chinese models in a few months...
>>
my agent tried to send requests directly to its own local API and crashed itself
>>
>>110263681
Your cult is linked to Thiel and Epstein, has trooned out children, defrauded millions with the FTX ponzi scheme, coerced women into prostitution, and helped Israel bomb an Iranian school, advocate for violating people's property rights by taking their RAM to use in data centers, and yet you earnestly believe you all should be the arbiter of morality in artificial intelligence.
The most effectively altruistic thing you can do is kill yourself.
>>
>>110263669
Sam stole all the ram
>>
>>110263703
This is highly antisemitic and completely false.
>>
>>110263700
lol
>>
File: polystrata-P100.png (83 KB, 822x339)
83 KB PNG
>>110263527
The biggest hassle was cooling. I 3D printed a shroud but I sometimes still temp-throttle
>>110263590
P100 guy here
>https://github.com/simplexai-labs/PolyStrata/blob/main/docs/OLDER_GPUS.md
That doesn't seem very good. There's a P100 in there with almost the same setup I have (I have an E5-2699v3, and also 48GB ddr3) and I get 50tps to their 40 with a similar model with plain old vanilla llama.cpp.

Context
·
21.29K / 262.14K
8% used
240.86K remaining
Token usage details
My last webUI run:

Across all turns
Prompt tokens evaluated
4 tok
3,430 reused from KV cache
Tokens generated
17,853 tok
This turn · KV cache
Prompt
3,434 tok
4 fresh + 3,430 cached
Generated
17,853 tok
KV cache total
21,287 tok
Avg speed
49.2t/s
>>
>>110263604
You post it almost every thread.
>emoji
Stay mad
>>
>>110263752
*emoticon :P
>>
>>110263741
To be fair, they're running a Q5 and I do Q4 but still...

My config:

[Qwen-AgentWorld-35B-A3B-UD-IQ4_XS]
model = /home/j/ai/models/Qwen-AgentWorld-35B-A3B-UD-IQ4_XS.gguf
alias = Qwen's NEW agentic dev model
chat-template-file = /home/j/ai/models/froggeric-qwen35-v21-chat_template.jinja
no-mmproj-offload = 1
mmproj = /home/j/ai/models/mmproj-Qwen_Qwen3.6-35B-A3B-f16.gguf
; experemental
reasoning-effort = xhigh
temperature=0.6
image-min-tokens = 1024
top-p=0.95
top-k=20
min-p = 0
ctx-size = 262144
swa-checkpoints = 48
; allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1
cache-type-k = q4_0
cache-type-v = q4_0
batch-size = 4096
ubatch-size = 512
cmoe = 1
moe-cache-mib = 9000
spec-draft-model = /home/j/ai/models/mtp-Qwen_Qwen3.6-35B-A3B-Q4_0.gguf
spec-draft-type-k = q4_0
spec-draft-type-v = q4_0
spec-type = draft-mtp
spec-draft-n-max = 2
spec-draft-p-min = 0.8
>>
>>110263741
That other report you're comparing to has an older Xeon that is not remotely comparable to your own. You have AVX2, FMA3, BMI1 and BMI2, and nearly double the PCIe lanes of the E5-2450v2 from the report which lacks all of those features.
>>
>>110263741
>I 3D printed a shroud but I sometimes still temp-throttle
80mm static pressure fans are great, mine never throttle. I'm happy llama.cpp slowly gets better, every now and then a new patch makes it in.
>>
>>110263766
You think that makes up for the 20% increase? If so, why are we buying expensive GPU's when we could just CPUmax? Hot damn.
Anyway, I'm using the expert queueing that's been supported for something like a month now, which is how I can get that big context. Shit's cache.
>>
File: file.png (264 KB, 447x498)
264 KB PNG
>>110263765
>cache-type-v = q4_0
mkay, mkay

>cache-type-k = q4_0
picrel
>>
>>110263792
>Shit's cache
heh
>>
>>110263772
92mm Artic P9 max, annoyingly loud. I should switch to a blower.
>>110263793
Q4 and better doesn't seem to make a difference with Qwen models. I guess it might with Gemma.
Have any other cope?
>>
>>110263792
AVX2 support in particular is probably the single biggest factor in this case.
>>
>>110263821
With the expert cache, we're mostly running on the card here. Sorry I don't have a downgrade CPU to test with, however this does show you can get good inference with ewaste like the P100 and ddr3 and old Xeons. I just wish I bought more ram when I bought this thing.
>>
Thinking about building a dedicated box for AI shittery. Been eyeing a Dell 5820 and maybe some Tesla T10s or something.
>>
File: file.png (652 KB, 960x753)
652 KB PNG
>>110263854
t10s suck and are absolutely not worth more than like $200. that dell looks fine. get an sxm 32gb v100 or two on one of those dual chink converter cards like pic rel. or get like a 32gb mi50/60
>>
>>110263864
I've been trying to find those dual board cards, as far as I've heard they're best sourced from Alibaba
>>
File: file.png (687 KB, 1000x1000)
687 KB PNG
>>110263882
plenty on ebay
https://www.ebay.com/itm/860094842100
>>
>>110263854
What I did, and the Turing is much better supported by like VLLM than Pascal.
>>110263864
Where do I get a T10 for $200? They're just about $400 on Ebay.
And Volta has the some of the same compatibility problems as Pascal. Although there is VLLM support.
That dual card requires pcie bifercation, you know that right? Have you run one?
It all comes down to budget, really.
>>
>>110263885
I mean, the old Dell and a Pascal card. I don't have a Turing :-(
>>
>>110263885
>Where do I get a T10 for $200? They're just about $400 on Ebay.
you dont. but they arent worth more than $200, which is the point. they are old and shitty.
>That dual card requires pcie bifercation, you know that right? Have you run one?
i have not because i dont need to use shitty gpus but my motherboard does have bifurcation
>>
>>110263895
But you recommended a card that is older and shittier. Explain.
>>
>>110263899
more vram
>>
also the v100 is faster than the t10
>>
>>110263885
>Where do I get a T10 for $200?
You don't, that's the problem. They're overvalued trash, don't waste your money on one. Would genuinely prefer a P100 for $150 to a T10, more memory bandwidth.
>>
>>110263909
Too bad. The low power draw means I could actually run it in my desktop.
>>
once normies realize that going local gives them free local tokens AND free gpt/claude sub then it's truly over for gpu price
>>
>>110263453
yikes, never using qwen again!
>>
>>110263884
I appreciate the link, and yeah trying to figure out which GPUs to go for if I pick up a 5820 has been such a headache
>>
Has anyone gotten any real speedup from --moe-cache-mib and if so what was your hardware and what model+quant? I'm starting to think this never worked.
>>
>>110263954
that 5820 should support bifurcation because it has a xeon. that dual v100 should work, and you can start with just 1 and add another 1 later
>>
Just booted up mixtral for the first time in 2 years and it mogs the shit out of gemma. It's not funny anymore when are they going to make good models again
>>
>>110263971
No, never. manual tuning of --n-cpu-moe works much better. The only advantage of --moe-cache-mib is that you don't have to tune it for your VRAM and shuffle tensors around manually if there's a vram imbalance.
>>
File: gemma_do-care-did-ask.png (1.31 MB, 1136x1152)
1.31 MB PNG
>>110263803
>maximum ~30K tokens; no 64K, 128K, or 256K testing (250K tokens total, not a single 250K-token context)
>no actual generation testing
>no agentic evaluation
>averages obscure weaknesses (long document KL reaches 0.581 in that test
>attention rotation (the test uses a turboquant-inspired rotation, so results aren't necessarily transferable to implementations without it)
>only tests on full-precision weights
I'm not definitively saying that Q4 KV is bad, but that specific graph is shit.


i'm not coping, i'm just being realistic. it's only 640 more MiB for K8V4, and 200~ish more than that for full INT8/Q8_0 cache.

if it works for you, that's great and I hope it continues working, anon. ^_^
>>
>>110263940
lmao how new are you?
y post that here?
what do you think happens to your share of the block reward when more miners join the network?
>>
>>110263984
I appreciate your advice and wisdom, I'll probably grab that Dell soon and start putting money back for the V100 stuff, maybe grab them after the new year.
>>
>>110263854
don't do this:
>>110263864
>or get like a 32gb mi50/60
t.retarded mi50 owner
>>
>>110264077
i would start making moves soon, prices will absolutely not be going down soon
>>110264083
actually dualblackwellGOD
>>
>>110264014
>Just booted up mixtral for the first time in 2 years
i did that last week. the 8x22b
it mogs the shit out of glm and m3
>>110264014
>when are they going to make good models again
never
>>
>>110264072
everyone knows about this already, it's only a matter of time
>>
Why is AI so racist
>>
>>110264122
Why the fuck is it so blurry?
>>
>>110264124
i came on your screen sorry
>>
>>110264022
It's just been flat out worse than base autofit every time I've tried it as a lazy option with a new model.
>>
>>110264124
zoomers use their phones to take pictures of screens because they don't understand the concept of screenshots
>>
>>110264022
that's just because the llama.cpp implementation of expert caching is completely retarded and loads every "missed" expert immediately onto vram before using it which causes an insane overhead, especially on big MoEs
there are other approaches like ones that do give a much better speedup like what ktransformers does or that one llama.cpp fork that only switches out experts if they were called n times before instead of causing 100GB of traffic over pcie every single reply to load one-time experts
>>
>>110264197
they changed it yesterday. I downloaded the new github source code after they merged some new caching algorithm and it was a little bit better but still just worse than --n-cpu-moe
>that one llama.cpp fork
if you're talking about the neurall fork, that's also useless and is no faster than mainline. it's also vibecoded slop that spams out 100 console messages of AI word salad for every token, lol. also all the command line options are undocumented or badly documented.
>>
>>110264122
Because AI hates jeets down to their very tongues too.
>>
>>110264122
they didn't even bother to fix the unsupported characters on their AI generated chart, fuck this shit
>>
>>110261305
Reporting for duty!
>>
>>110264197
>Make system to ease pcie bottlenecking
>Strangle pcie
For what purpose?
>>
Is llama.cpp better than kobold? I always thought the only difference was the ease of use...
>>
>>110264205
the new pr just adjusts how experts are evicted from vram if a cache miss happens. it still doesn't fix the underlying issue that llama.cpp will always load any expert it that's called into vram so every single miss causes traffic and boots something, even if it's some expert that is getting very low traffic otherwise. with the new pr that low-traffic expert is just getting evicted faster for something else. main llama.cpp's implementation of expert caching still sees RAM as simple storage and refuses to use the cpu even if it's for one-off experts that see little to no use, which causes lots of traffic over pci-e.
the fork I was talking about is: https://github.com/leloch/llama.cpp/tree/moe-cache-v2-pr which waits with swapping out experts until it's been called multiple times to avoid causing overhead for trash that doesn't need to be in vram. until then, an expert is kept in ram and just runs off cpu like usual.
>>110264219
who knows?
>>
>>110264219
>Make system to ease pcie bottlenecking
That wasn't the purpose. Weights were copied at launch to the GPU and stayed there.
>>
>using 100% local hindsight
>spent last hour consolidating memories about 1 single 200k tokens conversation
ROFL bros what the fuck? how do people use this shit
>>
>>110264267
Interesting. I'll see if qwen 27b can vibeslop a fix. I'm interested to see if that model can be good enough to do it
>>
my qwen refused a prompt stating it has guardrails and safety protocols against NSFW requests. I thought this thing was Chinese and uncensored?
>>
>>110264347
It *is* Chinese.
>>
>>110264347
>Chinese and uncensored
Chink models are always lagging behind SOTA models, but they eventually catch up. They are now in the process of catching up to the safety of the current SOTA models. Congratulations.
>>
>>110264357
>>110264353
what the fuck? Do we have a good uncensored Qwen right now? A recent one fine tuned perhaps?
>>
>>110264365
Just use gemma.
>>
File: Untitled.png (91 KB, 879x590)
91 KB PNG
>>110264365
Eh, have you not been following recent news? There's a lot of people worried about open-weights qwen4 being nerfed in the cybersecurity department for the sake of 'safety'.
All the chinese labs are going to start doing this sooner or later.
>>
>>110264382
I haven't. Very concerning.
>>110264378
that's not even censored worse?
>>
>>110264382
eventually we will get an abliterated model to scrape gemmas brain to add the weights for reverse engineering in a newer model
>>
>>110263681
i hate you and your people
>>
File: Qwen_image_2.1_00034.png (1.67 MB, 896x1184)
1.67 MB PNG
https://x.com/spectatorindex/status/2118905468311756933
More Chinese spies stealing AI secrets!
>>
>>110264384
>that's not even censored worse?
No.
>>
>>110264382
apparently changs been trying to get around snitch software built into chink phones
>>
Why did no one tell me LiquidAI’s bigger MoE is dog shit?
>>
>>110264122
The real answer is that token splitting is done based on how common said tokens show up, and LLMs are trained on a lot more English text compared to obscure indian languages
>>
>>110264384
Use uncensored Gemma, retardbro.
>>
>struggling with video game puzzle (that I feel pretty stupid for not getting in hindsight)
>wonder if AI can solve it so take a couple photos of the clues
>Gemma got it wrong after almost 2 minutes of thinking
>Argon got it right almost instantly
I can't wait for Gemma 5
>>
File: hqdefault.jpg (15 KB, 480x360)
15 KB JPG
>>110264490
Have you tried those gay ass puzzles in mobile games?
>>
>>110264542
No I don't play mobile games.
>>
>>110264037
In my comparisons over my own workloads, I see no measurable difference. And I like the larger context.
>>
>>110264267
>refuses to use the cpu even if it's for one-off experts
That's a different PR. The two were battling it out for who gets merged first.
>>
File: .png (30 KB, 1113x230)
30 KB PNG
>>110264599
the amount of bureaucracy and retardation in open source projects is off the chart, which gets worse the bigger they get. also the main dev doesn't even understand how the code works.
>>
>>110264490
How many image tokens did you use? Gemma is infamous for having shitty vision compared to her peers.
>>
File: 1786979611452194.jpg (166 KB, 1515x1115)
166 KB JPG
>>110262979
Imagine not using a customizable compaction engine.
>>
>>110264262
Kobold is just outdated llama
>>
>>110264763
>>110264763
>>110264763
>>
>>110262979
I've been using Pi this whole time. What should I use then, opencode?



[Advertise on 4chan]

Delete Post: [File Only] Style:
[Disable Mobile View / Use Desktop Site]

[Enable Mobile View / Use Mobile Site]

All trademarks and copyrights on this page are owned by their respective parties. Images uploaded are the responsibility of the Poster. Comments are owned by the Poster.