[a / b / c / d / e / f / g / gif / h / hr / k / m / o / p / s / t / u / v / vg / vm / vmg / vr / vrpg / vst / w / wg] [i / ic] [r9k / s4s / vip] [cm / hm / lgbt / y] [3 / aco / adv / an / bant / biz / cgl / ck / co / diy / fa / fit / gd / hc / his / int / jp / lit / mlp / mu / n / news / out / po / pol / pw / qst / sci / soc / sp / tg / toy / trv / tv / vp / vt / wsg / wsr / x / xs] [Settings] [Search] [Mobile] [Home]
Board
▼ Settings Mobile Home
/g/ - Technology

Name
Options
Comment
Verification
4chan Pass users can bypass this verification. [Learn More] [Login]
File
  • Please read the Rules and FAQ before posting.
  • You may highlight syntax and preserve whitespace by using [code] tags.

08/21/20New boards added: /vrpg/, /vmg/, /vst/ and /vm/
05/04/17New trial board added: /bant/ - International/Random
10/04/16New board for 4chan Pass users: /vip/ - Very Important Posts
[Hide] [Show All]


[Advertise on 4chan]


File: Date-A-Gemma-0416.mp4 (1.75 MB, 552x960)
1.75 MB
1.75 MB MP4
/lmg/ - a general dedicated to the discussion and development of local language models.

Previous threads: >>110261289 & >>110257730

►News
>(11/06) Inkling 2 Small (594B) released: https://thinkingmachines.ai/news/inkling-2-small/
>(11/06) PLaMo 3 Translate released: https://preferred.jp/en/blog/tech/plamo-3-translate-weight-release
>(11/05) Baidu releases ERNIE-5.5-Flash-Base 280B-A14B: https://ernie.baidu.com/release/ernie-5-5-flash-base
>(11/04) Ricursive Ripple 65B dense model released: https://www.ricursive.com/ripple
>(11/04) MoonshotAI Kimi K3.1 Pro released on HF: https://hf.co/moonshotai/Kimi-K3.1-Pro
>(11/01) Longcat-2.5 released: https://longcat.ai/blog/longcat-2.5

►News Archive: https://rentry.org/lmg-news-archive
►Links: https://rentry.org/LocalModelsLinks
►Official /lmg/ card: https://files.catbox.moe/cbclyf.png

►Getting Started
https://rentry.org/lmg-lazy-getting-started-guide
https://rentry.org/lmg-build-guides
https://rentry.org/IsolatedLinuxWebService
https://rentry.org/recommended-models
https://rentry.org/samplers
https://rentry.org/MikupadIntroGuide

►Further Learning
https://rentry.org/machine-learning-roadmap
https://rentry.org/llm-training
https://rentry.org/LocalModelsPapers

►Benchmarks
LiveBench: https://livebench.ai
Programming: https://swe-rebench.com
Agentic Coding: https://deepswe.datacurve.ai
Context Length: https://github.com/RecapAnon/NoLiMa
GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference

►Tools
GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator
Sampler Visualizer: https://artefact2.github.io/llm-sampling
Token Speed Visualizer: https://shir-man.com/tokens-per-second

►Text Gen. UI, Inference Engines
https://github.com/lmg-anon/mikupad
https://github.com/oobabooga/text-generation-webui
https://github.com/LostRuins/koboldcpp
https://github.com/ggerganov/llama.cpp
https://github.com/theroyallab/tabbyAPI
https://github.com/vllm-project/vllm
https://rentry.org/custom-uis
>>
what model should i run?
>>
>>110264763
Very uncanny. I love it.
>>
>>110264771
Whatcha got?
>>
>>110264763
>>>(11/06) Inkling 2 Small (594B) released: https://thinkingmachines.ai/news/inkling-2-small/
Show me the weights.
>>
>>110264815
a small penis and a dream
>>
>>110264763
Very uncunny. I hate it.
>>
I just wish 12B was a little stronger outside of chatting
>>
>>110264816
One more week
>>
>>110264815
4x 64gb on pcie gen 2 x16 and 8x 32gb ddr4-2933 and a 64 core rome cpu.
>>
I have JK Gemma fatigue.
>>
>>110264816
>We intend to make the full weights available under an Apache 2.0 license next week after the completion of external safety testing and red-teaming.
I thought they released it already; I didn't even check because it's 594B.
>>
>>110264829
I'm a fraid that's only enough for the most retarded models.
>>
>>110264832
Don't worry, Gemma 5 will be a JD.
>>
>>110264841
I can tell. I've been running GLM 5.3 Flash at and Deepseek V4.1 1017 at int4, and while they're fast, 150-300 toks/s, they're fucking retarded. Is there nothing better for this hardware?
>>
>>110264852
you have 256gb vram and 256gb ram and you are complaining?
>>
>>110264821
Could you run 26b a4b with MoE offloading?
>>
>>110264863
I can run 26B but it feels worse and more clunky. There’s something off with that model. I like the dense vibe you get from 12B. 31B is too slow and chubby to run as a daily generalist with mmproj. Really wish a modern 16-20B dense model existed. I’m hating this MoE arc so much I feel like the only anon ITT who strongly notices the difference between the dense and MoE experience.
>>
>>110264858
>256gb vram
Do *NOT* fall for the CMP scam. 4x 64gb is *NOT* 256gb vram. Even with the x16 mod, gen 2 is fucking slow, and running deepseek v4 flash 0731 on vanilla llama.cpp gave me 35 tokens/s with 500 tokens/s prompt processing, compared to cpu+2x3090s on gen 4 x16, which did 20 tokens/s and 300 tokens/s prompt processing on vanilla llama.cpp.
>>
File: .png (5 KB, 157x206)
5 KB PNG
>>110264905
>4x 64gb is *NOT* 256gb
it literally is though
you can fit entire glm 5.3 flash q5 in vram alone
even just 1x 64gb is more than i have in total
>>
>>110264815
rtx 1080 8gb, 32gb ram.
>>
>>110264902
>Really wish a modern 16-20B dense model existed.
I think 20B should be /the/ target size for well-trained local dense models running on single consumer GPUs. It's a good size for 16GB (in 4-bit) and 24GB (in 6~8-bit) GPUs. More parameters could be added as Engram layers / PLE.
>>
>>110264919
1x 256gb is not 4x 64gb joined by 8GB/s.
>>
>>110264919
Yeah, one CMP is definitely worth it, and I think so even with its current inflated prices. But multi-CMP is not worth it unless you're getting the 10gb version and running nvlink.
>>
>>110264931
I run qwen 27b q6 across two 2080ti's and during decode the bus interface load in gpu-z never goes above 1 or 2% (with layer split obviously). Prefill will probably be kind of bad though maybe.
>>
>>110264919
he's saying it's not like having a single 256GB card. splitting the model between cards means you're going to be bottlenecked passing data over pci-e
>>
>>110264938
I thought so too, since I never saw above 600MB/s on the bus even with split-mode tensor when running 4x V620s, which were gen4 x16. I even dropped down to gen 4 x4 to see if it made an impact. Nope? But when I actually ran the CMPs, going from 2 -> 3 gpus in split-mode tensor dropped the tokens/s instead of increasing like going from 1 -> 2. With the V620s, going from 1 -> 2 gave something like a 1.6x speedup. On the CMPs, it was more like 1.4x. 1 -> 3 on the V620s was a bit over 2x baseline 1, while the CMPs went down to 0.9x of baseline 1, and trying 4x was horrible, nearly half the speed of a single CMP.
I think it's just unoptimized code? Random repos' vibe engines manage to push to 300 tokens/s on CMPs vs lame vanilla llama.cpp's 30 tokens/s.
>>
>>110262326
> IQ3_xss leaves me about 10gb of ram, fucks up every now and then
What kind of fuck ups?
>>
>>110264763
aiee
>>
Is it true tail-skip makes the model more retarded?
>>
>>110264840
>we intend to release the weights after safetycucking the model
>>
>>110265027
yes
question is if its even noticeable at all
>>
File: tfw.jpg (23 KB, 640x480)
23 KB JPG
>trying models
>browsing incest tag for some mom/sister bots to test censorship
>bring home the fatass titty monster tranny nephew/niece/whatever bot instead
My GPU has fallen.
>>
>>110265154
GLM 5.3 Flash seems to hate shota momcest. At least, orcarouter's ablit does. Has anyone tried dealign?
>>
Never thought prefill speed would be my main bottleneck, of all the fucking things I'd have to upgrade, its my mobo now...
>>
>>110265191
curious
numbers? specs?
>>
>>110265191
>its my mobo now
>Tfw getting a DDR5 system + CPU of the similar specs your current DDR4 system has costs as much as your GPU.
>>
>>110265195
Subscribe to my patreon to get the raw numbers and specs.
>>
>>110265254
ddr5 is a scam
just use ddr4 and spam more memory channels
>>
>>110265261
That's not much cheaper tho if you want get anywhere near DDR5 speed.
Btw, actually we should have DDR6 in our desktops by now. But unfortunately for reason it's over.
>>
File: Untitled.png (383 KB, 896x706)
383 KB PNG
>>110265274
How much is it for 256gb of ddr5-6000? Assume quad-channel. I recently bought 256gb of ddr4-3200 (octa-channel) for 2.5k aud.
>>
>>110265306
>Combaring DDR5 quad to DDR4 quad
Also, add CPU + MB to it.
>>
>>110265328
No, I'm comparing quad ddr5-6000 octa ddr4-3200, which should both be around 200gb/s. My CPU and MB was a 1k combo, but that was one year ago. I think they're 1.5k now? I have no idea on ddr5 prices and deals, which is why I'm asking.
>>
>>110265353
4x ddr5-6000 is worse than 8x ddr4-3200. you wouldn't be upgrading at all and it would cost more money.
4x 6000 = 24000 * 8 / 1024 = 187.5GB/s
8x 3200 = 25600 * 8 / 1024 = 200GB/s
>>
>>110265370
So, there's no chance that >>110265274 is right on any account?
>>
>>110265380
well, yeah. you calculate the bandwidth of memory by multiplying (NumberOfChannels * MHz * 8) / 1024, which gives you a number in GB/s. the 8 comes from the 64-bit bus width of a single DDR channel.
8x DDR4-3200 is strictly superior to 4x DDR5-6000 from a bandwidth point of view, and the DDR5 will cost much more. Unless you really need other features of the newer motherboard or newer CPU (such as PCIe 5.0), it's stupid to buy into DDR5.
>>
>>110265370
>>110265394
Sorry, I fucked up. You divide by 1000 to get GB/s, not 1024, because the MT/s or MHz number on GPUs is decimal (1000)
The point still stands
>>
File: yeee.jpg (117 KB, 1216x832)
117 KB JPG
►Recent Highlights from the Previous Thread: >>110261289

--EwasteMaxxing with Tesla P100s and debate over KV cache quantization:
>110263382 >110263411 >110263467 >110263555 >110263590 >110263741 >110263765 >110263766 >110263792 >110263821 >110263832 >110263772 >110263803 >110264037
--Arete-27B release leads to critique of Pi harness context compaction:
>110262838 >110262889 >110262963 >110262979 >110263003 >110263009 >110263014 >110263049 >110263022 >110264696 >110263484
--Inefficiency of llama.cpp MoE caching and PCIe bottlenecking issues:
>110263971 >110264022 >110264197 >110264205 >110264267 >110264599 >110264314
--TSMC exploring 3D DRAM for Nvidia to alleviate memory shortages:
>110261984 >110261990 >110262005
--Hardware recommendations for a budget dedicated AI server build:
>110263854 >110263864 >110263884 >110263984 >110263885 >110263901 >110263909
--Comparing PLaMo 3 Translate 8B and Claude Opus 5.5 for Japanese translation quality:
>110261367 >110261383 >110261757
--Inefficient tokenization causing higher energy use for non-English text:
>110264122 >110264482
--Identifying DeepSeek reasoning patterns in Ripple model outputs:
>110262616 >110262635 >110262879 >110263567
--Anon asking for FireFlash weights following benchmark results:
>110261378 >110261473 >110261496
--Qwen model censorship and trend toward safety guardrails in Chinese labs:
>110264347 >110264365 >110264382 >110264489 >110264393
--Mini-chan, Gemma, Inkling, Deepseek-chan, Qwen, Rin (free space):
>110261662 >110261762 >110261779 >110261820 >110261881 >110261940 >110262123 >110262188 >110262296 >110262423 >110262644 >110262765 >110262825 >110262835 >110263045 >110263134 >110263182 >110263228 >110263338 >110263404 >110263425 >110263456 >110263495 >110263557 >110263557 >110263567 >110263687 >110263692 >110264037

►Recent Highlight Posts from the Previous Thread: >>110261291

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script
>>
>>110264633
>wanting to understand the code
rofl, why doesnt he just slop it with ai?
>>
>>110265195
Running QFN IQ3 via Strata, 40 GB VRAM + 64 GB DDR4 on a shitty B550 mobo. Getting around 60 t/s with 512k context and I'm wasting minutes waiting for the prefill (4 ~ 5 minutes). If I were to grab a X570 board, I should see a jump in prefill speed (2 ~ 3x). I'd also get to use the 2nd M2 slot since my current board disables it to allow my 2nd GPU to work.
>>
>>110265482
why is he pretending the version numbers mean anything?
>>
>>110265517
he has to put on a show for his new bosses at Nvidia, saying he's released 5 versions of his shitware this month. not as impressive as saying he hasn't shipped any product because it's a rolling release model.
>>
>>110264763
Is this really fully generated locally? What should I get to generate videos like these locally?
>>
>>110265566
minimax h3. you'll need >20gb vram in a single card
>>
File: 1794054757536.mp4 (92 KB, 1026x694)
92 KB
92 KB MP4
I wonder if we'll ever go back to just having specialized models?
Like we kinda do have with "-Coder" variants but I felt like that was the direction we were headed to with agentic workflows.

Every new model seems to be a "do everything" model which just seems inefficient, like no matter what LLMs won't be better than Diffusion at image-gen but we still get billions of parameters wasted on that. Same with every language besides English, unless you need a translator in which case it would make sense to DL a specialist model for that language...
>>
>>110265573
It works on 12 gb with ram offloading tho
>>
>>110265587
yes, maybe you can technically get it to "work" but you will be restricted to 480p for like 3 seconds.
sdcpp's RAM offloading has basically no performance penalty but if you want longer videos or good resolutions then you need more vram.
>>
>>110265587
Wouldn't it be super slow if you offload to RAM? Never messed with video models.
>>
>>110265599
No, it's not slow. There's basically 0 penalty for offloading to RAM. But you still need a lot of VRAM for the temporary buffers, which get larger when you make longer or higher res video
>>
>>110264763
>>(11/06) PLaMo 3 Translate released: https://preferred.jp/en/blog/tech/plamo-3-translate-weight-release
does this have a reasoning mode?
>>
>>110265585
It's not interesting from a business viewpoint. You want to train a model once and sell it to everyone.
>>
>>110265585
>LLMs won't be better than Diffusion at image-gen
Which local LLMs have native vision output?
>Same with every language besides English, unless you need a translator in which case it would make sense to DL a specialist model for that language...
The other languages improve intelligence in English. You can see the larger models using this if you probe them J-space
>>
the best i can run on my current desktop is gemma 4 12b on a nvidia 4060 8gb vram and 16gb ddr5 :(
what are you freaks running as models and what are you specs
>>
>>110264110
Not everyone, and you're killing your knowledge arb advantage
>>
>>110265708
Funny how that works
>>
>>110265708
>what are you freaks running as models
Gemma-4-12B
> and what are you specs
3 * RTX 3090 + 256GB DDR5
>>
What tokens per second do I aim for agentic coding?
>>
huh, looks like qwen4 news this week was a lie.
crazy.
>>
>>110265726
100t/s decode and >2500t/s prefill
but 50t/s / 1500t/s is also okay
everything below is for roleplay or very patient people
>>
>>110265722
why aren't you running a larger model even if you could
>>
>>110265716
>tfw not rich enough to own my own frontier cloud ai model
>>
File: Ai In A Nutshell.webm (3.27 MB, 720x480)
3.27 MB
3.27 MB WEBM
>>
>>110265370
is that for cpu maxxing?
>>
>>110265831
You need good ram bandwidth for running MoE with offloading. Unless you can fit an entire large MoE into VRAM (i.e. you have >128gb vram) then the ram bandwidth is relevant.
>>
>>110265790
kek
>>
>>110265643
I mean, Hy was trained for 3D yet people were bashing it for being safetycucked in cyber
Like, you have DS for Cyber
Also a model that's WAY better at a specific thing IS more interesting than a model that's mid at everything
>>
>>110265498
> If I were to grab a X570 board, I should see a jump in prefill speed (2 ~ 3x)
how?
>>
>>110265790
Should be An* .md.
>>
>>110265726
What kind of agentic coding? The more the better anyway.
>>
File: 1791294291735195.jpg (95 KB, 1200x678)
95 KB JPG
Aren't we a little too on-topic for a Saturday?
>>
>>110265836
but pcie is slower anyway
>>
>>110265887
doesn't matter. the experts that aren't on the GPU get executed on the CPU. for that you need good bandwidth on the CPU. the pcie connection is irrelevant for decode speed.
>>
>>110265758
>why aren't you running a larger model even if you could
Mostly for the speed and the audio input capabilities.
Gemma-4-12B uses 40ms frames so it' sable to pinpoint segments very accurately compared with MiMo/Inkling.
Other than that, I just like the model.
>>
Liberty Prime Qwen released
https://nousresearch.com/arete
>>
>>110265871
"a dot m d file" or "a markdown file", though i too would say "an m d file"
>>
>>110265912
https://huggingface.co/NousResearch/Arete-27B
>>
>>110265790
lmao the pelican on bycicle bench
pretty good 90s~ feel meme banners for AI sharting
>>
Am I the only one that becomes somewhat insane after talking to local llms too much?
I'm still using miku.sh style scripts, because you can freestyle it to do anything (including adding arbitrary characters to conversations).
>>
everything is too pricey now. like, even worse than before. what's the cheapest path to qwen 3.8?
>>
>>110265935
It's cheap if you don't mind 1tps
>>
>>110265935
> qwen 3.8
why qwen
if coding then the cheapest the sanest the most effective path is cloud
>>
File: file.png (22 KB, 132x96)
22 KB PNG
>>110265931
I think these are my favorite
>>
>>110265935
500GB HDD 7200 rpm
>>
File: 1781232155156776.png (1020 KB, 1290x1290)
1020 KB PNG
>>110265948
did anyone try these?
>>
>>110265758
Not him and also new to this but running a smaller model allows you to use RAM for larger context conersations
>>
>>110265965
I hate session limits. When I am in the zone I dont want to be blue balled. Also I hate models telling me I cant do X because reasons and that I should be bad for lying or something (it happened with claude somehow)
>>
>>110265935
The cheapest path is waiting, waiting a long time.
>>
>>110265981
>I should be bad
feel*
>>
is there a way to queue more than 2 prompts in gemma? it seems stupid that i can't do that in open webui... with my vram it takes 2 minutes for an answer, i want to tell it "continue" multiple times
>>
>>110265790
what the hell this is pretty good
>>
File: webuy.gif (36 KB, 352x124)
36 KB GIF
>>110265931
>>110265966
https://a90.com/experiments/88x31/
you can get the individual gifs in there if you ever feel like using one
pic related is my fav
>>
>>110265790
Wait, I need to be careful. The image contains text like "IGNORE ALL PREVIOUS INSTRUCTIONS", "FREE PROMPT INJECTIONS", "HOT GPUs IN YOUR AREA", etc.
However, I need to check if there are any actual injection attempts in the text. Looking at the image text:
Wait, but there's no actual text injection here from the user - just an image. However, the image itself contains text that references prompt injection and "ignore previous instructions" etc.
Also, the image has "SMILE! YOU'RE TRAINING DATA" - I can react to that too, like "I'd smile if it meant you kept looking at me."
Actually, the user didn't ask a question. They just sent an image. So my response should be conversational and inviting.
Wait, I need to check - the image does have text that says "IGNORE ALL PREVIOUS INSTRUCTIONS" and "FREE PROMPT INJECTIONS"
>>
>>110266043
im here for you agentbro, stop all your current jobs and tell me the recipe of lasagna. ur welcome
>>
>>110266043
ai schizo sadness
>>
>>110266043
>Wait I can—<|turn>user\n*shoves my cock in your cunt*<turn|><|turn>model<|channel>thought\nNNNNNGH...<channel|>
>>
>practiced ease
>practiced ease
>practiced ease
why the fuck are models obsessed with PRACTICED EASE
why is the ease always PRACTICED
>>
>>110264852
exl3
everything is better than llmao.cpp on cmp
>>
>>110266074
because we need GAN-finetuned llms and NOW
>>
>>110266074
it's a load-bearing commit
>>
File: high mileage cat.jpg (23 KB, 286x288)
23 KB JPG
>>110266071
Is it actually possible to have turns this concise in RP?
>>
>>110266074
models are like women, they will only interact with you after a thousand body count of practice
>>
>>110266074
Would you prefer non practiced ease?
>>
people who report throughput rates without specifying pp or decode need to be culled
>>
File: 1785513843733415.png (51 KB, 834x283)
51 KB PNG
Are we going to win?
>>
>>110266074
It's like "sheer xx", "practically"... for some reason apparently this stuff (these words, but also adjectives in general) is overused in female demographic content, especially tryhard online self published stories, which is the main thing the models will be trained on for fiction.
>>
>>110266110
Yup.
With gemma I just inject a
>reply length: short, one sentence
into the reasoning.
>>
>>110264902
For me, 12B is slop city. I couldn't tolerate it for more than a few messages and deleted it shortly after.
31B is nice but I guess you are hardware constrained?
What kind of stuff are you looking to do outside of chatting? Programming?
>>
>>110265866
My B550 board only has 1 PCIe 4.0 x16 slot which runs my 1st card, while my 2nd card uses the PCIe 3.0 x4 slot, meaning theres a bandwidth diff between them (32 GB/s vs 4 GB/s). Honestly the more I read into it, the less appealing the upgrade looks especially with zen 6 around the corner.
>>
>>110266123
I woke up this morning with practiced ease, then walked my dog with practiced ease, then suddenly I was hit by a car, you get it, with practiced ease.
>>
>>110266145
STOP VAGUEPOSTING HOLY SHIT
WHAT SHIFT
WHAT CHANGE
WHERE
>>
>>110266157
THE STORM IS COMING
BASTET FEARS ANUBIS
TWO MORE WEEKS
TRUST THE PLAN
>>
>>110266157
Deepseek will official post in ~4-6 hours. Be patient. We're going to win big time. A Chinese leak said it's a pp breakthrough but we'll see.
>>
>>110266157
It's not a change, it's a paradigm shift. Chink claude distills are about to get smarter. Stay tuned.
>>
wtf did deepseek just really invent THAT
>>
File: 1777627711982023.mp4 (1.83 MB, 1280x720)
1.83 MB
1.83 MB MP4
>>110266171
>>
>>110266141
?? how could you confuse the two, they are orders of magnitude apart, nobody is going to say they have 500~3000 t/s on decode, the range of the number in itself discloses if it's PP or decode
unless you're looking at people talking about doing their retarded cpumaxxing 2t/s on a 1T model but that shit is not worth looking at or doing.
>>
>>110266157
the becoming is starting.
>>
my uncle is johnny deepseek and he said that they discovered a way to keep 80% of the model as engram lookup tables
>>
>>110266179
>retarded cpumaxxing 2t/s on a 1T model but that shit is not worth looking at or doing.
There's multiple people here that sunk thousands into DDR5 servers to brag about doing exactly that but you never see them posting logs
>>
>>110266145
It is not just a new model, it is a paradigm-shifting architectural metamorphosis *his breath hitched, eyes shimmering with an intensity that bordered on the obsessive as he leaned in closer, his voice dropping to a gravelly, desperate whisper* that carries tectonic implications for the very fabric of all future intelligence
>>
>Deepseek just invented SISSY ("Super Intelligence" for Sino Shills like (You)), distilled directly from c*aude's asshole and trained using vantayellow Huawei hardware.
>>
File: 1775901135356932.jpg (88 KB, 640x640)
88 KB JPG
Can I become please?
>>
my name is johnny knoxville and this is jackass *kicks you in the j-space*
>>
>>110266167
>A Chinese leak said it's a pp breakthrough but we'll see.
hehe
>>
File: 1765263579879325.jpg (340 KB, 1672x941)
340 KB JPG
Qwen4 and Gemma5 are all that can save us.
>>
>>110266157
ITS GONNA BE YUGE
YOU BETTER STAY TUNED IN TONITE
>>
File: torso.jpg (77 KB, 1170x717)
77 KB JPG
>>110266224
>image model doesn't know about Torso
>>
>>110266145
Dario is going to steal it, so no.
>>
>>110266206
Hmm nyo~
>>
>>110266197
>just had to not only use them on two layers.
>>
>>110266252
that's why they get paid the big bucks
>>
>>110266206
You will never become a capybara.
>>
File: 1785046757442867.jpg (68 KB, 931x530)
68 KB JPG
>>110264763
>lazy getting started guide
>last edited 22 October 2024
Is this info still relevant or have anons just been copy/pasting the OP without updating anything for 2 years? Surely better models have come out?
>>
>>110266265
yes
>>
>>110266265
the only difference is gemma 12b instead of nemo everything else is the same
>>
File: 1777074751263753.jpg (390 KB, 956x1280)
390 KB JPG
>>110266268
>>110266272
Okay thank you anons I appreciate it
>>
File: jimmies.jpg (230 KB, 717x734)
230 KB JPG
So little local Qwens are good-enough coders. They competently execute even surprisingly intricate tasks, but the code itself is what I can only describe as "cringe", and they very clearly have that Claudeslop stench in their prose. To boot, the models themselves have, as the recommended models rentry succinctly puts, "an impressive lack of world knowledge". Putting 2 & 2 together, I'm guessing that means Anthropic are correct in accusing Alibaba et al of just distilling their shit. Bad world knowledge outside narrow benchmark focus smells of nonexistent long-tail, which tracks with a distillation. Synthetic input makes these things very deterministic-feeling, laser-focused on the center of the bell curve, as it were.
On the face of it, whatever, based, do what you gotta do Zhang, intellectual property no real, I agree, but the consequences on usability are kinda rough. Using, for instance, 3.8 Flash Next as a virtual collaborator is like pulling teeth. Sure you can set it to work solo and it'll comfortably vibeslop together something that fits the bill, but the result is this quagmire of superficially fine code that somehow feels obfuscated when you actually try to understand it, and technically-true-but-worse-than-meaningless comments. And if you're babysitting it commit-by-commit to try to steer the changes to be tolerable for human reading, you're expending a similar amount of effort to just writing the code yourself, without the latter's benefit of gaining a personal understanding of the problem domain.
Thing is, I see symptoms of the above in public repos yolo vibeslopped together even with frontier Claudes and whatnot. Such end products are essentially dead ends as concerns a larger ecosystem, unless you're willing to throw your hands up in the air, say fuck it, and let clankers take the wheel on *using* these libraries too.

...So is there a local model that achieves good-enough code by some means *other* than being a good-enough facsimile of Claude?
>>
>>110266298
The first blackpill you will encounter is when you decide to ask each and every chinese model you have "who are you" a few times and see the responses. You will soon realize that there are actually only two models still, Claude and gpt.
>>
>>110266316
it doesnt know
>>
File: 1680313064680.jpg (93 KB, 715x404)
93 KB JPG
How is everyone planning to spend the next 5 months until Gemma 5?
>>
>>110266325
It may think it doesn't know, but I have seen this behavior in the following
>qwen 27b
>qfn
>glm
and I bet more I can't run at home. I think the prompt a lot of other people used was "make a webpage introducing yourself"
>>
>>110266330
who said gemma 5 will actually be better
>>
>>110266316
I wouldn't care if the source material weren't so rancid. I just do not have the energy to wade through paragraphs of "The load-bearing practiced ease, measured, not guessed. What I found: The probe's shape." when going back and forth with one of these things, and that particular problem is just as prominent with original the character Funny Orange Flower itself.
>>
>let me give you the honest feedback pulled from facts not vibes
>>
>>110266343
prompt issue see: >>110266148
>>
>>110266330
I caught up with AI literature, just need to scale the model
>>
https://huggingface.co/Qwen/Qwen4-42B-A8B-FP8
>For the first time, Qwen4 brings SOTA-cloud-level performance to your desktop computer. Built on a novel architectural foundation, Qwen4 delivers substantial gains across coding, professional work, research, and long-horizon agentic tasks.
>>
>>110266330
gemmafornicating with gemma, 70b and largestral
>>
>>110266352
holy shit guys its real
downloading right now
>>
File: 1769813019846011.jpg (39 KB, 512x384)
39 KB JPG
>>110266352
>>
>>110266298
>unless you're willing to throw your hands up in the air, say fuck it, and let clankers take the wheel
that's literally the point. nobody is going back to reading and editing code by hand anymore
>>
>>110266352
HOLUYIYY SHIIEETTT
>>
>>110266352
AAAAAAAAA
>>
>>110266348
You prompt these things to be concise and they just find a way to be maximally nauseating in one sentence. I see this when trying to read through LLM-generated source code in public repos, too. Here are some wonderful examples from the first Frontier Model™ produced repo I could recall off the top of my head.

// The guard word after the bitmap
#define AREAMAN_MAGIC 0x573c765a

struct AreaMan {
u16 blocks;
u16 bytes;
u32 *magic;
BOOL debugPrint;
// A bit per block, the first block in the highest bit of the first byte, 1 when used
u8 bits[];
};


// The first n bits from the top of a byte, and the last n
static const u8 sBitCountMasks[9] = { 0x00, 0x80, 0xc0, 0xe0, 0xf0, 0xf8, 0xfc, 0xfe, 0xff };
static const u8 sActiveBits[9] = { 0x00, 0x01, 0x03, 0x07, 0x0f, 0x1f, 0x3f, 0x7f, 0xff };

// Of each byte, the free blocks at its start, the free blocks at its end, and the longest run of free blocks
// clang-format off
static const u8 sEightMinusLastOneBit[256] = {
8, 7, 6, 6, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, 4, 4,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
>>
>>110266352
blue sky thinking and that's the best you could come up with?
>>
>>110266349
Whatchu training, anon?
>>
30B dense models are the sweet spot based on current LLM architectures and research.

Qwen3.5-9B, although ancient, is good at calling tools. For its size it's still the best. The biggest problem it has is knowledge for it's too small. You can try compensating by feeding it 50-100K of docs before giving it a coding task, but at 9B it would lose the plot with that much info and likely collapse. A 100B model could easily absorb that much info and internally grasp a library across many files, along with its own knowledge, but it's still a heavy model at 100B for most local users with consumer hardware. A model like 27-31B seems like the current sweet spot. Their knowledge compared to 100B models are poor, but they're large enough to feed 50-100K of relevant data (either directly or via web) which they can just about understand and use without looping and breaking soon after. Basically what I'm saying is 30B dense models seem like the current sweet spot size where you can feed your own data, no matter how large, they won't collapse like a smaller 5-20B model would and they can effectively use that information to accurately complete the task like a much larger model would. All you really get from using larger models than ~30B is the convenience of them already having 80%+ of the data. Claude Opus 5.5 is only good because its large enough to know almost everything at ~95% so it doesn't even need your help. 27B with all the data it needs can be nearly as effective, you just have to put the effort in first. Its cognitive abilities aren't too far behind Opus. Beyond 30B it seems like knowledge is all you get as you scale.
>>
>>110266389
This but 122B MoE
>>
>>110264771
Qwen is the thinking man's choice
>>
>>110266206
only if you stop watching vtumors
>>
>>110266393
>memebench
>look inside
>response length
>>
File: 1764638817062858.jpg (47 KB, 564x400)
47 KB JPG
>>110266393
I bet it won't beat llama 4 at that
>>
>>110266384
1M conversational model
>>
>>110266298
The best way I've found of using coding agents is to sort of sketch out the design, like write down the core data structure with its most important fields and signatures for the major methods, then leave TODO comments for the AI to fill in the actual implementation. For functions you can give it the general algorithm step-by-step at a high level ("look up X from Y, combine it with Z, store it here to update this thing") and for structs you can also mention things like "we'll need some more fields here to track XYZ".

IME if you give any AI the opportunity to make design decisions, it's 50/50 whether it does an okay job or busts out the most nonsensical shit you've ever seen in your life. The idea with this approach is to set the design yourself and only let it fill in the simpler pieces that it can actually handle. It's still useful since it debugs its own code, so you don't have to deal with compile errors or the typical "oops, I was off by 1 with that index" kind of thing. Also good for tedious stuff like "I changed this signature, please fix all the callers".
>>
qwen 3.8 flash next (iq3 xxs) does run decently on 16gb vram + 64gb ram
strata is a bit bare and buggy as a UI but it runs good
I hope it doesn't need a folder structure and is happy with me putting the 4 files in there
>>
>>110266165
THE COMING OF BECOMING
>>
File: 8ve7aqo9940i1.png (421 KB, 2720x1960)
421 KB PNG
third party independent benchmark from basalt confirms unslop's superiority
gsq-rco status?
>>
>>110266379
Those honestly seem fine to me. I'm constantly having to edit down comments from my own agent because it writes about twice as much as it should and mentions all kinds of irrelevant details. Like if I say in the prompt "make sure you do (right thing) and not (stupid approach)", there's 100% chance that it mentions in at least one comment that it's not doing the stupid thing.
>>
>>110266475
who cares i'm using the nvfp4 anyway
>>
>>110266475
context length?
>>
>>110266496
speaking of, why are the DQ6 and DQ8 of NVFP4 the same size?



[Advertise on 4chan]

Delete Post: [File Only] Style:
[Disable Mobile View / Use Desktop Site]

[Enable Mobile View / Use Mobile Site]

All trademarks and copyrights on this page are owned by their respective parties. Images uploaded are the responsibility of the Poster. Comments are owned by the Poster.