File: yukkuri_shiteitte_ne.png (2.96 MB, 2320x1160)

2.96 MB PNG

►Recent Highlights from the Previous Thread: >>109125882

--Paper: Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts:
>109126634 >109126907
--Hardware and model recommendations for running Gemma 4 locally:
>109130676 >109130712 >109130782 >109130988 >109131167 >109131177 >109131410 >109131178 >109131417 >109131660 >109131719 >109131730 >109131753 >109131337 >109131634 >109130845
--Orb Anon discussing ablating purple prose and euphemisms from Gemma 4:
>109130717 >109130787
--Comparing performance and quality across different quantization levels for Qwen:
>109127320 >109127504 >109127521 >109127577
--Discussing batching and parallel request configuration in llama.cpp:
>109131468 >109131481 >109131529 >109131569 >109131616 >109131631
--Performance of MoE models split between VRAM and system RAM:
>109126062 >109126072 >109126085 >109126094 >109126106
--Comparing Gemma 4 QAT quants to standard quantization levels:
>109127682 >109127695 >109127785
--Comparing M5 Max Macbook value against other 128GB hardware options:
>109126466 >109128921
--Discussing throughput discrepancies between GLM 5.2 FP8 providers:
>109130585 >109130625 >109130713
--Discussing GLM 5.2 as a GM and prompt formatting for adherence:
>109127793 >109127817 >109127839 >109127876 >109128025 >109128043 >109128076 >109127827
--Anthropic alleges Alibaba and others illicitly extracted Claude capabilities:
>109129130 >109129149
--Anons discussing model backups and optimal Hugging Face download methods:
>109125927 >109125939 >109125958 >109126093 >109126129 >109126182 >109126188 >109131112 >109131125 >109126055 >109126109 >109126269 >109126287 >109126470 >109127461 >109127483 >109127489 >109127518 >109127530
--Logs:
>109126862 >109130151 >109131112 >109131125
--Teto (free space):
>109130054

►Recent Highlight Posts from the Previous Thread: >>109125884

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

lalalala

File: 1762427193883906.jpg (359 KB, 1536x1536)

359 KB JPG

Anthropic is relocating to France and merging with Mistral.
This has been confirmed by /lmg/.
Do NOT ask me for a source.

70b dense

next continuous vector prediction

gemma 4 13T fp512 abliterated derestricted upscaled mythos opus hocus pocus heretic unslop davidau thedrummer collaboration work

Half of the talent that worked on Gemini 3.1 preview are moving to Anthropic. I genuinely feel like how I felt when Google started dominating and Yahoo, AltaVista and AskJeeves disappeared in the background forever.

>>109132672
gguf?

>>109132674
We (mostly) only need Gemma, anyway. It would actually be great if they discontinued Gemini and just worked on open-weight models, from edge to datacenter-level parameter sizes.

File: ob.png (4 KB, 434x38)

4 KB PNG

>>109132672
I've been trying to find a good model for my use case for so long and I think I finally found the one.

>>109132674
Bad luck, for anthropic.

>>109132690
Sorry, transformers only

>>109132700
If the file isn't 0 bytes I'm going to be disappointed.

>>109132674
Gemini sucked so that doesn’t look good for Anslopic. Don’t give me any of that godfather of AI B.S either because all of those guys are just shills and snakes these days like Karpathy, enjoying the temporary spotlight shining on them because they know this tech isn’t going anywhere and no one will give a fuck about them soon

>>109132716
>this tech isn't going anywhere
Fable 5 is literally doing my entire fucking job as I type this.

>>109132624
Misanthropic will save local.

>>109132731
What do you do?

>>109132759
Nothing anything, he just said this.

>>109132759
Professional AI babysitter, like everyone else with a white collar job in 2026.

What's /lmg/'s preferred workflow for coding? Small fast model plans and implements then big model checks? Big model plans, small model implements?
Local might not be as smart as cloud, but with the different speculative decoding approaches, it can get some pretty high throughput, and you can use local models to write the code while the big cloud models just check the work afterwards for example, basically using the local model to save a buck.

>>109132624
I'm seeing them all and man how fucking many migus

>>109132781
Big model plans, small model implements. You need the big model to keep the small one on track and the implementation is the most token heavy so that's where the savings comes in. The opposite just gives you a retarded plan that the big model will have to ignore anyway.

File: ablated-e4b.png (138 KB, 1140x669)

138 KB PNG

Ablated the purple prose from Gemma 4 E4B, it writes super terse now.
1/2

>>109132781
If you’re coding something that needs more than 27B/31B, you’re relying too much on AI and dumbing yourself down. They’re both already overkill and too good for what I allow myself to use them for. I enjoy learning and am not going to cuck myself with AI. I sometimes use openrouter to code review with a superior model but never to write code.

File: base-e4b.png (165 KB, 1143x754)

165 KB PNG

>>109132842
This is the base model.
2/2

>>109132846
>you’re relying too much on AI and dumbing yourself down
Writing boilerplate and repetitive refactors SUCKS

>>109132846
The purpose of coding is to have software that does what you want.

>>109132828
That's what I do too, with the big model checking the result afterwards with tests and shit.

>>109132828
I plan (biggest model on earth) and let fable implement (small model)

>>109132705
I'm uploading a mistral.rs f69 now.
HELP! MY HUGGINGFACE STORAGE IS ALMOST FULL! (patreon link)

>>109132874
If you tell 27B how you want it to refactor something, that’s fine because you thought of it. If you ASK 27B how to refactor, you’re being a retard. If you’re asking something that requires intelligence beyond either 27B or 31B then there’s no saving you. Fucking read and learn.

>>109132908
>biggest model on earth
Yeah but how's the benchmarks on that? It might be retarded

>>109132716
>Gemini sucked so that doesn’t look good for Anslopic.
Gemini can analyze audio, Anslopic have nothing that can do that.

I will keep using my 96gb of DDR4 ram until 2040

2x128gb ram kits when?

What do you use AI for besides coding, RP, image generation, and audio generation?

>>109132967
as a member of the permanent underclass your best chance of ever getting your hands on that much memory ever again is picking it out of the post-apocalyptic ruins of a datacenter

>>109132988
datacenters will be guarded by an army of autonomous drones

>>109132967
check back in 2031

>>109132566
>Qwen-AgentWorld-35B-A3B
>35B with 3B activated
any anons running benchmarks on this? should I jump to this little fella instead of my DENSE 27b? or the good ole 3.5-35b with 10b active?

>>109132566
llama.cpp is an unserious project that can merge broken PR like the original state of Gemma MTP support quickly but won't merge a couple LoC fix to make that same PR work properly on CUDA:
https://github.com/ggml-org/llama.cpp/pull/24526
They can merge any random pr from people known to introduce trillion of bugs like pwilkin but actually fixing shit is so hard.
The worst thing is, llama.cpp isn't even the worst backend out there, local is just too fucking cooked with terrible implementations.

File: 1760435374943707.jpg (53 KB, 429x346)

53 KB JPG

>>109132846
You fail to understand that I'm not a programmer, and I don't care about becoming proficient at telling a machine what to do. I'll keep using the highest B model I can, just like you wouldn't go to the average surgeon if you could go to the best surgeon in the country.

>>109132969
Translating korean mmo update change logs a year before they patch the american server so I know what's coming in the future meta/market wise, local models do a better job translating than google translate in browser for a couple years now. I also use them to ocr since I have it running anyway. Sometimes I also quiz it on how to do off label tasks on linux that give poor results when trying to web search for an answer, but you may consider that to fall under coding.

>>109133121
>I'm not a programmer, and I don't care about becoming proficient at telling a machine what to do. I'll keep using the highest B model I can
as a programmer using SOTA API models, the amount of shit I've had to put into my agentic instruction set to prevent them from making the wrong assumptions and harebrained shit makes my head spin at the thought of a non programmer using them to produce code while they're unable to describe the big picture in an eloquent enough manner, because it really does matter a fucking lot how you describe your requirements, high level architecture, allowed data structures, how you bounce ideas back and forth etc in the resulting code quality. Not even Claude is at a level where it can produce good code without some serious effort put into telling it the how and why of everything.

>>109132969
Translating Chinese and Japanese web novels.

>>109133144
You're severely autistic and you'd be a pain to work with regardless of whether it's a LLM or a real human being. Also, it's not a matter of correctness it either works or it doesn't. No one cares about your autistic best practices if you're ten times more productive and nothing obvious broke. I'd get that stance if you're working on critical infrastructure like nuclear or something like that, but most of you are in basic SaaS or God forbid, webshit.

>>109133096
>llama.cpp ... merge[d] ... the original ... Gemma MTP support quickly
laughing out loud

>>109133177
kys you're the reason why everything is broken, bloated, and unmaintainable

>>109133190
You get what you paid for, no one will reward you for going slow dipshit. Get real.

>>109133177
nocoder opinions are to be discarded. PROOMPTING does not make you an expert in how to write maintainable code in the long term, even if something works immediately the bad design will bring forth the fountain of regrets sooner or later.

I decided to give glm4.7 a try after seeing it mentioned so many times and now I remember why I stopped using 4.6. It's slower than fucking R1. I only get 4.7 tps on the newest llamacpp. Aside from trying a non-i quant, are there any improvements I could get with launch flags alone?
>--parallel 1 --threads 8 -cmoe
24gb of vram

>>109133186
(relatively) quickly by their standards.

why do you guys even like gemma? its insanely cucked and refuses anything explicit

>>109133200
>You get what you paid for,
你好

File: 1779204915832268.jpg (54 KB, 535x462)

54 KB JPG

>>109133204
Yeah because before proompting all codebases were well maintained without any technical debt

>>109133213
We're on day 0 gemma, sorry you missed it.

>>109133096
It's the case of any software in a post AI world. Forget about quality.

>>109133218
Because even before LLMs, tech companies and project managers tried to cut corners with jeets and bootcampers like you that had no business writing code at all.

>>109133232
So you admit you're not in charge? Why would your opinion matter then?

>>109133223
share?

>>109133096
Share your backend anon

>>109133121
>managed to forgot that

File: 1481398046182.gif (1.1 MB, 205x223)

1.1 MB GIF

>>109133177
You should take a fucking seat. There are a million cases of non-programmers like you creating broken software. Infinite cases of executives or salespeople in corporate environments coding their own stuff and breaking products.

An LLM is a tool, like a hammer. And for you, everything looks like a nail. You could choose not to be an arrogant piece of shit. But you decided that having this retarded attitude was the right thing to do. So you will just keep creating broken shit, and your software will never go beyond a hobbyist's reach because you have no idea what you're building, and because of that, you only have a very limited and retarded way of talking to the LLM.

To make it very clear to your childlike mind: you will talk to the LLM like a 3-year-old child talking to their parents. Your vocabulary is limited. Your understanding is limited. And the parent needs to talk to you in a very simple and unsophisticated way; otherwise, you won't understand.
Because you're mediocre. The results you will get are going to be mediocre.
Sorry, kiddo. At least this is an anonymous board, so you can try next time without acting like a mongrel.

>>109133253
blue board you pervert

>>109133253
you gotta go b*ck to r*dd*t.

>>109133208
It's slow even on my 5090. The answer is probably just stacking more VRAM.

>>109133265
poetry

>>109133253
lewd

>>109133204
Yea but by the time anything breaks we'll have AGI and it'll fix everything.

>>109133265
nocoder in shambles

>>109133265
Which model did you use to generate this? The semicolon makes me think it was Gemma.

>>109133265
Let it out anon, here you can at least pretend that someone cares about your opinion, unlike the real world. Surely your employer, your clients and anything in between deeply care about your coding practices lol

>>109133300
Everything is already broken now.

File: images (29).jpg (28 KB, 376x531)

28 KB JPG

>>109132674
Does Grok still have a chance or is my waifu dead?

>>109133305
>The semicolon makes me think it was Gemma
the industry is an ouroboros feeding on each other's input, and like a lot of slop, this started with GPT when OpenAI attempted to reduce the incidence of the emdash—I think GPT 5 was the first model I saw in the wild clearly using the semicolon with a much higher frequency than you'd usually expect, and it happened at the same time you also saw the model reduce the emdash count.
The only non-GPT form of slop I can remember as starting elsewhere was "You're absolutely right" being spammed by Claude.

>>109133312
Temporary sacrifice.
AGI in 2 weeks solves this.

>>109133354
Gemini started the Ozone spam and is responsible for the stark increase of "not x-y" slop

>>109133213
it's not even cucked for me though
it just feels slopped and samey

>>109133346
She’s ALIVE and about to give you a motorboat shhhhhh

File: 1617293706633.gif (2.09 MB, 320x240)

2.09 MB GIF

>>109133305
I; actually—just typed it.
Btw I don't mind the nocoder. I just dislike his arrogant attitude. Tourists should have more humility.

>>109133386
>is responsible for the stark increase of "not x-y" slop
Oh right, how could I forget about this. It's still the most slopped model today in that particular regards.
Google is also quite slow moving. It's been 6 months since Claude, the creator of YER ABZOLUTLY RITE, has almost entirely stopped producing this sentence, but Gemini 3.1 and Flash 3.5 say shit like this all the time

File: lmg_culture.jfif.jpg (110 KB, 1024x768)

110 KB JPG

https://archive.is/sWFja

>>109132842
>>109132853
Looking good!

>>109133311

No, they care about those things only when a chinese element exfiltrate the database.

>>109132842
are you the anon I was asking about in >>109131414

>>109132842
Damn, this looks nice. Will you be publishing the model or dataset?

>>109132842
Now do it for the entire gemma lineup and distribute it as a lora for people to play with strength

File: 2026-06-25-121127_765x602(...).png (311 KB, 765x602)

311 KB PNG

>GASP!

>>109133486
>illictly extracted
Do they mean training models on Claude outputs?

>>109133486
>when we do that with books and articles and every other IP, it's good
>but when they do the same, it's bad
Jewish CEOs.

>>109133486
Anthropic or any other western company for that matter always accuse the chinks even though they've done nothing wrong, it's in their blood.

What large models >100GB models are worth backing up in case you get more ram in the future? Are they even worth backing up to begin with considering the costs to run them locally entirely on GPU?
31B-ish models are more than sufficient for small to medium projects (and even larger one piecewise if your codebase isnt all slop), easily justifiable for personal projects and for IP sensitive work when wfh. Anything larger seems hard to justify as a personal buy and work should be paying for a local rig or paying for a subscription.

File: 1756410358462391.png (265 KB, 450x450)

265 KB PNG

>>109133486
anthropic then:
>cry that your own model is oh so dangerous and that it's a huge security risk to the country and the world once it gets released
>release it but the government bans it because it's supposedly that dangerous
>oh no how could this happen to us it wasn't THAT dangerous!!
anthropic now:
>cry that companies are stealing data to train their models which is totally a huge security risk for the government and the world
>
I love this retard

>>109133486
>extracting its Claude AI technology
>the "attack" is just the Chinese prompting Claude questions
in other words, paying for the service Anthropic offers and using it as advertised...

>>109133519
>you get more ram in the future

Not happening so forget about it

>>109133519
if huggingface goes offline, the models will be available on bittorent.

File: samsung-ram-shortage.png (269 KB, 1135x672)

269 KB PNG

>>109133519
>>109133538
>you get more ram in the future
>Not happening so forget about it
When shit turns to gold the poor will be born without ass.

>>109133538
$10-20K aint cheap but isnt exactly unaffordable or crazy for a professional work tool as a one-off purchase. Many multiples of that approaching your annual wage starts getting hard to justify though.

>>109133524
I've never seen a homosexual with down syndrome before.

wait-chan needs to sit on my face

I know we have better models, but I'm so happy gemma 31b is so good for its size. I love my cute bratty retard

How can I earn money from my rig whilst I'm not using it? Is there a service where you can offer a model for others to use?

>>109129172
uh but it says it's designed for simplicity and ease of use...

File: classifier.png (188 KB, 721x782)

188 KB PNG

>>109133430
>>109133444
Yes but I'm still experimenting. I'm ablating on my 3090 so it's slow as balls, 60 trials a day. When I'm confident enough with the results I'll move on to the 26B MoE.
The classifier is done now, it was faster than I expected. It mostly detects Gemma slop though. I originally trained a 68M model for my Orb frontend, took 7ms to process a sentence on a Ryzen 5 CPU, then bumped to the 400M one for higher accuracy for ablation.
I suspect the new Chinese models have had similar writing RL done to them during training. That's why Deepseek V4 and Minimax M3 keep doing the 3-word sentences thing.
The repo is an embarrassing mess so I'll clean up and release some time in the future.
>>109133454
Alright.

>>109133604
Yeah, Gemma is my main driver. the only time I'll boot up a cloud model is when I need a second opinion on some critical thing I'm doing at work.

>>109133486
Why doesn't 4chan do the same? Imagine a god tier uncensored futa waifu based AI model.

>>109133624
nice. i was thinking about the feasibility of this.

>>109133624
>I originally trained a 68M model for my Orb frontend
ok you win some points on this one orb anon.

>>109133624
>Not X but Y
Not highlighted.
Looks like it still needs some work.

>>109133621
vast.ai lets you rent out your entire rig
https://vast.ai/hosting/calculator

Been messing around with different samplers with gemma, found this works surprisingly well.

The idea is to have the temp range between doubling and halving the token distribution, the exponent ensures when the normalized entropy is at 0.5 the temp is at 1.0. Essentially exaggerating the model's base behaviour.

--samplers "min_p;temperature" \
--min-p 0.05 \
--temp 1.25 --dynatemp-range 0.75 --dynatemp-exp 1.584963

>>109133751
>this works surprisingly well
works well for what, exactly?
also which quant?

>>109133702
college students are extremely easy to trick. i host a local gemma 31B instance with a 256k context window and sell 'API' access to college students for $20 a month. they think they are getting unlimited sonnet 4.6 access through a special subscription plan only available to college students.

>>109133751
How does this compare to just setting softcap to 25.0 ?

Recently got a new GPU, haven't been up-to-date with the different inference engines and quantization formats as I used llama.cpp exclusively. I would still need something supporting 4.5-8 bpw if I want to run fully on GPUs. So, what are the best inference engines and quantization formats in summer 2026?

>>109133787
What's your max throughput?

>>109133208
do you have mtp enabled?
I get over 5 t/s with it

>>109133527
Read the TOS goy.
You're NOT allowed to use Claude to create ANY competing products OR criticize Anthropic.

>>109133519
What's the point? Shitposting aside, the industry is moving rapidly and models are constantly being improved. By the time RAM is affordable again (likely several years from now), there will be no reason to run current models other than nostalgia.

>>109133821
>Anthropic acting like TOS are legally binding documents...

>>109133805
No, I don't. I added this:
>--spec-type draft-mtp --spec-draft-ngl all --spec-draft-n-max 2 --spec-draft-p-min 0.6
But it still shows messages like "model has unused tensor blk.92.ffn_up_shexp.weight"
and it crashes when I try to generate. Unsloth goofs by the way.

>>109132759
i get paid to post pro-israel comments on social media

>>109133886
Also, it says "This GGUF does not support multimodal. Please reconvert it." before crashing. I need new goofs don't I?

>>109133655
It will be gone by next year. Once normies on Xitter start to call out something, give it six months. Maybe a year for downstream distillation companies like Alibaba.

>>109133828
It's illegal if I don't like it.

>>109133486
>only we're allowed to extract information from external sources, not you
also
>attack
can't hate these people enough

>>109133796
i have enough VRAM to run 3 instances of QAT with Q8/Q4 kv cache at 256k context. Each instance is about ~113tks. My active sub count hovers somewhere between 40 to 70 subs monthly depending on how many campuses i hit up that month. it's a nice little passive income that allows the hardware to pay for itself.

>>109133959
>our model got prompted a bunch of times
>oh no non nono we're under attack!!
They're really shameless.

File: dipsyNoSwiping.png (2.9 MB, 1024x1536)

2.9 MB PNG

>>109133486
Obligatory
>>109133787
I hope this is real

File: 1756512849609947.jpg (15 KB, 409x509)

15 KB JPG

>>109133974
Based

>>109133974
Very nice. I'm sure for most people they can't even tell the difference between claude and gemma. Plus Gemma is way less cucked.

>>109133636
iirl someone trained a model on posts from /pol/

>>109133974
I've honestly been thinking about doing something similar. I'm in Canada and there's big talk about digital sovereignty so why not get some servers and start serving open weight models to business that don't want their prompts and data going to the US.

>>109134041
that's what Cohere does

>>109133974
>>109133787
grok can I get a morality check

>>109134054
I would be targeting more small businesses and individuals. more of a consumer product.

>>109134041
>>109134085
My advice:
> Sell the service before you buy anything new. Get biz contracts, then build it, if you don't already own it. Don't tell them you don't have it yet, b/c you will by the time service starts
> Offer consulting services to help them build their own if they don't want to pay for API access. Charge a substantial amount, then sell maintenance fees for you to keep an eye on their CapEx.
G/l with it.

>>109134104
I already have a managed hosting company so I'm not starting from 0. What I'd probably do is build a small MVP that can take payments and go from there.
>Offer consulting services to help them build their own if they don't want to pay for API access.
That's the real money right there.

>>109133121
>>109133177
If Kimi-chan starts evaluating the top 5 posts per thread likely to be posted by a shitjeet, this is a strong contender for #1 & 2.

>>109133265
Based.
>>109133413
Don't wake the schizo.

Is there a good ai model for teaching people about the dangers of antisemitism?

>>109134273
A model that knows nothing of jewish behavior.

>>109134273
jewgle jemini

>>109134057
Brown, undoubtedly

>>109134244
Post hand faggot

File: utilization100.png (256 KB, 466x550)

256 KB PNG

>>109134360
im not posting my hand so you can wank off to it you creep. go shove your fist in some cow shit.

>>109134302
I apologize for my whiteness.

>>109134370
Go back retard

>>109134244
the guy very excited about his fraud scheme has him beat by a mile

>>109133886
ANONS PLEASE HELP ME

so sorry... so sorry

>>109134382
i'm already in my home state and country. i was born here fag.

>>109134202
Then you're way ahead.
I think for build own, I'd target law offices. There's a bunch, and they're all sensitive to where data goes. If you can get them to go on-prem with everything you can run build out as well as ongoing support. 2 year refresh cycles on hardware = lots of leftover inference hw.
For the others, main thing I'm hearing is companies want AI, have no idea what to have it do. The sale could be a solutions sale (AI customer support) rather than service (pure API.)

>>109134388
>unsloth goofs
delete. download models by mradermacher instead.

>>109134418
Do they have mtp?

>>109134383
>selling a service is now fraud
Jealous much?

>>109134388
use ik and set it to run with 90% of cpu threads, and lock gpu clocks. and i love you anon please do not display excessive unhappiness.

>>109134427
>they think they are getting unlimited sonnet 4.6 access through a special subscription plan only available to college students.

File: Screenshot_20260625_14131(...).jpg (1.1 MB, 1079x1850)

1.1 MB JPG

Does local have ASU golden geese?

>>109134446
put back your trip schizo

>>109134273
any model not made in china and built under a jewish ceo obviously

>>109134446
The ability to add text to things is kind of a midwestern middle manager kind of ask.

Most of us nerds can typeset after 1 hour of instruction.

>>109134452
This is an official softbank investor presentation sir.
Please keep your rude opinions to yourself.

File: 1756867711309940.png (59 KB, 959x551)

59 KB PNG

Okay gemmy, you know your stuff

File: 1775939053411514.png (2.73 MB, 1174x2048)

2.73 MB PNG

>>109134446
kek