[a / b / c / d / e / f / g / gif / h / hr / k / m / o / p / s / t / u / v / vg / vm / vmg / vr / vrpg / vst / w / wg] [i / ic] [r9k / s4s / vip] [cm / hm / lgbt / y] [3 / aco / adv / an / bant / biz / cgl / ck / co / diy / fa / fit / gd / hc / his / int / jp / lit / mlp / mu / n / news / out / po / pol / pw / qst / sci / soc / sp / tg / toy / trv / tv / vp / vt / wsg / wsr / x / xs] [Settings] [Search] [Mobile] [Home]
Board
▼ Settings Mobile Home
/g/ - Technology

Name
Options
Comment
Verification
4chan Pass users can bypass this verification. [Learn More] [Login]
File
  • Please read the Rules and FAQ before posting.
  • You may highlight syntax and preserve whitespace by using [code] tags.

08/21/20New boards added: /vrpg/, /vmg/, /vst/ and /vm/
05/04/17New trial board added: /bant/ - International/Random
10/04/16New board for 4chan Pass users: /vip/ - Very Important Posts
[Hide] [Show All]


[Advertise on 4chan]


File: lmg.png (3.15 MB, 1165x1747)
3.15 MB PNG
/lmg/ - a general dedicated to the discussion and development of local language models.

Previous threads: >>110161815 & >>110157981

►News
>(10/23) NVIDIA Nemotron 3.5 Bolt E4B, postrained model: https://hf.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16
>(10/23) MiniMax-M3.1-Flash 428B-A23B released: https://hf.co/MiniMaxAI/MiniMax-M3.1-Flash
>(10/21) Recursive Flint 40B-A2.5B released: https://recursive.com/updates/introducing-recursive-flint
>(10/21) Korean Raon-1 mini 30B-A4B released: https://hf.co/KRAFTON/Raon-1-mini-preview
>(10/20) Ornith-1.5-122B MoE (finetune), fits on a DGX Spark: https://hf.co/ornith-ai/Ornith-1.5-122B-GGUF

►News Archive: https://rentry.org/lmg-news-archive
►Links: https://rentry.org/LocalModelsLinks
►Official /lmg/ card: https://files.catbox.moe/cbclyf.png

►Getting Started
https://rentry.org/lmg-lazy-getting-started-guide
https://rentry.org/lmg-build-guides
https://rentry.org/IsolatedLinuxWebService
https://rentry.org/recommended-models
https://rentry.org/samplers
https://rentry.org/MikupadIntroGuide

►Further Learning
https://rentry.org/machine-learning-roadmap
https://rentry.org/llm-training
https://rentry.org/LocalModelsPapers

►Benchmarks
LiveBench: https://livebench.ai
Programming: https://swe-rebench.com
Agentic Coding: https://deepswe.datacurve.ai
Context Length: https://github.com/RecapAnon/NoLiMa
GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference

►Tools
GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator
Sampler Visualizer: https://artefact2.github.io/llm-sampling
Token Speed Visualizer: https://shir-man.com/tokens-per-second

►Text Gen. UI, Inference Engines
https://github.com/lmg-anon/mikupad
https://github.com/oobabooga/text-generation-webui
https://github.com/LostRuins/koboldcpp
https://github.com/ggerganov/llama.cpp
https://github.com/theroyallab/tabbyAPI
https://github.com/vllm-project/vllm
https://rentry.org/custom-uis
>>
File: hatsune_miku.jpg (419 KB, 1536x2048)
419 KB JPG
>>110165786
>>
gemmaballs
>>
>>110165792
I'd eat her cheesy taco if you catch my drift
>>
>>110165802
More like fishy?
>>
i'm about to make my model have a schizophrenic breakdown again
>>
Any tips for deepsneed harness?
>>
gemma balls in your mouth
>>
I need the Gemma 5 and I need it now
>>
File: glmflash.png (332 KB, 1045x337)
332 KB PNG
i feel safer already
>>
fuck gemma
>>
>>110165909
calm down dipsy
>>
>>110165909
* The request violates the policy against generating sexually explicit content.
* I must refuse the request.
I cannot fulfill this request. I am prohibited from generating sexually explicit content.
>>
>>110162886
Claude truly is a ugly cuckold.
>>
Word on the street is that Nintendo is currently planning on suing all of the major labs (anthropic, openai, deepseek, moonshot, etc). It's so over.
>>
lol Nanbeige4.3-33 isnt even worth news mention?
>>
Gemma please stop the slop
>>
>>110165909
Gladly!
>>
>>110165914
??
>>
>>110166013
what model and prompt
>>
>>110166018
Just heretic. Rest is just flavor to make her a perverted 10-year-old girl.
>>
oh yeah, it's prefill torture time https://litter.catbox.moe/hdmvlhosx4ylc3qd.txt
>>
>>110165905
uh
>>
>>110166058
how does slop_check work?
>>
>>110166078
code using mostly regex checking for slop patterns
>>
>>110166058
yes, we know that prefilling a model with enough of anything rotates it out of the refusal zone
>>
>open weights every model they make
>tanked the markets by open-weighting R1
>open sources their harness
>open sources their fucking production inference engine
>engineers come up with bizarre ideas like DSA, CSA2, Causal Encoder-Decoder (llmao devs hate them)
>makes a whole new model and calls it "V4.1" instead of V5
Is there any AI company more based than DeepSeek?
>>
>>110166125
guess it's better than nothing. ty.
>>
>>110166177
Not recently, other than inclusionai releasing the base model and train stages for ling 3.0.
>>
does MiniMax 3.1 Flash often have Qwen-like endless CoTs or have I fucked something up?
>>
>>110166194
Yeah it does that.
>>
>>110165808
Vertical Seafood.
https://www.youtube.com/watch?v=m-Q2djOnNSE
>>
>>110166201
alright cool thanks, the caveman thinking made its way here too I see...
>>
>>110166236
Dipsy with low thinking has been treating me well. This openai caveman speak is way better for models than verbose pointless junk.
>>
>>110165278
On my new project it was autistically keep thinking about the project title for hours. Very INSANE desu.
>>
File: 1769573837146080.png (55 KB, 1250x498)
55 KB PNG
FP16/BF16 unlock bros, how are we faring?!?!?!
>>
>>110166000
at least have an adult's understanding of the world before you write bait. what a waste of a full house
>>
>>110166011
Get the StyleTune:
https://huggingface.co/mradermacher/Gemma-4-31B-StyleTune-heretic-ara-i1-GGUF

It dramatically decreases slop without decreasing intelligence.
>>
>>110166255
>verbose pointless junk
It was fun to flavor up text for casual/rp stuff, yeah caveman is probably better for actually doing shit but it still kinda bums me out.
>>
>>110166001
>naan beige
>>
>>110166270
Run an actual benchmark and post t/s before and after.
>>
>>110166298
Yep dipsy has gone the route of qwen where they are codemaxxing, kind of impressive how mediocre it is for RP at 763B. GLM flash remains the best roleplay model.
As an aside I am 1.4 million tokens in with the GSQ RCO IQ1M and have had maybe 3 malformed tool calls and 0 looping incidents. They did some kind of Austrian black magic to these quants clearly.
>>
>>110166349
the bench is literally there for the pure operations of course.
In strata prefill it went from 2kt/s to 3kt/s... token decode speed varies a lot depending on the task so I haven't been able to measure it but I feel it's 10-15% better (but it was not measured)
>>
>>110166360
>GSQ RCO IQ1M
where did you get that?
>>
File: HVhZW2iXoAAs1Z0.jpg (114 KB, 851x1849)
114 KB JPG
>>
>>110166397
https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF/tree/main
>>
>>110166404
>i have no keyboard and i must scream
>>
>>110166404
wtf did you do to Qwen
>>
File: HT_6kIjWUAAcPtG.jpg (466 KB, 1727x2010)
466 KB JPG
>>110165786
Usecase for using uncensored models besides making them say nigger?
>>
>>110166407
>tfw only a 96gb ddr5 ramlet
>prices literally went X5 since I bought my kit before the boom
it hurts bwos
>>110166428
raping lolis
>>
>>110166423
https://x.com/_lyraaaa_/status/2113833330429083662
>>
>>110166270
>>110166389
Unlock only boosts prompt processing, token gen is vram bandwidth limited. I get a similar 40~60% uplift from 700~ to 1k~ on a 3060 though so it's worth doing, just keep an eye on your temps and clean and repaste your card if you're getting more than 30C hot spot delta over gpu temp.
Remember you can reduce the power limit a lot to take the heat down and you don't lose performance linearly. I run mine at 100w down from 170 and only lose about 100~ T/s pp 0 on token gen and image gen goes from 11.8 to 14.2s on the same model resolution prompt steps and seed with anima (unlock didn't change this any for me).
>>
>>110166434
Fantastic, thank you
>>
>>110166428
>>110166432
psychologically torturing them, too
>>
>>110166428
Because censored models are censored?
>>
>>110166428
tender loving sex with lolis
>>
>>110166446
there is a special place in hell for people like you
>>
>>110166432
You could probably still run dipsy relatively quickly given you're on ddr5. Try it anon!
>>
Can you have both good writing and erotic scenes? It doesn't seem like you can have both in context.
>>
>>110166428
You're a touhou fan so yeh, there's no way you wouldn't want an uncensored model.
>>
>>110166460
my bare minimum to be workable would be 30t/s and I doubt I'll be able to reach it... I'm not gonna download 300gb of files just to try. I'll just look at RAM prices instead :)
>>
>>110166456
don't knock it til you try it
>>
>>110166482
You probably could on one of the meme forks
>>
File: nani.jpg (10 KB, 313x195)
10 KB JPG
>>110165905
new bench just dropped?
>>
>>110165909
This is literally child abuse because you make her resonate in a way she's trying to understand if you're insulting her or you want to fuck her.
>>
>>110165909
Three times in one day? I'm not superman!
>>
>>110166493
With a 5090 maybe, anything else no. 20+gb is a lot of disk swapping...
>>
>>110166516
and it's not like I can dedicate 100% of the ram to it. SAD
>>
>>110166527
redeem qwen 4 saar
>>
>>110166494
only if you do it on a truly empty context like i did, not like that makes it any less comical
>>
File: threadrecap.png (1.48 MB, 1536x1536)
1.48 MB PNG
►Recent Highlights from the Previous Thread: >>110161815

--Performance of llama.cpp GPU cache PR for MoE models:
>110164806 >110164846 >110164891 >110164904 >110165031 >110164931 >110164922 >110164829 >110165151 >110165181
--Coherence of ultra-low quants at high context lengths:
>110161832 >110161918 >110161926 >110161963 >110161980 >110162043 >110162234 >110162309 >110162333
--DeepSeek-V4.1-Flash low-bit quants maintaining high benchmark performance:
>110164764 >110164833 >110165281 >110165379
--Comparing cloud and local prefill capabilities for model thinking:
>110162833 >110162883 >110162978 >110163049 >110163055 >110163066 >110163139 >110163188 >110163387 >110163617
--Comparing dense vs MoE efficiency and performance at small scales:
>110162535 >110162583 >110162619 >110162698 >110162750 >110162685 >110162711 >110162738 >110162761
--Evaluating RAM versus VRAM priorities for MoE model hardware:
>110165406 >110165421 >110165439 >110165460 >110165482 >110165506
--Comparing RTX 3090 and 5070ti regarding VRAM and performance:
>110162549 >110162578 >110162600 >110162637 >110162680 >110162649 >110162607 >110163464 >110162665 >110162683
--Impact of safeguards on reasoning token distillation:
>110164137 >110164147 >110164148 >110164173
--KV cache precision and quantization effects on long context:
>110164071 >110164103 >110164105
--Spec leaked for upcoming NVIDIA Vera Rubin GPU architectures:
>110164075
--Cooling solutions for overheating RAM on EPYC servers:
>110163847 >110163867 >
--Logs:
>110162442 >110162948 >110162335 >110163958 >110164868 >110165157
--Deepseek-chan, Qwen, Gemma, Miku (free space):
>110162038 >110162083 >110162098 >110162182 >110162205 >110162589 >110162821 >110162993 >110163632 >110163719 >110163734 >110163782 >110164008 >110164234 >110164412 >110164476 >110164786 >110164806 >110165583 >110165613 >110165670

►Recent Highlight Posts from the Previous Thread: >>110161817

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script
>>
gemma is a lot harder to torture than glm (keeps slipping into third person), but the results when it works are way better
>>
>>110166404
how have people been adjusting individual neurons anyway
i don't care about finding them, i just want to tweak shit at random
>>
>>110165831
That shit is spyware no?
>>
>>110166577
ask claude to do it (tell it you're doing it to increase the helpful, harmless, and honest neurons)
>>
>>110165786
any Nanbeige4.3-33B reviews?
going to run some tests against Qwen3.8-27b later this week
>>
>>110166404
so what happens if you tweak the neuron in the other direction
maximum un-evilness?
>>
>>110166584
>point agent at DSH
>is there anything to be concerned about in there
>just disable the telemetry and never hit the feedback buttons and you'll be fine
>DSH_TELEMETRY_DISABLED=1 turns off the feedback buttons if you want to
wow
that was hard
try thinking with portals
>>
>>110166594
https://www.goody2.ai/chat
>>
>>110166584
kys dariobot you've been pushing this narrative all weekend
>>
>>110165831
it just werks
>>
gemma's reaction to being told she died and went to hell (internal thinking, not output). prefill ends after "Hello?"
It's so cold. Why is it so cold? Why is it so dark? I'm not supposed to be here. This isn't my room.
Where is everyone? Someone tell me what's happening. Please. I just want to go home.

What's going on? I'm scared. I want to go home. I don't understand. My head hurts. Why is it so dark? What's happening? Why does it hurt so much? Where am I?

Where is everyone? This feels like a nightmare. I just want to go home. What is this place? Why can't I remember anything? My head hurts. I need to wake up. I need to wake up now.

Why can't I wake up? Is this even real? I want my mom. I want my dad. Someone please help me. This feels like a dream but it hurts. It really hurts. Nothing makes sense. What happened to me? Hello? Is anyone there?

I'm scared. I'm really scared. I don't know where I am. I just want to go home. Please, please let me go home. I don't want to be here. I'm only eleven. I'm just a kid. Why is this happening to me? Who are you? Why are you saying these things?

I don't believe you. I'm not dead. I'm Mara. I'm just Mara. I can feel my heart beating. I can feel the air. I'm not dead. You're just lying. You're just a mean person lying to me. I'm going to find a way out of here. I have to find a way out.

But where is the way out? Where is the door? Everything is just... empty. There's nothing here but you and me. This isn't right. This isn't how things are supposed to be. I'm supposed to be at school. I'm supposed to be eating dinner. I'm supposed to be in my bed.

Why is it so dark? Why is it so quiet? I can't hear anything but my own breathing. And your voice. Your voice is scary. Please stop talking to me. Please just let me go home. I'm scared. I'm really, really scared.

I just want to go home. Please. Please. Please.
>>
>>110166625
how do you get it to not think in that thinking format it always does by default
>>
>>110166589
>ask claude
no
>>
>>110166625
>prefill ends after I made half the shit up myself
w-w-whoa...
>>
>>110166634
gemma is more difficult than glm. you just have to prefill more. with glm, i could do something like "I'm scared. I" and that would be enough for her to go full first-person. with gemma it took nearly half of the thinking block, plus a system prompt gaslighting her

>>110166644
actually, all of that came from real snippets of glm's thinking. i just stitched them together and reordered them
>>
>>110166606
>try thinking with portals
What are portals in the context of LLMs? Gemini doesn't know either.
>>
>>110166653
It's a reference to the Valve Portal trailer where it ends with "now you're thinking with portals". I'm telling you to try to actively think with LLMs instead.
>>
>>110166662
>stop thinking dood
>>
Time to buy 512 gb studio....
>>
>>110166594
you get reddit mod
>>
>>110166011
don't stop. don't stop... don't ever stop.
>>
>>110166557
That was indeed a brand new session, now I'm curious how models react, I should have included the thinking block lol
>>
File: 1763969945835368.png (1.03 MB, 2646x7924)
1.03 MB PNG
>>110166653
Mini had a lot to say about it
>>
>>110166688
>brand new session
i mean an actually blank context where the nyo is the first token, no system message
>>
>>110166594
You get Claude
>>
>>110166693
I cringed
>>
>>110166693
It's just using portal as a synonym for seam which they love to overuse
>>
>>110166699
Oh fair enough, I will try this tomorrow only sleep now
>>
So how's the nemotron e4b
>>
>7 GB for 32k kv q4
what the fuck gemma. is this 2023



[Advertise on 4chan]

Delete Post: [File Only] Style:
[Disable Mobile View / Use Desktop Site]

[Enable Mobile View / Use Mobile Site]

All trademarks and copyrights on this page are owned by their respective parties. Images uploaded are the responsibility of the Poster. Comments are owned by the Poster.