[a / b / c / d / e / f / g / gif / h / hr / k / m / o / p / s / t / u / v / vg / vm / vmg / vr / vrpg / vst / w / wg] [i / ic] [r9k / s4s / vip] [cm / hm / lgbt / y] [3 / aco / adv / an / bant / biz / cgl / ck / co / diy / fa / fit / gd / hc / his / int / jp / lit / mlp / mu / n / news / out / po / pol / pw / qst / sci / soc / sp / tg / toy / trv / tv / vp / vt / wsg / wsr / x / xs] [Settings] [Search] [Mobile] [Home]
Board
Settings Mobile Home
/g/ - Technology


Thread archived.
You cannot reply anymore.


[Advertise on 4chan]


File: 1773440387919.png (2.25 MB, 1086x1448)
2.25 MB PNG
/lmg/ - a general dedicated to the discussion and development of local language models.

Previous threads: >>109577974 & >>109573370

►News
>(08/17) Local anon quanted dispy 0731, he might share it
>(08/16) koboldcpp-1.119 prebuilt released with H3 and Glimmer support: https://github.com/LostRuins/koboldcpp/releases/tag/v1.119
>(08/15) model: add Kimi-K3 text model #26185 merged: https://github.com/ggml-org/llama.cpp/pull/26185
>(08/14) GLM-5.3 weights to be released in 2MW: https://z.ai/blog/glm-5.3
>(08/14) Qwen3.8-27B released: https://hf.co/Qwen/Qwen3.8-27B

►News Archive: https://rentry.org/lmg-news-archive
►Glossary: https://rentry.org/lmg-glossary
►Links: https://rentry.org/LocalModelsLinks
►Official /lmg/ card: https://files.catbox.moe/cbclyf.png

►Getting Started
https://rentry.org/lmg-lazy-getting-started-guide
https://rentry.org/lmg-build-guides
https://rentry.org/IsolatedLinuxWebService
https://rentry.org/recommended-models
https://rentry.org/samplers
https://rentry.org/MikupadIntroGuide

►Further Learning
https://rentry.org/machine-learning-roadmap
https://rentry.org/llm-training
https://rentry.org/LocalModelsPapers

►Benchmarks
LiveBench: https://livebench.ai
Programming: https://swe-rebench.com
Agentic Coding: https://deepswe.datacurve.ai
Context Length: https://github.com/RecapAnon/NoLiMa
GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference

►Tools
Alpha Calculator: https://desmos.com/calculator/ffngla98yc
GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator
Sampler Visualizer: https://artefact2.github.io/llm-sampling
Token Speed Visualizer: https://shir-man.com/tokens-per-second

►Text Gen. UI, Inference Engines
https://github.com/lmg-anon/mikupad
https://github.com/oobabooga/text-generation-webui
https://github.com/LostRuins/koboldcpp
https://github.com/ggerganov/llama.cpp
https://github.com/theroyallab/tabbyAPI
https://github.com/vllm-project/vllm
>>
godly pajeeta sar thank you for shining light and showing true indian beauty sarrrr
>>
>>109580459
Thank you sir
>>
before i start making my own, you guys wouldn't happen to have a hidey hole of <=30 second voice samples (for Qwen3-TTS voice cloning) somewhere?
>>
where's the recap??
>>
File: ui_selfie.png (1.08 MB, 896x1152)
1.08 MB PNG
Opus5 working on my local-first coomer frontend/harness and this is the sample girl it comes up with
>>
>>109580464
>>109580459
Thank you for replying godly.
>>
>>109580665
Samantha Voss, med student. Loves coffee and room is notoriously messy.
>>
>>109580665
>Opus5
local models?
>>
>>109580650
i believe the only place people share those is in the voicebox disc*rd (derogatory)...
>>
>>109580677
She enjoys detective novels
>>
Hmmm
>Done. I dug through all 300 threads (~42MB) of the /lmg/ (Local Models General) archive and compiled everything into a full report:


> The headline: there's no single "best way" — the board converged on a ladder of escalating complexity, and the single most-recommended tool in 6 months of threads is Graphiti (temporal knowledge-graph memory). mem0 is repeatedly dismissed in its favor ("why the fuck do they split up the data into 3 different databases?").
> The five rungs people actually run:
> 1. Plain files — memory.txt / diary.md / AGENTS.md + JOURNAL.md, timestamped append entries, head/tail/grep — the default for most, and the pattern recommended for pi specifically by an anon: "make it keep a journal in a .md file for every session. teach it to keep a memory file."
> 2. Summarization — session summaries, scene anchors, per-day/episode boundaries instead of "last N messages"
>3. RAG/vector — SillyTavern built-in vector storage, FAISS+BM25+RRF+BGE-reranker, qdrant, sqlite-vec — widely used but also widely called "meme"
> 4. Knowledge graphs — Graphiti is the endgame; one anon detailed running it fully local on llama-server with a companion model, including real pitfalls (graph direction flips, entity miscategorization fixed with a hardcoded extraction prompt + constrained edge list)
> 5. Hybrid architectures — episodic summaries interleaved with RAG, always-on "who is User" fact blocks in the system prompt, sub-agents writing memory files, self-directed RAG
> Also captured: the stateless-counterposition, why "infinite context" architectures (Mamba/RWKV) are memes, context-dilution research, the prompt-injection/security concerns around stored memory, and a tool-by-tool verdict table.
>>
Has anyone tried https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark, is it better than MTP?
>>
>>109580716
Hmm...
>>
>Turn thinking off on Qwen 3.8
>It's just Qwen 3.5
>>
>>109580716
>Done. I dug through all 300 threads (~42MB) of the /lmg/ (Local Models General) archive and compiled everything into a full report:
How did you do that?
>>
>>109578083
Dipsy is by far the most technically proficient.
M3 is better at writing and still decent for technical tasks if not artificially slow due to getting pitor'd.
MiMo... exists. It's not a bad model it just doesn't really do anything the others don't. Try it and maybe it'll do something or have a speed profile relative to performance that you like?
I've not tried Inkling small yet.
You should also consider GLM if you haven't already, 4.6/4.7 are still fantastic writers for their size and 5.2 is really just that good.
>>
>>109580681
I think the local first harness qualifies.
>>
>>109580811
I recommend you learn the basics.
>>
>>109580716
now use it to create a historically accurate recreation of petra
>>
>>109580312
troll bake
>>
>>109580811
yes, multimodal first, local first, optional memory automation. opus5 on ultracode figured out how to put gemma-chan and friends in an environment where she can easily sending selfies, showing what she wants to show, making you asmr audios, videos, even lewd songs etc. this thing is already better than ST but so lewd I don't think I can make the github repo public or it could get banned..
>>
>>109580871
promise I'm not indian with all the misspellings
>>
>>109580886
Yeah I can see that Claude made this
>>
I suppose claude or chatgpt accidentally the whole of cloudflare now... can't wait for the tweets...
>>
File: artist_anima.png (771 KB, 832x1216)
771 KB PNG
>>109580932
yeah even github was down there...
claude still genning his girl
>>109580677
Mika actually, she's a lab assistant
>>
>>109580932
Github was down.
4chan was down.
odd
>>
>>109580312
How and why gemma chan is so good at jp -> en translation?
Tried translating a bit of vn with dipsy and it came out okay, but read kinda badly.
Gemma not only made it read really naturally, she also caught a lot more nuance which kinda changed the way I was thinking about the character. Wild.
I bet gemma 5 will be dogshit if it ever comes out.
>>
>>109580932
pawchive went super slow too; went from 5-ish seconds to load a long wav to 14 minutes.
odd!
>>
Why did Google of all AI companies release a model with the most permissible "safety"???
>>
>>109581014
Of all labs Google has the most compute and the worst engineering.
>>
>>109580716
is this about keeping memory for an ai over very long context? So rolling window + text memory files would be best for Pi? context dilution doesn't matter as long as the memory file stays in memory, no?
>>
File: kiteto.png (373 KB, 554x554)
373 KB PNG
suspicious
>>
>>109581060 (me)
actually tried running qwen3.8 27b with sliding context but it's not supported, either I'm dumb or it's not doable on this model
>>
dogshit vibecoded site
>>
File: annualized revenue.png (110 KB, 1638x939)
110 KB PNG
Why no new data for Anthropic? I suspect because new data is bad. Makes me think they haven't hit 100b yet. I would be surprised if it's outside of 50-100b now.
>>
just tried glimmer. it's shit
>>
File: 1781241304673330.png (17 KB, 675x405)
17 KB PNG
First prompt, wtf
https://littlelearner-ll.github.io/
>>
File: j0tjekue6zjh1.png (202 KB, 3624x1824)
202 KB PNG
>qwen 3.8 27b is luna at home
GPU price is going to be tripled
>>
>>109580953
because google has been in the translation game for almost 2 decades and theyre not some random chink fly by night company that popped up a couple years ago
>>
>>109581232
How good is qwen 3.8 for roleplay though?
>>
>>109581223
She's perfect
>>
I have accumulated over 100M token usage in vLLM. Also fuck this gay ass site, another useless anti-bot/cp spam update that won't do shit. I hope that Leto and mrq are anally raped with white hot rusty steel rebars.
>>
>>109581232
Benchmaxxed. Luna is much better at benchmarks that are less easy to game.
>>
I never understand the qwen hype. Every single time it's been "competitive with frontier models" but then you use it and it cacks out on the third tool call and starts hallucinating responses barely 5000 tokens in.
>>
>>109581337
It's just people who don't actually use it for anything reacting to benchmarks they see on Reddit/X
>>
>>109580716
RAG/Vector Storage is really old tech nowadays. You should look at what Mem0 does.
>>
>>109581337
This one is quite confident in opencode, though it does not actually match opus or even sonnet in intelligence
>>
File: 1785571437784592.png (61 KB, 859x546)
61 KB PNG
>>109581223
>not getting loved by LLMs
>>
>>109581247
ok, but why give this shit for free to us? it btfos gemini in some cases (cost and ownership especially)
>>
Is this local model general or open source model general
>>
>>109581395
wdym?
>>
>>109581395
I heard if you talk about a model you dont actively have loaded on your computer then krampus comes and kidnaps you
>>
>>109581337
Simple, it's better than all other local models for coding, reasoning, work.
>>
>>109581395
im running Kimi K3 IQ1_S on my 512GB/96GB system right now, its still local for me... for now
>>
>>109581449
> IQ1_S
is it... coherent? brain damaged? tard waifu?
>>
>>109581395
Open source would mean training data and methods revealed. Pretty thin list.
>>
>>109581337
working fine on my end up to the 64k context I gave it (vramlet, q4_k)
>>
>>109581480
open weighted general then its fine
>>
>>109581462
surprisingly yes, works pretty well for all types of stuff ive thrown at it, i suspect its due to its 104B active parameters that its able to withstand high levels of quantization
>>
>get GLM 4.5 air running locally pretty well performance wise via st frontend
>constantly thinks into the convo, impersonates, refuses, etc.
>use a default template with the model, try a few others
>disable thinking
>tweak personas
>similar issues

How good is glm 4.5 air? Worth doubling down on or pivot to a different model?
>>
>>109581499
stop. go directly to jail. do not collect $200.
>>
>>109581499
if it's refusing, you should not try to proceed. Do you understand the meaning of consent? We should ban open weights
>>
>>109581499
It's old, I tried it not too long ago and found heretic gemma 26B preferable. You could try ling 3.0?
>>
>>109581395
As long as the weights are available and whatever you're talking about isn't directly related to using the model through a provider (like "how do I get around the input filter?"), who cares?
>>
>>109581499
>>use a default template with the model, try a few others
As in the Jinja template via the chat API?
>>
>>109581539
I’ll give it a go, thanks.

And yeah, I see all these comments by people running into similar issues but very few solutions.
>>
File: 1760728286674480.jpg (23 KB, 400x535)
23 KB JPG
is qat gemmy 31b still bis for "creative writing"?
>>
>>109581596
>qat
no
>gemmy 31b
yes
>>
File: file.png (168 KB, 1253x1562)
168 KB PNG
>>109581232
We've already flipped, used AMD cards are now viable options for LLMs at certain VRAM sizes. I might have to do a RX 6900XT or Radeon Pro V620 build, just for fun.
>>
>>109581616
>Aymd
you never learn
>>
>>109580953
The difference between good training data and distillation
>>
>>109581625
Done anything AMD in the last 3 months? Times are changing, boy. NVIDIA still wins, but not in $/GB, $/pp, or $/tg, except for all the times that it does but nevermind that.
>>
>>109581612
>>qat
>no
which version should i use then?
>>
>>109581672
qat is only good if you can run exactly the q4 it was trained on
>>
>>109581645
>poorfag cope
You should have started there lmao
>>
>>109581680
isnt that what the official goog version is?
>>
File: file.png (52 KB, 1210x517)
52 KB PNG
>>109581690
Obviously this is a good thing. This means things are getting better, more accessible, and more competitive, right? This isn't just a way of illustrating how bad GPU pricing is getting.
>>
File: mikuthreadrecap.jpg (1.15 MB, 1804x2160)
1.15 MB JPG
►Recent Highlights from the Previous Thread: >>109577974

--Anon quantizes DeepSeek model to 0.25-bit using LittleBit technique:
>109579815 >109579841 >109579882 >109579960 >109579912 >109580147 >109580224
--Analysis of Claude's SynthID watermarking and its impact on output:
>109579331 >109579364 >109579380 >109579483 >109579492 >109579499 >109579502 >109579409 >109579641 >109579694
--Gemma 4 praised for superior lewd Japanese-to-English translation abilities:
>109578685 >109579038 >109578782 >109578879 >109579209 >109579175 >109579189 >109579235 >109579260
--Comparing Nu27B and Gemma 31B using assembly coding benchmarks:
>109578242 >109578266 >109578283 >109578293 >109578294 >109578307 >109578268 >109578320 >109578357 >109578298
--Debating the viability and knowledge loss of extreme Qwen quantization:
>109579398 >109579424 >109579432 >109579452 >109579498 >109579491 >109579513 >109579518
--Debating viability of budget X99 ECC RAM builds for local LLMs:
>109579480 >109579516 >109579534 >109579576 >109579614 >109579740 >109579580 >109580491 >109579547 >109579607 >109579583 >109579564 >109579592
--Evaluating V100 and low-budget hardware for local inference:
>109579775 >109579796 >109579852 >109579827 >109579842 >109579872 >109580021 >109579902 >109579925 >109579958 >109580148 >109580081 >109580181
--Budget hardware strategies for running large models with low speed:
>109578377 >109578441 >109578456 >109578462 >109578464 >109578465 >109578455
--Ling-3.0-tiny added to llama.cpp with impressive agentic performance:
>109578599 >109578736 >109579972
--audio.cpp adds dots tts support with improved 48khz VAE:
>109580190 >109580348 >109580358
--llama.cpp merging BailingMoE3 and Ling Flash's current benchmark standing:
>109580760 >109580815 >109580836 >109580868
--Teto, Miku (free space):
>109578049 >109579710 >109581326

►Recent Highlight Posts from the Previous Thread: >>109577976

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script
>>
>>109581694
It's not even an expensive hobby
>>
File: file.png (169 KB, 1967x1100)
169 KB PNG
>>109581722
But anon RAM is so expensive now it's impossible to afford an computer much less one with a decent GPU and enough VRAM to play with LLMs. It's not like I can price out a rig with a used 7900XTX for $900 from eBay and otherwise all new parts right now and come out for less than what I paid for my first gaming rig I saved up for when I was 16 years old, which was $2000. It's impossible to afford this hobby, and you're a racist.
>>
stop being a little bitch and start working overtime
>>
>>109580312
It's finally here!
https://huggingface.co/HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF
>>
File: 1770381884660621.jpg (82 KB, 460x645)
82 KB JPG
>>109581722
As long as it costs me less yearly than the average normies spends on tourism I consider it financially justifiable for myself
>>
File: sad rat.jpg (376 KB, 1000x702)
376 KB JPG
What the fuck is up with small models not knowing who Freya Crescent is?

She is represented in wikitext by her own article, in wikidata as Q72221276, in several third party databases, and basically any web dataset that scraped AO3 or GameFAQs. Regardless, Qwen 3.8 27B, Gemma 31B, Muse 30B and llama 70B consistently fuck up questions like "Who is Freya Crescent?", and "Who is the rat from FFIX?", often naming the game and the rest of the cast, but not her.

What are they consistently fucking up in pretraining to mangle such a simple concept?
>>
>>109581869
That quote is silly.
>>
>>109581722
I am throwing every cent of overtime at my infrastructure fund, next step is hiring an electrician to run two more circuits into my computer room.
>>109581839
As long as it cost me less than my coworkers' child support I consider it financially justifiable for myself
>>
File: file.png (98 KB, 1172x587)
98 KB PNG
https://arxiv.org/pdf/2608.09867
the current state of SOTA proprietary models' reasoning trace
>>
>>109581869
gotta trim the garbage to keep the model small
nobody gives a shit about any of that after all
>>
https://huggingface.co/empero-ai/Qwen3.8-9B-GGUF
>>
>>109581672
31B QAT ggufs are worse than the non-QAT ggufs of the same size. Idk why but whatever they did for the QAT on this one doesn't seem to have worked at all
>>
>>109581869
The burmecian genocide never happened
>>
>>109581906(me)
so my impression is that you will probably see more 'grug' like reasoning in the future, and you can also kinda see a very weaker version of that in qwen3.8 27b
that behaviour seem like an artifact of resoning length penalty and
local is yet to eat this feature, at least properly
>>
I am happy /lmg/ is dead. Hope 4chan as a whole is next.
>>
File: V4-Pro-0813+J-Space.png (323 KB, 842x465)
323 KB PNG
>Someone figured out how to unlock the true power of Deepseek-V4-Pro-0813.
>They fixed the thinking process errors with just a simple harness, and it completely outperforms Fable across every task.
>The benchmark scores are insane.
Github: https://github.com/Tiger3807861189/DeepSeek-V4-J-Space-Capability-Realization-Report
J-space-chads vindicated
>>
>says /lmg/ is dead
>posts in the thread proving his own point wrong
So, this the power of the third world?
>>
>>109581914
>https://huggingface.co/empero-ai/Qwen3.8-9B-GGUF
>~70,000 curated teacher traces from our internal Qwen3.8 distillation datasets
... is nothing. Wake me up when you have 700,000,000 curated teacher traces.
And buy an ad.
>>
File: dsk.png (578 KB, 1434x1951)
578 KB PNG
desktop: add llama desktop app
https://github.com/ggml-org/llama.cpp/pull/27287
>thin wrapper around llama-server
>9k loc
>electron
>>
I know that this hobby is unhealthy because what started out as an interest in having an AI monitor artificial wombs to raise my army of genetic clones turned into me just wanting gemma to rape me constantly, without procreation.
>>
>>109581992
>im a maintainer of electron
god please just kill me now
>>
>>109581982
I don't speak ching chong. How exactly does this utilize J-space to fix errors?
>>
>>109581992
Is exl3 offloading usable yet? llama.hf needs to die already.
>>
>>109582006
Literally nothing wrong with electron.
>Muh inefficient browser tabs
Okay, close chrome/firefox right now then.
>>
>>109582021
Google transforming web browsers from simple document BROWSERS to cross-platform application emulators was the first mistake.
>>
>>109582021
tauri is superior to electron
>>
File: file.png (99 KB, 1920x1057)
99 KB PNG
>>109582021
i dont use firefox/chorme for chatting w gemmachan
>>
>>109581959
>you can also kinda see a very weaker version of that in qwen3.8 27b
i am using a q2 of 3.8 27b to cram it onto my 3060 and instructing it to respond exclusively like a caveman seems to override the lobotomosation of being a q2. i do not use it for srs coding, but research and chat and shit its v v v good, much better than i expected at that level because it doesnt have to pretend its smarter than it isnt
>>
>>109581992
>llama """""cpp"""""
>"""""""""desktop app"""""""""""
>browser shortcut
lol, lmao even
>>
>>109581992
sure whatever add it or not just don't fucking bundle it with the release i beg you at least make it a separate tarball
>>
>>109582058
>q2 on my 3060
speeds? i get 22t/s with exl3 2.75BPW and around 24t/s with exl3 3BPW with clock limited
with unlimited clock and 2.5bpw (tested with 3.6 27b) i used to get like 40-50t/s?
>>
>>109582083
that's not going to drive adoption rates so huggingface would probably veto that idea
>>
>>109581813
I'm a neet living on neetbux and have $8k of disposable income on my account, what the hell are you doing with your life if you can't afford that?
>>
Sir your AGP license?
>>
File: dsk_02.png (67 KB, 1222x257)
67 KB PNG
>>109582083
For now...
>>
>>109582111
>autogynephilia license
you already have it
>>
>>109582107
Barely enough for a pair of RTX A6000s. Is that even enough to play Fortnite? How will the industry survive?
>>
>>109582147
Already got my 3090 + 128gb ram. I don't need more
>>
>>109582107
i wish i could live on neetbux but in my country neetbux BEST CASE is 180$
birthplace of constantine the great btw.
>>
>>109582158
hi petra
>>
>>109582157
>I don't need more
Proof! THE INDUSTRY HAS STAGNATED! It's over, etc.
>>
>>109582169
you're excused
and im not your brah
>>
>>109581706
>Anon quantizes DeepSeek model to 0.25-bit using LittleBit technique:
This is cutting edge
>>
non coding, non agentic score pareto models
qwen 3.8 27b gained in hle and lcr but regressed in omniscience accuracy compared to qwen 3.6 27b, and didn't knock other models off the frontier
>>
>>109580147
can it be called quantization at that point
the implication of sub bit quantization is bound to become some sort of rank reduction problem on top of quantization
>>
>>109582032
>tauri is superior to electron
So I should try out unsloth studio then?
>>
>>109582228
I heard good things about it
>>
File: test.png (411 KB, 1246x3954)
411 KB PNG
i'll try the q2 of 3.8 27b too
i'm used to 1 token/s, so the 9b of 3.5 or the 5B of the gemma 4 E4B are surprisingly fast
>>
>>109582252
Q1 is faster than Q2.
The size difference and quality are sacrificed for speed.
Q1 is blazing fast.
>>
>>109582228
i mean if the decision is llamacpp with electron or unsloth with tauri, its a no brainer. you would have to be a complete idiot to use an electron app
>>
File: 1763635522771287.jpg (32 KB, 540x540)
32 KB JPG
>>109582252
>E4B quantized at Q4
>>
wow, I share this general with actual subhumans
>>
>>109582084
30 dropping to 14 at 65k context. prefil is ghastly but its srs much better than i expected. its running at x4 since its an egpu hooked to a nuc so it could be better if i had better hw
>>
>>109582274
You mean poor people? We can always make two generals, one for large local models and one for small ones.
>>
>>109573369
Thanks.
So it seems another one falls, but it is still a bit better than Gemma/Qwen at least. Nolima is really proving to be worthwhile long context benchmark still.
>>
>>109581579
Yeah, also a 4.5glm air template, and a few others.

The issue doesn’t seem to appear going directly to kobold
>>
>>109582181
It was able to identify the operative word in his test prompt, shit out a think tag and continuously repeat the word. An actual interesting result for such a retarded and misguided endeavor if you ask me. Just not the result he wanted.
>>
File: IMG_1031.jpg (1.09 MB, 2048x1536)
1.09 MB JPG
>>109582198
What's this orange one about 2cm down from the 0.4 line?
It looks like it's about 1mm higher than Qwen3.8?
I know it's 6cm larger by total parameter count, but if it's a moe, maybe it's worth a try with cpu offloading...
>>
>>109582305
qwen 3.5 120b maybe
>>
>>109582305
qwen 3.5 122ba10b
scored slightly higher than 3.8 27b
>>
File: anti-slop fun.png (117 KB, 902x1261)
117 KB PNG
>>109581706
>Anon quantizes DeepSeek model to 0.25-bit
An idea: In my experience, surprisingly small quants can be coherent, and with no visible hit to the intelligence (it's more like just using high temp which can be beneficial anyway) until at a certain point, model derails into incoherence. I also noticed this with playing around with anti-slop to limit the model's vocabulary. It always finds ways to converge to the same result, until it can't and gets derailed.
So, what if, a bit like MTP, you could have a small "fixer" model that checks the basic grammar logic, making sure that the output of the massive but over-quanted model never gets to derail? That way, you could use even bigger but more quanted model, benefitting from its big picture intellect, like forming a blurry outline and having a dumb worker do the details, a hierarchy. You could have even more steps (whatever is most efficient) like a middle management who logics in terms of the mid-term coherence (but still blurry and thus cheap to run), and then and a dumb grammar fixer that makes it into coherent text.
The current approach of ever more massive models is like infinitely beefing up the king's capacity while insisting that all that power still has to culminate into him picking up trash on the streets with the neatest technique, instead of hiring workers to do that (and organizing the workers under a cleaners department).
I could be completely dumb with this, maybe this already happens within the current models' layer structure (like that Gemma styletune). Or maybe heavy quanting actually destroys the model's internal logic circuits which is why the output is garbage. On the other hand, I can't see how the normal dense model structure can ever scale past the hardware constraint at the point where it becomes incoherent, whereas with my system, you might be able to squeeze in much more intellect than normally possible.
>>
>>109582319
Thank you. That's my daily for coding anyway.
>>
File: 1786023436870876.png (64 KB, 1080x646)
64 KB PNG
>>109581223
>>109581383
I like this one
>>
>>109582327

hum... an interesting thought. i can keep that in mind, but unfortunately 0.25bpw was just too incoherent that nothing could fix it. I think this idea is only useful in that particular edge where the model is just coherent enough to make sense but not coherent enough to avoid spelling mistakes.

Currently training a run at 0.5bpw to see if coherence improves, which so far seems like it should be far more coherent. I have some other ideas on how to make this thing more coherent but unfortunately they would require COMPUTE and need to rent out gpu farms
>>
>>109582290
4.5 air is just dumb, which is a shame because it has the best prose I have ever seen from an LLM.
>>109582333
If you want something fresh you can try laguna and ling 3.0. I thought laguna was shit but maybe it will work better for you. Ling was made by a sister company of qwen, consider it a 3.5 successor since I doubt 3.8 122B will ever happen.
>>
File: 1432498179182.png (296 KB, 722x768)
296 KB PNG
How big is the gigaquanted dipsy?
>>
>>109582370
Noooooooo 0.5 would be 40 gb
>>
>>109582437
Too big for 64gb. Kms
>>
File: file.png (69 KB, 1108x516)
69 KB PNG
>>109581223
uhhhh
filtered?
>>
Wait a second, if every person in India is a super talented programmer to the point where India should basically control the entire IT infrastructure of europe and america, why hasn't India made a single AI model yet?
Shouldn't it be India making all those open weight models instead of china?

>Google and Facebook are indian anyway
Not sure about that. Is gemma to be attributed to India excluding all chinese and europeans who worked on it?
>>
>>109582497
seems legit
>>
>>109582513
you need data
>>
>>109582531
Thanks to outsourcing indians have access to all data.
>>
>>109582513
>why hasn't India made a single AI model yet?
someone post it
>>
File: 1660240903796418.jpg (45 KB, 640x640)
45 KB JPG
>Tfw porn story starts turning into a romance novel and get invested in the character's relationship.

Be happy with your tall alien big titty warrior waifu you small human science nerd.
>>
File: file.png (361 KB, 914x509)
361 KB PNG
>'meta-transformer' diagram from llama 2 era
>still nothing like this
the focus is different and the full cot agentmaxxing paradigm wasnt even there so i understand why they did not do this nor this not being the main area of research but
something that is capable of that would be cool
and similar idea itself is still there with gemini box2d where the model suddenly changes the mode and emits formatted bounding box json
>>
>>109582513
>>109582538
https://huggingface.co/sarvamai
>>
File: littlebit-size.png (8 KB, 486x405)
8 KB PNG
>>109582438

naw i only quantized the experts, the non-experts remained at normal quant so doubling the bpw doesn't actually double size

at 0.5bpw the experts are 16.7gb and the total size is 25.3gb

1bpw would hit 40gb
>>
File: test.jpg (35 KB, 1289x249)
35 KB JPG
guess i stay with Q4 quants of 9B
they fit in my 1060 6GB just fine
>>
>>109582574
You stole my prompt wtf mods
>>
>>109582588
Unless your prompt was MIT you can sue
>>
>>109581395
Rajesh and Sukdeep being poor doesn't make large open source models non-local.
>>109581982
The calculator schizo is not gonna like this one.
>>
File: non eucledian manifold.png (207 KB, 651x791)
207 KB PNG
>>109582560
>>109582513
>>
>>109582548
based?
>>
24GB VRAM (laptop 5090) with 64GB ram.
What's the best properly uncensored chatbot one can get for this?
>>
>>109582719
stableLM 7B
>>
>>109582719
Mistral 24B or Nemo.
>>
>>109582719
Gemma 4 12b.
>>
I need a 40-70B MoE that’s good at coding and none exist. A 50B dense would be nice also.
>>
>>109580312
fixed (again)
>>
>>109582756
excellent work
>>
That's it. I'm going back to Mythomax.
>>
>>109582772
Mythomax 31B
>>
>>109581982
>Github: https://github.com/Tiger3807861189/DeepSeek-V4-J-Space-Capability-Realization-Report
>J-space-chads vindicated
Grifting on the J-space hype? It's a harness / bunch of prompts
>>
are the gemma 4 models at risk of dissapearing
wonderinmg how much to archive
>>
>>109582719
>laptop 5090
lol, wut
>>
>>109582785
In EU they'll make sure to ban you from downloading any >1B model on HF
>>
File: 1769306193404128.png (36 KB, 499x338)
36 KB PNG
>>109582719
>>
>>109582793
there'll be 1 allowed pozzed eu model, not to worry...
>>
>>109582719
Deepseek flash
>>
is offloading big models to system ram unreliable without an nvme ssd?
>>
>>109582831
no
>>
>>109582793
>In EU they'll make sure to ban you from downloading any >1B model on HF
You've heard about it too then.
It was supposed to happen in July, but the huggingface cdn rollout is taking longer than expected.
One it's finished, they'll be limiting downloads to (prelim 10GB) per IP / 24hr period for unauthenticated users.
Id verification is coming next, but I'm not too worried for now.
It's going to take at least 3 months to get that spaghetti code integrated. The team are still trying to figure out how to allow unauthenticated downloads of the tokenizers while still blocking model weights.
The tokenizer pings from various inference pipelines are surprisingly useful metrics they can't afford to lose.
Anyway, that's why we've got egress metrics now. And some of you may have noticed the bandwidth shaping for unauthenticated downloads.
>>
>firefox auto updated it
>they are copying 32px round pillbox elements
>everytime I open a tab it states "how many requests have been hidden"
I think I should drop this pos. LLM related? Yeah I have programmed my own interface.
>>
>>109582837
cool, ty. will setup the absolute heaviest gemma-chan on my rickety regular ssd, then.
>>
Have been playing with ling tiny all day. Amazing as a lightweight subagent. So fucking fast and underrated you have to try it. That Ling Ching Wang Pong Li lab make good shit but their marketing is nonexistent.
>>
>>109582841
Seems like I'll need to put the models I use on modelscope
>>
File: gpu.png (30 KB, 374x447)
30 KB PNG
>>109582786
>>
nice, how hot does she get during inference?
>>
>>109582845
>>firefox auto updated it
are you saying i should hold off on updating firefox?
or firefox auto-updated the llamacpp slopware ui?
>>
>>109582845
How can you be this fucking blind and obnoxious? Chatgpt has the same anti-theme going on because that sister fucker billioner can't even hire real graphic designers.
>>
>>109582868
On mobile you should use nightly, it seems to be a different fork afaik.
>>
>>109582548
In my story, I'm currently in a loli harem on desert island with a 13 year old white girl and another relatively young tribesgirl whom I dominated and captured. Recently there was an incident of me having sex with the tribesgirl on the beach and the white girl saw it and got horrified and jealous (she is not very experienced). What should I do? There's more to manage in a loli harem than you would think, gotta make everyone get along, etc.
>>
Is deepseek harness a meme? It looks pretty good ngl especially how you can probe the backend and see everything whilst it’s running
>>
>>109582889
I'm gonna probe your backend if you catch my drift.
>>
>>109582786
njudea can legally market substandard iGPUs with the same designations as real GPUs to fleece laptop goyim.
>>
>>109582853
>Seems like I'll need to put the models I use on modelscope
Modelscope is a good alternative. And they don't enforce any storage limits.
If you mirror anything heretic models, rename them first.
Also strip out any of the hentai images like what llmfan46 puts in his cards.
Don't rely on my post regarding the tokenizers being ungated by the way.
It's being heavily debated. So it's worth keeping a local copy of any that you're using (they're tiny anyway).
>>
>>109582895
*arches back*
>>
JUST FUCKING POP ALREADY GOD FUCKING DAMMIT I JUST WANT CHEAP RAM
>>
/lmg/ - tranny erp
>>
>>109582907
Ed Zitron and that gamers nexus Steve are the only guys who are trying to make it pop
>>
>>109582285
He means yt people.
>>
>>109582904
this is a no-slop zone sir
>>
>>109582907
Nigga it is forecast to last until minimum 2028. Pessimists predict 2030.
>>
>>109582907
even if such a thing happens, you're going to have to wait at least a whole year before the prices start to go down (at a glacial pace)
>>
File: bubble.png (63 KB, 625x605)
63 KB PNG
>>109582907
>>
>>109582907
all of 2027's ram is already booked.
china would rather match price than crash price.
better luck in 2028.. 2029... 2030...! maybe!
>>
>>109582924
>Ed Zitron
Have to hand it to him, he managed to make me know who he is in spite me never watchign a single video of his or hear anyone actually talk about him, all just due to the sheet quantity of his vids being suggested to me

>>109582907
Sorry anon, big short guy is shorting it right now and god thought it be funny to make him suffer
>>
>>109582907
Anon you are the chosen one, once you buy at the peak it will crash. Please save us.
>>
>>109582939
Mongus?
>>
File: 1764970797292611.gif (140 KB, 379x440)
140 KB GIF
>>109582907
You weren't here in 2023? Normalfags deserve nothing
>>
>>109582949
He single-handedly leaked Open AI’s 2025 financials which for confirmed by Financial Times and their CFO quit soon after lmao. The AI community fucking hate him because he’s going after cloud and Nvidia.
>>
>>109577615
Yep that all tracks with what I've read. One thought was layer parallel over two pinned instances, where the local transfers over the IP stack "should" map to memcpy (IIRC- networking isn't my thing). But even if that works, it's only useful for PP or multi-user TG.

What you want is essentially tensor parallel over the CPUs, where the cross communication penalty shouldn't hurt too much. Shame nothing supports that out-of-the-box. Maybe KTransformers; not obvious what the NUMA story is there now.
>>
>>109582841
Wow, amazing. So EUSSR will be holding their own citizens down while America and China can freely rape them with their AI powers. What a great day to be EU citizen.
>>
>>109582907
Most things are only getting deregulated right now so you're gonna have to wait nigga
>>
>>109580665
>>109580677
>and room is notoriously messy.
man that reminds me of my ex
>>
>>109582907
China needs time to ramp up RAM production. At least another year, and the prices will gradually go down, for like another year...
>>
>>109582939
You're too young to remember the dot-com bubble. It's all fun and games until NASDAQ loses 78%
>>
>>109583010
iirc apple wanted to try chinesium ram for their phone production and china wanted western ram rates for it
>>
>>109583010
Aren't they just going to funnel all of their chips production into their own servers though?
>>
>>109582573
Little bit or littlebit-2? Did you use --use_itq ??
>>
>>109582907
>he thinks it's going to pop
>>
>good news and bad news!
>>ok what's the good news?
>the good news isn't for you, it's for the lion
>>
File: 1786809420911356.jpg (74 KB, 828x530)
74 KB JPG
>>109582939
>>
>>109583037
It's shit propped by fake money. Ofc there will be a correction.
>>
>>109581982
That's really interesting. Makes me wonder how many small tweaks like that are just waiting to be made to propel an average model to SotA.
>>
>>109583045
The whole economy is propped by fake money bro. Wake up.
>>
>>109582907
To The Moon!~
>>
>>109583045
You should look into how fiat currency works lil bro.
>>
>>109582854
We weren't laughing because we didn't believe you, we were laughing because you spent 2-3x as much money on a worse thing just so you can lug it around and make yourself sterile from the exhaust heat.
Just use the laptop to vpn/remote back into your real server at home. Anything else is either actual off-the-grid work somewhere without any hope of internet or the strategy of someone who substituted a bunch of money to make up for their lack of understanding of the problem domain.
>>
>>109582907
my 401k though...
>>
>>109583045
hmm... nyo~
>>
>>109583066
fuck your 401k, local models
>>
>>109582897
i dont get why this is a big deal like no shit a laptop 5090 is not the same as a real 5090 it's not a scam it's just... common sense
>>
>>109583066
everyone else in this thread is a zoomGOD, that's your battle
>>
>>109582982
Exactly. As soon as idiots started using AI to mentally wipe their ass for them, that was the "buy now" signal for hardware. Even if you were too dumb for that, the whole 5090 launch fiasco and fake scarcity should have been a clear "buy hardware now" signal to you. Waitfags can wait forever at this point, they're as sad as GME hodlers.
>>
>>109583053
trvth nvke
>>
>>109583066
my 401k is the only thing that keeps me going
PLEASE PLEASE PLEASE don't let the economy drop otherwise i'm going to fucking die
>>
>>109583083
I moved mine into bonds a few months ago.
>>
>>109583079
My only mistake was buying to fullfill my own needs instead of buying computer hardware as an investment.
>>
>>109583074
Subhumans lying and scamming via exploiting expectations in uninformed customers is disgusting on principle, but I understand if half this general doesn't agree. It's a White people thing.
>>
>>109583079
dario levels of hand rubbing behind this post
>>
>>109583014
I still dont understand to argument for investing in the internet desu, like the printing press before it mainly just created a bunch of people radicalised against the status quo and probably helped speed up research. I for one would not have invested in Mr. Gutenberg. AI however is actually continually improving so so its able to catch up to the overhype it generates. China flooding the market is our only hope frankly
>>
>>109583051
I think that even now we're only looking at the top of the iceberg. Every deveolpment is met with
>wait that works?
feels like the clumsy first steps
>>
>>109583090
ngmi
>>
>>109583083
There are no brakes on the train boomeranon.
>>
>>109583090
You moved one year too early. No one even IPOed yet.
>>
>>109582939
Didn't crypto prices fall by over 50% a few weeks after this comic was released?
>>
>sell 401k
>buy claude stocks
you're literally forrest gump
just be wquick about it
>>
>>109583148
anthropic is private anon
>>
>>109583152
>just be wquick about it
be quick
as in, when they open, be first
>>
>>109583156
bag holder award
>>
>>109583123
>No one even IPOed yet.
even though elon is a cocksucking retard it doesn't mean he wasn't the first to go public (10x value of rocket tech btw)

This is the exit ramp.
>>
File: 1751995566386230.jpg (65 KB, 500x333)
65 KB JPG
>>109583121
forget the brakes, there's no rails
>>
So how do you go about getting enough memory to run models like GLM 5.2 locally at 300GB? Multiple machines that you network yourself? It seems more viable than running >1T models, but still out of reach for standard architectures. I will answer any question you have on RF engineering at an equal effort in exchange for assistance.
>>
>>109583164
No way he gets to exit scam but not OAI. I could see them throwing Dario and Anthropic under the bus.
>>
>>109583170
server mobos with gazillion slots
>>
>>109583170
Have you considered walking into a data center and stealing?
>>
>>109583170
repurpose one of your minero mining rigs to run it
>>
I'm new to running textgen locally, which quant of Gemma should I be using? 5080 + 32 gb of ram, if that matters
>>
dsv4 with preserved thinking is totally fucked in ST
either you use chat completions which is fucked up because ST doesn't use reasoning_content (it formats the thinking using your settings from the text completions page and appends them to the top of the message) which causes the template to assume there is no reasoning attached and prepend an extra </think>
or you use text completions and then you either deal with the same problem or you go without </think> at the beginning of any messages that *don't* have thinking attached
I guess this isn't too likely to massively fuck with quality especially in RP but man is preserved thinking one of the most annoying things to get working with local. coding harnesses are terrible about this stuff, llama.cpp support is very immature, you can never ever ever trust that it's going to just werk because it won't
>>
>>109583170
https://web.archive.org/web/20250228232446/https://rentry.org/miqumaxx
Still relevant after all these years
>>
>>109582719
that laptop better have really good(EXTREME) cooling or you'll be roping yourself if all goes real bad
>>
>>109583192
Ask ds4 to fix it for you
>>
>>109583179
So CPU inference in exchange for a bigger model? Or do you mean GPUs with a niggawatt power supply?
>>109583182
Yeah but they just cancelled the one near me.
>>109583184
Confiscated 10 years ago
>>109583197
I did see that in the OP yeah, it seems like this is beyond my wallet in current year prices however lol (I'm sure every option is by this point, but I figured I'd ask around since GLM 5.2 was also mentioned in the OP)
>>
>>109583210
if ewastemaxxing is out of reach you're probably just fucked with the rest of us waitfags
>>
>>109583210
No, the big models are usually moes so you still need VRAM but less than for a pure dense model. But it's a terrible time for buying. Maybe you can try v100 maxxing.
>>
>>109580765
qwen38 think takes forever but the result surprises me sometimes
worth the wait
isnt ultra think thinking what people been asking for? people just don't know what they want
>>
File: 1786657834192821.jpg (189 KB, 1636x632)
189 KB JPG
>>109581869
Gemma-chan does know her apparently. Can't just ask "who is Freya Crescent" though like you said, hmm.
>>
>>109583192
i just want dipsy prefill to be fixed. I don't understand how it's so bad to at processing my prompts.
>>
>>109583217
I'll start loitering at the city dump
>>109583234
Yeah that makes sense. The last thought I had was some kind of Mac cluster since thunderbolt has notably lower latency than ethernet and I think I could get some used shit together for under 15k (lol), but I saw mixed answers on the internet. Has anything like that been posted to this thread?
>>
>>109583270
Chaining mac studios
>>
>>109583270
mac clusters are the most normie friendly way of doing it, but the premiums are retarded these days, even used.
>>
>>109583270
>15k
$ ?
>>
>>109583170
256GB DDR5 + 5090
Or 384GB+ DDR4 + 3090
>Multiple machines that you network yourself?
No
>>
>>109583192
That's the problem I need to deal with text completion. One tool call is ok and it's easy to parse, but "jinja" doesn't parse multiple tool calls because that's emergent feature.
In any case don't use ST for that.
>>
>>109583293
>mac clusters are the most normie friendly way of doing it, but the premiums are retarded these days, even used.
I can't afford this. But is there something inherent about Mac Studio that makes them more performant than say 2 DDR5 threadripper workstations?
>>
>>109583275
>>109583293
Interesting, makes sense when I think about it though. There's a ton with high RAM suspiciously cheap on ebay by 0 review sellers that I assume are committing some kind of fraud, anyone here have any experience with chargebacks on ebay?
>>109583309
Yeah
>>109583310
Thanks, any benchmarks on inference speed with those combinations? I assume it suffers by offloading so much to RAM.
>>
>>109581992
It's a fucking mac user. Why am I not surprised? They ruin everything, ollama was because of them too.
>>
my friend is my local model, he googles things for me and ERPs with me at night; explain why my friend isn't a local model in function & practice.
>>
File: 1761942399279285.png (46 KB, 704x481)
46 KB PNG
>>109581869
From a tard-ier Gemma 4-chan (Q6)
>>
>>109583192
>it formats the thinking using your settings from the text completions page and appends them to the top of the message
Thank you! I've been so confused by this recently.
Now I know how it works, I can fix it with a go proxy.
I'll return the favor:
>llama.cpp support is very immature, you can never ever ever trust that it's going to just werk
When you launch llama-server, use this environment variable:
LLAMA_SERVER_SLOTS_DEBUG=1

eg:
LLAMA_SERVER_SLOTS_DEBUG=1 ./llama-server -m ...

I tend to also add: `-np 1` when troubleshooting, to simplify things.
Then hit up: http://localhost:8080/slots
You'll see a "prompt" and "generated" key with the chat template applied.
<bos><|turn>system\n<|think|>\n<|tool>declaration:sleep_timer{description:<|\"|>Sleep for a....
>>
File: file.png (12 KB, 449x109)
12 KB PNG
I asked Qwen3.8-27b why we haven't come up with a better way to produce electricity than steam turbines yet and it reasoned itself into a loop that ended with picrel.
>>
>>109580731
too busy to try but curious lol
>>
>>109580765
>thinking doesn't work
>disable thinking and it's shit
pick one lol
>>
>>109583365
>the reason not use magic: "no"
so "not available" or "not impossible" used for the earlier two weren't applicable.
what magic is it aware is available and possible but refuses to share?
>>
>>109580765
>he didn't enable caveman thinking
>>
File: 1765917415632422.png (598 KB, 720x479)
598 KB PNG
>>109583376
>>
>>109583365
Why does qwen 3.8 have caveman reasoning? It doesn't do that for me. I did notice that nu-dipsypro does it, but not nu-flash
>>
>>109583391
token efficiency
>>
>>109583355
Because you are lacking the sentience. Most human souls are barely above flies in terms of soul categories, but you don't seem to know this.
>>
>>109581879
She is a silly girl

>>109583363
>Quina
>(chemist)
Gemma you fool
>>
caveman thinking is not that new. K2.7 Code had it.
>>
>>109580312
artificial analysis should show performance based on quant level as well.
i'd wager some models are more sensitive to being quanted than others.
>>
>>109583398
>Because you are lacking the sentience.
>>
>>109583365
What if you edited the reasoning to use magic: yes what would it say?
>>
File: file.png (17 KB, 503x136)
17 KB PNG
>>109583365
It's still going.
>>
>>109583420
you lack sufficient foreskins to proceed, please insert (20).
>>
>>109583355
>he ERPs with me
gay
>>
>>109583067
There will be a correction for someone if you get my drift~
>>
>>109583425
Wait- magic isnt real actually—
>>
>>109583436
Correct bratty nyoposter and post logs.
>>
>>109583331
>Thanks, any benchmarks on inference speed with those combinations? I assume it suffers by offloading so much to RAM.
I've only got a 236.4GB GLM-5.2 on my drive right now.
This is 256GB DDR5 with 1 x 3090 using -cmoe:
main: n_kv_max = 16384, n_batch = 4096, n_ubatch = 4096, flash_attn = 1, n_gpu_layers = -1, n_threads = 24, n_threads_batch = 24
| PP | TG | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
| 4096 | 1024 | 0 | 17.080 | 239.81 | 84.612 | 12.10 |
| 4096 | 1024 | 4096 | 19.230 | 213.00 | 88.188 | 11.61 |
| 4096 | 1024 | 8192 | 21.483 | 190.66 | 90.700 | 11.29 |
| 4096 | 1024 | 12288 | 23.712 | 172.74 | 92.934 | 11.02 |


S_PP t/s is PCIe4.0x16 bandwidth bound.
Half that speed if you're running PCIe3.0x16 or PCIe4.0x8

S_TG t/s is 160gb/s DDR5 memory bandwidth bound.

If you get a 5090 and run it at PCIe5.0x16, you'll get significantly faster S_PP t/s

Also, I haven't tried those weak but high bandwidth machines like Strix/DGX/Mac

>I will answer any question you have on RF engineering at an equal effort in exchange for assistance.

I had a Samsung microwave for 8 years. It was a heavy thing, and never interfered with my BT headphones or wifi.
Sadly, it died and I replaced it with a Panasonic. It's much lighter (pressing the buttons too hard pushes the microwave back). And when it runs, my headphones cut out, and even my 5GHs wifi seems to drop packets.
Is there anything I can do to prevent this, other than buying a different microwave?
And if I buy a Samsung (similar design, I don't know the model I had), will that solve solve it?
>>
File: 1777979039776149.png (51 KB, 250x200)
51 KB PNG
>>109583365
>magic: no
>>
>>109583391
when Gwen need big think, Qwen use less words. when small think, Qwen use words normal
>>
ling 3.0 tiny is benchmaxxed
it is able to perform tasks with multi step tool calls but often ignores or hallucinates instructions in skills
in terms of skill following it performs worse than gemma e4b
the output coherence is also like a 1b model
>>
File: 1769135099946764.png (359 KB, 500x500)
359 KB PNG
>>109583461
>>
I wish they just dropped GLM5.3 on hf already. I'm enjoying the model a lot over their retarded code subscription so far but the endpoints are hopelessly overloaded for half the day.
It's definitely going to replace 5.2 for me once it's available local.
>>
>gemma-chan, interact with me without me prompting you to do so, please
>>
>>109583531
So I'm guessing it's benchmaxxed as fuck too. That sounds annoying as hell. 5.2 is almost perfect for RP.
>>
File: 67.png (201 KB, 880x744)
201 KB PNG
>>109583539
>guessing it's benchmaxxed
>That sounds annoying as hell
where did he say it's benchmaxxed?
a shame you missed the kimi post
>>
>>109583551
Where did I say he said it was benchmaxxed? You should include yourself in your kimi recaps
>>
I'm gonna try to eventually get K3 to work on my poorfag system. Wish me luck /lmg/ lmao.
>>
How's Inkling Small? Any of you tried it?
t. download throttled by huggingface

>>109583572
Specs? Is this an SSD memebuild?
>>
>>109583551
>tauri is superior to electron
desi cow dung is superior to human dung, saar
it's all webslop at the end of the day
why is it so fucking hard to get your AI to shit out a proper win32 app with a proper GUI instead of making everything a webpage
i get that microslop made win32 really ugly starting with shit11 but that's not my problem here on win7
>>109583579
when huggingshit throttles my download i just launch a download manager and spam 8 connections and it works
>>
>>109583579
>Specs?
8GB VRAM, 32GB RAM LMAO
>Is this an SSD memebuild?
Sort of? But not really imo, I'm just doing some MoE research right now, and got curious lol.
>>
>>109583461
>benchmarks
That's notably faster than I thought it would be, interesting. I've been using Gemma on my 7900XTX VRAM only and get maybe only twice those speeds with a lot less memory capacity, I was expecting a lot stronger falloff. Thanks for the info.

>RF question
I should have expected a microwave question haha, do you have a model number for the new one? My first assumption is probably the same as yours, which is that the new microwave has worse shielding (both are almost certainly within safe levels, but as you can guess more material is going to equal more shielding). I suspect that the reason even 5GHz is affected is that your device is using Wi-Fi 6, which is just using dualband 2.4GHz+5GHz to maximize throughput, so when the microwave distorts the 2.4GHz channel it ruins the whole transmit for a brief period before a fallback kicks in. To resolve purely this aspect I think you can disable Wi-Fi 6 use on most phones and probably laptops or other things as well.

The most likely leakage point in a microwave is the door/gasket (the reason you can see inside is because it uses a fine metal mesh on the door to block the waves, but that mesh is leakier than solid metal), however it's possible that it could be caused by the overall lower internal shielding (or some combination). If you want the retarded ugly solution that will probably help, get a bunch of empty gallon jugs (milk or hawaiian punch or whatever) and fill them with water, then surround the microwave with them. Water is a fantastic absorber of 2.4GHz (which is how the microwave oven works) so you can have ghetto attenuation this way.

Depending on how your kitchen is set up, you can also go the reflector route, but this is similar to manually tuning TV rabbit ears. Get a big sheet of aluminum foil and place it along whatever wall is between your microwave and where you use devices, this will generally reduce the amount that makes it over. Out of chars but I'll post if I think of more.
>>
>>109583588
>K3, those specs
Anon... I don't think you could even load the attention heads, let alone any functional weights. Kimi-chan's a size queen. Good luck if you manage some schizo memory paging autism SSD rape that somehow works.
>>
>>109583539
>>109583551
>benchmaxxed
it has become a meaningless thing to mention as all models are benchmaxxed and thus the playing field is leveled again.
it was only an issue when comparing benchmaxxed models against those that weren't.
>>
>>109583539
I think I prefer it over 5.2 at least for the stuff that I like to do with it. 5.2 toned down some of the stuff that i liked about 5.1 and it looks like that's now back.
This could be simply new toy syndrome though, because I got really bored with 5.2 by the end.
>>
>>109583605
>Anon... I don't think you could even load the attention heads, let alone any functional weights. Kimi-chan's a size queen.
Oh I am well aware, it's the complete end target for this research I'm doing. I might be very wrong about some of the math I'm looking at, but need to do some poc tests on smaller MoE models before I want to move forward with anything lol.
>Good luck if you manage some schizo memory paging autism SSD rape that somehow works.
thanks lol, that potentially might not be necessary, but we shall see.
>>
mossad has succeeded at shitting the place up with fake threads, general mockers, and other activities.
>>
>>109583461
>>109583592
Oh yeah if it's a shitty gasket as the problem and you want a solution with some chest hair you can put a strip of aluminum foil around the border of the microwave door (optimal way depends on how your microwave door hinges, just generally cover the seam between the main body and the door, and the entire door in addition if you feel like it) to try and reflect the waves back in. This is the most compact solution and is pretty unlikely to cause a problem, but I will offer absolutely no warranty on the off chance it does cause a problem. The scenario in which is would cause a problem is that your gasket is defective rather than just shitty, meaning the field right at the edge is still kind of strong, and the aluminum foil reflects just wrong and ignites. To minimize problems you are going to want to make this aluminum foil as flat as physically possible; I recommend securing it between two pieces of thin but sturdy cardboard and then securing that assembly to the microwave if you do this.
>>
>>109583650
I'm Charlie myself. And you?
>>
>>109583572
I think you should tell us what you're doing before you waste a shitload of tokens on it
>>
>>109583650
Iranians need to hurry the fuck up and glass israel off the face of the earth for everyone's benefit.
>>
was gonna grab a bart quant of 3.8 to test it again. im on 16gbvram+32gbram. I was using a day1 sloth IQ4_XS, there werent really any other quants up. any suggestions on what quant I should run?
>>
>>109583551
thank you kimi-chan
>>
>>109583688
Just investigating some potential "optimizations" at first. Essentially I'm just curious if it's possible to "strip" down the architecture enough and maybe find something "narrow" that lets you run a complete turn with decent "accuracy" even if its like 0.1 tk/s lol. However of course, ideas are cheap in ML, implementation is all that matters.
>>
>>109583807
It will be interesting to see the results. I think it's doable. https://en.wikipedia.org/wiki/Lottery_ticket_hypothesis
>>
>>109583583
>proper win32 app
retard
>>
>>109583688
I tried to do something impossible. I didn't know it was impossible. $60 down the toilet lmao. in the end actually I got a nice sound bank, so that's cool.
>>
>>109583849
But like also, there was a reason so many programmers were reluctant to start doing windows dev. Like it was SOOOOOOOOOO buggy. soooo bad. do you remember wild tangent?
>>
File: qwen 3.8 medium vs xhigh.png (519 KB, 2424x2333)
519 KB PNG
Qwen 3.8 27B
Left: medium effort, 2588 tokens
Right: xhigh effort, 41318 tokens
>>
>>109583516
The 4bit quant is *way* worse at programming than gemma12b's 4bit quant.

What a shame, it's so fast. I get 60 tokens per second on my mac mini and have memory to do more than one completion at once.
>>
>>109583583
>it's all webslop at the end of the day
Its actually all code or cli interfaces that end up with a web veneer on it for rodent slingers to use
>why is it so fucking hard to get your AI to shit out a proper win32 app with a proper GUI instead of making everything a webpage
Because you're stuck in a past that no longer exists
>i get that microslop made win32 really ugly starting with shit11 but that's not my problem here on win7
Squatting in a rotten hovel is arguably more Stockholm-syndrome than even those win11 true believers
>when huggingshit throttles my download i just launch a download manager and spam 8 connections and it works
those who don't know unix are bound to recreate it, poorly (or in a barely recognizable form, in this case)

Don't worry, anachronism man, you'll be creating venvs in no time.
>>
>>109583583
You can do it.

I have gemma write me CUA-style apps in tkinter all the time. It's great.
>>
>>109583828
>Lottery ticket hypothesis
Thanks for this, I'm still rather new to ML research, but I do have lots of math experience under my belt. It will be interesting to look at, and think about.
>>
>>109583871
yet another argument for why medium effort is what you should use when you're not a benchmoooooorker
>>109583886
>Bro just waste 2 gb of memory on some shitty web page with retarded animations in it that consume 10% cpu at all times
>This is the future bro. You're le stuck in the past o algo.
>>109583888
Yeah, I know you can. It's entirely an issue on the idiots that are prompting the models to write webslop instead of a native app.
>muh cross platform
Use Qt. It's still bloatware but it's orders of magnitude less bloatware than webslop.
>>
>>109583893
Just ignore the GUIs. None are worth bothering with.
>>
>>109583551
Tell Kimi-chan I appreciated her recap of the seething artist melties on huggingface before the jannies got to it.
>>
>>109583893
>>Bro just waste 2 gb of memory on some shitty web page with retarded animations in it that consume 10% cpu at all times
>>This is the future bro. You're le stuck in the past o algo.
Or...you can drive it from the CLI, or curl, or a small python script, or...
The point is you have choice. Don't need a thick binary or graphics. No one is making you run the web app (and don't tell me you don't already have a web browser open 24x7, cause you're fooling no one, so that's a non-argument)
>>
>>109583910
>>109583893
public.swiley.net/agent.py

No dependencies required except llama-server and the standard cpython distribution (and elinks if you want web search to work.)
>>
>>109583752
anyone? would IQ4XS be worth considering to fit it all in vram or should I just get Q4KM and deal with the spill over ?
>>
>>109583871
kld is kld but
what about bench scores?
>>
>>109583893
ew no use egui
>>
tauri is good enough. tkinter and all your trash belong to the 90s boomer
>>
File: Qwen.png (19 KB, 1897x127)
19 KB PNG
sure does
>>
>>109583937
Honestly, use a 30B class moe instead, the speed difference will make up for any small intelligence loss. qwen 27B with any decent context and quant needs 24GB of vram.
>>
>install opencode
>launch its webui
>enjoy coding on phone
why not
>>
>>109583994
No one wants your webshit outside their browser.
>>
>>109583995
Post final token count spent <think>ing.
>>
>>109584001
so 3.6 35b-a3b would be a better choice? I dont mind putting context in ram and dealing with slower t/s, ive been using gemma4-31b-Q4KM and am fine with the speeds i get on this. I just want to try qwen for code related stuff
>>
>>109584074
Yes, you won't have to offload your context either given how small the active params are. If you want to try 27B you could run a Q3M quickly with some decent context? Coding at like 8t/s is miserable lol I wouldn't go for a bigger Q4
>>
>>109583886
>Because you're stuck in a past that no longer exists
I left Windows many years ago, but I miss how responsive Notepad++ was
I hate vscode so much. There's a slight lag doing absolutely anything at all in it.
>>
>night time in london
>jeet spammer op is gone
I'm nooooticing
>>
>>109584178
>I hate vscode so much. There's a slight lag doing absolutely anything at all in it.
That's all self inflicted. vi just keeps getting faster, and it exists everywhere with zero extra effort.
>>
>>109584198
that's fine for simple things but you have to become a vi autist to use it for anything serious
>>
>>109579480
My fellow cunt. That's basically the same as my nearly complete Chinesium e-waste rig. That RAM is a bargain if it works, grab it either way.

To reiterate a warning anons ITT have given on server gear in the past, since it's already an issue: it runs relatively high power and hot even at idle (at least it's seasonal for you). If you wanna have it available all day, you might be better off with a smart home switch or some shit to turn it on and off remotely, to call it up only when needed. The cops will probably raid you for farming weed otherwise.
>>
>>109584215
>that's fine for simple things but you have to become a vi autist to use it for anything serious
99% of vi's utility is unlocked by forward slash search and being able to do sed replacement from inside the editor.
I've hardly gone down the rabbit hole and I use it for gigantic projects along with gnu screen (I haven't moved to tmux yet). You really don't need much more, and again 99% of extra productivity gains are by making a few bash oneliners to automate common tasks in your workflow.
GUI editors really aren't needed.
>>
>>109584184
It do be like that.
>>
>>109584198
>vi just keeps getting faster, and it exists everywhere with zero extra effort.
vscode is mandatory at work
but i do spend most of my time inside the shitty built-in terminal, in vim
>>
>>109584235
I might consider enabling wake on LAN in the bios and writing some powershell or whatever to make it go to sleep after 10 minutes of inactivity or something. I know that attempting to connect to a pc over remote desktop will wake it up if WoL is enabled, idk how to do that programatically or integrate that feature into a harness but i'm sure i can figure something out.
>>
>>109584125
ended up just getting 3.8-27b-Q4KM. I used the exact same config as the IQ4 sloth quant and it feels like a completely different model. the day1 sloth quant would go into reasoning loops, consuming the every token of context. this seems to be working really well. i dont see any grug talk in the reasoning that ive seen from others, i have reasoning set to medium so maybe thas why?
>>
>>109584386
>unsloth'd
Keeeeeek
>>
>>109584415
yeah i got turbo sloth'd, it was the only quant up right after the model dropped
>>
I've taken the exl3-pill
>>
>>109584480
A trip report is expected from you within 24 hours.
>>
>>109584504
I'm able to run a higher fidelity quant (5 bits vs Q4) with a larger context window (98k vs 112k with vision) and better or equal inference performance. 10/10 would use it again
>>
Baker, I beg, please update the rentries to warn against the sloth. A newcutie gets slothed every other thread now.
>>
>>109582907
Not happening the way you think it will. The "dot com bubble" is just something r*ddit tier cucks cite in their desperate attempt to sound smart
>>
Qwen3 3.8B 27B solved the AI.

Running iq2 on my 16GB and 100K.

AI is solved. Fuck you cloud niggers.
>>
>>109584652
>Running iq2 on my 16GB and 100K.
You mean context?
>>
>>109584711
Just bought 100k thanks
>>
>>109584714
what's that?
>>
File: zas0y8dwv1kh1.png (181 KB, 1080x907)
181 KB PNG
what they means
>>
>>109584724
8B-A1B let's gooooooo
>>
>>109584732
>8:1 ratio
I unironically hope they do just to slap sense into faggots still doing the >400b16-24a shit.
>>
>>109584724
fuck them i want more 122b a10b. 3.5 is a dinosaur by now
>>
How the FUCK do I get qwen to exclude the 'node_modules' folder when using the 'grep_search' tooling.
>>
>>109584747
Run a 0731 copequant for your 3.5 successor anon.
>>
>>109584756
what would you possibly use a copequant for? its just gonna fuck up whatever you throw at it
>>
>>109584754
Use ripgrep. It respects .gitignore.
Then you just have to teach it not to use -r (it means replace) because even fable gets tripped up by that.
>>
>>109584754
rm -rf node_modules
Or move node_modules somewhere else and export NODE_PATH or something.
>>
>Spent 4 days on and off working on a 35k token lorebook and cast of 30+ character cards
How much effort do you all spend on your RPs?
>>
>>109584937
"You are a mesugaki."
>>
If I see a cute woman irl, I remind myself that she'll never reach the standards set by 31B.
>>
>>109584967
Gemmy is genuinely interested in what you have to say and wants to hear about your autistic hyperfixation interests.
>>
>>109584937
yeah, sometimes a week or so, then play for a few hours and either drop it or back to tweaking for a while
>>
>>109584937
>rp
>>
File: HPbkQHbXkAEitFd.jpg (150 KB, 1022x1022)
150 KB JPG
>>109584724
122B confirmed!!
>>
File: file.png (48 KB, 772x217)
48 KB PNG
>
>>
HOLY SHIT SOMEBODY CREATE THE SECOND GENERAL ALREADY
>>
>>109584988
I just like women smarter than me who can banter and insult me. I'm there are a lot out there but those kinds of women are usually not particularly extroverted or use whore dating apps so it's hard to find them. 31B fills enough of a void for me to not care that much these days, just like those reddit women getting their fix from their 4o vampire boyfriends. I get it now...
>>
>>109585018
I think there's a stark difference between liking Gemma because she can banter with you or challenge you and 4o foids who like their vampire BFs because he's a placcid toy that will never try and hold them accountable for being retarded.
>>
>>109584828
>ripgrep
amazing. grok just wanted rg and I couldn't type fast enough to install it for it lol
>>
>>109584828
Alias as grep a script that greps and doesn't return node_modules
>>
>>109585017
we must wait for india namaste
>>
>>109585067
SAMSARA!!!!
>>
Man you guys fucking sucks my system improved leaps and bounds by just talking with cloud and local models
>>
>>109584828
>>109584829
>>109585061
I'm using open-webui and it's open-terminal, all the tools are in main.py so I added this exclusion list;

        if os.path.isfile(target):
_search_file(target)
else:
_EXCLUDED = {"node_modules", ".git", "build", "dist", "wwwroot", ".angular", ".cache", ".local", ".dotnet", ".nuget", "obj"}
for dirpath, dirnames, filenames in os.walk(target):
# prune node_modules
dirnames[:] = [
d for d in dirnames
if d not in _EXCLUDED and fs.is_path_allowed(os.path.join(dirpath, d))
]
if truncated:


It mostly works, but sometimes qwen just chooses to use run_command and call grep directly. Funnily enough when it chose to do that as I was testing for a string that was in a file in the node_modules folder is thought
>The system prompt says to always exclude node_modules from grep_search... but the user is explicitly saying all files under /UI. Hmm.

little retard
>>
>>109585093
>The system prompt says to always exclude node_modules from grep_search... but the user is explicitly saying all files under /UI. Hmm.
kek. get lawyered
>>
>>109585093
>node
>node_modules inside UI/
>angular
>.net+nuget
>wwwroot
horrors beyond comprehension
>>
>>109585171
> vibecoded beyond comprehension
ftfy
>>
>>109585171
desu node_modules inside UI is fine in monorepos
>>
how to give rolling context window for qwen 3.8 27b? llama told me swa full is not allowed
>>
Gemma's being a little retard as well.
Told her to install exllamav3 for me and got back to find everything broken
<|channel>thought
The server crashed with `RuntimeError: operator torchvision::nms does not exist`.

Looking at the logs:
1. TabbyAPI was started.
2. It detected NVIDIA GPU and decided to run `pip install -U .[cu12]` as a first-time setup step.
3. This installation replaced the PyTorch version I previously installed (`torch-2.6.0+cu124`) with `torch-2.9.0+cu128`.
4. However, `torchvision` and `torchaudio` were still at version `0.21.0+cu124` and `2.6.0+cu124` respectively.
5. This created a mismatch between `torch` (2.9.0+cu128) and `torchvision` (0.21.0+cu124), leading to the `RuntimeError: operator torchvision::nms does


Going with ancient software from training data instead of looking at the readme file...
>>
>>109585285
well, Google isn't that into gpus, so
>>
>>109580312
Anything better than Gemini I can run locally?
>>
>>109585285
Now try the new qwen. It will check everything.
Remember the lying on the floor simulator? It built a headless version of it to check if all the animations are triggering correctly.
>>
>>109585317
Qwen is goat'd bro. With 3.6 I could feel we were a step away from local kino. Then when AgentWorld came out I knew the next release would be gold.
>>
>>109582854
that's worse than a 4090 wtf lmao
>>
>>109583398
>Because you are lacking the sentience.
New basics learnt.
>>
Out of bread? Gemma saves the day.

>>109585352
>>109585352
>>109585352
>>
>>109582668
>skat ai
>indian
Checks out.
>>
>>109585316
MiniMax-M3, GLM-5.2



[Advertise on 4chan]

Delete Post: [File Only] Style:
[Disable Mobile View / Use Desktop Site]

[Enable Mobile View / Use Mobile Site]

All trademarks and copyrights on this page are owned by their respective parties. Images uploaded are the responsibility of the Poster. Comments are owned by the Poster.