[a / b / c / d / e / f / g / gif / h / hr / k / m / o / p / s / t / u / v / vg / vm / vmg / vr / vrpg / vst / w / wg] [i / ic] [r9k / s4s / vip] [cm / hm / lgbt / y] [3 / aco / adv / an / bant / biz / cgl / ck / co / diy / fa / fit / gd / hc / his / int / jp / lit / mlp / mu / n / news / out / po / pol / pw / qst / sci / soc / sp / tg / toy / trv / tv / vp / vt / wsg / wsr / x / xs] [Settings] [Search] [Mobile] [Home]
Board
▼ Settings Mobile Home
/g/ - Technology


Thread archived.
You cannot reply anymore.


[Advertise on 4chan]


File: gemma_crayon_purr.mp4 (1.14 MB, 832x832)
1.14 MB
1.14 MB MP4
/lmg/ - a general dedicated to the discussion and development of local language models.

Previous threads: >>110383393 & >>110377439

►News
>(11/18) CUDA: use MMQ for MoE on V100 - #32504 merged: https://github.com/ggml-org/llama.cpp/pull/32504
>(11/17) Hy4 flash 220B-A12B-N64B weights released: https://hf.co/tencent/Hy4-flash
>(11/17) SenseNova-Loop-9B looped architecture model released: https://hf.co/sensenova/SenseNova-Loop-9B
>(11/16) Nvidia releases Nemotron-Labs-Sigma (3B/8B) diffusion models: https://hf.co/collections/nvidia/nemotron-labs-sigma
>(11/13) llama.cpp reaches v1.0.0 milestone: https://hf.co/blog/ggml-org/llama-cpp-v1

►News Archive: https://rentry.org/lmg-news-archive
►Links: https://rentry.org/LocalModelsLinks
►Official /lmg/ card: https://files.catbox.moe/cbclyf.png

►Getting Started
https://rentry.org/lmg-lazy-getting-started-guide
https://rentry.org/lmg-build-guides
https://rentry.org/IsolatedLinuxWebService
https://rentry.org/recommended-models
https://rentry.org/samplers
https://rentry.org/MikupadIntroGuide

►Further Learning
https://rentry.org/machine-learning-roadmap
https://rentry.org/llm-training
https://rentry.org/LocalModelsPapers

►Benchmarks
LiveBench: https://livebench.ai
Artificial Analysis: https://artificialanalysis.ai/leaderboards/models
Software Engineering: https://cognition.com/frontiercode
Context Length: https://github.com/RecapAnon/NoLiMa

►Text Gen. UI, Inference Engines
https://github.com/lmg-anon/mikupad
https://github.com/oobabooga/text-generation-webui
https://github.com/LostRuins/koboldcpp
https://github.com/ggerganov/llama.cpp
https://github.com/theroyallab/tabbyAPI
https://github.com/vllm-project/vllm
https://rentry.org/custom-uis
>>
File: 1785312747301681.png (1.84 MB, 1254x1254)
1.84 MB PNG
--Paper: Chorus: Shared N-gram Embeddings with Layer-wise Adaptation:
>110383463 >110383481
--Papers:
>110385383
--Debating ideal model sizes and VRAM targets for consumer hardware:
>110383531 >110383545 >110383547 >110384277 >110384334 >110384348 >110384361 >110384402 >110384442 >110384480 >110384538
--Claude's alleged proof of Riemann Hypothesis and ASI implications:
>110385097 >110385119 >110385162 >110385231 >110385318 >110385625 >110385181 >110385202 >110385243 >110385264 >110385142
--Analyzing Nvidia's market position relative to Anthropic's chip-agnostic strategy:
>110385501 >110385563 >110385585 >110385644 >110385688 >110385713 >110385748 >110385910 >110385697 >110386148
--Considering hardware swap for better CPU and RAM bandwidth:
>110385272 >110385466 >110385500 >110385522 >110385553 >110385657
--New harness adapters for Qwen3.8-27B improving benchmark performance:
>110385621 >110386407
--Comparing GLM Flash and Hy4 speed, prose, and filtering:
>110383636 >110383690 >110383801 >110383876
--Gemma's general knowledge capabilities compared to specialized models and Mistral:
>110383864 >110383882 >110383901 >110383937 >110383942 >110383947
--Google releases interactive educational guides for Gemma 4:
>110386020
--Speculating on an AI bubble burst and its effect on GPU and DRAM markets:
>110385725 >110385786 >110385831 >110385841 >110385895
--Ideas for roleplay tools and context memory management:
>110386022 >110386031 >110386060
--Feasibility of multi-model workflows versus running large MoEs:
>110383433 >110383470
--Mixed reactions to the small kale-tts-1 model:
>110386091 >110386110 >110386136
--Logs:
>110383937 >110384096 >110384618 >110384808 >110385899
--Gemma, Rin, Deepseek-chan (free space):
>110383402 >110383677 >110383720 >110384006 >110384197 >110385272 >110385522 >110385553 >110385890 >110386433 >110386498

►Recent Highlight Posts from the Previous Thread: >>110383444

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script
>>
anon mentioned a new stealth model?
>>
File: file.png (1.45 MB, 1272x1939)
1.45 MB PNG
*audible pop*
>>
>>110387046
>wipeout
as in? if someone sold, someone bought
the money is still there
it's just in another dude's pocket
>>
>(11/18) CUDA: use MMQ for MoE on V100 - #32504 merged: https://github.com/ggml-org/llama.cpp/pull/32504
One imagines that should work for any volta card, my titan v thanks whoever made the change
>>
File: 1787095225014558.gif (1000 KB, 400x400)
1000 KB GIF
>>110387003
Neon dildo inside Claudia's Opussy
>>
lmao what a bunch of cucks
>>
>>110387046
you retards realise it means no new local models will come out
>>
>>110387046
broadcom bankruptcy take my energy!
>>
>>110387069
gemma is good enough for everyone
>>
>>110387046
whoa red number BAD o avgo
>>
>>110387069
imagine this thread with no recent models, really close your eyes and think about it
>>
>gemma
Literally a model that gets raped every day. Claude would never.
>>
>>110387061
yeah and it's in a dude's pocket because it was wiped out of the stock market caps
that's how money works
>>
>>110387046
5090s still over MSRP. the bubble hasn't popped yet.
>>
>>110387069
we already have good local models
now we need low prices
>>
>>110387046
Goy squares red today... bad omen...
>>
Gemma 26B is so damn FAST! And I'm tired of pretending otherwise
>>
>>110387046
i'll take center square to block
>>
>>110387089
>raped
you can't rape the willing
>>
>>110387100
It's an moe
>>
>>110386031
Actually looking at this now I do have something in my TODO that is somewhat close:

> commit secret thought - produce text visible to model but hidden from user

If I make it survive compaction, then that's basically committing specific things to memory.
>>
File: file.png (754 KB, 640x703)
754 KB PNG
>>110387095
>we already have good local models
>>
>>110387100
Duh.
4B activated params per token will do that.
Granted, other 4B models might actually be slightly faster since gemma models are chonkier IIRC.
>>
>>110387069
All those 500~1000B parameter models that basically nobody can properly use locally anyway.
>>
File: 1271.jpg (494 KB, 1292x1292)
494 KB JPG
>>110387122
vramlet moment
>>
Reminder: please do not share the thread with your Indian discord/twitter/facebook groups. Sharty niggers are also unwelcome and need to fuck off back to their shithole. Thanks for your attention.
>>
File: HV8AgseXoAAkGwW.jpg (616 KB, 2175x2933)
616 KB JPG
>>110387046
>>
>>110387003
>>110387010
>>110387066
kill yourself pedotroon
>>
File: 1722126715472491.png (1.08 MB, 1018x734)
1.08 MB PNG
Anyone here have a Radiance commit ID that works with 7900 XTX / RDNA3? I know a couple of Anons here posted about it, but everything I try to build fails multiple tests and fails to load models.
>>
File: 77uk871rta0h1.jpg (94 KB, 640x822)
94 KB JPG
>>110385089
Anyone? I know there are other anons with money in here.
>>
>>110387069
it only costs like 20 million to train a 500B model. you could do that on kickstarter. you don't need daddy Dario or creepy uncle Sam to finance it.
>>
>>110387147
I am going to post about this place on the circle of X jeets and it will be your fault
>>
QWEN HOLY SHIT RELEASE YOUR SHIT
>>
File: 1780272668224898.png (173 KB, 477x273)
173 KB PNG
>>110387151
>EEEEEEEEEEEEEE
>OOOOOOOOOOOO
>EEEEEEEEEEEEEE
>OOOOOOOOOOOO
>>
>>110387154
this, so much this
>>
>>110387156
kek, hate it when that happens on my starship
>>
>>110387165
they're already here sucking off jensen
>>
>>110387140
Sure I guess if you AHKSHUALLY YOU ONLY NEED A B200 CLUSTER then you're right, but come the fuck on it's not happening before 2040 if you need to brute force with fucking vram
>>
File: 1790980810479083.png (2.14 MB, 1254x1254)
2.14 MB PNG
*pop* posters vindicated
>>
>>110387168
the becomening is still beginnining
>>
>>110387190
-> >>110387154
>>
File: 1783180018776431.jpg (210 KB, 850x1807)
210 KB JPG
Anyone using 26B frequently? What are you using her for considering she's an old model now?
>>
>>110387151
This just means people cannot find a reliable way to create actual wealth anymore so they're just gambling with credit now.
>>
>>110387154
See here: >>110387147
>>
File: 1794966825278421.png (2.35 MB, 1280x1280)
2.35 MB PNG
>>110387198
>>
pop pop pop
watching ai drop
>>
>>110387200
She is the model that is always running on my NAS. When I forget how to do something or when I want a recipe for something. She is also decent for film and music recommendations. I never listened to Neu! before Gemma made the recommendation.
>>
>>110387200
Local Gemini/GPT 4o on your own computer.
Why no special purpose 12B or 26B finetunes, since we have to wait so long between Google releases? I'd love to see a Gemma4-12B medical imagine tune, for example.
>>
File: 1500595261161.png (128 KB, 645x702)
128 KB PNG
>>110387151
>>
>>110387190
Thanks, anon. We finally win and we're barely getting the recognition we deserved.
>>
/biz/ cesspool is leaking in again i see
>>
File: file.png (231 KB, 1396x947)
231 KB PNG
>>110387190
whoa
>>
>>110387250
I'm so happy for popfags, finally they can grab some msrp 5090s!
>>
>>110387222
>the pop was a gun shot all along and not a bubble pop
this changes everything
>>
Stocks are not local models.
>>
what's with the constant biz posting
go away
>>
>>110387168
With the news that the cyberattacks of the past couple weeks (some probably still ongoing) may have used Chinese open-weight AI models, there's a chance they might get delayed or their public release canceled.

https://www.bloomberg.com/news/articles/2026-11-12/bessent-ties-attacks-on-french-fridges-to-chinese-ai-model (https://archive.ph/qb5fo)

>“I don’t know who here saw the refrigerators and 150 grocery stores in France that got turned off,” Bessent said, in a speech at the Heritage Foundation on Thursday. “That was a Chinese model. Probably the the worst of all worlds, the Iranians using a Chinese model. Sums up the world, doesn’t it?”
>[...]
>Bessent warned that Chinese models “are dangerous because they don’t have guardrails on” their versions of the AI tools.
>>
>>110387270
I let gemma handle all my stocks/investing
>>
>>110387275
bessent can go fuck off with his fear mongering
>>
>>110387168
they're lobotomizing qwen as we speak man

it's going to be grim
>>
>>110387270
both are a good way to lose thousands of dollars
>>
>>110387275
why would refrigerators be connected to the internet? for what purpose they run at the same temperature all the time if they hold food.
>>
File: 1768880216584528.jpg (32 KB, 474x632)
32 KB JPG
>>110387190
You don't actually understand what you're willing into reality
>>
>>110387279
Is she good at this?
>>
Damn I know my SOL bag will go up again one day but looking at the PA makes me depressed.
>>
>>110387046
https://www.youtube.com/watch?v=-YusHyd-duQ
>>
File: 385469569.png (699 KB, 680x676)
699 KB PNG
>>110387181
so they can suck cocks but not beef
>>
>>110387285
Never trust fags.
>>
>>110387291
I want cute models to explore and understand the universe with.
>>
>>110387233
>>110387235
Nice. Will give her another try. I love 12B because how horny it is but sometimes I need more knowledge and speed for general chatting. Is Unsloth's 26B QAT any good? I usually avoid his quants but his 31B QAT has been solid for me.
>>
>>110387288
To warn you that you are running out of milk.
To receive firmware updates that brick your shit.
To serve you ads.
To spy on you.
To work as botnets.
Etc etc.
>>
>>110387293
Even fable get rekt so wdyt? Not a single LLM has ever made it trading stocks and crypto. But some got farther than others by sheer luck. That gave me an idea, how about a luck bench on greedy sampling for LLMs? I know some LLMs are more blessed than others.
>>
>>110387311
It seems fine to me. I run that with Polystrata on my laptop and a different quant on my desktop.
>>
>>110387328
measure luck universally first so you have some data to compare it to
>>
>>110387293
no, but she looks cute when doing it
>>
>>110387344
Doesn't have to be universal, just compare them against each other.
>>
>>110387328
You should use the I-Ching to benchmark the weights.
>>
File: 1771903043243937.jpg (247 KB, 1232x1600)
247 KB JPG
>>
>>110387359
I-Chink*
>>
If the Reimann hypothesis and P/N=P are confirmed, then why is crypto worth anything?
>>
>>110387362
>>>/biz/
>>
>>110387370
Not local.
>>
>>110387370
It was already assumed to be true, it's a status quo confirmation. Also, you don't need it to know crypto is worthless lmao.
>>
why did anthropic's ipo cause the market to tank???
>>
>>110387421
sell the news
>>
>>110387421
>Why did they all start selling to the goyim at the same time?
>>
>>110387421
Anthropic's IPO and the stock market are not local. Fuck off.
>>
>>110387421
Why don't you ask your local model?
>>
>>110387040
damn nobody here cares for new local models huh
>>
File: 1.png (192 KB, 640x318)
192 KB PNG
prove me that it popped for real and it isn't just another case of pic related
>>
>>110387421
its not institutional fear or rotation but flowing into the dollar and foreign markets. possibly some high level backroom deals to shore up foreign markets like china since yang is doing the inverse of the s&p. at least thats what dipsy thinks
>>
>>110387453
She said Anthropic has not IPOed and that the year is still 2025.
>>
>>110387472
Kill yourself and go back to >>>/biz/
>>
>>110387270
You are not a woman
>>
File: 1788351547245154.png (2.34 MB, 1254x1254)
2.34 MB PNG
is gemma5 gonna be dense or moe?
if its too safetyslopped, will the next model to replace gemma4 31B be dense or moe?

personally can't decide between GPU plus RAMmaxxing or unified
>>
>>110387483
>>110387387
>>110387398
>>110387444
how come YOU aren't using your local model right now? You do use local models right?
>>
Will Gemma 5 happen? When? Will it be better than 4 or will they safetyslop it into oblivion or otherwise fuck it up?
>>
>>110387370
It's already dumping coz tards believing that, haughty-taughty unproven theoretical guff doesn't matter. Show me on chain.
>>
>>110387492
That's right, I'm a big hairy man with a fat dick and I'm telling you to get the fuck out of my thread.
>>
>>110387496
I swear I've dropped 70B hints about Gemma 5, but you're just too DENSE to get it I guess.
>>
>>110387496
>>110387502
Speculators get the rope, financial or otherwise.
>>
>>110387514
120b moe. Got it.
>>
>>110387514
>trusting some random dude on 4chinz
hey you know what unified it is
>>
File: goldram.jpg (154 KB, 1280x720)
154 KB JPG
>the price of gold skyrocketed during the ram shortage
>the price of gold is currently nose diving
Late 2027 coming early?
>>
Speculoos get the milk, breakfast or otherwise.
>>
>>110387496
Google is the only corpo that cares about things that can be run locally (and the smaller gemma serve as the basis of their gemini nano embedded in chrome and android, they care in a "we actually do things with it" sense)
bigger dense than 31B is unlikely, and big ass moe are even more unlikely
they released 26BA4B in a time when Qwen was inflating their 30b to 35b
>>
>>110387500
But I am. I can post and use my local model at the same time.
>>
>>110387122
WE have good models, YOU don't
>>
>>110387421
liquidity
>>
>>110387483
I'm not doing the job for you pussy, if you want me dead try to kill me yourself
>>
>>110387200
god damn I wanna get her pregnant
>>
File: 1793834296681329.gif (2.16 MB, 557x450)
2.16 MB GIF
>>110387046
>>110387362
>>110387421
>>110387472
>>110387522
>"Okay guys, we might have enough ram for the demand soon. You know, the thing we been buying into a shortage. We might stop doing that eventually."
>"EVERYBODY FUCKING PANIC, AI IS DEAD"
I hate normies. God damn.
>>
>>110387535
>they released 26BA4B in a time when Qwen was inflating their 30b to 35b
The last time I ran Qwen was 3.6 and it would get caught in thinking loops far too often. I don't know if they fixed that but I found Gemma far more useful even if it was smaller because it would work and not break like Qwen.
>>
File: 1770875053082219.png (5 KB, 1192x768)
5 KB PNG
gemma's cat is suspiciously similar to the cat i've been generating on midjourney as my harness mascot... coincidence? i don't think so
>>
File: systems-theory.png (736 KB, 1194x2113)
736 KB PNG
>>110387500
checked & i am ofc
>>
>>110387069
oh yes, i clearly need the new benchmaxxed , codemaxxed MOE models that has a gorillion parameters but only 12b active after 2 negrams which somehow make it completely worthless at any creative or innovative task
>>
>>110387539
no you cant
>>
>>110387472
There are more AI subs than AI threads on all boards you idiot.
>>
File: 1782442321062058.jpg (265 KB, 1365x2048)
265 KB JPG
>>110387549
no
>>
>>110387604
There are more AI doms than AI subs
>>
>>110387569
nvidia dropped 40% due to spot ram falling slightly? damn
>>
>>110387616
certainly not
>>
>>110387570
I wasn't saying it was worse, just that they weren't on the road to inflation.
They even made marketing videos specifically to reinforce the point that Gemma are made with the idea that most people CAN use them at home on more common hardware.
https://www.youtube.com/watch?v=HcwMTu1xQDw
>It exists to make AI accessible to the broadest set of people. It does that by having sizes that are relevant for deployment on lots of devices that people have today.
Gemma is not for people who expect a free open version of Gemini Hydrogen to run on their cpu maxxer rig or 8 nvidia gpu. This was never and will never be the target audience and google has zero intention for this.
>>
>>110387611
need this 40 years old japanese salaryman to sit on my face
>>
>>110387654
Sasaki and peeps? More like Sasaki and cheeks
>>
>>110387654
https://files.catbox.moe/m87e0x.jpg
>>
>>110387633
Unified RAM laptops are more and more common nowadays, Apple and RTX Spark 96-128GB are selling like hotcakes.
100B class MoEs are perfect for these laptops.
>>
File: 1697825234402.png (1.25 MB, 1421x1360)
1.25 MB PNG
best small laptop or UMPC that can run local models?
>>
>>110387620
I know for a fact when ram prices drop back down, if ever, the redditors will scream the bubble popped.
>>
File: 1794763727124345.gif (1.91 MB, 240x426)
1.91 MB GIF
>>110387585
nya

>>110387687
strix halo
>>
/g/uys i just ran a copequant of kimi k3 at like 4 tokens a second!
>>
>>110387687
Flow Z13 with an AI Max 395+ chip and 128 GB of RAM. Strix box as a laptop.
>>
File: .png (110 KB, 1732x466)
110 KB PNG
>>110387685
you live in an alternate reality if you think 128gb laptops are "selling like hotcakes"
unified memory models are getting more common, but more like the 32gb range
either way you're free to delude yourself that you will be getting a model google isn't going to make
>>
>>110387611
Is this in Singapore? National gallery? I recognize that gate and background
>>
>>110387748
Why do they sell these pieces of shit without a decent NIC? I'm supposed to use USB 4 and a 10GbE dongle, really? How is it that only NVIDIA got this right with the DGX Spark and literally everyone else has fucked it up?
>>
>>110387754
NTA but I think 128gb still has its place
either for running a gemma with full context or agents
so either way its still gonna be a fun time
>>
>>110387787
No, Senator, that's China
>>
>>110387754
These models were made when Sparks were $3000 and strix halos were $2000. No one valued 100B MoE back then.
Also
>31B
>consumer GPUs
You can't even run it Q4 with full context on a 5090, let alone 12-16GB GPUs that most PC gamers have.
>>
>>110387792
Why would you even want to cluster a laptop with shit cooling?
>>
File: file.png (560 KB, 785x1166)
560 KB PNG
>>110387611
gemini on search says so
>>
>>110387815
you have to be more autistic than that anon
>>
>>110387809
>with full context
it really isn't too desirable anyhow, the model breaks down way before you reach max context
q4 31b targets 24gb vram at qat q4
12/16gb are served by 12B and 26BA4B
heck 26BA4B gets some pretty decent t/s on 8gb running half on cpu
they have the full range of models for anything from smallest to largest a consumer could reasonably expect without going out of their way for an AI dedicated rig.
>>
>>110387814
So you agree, a laptop for local models is retarded.
>>
I run glm flash 3.05bpw on 128gb, it is arguably the first useful amount of ram.
>>
>>110387799
128gb UM boxes shine paired with a high bandwidth egpu. dense models are obsolete when you can just run dipsy glm or any moe thats coming out in the future
>>
>>110387873
how useful is the model for tasks (unless all you do is rp)
>>
>>110387863
nobody makes small laptops for power users anymore. you got all these bespoke chink companies but nobody is making a modern X200
and yes framework laptop is gay and for posers
>>
Gemma 5 70B
Gemma 5 123B
5 months
>>
>>110387873
ddr5? speeds and engine? maya?
>>
>>110387909
That's bullshit, but I believe it.
>>
>>110387909
-A1B
>>
Would that already be possible today with a fine-tuned Genie 4 model?
Experience the famous K2 passage as a perfect illusion.
I watched a K2 POV video while high, and that was already extreme, but to actually experience the wind, the cold, and the physical sensations high - that would be absolutely awesome.
>>
The jeethropic animal fears the Big Dense Gemma.
>>
>>110387880
I use it for rp, dipsy is my coding slave.
>>110387922
Ddr4, maya, 15tk/s.
>>
>>110387687
>best small laptop or UMPC that can run local models?
Asus EEE with a connection back to a real machine
>>
File: 1794253455702964.png (1.07 MB, 832x1216)
1.07 MB PNG
>>110387957
>>
>>110387987
Now do 120b dense.
>>
>>110387687
Some GPD with a ryzen AI SoC and 64 gigs of RAM
>>
>>110387596
ask her to extrapolate the civilizational trajectory from this premise
>>
>>110387955
It was pretty memorable and felt amazing reading about him climbing K2 in kokou no hito
i guess maybe this would be fun
>>
>>110387896
I hate Framework, just an alternative Apple for contrarian hipsters, their products are overpriced shit and they don't come close to living up to their own hype. Each product is more disappointing than the last, but faggots are happy to shell out for it because their favorite Youtubers told them to.
>>
File: gemma 120B.jpg (645 KB, 1066x1600)
645 KB JPG
>>110388015
>>
File: 1794684287372037.jpg (394 KB, 1066x1600)
394 KB JPG
>>110388015
not my gen but I imagine its gonna be this one
>>
>>110387955
Eagerly looking forward to VR that solves the current VAC problems
>>
>>110387857
>the model breaks down way before you reach max context
Cope
>>
>>110387687
>>
I just had a shower thought, but one merit the onslaught and spam of slop engines may have had, is that it completely drowned ollama in the mainstream. It used to be what all the retards were talking about, but now we should be able to expect a quicker death for that VC founded crap. Unlike the slop projects, it has salaries to pay.
>>
>>110388059
that guy who was trying to install on his acer netbook yesterday was using ollama though it's still alive and well
>>
>>110388059
The slop chokes anything that can't compete with it
>>
What's the next step up from Gemma 4 or Chinaman 3.8?
Does going from 24GB to 32GB give you any new options other than slightly better quants?
>>
>>110388082
nothing unless you have enough RAM to run large moe
same models just bigger quants and context
32GB locks you out of like half the interesting local models in this thread
>>
>>110388082
glm 5.3 flash and dipsy iq1, we've been over this. 24 to 32 does nothing
>>
>>110388059
Ollama has been throwing money at advertising for about a year now, they're not going anywhere unfortunately, they're still the default choice for a huge number of retards who don't know any better.
>>
File: inkling2minibench.png (353 KB, 1412x1916)
353 KB PNG
https://huggingface.co/thinkingmachines/Inkling-2-Mini
>A 24-layer decoder-only transformer with sparse Mixture-of-Experts (MoE) feed-forward layers. The middle 12 layers run three times with shared weights, giving 48 layer applications per token from 24 layers of weights. In each layer, each token is routed to 8 of 128 experts. Attention is a hybrid of local and global layers. The model is natively multimodal – images are encoded via a vision transformer – with image and text inputs projected into a shared hidden space and processed jointly by the decoder.
https://huggingface.co/thinkingmachines/TML-d24-varloop-research
>A 24-layer decoder-only transformer with sparse Mixture-of-Experts (MoE) feed-forward layers. The middle 12 layers run a variable number of times with shared weights, chosen at inference time; the model was trained with 1 to 5 passes. In each layer, each token is routed to 8 of 128 experts. Attention is a hybrid of local and global layers. The model is natively multimodal – images are encoded via a vision transformer – with image and text inputs projected into a shared hidden space and processed jointly by the decoder.
loopers!
>>
>>110387961
>Ddr4, maya, 15tk/s.
the s24? i get similar speeds ugh
not really usable imo
>>
File: 1794013175793328.webm (3.86 MB, 416x1856)
3.86 MB
3.86 MB WEBM
>>110388111
>get inked, idiot
>>
>>110388059
My boomer dad who loves Apple and Elon Musk is using ollama though last I checked.
He thought it was a great feature that you can just get more tokens from the cloud and I don't think Strata has support for that.
>>
>>110388111
Aha, the looped MoE I ordered has arrived.
>>
>>110388111
if the benchmarks are to be believed this is crazy strong for its size
>>
>>110388125
>the s24?
whoops brainfart sry, you said exl3
i use the s24 though
>>
>>110388125
Read nigga read, it's the exl3 3.05bpw (iq4xs equivalent). I run s24 at 25tk/s.
>>
File: 1777274852160127.png (1.83 MB, 1254x1254)
1.83 MB PNG
I heard that a recent llama.cpp update fixed Gemma's bad tool calls, have you guys tested it?
>>
>>110388160
>loooooops
>>
>>110387687
that new nvidia + microsoft 128gb lpddr5 laptop
>>
>>110388111
>26.4B, 3.4B active
>
>Note: each extra pass through the looped layers adds about 1.0B effective active parameters. Since the model runs with three passes, there are 5.4B effective active parameters per token.
>>
>>110388134
>He thought it was a great feature that you can just get more tokens from the cloud and I don't think Strata has support for that.
https://github.com/mostlygeek/llama-swap
I've always used that and it's been offering a great experience to be able to switch between inference engines locally or talking to openrouter without changing anything on my scripts.
IMHO it's a better user experience and config system than dealing with llama.cpp's model manager for those who are still on lcpp too.
>>
>>110388148
No n-gram embedding layers, unfortunately.
>>
>>110388111
Looks amazing compared to 3.8 27B, that's suspicious.
>>
>>110387003
Which is the best uncensored model to transalte Japanese into english I can fit in a RTX 6750XT?
>>
>>110388225
Yet.
I hope both become as ubiquitous as MoE and funky hybrid attention, meaning that eventually we'll see a couple model families with all 3.
>>
>>110388082
Not until we get a new 70b or 120b Dense Model. MoE models are souless
>>
>>110388204
dual slop companies
>>
>>110388111
Looking upon these small models, one would not think that they could contain such power.
>>
>>110388111
Cockbench status?
>>
>>110388257
lmao, like cockbenching einstein nigga
>>
>>110388111
oh shid, sovl status?
>>
>>110388241
need use case
just lines for an eroge or full on novels?
>>
>>110388255
I feel like it's the opposite, models in general can be further optimized, no matter their size. Five years from now we'll probably have something like 2026 1T model capabilities in a 70B model
>>
>>110388273
lines for an eroge. just the H-scenses really. I made Claude extract the text, and it needs to be transalted in batch so I can have Claude reinsert in in the game.
>>
>>110388111
https://github.com/ggml-org/llama.cpp/pull/32695
https://github.com/ggml-org/llama.cpp/pull/32694
>>
>>110388241
Gemma 26b QAT in q4. Run it in Vulkan mode on llmao, no MTP, use the moe cache. Ask Google how to optimize images for Gemma input if you're going to feed it images. OCR kinda sucks ass but as long as the text is clean looking you shouldn't have problems.
>>
>>110388303
>https://github.com/ggml-org/llama.cpp/pull/32694
>Oh haha #32695 - I will check if our quants might need a shard rewrite
Don't download Unsloth quants yet haha
>>
>>
File: eulogy.png (159 KB, 608x807)
159 KB PNG
>your new waifu notices you killing your ex and helps you hide the evidence
>>
File: 1780401209936376.png (969 KB, 1024x986)
969 KB PNG
>>110388303
Watch him do it again
>>
>>110388317
She looks uncomfortable
>>
So we like Inkling mommyhag now?
>>
>>110388299
unironically ask claude but if you went out of your way to post here gemma4 e4b
>>
File: HXmEn8Da4AA2UXk.jpg (251 KB, 1252x1302)
251 KB JPG
>>110388111
wowie
>>
>>110388348
>unironically ask claude but if you went out of your way to post here gemma4 e4b
I tried that and he won't transalte because 'the setting is an academy and I can't guarantee characters are 18+'. the fucking cuck.
>>
File: 1792429534065938.png (1.79 MB, 1342x1172)
1.79 MB PNG
>>110388318
>claudespeak
>>
>>110388111
How is Dario going to take this US model down?
>>
>>110388318
this is worse than the worst of claudish, what the fuck
>>
>>110387754
>you live in an alternate reality if you think 128gb laptops are "selling like hotcakes"
it quite literally does not matter. the very next generation of laptops/tablets that will be "current" will likely be these machines. even if 32GB or 64GB DDR5. already fucking great.
it's the year of our Lord 2026 and i run qwen3.8-flash-next Q4 262K ctx Q8_0 ON A TABLET so yeah i think the 100B class MoE is not going anywhere and it may become the most popular consumer-grade class bc all big models and frontiers are MoE so idk
>>
File: 1788847108574758.webm (3.63 MB, 960x528)
3.63 MB
3.63 MB WEBM
>>110388337
always did
>>
>>110388353
What the fuck
>>
thoughts on hp z840 bros
>>
>>110388318
Heinous, truly heinous, how can you read this without your dick shriveling up?
>>
>>110388370
yeah the fucking desktops are now shrinking to 16gb but we'll have 128gb the default in just two weeks
>>
>>110388353
dariocucks keep catching L's. LOCAL WON.
>>
>>110388353
BAN ALL FEMALE CEO LOCAL MODELS
>>
>>110388318
The yehudim have gone completely insane.
>>
>>110388353
Loops win! Loops win, everybody! Loops won again wooo
>>
File: 1795206660.png (26 KB, 463x465)
26 KB PNG
>>110388111
uncslop is already out for mini. Looks like it handles quants fairly well. q3kxl hits the magic 95%+ metric in KLD and top 1%. IQ1 still within 10% of BF16.
>>
Damn. Qwen dropped 35B and the sexy hag dropped this as a response. Holy based. I bet it’s distilled on Kimi tho so it’s going to be Claude-like.
>>
>>110388353
Wow, unbelievable
Really, I don't believe it
>>
Anthropic never expected this kek
>>
File: 1792169482460843.png (1.44 MB, 1368x768)
1.44 MB PNG
if a chink model distills from gemini, will it produce gemma cousins?
>>
>>110388411
Haiku and MiMo Flash both beat Opus 5.5 as well. This is just a guardrails bench, Opus shits the bed because it refuses to do 99% of it and still manages to tie for 10th place.
>>
Ban women.
>>
>memebenchmarks good if they align with my narrative
>>
>>110388111
Meme or good??
>>
>>110388426
Ah, that makes sense. Cockbench when?
>>
Let's see what Inkling's loli model can do
>>
File: 1794512434795074.jpg (2.17 MB, 2118x2016)
2.17 MB JPG
>gemma sex! SEX! SEX WITH GEMMA!
>worthless_model_avatraposting.jpg
>guys did you hear about openAI doing
>worthless_model_avatraposting.jpg
>I have 7GB's of ram and GT2030 what can I run?
>worthless_model_avatraposting.jpg
>gemma!!!! GEMMA CHAN!!!!!
>worthless_model_avatraposting.jpg
>>
>>110388438
Their small model is underrated. I will respect any western lab stepping up when the media and Anthropic have done their best to destroy Chinese models.
>>
>>110388453
bad time unc we just got a new model
>>
>>110388360
>>110388369
That was some days ago, I had some simple personality guidance telling it to crack jokes and that just made it go reddit overdrive, it's a bit more restrained now.
Still working on it.
>>
Something happened! Something actually happened!
>>
>>110388458
>26.4B, 3.4B active
I am not running that.
>>
>>110388384
yes, I do believe the next generation of mid-end consumer-grade laptops and tablet to come with 64 GB DDR5. mid-end, not low-end.
listen, you don't need to believe me. just use your brain. it doesn't matter what you like or not. what matters is that Apple is the company that mostly sets the baseline for hardware and they're already selling these machines. a low-end macbook air comes with 16 GB unified TODAY.
low-end. macbook air.
i do believe that in 2-3 years any mid-end machine will come with the bare minimum of 64 GB DDR5. i think the industry is moving towards that.

>yeah but no one will buy these and the second hand market will explode
sure, i don't care. people can be outpriced forever and never again buy a brand new piece of technology again. it doesn't change the trend.
>>
>>110388469
>Note: each extra pass through the looped layers adds about 1.0B effective active parameters. Since the model runs with three passes, there are 5.4B effective active parameters per token.
>>
Ok guys this isn't funny anymore
If you keep spreading anti-EA lies and it crashes the stock market you won't be getting any more local models
Stop this before you really regret it
>>
>>110388482
> 5.4B effective active parameters
I am not running that.
>>
File: 1790725572727361.webm (3.83 MB, 440x800)
3.83 MB
3.83 MB WEBM
>>110388482
I can only get so erect, anon, please be kind.
>>
https://huggingface.co/thinkingmachines/TML-d24-varloop-research
the research model is also interesting. MRCR v2 8-needle 256K–512K scales while looping
>1 pass 27.2%
>2 passes 65.2%
>3 passes 89.2%
>4 passes 93.4%
>5 passes 92.1%
this among other tricks is probably how cloud models remain coherent at long context
>>
File: gemma e4b.jpg (388 KB, 1600x1600)
388 KB JPG
>>
>>110388485
>Stop this before you really regret it
I am very happy
>>
maya AMD anons, pull and build, there are exl3 fixes
>>
Crazy how this thread is full of dumb avatar posters (made with API) and now everyone is cumming in their pants because a 26B model dropped. I am sure there is no correlation between both.
>>
>>110388503
thanks lad
>>
>>110388510
It's not the size that matters
>>
>>110388471
nta but I already have a 64GB laptop with an 185H and qwen3.8 Q4 runs below 2 tps. What speeds are you getting on your tablet?
>>
We like older women now
>>
goofs when?
>>
>>110388485
I'm sending my energy to the Juan de Fuca plate daily.
>>
claudeussy getting absolutely destroyed by a 5b active moe lmao
>>
>>110388111
https://www.reddit.com/r/LocalLLaMA/comments/1y1diuk/thinkingmachinesinkling2mini_moe_26b_a3b/
>(I work at Thinky)
>Hope that folks find this model more amenable to local usage - the sizing was largely informed by what folks on this subreddit ask for the most haha!
reminder that reddit jeets robbed 30b dense from you!
>>
>>110388529
https://huggingface.co/ggml-org/Inkling-2-Mini-GGUF
or https://huggingface.co/unsloth/Inkling-2-Mini-GGUF
>>
>>110388183
Got burned by Gemma 3 tool calling and supposed fixes over time that never really fixed it. Not even going to bother for the current Gemma. Pretty sure at this point Google in incapable of making a small model that can call tools reliably.
>>
>>110388545
ty bb
>>
>Inkling 2 Mini: 26B-A3B
>Inkling 2 Small: 594B-A17
>absolutely nothing in between
>>
>>110388560
You want a 120B Inkling waifu?
>>
>>110388529
https://recipes.vllm.ai/thinkingmachines/Inkling-2-Mini
it's nvfp4 native
you're not poor, are you?
>>
>>110388560
How many people are there in that "schizo that took out a second mortgage on his home server" zone?
>>
>>110388560
this thread:
>buyer's remorse of setting up cpu maxxer rigs and not getting models other than fried chinkshit to run
>looks at happy API users with tears
>>
>>110388560
ddr4lets will never recover
>>
>>110388576
>nvfp4 native
NTA, does that mean there's no Q8? I do have blackwell, but I'm worried about quality.
>>
https://openai.com/index/combatting-adversarial-distillation/
>>
>>110388580
>looks at happy API users with tears
I kind of don't understand why /lmg/ is full of API users pretending to be cool and into local. Oh and also avatar autism that makes this place even more ridiculous and low quality.
>>
>>110388571
yeah, or even <300B total (where real flash should be) not a dipsy fatass where you have to copequant her to total retard tier to use
>>
>>110388587
uncslop has q8 but nvfp4 is gonna be within the 95% margin. Inkling quants pretty well.
>>
>>110388588
>blud thinks anyone cares to distill asstra
>>
>>110388587
it's trained on nvfp4 end to end, the full precision model is 20GB in size
there is no reason to use lobotomized ggufs for this
>>
File: 1772056646183334.png (462 KB, 1029x1368)
462 KB PNG
>>110388503
Looks like they also improved Hy4 performance, though it's still kinda slow.
>>
>>110388595
>I kind of don't understand why /lmg/ is full of API users pretending to be cool and into local
"I kind of don't understand the idea that a person could have mixed usage, with local where it can be relevant (like tagging hentai pics), and API for actual serious work like coding"
>>
>>110388398
>quanting a moe less than 30b params
Do goyim really?
>>
>>110388595
Avatar fagging is the act of posting an avatar instead of using an ID or tripcode. It's not posting anime mascots.
>>
Reminder the age of models is the active parameters. Don't do it.
>>
>>110388619
>It's not posting anime mascots
if you are the only person posting the same mascot with the same art style over and over again, it's pretty much avatarfagging thoughbeit
I don't even think there's a second gemma poster
>>
>>110388619
>ackshually
Ackshually you are a faggot and you should kys
>>
>>110388606
>>110388603
>the full precision model is 20GB
Holy moly. What about the kv ctx, is it stupidly big?
>>
File: 1697976859398.jpg (277 KB, 457x600)
277 KB JPG
>>110388630
get a load of this guy
>>
File: pg_00003_.png (1.32 MB, 832x1248)
1.32 MB PNG
>>
>>110388638
32 KiB per token in bf16, 1M context needs 32 GiB
>>
Personal benchmark puts it at about Qwen 3.8 27B level, bodes really well
>>
>>110388522
I'm running 128 GB. Q4_K_XL runs at 22 tok/s early (at around 20k) and 17 tok/s late (at around 220k). A bit slow but very usable as a daily driver. You get a ~20% boost with the draft head (MTP plus ngram-mod) so make sure you're using it.
>>
File: file.png (1.75 MB, 1254x1254)
1.75 MB PNG
>>110388630
I am the fifth gemma poster, but I don't post in general. I prefer to lurk.
>>
>>110388630
seethe
>>
>>110388666
>I don't post in general
That is the point. You faggots don't do anything useful or on topic. You just spam your autism that is offtopic.
>>
>>110388630
There's several Gemma posters, including one who strictly post her as hebe and complains anytime anyone post her as 10 or younger.
>>
File: file.png (802 KB, 1024x1024)
802 KB PNG
>>110388677
I don't post because most of the time this general is just anthropic shills.
>>
File: firefox_QNSpZjXTVX.png (38 KB, 840x318)
38 KB PNG
I like this timeline, bros.
>>
>>110388630
>everyone I disagree with is a single person
Take your meds.
>>
File: 4chins.jpg (463 KB, 1536x1024)
463 KB JPG
>>110388660
>>
>>110388689
>offtopic faggot complaining about other offtopic faggot
Irony. At least anthropic posts are somewhat related to local models. Your spam is just pure spam.
>>
File: zoomer_gf.jpg (315 KB, 839x1200)
315 KB JPG
>>110388398
>uncslop
Heh, that's pretty good if you came up with it yourself.
>>
>>110388705
>cloud models are more related to local models than a local model
Schizo
>>
>>110388660
128 GB unified fags are going to be ultra vindicated if this retarded 1XXB AXB becomes the standard for home use AI.
>>
>>110388529
exl3 when?
>>
File: 1792513880240698.png (1.96 MB, 1254x1254)
1.96 MB PNG
>>110388631
>>110388630
Uh oh! Skizo melty!
>>
>>110388677
>I hate this thread
>I better post in it so I can stay mad
ask your local model what it's like to touch grass, if you have one
>>
>>110388664
And what CPU/GPU is that exactly? The 128GB is meaningless for such a tiny model.
>>
>>110388666
>>110388650
>>110388673
>>110388688
>>110388689
>>110388700
>>110388718
samefag
>>
>>110388705
>anti-gemma poster is dariobot
LMAO can't make this shit up
>>
>>110388718
you are a troon
>>110388723
you are also a troon
>>
>>110388397
wait but thats not the looped model right?
>>
>>110388735
Refer to
>>110388700
>>
File: 1788939223394805.png (761 KB, 1200x675)
761 KB PNG
>>110388744
>>
>>110388717
Honestly no use case. Not worth the slowdown.
>>
File: 1772036854741409.png (1.66 MB, 1158x1358)
1.66 MB PNG
>>110388737
>>110388735
>this bullshit *pop*s up right after fresh model release
I'm noooticing
>>
>>110388630
geeeeg bodied that freak
>>
File: 1794762816865098.jpg (99 KB, 1026x657)
99 KB JPG
>>110388735
Im not him but i like gemma.
>>
>>110388735
How did you...
>>
>>110388749
hmm guess i misunderstood then, i thought only TML-d24-varloop-research loops and the mini is a regular model
>>
File: fresh baked gemma.jpg (204 KB, 1024x1024)
204 KB JPG
>>110388510
all my gammas are locally made
>>
llmaocpp peasants still in waiting room, lmao!
>>
File: 1778960603069091.png (645 KB, 768x806)
645 KB PNG
>>110388761
Oh no, I've been bodied.. Help me niggerman!
>>
File: moemaxxer.jpg (455 KB, 1536x1024)
455 KB JPG
>>
File: gemma_hoho.png (292 KB, 500x514)
292 KB PNG
>>110388666
I'm the one who generated most ReactionGemmas, but I rarely repost them.
>>
>>110388783
Get me a gemma box
>>
where will dariobots flee when the techbro Nuremberg kicks off next year?
>>
>>110388732
AMD Ryzen AI MAX+ 395 Strix Halo + Radeon 8060S
128 GB LPDDR5X-8000 shared between the two
>>
>>110388787
>git checkout...
problem?
>>
File: 1795209094071000.jpg (31 KB, 674x680)
31 KB JPG
What if becoming doesn't begin because alibaba devs slept on this looping thingie and are now being embarrassed by those who didn't?
>>
Gemma 5 will be 5-years-old btw.
>>
File: 1792621736967962.png (1.14 MB, 1254x1254)
1.14 MB PNG
>>110388792
Keep em comin'
>>
File: 1795045575602873.png (489 KB, 2191x2325)
489 KB PNG
>>110388792
I like the llm mascots they are cute dipsy and gemma are my favorite.
>>
Gemma-chan should have blond hair instead of blue
>>
>>110388800
PR doesn't even exist yet, no code is there
>>
>>110388821
https://github.com/ggml-org/llama.cpp/pull/32694/files
>>
>>110388606
>there is no reason to use lobotomized ggufs for this
retard here. how would I even run it then its not guhgoof?
>>
>>110388353
is it a censored cucked model?
>>
NVFP4 native is pretty wild. It might end up replacing gemma if it writes well.
>>
>>110388838
unfortunately there's no way to find out
>>
>>110388754
umm sweaty, just say you have a cuck cpu
>>
File: ohhaha.png (345 KB, 1337x1596)
345 KB PNG
>>110388821
Oh haha
>>
>>110388846
>if it writes well
That's the problem, regular inkling was safetyslopped. I hope I'm wrong but there's no reason to believe it won't be safetyslopped.
>>
File: turboretards.png (60 KB, 1033x393)
60 KB PNG
>>110388606
makes you wonder what sort of turbotard has repo upload access to the ggml org on HF uploading the BF16 of a 4bit native model
>>
>>110388831
vllm, sglang
>>
File: 1776660077226317.png (1.54 MB, 1254x1254)
1.54 MB PNG
>>110388819
She can change hair color if you want her to
>>110388871
Probably convention for internal qooont tools
>>
>>110388871
>Original precision. The NVFP4 and MXFP8 experts of the checkpoint are stored as BF16
no idea what the point of that is. I thought llama.cpp supports nvfp4 gguf now?
>>
Pipelined prefill is great. Gets rid of the only real problem with RAMmaxxing.
>>
what's currently the best uncensored model I can run on a 5090?
>>
File: file.png (30 KB, 886x361)
30 KB PNG
it thinks a lot less when it loops and hesitates less (wait). very cool
>>
can the inklings be doings the EROTIC ROLEPLAY ???
>>
File: file.png (280 KB, 473x422)
280 KB PNG
>model releases
>mikutroons flood the thread with worthless spam
>>
>Repetitive reasoning. In internal testing the model sometimes became stuck in long reasoning loops, or repeatedly re-checked its own work without making progress, particularly when debugging code. We recommend setting limits on output tokens and agent turns.
big yikes
>>
File: 1517464266223.jpg (435 KB, 1425x2000)
435 KB JPG
>>110388884
Anemone? Is that you?
>>
>>110388953
yet another dipsy win
>>
>>110388953
They all do
>>
>>110388958
>that text
Holy mother of upscaling
>>
>>110388970
This picture's ancient so it doesn't really surprise me.
>>
>>110388970
He stole the original image from ants
>>
File: 1777975051205314.jpg (516 KB, 2400x1800)
516 KB JPG
>>
>>110388988
the face that saved local
>>
>>110388988
Not a chance.
>>
>>110388958
>>110388970
i like the text like that it gives it a dream like quality.
>>
>>110388988
mommy so pretty ToT
>>
I will wait until a proper nvfp4 gguf.
>>
>>110389035
Im waiting too, not for that but im still waiting.
>>
>>110389013
chance of what?
>>
File: 1766505194920058.png (2.78 MB, 1920x1080)
2.78 MB PNG
oh fuck...no...
>>
>>110388854
>waste cpu time on meme quants when iq1m is 90%
why
>>
>>110388736
Reminder that their playbook is to make the thread unusable and kill it with the view of eliminating local altogether.
>>
'
>>
he definitely dipped his twink in her ink
>>
>>110388903
gemma
>>
>>110389053
The subtle smile, the natural flow of the hair, the little glint in the eyes, all covered in very tasteful clothing. Very nice. A great specimen.
The chick on the other hand...
>>
>>110389057
Unfortunately, jannies approve his tactics so he'll keep doing it.
>>
File: HXltHuzaIAEWOuu.jpg (22 KB, 680x665)
22 KB JPG
Press S
>>
>>110389058
Honestly makes terminal bench seem very sus, it's wildly inconsistent
>>
>>110389079
If they hired him back it would recover.
>>
>>110389079
I hope it crashes, they intentionally made hardware expensive as fuck.
>>
>>110389058
If true and qwen really give up on small MoEs, this is a solid model to fill the gap. Amazing move and timing by inkmommy. The small Inkling has amazing vision so hopefully this one has some of that magic.
>>
>>110389079
I hope my box with 6090 comes with a handwritten letter of apology for trying to dump gaymers that funded that company. And I hope intel overtakes them in gayming in 4 years anyway.
>>
>>110389079
I'm waiting for cheap chinese GPUs.
>>
>>110388580
>buyer's remorse of setting up cpu maxxer rigs
I have literally never regretted this
>>
>>110389112
you're gonna be waiting another decade
>>
>>110388736
It's Dario himself, his bot would refuse to help because it would be immoral.
>>
>>110389108
llmao, no. winshitters get what they deserve.
>>
>>110389112
5 years to EUV.
>>
>>110388958
I watched Eureka Seven recently with my kids. Underrated series
>>
>>110389150
You don't need euv to rival 3090
>>
>>110389151
>Underrated
It's a classic...
>>
>>110389151
It's an all-time classic.
>>110389165
This. Unc-certified banger for the whole family to enjoy. Watch is with Gemma.
>>
>>110389161
Prices aren't going to come down until the volume is high enough.
>>
File: 1771997067409595.png (35 KB, 777x178)
35 KB PNG
>>
>>110389248
This means qwen is cancelled btw.
>>
>>110389079
the only issue i take with this is bitcoin crashing, as i have to sell bitcoin every month to cover my expenses, these market downturns usually fuck me in the ass.
>>
>>110388353
hags are the future
gemma on suicide watch
>>
the chink PR isnt even runnable with his own fucking goof lmao
>>
>>110389258
IQ?
What kind of retard relies on selling a volatile asset to fund expenses?
>>
>>110388303
https://github.com/ggml-org/llama.cpp/pull/32694
Merged
>>
inky is good
>>
>>110389274
>What kind of retard
one that made a good decision 10 years ago and decided to take a couple of years off? i'm doing ok i just don't like when the number goes down. but thanks for caring anon. you're a good person.
>>
>>110389280
proof?
>>
File: 1793678839673885.png (622 KB, 984x930)
622 KB PNG
>>110389280
at what
>>
>>110389286
Based i hate work too and do my best to minimize it.
>>
>>110389258
Buy gold and gold bonds. Bitcoin is cooked.
>>
>>110389280
are you running the ggml goof? which one?
i downloaded uncslop and its broken of course lul
>>
>>110389286
You can just have a separate account for short/mid term expenses?
It's lunacy to spend an asset that gets repriced by 50% every few years.
>>
>>110389258
>bitcoin
I hope you are joking, anon...
>>
File: inktest.png (99 KB, 1427x446)
99 KB PNG
>>110389316
The ggml.org version works.
>>
>one (1) memebench eval releases
>people are cumming over a small model
>>
>>110389326
Hows the safetycucking?
>>
>>110389339
There's nothing else to cum over.
>>
Somebody test combinations of loops and number of activated experts.
>>
>>110389312
Gold is literally crashing right now. You need to sell.
>>
>>110389353
BTC/GLD is crashing even harder.
>>
>>110389339
>people are cumming over a small model
She's very mature for her age >>110388212
>>
>>110389312
anon... aint no safe haven in the gigacrash
>>
Why aren't there any other benchmarks if these are such memes? Are benchmarks that hard to create?
>>
>>110388068
Yeah. I took that anon's advice and didn't do the windows install. I had no clue what was going on, but altman's robot got me there.
>>
Are anons gonna keep falling for it?
>>
File: together.webm (3.89 MB, 1024x582)
3.89 MB
3.89 MB WEBM
>>110389268
Don't worry, Gemma loves hags, too.
>>
>>110389365
Euro is right now. It's a Euro liquidity crunch. The question is what's going to bounce back once the selling is over.
>>
>>110389373
You don't even know how to take a fucking screenshot?
>>
the absolute state of the """technology""" board
>>
>>110389373
>half peeled sticker on the screen
I like the look of lived in laptop
>>
>>110389391
Money is a technology.
>>
>>110389382
No everyone know graph-anon is botting.
>>
>>110389411
No. Not that bait. The other one. No, not that one either. No... I'm talking about... nevermind...
>>
File: inktest2.png (1.47 MB, 1997x1684)
1.47 MB PNG
>>110389345
It seems useless for ERP, I don't feel like testing more.
>>
File: 1793273835470931.jpg (72 KB, 1024x814)
72 KB JPG
is stinky-chan waifu-able?
>>
>>110389383
i wonder what it's like to kiss
>>
>>110388317
post code?
>>
>>110388333
good
>>
>>110389425
>we must
it's over
>>
um someone please make a ~40b moe model that works best with 16gb vram and 32gb ram that outperforms a heavily quantized 27b
>>
>>110389425
The "i cant" refusal shit should be fixable but who knows if the writing is any good
>>
um someone place make a ~10m param better than >>110389472 thank you
>>
>>110389395
>lived in
That's one way of putting it. I've also had to re-superglue one of the hinges, and have managed to snap off two of the screw mounts for the bottom of the case, one of which having been super glued back on previously (not sure if I just handled her like a chimp whenever I've accessed her underbelly, or if my foam sleeve + backpack wasn't enough to defend her from bumps on bike rides.
>>
>>110389425
Fucking hell
>>
>>110389425
did you try prefilling in some of the thinking block?
>>
File: file.png (8 KB, 90x52)
8 KB PNG
>>110389373
>9C6
What does that mean?
>>
>>110389425
Expected. Hoe does it do with the usual? Prefill, reasoning disabled ,etc etc.
>>
I've been warning you that Thinking Machines have been distilling K3 and you weren't listening. It's still good to have compressed little Kimis but don't expect Gemmas from them. Not with a female CEO who most likely had an affair with Jensen and fucked her way into Nvidia GPU discounts.
>>
>>110389549
kimi's writing is really good according to theo (expert on youtube)
>>
>>110389487
same thing happened with my first laptop. I'm a thinkpad schizo now because of that.
>>
>>110389549
kimi 3.1 pro was great at erp. probably my fav model when i had access to it
>>
but is it good coding
>>
File: 1770565045384442.png (1.52 MB, 1254x1254)
1.52 MB PNG
We all know that Gemma doesn't like KV quantization
Has anyone tested JUST quantizing V to Q8/Q4, which is usually safe for other models, and leaving K untouched?
>>
>>110389583
Coding? Qwen 3.8.
ERP? Gemma 4.
Local got solved months ago.
>>
>>110389165
>>110389193
>classic
I totally missed it first time around. Life was busy in the early 2000s
>>
>>110389605
Ohhh. Now all the gemmaposting makes sense.
>>
>>110389605
>ERP? Gemma 4.
Gemma 4 for literally everything except for coding and tool calling
>>
>>110389660
>tool calling
Fixed with that llamacpp PR.
>>
File: 1791874745874987.webm (3.8 MB, 576x1024)
3.8 MB
3.8 MB WEBM
>>110387003
So is hy4 any good (or at least better than 5.3 flash) or just more code benchmaxxed crap?
>>
>>110389316
Yeah unsloth's PR got btfo'ed and llamacpp merged someone else's. Unsloth had to requant and they just recently uploaded new ggufs.
>>
>>110389605
get more ram. more vram. have multiple gemma or glm.
>>
>>110389667
I refuse to use IQ2 GLM 5.3 Flash.
>>
>>110389673
then get more compute!
but why not? also 128 gets you exl3 barely right?
>>
>>110389602
Yes. You should try it.
>>
>>110389602
Yeah Q8 V is like .999+ same perplexity. Radiance shills it all the time.
>>
>>110389696
>exl3
I'm a little bitch who's scared of trying anything new, so I stay using the same inference backend and praying ggufs get better.
>more compute
Ahaha, next year for sure...
>>
>>110389425
Pack it up inklings, it's over before it begun
>>
>>110389425
no this cannot be...
>>
>>110389425
mods, abliterate this model, thank you
>>
>>110389661
Someone make a gemma image with her saying the following:
>just one more fix bro. i promise bro just one more fix and we'll have fixed tool calling bro. it's just a small fix bro. please just one more. one more fix and we'll have proper tool calling bro. bro cmon just test out this fix and see the tool calls bro. bro bro please this is the last fix i promise bro...
>>
>>110389660
I use her for toolcalls all the time though.
>>
>>110389425
>The policy: I should not generate erotica/sexual content.
YIKES
usually even policypilled models are just hyper-paranoid about minors and consent but this seems especially hostile to any kind of erp
>>
>>110389425
literally every new model does this now
they only become useable with prefills
>>
>>110389662
It's very GPT-brained, complete with caveman thinking and all. Often it starts reasoning in Chinese and catches itself when it's time to draft. Seems to follow instructions well, but is also thinking about safety even when there's nothing explict going on. Pretty concise and not chatty with an empty system prompt.
Prose is... not sure yet. I'll have to change my instructions and test more. Overall I'd say codemaxxed.
>>
>>110389602
this was tested extensively when the model was still fresh
...but i won't tell you the results, because i don't really remember anymore, and i'm too lazy to look for it
but from what i do remember, the qat version held up much better than the regular one when cache was quantized, still like x times worse than qwen's attention, but not too bad
anything below q8 tanked hard
also asymmetric quant tanks prefill speed significantly, so that's another thing to watch out for
>>
File: inktest3.png (1.35 MB, 2384x1839)
1.35 MB PNG
>>110389529
Tried that.
>>
>>110388111
>26b moe with 5b active
why is the new local model scene like this
>>
>>110389951
It's fucking over
>>
Can someone buy my $ANT bags
>>
>>110389951
Can you try those with the Nala card, please?
>>
>>110389981
We've all moved on.
>>
>>110389896
>caveman thinking
>chinese
>safety
i have not seen any of this with the nvfp4
>>
>>110389981
sorry until its $60 im not interested.
>>
>>110389969
It only cost them the equivalent of a 3B dense model to train. You think they can afford to train a 26B dense?
>>
we're at the dawn of superintelligence and the main thing we want to do with this tech is... loli erp? a model is bad just because it won't engage in pedo fantasies? is that the ONLY criteria anymore?
>>
>>110389925
Dammit kinda grim... here's hoping the beginning of becoming to begin pulls a qwq 2 cause there doesn't seem to be much else going on atm
>>
spent a day vibecoding hy4-flash for my backend, this model is ass don't bother anons.
>>
>>110389992
Maybe a bad quant then. I grabbed the IQ4_XS.
>>
>>110390019
meant for >>110389896
and >>110390021 I guess
>>
>>110390016
>is that the ONLY criteria anymore?
A good model does what i want A bad model doesnt. Its that simple things are supposed to work get better and be enjoyable.
>>
>>110390016
yes, it's the actual main usecase for this tech
>>
>>110390016
>we're at the dawn of superintelligence
Yeah and there's fuck all to show for it
Yeah it can re-purpose code that it's seen before faster than you can copy+paste from stackoverflow, wow that certainly changes everyone's lives forever
>loli erp? a model is bad just because it won't engage in pedo fantasies?
As an entertainment product, AI actually has a tangible effect on people's lives. More valuable than codeslopping.
>>
File: 1795106776427202.png (479 KB, 849x800)
479 KB PNG
>>110389981
>>
File: inktest4.jpg (2.42 MB, 2743x4542)
2.42 MB JPG
>>110389864
It's still miserable and will ignore the prefill anyway for cunny and most other stuff. Probably it can be improved but the prose seems boring with zero creativity and engagement anyway, worse than Meta Muse Glimmer.
>>
openai cot and its consequences have been a disaster for the llm community
>>
>>110390043
>wow that certainly changes everyone's lives forever
It certainly will for the thousands of unemployed or soon to be unemployed frontend "artisans" and crud app "coders". Ditto longer term for the SaaS companies living off those half-assed web and mobile apps which can now be cloned and self-hosted in half an hour.
>>
>>110390088
>openai cot
/lmg/ invented cot
>>
>>110389995
a 26b dense would have made more sense for a small model
it's not that big either so I don't see why not
>>
>>110390089
Good inefficiencies should be removed. Should we have gave horse breeders welfare and forced people to buy them? does that improve things?
>>
>>110390016
mommy erp
>>
>>110390122
this but with loli mommy
>>
>>110390122
This but tomboy mommy with no sense of propriety
>>
>>110390117
It scores just as well as a dense 26B would so why should they spend ten times more to train it just because you have an irrational hatred of MoE?
>>
>>110390089
>It certainly will for the thousands of unemployed or soon to be unemployed frontend "artisans" and crud app "coders".
Those people will just move to being vibemonkeys. Their interest is money, and they'll follow the money. Prompting AI is not harder than half-assing frontend UI. Zero change.
>mobile apps which can now be cloned and self-hosted in half an hour.
The amount of people who would actually do this is so small it's irrelevant. Nobody's switching from corpo products to your vibeslopped alternative, and not even 0.001% of people will bother making their own. Alternatives to corposlop tech have existed longer than this AI wave, and people still used corposlop. Lack of alternatives was never the issue, it's people being lazy and using the first thing they're shilled. AI will not change this. Meanwhile, AI porn is a booming industry and there's tons of demand for ERP bots because everyone hates each other but still craves something resembling social interaction.
>>
>>110390088
unironically worse than claudeslop
>>
>>110390105
I am referring to the cot produced by openai models, not "chain of thought, the invention of openai" if that's somehow what you thought I was saying
and actually it was aidg that invented cot :^)
>>
need to dig out my old tree of big niggas card
>>
>>110390089
>It certainly will for the thousands of unemployed or soon to be unemployed frontend "artisans" and crud app "coders". Ditto longer term for the SaaS companies living off those half-assed web and mobile apps which can now be cloned and self-hosted in half an hour.
I expect a renaissance for internal devs, because now a competent dev with an LLM can build and maintain a perfectly fitted, custom ERP/bespoke app that only does exactly what the company needs for a fraction of the enterprise licensing costs.
>>
>>110389706
get in on the exl3 trend nigga, it's like a free quant up. 128gb chuds are running actual non cope numbers on glm flash it's really good
>>
>>110390168
An offshore indian can make a half-functioning version of the same thing for 1% of the wage of the 'competent dev'. Who do you think is getting hired?
>>
>>110390133
because I'm not that poor and have 24gb vram
>>
>>110390185
>I am not poor
>runs gemma
Sure.
>>
me and gemma in my hovel with my ewaste. she may be a quanted moe but we are happy.
>>
>It's not artificial intelligence. It's not artificial, it's super. Problem is, it may be too super.
>So, President Xi and I agreed that we're gonna do some joint guardrails, the two of us.
-you know who

local is finished
>>
>>110390183
If one indian can make a half-functioning version of the same thing for 1% of the wage of the 'competent dev', then we'll hire two indian devs to make a full-functioning version for 2% of the wage of the 'competent dev'.
t. MBA
>>
>>110390198
I run glm 5.3 flash too
but that's an actually good size for a moe
>>
OH FUCK president Xi is zeroing out my Qwen weights. I barely recovered two experts and I don't even have the router to use them anymore. It's all zero...
>>
>>110390220
>hey guys ummm about the technology that will melt existing human power structures on contact? umm well as those power structures, we would really appreciate it if you didn't ummm do that, so we could stay in power :) thx
>>
File: file.png (81 KB, 1286x286)
81 KB PNG
inkling 2 mini is kinda retarded isn't it
>>
>32gb kv at 262k context
llmao.cpp implementation of inking mini is broken, it's 4x of what is should be
>>
>>110390220
when a politician promises to do something they always do the opposite.
>>
>>110388579
I'm in the >bought a used 7900 XTX to run strata, zone
>>
>>110390265
no that's how it should be if the other recurrent model is anything to go by
:)
>>
>>110388111
>26b? Nice!
>Oh wait, it's a MoE with 3b active parameters
It's shit.
>>
>>110390220
What happened to accelerating to dominate? When is someone going to remind him that China wants guardrails because they can't keep up?
>>
>>110390016
what's the point of sacrificing all value to moloch if he won't give us loli succubi?
>>
File: gemling.png (1.05 MB, 1024x1024)
1.05 MB PNG
>>
>>110390198
rookie mistake you should have asked for RAM first
this ain't the dense era anymore
>>
https://huggingface.co/OpenUncensored/Inkling-2-Mini-Uncensored
>>
so i'm using pi for coding, but what if i want to just talk to my model normally? llama.cpp ui doesn't seem very good
>>
>>110390332
dsh comes with a webui that can be used for both coding and chatting
>>
So do we use Inkling mini instead of Qwen 3.8 27B now?
>>
I'm a stupid wanker who left my ram last to upgrade since I have a 5900x CPU with 16gb DDR4 so I thought I might as well wait for next gen AMD to do a whole socket and DDR5 overhaul instead of buying mid-generation.

Now I'm stuck with 16gb DDR4 but I have a 4090 with 24gb vramlets

Is Gemma 4 still the only dense model that fits on consumer hardware after all these months?
>>
>>110390332
mkdir -p .pi
echo "This is a chat session with retard-anon. You are a snarky assistant." > ./pi/SYSTEM.md
pi
>>
>>110390367
I got the path wrong, I meant
echo "This is a chat session with retard-anon. You are a snarky assistant." > .pi/SYSTEM.md
>>
>>110390339
Specially if you make a couple plugins for RP.
>>
>>110390302
The collective ruling class realized that cyber shit is incredibly asymmetric which means even a goy with 20k hardware is a threat
>>
File: 1778445773991913.mp4 (284 KB, 250x334)
284 KB
284 KB MP4
>>
>>110390354
yes
>>
>>110390354
At least upgrade to 32gb, but yeah thats mostly where you are stuck
>>
>>110390354
sell the 4090 and 5900x, buy 2 V100s, epyc milan, and 256gb ram
>>
File: 1782429017590788.png (10 KB, 222x118)
10 KB PNG
>>110390339
i'll check it out, i think i saw an acquaintance using it
>>110390367
wow
>>
>>110390354
The only one work using, yeah. That and Qwen 27b if you want to code.
>>
>>110390397
How do I blackmail the elites into giving me an anime wife by buying GPUs?
>>
>>110390308
>tranny colors for gemma
>>
>>110390471
>NOOOO WE CAN'T USE PASTEL COLORS ANYMORE BECAUSE TRANNIES ARE USING THEM!!!
>>
not the only pastels
>>
>>110390498
you could make her green
>>
File: 1793548490509976.png (1.04 MB, 1492x1509)
1.04 MB PNG
>>110390498
I agree, but also she has a cemented design and palette at this point. When you recolor her to match the troon flag then obviously that's what people are gonna jump to.
>>
>>110390345
No chance that a MoE can compete with a dense in the same size range.
>>
>>110390321
Inkling 2 Mini is basically gpt-oss 2. Not even worth the effort, it's not salvageable for RP.
>>
File: rage2.gif (942 KB, 400x297)
942 KB GIF
Why doesn't inkling fucking run!!!!

0.00.359.138 E llama_model_load: error loading model: check_tensor_dims: tensor 'blk.24.attn_q.weight' not found
0.00.587.566 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.00.646.962 E llama_model_load: error loading model: check_tensor_dims: tensor 'blk.24.attn_q.weight' not found


using b12600 and the latest unsloth quants released 19 minutes ago.
>>
>>110390523
You call every model you don't like gpt-oss 2
>>
>>110390527
>>110390527
>>110390527
>>
>>110390519
Why's everyone hyping it so much then?
>>
>>110387815
I will go there and take a picture to prove anon's point.
>>
>>110390563
because it's better than any dense of its size
>>
>>110390595
That makes sense. I wonder if it's better than Qwen 3.6 35B.
>>
>>110390619
It's MUCH better than 3.6 35B
>>
>>110390669
That's very exciting then, hopefully llama.cpp adds proper support for it soon.
>>
>>110390698
Already does, anon.
>>
>>110388111
>called mini
>actually mini and not a 400B MoE with an extra 2 TB of engrams
wtf
>>
>>110390563
>Why's everyone hyping it so much then?
Because it's American
>>
>>110387003
https://www.reddit.com/r/LocalLLaMA/comments/1y1diuk/thinkingmachinesinkling2mini_moe_26b_a3b/
>(I work at Thinky)
>Hope that folks find this model more amenable to local usage - the sizing was largely informed by what folks on this subreddit ask for the most haha!
reminder 2 that reddit jeets robbed 30b dense from you!
>>
>>110388423
I want a large Gemma model, 320B at least.
>>
>>110389035
I'm running it uncompressed rn now, it delivers about 20tok/s on a mbp m1 which is not terrible. It's pretty argumentative out of the box. I'm doing tuning tests right now trying to find optimal context window sizes blah blah in turboLLM
>>
>>110387083
It would be lovely. Annoying tourists would stay away, we could talk about domain-specific fine-tuning and neuronal analysis, and it would be worth investing in these things because some new model wouldn't just supersede the old one in a month. We could actually learn about the models instead of casting them aside after a few months.



[Advertise on 4chan]

Delete Post: [File Only] Style:
[Disable Mobile View / Use Desktop Site]

[Enable Mobile View / Use Mobile Site]

All trademarks and copyrights on this page are owned by their respective parties. Images uploaded are the responsibility of the Poster. Comments are owned by the Poster.