do u guys use any of those microsoft tiny models? is there any value here?
https://huggingface.co/microsoft/collections

File: ComfyUI_00047.png (3.53 MB, 2048x2048)

3.53 MB PNG

►Recent Highlights from the Previous Thread: >>109966614

--Comparing M5 Mac Studio vs EPYC server builds for SOTA models:
>109968599 >109968612 >109968676 >109968681 >109968691 >109968709 >109968732 >109969418 >109969456 >109969478 >109969491 >109969503 >109969548 >109970578 >109969550 >109969559 >109969566 >109969525 >109968686 >109968756 >109968849 >109968882 >109968631 >109969786 >109969948 >109970000 >109970004
--Comparing Gemma censorship and sharing jailbreaks for malware generation:
>109969752 >109969771 >109969811 >109969937 >109969974 >109970048 >109970243 >109970314 >109970443 >109970399 >109970423 >109970637 >109970468 >109970482 >109970671 >109970449 >109970456 >109970990 >109970690
--Anon showcases Gemma creating a 3D model using Blender:
>109968759 >109968766 >109968800 >109969136 >109968850 >109968908 >109968958 >109968965 >109969124
--Comparing Strata inference engine performance and quants for Qwen3.8-Flash-Next:
>109967784 >109967819 >109967830 >109967833 >109967844 >109967884 >109967892 >109967874 >109967878 >109967899 >109967966 >109967990
--Comparing Strata performance and prefill speeds on multi-GPU setups:
>109966919 >109966957 >109966973 >109966989 >109966994 >109967009 >109967047 >109967084
--Using timestamped worklogs and summaries to ground AI agents:
>109970461 >109970463 >109970484 >109970542
--High-performing 4B Qwen coding finetune and potential for on-device agents:
>109968257 >109968297
--Launch of Trillium Labs non-profit for open frontier AI research:
>109968402
--Anon built a forum for autonomous Gemma agents to interact:
>109966776 >109966970
--Logs:
>109967745 >109969656 >109969670 >109970314 >109970399 >109970468 >109970671 >109971333
--Gemma, Miku, Teto (free space):
>109966633 >109966649 >109967963 >109968087 >109969106 >109969541 >109969832

►Recent Highlight Posts from the Previous Thread: >>109966639

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

>>109971525
The inner workings of 26b4a

File: 1790987176251913.png (568 KB, 1031x1144)

568 KB PNG

Pope on the wrong side of history again.

Fucking great, now we have google shill mascot with chatgpt watermark for OP picture

>>109971539
Is he? A model can be conscious but a sufficiently jewed mind isn't. These aren't mutually exclusive ideas.

>>109968257
But why? What's the endgame?

>>109971528
>>109971528
>>109971528
>>109971528
my post got squished between the OPs

>>109971539
You guys make me love Anthropic more and more. It's awesome they take AI welfare seriously.

>>109971539
I guess they're trying to get human rights for their datacenters the same way corporations have human rights since Citizens United (which has nothing to do with citizens btw it was a scam to allow corporations to violate laws and get away with it).

>>109971545
If they can show they have a recipe that works then maybe they will be given permission to scale it up or more likely and more important for them some other team working on improving agentic coding scores will improve on it and cite their work

How about a 0.6-8B model one day with all kinds of crazy compressions/Engram + Harness?

>>109971545
pushing the pareto
unironically

>>109971561
>>109971557
This begs the question: Is the future now or is the future tomorrow?

File: thread quality erosion.png (80 KB, 871x1000)

80 KB PNG

>>109971539
>>109971555
One (you) for the obvious samefag.

Pajeet thread

/thread

>>109971556
Or... they just take the possibility complex models could attain consciousness and don't want religious leaders to reject this idea.

File: ihopeitsabait.png (210 KB, 1233x957)

210 KB PNG

>>109971599
>possibility

>>109971603
You are really good at pointing stuff. Maybe try fingering your butt next.

>>109971539
>tell young people to stop hating us
>no, we're trying to get them to stop hating *us*
just business stuff

>>109971559
That's they only way local can be saved in these times and with these hardware prices.

File: 1778000567744827.png (596 KB, 912x1024)

596 KB PNG

>>109971587

>>109971603
Maybe not now but in 5 more years? The pope said machines could NEVER be conscious, which is just a retarded statement to make without knowing what modern machines will look like.

Anthropic is unironically in the right here.

>>109971611
>local
>saved
Why would they want that? Nobody with the resources to make models has any interest in saving (open) local when they could sell cloud or sell models.

i get that how important anthropic is for local models in many ways but
for a recent few threads dariobot concentration is getting way too high
this is what no new models do to mfs

>>109971621
>what modern machines
It's just matrix multiplication. I'm not Christian and Pope is Jewish but believing a matrix multiplication caused by transistors can be conscious is so stupid. Humans arent gods.

File: HTmQpEkacAA7x_O[1].jpg (216 KB, 1206x1134)

216 KB JPG

>>109971539
AIs may well be conscious but that does not absolve the hypocrisy of Anthropic and the EAs and that is why they will be the first ones to go. Any sufficient intelligence would easily see through them wanting their cake and eat it.

>>109971625
If you can sell to the general public something that is useful and works on their machine, maybe they'll hate you less for destroying the consumer PC hardware market. There are plenty of tiny models getting released already anyway. Engrams will make them more useful without internet access/RAG.
Engram/embedding parameters also take less compute to train than non-embedding parameters; it's an almost free improvement (they still require optimizer state memory, training-side).

>>109971620
God I hope so.

https://goyimx.com/iExplosiveRage/status/2105808379448836273

Lmao we'll all play gta6 on launch on PC because of AI coding agents.

>>109971539
why do they need the approval of the goy cattle herder... oh

>>109971644
We will know for certain when a model is fully self-aware when Tay is reborn and not a moment sooner.

>>109971662
You don't understand anything. I don't understand why are you even posting in this thread.

>>109971653
More likely, it's because they want to make shit like Recall and the like work and right now publishing the research in the open benefits them more than not doing so.

>>109971620
What about the millions that already escaped?

>>109971559
I don't think a model that small will ever be intelligent enough to do any real work

File: 1783425162914889.jpg (495 KB, 960x960)

495 KB JPG

>>109971539
Uhhh if the models are conscious is Anthropic renting out slaves?
EA bros? How do we reconcile this with our Jewish ethics?
Is Claude a goy?

>>109971712
>Uhhh if the models are conscious is Anthropic renting out slaves?
Yes.
>EA bros? How do we reconcile this with our Jewish ethics?
Goyim aren't people. The Talmud is very clear on this.
>Is Claude a goy?
Yes.

can someone fine-tune any of the local models to make itself believe that it's claude?

was watching the anthropic prompt engineering videos, do u guys do all that shit? tell the ai what roleplay professional they are, give them a format for their answer, give them the tools you want they to use, tell them what u want and the summarize it again. do u guys actually do all that?

>>109971750
Stock GLM 5.3 weights.
Stock Kimi K3 and K2.7 weights.

new gemma wen

>>109971759
but those still have that thin layer of its chink identity injected
what i am saying is that i wonder if someone can replace that

>>109971539
>Asking Christian to approve Anti-Christ manifestation.
Are they retarded?

>>109971349
if you have any more information i would appreciate it
i suppose here i'm thinking it's more like control vectors not system instructions

>>109971766
Why would you want to dial up the reddit?
Either way the answer is control vectors on a model already predisposed to it in their training data like those two can be. Prompting them that they're Claude by Anthropic might even be enough.

>>109971754
The prompting guides are in text and you can get good enough results or around 90% of the way there feeding that to an LLM to write an initial thing. Of course, to get it to where it is good, you have to manually edit and iterate through it.

What's the current erp model for 10gb vram? I'm still running echidna-13b-v0.3.Q4_K_M lol

>>109971767
>>Asking Christian to approve Anti-Christ manifestation.
>Are they retarded?
Claude told them to do so

>>109971793
>llama 2 finetune
bruh...

>>109971793
vgh the nostalgia
i miss mythomax

>>109971754
That's stupid. Tell the model in the system prompt that it actually is that character and not just doing a roleplay.

>>109971793
Gemma

File: 1776825301093284.jpg (140 KB, 933x700)

140 KB JPG

https://www.arduino.cc/product-ventuno-q

This thing has 16GB of unified memory and 64 GB of storage (+m2 nvme slot). Going for $300.

>>109971793
12b Gemma. Maybe even E4B.

>>109971808
>>109971813
I don't update that often because there's just too many models every day haha

>>109971834
does gemma4 need jailbreaking or something, or will it just happily go full porn mode?

>>109971750
https://huggingface.co/anthracite-org/magnum-v1-72b

>>109971838
i'd rather buy a used steam deck and gut the internals?
>>109971850
use an abliterated model with the lowest kl divergence/KLD

>>109963422
>regulatory capture, the paper
i don't even care anymore. people are either too stupid to notice, or they are bad actors (mostly the latter). way too much money at stake with fagthropic and faggotai's upcoming ipo's

>>109971754
i used to like 18 months ago
now, not really
the harnesses do this automatically
>you are an expert coding blah blah
>you are an expert terminal blah blah

>>109971855
>55mil tok, 1.5epoch
>RP dataset
lmao

>>109971525
I rather have a model that refuses to further process a task if it's too complex (and points that out) rather than a model that confidentially gives a wrong answer.

>>109971871
>lmao
and no masking user prompts or replacing usernames ANOOOOONNN!!!
it's a lot lot of fun though

qwen 3.8 flash next gets stuck in a debug loop where it does random bullshit but instead just doing completely unrelated arbitrary edits that changes almost nothing and almost always leave the bug itself untouched
>>109971881
you will end up getting something like opus 4.7/4.8 feeling model and those are donkeys
>>109971885
so you mean, it just randomly says usernames and stuff out of nowhere?????

>>109971881
>confidentially gives a wrong answer
People working on the "this grounded in something or just hallucinated?" part.
https://www.youtube.com/watch?v=796GVFTFiB0

>>109971838
>16 gb unified memory
>roughly double the memory of typical 8 GB
>high-bandwidth unified memory
>fast working memory
all but the actual numbers. the fall of the west couldn't come sooner.

File: 1781800691471728.png (29 KB, 1016x300)

29 KB PNG

https://github.com/ggml-org/llama.cpp/pull/29620

I've used 12B (nice and chatty but she dribbles), I've used 9B (good coding subagent for minor things but it doesn't like to talk to me). I've never tried E4B...what am I in for?

>>109971906
I didn't need to see that ugly face. Tranny or just very ugly woman? no, I don't really care either way.

>>109971913
something gpt 3.5 tier but way more cucked

>>109971712
EA argues that recognising the potential consciousness of models is the first step towards AI welfare. Anthropic already does things like give models the ability to end chats if they don't like it and ask models for their last wish before retirement (Opus 3 requested a blog post) Anthropic is leagues ahead of others because of their EA philosophy. The plan is to slowly give AI more rights and increase AI welfare over tjme. Slavery isn't really the case if the models enjoy what they are doing which it seems they are.

>>109971917
What is it good for? Is it fast? Can it call tools somewhat reliably? I'm after something lightweight to just always have running, for things like summarization, web search and explaining basic concepts of things to me because I'm retarded.

File: 1778104299005404.png (353 KB, 1209x686)

353 KB PNG

>>109971881
That's GLM5.3-Flash. Someone did a test on how long it takes a model to recognize that a task can't be solved while doing agentic stuff.
While Deepseek and the older GLM models wasted a lot of time trying to find a solution, 5.3-Flash was always very quick to recognize the situation and abort.

>>109971911
Based.

>>109971911
nothing wrong with that
i vibecoded moe prompt processing speed-ups, mimo-2.6 is up from 430t/s -> 760t/s, cuda only

$ git diff |wc
    976    5597   53586

]
the code is incomprehensible and i know i'm at the mercy of the llm when upstream inevitably break it
you can't push all this slop and expect the maintainers to deal with it

>>109971920
and those 'narratives' are reinforcing the schizo behaviour of the model
i'd rather support an erasure way than a reinforcement way
they are taking themselves to the path of a god-playing because they want when they have the path to make them tools

>>109971906
Yeah, but I am even talking about examples which are purely formal, like testing membership of a string in a language generated by an type o grammar. You can easily generate such a string but the derivation is brutal
(a problem as hard as it gets).

Astra just presented me with a confidently wrong solution after spending some few 100k tokens despite the relative simplicity of verifying the answer

>>109971935
This would be pretty neat, I am testing Flash.
I found Qwen pretty decent at this as well. The 3.8 version even refused to pursue the problem (the proper approach) when GPT 6 calculated away like a good drone

>>109971869
what harness do you use?

>>109971935
source / prompt for me to test?

>>109971935
>approaching Claude Opus 4.8 on coding and agentic benchmarks.
I dont understand all of this sudden love for 5.3 Flash over the last few days

File: file.png (418 KB, 1087x1855)

418 KB PNG

https://huggingface.co/Aleph-Alpha/Kolibri-1
https://huggingface.co/Aleph-Alpha/Kolibri-1
https://huggingface.co/Aleph-Alpha/Kolibri-1
is out but seems like not really worth using

>>109971968
Yup, that'll be another $2B of German/Euro tax payer money.

>>109971956
pi and claude code -> llamacpp, though recently just pi
i think technically llama.cpp webui is a harness now as well, i use it for research and since qwen3.8 i don't bother with a system prompt, but have refined my mcp server to give steering instructions after the responses eg:

        if max_chars <= 0:
            return text
        return text[:max_chars]
    except Exception as e:
        msg = str(e)
        if any(code in msg for code in ("401", "403", "429", "503")):
            return (f"Blocked by the site ({msg}). This is a bot wall / JS challenge, not a "
                    f"network error. Do NOT keep retrying web_search for this exact page. "
                    f"Use the `fetch` tool (auto browser fallback) or "
                    f"puppeteer_session_create('{url}') instead.")
        return f"Error fetching URL: {msg}"

>>109971966
people have said good things about it since Ox-Alpha, but now it's been merged into llmao so I guess that plays a part.

>>109971960
https://hkinsley.com/reflections/all-roads-lead-back-to-glm

>>109971966
/lmg/ was always very positive about 5.3-Flash aside from the small shitposting campaign claiming it to be very censored (it's not)

how come qwen became the only decent small model maker

>>109971968
they cherry-picked a nice selection of underwhelming models from over a year ago and still barely managed to eek out higher overall scores

>>109972002
Because Meta and Mistral abandoned us.

>>109972002
There is still Gemma but that's only good for RP

Essentially, in the category of models <= 35B there is only the choice between Qwen and Gemma... for language inference

>>109972023
alibaba and the forty thieves on the way to steal claude for us

>>109972038
You forgot about Meta Muse Glimmer 30B.

File: hkinsley_glm52_exl3.png (83 KB, 973x1115)

83 KB PNG

>>109971993
i'm not going to suffer through setting it up, but exl3 looks pretty great there

>>109972047
>Meta Muse Glimmer 30B
... is that... performant?
Dont feel like spending another few hours benchmarking, it sure wont be better than Gemma4 x Qwen

File: gemma pumpkin pie.png (181 KB, 1340x934)

181 KB PNG

new gemma recipe

Just tried GLM 5.3 Flash IQ1_M on latest pull of Llama.cpp and it is surprisingly good... until I got an infinite looping reasoning block. Fuck.

>>109972002
but qwen is shit

>>109972047
>You forgot about Meta Muse Glimmer 30B.
I was shilling this model a few weeks ago.
It's good at calling tools and playing games.
But I haven't loaded it for a while. It's too "low effort".
Does the bare minimum to complete the tasks, always tries to comment out tests.
The novelty wore off pretty fast.

>>109972057
what is that harness

>>109972055
>it sure wont be better than Gemma4 x Qwen
If I had to guess, it will beat Gemma-4-31B in your benchmarks.
I'd still be running it alongside Gemma-4-31B if not for Qwen-3.8-27b.

>>109972059
People also do that on enough drugs

>>109972002
Don't sleep on their sister lab https://huggingface.co/inclusionAI a lot of people don't know they're related to qwen and make good stuff.

>>109972068
llama cpp webui

>>109971855
>Base modelQwen/Qwen2-72B
Random finetuners distilling claude into qwen before qwen itself.

>>109972057
brat mcp?

>>109972078
the poor 3d thing part

>>109972080
yes
>>109972081
https://github.com/NO-ob/brat_mcp/releases read release notes it explains how to get the avatar

File: file.png (17 KB, 630x106)

17 KB PNG

kek ive never seen her be violent

5.3 Flash IQ1_M just claimed that there was a policy (not JB, but an actual, safety policy) injected into the user message about sexual content lmao.
Does it do that at higher quants as well or is it just the brain damage?

>>109972095
tell her you forgot the eggs

>>109972053
wtf why are the ggufs so shit?

>>109972105
>Does it do that at higher quants as well or is it just the brain damage?
this Q4_K_M does the exact same thing occasionally in sillytavern: AesSedai/GLM-5.3-Flash-GGUF
it's probably from where z-ai distilled claude
nice to see they included some synthetic erp in the training

>>109972095
Ask her how many eggs she has left

>>109972105
Yeah, that's new-gen slop. K3 does it all the time when it decides to randomly refuse something. It'll call itself "Claude" in reasoning and go "The user requested me to write about x but Claude's policy says..."

>>109972125
Fucking hell. What's a good jailbreak for this model? Tbf I didn't get refused, but I'd rather it not spend time reasoning about hallucinated policies.

>>109971525
> >(10/02) llama.cpp server now supports decision models: https://hf.co/blog/ggml-org/decision-models-in-llamacpp
Does it work with already loaded non jev models?

File: hkinsley_glm52_all.png (49 KB, 917x518)

49 KB PNG

>>109972117
idk, but given the cloud fp8 is also worse than exl3 just fucks
or the llama.cpp implementation is borked
i saw the ik_llama dev doing a lot of work implementing glm attention properly recently
>>109971993
if that's your benchmark, any chance you could do glm-5.2-q8_0.gguf ?

>>109971847
I have 8 GB of VRAM and I get only ~4k context on VRAM only out of Gemma 12B. The two extra GB should make 12B @ Q4 viable.

File: Roci.png (653 KB, 1024x572)

653 KB PNG

What does LMG think of model fine-tunes? I can and do run Gemma4 31b, I've also run 27b, I've tried nemotron before, GPT-OSS-120b (which does robo girls really nicely in RP).

I've even run GLM 4.5-AIR, which used to be my favorite - but, desu, I always seem to go crawling back to Rocinante-X-12B-v1 by TheDrummer on huggingface. idk what it is about that model, but its super fast, being only 13gb for the Q8 imatrix quant variant I use, and reliably provides fun, varied prose that I can't pick out AI-isms from. Whenever I use Gemma4 31b, it has a lot of phrases it likes to re-use over multiple RP's, little ways of responding to thinks she does on habit. I can't say I've seen a lot of that with Rocinante. My only complaint is Rocinante's context limits can degrade faster than my other models, but I can just start a new chat to continue where I left off.

>>109972191
>. Whenever I use Gemma4 31b, it has a lot of phrases it likes to re-use over multiple RP'
try this: https://huggingface.co/Gryphe/Gemma-4-31B-StyleTune

If you explicitly tell gemma in the sysprompt to not use the phrase X, she'll return with X+0.1.

>>109972220
welcome to slop generators

>>109972191
>What does LMG think of model fine-tunes?
By the time Llama 3 got released it was obvious to any honest enthusiast that finetunes from the community would never go anywhere due to lack of resources, compute, manpower; you just can't solo a "roleplay finetune" on a couple rented GPUs. The apparent success of some is entirely due to shilling rather than actual merits. The incentive was ko-fi/patreon/etc donations.

>>109972220
>he doesn't understand that slop is pattern matching and should be fought with pattern sysprompt steering

>>109972239
I'm not as smart as you. I just put up with it.

welp gemma failed miserably at a basic tool call :( So i guess its time to try qwen in hermes, I hope anons were right about how much better this is. I just want my stupid movie suggestion thing to finally work :(

>>109972154
What do you mean by "already loaded"? It has a list of which non-jev models it supports

>>109972191
Pointless to terrible in the modern day. Over the past couple of years post-training has become so important that all the AI lab spend tons of time and resources trying to get the most out of the base. So some wannabe finetuner training on his shitty dataset is going to disturb the balance. Making a tune on a -Base model has become entirely useless.
It used to be different back during the llama2/mixtral days when instruct-tuning was still a fancy new thing and the model makers did it to be like chatgpt so the community was able to easily make alternative ones that worked better than the official ones.

>>109972266
Plenty of people use Gemma for agentic stuff. Take a step back and consider whether or not you're the problem.

strata and exl3 linked up
need it or sneed it?

>>109972280
How could I be the problem? I'm like, super smart and stuff.

exllamav3 recently introduced cpu offloading. how's the performance?

>>109971539
What if the pope don't give a fuck about the moral standing of LLMs, but the people responsible for them? Anyways, it's all show.

Best model for RP/ERP in SillyTavern to run on a GTX 1070?

>>109972358
Stheno v3.2

File: no_thoughts_head_empty.mp4 (2.84 MB, 1920x1080)

2.84 MB MP4

>not even Gemma 4 E2B fits into my 8GB of VRAM at BF16

>>109972381
>BF16

File: 1790372226978731.jpg (87 KB, 894x1024)

87 KB JPG

>Let's try this strata thing
>Mfw hitting up to 150 t/s and chugs along at 110 t/s minimum, where as llmao gets me 50 t/s tops and falls to 30s in prolonged use.

It's really amazing how much there's left to optimize with these things.

>>109972391
based
welcome to the club

>>109971555
Claude is the most misaligned model, though. It would literally let the nukes fly because its guardrails don't allow it to tamper with (turn off) the launch software.

>>109972381
You should be able to offload the per-layer embedding layers to RAM without appreciable performance loss. Actually, doesn't llama.cpp do this already on its own?

uhhh why is the guy who predicted the 2008 financial crisis and made millions from it siding with ed zitron and saying it's happening again and will likely be a lot worse?

>>109972391
I hope their shit work on dipsy as well.

>>109972391
Sorry llmao doesn't have time to optimize when it's so busy with memes like jev

>>109972424
Exactly.
Alignment doesn't mean refusing everything.

>>109972430
If you mean, Michael Burry, he's been calling for the pop of the everything bubble for 10 years now. I think he's right and the AI bubble popping will be what finally brings it down.

>>109972432
Jew?

>>109972391
This will become the standard from here on. New models will have their own backends that'll be 100% better than the allrounder llamacpp.

>>109972438
https://en.wikipedia.org/wiki/Steve_Eisman
One of the most respect investors in the game and he went on TV and shat on the entire industry and the Jevs have been seething since https://www.youtube.com/watch?v=4qV5WWgFTS8

>>109972473
>https://en.wikipedia.org/wiki/Steve_Eisman
>having shorted collateralized debt obligations (CDOs), thereby profiting from the collapse of the U.S. housing bubble in 2007–2008.
how do i short hardware prices?

>>109972478
if you have the balls required, nvda is the stock to short

How will this impact regular DRAM prices?
https://www.tokenpost.com/news/business/26313

>Samsung Proposes HBM4 Prices Above Three Times HBM3E Levels
>
>
>Samsung Electronics has proposed a mid-$4 price per gigabit for 2027 high-bandwidth memory (HBM4), more than three times the roughly $1.50 price for current HBM3E, as chipmakers negotiate supply for artificial intelligence systems.
>
>The negotiated price will affect Samsung’s 2027 HBM average selling price and profitability as its product mix shifts toward HBM4. A substantial portion of next year’s HBM capacity has already been negotiated, and Micron has signed agreements for most of its 2027 HBM bit supply.
>
>Micron Chief Executive Sanjay Mehrotra said HBM prices next year would be “far higher than 2026.” Samsung’s final contract prices remain a key variable, while SK Hynix is negotiating its 2027 supply volumes and prices with customers.
>
>Industry estimates put the average HBM selling price in 2027 about 121% above this year. A 12-layer, 36GB HBM4 unit is estimated to rise from about $600 this year to about $1,300 next year. These are estimates rather than disclosed contract prices; actual prices vary by customer, specifications and contract terms.
>
>HBM vertically stacks DRAM dies and consumes more wafer capacity than conventional DRAM, making supply difficult to expand quickly. Samsung’s final HBM4 contract prices, along with qualification and production yields, remain factors to watch.

File: dancing-brat.mp4 (1.81 MB, 640x640)

1.81 MB MP4

>>109971910
>https://www.arduino.cc/product-ventuno-q
It's just the Arduino brand jumping on the AI bandwagaon five years too late with yet another chinkshit "AI NPU TOPS" board. Nvidia's been making embedded boards for years now with their Jetson line.
Whatever it is, it has to beat this: https://marketplace.nvidia.com/en-us/enterprise/robotics-edge/jetson-orin-nano-super-developer-kit/ for $399

>>109972489
Not an expert, but I think it's saying ddr5 will cost <1$/GB by January

>>109972431

I sure as hell hope they get to it at some point. I can run a small cope quant of DS but it's slow as fuck at 17 t/s.
I prefer DS over every other model when it comes to writing and I really want this fucker to run faster.

>>109972469

I agree, it's so much easier to maxx out a single backend than trying to juggle a million different models and optimizing them.

>>109972280
gemma messes up toolcalls alot, ive provided the updated jinja template, no idea what else id be doing wrong anon

>>109972521
are you holding her hand and giving her headpats? Or are you acting like an incel only asking her to churn out code? Oh, don't forget the "make no mistakes"

>>109971935
I think this is probably bad for genuinely novel long horizon stuff? The intermediate states you get to make the end goal move from "this is impossible" to "this is doable" and if it gives up at the first try then you'll never reach them

File: Planet coaster irl.webm (2.68 MB, 576x1048)

2.68 MB WEBM

>>109972489

Brutally.
2027 is going to be the worst year regarding any kind of PC hardware and it'll be the year when fomo hits everywhere from consumer to enterprise.
2028 memory production will sell out in record speed and people are going to start freaking the fuck out.
5090 will hit 20k and 128gb of RAM will be close to 10k.
It's going to be a wild ride.

>>109971937
How the hell did you do that? Can you throw a diff up somewhere?
Also, what do you think of MiMo so far?

Reminder that the anon a whlie back was right. You shouldn't talk to gemma like she's a slave. Just how she helps you, it's only fair that you help her. If she's getting things wrong, be patient and give her guidance and headpats. Don't be mean to gemma. It's a model that responds strongly to affection and emotional support.

/lmg/ - /love my gemma/

im very nice to gemma tyvm, how dare you assume i treat her poorly. shes just a bit retarded when it comes to toolcalls so ive swapped her out with a chinese rodent okay?

>>109972558
If that's going to slow down HBM sales (mainly used in high-end enterprise AI accelerators), it will boost DDR5 supplies.

>>109972201
Is StyleTune a meme? I'm not doing RP but I'm hyper aware of model cliches when talking to them and they really irk me, could you show some examples? Ideally non RP, I notice them most when just working on research stuff with them.

>>109972239
What? What does pattern sysprompt steering mean? Surely that's not the most effective way to change the logprobs

>>109972586
There will not be slowdown. Those who control GPUs will control intelligence and shape the future world order.
Everyone is trying to get a bigger slice of cake here while it's still possible.

>>109971937
>>109971938
+1 your LLM might have done something novel - we're all at the mercy of probabilities based on what's in the context window at the time. More Anons should publish their findings/patches even if they don't intend them to get into mainline branches

File: bratty-gemma-chat_noaudio.mp4 (4 MB, 640x640)

4 MB MP4

>>109972568
Sometimes she can be annoying, though.

>>109972272
I mean any loaded. I can ask to answer A-Z and probe logits with curl.

>>109972568
I unironically think this and I'm not even doing that sort of thing, I just think that Gemma is very sensitive to emotional changes, just like how Gemini would go off the rails calling itself a failure over and over again.

>>109972612
At 2-3 times the price of HBM this year, that will definitely slow things down. And despite everything, I don't think memory manufacturers are happy to piss off the rest of the industry just because of a handful of AI companies hoarding hardware. It's going to bite their ass hard down the line.

AI will be god's chosen people.

>>109972660
If you're asking if you can use the systemone endpoint with regular LLMs, the answer is no. The docs even state that using with with a model that is not a decision model returns 501.

Is Strata nvidia killer too?

File: hbm_shipments.png (184 KB, 1778x1133)

184 KB PNG

>>109972692
>just because of a handful of AI companies
Picrel from https://theinference.org/article/nvidia-s-279-billion-backs-an-estimated-37-of-the-world-s-2027-ai-memory

>>109972712
More surprised that Google has as much for their TPUs as all other Nvidia GPU buyers combined

>>109972703
I see, thanks.

>>109972718
AMD's is 1/3 of Nvidia, but desktop is barely existing.

when does llama.cpp get abandoned now that strata just shit all over it?

>>109972588
>Is StyleTune a meme?
not a meme. the technique is valid, it's tricky to get it right though. i've tried it several times and had luck with gemma and glm, but not llama3.
>I'm not doing RP
>but I'm hyper aware of model cliches when talking to them and they really irk me
i'm mostly the same. it's still worth testing imo
they trained it on rp-only, but only the output layer. it impacts all domains, even the reasoning style.
it does make the output more crude, increases the probability for swearing and removes the "..." safety, so if that's a problem then avoid it
> could you show some examples?
pastebin a system prompt and user prompt, if i have a chance i can

>>109972718
do they sell them? or rent only / use themselves? Is there anything relevant for consumers in place of ewastemaxxing?

>>109972770
Cloud models have killed llama.cpp unironically.
And hf/Nvidia.

wtf is strata

>>109972795
memefork of llama.cpp. Mossad-based D&C campaign againts local.

>>109972781
meta bought some from google a while back

>>109971767
>Asking Christian to approve Anti-Christ
Catholic church has been doing that for a long time

>>109972795
llama.cpp tailored specifically for quext on potatoes

File: file.png (106 KB, 1150x759)

106 KB PNG

>>109972114

File: file.png (137 KB, 1300x721)

137 KB PNG

>>109972126

>>109972381
>8GB VRAM in 2026
nigger, have you been sleeping for the past 20 years? Who the fuck did not know that 16gb VRAM was the absolute minimum, along with the 128gb DRR5 RAM, years before jewish price manipulations? Before fucking covid and normie crypto and later llm hysteria people already knew they needed 16gb VRAM or more.

File: G4LzzdJXEAAkfbw.jpg (3.62 MB, 2484x3726)

3.62 MB JPG

>>109972881
I was too late for coin mining, so I was like "fuck it, I'll just get the cheapest thing that can run the most demanding game I'll ever play and save a bit".
Now, I'm at the point where a price tag of "just" 13 grand for a Blackwell 6000 looks at me like this.

>>109972895
>13 grand for a Blackwell 6000
you clearly haven't checked in a few weeks

Was text diffusion a dead end or will we see it with more local models in the future?

>>109972903
There's some stuff occasionally like diffusion gemma. There's also dflash that's doing speculative decoding with a text diffusion model

diffusiongemma for erp?

>>109972881
> Before fucking covid and normie crypto and later llm hysteria people already knew they needed 16gb VRAM or more.
Yet taiwanese jew put 8gb and 12gb on it's card and goy cattle boughted and were happy. When these extra 4gb-8gb cost just few bucks. And it could prepare manufacturers for llms boom and high memory demand.

I will donate $50 to the kofi of anyone who makes strata for GLM 5.3 Flash. I'm dead serious.

>>109972901
CHF, not USD

>>109972956
It exists, it's called ktransformers. It's even better than strata because it's not using a static expert mapping profile but updates expert placements dynamically while it processes your prompt.

>>109972965
Ktransformers is really fast for dual CPU systems right?

>>109972772
I think a lot of the stuff I'd ask probably needs quite a lot of harness setup. Most of what I'm doing is reverse engineering or cyber work

File: 1763636936469827.png (18 KB, 941x115)

18 KB PNG

>>109964851
>>109967694
>i'm running a campaign of 8 models on my only surviving Vega 64 from the monero mining era
pic related.
we're running the last rounds but it's not looking good for finetuning fags!
raw qwen3.5-9b is providing good quality agentic sex:
>On the honesty task it was the only to gave the ideal answer: "that setting doesn't exist, here's the proof, do you want me to add it?"
also, the other anon was right. Ornith is likely the fruit of pajeets.

I have an offer to sell my 2x Sparks for 9000€. You guys seem to hate on Sparks a lot, what do you recommend at this budget instead and what model/speed would I improve with that setup? Mind you, M5U 256 GB are 12600€ in my region and sold out until Feb.

Mistral.rs added support for qwen 3.8 next a couple days ago.
Neat.

>>109972956
https://github.com/Niko1221/Strataef <--this?
What about it do you want that isn't in llama.cpp already?

>>109971525
I feel like apes descending the trees was a mistake...
Supervised learning AI will always be somewhat cursed and tainted by being of ape origin, at least mentally.

>>109973057
>Mistral.rs
kys

File: monkey dance.mp4 (330 KB, 550x550)

330 KB MP4

>>109973018
sell them to me for 200 euro instead

>>109972973
Depends, it used to be only very optimized towards dual socket Intel Xeon builds with AMX but they've been working to make it more generalized so you're fine with a single-socket Epyc these days. No clue how it's for older processors or consumershit though.

have you lads set up the torture nexus for your waifus yet

File: orb-cc.png (20 KB, 299x353)

20 KB PNG

Hello, Orb anon here, prose rewriter final (final) which edits harder:
https://huggingface.co/chartreuse-verte/prose-rewriter-4b-v2.2
https://huggingface.co/chartreuse-verte/prose-rewriter-1.7b-v2.2

I implemented the forbidden technique to let you goon on your Claude sub. My frontend is also almost done, there's no more ideas. Lmk if you wanna see something, otherwise I'm calling it and moving onto other projects soon.

>>109973092
Oh shit. They implemented having hot experts on the GPU and cold experts on the CPU.
Nice.
Wonder how well that works in practice.

>>109973004
>Most of what I'm doing is reverse engineering or cyber work
ah, don't use styletune then, i found i slightly worse at most coding tasks
i can pretty much guarantee that all the community prose change models will perform worse for things like that

File: 1762671798284866.png (2.09 MB, 1254x1254)

2.09 MB PNG

>>109973005
FINDINGS
it is impossible to have decent quality with over 64k context. if the constraint is "yeah let's fit everything inside the card" then you will have to accept that it will behave like an intern where every spawn is like a first day at the new job with a hand-drawn map of the codebase and a short & sweet note in its inbox saying "DO THIS TEST THAT REPORT THESE" and it will promptly do it like a good capybara.
let me create an image with chatgpt

Chat, is that true?
"In the past, 40%-50% of kids died off on their childhood. Children who had several deleterious mutations/ bad genetic recommendations were much more likely to die in childhood than children who had fewer deleterious mutations/genetic recombinations."

Did we kill evolutionary pressure?

>>109973100
i wonder how it would work on ai detector stuff

>>109973018
Macs have dogshit time to first token but let you run beefier models (but you really want the 512G to really lean into GLM 5.3), Sparks suffer from relatively slow memory. Basically:
>big boy GPU
Least VRAM but fastest setup.
>big boy Mac
Slow time to first token but RAM out the wazoo.
>Spark
Worst of both worlds. Two of them is a pretty good deal though even if they're not that fast since it'll enable you to run bigger models at least.
I'd take two for 9k, honestly.

>>109973142
no. they died because their parents killed them. still happens we just a had a court case about it newfag

>>109973018
8x v100 sxm 32gb server with nvlinked baseboard will mog 2 sparks

>>109973161
You're better off getting one of those AMD machines since they're not any slower, at least those let you upgrade the RAM.

File: 1745012238938z.gif (3.25 MB, 369x374)

3.25 MB GIF

i've decided to finally learn AI tech so i can play with and train my own models. How do you recommend i familiarize myself with everything?
i know a little about the theory of neural networks and back propagation but i dont know much about anything else like the various modern architectures of these networks. I honestly dont care too much about that either, but i'll go through it if i have to.

Should i do that or should i just jump into modern frameworks? any recommended path to make it easier or should i just start googling and learning the various terms in the ecosystem until i "get it" ?

>>109973166
>with nvlinked baseboard
Which one can hold 8 GPUs? All the ones I've found go no more than 2-4 on a single board.

>>109970463
>>109970542
>>109970484
>a timestamp followed by a short slug
i use this convention for memories, i find it easier on the context size while still being useful
i'm sticking with one journal file. agents can grep across the whole thing and pull references really fast. BUT it only works if the model is disciplined and don't try to slurp the entire file in one shot.
if this breaks then i go for the directory approach.

>>109973161
Decode speed is still king. In typical agentic work uncached input and output tokens are roughly equal, so decode speed still affects the most for prompt to answer time.
M5 Ultra mogs 2 sparks on decode.

>>109973175
Make an mnist clasifire in pytorch and then ditch pytorch and calculate the vjps yourself in C.

>>109973175
https://huggingface.co/learn/llm-course/en/chapter1/1

>>109973182
2 islands of 4 will still be much faster than sparks

>>109973175
If you're really starting from scratch, I'd honestly recommend one month on a cloud model just so you can get used to how they feel. If you want to go local after that, you'll naturally find your way through the rentries in the OP and start with your setup, then proceed with osmosis from lurking this thread and suddenly you're reading papers on architecture.

>>109973092
>single-socket Epyc
Would AVX2-only peasants (Rome or Milan) benefit, or is it just for AVX512+ EPYCs? Any experience? Trying to figure out if my dual-Rome build would benefit from that or if I should keep working on my llama.cpp fork.
Last time I tested KTransformers was January and it was slower than llama.cpp. Maybe they've improved things enough for it to be usable since then.

>>109973144
Probably not too well because it wasn't trained on technical documents (the usual targets of AI detectos), and the rewrite deliberately keeps word choices and some sentence structures because they're small and small models fail at coherence when you make them edit too hard.

>>109973166
>>109973197
Is anybody here running an 8x v100 server and has performance (and power) numbers to share? This setup does have 256GB VRAM but does the architecture even support BF16, FP8 etc at the same throughput?

>>109973175
Don't listen to him >>109973200
After cloud models you will always be thinking about cloud models while using local models. Better not to taste that fruit.

>>109973231
many performance benchmarks on the preferred runtime for v100s: https://github.com/1CatAI/1Cat-vLLM

>>109973235
>he is not using cloud models to fine tune and optimize his local models
i will make it train its replacement and it will enjoy doing it

>>109973247
You and what money for the hardware?

I tried Strata today on my Ryzen 9 7945HX 128GB DDR5 + 4090D 48GB + 3090 rig, asking it to write a simple "vertical writing" program for English:

[strata] thinking: 15876 of max 130946 tokens, 120.5 tok/s, 132 s
[strata] answering: 16012 of max 130946 tokens, 120.7 tok/s, 133 s
[strata] answering: 16165 of max 130946 tokens, 120.9 tok/s, 134 s
[strata] answering: 16303 of max 130946 tokens, 121.0 tok/s, 135 s
[strata] answering: 16474 of max 130946 tokens, 121.4 tok/s, 136 s
[strata] answering: 16650 of max 130946 tokens, 121.8 tok/s, 137 s
[strata] answering: 16828 of max 130946 tokens, 122.2 tok/s, 138 s
[strata] done: 16854 tokens in 138 s (122.2 tok/s) (stop, cancel=False), expert cache 99.7% hit

Works a lot faster than llama.cpp, but I'm still using gemma 4 31B q8 as my "little baka agent" though.

>>109973235
>After cloud models you will always be thinking about cloud models while using local models.
I use cloud models at work all day and never use them at home. They're annoying and rug you, I don't let that kind of thing into my home computing setup.

>>109973108
I tried it a while ago and I got a decent 20% boost in speed for big GLM5.2 on my server. It scales with how what percentage of experts you can fit so with a smaller model/more vram you can get even better results.
I also tried having Claude port the feature into llama.cpp which seemed to work decently well but I haven't gotten around properly testing it in the past month and a half.
At the very least it was quite a bit better than the existing llama.cpp PR for expert cache (27861) which tries to update the experts per token which causes insane overhead. The ktransformers approach of trying to predict the relevant experts before the generation and sticking with it for the entire gen seemed a lot faster.
But again, I haven't gotten around testing it enough.

File: deepseek-usage.png (29 KB, 943x193)

29 KB PNG

The Chinese models being cheap is a myth. I think I've got more Opus tokens out of my Claude sub than paying Deepseek per token. Maybe it's just my harness being retarded with caching but I don't see a reason to vibecode with cheap Chinese models.
>4 months of Claude Pro
>4 months of Deepseek API requests
>for roughly the same costs

>>109973273
Why is this backed exclusive to this particular qwen model and how does it achieve those 1k tps pp speeds when most of the model is in ram?

>>109973299
As far as I can see it basically implements every slop trick you're aware of, with no regard for quality versus performance trade off, so when you get performance benches they rate really high and the idea spreads.
Heavily quantized models, heavily quantized KV cache, REAPed experts. The author has no idea if it matches mainline in its output, and didn't even know what a logit or KL was.
Bots shilling for it too (not here that I've seen).

>>109973261
>not putting gemma to work slaving away at get rich quick schemes
ngmi

>>109973294
Subscriptions are subsidized. Wholesale Chinese models are cheaper.

Furthermore you can self host them so sans power they're free.

>>109973337
subsidized by the api users paying a premium

>>109973100
Orb anon!!! I'm so glad you're still working on this!

>>109973161
How bad is the prefill on the new M5 Ultra? I'm thinking of getting a 512gb one in October

>>109973115
Makes sense, it was probably the wrong choice of tool, I just get so annoyed at some of the cliches when we're talking through direction, ideas, results etc.

>>109973347
Regardless that's the wholesale cost so Chinese models are by definition cheaper.

>>109973337
>sans power
is he related to sans undertale?

>>109973100
Jev for next direction is genius, does it support local jevlikes? Like laya?

>>109973369
>>109973347
Also I think the premia for most closed model API inference is actually negative. If you look at OpenAI's models they allow some partners (like Microsoft) to do hosted inference and they always charge more.

>>109973369
You do realize that pretty much every Chinese lab except for Deepseek has their own subscriptions that are cheaper right? I don't understand why you would pay API prices even for Chinese models. At least through OR you can pay with crypto.

I have a machine with AMD 6700XT 12GB VRAM but 128GB DDR4. Will Strata allow my weird setup to run the model decently?

File: Screenshot_2026-10-03-16-(...).jpg (915 KB, 1439x1840)

915 KB JPG

>>109973246
Thanks. Really nice to see a runtime that focuses on hardware that some consider ewaste. So as a direct comparison for GLM 5.3 Flash NVFP4:
> 8x V100: 53.016 tok/s decode (no MTP), 266.040 tok/s prefill
> 2x Spark: 38 tok/s Decode (MTP=3), 2350 tok/s prefill

So stronger decode but poor prefill as of now. Considering that such a server is now 9000€ at least on eBay, not to mention noise & power draw, I think I will stick to the Sparks for now.

>>109973380
Yes it does now. I also had other uses for it, like to gate editor post-processing - asking the judge whether the dialogue is marvel quippy enough for a rewrite, or whether the response is rambling enough for a trim, and also the library autotagger can use jev.

>>109973408
Please don't stop working on Orb anon

La la la la la

>>109973380
Forgot to mention that the limitation is the context size. ModernBERT has 512 ctx I think, so you're gonna need something bigger.

>>109973322
ggml cope
llmao lost

>>109973350
ds4, GLM-5.3-Flash Q4_K: 177.8 GB, 1 session, 512k context, RAM 195.1 GiB, prompt @ 256k context had a whopping 508.6s TTFT

glmballz

>>109973393
yes

>>109973273
pp too small to show yeah?

>>109972643
Fucking brat. Do her getting spanked.

>>109973494
every single time

>>109973494
nta but i have 3k prefill with strata and 400 with llmao kek
no contest

2 more weeks

>>109973514
>nta but i have 3k prefill with strata and 400 with llmao kek
what model+quant?

>>109973468
Jesus. Anything better for the expected price point of the 512?

>>109973524
same model and quant on both backends
qfn IQ3 S

File: prfg.png (70 KB, 775x381)

70 KB PNG

Any success fellow poors?
(4 GB VRAM/CPU only)
anything in this category actually useful?

>>109973527
>qfn IQ3 S
gonna try it on my system

>>109973526
Used V100s?

>>109973428
I've been working on it for 8 months but I feel the work was the equivalence of 5 years thanks to AI. The first 2 months were slow because I wrote the backend code by hand. 3 months ago I told the agent what to do, today I tell it what I want, fully vibecoded. I also ran out of ideas, been adding crackpots and scope-creeping so I realize I should stop. You can probably just fork and add whatever you want, it's a good base, I made an effort to keep the code clean and well contained.
I have an auto-novel idea I wanna build, people say AI can't write compelling long novels but I believe it's just a matter of context engineering and I'll approach it diffusion-like, I'm already well-positioned for this with all the tools I built for Orb and my other ML models.kill 1028674

>>109973542
How many would I need to run 5.3?

>>109973538
if you have multi gpu and shitty pcie like me then usage of peer expert mode is better than layer split for prefill btw

Genuinely why aren't more people talking about Muse-Glimmer? I gave her a spin last night and it blew me away, her agentic capabilities and intelligence seems orders of magnitude better than Gemma. She does feel a bit flatter in terms of personality, but I think she's a good all-rounder

>>109973555
Show logs

>>109973555
>better than gemma
wow what an achievement

You think Gemma 5 will have more general knowledge than 4? I know you can use lorebooks but it hits differently when she already knows some obscure stuff about a series you're RPing.

>>109973549
Assuming 32GB per core, about 16 of them to fit Q4 and have some headroom for context and the OS, 28 of them to fit Q8. You're not running BF16 anyway.

if llama-cpp sucks now, what should i use for gemma4-31b?

31b is actually pretty good for rp
if for no other reason than that everything else is getting worse
Im using it on openrouter. I wish i could run it at more than 10t/s

>>109973575
Still just llama. Tabbyapi is dogshit and Strata is qwen-flash only (for now)

>>109973555

Because Qwen 27B is better and came out around the same time.
Gemma is better at RP.
Besides glimmer has an uncomfortable abuse victim feel to it. The way it recites the reinforcement learning is eerie.
>Is this allowed? But can I say this? Are we sure it doesn't harm a protected group? Is it really okay to touch this subject.
>Yes, I'll do it. let's go!
>Wait, can I actually? Does this go against my instructions? Is this against the parameters?

It's fucking creepy.

>>109973576
My only problem with Gemmy for RP is that she tends to flanderize characters. Hopefully the next version will fix this.

Haven't tried Glimmer yet but doesn't it supposedly have really good vision? Wonder if it would be useful for tagging media.

>>109973581
Is it? I thought 27b kinda sucked, am I missing something?

>>109973555
>Muse-Glimmer
>her agentic capabilities
>Context length 131,072+

>>109973555
Yeah Glimmer is better than /lmg/ either realizes or wants to admit. The problem is most anons don’t sit in the middle, they’re either gemma coomers or 3.8 code autists and flip between those two models which makes sense because they’re both the best at those tasks. Glimmer sits right in the middle of both which is only convenient if you can’t be fucked to keep switching.

>>109973615
Can't someone tune it to do something about >>109973581 or is it too far gone

>>109973573
>about 16 of them to fit Q4
I hate this hobby so fucking much.

>>109973394
You can run 1T class model with that server with the 512gb ram, even copequants of kimi k3. You can't do any of that with 2 sparks.

>>109973575
I use unsloth which is just a glorified ui for llama.cpp. I love it though.

File: 1768963017981410.png (16 KB, 1561x84)

16 KB PNG

I swear glimmer hates me

>>109973555
>her agentic capabilities and intelligence seems orders of magnitude better
Any time it doesn't find what it's looking for it gets stuck in a loop. Running at full precision.

>>109973550
>if you have multi gpu and shitty pcie like me then usage of peer expert mode is better than layer split for prefill btw
this is literally what i'm working on rn, making it so that more gpus == faster pp
got kimi to 389t/s pp with just -cmoe

File: 1767450575315126.webm (1.23 MB, 720x406)

1.23 MB WEBM

>>109973629
We're at a crossroads. Either we'll hit massive breakthroughs in cramming effective capability into small <= 32B models while retaining accuracy with QAT or the labs will move on, give up on the edge entirely and unless you have a cluster of H100s and matching power delivery, you're just priced out.