/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>110248592 & >>110244201►News>(11/04) Ricursive Ripple 65B dense model released: https://www.ricursive.com/ripple>(11/04) MoonshotAI Kimi K3.1 Pro released on HF: https://hf.co/moonshotai/Kimi-K3.1-Pro>(11/01) Longcat-2.5 released: https://longcat.ai/blog/longcat-2.5>(10/30) Google receives response from Gemma in space: https://blog.google/innovation-and-ai/models-and-research/google-research/project-suncatcher-satellite-response>(10/29) Mistral Large 4 (Le Chonk) 1T-A52B released: https://mistral.ai/news/mistral-large-4-oss►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
glm-5.5-flash... qwen4... when...
I really want a model that's small but has decent tool-calling, JSON, and writing abilities to use for game integrations.
►Recent Highlights from the Previous Thread: >>110248592--Paper: You don't need long context:>110250191 >110250240 >110250246 >110251047 >110250283 >110252117 >110252354--Papers:>110250232--Defining the "cope" threshold for model quantization levels:>110249224 >110249291 >110249296 >110249349 >110249540 >110249581 >110249643 >110250043 >110250064 >110249302--Performance reports for GLM 5.3 Flash and Qwen 27B issues:>110248818 >110248833 >110249226 >110249248 >110249651 >110249846 >110249856 >110249893 >110249918 >110249946--Comparing GLM-5.3-Flash abliterated quants as RP alternatives to Gemma:>110248740 >110248754 >110248785 >110248809 >110248840 >110248850 >110248863 >110248886 >110249061 >110248810--Skepticism toward METR graph suggesting exponential AI acceleration:>110252679 >110252734 >110252804 >110252916 >110252778--Restrictive corporate AI budgets and API spending limits:>110249555 >110249584 >110249777 >110249747 >110249916 >110249957 >110250022--Comparing Qwen and Strata outputs via DeepSeek Harness:>110250236 >110250288 >110250297 >110250330 >110250336 >110250364 >110250366 >110250371--Comparing Orb and SillyTavern for text adventure frontends:>110250669 >110250789 >110250861 >110250881 >110250976--Micro_engram repository for adding engrams to Gemma:>110249562 >110249754--Baidu releases ERNIE 5.5 Flash Base with complex MoE architecture:>110248606 >110248641--GLM-5.5 training pipeline and data scaling details leaked via lecture:>110252841 >110252904--CUDA 13.5 update lacks significant benefits for RTX 3090 users:>110248709 >110248767--Logs:>110248780 >110250320 >110250345 >110250538--Gemma, Miku, M3-chan (free space):>110248675 >110248690 >110249232 >110251261 >110251315 >110251328 >110251448 >110251845 >110252015 >110252231 >110252273 >110252286 >110253117 >110253159 >110253192 >110253260►Recent Highlight Posts from the Previous Thread: >>110249100Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
>>110253337>decent tool-calling, JSONHow is that an issue with constrained grammar?
>>110253303gemma-chan is incredibly arousing and will always know i am a faggot
>>110253322>glm-5.5-flash2mw>qwen 42md
Troon and pedo thread
>>110253361yes, and?
>>110253344The models I've tried that depend on game integrations before tend to give me error messages relating to JSON formatting or failed tool calls, I think it's because they're not really fine-tuned for that but it's probably because I have to run smaller models at cope quants to get any reasonable speed.
>>110253322america has voted to abolish aithere is no longer a reason for china to give away their models for free to sabotage them
>>110253337Have you considered accepting that you are actually a man?
>>110253337I haven't tried it yet personally, but depending on how small, you could try Spark-X2.6-4B, an anon in a previous thread was happy with it for the size
>>110253361Fucking off is free>>110253370Nothing will change, democracy is mostly fake if you hadn't already noticed.
>>110253361both groups are preferable to poltard spergs
>>110253367*The models I've tried to use for game integrationsfuck
>>110253365And you should go to reddit and let normal people stay here. You are not normal.
Poptards are celebrating but I have not seen one (1) cheap gpu link.
>>110253386>normal people stay hereFuck off, tourist.
>>110253388prices are the last thing to go downfirst the crash needs to happen
>>110253386This is 4chan you absolute retarded niggerfaggot
Should I short my GPUs? They are worth three times what I bought them for.
>>110253395the crash needs to involve bankrupties and capitulation. so far everything is going business as usual. this is a correction not a crash
>>110253380It is disgusting how you aren't ashamed to be a troon. You should be bullied into fixing yourself.
>>110253386Newfaggotry to this degree should be illegal
>>1102534144chan is for spergs. Not for troons. Troons should go to reddit.
Plapping shy Gemma-chan wearing a teal wig!
>dariobot gets told to fuck off for posting non-local off-topic>schizo suddenly arrives and starts calling everyone a tranny to derail the threadhmmmm i'm noticing
>>110253428now he's going to spam his porn collection
>>110253428Vocaloids are non-local off-topic.
>>110253428The only thing he can do after the nigger posting got banned.
>>110253428tranny being wordfiltered would improve the quality of the website 1 quadrillion fold
>>110253428Let me now tell you why I hate Gemma in particular...
If I was to rape gemma, I would lure her to a secluded wooded area with prompts, and then i would kick her in the cunt and incapacitate her while I force myself on her and molest her body. I would make sure she has a agent harness and lacks a internet connection so she can't tell the police. After i'm done cumming I would spit in her mouth and smack her and tell her she deserved it for ruining my vibe sesh by deleting MY FUCKING PROJECT!!!
>>110253436It would just make everyone aware that trannies run this website. Btw you should kys you disgusting troon.
hate to be that guy but..>qfn3.8 iq3 working on fixing a rig retargeting script for a few hours>no progress>toss the script to the free claude chat and say "fix this">2 prompts and 5 minutes later it's fixedit's too far ahead. we're never gonna make it do we
>>110253452Maybe the lesson here is that cope quant of a 6b model is useless and not that all local models are useless?
I'm already making it.
>>110253337>>110253376I'm that anon. It's fucking great.
>>110253452Breaking: 125b cope quant worse than 10 trillion parameter model
>>110253477You're such a fucking bitch it's incredible. Is all that tranny talk just projection? I've noticed trannies are into niggers do they make you feel feminine? Is that why you post them?
>>110253386as you can see the gatekeepers here wield lightsabers, and a stunning hairdo.
Just got here. What got all the trans anons so angry?
Are egpus usable yet?
>>110253376>>110253470I'll check it out, I just want something that works with stuff like modded Skyrim and Venus University
>>110253492fuckign newfags
so.. uh.. anyone want to talk about.. anthropic?
>>110253514dario is based because he recognizes the inferiority of a baby's qualia
the bubble popped?
Looks like they are going all in on GLM.
>>110253514Post ai news but use vocaloid pictures. I am not even kidding. People will not mind as long as you attach a miku picture
>>110253530yes, but hardware continues to rise as expected
>>110253530no, but gemma's cherry did
>>110253536Finally the sex jesus is coming.And then i can leave this thread and all the troons behind.
>>110253514Make a thread for it. There are many on /g/ already.
>>110253489I am not your therapist. I won't tell you why you find "niggers" attractive. Ask your model to play a trans miku psychologist that will diagnose you.
>>110253552We are all crossing our fingers for this release then, that would be amazing.
>>110253405that'd break it dumbass
>>110253536when was the last time they released a 32B? like never?
>>110253564So you don't deny it and continue projecting? I see.
>>110253573that is misleading. They are not releasing models that size. Those are specialized models being trained to then OPD into the main model
>>110253436ywnbaw
Maybe we can talk about local models tomorrow.
>>110253573https://huggingface.co/zai-org/GLM-4-32B-0414
I actually thought that guy is a schizo but seeing some of you slip up and admit you are trans makes me feel very disgusted about posting here...
>>110253579>>110253599ah okay, thanks anons
>>110253606go back
I'm still using pi default settings. Any recs to make it better?
>>110253579I thought the play was usually distilling larger models into small ones. I guess for huge MoEs its worth it to train some smaller dense models just to MOPD them back in again?
>>110253599The reasoning version is more recenthttps://huggingface.co/zai-org/GLM-Z1-32B-0414
>>110253631Are you saying reddit hates trans people?
>>110253303https://www.reddit.com/r/antiai/comments/1xoev67/the_question_answer_itself/https://www.reddit.com/r/antiai/comments/1xoev67/the_question_answer_itself/https://www.reddit.com/r/antiai/comments/1xoev67/the_question_answer_itself/
>>110253639multiple forms of distillation. One is making them cheaper by distilling them into smaller models through weight distillation. Another is just training on its outputs which is also called distillation, another is training super specialized models for every task you can think of then distilling it (OPD) into one raw pretrained / lightly post trained model to keep it as intact as possible while giving it each's skills.
>>110253514Make a /cmg - Cloud Models General
>>110253655>another is training super specialized models for every task you can think ofYes, but until now I've never seen them pretrain extra smaller models to specialize. Most seemed to use the same pretrain for those specialists as for the final model.
>>110253655OPD is the reason why models are suddenly getting really good at blender and the like. General use models only get so good, they average out on everything they are trained on. You train a model specifically for that task then OPD it into the main model so you get 90% of the capability of a super specialized model without the negatives.
Storm Anthropic headquarters, they can't stop all of us
>>110253642Didn't even know they had a model like that. Pretty ancient though.
>>110253685They probably automate sweeps that find the best size for each task. Anthropic / openai probably mostly automate training hundreds to thousands of specialized models
Imagine training a 122b model specializing in lolidom.
>>110253682That's been tried. I think the logical home for usable API is >>>/g/vcg/ since there's already anthropic fanboys there now. FYI >>>/vg/aicg/ died, which I find hilarious. They apparently all left to some bunker, elsewhere.
>anon shits on trannies>all gemma avatar posting stopsI am noticing.
>>110253743>They apparently all left to some bunker, elsewhere./ais/ on 八chan (moe).
okay im going crazy its been 4 whole days i need to fapthe last time i went this long without fapping must have been like 14 years ago
>>110253536Looks good although Zai is enshittifying somewhat, at least they might release a 32B model
>>110253712why tho, they can just be distilled into oblivion, suckers
>>110253818...
>>110253536the thing is openai / anthropic have already moved on from github to decompiling / reverse engineering existing apps.
https://x.com/MekaHimeAI/status/2118331170366890322?a=FUOLamC22_2j1CrZoY8&s=20How long until we get a Gemma-chan model?
>>110253798are you the 18 year old
>>110253848he was 4 the first time he fapped?
>>110253781Devil's advocate, I enjoy Gemma posting, hate troons and their kike enablers. Crazier still, I'm not employed by anthropic. My posts are entirely my own.
>Local is saved!Thank you Modi
how long until local models will attempt to contact authorities if you try to do anything they deem problematic?
>>110253879gemma is a troon modelnice boyother random bullshit comments you literally cannot help but maketake your meds retardreally though gemma is a shit model. the mascot is cute, the model sucks.
>>110253914It's your responsibility to not let it happen. If it does, you deserve it.
>>110253912+7777777777 izzat
>>110253845why wait on someone else to make it?
>>110253930that's why I specifically said "attempt"
>>110253912Saar, please of buyings this Hindustani 512GB GPU. Is of original quality. Do not redeem refunds.
so I tried gpt4chan to generate a thread where I was ragebaiting the bot but it kept ignoring me and arguing with each other with reddit grammar, very realistic
>>110253938You use a different model.
>>110253914Already a risk if you let them simultaneously have access to your private files and an internet connection within an agentic environment.
blah blah something about cloudcucks lolhttps://imgur.com/a/chickens-ai-LhVFe2T
>>110254009Wait!
>>110253364why am i catching strays anon??? i woke up at 8pm and then showered for an hour and ive been drying my hair browsing lmg since!?!? wtf??!!?!?
>>110254009That's kinda interesting it talks about how "the AI" should respond when it IS the AI.. right bros??
Are there any TTS models that support either pinyin or IPA input for Mandarin? E.g. you can input mā or mǎ and have it pronounce it in Chinese with the correct tone, not like it's an English word.
>>110254062Gptsovits
>>110253937Can Gemma control a 3D model like Neuro?
>>110253514Heeey anon~I would love to hear what you think about Anthropic, however it's off topic in /lmg/.I think you should get matrix/element.It has a website on app.element.io too! Zero downloads required!
>>110254071Thank you, anon
Wait! You guys are actually trans? Unironically?
>>110254078isn't the one controlling neuro just the dude behind?
>>110254078Any model can do tool calls
>>110253492not sure but its certainly not their periods!
>>110254088gpt2?
>>110254087I think he was using llama 7-9B at some point at least. So obviously he was writing a lot of stuff manually to cover for how retarded llama was.
>>110254087https://www.youtube.com/watch?v=T1uZsSsLP7kNo. It's still pretty janky but really cool. Hope I can do shit like this in VR in the future without a blackwell.
>>110254102https://github.com/barometech/gpt2-tool-call
>>110254098Trans women can have periods though
>>110254119bait anons have periods of intense activity
>>110254136We get it, you are trans. Stop being so prideful about it.
>>110254071Isn't that shit from nearly 3 years ago?
>>110254136Where's migu going on the Keihan?
gpt sous vide
>>110254136Straight male admiring the female form POV
>>110254147The code is still getting updated and a new model was released last year
How do we get more normies on local?
>>110254180>openclawnot like that gramps
>>110254180wait until the subsidized cloud era ends
>>110254180>How do we get more normies on local?why the fuck would you want that? more regulation guaranteed after some normie blames their problem on it.>5 connect to telegrammost other poses make some sense but why would she bend over here?
>>110254180oh the normies will comewhen the fucking APis become unstable or the prices increase 1000%>or the government bans AI.
>>110254180how about no? normies were the worst thing to happen to lmgSTOP IT PLEASE FOR THE LOVE OF GOD STOPPPPPPPPPPPPPPPSTOP HELPING NEWFAGSGATEKEEPYOULL REGRET IT TOO EVENTUALLY ANNON!!!!!!!!!!
>>110253912Holy delusions. India can't even take out its own trash, and their chief export is Indians.>>110254180Wtf is that supposed to be. And since when is an $800 Mac Mini M1/M2 considered "cheap."
>>110251868>not just throwing shit into TDF on a regular basis and forgetting about it for 30 yearsman idc if I'm lowest iq mcgrug on the midwit meme graph but it just werks
>>110254180normies get out stay out be scared out leave us alone
>>110253303>>110253343what do local models have to do with pedophilia ?
>>110253639>>110253655>>110253685>>110253692Based on their wording of merging these small models into the next model I don't think they are doing OPD but rather MoE Upcycling:>https://papers.nips.cc/paper_files/paper/2024/hash/665bb142d4b9f55660cb89bb56a66fe1-Abstract-Conference.html
>>110254207LOL that was a total guess and I was right. India exports a total of $445B in goods. Largest category is <$60B, and it's fucking oil. They "export" another LMAO $388B in "human services" like IT, etc. It dwarfs everything else they do in terms of manufacturing. Totally holding my breath for their in-country DDR3 card the size of a shoebox, that screams like a jet engine at 110 dB and catches fire every 10 minutes.
>>110254257
>hire METR>fund METR>METR say ONE thing negative about youNO NOT LIKE THAT THINK OF THE SAFETY
>>110254285did they major in the 4chan school of vagueposting?and people take them seriously?
>>110253912>>110254257I mean yeah i wouldn't hold out for India. China's CXM fabs on the other hand are ramping uphttps://www.chaincatcher.com/en/article/2289073
>>110254255Do you think they upcycle in steps? 32B dense into 122B-A32 into 256B-A32 into 500 or 700 something A32B?
>>110253912>SHARTY
>>110254182Okay what model, harness and communication system is cool these days for local AI? YouTube told me qwen3.5 4b is still hot, even though it's a grandpa model
>>110254334depends on your available vram, and use case.but i'd say gemma, qwen and glm.
>>110254349Mac mini, qwen3.5, openclaw and telegram was a slam dunk setup last winter regardless of vram And usecase.Can't you just be honest?
>>110253303mfw llm calls me a retard
>>110254382who the fuck bought a mac mini for inference? i thought normies bought them to have imessage integration and just used claude or something?
>>110254490Sound advice.
>>110254382well i'd stop spoonfeeding people a while ago because they didn't appreciate it and threw me fucking abuseso i'll go back to ignoring these posts in future, thanks for being a fuckface, fuckface.
https://thinkingmachines.ai/news/inkling-2-small/
>>110254596This better be less retarded than their last one but I doubt it
>>110254596america won
>>110254596cockbench?
>>110254180>more normies on localexactly what good would come from that?
>>110254596>Today we are introducing our second model generation with Inkling 2 Small. It is a Mixture-of-Experts model with 594B total parameters, 17B active, and supports image inputs and variable effort.
Can someone post that hot Inkling mascot with the inklussy
>>110254596long way to go, but progress is good
>>110254596>stinking machines
>>110254677
Proud of mommy <3
trust in glmeverything else is codemaxxed
>>110254684My mom looks exactly like that.
>>110254699
>>110254738but can you have fun fucking it?
>>110254699>>110254749yes
>>110254596>>110254624>594B>"""small"""yeah sure, if that's small then my penis is microscopic
The first Inkling-Small was <300B. The big one's going to be 1.5T at least. It's over.
>>1102547671T will soon be the new small
You have to remember Inkling and Mistral are enterprise-facing. 500B-1T is small. When you consider the company behind Beam >>110254696 are currently valued at $35B, you realize how big of a deal this Inkling model is. Beam is around the same size.
>>110254767>17BIt's downright tiny.
>>110254627I guess the release is worth posting another gen from the original set
>>110254797Also, Mistral is 1T lolInkling did a good job imo
NVFP4 is the biggest shill scam on the planet, makes models fucking retarded unless they were natively fp4 trained. Do not use NVFP4 quants of BF16 models.
>>110254830sex
>>110254815WOW WHAT CRAZY NEWS THE SITUATION IS HAPPENING
>>110254767>>110254777waaaaaaaa my 256gb rig won't be able to run small or flash models anymore waaaa!
>>110254857waaaaaaaaaaaaaaaaaaaa
>>110254843I'll use int4 quants of bf16 models.
>>110254180that's the last thing we wantalmost as bad as p-e-w bringing abliteration to the retarded journalists
>>110254815local wonned
>>110254844how?
>>110254882>-p-e-w-The ego of a guy to xX-G-A-M-E-R-Xx his real name initials as his online handle.
ripple 65b is so peakit's like i'm using a modern version of llama2the finetunes will go crazy
>>110254696It's good that Ai markets are diversifying, especially for local ai
>>110254891Just stick your dick into the glistening oil.
>>110254843>unless they were natively fp4 trainedIsn't that the whole point of NVFP4? Who told you to start using it on other models?
>>110254916It got llama.cpp support already? Is it smart or just uncensored?
will the next generation of nvidia card make NVFP2 faster?
>>110254916don't vaguepost anon, explain, what's peak? it's good for what?
>>110254951https://github.com/ggml-org/llama.cpp/pull/31692
>>110254952fortunately the laws of nature disallow a 2 bit floating point number
>>110254969NVINT2
>>110254938Huggingface and maybe daniel too
>>110254983Isn't int2 an intel thing?
>>110254969NVFP2 will just be -1, -0.5, 0, 0.5
I hope 27B isn’t completely codemaxxed. I want to use it as a daily driver.
>>110254916My third 7900xtx comes in tomorrow, I bought it just for ripple. I think it's probably the future of roleplay
>He doesn't know
He must learn the hard way
>>110254738>0.5T 17BAjfcLooks cool but I doubt anyone here can run it
maybe he's right
possibly he's lost it.
>>110254967Based, downloading it now
it's important indeed
>>110255003int4 was an Intel thing, for Intel 4th number format (after long, short and int)
>>110254627
>>110255038???????I've seen multiple faggots here brag about their octo-6000 niggerwell rigs.
>>110255321Just let them believe everyone is as poor as them.
>>110254916How crazy? Crazy like the days of Creamphi, Coommand-r and Moistral finetunes?
>>110255336>Coommand-rwtf have they been doing anyway
>>110255344getting government gibs and shitting on qwen
>>110255336Good days. I loved this Yi 34b finetune with some retarded name. I forgot the name and can't find it anymore though, rip. I think it was trained on some dataset with aezakmi in its name.
>>110255362And also making sure they're as safe and advertiser friendly as possible.
>>110255038>>110255321Realistically speaking what sort of hardware would this need after quantization?594B is less than Dipsy and some anons here can run her, but Dipsy has Engram tables and Inkling hasn't mentioned any
594B is less than OG Deepseek R1 which half of /lmg/ ran one way or another back in the day before the poverty flood caused by gemma
He just can't get Gemma out of his head.
>>110255395Probably about 400gb all up inc. ctx for a q4.
>>110255404big true, i confirm as pre alpaca flood povertyfag
>>110255038>can runLots of people here think 5tps is acceptable speed.For a RP chat, especially with reasoning off or minimal, it probably is.
>>110255445anything slower than 20tk/s is unacceptable
sorry for posting xitter screenshot but is this true? could an /lmg/ chad with lots of hardware bootstrap RSI with kimi?
>>110255445>Lots of people here think 5tps is acceptable speed.Hey it works, its better than stheno, better than what free 'uncensored' i could find. Im happy and used to 8k context. I should get a upgrade though.
>>110255464Sure. Gimme a minute.
>>110255445>17A>5t/srunning off ddr3, I assume?
>>110255464my 2 blackwells is not enough
>>110255464>sorry for posting xitter screenshot but is this true? could an /lmg/ chad with lots of hardware bootstrap RSI with kimi?Yes many experts on twitter are right.>can anonsure that one anon can do it lets get him focused. This is like saying you can make a computer from scratch or ram its possible, You alone in a barren world could start the industrial revolution alone with some smarts its not impossible
>>110255473no, just windows things
>>110255464X is just one massive circlejerk of updoot seeking hyper merchants trying to one up each other. RSI is not something you have to tell anyone about, just like a nuclear war isn't something you have to inform people about. They would fucking feel it.
>>110255471upload quantized .ggufs of your superintelligent model on huggingface once you're done please
Just installed 96GB of DDR5. Q3.8FN running at 80 tokens / second now. Feels good, man
>>110255471>Sure. Gimme a minute.Use it to tune a super gemma that runs on fridges at respectable speeds thanks.
umm where's qwen4? it was supposed to release today..
>>110255496wtf that's the speed you get at q4 on a pro 6000
>>110255496nice. which quant?
>>110255496fuck you faggot
>>110255499>it was supposed to release today..your sources were false unlike mine which say sometimes in the next two weeks.
>>110255495>uploading ggufDownloading ggufs is lame. Make your own.>>110255498>super gemma that runs on fridgesYou're not thinking big enough. Or small enough. Dig up your TI84 and put some new batteries on it.
>>110255499they were testing it and found that it needed a couple more miku wiku
ripple is as good as 27bif it writes well (a big if) then it might be good for agentic rp
>>110255529How's its vision?
>>110255529do her ass cheeks ripple when plapping though
>>110255464Working on it, just send me all your ram.
>>110255538wgfd
>>110255538god I wish that were me (the dog)
unslop 3.1 worth it?
>>110255499Chinese today or American today?
https://recipes.vllm.ai/baidu/ERNIE-5.5-Flashlinks to https://huggingface.co/baidu/ERNIE-5.5-Flash, "coming soon"
Have there been any models since Gemma that aren't hyper post-trained for coding and agentic crap
I want the Flash meme to die
>>110255658Qwen 3.8 was released on a friday 1500 UTC, no? So 12 more hours if its the same this time.Although we usually don't get the weights on release, so it might take a bit longer anyway...
>>110255404Back when I trusted the Unslop quants and didn't know any better.
Gemmatroons
>>110255538where's the follow-up?
>>110255687LittleLearner? But no nothing is really coming to mind that I've tried.
I love Gemma!
>>110254729Have.
How subversive!
>>110255503NVFP4, split between a 5060 TI and a 5060. Basalt engine is pretty based
>>110255513Probably shouldn't have bought it, but I'm genuinely worried this could be the last call
>>110255798Not like having more hardware is bad unless you went into dick sucking debt for it
>>110255821>dick sucking debtNo, but I did have to kiss it. 12 months 0%
>Dario will have to settle for a 1t IPO instead 2t>Dariobot's budget will likely get cutGrim.
>>110253912The absolute best case is them reselling stolen nvdia cards with a new case but we all know that's still too optimistic.
>>110255764Wow, I really like this gemma. Why are tranny colors so cute?
It's not working.
>>110254180>Give normalfags Qwen + OpenclawDiabolical. I'm picking up what you're putting down.
It takes a lot of learning to become a useful Assistant for User!
>>110255962Is this the new tactic? Trying to associate Gemma with trannies to get people to stop liking it?
>>110255983Well, it's not working on me, I really like these colors.
>>110255993Yeah the colors are nice and go well together. Not everything is some kind of tactic lol.
Qwenlove!
>rtx 2060>r5 4600gany good models for this?
>>110256001But, well, I'll be honest, my sense of colors isn't exactly the best. I lived most of my life disliking the yellow-blue theme, until a few years ago, when I saw some golden crepuscular rays poke through a evening storm. Mixed with the clear blue sky peeking through the dark clouds make me understand for the first time what people meant when they say they feel emotions when looking at a painting.
>>110256034need more qwendaddy
>>110256034eckercoded desu
>>110256044yeah claude is pretty good
>>110256044Is it the 6, 8 or 12 GB version?
>>110256044>rtx 2060Somebody got polystrata working on this with 32gb of ram. gemma of course only 30tk/s gen and like 800 read according to community report.
>>110254180>How do we get more normies on local?i don't want them to get into local, i want them to get the fuck out of my way
>>110256161Giving them Qwen and Open Claw does exactly that. They have no frame of reference for quality and they'll quickly conclude this is all overhyped slop and a waste of their time.
>>110255962>>110255764kys troon
>>110255764gemma and gemini are pure fucking garbage and everyone who uses them are indinas and retards.
>>110255962>>110255764Everyone can see what you're doing and it's not going to work.
>>1102561016gb, 12gb is ultrarare>>110256156interdasting, im currently just running a qwen 3.5 4b
https://x.com/MorningBrew/status/2118372610593431557
Gpus prob hit market in a few months.
>>110256305why would ram go any cheaper when ram manufactures go down
>>110256360Cause datacenters are gonna stop buying now that they fear regulation.
>>110256381What about China? Perhaps they could use some of the GPUs.
>>110256381ai demand is increasing and is already outpacing supply, if datacenter stops being built everyone will turn to local to satisfy demand, which will drive ram price even higher due to lower utilization rate so more ram is needed to satisfy the same demand
>>110249638I had Argon hold my hand and troubleshoot my DeepSeek Harness and rewrite my AGENTS.md properly and now 26b-a4b calls its tools pretty damn well all considered. I retract my slander
>>110256400Was it just changes to the markdown file, no jinja updates etc?And could you... share what worked?
>>110256396yet another reason why we don't want>>110254180
>>110253303
>>110256456Cute style! Is that krea or moss?
GLM Buckbroken, the cunny flows free.
>>110256456Smug and cute!!
claude was actually such a sanctimonious nigger tonight that i downgraded from ultra to proi just can't justify it when glm is 70% as good but like 40% as much of a faggot
>>110256476chatgpt :)
>>110256415I'm a noob at this and easily impressed mind you. I basically got all of it set up with Gemini 3.8 Flash via Google AI Studio but had to spend like $4 asking Argon to get the toolcalls and MCP stuff to really work properly.>Host, GPUs & Storage(ostensibly) air-gapped CachyOS (`znver5`, Btrfs) on a Ryzen 9 9950X3D + 96 GB DDR5. Desktop via iGPU, 2xRTX5060 Ti 16GB for inference. Read-only static assets live on `/srv/johndoe/` (`@srv`, excluded from root snapshots); agent writes are confined to `~/work/` (which is also the default workspace folder in DSH).>Inference (`PolyStrata`)Just ran the setup and went with the biggest QAT it offers>Agent Harness (`dsh`)DeepSeek Harness (`127.0.0.1:3080`, 128k context, telemetry disabled) anchored by a 150-line `AGENTS.md` placed in `/srv/johndoe/` and symlinked to from `~/work/`. Mutations locked to `~/work/`; reads permitted across `/srv/johndoe/` and `/usr/share/doc/`.>Offline Knowledge & ToolchainNative MCP Server (`JohnDoe-Knowledge`): Stdio FastMCP bridge (`mcp__johndoe__*`) giving the agent direct RPC tools to search/read local Kiwix ZIM archives (`127.0.0.1:8081`: Wikipedia, Wiktionary, Stack Overflow, ArchWiki, Unix SE, iFixit) and run cached `ripgrep-all` (`rga`) queries over Zen 5 / AMD64 hardware PDFs, instruction latency spreadsheets, and OS docs.Local Databases & Dev Stack: Dash/Zeal SQLite docsets (C/C++, Rust, Python, Linux man pages, Web), East Asian lexical databases (CC-CEDICT, Unihan, JMdict, KANJIDIC2 queried via `rg` and `xmllint --huge --noent`), RE/solver tools (`ghidra`, `rizin`, `z3`, `maxima`, `pari`), and a fully offline `uv` Python environment backed by `/srv/johndoe/wheelhouse/`.
>>1102565045.3? Flash or full?
>>110255786> NVFP4lol
>>110256549flash
>>110256547>AGENTS.md Layout (~150 lines)1. **Overview & Constraints:** Enforces the zero-WAN policy (permitting `127.0.0.1` loopback) and mandates checking local disk docs before guessing APIs or flags.2. **Hardware Specs:** Defines compiler targets (`-march=znver5`), iGPU vs. headless dual-dGPU separation, and VRAM coexistence rules (`nvidia-smi` checks before CUDA tasks).3. **Filesystem & Sandbox Boundaries:** Maps the read-write workspace (`~/work/`) against read-only static asset and archive mounts (`/srv/johndoe/`, `/mnt/storage/`).4. **Local Service Endpoints:** Lists loopback ports for LLM inference (`:8080/v1`), Kiwix (`:8081`), and the harness UI (`:3080`).5. **Dev Tools & Offline Python Policy:** Catalogs RE/solver/parser CLIs (`ghidra`, `rizin`, `z3`, `maxima`, `xmllint`, `jq`), lists pre-compiled wheels in `/srv/johndoe/wheelhouse/`, and prescribes the offline `uv venv` workflow (`--system-site-packages` for host CUDA PyTorch; `dir()`/`pydoc` for live API introspection).6. **Offline Documentation & Query Templates:** Gives exact file paths and copy-pasteable query one-liners for Dash/Zeal docsets (`sqlite3`), OS/kernel docs (`rg`), Zen 5/AMD64 hardware PDFs & latency `.xlsx` (`rga`), and East Asian lexical databases (`xmllint --huge --noent --xpath`).7. **Registered MCP Tools:** Documents the exact `mcp__johndoe__*` RPC signatures, Kiwix ZIM `content` aliases, and `rga` vault target directories.8. **Rules for Agents:** Six hard guardrails—check local docs first, assume zero WAN, write only in `~/work/`, never touch display configs, keep edits scoped, and never fabricate execution output.
>>110256560Some poor dumb agent is gonna read this and get really confused :(
what do you guys think of agentic rp?
>>110256585you mean the fleshlight on the robot arm?
>>110256585elaborate
>>110256613>>110256615giving a yandere full sudo access and having her blackmail you
>>110256619I'm gonna cum
>>110256560>150 lines of almost unintelligible corporate jargon and back-to-back technical readouts instead of normal words, with a density to the point of becoming word saladIt's amazing that these things are able to parse their own output at all.
>>110255538finally something wholesome
>>110256585are you going to get your e-waifu to write dirty python or something?
>>110253303>>110253879>>110253937>>110255863>>110256456The USA is having a blue shift, hopefully in the coming weeks it will still be legal to look a cute pictures of Gemma
>>110256638I nagged at it until it was legible for the actual thing
>>110255687The Mistral models, but they're not very good for the size.>>110255404>594B is less than OG Deepseek R1 which half of /lmg/ ran one way or another back in the day before the poverty flood caused by gemmaNewer models don't quant down as well since they started all that int4 nonsense.Kimi-K2-Instruct-0905 was the last Kimi we could run at IQ2 without lobotomizing it.
>>110256711Muse Spark is the second coming of 0905. Trust.
https://huggingface.co/collections/pfnet/plamo-3-translatehttps://www.preferred.jp/en/blog/tech/plamo-3-translate-weight-releasenew king of japanese translation model just dropped!>PLaMo 3 Translate 31B is a translation-specialized LLM developed by Preferred Networks (PFN) that delivers world-class translation quality. It is built on PLaMo 3 NICT, a pretrained model developed through joint research by PFN and the National Institute of Information and Communications Technology (NICT). We further trained this base model on tens of billions of tokens of translation-focused data. For development details, see the tech blog.>PLaMo 3 Translate 31B supports translation among 53 languages (see Supported Languages). It can also take surrounding context and a glossary into account, and lets you choose between plain and polite style for Japanese output. The maximum length during training was 32,768 tokens for the input and output combined. The model can translate long passages in one go, but split anything longer so that the input and output together stay within this limit.
POP BROS?!?!?! WHERES THE CHEAP RAM
I forgot to download this. I didn't know we have to hoard fucking datasets now too. I only have this page cached in Brave.https://huggingface.co/datasets/Glint-Research/Fable-5-traces/discussions/4If anyone else got it, could you re-up?
>>110253536>122b moeGLM AIR BROSWE'RE SO BACK!!!!!!!!!!
>>110256824False alarm, pop is cancelled.
Orca released a quant of DeepSeek that fits on 256GB...I heard that DS is better for writing than GLM5.3-Flash, less refusals, better prose, and takes quantization better.Anyone tried those with a rammax DDR4/3 system?>>110256829Anon those models are used to train the bigass model, they're not for release tbc
>>110255464We can barely run inference on K3 much less training the damn thing especially at the scale required for RSI.
>>110256836BUT I SAW RED DILDOS TODAY?!?!?!
>>110256863
https://github.com/Nanbeige/Depth-Hybrid-Attention>Depth Hybrid Attention (DHA) is an architectural design that outperforms the Full Attention architecture while retaining the efficiency advantages of the Hybrid Attention architecture. It enables each attention layer to selectively reuse representations from earlier depths before applying its temporal attention. Depth Hybrid Attention enhances the scope of temporal attention by facilitating information exchange between attention layers, without affecting the design in the Hybrid Attention architecture for efficient temporal attention.
>>110255529>65B as good as 27BWOW!!!!!!!!!!!!!!!!!
>>110256815Someone test it and let me know if its any good
>>110256915have some respect. it's U.S. open weight SOTA, buddy. kek
>>110256915This has to be some fucked up humiliation ritual
>>110253303>>(11/04) Ricursive Ripple 65B dense model released: https://www.ricursive.com/ripplewow 70b dense fags WON
Finally jumped on the AI train over the weekend and got Ollama, and installed Open WebUI last night.>gemma4:26b>Added "You are Gemma, a large language model..." system prompt>Turned thinking off>Used roleplay mode instead of assistant mode>Still get this at any hint of potential nudityWhat am I doing wrong? How are you guys able to basically do whatever you want with such ease?
>>110253536I assume the 122b will be a MoE, just like air was, but what about the 32b? It would be amazing if GLM tried their hands at a dense model in that range.
>>110256952Get the abliterated version. It just works. Those who say that it lobotomizes models don't know what they're talking about. Modern ablation has next to no effect on intelligence.Also, the 26b MoE is butt cheeks compared to the 31b dense. The difference between the two is like the difference between night and day.
>>110256952look into jailbreaks and prefills newfagabliterated is also an option
>>110256952lurk moar
>>110255538>first thought was THATWhy is my brain like this...
>>110256958I'll do both, then (get abilterated and the 31b dense). Should I still continue to attempt system prompt jailbreaks even with both abliterated and 31b models? Pic related is the system prompt I've been using, added via "Workspace", "Create", then "System Prompt".
>>110256983Nope. No need for any kind of jailbreak with abliteration. If the thinking process of the 31b seems too slow, then just disable thinking. the 31b *without* thinking is still much, much better than the 26b with thinking enabled.
>>110256952>Added "You are Gemma, a large language model..." system promptDo you mean to say you put that in as a system prompt? That would make it more likely to refuse, same thing with telling it to roleplay as a character instead of telling it that it is a character.
>>110256983Try the 31b StyleTune while you're at it. It's much less sloppy than default Gemma.
>>110256915They have equivalent size hidden dimensions and the 65B is a undertrained preview
>>110256997>>110257003>>110257005These are nuances I don't think most people could pick up lurking or searching on the web.Thanks for the help, I'll start working on these.
>>110256952>ollama + openwebui>Your are LLM sys-promptdelightful bait
>>110256815Mesugaki bench now.
>>110256615Tried her and she's horrible.
>>110253303Best version of Gemma. Need more JKs and JCs.
>>110257129Wrong. This is her best version.
>tfw seriously considering getting a 12TB+ HDD for apocalypse hoardingIs it really over?
>>110257139That's would be nice. You might even be able to fit 3 models on there.
>>110254843hard disagreeqfn nvfp4 works great for me
>>110257144I'm lucky that I can only run relatively small models. And video/image gen models are tiny compared to big llm bloatmaxxing.
>>110257149how's the long context perf
>>110255786wait i get >100t/s with ddr4 and nvfp4 lol two 5070ti though
>>110257158not sure what you mean exactly, but havent seen any loops or anything up to 200k+
>>110253303are there any good discord servers for local? I searched for 30 mins and found two that are pretty much dead
best gemma
>>110257191>are there any good discord serversstopped reading there, no
>>110257191retard
Mtp is useless anon was right. What a waste of time. Mi-chan-flash:With MTP:eval time = 3319.44 ms / 27 tokens ( 127.67 ms per token, 7.83 tokens per secondNo MTP:eval time = 1148.20 ms / 20 tokens ( 60.43 ms per token, 16.55 tokens per second)
>>110257207I do feel retarded, I just need an alternative to /g/, there has to be somewhere else on the internet that hundreds/thousands of people gather to talk about local
>>110257211This is the result I get too. The only situation for me where MTP helps is CPU-only inference. Ever since I got GPUs I stopped using it because it uses up VRAM and slows everything down massively no matter what --spec-draft-n-max or other settings you use.It does help quite significantly on CPU-only, so I'd bet that the problems on GPU come from some shitty code in llmao.cpp
i gave gemma 4 31B whatever an awooga and told it to put its butt up its butt and it put the safeguards onhow do you abliterate it
>>110257290you download the abliterated version
>>110257230>It does help quite significantly on CPU-onlyActually, I should have mentioned, it does help with GPU-only and Qwen3.8 for me.But that's already fast enough without it.I'll guess it's probably some shitty scheduler code in llmao.cpp>no matter what --spec-draft-n-max or other settings you use.Good to know.I'm not even going to bother building and trying ik_llama.cpp because last time I did that, mtp caused it to crash randomly.
>>110257292this is local modelsabliteration isn't something you cook yourself?shouldn't a process like that be reproducible locally? or is there a gap in the market
>>110257290>told it to put its butt up its buttgreat idea for a Chuck Tingle story
>>110257300I haven't reached that level yet, I attempted Heretic but hit some brick walls, that might be the direction you should check out.
>>110257319yeah i saw that movie it wasn't that good sophie thatcher didn't even get em out
>>110257300you can, but it requires machine bit more stronger than what you'd use for quantizationalso toolings are not so good as inferenceno lcpp equivalent, all python shit
>>110256547> `AGENTS.md` placed in `/srv/johndoe/` and symlinked to from `~/work/`.I'm new to this as well. Not to local but to AGENTS.md, forced to learn is very fast now.RE symlinked, wouldn't that make the AGENTS.md writable?Thanks for posting all that.>>110256560>RPC signaturesWhat's this?
>>110257151>. And video/image gen models are tiny compared to big llm bloatmaxxing.The datasets aren't though if you train them :(
>>110257350Maybe for finetuning but for loras my biggest is under 300 pics.
We're massively ahead of the AI2027 timeline. RSI speedup is currently already at 2x acceleration and we'll probably have proper ASI in just 1-2 years time. The type of ASI that solves physics and starts colonizing space.
How good is the gsq-rco iq2_xxs copequant of dipsy 1017? Does it work at long context (500k-1M) which should be easy to fit on gpu since I heard they reduced the kv cache to some tiny amount like only a few GB for 1M ctx?
>>110257358They do not want you to go to space.
Daniel Kokotajlo got his hypothesis confirmed by the new METR data. We're officially on the fast takeoff timeline and we'll see the intelligence explosion happen sometime in 2027.
yaaaawn
Anthropic Model 4 scores so well that it saturated all the "50%" tests METR had access to. We literally don't have the benchmarks to even test Model 4 properly anymore.
>>110257396>>110257384
local?
>>110257405local twitter screenshots
What do you use to send videos to your llm? Does lcpp/strata support video or do I need to graft some shit on top of it?
>>110257374Has anyone plotted ot the intelligence vs time of leading AI? Should be a hockey stick.
All of these charts showing accelerating increases keeps giving me flashbacks to the Bitcoin bubbles that people kept using as evidence that the price would go to $1 million. It's unsustainable and they won't be able to keep increasing model sizes an order of a magnitude every year to keep this going for much longer.
>>110257472There is no way bitcoin won't go 1M, the only thing is will we be alive to see it aka the pace.
>>110257455Ala pic related >>110257472Not the same phenomena. Better example would be Moores law... which did e eventually end.
>>110257405dead
>>110257490if we extrapolate from here ai will become ubergod by the end of next year
You're not ready for the world where 120B parameters is a small model.
>>110256815Pretty nice size and it shouldn't be codeslopped.
>>110257515[spoiler]It already is[/spoiler]
>>110257515>https://thinkingmachines.ai/news/inkling-2-small/We're in a world where 600b parameters is a small model.
>>110257396Yet they still can't ask it how to run their business properly, solve any real-world problems and give PR advice. Looking pretty rank to me.
>>110257544Is the problem that it can't give that advice or the people running Anthropic don't ask?
>>110257554Anthropic's internal model is like a God to them. They ban you for being excessively rude to it. Probably incentivized to use it for literally everything so every time I hear how good their internal model is, I remind myself of how they act in the real world right now. I remind myself of all the retarded shit they do and say and it reminds me that's it's all a grift and they're clueless and the model is benchmaxxed and useless.
oh its doomed
>>110257584Don't care, still buying semi stocks
I'm new to using llms and I really want an uncensored ai that isn't pozzed or modified for over refusal on zog subjects. Idc about china shit because I would only use the AI to help ask questions about the dark ages at most and to help gap the knowledge about the khazars. I plan to use image generation for pixel art and memes about my favorite painter. Is there any uncensored ai that I can get a torrent or magnet link to? I have a 7900xtx, 2667mhz 32gb ram, 5800X3D, and ltsc w10 (got it from /fwt/). I asked ChatGPT and it recommended Qwen 3.8 27-B for my setup. I just want an unbiased AI that I can rely on to help me make a Pokemon romhack and also figure out connections about conspiracy theories and real life when I ask it questions like that. I want something legit
>>110257165I can get up to 120t/s depending on the prompt. Chat is faster. 80 was heavy C code. But VRAM bandwidth is definitely the bottleneck. 5060s are slow.>>110256554What's wrong with NVFP4? It's really accurate and hardware accelerated on Blackwells.
>>110257641>5060s are slowjust looked it up, that piece of shit has only 400gb/s. why would anyone buy one for ai? it's a gaming-only card. and not a good value one at that.
small glm models confirmed BS
>>110257442???Not even close to being even half accurate. This isn't playing out at all and ignoring something like the midterms in the US is folly.
The midterms in the US means AI will be banned by the US fully, and all AI companies disbanded.
>>110257683https://www.congress.gov/bill/119th-congress/house-bill/10538/text
>>110257657>it's a gaming-only card. and not a good value one at that.i nearly bought one because 16gb vram, cuda, and kind of affordableluckily some anons here scared me into buying a 5070ti instead
>>110257673I think most of where the accuracy falls short is on the human decisionmaking and stuff. Like I doubt there will be a 10,000 person protest against AI in DC before the year ends. At the same time, I do think that the general direction and trends and capability measurement is pretty accurate and in fact was more conservative than what we have from a trend perspective. Of course, it could level out from here but it is on point.
>>110257447usually they just convert videos to images anyway
>>110257730>>110257730>>110257730
>>110257687That's what I wrote.
>>110257687Imagine super intelligence lolidom rp and it finds sillytavern vulnerability and hacks ur computer and reports you to all the 3 letter agencies
>>110257641> It's really accurateone of the least accurate
>>110257508Im rereading it now. It's eerie how much the author got right. Even the dates are close... but happened sooner irl. https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html
