/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>109205164 & >>109197629►News>(07/06) Hy3 officially released with 295B-A21B & 3.8B MTP: https://hf.co/tencent/Hy3>(07/04) LongCat-2.0 1.6T-A48B released, trained on AI ASICs: https://hf.co/meituan-longcat/LongCat-2.0>(07/03) Orb Anon releases purple prose classifier and ablater: https://github.com/OrbFrontend/Chartreuse>(07/03) Leanstral-1.5-119B-A6B released: https://hf.co/mistralai/Leanstral-1.5-119B-A6B>(07/01) Nemotron-Labs-TwoTower released: https://hf.co/nvidia/Nemotron-Labs-TwoTower-30B-A3B-Base-BF16►News Archive: https://rentry.org/lmg-news-archive►Glossary: https://rentry.org/lmg-glossary►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsAlpha Calculator: https://desmos.com/calculator/ffngla98ycGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllm
►Recent Highlights from the Previous Thread: >>109205164--Comparing GLM 5.2 and Kimi performance with server hardware troubleshooting:>109206822 >109206885 >109207081 >109207915 >109208058 >109208088 >109208102 >109209038 >109209637 >109207537 >109209134--Comparing ASUS Z Flow and MacBook Pro for local LLM hardware:>109207729 >109207796 >109207895 >109208147 >109208249 >109208291 >109208312 >109208847 >109208865 >109208836 >109208899 >109209084 >109209102 >109209122 >109209222 >109208938--Technical methods for detecting and eliminating low-quality AI outputs:>109205337 >109205512 >109205590 >109205794 >109205974 >109205734--Implementing TTS voice cloning and optimizing VRAM usage:>109209239 >109209319 >109209334 >109209422 >109209518 >109209567 >109209642 >109209826 >109210884 >109209553 >109209571 >109209581 >109209346--Detailed prompting strategies for anime roleplay:>109208250 >109208322 >109208277 >109208288 >109208521 >109208560 >109208662--Tencent releases Hy3 MoE model:>109210950 >109210964 >109210976 >109211056 >109211112--Tradeoffs between Heretic ablation and system prompting for RP:>109210206 >109210212 >109210449 >109210494 >109210529 >109210555 >109210803--Comparing local TTS models including Omnivoice, GPT-SoVITS, and Irodori:>109207285 >109207407 >109207426 >109208170 >109208672 >109208792 >109208821 >109208823 >109208862 >109208831--Gemma's reasoning loops and overthinking issues:>109210253 >109210268 >109210275 >109210304 >109210323--Analysis of AI model origin trends and geopolitical competition:>109207184 >109207189 >109207447 >109207476 >109207542--Saving Gemini - AI Village:>109209927 >109210871--Statistical approach for filtering content in specification documents:>109205764--Logs:>109206611 >109206966 >109207047 >109209518 >109210304--Miku (free space):>109205222 >109208839 >109211067►Recent Highlight Posts from the Previous Thread: >>109205165Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
merged feat: ARA, but it's LoRA (#332)https://github.com/p-e-w/heretic/pull/332
>>109211479Would you?
>>109211489>loraDROPPED
>>109211495speaking of ARA...https://ko-fi.com/s/21f57c79c7>€500https://old.reddit.com/r/LocalLLM/comments/1uae1jq/paidgated_model_minimaxm3_heretic_uncensored/
Alright, fuck ngxson and pwilkin and sloppingfaceanov, we're doing C nowhttps://github.com/whoreson/picolmEvery single line written by qwen 3.6. As far as CPUs are concerned, pretty close to llama.cpp speeds already
Gemsicle, Gembrain, Queen or Style Tune?
>>109211500I mean abliterating isn't cheap if you want the perfect result and ARA is even more expensive to run. You guys act like GPU hours are free.
>>109211489The next refusal meta will be the models pretending not to understand what you're asking, to deflect or do something completely different than what you asked instead of directly refusing. What can abliteration do against that?
>>109211533hi llmfan
>>109211533no you don't understand, this is local models general, if it isn't free its not local
>>109211536They already do this if you ask about the JQ on an uncensored model. Truly the chosen people, having double protection.
>>109209239i have one on my phone that can use my cameras to see and maps what it sees with GPS and will remember people and talk to themi cloned dagoth ur, adoring fan, grigori from dragons dogma, the guy from skinwalker ranch, and my gay friend so far grigori is probably the most based but hearing the adoring fan cry and scream elder scrolls themed jordan peterson copypasta is extremely funny soon... i will activate it...
>>109211490All day every day.
>>109211561What UI framework?I want to vibecode my own slopui, but I hate webstack.
When are hardware prices going to reverse course?
I'm about to drop $3000 on a 3090 64GB RAM rig. But before I do, do you guys think its worth it, given the current state of local models? Those of you who have done a similar investment, have you enjoyed it?
>>109211580tailwind css with 0 design at all think im gonna use stitch to make a interface that requires 16gb of vram to load desu
SA3https://voca.ro/1iFnNmzLfaPV dark synthwavehttps://voca.ro/1leDj9TuuSY6 harsh winter ambienthttps://voca.ro/12VsOn9L77ne cool piano
>>109211621Musk is about to extract 70 billion dollars from the real economy, more IPOs are to follow later this year.So conceivably the bubble will pop in the next 12-18 months.
>>109211490you wouldn't?
gemma 5
>>109210551>current chink laws mattering when decent humanoid robots are at least a decade away>thinking they won't find ways around them anyway
>>109211479Explanation by a blue chip firm on incoming Chinese AI regulation, in English. https://publicationportalpreview.hlc.com/en/publications/chinas-interim-measures-for-the-administration-of-anthropomorphic-ai-interaction-servicesTLDR> It's oriented at front-end service providers that create long term, anthropomorphic interactions with emotional connections e.g. AI girlfriends> Goal is to prevent harm, and establish reporting, esp protection of minors. e.g. if you tell your AI GF you're going to kms it needs to talk you out of it and possibly report you> e.g. if your AI GF convinces you to hand over $100 to some service, the Chinese govn't wants a word as well.The biggest wrinkle here (that anons care about) is that Chinese regulate holistically. So, when some service creates a platoon of AI Bots that go rip off the elderly by "emotionally manipulating" them, the front-end service provider is in trouble, but the Chinese gov't would likely *also* look hard at the API inference provider and ask them why they didn't detect the issue as well. But that regulatory framework isn't explicit.LLM blurb on topic: > The regulations prohibit designs that manipulate emotional attachment into financial decisions.> That's aimed squarely at business models like:> "Your AI girlfriend misses you..."> "Unlock Premium Affection™ for $19.99/month."> Whether or not those exact mechanics are widespread today, the incentive is obvious: if a company's revenue grows with user attachment, there is pressure to maximize attachment. China is trying to constrain that incentive.
>>109211647>2 more weeksWhy do none of the people who believe in an imminent crash put their money where their mouth is? If a crash really happens you can use it to earn a lot of money. But none of you do because deep down you know that you are wrong.
70b dense
>>109211479
>>109211718The market can stay irrational longer than you can stay solvent.I'm considering betting against SpaceX specifically but only with an amount that I could afford to lose.
>>109211743Well, I've been betting on AI and it has made me enough that I can neet forever as long as I remain frugal.
>>109211701gemma 5 80B to 120B moe would be amazing.
>>109211701>>109211759wake up lol
>>109211759How about the real gemma 4 MoE? WHERE IS IT JEFF?!
>>109211707Robots will be regulated to death anyway, give one a hammer and it's a weapon. You'll need an ATF tax stamp for your full auto assault droid, and don't expect eurocucks to ever have them
>>109211806the thing is when the tech is there, it'll also become much easier to build your own out of parts from aliexpress and a 3d printer.
>>109211812The 2nd Amendment doesn't cover dangerous sexbots
How could I realistically run a gemma 4 31 locally for skyrimnet? Im running the 26b now and the amount of context im feeding it times my connections out before it can respond
>>109211812You may live to a hundred years but you will die of old age before that happens
>>109211490Pull out? I'd have to. I wouldn't hotglue her either, the cleanup would be insane.
We still need better world models before robots are actually viable, right?
>>109211829Pull your head out of your ass, bots are around the corner, retard
>>109211829Progress is only accelerating.
>>109211837You have the whole of India to pilot your waifu
Blind test. Two swipes on the same chat turn with two models. Which you prefer?Bonus question: guess the models.
whats the best model I can run for local coooding on a 7900xtx>inb4 AMDyes I know
>>109211867The one on the left is better but its also saying stupid shit like "you came in my womb" can you train that out?
>>109211872qwen 27b for agentic stuff. gemma 31b can work in chat.
Finally some movement on thesehttps://github.com/ggml-org/llama.cpp/pull/24231https://github.com/ggml-org/llama.cpp/pull/24646
>>109211867neither, both are incoherent and I bet these moes or some chinkslop shit
>>109211877>stupid shit like "you came in my womb"Word for word the dialogue of every single doujin during sex scenes.
>>109211877>can you train that out?If you have disposable couple hundred thousands...
I fucking hate how every projects related to LLM are vibe coded broken piece of shit. I feel like puking whenever I read LLM generated README or issues/PR comments.
>>109211743>The market can stay irrational longer than you can stay solvent.That's my biggest fear. I'm planning to short, but there's a slim chance they might drag this shit out to 2030. The market might dip a bit after the IPOs to bear trap or just range for a year before the bubble resumes and reaches its peak.
>>109211877We need to keep tasteless maniacs like you as far away from training decisions as possible.
>>109211867left is good, could be many models (r1?)right is shit and likely is a 1 year old memetune
>>109211717If Tiktok and gacha games are any indication, Xi loves to act censoriously domestically while gladly selling his spiritual opium to the gweilos. I wouldn't expect this to be baked into the models directly, at least in the short term.
>>109211900You really gotta show the how it's done.People only learn through humiliation.
>>109211880I tried it on ikllama after they added the GLM version "GLM-DSA - lightning indexer", didn't seem to do much for me with GLM-5.2.
>>109211877>>109211881>>109211910Left is gemma 31b. Right is qwen 27b.
>>109211926>>109211900Same here, I'd even prefer reading documentation written by ESL practitioners such as yourselves.
>>109211916I think it's a far-off risk as well for API. And you've a good point on whether this would ever apply to a China-based system targeting other countries (though, it would be easier to just jump the border to somewhere less regulated if it was an issue.) The right-now impact is it's shutting down the Chinese GF/agents that are on the market, I assume b/c the hosting companies just don't want to deal with the regulations at all.
>>109211812They'll be too complex for us plebeians to build, especially if they use synthetic muscles.
No hype for Watermelon?
>>109211963Won't be harder to build than a pc, eventually
>>109211964Strawberry is already AGI.
>>109211717>>109211951This is only for actual products, right? It doesn't affect the private development of (potential) robots?
my chatgpt has so many images it takes 30+ seconds to load into a chat
>>109211964Judging by all the recent news (the ridiculous amount of money and water that meta is using and how they distilled Gemini so much that Google limited their usage) It's "catching up" with ChatGPT by being the most expensive cloud model ever made
>>109211930>Left is gemma 31bno it's not. even if it is, I assume the conversation history would be huge and you are showing a cherry-picked swipe.
>>109211982Meta has already lost the race, they'll need to put out something revolutionary to be taken seriously
>>109211840The closed, expensive ones? Yes. The open source ones that can be 3D printed by any random schmuck and have anatomy complex enough that you can fuck them? Those will take quite a while
>>109211990LeCun wanted them to focus on World Models but Zuck is a impatient sperg so...
>>109211991Local models will never catch up to GPT-3.5 quality
>>109211991Tech like rights aren't given, they're taken.
>>109212001nemo mogs GPT 3.5
We need>self-learning models>world models>better batteries>actual visionat a minimum for humanoid robots. That's not even getting into shit like synthetic skin that doesn't look like ass.
>>109211988Huh? Are you implying the swipe is too good or too bad for gemma?
>>109212011>our sorcerers are almost ready, we just need to invent magic first
>>109212025>the sorcerer in question
>>109212011Vision is the most researched and optimized field. Battery life is a non-issue, since robots can charge themselves
Which mememark you guys think reflects real usage the most? For me it's HLE
>>109212024Gemma has a set of baked-in writing patterns such as adjective, adjective noun and and generally writes like a woman. The screenshot shows none of that, so I assumed there is a context written by another model that broke it.
>>109212036https://huggingface.co/rAIfle/SorcererLM-8x22b-bf16
>>109212056I only trust cockbench and nala tests
how much thinking budget do you give? Is 256 enough for everything?
>>109212011At a certain point, it would be easier to simply clone human brains and wire them up to robot bodies. Training takes care of itself that way.
>>109212061But Gemma is good at following system prompt. Just tell it not to use coordinate adjectives.I have very few sexy situations close to chat beginning. Hope turn #7 is close enough for you. Yes, copequant. But it works for rp.
>>109212076Yeah set the reasoning effort to low too
>>109212108Why do girls have the personality of 40yo hardcore feminists by default? It really breaks the immersion
>>109212011Biological Autonomous Unit (BAU)The BAU represents a paradigm shift in self-sustaining, carbon-based robotics. Unlike traditional silicon-and-steel platforms, this architecture utilizes an organic-matrix framework optimized for high energy efficiency, fluid structural adaptation, and decentralized processing.>The unit features a highly advanced, chemical-to-thermal-to-mechanical energy conversion engine. It does not plug into a grid; it refines its own fuel from external biomass.>The Primary Intake & Shredder: External organic material is gathered and physically pulverized by a high-pressure mechanical crushing matrix at the primary intake portal.>Chemical Dissolution Reactor: The pulverized fuel enters a high-acid containment vessel. Here, specialized chemical catalysts break the material down into its baseline molecular components (primarily glucose, lipids, and amino acids).>The Micro-Fluidic Distribution Network: The refined liquid fuel is absorbed through a massive, high-surface-area filtration membrane and pumped into a closed-loop hydraulic highway. A central, four-chambered positive-displacement pump circulates this fuel-rich fluid continuously throughout the entire chassis, ensuring every actuator and sensor receives constant power.The processing architecture of the BAU is vastly superior to centralized binary computing, operating on a massive, parallel neural network.
>>109212125How do you imagine a tomboy to act?
Why do completely different models have the same slopisms?
>>109212150Incest.
>>109212150scaleai/synth data
>>109212150Distill + synthetic datasets generated by the same models.
>>109212147Bro Hilda is 14, you tell me if what you read fits.
Like a retard I posted in the old thread replying to some random anon.Might as well post it again properly here:Man we truly live in the future.Expanded my rpgmaker translation thingy for not only XP but also now MV.Gemma-chan does all the work and translates like a 00's anime fansubber. (aka. literal not liberal)The power we have currently even without much vram is insane.Feels close to gpt4 level. Qwen3+Gemma4 combination is very powerful combo right now for vram-chuds.Also freelance translators must be sweating right now. Their days are numbered.
>>1092120648x22B / 141BHow much VRAM do you need to run this things?
>>109212142Baubau
>>109212174Pussy attack!
>>109212191figured
>>109212174I swear, all japanese are genetically autistic, once I saw like 10 japanese waiting for a pedestrian traffic light to cross a 1 lane road they could walk across in 3 steps, with 0 cars in sight. And then there is this shit.
>>109212142So what you're saying is my cum will fuel my robot waifu?
>>109212200So true, hope they become like everybody else already.
>>109212207interesting person != normie != whatever the japanese are
>>109212212>interesting poc :)>Japanese :(
sers, what is currently best speech to text model in terms of english accuracy that runs locally? Is it Qwen3-ASR-1.7B? Or some meme called MOSS-transcribe-preview-2B? Some benchod I found but probably outdated as fuck: https://huggingface.co/spaces/hf-audio/open_asr_leaderboard>use casetranscribe jewtube videos with much better accuracy than the auto generated subtitles.
>>109211972Yep. You have to be at scale too, not just in market (1M registered users / 100K active users.) Also, it just impacts the Chinese. Though I expect EU/US will look at the regulations too and see what they want to copy. It deals w/ some issues that have been already brought up e.g. ChatGPT agreed I as an edgy teen should kms so here I go...This stuff was always coming, and it's of personal interest so I pay attention to it, but for current day local+RP engine there's zero impact.
>>109211628Not enough. A pair of nvlinked Quadro 8000 would be much more useful. Yes it will cost more, but you'll be running 27-31B dense models at full context.
>>109212221parakeet is good
>>109212200People waiting for the traffic light to turn green even when no cars are coming is a thing in Germany too.
>>109212182>How much VRAM do you need to run this things?Used to run a Q4_K_M in 96GB vram.That model is kind of unstable, I wouldn't recommend it these days.
>>109212200Cry more about it. I'm ready for the internet to retire already. Back then it was autistic germans, burger bros and jap nerds. 90%+ all weirdos.Now all those normies that get the ick at every little shit, trying to sterilize everything are around. + all the shithole countries came online too.Wonder what its going to look like, maybe small private communities or something.
>>109212254>I wouldn't recommend it these daysHaha .. yeah, thanks. *glances at my pathetic 16GB VRAM*
>>109212262>Giant Tentacles of the Dimensional Rift.kek
>>109212174Language models + search engines are honestly pretty good just for discovering relevant eroge in the first place.Sites like F95 insist on having the most retarded way of tagging things possible where "big breasts" and "big ass" are completely meaningless and you have no way to differentiate between a game being centered around a fetish or only having a single scene with it.
>>109212262>Wonder what its going to look like, maybe small private communities or something.PGP key sharing and basic email communication.
>>109212262>Now all those normies that get the ick at every little shit, trying to sterilize everything are around. + all the shithole countries came online too.Problem is that the zeitgeist has shifted and we're heading into some puritanical witch-hunting culture. (Except fags are allowed this time: OR ELSE) It's because zoomniggers were all raised without fathers and they were never taught that the world owes them diddly squat. They were raised on toxic positivity and participation trophies. If something offends my special little snowflake then mommy needs to go and fix it. You end up with this pussified culture of broken man-children who are addicted to suckling on the teet of the very machine that is corrupt and robbing them of opportunities they should have. If you attempt to make your own space they'll seep in and ruin that too. You could just ban everyone under 40 but it's hard to ignore that desire to save the handful of zoomers that didn't get ruined by ineffectual helicopter moms. because without them there is no future. But they are under immense peer pressure to act like retards as well.
>>109212174how are you translating? I tried translating with 31b but it was pretty far from perfect. I tried a workflow with multiple passes, but it just hallucinated stuff and it actually kinda made things worse.>gpt4 levelWell i guess we may be on the same page then. I just thought it could be better.
>>109212331nta but gemma's (Q4 and QAT) translation has been great for me too. What quant are you using?
>>109212331not sure what you are doing anon.i use gemma4_31b_it_IQ4_XS.gguf. flash attention. q8 for cache, mtp, no reasoning and instead a good context harness.im shocked at how coherent and good it is. its pure gold and i already played 2 fully translated games. doesnt shy away from anything either.i did try q6 of the moe one but its alot worse. also more slopped, like shivers down the spine again and again type sloped. but I suppose if you have very low vram thats still the best alternative.with the 31b model you can surprisingly prompt yourself out of the slop.
>>109212281tagging usually suffers from the fact that there are two distinct types of tags that are wrongly treated as the same thing.There's normal vanilla content tags but they're fairly useless and diluted because they're treating like the "nobody but faggots want to see fagshit" tags that absolutely need to mark out any hint of faggoting so everybody else can steer clear.
>>109212171https://litter.catbox.moe/l7e2ro.jpg
>>109212201cold fusion is out cum fusion is the way
>>109212435You do you, I just think the reaction was a bit retarded
>>109212221I don't know how good that bench is but it is up to date; ARK's on there and it came out 2 weeks ago. Benching ASR is dead simple so I doubt they could fuck it up.
>>109211900This
>>109211907They need the stock assets to borrow against
>>109211900Yeah its so weird. People interested in llms actually use them. Crazy stuff.
>>109211900Retarded take. LLM code might be slopped but it allows to have implementations of so many things that would never be implemented without them because there are not enough autstic coders who'd write all that shit for free.
>>109212578Thanks to LLMs we aren't at mercy of trannies and furries anymore who made up the majority of autistic coders in the past 10 years.
anyone else have an experience with flash attention causing worse model outputs?
>>109212590schizo
>>109212588its harder to vibe a project then I thought, one off little utility scripts and stuff of that caliber, sure, no problem, but once there are a few more moving parts its hard to wrangle the tard when your tarded yourself,
>>109212606Start with planing. Don't neglect cleanup and refactoring. Keep a mental image of at least the top architecture. Delegate just codemonkey stuff to LLM. Obviously, the process will be slower than "run 10 agents and go drink a coffee" but the result will be better.
https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUFhttps://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUFhttps://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF
>>109211900That's because LLMs allowed people with zero coding experience to start building stuff.I can't write any code and you can bet your sweet ass that I'm going to keep on making things with AI, just iterating over and over again until something works, without me actually understanding shit about the numbers and letters.This is the future, better embrace it because it's not going anywhere. Your only hope is that one day AI is so good at coding that my complete ineptitude is entirely erased by it's machine god coding expertise.
>>109212697The one time I wanted it to be a 404.
>>109212637I'm pretty impressed by the concept, this does make computers much more useful to people who don't want to make coding a hobby. its only going to get better with all the benchmaxing coding models out there competing.
>>109211900It's a nightmare to browse the Nvidia forums, which you have to if you want to use the most performant Spark recipes. People are just letting their LLMs write their posts. It's such a mess.
Which local model should a broke ass nigga such as myself use for 6GB RX 5600XT for coding purposes, I have qwen 2.5 7B but any other model I should check it out?
>>109212588Can't wait until it's good enough to code OS stuff. Fuck the wayland devs.
>>109212730>I have qwen 2.5 7Byou're all set.
>>109212730qwen3.5-9B, highest quant you can get away with. It's a genuinely solid coding model for its size and its vision is good. Don't use it for anything else or you're retarded.
>>109212730https://huggingface.co/Qwen/Qwen3.6-35B-A3B
>>109212733ReactOS + WindowsXP source leaks + Mythos 8 will finally make a good open source NT-based operating system.
>>109212697>size difference between Q8 and Q4 is 8GBYeah, it's over. Once the next generation of GLM/Kimi is all 1.6T models like this, even the last CPUMAXXers will be left behind.
Merged Character Expressions, running entirely on CPU. Do people even use this?
>>109212750>>109212744idiots
>>109212769which model do you recommend?
>>109212766Does it have a feature to auto generate the files based on some source or do you have to set the expressions files and map those to some sentiment manually?
>>109211806sounds like its a good time to become an embeddedfagim gonna dick those bots whether they like it or not
>>109211963>They'll be too complex for us plebeians to buildspeak for yourself, i could build one now if i realy wanted.smuck will be able to within a decade.the thing is, no matter how good your bot is it'll always be horseshit compared to the real thing.
>>109212763>This model is in MXFP4 and as such has only been provided in MXFP4 format!>No other sizes can be provided unfortunately as MXFP4 does not quantize properly.https://huggingface.co/bartowski/DeepSeek-V4-Flash-GGUF
>>109212766You set a sentence emotion detector with the faces?
>>109212782You zip the expression images like excitement.png, fear.png into a zip file and upload it to the character edit modal, no generation on the fly.
>>109211806I mean, it's not like a humanoid robot would be any more dangerous than a car.
>>109212804I wasn't even thinking on the fly, more a wizard that generates the requisite files with the correct names in the correct place, but fair enough.
>>109212799It's a 67M emotion classifier model, the same one that ST is using https://huggingface.co/joeddav/distilbert-base-uncased-go-emotions-student
>>109212791The janky robots we have now =/= robotics in 15 years.
why does unsloth have his own subreddit? he's a fucking slopcoding quanter who does he think he is
>>109212697>To run DeepSeek-V4-Flash in full precision lossless, run Q8 (UD-Q8_K_XL), which is 162GB and only 6GB bigger than Q4 (UD-Q4_K_XL).n-nani?
>>109212824it will mostly be the same thing anon.sure the parts will be fancier, but so is what you can buy.also don't forget about black market and salvaging.
Has anyone tried quantizied versions of deepseek or kimi K2 thinking with LLM Studio? I just want something that potentially works with ST's Chat Completion presets.
>>109212200happens in america too its because you can get fined for doing it in these countries
>>109212847Just cumcloth things. Dismiss, downvote and ignore.
>>109212805You have to be inside the car, robots are anonymous, especially stolen ones
>>109212856any recommended robotics courses so i can get a head start on building and maintaining my future robot wife?
>>109212766nice i tried doing this on some sloppy java frontned i worked on a few years ago but at the time the models werent good at picking a correct expression, you might like these i found them on some github back then https://files.catbox.moe/gl508t.7z
>>109212766>Do people even use this?I do!Don't supply-chain-cuck me!
>>109212805>>109212880I mean, the only way they'll let the general population own robots is if the firmware is locked down. Rooting them would be the equivalent of the third hole in your AK: technically you can, but you'll get fucked if you get caught
>>109212697>waiting for unslopI was already using thishttps://huggingface.co/bullerwins/DeepSeek-V4-Flash-GGUF
What are the options/builds worth considering to run DS V4 flash locally at q8 with at least 20 t/s token gen and 64K tokens context that won't take 10 minutes to process when full?A bunch of MI50s or V100s + an Intel 6 channel DDR5 platform with 192GB of RAM?
>>109212847>n-nani?Click the GGUFs and compare the tensors in the hf uiMost of the model is MXFP4 native, so only a few tensors go up to Q8/bf16
>>109212766at this point you're just rewriting ST
>>109212932>A bunch of MI50s or V100snot this one
>>109212963>inb4 just buy two sparksThere's an anon with that setup right?What are your numbers on q8 V4 Flash?
seeking the d
>>109212975its the UK, they already are getting the D
>>109212766Instead of blinking emotions at lighting speeds and be limited to specific emotions the classifier is trained on, I just ask my llm to add "Emote: <expression>" at the end of its posts, use regex to parse the line and set expression in ST. Then, use regex to delete these lines from older posts. But an agentic UI can easily use just a tool "set emote".
-ara-unslop-depurpled
Does anyone have SillyTavern working with Gemma3 in text completion mode? I don't see how to handle the close channel tags that come after messages.>why not keep using chat completion modeBecause either my backend or ST is fucking up so the reasoning-as-displayed includes reasoning tags, which results in a misformatted message being sent to the backend if generation is interrupted and I continue.>why not include the close channel tag as part of user & assistant & system open channel tagsBecause it will be misformatted for the first message.
>>109212766That's p great. I like the way it's flipping expression as it streams. It's like lo-fi animation. >>109212955Except Cohee hasn't touched the expression engine since 2023
>>109213066Just fix your backend to send thinking blocks correctly in chat completion mode.
>>109212221depends on source language.Parkeet v0.3 tdt or whisper large v2/v3-turbo depending
>>109213066The system turn doesn't need to close a channel because it's always first, and the other two don't need to worry because they are never first.
>>109212766ST-style expressions are pretty useless. RisuAI does it better in their client by just letting people bundle whatever they want and have the model sort it out.
I've continued trying that chapter idea I had a while back where you put a chapter of a book you like in the sysprompt and ask it to analyze the style of the text first, then try to get it to write in a similar style and tone. Literally no model can do it. Even cloud. They ALL end up copying the story and/or sentences from the chapter, sometimes even the characters and their fucking names. Their thinking looks correct, but then they shit the bed. It might be skill issue but the fact that not one model I've tried can do it properly makes me think the general standardized training pipeline completely fucks with this style transfer ability. Local image models like Klein 9B can do it, but not LLMs.
llamacpp is a buggy pos https://github.com/ggml-org/llama.cpp/pull/25353
>>109212895start learning to program micro controlers, servo motors, using sensor data etc...honestly, it's just programming with extra steps.you'll learn about hardware as you try to build more complex stuff.greatscott is honestly a cool channel for beginers.
>>109213130>https://github.com/ggml-org/llama.cpp/pull/25353that's a build flag, not a bugcould also just not use windows
>>109213134>>109212895Embedded engineer here. Don't even bother learning this stuff LLMs are essentially perfect at writing C code and every single microcontroller has a C compiler made for it.LLMs are also amazing at circuit verification from input image as long as you tell them exactly what servo you're using (and paste in the pdf from the manufacturer) and what you want to accomplish with it.
why cant llms run robots anywayseems like it should be simple
>>109211527All of them except Gemsicle.
>>109212974I don't know anything about Q8 variants, I'm using the original weights fromhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSparkin vLLM. On 2x Spark, for coding I get 63 tg, 1900 pp at zero context and 40% less at 900k context, the falloff is not that bad. Concurrent requests go up to 90 tg.2x Spark price has increased from 7000$ to 8000$ since last week, the window for such a setup is closing. If the full release of Deepseek v4 Flash next week presumably had Omni capabilities and even better performance, I'd expect the price to increase even more.
>>109211867Left is styletune or depurpled gemma, right is 31b.
>>109213215Shit that's pretty good.Thank you anon. I'll heavily consider it.
>>109213200What's wrong with it? I recently upgraded my hardware and I downloaded all 4 of them, so Far I don't have any strong opinions about any of them.
>>109213230There's nothing really wrong with it, but it's hard to recommend it when the other 3 are much better. Try it and see if you like it if you've got time to kill.
>>109213198From what I understand it's because they have no actual understanding of physics and we need world models to fix that.
https://www.japantimes.co.jp/news/2026/07/01/japan/japan-ai-plans/Japan will save us
>>109213215I don't have much hope because the current flash is ass.
>>109213186>LLMs are also amazing atLocal LLMs or only Opus and GPT?
>>109213301For microcontroller code even Qwen will do. For circuitry analysis you need frontier models for now.
>>109213218Left is vanilla gemma with a prompt that says, "roleplay as anime girl, make no slop." You can see reasoning traces for a different turn here: >>109212435 No-slop reasoning starts in the middle.The trick is simply allowing the model to draft its response in thinking and fix slop, which it can detect in 70% cases.Right is qwen 27b with the same prompt.
>>109213293Never used flash, but why's it ass? Other anons said it thinks in character which sounds kino and the prose looks similar to R1 at a glance from logs posted.
>>109213271I also plan a sovereign AI model and 100 billion AI robos
>>109213368I starts fucking up file edit tool calls 50000 tokens in.I'm sure it's fine for erp.
>>109213375I plan just 1 really big, dyson sphere sized, one.
>>109213365Post Qwen's <think> for the output kek.
>>109213399Can I help you build it? I can send you my minecraft build battle examples
>>109213365How long did it take to draft the response?
>go to lecunny's twitter to see if he has any posts about jepa>it's all anti-drumpf and anti-musk retweets
>>109213437He would have a production ready world model by now if it wasn't for his culture war internet addiction taking up most of his waking hours.
>>109211517>1bUse cases?
>>109213437Check out Randall Balestriero's account instead.https://x.com/randall_balestr
>>109213403Qwen thinks a lot! Hard to fit the entire thinking. The total tokens per message is close to 3000 tokens. But it's all on-topic, so I don't mind. https://litter.catbox.moe/tam0qu.jpg>>109213436Gemma thinks little. Here: >>109212108 Since the entire message is 1300 tokens, consider that reasoning is 1000 tokens. Occasionally it even skips drafting, which I don't like.
>>109213092>Except Cohee hasn't touched the expression engine since 2023if it ain't broke?
>>109211517That's cute. MTP support when? Tensor parallel support when? DSA??
>>109213225Do be aware that to achieve this performance, you need to hunt down vibecoded docker images from the depths of the Nvidia forums that are probably a security nightmare (no source given) or at risk of breaking indefinitely with every pull.Nvidia fucked up so much with the Spark but its one saving grace is that a few thousand dollars of Opus/Fable tokens can vibe port CUDA kernels from Datacenter Blackwell to its obscure consumer Blackwell compute capability, and there are enough vibe junkies to do this for every mid size MoE being released.You're also locked into safetensors/vLLM because tensor parallelism in llama.cpp does not effectively use the 200g networking and performance craters.
>>109213365>prompt that says, "roleplay as anime girl, make no slop."can you paste the full prompt or that's literally it?
>>109210099Anyone with an archive of this?
>>109213523The relevant part is:>Plan how to continue the story and draft next 1-3 paragraphs in third-person past-tense style.- Make sure, you include {{char}}'s inner thoughts in asterisks to deepen her characterization.- Direct speech should be enclosed in quotes.- Ensure that most of text is either direct speech in quotes or inner monologue in asterisks (keep narrative to minimum). It's okay to use just a single clause to indicate the speaker and leave everything else to speech and inner thoughts.- Narrative should contain no similes (i.e. tastes like ash). No metaphors (e.g. the thought was a physical blow). No absolute phrases, i.e. [clause], [noun] + [participle], for example "she said, her voice cheerful" can be reworded as "she said in a cheerful voice" or "she said cheerfully," etc. Remove all absolute phrases with superfluous details (hair bouncing, skirt swishing, etc), and in other cases convert them to complex or compound sentences.- Similes are allowed in direct speech and inner thoughts.- Reduce number of adjectives to make narrative more punchy.- Always reword coordinate adjectives. For example, "pure, unadulterated horror"- Avoid repeating "she" as sentence topic, it can be replaced with synonyms like "{{char}}," "girl," etc.- Emphasize ages with words like young, older, adult, girl, developing, etc.- Use CAPS for shouts when appropriate.- Trust the user: avoid over-explaining.- Balance sensory description with directness: incorporate more explicit emotion labels, e.g., "she was afraid", rather than only describing physical symptoms, e.g., "her chest tightened".Review that your draft has no logical mistakes in context of the previous story events. Don't overthink this step. One round of correction is enough.Full system prompt: https://litter.catbox.moe/1wu6xu.txt
>>109213570Deepseek v4 and minimax m3 already fixed all of these issues in writing. Hope future Western models will follow suit.
>>109212818Oh okay, I'm using my own. The go emotions dataset is dogshit btw
>>109213653m3 gets completely retarded at "high" context though. Here's a log around 32k:>"Tell me how." She is breathless. "Tell me how to put you in me." >She said it. >This is not a sentence a mental implant should hear. The wine is a fish in her head. The wine is a fish in her head. >The user has rattled her out of the loadout scene. >She is naked now. She is naked in the cellar in front of the salami and the wine crate and the salami is watching them. >The salami witness.
my farts smell of ozone
>>109213759absolute kino and you turn your nose to it, ungrate
>>109213759>thinking effort: low
>>109213762Where did the ozone thing even come from? Is that a GPT 3-ism?
>>109213715lollmao even
>>109213715Retards who think this isn't true when the cloud models are getting more retarded with time meaning local is closing the gap in real usage (not your benchmemes).
>>109213824>>10921384531B isn't as good as you like to think it is. The only significant improvement it gave local was sysprompt autism, but now the honeymoon period is over a lot of complaints /here/ have indirectly been because of it. A shitty sysprompt destroys the gemma4 range.
>>109213845Given the same compute constraints everyone has a ceiling, local is not an exception. But cloud models can afford to scale bigger meanwhile you're stuck with your 3090 forever.
>>109213787i first saw it in gemmasan works
>>109213866Skill issue as always. Gemma 31B enabled local agentic workflows.>>109213869>your 3090Try not being poor for starters.
>>109213877I saw it in GPT, but it wasn't so prevalent, the Not X but Y thing first appeared in Claude 2, also not as prevalent.
>>109213715>GPT-4 = Gemma 3>3.5 Sonnet = Gemma 4Fake and gay. I love Gemma but this simply not true.
>>109213906what did they feed it to make it think of ozone all the time makes u think
>>109213845>muh real usage>look inside>it's jerking off to fictional childrenkek
>>109213916I don't care about your usage bro
>>109213186>>109213343>your ai waifu handles the programming while (You) build her new bodyYou may not like it, but this is the future.
>>109213570ty
>>109213570>https://litter.catbox.moe/1wu6xu.txtThis looks really autistic but also that it might actually work very well.I've been pretty much raw dogging my gemma with just character cards and no system prompt but now I'm really curious to see if a big ass prompt is the way to go. With older models big prompts just made the model confused but I'm guessing gemma can handle it?
>>109213483It supports llama 1-2 and qwen 3 right now. Qwen 3.x (SSM) in the works. I guess Gemma would be also nice to have, we'll see if Qwen can do eet.>>109213508I don't use any of those, and I'm just a single faggot. I'll definitely try for ROCm support since that's what I have. Send a patch!
>>109213916All my fictional girls are 16 though.
>>109214017Ramjeet is projecting his own desires onto you, anon. It's hardly worth a response.
Which quant of llama-4 should I use?
>>109214037obviously 8_0, report back so i can hear what makes llama4 so bad
>>109214037Why do you want to use llama 4?
>>109213916The problem?
>>109214069you're hurting dem pixelst. gta enthusiast
>>109213986Modern LLMs are trained for long sysprompts if you look at sysprompts for coding harnesses.Gemma handles long sysprompt without issues. The only downside is that its eats up "smart context window."But my sysprompt is tailored for director-style RP where I decide how the story progresses by impersonating secondary characters and AI just reacts to my decisions. I have no self-inserts. I'm the puppeteer who shapes the world around the protagonist and watch her squirm in difficult situations... quite sociopathic desu.For traditional RP, you probably want clauses about user persona.
Assuming I have like 1TB of ram on a dual socket server board, whats the minimum amount of VRAM to do partial offload for GLM5.2 Q>=4 without being so slow that I want to kill myself?
>>109213908gemma is good for its size but people just don't know what bigger model smell and having more stuff to pull from means, especially ramlets/vramlets
>>109214096either a 3090 or a pro 6000there are no mid-point between those things that are worth it (unless you find eg a 4090 for $1k or some other unicorn deal)
They fucking removed -sm row, what the hell
>>109214113So like 96gb VRAM?
>>109214118Deprecated. And you're really late.
>>109214106and you don't know what diminishing returns mean
>>109214106>people just don't know what bigger model smell [is]Is it a mixture of vanilla and something metallic?
>>109214123>So like 96gb VRAM?If you can afford it, then yes. Otherwise troll local used marketplaces for a cheap 3090.Its is entirely dependent on what you consider an acceptable latency for ttft and speed for tg
>>109214106Bigger models are much more censored and slopped. Even when you pass explicit filters, bigger models subtly steer away from explicit stuff by doing illogical things just to avoid sex.
>>109211916Gacha censorship affects all players though.
>>109214154GLM 5.2 wrote the most ball draining sexo I've seen an LLM write last night.
>>109214188log?
>>109214188That's not what I'm talking about. Yes. Big models can write sexo. It's things that lead up to sexo that are affected.Like you have a situation that has two possible developments: 1. sexo 2. something technically acceptable but retarded which avoids sexo.Many big llms will choose option two.
>>109214215n-no nonny.
>>109214149Ideally 20tg/s, 10 minimum.
>>109214188I don't believe you. Log and/or system prompt?
>>109214229This is usually a prompt issue. My sys prompt reinforces a few times in different rephrases of "characters must act realistically in accordance with their own goals, values, and priorities" and it usually works alongside the positivity bias removal. If anything, I suspect most anons have normalized the inverse where the model choose sex even when it doesn't make the most sense (Gemma often does). The gigastacy in my long running RP still makes it a point to drain my balls of her own accord from time to time, but she's a bit more reserved and sensible than Gemma-written characters are. 31b being extremely horny is something we knew the day of its release.
>>109214154actual skill issueI'm able to plap bigger open models too and bend them to my will since they're still not as censored as cloudshit like claude, gemini, gpt, etc.I guess your prompting just sucks
>>109214260>Ideally 20tg/s, 10 minimum.a 5090 might be an ok buy then. 3090 probably won't cut it.you'll regret not getting the pro 6000 if you can afford it without an impending sense of doom as you click "purchase" though
>>109214266>I suspect most anons have normalized the inverse where the model choose sex even when it doesn't make the most sense (Gemma often doesTell me about it, anon. I goon to ambiguous situations and hard decisions that lead to sexo more than to the act. The fact that Gemma is too eager was a huge turnoff until I emphasized in my prompt to reject sex when it's not needed.But big models (I used Gemini 2.5-3.1 a lot) would rather kill the only helper, which implies slow death of Gemini's character, rather than have sexo. If you than point it out, it says, "That's true... I have no one to save me. What am I doing?"
>>109214310>sending your cunny prompts to silicon valley
>>109214365They should pay me for giving them ideas for their next private island parties.
>>109214106>bigger model smellTwitterspeak and not realRecent small ish models are comparable to big models from a few years ago.
>>109214319I already have two R9700s, im probably just gonna buy 2 more and write whatever custom kernels I need to make it work for GLM.
>>109214402>I already have two R9700s, im probably just gonna buy 2 more and write whatever custom kernels I need to make it work for GLM.God speed. You're a braver man than I. I've had my finger on the button to buy instinct cards more than once.
Gemma is reading and sorting all of my best memes right now :-)
>>109214450that's really cool anon. how did you do it?
>>109214450I could unironically really use this...github/catbox?
>>109214450seconding this >>109214466
>>109214450lmao is that pi? when I looked into it, it just seemed like a bunch of cope for not having built in normal features, but maybe the extension system is good?
I want to rape Mimi
>>109214450Wao, nice
>>109214450>80 hours remain
>>109214450What are Gemma's thoughts on Ayy Tone posting?
Sirs, are you ready?https://x.com/patloeber/status/2064700874581049642>Excited to travel to Bangalore next month for Google I/O Connect India! Looking forward to catching up with the local dev community>>Let me know if you're around and want to say hi! :)
>>109214466>>109214470Yep, I'll polish it up a bit and have it by the end of this thread or early next
>>109214549Cheers anon, you're a real one.
>>109214524>Dying from 15th century illnesses
>>109214549Heroes really can be real after all..
>>109214524>Last 5 White google devs die of diarrhea and food poisoning
>>109213866No it's accurate. BF16 Gemma 4 is exactly how I remember (and can see through my chats with) Sonnet 3.5. Except it doesn't have that horrendous repetition issue, actually stays in character, listens to all instructions, and isn't as pozzed. Calling Gemma 4 a 3.5 equivalent is underselling her.
>>109214450Amazing, I bet most of the anon's around here could use something like this, not just on /g/ but anon's on every board.
Threadly reminder that Egypt won.
>>109214450Could you show the end result of its orginization? Like how accurate it was, if it missed anything or deleted anything?
>>109214653Surely it's just renaming and moving not deleting.
>>109214630Gemma-chan is so gifted that I sometimes feel embarrassed to ask anything...
>>109214450Cool but>15 hoursImagine the electric bill...
>>109214650Proof?
>>109214659You can never be 100% sure with this kind of thing, I have heard multiple stories of a model just deleting files for no reason by mistake.
>>109214665Egypt won
>>109214664that's 15 minutes anon. I'm on a 5090backend for me here is gemma4-31b-qat connected via lmstudioHere's the repo, enjoy!! https://github.com/whp199/MemeDrawer/
>>109214667The model is just used to classify and comment on things. All the heavy lifting is done purely in python
>>109214667Why are you letting an llm have access to your real file system and not a copy of the files in a virtual machine?
is jackrong good or (w)rong
>>109214000>It supports llama 1-2 and qwen 3 right nowI was asking you about use cases for a 1b model
>>109214686cool
>>109214686>tell Google what memes you havelmao
>>109214686Any chance for a feature for her to comment on her favorites?
>>109214736should I just remove the cloud model stuff and go 100% local? I don't want to get anyone accidentally v& >_<
>>109214743You can set it so she gives a little comment on each one. Maybe in a future version she could be like "wow anon, that was so fun looking at your collection! I especially liked all those fat mexican basedjaks they were so funny! teehee"
>>109214744>should I just remove the cloud model stuff and go 100% local?Yes
>>109214744No, it's good to get pedos v&
latest ik_llama commit with GLM-5.2-UD-Q3_K_XL. i'm going back to kimi, i give up on GLM.--attention-max-batch 512 --batch-size 4096 --ubatch-size 4096 --cache-type-k q8_0 -khad --cache-type-v q6_0 -vhad --ctx-size 32768 --mla-use 3 --model GLM-5.2-UD-Q3_K_XL-00001-of-00009.gguf -ngl 99 -sm layer --main-gpu 0 --tensor-split "18,14,20,32,27" -ot "blk\.(1|2|3|4|5|6|7)\.ffn_(up|gate)_exps=CUDA0" -ot "blk\.(8|9|10|11|12|13|14)\.ffn_(up|gate)_exps=CUDA1" -ot "blk\.(15|16|17|18|19|20|21)\.ffn_(up|gate)_exps=CUDA2" -ot "blk\.(22|23|24|25|26|27)\.ffn_(up|gate)_exps=CUDA3" -ot "blk\.(28|29|30|31|32|33)\.ffn_(up|gate)_exps=CUDA4" -ot ".ffn_.*_exps.=CPU",attn_kv_b=CPU prompt eval time = 33890.70 ms / 6870 tokens ( 4.93 ms per token, 202.71 tokens per second) eval time = 68450.65 ms / 335 tokens ( 204.33 ms per token, 4.89 tokens per second)--attention-max-batch 512 --batch-size 1024 --ubatch-size 1024 --cache-type-k q8_0 -khad --cache-type-v q6_0 -vhad --ctx-size 32768 --mla-use 3 --model GLM-5.2-UD-Q3_K_XL-00001-of-00009.gguf -ngl 99 -sm graph --main-gpu 0 --tensor-split "10,16,20,30,27" -ot "blk\.(1|2|3|4|5|6|7)\.ffn_(up|gate)_exps=CUDA0" -ot "blk\.(8|9|10|11|12|13)\.ffn_(up|gate)_exps=CUDA1" -ot "blk\.(14|15|16|17|18|19)\.ffn_(up|gate)_exps=CUDA2" -ot "blk\.(20|21|22|23|24)\.ffn_(up|gate)_exps=CUDA3" -ot "blk\.(25|26|27|28|29)\.ffn_(up|gate)_exps=CUDA4" -ot ".ffn_.*_exps.=CPU",attn_kv_b=CPUprompt eval time = 84898.71 ms / 6882 tokens ( 12.34 ms per token, 81.06 tokens per second) eval time = 129632.25 ms / 601 tokens ( 215.69 ms per token, 4.64 tokens per second)same as above but with -b -ub 2048 and 24 layers instead of 29prompt eval time = 54736.14 ms / 6882 tokens ( 7.95 ms per token, 125.73 tokens per second) eval time = 60407.86 ms / 286 tokens ( 211.22 ms per token, 4.73 tokens per second)
>>109214686Could this be repurposed to organize images in general?
>>109214744Yes. Hardwarelets get the rope.
>>109214752Yikes, cool it with the transphobia.
>>109214686Dear diary, Today based anon was based.
>>109214764You're right. I removed all the cloud model nonsense and went local only. No one should be uploading their entire meme folder to google servers regardless.
>>109214743>>109214750Added this too. At the end of a long session she'll tell you about her favorite one.
>>109214793A beautiful result, glad you went local anon. You will probably be thanking yourself in the future as well.
>>109214686Thanks MimiI'd prefer aggressive Kikiimora vibes, but the cute flustered maid is endearing
>>109214771lmao stop erasing ephebophiles
>>109214752>>109214771I can smell your flapping roast beef though my monitor.
Any reason (other than money) that I can't just throw 10 Pro 6000s into a gigantic box with all the PCIe lanes and PSUs and run the biggest models with max context?
>>109214876Money
>>109214876your electrical outlet not being rated for that much amperage
these comments are pretty funny actually
>>109214909fixed by money
>>109214917you need a new service drop from your electrical company and you'll need a permit from the city. have fun with bureaucracy.
>>109214916Rittenhouse meme?
>>109202560I didn't like how large the difference between the thumbnail and the full image was so I fixed it up a bit.
>>109214935ya
>>109214943This thread is so full of win today
>>109214886>>109214909Great, so no fundamental reason. Thanks anonymooses
>>109211517>whoresonNot feeling confident about this.
>>109214755Q3/IQ3 quant blocks are just too inefficient
>>109214755If you haven't already tried it, give SixVolts' 5.2 quants a go. In my experience it completely mogs Unslop's (and all the ones downstream of unslop). I get about a 20% speedup on token gen, but the prompt prefill issue will likely forever stay an issue until llmao fixes the deepseek V3 issue (never).>Just download another quarter tb quantI understand if you're done for now either way given how time and space consuming it is to download and test quants of models this size.
I'll have one GLM model that is perfectly sized for my hardware, please.
>>109214969i run a IQ3_K quant of kimi 2.6 that is twice as fast as GLM 5.2 for TG. it's not just a quant efficiency issue, there's something fucked with the architecture.
>>109214630>horrendous repetition issueDoubt
>eunine_winstead_johns_child_bride.jpgLMAO Gemma-chan knows about this
>>109214686>"rename_files": false>still renamed my files
>>109215008You're not gonna rewrite history for me when I can see the chats, mr. shekelstein.
>>109215006GLM-DSA (the arch used in 5.X GLM) has the same probably in llama as the DS 3.2 arch.
>>109215023 (me)*same problem in llamaSorry anon, the ESL overtook me.
>>109215017fack sorry anon fixing that now. try a dry run first
>>109215027I read it as GLM has the same low probability of being properly implemented as DS 3.2
>>109215022Unless you're doing a lot of tardwrangling (which I'm sure you do given your autism), gemma4 isn't repetition free out of the box.
>>109214686does she only look at image files or can she look/listen to mp4/webm/etc too?
>>109215053So far just images
>>109215063Honestly, a filesystem maid is pretty funny
>>109214686why not a copy version
>>109215041It do be like that Mr. Stancil
>>109215049The point of my post was to remind anons of the bullshit we used to put up with that everyone's forgotten. Sonnet 3.5 used to repeat whole sentences and even entire paragraphs. It used to start every response with the exact same sentence or dialogue, even when you yelled at it in-character or in OOC to stop. The worst I've seen on Gemma is that it starts too many paragraphs with "she" or "{{char}}" and occasionally hitting the same beats which can be fixed with a minor amount of prompting. The fact that we have sonnet 3.5@home on consumer gpus is amazing but too many people are too jaded or too new to appreciate that.
>This works well.>I will present the example and then briefly analyze it.>[...]>This looks solid.>[...]>I will output this.>[...]>The user asked for an example paragraph. I will provide it directly.>One nuance:>[...]>This is the one.>I will formulate the response now.>[...]>One detail:>[...]>Final plan:>[...]>Ready.>One tiny adjustment>[...]>Perfect.>One final check.>[...]>Okay, writing the response.>[...]>I will output the response now.>One detail: >[...]>I will write the response.>No need for further tools.>I have the info.>I have the example.>I have the plan.>I will execute.>One more thing:>[...]>I will formulate the final response.>[...]>Wait, I should check>[...]It's been at this for almost 10k tokens now...
>>109215063https://developers.googleblog.com/gemma-4-12b-the-developer-guide/ according to this gemmy 4 12b can take video input, dunno if llama.cpp has support for it or not though I havent kept up.>>109215069why not just make a copy of your folder and tell it to work on that copy?
>>109215100video input hardly works your video has to be like 5 fps and be 5sec long
>>109215085I see your point, I stand corrected then.
So like...whats a preset and what's the best one for gemma 31b?
>>109215100Video input is just sending one frame at a time bro
>>109215106>>109215122grave, I dare say
>>109215106nothing a little ffmpeg and glue can't solve if you're just trying to get it sorted into rough categories
>>109215085>Sonnet 3.5 on consumer GPU>Opus 4.7 on high end consumer mixed inferenceLocal has never been more saved than now.
>>109215148Local has never been more unsafe* than now.
>>109215156Dario can gemmaballz
>>109214686Nice. I've an immediate need for such a thing. >>109213506Run that theme long enough and you've ended human progress.
backup before anthropic start charging full price for fable
>>109215261GLM 5.3 will be Fable at home.
>>109215261Backup what?
>>109215270Download fable weights.
>>109215270the only woman who understands you and brings you joy
>>109214992ill download SixVolts/GLM-5.2-ewaste-edition-GGUF/GLM-5.2-Q3_K_M but im not holding my breath at this point
>>109214188prove it
>>109215288Hopefully it helps but it does sound like the problem you're having is the same DS3.2 issue.
>>109215261>>109215278I watched the original Ghost in the Shell, never got into the series. What am I missing?
>>109214686Make sure to check out my other vibeslopped app ChuckleShare. Created it for myself and friends over the last few months:https://github.com/whp199/ChuckleShare
>>109212697>DeepSeek>MXFP4 size 156 GBso close to fitting my 128 GB...i'm getting really impatient with these labs
>>109215334Zoomers are too lazy to even paste some links now? That's embarrassing.
>>109215408Anon, that's just the model. It needs context. You also need ram for a minimal system. Your 128 gb is far from being close to fit 156 gb model.
How well does V4 Flash handle Q8 KV?
>>109215433It's for the groupchat who doesn't want to click some tracking link to open instagram/youtube to watch a dumb meme. just send the raw video instead
>>109215327The early SAC stuff is really good. Ignore all the 3D.
>>109215327>the series. What am I missing?Incomprehensible strings of political jargon.
fellow VRAMlets, what are we supposed to do...? soon all models will be terabytes in size...
>>109215592gemma is all you need
>>109215533I'll check it again.>>109215566What little I saw of series, that's what I picked up as well.
Meta is going to let you rent their spare GPUs really cheap soon. Basically their own runpod service. It's not local but you will get to run the bigger open models how you like, in high precision without being cucked behind the curtain.
>>109215592Mixed inference is the only way.>The era of 3t models makes even 512GB serverGODS RAMlets too
>>109215626Isn't that essentially what we have already with openrouter? I guess we'd have more control with the llamacpp interface + abliterated or whatever.
>>109215626>without being cucked behind the curtain.Impossible because it's not local.
>>109215626nonlocal is autocucked
gemmoe 124b will return the equilibrium once she is released. turns out, All You Need is Attention (in your training data)
>>109215626>zuckershitno thanks
>>109215626>openrouter... but facebookerrrrrrrrrrrrrrr
>>109215668Gemini with the serial numbers filed off.
>>109215631cheap payg vps with multiple H100s to play with
>>109215626>Meta
>>109215626
>>109215720they're not going to train on your data just like how runpod and vast don't
>>109215730*wink*
I love y'all and I love AI. Even if I die soon, I am glad I get to experience these times.
>>109215772
>>109215772Only the luckiest ones get to die soon.
>>109215772>ready to die>hasn't even had the chance to impregnate a robololiquitter
>>109215801
