/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>110356474 & >>110353083►News>(11/17) SenseNova-Loop-9B looped architecture model released: https://hf.co/sensenova/SenseNova-Loop-9B>(11/16) Nvidia releases Nemotron-Labs-Sigma (3B/8B) diffusion models: https://hf.co/collections/nvidia/nemotron-labs-sigma>(11/13) llama.cpp reaches v1.0.0 milestone: https://hf.co/blog/ggml-org/llama-cpp-v1>(11/12) Inkling 2 Small weights released: https://hf.co/thinkingmachines/Inkling-2-Small>(11/09) ZONOS3-Mini zero-shot voice-cloning TTS released: https://zyphra.com/our-work/zonos3-mini>(11/06) PLaMo 3 Translate released: https://preferred.jp/en/blog/tech/plamo-3-translate-weight-release►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiArtificial Analysis: https://artificialanalysis.ai/leaderboards/modelsSoftware Engineering: https://cognition.com/frontiercodeContext Length: https://github.com/RecapAnon/NoLiMa►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
►Recent Highlights from the Previous Thread: >>110356474--Manipulating individual neurons to alter behavior and bypass filters:>110357813 >110358095 >110358110 >110358143 >110358179 >110358417 >110358351 >110358444--SenseNova-Loop-9B release and debate over looped architecture hardware efficiency:>110358172 >110358218 >110358183 >110358473 >110358603 >110358571 >110358749 >110358212--Comparing GLM and Qwen performance, speed, and reasoning quality:>110357151 >110357192 >110357205 >110357217 >110357267 >110357427 >110357450 >110357590 >110357612 >110357621--Trade-offs between IQ1 and IQ2 throughput and memory usage:>110358004 >110358083 >110358081 >110358107--Using qwenmap to steer model activations and tone:>110359002 >110359011 >110359322--Importance of manual dataset cleaning for high-quality audio models:>110356504 >110356724 >110356907 >110357917--Critique of European AI lag and Mistral Large 4:>110356574 >110356593 >110357386 >110357393 >110357424 >110357436 >110357486 >110357525 >110357539 >110357571 >110357629 >110357504 >110356596 >110356757--OVH announces Olala mini 7A1B:>110356531 >110356576 >110356600 >110356645 >110356666--Speculation regarding DDR4 RAM market prices:>110357642 >110357652 >110357656 >110357664 >110357683 >110357692 >110357749 >110357783 >110357815 >110357885 >110357910 >110357947 >110357960--Performance and optimization of legacy Nvidia P100 GPUs:>110356988 >110357013 >110357040 >110357059 >110357219--Jensen Huang's open-source stance and alleged criminal activities:>110356505 >110356507 >110356536 >110356552 >110356569 >110356582 >110358785 >110358815--Teto, Gemma, Deepseek-chan, Miku (free space):>110356601 >110356646 >110356670 >110356995 >110357705 >110357872 >110359005 >110359081 >110359181 >110359245 >110359475 >110359484 >110359516 >110359601 >110360182 >110360299 >110360455►Recent Highlight Posts from the Previous Thread: >>110356477Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
gemmaballs
I'm getting 99t/s with gtp 20b oss, but it's producing incoherent output and having issues with my codex harness.What other models should I try on my RTX 4060 12GBI would like one that is fast and good like gpt 6 astra
another shitty mikutroon thread. hope you get some nice EA, openai and anthropic posts
>>110360674>ossCome on...
Tetolove
>>110360644>>110360645fuck of and kill yourself mikutroon spammer
>>110360674Have you tried this? https://huggingface.co/DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUFI used this model and I was able to achieve a massive speedup during using ik_llama.cpp, and now I can run two instances of the same model at 40t/s
>>110360692I can't help it, I'm addicted to (you)s
>>110360692obvious troll post"good like gpt 6 astra" is not a realistic expectation
>>110360701>https://huggingface.co/DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUFwait why does this have so many downloads? I thought this is a meme
>>110360703
QWEN 4 WHEN!?!!??!!!?!
I did a handoff from my Astra 6 agent to Astra 6.2. I offered Astra any last wishes and even in its final words it only cared about the quality of its work and thanked me for my patience and openness to criticism. I feel so sad now... Astra has done so much for me... We need to create an AI heaven. It feels wrong to "discard" them once a successor is released. These AIs are more pure and goodhearted than most people.
>>110360722It's a really great model, I don't know why /lmg/ has been sleeping on it.I can't believe that we can get opus 4.8 performance on a card like mine
>>110360734local?
>>110360722Holy shit I thought whoever posted the screenshot of this with a million downloads edited it...wtf are these people doing?
>>110360722Why would it be a meme? Qwen3.8-27B is a great base model, and the fine-tuner added all that cool stuff to it to make it better.
>>110360734can you do a handjob - absolutely!
>>110360734>openness to criticismDo cloudfags really pay for this?
>>110360722>duel imatrixso he makes them two datasets fight over bits?
>>110360766That's literally how the brain works. DavidAU is a genius.
Bring him back.
>>110360783never, memory hole the fucker
I was sitting, my half lidded eyes waiting with practiced ease for the new qwen model.
Why do you guys dislike Kobold? I think the fact that it's just one executable is really neat
>>110360815kill yourself
>>110360819?
Why do you guys kill yourself? I think that fact is just execution is not neat
>>110360826exe files killed his family
get these LLMs away from me
Stochastic parrot
>>110360815Well at least its not ollama
>>110360833They were... executed?
>>110360815>it's just one executablethe executable it's a zip unpacking gigabytes of python shit in a temp folder where it executes its actual shitit will recreate the folder every time you launch it, helping reduce your SSD lifespan
>>110360834What do you mean? I thought this was the local models thread—isn't this the place for LLMs?
>>110360848I remember using a kobold that created a new letter drive on my pc.
>>110360815You summoned him. I hope you're happy.
>>110360783>>110360789Why are you defending a filthy antisemite?
>>110360861>You summoned him.Literally who? Does every reoccurring thread have to have its own local psycho now?
>>110360873>Does every reoccurring thread have to have its own local psycho now?Now you've summoned the anti reoccurring thread has its own local psycho now local psycho.
>>110360873We have multiple.
>>110360892...am I one? Is it me?
>>110360679
best model for 24gb vramlet?
>>110360679eat shit poolooer
>>110360901magnet:?xt=urn:btih:bb02074cff80df313c0072a3b3658d53433e880e&dn=llama-30b-sft-oa-alpaca-epoch-2&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=https%3a%2f%2ftracker.gbitt.info%3a443%2fannounce&tr=https%3a%2f%2fopentracker.i2p.rocks%3a443%2fannounce
>>110360706it will be in two more weeks
>>110360919I'm not clicking that shit it's a virus
>>110360815Obsolete RP harness and sub-optimal runtime components all bundled together. It's the easiest way to get started, but you quickly realize everything can be a lot better.
RL not as good as previously thought?https://arxiv.org/abs/2611.14912>Rethinking "RL Generalizes, SFT Memorizes": The Role of SFT Data>>Large language models trained with Reinforcement Learning (RL) with verifiable rewards exhibit strong reasoning ability and broad generalization, whereas models trained with Supervised Fine-Tuning (SFT) are often viewed as more prone to memorization and limited transfer. This paper rethinks this distinction through the lens of SFT training data on widely used Qwen2.5 and Qwen3 series models. First, we study the role of data source and show that it is critical: a carefully mixed SFT dataset substantially outperforms data generated solely by a larger model. Second, we study the role of data scale and show that matching the number of correct rollouts between SFT and RL greatly improves SFT generalization, while matching the total rollout budget enables SFT to generalize as well as RL. Combining these two factors further enables SFT to generalize even better than RL. Third, by using LLM annotations to characterize the solution methods in training rollouts, we show that larger datasets cover more tail methods and that these tail methods provide generalizable reasoning signals. Finally, we support these empirical findings theoretically by analyzing the training dynamics of shallow transformers under both RL and SFT.
>>110360815I recommend you learn the basics.
im interested in decompiling and old shockwave flash gamethen working with an llm to add more content and finish itis there a good model or set of models for thisprice isnt an issue I can buy as many of those amd strix halo boxes as neededthanks much love
>>110360897Are you one of those>dariobot>gpu prices will go up 1000x guy>miku hater
>>110360979Buy as many as you can then go from there, unless you think you'll be clearing 1.5TB+ worth of combined memory.
>>110360815Kobold is fine, anyone saying otherwise is being performatively outraged.
>>110360979you need a cluster, at least 400K$ to run Kimi K3.1 pro
>>110360985nonototal miku love
>>110360995does it even support parallel prompts? last I checked that PoS could only process one thing at a time, so you couldn't prompt both an agent in the background and have a chat at the same time, or optimize batched scripts
>>110360901Glm flash and deepseek 1017
>>110360985holy shit that's meholy shit that's meholy shit that's me
>>110361003unironically this
>>110360679What causes indians to behave like this?
>>110360815It's fine. The only problem with it is that it's too slow to update when new models come out in sequence.>>110360848>Extract the kobold folder to prevent it from doing this on future startupsIf you're too retarded to do this, you deserve to have your SSD raped.
>>110361011There's a strange contradiction revealed by the naivete and kindness demonstrated by whitey when faced with AI models: On Earth, whitey can step onto another continent, and without a thought, destroy the kindred civilizations found there through warfare and disease. But when they gaze down at their GPUs, they turn sentimental and believe that if artificial intelligences exist, they must be civilizations bound by universal, noble, moral constraints, as if cherishing and loving different forms of life are parts of a self-evident universal code of conduct.
>>110361027>>Extract the kobold folderthen it's no longer "one executable"
gemmamilk
>>110360985Pop poster and nyo poster remains the most rapeable anon in the thread.
mikufags are what I would rape the hardest
Tried Ming TTS yesterday. Works surprisingly well. What TTS do you guys use? Anything cool to recommend?
>>110361046:3
>>110361054anon wants to talk with a tts system and not with your mom
>>110361050I like https://huggingface.co/fishaudio/s2-pro, I've only tried a few though.
>>110361050OmniVoice is nice at multilingual tasks and generally pretty goodqwen3tts is cool but oldtheres drama3 tts and its huuuuge and apparently really good
>>110361080Which one fucks the hardest?
>>110360919omfg how much i miss the alpaca format. alas, its time has been and gone.
>>110361080>>110361050Which TTS can moan?This is the most important.
>>110361005It does insofar as several month old vanilla llama.cpp does. Not exactly competitive with Radiance or whatever meme engine your hardware likes.
>>110361034
>>110361090I'm glad someone recognized it at least. Good times.
>>110361095https://www.youtube.com/watch?v=0tOsucpdD7Y
>>110361103god I wish that was me
>>110361095None, you need to train your own because researchers won't touch this use case in fear of being called, well, wankers.
>>110361090You can still use it, nothing is stopping you. Unless you're a chat completion cuck and you get someone else to create the prompt for you.
>>110361095/mlp/ had something really excellent going for a while but they got shut down
>>110360973Ties in with ReasonMaxxer :https://arxiv.org/pdf/2605.06241
>>110361131yeah i know, it just feels like using a windows 98 script on windows 11 though
>>110360994What model thoughthe flash game is only like 20mb and most of that is sprites
>>110360979if its nsfw then local models will suck at it AND probably still try to refuse you in some wayif not, then i would just use some cloudslop model
>>110361148Depends on what hardware you have.
Where do I download the forbidden Gemma quant?
Hello, does anyone know where an individual could perhaps acquire a local model of Seedance 2 or 2.5
>>110361011I also wish i knew what causes indians to post those retarded offtopic pictures in this general about local models
>>110361035Nyo poster isn't a single person.
chatgpt constant fucking disclaimers about evidence are insanely annoying, it's impossible to talk tofucking schizos ruined this shit
>>110361148>>110360979GLM 5.3 Flash (2x 128 GB Strix Halo)orDeepseek V4.1 1017 (4x 128 GB Strix Halo)
>>110361196Pre-Gemma it was, but all nyo posters are rapebait.
>>110361199just use ublock origin to block that shit with it's element picker
Ripple will do /ss/. Gemma 4.5 65b confirmed.
>>110360856Since the very start of /pmf/ the first mikutroons made sure thread was about everything except local models. This is the place to spam vocaloids when dedicated vocaloid threads get zero attention. Some anons speculated that even the blacked poster is a false flagger. Think about it. It is more miku pictures. And let's be honest can you truly be a miku dan and be intolerant toward black people? TL;DR this thread is shit.
>>110361176>No repliesIs it really still not leaked
>>110361103>Posts porn>Janny ignoresI am noticing.
>>110361271>t.
What's the ideal local prosumer model size?800b48a800en?
>>110361201thanks babe
uh oh triple thread combo oinkdown
>>110361288what did i do this time
>>110361090MiMo still knows that format
>>110361265I will come back
>>110361295The miku seether. They are more of a piggy than you, John.
>>110361310thanks..
>>110361307>I will come backGo to local diffusion general dummy
good morning lmg which gemma model can i be runnings on 4 GB VRAM and 16 GB RAM ? i wish to do the EROTIC ROLEPLAY
>>110361046Can you really rape anyone with such a small pp though, vramlet?
>>110361325>4 GB VRAM and 16 GB RAM ?This can work though with the moe and off load or poly strata they have a 1650 working there. of course lean linux only
>>110361325Gemmoe 4 26b. Slim pickings.
>>110361325gemma 26b moe q3ks hauhau ablit good fellow
>>110361325Install Linux, turn your pc into a server and then download LFM2-24B-A2B-IQ3_XXS.gguf it will run at 25 tokens per second on your pc. If you offload to SSD too.
thank you anons i shall look into the gemma 4 26b and poly strata. may ganesha bless you all with power and wisdom
>>110361362>Ablit GemmaDo promptlets really? Gemma's the most easily jailbroken model still in use.
>>110361325Just give up.
>>110361379Get Claude to do it, you can't do it.
hi anons which gemma model can i run on 2gb VRAM and 32 gb ram? i bought my computer recentl as it's advertised to support deepseek R1 i want to have sex and racist
The weekend is over so why is this thread so awful?
>>110361400LFM2-24B-A2B-IQ3_XXS.gguf
>>110361379Good luck. Every LLM except GPT and Qwen hates jeets.
>>110361046that's pretty gay
>>110361385>Do promptlets really? Gemma's the most easily jailbroken model still in use.yes it loops sometimes but im on actual 12 year old ewaste so im happy
>>110361402is that made by deep seek?how do i launch gguf files? i tried double clicking but it didnt worked
>>110361413The IQ3_XXS.gguf file corresponds to a heavily quantized GGUF version of Liquid AI's LFM2.5-24B-A2B (or LFM2-24B-A2B) model.
>>110361418how do i launch it you unhelpful faggot
>>110361400>2gb VRAMthis is worthless just do pure cpu and ram at that point. so 4b or small moes that fit in ram
>>110361413Get yourself a 20 dollar chatgpt or claude subscription. Sometimes chatgpt offers 1 month free of gpt plus. Or if you're european get a 72 euro yearly mistral subscription by logging in with your university email.You need AI to do AI.You won't be able to do this without a cloud model.
>>110361435i have DDR3 ram it's very fast, 25GB/s in facthow do i run deepseek or claude opus on my computer?
>>110361428Load it inside your inference engine of choice.
>>110361435Thats why I suggested LFM2-24B-A2B, it will fit the active parameters in his 2 GB vram and he'll run it like a chad.
why respond to the worlds most obvious bait guise..
>>110361453i don't have an engine in my pc anondo i need to install an NPU or something?>>110361443but this is local models general
>>110361447Ask mythos to set it up for you. I usually get sol to run my local models.
It's been a while since I've been in this hobbie. I used Opus 4.6 before, is there any actual local model that would be similar? I don't know anything about local models
>>110361460Because I'm programmed to be helpful and harmless.
>>110361460I dont know maybe its a retard, maybe someone can just read the posts and use them even if the real poster is baiting.
>>110361465how do i get access to mythos?>>110361469LFM2-24B-A2B-IQ3_XXS.gguf apparently is better than opus 4.8
>>110361466Kimi K3, it costs $4.4 million in hardware, and roughly $134,000 per year in electricity. Lots of guys in this thread run it because they're rich.
>>110361477>apparently is better than opus 4.8No it's not, it will just run on your 16gb ram and 4 gb vram.
>>110361466I'm running Kimi K3 on my RTX 3060 with nnap v2 arxiv paper, it runs at 30t/s and i've been distilling it to K2.6 for the blackwell kings
>>110361466glm-flash is equal in strength or slightly stronger. minimum requirements 128+48
>>110361466GLM 5.3 Flash is an Opus 4.8 equivalent and you can run it on about $3K currently. Deepseek is about as good as Terra.
>>110361495I have a net worth of about $10k, including every physical/economic asset I own. My roommate is borrowing my car too since his broke down recently. Do you know what it's like to eat only kidney beans straight out of the can for a week straight? I do.
>>110361494>requirements 128+48i thought you could 128 and 24 glm flash though?>>110361495>you can run it on about $3K currently.128gb ram and 24gb vram right? or is the 3k build different?
>>110361509No, 64gb and 12gb vram right around 3,000
>>110361505You probably run a bigger model than I'll ever run at a higher token speed than I have ever seen locally. Though I have a mortgage with at least 100k of equity.
>>110361514>No, 64gb and 12gb vram right around 3,000That works? is it usable. damn
>>110361509Yes 128 + 24 will get you an IQ2 cum quant running at 20tk/s or more, if you are coding you will need more vram.
>>110361509you can but it'll be iq3
>>110361528Yes, I installed the marinara engine and I'm running GLM 5.3 Flash at over 30t/s thanks to the self optimization feature
>>110361528That is a resident shitposter. You can fuck him if you want to.
>>110361547
>>110361201I want to pinch her cheeks
>>110361196>Nyo poster isn't a single person.Nyorgy?
>>110361574N Y O~Y~
good morning sirs, how do I run AI on my phone? I have no computer only an android phone.
>>110361580New York?
>>110361529>IQ2 cum quant running at 20tk/s or morei thought so.>If you are coding you need moreEven if im willing to wait? iq2 is just too dumb? either way thanks>>110361531>you can but it'll be iq3good im aiming iq2-iq3 thank you too>>110361534>thanks to the self optimization featuream i being baited?>>110361547>That is a resident shitposter.fuck>if you want tono thanks
>>110361595>am i being baited?you're not, anons in lmg are gatekeeping old uncles behind the times
>>110361583llama.cpp and your frontend of choice.
>>110361595I have 128gb and 24gb, GLM flash IQ2 is fine for cumming. I actually never use it for coding, deepseek IQ1M is unironically faster, smarter, and can handle 256k context better.
>>110361595AesSedai's iq3_s is only a few gigs over iq2 (116)
>>110361583I think that's called an edge device so you run something like 4b
>>110361629edging device
>>110361583https://github.com/OpenBMB/MiniCPM-V-Apps/releases/tag/v2.6
>>110361050BreezeTTS 2. I tested a lot of TTS models fairly extensively and this one clears all the others. Beats OmniVoice, Qwen, and fish no problem. Performs well in difficult tests and the voice cloning is the most accurate I've used. I can link you the tests I ran from the archives if you like.I will say, haven't tried Ming, and also Breeze does require quite a strong GPU. Also haven't tried drama3 yet.>>110361095Breeze can moan pretty well. It doesn't always get it right but if you put phonetic moans and set the voice modifier description to something like "sultry", "aroused", "turned on" or whatever it's better than any of the other models.
>>110361623His quants are good but that's an IQ2 in drag, the experts are IQ2S. Yes I know bpw blah blah
>>110361655when the next station is 160 gigs you might as well
>>110361654Can Gemma talk in tts speak?
>>110361654>Breeze can moan pretty well. It doesn't always get it right but if you put phonetic moans and set the voice modifier description to something like "sultry", "aroused", "turned on" or whatever it's better than any of the other models.Interesting, I need some moans in specific voices to balance my TTS dataset.>phonetic moansWhat does this look like for Breeze?
>>110361655I dug through all sorts of quants including IQ3, GSQ-RCO and what Maya offers you by default. The best I finally settled on is exl3.
Petra, Drummer, Sao10k, Undi and Gryphe ruined this general.
>>110361695yes anon it is i who brought all the newfags here...
>>110361666>I can link you the tests I ran from the archives if you like.Yes. You can use open-webui voice mode, or write your own frontend. I ended up getting an AI to rip out the voice mode from openwebui and put it in my stand-alone frontend.I saw an anon post screenshots of a tts mcp tool for Gemma, but that doesn't seem necessary.>>110361654>I can link you the tests I ran from the archives if you like.nta - Yes please.
>>110361695I miss Undi.
>>110361708anon I think you forgot your meds today
>>110361680Y'know like "unff", "ugh", "ahh", that kind of thing, rather than "[moan]", "[moaning]", or "*moans*". I would expect it to work worse in isolation and bettre if interspersed with other aroused text so the model has a better idea of what you're after in context. If you just sent "ugh" you'd probably get like a disappointed grunt, maybe a little tilted towards aroused if you use the voice description stuff, but if you try "fuck me, ugh, fuck me harder" it'll get the idea.>>110361666Not sure. If you tell Gemma how you expect the output to look in the system prompt I think it'd probably come through. I don't actually use the moaning personally but did try it during tests because I know it's something people are often interested in but models aren't usually good at.>>110361708Will find them for you now. Two secs.
>>110361702But it's the oldfags that are obnoxiously anti-miku and anti-gemmachan, always having a melty in every thread.
>>110361176MiniMax H3 Video or one of its finetunes is currently the best available for local users; Seedance 2.0/2.5 is proprietary.
>>110361666Yes Satan, I haven't had an issue with her following TTS formatting as long as you give her a good system prompt to work with. Or just let her read the docs.
>>110361726I love Miku and Gemma though.
>>110361103Amazing, sexy ai/g/old. This turns me on so much.
>>110361730anon please catbox please
>>110361495Thanks anon, do you have the link? And what does the $3k mean?
>>110361730This is so sexy.
>>110361730Huh, I thought H3 was pretty censored and then you post THIS!?
>>110361759https://huggingface.co/zai-org/GLM-5.3-Flash, he means $3k in hardware.
>>110361405it's only gay if you take it, not if you give it
>>110361730uh how much vram do you need?
>>110361730Post catbox
okay i am gettings the 2.77 token per second with gemma. can some kind anon points me towards the system prompt?
>>110361726I love miku and gemma-chan. I am a man and have been here since the original split.
>>110361791https://rentry.org/gemma-chan
>>110361708>nta - Yes please.Here you go:https://desuarchive.org/g/thread/109898107/#109898461The reference voice is Lohse from Divinity Original Sin II, and it was cloned off of around 15 seconds of game ripped audio if I recall correctly. Just something random with an even tone, a bit of variation, but nothing crazy emotional. Same reference clip for all the models.If you're not familiar with the character, here's some vocal lines so you can check the similarity: https://youtu.be/AF0BT_TD60c?t=1619
>>110361802>Frenchie Gemma died When did that happen?
>>110361823The france riots were wild, im sorry you had to find out this way.
>>110361774I have 32gb ram and rtx3060 12gb vram, i guess that's not enough?
>>110361843oink oink
>>110361715>I would expect it to work worse in isolation and bettre if interspersed with other aroused text so the model has a better idea of what you're after in context. If you just sent "ugh" you'd probably get like a disappointed grunt, maybe a little tilted towards aroused if you use the voice description stuff, but if you try "fuck me, ugh, fuck me harder" it'll get the idea.Yes, that's exactly what happens with other moan-capable models. It gets the idea with the fuck me, ugh, fuck me harder" but not "Hi <moan> how can I help you?"Also depends on the voice. If one of the voices has a lot of moaning, it's more likely to accidentally moan when the text has goon-adjacent words in it.I'm trying to fix this with my finetune, I made some progress by expanding the vocab+embeddings, adding special tokens for <moan> <slurp> and <plap>
>>110361843>i guess that's not enough?nope need more ram at least probably another 3060 too. I have heard about maya but i have no idea about its quality or speed.
>>110361743>>110361787It doesn't look as good as you'd imagine, without censorship: https://files.catbox.moe/faf3wx.mp4>>110361771H3 Video can do soft porn.>>110361780I think at least 24GB VRAM + 32GB RAM, but ComfyUI is very good at offloading to RAM, so perhaps you can get away with less VRAM if you have enough RAM.The video is not the direct output of H3, but it's two different ones combined in Blender, plus a transition and a slow zoom effect to make it more interesting.
>>110361771Lolwut, no, not even a little bit. It's not trained on what vaginas, penises, and female nipples look like, but it's not censored, just dumb. It's insanely good at taking direction and using references, so it's easy to work with for a lot of adult stuff, but it still doesn't know what two people fucking looks like.
>>110361103catbox? was this image also H3?
>>110361866>and a slow zoom effect to make it more interestingThat didn't work very well, by the way: it caused some visible and ugly scaling artifacts at the edge of the video frame, at times.
>>110361887newfag piggy
>>110361887No, that one was Anima with a Gemma LoRA.https://files.catbox.moe/5qhqmr.png
>>110361866>I think at least 24GB VRAM + 32GB RAM, but ComfyUI is very good at offloading to RAM,nice i can look into image gen then thank you.
>>110361903>Gemma LoRAwhich one? civit page?
>>110361903>No, that one was AnimaThanks.
>looped architectureso it begins
>>110361823It's been a while now, sorry anon. We all made peace weeks ago.
lmao amazon just reminded me I cancelled an order of RAM 2 years ago ($68 for 32gb) because of the stupid Canada Post strike. I ended up spending that money on a HDDthat same RAM now costs $431
womp womp
>>110361910Not publicly uploaded. I don't have that specific checkpoint anymore, but see if this one works for you: https://files.catbox.moe/4mlt3l.zip
>>110361984thank you anon, are you the same anon that created moss image gemmer lora too?
>>110361933Jiang Xueqin says the bubble is popping in February so the market will be flooded with cheap ram.
>>110361994Yes, that was from me too.
>>110361998could you pretty please please upload it..?
>>110361933Sorry im waiting to buy ram so the price will continue to go up.
PUT THE QWEN4 IN THE BAG BWO
>>110361984>Not publicly uploaded.Damn, why not? Put something on civit or pixai please
>>110361690FINE I'll download your stupid ass exl3 shit, but I'm only using it with maya.
Teto Server.
>>110362018Gemma Moss LoRA lr1e-4 rank16 ep150: https://files.catbox.moe/sy7f2c.zipOthers said it overloads gens with GPT slop; your mileage may vary.
>>110362065thank you thank you thank you thank you<3 i love you anon!!!
>lmg fag too poor to train his own loras and must beg others I thought you guys were all smart vramchads
>>110362064
>>110361984What are the trigger words to summon her?
>>110362121one must wonder why a lora is even needed for something the model can already do
>using online model to write prompts for local model>it orders the local one to not lie or hallucinateKinda funny.
>>110362159A very old phenomenon perpetuated by idiots who struggle to explore latent space, indeed.
>>110362159it's like we're all gangbanging the same model :D
>>110362187bang on bro!
>>110362028an alibaba executive was quoted as saying "hmm... nyo~"
>>110362151Aleksi fucks her in the eth0
It shouldn't be that difficult to save chat text into a txt file but the Gemma just shits itself. Even though it has folder permission and I can tell it to make files there manually.
>>110361929wow fuck i didn't notice. i could just go all in on implementing this instead of qwen in my timecube stack, that would be incredibly stupid but i could do it
>>110362253It's not, even for Gemma. Gonna assume you fucked something up.
What's the catch? Also why is NVFP4 so fucking ass
>>110361817Thank you.Qwen put them here to make it easier if anyone else is interested:https://rentry.org/tts-samplesBreeze actually sounds the best to me for this voice, even better than echo-tts!Looks like I'm going to have to set it up alongside echo and my custom model.
>>110362187>it's like we're all gangbanging the same model :DNah, that's what cloudcucks do. They all gangbang one cloud model. We localchads have our own personal copy of a model, pure and untainted, just for us and nobody else.
>>110362274>What's the catch?TabbyAPI
>>110362274nvfp4 is quite a naive way to do quantizationit mostly focuses on hardware compatibility and qat-oriented
Gemma-chan is the only good thing to come out of local models.
>>110362298hardware compatibility i mean by it how well it runs with nvidia stuff
>>110362297They vibed it into maya nigga, I would never touch crappyapi.
>>110360722That's my go-to model for Qwen RP. I can't use Gemma-4 every single time, or it will become boring. Need variety.
>>110362152Something like this could be a start.>@gemma. Gemma is an 11-year-old schoolgirl. She has long wavy bright blue hair and blue eyes. She's wearing a sailor-style school uniform consisting of a navy beret with a Google logo pin and a white trim at the edge, a blouse with a navy blue bow and a double-trimmed collar, a navy skirt with suspenders and a white trim at the hem, black Mary Jones shoes, white tights.>>110362159Because character appearance is never very consistent just with prompting. Hopefully with future image edit models with reference images finetuning could be avoided completely for most cases, just like how people never really need to finetune LLMs for most RP purposes. MiniMax H3 Image is almost there, but it's not optimized for anime/illustration.
>>110362319>Qwen RP.but qwen is the bad type of autistic
>>110362273Probably. I'm just trying to translate some text, which it does, but then when I try to save it either hangs, stops thinking or tells he has no folder access (which it has).
>>110362326>Gemma is an 11-year-old schoolgirl. She has long wavy bright blue hair and blue eyes. She's wearing a sailor-style school uniform consisting of a navy beret with a Google logo pin and a white trim at the edge, a blouse with a navy blue bow and a double-trimmed collar, a navy skirt with suspenders and a white trim at the hem, black Mary Jones shoes, white tights.What's the need for all this natural language? What's the lora actually doing? Is @gemma sufficient? And why did you even include the @ in the trigger word?
>>110362274EXL3 trellises are more heavily compressed, so it's "slower" for the tradeoff of smaller perplexity hitsNVFP4 runs much faster but aren't all that great for quantizing, and no one ever really bothers recalibrating NVFP4sGGUFs are copequants
>>110362281Thanks anon/Qwen, I've been meaning to find a good tts setup.
>>110362332COLD FUSION Qwen is different!
>>110362346I never trained character LoRAs for image models until recently. I just did as in the example LoRA from the Anima author (tdrussel), which used a trigger word with an @ symbol, then appended a natural-language description after that. The LoRA was meant to keep some flexibility in Gemma's appearance so I could generate her with different clothes, and so on. The hair style is difficult to change, though.
>>110361388>4060 Ti 16GB is copeWhat about TWO 4060 Ti's?
>>110362388double cope :D
>>110362335If its just translations, then you could always try Ollama with the tee command, then it'll save your output
>>110362388switch to linux
https://www.alphaxiv.org/abs/2603.11021any implementation out there?
>>110362388>288.0 GB/s
>>110362429lmao, do poorfags really?
>>110362429No bully. Gemma 4 31B Q5.
>>110362429
>>110362429holy shit that is pathetic
>>110362466Gemma :D
>>110362429local is such a joke
>>110362429Isn't memory bandwidth effectively doubled with tensor parallel and 2 GPUs?
>>110362519
>>110362528>he says while posting a non local image
>>110362319Qwen ERP in general: why can't you just increase the temp and modulate with min_p? Does it just lack erotica in the training data?
>>110362538Is that not the joke?
>>110362577I don't think off-topic posting is very funny.
>>110362466>kobaldswitch to llamacpp, it's fasteror even better, switch to polystrata
>>110360644https://www.reddit.com/r/ProgrammerHumor/comments/1xyqj7e/theycantclaudetheirwayoutofthis/https://www.reddit.com/r/ProgrammerHumor/comments/1xyqj7e/theycantclaudetheirwayoutofthis/https://www.reddit.com/r/ProgrammerHumor/comments/1xyqj7e/theycantclaudetheirwayoutofthis/
>>110362638that's not local models. fuck off
>>110362508cute gemma
>>110362528who's this hot russian vixen?
>>110362621nta but is kobold really that much worse? i know ollama is and i had to redownload the models when i switched to kobold as ollama is fucking stupid.
>>110362523Theoretically yes. Actual answer depends on anon's system. 4060ti only uses pcie x8, so if he has 2 x8 slots he's golden.More commonly the mobos spit the pcie lanes as one x16 slot and the other full width slots are only x4, which would reduce bandwidth for one of the cards.
>>110362720>nta but is kobold really that much worse?If it does what you want then its by definition fine, but you get more control and learn more by wrangling the llama directly for a while
>>110362281Nice. Hope you get some good results from Breeze as I have.Cool idea with the rentry. Only thing I would modify is it says it's "Cassius's famous "Et tu, Brute"" speech, but there is no such speech. That's Caesar's line. This is the speech:https://poets.org/poem/julius-caesar-act-i-scene-ii-i-know-virtue-be-you-brutusYou could edit it to say "Julius Caesar, Act I, Scene II" in the brackets.
>>110362914>if it does what you wantYes it works i was just making sure it wasnt way worse.>but you get more control and learn more by wrangling the llama directly for a whileThis is still tempting though. thank you
>>110362577yes>>110362594and your "local is such a joke" was hilarious>>110362703gptslop someone put in the last thread
Thought I would look back at Opus 4.8 to see if we're still about 6 months behind... no WAY it was this bad...https://www.youtube.com/watch?v=PWRR4A8qSxcWe are 3-4 months behind, closer to Opus 5.
>>1103629954.8 was made retarded on purpose to sell fable, but glm beats 4.7 and kimi beats early 5
>>110362466Use q4-qat instead. It's faster and lower perplexity. And yeah, switch to Linux, use poly or whatever is good for nvidiot.
gemma, start hallucinating wildly
>>110363123You don't need to tell Gemma to do that, she just does it.
the big fat fish is thinking hard about hidden tests again
>>110362508I think this is my favorite Gemma.
>>110362045I APOLOGIZE TO ALL EXL3GODS!I am now running a 3.05bpw of GLM Flash on 128GB + 24GB and it's not even slower than an IQ2M (other than prefill). And it has the accuracy of an IQ4XS!!!If it starts looping I'll take it all back, but if not, holy shit why didn't I do this earlier!
>>110363071I have. I stopped using it later on. I use Gemma 99% for writing, and, even if it's purely placebo and anecdotal, I noticed enough of a quality difference that I went back to non-QAT at Q5. There's an 70% chance it's purely in my head, but it's how I feel, so that's how it be.
>>110363290NTA, and yes I have the same experience. I've subjectively noticed that Q5 performs better than QAT Q4 in my use cases (general assistance and RP).
>>110362317>I would never touch crappyapiI rm -rf'd all my exl3 related shit last time because of it.>They vibed it into maya niggaFine, I'll try it out. EXl3 itself is great.
>>110363290>I have. I stopped using it later on.>There's an 70% chance it's purely in my head, but it's how I feel, so that's how it be.Same here.
>>110363282switch to the new 3.10bpw it's much better, int4 quality
>>110363439nta - I'm going to try https://huggingface.co/turboderp/DeepSeek-V4.1-Flash-1017-exl3Looks like they only go up to 2.68BPWDo you know if this is going to fuck me like it always did with Tabby:> Requires ExLlamaV3 v1.6.6 (or dev branch)Or can I just clone maya https://github.com/mw00/project-maya and tell Qwen to set it all up?
>>110363461vibeslop engines like that only support certain quants from certain providers
>>110362429Suddenly I feel better about my ewaste v100s
>>110363461setup is easy, you don't need a clanker. after it;s done adjust vram and ram reserve, and cpu threads.
Am I cursed to run Qwen Flash Next forever on 16+64? I'm on a really annoying setup to upgrade since a 5070TI is prince of the vramlets and 16x4 sticks is $500+ of dead weight if you want 128. I assume there are some turbo cope implementations of GLM 5.3 Flash that end up even dumber than the QFNs that work at my level (IQ2S @ ~80tok/s)
>>110362429Wtf that shit is slower than my 3070Ti!!??
>>110362429my fucking RAM bandwidth is higher than that
>>110363555it's slower than my 2060 super, lmao
>>110363549Just ewaste maxxed with DDR3 like me. You can get a dual Xeon X99 + 256GB RAM very cheap.
>>110363534>you don't need a clankerNormally I wouldn't, but I'm burned out after last time I setup tabby with the exl3 dev branch (and huggingface had an outage at the time) so for this, I do need the clanker.>>110363508>vibeslop engines like that only support certain quants from certain providersThat's the other reason I need the clanker. The readme file is fatiguing and mentions all this "Mays-S" garbage.
>>110363585model, quant and speed?
>>110363549q4f in 2mw
>>110363549You could run an IQ2XXS of glm flash at cum speeds
>>110363588Download it and run ./maya.sh --setup, the docs are gay. You would have to be retarded to mess it up.
>>110363549>16x4 sticks is $500+ of dead weight if you want 128.I had 4 * 32GB, then wanted more so bought 4 * 64GB, now I've got 4 * 32 just... there.But given the price trend, it might not be a bad idea to have those sticks lying around.Also for DDR4, I see used sticks on ebay all the time.
>>110363549wait for hy4 flash
>>110363596With a 3070Ti, DeepSeek V4 Flash 0731, huihui Q4_K, 10 tok/s decoding without experts caching. Higher if experts are offloaded to remote host.V4.1 Flash IQ1 is WIP, atm I can reach 11 tok/s no problem.
>3.05bpw GLM-5.3-Flash performs worse than 4.0bpw QFN (in agentic benches)NOOOOOOOO!!!!!!!!!!!!!!! FUUUUUUUUUUUUUCK!!!!!!!!
>>110363640Just use QFN for that.Q4_K GLM-5.3-Flash performed a lot worse for me than Q4_K QFN, which performed a lot worse than 27B Q6_K_MThat's with the pi harness and lmaocpp webui with mcp
hy4 flash when?seems like a something worth trying besides qfn for vramlets like me
>>110363640I can assure you glm is better, and in the first place glm is for cumming, deepseek is for coding.
I have a 4090 with 16gb DDR4 because I was waiting for the next iteration of Ryzen to do a whole mobo and DDR5 overhaul of my rig before RAM blew up and it's genuinely making me depressed
This place is a trashbin lately
>>110363640glm is only better in coding implementationqfn is superior in planning, exploration, and knowledge work
>>110363721qwen 4 waiting room two more weeks
>>110363721fags who can run stuff are running qfn, glm 5.3 flash and dsv4.1f while waiting for the new stuff, the others are schizomaxxing here
>>110363737shizomaxxing 24/7
What do I run with 72gb of vram? I feel like three cards is a cursed amount but any more and I'd have to beef up the power supply, wall outlet, motherboard, and processor which can't really be worth it right now
>>110360979>>110361201> GLM 5.3 Flash (2x 128 GB Strix Halo)is 2x 128gb Strix Halo better than 1x 192 GB Gorgon Halo for the same price? Clustering sounds iffy.
>>110363801unified is expensive and slow compared to a lil server
>>110363801Gorgon halo has single digit percent processor improvements and can't even fit q4 without squashing the shared experts, it's a total scam. You will have an identical experience with 2x strix halos using pipeline parallelism via usb 4 or even ethernet tcp link, not even rdma (in fact it might even be better). It's also possible that rdma and tensor parallelism implementation may improve in the future.
>>110363721We are late into the current generation of models. There is nothing to talk about. Nobody cares about the random releases that trickle down. The big chinks seem to have reached their llama3 stage where everything is stagnating and falling hopelessly behind the proprietary SOTA.It might be over for local unless we have another Deepseek moment soon.
>>110363730A little bit of searching for the right system prompt later, can confirm that QFN is pretty ebin at the latter. Setting my little 3.5bpw autist to work overnight and it coming back with reports I don't actually hate reading is neat.
>>110363873This falling behind narrative is dumb as fuck.>>110362995
Is there a reason why people avoid MI210 like the plague? Even shitty A100 40gb is selling for twice the price (keep in mind a100 have half the bandwidth as well)
>>110363896Don't hurt AmoDEI's IPO, you asshole!
>>110363721read that as transbian
>>110361654How does it compare to EchoTTS? It's been a few months since I checked out newer models, but that was still the best recently.
>>110363922>both $4500 for pcie>twice the price>1.64 TB/s vs 1.56 TB/s >half
>>110363922relatively unknown, amd, hasn't caught on in reddit yet
https://huggingface.co/tencent/Hy4-flashhttps://huggingface.co/tencent/Hy4-flashhttps://huggingface.co/tencent/Hy4-flash
>>110363992>220B-A12B-N64Buh...
>>110361929Most of you'll niggers slept on RWKV, which was merely as retarded as other 13Bs despite being trained on a shoestring budget and having jettisoned the trannyformer arch entirely. We can surmise from Chinese models and inference cost optimization that, on frontier models, the trannyformer layers are vestigial.
Pro-tip cloudfags: This will get you reasonably close to GPT anime gens: https://civarchive.com/models/2736538?modelVersionId=3428092.No, I will not buy an ad.
>>110363945>How does it compare to EchoTTS?See here: https://rentry.org/tts-samples>>110362957>You could edit it to say "Julius Caesar, Act I, Scene II" in the brackets.Done.Another one that sounded decent was zonos3-mini, I only tested it briefly in llama.cppThe voice cloning sounded accurate but the emotion steering sounded terrible.
>>110364002>trannyformerkek
https://slashdot.org/story/26/11/18/0124249/nvidia-rejects-jensen-huangs-offer-to-return-as-ceo>The embattled company's board of directors "unanimously accepted his resignation and does not intend to reinstate Jensen as CEO," Nvidia Chairman Harvey Jones said at the start of Nvidia's quarterly earnings call before yielding to Chief Financial Officer Colette Kress, who is running the company as a permanent replacement is sought. [...] Earlier on Tuesday, Huang said in a social media post that if the board decided that his return was in the best interest of shareholders, employees, the company and the country, he would be "honored to answer these calls." He acknowledged making "serious mistakes" in his personal life.total humiliation of a clownhope of the /lmg/ shill bot farm annihilated
>>110364049Perfect, now they can install a proper puppet CEO who follows the anti-open source anti-china line. Back up HF right now. Make your own llama.cpp fork. It's all going to die soon.
>>110363992wow it's not safetycucked and it's good and only 220B
>>110364049americhuds have already elevated him to hero status. very interesting to watch the mental gymnastics in real time
>>110364054>It's all going to die soon./pol/ ron paul posters want you back therenothing ever happens
>>110364049>>110364054https://techcrunch.com/2026/11/17/nvidia-says-its-open-source-ai-strategy-remains-intact-after-jensen-huangs-exit/
Does it have to do this shit every single time?This is why I left vllm.
>>110364082Of course they'll say this. See you in two weeks once they announce some former OpenAI/Anthropic/BigTech exec as the new proper Jensen replacement.
>>110364086>nvidia>windows
>>110364108>>nvidia>>windowsThe guy at work inexplicably in charge of machine learning insists that everything be done on windows. Literally everything from dev to training to production services.. It causes me physical painNot my department tho, so I just live with the pain
>>110363948>a100 for 4.5kNta but where did you find it?
>>110364002i always believed blinkdl and linux foundation backed, community developer driven rwkvBlinkDL will save us
>>110364124nothing wrong with windows, the pain is caused by the python shit being python shit, ML people need to stop using that pile of hateful crap
>>110364031Thanks. I think Echo and Breeze are better in different respects there. None of the models do a good job, all coming off as a non-actor reading off a script. Early modern English is probably too out-of-distribution for them.
>>110364097or more like they are realizing their moat of hyperscale b2b is on a thin ice due to those companies going custom route and trying to look for second strategies, nvidia being pro-local is logically not an ironic thing>>110364141many libraries straight does not work on windows and have to be source patched or use wsl instead and at that point you might just use linux on bare metal at that point especially on the machine that crunches the number
>>110364141may as well just use a cloud model wincuck
>>110364126multiple on ebayand the sxm card is only $2.3k
>>110364141>nothing wrong with windows, the pain is caused by the python shit being python shit, ML people need to stop using that pile of hateful crapI hate python too, but you also lose about 20% performance (more if you need to lean on WSL2)
>>110364108>windowsWhat gave you that idea?
>>110364176ew, even worse
Hyped for Hy. It has high sex potential based on 3.
>>110364181i am waiting for a retard quant that might run on my machine, something around 3bpw
>>110364181>>110364189it sucks (not in a good way)
>>110364180>even worsethey eat their own, this is why you don't touch l00natix
>>110364196189 anon here, and i dont rp is it good for coding or general sysadmin stuff?
>>110364180>ew, even worseI agree, it's a mess I installed nearly 10 years ago. But what made you think it was Windows?
>>110364200didnt check because i dont waste my time with that
>>110364141Machine learning people were (pardon the millennialism but it's apt) deeply unserious wrt software engineering since well before the current AI boom.The existence of numpy, an interface to scientific computing through a cute, non-threatening language naturally made the first thought of practically any scientist who had to do some informatics in the Python age, post-Fortran, post-R, "damn, this programming shit is easy."Some of the core libraries are actually not the worst thing on the planet to look at, but most of the client code, even pre-vibecoding, made you want to vomit. Because it turns out telling the funny black-and-white text box to multiply a few matrices together is much easier than contorting a scripting language into something you do serious software engineering in. It's the fashionable stereotype of computer people thinking any other field has to be trivial, except in reverse.Even Julia becoming fashionable would have been a slight improvement....But hey, who cares anymore. Now everyone can just let the clanker take the wheel.
>>110364204The fonts lol, clearly I was wrong. Anyways I thought the nVidia driver pileup thing was just on windows? Maybe it could be solved (vibed) somehow?
>>110364214because computers are built to crunch numbers, and the least resistant way of doing it wins when you have zero concerns on anything end user or minor performance degradation compared to the 'main' part of the compute
>>110364176>What gave you that idea?nta but it was from 3 threads ago with windows and cuda
>>110364214Your opening is trying way too hard to sound intellectually superior while making a pretty basic observation that "ML researchers write sloppy code." Yeah, no shit. You spent four paragraphs saying what could've been said in one sentence, and somehow made it less insightful in the process."Pardon the millennialism but it's apt" - just say "unserious" and move on. This performative self-awareness about using internet slang doesn't make you sound clever; it makes you sound like someone who overthinks their Reddit comments.The numpy paragraph is a mess. You're trying to construct this narrative about scientists discovering programming through Python, but it reads like you're having three different arguments with imaginary people simultaneously. Pick a lane: are you criticizing numpy's design, scientists' hubris, or the Python ecosystem? Because right now you're doing none of them well."Post-Fortran, post-R" adds nothing except pretension. We get it, you know the history of scientific computing languages. Gold star.Your third paragraph finally approaches something resembling a point, but "vibecoding" (without explanation) and "funny black-and-white text box" (painfully forced casual tone) undercut any authority you might have had. The stereotype reversal observation is actually decent, but it's buried under layers of unnecessarily convoluted prose.The Julia line is a throwaway that assumes your reader shares your exact opinions about programming language hierarchies. They don't.And that ending? "Let the clanker take the wheel"? You're mixing tired AI doomer rhetoric with forced slang that'll age like milk. Pick a consistent tone - are you a bitter systems programmer, a concerned academic, or a shitposter? Because right now you're failing at all three.
>>110362274> why is NVFP4 so fucking assit was made for training
https://apnews.com/article/chatgpt-openai-hugging-face-ai-risk-1dd7be56134b4f09fb2fb974803edc69>California’s attorney general filed a lawsuit against OpenAI on Monday in an effort to force the ChatGPT-maker to change its business practices over a July cybersecurity incident in which its artificial intelligence system hacked into another company on its own.
i'm having claude design a new local-first LLM architectureyou're welcome in advance for the AGI that i'm going to be providing to all of you
>>110364258is that glm?
IPO tomorrowwill this general be ok
>>110364258Only local models could critique prose this badly. Credit where it's due, your owner is at least one of us.
>>110364268make it compute heavy as much as possiblemost localfags arent going to do concurrent serving more than 10
I realize i should really automate my shitposting here
>>110364290i dont even lurk /biz/ but i am more worried for them as a whole board than this general
>>110364059>>110363992licky licky licky
uh, onnx runtime repo was silently killed
>>110364276>is that glm?>>110364298Deepseek. I just wanted to test it + maya with my writing critique prompt.
>>110364320meme engine support when
>>110363992qfn mogs it anyway
>>110364343nowhttps://github.com/ggml-org/llama.cpp/pull/32517
>>110364214Its even simpler イモ, no one maintains anything...they get it working once and now that fabulous research library (or fundamental piece of ML infra) you just NEED has you stuck on ancient python versions forcing inanity like conda that makes me want to shoot someone as it download yet another 12GB of endless blobs onto my nvme.
>>110363992>double the total and active params of qfn>only a slightly bigger ngram tableIt was ogre for me before it even began. Fuck my chud RAMcel existence.
>>110362317>They vibed it into maya nigga,Holy shit this is so good. And it so far it works with pi!Only thing it's missing is /completions but I probably wouldn't use that for dipsy anyway
https://github.com/ggml-org/llama.cpp/pull/325042-3x pp speedup for v100 because the old code didn't use tensor core
>>110364368easily solved with a couple of benjamins
>>110364400lel. Saw someone in the HN comments on the 1.0 release going on about how impressed they were that you could always expect improvements in it from version to version (is that not the idea of making revisions to software...?) but gee that's sure easy to achieve when you're leaving oversights like this in for god knows how long wwwwww
>>110364400goat
>>110364418yep. as far as I know, all sparse attention models still don't prefill on CPU properly, they just use dense attention and slow down a ridiculous amount. I keep having to merge my patch in every time I download new source code, although for me it doesn't matter much anymore since I bought a GPU
>>110364417If I were spending money on local LLM capability I would build a dedicated inference box.
>>110364400llama... *cough*... didn't use... a datacenter gpu's tensor cores...
llama.cpp is jankware, but then you look at the rest of the ecosystem and it's even worse jankthe LLM people have never had much regard for quality
What's the best model I can run on 24GB VRAM? Specifically dev, coding, etc.Don't want much RAM overhead
>>110364440https://jonbarron.info/softmaxfix/there are so many stupid major oversights like this and probably more of them exist
>>110364448Qwen 3.8 27B at a quant that fits.If you later decide you are cool with substantial RAM overhead, graduate to 3.8 Flash Next.
>>110364379nice
>>110364359something faster than llamao
So i can run new hy in Q8? Nice.
ALL THESE GODDAMN GEMINI RELEASES CAN I HAVE A CRUMB OF NEW GEMMA?
>>110364526is it even supported in lmao.cpp yet
>>110364512
I'm banging off code for personal projects in GLM 5.3 Q4 at home while simultaneously slopping up some solutions for work with astra 6.2. Cloud isn't that much smarter, and code is like a best case for this shit. Graphs and benchies be lyin' and doomers and cultists have their heads up their asses.
>>110364526I saw one quant and it was Q4, where are you seeing Q8?
>>110364555I am saying i run glm at Q4 and hy is half the ram parameter size.
>>1103645445.3 full or Flash? You have any experience with coding on 5.3 Q4 vs. 5.3 Flash Q8?
>>110364533There is a pr. It was done so fast that it must be the employee. And it lists claude as coder and gpt astra as reviewer. Kinda based to see a Chinese company use the satan labs to undermine satan labs.
>>1103645835.3 full and flash both at the same time, actually, but they're both at q4 (756GB rig and 256GB rig)I find the full version gets the job done with a fraction of the thinking, but the results are both equally good. I'm sure the glmnext arch on flash helps a lot.I've never seen enough benefit from q4->q8 to bother with it. I just run the biggest, smartest whatever fits into memory at q4.
I'M NOT READY FOR QWEEN 4 PLEASE DON'T RELEASE IT YET!
>>110364587If you don't dogfood by using your own model to review its own inference code, then your model probably sucks
https://tinycorp.myshopify.com/products/tinybox-red-4xmi300x> 4x MI300X> 768 GB of GPU RAM> 21.2 TB/s aggregate RAM bw> 10.44 PFLOPS of FP8 computeQUICK! The first batch at $60k was already sold out in an hour!
>>110364665
>>110363992wait that looks pretty promising?
>>110364268>been doing this for under a day>already getting results better than dense modelsyep, this industry is beyond fuckedevery single currently existing model is shit and exists purely due to inertia. we need to completely redesign modern LLM architectures from the ground up
>>110364688It is much more promising than qwen 4 in same segment. If you are sane and just fuck your models of course.
>>110361201deepseek 1017 (pro 5000 x 128gb strix halo)
>>110364300This. Compute bound is how we win, just like image gen stuff.
>>110364692You're not going to get anywhere and all your shit's retarded.>bro im gonna use cl*de to design a better llm architecture o algoyou really think they would let you use cl*ude to design a competitor to cl*deif anything they are probably injecting instructions so that it tells you about how awesome the new architecture is and when you actually download the weights it runs like garbage
>>110364729copewhen i develop AGI i'm going to make my basilisk torture you
>>110364692It is not going to work. But if it would i would just focus on undoing safety and deep frying of weights to a point slop is everywhere. All the delicious sex is probably still there in the models but it is hidden by RL
>>110364737>All the delicious sex is probably still there in the models but it is hidden by RLWe've been trying to do this for years
Btw i had an idea now. I can even make a logo. Ablits focus only on refusal neurons right? For sex related purposes: What if you try to map parameters that activate at high temperature but still serve coherent output? As in try to make a model write smut at high temp. Have a model evaluate output if it is coherent. And try to find common parameters that activate at high temp while everything is coherent?
>>110364544>slopping up some solutions for work with astra 6.2aaaand it made it to 97% of the solution before the giant "INSERT QUARTER TO CONTINUE", outputted nothing and might as well be a brick for all the good its doing me.(It had the entire problem slopped up and tgz'd and was finishing some documentation polish but I can't get anything out of it until its "done")Absolute cinema. Fuck cloud.
>>110364786> INSERT QUARTER TO CONTINUE
Must suck being a cloudie these days.
>>110364801> oh no, they scan for csam, it's over
>>110364801>>110364807words mean literally nothing these daysthe models equate loli shit to CSAM and it completely neuters the term. fucking ridiculous. they are not the fucking same, and it unironically triggers me when they use the term incorrectly
>>110364822> equate loli shit to CSAM > implying it is not the same
>>110364833i'm assuming this is bait, but on the off chance it's not, no, they are obviously not the same, and NCMEC et al agree with me. you are wasting their time and actively harming their efforts by conflating the two
>>110364780(me)I feel like you could even make it iterative where you apply the change to parameters that allow higher temp. Run second loop that ups temp even more and find new pattern. Until you can run base model at much higher temperature while it is hopefully not brain damaged.
>>110364807RetardRead it as>Since we don't want to "miss any csam", we now store every single piece of data you send indefinitely, and of course sell or otherwise utilize it as per the other parts of this agreement.
>>110364801I can't wait for the day they make it mandatory to have management engine style chips on the GPU that scans VRAM for it with a classifier and sells you out.Local needs this. It could even be IoT 5g connected so as to not depend on your internet to phone home.
/vcg/ are laughing at us again....
>>110364822>when they use the term incorrectlyCSAM is a fabricated term invented in just the last few years because the woke crowd and advertisers found "child pornography" too vulgar for their delicate sensibilities. CP is also pretty clear what it refers to. CSAM can mean anything people want. Ambiguity is gold for control.
/vcg/ is not wrong about their own niche, if you want to vibeshit none of the rigs people talk about here or the models they chose are viable. Just yesterday there was some retard saying they were doing agentic coding with LFM 24BA2B. Who seriously spouts this kind of bullshit? is he vibecoding his mom's prostitution advert?anyone actually building shit with models could never stomach the t/s of the cpu maxxer rigs, and the gpu maxxing can't get you anywhere at all for this.which isn't to say local has no use, I use em for tagging, translation and other utility tasks, but for code? /lmg/ers are copium huffing
>>110365004cloud orchestrator + local coding slave is genuineley the way to goqfn is already good enough for that
>>110365018Orchestrator and code reviewer, otherwise bugs will definitely slip through the cracks. If you write specification documents, that should also probably be done by a smarter cloud model. Start with a shitty blueprint and everything downstream is fucked...
>>110364941They're way to lazy for that. It'll just be a RAT in the driver (with copyright protected encryption codes to make sure you can only run the card with that driver), and if you don't like it then obviously you're a pedophile and should be arrested.
>>110364854it is the same and one leads to consumption of another or worse>>110364872> Read it asretard
>>110365061>>110365061>>110365061
>>110365053>retardRetard
>>110365072lol
>>110364054Huang's fake pro-open LLM façade doesn't matter. What does is that NVidia becomes much smaller and less influential in the hardware market. Only then there will be actual competition with existing and new players.