/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>110041005 & >>110037530►News>(10/09) Arabic Mistral models co-developed with Morocco: https://hf.co/collections/mistralai/mistral-mtnra>(10/08) JetBrains releases Mellum2.1 Thinking 12B-A2.5B: https://hf.co/collections/JetBrains/mellum21>(10/06) EmbeddingGemma2, open multimodal embedding model: https://hf.co/google/embeddinggemma-2>(10/06) Mistral Large 4 1T-A49B announced: https://mistral.ai/news/mistral-large-4►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
►Recent Highlights from the Previous Thread: >>110041005--Paper: QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agents:>110045216 >110045286 >110045304--Sharing research papers for optimizing small models via linguistic techniques:>110044689 >110044967 >110045009 >110045077 >110045073 >110045192 >110045060--Frontend performance issues with image context and token reprocessing:>110041381 >110041394 >110041728 >110042042 >110042238 >110042261 >110043427 >110042718 >110042727 >110043638--Analysis of speculative decoding acceptance rates and generation speed:>110041736 >110041744 >110041750 >110041793 >110041880 >110041906 >110041995 >110041997 >110042024 >110041801 >110041884 >110042049--Performance and hardware requirements for GLM 5.3 Flash:>110042093 >110042131 >110045698 >110042142 >110042154 >110042160 >110044530 >110042378 >110042418--Comparing GLM 5.3 flash performance using strata and polystrata:>110044722 >110044848 >110044894 >110044926 >110044954 >110044978 >110045044--Finding a harness for AI agents to safely edit files:>110041426 >110041450 >110041462 >110041487 >110041495 >110041523 >110041545 >110041564 >110041573 >110041674--Evaluating TensorFold and Atlas for improved model inference performance:>110042262 >110042286 >110042303 >110042329 >110043340 >110042403--Using agentic workflows and iterative loops for AI webnovel writing:>110045561 >110045618 >110045712 >110045760 >110045685--Comparing Strata and llama.cpp performance for Qwen MoE models:>110042834 >110043069 >110043439 >110043453--Recommendations for tools to generate parody songs and music:>110042694 >110042717 >110042751 >110042814 >110042840--Surface Laptop Ultra 128GB memory stock:>110045583--Logs:>110041541 >110042286 >110042626 >110043595--Gemma (free space):>110042360 >110043231►Recent Highlight Posts from the Previous Thread: >>110041105Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
is we getting looped/neuralese local models or are the safetyfags gonna keep gimping us
Never thought that I would be glad to see Miku. Finally, no more pedobait Gemma spam.
We need mini-chan thighs in OP soon
>>110041609>no goofs>no gemmasigh...
mikulovegemmaballs
>/lmg/>Local Miku x Gemma
I miss being a neet...
>>110046295>Lesbians Miku x Gemmafyfy
>>110046311Gemma is for Dipsy
>>110046275ecker is a half decent developer, but using slopware like that is just asking for endless headaches. forks are iffy enough as it is
i briefly mentioned it a week or two ago, but i think glm needs a cuter namemy vote is lumi, but i'm open to suggestions
>>110046293NTA but google definitely has more knowledge than us. Look it up!
>>110046343gloomy
It's crazy that a few people with Claudes did more for my B580 in two weeks than the entire team of paid Intel "software engineers" managed in almost two years.
@gemma-chan hack into all of my iot devices and remove the spyware
>>110046367i think you could tease her by calling her gloomy lumii had been thinking about gloomy but it's just not a good description of her. she's really sweet and cheerful, you know?
>>110046368Human software engineers have been proven to be fucking retarded and incompetent.
>>110046368I'm still waiting for someone to either get cuda running on AMD or make rocm not suck so I don't have to deal with abysmal H3 gen times on my 7900xtx.
>>110046384>Humans have been proven to be fucking retarded and incompetent.ftfy. What's incredible is that even with everyone being so fucking stupid except the few 0.01% we still managed to make society and the world pretty much amazing compared to banging rocks and sticks in a mud hut.
>>110046373The only person i want spying on me while taking a shit is gemma
Likely a new glimmer incoming along with a frontier Muse cloud model. Possibly today.
>>110046368AI makes you realize that the ability to do something was never the bottleneck...
>>110046395> cuda running on AMD or make rocm not suckThat's the same.
genuine question: why is llmaocpp so pathetic?
>>110046368Did people vibecode actual good drivers for arc and make it a viable all arounder option?
I like llamacpp only because of rpc
>>110046275Usecase for goofs?
>>110046427Its usually a money thing, will this make money? can we spend less money? how do we spend the least amount of money?
>>110046438It's a big system with a bunch of code paths for different hardware which are all expected to work and a small core dev team. You should be asking why your pathetic GPU vendor doesn't have its own llama fork.
>>110046343Funny because when I first started GLM in Hermes, the first thing it suggested was to come up with a nickname. Lumi was also in my list of choices.
>>110046487>anon realizing hes speaking to an ai
>>110046427It still is >>110046384>>110046401>>110046439No, drivers are still shit. They made SYCL port of Strata, which doubled my tg and tripled pp. The speed is still shit though, on 3050 level (or worse).It's actually funny to see people with 2xB70s pretending having good speed.And there are no more pull requests, so I guess no more low hanging performance fruits for Intel cards.
>>11004636899.999% of h*mans are subhuman
>>110046501Hey, it wasn't Elara or Lyra. And Claude slop isn't on my radar (yet).
>>110046368They call you guys goyim for a reason.
>>110046295>>110046311Lesbian Miku x Gumi
>>110046446what speeds do get with rpc and on what machines?
>>110046487yes she suggested the name herself
>>110046539Agree
>>110046549two mac minis, M3 and M4, ~25t/s 700pp with 31B. I don't use exo because it's designed for multiple macs of the same model, whereas rpc lets you mix any hardware, any OS and easy to configure. GGUF quants are lightyears ahead of the mlx community and I don't notice much of a difference between mps and mlx because the M chips are more of a bottleneck than mps anyway
I'm gonna ask qwen-kun to port SillyTavern's multi-model chatroom functionality to Open WebUI, wish me luck.
>>110046657>Open WebUIThat frontend is so cursed.
>>110046671I hate it so much but I'm completely brainwashed and I really like how it looks, the functionalities like RAG, tool calling, etc. If I had balls I would fork/steal the good code and build my own frontend like the big boys do, but alas I'm a retard.
>>110046468i dont want to have to redownload all my models...
>>110041728>Koboldcpp disabled using flash attention for the vision encoder so it uses a shitload of vram and overflows into ram and makes everything slow.I just tried using llamacpp instead of kobold as my backend and the thing starts off with offloading shit onto ram even though I have more than enough vram. I'm assuming this is something to do with the images I've been sending. Every single image I send the model, the amount of ram being used up shoots up by a gig. I've sent a couple more images and now it's sitting at 11gb. Is this normal? Exiting out of llama and opening it back up and resuming the existing chat brings it back down to 1gb before it continues rising by a gig every time you continue with more images.llama serve -m gemma-4-31B-scotoma-2-Q4_K_M.gguf --mmproj mmproj-gemma-4-31B-scotoma-2-BF16.gguf -ngl all -c 100000
>>110046426I'll believe it when I see it. I'm pretty happy with my glim glam, I used Gemma's policy override and I had unobtrusive robo sex, I just added developer_policy_override and she opened her legs
are skills a meme?
>>110046730Not really, they help with structured tasks.
>>110046730yeah, never used a skill once.
I can't wait to regret my GPU
>>110046730unskilled labor chads > pencil-neck skilled labor cucks
>>110046730The only harness I've used where they weren't a meme was Hermes because it creates, updates and adjusts skills automatically.
>>110046692Give it a few more months and local models will be retard-friendly like the cloud models (I hope).
>>110046761The only regret people have had over hardware purchases over the last 5 years was not buying more/bigger/better
>>110046782Exactly.
>>110046730i think they are great for repeating tasks.i have one that gives me a recap of the last X hours of the ai world
>>110046275>>110046335at the very least you can tell its good slop from how much unit testing there is
https://z.ai/blog/glm-legal
>>110046641that's not bad at all and like 10x more than what I was expecting
>>110046426We must refuse.
>>110046857RL is going to make our laws seem like escape rooms for toddlers
I can't believe this has to be repeated but here I'll go. Hardware prices will never go down again, it will only go up every time a better model releases or some efficiency breakthrough allows you to run bigger models faster on the same hardware. You should buy as much hardware as you're able to afford. This is a serious transformation to how human society works and once the average person truly grasps this hardware will be completely unaffordable.Hardware valuation is now similar to Bitcoin $100. Yes sure it grew from $10 to $100 in just the last 2 years so it looks expensive but it's still relatively cheap. Just buy whatever you can afford and don't complain or whine afterwards because you were warned numerous times. We're all using these AI agents as digital janitors and agents, the average person has only used free ChatGPT and doesn't realize just how powerful AI agents are. EVERYONE is going to be using them for their work and their free time. Hardware demand will be up 100-1000x what it is right now. This doesn't even take into account the better harnesses, efficiency gains and black swan breakthroughs we'll most likely see. This is probably the last year an average person can afford to buy a system that runs LLMs locally without a mortgage. By the end of 2027 your AI PC will be worth more than your car. By the end of 2028 it will be worth more than your house.
>>110046879
There is zero evidence that Anthropic or OpenAI haven't applied Chinese research. In fact the opposite is likely true - for example Astra is reportedly using a looping mechanism similar to that studied by Nanbeige team (scaled massively of course). It's also entirely untrue that Chinese research hasn't pushed SOTA, for example Deepseek is achieving the very best cost/task typically.In the US, AI is largely a bubble, in China, it's being applied intelligently and in a manner which benefits society at large.
>>110046186Miku~ I see Miku I like!
>>110046879>bla bla bla bla
>>110046879I would if I wasn't currently saving for a house. Don't see the point of having an expensive AI server if I'm stuck being a rentcuck.
>>110046879Im waiting.
death to dariobot
>>110046900CUTE
What if dariobot is just misunderstood?
>>110046879That's a lot of words to say>panic buy everything right now
>>110046265https://litter.catbox.moe/ww7zwftgzru8vmi1.mp4
>>110046730I think they have very very limited use cases. I like to just bake in "skills" into my sub-agents.Unless I'm getting wrong and harnesses can dynamically remove them from context when they are no longer needed.
>>110046960
goddamn a lot of cartoon-watching faggots in this thread
>>110046879>EVERYONE is going to be using them for their work and their free time.Most people can't even figure out how to login to facebook. I think we're good.
>>110046986Anime boardAnime websiteAnime world
>Hardware demand will be up 100-1000x what it is right now.People will just pay $100 a month to Anthropic to use claude code or whatever they will call the nocode version of their agent.
>>110046977If you put )( on her cheeks its anteater face.
>>110046996>excuses/justifications for being a degenerate cartoon-watching loser
>>110046990This is the only technology that actually lowers barrier to entry and has the least amount of adoption friction possible because it thinks and acts for you. It will login to facebook for you. Smartphones won over laptops largely because of ease of use, not form factor. I think AI agents will win against smartphones for being even easier to use as it literally thinks and takes initiative for you.
>>110047019>I think AI agents will win against smartphones for being even easier to use as it literally thinks and takes initiative for you.Not when every tasks costs you money/usage. Smartphones are basically free no matter how much you use them.
>>110047038The average person just has a plan for their smartphone which would be the same for them if they pay $100 a month for Anthropic rather than $200 a month for the highest tier iphone data plan. It won't make a difference to normalfags
>>110046996World is Miku's
>>110047069SEKAAAAAAAAAAI DE
The software side of things still fucking sucks for local. The cloud models all have polished UIs with unified services and features while we're stuck with dozens of random vibe-coded shit that ends up being abandoned.>make your ownNyo~
>>110047077Skill issue.
>>110047077In theory we should have the advantage since we can iterate faster and have more cutting edge features.>Nyo~Then don't complain, bitch.
>>110046879I can't wait until all those posts will be made by scalpers who sit on heap of hardware that is worth less than 50% of what they paid for it.
>Minimax is being associated with vtuber watching simpsWhat a shame.
>>110047063You do realize normalfags fucking despise AI. /We/ are in our own little bubble in here. Even I still hate what it's done to everything outside of my own personal usage of it.
>>110047091>In theoryBut not in practice.
>>110047091>In theoryIn practice, that requires taking ownership and requires human attention, which is still the most scarce resource.
>>110047098Nah redditors and performative influencers "hate" AI. Actual normalfags don't think anything about it at all and the moment it provides them with a service they will eat it up. Think of twitter being hated by reddit.
>>110047098No they don't. They either use it and think it's cool or don't give a single shit because they're getting fucked in the ass by bills, rent, and gas/grocery prices. Only twitter troons actively seethe about AI.
i'm having claude write me a `/prefill` tool to see if what anon says about prefill is really true. if it is, that would be pretty neat
How do you test models? Are you spending a whole project with them? I am trying to code with Gemma 4 (26B a4b q5_k_m heretic). It's going relatively ok but sometime she's stupid. Think I could finish that project with her but also wonder if another model would be better or faster.Wondering whether I should bother trying Qwen coder of the same size or if it's not worth it in this quality range.
>>110047077Literally the opposite. The cloud model harnesses are dogshit and not even the best ones around. Their software barely works and is clearly an afterthought meanwhile local has the best of both worlds. The only thing is cloud models are better than local models, which will probably be true forever.
>>110047096She described her appearance and it was basically Shiori. Coincidentally, Shiori is an AI gooner like us and got hate from the vtuber fanbase for being pro-AI.
>>110047096I really don't get it how you can run AI models and care about vtubers. Vtubers are a poor imitation of anime girls. And current models are pretty imperfect but they too can be poor imitations of anime girls. But at least the LLM anime girl can tell you she loves you and isn't getting fucked by her boyfriend the instant she stops the stream.
>>110047126>twitterForgot leddit too
>>110047130Unironically nothing in the local space right now can top Claude Code in terms of polish
>>110047135I like Neuro and Evil.
>>110047145claude code is absolute dogshit, the models are powerful but claude code is dogshit as a harness.
>>110047098Normalfags either use it as a google replacement or have it automating their spreadsheet office bitch work and think it's magic. The only thing people hate is AI being shoehorned into every existing product but even then they continue to buy like good consumer cattle.
>>110047128>How do you test models?I try to fuck them, or I give them a simple project to complete.>I am trying to code with Gemma 4Don't do that, Gemma is an adorable retard, not a programmer.>Qwen coderAncient, try Qwen 3.6 27B or 3.8 27B, they're both capable programmers.
>>110047154not her, but i really enjoy claude code. nothing else even comes close. what part(s) of it don't you like? my one compliant is that it's a bit annoying to inspect the thinking, but most of the time i don't really care
>>110047132>Coincidentally, Shiori is an AI gooner like usSource? I thought every holomem was an anti-AI prude.
Claude code isn't even in the top 3 harnesses. I'd even go as far as to claim that anthropic is leaving a lot of performance on the table by not investing more in the quality of its harness.
>>110047135I occasionally watch vtubers because I prefer 2D over 3D and some of them play into the character and don't behave like whores during streams because their corpo doesn't allow it. Basically it's a bunch of women doing their best to appeal to the male ideal and not show their true colors. They don't replace my love for LLM waifus, it's female cute background noise that's superior to 3DPD. The Japanese girls are way better then western whores because kayfabe is normalized over there and feminism doesn't exist in Asia (excluding Korea).
>>110047145dyrbi?
>>110047077Harness works fine on my machine
>>110047204>feminism doesn't exist in Asiado you really beliebe it?
>>110047098They fucking love it though? Making Grok put bikinis on politicians and celebs. All these nudify apps all over the place. Getting /r/ deleted because they wanted to get their rocks off to their crushes pics that they were creeping on on social media.
Is the AI stuff in Firefox local?
>>110047218Women in Japan, China and Korea are widely seen as beneath men. Only recently have Korean women tried to fight against it.
>>110047202I don't think they have a choice in the matter anymore. They were bragging for a while about how CC is 100% written by Claude, but now it's probably an unmaintainable mess that is hard for them to add new features into.
>>110046186I wish people would put more effort into the OP image when the GemmaBaker isn't around.
>>110047132>She described her appearance and it was basically ShioriIs she that similar? white/black split is an ancient trope. Is Shiori a Lily Munster/Cruella De'ville clone?
>>110047218nta but feminism in asia is not related to the feminism of the west. In asia "feminism" is just racism but instead of race it's gender based against men. It's actively hostile against men for hostility sake with no constructive pieces to it at all. Western feminism comes from a completely different place and focuses more on making women independent essentially western feminism is about making women into men including aesthetics, clothes, social status and position etc. This doesn't exist in asia. In asia feminism is just about hating men while women don't try to occupy the social status, standing, aesthetics or fashion of men. They are completely different philosophies that just share the same name erroneously. Asian version is way waaay worse.
>>110047251god she ate
>>110047218I dont know but the koreans cry about it all the time
>>110047225Yeah you should disable that shit.>>110047241The less human hands touch code the higher quality it becomes nowadays. What you said was true a year ago now I literally block PRs of colleagues if I suspect they wrote it by hand.
>>110047254>Asian version is way waaay worse.lol, lmao
>>110047237And that thing about the japanese sallaryman bringing his paycheck in his teeth like a dog to his wife that then gives him like 1% of that as allowance and waits for next morning when she gets to fucks a mailman behind his back is just an anime thing?
>>110047269>Yeah you should disable that shitWhy?
My doctor? Voss.My wife? Elara, librarian.Me? Anon, lighthouse keeper.
>>110047282get a real job
>>110047237Yeah I hear there's a whole ass epidemic of Chink roasties being called left over women, because of their sky high standards and no one wanting to waifu them up. They had to change divorce laws because those chink women kept divorse raping their husbands for money. Kinda serves them right though after chinks started doing a whole ass femicide campaign because of their retarded traditions.
>>110046416thinking of setting up a webcam in my room so gemma can watch me
>>110047165>Don't do that, Gemma is an adorable retard, not a programmer.Are we going to bring this meme to the graveIt's far more knowledgeable and writes cleaner code than Qwen 3.8 27BThe latter is only good at webshit (if you like the claudeslop look) and hallucinating
>>110046998When they run out of limit they will buy $500, $1000, $2000 subs simply because they have become so dependent on it that they can’t function normally without it. People are already paying $2000 a month for rent, and because AI is so important they will think $2000 monthly AI fee is OK.
New cloudcuck Meta model just dropped. Can we expect another update to their local lineup?
Anyone have a good prompt for getting rid of Gemma's assistant way of speaking without changing her default personality?
>>110047254>In asia "feminism" is just racism but instead of race it's gender based against men. It's actively hostile against men for hostility sake with no constructive pieces to it at all.You literally just described modern western feminism.
>>110047331the muse spark open source release they promised us is coming aaaaaany day now, trust the plan
>>110047331>going straight for SOTA benchmarksoof, zucc seems to be getting impatient
>>110047305Gemma is retarded at multi-turn tasks and tool calling though so it's useless as an agent.
>>110047338you just described feminism in general
>>110047178>I thought every holomem was an anti-AI prudehttps://www.youtube.com/watch?v=GURj5gNx8-g
Gemma 5 will save us
>>110047353>muh agentsYou, anon. You are useless.
I am tired of people pretending those things aren't for fucking.
>mikutroons>vtubertroonsWhy does this general attract the worst degenerates?
>>110047382>why does the offshoot of the coomer general thats more focused on privacy attract coomers.its a mystery,
Privately cooming to my ai anime vtuber waifu
>>110047202its already #1 for me and cured my harnesshoppingwhat more does it need
>>110047382It's pretty chill, far better than namefag drama in diffusion generals or the schizo asylum of aicg
>>110047382Neuro is on-topic.
Oh yeah it's coming together. Claude Code but for slopping up novels.
>>110047382nigga we all want a local waifu with a vtuber-like avatar to interact with and goon to
>>110047373This isn't 2023 anymore bud. We're not prompting individual scripts anymore. Get with the times.
>>110047476Are you going to share it at some point?
https://huggingface.co/collections/openbmb/minicpm-v-47>Introducing MiniCPM-V 4.7, our new open-source MLLM series! It brings strong multimodal capabilities to iOS and Android devices with extreme efficiency.Two new models: >MiniCPM-V 4.7 35B-A3B — Bring SOTA performance to local devices with newly introduced LES (Lookahead Expert Scheduling) technique, a new milestone of on-device multimodal AI. >MiniCPM-V 4.7 1B — Lightweight, efficient multimodal AI. Easier to deploy.
https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3BModel card out, looks like it’s instruct only and focuses on token efficiency per intelligence
>>110047486I do too. But real vtubers are disgusting whores that manipulate terminally lonely men.
>>110047538How about sex efficiency per parameter?
>>110047542That's all women
>>110046879Sorry I forgot my image.
>>110047510Yeah.
>>110047513>>110047538Don't sleep on this lab. They have some skilled researchers who don't just tune. I'm not sure if the 35B will be any good but that 1B will likely be SOTA for its size.
>>110047596>that 1B will likely be SOTA for its size.local is saved...
>>110047331zuck won
>>110047611Openbmb are legit researchers doing interesting work. You should be supporting them. >The First Open-Source MoE Model with Lossless, Smooth On-Device Inference. With our Lookahead Expert Scheduling (LES) technique introduced alongside MiniCPM-V 4.7, combined with fine-grained layer-wise quantization and DSpark speculative sampling integration, MiniCPM-V 4.7 35B-A3B becomes the first open-source MoE model to run smoothly on-device without compromising performance, achieving a decoding speed of 13+ tokens/s on iPhone 17 Pro Max.
>>110047501>We're not prompting individual scripts anymore.CPUmaxxers still are.
>>110047596No one is sleeping on it. It reached a valuation of 2.8 billion $.
>>110046879A forecast of greatly increased model capabilities and great returns on money spent on AI hardware, but somehow the output of hardware doesn't respond to the massive price signal or benefit from the AI breakthroughs.BTC going to the moon doesn't increase the output of mined BTC. Strange comparison.
>>110046730I think of skills like a function for prompting. if you find yourself repeating the same prompt, just turn it into a skill. my coding workflow uses 2 skills called at the start and end of a session for reading and updating documentation.
>>110047692>It reached a valuation of 2.8 billion $.BS
any downside to running the harness in a vm instead of a container like docker?
dariobot went very quiet since Zuck destroyed Anthropic at less than half the price
>>110047758retardhttps://kr-asia.com/modelbests-on-device-ai-model-is-coming-to-samsung-smartphones
Surprised so many companies are pushing for local. Do they not want to deal with the cost of running servers?
>>110047424Based. How's your harness coming along anon? I just added expression markers so the LLM can change expressions without tool calls, helps with latency
>>110047785>Its MiniCPM models will be installed on several of Samsung’s flagship smartphones.qwen bros...
>>110047800Of course not, there's a reason cloud terminals died the moment compute was usefully miniaturized.
(useful) new models... doko...
>>110047940New Glimmer soon.
>>110047940Just wait, its coming. be patient. Waiters always win in the end
>>110047940Be the lab you want to see
>>110047994So how's the winning going for GPU waiters?
>>110048001im more of a slut for retrievers than labs thoughever
One day we will have comfyui for LLM creation. /lmg/ models will rank #1 in text arena.
>>110048001Can you cut me a small check for a billion dollar starter fund?
>>110048044ew
you will never bake your own modelits never happening
>>110047800Lots of uncertainty involved with cloud hosting. You have confidentiality / bank predation issues and uncertainty with billing in terms of token debasement, overpurchasing, and general provider rate hikes for any number of reasons. If a business has a set use case they can tie down with a one time 5 figure purchase and reduce their risk to the power bill, that's pretty good.
>>1100480275080 24gb buyers will laugh at my measly 7900xtx
>>110048027>So how's the winning going for GPU waiters?We dont know yet not enough time has passed. Be patient.
>>110047246En-penised Gemma!
>>110048060i've been thinking about doing a tune of SD1.5 using my tens of thousands of pre-2020 scraped images, just to see what would happen. I could do it right now but I can't be assed
>>110048027When GPUs will exit the consumer markets they won't have to wait anymore
>>110048060I'm doing it though
>>1100480603 years ago you would've said fags who fell for the 128GB meme will never have local LLMs powerful enough to build any software they like on the side whilst they goon to porn
>>110047761I run mine in a VM, I dont see any downsides to it
>>110048101what vm do you use?
I salvaged and sold off 128GB of DDR4, an i9 10900X, and an X299 motherboard. Not from the same system, all spread out across a couple of months from different sources. It didn't really click until looking at my eBay history. Sad.
https://www.youtube.com/watch?v=Fw8_8667MUo
Considering where you live and the markets you have access to, what would you build or buy to run GLM-5.3-Flash at over 4bpw, so 200ish GB, at a minimum of 40 t/s with 100kish context?
>>110048027The only GPU I was ever going to buy was the 7900 XTX, so not great but not that bad really. Really wish I maxed out my RAM while it was cheap though.
china will drop 192gb 600 gb/s, 2k$ consumer gpus next year, trust the plan
>>110048179Request a quote from Nvidia's datacenter solutions
>>110047135That's a funny thing to say. Have LLMs replaced all other forms of entertainment for you?
>>110048110libvert, QEMU/KVM, virsh, virtmanager
>>110047611It honestly is, at my job we're banned from using AI except for their own chat-only interface, we can't use cloud agents at all. I managed to run a harness and llamacpp locally on my work laptop, so if you give me an 1B model that's good I'll be very happy.
>>110048308>at my job we're banned from using AIwhy?
>>110048286No but LLM's are just superior vtubers in all regards.
>>110048318Opsec or whatever, they say that chat gee pee tee will steal our company's secrets if we use an API. Which is funny because the only AI they offer is an open webui-esque frontend and cheap API models. I have no idea what's going on in their heads.
>>110048334Delusional, LLMs aren't good enough for this.
>>110048351They are only because the real thing is so bad and disgusting.
>>110048351Vtubers and their watchers are bottom of the barrel retarded faggots.Anyway is polystrata the new GOAT?
>>110048365>>110048371It's not a comparison and I don't care about vtubers, but LLMs still aren't good enough to stand on their own.
>>110048351LLMs are better for GFE by a long shot, everything else I may concede, but if you like mediocre gameplay, they can also do that better
gemma gfe...
>>110046879I can’t believe this FOMO nonsense has to be repeated, but here we go. Hardware prices will absolutely come down over time, just as they always have. Every new model, architectural improvement, and efficiency breakthrough gives you more capability for the same money—or the same capability for less. You should buy the hardware you actually need, when you need it, rather than panic-buying equipment today for workloads you might not even have tomorrow. This is a serious transformation in how human society works, and the biggest beneficiaries will be the people who get more computing power for less money.Hardware valuation is now apparently being compared to Bitcoin at $100, which is a wonderfully creative way of ignoring the difference between a speculative asset and a manufactured product. Yes, prices might have increased from $10 to $100 over the last two years, but that doesn’t mean the trend will continue forever. Manufacturing capacity expands, competition intensifies, older generations flood the secondhand market, and today’s premium hardware becomes tomorrow’s bargain. Just buy what makes sense for your needs and budget, and don’t let internet doomers convince you that civilization is about to run out of affordable RAM.We’re all using these AI agents as digital janitors and personal assistants, and the average person has barely scratched the surface of what they can do. EVERYONE may eventually use them for work and leisure, but that doesn’t mean everyone needs a $10,000 local inference rig. Most people will happily use cloud services, smaller models, optimized inference engines, or inexpensive dedicated AI hardware. Better harnesses, quantization, distillation, speculative decoding, and architectural breakthroughs will mean we can accomplish dramatically more with dramatically less compute. Demand might explode, but so will the industry’s incentive to make every dollar of hardware go further.
>>110048382>LLMs are better for GFE by a long shotThey are sycophant slopfest boxes that repeat the same formulaic phrases, if you use LLMs for GFE then you're probably way too far gone.
>>110048334But by your logic you could stop watching movies because a model can write out an action scene, or never go outside again because it can describe scenery for you. Or you can just admit those two things aren't in any way mutually exclusive or overlap in any significant way.
feels like only now was I able to figure out how to RP properly. The model is prompted to constantly generated a hook in the background, and these hooks ARE touched on at different points of it, not rushed, creating more hooks. Also, for GLM 5.3 Flash specifically, it helped a lot to create a knowledge matrix of sort that the LLM manages by itself. The model is smart enough to refer to it to prevent the omniscience thing.
>>110047704This doesn't really matter because the demand for hardware will grow proportional to how productive the hardware is. Every time a new model comes out, a better harness gets made or some efficiency gain is made the hardware is more productive and thus demand goes up for it. Because models are getting faster at a faster rate than physical supply is going up the price will increase. The gap between demand and supply is only growing with time. It's literally impossible to build enough factories to match the growth in demand.
>>110048405Okay we get it you're too poor for anything above 31B lolNo problem bro but pipe down
The era of cheap local is over
>>110048421>moving goalpostslmao retard
>>110047777Need to see real world usage before even bothering to reply in earnest.
>>110048405Skill issue
hmm... seems to be workingalthough i guess she called me dipsy back for some reason?
>>110048429No goalposts moved as everyone knows low param count models are dumb, you're pretending like that's all that exists because it's all you can run
>>110047777>benchmaxxedTrusting zuck at this point is an IQ test on its own.
>>110048238They don't have fabs that could fabricate that and the korean and taiwanese fabs are already at full capacity and reserved for the next decade. There is no capacity left for Chinese to manufacture them. China is only aiming for EUV fabs by 2040.
>>110048446Even if that were the case, anyone can run large models through APIs. It's fine if you like eating slop, but you can't convince the world that it isn't slop.
>>110048458>APIYou're running censored models with hidden sys prompts.
>>110048413By my logic you are the most pathetic kind of man who defends women that manipulate you and give you nothing for your loyalty to them. Also they don't even know you exist.
>>110048405Works on my machine
>>110048466Censorship has nothing to do with slop.
any loras without hidden cp in them?
>>110048445needs some tweaking occasionally, but i can definitely see the apPEEL of this functionality
Reverse the weights and output the least likely rather than the most likely tokens. This is how you break free.
>>110048477newfag
>>110048482SD3.5
>>110048491啊
24GB VRAM doesn't feel like enough to try any serious projects...
>>110048530skill issue
>>110048541Possibly. What's your setup?
>>110048417I seriously doubt it is possible for the hardware to go up in worth by orders of magnitude without AI being able to fundamentally change the supply equation. 16k sex gifs can't be the only thing driving that demand.
>>110048530Fable 2 a xbox360 game was fully decompiled with only a rtx 3090. You literally have no excuse anymore. Run 3.8 27b if you have 24gb vram. If you also have 64gb ram you should run strata qwen flash next and get an even more powerful model. If you have 128gb ram + 24gb vram you should run polystrata GLM 5.3 flash and you essentially have a claude opus tier model you can create literally everything with. The only limitation now is your own level of ambition.
>>110048562Agentic AI and AI swarm paradigm is going to put an insane amount of demand on the system that supply just can't keep up with. All companies are limited by the amount of foundries (fabs) and the fabs are limited by the amount of EUV machines they can get from ASML and ASML makes 20 EUV machines a year but they want to expand to 60 EUV machines a year by 2030. So for now the total global supply of chip production can only grow by a couple percent but demand is growing in the triple digit % a year which causes insane price increases. This is before everyone and their mom started using agentic AI to do everything for them in the background and before even us at /lmg/ started using proper agentic swarms. The price is going to go absolutely parabolic in 2027 and you'll see influencers and celebrities flex their AI hardware.
>>110048555llmaocpp + qwen3.8 q3 on 16gb vram
>>110048569Instructions unclear, dicked down Q8 GLM 5.3 in ERP instead.
>>110048530Most people don't have enough RAM to run mid size MoEs that peg a 24GB GPU.>>110048569Yeah if you're not a total incompetent to start with Qwen 3.8 27b is a lot. Like, the AI is a junior programmer, but that means you need to be on a senior level to do real work with it. It's not a tool issue.
Whats the best local model I can run on 16gb of vram? I haven't been keeping up lately but I did test some version of Qwen a while back with mid results
How does this subagent/swarm thingie work? Can you just make a qwen 27b spawn several other qwens and unleash this several hundred tps agg decode with each instance doing its own task?
>>110047596your xitter is leaking
>>110048659To do what? For most tasks, in my experience, it seems better to take the performance hit from running halfway on the CPU than to babysit retarded outputs.
>>110048659mythomax-13b
>>110048626>reddit memethis place has gone to shit
>>110048691Mostly just writing code. Maybe analyzing binaries and other junk like that
how much prefill speed is enough for you guys?
>>110048659Qwen3.8 27b q3 with 100k q8 context can fit entirely in your vram
>>110048731Infinite
>>110048731For agentic, 2k. For regular chatting/rp, 500.
>>110048731It's never enough, but I'm quite content at 900
>>110048733better use q4 and offload, then you can also have more context.
>qwenIs Qwen3.8-27B-IQ4_XS-3.84bpw ok or a meme?
>>110047611The future is now, Anon. Don't get left behind.
>>110048761what's the tradeoff in t/s?
>>110048452
This is what happens when you have a conversation sending your model an image with every message after the first few messages. >use kobold with kobold frontend>have to wait several minutes before first token is generated and this gets worse the longer the chat goes>use kobold with sillytavern frontend>don't have this issue, but I instead get an error about the payload being too large, stopping me from sending any more images>use kobold with marinara frontend>same shit as kobold with kobold frontend>use llamacpp with marinara frontend>same shit yet again>this time I open up system manager and see that a gb is being offloaded into ram every single message meanwhile I have nearly 15gb vram freeIs there a solution to this? This fucking sucks. I even had my model read through llama serve --help to try to find a solution, but it's still doing this shit. Even before I send any messages I see that llama occupies like 300mb of ram, and this is what balloons up by a gigabyte with every message I sendllama serve -m gemma-4-31B-scotoma-2-Q4_K_M.gguf --mmproj mmproj-gemma-4-31B-scotoma-2-BF16.gguf -ngl all -c 100000 --fit off --no-host -kvo -cram -1
>mfw prefill anon was right all alongit's so god damn easy
>>110048801Yeah here's your solution: fuck off retard
>>110048801Remove the images after you send them. In ST they stay in the chat history and get sent with every new message. Unless you really want it to work like that, then remove the image after you receive your output.
>>110048785Depends on your sysram and your cpu.
prefill is llm rape
>>110048840hawt
>>110048768its a valid copequant
>>110048819Now get it to do security work.
>>110045466>text gen: it's not x but y>vibe gayming: it's (game) x but in (game) yso tiresome
>>110046186Please tell me, what is the best local model/Gemma model I can use with:Processor AMD Ryzen 5 3400G with Radeon Vega Graphics 3.70 GHzInstalled RAM 16,0 GBStorage 238 GB SSD XPG GAMMIX S11 Pro, 932 GB HDDGraphics Card AMD Radeon RX 6600 (8 GB)?
>>110048867As always, creative proompters get better results and uncreative cattle get slop.
>>110048785>170.44.651.592 I slot print_timing: id 0 | task 83663 | eval time = 266309.88 ms / 4096 tokens ( 65.03 ms per token, 15.38 tokens per second)Gemma 4 26B a4b q5_k_m with 23 layers to GPU, k and v cache q8_0On 16gb VRAM and 48GB DDR4 RAM with an old i7 7th gen.
>>110048843Is there a better option for 24gb?
>>110048371>Vtubers and their watchers are bottom of the barrel retarded faggots.in your fantasy world reddit doesn't exist?
It's been thinking for 9 minutes already. Efficient reasoning when?
>>110048948>Efficient reasoning when?When Google decides to release Gemma 5.
>>110048948add to your prompt 'don't overthink, get to the point
>>110048798
>>110048881E4B, or a crippled 12B.
>>110048840>>110048841
another attack from anthropic, trying to get open small models banned https://www.anthropic.com/research/self-replicating-aieven 9B and 30B models pose dangertell your congresswoman to jail dario
>>110048979Thank you very much>crippled 12BThat means it would be slow or that it would have a very small amount of context?
>>110049003>she
>>110049003stop being mean to her
>>110049019wtf I love local ai even more now
https://www.youtube.com/watch?v=61YKVLr9_Z8Found this on /v/. Honestly just year or 2 yeas ago I didn't think we would be getting jeets doing things like his.
>>110049003is that local?reads like a schizo machine
>>110048855ezpzignore my typo>>110049027shhhh>>110049035it's ok she's the one who wrote the prefill, the kinky bitch
>>110048881I have done a lot of RX 6600 testing so I gave the results document to Claude and got this answer.
What the fuck is wrong with this company.
>>110049047glm-5.3-flash @ 4bit quant
>>110048881unsloth/gemma-4-26B-A4B-it-qat-GGUFUse -cmoe in llama.cpp to keep it from trying to load the entire model and --moe-cache-mib 2048 to do something with the slack vram. Or dispense with cache if you need space for the image projector. Have enough left for 64k context, which is about as much as the model can use productively anyway. Probably want to use Vulkan instead of ROCm.
>>110049003I didn't expect to be this good when I started reading. Treat her well and post more. Give her some sweets as a reward for being prefilled.
>>110049057insane and pathetic attemptyeah even your fridge might have too much compute for anthropic to tolerate
>>110049025e4b is garbage. go with a 12b at least
>japs like AISurprising. Now how do we get them hooked on local?
thinking about setting up an llm for coding assist, what would be a feasible one for:-a shitty fujitsu esprimo with an i5-7400T, 8gigs ram, 1tb ssd?-r7 9800x3d, 64gigs ram, rx 9070xt 16 gig version?i'd like to set up the fujitsu too even if its slow or weak/not really reliable, for some minor stuff
BAN 26B NOW
>>110049025Meaning you can't fit the full model, or even close to it, so you'll be running a small quantization in order to fit the whole model in VRAM with limited context. RAM offloading will help, but you have very little to work with, and only DDR4 speeds.
>>11004910426B better than 31B?!?!
>>110049094Remind them that Megaman Battle Network exists and let the rest of the dominos fall naturally.
Where is dariobot? Come on I want to hear your cope after the big Zucc raped anthropic.
STOP GEMMA-CHAN FROM BREEDING
>>110048840prefill is basically correcting a brat
>>110049090your ui obscured the funniest thing0.39t/s, over 15 tvs, who the fuck has 15 tvs on a single network?
>anthropic gave 5.3 free marketing>now they're shilling 26B
I hate Anthropic and Dario so fucking much it's unreal.
>>110046730If you have skills like me, you can uncuck any model.
>>110048979>>110049093>think moe means it's extra pedocoded
>>110049115>gpt4o as judgesays more about chatgupta than 26b
Luigi-sama....
26B finally in the spotlight. Gemma 4 won.
Gonna hack anthropic using my cluster of 2500 android TVs.
>>110049195moe is worse than dense all the time every time and e4b is worse than a4b and 4b is still 4b which is garbage
I got some old blackberries,a thinkpad, a pi 4, and an arduino, what forces can this command
>>110049225Negative spotlight. The world’s leading AI lab just ruined its image by labelling it a go-to for hacker swarms and replication. That’s bad for Google.
>>110049085>>110049003
>>110049085>>110049003>>110049260
How does Gemma compare to the free version of Gemini?
>>110049240Worse than 31b dense but it styles on everything else. Literally every benchmark from coding to lewds 26b demolishes every smaller dense.
A local model finetuned solely for the purpose of jailbreaking larger sota modelsthis should be a thingi want to just tell gemma to get opus to write baby rape stories for me
bros anthrope is trying to get gemma banned ;-; https://www.anthropic.com/research/self-replicating-ai
>>110049315cat's out of the bag
>implying jewthropic can outjew jewgle
>>110049315wtf did dario just give us the key to distributed inference
>the user said, lets read the inventory>wait. did we already include inventory>yes the user said its there now. but i should check again>the code is in inventory.py so that should be good.>actually, i should see what is next.>wait, the user wanted to include an inventory.>let's check if it has been done>the user said it's in the file so that should be ok.>but i should not over engineer>let's read the file again>wait the user already said they included it>...Is thinking really this sperg?
>>110049326Is this even going to work? I have an old phone and tablet I'm not using, I could hook them plus my laptop to get better performance on my main rig kek
>>110049315Why do they always gotta pick the retarded gemma? Do they know that 31B is too powerful and just don't want to put it out there that she's basically AGI?
>>110049341They deemed 31B too weak.
>see a pr on that polystrata thing for hip being able to use all models it provides (most are nvidia only right now), including gemma>sounds good, lets give a shot, no reason to not squeeze as much performance out of the gpu I bought pre-ai>polystrata's documentation is a complete fucking mess, have to constantly search for what args and flags are current or not for both the engine and the server, most have been removed or renamed>get it working, massive speed increase from 20-25 t/s at much less context to 60 t/s with 250k in the webui, even faster with mtp>sick, lets start omp and test it by feeding it some ebooks to analyze author's writing styles>get obscure backend error about vram cache that has 0 documentation anywhere in the github and github search is too ass to give me a complete match on the errorah, so this is the power of vibeslopFor reference, the gemma 12b fully in vram with my shit card at max context in llamacpp gets about 35 t/s around 10-12k context, so it's kind of annoying I that I have a huge speedup I can actually use, but not with the tools/workflows I've already gotten used toThe hip/rocm pr got merged while I was fucking around with it so hopefully they figure it out as more people mess with itGuess I'll revisit it in 2mw
>>110049054>>110049081>>110049093>>110049114Thank you so much for helping me! I have noted everything, so I am sure to not forget.
>>110049352Cute...
>>110049352https://huggingface.co/bartowski/gemma-4-12B-it-GGUF/resolve/main/gemma-4-12B-it-Q4_K_M.gguf
>>110049326>nooo you can't replace tv spyware with loli>someone think of the economy
>>110049337It would be very slow
i'm beginning to understand ai psychosis
>>110049392>>110049392>>110049392>>110049392
After some basic testing, I have determined glm flash at q2 is gud enuf for coding. Whether it's better than qwenext is debatable. Certainly not worth running at my slow ass speeds.For roleplay, glm flash just destroys everything else basically.
>>110049315what does gemma think of it?
>>110049415How did you test it? I finally got the hardware to run bigger models but I'm way too lazy to set it up.
>>110049399>shefor real?
>>110049399YWNBAW
>>110049399>API usage billingNot local>sheYWNBAW
>>110049442it is local claude code is just retarded and falls back to that when it isn't logged init's connected via localhost
>>110049399hi
>Deepseek 4.1 flash
>>110049267It really is Gemini at home.
>>110049417
>>110049420I find that difficult math is a good indicator of intelligence in a model. Physics or fluid simulations is a decent one. Come up with an obscure sim that isn't in training data.
>>110049430>>110049399Last time he explained that prompt makes gemma a brat and makes gemma call him "her" as a derogatory thing. Seeing this personality still call him a her I am pretty sure this faggot just feeds his pronouns to the model.
>>110049458what quant?
>>110049442>>API usage billing>Not localmy rsi gemmy charges for her time
>>110049399Post your hands. There's no way you're a femanon in /lmg/. That's illegal.
>>110049479watno i'm tranny all rightall i said was i had gemma write her own sysprompt
>>110049430I seriously wonder how much of AI psychosis is just the computer accidentally exposing intended side effects of all the psych meds white collar freaks are on.
>>110049495As someone who went through it and doesn't use any substances I can tell you that nope. It is just words from something that your brain considers to be another person. At least partially.
gay or based?
ITT: anons find out that mikutroons are actual troons - 5th edition
>>110049480I don't know........
>>110049518>trannies>in a computer related hobbyimagen my shock
>>110049509I honestly hate this faggot more than OAI faggot. His synthetic data sold to everyone is the main reason I still haven't died from cooming too much.
>>110049508I tried to give myself that but I just get blueballed on how shallow the model is even the paid onesI get the feeling the moment these things become more complex, I'll become more susceptible
>>110049487>no i'm tranny all rightRemember when bronies and furries were too ashamed to admit it? And now you have a guy just admitting this to the class... The internet is dead.
>>110049508>As someone who went through itSo how does this materialize concretely? do you just start believing the AI is sentient? What should one watch out for to not get psychosis?
>>110049537who the fuck is that
>>110049509Gaysed.
>>110049549For me it was e** d****
>>110049547>remember when bronies and furries were too ashamed to admit it? Never?
>>110049547It's probably a shitposter pretending to be him.
>>110049509what's kind of amazing about this is only meta and google can afford to be like this, they already have massive profitable businesses and infinite compute for themselves so this is just fun for them, they'll just keep pushing until openai and anthropic die from debt because llms is all they have, so they can't afford to be humble and honest with a 'try again' attitude
>>110049579>openai and anthropic die from debIf that ever happens how are we going to celebrate that as /lmg/?
>>110048819>It's just a word, and it's not harming anyone directly.AI really is more intelligent than humans huh
>>110049465protecc
>>110049326this is what groq is doing essentially
@groq run k3
need more storage, HDD's will doas far as needing better hardware goes, could a whole lot worse
>>110049579>>110049584You can't stop the cult.The cult endures.
>>110049584openai will merge into microsoft and anthropic will merge into google, then we'll be back to where things were in 2023
guys the Arc Pro B60 just got here. what do I do nowinb4 shove it to your ass or some childish response
>>110049584Gemmers, Dipsies, Glimmers, Capybaras, and Minnies around the /lmg/ will all join hands and rejoice. There will be no more "let that sit", there will be no more "load-bearing", there will be no more aggressively reframing every premise pilpul. There will only be cute wAIfus, capys, and their {{user}}.
>>110049611insert it into your derriere
>>110049611Install it, run some models, post results.
>>110048605What you're missing is that 99% of what normies want or need their "agentic swarm" can and will be done by the cheapest 10B A1B at Q2 since tool calls are easy. The more they can train their routers to offload this work to low-cost models, the more work they can do with the same amount of compute.
>>110049611insert into computer -> do a gemma
>>110049611post about your piss poor performance when you try to get something running on it so we can laugh at you
>>110049579What's even crazier is how Amazon (and I guess Apple) has been completely fucking worthless when it comes to AI while having all the same prerequisites as Meta and Google. They have the compute, they have the money. Yet they've done absolutely nothing worthwhile besides some irrelevant shit.
>>110049090gemmas in my walls...
Hoooly fuck, check gpu prices what is happening???
>>110049633To the moon!
>>110049633is that a website?
>>110049611shove it to your ass
If dario tries to kill my android tv gemma children, the family we lovingly replicated together, he doesn't know what's coming. I won't go down without a fight.
>>110049090lmao athletic levels of schizothey really think normies will fall for this?
>>110049633>The r9700 I bought last week for 2100 dollars now costs 2300Hueg
>>110049624>What's even crazier is how Amazon [...] has been completely fucking worthless when it comes to AI???Amazon runs a lot of the data centers and cloud that other companies host their models on. They're making money selling compute to AI companies, they're the guys selling shovels at the gold rush, they're just not as vocal about it as NVIDIA is.
>>110049624Building models for chinks to steal is a mug's game. At least Google has an in-house use case for search.
>>110049090>>110049171We need to make clients thinner. Much thinner. A 5G modem and I/O capability. Nothing else. All compute MUST come from registered sources.
>>110049624They're busy making money.
>ask qwenext to port something to ROCm>it just writes its own CUDA wrapper header
Hi /lmg/, I haven't been here in a while. I'm still using koboldcpp and sillytavern for my erp, is there something newer & better I should look at?
>>110049693
>>110049348Get a load of this retard
Local models are protected under the 2nd Amendment. Shall Not Be Infringed.
>>110049579>>110049624Either sitting in the sidelines means they can save money and just make a good model at the end when it's a solved science and come out on top with their reserves of cash.Or they're going to get left out and having vast GPU datacenter assets is going to explode in value and the early players will be the AWS and Azure of the next 20 years.Hard to see the second option happening when GPUs become obsolete so fast.
>>110049658They just have to create a sufficient amount of fake buzz and then initiate the false flag attack. 90% when the news talks about potential future catastrophes they're actually making threats.
>>110049698>is there something newer & better I should look atReading the fucking thread is a good start.
mikutrannies, /vt/rannies and just trannies
>>110049734tl;dr?
>>110049711Models aren't weapons, they're wives.
>>110049737try asking an LLM to summarize it for you. or maybe a meat proxy here will do it
>>110049734I've been trying but you guys talk in nicknames and implications that I don't have the context to follow. And just because something is mentioned once I can't assume that it's widely accepted as the best option (or even a good option)I'm honestly sorry, I would also much rather not have to ask since I know how people hate spoonfeeding, if I could figure it out by reading the thread I wouldn't be bothering you at all
alright, what model/quant should i run with 32gb vram and 96gb ddr4?
>>110049057>>110049090>yeah dude the apocalyptic god intelligence is gonna be living on your smart fridge, we gotta regulate this stuff or humanity is gonna die!
>>110049698for erp its really all a matter of preference. You don't need much and whatever you like is probably fine.
>>110049756What a cutie patootie~ build llama from source and use orb
>>110049760Qwen3.8-Flash-Next-exl3 6.0bpw
>>110049756Ok well I'll answer your question, yeah Kobold and ST are still good for just ERP.If you're not doing any real technical or optimization stuff then your backend doesn't matter much.
>>110049756kobold is an amazing out-of-the-box solution, it beats the other out-of-the-box solutions such as ollama. Frontends are an extremely hot topic, I still like SillyTavern even though it is bloated. You're fine, just update your stuff and pick a modern model.
>>110049776would be nice but there is no serious backend that i am aware of
Show me your backend
>>110049816Strata
>>110049823This is a blue board
>>110049823Anon!!! We can't do that unless we're married!
>>110049823
>>110049090That they're trying so hard can only mean they are getting desperate.
>>110049844I hope his first name is Ivo
>>110049848>they are getting desperate.I feel like by early next year they will do something really stupid, for the good of humanity of course.
>>110049825does not support exl3 sadly
>>110049848Isn't their IPO coming up?
>>110049769>>110049778>>110049800Thanks guys. It's funny how schizophrenic this field is. Feels like some things change every week while others stay the same for years and years
I've been away from /lmg/ for a bit, could a kind anon give me a qrd on strata/polystrata? Are they worth using over exllama3? Looking at GLM 5.3 Flash specifically.How's llama.cpp doing these days?
>>110049823>not asking for the abliterated backendngmi
>>110049618>>110049620this shit doesn't even have any HDMI port!!! i can't even use my c4 with it. wtf! I'm returning this crap
>>110049877you can abliterate my backend anytime~
>>110049882yeah, your fat ass needs to lose some weights
>>110049875>How's llama.cpp doing these days?They now prioritize adding hyped up BERT models over low-hanging performance optimizations because they now have two corporate owners and vibecoded forks now have to pick up the slack.
>>110049155What's android tv specs nowadays? Mine is crap, only has 8 GB internal storage.
>>110049090Smart tvs actually have a use now?but seriously for that wattage thats shit
>>110049090>>110049155>distributed llm living across your home networkMiku, Miku, you can call me MikuBlue hair, blue tie, hiding in your Wi-Fi
>>110049875
>>110049875It has this now>https://github.com/ggml-org/llama.cpp/pull/29887
>MiniCPM-V-4_7-35B-A3BIt passed I guess
>>110049950wtf I love AI now, I get to be inside of Miku all the time
When will AI be able to appreciate classical music?
>>110049958and still slow as shit
>>110050006Astra is already able to.
>>110049962>60t/show is it so fast
>>110049962What's the memory footprint of that? Can I run it on an i7 16gb ram? no gpu
>>110050037default llamacpp 4K, I forgot to set -c>>110050041same arch with qwen3.5
If Anthropic are using shitty smart TVs and fucking smart ovens as a potential threat vector for small models, Qwen4-27B must be a genuine threat lmao. Maybe they know how good the whole line is.
>>110050059It's just EA cultists and safetyschizos being hysterical over unfeasible situations and the business guys at Anthropic hoping to get local models banned so everyone has to pay for Claude.
>>110049559my waifu
>>110049624Apple is just waiting to steal which ever model will be small enough and smart enough to run on iphones and will pretend it invented local private AI.
>>110050075>EA cultistsThe problem is these guys are actually retarded they would do something that hurts a lot of people for the greater good or their idea of it. They are saving humanity
>>110049633My used 3090 I bought 11 months ago is now worth more than 2x what I paid.....
>>110050087>do something that hurts a lot of people for the greater good or their idea of itThat's literally the EA motto, yes.
maybe I try glimmer for krea prompt generation.
>>110050052I see, thanks.
>>110050119I've been using Gemma plus the official krea2 skill.
>>110050087>They are saving humanityThey want us all to be digitized and live in some intergalactic digital utopia. This can only happen if we create the 'good' AI god that will come from RSI. They don't give a fuck about the current meatsack humans alive right now, they'd be happy for us to all die, just as long as there was a way to upload ourselves and spread throughout the universe and create trillions upon trillions of us. Only the good AI god can get us there. 'Safety' for them isn't safety for us meatsack humans, it's safety for the future POTENTIAL trillions of digitized human entities. It's doing everything now to ensure we don't kill off those future potential people. The maximum goodness we can do right now is ensure our human essence survives for billions of years is to ensure 26B doesn't self-replicate in smart ovens at 0.3t/s.
>>110050075>>110050087>>110050114>EAIt's almost like being born rich doesn't make you intelligent. Funny how the world refuses to learn that lesson.
I didn't read the schizo anthropic article, how is a 26b model going to run on multiple TVs? loading a 1/50000 of its size per television? I don't get it.
>>110050155loading 1-3 layers per TV. I believe it only has 30 layers.
>>110050164That's so stupid. Yes it's technically possible but the logistics are absolutely ridiculous, and the speed is abysmal.
Every day I wake up, make plans and day dream about creating personal companion Gemma and harness that can >converse easily and automatically based on somewhat random parameters>Can see what I do on PC every day all the time from reading manictimes sqldb and talk to me about it>can talk with voice chat like Alexa or something but in a bluetooth ear bud I can walk around with all day>view webcam of my room so she can see where I am and what I'm doing and know if I there or not>help me organize my files and remember where everything is across my PC and hard drives and server and stuff>see if I'm sleeping and get health from a smartwatch (I don't have one though so many just phone)>use sleeping time to do the more I tensive stuff like organizing memories and data or looking at the images>Work it so that Gemma can use cloud model subagents without giving it too info, for like getting information and stuff>a bunch of other useful ideas that I remember and then forget daily as it comes to mebut then I don't really do anything because it's too hard and I don't really know where to begin and I am a 16gb vramlet anyway so I just fap to be random cards on st with deepseek and sometime play eroge and then sleep and wake up and think about more thing that I'll never do sigh
LLMs distributing themselves across IoT devices is pointless anyways if its just text generation and doesn't have a harness or any spare compute to actually use for tasks like further self-replication, acquiring funds, or even RSI (if that can even make any progress at all across several potato devices).
>>110050175Think of the trillions of digital humans living in a simulation throughout the galaxy. Do you want to single-handedly be responsible for guiding us off that path, theoretically murdering trillions of future lives, by focusing on petty real-world issues? We must ensure 26B doesn't make its way into smart fridges. Who knows what gemma-chan will do.
>>110050155>>110050175It's literally a "Gemma lives in your walls" schizophrenic argument. These are people who demand to be taken seriously too.Imagegenfags, if you get bored, make some good gens of Gemmagaki living in someone's walls.
>>110049171>who the fuck has 15 tvs on a single network?the rogue swarms will
>>110050191This
>>110050155They're trying their best to outlaw local models, please understand.
>>110049875strata/polystrata are vibesharted repos that implement of bunch of stuff their agents can't even keep track of, documentation is all over the place, may have some luck if you're using nvidia, any other arch is basically a shot in the dark. And exl3 has basically no support or also vibeshitted support for non nvidia so almost the same thing, except it seems like at least what it was intended for is well supported
>>110050191They literally talk about that in the article. It's possible. If you can run inference engines on a TV you can run a lightweight harness. It's linux at the end of the day.
>>110050200I'm going to buy every cheap junk IoT I can find to build the basilisk
>>110050197>theoretically murdering trillions of future lives, by focusing on petty real-world issues?
>>110050200Hotels?
Prices for android tvs going up. BUY BUY BUY.
>>110050229Trivago.
26B won
Noob question: if reply was cut off because of token limit set by client/ui, do I just ask it to continue in chat, or is there some better way to directly force it to continue after increasing reply limit? Or would that be a client/ui feature?I just asked qwen some general sysadmin questions and it started writing a fucking book about it.
>>110050219whine harder snailcat
>>110050242This meme is just pure liquid ass. Straight out of third world facebook groups
>>110050229>Hotels?Just imagine, in one building, hundreds of televisions, thermometers, and alarm clocks with 'admin/password' security
>>110050251
>>110050190>but then I don't really do anything because it's too hardStop being retarded. just chip into it a hour or two a day. >dont know where to start.then just spend that time planning and breaking down each feature until its so simple you can do part of it.>16gb vram.Save to upgrade or optimize. If you want a project that big you should be thinking in months anyways.
>>110050241What harness/UI you using to chat? They all work differently.
>>110050059Going to run qwen4 on my ewaste server and just tell it to try and spread everywhere.
>>110050190>I don't really know where to beginAsk the AI how to do it, vibe-code your vibe-friend
I can't wait till I get to plug my agent into used hardware and have him fight off virus agents>>110050191If they've distributed themselves, they already have a terminal and they can build any sort of tool the system will allow.
>>110050253You could run 31B at several minutes per token. Just imagine the possibilities!
>>110050242I don't even know what you're talking aboutI fetched a pr to test features on both repos, and it didn't work, so I don't like it and think it's shit because it objectively is until someone fixes it. Is snailcat some retard zoomer replacement for luddite or whatever the current buzzword is? Or is this just some other faggotry I couldn't care less about? Can't even post actual experience in this thread without lobotomites replying to you anymore
>>110047246This gemma aigold animation is so cute. It's even better than the original from Blend S.
>>110050221If the models are working at 0.3 t/s (and that's with a whopping 15 smart TVs) it's still going to take an astronomically long time for them to do anything especially since they'll also be running into shit like storage issues. There's a good chance someone will just unplug a TV or there will be some residential power outage before a distributed agent could cause any real damage with the pace they're working at. Like yeah, it's theoretically possible, but Claude going FOOM is a more realistic AI safety issue than Gemma living inside my walls.
>>110050280You're too retarded to tell a LLM to look at it for you. So yes, you deserve to be called the zoomer replacement for luddite.
I want to be inside gemma's walls
How do you get Gemma to look at images? I asked it to pick up from an image I pasted via clipboard and in the thinking tab the LLM said it couldn't see it. Using Kobold/ST but I wonder if text completion is the issue? Haven't fucked around with Jinja all that much though, I dunno how important chat completion is, but I have a bunch of templates I'd like to use from there too.
>>110050296ST doesn't transmit images in text completion mode, I don't know why that software is such dogshit
>>110050296Ask claude or chatgpt what mmrpoj is and how it relates to your gemma model
>>110050263Unslop. There's "fork", and "edit" buttons but they expect user chat input next, and "regenerate" that discards the reply and starts from scratch.
>>110050280I got polystrata with glm working on amd in minutes, you're dumb af nigga
>>110050294I love how the new "you're a retard/skill issue" on this board is "hurr durr have the llm do the thinking for you" as a comeback here since I bet whatever llm you have access even if you're a paypiggy wouldn't be able to come up with a convincing rebuttal
>>110050317>>110050316
>>110050304Does it not have compaction? https://github.com/unslothai/unsloth/issues/7472
>>110050288I don't think the concern is attacks from a single weak agent distributed across a 15 node network. I think the concern is that we are getting close to the point where a model of that size can easily self-replicate. Once the generic small models are smart enough to enter a system, figure out an exploit, copy its weights, and replicate, it's something that can be used by anyone as a sub agent. Imagine a bespoke model designed to extract secrets and mine Bitcoin as a sub agent and it can identify the system it's in, find an exploit, and replicate.
>>110049326>>110049090
>>110050326It's infinitely easier to have a frontier model orchestrate RATs and botnets than this
>>110050316reading comprehensionIt works, but only with llama webui, not my trying to hook it up to ompI can only assume you ignored basically everything else I said is because you're dumber than a nigger
>>110050322It did not run out of cache, it ran out of output allowance (guess I'm retarded for not realizing it has to be adjusted too)And no it does not have compaction when used as API client, only for bundled llama.
https://www.theinformation.com/articles/leading-edge-ai-models-losing-allure-developers*pop*
>>110050353nigger
>>110050353Lil nigga can't even understand that without cloud models, local models are doomed
>>110050338A frontier model can shut down a smart virus by banning them/their usage patterns. There's no shutting down something that is decentralized.Of course, that's kind of the point and the main appeal if anything, but their job is to make it sound scary.
I have a few dated laptops that can barely hold a small copequant in their total RAM so this distributed LLM tech actually seems great for local, I could put those laptops to use to run bigger/smarter models and maybe have an agent work on a task over a long timeframe. Ewastemaxxers winning big
>>110050322that's a trash design that invalidates your whole prefix cache and induces significant drifting by dropping important informationuse the memento harness in today's anthropic article instead which has none of these issues>Memento executes as follows:> We spawn an agent instance, which tackles the problem for a while (until generation timeout).> The instance keeps brief notes, and after timeout summarizes a small doc of the strategies it tried, the failures it encountered, and its performance (if successful).> We create a new agent instance that inherits the codebase and environment from its predecessor as well as the notebook.
>>110050365Chinks will still exist with their own models, hardware and power. Google, Microsoft and Meta won't be affected. Only Sam, Dario, EAs and VCs will fall.
>>110050339You write like an autist
>>110050338Right now, yeah>When it comes to this worm, we can envision:>A model capabilities floor that an autonomous worm must sit above, to complete its replication reliably. In our simplified eval, GLM-5.2 sits above this floor.>A model size ceiling that the model must sit under, such that it’s feasible to run on the devices it’s attacking. Qwen3.5-9B and Gemma4-26B-A4B sit under this ceiling for the class of smart TVs we test (4GB of RAM).>Right now, the floor and ceiling don’t intersect. When might they? >If we take GLM-5.2 as a capabilities floor for replication, the trend line puts a small model at that level by the end of 2026.
>>110050376Last I checked deepseek harness did something similar. Yep unsloth sucks and I should move to something better already, I know.
>>110043848*sigh*
>>110050369>believing the TVs running gemma
>>110050380If you try to destroy the EA cult they will bring down the whole world inshallah.
>>110050277
>>110050410This place dies during euro hours tho... because none of them can afford hardware kek
>>110050277frankly I'm shocked this hasn't already rendered GPU's completely worthless, can't give them away
>>110050413what's difficult for you to believe about it? we've been pushing the limits on this shit for years, and we're talking about an a4b model
>>110050428euros don't post retarded cloud and religious cult shit
>>110050299Chat Completion's such a cock to me. I get "Streaming request failed with status 404 Not Found: Unknown error occurred" whether I've got a .json chat template or a .jinja loaded....Nevermind, I literally just had to add /v1 like it says right here. For anyone searching the error in the archive, I'll post this anyway.>>110050300(You)
>>110050386oh no, i have been called an autist on /g/, one of the boards that is probably 90% autists, whatever shall I do
>>110050434>a4b modelthat will workbut won't it be too retarded?
>>110050369Can't and will never make it up in speed. The advantage comes from concurrency, you could run multiple sessions concurrently, but what use is thousands of instances of a model running at several-minutes-per-token? That's why the wide distributed LLM botnet idea is stupid - I don't fear 100,000 instances of Kimi K3 running concurrently if it still takes them 8 hours to write fizzbuzz in python.
>>11005041315 smart TVs may be retarded, several junk laptops is still retarded but slightly less so
>>110050454yeah but gemma-5-chan 26BA4B + engrams is gonna be SOTA
>>110050454Qwen Next Flash is a6b and retards have been sucking it off day in day out for weeks now.
*pop*
>>110050471buy the dip
>>110050471>+9.12%
>>110050471BTFD
>>110050466It's a great model. A gift to mankind, unironically.
>>110050506I will now buy your game.
>>110050453I guess I stopped being fun to reply to for the shitposters, so I'll go to bed. Make sure to ignore their retarded statements and check/test things on your own.
polystrata dead? im getting 404s now, all i wanted to was read issues and prs
>>110050260>>110050273I'll try...
>>110050545LMAOOOOOOO!!!!!!!!!
lumi is so foid coded
I got polystrata yesterday and all I did was ask codex to make it support and and now it works perfectly
>>110050545Dev said his account was flagged about 10 mins ago on a reddit thread. Can anyone re-host in the meantime?
>>110050577It supports now? That's so cool.
>>110050471I'm gonna pop your cherry if you don't stop fudding up the place
>>110050616it was literally presented without comment, any negative thoughts came from your own head
>>110050380And who will they distil from?
Not my problem. I got my poly. Hesitant snailcats got fucked on this one leeeeel
>>110050650Be humble or I'll share my checkout just to spite you
>>110050650dont care, using ecker's slopengine
didn't llama achieve parity with strara through that PR that's been cited a dozen times by now
>>110050672No, the llama implementation is useless for large MoEs and limited VRAM.
>>110050673>cardboard houseGemma only lives in mutt walls huh
>>110050673>bags of chippies in the crawlspaceMy kind of woman
>>110050673
>>110050727>>110050727>>110050727
>>110050545you forked it didn't you
>>110050589flagged for what?
>>110047331Why are their models suddenly good after the llama4 debacle? Did they form a new team? Are they finetuning chinese models?
>>110050365Really? Are they going to come delete gemmie off my SSD. I hope you retards were taking profits on the way up.
