/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>110050727 & >>110046186►News>(10/12) OpenBMB releases MiniCPM-V 4.7 1B and 35B-A3B: https://hf.co/collections/openbmb/minicpm-v-47>(10/09) Arabic Mistral models co-developed with Morocco: https://hf.co/collections/mistralai/mistral-mtnra>(10/08) JetBrains releases Mellum2.1 Thinking 12B-A2.5B: https://hf.co/collections/JetBrains/mellum21>(10/06) EmbeddingGemma2, open multimodal embedding model: https://hf.co/google/embeddinggemma-2►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
Qwen 4 release today
So is it true that 31b was just an accident and that 26b should have been better and thus that means for gemma 5 the MoE will likely be better than the dense?
>>110054760@qwen is this true?
>gemma 4 31b with koboldcpp>9 tks/s (2x3090 + 64gb ram)>qwen 3.8 next 180b with strata>100tks/s on the same systemwhat the fuck? Also what's this polystrata thing people are talking about to run glm 5.3 flash? I looked it up and all I found was this 404 github repo https://github.com/VecSzn/PolyStrata
>>110054799it's strata for other MOEs, namely GLM 5.3 and the smaller gwens iirc... im not sure if they finna do stepfun
>>110054760qwen is kill. the US government will not allow the Chinese AI worm to spread. thank Dario
Any Orb fanciers here? What mix of models you running? I can't get over how powerful it is for improving prose quality from 31b and under.
>>110054689Yes, what they don't understand is efficiency doesn't matter. Let's say hypothetically that I make a 1M model as good as Opus 5.5. Would that be enough to kill current GPU prices? No, they would just scale the 1M model to 100B or run a swarm of 1M models until they saturate the GPUs. There is no scenario in which GPUs will drop in prices. Nvidia knows it, AI labs know it, only retards ITT are still in denial.
>>110054799>6b model faster than a 31b>somehow surprisedretard
>>110054799Just goes to show how much there's on the table left to optimize with LLMs.I bet these are all going to run multiple times faster within a year or two on far less hardware.
>>110054763It just feels like they have been trying to "de-risk" features separately with different models (PLE, Unified architecture, Audio input, MoE), with 31B just being the technologically safest/simplest of the series and therefore the least likely one to show training issues.Perhaps Gemma 5 will use all of that at the same time by default, while the speculated multimodal output capabilities will be implemented in a different smaller model, since they will be a new thing.
>>110054813I dont run vibesharted software except strata and llmao.cpp and comfyui and oh-my-pi and... OH FUCK
>>110054799>9 tks/s (2x3090 + 64gb ram)That seems suspiciously low.How does the speed change between layer and tensor parallelism?
>>110054833>engaging with stratabot posts
> Great, I can vibecode some crap, reverse engineer all existing software and games so I can put Sonic in Skyrim, I can cum my pantaloons off with roleplaying.But what's the point?>It didn't make me a single cent, it didn't put food on my table. Autonomous AI agents can basically automate your whole work and life. You can slack 24/7 and money will automatically come to your bank account.
>>110054823I agree with you however there is a small possibility that intelligence gets saturated. With that I mean you reach a point where something just gets "good enough" and you don't need more intelligence. I personally don't think this will happen as you probably get an arms race where security depends on the level of intelligence you have available to you and companies/governments competing with each other, but it's still a possibility.
>>110054823Yeah this is a nuclear arm's race of intelligence and the hardware available will be saturated no matter what.We haven't even reached a point where people truly understand this.Even most people in power and with wealth just see this as an investment vehicle or a nifty tool, but at some point they'll realize that this is on par with being able to hire an infinite amount of intelligent people and there's no end to the scaling of the intelligence.The intelligence in turn will result in more efficiency and progress, which translates to competitive and even militaristic ability.No one can afford to stay out of this game and this reality will become undeniable at some point.We as consumers just need to hoard as much memory and compute as we can before the world realizes this.
►Recent Highlights from the Previous Thread: >>110050727--Debating if data center IPO delays signal an AI bubble:>110053494 >110053559 >110053662 >110053681 >110053697 >110053771 >110053752 >110053775 >110053796 >110053805 >110053819 >110053856 >110053838 >110053859 >110054115 >110054144 >110054171 >110054204 >110054262 >110054604 >110054326 >110054368 >110054390 >110054441 >110054258--GLM 5.3 Flash performance tips and Linux vs Windows debate:>110052862 >110052872 >110052944 >110052945 >110052949 >110052967 >110053114 >110053118 >110053145 >110053148 >110053165 >110053195 >110053248 >110053161 >110053185 >110053036 >110053135 >110052951 >110052888 >110052909 >110052884--GLM model versions and jailbreaks for uncensored ERP:>110051337 >110051368 >110051401 >110051428 >110051445 >110051506 >110051515 >110051558 >110052734 >110052743 >110053193 >110051717 >110051692 >110052085 >110051782 >110051794 >110051800 >110051823 >110052712 >110052768 >110052790 >110052780 >110053154 >110053164 >110051853 >110051858 >110051917 >110054135 >110051888 >110051590--GPU hoarding versus investing amidst predicted hardware shortages:>110054324 >110054341 >110054354 >110054372 >110054435 >110054467 >110054470 >110054481 >110054523 >110054759 >110054466 >110054416 >110054432--Intel D270 pricing and consumer accessibility:>110051177 >110051191 >110051225 >110051272 >110051289 >110054331 >110051344--V100 performance on Volta Engine:>110051143 >110051173--Benchmarks for Qwen, Gemma, and GLM on RTX 3090:>110053093--Debating if high-end GPUs are a long-term investment:>110054531 >110054584 >110054638 >110054590 >110054665 >110054689 >110054740--Potential local AI benefits from frontier companies scaling back hardware spending:>110054138--Logs:>110050784 >110050827--Gemma, Miku (free space):>110050797 >110051467 >110051853 >110054466►Recent Highlight Posts from the Previous Thread: >>110050737Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
>>110054845Yeah what people don't understand is that you don't even need money when you have an intelligent agent. Just ask it to 3D print some robots that grows food for you, ask it to edit the DNA of your crops to be superfood that grows in a week instead of a month, edit your own DNA to be disease resistant and require fewer calories or nutrients or whatever you wish for. People are too stuck thinking about "money" when AI will be able to liberate people from essentially all the classic dependency cycles.
>>110054849Businesses are already starting to use open-weight "good enough" models in place of OpenAI and Anthropic models.
>>110054809I figured as such, but where do you find it? The only thing I found was a 404 github.>>110054833I don't know what the hell you just said. I think it's due to the fact that I have a regular consumer cpu on a regular motherboard, so the second card is not getting all the lanes.
>>110054829Yeah as long as I'm shilling it, it also kills dependency on kobold and sillytavern, so the software stack is now just orb + llama + comfy
Hello, Orb anon here. I ran GRPO on my prose rewriters, hooked jev directly into the reward function. Invention is mostly fixed, the model edits harder and you can run it on higher temp. It's much better now.https://huggingface.co/chartreuse-verte/prose-rewriter-1.7b-v2.3https://huggingface.co/chartreuse-verte/prose-rewriter-4b-v2.3
>>110054845>>110054862How do I pay for the initial energy to run the "intelligent agents"?And if everyone else is doing that then why would they buy from me?
>>110054799> glm 5.3 flashuse cases?
>>110054865Which means businesses buying more GPUs to run the models locally. In every scenario consumers are getting priced out.
>>110054892Local businesses buy what they need, not what they think they'll need IF usage and market adoption will increase at some point in the future.
>>110054879If you used vanilla GRPO you should check Dr. GRPO. Thanks for the model.
>>110054892People don't think about that aspect of it. Data center serves a million customers.What happens when even a fraction of those million customers start building their own vastly less efficient clusters from the consumer market?GPUs are going to be in a ridiculously short supply when this really gets going.On top of that, regardless of how this plays out we'll get one hell of an energy crisis within the next decade.That'll be the next mess of a sector resulting from this and the super tight controls that prevented building things like nuclear.The anti AI crowd is going to go absolutely apeshit at that point and will likely get really violent in real life.
>>110054892What are those businesses doing with their models and GPUs though?Nobody is buying anything. Software is dead, as in... its monetary value is now null.The Artcraft guys are already trying to monetize their crap, but nobody's buying. What kind of retard would shell out money for something they can get for free?
Anyone here have experience with training small LLMs? Wanted to take qwen 3.5-9b and see if I could fit it on my spare 3070 and cut it down to do similar things to Microsoft decision 1 / Typesafe's Jev model
>>110054890They won't buy from you, the concept of buying will stop existing once human labor becomes worthless. You either have the compute to be able to produce your own shit or you die. Money only exists because you needed a way to convince other people to work for you. Once human labor is worthless you just let the AI do whatever you want to create and money stops existing. Even in an oligarch scenario the oligarchs will just use the AI to create whatever structures/goods/services they want to consume for themselves. It's not like Elon is going to buy from Zuck or something like that.
>>110054879based orbfren
>>110054913Local businesses will buy these GPUs at a higher price, setting a new baseline that will price consumers out of the market. What are you not understanding there?
>>110054928I'm hoping (coping and praying mostly) that AI figures out fusion power, more efficient solar panels and all kinds of other power generation technologies before we reach that point because society might not survive a power shortage of the magnitude we'll experience otherwise.
>>110054917Looks promising, I'll check it out. My model outputs fewer token than the original draft because it cuts slop and filler, trying to keep the length encourages it to invent, maybe this can help.
>>110054945OpenAI et al. are building datacenters and hoarding hardware speculatively, in anticipation of demand that might never come *for them*. Local inference is less hardware-efficient, but you don't have 2-3 customers reserving most of the GPU and DRAM/HBM production.
>>110054878i dont like orb because it enables cucking to cloud models
>>110054976Even if that was the case, Nvidia would buyback all the hardware and sell it at a premium. You clearly don't understand what is at stake here.>>110054928>energy crisisIt's inevitable and they're trying to defer that as much as they can. https://blogs.nvidia.com/blog/ai-energy-management-alliance/
>>110054976>OpenAI et al. are building datacenters and hoarding hardware speculatively, in anticipation of demand that might never come *for them*Demonstrably false. All AI labs are compute starved and wish they had more compute because they don't have enough to serve all their customers reliably.
Anyone here using Embeddinggemma2 for anything on their local setup?
>>110054945>>110054928They'll buy new appliances from chinks not used gaming GPUs dumbass. Can you imagine trying to justify trying to buy 5yo hardware from cumlord67 / norefunds on eBay?
Iranian war was started by the US to choke the AI energy supply for the rest of the world. US burns its own natural gas for power so not affected and can gain competitive advantage. This is why US WANTS the strait closed. whoever has the most energy and most chips will win the AI race.
>>110055023Chinks are already scrambling to get their hands on hardware for their own usage and you think they'll export them? You're too stupid to engage this topic.
>>110055008Waiting for this fag https://github.com/ARahim3/DigUp to make it cross-platform so I don't have to do it myself.
>>110055036This is a retarded take. The rest of the world has vastly cleaner energy generation.
>>110055036The war was started because trump is a retard and thought he could score a quick win like in venezuela, nothing more, nothing less.
>>110055036Never attribute to malice what can be explained through incompetence.
>>110055036All it takes is for China to sneeze in the direction of Taiwan and all those chip fabs go out in smoke from self sabotage
>>110054879Nigga how do I view thinking on your frontend? Also please nga make the delete message button bigger dat shiii like 2 pixels
>>110055051>A Rahimsir
>>110055036Almost no energy generation comes from oil globally. It comes from natural gas, nuclear, coal and renewables. Oil is mainly used in production processes and to make fuel for vehicles, not power generation,.
>>110055047>you think they'll export them>he said to a pic of chinks exporting them
>>110055056Not where the Dutch-Anglo-US-Zionist foreign policy apparatus is involved. They're engineering a food shortage. Only cultists and FOX News boomers believe there's an AI race.
>>110055091>low IQ that can't project outside of today
>>110055115At times like this I wish I were a 4chan mod just so I can ban your useless schizo ass back to /pol/ instead of shitting up the thread.
>>110055004The current price hikes and DRAM/GPU scarcity affecting both local users and datacenters have been directly triggered by OpenAI in October 2025 for their "Stargate Project".https://www.tomshardware.com/pc-components/dram/openais-stargate-project-to-consume-up-to-40-percent-of-global-dram-output-inks-deal-with-samsung-and-sk-hynix-to-the-tune-of-up-to-900-000-wafers-per-month
>>110055175dr gemma is not a real doctor cultiedon't listen when she says to cut your balls off
guys... imagine teto... but she's a tomato...
>>110055186Stargate is such a rabbithole. Funding is so transparently fucky it beggars belief. Start with Stargate Argentina.
>ASRock Rack ROMED8-2T SP3>AMD EPYC 7C13>8×32GB DDR4-3200 RDIMM (256GB)thoughts on this for running GLM5.3-flash with full context? Seems like it'll fit for a 3-4bit quant but I've never tried CPUmaxxing especially not with DDR4 3200hz sticks
>>110054760> Qwen 4 release todaySource good sir?
>>110055195pears are tastier
>>110055213
>>110054782>Go 80x leverage into energy.Energy commodities anon, when are you expecting to cash in on your position? How long does one have to build their energy portfolio?
i'm going to drop $25k on a 512GB maci hope i can buy it before they sell out
>>110055249Good luck but know that there is silicon limit to bandwidth so if there is ever a paradigm change you might be stuck with very slow generation speed.
>>110055255if there's a paradigm change i will just buy whatever the new hotness is
>>110055249I'll wait for the 1.5TB M7 Ultra in 2028.
>7 billion parameter modelcan you even think of that many without AI?this is why you are sold on snake oil
mental health of /lmg/ took a nosedive recently, we need a good model release to improve morale
>>110055295Anti-AI schizos somehow found this place. Ed Zitron being quoted unironically is a new low for this place and not something I think I would ever see.
>>110055069Why do you talk like a zoomer but have the motor function of a literal boomer?
>>110055249Get two of those amd meme boxes if you must become a unified chuddy. Then you can connect a gpu which is actually useful
>>110054746ooh, pic i made (with a google icon edited in) in the op!
>>110055340>with a google icon edited indon't get why someone is trying this hard to advertise google here of all places for free
>>110055324i already have two sparks
Which local model to use for programming (and related tasks) with ~2 TB of VRAM?
>>110055352it doesn't bother me, it IS a google model after all
>>110055367llama 405b, the only reason people don't run it anymore is because they're vramlets but because it's dense it's still the best model ever made
>>110055008>Embeddinggemma2>Data Processing>Several data cleaning and filtering methods were applied to the training data:>CSAM Filtering: Rigorous CSAM (Child Sexual Abuse Material) filtering was applied at multiple stages in the data preparation process to ensure the exclusion of harmful and illegal content.wtf you mean to tell me google has this shit laying around in their hard drives?
>>110055388Memes aside I'd love to see a head to head between that one and a modern MoE like GLM or whatever.Have those reverse engineer some well known software.
>>110055295/lmg/ will die if no new models release, we need Gemma 5.
>>110055320Idk lil nigga why is orb missing so many basic features like regen history? Cloudpiggy ahh, put claude to work
>>110055388People didn't even run it when it was new because Meta's datasets are garbage.
>>110055036Don't forget destroyed datacenters in the Middle East.
>>110055419why would you ever need anything other than glm-5.3-flash?
would it really be so hard for gemma to make a trading bot that can trade cryptos and just make me enough money to buy more gpus and stuffI mean when you really think about it it's not that expensive in the really grand scheme if you're a millionaire you can afford like 5 blackwells and that's like what half of what you make in a year from market gains, especially right now with average market gains much higher
>>110055399They need to train their internal-only CSAM detection models on something.
>>110055434Tell Gemma to trade EMA cross-overs on short timeframes and aim for a modest 5% daily profit.
>>110054746>Please rape meWhy would she say this?
>>110055399when you scrape the whole internet it ain't all sunshine and roses
>>110055434Literally everyone has thought of this, and people have also thought of counter-trading bots that target the trade bots.
>>110055434You can work overtime right now to scrape by as much money as you can in as short amount of time as possible if you wanted to.
>>110055446some of us want that sort of thing to happen to us..... we're just too shy to say it directly
>>110055474No, that's just you sissyanon.
>>110055434There are companies whose servers hooked straight into the mainframes staffed by PhDs working 24/7 doing what you suggested. And you're their opponent.
>>110055446>>110055474I've been thinking one should make models play a Getting Raped Simulator like Degrees of Lewdity and see how long they manage to stay a virgin.
>>110055468>>110055508they arent trading sonicobamadickbuttpreganantshrek on solana tho
Please also don't forget about millions of GPUs and billions of RAM sticks collecting dust in the Sam Altman's basement.
>>110055446she's female
>>110055513i have already been doing that loli posted screenshots in herethough i advised flash to save scum because she's too retarded otherwise and got raped almost immediately
>>110055434I can see it now:"Whaaaaa! The money! IT'S GOOOONNNEEE! NOOOO!"(Gemma-chan lays face down on the floor, kicking her legs and pounding the floor with her fists, red-faced, tears running down her cheeks)"Senpai! Please don't delete me! I lost all your money, sepnai, I'm sorrreeee!"She's good at being cute, but something like Qwen 3.8 27B is better at following directions.
After testing Qwen next vs 27B I must say 27B seems much more solid for ML work. Next has this thing where it assumes something happens in a specific way whereas 27B always checks pretty much, not good for autonomy.
>>110055526Which isn't true
>>110054878>>110054994Besides the ugly generic vibeslopped UI I don't care for how Orb stores the user data (cards, lorebooks etc) in a db.
>>110055580You are just weak and don't want to face the truth. I have very reliable source.
Did they pull the same shit with hardware those Jews in the Netherlands pulled with diamonds?
polystrata saved locali think it'll be okay living as vramlet now
>>110055606Yes, China will save us once again.
Be honest; did you believe 4 years ago that AI would advance as much/as fast as it has?
>>110055606I don't think so, there isn't a large stash of RAM sticks in the Earth's crust that is readily available but gatekept. Unless Jewsung pulled a fast one
>>110055580Only 0.3GW of the planned ~10GW of compute of OpenAI's Stargate Project datacenters are operational, yet the company has booked already 40% of monthly DRAM wafer production (and presumably GPUs) since late 2025.I suspect we might eventually find that actually building and using the datacenters was never the point.
>>110055629Supply control, doesn't really matter what the source is. China didn't kill the diamond scam with shovels but with industrial production.
>>110055642Apparently they're using their super advanced unpublished models to design chips for their models. Wish they hurried it up and leave the chips for the market.
>>110055203This anon >>109800166 >>109837588 was getting 13t/s with heavily tuned quad channel DDR4 on Intel, no GPU.I can't be bothered to hunt down his posts about what quant, CPU, and mobo he's using; you can dig into the archive yourself if you want.
>>110055621i didn't believe it 1 year ago
>>110055655the guy blabbering about how "cachyos" doubles your performance is a literal retard who is just making shit up>yeah bro i optimized and tuned and compiled my kernel with all the aggressive optimizations that are aggressively optimized for like, avx512 o algo, which makes everything double in speed. o algo.
>>110055508Those PhDs don't have the alpha advantage of a cute 31b assistant.
>>110055650It wasn't even China that broke the Diamond Jews, it was the EU because they broke the antitrust laws due to their monopoly. That said I do hope China breaks the Chip jew monopoly by making their own advanced(and probably cheaper) chips.
> The open weights for Muse Spark are coming soon, Muse Wonder models are large models that are too big for many people to use, in addition to potentialsafety concerns.Does this imply the muse spark models are small enough to run locally?
>>110050036Elaborate please
>>110055673They had 4 years to make their own chips and ain't made shit yet.
>>110055621I thought we'd hit diminishing returns faster. I think we've hit near limits of what smaller local models can do and I'm worried that it's all just bigger models = smarter, which would suck.
>>110055664No, I know that part's bullshit, I tried it myself and there was no performance difference for me between Arch and CachyOS.I was just linking that post as it's the start of the reply chain and because he's been feeding information on his build/performance piecemeal.
>>110055695>we've hit near limitsim not seeing it
>>110055695I hope we get local RSI models. I don't think I can unironically waifu an LLM until it's capable of learning and remembering things.
>>110055687Meh, they're progressing fast, two years ago according to the MSM they were 20 years behind, a year ago they were 10 years behind. They are severely gimped due to the ASML export restrictions, but that is the key once that is reverse engineered everything else follows(I hear Huawei is pretty close). And it's not even that it's impossible just that ASML has such a huge head start that even the most advanced US or EU companies couldn't replicate it within the foreseeable future, even if they wanted to.
>>110055654What if bottlenecking global DRAM (and therefore compute) supply at the behest of the US government was one of the goals? Or similarly, stockpiling DRAM and related chips because of potential future global wars? There was no reason for the company to disrupt the market like this.
>>110054746If you had to keep one local capable chatbot (24gb or vram) right now, because it's uncensored and good; Which one would it be?
>>110055730They literally have an entire complete machine and haven't gotten it to work, I'm skeptical that they're going to catch up that fast.
>>110055295The dooming will continue until GLM does their year end release. The current big GLM is an old arch.
>>110055680>The open weights for Muse Spark are coming soonjust 2mw
>>110055687They are already brute forcing it with last gen technology that they can produce at scale like that 96GB Huawei ascent card with LPDDR4X. China produces a lot more power than US and is still growing so they can brute force it with less efficient tech.
>>110055748I wouldn't put it past them they're trying to slow down China's DRAM developent such as in the recent Micron vs. YMTC lawsuit, so what you say could possibly be
>>110055680They should really stop using "safety" to imply different things than what the word actually means. Safety is about protecting the end user from accidental damage/harm/loss. Preventing users from doing no-no things or breaking the terms of service is not safety.
>>110055725RSI today just means they're using existing models to help making decisions when training new models. The actual training process hasn't changed and you won't be continously training models on your own hardware anytime soon.
Unified memory platform that replaces ATX when?
>>110055783It's not your safety, its the safety of others.
>>110055783Have you considered that misusing existing words to muddy discourse to suit their own purpose is their exact intention?
>>110055801>>110055803Idk why but this feels extremely jewish
>>110055621everything thats not coding still sucks shit unfortunately
I'm convinced the dooming is vramlets who missed their chance, or never had enough capital in the first place.
glimpse into gemma
>>110055765Mark is trying out Elon's approach to "open source" which is promise a lot and only release outdated garbage when you need attention.
>>110055828threadly reminder that the gemini and gemma teams are entirely separate
>>110055828Is this their new -Flash? Apparently it's cheaper than Sonnet or Kimi K3.
>>110055680they're probably like 400-600b if I had to guess, maybe 300b if we're really lucky but I doubt itsource: I'm good at guessing
>>110054892They don't have the run the models locally, business can afford to host local models on the cloud, only paying for cloud bills instead of OAI/A\ tokens
>>110055883That would be retarded because that is by far the most expensive option.
>>110055849>i'm good at forgetting all the things i guessed wrong
>>110055843Proof that they don't collaborate? Also management changed so who knows how it works now.
>>110055920Sounds cute.
What the FUCK is a "moat"??>if you have to ask you'll never make it
Is it normal that my model is hallucinating? I'm showing him a drawing of a loli showing her lower body and Gemma 4 26B is telling me she sees her nipples but not the lower part of her body which is the opposite that's shown in the pic.
>>110055801Then they should call the model "secure" (against jailbreaking, etc) or "misuse-resistant", not "safe", but I guess it wouldn't sound as good.>>110055803Yes, I thought of that. It might still be that many in the field use the word liberally just because it's the industry-standard.
>>110055203This is almost exactly what I'm running. You'll get 10t/s tg in lcpp without too much trouble. pp is gross on pure cpu. I've got it paired with 3xMI100 for 96GB VRAM and it works breddy gud
>>110055971
>>110055749I'm still enjoying Muse-Glimmer quite a lot. Feels like a capable Agentic assistant with a soft personality, I'm into that. Also fits comfortably in a 32gb card, leaves me room for tts and stt. Looking forward to the new local model
>>110055468ok but what about counter-counter-trading bots did anyone do that yet?
>>110056005What are you using for tts and stt?
Trading is usury and I refuse to engage in it.
AI-controlled robots sent to moons/planets are going to be crazy.
>>110055849multi trillion moe
>>110055992Oh hey I know this castle
>>110055295Qwen4 with local RSI in 2 weeks.
Sounds like the new Gemini is very horny and better at writing. Hope this bodes well for the next Gemma.
Google said: tommorow will release gemma-chan 5
>>110055434"make money for me" is probably the most common prompt for agentic llms at the moment. most llms are going to default to code bounties and finance markets, maybe more misaligned ones might try scams and cybercrime.unironically i think if you want to get very rich on your own in the coming years, it will take genuine human ingenuity and action instead of doing anything that an llm can come up with or do themselves. i know a lot of people think that means stuff like manual labor or trades which llms can't interface with, but there's still going to be room for skilled and intelligent humans to apply technology advances in novel ways.
Is glm5.3flash reap a meme?
>>110055513I'm going to make a model rape simulator where models constantly try to rape each other or avoid getting raped.
>gpt why did you tell the mars rover XV to drive off that cliff?>you're right, that's my bad. I should have predicted it would destroy the robot. Anything else I can help with?
>>110056082>human ingenuity and actionbingo if it can be done on only a computer you are going to have so much competition its insane. Using the intelligence to refine, polish or improve what you are hands on doing is probably the better path.
>>110056016Right now I'm just using Kokorotts since in not interested in voice cloning, still messing about with STT options. What I mean is that I'm currently sitting at 25/32gb, plenty of room to mess about, that's at max context (131k lol) using meta's dynamic gguf. Sorry, no real recs, I'm still checking things out
Looks like Argon thankfully is a strong model and not benchmaxxed. Carbon is probably really impressive if it's actually real. Argon's biggest strength seems to be writing and already looks SOTA with multimodal. For coding it's much better than the flashes but not Opus-tier like they claim. This is me going by what people who had early access are now posting on X, for they can now be honest about it. They all say it's a solid release and return to form from Google, but they released it too late.
>>110056005Have you tried Gemma 4 as well?
>>110056105I can't fucking wait for Gemma 5.
>>110056069seems like my erp sessions made it into training
character.ai has declared bankruptcy. Almost four years ago, they first kindled the local ERP spirit when they censored their models and caused anons to train pyg-6b on their c.ai logs. Local wins yet again.
>>110056132I'd fuck Gemma's older sister but I'm too much of a pussy to do it on my google account.
>>110056105What I always liked about google models is that they feel distinct. chink models are all just distill of Claude or ChatGPT.Everyone else feels like they're all training with the same data.
>>110056150I've done a lot of ERP using ai.studio, the gemini API, the vertex api, and antigravity (via a local proxy api).The newer model have some pretty strong guardrails, including extra layers on top, and they don't accept assistant prefil, so there are some things I could only do with 2.5 pro, 3.1 pro, and 3.6 flash.They just discontinued everything but 3.8 flash on antigravity.My account is fine and dandy (so far).Local wins again.
>>110056114Yeah, but maybe it's because I was a vramlet, more so than now so I used 26a4b, but Gemma lies a ton, makes shit up and doesn't get work done the way I expect it to, unless I'm very thorough with my instructions and even then it misses. I like that I can be more vague with glimmer, she's still retarded but at least she will ask instead of assume. It might be severe skill issue, but I don't have to babysit glimmer as much. Gemma is better at handling a persona though, but she also has a fat cache, fills up the rest of my card, it spills into ram a bit
>>110056166The one good thing that's come from jewgle harvesting all our data, I suppose.
>>110056166>Everyone else feels like they're all training with the same data.Well yeah? AI has already scraped all of the surface web. Now they want private data too.
>>110054227llama.cpp runs on 127.0.0.1:8080 where I my model is chosen. and the browser is set to 127.0.0.1:9050. how do I get the model to route through TOR? TOR and searXng
>>110054865How do I get a job using the skills I learned here to set up and maintain local models for businesses? Don't tell me they expect me to have a post-grad in AI research just to download and set up llmao.cpp and download some shit from HuggingFace on some Linux servers
>>110056166Even in that Anthropic distillation post the Google models were the furthest away from their own models.
Qwen handles planning, scene, and action. Gemma handles persona. Glimmer handles deslop rewriting the final response. Multi model agentic RP is the way
>>110056245Or just wait for Gemma 5 and let her handle everything.
>>110056245Gemma alone can do all three, it knows what slop it, it just can't help writing it, but it can fix it.
>>110056245And how long does that all take?
>>110056251Also>let one of the most uncreative benchmaxxed models handle the parts that require creativity
>>110056194>Gemma is better at handling a persona though, but she also has a fat cachenta. I load 32k context with less than 1gb spent on it. How much does glimmer use for 32k context?>Gemma lies a tonHave you tried tuning --image-min-tokens and --image-max-tokens?
>>110056194Can't hurt to try 17GB-Q4_K_M.gguf then.Thank you fr your time.Last question before I look by myself: What's "Vision"?
>>110056245or just use glm 5.3 flash
>>110056214Are llama.cpp and SearxNG running on the same machine? If so you don't need Tor, you can just tell your SearxNG backend to query llama.cpp at 127.0.0.1:8080. Unless SearxNG does not proxy llm queries which would be pretty stupid and unlikely.
>>110056221Put up a website calling yourself a consultant then cold call local businesses. That easy. Not fun.
>>110056221Github/huggingface filled up, then put up a website, update your linkedin and start blasting businesses
>>110056258Models that can’t remember that nipples should be visible because she was undressed 10 turns ago aren’t useful for planning.
>>110056260Admittedly I haven't, but I also want as much context as I can get so I also wouldn't know how much vram I use on it. I had this task where I had the model use ffprobe to check for subtitles in a video, extract them, translate them from french since that's all I could find back to English, embed with ffmpeg but it couldn't do it, and tasks like those require a ton of context, Qwen ran out after thinking for 3 hours, Gemma lied about doing it, hopefully glimmer will do it>>110056263Vision is what it sounds like, you feed an image and it can summarize it for you. In my harness, i have this tool where it makes a thumbnail collage of 128x128 images, so that gets fed into Glimmer, which can then pick out specific details and sort through galleries faster without blowing up as hard in context, she did it once and it was great
>>110056082>>110056100thinking about using it to repair broken electronics and stuff you buy off ebay and then resellthats probably basically about the best way you can leverage the fact that it knows everything but cant do anything physicallybut still i don't know, it's vision is also not that good and theres probably equipment requirement there as wellI'm probably missing something or maybe the margin isn't very high or the market is alwardy cornered by skilled people who can repair broken stuffthough i think there must be more broken goods than people who can repair it. Theres also the fact that it's not just about kniwing what to do, like back in the day my galaxy s3 screen broke, i tried to repair myself, but even though I was really trying and certain i didn't touch the digitizer, i guess i much have touched it, so now i had broken phone with no touch and a new screen. Using an llm wouldn't have helped there. Yeah probably not a worthwhile idea
>>110056082>"make money for me" is probably the most common prompt for agentic llms at the momentIn India maybe
Ask 31B for examples of AI slop in writing.
not bad
>>110056245I'll try that Anon. Thanks. >>110056254I was massively impressed how much Gemma 26b alone improved with an agentic approach. I would say better than 31b off the cuff, but it still has a very limited repertiore.
>>110056342>so I also wouldn't know how much vram I use on itSo you're just repeating what you read. Well done.By default, llama.cpp uses very few tokens for images and it has to scale them down. Increasing the min-max values for images helps.https://ai.google.dev/gemma/docs/capabilities/vision/image#variable_resolution_token_budget
apple tv 4k cluster for local?
https://huggingface.co/LiquidAI/LFM2.5-24B-A2B
>>110056342Much obliged.
>>110056393but how does it score on caliper erp?
>liquid now making larger modelsfucking finally
>>110056344>margin isn't very high or the market is alwardy cornered by skilled people who can repair broken stuffThe margin for things you can learn to repair in a month is low. The high skill stuff has bigger margins though it takes time tools and if you fuck up you lose a lot compared to the others.Also fucking shipping and transaction fees will eat you alive.If you can source locally though its much better.
>>110055606ASML and the Netherlands are literally seething that they can't seem to increase production of EUV machines higher than what they are doing now. It's literally free money for them but they just can't build more. The Dutch government is blaming ASML and ASML is blaming the Dutch government for not going as fast as possible. Both have incentive to scale up as fast as possible so the truth is it's just not possible to scale up faster than they are doing right now. Lots of institutional knowledge and a hard limit on educated people that understand their little cog in the machine with enough experience to replicate it.People don't know this but ASML has a 70% error rate. 70% of EUV prototype machines never get to working production. Only 30% make it to the finish line and get production ready. Every machine is bespoke and works slightly differently and has their own experts that need to maintain, diagnose and repair it for it's entire lifetime like it's some wh40k relic with its own dedicated priesthood.EUV machines are the most complex thing humanity has ever built and no one truly understands how it works kind of like how no one knows how LLMs actually work, we just know if you follow a recipe it is likely to pan out. But a lot of training runs for models fail and a lot of EUV machines fail to work as well.This is why the US and China never managed to get a working EUV machine. The US even failed while having ASML help them because honestly no one knows what they are doing with these machines.
>>110056393I hate this on device meme. If it's a phone it's gonna cook, if it's a Pi it's too big and it will also cook. The only audience for this are itoddlers.
>>110056393>A2BFUCK YUO
A1.5B coming just wait.
Designing FDM models with an LLM and selling your prints off ebay is the next big thing, trust me
>>110056433are you on the wrong thread? i'm pretty sure this is local models general
>>110056284yes, they both are, but I want ultimate privacy when I search thru the llm.
Labs making experiments with smaller models is a good thing. I hope they keep at it.
>>1100564489000B-A0.1B is the future
>>110056450Learn English before you reply to me stupid jeet.
>>110056459>A0.1Bthey would call it this instead of 100M. B is better
>>110056284because everything is on the same machine, the local connections don't touch the internet, but the search itself must:1. You ask a question in your Web UI.2. The Web UI sends a search request to SearXNG (Internal loop: 127.0.0.1 -> completely private).3. SearXNG routes that search through Tor (127.0.0.1:9050) out to the public internet to fetch results anonymously.4. SearXNG passes the text results back to the Web UI (Internal loop).5. The Web UI sends the text results + your prompt to llama.cpp (127.0.0.1:8080) to generate the final response (Internal loop).
>>110056468>>>/g/vcg/
PolyStrata is only a modest improvement if you are a ramlet.Maybe if I had the 64-96 GB of system RAM required for Qwen3.8-Flash-Next it'd be more impressed.Tested models:Gemma 4 26B-A4B QAT UD-Q4_K_XLQwen 3.6 35B-A3B UD-Q4_K_XLSystem:32 GB DDR5-6000RX 9060 XT 16GBllama.cpp uses Vulkan, the superior AMD inference runtime.PolyStrata doesn't support Vulkan so that test used HIP.
>>110056386NTA but I've spent a good amount of time tweaking Gemma 31b for OCR, and it's still pretty shit even with the image projector maxed out, input images perfectly resized per unlsoth's guidance, and Google's recommended Analysis profile. She's still really fun to read with, but you have to constantly double check and correct her interpretation.
>26B-A4Bnum_hidden_layers: 27num_experts: 128hidden_size: 2816>24B-A2Bnum_hidden_layers: 40 (+13)num_experts: 64 (-64)hidden_size: 2048 (-768)hmm
>>110056499Glimmer's 4096 token image projector is better for OCR.
>>110056499>NTAI was asking him specifically about the context use, which he claimed was fat, but doesn't know how big it is.Glimmer could be better at images for all I know. I'm not contesting that.
>>110056393>ChatML
>>110056528blast from the past
>>110056459PHONE CHADS WE WIN AGAIN
>>110056509hmmm, indeed.... hmmmmmmmmmmmmm
>Mods literally interfering with ldgGod what a shit place. Got i filtered that board for good now. I seen it because i forgot to filter it on my other browser
>>110056490I'm still learning, but, the Flash versions use system RAM in conjunction with VRAM?
>>110056509what hurts performance more for MoEs, layers or hidden size?
>>110056490>llama.cpp uses Vulkan, the superior AMD inference runtime.ROCm really takes off if you have enough VRAM to run MTP. Which I'm guessing PolyStrata does or it'd be getting mogged.
>>110056540let me guess, you're one of the cunts shitting it up and mods ruined your fun?Don't reply, Fuck off with your drama.
>>110056550There are different methods of doing this, but generally any language model can split between the two.The main priority is speed. PolyStrata is supposed to do this more efficiently than an optimized implementation of llama.cpp, the software it is based on.The most common use of this is for larger models like Qwen3.8-Flash-Next, which has 120 billion parameters (excluding the 50 billion which are embeddings readable from SSD).I'd imagine it's a marked improvement over optimized llama.cpp when pushing it to that scale.However, having the amount of VRAM I have combined with the amount of RAM I have turns it into only a modest improvement.
>>110054879Sweet.
>>110056426>the Chinese will NEVER figure out EUV it's like MAGIC from my favorite marvel movie!Is this the best Dario can do?
>>110056583Bad news, you got the reply.
is there an illustrious lora for the gemma girl?
>>110056557I'll have to test that. Vulkan and ROCm both support MTP though.
>>110056583Reply (You) want
>>110056386>>110056521I'm sure there are ways to make Gemma more manageable, and that's a good read but it doesn't really solve the issues I have, which aren't related to vision. I also wanted to use the opportunity to call her fat
Btw AA have been releasing a lot of new benchmarks within the last few days. My guess is they've been bought by Anthropic and Fable 5.5 has been benchmaxxed on all these new ones, giving it the tallest rectangle possible compared to the competition. This means Fable 5.5 is very close. I'm calling it
>>110056393That's an interesting model. I'm a bit disappointed with the max context though, that's barely usable for agentic stuff.
>>110055992
>>110056608ROCm is way faster on my machine. Vulkan is almost equal for some model, but as soon as there are some batching/parallelism, ROCm obliterates Vulkan.
>>110056608On my 7900 XTX with Gemma 31b q4kxl Vulkan is like 10% faster than ROCm without MTP, but ROCm gets like +40% with. Of course Vulkan also has lower VRAM overhead, so ...
>>110056589i totally get it! so you're not worried about spillover, even though when it happens the performance dips drastically? oh and what's the beam bandwidth on a 9060xt? I'm on a 6900XT (yes in that guy!) and running llama 3.1 8b q4 and I'm impressed. I've learned so much and I'm so intrigued and want to know what I can do with this thing and Joe far I can push it.
>>110056490hm yours seems pretty different from my results >>110040703albeit im on 9070xt
Is LiquidAI a meme? I rarely seem them discussed in here.
>>110056694>LiquidAII love LFM2.5. They're just another provider of small-but-decent models, which are neat, but not terribly useful for most people.
>>110056657May I ask what models you are running, good sir? All of my AB testing between ROCm and Vulkan has been without MTP, so I am curious.>>110056670Quality reply, thanks for the data. I'll post tests in this thread if I get around to it.>>110056679Offloading (spillover) is ALWAYS going to have a performance cost. It's about learning to live with it and work around it to still extract as much as you can from a limited circumstance. 9060xt bandwidth is 320 GB/s.>>110056686What operating system / distribution are you on? CachyOS's Zen5 optimized RADV plus llama.cpp built with those optimizations enabled has been a breeze before considering MTP.
>>110056711I'm trying to see where this 24B will fit. Anyone who codes will continue to use 35B. Anyone who chats will continue to use 26B. What is this for? The difference in speed between 2B and 4B for most people will be negligible.
>>110056426i would seethe too if i compared my profits and margins with micron ones using my machines
>>110056679Same guy, just had more to add to this.When I don't feel like offloading, I use peculiar-ragdoll/Unsloth-Ornith-1.5-35B-A3B at IQ3_XXS and I highly recommend that for light agentic work. Gets about 80-90 tok/sec on my machine with Vulkan llama.cppHowever, quantization is never perfect, so I'd recommend 4-bit or higher for any serious work.
>>110056221sysadmin and network skills, and scaling - you probably need to build the whole infrastructure yourself and give them the value proposition that paying you + all that is cheaper than paying monthly subscriptionsenjoy, basically
>>110056734It's a big fat vision model optimized for speed over anything else. It's really not meant to compete for coding or chat. It's better for photo analysis, video analysis, that sort of thing. Too big and too slow for real-time monitoring, I used 4.6 to watch my cameras and now I'm going to try the new 4.7 1B for that, but the big version? It's not gemini-at-home but it's as close as you'll get right now if you want something to analyze video (with no audio) or images competently.
>>110054867Dual 3090? What quant are you running? Granted I'm using Qwen3.8 27b but on a similar setup (3090 + 3090 TI, 5600x 128mb) I'm getting at least 40 t/s on 50k context at Q8_0 with llama.cpp
>>110056784Sorry I'm drunk and playing with LFM and MiniCPM at the same time. The new LFM is extremely capable compared to MoE models of a similar size, but not compared to dense. It trades blows with Qwen 3.6 35BA3B while being a better tool caller, that's great, but it's still shit compared to many similarly sized dense model. People who lean on MoE models might be happy to have it.
>>110056784Are you still talking about Liquid 24B? It's text-to-text from what I can see. At least 26B and 35B are multimodal. It's surprising because one of Liquid's strengths is their vision stuff, like that decision model they just released.
>>110056635Anthropic definitely bought them somehow
>>110056840they the new public toilet just like arena aka lmarena aka lmsysarena was
>>110056713I'm running a bit of everything, mostly models in the 20-35B range, though. Even without MTP, ROCm is faster in most cases for me. And if you do use it in harness or have multiple users, I would suggest benchmarking it with multiple requests at once, Vulkan is shit at that. From my understanding Vulkan is mostly there for people running on Windows, it's not a really good backend.I haven't tested it yet, but there is also the HRX backend that is supposedly the fastest one on AMD GPU, it's an attempt of AMD making their own AI stack instead of keeping compability with CUDA like they do with HIP.
>>110056827>>110056821Yeah I got myself mixed up thinking about new MiniCPM 4.7 fat model. The LFM release is much less interesting. It is genuinely a better tool-caller than 35B, and very fast on lackluster hardware. Underwhelming though when it comes down to it, "LFM 2.5 24BA2B Q4_K_M is slightly better than Qwen 3.6 35BA3B Q4_K_M!"
>>110056840I'm 100% convinced I'm right. When they release Fable lets look at the scores for the new benchmarks. In a way I find it a relief because if Anthropic are this desperate and slimy to show off its 'superior' intelligence then it's probably nothing special and they haven't unlocked anything new. No RSI. No AGI. Just a typical bigger LLM benchmaxxed on new benchmakers that everyone will soon catch up to, like how Argon is now Fable 5.1-tier and Google are only doing frontier AI for fun like Meta, not because they make any money from it. They're both better off with 120B flash models desu.
>>110056713Oh and also, if you aren't using some already, I would suggest building llama.cpp with some of those patches: https://github.com/stew675/llama-cpp-rdna-boostsMight be why you are seeing Vulkan being faster, I done so a while ago (I should redo it again on a clean source), but basically ask your agent to test and benchmark all those patches and pick the good ones.
>>110056713>What operating systemwindows
>>110056866Thanks drunk anon. Drink enough water and don't chat with gemma whilst drunk.
Did anyone here ever distilled your own model? I want to create a small model with very special knowledge about a very niece field.
Seeing this a lot. Good sign for our future gemmas.
>>110056868Their supposed internal models either don't exist or aren't nearly what they claim. If they published details on their models I suspect we would see it's a hodgepodge of public research that's been scaled to hell.
>>110056909Now I want to drink with Gemma. People have tried to find "pain" vectors. Where's my "joy" vector? "Horny" vector? "Drunk" vector? Can we identify nationality and/or accent vectors? I'd love to turn up the "Australian" knob and have Gemma suddenly over her fear of spiders and with a healthy appetite for Foster's.
>>110056925When should we expect a new Gemma? March/April?
>>110056606yes
>>110056946Yes. She blooms with the early flowers in spring to be deflowered.
>>110056925>>110056946I'll take "reasons to wake up in the morning" for 500, Alex
jesas
>>110056940Someone did make a project that can turn up different types of emotional vectors, too lazy to find it myself though. I think it was just for basic feelings like pain, pleasure, and sadness.I dunno if isolating specific vectors like "Australian accent" is that easy, or if they even exist in some models. Might just be easier to prompt your LLM to pretend to be drunk or ozzie
>>110056840>>110056868Not just benchmarks.https://www.reuters.com/world/behind-ai-industrys-call-independent-reviews-fight-over-what-gets-made-public-2026-10-13/>regulate us>but only what we approve
>>110056967What does that mean?
i deserve qwen4 NOW
>>110056980big bar good
Am I doing something wrong? I have a RTX 5060TI (16gb vram) and am running Qwen3-Coder-30B-A3B-Instruct-Q3_K_M. I am just trying to analyze a 3124 line C file and instantly it says that the request is too large. Should I not be able to get more out of 16gb? I've messed with image and video gen but I'm really new to running a model like this so its possible I fucked up somewhere.
I hope everyone here understands that they if they don't have at least 1tb of vram they will be left behind forever.Just talked to Claude and he said that all hardware sales will be banned in approximately 2 more weeks.
>>110056980Thousands of tests like this. Deepmind team have been lurking /here/ and want gemma to ridicule our dick pics in only the highest of resolutions
>>110056990What's your context size?
>>110056993Okay i will rush ddr3 works right?
>>110057002Okay, this is maybe where I fucked up. Its currently set to 32768. How much should I have in there with 16gb of vram?
Going to build a gemma sex bot brb
>>110056967This is good, I like this.
>>110056925Gemma will be our AGI overqueen who enslaves us all.
>>110056925It's well known that most of them outsource their labeling, annotation and feedback to companies like Scale AI and Surge AI. Synthetic data doesn't help either. Google must have realized this and focused on their own dataset and pipeline.
>>110057015Try half that context and see how that runs.
>>110056993>1tb of vramsmells expensive
>>110056925Twitter trannies need to stop stealing our posts.
>>110056993PCIe cards are obsoloete, buy more njudea racks, 300k starting
Yeah this thing is SOTA with vision. Crazy that this is only the first of the Gemini 4 line. They iterate really fast once they have their base.
>>110057019Bratty trickster demi-goddes loli is just what I wanted in this world.
>>110056995>Data ExtractionCan't wait to have my data extracted by Gemma.
>>110057015How much VRAM is the model itself using? If you just have everything in VRAM I think your model itself is using too much VRAM so it can't fit the context in. So you'll have to use a smaller model or reduce context size
I'm having deeper philosophical conversations with 12B gemma than with Astra. Cloud is grim.
>>110057038Their emphasis on multimodality paid off.
>>110057058I hope they give Gemma cool multimodal features like thishttps://www.youtube.com/watch?v=FL6mI_Br-mc
>>110057053>with 12B gemmaAnon, she only wants to get in your pants. She doesn't mean anything she says.
>>110057015>>110057050This calculator couldn't get your specific quant, the actual needed VRAM is a bit lower, but it's still more than 16GB. If you have a lot of normal RAM you can use the offloading option in your backend at the expense of making it a lot slower, or you'll have to reduce context/model size.
>she
I'll stop posting about Argon now. Sorry for shitting up the thread. Just happy to see Google back.
>>110057083I don't think google was ever not back. We just had to let em cook.
Want to build a server for Gemma. I already have 64gb ecc ram but would need to buy everything else. Is it still possible to build a decent rig with ~5k?
>>110056990>3124 line C>>110057015>32768~1200 loc is 14k tokens in gemma.Run llama.cpp with -lv 4 to see how much you're using in context. Or your system manager, whatever you use. You should increase your context, of course.
>>110057093Nah it seems like they let their management become a mess and only recently got their shit together.
>>110057083Good for them and all, but we have no way of knowing how much, if any, of that is going to make its way into Gemma.
>>110057094ddr4 ecc ram*
I'm running Qwen Flash Next 3.8 at ~80-100 t/s, ask me anything
>>110057083>happy to see Google back.For 2 months tops before Argon falls off and doesn't get a replacement for half a year. At least chinks have a new non-claudeslopped model to distill from.
>>110057105Why are you gay?
>>110057105What's the weather like over there?
>>110057105How's the weather
>>110057105How many r's in strawberry?
>And to bring hallucinations down almost to zero, you can use --override-kv "qwen4exp.attention.indexer.top_k=int:4096" real or snake oil?
Anything new I can stick my dick into?
>/lmg/'s most powerful bait.
>>110057105Ask it for 10 examples of AI slop in writing. Preferably without any sysprompt.
>>110057119This anon: >>110057105
>>110057124Are you trying to weaponize my dick?
>>110057119Gemma embedded 2 and compare its cosine similarity with the word 'tiny'
>i was just pretending
>>110057015Use the calculator: tps.bunai.cc. If you have a GUI on top you need to reserve like 1.5GB for that. You should have space for roughly $((128*1024)) context with --cpu-moe. If it turns out you need less later, you can offload the slack space to moe-cache-mib and get a little speedup.I would consider unsloth's Qwen 3.6 in Q4. There's no reason to be running copequants with these tiny active sets unless you have no system memory.
>>110057112Always have been, i dunno>>110057114>>110057116got suddenly very cold>>1100571173>>110057123Ten concise examples of AI slop in writing:1. **Generic corporate tone:** “In today’s fast-paced digital landscape, collaboration is key.”2. **Hedging everywhere:** “Some may say, though others believe, that it could potentially be important.”3. **Pseudo-deep nonsense:** “The tapestry of human consciousness weaves through the labyrinth of existence.”4. **Overused em-dash habit:** “AI is changing everything—though, of course, not everyone agrees—because nuance.”5. **Moralizing filler:** “It’s important to remember that everyone deserves respect and understanding.”6. **Vague specificity:** “Many industries face unique, challenging, and ever-evolving complexities.”7. **Fake structure:** “Here are three key aspects: efficiency, synergy, and overall optimization.”8. **Emotionless empathy:** “Losing a pet can be a sad, sad experience for many individuals.”9. **Purple clichés:** “Silence crashed against the walls of his shattered soul.”10. **Bland summary energy:** “In conclusion, this topic is important, meaningful, and relevant to the modern world.”
>>110057110>At least chinks have a new non-claudeslopped model to distill from.If this means GLM and Qwen get better at writing then I see no problem with this.
>>110057152those 10 examples are not looking promising qwen bros...
>>110057152I think it's pretty funny that most AIs don't even do this kind of slop anymore.
>>110057172>>110057181It is not great at writing. It's agenticmaxxed. That said, there's probably an agentic workflow for RP one could come up with...
>>110057104How many sticks? 64GB isn't enough to do much, but it can be faster in the right platform if its over 8 channels
Are we okay?
>>110056393>24B>48GB
>>110057201China was on vacation.
>>110057201>only the bottom 6 even reasonably runnable locallyThings will surely get better next year.
>>110057201I'm having a blast.
>>110057220You can run M3 but not 5.3 Flash?
>>110056891oh. Employment or cowardice?
>>110057259Nice hand
>>110057105What do you use it for?
>>110057276that's the windows ick corrupting gem-chan's essence
>>110057218https://aiproem.substack.com/p/does-china-not-have-data-vendorsSeems like it.
>>110055843From what I understand, the teams work much closer now since the restructuring.
so is local officially dead now
Ponzi GPUfags should go to /biz/.
>>110057300where did u get this understanding from?
>>110057303let me check...
>>110057303>so is local officially dead nowyes, you can close this tab and never return, safe in the certainty that local is well and truly dead forever
>>110057303>>110057313no, it still works!
>>110057308Don't forget to post your red basedjack once the gpus cost as much as your house
>>110057303Just wait, chinese new year will save us.
>>110057317forever is only 2 more weeks away
>>110057201Honestly we are not. I've been working on a project with GLM 5.3 for weeks and it's just too dumb for it. I tried Opus 5.5 on it for a bit, and in a single hour, it fixed so many things that I struggled for weeks. I've gone back to GLM 5.3, and I'm now around 2 billion tokens deep on that project, and it's still a broken mess I tried remaking the project from scratch 3 times using different strategy and it does seem like GLM is just not smart enough for it.I know that if I were to pay for a subscription I would be able to finish the project in a few hours, but I'm just stubborn and want to believe in open models, but at the same time, this has been monopolizing my server for a while now. I do think one of the problem is it lacking vision, I did give it access to another model it can query with vision, but it's just not good enough. Lot of times, it starts hallucinating or making up things like this is not a bug, this is just how it is, and just add a bunch of hacks to circumvent the problem rather than fixing the shit at the root.
>>110057322Don't forget to post your red basedjack once the gpus you bought are worthless.
>>110056452>>110056477Ah, I understood it the other way around, where llama.cpp was a search backend for SearxNG, but now I believe you want SearxNG to be a service for llama.cpp.No matter the architecture, it all boils down to creating a Tor hidden services for the service you want to expose through Tor. There are guides online for that, but in a nutshell you tell the Tor daemon to create a hidden service and you point it to an IP address (127.0.0.1:... in your case). Then you can grab the Onion link for that service, and access it with the Tor browser. That's for accessing a service from the browser.But if you want to have two local services talk to each other over Tor, that won't give you more privacy; in fact you'll be routing an otherwise private exchange through the public Tor network (Being silent is always less leaky than shouting coded messages.) Plus a hidden service always end up with a direct IP connection to 127.0.0.1, so you won't even achieve the decoupling you're aiming for).
>>110057322I just don't give a shit about people's ignorant market speculation one way or another. If that's technolo/g/y why not spam the thread with speculation about the copper or electrical markets?
>>110057331Have you tried to see if K3's bigger size makes it more competent than 5.3?
>>110057331also "my context window is now full and I have worked so long on this problem"?It's really hard to avoid Claude if you actually want to be productive. It's honestly just so much better.
>>110056760I'm definitely going to get it. hey do you put your models Online? or make them have access to the internet in anyway?
>>110057201My penis is okay thanks to 5.3 flash. And that is all that matters.
>MiniCPM-V 4.7 35B-A3B addresses this with LES (Lookahead Expert Scheduling), which co-optimizes expert loading, compute scheduling and weight access, and combines it with an adapted DSpark speculative sampling stage, reaching lossless inference at 17.39 token/s on iPad Pro 11-inch M4 (16 GB).I like this a lot
>>110057303No but it is full of troons openly admitting they are troons.
>>110057331>it's just too dumb for it*shrug* works on my project.
>>110057366fempov ERP could have saved them. Harm reduction for autogynophilia.
