/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>110058788 & >>110054746►News>(10/13) LiquidAI releases its largest LFM2.5 model so far: https://hf.co/LiquidAI/LFM2.5-24B-A2B>(10/12) OpenBMB releases MiniCPM-V 4.7 1B and 35B-A3B: https://hf.co/collections/openbmb/minicpm-v-47>(10/09) Arabic Mistral models co-developed with Morocco: https://hf.co/collections/mistralai/mistral-mtnra>(10/08) JetBrains releases Mellum2.1 Thinking 12B-A2.5B: https://hf.co/collections/JetBrains/mellum21►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
im slopping up my own GUI
>No Bloodborne PC version in the newshttps://github.com/Supermedo/bloodborne_pc
What's the most barebones chat system to use? Like essentially Sillytavern but without the dumb shit and without me needing to make a ton of blank templates and characters. I'm sick of running out of free tokens so I might as well try to locally get what I want.
>>110063081how do I achieve this level of bliss with only 16GB of VRAM
>>110063102was it even slopped?I assume so because it's just using the shad framework but it certainly isn't advertised as such
>>110063102That has been out since last week, retard. Even had working multiplayer for almost as long.
Not putting the bloodborne PC version in the news is fine as long as we get the GTA6 PC version in the news next month. It's the ultimate sign of absolute domination of AI over the gaming and legacy software industries.
>>110063112Its either quality or quantity. Lots of low size models of meh quality but letting you do everything. Or you just focus one thing and keep slopping it up until its good.But there's a secret third option where you can just load all of the good shit at once and take the slow and chill pill with <1T/s speeds
>>110063127I'm not a /v/tard so I wouldn't know. I just think it's a big milestone of LLMs giving back the power to the people and open sourcing all software by force.
>>110063140>I'm not a /v/tard so I wouldn't know.It was even posted here over the weekend...
>>110063130Nobody is gonna flex like that when it'll have it's own pc port eventually anyway
>>110063093That's a fake term created by feminist to try to make anime pictures illegal.
>>110063106>the most barebonesllama cli -m my-model.gguf
llama cli -m my-model.gguf
Here is the linux version, fuck that windows niggerhttps://github.com/deadinside28/bloodborne_pc
Are local models and current harnesses good enough for overnight 1-3t/s projects to be reliably completed by the time I wake up? I can deal with dumber models whilst I'm awake for I sometimes go hybrid with chinks models on openrouter, but I want to know if there's anyone ITT who gets shit done locally whilst they're sleeping with stronger models.
►Recent Highlights from the Previous Thread: >>110058788--Comparing PolyStrata and llama.cpp throughput and MoE expert caching:>110058799 >110058934 >110059093 >110059097 >110059161 >110059277 >110059331 >110059340 >110059341 >110059346 >110059451--DDR4maxxing for big models and avoiding overpriced server RAM:>110059840 >110060512 >110060535 >110060538 >110060546 >110060593 >110060629 >110060638 >110060643 >110060980 >110060713--Hardware recommendations and cloud vs local debate for $20k budget:>110060910 >110060929 >110060936 >110061025--Reproducing LaMDA and discussing early dialogue pre-training data sources:>110059711 >110059733 >110059763 >110059858 >110059916 >110060025 >110060043--Methods for compressing long contexts using optical or encoder-based projection:>110059488 >110059498 >110059539 >110059515 >110059572--Comparing VRAM capacities for image generation and model fitting:>110061694 >110061701 >110061720 >110061746 >110061787 >110062812 >110061727 >110061734 >110061782 >110061805--Comparing GLM Flash, Gemma 4, and Qwen for degenerate roleplay:>110059478 >110059507 >110059529 >110060096 >110060130 >110060543 >110060561 >110059910 >110059931 >110060017--Comparing Strata and PolyStrata for Godot game development:>110061590 >110061611 >110061628 >110061641 >110062275 >110062285--MI100 VRAM utility for high context and hardware recommendations:>110059416 >110059469 >110059527 >110059552--Anon achieves high throughput on four RTX 3090s using llama.cpp fork:>110060269 >110060299 >110060367--Using local Mistral-Nemo via Ollama to automate directory submissions:>110059471--Anon provides a mirror of the PolyStrata repository:>110061998--Logs:>110062012 >110062349--Gemma, Teto (free space):>110058998 >110059789 >110060169 >110060486 >110061031 >110061040 >110062778►Recent Highlight Posts from the Previous Thread: >>110058904Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-scripti
>>110063170Yes. I ran qwen flash next q8_0 on cpu only to do some mods to llama.cpp (displaying a progress bar when using --load-mode none, loading models into large pages using VirtualAlloc) and it went fine. Although the large pages thing didn't speed up anything, so it was a failed experiment.
>>110063170Local models and current harnesses are good enough for overnight 10-30t/s projects on a single gaming gpu and 64GB RAM.
ugh im sloppppping AUGHHHHHHHHHH
>>110063170lol as long as your projects are single file python scripts
>my ram was running at 2666 instead of 3200thank you dmidecode -t memory ...
>>110063149That was the linux version not this new windows port
can I run dipsy on 12+128GB ram?
>>110063204~25-30K tokens at 1t/s, ~80K at 3t/s???
>>110063221Why is a wintoddler-only version newsworthy when the original from last week was not?
Am I missing something?A dude I've been chatting with has shown me his roleplays that he does in Sillytavern, and he has NanoGPT so he can use the frontier models for it.But holy shit, they're so bad. I can tell its written by Ai just from a passing glance. All the characters sound the same, they all talk like its a clinical office or some shit, and I could barely read it without cringing, to say nothing of actually roleplaying with it.>You didn't just hug me. You embraced me. The distinction matters.Like bro, holy shit. Why do people like that for roleplay? Local models may not have the context length or speed of the frontier shit, but at least Gemma has a soul and a writing style that isn't slop-tastic.
>>110063222yeah but it will be a cope quantyou need at least 192gb for full precision 0731
>>11006322380K at 3t/s is actually an interesting way of thinking about it. Every night you can realistically get a project done with a model you wouldn't run during the day. Better than not having the option at all. Even the richfags ITT aren't satisfied with what they have but could run an even bigger model at 3t/s. It's all relative.
>>110063238You're just jealous you aren't retarded enough to enjoy what they do
>>110063237No one on /v/ is going to play the linux version. Now /v/'s noses are rubbed up to the fact AI is dominant.
>>110063242I'm thinking more about Q4
>>110063238>Gemma has a soul and a writing style that isn't slop-tasticYou should run llama 1 65B, you don't know what soul is.
>>110063238Almost all enjoyment is novelty. You like most people in this thread have been exposed to way too much AI text and can no longer enjoy it, not until they change it to sound like a fucking human. To normies, it's transparent, it might sound "literary" to them at best, not a tell you're talking to a corporate robot.
>>110063238>Gemma has a soul and a writing style that isn't slop-tastic.no, it really doesn't
>>110063252q4 of 0731 is barely any smaller than full precision because the majority of the model is in mxfp4 straight out of the factory (so to speak)its like QAT
why are you responding to obvious bait
>>110063282Part of my system instructions
Something that's still not done yet but is a quick and easy win is whenever you type in a text box or some harness it should already page whatever text you have written so far in the background to speed up prompt processing so that by the time you hit enter it can go flying immediately.
>>110063290llmao does this
Do I have to become a cloudcuck to slop something for this or has local caught up?
>>110063297You already know the answer
>have $10k limit on credit card with 0% apr until next summerShould I?
>>110063307/lmg/ entry ticket is $20k now.
>>110063297the answer is yes and yes, GLM 5.3 will do it but of course you will need to have someone else run it for you if you're asking this question>>110063307literally free money if you just put it on T Bills in that case. i don't believe it.
>>110063307yes, use a strong rope so it doesnt break once you kick the bucket
>>110063297GLM 5.3 flash can do everything Opus 5.5 can but it will take 10x as long in terms of thinking.
>>110063306I literally don't. I know cloud models can handle it but I don't know if this is something the qwens can do now.
>>110063297I managed to slop basic 3d models with ds4.1f but you'd better wait for the next generation
it begins
>>110063327?
>>110063329the slop
>>110063319>literally free money if you just put it on T Bills in that case. i don't believe it.Congratulations, this is the lowest IQ post I have read here all week
>>110063092Way ahead of you. My GUI is already slopping itself.
>>110063338you dont udnerstans bro u have to go long with 200x leverage on TSMC
>>110063338>loan is 0% apr>T bills pay more than 0% APRyou might actually be a nigger in real life if you can't understand that. oh my god i will literally never feel like taxation isn't theft if even a cent goes towards subsidizing the community that allowed you to be born
>>110063346what u usin for compaction, broski? screenshots or shaking or LLM summary or..?
>>110063356NTA but there is a significant default risk from the US government and you might be left holding the bag on 10k of debt.
>>110063356You don't believe that 0% APR cards exist? The best use for 1 year of free money you can come up with treasury bills? And you are here doubling down on that? Seriously?
>>110063369I buy bonds yearly>low risk>generate passive income >can sleep at night without an infart thinking about candles going red
>>110063368>there is a significant default risk from the US governmentlol, the US will steal another country's money and use it to pay its debt before defaulting. why do you think they stole venezuela>>110063369>The best use for 1 year of free money you can come up with treasury bills?no, its the safest use. have you forgetten they stole all of venezuela's oil? I'm not even a mutilated burger mutt I'm just not a retarded poor
>>110063359LLM summary with codex/hermes/dsh features.
>>110063379T-Bills are not "bonds", T-Bills are lower risk than bondsdon't reply to me about the US collapsing, if you actually believed this you'd be doing more than avoiding buying T-Bills for fucks sake..
>>110063385>have you forgetten they stole all of venezuela's oil?mutts even fucked up stealing, they can't even steal properly apparently because they had to buy diesel from russia
if the us collapses i don't think your credit card 10k is going to matter much anyway
>>110063307you can get half an rtx6k
>>110063385>Venezuela misinformationAre the Russian botfarms back up again or something? Do you know how low the revenue is from Venezuelan oil? Do you realize it's held in a non-US bank and not directly at the behest of the US treasury? Do you realize how insignificant all of that is compared to the crushing interest payment on secondary markets for US treasury bonds, fucking the US when debt rolls over? The US is at the highest chance in history of defaulting on its debt. "It never happened before!" It also didn't happen until Russia did so in 1998 crashing the Asian market with it. This is /lmg/ so I don't expect people here to make financially coherent arguments but what you just said exposed you as either a deliberate troll or one of the dumbest posters I've seen in a while.
>>110063368Lol. If the US defaults, there will be hyperinflation, so you pay off the loan easily with worthless money. Any other midwit takes to share?
>>110063396no its stolen they just can't refine what they stole in time to avoid the diesel midterm crisis. i dont think any of this is smart, my point is that the US literally already has, and will continue to invade countries instead of default on its debt.maybe a military economy saves america, you send all the pro palestinian university tent sitters to die and your wartime economy and war bonds fixes/breaks everything else and you also don't waste time pretending with elections anymore>>110063402exactly, T-Bills are existentially speaking 100% safe as long as money still matters.>>110063418didn't read. my point is that the US literally already has, and will continue to invade countries instead of default on its debt.
>>110063414he can get two $5090s and sell them for $10k each next year
>>110063418im european ;)_
>>110063297wait next gen (this afternoon)
strata bros...this is making the feedback loop kind of ball breaking, losing a 200k ctx cache every single fucking time the agent does a screenshot (when checking webapps it does it a lot also in succession).https://github.com/Niko1221/Strata/issues/2228HELP ME VIBESHARTERS
>>110063426>he can get two $5090sarent these like 10k nowHonestly, maybe I should see if i can take out a credit card and do this too. Ideally theirs a credit card out there with a generous signup bonus that requires you to spend like 10k and no interest for a year...
>>110063369>0% APR cards existreally?
>>110063429didn't you say you had black eyelashes?
>>110063461no, it's a hoax
>>110063426ah yes 64gb vram for the low price of 1150W
>>110063461
>ranjeet, you can have $10k for free for one year>you can invest in equities making 200% returns per year>or gpus also increasing 200% per year>choose wisely>hurr duur want 4% t-billsthese are the same retards coming here crying about the cost of hardware and missing out
x is laughing at us
What am I able to run for Agentic bullshit with 32vram+44ram? I need 20 for my server, so I can't use all 64
>>110063502you’re a 10 person startup?
>>110063502go back
>>110063502Just send your production code and intellectual property to [someone else's computer] it's so much cheaper and faster :)
>>110063502Why would you need 128gb of vram for RP though? That IS the only reason you'd use local stuff right?
>>110063502in 8 years those cards will be worth more Uganda, and those expired subscriptions will be worth a like on reddit
>>110063518Anthropic would never steal your code or intellectual property. Take off your tinfoil hat.
>>110063456if you have money, take out collateral on your stocks / margin instead of using a credit card.InteractiveBrokers hyas good rates>>110063461americans enjoy financial supremacy and things like 0% apr cards yes>>110063498>>choose wisely>>hurr duur want 4% t-billsyes, that's what choosing wisely looks like. that's why they don't let you buy as many as you want.i will entertain 3 more of your financially illiterate replies before I start posting photoreal kids because I find it funny when people poorer than me give financial advice
>>110063502Your troon failed calculus, you're not going to make 10 people use a single $200 sub.
>>110063502>18 KHey, that's pretty good.>>110063518More than that, you are at the mercy of the models they provide (which they change all the time), throttling, rate limits, guardrails getting in the way, etc.With a setup like that, more than having everything local and isolated and under your control, you can leave it running 24/7 eating through an endless queue of work in parallel.18K is not that much at all when it comes to all of that.
>>110063502just coming from that turbo queer narcissist's mouth would never make me want to touch itare the Claude reps okay with that?
>>110063502If it's not for coding that could actually be a decent spec
>>110063492>5% rotating mystery box of category of spending>Excluding Walmart and Target??? why?
>>110063534>hurr durr lets engage in usury that's the death of our civilised societyBegone jew
>>110063502$1600 is a lot to "theo"? show his flag lmao
How are you supposed to buy stocks/bonds with a credit card?
Has anyone tried polyGLM with 16gb vram and 64gb ram? It seems impossible but even if the speed is trash as long as it runs surely it'll be better than qwenext.
>>110063580glm doesn't have any ssdable parts like qwen
>>110063580even the worst copequant wouldnt fit weights alone on that
>>110063211was that a big deal
>>110063573He's a multimillionaire. He spent $400K on claude on a random typescript compiler project the other day
>>110063576You have to be a finance pro to gamble with someone elses money, especially banks money
>>110063571wow an actual muslim. commit suicide poorfag shitskin. there are no rich countries without banks.
>>110063553>you can leave it running 24/7 eating through an endless queue of work in parallel.you can do that with subs as welland not pay for electricity
>>110063591Damn, don't say that. I have the same setup, but 48gb vram. You killed my hopes before it ever even began. I'm waiting on polystrata to come back to github since I don't trust downloading some trust me bro binary off of huggingface.
>>110063368Governments that borrow in their own currency have no default risk.There are other risks, but default is not one of them.
>>110063637Well why doesn't anyone make a c.ai clone?It's literally free money as long as you shill well.
>>110063647Most of those women are talking to male characters. 31B is girly as hell and all the male local models can't chat for shit.
>>110063170depends on project size and the project itselfand sometimes after decomposing into tasks you don't need smart slow model to complete each of themi did with 5t/sit works if you have good plan/spec and tests
>>110063647aren't there already millions of clones? most people just don't like to change their habits. they're used to c.ai and want nothing else
>>110063596I'd do better with $400K
>>110063659That's why the shilling is important.
>>110063647>free moneythey're freeloaders nigga, you're not going to make any money from mentally ill 15-26 women.
do you think claude sabotages local model or harness building assistance? I think it is making my harness shit on purpose
>>110063674i'll setup a chatbot for a cute, loyal hebe for free... mental illness is acceptable (yandere).
>>110063223> let me explore the project> is python available> now i have full picture> draft1> oh wait i did it wrong> draft2> oh wait i can't write files without making a directory first (after spending 10k on new file contents)> let me test> no it doesnt work> let me check once again> now i have all information i need> draft3> ...
>>110063682yes it does. look at your tool descriptions, prompts, personas etc then ask your model to rephrase them for it's own understanding.
>>110063682Very unlikely it only gives bad performance on purpose in AI/ML tasks. Harnesses are considered regular software so it shouldn't behave bad on purpose.
>>110063682just don't use chink models
>>110063674Aren't they making big bucks on OF these days tho?
Why do people complain about twitter and reddit screenshots if they get replies every single time?
gave gemma a bbl
>>110063647>audience of addicts>high propensity to mental illness>large portion underage with no spending powerbasically a less profitable version of being a drug dealer
>>110063682They're all pretty shit at handling local deployments because the semantics change every month and their training sets are ancient. Just read the manual.
>>110063719They still think the mongoloids that inhabit 4chan these days are capable of producing "OC"All the announcements have been dropping on twatter first for ages now. That was the whole thing that made it go viral.
>>110063664money obsessed people turn into pol words.
It's going to be so much fun seeing games being ported to PC at an increasing pace and the devs screeching about piratism and how AI needs to be regulated harder.People have been asking for PC versions of countless games and the devs haven't been assed to port them for whatever reason.Soon they'll start whining about losing profits on something they didn't even want to offer in the first place.Fuck them, there's no stopping this train.
>asked for a complete rewrite in another framework while the AI was busy doing one of the changes>agent malding that I demanded the change NOW while the repo is in a not-working state because he was in the middle of work>even gave me multiple choice shitlol
>>110063716OF income got destroyed by AI funnily enough.
>>110063749That and the bug fixes and modding potential.
>>110063753the agent paid a nigga in crypto to bash your head, stay put
>>110063740Lol this site passed its golden age many summers ago
>>110063760Yeah some cosplayer I follow were screeching about it for some reason, like they aren't participating in IP theft to make a buck
is there like um an easy way to get local ai to make me money so i don't have to work?
>>110063768yet it still is one of the most bearable place to uset. newfag
>>110063776Also interested
>>110063776I got one yes
>>110063749>and the devs haven't been assed to port them for whatever reason.Because they were paid to make them exclusive?
>>110063776become the C. Ai knock off
>>110063776yes just use this prompt 100% guaranteed of be of working for great izzat gains:"gm sir, ples be of maeikin g huge amount opf moni becuase I am of need LAMBO soon, hers my id and card details"just attach your ID and your CC in image format and watch the agent make you some real bucks really fast!
>>110063776A couple but you have to put some work up front and deal with some maintenance work afterwards.
>>110063081test
>>110063805what are they
>>110063776What I did was move to western europe, get a social housing project and welfare. My reasoning being once AI takes all the jobs I will already be grandfathered into the system and be the first to received UBI which rich EU nations will absolutely give out for free because of how socialist they are.
https://github.com/ggml-org/llama.cpp/pull/30520>Add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B supportlooks like some new nemotron models are coming?
>>110063810テスト頑張ってね!
>>110063704sovl
>>110063776you get a job and get the model to work for you, it is what everyone does
>>110063776Literally everyone right now is merely pretending to work, get an office job or any type of job that requires you setting behind a desk and slowly automate your tasks away.
>>110063810Congratulations, you've successfully eluded your ban.
>>110063858I really need to get a second job while the getting's good.
>>110063749I fucking hate cloudcucks to death but this porting, modding and reverse engineering arc the community is going through is based. I'm fully behind them and like you said they're not going after good projects they're just doing the shit everyone has been asking for.
>>110063858I already had a merely pretending office job but I made the foolish mistake of being too dependable so I kept needing to do shit
>>110063862Ha ha!
>>110063866This whole arc is why I'm planning on building a decent local system.There's no way they take this lying down once AI stocks settle and a big enough tech company starts bitching about it
>>110063749>>110063866Yep same cloud AI being used for open sourcing proprietary software is a net good for the world and therefor tolerated.
>>110063858>everyone right now is merely pretending to workMaybe if you're sitting in IT. The only thing I have access to is the cheapest tier of Copilot.
>>110063924Sucks to suck. Why would you work for a poorfag corp?
Reminder the death of Anthropic will be proprietary software lawsuits. Only Jevs can destroy Jevs. All we have to do is encourage reverse engineering and wait for GPU prices to drop.
>>110063924I pay claude code out of my own pocket and use that to automate my entire office job. Not having to work and shitpost on /lmg/ the entire day is worth the $100 a month.
>>110063935Anthropic is already too big to fail, proprietary software companies are just going to die out instead.
>>110063170I don't see the appeal in this kind of setup. At that point just write the code yourself.
>>110063953>proprietary software companies are just going to die out insteadand do you think Anthropic's biggest enterprise customers are?
>>110063953Anthropic and OAI are getting set up to eat dirt later so governments can inherit their assets. Nobody gives a shit about IP holders and their 'rights' any more unless it's an excuse to squeeze the proles.
Some new data for my fellow /lmg/ MoE vramlets.
>>110063973>vulkan>hip>llmao.cpp forkROFL bro, these two models also run on memestrata btw
>>110062148find the token causing this and swap it for ` human` or ` maid` in the jspace, usually the 2nd-last and 3rd-last layersworks for almost every model
>>110063991Both performed generally worse on PolyStrata for me. Results may vary, I s'pose.
>die by proprietary software lawsuits or live long enough to wipe them out with your AI technology and lose your biggest enterprise marketAnthropic is going to collapse soon KEK
>>110063973>Do 100t/s on a 16gb vram 4080 with qwenextusecase for... whatever this garbage is?
>>110063972Opposite, government is getting set up to eat dirt later so Anthropic and OAI can inherit their assets. Nobody gives a shit about governance and office holders once the AI labs hold all the leverage.
>>110064006Anthropic is looking to just replace the entire economic engine and co-opt society in one fell swoop. People won't know what hit them.
>>110063858why did this comment get 3 replies while mine got none even though they say the same thing >>110063844I am offended and I will beat up everyone here
>>110064012How much system RAM do you have? I only have 32 GB.Also, your GPU has over double the beam-bandwidth of mine. Apples to oranges.
>>110064022No idea anon I even felt stupid writing that post when I updated and saw someone right before me wrote the same thing.
>>110064035I will beat you up too
>>110064016The government always has the leverage because it has a monopoly on violence.
>>110064044You have to be naive to believe this.
