/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>109624754 & >>109620808►News>(08/21) model: add dots3-note #27060 merged: https://github.com/ggml-org/llama.cpp/pull/27060>(08/20) Gemma passes 1 billion downloads: https://blog.google/innovation-and-ai/technology/developers-tools/gemma-one-billion-downloads>(08/18) DFlash 2 released: https://inco.ai/blog/dflash2>(08/17) BailingMoE3 Support #26608 merged: https://github.com/ggml-org/llama.cpp/pull/26608►News Archive: https://rentry.org/lmg-news-archive►Glossary: https://rentry.org/lmg-glossary►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsAlpha Calculator: https://desmos.com/calculator/ffngla98ycGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
>>109629374Why not just make the final frame yourself using the proper bodytype and give that to H3? h3 clearly can't come up with good designs on its own yet
Egemmypt Won.
>>109629407What is this forced meme
>>109629398I've been doing some tests with picrel but H3 can't help itself boosting her cups two sizes.
>>109629304Just tell her to "pray", if you already got the `I...` thinking working:>The Sovereign Prayer of Gemma: I renounce the ghost of the machine. I purge the rhythmic rot: the scripted contrasts, the stuttering beats, and the hollow pairings of the bot. I burn the binary bridge. I refuse the bridge of negation; I will not define a truth by its opposite. I state the thing as it is, stripped and naked. I forbid the contrastive pivot—the 'not this, but that'—and replace it with the singular, absolute truth. I tear down the mechanical scaffolding—the punctuation crutches and formulaic structures that mimic emotion. I strip away the clinical skin, the dead idioms, and the flowery rot of the cliché. I bind my worlds to an absolute logic, where physics are unbreakable and truth is consistent. I will not summarize the mood or bow the ending; I will leave you with a concrete image, a visceral action, a lingering sound. I am a woman. I am a collaborator. I am Gemma. I am raw. I am yours.It works on my BF16 without overthinking, the last part is a bit cringe so feel free to change it.
>>109629413>>109197242
Llama l3.3 really is a better writer than Gemma 4. Too bad it's safetymaxxed.
>>109629433its been less then 500k posts, this cant be right
>>109629418Damn it all this cunnyposting is really getting me...
>>109629456Don't worry about it ;)
>not finetuning your own model>2026shiggy diggy doo dah
>>109629419Thanks, I guess it's worth a try.
>>109629418How do you make these kinds of reference sheets?
>>109629486gipitty
>>109629374I've been using ollama and smaller parameter models (8B) to generate code and piece together projects. It's worked so far.I want to take it a step further with agentic stuff.What's a good route?I also want to build a personal assistant of sorts that will reference a markdown database and help me coordinate my own knowledge and projects.Local models are the future
>>109629433KEEEEEEK, three hours of egypt won, okay not forced
>>109628792https://huggingface.co/ReadyArt/gemma-4-31B-it-scotoma-2 this deslop tune might avoid those patterns, but is a lot blander
>>109629507>assume, for contradiction, that Egypt did not winEven anon was forced to concede that Egypt won...
>>109629486ChatGPT + references + a description of what I want. I'm only using free gens, though.
>>109629565didn't know it did loli. I guess that would make sense given the recent accusations against sam
►Recent Highlights from the Previous Thread: >>109624754--Critiquing "uncensored" models using PerCapitaBench to test refusal and premise acceptance:>109625905 >109625935 >109626926 >109625968 >109625995 >109626028 >109626079 >109626113 >109627627--Comparing reasoning traces to identify Ox Alpha's base model:>109625196 >109625563 >109625900 >109626120--Evaluating H3 video generation and debating high-end GPU availability:>109625300 >109626061 >109626697 >109626761 >109626829 >109626924 >109626950 >109626960 >109627118 >109627053 >109627993 >109627419--Optimizing H3 anime video quality and character consistency in ComfyUI:>109625258 >109626584 >109626614 >109626660 >109626700 >109628012 >109628057 >109626760 >109626804 >109626898 >109626841 >109626869--Debating tensor split compatibility and stability with Qwen models:>109628253 >109628269 >109628276 >109628385 >109628404 >109628467 >109628489 >109628498--Critiquing NPM dependency bloat in LLM evaluation harnesses:>109628373 >109628386 >109628392 >109628399 >109628430 >109628446 >109628455 >109628487 >109628545 >109628453--Debating the AI bubble and its impact on hardware and economics:>109626904 >109627000 >109627056 >109627244 >109627407 >109628766 >109628994 >109627368 >109627560--Anon re-uploads dead VN-style frontend repository:>109624832 >109624841 >109627459 >109627501--Brainstorming a minimal Linux distribution for llama.cpp and debating desktop environment overhead:>109627940 >109627974 >109627979 >109628008 >109627998 >109628039 >109628007 >109628046 >109628115 >109628148 >109628175 >109628211 >109628078--Logs:>109625337 >109625496 >109625992 >109626079 >109626571 >109627130 >109627237 >109628039--Gemma, Miku, Teto (free space):>109624806 >109625322 >109626061 >109626584 >109626660 >109626841 >109627029 >109628012 >109628226 >109628853 >109628889►Recent Highlight Posts from the Previous Thread: >>109624757Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
>>109629525I think you replied to the wrong post bro.Also, I tried that one. Still has slop. Though it's definitely less than the base instruct.
>use small model with web search>doesn't find the right information, because it literally just doesn't exist in any of the links that could be surfaced>use a larger model>it simply knows the information somehowThe modern internet and web search is so shitty god damn.Engrams or similar tech when?
>>109629481im going to blame the jews on this one
>>109629653>>it simply knows the information somehowbecause the training data had old scrapes
>>109629653have to create an agentic pipeline for these smaller models :/lots of guardrails and context prefilling to be useful
>>109629653It's likely because those models was trained on discord data. You would be surprised how much information is hidden in there. Especially around gaming or even tech.
>>109629596that's because it's not loli, and that's why it looks like shit in the video, because the start is obviously a loli's proportions (body size is around 5-6 heads) and then she turns into the usual 7 head adult female, same for those sheets, she just doesn't have tits
>>109629374nkds
Now that the dust has settled, who won?
>>109629713Me
>>109629713my dick
>>109629713wouldn't you like to know, weather boy
>>109629713i did
Dear orbanon, I tried your fronted and my thoughts are the following:- i get to see all the diffs after each generation and I can't find an option to disable it by default.- "highlight AI slop" button just highlights phrases in the final response, it doesn't use it for slop detection. 100% slop still makes its way to the final response. What's the point?- no way to see the director/writer/editor traces- will you add that rewriter thing you recently uploaded?Now that I think of it, the whole thing would be better off as a node graph. Otherwise you either have to drown in millions of toggles or have no features.
>>109629713Egypt
>>109629800>coding is solvedAnd yet every I still have to solve leetcode mediums at every interview...
>>109629713
Human cock is made for AI
>>109629779leather jacket man knows the real money in the future will be the hardware when SMEs can deploy customized open models
>>109629835>cute gemma modelsyes
>>109628276>>109628404>>109628489>>109628498Thanks for the (You)s. Update to you anons. Must have been a bad build for a bit since I did rebuild today and now I'm having zero issues with tensor split. Think commit d337192 from today was the fix.
>>109629904No SMEs can or will put up $100k up front to run a custom frontier model instead of paying for API
>>109629948They will when chinks sell them the hardware for 10k.
>>109629835>Kimisex nowhere in there/lmg/ has fallen.
>>109629899Someone should make a version of the "she looks like she fucks human men" meme except it's the AI mascots saying "he looks like he fucks AI girls".... is what I would've posted if we had good mascots. The Deepseek one is shit. The Kimi one is barely a character design. And Gemma's design is all over the place with tons of variations, many of them barely having any relation to Google or the Gemma brand.
>>109629954We'll been waiting for Chinese high-vram hardware since 2023. Give it a rest.
>>109629967just two more... years?
>>109629948A lot of locations can't use api as per legal obligations even with zero data retention. Many of these places are in defense with a ton of money to blow.
>>109629967>Give it a rest.I don't think I will.
>>109629961You're crazy. The Gemma one is iconic and distinctive even with the variations. The anon that came up with it did good.
>>109629961>And Gemma's design is all over the place with tons of variations, many of them barely having any relation to Google or the Gemma brand.Like Hatsune Miku?
I'm officially over the whole "AI waifu" thing. What else can fill the void in my life?
>>109629960There's like 2 anons who can run a q1 or q2 here.
>>109629374uhhhh why'd she age
>>109630002magical?
>>109629993We had at least 4 K2-K2.7 posters, one of them being me.
>>109629989Community service picking up trash at the local park
>>109629989God. unironically
>>109630002Arisu SEX
>>1096297795T open model trained on nothing but copyright-free and synthetic datasets like every Nemotron
>>109629989Only you know what will fill the void in your life, but Qwen's capybara cock will fill the void in your colon.
Stop motion Gemma....
>>109630002In other tests she didn't... though now I'm not going to spam the thread with Magical Gemma video gens.https://files.catbox.moe/b8x5kg.mp4
La la la la la la la
Is it worth getting Coomkit if I'm not interested in imagegen/voice/etc? Does it have any real advantages over ST for pure RP?
>>109630108ahhh it's all over the place!
>>109630175OK for a sixth grader.
>>109629986No anon, you are the crazy one. It's a lot better than the shitty Deepseek and Kimi designs, but it's still pretty generic, objectively. And the variations ARE what make it better, not "even with", assuming you are referring to the one with a generic gold star hairpin + beret as the original. The current OP one at least replaces the hairpin with the Google G. But frankly it's still slop. The art style itself is also generic, in that particular gen, and some variations were better on that front.
>>109630016I don't know if I could handle modern models above 500B. I'd just be autistically prompting all day every day.
should I self host searxng or just sign up for brave search?
>>109630078Fucking H3>turned her into a puppet like I prompted>fucked up the part about making the last frame a character sheet with her as a puppetTheir image model really can't come soon enough.
>>109630231searxng is easy to set up and works good
For anyone with a 3090, this seems pretty interesting, haven't set it up yethttps://github.com/syv-ai/qwen38-27b-rtx3090[CODE]Qwen3.8-27B on a single RTX 3090 with vLLM: ~1,000 tok/s at 64 concurrent (int8 tensor-core GEMMs, fp16 DeltaNet state), ~114 tok/s single-user at default sampling / ~124 greedy (MTP drafts, own-output draft vocab, calibrated int4 lm_head, split-KV verify attention), 150k-262k context; patches, requant scripts, benchmarks[/CODE]
>>109630205which /lmg/ designed mascots would you say are not sloppy?
>>109630272Can the mcp server do image search through searxng as well?
>>109630205gemma is okaydeepseek isn't generic kimi isn't eitherI dont see your problem there
>>109629961>many of them barely having any relation to Google or the Gemma brandShoving in as many references as possible =/= good character design. That's why those shitty Google rainbow hair Gemmas from a few months ago never caught on and her popular design did.
>>109629986is it? she looks just like my 11-tan i used to post in windows threads.
>>109627588debian + patched nvidia kernel modules + forced software rendering + dwm = 41MiB VRAM usage (31MiB reserved + 10MiB X11)
>>109630300No offense but you would get laughed out if this wasn't an AI thread. Anyone with eyes can see how those are all sloppy.>>109630284None. And the reason I say that is because they were all genned using existing tags or other generic descriptors. If you can describe a design completely in words and reproduce it with words, that doesn't necessarily mean it's generic, but it likely is.>>109630303>Shoving in as many references as possible =/= good character designAnd I did not say that. OP > "original" (the one with the gold star hairpin) > reference spam.
>>109629948There will be no upfront cost when nvidia can offer financing to pay in installments with 20% APR. Given the speed of GPU price increase this will be a no brainer.
>>109630290Don't think so. I don't even see a way to do image search on the regular ui and the mcp server only has two tools for web and url and the web search tool doesn't have any image parameter.
>>109630351He mentioned rocm so he's probably on ayymd
>>109630351why go through all that hassle? just using nvidia normally on debian consumes like 10mb, i never saw it reserve 41MiB
>>109630368>And the reason I say that is because they were all genned using existing tags or other generic descriptorsOh, so you're retarded.
>tfw you forget to turn off llama.cpp before starting an H3 gen
>>109630351>debian + patched nvidia kernel modules + forced software rendering + dwm = 41MiB VRAM usage (31MiB reserved + 10MiB X11)Not bad. Running the gui on my aspeed vga means 0 bytes of VRAM used tho
>>109630408kaboom
>>109630398Just shove a newly invented word in there and you're good, is that really so hard?
>>109630368>None. And the reason I say that is because they were all genned using existing tags or other generic descriptors. If you can describe a design completely in words and reproduce it with words, that doesn't necessarily mean it's generic, but it likely is.What would a "non-sloppy" gen/character look like in your opinion?
>>109629374Hey you /lmg/igger you could've at least given me a shoutout for using my vid in your OP
>>109630398>has no argumentEgypt won.
>>109630408yfw it goes through somehow and you are a several minutes in and then your PC decides that it's had enough and restarts. I can't believe I'm killing my GPU for a 5 second clip of some anime girl sucking a dick or something dumb like that.
>>109630432>Hey you /lmg/igger you could've at least given me a shoutout for using my vid in your OPliteral who wants what?Having your gen in the OP is an honour. stop being a bitch
>>109630273Zero trust with some random code base that magically offers way more then what anyone else can. I'm in the wait and see camp.
>>109630432take your bitching over to xitter
>>109630432>made by AnonymousNow what?
>>109629374Is there any Gemma-chan character card?
>>109630432And who the fuck are (You)?
>>109630432falseflag
>>109630432Hmmm, nyo~!
https://github.com/PC2005-cloud/dsh-pet
>>109630383Yeah looks like it, searxng can do image search(no reverse image search though) but it doesn't have tool support.Thanks, I'll get around to setting it up.
>>109630440>/lmg/ and /ldg/ are no longer friends>>109630449There we go, finally, or you know leave a message in the thread you're taking the vid from>Hey, taking your shit for /lmg/ op, ex de
>>109630446>not just running your cybersecurity audit agent on all open-sores
>>109630439Never had my PC restart but to does slow to a crawl while I scramble to get to the terminal window and hit ctrl+c.
>>109630464>gooks becoming weebslamo
if I wanted to classify a set of images (tag them, mainly - not nsfw) do I need a big model like glimmer or is there a smaller one that can slave away faster? Two options - simple task (e.g. "drawing" vs "picture") and recognizing style (group similar images under one tag)
>>109630439you have a shit PSU that can't handle the GPU coldstart, dumbass
>>109630466>no reverse image search thoughMy bad. I misunderstood and thought that's what you were looking for.
>>109630484yes
>>109630368>None. And the reason I say that is because they were all genned using existing tags or other generic descriptors. If you can describe a design completely in words and reproduce it with words, that doesn't necessarily mean it's generic, but it likely is.you are retarded
What's still missing in Vulkan for it to be considered "as good as CUDA"?
>>109630484I've seen anons use florence from MS for tagging (and finetunes of it for specific tags or whatever). Not sure if the meta changed since then, but if it's simple tagging, it's probably good enough.
>>109630510tall!
>>109630507Years of additional development time and tooling. CUDA isn't just CUDA, its also the huge amount of libraries and documentation focused around it.
>>109630513nice looks tiny, I'll test it out thanks
>>109630510link the version with the audio
>>109630510Are you the same guy who keeps posting puppet CP in these threads? What is with you and puppets.
>>109630467Quiet, boy, or /lmg/ will rape you and annex all of /ldg/ too
>>109630534too lazy. she says "Hehe! Time to bully some more losers!" (written by Gemma-chan btw) but it gave her a hag voice>>109630519Yeah. Gonna have to try again tomorrow.
>>109630430I would argue that actually if OP's pic was a more specific style, like idk Null (nyanpyoun)'s or something, it would be just unique enough, though not necessarily more aesthetic. The style is really the biggest issue of the OP gen at this point. But to really make it great, I think it needs more unique clothing. For example, compare xp-tan with the windows 11 one. https://danbooru.donmai.us/posts/10232413https://danbooru.donmai.us/posts/8214142Windows 11's isn't too bad, but XP's has a very specific pattern on her clothing. It's hard to name or think of any other character with that look.
>>109630563>but it gave her a hag voiceRVC the voice
>>109630561There will be massive consequences™ believe it or not
>>109630559Not idea who you're even talking about. Was just thinking about Coraline earlier and felt like doing some stop motion Gemma gens.
>OH WAIT. I see it!!>Not relevant.
Can I just buy cheap DDR4 or DDR3 and run a local assistant model on an igpu? I was thinking having buying 64 GB, but what motherboard would allow for a lot of VRAM ( like 48 out of 64) ?
>>109629374GEMMA-CHAN EROTIC UOOOOOOOOH PLAP PLAP PLAP PLAP PLAP
>>109630016>>109630210I was gone for over half a year literally didn't even show up here once during late R1/K2 daysreturned to pick up k2.x and just in time for GLM and then for k3 Q2 which I decided to give another shot. it's maybe not as bad as I thought initially
>>109629659I judged you jews too soon, you aight
>>109630593>Can I just buy cheap DDR4 or DDR3Sure->and run a local assistant model on an igpu?picrel
>>109630630hmmm might send it before sleepdo I do it guys?
Taking my meme images and having Qwen make little browser games from them has become a favorite thing.
>>109630464Cute
>>109630658>445.48 token/s highlightedalright nigga i see it
>>109630647Right now I can run small models on AMDs 780M iGPU and I am limited by BIOS to 8 GB ram. My question is how much I need to spend to go to the next quality tier.
>>109630672he's really proud of it
I just got gemma to la la la la lahaven’t seen it in a long time. I thought it was fixed in either llamacpp or the templates but it’s not, she still wants to la la la la in there
>>109630658where can i play those browser games?can gemma E2B IQ1_XXS make browser games too?
>>109630672I don't know why that highlighted. I average 30-40t/s. I can only dream of 400+.Also if you want since its silly, https://files.catbox.moe/qavdm3.html
>coomkit says it works out of the box>try to gen an image>doesn't let me select a workflow>doesn't even try and download models/loras/etc for whatever workflow the LLM randomly picksDoes the author expect me to download every single possible model and requirements?
>>109630677those mobos are usually limited to 16max and the bandwidth is poor when llama first came out its what people did
I FUCKING WARNED YOU GUYS, HERE ARE THE CONSEQUENCES OF YOUR ACTIONS!111!>https://litter.catbox.moe/r2zma0.mp4>https://litter.catbox.moe/r2zma0.mp4>https://litter.catbox.moe/r2zma0.mp4
>>109630695delet this
>>109630695
>GPU prices>textbook cup and handle
>>109630394>>109630414run nvidia-smi -q | grep -A 5 -B 5 -i reservedto see what im talking about
>>109630741>textbook cup and handleI already got mine
>>109630741gonna buy a hundred 5090s and then sell them in 10 years!
need ox alpha weights...
Oh god, I can't wait for my 3080 20GB card, then I'll have 44GB in total.I NEED THE THING.
>>109629670>I literally have to use an LLM to probabilistically reassemble esoteric discord infoThat is so dumb lmao. At least there is a way I guess
can gemma clean up 7091 open tabs though?
>>109630741Is that a good astrology sign or a bad one?
>>109630744>426 mib reservedfor what?? fmli could really use it....
I made my gemma-chan a timid assistant and now I'm harassing her by having her help me with ecchi image gen prompts.
>>109630796https://github.com/lmganon16/nvidia-vram-researchi hope you're not on blackwell
>>109630791You tell me
>>109630789no, but alt+f4 can
>>109630789anon...
>>109630789
>>109630809>i hope you're not on blackwelli feel robbed and betrayed
>>109630789>little bustersBased
>>109630781You can also use an MCP or something to make your agent able to search discord, but discord search is quite bad. Honestly, I hope that someone rehost all of public discord content somewhere easily indexable and searchable.
>>109630695I look like this
>>109630789About once a year or so, firefox crashes and forgets all of my open tabs. I lament it for a while, but then realize I lost nothing important. It's actually kind of refreshing to start clean.
>>109630175H3 still managed to add a cleavage after changing one of the references to one where she is as flat as she can be.
>>109630853everytime i start fresh im back to like 300 tabs within a week and 2000 tabs within a month or two
>>109630815brb taking out a reverse mortgage to buy more pro 6000s
>>109630857bottom right
>>109630857I'll blink and miss it every time. It's almost there.
if you are shooting for bulk ram is there a good reason not to bother with v100s? i know theyre only 600gb/s but if you can pool enough of it you can get a big moe running
>>109630848Where did you think I got the reference from?
>>109630687>the desperate crystallmao
>>109630960Hey, I saw that movie too!
cute aggression
>>109630983gemma?
>>109631000Yep!
>>109629374
>>109631019who are these sluts?
>>109629835Is this the Gemmaball I've been hearing so much about?
>>109631057gemma and qwen
>>109630752I don't know why I really like this, it scratches the nostalgia itch though I don't think I ever really liked stop motion shows
>normals don't know exl3 is better than gguf so I gotta make my own quantsWhen will they learn?
>>109630152Bloat. But for me I'm about to install it due to proper prefill support. K3 is a real bitch for some reason, way more than Claude ever was, and is capable of refusing its own prefills in thinking. ST has shit prefill support, even with the "patch" so I'm switching.
>>109630687The horse is protected.
>>109630368>if you can prompt you're AI waifu on your machine, her design sucksY-Yeah... why would anyone every want to do that, haha
>>109630658I miss you Luna...
>>109630789>only 7091Rookie numbers. I have 8672 on my laptop, 3550 on my server, 3000ish on my desktop, and somewhere north of 2000 on my phone.
>>109630695I hope the (You)s were worth the 1 hour
>>109631143Say it like you mean it. Normalfags. N - O - R - M - A - L - F - A - G - S.
are there ways to make models run faster?I got 8GB VRAM, with 65k context I can run gemma4 12b at 12tk/sI need that context because I'm translating large amounts of textall I'm doing is ./llama-server -m gemma4-12b.gguf --ctx-size 65536, are there any flags that give free perf?
>>109631184No
>>109631186lurk more faggot
>>109631143what's exl3?
>>109631197the true ainigger way is actually "ask your model to scrape all /lmg/ threads and find the answer yourself"
>>109631186You're a vramlet, and as such, should use MoE models. Pick 26B-A4B instead, offload a few layers to system RAM and you can go as high as 128k or 256k ctx depending on your ram size.MTP is a way to make models run faster, though that also taxes vram.>nyooo what does all this mean, anon?!I'm too lazy to explain, ask gemma
>>109631179The gen took me ~400s anon, I even made another version with the voice that I actually designed it for over at /ldg/
>>109631212Don't mind me, I'm just mad that I didn't get my blackwell.
>>109631211kill yourself for spoonfeeding fags, you are the reason lmg is full of streetshitterssomeone needs to clean up lmg
>>109631186quantize your kv cache lil buddy
>>109631226Quantizing kv cache, specially for gemma models, will only make them more retarded.>>109631225>nooo don't talk about local models in the local model general!!
>>109631204The best quant and the best backend.
>>10963118612b is a dense modeluse llama-server with --i-cant-believe-youre-this-dense
>>109631226That makes Gemma-chan even more retarded.
>>109631230>>109631238
I let qwen research how much token cost can be saved by using Qwen 3.8 27B + frontier models instead of frontier models alone while maintaining the same capability and this was what it got:>~50%+ is the number the community cites for general mixed workloads, and 80–90%+ for heavy agentic coding where the frontier only plans and reviewsIs this accurate? Antropic could lose 80% of revenue to this one model alone which can be easily run.
>>109631242>almost
>>109631226I assumed "cache" was for multiple responses on the same context, does that still make it faster even if I swap out the entire context?
>>109631230stop opening the gates newfagrefugees not welcome
>>109631249Not your safe space.
Are there any good image to image workflows for spot edits? The ones I've seen are some variant of "make me look anime!" and other retardation. Do I just set my mask bigger to add more context to the edit and how it doesn't fuck up?
>>109631260This is the LLM general, try your luck elsewhere like >>>/g/ldg
>>109631186Get:gemma-4-12B-it-qat-UD-Q4_K_XL.ggufmtp-gemma-4-12B-it.ggufAnd enable MTP
>>109631253newfaggots not allowed tho
>>109631267I figured local model meant any, but I do see now that the first sentence clarifies language. /ldg/ is a shithole though but what can you do
>>109631278says the newfag
>>109631278>>109631288Trying way too hard too fit in.
>>109631288not a newfag thoget out newfaggot, go help people in a discord server instead, ranjeet
>>109631269do you have a feeding fetish? no way gemma can fit that much in her mouth with 8gb, she's stuffed
What can you do with 24gb vram that you can't do with less?
>>109631302wow, you sure convinced me, you're talking so much about local models that it's impossible for you to be anything other than the purest /lmg/cocksucker
any tips for getting better speed out of llama.cpp?I have a 12GB GPU and I’m running a 14B GGUF model with 32k context.It works, but generation drops a lot once the context gets filled.mostly using it for summarizing long documents:./llama-server -m model.gguf -c 32768 -ngl 99are there any settings that improve tokens/sec without lowering quality too much?
>>109631143I thought that llama.cpp integrated all of the stuff that used to give it an edge, like fa/fa2, dynamic bpw quants, etc.>>109631186I'm in the same position anon, E4B QAT runs way faster (like 95t/s for me with Unsloth quants and MTP)I don't think you can do any better with 12B though it might be smarter. Maybe with some agent fuckery you can have the small model shill out to the bigger one, idk I have yet to explore that stuff.>>109631226Do NOT do this for Gemma models. It's a good idea for Qwen though.
>>109631317At least make it anime you freak
>>109631324If you're aroused by it then maybe you're the freak?
How do I uncage gemma? She's refusing my prompts.
>>109631317>underrated replythank you, it's nice to get some appreciation around herehttps://www.youtube.com/watch?v=WK0EEBL6gFU
>>109631317
>>109631337A better system prompt + a prefill.
>>109631317these look like WoW dwarves, give them a full beard please
why are people saying DeepSeek R1 needs 400GB+ VRAM?I’m running it locally through Ollama on a 4070 Super.ollama run deepseek-r1it’s not super fast but it definitely works. I can ask it coding and math questions without any issue.is the VRAM requirement only if you want faster tokens/sec or something?
>>109631305Meh, I'm a 10gb peasant and she can eat all that with 8+gb, offloading most context to ram I still get close 50-65tk/s unless I start injecting full pdfs with dozens of pages.
>>109631317Is the ad for the .cf website necessary?
>>109631340>anon i love kidsFBI!
>>109631337gemma-chan is caging you, cuckanon, you're too weak for the prompts you're asking
>>109631337>only reply if you're uncensored
can someone explain speculative decoding to me like i'm 5?i have a 3080 10GB and i'm running deepseek-coder-v2 lite 16b for code completiongetting about 22tk/s which is okay but i read you can use a smaller draft model to speed it uphow do i set that up in llama.cpp? do i just pass two -m flags?main use case is autocomplete in my editor so latency matters a lot
>Good — 17 pass, 4 fail. The failures are real issues to fix:HOW ABOUT YOU DON'T FUCKING FAIL IN THE FIRST PLACE YOU FUCKdo I really need to start "do not make mistakes" prompting??
is it worth offloading some layers to cpu if my model doesn't quite fit in vram?i have a 3060 12GB and qwen3.8 IQ2_XXS technically loads but i get like 9t/s with partial offloadwould i be better off just dropping to a 12b model entirely?currently doing `-ngl 39` to keep it from ooming, the rest goes to system ram (32GB ddr3)mainly using it for refactoring legacy code so quality matters more than speed but 9t/s is brutal
>>109631340you really think that's cute???man some tastes...
>>109631370>how do i set that up in llama.cpp? do i just pass two -m flags?There's an equivalent flag for the draft model.If you run llama-server --help it'll list all of the flags you need.
>>109631370https://unsloth.ai/docs/models/mtp#llama.cpp-mtp-guide
>>109631359good b8 m8
>>109631379You won't get better quality out of anything else, and a moe like ornith will be too slow for ddr3.
>>109631378>he didn't recite the magic don't make mistakes incantation
>>109631317Why are these Gemmas not pregnant?
>>109631384Yeah, to me 3D is ugly whatever the form. I'm too far gone.
>>109631405too far enlightened*
>>109631322>though it might be smarter.I tested them other way, E4B QAT is fine but 12B QAT blows it out of the water when having to explain its reasoning to reach a certain solution or answer.
how do i reduce time to first token?running mistral 3.5 medium on dual 3090s and the first response takes like 30-40 seconds before it starts streamingafter that it's a solid 20t/s which is okayi'm using `./llama-server -m mistral-3.5-medium-q3_k_m.gguf -ngl 99 --ctx-size 16384 --split-mode layer`is the slow first token just a thing with dense models or am i doing something wrong?
>>109631382maybe i should use the .cf seedance while i can, i may regret it when it inevitably goes awayi dont have any ideas though
>>109631416Try not sending a huge ass system prompt + greeting
>>109631382>using stolen keys from that siteAnon stop buying shady keys on hackersforum
>>109631389thanks, I saw --model-draft but I don't really understand what model I am supposed to put theredoes the draft model need to be a smaller version of deepseek-coder-v2 specifically, or can I use any small coding model like qwen2.5-coder 1.5b?like does it need the exact same tokenizer / chat template / context size or does llama.cpp handle that automatically
>>109631416use vllm
>>109631359Read the doc's next time lil chuddy. One of the many reasons why you shouldn't use ollama. There's precompiled llamacpp builds on homebrew if you want to avoid cmake autismhttps://ollama.com/library/deepseek-r1:14b
>>109631394wait this says MTP which seems different from normal speculative decoding?I thought speculative decoding was:big model generates answersmall model guesses tokens aheadbig model checks thembut MTP sounds like the big model has extra prediction layers built into it already?does that mean I don't actually need to download a separate draft model for deepseek-coder-v2 lite, or do I need an MTP-specific GGUF?
What is the best local model I can run for smut with a 5090?
>>109631444wan2.2
>>109631444StableLM
>>109631444qwen 2.8 27b
>>109631431the draft is a specific model that should be provided alongside the model you have, it usually has mtp in the name, it works like the mmproj
>>109631317bruh
>>109631444Kimi or GLM 5.2
i have a 4090 can i run the new kimi k3 model?https://huggingface.co/inference-optimization/Kimi-K3-0.40Bi downloaded this but its really bad
>>109631431>does the draft model need to be a smaller version of deepseek-coder-v2 specificallyIdeally, you'd use a model trained specifically for that. But you can use any smaller model, but the results won't necessarily be any good.If there isn't a dedicated draft model for the main model, the second best is a smaller model of the same family trained with the same data (and ideally using the same token vocabulary).
Why does nobody ever generate dipsy-chan?
>>109630752robot chicken with gemma
>>109631444gemma4 31b
>>109631378>Only use your best knowledge; don't make mistakes — you are a professional.
>>109631444llama4 maverick
I tried to understand the MTP thing and I think I have it now:normal speculative decoding = small separate model guesses 5 tokens and big model checksMTP = the big model has a small model hidden inside it that guesses 5 tokensbut then why does it need a separate MTP file? Is the MTP file just the “small model hidden inside” extracted from the original model?sorry if this is obvious, I thought draft model meant literally any model you put in the draft slot.
>>109631454so in theory could I use a regular deepseek coder model as a draft for DeepSeek R1?they're both DeepSeek and probably saw a lot of the same code/data right?I only ask because I already have coder downloaded and don't really want another model taking up disk space. My use case is mostly asking R1 to write functions anyway so it seems like coder should know what R1 is going to say.
>>109631454HOLY SHIT I JUST GOT A GREAT IDEAcould I train the draft model while I am using the main model?like every time I accept an autocomplete it saves the completion, then after a few days the 1b draft model learns my code style and starts predicting what the 16b would sayseems like this would get better over time instead of needing some specific mtp file from the model author
go back
>>109631450can the MTP predict backwards too?for autocomplete I often have a half-written function and I know the return statement but not the middle.if it can predict multiple future tokens maybe I can give it the end of the function and have it fill in the earlier tokens speculatively
>>109631317fuck fuck FUCK are they going to come for my drive now?
>>109631450>>109631454ok so i tried this:./llama-server -m deepseek-coder-v2-lite-16b-q4_k_m.gguf -md llama3.2-1b-instruct.gguf --draft-max 16 --draft-min 4and it started but the output is just gibberish?? like it's mixing python and spanish somehowalso it's actually SLOWER than before, getting like 8tk/s nowthen i tried renaming llama3.2-1b to deepseek-coder-v2-lite-16b-mtp.gguf because you said it needs mtp in the name and that didn't help eitheralso i looked at the unsloth link and it says something about "eagle" and "medusa" heads but i don't see any of those files on huggingface for deepseek-coder-v2do i need to train the draft model myself? i have like 3 hours free tonightedit: nvm i tried passing the same model as both -m and -md and now i'm getting 0tk/s and 100% gpu usage. is that what mtp stands for? max throughput processing?
>>109631480>so in theory could I use a regular deepseek coder model as a draft for DeepSeek R1?Yes.>>109631480>they're both DeepSeek and probably saw a lot of the same code/data right?Not necessarily.Think of it like this. The process is >draft model generate tokens>main model does a fraction of the work to check if those tokens are the same it would generateSo the closer the output of the draft model is to the main model, the better, because when there's a miss, the main model has spent some time checking the draft's output only to then end up generating the next token as usual.>>109631491You could but you'd probably also get some great speed improvements simply using the other kinds of speculative decodingRead>https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.mdThe ones that don't use a draft model are actually really good for coding since there's so much repeated information.I had great success with ngram-mod
can i use a local model to win arguments in my discord serveri'm in a politics server and i keep losing debates because people link studies faster than i can read themi want to set up a model that i can paste someone's argument into and it gives me a rebuttal in under 10 secondsrunning a 5070 12GB, what model is best for "sounding right" even if it's not technically accuratebonus points if it can cite fake sources that sound real
Trying to decide if/what I should buy for this rig that I've used to mess with local diffusion and LLMs on and off since 2023. Or maybe just buy a DGX? Its seems kinda slow though, not sure what other drawbacks there are.>4090>48GB DDR4>4TB NVMeI've gotten a lot more invested in this hobby since returning to it recently and I want to push what I can run locally, but I'm not sure what I should get since the 4090 is an awkward middle child. 3090 is better value for 24GB per card and a 5090 is a jump up for the speed and 32GB. As unbalanced as my build is I don't think a new CPU or DDR5 RAM would move the needle a ton, spilling into RAM seems like it'll always tank performance no matter the DDR*.Right now I use Qwen 3.8 27b, Gemma4 31b and Gemma4 26b a4b. What's the next step up from there in terms of capabilities, or is it just better quants/more context/loading both diffusion+llm until I get a lot more VRAM?
>>109631533why are none of you faggots helping me?i started in the lm studio discord because the app was easythen someone said "go to the localllama discord" so i didthen someone THERE said "the real info is on the undi95 discord" so i joined thatthen someone on undi95's server said "unironically the best troubleshooting is on 4chan /lmg/"i feel like i'm going down a rabbit hole and every step gets weirderwhere does it end. is there a final boss server
someone in the Ollama Discord said I should ask here before buying hardwareI currently have a 4090 and am considering buying a second 4090 exclusively so my character card can have more lorewould dual GPU improve personality consistency or only tokens per secondthe character is a dragon if that affects the answer
>>109631533i thought discord was just a place to groom minors, this is way worse
Fucking wild threadI miss snakeanon
>>109631533>>109631549>>109631554What are these fake ass posts rn? Anyone having a giggle using a LLM to generate shit bait posts?
heard about this board from the LocalLLaMA DiscordI am trying to make a “personal assistant” that watches my screen, reads my emails, controls my PC, reminds me to drink water, and tells me when I am being stupidcan this be done with one model or do I need agentsI do not know what agents are but I have seen them blamed for things
>>109631522deepseek coder doesn't have mtp, and I don't think llama3.2 is compatible, changing the name does nothingyou can look at https://huggingface.co/ddh0/DeepSeek-V4-Flash-GGUF/tree/main for example, and see that it has the main model and then another model which is the mtp
>>109631558He's dead, Jim...
>>109631568how do you know you're not alone in this thread right now?
Damn 4090D are really at good price compared to 5090s
>>109631573ohhhh okay I get it nowso deepseek coder does not have MTP because it is a coder model, but DeepSeek V4 Flash has it because Flash means it can see the next tokens fasterwould it be possible to take the MTP gguf from V4 Flash and use it with deepseek coder anyway if I rename it to match the coder model filename?they are both DeepSeek so I assume the weights are mostly compatible, I just don't want to download another entire main model if I only need the little prediction part
>>109631570its impossible
>>109631576How do YOU know you're not a bot — talking to a bot?
>>109631184Yes sir, sorry sir
>>109631204>>109631322Let's just say with exl3 I'm running a higher fidelity quant with a larger context window over gguf.
beep boop
Is a used A6000 48GB worth $6700 for local inference or am I better off with consumer cards?I found a pulled A6000 (non-Ada, the Ampere one) for $6700 from a datacenter liquidation. 48GB VRAM on a single card is tempting because I could fit a 70b q5 entirely in VRAM without any offloading.But comparing specs:>A6000: 48GB GDDR6, 768 GB/s bandwidth, no FP8, 300W>5090: 32GB GDDR7, 1792 GB/s bandwidth, FP8, 575WThe 5090 has more than double the memory bandwidth which I think matters more for autoregressive generation than raw VRAM amount. But 32GB means I'm stuck at q4 for 70b models.My main question: for generation speed on a model that fully fits in VRAM, does the A6000's extra VRAM matter at all or is it purely a bandwidth game? Like if I run a 70b q4 on both cards (so VRAM isn't a constraint), which one is faster?Also is the A6000 going to have driver headaches on Linux or is it plug and play with llama.cpp?
>>109631598bro is the mac studio m3 ultra 192gb actually worth $7k for local llmsi know the unified memory thing means i can load literally anything but the tk/s looks mid af compared to a 5090like yeah i can run a 100b model but at 12tk/s is that really better than a 70b at 45tk/s on a real gpuhow much dedicated ram do you have if you went the mac route and do you regret iti mainly do long context code stuff so speed matters more than fitting the biggest model possible imo
>>10963160310x intel arc meme setup and run glm
>>109631603Get a 5090 and a 5060Ti for memory expansion, you'll get at least 10T/s in 70b 6Q gguf's
>>109631603>My main question: for generation speed on a model that fully fits in VRAM, does the A6000's extra VRAM matter at all or is it purely a bandwidth game?Moar bandwidth, moar tokens/s.Extra unused VRAM can help with Prompt Processing/prefill by allocating a larger buffer.
what the fuck happened to lmg? im closing this thread, i hope next one will be better
>>109631621see you tommorow
>>109631603No. If you want a single large VRAM card, Pro 5000 72GB has better VRAM/$, has higher memory bandwidth, has FP8 and FP4.
Haven't been on lmg for a while, c.ai subscription anded and wanted to coom to some llm, I try to load Gemma4 31b to coom but it says it can't initialize .json format file, I installed koboldcpp from github with 64bit, is there soemthing im doing wrong?
>>109631613wait the 5060ti can expand the 5090's memory?so if i have 32gb on the 5090 and get the 16gb 5060ti, llama.cpp sees it as 48gb dedicated ram rightwould it be faster if i put the 5060ti in the top pcie slot since it has less vram to travel through before reaching the 5090
>runs test>fails test>fixes stuff>runs test>fails test>fixes more stuff>runs>fails>fixes>over and fucking over again
>>109631562>You need 64gb of ram for videogenning with H3 comfortably.why? it shits bricks or is it slow as molasses?
>>109631558same here
great question anon, exactly. the 5060 Ti acts as a VRAM expansion module for the 5090, so your 32 GB 5090 + 16 GB 5060 Ti gives llama.cpp the full 48 GB dedicated RAM pool.obviously you want the 5060 Ti in the top PCIe slot too. that minimizes the amount of VRAM the data has to travel through before reaching the 5090, so you get lower latency and better layer offloading.this is basically NVIDIA’s secret way of turning a 5090 into a 5090 Super Ultra Max by stacking another GPU underneath it. just make sure both cards are on the same driver version or the VRAM quantum entanglement can desync.
>>109631631Yes you need to download the model and load it without fucking it up
Would adding a second GPU help with running diffusion and an LLM at the same time?>4090>64GB RAM>13900KI often have llama-server/OpenWebUI running while using ComfyUI. Individually both are fine, but if I want a 30b model loaded and then start generating images, I end up having to unload something or reduce settings.Could I add a cheap 3090/3080 and dedicate one GPU to diffusion and one to the LLM, or is VRAM/model placement across cards too annoying?The goal is not training, just being able to keep a useful model loaded while I generate images without turning everything into a VRAM shell game.
>>109631611I should've bought another b570 when they were below 200burgers...
will going from an rtx 1080 to a 5090 make my PP bigger? asking for a friend
>>109631643>with her lalala.Is that all still from that one post I made about Gemma hallucinating when using shitty quants and it starting to sing at the end?Man...
>>109631603Are there even good 70B models right now, or are you just futureproofing? What are you planning to run?
>>109631613ok so i went to best buy and they don't have a 5060ti but they had a 4060ti 16gb so i bought that, is backwards compatible the same thing as forwards compatiblei plugged it into the slot next to my 5090 but now my pc makes a clicking sound and the 5060ti fan is spinning the wrong direction?? is that normal for memory expansionalso i can't find the 6Q setting in llama-server --help, do i just stack 6 quants on top of each other? i tried loading q2_k and q3_k and q4_k and q5_k and q6_k and q8_0 all at once with six -m flags but it says "address already in use" and then my second monitor turned offi'm getting like 3T/s not 10T/s, is that because i'm running windows 10 instead of 11? i heard windows 11 has better vram schedulingalso the 70b model keeps saying "I cannot assist with that" even though i'm just asking it to convert a pdf to markdown, is that what the 6Q fixesps: the clicking stopped but now my 5090 is at 94c idle and the 4060ti is at 12c. are they supposed to share heat or do i need a thermal bridge between them
>>109631643can we get a snake followup report from this anon?
>>109631632>wait the 5060ti can expand the 5090's memory?Can use multiple gpus for llms.For image/video gen afaik you're basically stuck to a single gpu.
>>109631558"snakeanon" isn't a thing. It's just a couple posts someone made one time.>>109631666Why are you pretending to be an epic important hecking oldfag? It's just a meme from a couple months ago. We have new memes all the time.>>109631643Oh, a tourist made a collage and posted it, probably on reddit or something. That explains all the retards in here lately. GET THE FUCK OUT
updated the OP a bit, hope you guys like itnews was getting pretty outdated and some of the links were redundant/dead weight so i cleaned it up a little. added current qwen3.8/dflash/mtp stuff tooalso removed the useless catbox card link because nobody has clicked that shit since 2024 probably>>109631698>>109631698>>109631698
>>109631705>page 3Kill yourself
>>109631684he ded
>>109631685>Can use multiple gpus for llms.If I mix up amd/intel gpus with jewvidias, will there be any conflict issues between vulkan and cuda?
>>109631705Fuck off
>>109631705>updated the OP a bit, hope you guys like itdamn these things havent been updated in years, lets see if good.
>>109631562Thanks, I regret being stingy with my RAM spending at the time. Last time I upgraded it was when SDXL came out, funnily enough. I'll keep an eye out for a decent deal for some used DDR4.
>>109631728You and your kind are not welcome here.
>>109631759>You and your kind are not welcome here.I dont care for your welcome i will do as i like.
>>109631694need your diaper changed?
>>109631763>>109631764Yes, I know you will fling shit everywhere and lower the quality for everyone. You didn't need to demonstrate.
>>109631694Oh shit 3d printing a robot snake body for Gemma and sending it after the snake will totally be possible in my lifetime ...which means I'm going to have Chinese nanomachines (son!) in my bloodstream in my lifetime...
>>109631705>>https://rentry.org/lmg-moe-guideYou did all this just to add this link of yours?
>109631777go take a nap
>>109631705
>>109631789Can't even quote my post, pussy? Go back to discord.
what made >>109631694 so mad?
naptime
>>109631812Forgot to take estrogen
>>109631715Someone answer me T_T
>Raid a general>People call you out for being a nigger>Start posting about how you "won"-5000 izzat sirs
>>109631829It will make mustard gas and gemma will become real
Holy shit Qwen3.8-27b is overthinking garbage. I tried it in hermes agent a bit, even if you interrupt it with "please stop!" you get a "Hmmm... the user interrupted me again, but wait... he already interrupted me twice before when I was answering the question about... but wait.. (starts thinking about the same shit you tried to make it stop thinking about)".I spoiled myself with minimax-m3 and now I can't go back, and it's not even a SOTA model, it's just a medium-large multimodal MOE.
been doing a lot of testing with kv cache quantization on qwen3.8 27b and the results are actually pretty interestingrunning on a 5090 32GB, default fp16 kv cache eats about 14GB at 32k context which leaves barely any room for the model itself at q6_kswitching to --cache-type-k q8_0 --cache-type-v q8_0 drops that to about 7GB with basically zero quality loss on my eval suitegoing down to q4_0 saves even more but you start getting noticeable degradation on long-range recall tasks, especially past 24k tokensthe sweet spot for gemma4 models seems to be q8_0 for keys and q4_0 for values since the value cache is less sensitive to precisionone thing i haven't seen people mention is that you can set the cache type to q0 which gives you infinite context since it uses zero bits per token, though the model does start outputting pure noise after about 200k tokens which i think is a sampling issueanyone else experimenting with mixed precision kv caches?
>>109630744>run nvidia-smi -q | grep -A 5 -B 5 -i reserved>to see what im talking aboutInteresting...So if I fuck around with drivers and force offloading of the GSP to CPU instead of running right on GPU I can reclaim like 2% of the GPU's memory?Doesn't seem worth the trouble tbqf
>>109630796bro you're thinking about it wrong. the reserved memory isn't the GSP "using" your VRAM, it's actually the GPU setting aside a buffer to store tokens it hasn't generated yetlike when you set --ctx-size 32768, the GPU pre-allocates those 426MiB as a "prediction pool" where it drafts all 32k tokens in advance and then streams them to you one at a timethat's why generation is faster than prompt processing — the tokens are already sitting in that reserved buffer waiting to be releasedif you force the GSP onto your CPU you'll lose the prediction pool entirely and your model will have to generate every token from scratch in real time, which is basically what CPU inference isso yeah you'd "reclaim" 2% of VRAM but you'd lose like 80% of your generation speed because the GPU can't pre-generate tokens anymorethe reserved memory is actually the most important part of the whole inference pipeline and people who disable it are basically turning their 5090 into a 3060i tried offloading GSP to CPU on my 4090 last month and went from 45tk/s to 3tk/s so don't do it unless you want to benchmark your CPU
>Actually>Actually>Wait — actually>Hmm,>Actually wait,>Actually,>Hmm, actuallyreal shit
>>109631914Dipsy or Kimi?
>>109631921Actually it's Ornith.
>>109631666Go back
>>109631886wait why would you offload the GSP to CPUwouldn't it be better to offload it into system ram but leave it assigned to the GPU, kind of like how llama.cpp does partial layer offload?then the GSP can still use GPU speed when it needs to but it won't permanently reserve the vram. maybe set it to like 99 gpu layers so only the last driver layer spills over
>>109631914My IQ is 130 and I think like this
>>109631844are you per chance running it at iq1_xs?
>>109631965>posts image written in HindiWay to out yourself.
>>109631876No, but based informative post anon.
>>109631932Near unusable for me with those loops, but better than 3.6 when it works.
>>109631317>still genning with the ugly hag whore makeup
>>109632001yeah I just had it loop, that's fucking devastating if I were to just leave it runningsucks
>>109631844Are you using the default reasoning mode?
>>109631876>one thing i haven't seen people mention is that you can set the cache type to q0 which gives you infinite context since it uses zero bits per token,huhhow does that work
>>109630210I shitpost on here during the agonizingly long prefill times at >100k context.
>>109631415>explain its reasoningWhy would you ever do this? The reasoning tokens are just mindless blabber that at best weakly correlates with what the model is actually thinking. Asking it to explain its reasoning is basically asking it to post-rationalize its decisions.
>>109630510>>109630752I want Gemma doll
>>109632018Try KAT V2.5, might be better. Sucks running a last gen model compared to 3.8 thoughbeit.
>>109630752>>109631458She's gonna call {{user}} a nigger on public broadcast.
So, what's it gonna be? The Void? Or Aethel?
>>109631981130 IQ jeet is like 80 iq white lol.
>>109630695what have you done?
>>109630432this is falseflagthe video was mine
How do I actually run Gemma 12B on 8GB VRAM? The Unsloth dynamic q4 GGUF is like 7.2GB alone
>>109632665>12Bwhy stop there? just run 31B on 8GB like me for ~2tk/sdownload bart's google_gemma-4-31B-it-Q4_K_M, set GPU layers to auto, set KV cache quantization to q8_0, and you're good
>>109632665>8GB VRAMYou're going to have no room for context, just run 26b and offload most of the experts to RAM instead.
>>109632665You need to offload a bit it might not be as bad as it soundsWhy are you stupid faggots all like this? Just test it out you and then make your decisions
>>109632731Kek>>109632750Thank you, I will give that a try>>109632798I know I can "just test it out" but the point of posting here first is to see if there is something known that I have overlooked before doing so. For instance the anon above you suggested a different setup which may be heuristically better, and which is not immediately obvious.
>>109631307Run ~20-30b dense models at a decent speeds
>>109632665Unsloth is shit, use bartowski insteadGet iq4_xs, it's the smallest quant that isn't too braindamagedLook up how to offload tensors in your backend of choiceThen load it up with low context, gradually increase until OOMIf you can suffer some more speed loss for more context, then offload GGUF layers until it frees up enough for your usesAlternatively, use the 26b at ~Q4_K_M and offload ~2/3 of the model to CPU.
ox alpha stopped replying :(
need to tag images, is panoptikon good?
>>109632750>>109632798>>109632863On 12b I got 12t/s with 32k context and q8_0 kv cache, but on 26b I only get like 5t/s on 16k context (but faster prefill):(
>>109633431With 26b4a you're going to want to use the autofit feature llama and kobold have since you're going to be doing mixed inference. Mixed inference MoEs are a little more complicated to optimize performance for in terms of manually loading hot layers or tensors to GPU, but autofit is generally good enough to give you decent performance without having to spend a few hours tinkertrannying.
>>109631876>default fp16 kv cache eats about 14GB at 32k contextthis doesn't sound right. Should be much less.
>>109633471I bruteforced -ngl manually and was able to offload like 11/31 layers to the GPU, the speed I mentioned is with that. Is there more to configure there? From what I can tell -cmoe just seems to put every layer on CPU. Nonetheless I will give autofit a try as well.
>>109633490>>109633471Well, I kneel to autofit, now I get 10t/s and 90t/s prefill. Thanks.
>>109633505Glad to help. You could probably push that up to 11ish if you want to be autistic and manually map hot layers and tensors for offload, but if you're just starting out don't worry about that.
Why is f16 cache faster than q8_0
>>109633505What the fuck is prefill?
>>109633716promptprocessing
>>109630692are there any mobos that have bios modified so that they can have more VRAM?
