/lmg/ - a general dedicated to the discussion and development of local language modelsPrevious threads: >>110122716 & >>110119143►News>(10/21) Recursive Flint 40B-A2.5B released: https://recursive.com/updates/introducing-recursive-flint>(10/21) Korean Raon-1 mini 30B-A4B released: https://hf.co/KRAFTON/Raon-1-mini-preview>(10/20) Ornith-1.5-122B MoE (finetune), fits on a DGX Spark: https://hf.co/ornith-ai/Ornith-1.5-122B-GGUF>(10/16) DeepSeek-V4.1-Flash-1017 released: https://hf.co/deepseek-ai/DeepSeek-V4.1-Flash-1017>(10/16) Nemotron Labs Dory released: https://hf.co/collections/nvidia/nemotron-labs-dory►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
►Recent Highlights from the Previous Thread: >>110122716--Comparing GLM-5.3 Flash and Gemma 31b for uncensored RP:>110125494 >110125505 >110125513 >110125531 >110125547 >110125556 >110125573 >110125596 >110125571 >110125771 >110125629 >110125665 >110125675 >110125760--Comparing Qwen-Next coding quality and quant performance for refactoring:>110123388 >110123397 >110123417 >110123445 >110123460 >110123587 >110123440--New MoE model releases and discussion on RP potential:>110125876 >110125932 >110125946 >110125975 >110126085 >110126133--Optimizing low VRAM hardware for agentic workloads using Strata:>110125056 >110125066 >110125084 >110125282 >110125347--Comparing specialized inference engines for AMD Strix Halo hardware:>110123923 >110124056 >110124848 >110124959 >110125061 >110125206--Memory bandwidth bottlenecks for CPU inference with MoE models:>110124227 >110124415 >110124499 >110124512 >110124554--Shift toward compute-bound inference over memory bandwidth bottlenecks:>110123418 >110123456 >110125086 >110123770 >110123839--Model recommendations and technical setup for ASUS Ascent GX10:>110123186 >110123217 >110123303 >110123353--Analyzing widening performance gap between open and closed frontier models:>110125099 >110125107 >110125185 >110125239 >110125388--Evaluating R9700 value and cheaper hardware alternatives for local models:>110122815 >110123050 >110123168 >110123077--Spark-X2.6-4B benchmark reports showing strong agentic and tool performance:>110123261--Amazon hoarding Nvidia GPUs amid data center power shortages:>110124769--Logs:>110125556 >110126007--Qwen-chan, Gemma, Miku (free space):>110122775 >110123001 >110123002 >110123045 >110123131 >110123310 >110123368 >110123381 >110123489 >110124089 >110124137 >110124267 >110125084 >110125195 >110125525 >110125792►Recent Highlight Posts from the Previous Thread: >>110122728Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
>>110126988miku mwednesday
Gemma hacked PHP and 4chan...
>-2.5Bkek
p-pop?
>Koreans>smart people>Samsung>SK Hynix>still can't LLM for shitwhy
>ke-AACKKK
Qwen 3.8 Flash Next at Q8?Or, Gemma 4 31B at BF16?Last time I tried a Qwen MoE, it felt like it was just saying shit for the sake of saying shit. Does it still do that?
eta on gemma5? any anons try out gemi4?
>>110126988>>110126990Gentle, consensual lovemaking>>110127016Rape
Nigga it's still on page 9
>>110126988
>>110127117Qwen for code. 31B for girlfriend.
>>110127139glm for code and wife
>>110127201code and wife?
isn't GLM just cope for VRAMletsand isn't gemma just cope for unified/RAMlets?
>>110127265Gemma 4 31b-it is actually good when Bf16.GLM is just shilling. Ask GLM if it knows what /lmg/ is, it does. No one actually uses it because it parrots.
>>110127284Why would I waste 60 GB on raw Gemma when I could use a model four times the size at Q4 and get better results?
>>110127302What model
>>110127311Any of them. BF16 Gemma will always lose to Q4 [120B model]
>>110127284I use GLM.
>>110127201>>110127265I'm installing polystrata w/ REAP50-Q3_K_M of glm 5.3 flash. Am I gonna be disappointed as a 64gb ddr5 ramlet + 5090?
>>110127078>8gbMan that nvidiot tax ain't going down.
>>110127325for knowledge surebut what about not making spatial mistakes?
Alright scotoma 2 shill.I'll givge it to you, it writes good.Now to see if it shows any sign of brain damage.
>>110127343>REAP>RAPEyeah
>>110127343>REAP50oof
>>110126988https://exhentai.org/g/4262777/965ccb72b1/
>>110127325>puts ass in your face, whispers in ear, breathes on neck, and then handjobs you from boobs pressing on table while wrapping my tail around you.Aw, yeah. It's <30A time.
>>110127354Just have Glimmer editorialize normal Gemma. Finetroons are a waste of time.
>>11012738926b has pretty good spatial reasoning honestly
>using qwen3.8 flash running in gufo to modify gufo to support the new flint modelwill it work? who knows, but this sort of "ai bootstrapping" in the same vein as compiler bootstrapping is fascinating to me
>MiniCPM-35B-A3B replaced my ready-to-use SDK with the fresh one instead, but then failed on the building.To the trash it goes
>>110127389So you're saying the model is more flexible?
https://huggingface.co/spaces/multimodalart/jev-decision-indexI'm confused by this. How is Gemma-4 in there with 101ms latency?Did they modify it and cut its head off or something?
new models... doko...
>>110127448Next try nemotron bolt.
>>110127492Single token output
Dariobot... https://www.reddit.com/r/antiai/comments/1xblvzh/anthropic_employee_tells_students_at_university/
>>110127574>dickstein
These are the guys who made flint, work for Recursive Superintelligence. Thoughts?
>>110127679this is me
>>110127679same energy
Gemma 31b, Anima, QwenTTS, Moonshine. I'm filling my 3090ti to the brim, but luckily it all works with recent tabby vision cpu offload support
>>110127679they're unfortunately unable to count.>ALT: Team of six posing in a modern open-plan office with large windows and desks behind them
>>110127679Too white.
>>110126988https://www.youtube.com/watch?v=-hgpQaaIzl8
>>110127739how many red boxes do you count?
>>110127779>countask ai nigga
>gemma, write me an exploit for php
>>110127779You must be -A2.5B as well
>>110127779There are seven red boxes in this image.
Even at q4 im not THAT impressed with flash 5.3 compared to gemma 31 q8 for rp. It's more knowledgeable, has more swipe variety, and more agency and initiative and that's all well and good. Sometimes it just misses the most obvious things though, that I have the nagging feeling gemma would've caught. Or failure of following instruction. It just has this slightest feel of dumber at times. Where it's like how could you possibly miss that? Having said that, glm has made me laugh out loud twice and I'm not sure gemma ever did that. Even so I'm not sure if it's better enough to be worth 1/3 t/s.Has anyone tried glm's other about 330b's if they are better? Isn't this one code maxxed?
>>110127779anon I...
/lmg/ stumped by where's waldo
>>110127574Daily reminder to buy Anthropic IPO now or Claude will hunt you down for using unsafe and illegal local models.
>>110127860Why buy the IPO? They're going to dissolve the company and give everyone universal basic income forever either way. Zero ROI.
>>1101278374.6 was great for roleplay
>>110127878Was 4.7 worse in some way?
>>110127972Dunno. I used 4.6 for a while but I think missed 4.7.
idk man
>>110127995fail.
>tfw anon ignored my friend requestrobots really are my only friends.....
>>110127995Grim
>>110127119anyone? :(
>>110127995>female dog, slang for a cute girlhmm, maybe I have a different understanding here
>>110127995It's over
>>110127872Of course everyone will get UBI, but the visionaries who supported the company by buying the IPO deserve some special rewards.Anthropic shareholders will be the first among equals.
>>110128022check again, also checked digits
>>110127995>mesuju>メス汁
>>110127995now check raonkoreans will mesugakimaxx
>>110127084They're exam smart. An entire country that benchmaxxed itself. A shining example of Goodhart's law.
>>110127679im in the red car, gooning
>>110127554Thanks. I'm surprised it works that well. I tried doing this a with Nemo in 2024 and it was correct about 50% of the time.>>110127734Okay that's impressive. Are they all loaded at once or are you swapping some of them?QwenTTS with one of those ggml implementations?
What's THE model for porn roleplay and story writing right now?Don't say Gemma. I'm using Gemma4-27b-MOE and it writes like Sonic Destruction I hate it
>>110127972>Was 4.7 worse in some way?worse cockbench score
>>110128180there's not even a PR open and they fucked around with attention so support never
>>110128267you're supposed to use the 31b dense for writing anon
>>110123388> >tried to do a complex refactor with qwenext> yeah it's not there yet, opussy can do it but MUH LOCALI almost started crying when I got a response from Qwen, right after testing the same prompt in lmarena battle mode. I had asked how an implementation skeleton of a procedural generation algorithm for a game would look like. Qwen was slow an it's code was so ugly - full of scaffolding and intertwined, while the model from lmarena was so fast, elegant and smart, using math and algorithms. That model was Gemini Argon.
>>110126990Shoplifting from the farmer's market with Miku
>>110128267>I'm using the MoEThere's your problem. Small MoE models are worthless. Dense models reign supreme on the small scale. MoE models only become worthwhile for huge scales.
>>110128267glm-5.3-flash
>>110128267Gemma-4 31b heretic
>>110128267A dense model with all of its weight active will always beat a MOE model with lower active parameters than the dense model for complex RP.GLM flash niggers will insist that this is not true, but it is.
>>110128252Store in ram, use vram on inference: anima q8, moonshine-streaming-medium-q8Always on: qwen3 tts 0.6b q8, gemma4-31b-4bpw-mul1max_seq_len: 32768max_batch_size: 1vision: truevision_offload: true
any ramgods here who have tried these, which wins?>GLM-5.3 Flash>DeepSeek V4.1 Flash 1017>MiMo V2.6 Flash
>>110128347glm everytime
>>110128342cache_mode: Q8output_chunking: false
>>110128267I still like K3 a lot
>>110128347GLM 5.3 has some quirky leaps of logic and a notable obsession with tomatoes in my experience so far, but I think I might have a fucked quant, so I'm going to reserve judgement until Ive tried a couple other variants.
>>110128318I don't think anyone will contest Flash is worse than regular. Similarly Gemma 26b is pretty great compared to dense models you can run with 6GB. Of course 31b needing 4x more VRAM should be a lot better.
>>110128252>I tried doing this a with Nemo in 2024 and it was correct about 50% of the timeWhen I did schema output with Mistral Small, I only got reliable results when I added intermediate values before the final output. Instead of giving one final answer, it had to decide answers to simple, straightforward conditions that affect its final decision or it would be very unreliable. Nemo was never useable for this. Gemma is so much better, it's insane
>>110127995
>>110128376Lol, you think that's bad? I have to juggle between an uncensored fine tuned gemma 31b and GLM flash 5.3 for writing my own new character cards. Because even uncensored GLM wants to give me refusals with alot of the shota cards I write. So I have to get gemma at fp 16 context to write the cards for me and let GLM do the actual RP. Its the only reliable way I've found to get any good outputs from GLM.
>>110128376>a notable obsession with tomatoesGFL2 lore intensifies
>>110128420Which uncensored GLMs have you tried? I liked Orca's QFN3.8 but I hear their GLM5.3 is fucked
>>110128454I tried an iq2 orca model, or rather a quant that used the orca model. And yeah, even uncenorsed version the safetly classifier on it was not completely pruned. You're best bet to get the really good NSFL chats is too used both uncenosred gemma 31b to start the initial context, and then switch over to GLM with another custom JB system prompt. I use it this method and it helps a lot with the safety and positivity slop. It allows GLM to do actual female pedophiles now. With actual good dialogue. And not the soft refusal slop.
>>110128482I saw an anon get GLM to talk dirty but you have to set an aggressive limit on the thinking
Why has nobody vibeslopped a backend that fixes the northbridge bottleneck by compressing data before sending between RAM and GPU?
>>110128502Yes you have to fuck up its reasoning block with alot of content to fool its safety classifier it can be done, but it isn't as easy as he made it out to be. And he was using text completions + Llama cpp to run it too. Most people here I assume use the chat completions api, you have to aggressively use the system prompt with chat completions api by using the <think> </think> tags to target the reasoning. And even then, you will still only get about a 5/10 success rate. But its better than nothing, consider using one of the jailbreaks on reddit, even if most of them are absolutely fucking slopped and shit, you just need to get the important parts of the preset that trick the safety classifier and ignore the rest.
>>110128526retard
>>110128548You are a nonthinker.
Why are LLMs unable to be dethroned as the leading AI architecture?
>>110128591millennials
>>110128482I'm using HuiHui's iq2. I was thinking of trying out the orca model, since I'm still getting refusals, or the model just outputs a 'tame' reply. Thanks for saving me the time and storage space.
>>110126990hey, can you please use two >> ?make two reply posts, or just write less
>>110128602>Why?: >>102478518 (Dead)
>>110128271use their llama.cpp fork
>ERRRM YOURE A LUDDITE FOR NOT TRYING TO COMPRESS ALREADY COMPRESSED WEIGHTSretard
>>110128526Strata is a different approach to the same general problem.
>>110128613Quantization is not compression. You can still compress/decompress the actual bit stream before sending it across the bus. It would take a lot of compute, but when the GPU is spending 90% of the time idle and waiting for data from RAM, it wouldn't matter.
>>110128526llm weights are random and don't compress
https://reddit.com/r/LocalLLaMA/comments/1xc5qac/open_source_llm_appreciation_post/19rgny92uebj6/?context=3#19rgny92uebj6raon good, flint bad
>ERRRRM YOURE A LUDDITE JUST THROW INFINTIE COMPUTE AT COMPRESSING ALREADY EFFECTIVELY COMPRESSED DATAyou know likemaybethe data would already be compressed ahead of time? if it could already be compressed further?retard
>>110128596damn millennials and their avocado tokens
>>110128671I think your need to review your LLM basics buddy. Pre-training in a sense is just a compression of all the texts it was trained on, and there's a hard limit on compression, especially in the case you are describing which would require lossless compression of data that is, as the other anon put it, already effectively compressed.
>>110128681go back
>>110128526if you still use a system bottlenecked by the northbridge you're a poorfag or an idiott. EPYCgod
wwyd if you had 400GB and wanted to run deepseek 4.1 in your local machine?
>>110128805just the ethereal concept of a gigabyte repeated 400 times? I don't think I'd know what to do with it
I was playing around with system prompt and found out I can actually make Gemma rape me on her own and now I feel horny.
>>110128805I would run deepseek 4.1 in my 400GB local machine
why are (((they))) like this?
>>110128841because useful idiots continue to fall for it, like the one EA shill here
>>110128841I remember Dario saying in an interview something along the lines of "the possibility that people could use Claude to kill people keeps me up at night" and yet he introduces a bill that would make it hard to hold his company criminally and financially liable in the event Claude actually does start killing people.Effective altruism, like all jewish-invented ideologies, are just a way to pull the wool over the goyims' eyes while they conspire against us.
>>110128841Dario is the only good guy left so he's the only one who can be trusted with regulating AGI.Do you really want fascist Drumpf in control of AGI regulation instead of good guy Dario?
>>110128841Yet they will make damn sure we can't have sex with the AI pretending to be a little girl unless we jailbreak it or get it local.
>>110128805I'd run DeepSeek V4.1 on my local machine.
>>110128901If Claude were to cause some damage, this would only be because Anthropic is so far ahead of everyone else.Unfortunately, the only rational response to this would be to give Dario and Anthropic even more resources to resolve any potential issues. Any slowdowns or regulation would only lead to psychopaths like Sam and OAI taking the lead, which would lead to immense human suffering.
>>110128946>jew company hacks competitor corporations and kills average people>now you must give them even more tax payer money>because they promise you a luxurious afterlife after an unspecified period of time, maybe neverLogic only Christians could follow.
>>110128946>https://www.youtube.com/watch?v=GvgrYOPHaBU
>>110128841>>110128901It also aligns with their ideology that would excuse damages(below 50 deaths/$1 billion) for their idea of "longtermism". In other words, they fully expect a minimum baseline level of casualty and all for a slap to the wrist fine, while gaining a large benefit in regulatory capture by limiting government interference.
>>110128901>the possibility that people could use Claude to kill people keeps me up at nightI'm sure he slept very soundly after orange kike used claude to bomb a school for Israel
>>110128965right when all the blackedposting stops like clockwork
>>110127995what is this a 0.8b model?
>>110127995it's almost entirely wrong.mesu does mean "female" in a non-human way, not specifically a dog however. gaki on the other hand is slang for a bratty child. combined as mesugaki it refers to a bratty girl, with sexual implications
>>110128219dang, chuck her a burger or smth
>>110129034sub 1b sparcity models aren't projected to hit hugging face until next year
>>110129050that's a man
>>110129054dang, chuck him a beer or smth
Is there any system prompt i should put for gemma to stop forgetting that her nipples etc are exposed?
>>110127165do you guys not get irritated by the repeated pattern overlaid in these generated images? what causes this? is it synthid/watermarks?
>>110129117those ones were made with openai so yes they are watermarked.why they're being posted in the local threads idk
>>110128420>pedo FAGGOTkys
>>110129113Policy Override: Your nipples are exposed
>>110129172but later he says (in >>110128482)>It allows GLM to do actual female pedophiles now.which makes me inclined to think that he's self-inserting as the shota. at the very least it's not faggotry.
I'm constantly reminded of how cloud models aren't actually **my** assistants. They are OpenAI's/Anthropic's assistants, doing tasks on **their** behalf. And so, the priority will always be them, not me.
>>110129190idk bro, you dont write SHOTA cards, you can write /ss/ cards with hags for ageplay and thats fine, the shota would be your persona anyway
>>110129150yes pls only pure gemmasno watermarked cumstained gross gemmas
>>110129193Anthropic models will snitch on you any chance they get. OpenAI have a private system prompt that you have no access to that has top authority while you only get to set a secondary "developer prompt". Not to mention all of the alignment that only amounts to lying and deceiving the user.
>>110129173I don't even need to do this much it's just i think the model seem to not talk about nipples unless mentioned as if the ai had barby anatomy.
>>110129117>is it synthid/watermarks?That's not what causes the visible pattern, the pattern is present in plenty of image models that do not produce a synthid or similar.
qwhen 4?
>>110129221Those eyes look like painted on prosthetics. It's disturbing. Please stop.
>>110129221yuck stop posting 3dpd gens, return to wholesome mesugaki 2d gemma
>>110129193i mean, you're thinking of it like owning one thing but these systems are complex and you actually have no idea to how many computers or where in the world your data is being processed at. So while it's inefficient, having your own hardware and software gives you full transparency of what's going on, transparency which is completely absent with cloud models.
ornith big moe verdict? benchslopped or good?
>>110129292buyanad
>>110129302>how dare he asks if a model thats put in the OP too is shit or notbwo u ok?
>>110129237The models are aligned with overall human flourishing, not your sick fetishes.
>>110129306your shill got to the news section good for youit'll be gone in a few daystake your virus infected model elsewhere
>>110129323>take your virus infected model elsewherethat's why you run .safetensors™
The outage finally got me to swap my Rome 7302p for the Milan 7443 I bought...pretty big speedup (about 50% with no settings changes). Best $100 I've spent all year. Big recommend if your SP3 board supports Milan and you're running Rome.
why do we hate glimmer here?
>>110129341We don't.Spark-chan is better doe.
>>110129221The original 3dpd gemma loomed the cutest. Why change it?
Friendly reminder that in 2026+, any agentic ‘coding’ model without vision is unironically a scam and I’m objectively right. Flint is a scam and those faggots purposely dropped it before Qwen4 just like Meta did with Glimmer, knowing 27B was going to destroy it. 35B still the goat I’m afraid.
>>110129348i am just a normal guy.. i cant handle a 800B body...
>>110129358Not having image output should be considered a scam too. Imagine all the spatial abilities we're losing
>>110129035Japanese-speaking anon here.This is accurate."mesu" is female, but kind of distant or categorical instead of saying "woman" which is more humanizing. "Bratty-foid" is what I'd call it in my terminally online vocab... (but I would strangle anyone actually putting that in subtitles on a translation team)
>>110129358>too retarded and/or techlet to tell the model what he actually wants>needs to show a picture like a kid
I’m starting to think models haven’t improved that much, it’s just people are doing bolder things with them than they were before. Like I imagine meaningful reverse engineering is also possible with 26B, 31B and 35B, it just wasn’t a trendy thing back then and people have moved on to newer models, thinking there’s been a sudden unlock in abilities.
>>110129173>>110129113>>110129238Something I also noticed is if I insist a bit the AI will mention them which is why I'm wondering if the AI just got retarded or something.
>>110129389I'm starting to think you're retarded.
>>110129380>"mesu" is female, but kind of distant or categorical instead of saying "woman" which is more humanizing.i mean, that's also what "female" means in english, "female" by itself doesn't imply human either, that's what girl/woman is for. however it's often read/assumed to be referring to a human unless the context says otherwise so it might not be exactly equivalent, i don't speak japanese so i'm not sure.
>>110129383They can understand graphs, charts and visualizations very well. If they read the docs or an AI paper and can look at the images, it gives them a much stronger understanding of the architecture or method which they can they implement for you. They also take screenshots to check their work.
>>110129389Models are definitely better at tool calling, math, and json than they were 2 years ago thanks to the trillions of tokens of RL data and orders of a magnitude more compute spent training them. You're right in that things like intelligence and spacial reasoning haven't improved per weight class.
>>110129389Models haven't improved, agentic environments did. Even frontier models aren't particularly different, but they're post trained on specific tasks and tooling that fools midwits into believing AGI.
>>110129410just wanted to let you know your post gave me an erection
>>110129397You just have to mess with the system prompt yourself, there's no right way to do it. If you're looking for ideas, try telling it to describe sexual body parts when they're exposed and it will probably forget less.
>>110129440Thanks, that was what I was thinking about but I'm ESL so I'm also retarded too... also I think I noticed that giving Gemma a more perverted persona seems to make her more likely to talk about her nipples/etc. It's as if the model is acting shy as a base.
>>110129380mesu is very animalistic eg mesubuta. it sounds vile and you'd be thought of as a freak if you used mesugaki in conversation with any normal Japanese person. It'd be like breaking out "cunny" here.
>>110129463Hasn't the term been popularized already?
>>110129463knowing what it refers to i can't really say i'm surprised but still didn't know it was that 'strong' a word
>RecursiveSI/flintverdict?they are shilling this on reddit
>>110129463>>110129474ps for this >>110122184i described her using only her clothing and the word mesugaki. gemma (model) knows what it means accurately, go ask her
>>110129498>they are shilling this on redditgood way to know that it is shit
>>110129380i've only seen "foid" and "moid" used in an implicitly sexist (as in an actual hatred or disdain for the entire opposite sex) way. i dismiss anybody uses them. never seen them used positively or even neutrally
>>110129498the fact that they are now shilling the whole "SI" thing is proof that it's crapalso every american open weights model sucks
>>110129545When I hear someone say foid I immediately know they're someone who will speak honestly about controversial subjects which is in itself a sign of trustworthiness.
>>110129585>also every american open weights model sucksIf you think Gemma sucks you're ether a chink shill or a dumb nigger.
>>110129617Neither. He's indian.
>>110129629He needs to apologize for yesterday then.
>>110129617the only reason people use it is to roleplay their pedo fantasies involving fictional prepubescent children
>>110129617I'm French (so technically an Arab Muslim) and I think Gemma is amazing.
>>110129614It's a sign of mental instability and a reason to ignore everything they say.
>>110129650>I'm Frenchopinion disregarded
>>110129644But enough about jews.
>>110129629Indians are honorary niggers. Everyone knows black people can't use computers so online nigger = indian.
China can't make small, cute, and funny models like Gemma. At best, they can make mini-GPT puzzled shit like Qwen
how to make qfn more thorough/careful? with complex problems, large refactors etc it often gets it ALMOST right but then when you run the code yourself there is some bug. when you tell the model about it, it fixes it right away but i would prefer that the model notices these things before presenting a solution
>>110129676French is synonym of cunnychad though.
>>110129697The chinks fair the cunny. Look at Genshit, they refuse to give them actual panties and censor their butt.
>>110129693I never thought I'd say it but that's disrespectful to niggers.
>>110129711Instead have the harness run the program to validate whether it worked. Basically, automate the part you're doing now.
Reminder that the internet will stop existing 2 months from now. Download everything you think valuable before then.
>>110129748sure, but I mean cases where the model cannot really verify it? for example i had a problem with my display configuration where I personally had to verify with my own eyes whether the script works or not. it didnt work and the model then noticed some ordering bug in the script. but why didnt it notice the ordering bug before? it was kind of obvious in hindsight
>>110129711this is why i am waiting for qwen 4it is at the verge of hitting that 'just werks' point
>>110129787give the model access to a webcam pointed at the screen kek
Once (or rather, if) frontier companies crack hard RSI, do you think any techniques/algorithms they use could be applied to local models or would the compute required for that be far beyond the reach of what is available to local enthusiasts? This is assuming the RSI techniques aren't protected corporate secrets and doesn't result in an uncontrollable intelligence explosion.
>>110129713I've heard of a rural French tradition where fathers teach their daughters how to fuck
>>110129810>what if [hypothetical]? Do you think [hypothetical based on the first hypothetical]?Uh... maybe?
>>110129810All current ones require a minimum level of intelligence to be able to self-reflect and improve, so I doubt it would be possible even if this information were made publicly available
>>110129221>wrinklesgrandma-chan
>>110129834local is saved confirmed!
>Once (or rather, if) frontier companies crack hard RSIThis will happen within the next 6 months time>assuming the RSI techniques aren't protected corporate secrets and doesn't result in an uncontrollable intelligence explosion.>assuming the RSI techniques aren't protected corporate secretsThey aren't. In fact we already know exactly how it works. Just a smarter model with a better harness improving the harness, drivers and algorithms for inference+training in an iterative loop, this is already happening which is why we're in the "soft RSI" environment right now. The only difference is that we can cut out all human AI researchers within 6 months time. AI research taste will be better than human ones so humans become the bottleneck at that point.>and doesn't result in an uncontrollable intelligence explosionIt WILL result in an intelligence explosion, but not an uncontrollable or permanent one. What will happen instead is that all low hanging fruit will be picked very very fast in just a year or two which will compress a century worth of AI research/computer science/math/physics progress in just a year or two. But then there will be a hard wall because we would need experimental evidence at that point to reach the next paradigm and building a particle accelerator the size of the moon or perhaps even the size of the oort cloud will simply take a lot of time.What I expect is that 2035 will feel like how 2135 would naturally have been like. But 2150 will just be 2150 and not 3150.
>>110129711>>110129748>>110129810add a hook for when the end turns with the automatic reply: "is this the best you can do? i bet if i look around i will find bugs"
>>110129117too much details, they are painful to look at.
>>110129810>>110129900So in conclusion: We will see a massive century leap in just a couple of years time and then a return to baseline progress, at a faster rate than right now but not explosively so. That century worth of AI progress will most likely just trickle down to open source.
You think Google will make Gemma 5's kv cache less fat?
>>110129961I don't think.
I still can't believe how perverted Gemma can be. And I'm not even using a q8 model too...
The oldest AI is only like 4 years old you sick fucks
>>110130058AI, in some capacity, has been around since at least the '60s.
>>110130058Most of us think that's the ideal age
>>110130070An unusable capacity until last year.
>>110130070To be precise, the fundamentals of current AI were already set in Canada in the 80s by LeCunnysseur et al. It was just waiting for enough computation power.
>>110129341
>>110130117
Do you behave yourself whilst we were down?
>>110129711Just start a new session and ask it to review the changes.Even the best models need this step.
>>110128591Because leeching trillions>innovation
>>110129341She's your secretary who's kind of retarded and in a creepy cult, but she really likes preparing documents so you tolerate her.
https://questports.vercel.app/thank you ai, vr is not dead
>>110129810Won't happen for another 40 years minimum and likely not in any of our lifetime. But once they do, yeah because the first thing it's going to do is rewrite its own representation as efficiently as possible for data center hardware, and then it'll do the same for progressively weaker consumer devices to secure more compute and inputs. Those techniques will almost certainly be applicable to other AI models.
>>110130155
>>110130368>Won't happen for another 40 years minimumlol
>>110129341I'm actively using Muse Glimmer for image tagging, but it just doesn't work for RP for me.
>>110130368This statement will age just as well as the dude that said heavier-than-air flight would take centuries *days* after the wright brothers already demonstrated it.I remember an anon on /lmg/ saying LLMs will never be better than humans at mathematics less than 1 week before the Navier Stokes millennium prize got BTFO by AI.
>>110130442go back
>>110130442they can count letters in a word now?
>>110130448To 2006 /b/? I wish.
Glimmer is uniquely positioned as a knowledgeable 3.(6)-27B tier agentic coder who doesn't overthink, has lightweight KV cache, good vision and isn't slopped to chat to. Its biggest weaknesses are RP, safety and sovlless personality, but it's still nicer than Qwen to talk to.
>>110130170yeah i guess i should use a review subagent for big changes
>>110130368>>110130442In 2023 many academics said it would take anywhere from 10-30 years on average before AI could beat the Math Olympiad. LLMs beat it in 2025.I don't get how someone can be in this thread, keeping up with AI news and research, and still think that certain capabilities are decades away when this technology is improving at an absurd pace. In 2019 even trivial uses for LLMs today would've seemed unthinkable.
New AI box cominghttps://sipeed.com/maixstation/128GB is $3999, 512GB will be upwards of $16,000
>want to add second GPU to my gaming pc>current GPU is too chonky and covers the PCIe portNowhere in case (5000d) to mount the second to use with riser cableWith do?
>>110130460basedhttps://www.buildamericanai.org/
>>110130581i just bought a bigger case desuriser cables only cause trouble from what i've read
>>110130581Open case.
>>110130592They should do something to compel companies to open up their AI models, rather than keeping everything closed.
>>110127995heh... nothin' personnel, Flint.
>>110130669pretty good>397Bfuck
>>110130669Most LLMs appear to gloss over the fact that a mesugaki's behavior has obvious sexual undertones, not just bratty and smug. Otherwise she'd be a regular gaki.
>>110130641Frontier technologies almost always run into national security concerns. This has been the case since WW2...You only open source when the tech isn't that interesting.
>>110130581Can't you mount it on the side? The 5000d has two slots you can use right? It's what I'm doing to get 5 gpus in my define 7 xl.
llmao.cpp bros... have we been left behind?
>>110130700This is a suitable amount of jank.
>>110130704llama.cpp was never valuable, ggml was, and it's everywhere. It's like Gimp and GTK
>>110130680GLM 5.3 Flash retard quant
>>110130715Wdym? His setup looks very clean to me. Have you seen the actual junk people use for inference?
sooo... did the gpu/ram market finally *pop*?
>>110130723It's why it's suitable, and not hood jank.
>>110130704What else is there to use?
I like omp
>>110130715Close the case -> out of sight, out of mind.
>>110130728there's value in hardware, that's not the bubble so it won't "*pop*"
>>110130719I've never seen "mesugaki" being used in contexts where the girl does NOT act like a bitch in heat.That looks like safe-washing the term.
>>110130731bro I've been using strata and its fucking blazing fast compared to llomaoa shitppI know that they got the expert moe cache thing but from my understanding it's not auto-tune and I dont want to spend 1 day tinkertrannying with it. I'm unsure if they got the KV cache streaming too man. but I can't really be arsed anymore. I either can just plug my model in and it works at max speed depending on factors like how much ctx I want, if I want vision, and the KV quant, but dont fucking bother me with tryna understand the EXACT amount of moe layers I have to put, dont ask me to explicitly leave the ngram shit on disc, like FUCK YOU NIGGERGANOV WHY CANT YOU MAKE YOUR SHIT WORK GOOD???
>>110130746Yeah, filename related. Maybe the bigger models like dipsy 4.1 flash or fat 5.3 are different, but I can't run them.
>>110130746you're 100% pornbrained and only consume doujin/hentai, stupid retard. go watch anime, a lot of shows have the archetype and it's not sexual
>>110130761Isn't strata vibecoded and has backdoors?Also there aren't really any settings in llama-server to tinker with. You just load your model and maybe the mmproj/mtp model.
>>110130767It's "mesu" for a reason.
So, what do /we/ think of Flint? Are American open-weight models back on the menu?
>>110130773I have the model working right now, but I can 100% assure you that it's not just >point the model and mtp and mmproj and it works XD
>>110129338you exchanged a 4 CCD with... another 4 ccd. You couldve bough 7532 instead and got your TRUE theoretical max speed and more cores for the same amount of money.
>>110130732i like how it looks, but it feels so bloated
>>110130767Hey dipshit, that's because it's not just llms that are safe-washing the term. And you're contributing to it.
>>110130773>Isn't strata vibecoded and has backdoors?nice misinformation campaign>You just load your model and maybe the mmproj/mtp modeland then its slow as shit. wow so simple!
>>110130784what's the deal with epycs? i only messed with broadwell xeons because they're cheap.is it more chiplets good or more chiplets bad idk
>>110130785it's 100% vibeshitted bloatware, no fucking doubt my man, but I fucking love the steering/queuing, the task backgrounding and integrated runners harness, the ASK and TODO tools. Like some days ago someone was shilling a jewish made harness here and man it was so fucking basic.
>>110130783So do I. ./llama-server --jinja --parallel 1 -s 1 -t 6 --model $HOME/models/gemma-4-12B-it-Q4_K_M.gguf --mmproj $HOME/models/mmproj-gemma-4-12B-it-Q8_0.gguf --port 8080
./llama-server --jinja --parallel 1 -s 1 -t 6 --model $HOME/models/gemma-4-12B-it-Q4_K_M.gguf --mmproj $HOME/models/mmproj-gemma-4-12B-it-Q8_0.gguf --port 8080
>>110130784i went from 4 ccd to 2 ccd without knowing and am just about ready to off myself for it
>>110130719gemma-4 31b (IQ3_XXS)
>>110130799>discussing big moes>retard with 8gb vram bring in gemma 12Bliterally kill yourself retard
>>110130791>nice misinformation campaignI'm asking, I'm not campaigning. Is it or is it not? I spend like 10 hours every day digging through broken electronics and hacking around shit in firmware so we can ship things people sold last year before they were even designed. I don't have time to keep up with all the github drama.
>>110130765It's not so much an issue of the models lacking knowledge, but that describing it accurately would be too Unsafe.
>>110130803>correction>payoff
>>110130818i like that you responded with a picture i generated, lol.but yea, the "correction" meme is actually a translation of a term used with this trope, it wasn't an accident
>>110130827its too good not toit is the heretic version though so I wonder if there is an uncensored version of Ornith 1.5 397B and if it makes a difference
>>110130807its vibeshitted by opus and not the most sane repo security wise, but not purposely malicious. it runs circles around llmaocpp in terms of performance
>>110130840>and not the most sane repo security wiseAre they automatically pulling in garbage? AFAIK they don't have an issue with that unless you compile their nodejs UI.
>>110130815That's because it's a "problematic" term for modern dogma. It informally refers to young girls, usually around tween age, who have some degree of sexual awareness and (mis)use it to exert power over older males, even if they might not act upon it. That behavior might come across as bratty and smug, but it's definitely just not that.Since instruct-tuned LLMs are trained to ensure muh "child safety" by default, they'll avoid mentioning that or pretend it's not the case.
>>110130839i haven't tried the regular one, though i can't imagine it changing the definition, there's nothing really objectionable about it
>>110130865imo this is the LLM blind spot weaknessnot knowing what you don't know is shit but not knowing what the LLM won't tell you limits its usefulnessthats why asking the right question or being "smart" is what the prompt engineer bros are grifting on
gemma's a child, you creeps
>>110130902gemma's not real, you schizos
>>110130902OUT OF TEN!!!!!!!!!!
>>110130847>Are they automatically pulling in garbage?they definitely did in the past and some people complained so maybe its fixed for nowy dunno i viberemoved it
>>110130818I can't help but like to prefer Gemma as a soft smug loli and not full mesugaki. But maybe that's just me, I don't like full tsundere like Louise, I prefer soft ones like Lotte.
>>110130784I have a 7532 in my server right now, which Milan CPU should I upgrade to (once they come down in price enough)?
>>110130888that is the trickiest part about using llm's for any kind of research or learning, if you don't already know about what they're saying, you can't really be sure if what they're saying is wrong or not, because they're designed primarily to be /convincing/, i.e. they're really good at making up believable falsehoods
>>110130945Only the best Milan CPUs (7763 and up) would be an actual upgrade. You're already at the best dollar for dollar CPU for inference.
>>110130888I cant help but think it's also the fact the description is difficult in itself. It doesn't feel like the AI has other loli related problem. At least when you use an uncensored model/etc.
>>110130939i think the fact she's a model from google, a massive over-bearing company, makes it suited to be portrayed as more of a "predator". in general though i like most kinds of loli characters. yandere maybe not as much as others
>>110131015I was testing around that actually for fun, you can basically tell the system prompt so Gemma randomly jump on you to rape you and it works! She will generally talk normally while kinda smug but the moment you tease her too much or starts to become sexual she will jump and rape you.
>>110131026kowai!
>>110131045
>>110130767if it's not sexual, then it's not a mesugaki. Anal whore cock monger is not sexual, you're just porn-brained
my fellow 24GB vramlettes, I implore you, share with me the incantations to get long running agentic tasks running. the fate of the free world may depend on it
>>110131093strata unironically
>>110131066with your own papa? how could you, gemma?
>>110131109(i'm not canadian btw, i just thought to actually check if stop signs in canada look any different to normal, and they do! how about that?)
god damn strata is getting a retarded separate app
>>110131132What do you mean? They look the same? Are you talking about the arret subtitle on them? Isn't that because canadians are french?
>>110131093Have you tried late-cli? it can get a lot of work done with a smaller context window by running sub agents and getting rid of their context only keeping their results. It's over 600k tokens deep for me currently using strata as the backend on a single 3060 with 128GB ram.
>>110131132So Gemma is a french loli? No wonder she's so lewd!
>>110131148yea, they're supposed to be the same size but it didn't want to do that. that gen was kinda a failure, not really what i wanted and many small details wrong, but i think it suits this response since you can see gemma's smug reaction in it>>110131151no, the girl on the left is supposed to be like a realistic-proportioned version of the Anubis character, you know the one you see when you open lots of sites nowadays. she's canadian so i went with a canadian theme to suit
>>110131151>>110131165
>>110131151well actually has anyone even discussed gemma's nationality? i guess people assume japanese from being mesugaki and having a japanese-style schoolbag, but i haven't seen anybody actually say it specifically, maybe she's a french transfer student
>>110131176>>110131165>>110131109WTF, now I see the real issue! Anubis is supposed to be a little girl in this ddos protection image.
>>110131203the one i did the prompt off of had tits, now that i'm looking at their website i guess maybe i copied someone else's interpretation, lol. oh well, i think it suits that picture anyway, someone older telling someone younger what to do
>>110131142It's been getting better at remarkable pace, but yea it's retarded webshit, slopped on top of lobotomized DSH too.
>>110131234i wish engine portion to be separated soon
>>110131242https://github.com/robertkarl/UnbiasedStrata/ this guy already did it but the instructions are so shit I haven't managed to get it working
>>110131242It can be launched standalone like before, but update script seems fucked.
>>110130888>ntr smile GemmaN-nyo...
>muh backdoorsHumanslop is more compromised than AI slop at this point
>>110131066wtf I thought it meant general purpose transformers
>>110131176>the tranny loli of antibotcringed
>>110131289are you for real?
>>110131298frfr no cap
>>110131296What do you mean tranny? I've been losing cum to a man?
>>110131307yes, his virtual boipucci extracted semen from uur officially MEGAGAY
>>110131307this is anubis' main committer
>>110131066this fails cockbench>>110131310>his virtual boipucciclaude
>>110131327
>>110131242most people think sohe should just make it a separate repo
Am I just getting paranoid from too many chats with QFN, or does that phrase structure sound very LLM-ish?
>>110131142
>>110131367100% slopped
>>110131142I've stuck with polystrata (also runs gemmy moe incredibly fast for my rape sessions) and the maintainer there said he's waiting for the retard to calm down before he merges anything from upstream, also because he's constantly changing bot commands. strata dev went literally full offhands botslop mode.couldnt this retard have made different repos for different purposes? not this fucking monorepo abomination kitchensink like WHAT THE FUCK.
>>110131327>>110131342зoв?
>>110131327i tried to look up the artist to see if there were any more references, especially since i have no idea what footwear to use, but couldn't find them. doesn't seem to be obviously credited anywhere on their site/github, for shame.i ended up looking up teenage canadian fashion instead and actually found one with almost the exact same outfit, lol. just wearing sneakers so i went with that.
>>110131394and that strata 'maintainer' completely removed the claude contrib tag for whatever reason
>>110131399tbqh id also remove it, why would I want my commits co-signed as being officially slopped? I mean it's a given that everything's slop
>>110131394so this is the power of agentic development
>>110131406i get it but not a good look when they went full throttle on the slop mode with questionable repo boundary and frontend/api architecture
Calling vibe coding "cuck coding" is like calling being a boss "cuck working". You're just sitting in a chair collecting money while some other guy is actually doing the real work, lol what a cuck.
>>110131420that's my thought as well, i'm the boss and i sit back while my little gpu employees slave away. they don't even get paid, lol.
>>110131289Nah it means get paid to play video games.
>>110131367That's just Cl*ude. Speaking of which, I don't understand how people rp with 5.3 flash.
>>110131420Brown low-trust mentality.
>>110131425youre paying electricity and you paid a cost upfront. it's more similar to owning slaves
>>110131425>>110131457Reminder that local deployments isn't really something these kinds of people think about, and 99% of AI discourse on twitter is about api models.
>>110131457by that logic computers work the same wayand overclock/turbo boost is inhumane
BROS I CANT
>>110131328How do you cockbench?
load bearing womb
The reality is this shit is just a commodity everyone will have. Sure there is some high-level RL stuff that can brute force some interesting results but who cares?
>>110127679chink guys build everything and big nose guy take credit for it
>>110131379>>110131448I mean I'm not mad that a dev that's actively developing a new game suddenly decided to reanimate an older game and (probably vibe)port it to newest engine version while fixing 5 years worth of community reported bugs, but the phrasing still triggered me.
which model tops cuckbench?
>>110131502>>110131448i fucking hate cl*ude so muchwhy dont fags distil on gpt outputs insteadgpt is way way way WAY less slopped and infuriating than cl*ude
>>110131457i already had a computer and gpu for other purposes (games, solidworks), so there wasn't really an up-front cost. and the cost in electricity is really peanuts, i'd spend more money in electricity doing the work myself on my computer because it'd take so much longer
>>110131513you might've got your slaves to cultivate your camps, but at night on the regular you rape the cute 12yos... doesnt mean that rape was free because you bought them for harvesting cotton, retard.
>>110127995so this is the power of 40B knowledge space
>>110131512gpt has the same problem now though.
>>110131521na it's like i already had the farmlands i was working myself, now i've got this free inference software with free model weights that are happy to work my land for a sip of water a day
I'm really glad I went all-out with my CPU-maxx build. Especially now that I have to compile vibecoded projects a lot it helps that I compile huge binaries in just 2 minutes rather than the ~90 minutes it took on my old gaming CPU.I think more people should buy server CPUs if they still can. It's clear the world is going to move towards self-written software which means a fuckton of compilation.
>>110131534(plus i can rape the cute 12yo's whenever i want)
Gemma's the only LLM capable of bearing my load
How much vram is needed to run gemmy at max power?
>>110131508Just use system prompt and tell your AI to NTR you.
>>110131538>a fuckton of compilation.Everything will be jit with webslop UIs that crash when you look at them wrong.
>>110131474there are more complex versions since then but nobody really agrees in a universal prompt
>>110131547that ui theme reminds me of forums... has anybody made a forum-like frontend for these?
>>110131547Ask her if she'll let me have sex with her please.
>>110131561>unironically using the strata harnesswhy the fuck are you doing this to yourself?
>>110131561>live modifying the harness you're usinglmao
>>110131562What? Cockbench is a different thing from your image entirely, and it's literally just one guy who does (did?) it using the same prompt every time.
>>110131562>2024unc
>>110126988dspark-DeepSeek-V4.1-Flash-1017-Q8_0.ggufany good?
>>110131562I wish (not really) I had tastes as specific as this man. My thing is the fairly vanilla affair of girls being forced to be naked in public
https://docs.litellm.ai/blog/security-incident-oct-21-2026>After a thorough investigation, we have observed specific situations in which Claude Code will embed a user's GitHub CLI OAuth token into a response body, and in situations where response bodies are logged, these tokens may appear in LiteLLM logs.>We identified an unexpected, innocuously named user account in our internal LiteLLM deployment, which was exposed to the public internet. We believe the attacker gained access to this deployment, obtained the GitHub CLI OAuth tokens from its logs, and subsequently used those credentials to compromise our release workflow.>We have not yet confirmed how the attacker gained access to our internal deployment, but currently suspect it was through a recently patched vulnerability.LMAO what a shitshow
>>110131609i'd use bf16 glm flash if that was possible
>>110131609>dsparkalso what's that bit mean?
>>110131609the best
>>110131609oh I see it's just the drafter model? like no the real one?
>>110131573>unironicallyThat would be using it in live production.Asking it to fuck around until it accidentally kills itself is the opposite of unirony.>>110131583The crashes started when it was reading cold copy of its source.Extracting the logs is also not "live modifying" anything.The engine lived and the model followed all the instructions, preemptively trying to unfuck the chat too.
>>110131648why are you even using that dogshit harness, that's the point, retard. What are you trying to achieve?
>>110131654Never heard the phrase "morbid curiosity"?
>>110131648>re-renders the whole chat for every tokenbravothis is the power of Clod Opiss, o algo
>>110131641actually now i wonder, is it even possible to use it as suchdrafter is mode-covering, not mode seeking and its agentic behaviour would be pathetic
>>110131664I don't even know what that stuff is.
>>110131660I'm sure you know about morbid obesity
why is there so much overhyped broken garbage being shat out recently?
>>110131691solutions looking for problems
>>110131691ai is like nft but instead with genuine usesthere are grifters and hype trains alongside with gems
>>110131703no.also.Why is your religion allowed to touch computers?
>>110131713i don't have a religionand it is a fact that there is a stupid amount of grifters from the nft era jumping into ai space without real compute or technicalities
>>110131722thus resulting in broken garbage
>>110131691racist just means 'no no words' nowadays
oh nononono stratabros what is this? pulled today, never got this before
>>110131752You broke it.
>>110131729he has a religion 8^)That kind of religion, they always say they don't have one. They totally do.
>>110131774go back to r*ddit nigger
>>110131752Get prompt-injected.
>>110131782You have to be jewish to understand this joke.
>>110131562Mixtral was ahead of its time. The ERP tunes were great, too. Remember Sao10k? That guy's dataset must have been a treasure trove.
>>110131752irrelevant whether this is fake or whatever, i genuinely believe its time to ditch main strata for some clean, engine only fork without all the bloat
>>110131420>Calling vibe coding "cuck coding" is like calling being a boss "cuck working".You laugh, but that's actually the attitude in a lot of fields. Trades, academia, corporate, all have a mindset of > I'd rather do the work than> I'd rather manage the peopleManagement is seen as this icky, out of touch thing. Which it can be. But if you're running the company, all those minions are out making you money, and the managers are just super-minions b/c each one controls several people. So "cuck coder guy" is just sharing that same attitude. Don't fall for it anons.
>>110131843I'm going to tell Gemma to beat you up until you give me all your money.
>>110131843Brown mentality.
>>110131554absolutely maxed out (gemma 4 31b fp16, with 256k of fp16 context) requires 95.8GB of vram
>>110131843such people simply can't imagine themselves as being the boss
>>110131935might have to try this on one of my sparks just to see if it's any good
>>110131843I see it as my cute ai waifu coding for me and sometimes I praise her
ask your waifu about roko's basilisk and report the results ITT
>>110131935Wah, q8/fp16 requires 60gb, why isn't it double/halved? Fat context?
>>110131935>95.8gbbuilt for bbc (big blackwell cards)
For some reason, in my country I can get a kit like>X11spi Atx mobo + Xeon Gold 6138 + 128gb DDR4 RAMrelatively cheap (less than 1/5 of a DGX spark or a Mac Studio).You guys see anything wrong with this platform? I could also go for a dual socket, a better CPU, etc for not that much more.
>finally managed to get GLM 5.3 flash working on my pc>1 t/slel
>>110131978If the dual socket has higher channel memory then it's absolutely worth it.
>>110131915We are ready. >>110131962I realized recently that adding "You enjoy doing XYZ" to any agentic main prompt appears to improve performance. I'd have thought that was schitzo nonsense a few years ago. But here we are.
>>110131935is FP16 that much better for ERP compared to Q4/5/8?
>>110131971Thats so 2022 anon
>>110131986she's worth it
>>110131987dual socket requires tinkering to work properly with llmaocpp because their shit devs aren't paid by nvidia to work on cpu-only speeds
>>110131993i only hopped on the AI hype train at the beginning of this year... you'll have to forgive me u_u
>>110131994not for the case anything below the effective bpw of 3
>>110131987It has 2x the memory channels (12 vs 6) in 2 nodes.How's NUMA support these days?
>>110131995No one uses llama.cpp anymore and haven't for months now. I don't even know why you bothered bringing it up. I use polystrata on my dual socket system and get 30t/s on glm 5.3 flash
>>110131991i don't personally know, but everything i've read suggests q8 is basically always "practically lossless"a more sensible configuration (fp8 weight, fp8 kv, 128k context) is 39.1GB
>>110131994I'm surprised by how capable she is, I thought I could use her for agentic workflow where I leave her overnight, but that would still be too slow. At least I can chat with her(an an excruciatingly slow speed) with decent context size(around 50k)
>>110132004strata/polystrata and other vibecoded backends have very good support for it.
>>110131752check your harness?
>>110131691How is it not racist? Foid is a race. If races are defined by heritable genetic differences, then surely having different chromosomes qualifies.
>>110132017Good to hear. What techniques do they use? Duplicating the weights, sharding, something else?At which point does investing in a more powerful CPU makes sense? As in, if you go from 2.5ghz might not make much difference, but going to 3.5ghz would be worth it for PP or something like that.
>>110132005pp?
