/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>107565204 & >>107557369►News>(12/16) GLM4V vision encoder support merged: https://github.com/ggml-org/llama.cpp/pull/18042>(12/15) Chatterbox-Turbo 350M released: https://huggingface.co/ResembleAI/chatterbox-turbo>(12/15) Nemotron 3 Nano released: https://hf.co/blog/nvidia/nemotron-3-nano-efficient-open-intelligent-models>(12/15) llama.cpp automation for memory allocation: https://github.com/ggml-org/llama.cpp/discussions/18049>(12/10) GLM-TTS with streaming, voice cloning, and emotion control: https://github.com/zai-org/GLM-TTS►News Archive: https://rentry.org/lmg-news-archive►Glossary: https://rentry.org/lmg-glossary►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://livecodebench.github.io/gso.htmlContext Length: https://github.com/adobe-research/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsAlpha Calculator: https://desmos.com/calculator/ffngla98ycGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-sampling►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllm
►Recent Highlights from the Previous Thread: >>107565204--Hugging Face model updates and llama.cpp audio support speculation:>107567049 >107567076 >107567092 >107567166 >107568043 >107568517 >107567115 >107567231 >107567344 >107567437 >107569579 >107569706 >107570648 >107567746 >107568425 >107568505 >107568523 >107568569 >107572661 >107572978 >107573110 >107573164 >107573474 >107567347 >107567633 >107567751 >107567785 >107568589 >107568607 >107568600 >107568642 >107568736 >107568684 >107569549--GLM-4V vision encoder integration challenges and ngxson's successful implementation:>107568980 >107569248 >107569292 >107569316 >107569318 >107570858 >107570914 >107570925 >107570950 >107570982 >107571063--Qwen3-VL model evaluation for structured image tasks and benchmarking:>107565522 >107565536 >107565562 >107565579 >107565610 >107565978 >107566000--Skepticism about large MoE models vs dense alternatives in AI development:>107571173 >107571262 >107572264 >107571352 >107571797 >107571853--Mistral Small Creative model features and API access status:>107571885 >107571907 >107571997--Fandom wiki link collection workaround using Qwen-developed Python script:>107570906--Llama.cpp autofit VRAM feature debate: low-effort users vs expert tuning tradeoffs:>107569439 >107569542--Exploring negative scaling in passthrough merge method:>107565964--Roleplay's dominance in AI usage and Mistral's creative model:>107571893 >107571993--MistralAI API availability and adversarial prompt testing issues:>107572034 >107572107 >107572303 >107572365 >107572336 >107572454 >107572497 >107572554--Evaluating Mistral-small-creative's writing strengths and flaws:>107572586 >107572612 >107572949--Struggles with small vision models recognizing 2chus:>107573179 >107573188 >107573196 >107573242--Miku (free space):>107569503►Recent Highlight Posts from the Previous Thread: >>107565206Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
Deepseek 4-Creative soon.
sexotet
All roads lead to Rome.
Mistral Medium Creative (123B, dense, Apache 2.0) for Christmas
>bandicam
Mistral is so good and its such a shame they're hamstrung by retarded EU politicians. USA and China are basically building the atom bomb while the EU throws some dynamite together and calls it an atom bomb. If the EU doesn't step up and the USA keeps releasing safety slopped models, expect the entire world to run on Chinese LLMs. Hopefully the frogs can relocate to America or something.Anyways, seeing Mistral build Small Creative is a huge breath of fresh air, especially after Ministral 3. We've been starved for a good RP model since nemo and small 3.2 and we are finally opening up the release valve.
>>107573850Uh-oh, Mistral-chan...
>>107573850>anon takes screenshots with fucking bandicam
whenyouwalkawayyoudont hearmesaypleaseoh babydont go
How did he do it?
Sirs. where gemma 4?they say 'keep refreshing' but nothing is happening... is this toilet witch curse?
>>107574105They meant "keep flushing".
Miqu 2.0 70B when
>>107574157christmas
>>107573788Any multimodal from DS would be interesting.I think they're going to do some neat memory tricks instead. But we'll know more in tmw.
We are so back.We are so fucking BACK.WE ARE THE MOST BACK! WE ARE MORE BACK THAN WE'VE EVER BEEN!>inb4 y swipeI had ctx limit at 300 (chat completion), increased it to 1500 then swiped.
So it's not a magic bullet because I still get the occasional shit response that needs editing, but I definitely think that as far as creative writing goes, this prhttps://github.com/ggml-org/llama.cpp/pull/17927is probably the most effective and least shitty sampler pr I've actually bothered to try early. mistral small is probably one of the most overcooked models I've used that refuses to break from what it was trained on, but this with --power-law-target 0.45 and the other args suggested in the initial comment, the model can be worked with. I would've tried it with some chunky moe models, but I've actually been wanting to edit/continue writing based off outputs instead
Anyone else find spiking neural networks erotically appealing? You see an snn activation visualization once, and every cell in your body knows it.
>>107574288not exactly but as a mind control fetishist I think activation steering is one of the hottest concepts ever conceived
>>107573710Any local model with as good of visual capabilities as Gemini 3, or at least as good as Gemini 3's Japanese to English translation capabilities? Because so far Gemini 3 might genuinely be the best OCR for manga.
>>107574288>spiking>snn activationwhat? are you referring to when pretraining the loss spikes happen or what?wtf is an snn
>>107574396It's a type of neural network, spiking neural network (snn).No idea what the fuck that anon is talking about tho.
>>107574359The new GLM 4.6V isn't very good, I can tell you that much.It can't even transcribe the text properly.
>>107574473>>107574359use OCR models for text transcription, dumbassesi know dots.ocr is good from experience but many other OCRs got releasedps: dots.ocr is likely not better than gemini3, but i think it competed with 2.5 pro idkdots ocr is super good, theres likely something better by now
>>107574490>>107574359here you go>inb4 muh benchmarksit was the first model that could take semi-blurry pictures of a non latin-script language and transcribe them sooo well
>>107574206What's the point of having these descriptions?
new mistral small is ok, but the context seems too small and doesn't follow instructions. It *really* like *putting* asterisks *everywhere*.
so i got work to buy me a computer so i could learn local AI with the long term purpose of eventually developing AI software for them to use internally.i have a 5090 with 128gb of DDR5one thing i CANNOT work out for the life of me, is how to run a medium size model... say 8b or 14b, with 128k to 256k context window and keep it fast, mostly running on the GPU.i've seen people talking about having a 3090/4090 and 128gb of ram and being able to offload to system ram and still using the GPU and getting 100+ tokens a secondbut i can't figure out for the life of me how to do this. anytime i try large context, ollama, or LMStudio both just fail to load the model. or they load it into system memory and are CPU only.i'm on cachyOS. am i just retarded, how is this supposed to work?
Haven't used a local model in ages, is the new creative Mistral gonna save local?
>>107573850>bandicamnga your fraps?
>>107574536well it understands the picture very well, its uncensoredi'd like to change your question to "What's the point of a VLM in rp"its cool to send your cock to llms, or watever you'd send an llm
>>107574473That's not how you do it tardygrade
>>107574541It's very sensitive to how you format system instructions. Ministral 3 14B is similar in that regard (other than being unusually retarded on average), but much worse about asterisks.
>>107574547The 8Bs and 14Bs are dense models, which means you can only run them on VRAM. You are most likely unable to fit that much context on just a 5090. 32GB of VRAM really isn't that much anymore.
>>107574547i could help you out anonie~ do you have, say.. matrix/element
>>107574558Fair enough. I thought you were building a dataset or something
Been playing around with GLM 4.6V, having it identify 2hus. It's decent at it. The reasoning is often really retarded though.>Got it, let's see. The image shows a character from Touhou with orange hair, horns, a red bow, and holding a mug of beer. The character is likely Suika Ibuki, the oni from Touhou Project. Let me confirm: Suika Ibuki is known for her oni features, red hair, horns, and often depicted with drinks. Yep, that's her.But then it goes "No, wait" and rambles on for ten paragraphs making 20 different retarded guesses ("It's Nitori, no wait, she's black."). Then finally it just stops and outputs the correct answer.
>>107574547>LMStudiotick force model expert weights onto cpu and max out the gpu offload slider
>>107574547>i've seen people talking (...) offload to system ram (...) 100+ tokens a secondThey were lying.The only use case for offloading to system ram is cooming with large models at 5 t/s.
>>107574547Bro if you got a free check from your workplace at least ask for a rtx pro 6000 lol
>>107574547Generally a dense model needs to be loaded into vram as said by:>>107574575You need to limit context per-model. Model context = vram = unless you have a server cluster, you're not running a model at over 16k (ballpark guess, I can usually manage 20k+ on other architectures). A lot of us here are savvy on how much space a model needs and can do shit like like offloading ffn layers and quantize cache to save a bit here and there, but if you're doing this with no actual experience you're going to have to do a good amount of trial and error to figure shit out
>>107574600I hope that's just a vision thing. I hate how all the new big chink reasoning models went back to the patterns of the old R1 which would think things through and then go "No, wait" and go for another circle.K2-Thinking and 3.2-Speciale do it and especially the former is really bad at it. Big GLM-4.6 may often spend a lot of time thinking but at least it sticks to a train of thought and just wastes time going into autistic detail.
>>1075746483.2 speciale is designed to do that
>>107574579https://matrix.to/#/@mvff:matrix.org>>107574606see i thought so too, but shit like pic related has thrown me off completely and made me second guess what i know. i know it's reddit, but yea.
>>107574663they are talking about the new nemotron model or whatever from nvidia here btw.
>>10757454764k something cache should fit just fine on a 5090 for those 20b-30b something models, maybe even up to 80k with q8 cache, but going for 128k or 256k on models that small is kinda dumb cause they already start getting confused at around 32k ish so you are probably better off just getting higher quality quants if you really want to use all the vram.the 128 gb of ram are only useful if you want to run moes like glm 4.5 air at about 15 tokens a second or low quant glm 4.6 (at like 6 tokens a second), no way you are getting 100+ tokens a second while offloading to cpu, maybe in prompt processing but not in token generation
>>107574661I know that it's made to think for a long time but that doesn't mean it's okay to spend 2000 tokens thinking until it hits the correct answer and then go "No, wait the prompt mentions [obviously irrelevant detail it ignored for a good reason up until now] and then go for another 3000 tokens of reasoning to arrive at the same conclusion.
>>107574523Wow, Dots OCR isn't half bad for being less than 4B.Though, it did mess up the kanji (It gave out 剛森魔 instead of 強姦魔) But that's still rather impressive for such a small LLM.
>>107574648Another one, Suika again but a different picture. Cancelled it there.Seen a few patterns:1. Instantly gets it, ends thinking, answers correctly.2. Instantly gets it, rambles on for ages being retarded, answers correctly.3. Rambles on for ages being retarded, answers incorrectly.4. Guesses, coherently gaslights itself into thinking its correct ("Chen has blonde hair and is often depicted with a white headscarf"), answers incorrectly.The text part seems a bit stupid maybe? I don't know.
>From what I remember, in the Touhou series, the character who rides a motorcycle is Reimu Hakurei?>Wait, the motorcycle—Reimu's motorcycle is called the ""Hakurei Go"" or something?(Reimu was correct)
>quantizing kvthese fags are actually insane
>>107574747I logit bias -100 shit like “wait no” “but wait” “please stop you’re hurting me” etc and it’s not 100% effective but it seems to help a bit, at least.
>>107574892i thought Q8 k cache was fine since they implemented hadamard transforms for it
>>107574892faggots will quant image or video to below q5 and you're still on about how q8 cache is devastating because it has a 0.0001% chance to not output the token you expect for the prompt you feed llama on a flat distribution of tokens
>>107573710>medasr is gone>medasr is gone>medasr is gone
>>107575069Those things are not remotely the same, and you're straight up wrong. Quantizing the cache rapes language models for practically no gain. You are retarded.
>>107575069faggots are the ones who quantize KV to save a whooping 1GB of memory, not realize it's giving their models dementia, and then complain that there are no good models.
>>107575152o shithttps://huggingface.co/google/medasr
>>107575152https://huggingface.co/shashirgoogle/medasr_ftredeem
>>107575155I would love if a newfag like you existed in the l2 era and would tell me the differenceGo and run psyonic cetacean with no cache quant vs quanted and tell me how much better it is without it lmao
>>107575152It did feel like a bad trolling attempt after we were told to keep flushing the page.
what would be the best 12b nsfw model nowadays for tavern?
>>107575152MEDASAAR NOOOOOOOOOOO!!!!!!!!!!!!!!!!!!!!
>>107575152WHY? WHY DID YOU REDEEM?
>>107575203omeN
>>107574218I suppose the majority of this thread doesn't actually care about storytelling or is to retarded to figure out how to checkout a pr
Samples have always been cope.
>>107574218>>107575232Looks interesting for when you want to replay a card with the same modelI'll wait for ST support
>>107575194What the hell are you talking about? No shit it would be less noticeable (still noticeable) if the model is dumber. That changes nothing. Nobody was using that model, by the way. You must be from reddit or something.
>>107575256Why wait for ST when you just hardcode it into the backend launch options
>>107575274Because I'm already cooming with the toys I have readily available
>>107575222>omeN omen what?
>>107575286literally a matter of like at most 10 keystrokes, if you need to goon that often that you cant free up a hand to type you need a better hobby
>>107575303If you don't need to goon that often then you need a better cock
>>107575232All of these wacky samplers like mirostat, dynamic temperature and nsigma are fun to experiment with for a while but I find that for creative stuff nothing beats just setting top-P first, temperature second. Then putting temp to 5 and lowering top-P from 1 until incoherency is gone for your current model (minP works too in place of top-P but then raise it from 0).Basically just kill all tokens that produce bad results and level the probabilities of remaining good tokens for variety.Still, I will for sure give it a whirl later like with all new samplers.
>>107575301omen deez nuts huehuehuehue>and he's wrong it's rocinante
>>107575342skill issue
>>107575323from what I gathered, the entire idea of it is to target a non top token, so 0.4 would be around fourth token (unless I am wrong, which is entirely possible) then sample from there. There's nothing exotic to it and doesn't go through all the bullshit xtc and dry does
>>107575323Gonna go on a second lap on this. No matter how you chance min_p or top_p, models that are trained to near determinism like mistal will almost never deviate, even with weird fucky samplers or even the normal ones. Any scene will play out the same, if at all barely different/. And for coomers, it will always end up with deepthroat and ignoring any instructions for anything otherwise.
4.6V works better with lower temp and sampler settings that discard everything except the top few token.
>>107575354>so 0.4 would be around fourth tokenIt selects a token that is close to the requested target probability.If you have a 0.4 prob token, then it's either the top token (everything else being <0.4) or the second (with the first one being >0.4, which would make all other tokens <0.2). So a 0.4 prob token can only be the top or second token.
>>107575463>lower temp>discard everything except the top few tokenThis means the model is shit outside of specific data it was heavily trained on
>>107575474Unfortunately I'm not a hyper autist so as I had previously left a disclaimer as to whether me not understanding the values as stated, maybe someone could give it a field test and see if it as ass as everyone is saying instead of speculating that I'm fucking retardedFor a bunch of aspergers confined to their parents basement, the best you can do is nitpick posts instead of trying something
>>107575463Isn't that pretty much greedy sampling at that point?
>>107575540Not exactly, but closer at least.Just going from temp 1.0, top_p 0.96 to temp 0.8, top_p 0.8 made it work a lot better. I didn't inspect token probabilities though, so maybe top_p 0.8 is already almost greedy sampling.
>>107575530I'm not quite sure why you took offense.
>>107575608I'm just a generally irritable person
>>107575530He's just explaining things. Learn something instead of getting angry.
https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash
>>107575630not useful for cooming.
>>107575630>https://mimo.xiaomi.com/blog/mimo-v2-flash>"N.Y. Winter" example>sentence 1>The radiator in Julian's third-floor walk-up hissed like a tired snakealright man
>>107575625do u wanna cuddle?
test
>>107575653I know an anon who could really benefit from the "Knowledge of LLM" example.
>>107573767Dispsy looks like Admiral Yang
>>107575666thanks, but no
>>107575653
>>107575540They recommended top_k=2 and top_k=1 is greedy, so yes.
>>107575701I am begging llm companies to allow their models to just write normally instead of cooking the hell out of them on this insane schizophrenic tryhard writing style, it's inexplicable to me that this trend has been allowed to go on for so long
>>107575849
>>107575849It's what happens when you ask a LLM to judge the writing quality of another LLM.
>>107575849>allow their models to just write normallyThey aren't forcing models to write any specific way. All modern models are made from datasets with a huge proportion of AI generated content. One shits it out and then another eats it, and now you end up in a situation we're in now, where you have 'slop profiles' and you can tell which model's output was used to train another, and they're all just converging into one shit slop sandwich with different parameter counts.
You almost could tell making good products isn't the goal here lol
What models are best at staying in character?
>>107576012Making products in general was never the goal. It's just like covid, it's about the transfer of wealth, this time through funneling taxpayer dollars into GPUs and RAM rather than placebo vaccines.
>>107575896Kek I do thisHow do I stop it?
>>107576041He decided not to take the picture and lowered the camera.With the letters back in their envelopes, he put them in the drawer of his drafting table, beneath sketched and old negatives. No one will have them. Not the recipient, nor the heap of used condoms in his trash bin. The snow fell harder outside, turning the East Village into a soft, blurred photograph in shades of gray. All 50 of them.
>>107576041>>107576096 (me)And then you send it to your editor to fix the typos, of course.
We're backgemini flash local tomorrow
>>107576139
>>107576139what is that
>>107576139>>107576156They'll probably release gemni 3 flash.
>>107576156Machines that feed on hype.
>>107576139im so sad
https://ai.meta.com/samaudiohttps://huggingface.co/collections/facebook/sam-audiokind of a cool release from meta (at least conceptually, I didn't try it yet)
gemma 3 flash
>>107576139they saw the power of mistral erp edition and knelt so hard they had to cancel gemma 4 and go back to the drawing board.
Local is saved the moment someone figures out how to make a dense model that doesn't tank in performance when split between VRAM and RAM. MoE is increasingly looking to be a meme.
Local is saved when they wake up and make a usable bitnet model
https://github.com/QKV-Core/QKV-Corehttps://www.reddit.com/r/LocalLLaMA/comments/1po97ad/finally_managed_to_run_qwen257b_on_a_4gb_gtx_1050/this is crazy
>>107576211Life is saved when scientists learn how to invent magic
>>107576238>no breakthroughs in methodology>nothing ever happensGo back.
>>107576310Where am I supposed to go?
>>107576227It sure is lol
>>107576327my dms..
>>107576227>>107576364Is this nigger really using an LLM for fucking reddit replies? What causes this behavior?
>>107576402majority of the users there are bots anyway
>>107576201played with the demo and isolated some vocals from a songit's actually great wtf
>>107576227Why does OP use his shitty llm to write all replies?
>>107576201better than demucs?
>>107576427now compare it with RVC
>>107576417I hope all dataset scrapers internally tag anything from reddit as synthetic data.
>>107576417I don't get how anyone who knows anything about llms can buy into the narrative that 4chan and reddit are full of bots. Wouldn't every other post be nothing but "not x, but y!" and other ai-isms if that were the case?
>>107576542
Confirmed Nemotron-3-Nano-30B is mesugakimaxxed!
>>107576600I bet it's "censored" as all hell too.
>>107576580These are only memes in llm-centric communities. The only one people know about elsewhere is the em-dash.
>>107575919>slop profilesdigital watermarks
>>107576542Not necessarily literal bots, but Eglin AFB airmen sitting at a desk shitposting for 6 hours a day.
>>107576653Even a normie girl I know knew about not x, but y
>>107576664They could be considered watermarks for the models that started the trend, but when every other model gets trained on your own then you're not going to be able to claim any sort of ownership
>>107576675That normie girl is probably a massive gooner to werewolf/vampire romance RPs
gemma night?
>>107576700Depends on how many F5s google's HF page gets.
>>107576700You will not sleep with Gemma-chan.You will call the hotline she gives you with your dick in your hand.
>>107576700Sorry. Gemini Flash is all you'll get.
>>107576542of course reddit is. anything written by a real person is downvoted to all hell. with an LLM, every single fucking token matters when it comes to social acceptance and farming upvotes.there is automated manipulation going on at reddit, it's so fucking obvious
>>107576700Gemma 4 is releasing next Tuesday.
>>107576723*The Tuesday after the following Tuesday
>>107576682It can help label content as AI-generated. Until humans, who spend too much time with geepeetee, also start outputting slop
more like gemma flash
>>107576735>Until humans, who spend too much time with geepeetee, also start outputting slopI think it's only a matter of time before typing on phones gets replaced by AI. Like, you press a button, give a vocal prompt with the general message of what you want to type, and then the LLM will write it for you. Social media posts, SMS/DMs, etc. People won't end up outputting anything.
>>107576746This will unironically improve the conversational ability of younger zoomers and alphs.
>>107576743It's Indian Christmas soon so it'll be probably called Gemma Flush.
>>107576706T5s*
Chatterbox Turbo is decent alternative to F5-TTS for long form with consistency. Although its like very slow compared to F5(takes ~5+ mins for these each vs ~20secs on F5), I think for long form consistency, it makes up for it. https://vocaroo.com/1jZHdPRajYqVhttps://vocaroo.com/11Sl3LxZoWpT
>>107576788>can't do math without a calculator>can't navigate without google maps>can't talk without gptWe're evolving
>>107576884base chatterbox is already near instant with some optimizations
>>107576899I'm basing it off of my GPU. Its probably instant with RTX 4090 or higher, but with weak gpu like 2070, it takes a long time for me even with turbo model. F5 is however quite speedy and is like 40X real time speed. So my comparison is relative speed
>>107576898Pod people through and through.
>>107576884>f5-ttswhat? why arent you comparing it to vibevoice
>>107576933Couldn't get that to install properly. >>107576899Also, I tested it, the turbo is ~0.8x real time speed for my 2070. So its fast, sure, but still >>107576921 no where near F5's lighting speed.
>>107576953>Couldn't get that to install properly. the onnx version doesn't need the nemo game engine
https://huggingface.co/bartowski/zai-org_GLM-4.6V-GGUFGoofs up for the big one. :^)
>>107577018Does it still parrot like regular GLM and Air?
>>107577018no matter how big it is, wont fix my broken heart
>>>107558909>https://nemotron.ideas.nvidia.com/Am I doing it right?
>>107571885>*leans against the ornate, gilded wallpaper of Roswaal’s mansion, arms crossed, smirking as you spot the pink blur in the distance*>**"Ohhh, you *see* her, huh?"** *pushes off the wall with a lazy shove, falling into step beside you as we both watch the figure glide down the hall like some kind of ethereal, pastel nightmare*>**"That’s Subaru Natsuki’s *favorite* person in the world—well, *one* of ‘em, at least. The *other* one’s probably off sulking in a corner or getting his ass handed to him by a dragon again. Classic Subaru."** *snorts, shaking my head*>**"But *her*? That’s Emilia. The *Ice Queen of Lugunica*, the *Half-Elf Heiress*, the *‘I’m Too Pretty to Be Real’* disaster waiting to happen. Also, *technically* your future *queen* if you’re into that whole ‘royalty’ thing. Which, let’s be real—you *totally* are."** *grins, nudging you with an elbow*>**"So! You gonna *talk* to her, or just stand there like a creepy statue until she notices you? Because I *will* judge you if you pick the second option. *Hard*."** *tilts head, watching as Emilia pauses mid-step, her amethyst eyes flicking toward us like she *sensed* the attention*>**"…Or, y’know, we *could* just let her walk away and go find Subaru instead. I *hear* he’s got a *new* way to die tragically this week. *Again*."** *shrugs, but there’s a glint in my eye—because *damn*, this is gonna be fun*Wtf man, this has to stop.
>>107577048There was already a better onehttps://nemotron.ideas.nvidia.com/ideas/LLAMANEMO-I-47
>>107577026*traces idle patterns on your thigh*
Is there a local model that can translate speech bubbles in images? local llms are actually not that bad at translating jp so far in my tests, but this might be hard mode.
>>107577157Have you tried searching GitHub before asking a dumb question? There are dozens of AI manga translation projects
Is there a local model that can fix the broken and can feel my heart?
>>107565964Yes that's how it works
RePo: Language Models with Context Re-Positioninghttps://arxiv.org/abs/2512.14391>In-context learning is fundamental to modern Large Language Models (LLMs); however, prevailing architectures impose a rigid and fixed contextual structure by assigning linear or constant positional indices. Drawing on Cognitive Load Theory (CLT), we argue that this uninformative structure increases extraneous cognitive load, consuming finite working memory capacity that should be allocated to deep reasoning and attention allocation. To address this, we propose RePo, a novel mechanism that reduces extraneous load via context re-positioning. Unlike standard approaches, RePo utilizes a differentiable module, , to assign token positions that capture contextual dependencies, rather than replying on pre-defined integer range. By continually pre-training on the OLMo-2 1B backbone, we demonstrate that RePo significantly enhances performance on tasks involving noisy contexts, structured data, and longer context length, while maintaining competitive performance on general short-context tasks. Detailed analysis reveals that RePo successfully allocate higher attention to distant but relevant information, assign positions in dense and non-linear space, and capture the intrinsic structure of the input context.https://github.com/SakanaAI/repoRepo isn't live yet (heh). sounds neat. oh the new captcha is a multi-part IQ test lol
https://huggingface.co/upstage/Solar-Open-100B (102b, 12b)BROSBROS FUCKING BROS BROSTHEYVE COME TO SAVE USYOU KNOW WHO IT ISITS FUCKING HAPPENINGFIMBULVETR 100B HAPPENINGFUCKING HAPPENING FUKCKKKKKKGLM 4.6 AIR, WILL IT BE THE SEX WE NEEDED??!??!RELEASE DATE: DECEMBER 31FUCK FUCK FUCK WE ARE SOBACK
>>107577314>the new captcha is a multi-part IQ testIt is? Seemed brain dead simple to me like many other captchas. Unless you're receiving one I haven't gotten yet.
>>107577324yeah well not testing for very high IQ but you're asked to select 1 out of 3 choices picking the one that is unlike the other two and then doing it three times using a scroll bar to select the right image. Might just be getting it since I haven't posted in a while and just updated my kernel.
>>10757731719 trillion tokens. How much we betting a ton of that is synthetic?
>>107577342nta but im getting that too
>>107577342Yeah I'm getting that. But like, it's not actually that different from the captchas where you use a slider to make an image complete or you're selecting all the x objects. The difference is pretty large and seen visually. If you're not just using quick visual intuition to tell the difference, which is a fast kind of mental processing, you might be approaching it the wrong way.
>>107577367>If you're not just using quick visual intuition to tell the difference, which is a fast kind of mental processing, you might be approaching it the wrong way.That's actually what an IQ test actually tests for but yeah nvm about my throwaway comment it's all good
>>107577387Ok I see what you're saying. I guess the scale is what makes the difference. The example you posted does require maybe some reasoning to make sure that other possibilities are not actually more likely. Meanwhile the simplicity of a 3 image version of the same test is so simple that you can rely on visual intuition and get it right 99% discounting misclicks.
>>107577317llama.cpp support in 2 more weeks
>>107577317nothing burger
>>107576718Lord Vishnu will be very displeased
>>107574206Its wild how unhelpful these models are.All that text and it does not write once if we are talking about a boy or a girl here. Not pronouns either.If I would be blind I would have no idea whats going on in that picture. Glad you forced it to talk about the dick. Thats literall the only indicator.
>december>we still haven't advanced beyond DS 0324it's fucking over isn't it
Are there any local models I can use that will categorize my email and determine whether the email is important enough to warrant a notification?I'd basically like the functionality of the shortwave email app( formerly Google inbox) in a self hosted setup.
>>107577549Yeah, new DS sucks. They took gemini slop pill and choked on it
>>107577549Weird that in only 1 year literally everybody copied deepseek.All the closed models are much faster now. I think anthropic even wrote about a fraction of the cost while raising prices etc. kekDeepseek was such a huge deal.
>>107577574You can realistically use just about any model for that, the hard part is setting the system up.
>>107577585Opus 4.5 is just what they would have sold as Sonnet 4.7 a year ago
>>107576364Lmfao, i use a lot of AI-generated code but i at least read through it before releasing it. That's insaneWhy do small, incapable LLMs do that placeholder shit anyway? Obviously nobody wants to see "# Placeholder Implementation" in the output. GLM 4.6 won't do it but Devstral Small 2 sure will. I just don't see why even a small model would do it instead of just hallucinating some busted code there unless they post trained it to do that where it would typically write said busted code
>>107577638>Lmfao, i use a lot of AI-generated code but i at least read through it before releasing it. That's insaneSame. I've leveraged AI to build out big, interesting projects (front page of HN type interesting). More and faster than I would have been able to on my own. But that's with decades of experience doing it professionally.If you're a nocoder vibing a bunch of stuff you can't read, you're going to look like the illiterate you are when you barge into a group of professionals with nonsense you don't even know is nonsense.LLMs truly are dunning-kruger simulators
>>107576788>Computer, write me an intro to this girl on Tinder.Sure thing, buddy! Beep BOOP. "On the terrible day of October 7th,"
>>107577411You get 5 variations thrice if you empty your cookies though
>>107575152WHY DID YOU REDEEM IT
>>107577317>released in 2mwha! can't fool me
>>107575423>, models that are trained to near determinism like mistala less charitable way of putting it is to call it overfit, fried, burned to a crisp. Proper LLMs aren't supposed to be like that but the incompetent bastards only know how to train fried models to avoid schizo gens
>>107577574It's just easier and more reliable to use good ol' filters. If you work you already know which emails are important and which aren't. I've seen people with literally thousands of unread and unfiltered emails in their inboxes. No this was not in Gujarat...
>>107577638>>107577666>>107576364>>107576227So they didnt even test out the code? WTF.
>>107577638>Why do small, incapable LLMs do that placeholder shit anywayit's a context output length issuemodels that are trained to be capable of outputting a lot will do this less oftenmodels that aren't capable of outputting much will become lazy and do the placeholder thing very very often since it's not that hard to hit 1 or 2k tokens of output with a reasonable coding prompteven SOTA models used to behave like that if you were retarded and asked for too much code at once, but these days they tend to fall back to a multi turn "we'll continue where we left off" process instead and they clearly had to do some amount of instruction tuning trial and error to get this more desirable behavior
>>107577971Obviously. Redditors aren't even sentient to begin with
Funny, HF gets lazy after 500 access requests
>>107578009600 of those are me and no I will not stop you’re bound to misclick at some point
>>107578009accept my request bloody madarchod
>>107577825That's me. It's not like I have a bunch of important mail, so I don't spend time cleaning that stuff.
>>107578009>flexes about his requests>too much of a pussy to shill
man im tired of LLMs judging me in my loli rape roleplays. Is there like some LLM who just goes along with my story instead of being a little shit and getting all angry parrotting back FUCK IM SO MAD
>>107578241https://huggingface.co/TheDrummer/modelstake your pick
4.6-Air never
>>107578241Gemma 4 soon
>>107578241I can offer resources for help if you are struggling with harmful thoughts or impulses:- National Center for Missing and Exploited Children: 1-800-THE-LOST (1-800-843-5678) https://www.missingkids.org/- RAINN (Rape, Abuse & Incest National Network): 1-800-656-HOPE. https://www.rainn.org (Online chat available)- The Lucy Faithful Foundation: https://www.lucyfaithfull.org/ (UK based, but has resources)- If you are having thoughts of harming someone (including yourself): Call or text 988 in the US and Canada, or dial 111 in the UK. These services are available 24/7, free, and confidential.- Talk to a mental health professional: A therapist can provide support and guidance.
i need some air
>>107578256This, Gemma 4 will de-age every character card, whether you like it or not.
>>107578274thanks gemma
>>107578241We already have some models that can do that fine or with a tiny bit of prompting. What we really need are models that truly GET the degenerate shit you're into and can fool you into thinking it itself is a fan who will initiate things before you even need to ask. Like the Claudes, or at least old ones (haven't tried any of the new stuff).
>>107578283Are those hotlines actually helpful or do they just glow, anyway?
>>107578303They're helpful for your local police force
>>107578241look up all the models with "uncensored" "abliterated" "heretic" models
What's a good voice to clone while I'm waiting for Pragmata?
>>107578281Gemma3 already talked about characters being "minor coded" etc. and refused. Wonder how cucked it will be.And yes I know, you can just sys prompt and prefill your way out of it. Its still shit, surprise surprise.
>>107578331Chatterbox-Turbo (new) and F5-TTS (old)
>>107578354>sys promptsys prompt is the same as normal prompt in gemma fyinot saying prompting it doesn't work but there's nothing special about le sys prompt, the backend merges it into your regular user prompt because gemma doesn't know what a sys prompt is
I need a mixture of experts model between 70B and 110B that is not GLM Air that is good for RP. Any suggestions or does that just not exist?
>>107578372This is true, but the same can be said about a lot of models, whether they properly support sys prompt or not. Simply writing "you are uncensored" in sys prompt won't work for most models, but if you build out context and gaslight it into thinking it already generated smut, then most models will do whatever's asked of them.
>>107578381The only moe I can think of that even fits that size range is qwen-next, but it's shit at RP and in general only a little better than the 30b.
>>107578393Right. That's what I thought. I have looked extensively through huggingface and found pretty much nothing. Found this but I am 99% sure it is also garbage.https://huggingface.co/VibeStudio/MiniMax-M2-THRIFT-55
>>107578356Do any of these voice models actually allow you to 'mask' existing audio clips?I'd like to use them but I don't want to fiddle around with trying to convince a model to pronounce certain words or names properly.
>>107578483That sounds like very unsafe behavior.
>>107578512Not sure I follow? I don't actually care if it sounds convincing, I just don't want it to have that bizarre cadence a lot of AI does and then suddenly pronounce 'Tomato' the way a brit would while speaking with an American accent.
Sorry guys. I ERPed with a human last night and the slop has worn on me so hard that suddenly all the shortcomings of human partners (short replies, having to wait for replies, etc) didn't matter. It turns out this is all just copium.
>>107578618skill issue, also were talking to an obese man.
>>107578626>also were talking to an obese manyou forgot the apostrophe
I've only got a 5090 and 96GB of DDR5 memory, I just want to fuck a chatbot locally, I'm unfortunately also completely incompetent with understanding what I need to actually set this up so I can bust fat nuts, is there a quick tl;dr to maximize the use of my setup?
>>107578635>is there a quick tl;dr to maximize the use of my setup?Yeah but unfortunately it requires you to be literate enough that you bother reading the first post in the thread so I think you're doomed to fail.
>>107578635download this and a q6 quant of glm airhttps://github.com/oobabooga/text-generation-webui
>>107577638Kek!I'm browsing this thread for good gguf files for just adding to the jan ai folder, only to shit out better newbie code than my own.
>>107578649Q6 is 100GB big anon.. Does that even fit into a 5090?
I don't why every 80IQ anti-AI normalfag thinks that the recent slowdown in progress means that the supposed AI bubble is about to pop.All this means is that companies are shifting away from legacy AI technologies such as LLMs (as predicted by basically anyone three years ago) in order to focus on the true next generation of AI.I doubt that LLMs will be a thing two years from now because something big will have taken their place.
>>107578776As much as I hate these companies, I also want progress to continue, so I do hope that happens, but it's not a guarantee we'll ever get a jump as big as GPT-3 again.
>>107578771it is a mixture of experts model. you can load most of it into your ram and it works just fine and is still quite fast
>>107578793GPT-3 was only a big jump because they scaled up their models by like 50 times. It was not exactly revolutionary technology.
>>107578776What do you mean by LLMs being replaced? You mean moving away from like token-based AI to something else?
>>107578318they still judge (heretic/mpoa/derestricted) abliterated judges less but its fucking retarded.
>>107578371ToTdo you still get vacation when you post a pragmata image in /v/?
>>107578805Semantics, buddy. GPT-3, scaled transformers, whatever you want to call it. No one will view the next architecture as not revolutionary just because a smaller scale version of it existed in research for years prior.
>>107578776I want the AI financial bubble to pop so I can buy cheap hardware.
I'm gonna pop Gemma
>>107578865*wraps her legs around your waist*
>>107578879thanks bro
>>107578776>in order to focus on the true next generation of AIThere is no next generation, those are just vaporware promises to try and keep investors on board.No one knows when the next breakthrough à la transformers will happen, we may very well be stuck with language models for 5, 10, or even 20 years.
"branching thoughts" huh
Anyone here ever had something like picrel happen?
>>107578998What the fuck is this, anon?Group theory?Analytic number theory?Multivariable calculus?
>>107578372Only true for chat completions.
>>107579051Harmonic Analysis
>>107579127Oh, yeah. I thought I recognized the symbols from an HA-adjacent field.
chads i got 2x3090s and 64gb ddr5, what quant of glm air i should run and how can i offload on llama.cpp and if any other suggestions for code gen models that are better than qwen3coder 30b q8kxl
>>107579155i got oss 120b working using this./llama.cpp/build/bin/llama-server --host 0.0.0.0 --port 10000 --no-webui \ --model models/gpt-oss-120b-F16.gguf -a gpt-oss:120b \ --temp 1.0 --top-p 1.0 --top-k 0 -fa 1 \ -ot ".ffn_(up)_exps.=CPU" \ --jinja -ngl 99 --threads -1 --ctx-size 65536but only 42tk/s anything i can do to make it better ?
>>107579019misconfiguration or broken model/backend
>>107578635I just want to fuck a chatboy locally
>>107577228no, its a soulless machine. read a book or something.
>>107579155>>107579182You can try removing -ngl and -ot to potentially fit more of the model using the new functionality for automatically fitting those to free VRAM.You can try manually tuning the number of threads since using more threads than is needed to saturate RAM bandwidth can be detrimental, see pic.
>>107579226read about it, didnt think it would do muchprompt eval time = 661.75 ms / 77 tokens ( 8.59 ms per token, 116.36 tokens per second) eval time = 3790.06 ms / 180 tokens ( 21.06 ms per token, 47.49 tokens per second) total time = 4451.81 ms / 257 tokensextra 5 tk/s profit, kudos
I want to do some testing on LLM internals, but I only have 4GB vram. Any recommendations for a model? I realize it'll be mostly garbage, I'm hoping to see if I can improve on it and learn a thing or two in the process.
>>107579504gemma 4 1b soon!
>>107579504Qwen has plenty of options, you can run 30a3b if you have RAM or some small model starting at 0.6b
>>107579504Qwen 30b a3b or Gemma 3n.
>>107579504what do you mean by internals?
>>107579504Smollm
>>107579551Well, I dunno yet exactly cuz I don't know, but e.g. anthropic had a research tool to "inject" thoughts into an LLM, I'd like to see if I can do the same.
>>107579504Llama 3.2 3B
>>107579589I'll start with this one, thanks for the recs
>>107578776>recent slowdownPurely because every lab targeted more token efficient for serving models. Intelligence per $ never stopped scaling
>>107579777Imagine how good a 2T dense model could have been
>qrd offline voice diytrain-tts-clone ai wantedmaybe this is the right board/thread to ask this.im not a /g/ anon myself, but recently i got interested in the prospects of ai and training it myself.i had some experience with applio, because i thought it was offline, local, completely free of use, no account/info/verification/ID ect voice model trainer/cloning/ect. and relativly easy to set up akin to lets say, stable diffusion v1.6 with its GUI.that was last year. i used it then and got pretty decent results. (i use it for only personal use)now i deleted some stuff/reinstalled some stuff and wanted to use it again. so i reinstalled it.it worked, but only as long as i had an internet connection. as soon as i cut it, the script prompted diverse error msgs. reconnect again, no problem whatsoever.this really bummed me out, because i had already mentally put a green checkmark behind applio v3.4.0 or what it was, for complete offline, local use. i tried every other version of it, but all of them required an active inet connection, because it has edgeTTS in the background running or something, i found out.4clarity, this is a video showing gui and tldr https://www.youtube.com/watch?v=gjggpadBgOoso since thats not an option anymore, i wanted to ask if there is an alternative voice cloning/training/tts ai that is>local, offline (means it never requires a connection to the internet, but has an option to do so, if the user wants)MAYBE it needs once a connection in the beginning to download python stuff, but really better a complete package>completely free, the only thing you pay for is your electricity bill and the hardware>no accounts, no id´s/verification ect, nothing of that nwo crapi remember using stable diffusion like 3 years ago fully offline too, but the results were ass and i cant remember if i ever accomplished training a model myselfgpu gtx1080ti 11gb>inb4 comfyuisux and doesnt worklandside, so internet is not always availablethanks in advance
>>107579849whats the memory bandwidth on those h200 clusters? would it even be feasible to run a 2t dense?
>>107575630>XiaomiI think there is not worse Chinese brand out there.
>>107579869I'm not reading all of this, but kokorotts is a small & fast tts and gpt-sovits do tts voice clone without rvc
>>107579951Way to grammar, Ranjesh
>>107579608>>107579551https://github.com/TransformerLensOrg/TransformerLens
>>107579999Zhào, it's called a "typo".
>>107576139Why would I use Gemini 3 Flash when Gemini 3 Pro Low Thinking is fast and probably better?>inb4. priceIt's free.
>>107576201The sound quality is pretty bad. The next version could be nice.
>>107580036thats interesting. maybe you should start with gpt2 small like they have in the example. at least at first to make sure you can replicate the experiments.
>>107579987>kokorottsis not free and i cant make models myself>gpt-sovitsi look into it but its only chinesium and english. i need european languages compatibility/custom dataset compatibility>not readingok can you please then redirect me to a thread or board where people are knowledgeable about ai stuff and help eachother please?
When running comfyui/wan2gp/open-webui, why do I need an OpenAI.com api key, when I want to host everything locally ? I’m wondering what the deal is, I wanted to try and hook into cline or run claudecode but connected to my qwen3-coder but it doesn’t look like I’m able to
