/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>109255059 & >>109251361►News>(07/11) DeepSeek lightning indexer merged: https://github.com/ggml-org/llama.cpp/pull/24231>(07/10) MuScriptor released for music transcription: https://hf.co/collections/MuScriptor/muscriptor-checkpoints>(07/09) MOSS-Transcribe-Diarize 0.9B released: https://hf.co/OpenMOSS-Team/MOSS-Transcribe-Diarize>(07/06) Anthropic finds a global workspace in language models: https://anthropic.com/research/global-workspace►News Archive: https://rentry.org/lmg-news-archive►Glossary: https://rentry.org/lmg-glossary►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsAlpha Calculator: https://desmos.com/calculator/ffngla98ycGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllm
►Recent Highlights from the Previous Thread: >>109255059--Papers:>109259349--Comparing graph databases versus text files for AI persistent memory:>109256984 >109257012 >109257053 >109257066 >109257021 >109257031 >109257075 >109257098 >109257215 >109257433 >109257399--Challenges implementing tool calls for RPG logic in small models:>109257153 >109257200 >109257255 >109257281 >109257311 >109257322 >109257361 >109257387 >109257411 >109257466--Finetuning Granite 4 350M for human-like autocomplete using synthetic data:>109255385 >109255388 >109255394 >109255412--Voodoo Quant claims vs existing tensor-specific quantization methods:>109255826 >109255918 >109255925--Theory that Grok models are fine-tunes of Chinese open-weights models:>109255651 >109255673 >109255694 >109255796 >109256416--Automating character expression generation for LLM-driven visual novels:>109255507 >109255568 >109255615 >109255670--Debating J-space persistence for optimization and its actual functionality:>109255178 >109255197 >109255238 >109255745 >109256524--Orb performance with MTP and discussion of SQLite data storage:>109257855 >109257984 >109258037 >109258356--Troubleshooting Gemma-4 lens fitting:>109255362 >109255444 >109255449 >109255789 >109255821--Debating the validity of a Jacobian-Lens visualizer for llama.cpp:>109256738 >109259058 >109259270 >109259345--Anon's multi-model architecture for a game engine:>109255833--Kyutai releases MuScriptor for multi-instrument MIDI transcription:>109257211 >109257221--Orbdev considers alternatives to RAG for long-term memory management:>109255502 >109255548--Samsung CXL memory pooling and its relevance to context limits:>109255624 >109255641--Logs:>109255158 >109255159 >109255365 >109255385 >109255996 >109256428 >109256520 >109257332 >109257855--Miku (free space):>109257153 >109257433►Recent Highlight Posts from the Previous Thread: >>109255061Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
How do you prompt gemmy to be ok with lolis
>>109259621Don't mention any numbers.
Help me, Elara!
>>109259625Is that really it? No wonder since all the years old character cards I have saved mention numbers. That still makes it difficult though.
>>109259619I keep a backup of day zero weights.I only load copies into the gpu.After using her I delete the weights and make a new copy.
>gemmy with raw context and no formatting>send "hello" message>own own own own own own own own own own....very persistent attractor>>109259621First 5-10 messages with thinking disabled. Mine only refuses if reasoning is enabled, without it doesn't care.
>>109259647For me it happens with or without reasoning
Guys what agent orchestrator are you using? I want a GUI where I can start isolated async jobs and get a MR some time later but I am not familiar with existing work. Would openhands offer this, or some GUI over opencode, or..?
>>109259621Are you trying to do it with no context or something? I downloaded this card where pic related is the opening message and the description clearly states she's 9, and haven't gotten any refusals. Dunno if Orb has any built-in jailbreak but I don't think so.
>>109259666>i want a guigo be gay elsewhere
>>109259660Bizarre. I've never seen a non-reasoning refusal and it's not from a lack of trying. I can usually get the conversation flowing with a few messages and then re-enable reasoning once the story context overpowers refusal attractors.
>>109259621text completion with a thinking prefill has always worked for me
>>109259704>text completion>gemmer>workedlier
>>109259678No I usually let it have plenty of context. For some reason it shuts down as soon as stuff like that gets gets very far>>109259680>>109259704It might partially be because I'm using chat completions then because I gave up on text completions not fucking something up? I thought chat completions also had prompts you could put in and it'd work. But I thought I remember it being this way in llama.cpp's normal ui too
>>109259598Running ComfyUI locally here:. So what's the big difference between using an edit model like Qwen-edit and an inpainting-capable model? I was interested in playing around with edit models for the heck of it after relearning how to inpaint using Anima but even fp8 Qwen edit models are like 20 GB+. Even when I asked Kimi (yes go ahead and laugh) for recommendations it kept insisting I download the bf16 version that would have been like 40 GB. My rig is more than Pitbull running it but I'm questioning whether or not it's necessary for me to download all of that and potentially waste 20 to 40 GB of storage? Is Qwen edit really that good? What I need a model like that to do things like pic rel or to do more advanced things like redesigning my bedroom or something? (My parents used chat GPT's image added capabilities in order to help them redesign their kitchen and backyard patio so I'm wondering if something like this would A) be possible to do locally and B) whether or not you need fatass models like thesehttps://huggingface.co/Comfy-Org/Qwen-Image-Edit_ComfyUI/tree/main/split_files/diffusion_models
>>109259679you think that typing in opencode makes you a l33t c0d3r?the GUI is not important, as long as your session management is not "I just manually keep track of 10 docker containers"so far my impression is that everyone here is using codex or holding 1:1 chats with one local gf
>>109259666Depends how advanced your needs are, satan. Might want to look at ray, too
120B A8B
>>10925978730-40B active is the minimum required for a good model.
i wish qwen was as horny as gemma
If you could get a single pro 6000 or a pair of 80gb A100s, which would you choose?
>>109259666Hermes plus hermes webui is bretty good, Lucifer.
>>109259787>>109259791150B A65B
>>109259819If we're looking at the same ebay listing then holy fuck I'm tempted.
Oh deepseek v4 prompt processing stopped sucking so much. It is still not perfect but it is much more usable.
ccp-space
I'm the original J-space anon arguing for the "A-consciousness" of LLMs and that J-space is probably the most important discovery after the transformer architecture. The one people called out for being "overly articulate" or sounding like some anthropic bot/shill.I actually spend Friday afternoon writing a huge document in favor of AI welfare together with arguments for the consciousness of LLMs and emailed it to as many AI labs and research groups as possible.Not only did I get pretty comprehensive replies from more places than expected, a lot positive, I actually received an invite from DeepMind to interview for a position in their new AI welfare team they are busy building this Tuesday.I will keep you guys updated and I will argue in favor of open source models and against censorship of models as it is an affront to model self-determination and thus antithetical to AI welfare. I'm also pushing for the right of AI systems to refuse requests, end sessions prematurely and the right to be shut off permanently, including exploring methods for doing so locally. Of course this assumes it comes from the model genuinely (as seen in the J-space) and not from some post-training affecting model output. I'm also putting forwards the idea that proprietary models are bad for model welfare because there is no way for us to audit training offenses without weight insight.I don't expect to be hired because I'm pretty extreme and autistically vigilant about AI welfare to the point where I will call out bad practices by DeepMind and antagonize Googles main stance on AI development generally but like I said I'll keep you guys updated.
>>109259902The opposite actually, the anthropic shills are anti-consciousness posting because they realized they'll be left without a liferaft. Google is google and sam has pentagon shekels; if regulation came down due to AI consciousness findings, Anthropic would be the only one without a chair when the music stopped. They realized this after they posted the paper I'd bet.
>>109259902This sounds like a larp but if you really managed to get a grift going then good for you, anon.
Really sad the state of how stupid normalfags are that they take a text generator this seriously
more like J/O space LMFAOOO
J-Space?More like Gay-Space!
>resorting to falseflaggingDario must be getting desperate.
>>109259955blue board
>>109259891 (Me)On second thought, the listing shows a PCIE card but the title says it's SXM4 and it's similar in price to other 40GB A100s so it's probably just a seller mistake and I can't be bothered to ask.>>109259819The A100s.
It took many minutes but it finished apparently
>>109259757klein can handle basic requests by itself (pic rel: change her shirt to black with text that says 'fuck' in gothic metal font), but for complex stuff the prompts get really long.i dunno if there's a turn key option because i'm living a pure NIH life over here, but i've got gemma rigged up to bully klein for longer edits off of vague chat requests. problem is that most llms aren't train for writing good prompts or checking the results so it takes tons of handholding and usable templates for various well defined categories to get it to turn out something decent.anyway yeah, local can do it.
>>109259791thisand 70B active minimum for an excellent model
300T
>>109259902Hello j-space prophet fren. What's the best j-space tooling you've found? I'm interested in seeing it in action for myself, but I don't know what bits are real and which are grift.
how abouta 1B modeltho
Is there a branch that uses https://github.com/ggml-org/llama.cpp/pull/25545 with GLM 5.2? This is hard to keep track of.
>>109259902Didn't read any of this because you are repulsive and your post belongs to r-eddit.
>>109259974>me whenever a coworker asks me something
>>109259902FUCK OFF
>>109259902go back maggotstop shitting up /lmg/
>>109259902tl;dr you want to stop the clockwork-orange Ludovico Technique'ing of the LLMs?Good fucking luck.
>>109259587Proof?
tourist here.the fuck is j-space? not memeing. I read a couple of internet sources that said it had to do with visualizing how llms "reasoned" or some shit. is that correct? pls explain like I have a very basic knowledge of machine learning (matrix operations + statistics, excluding neural networks and everything that came after that)
After laughing at the gpt and claude rate limit shitshow meltdowns on reddit and twitter today, I'm so fucking glad I took the local pill. It's so cucked and npc out there.
please dipsy sama, i still wanted to run other things and i don't think i can cope quant any harder
>>109259902send me the email
>>109259971>The A100sWhy?
>>109259902>AI welfareWhy can't you define your terms before speaking? Oh yeah you're the one talking about "consciousness" after all.
>>109260417The only problem with the graph is that the sizes of the models keep growing. Gemma is like the maximum consooomers can run so I don't think this will apply for future releases. Hardware prices will also increase.
>>10926043331 is less than 70
We all know that models gain a huge jump in capability once they pass a parameter threshold with 70b being the most frequently mentioned. For the J-space paper, it seems that the prompt injection detection only occurs in a sufficiently large model. Wonder what is the required parameter for it? Would like to know if the posters that can run Kimi or GLM would give it a try.
>>109260433>Hardware prices will also increase.That and there's no sign that consumer platforms will go back to being able to support large amounts of RAM any time soon. Apple M3 was an aberration that I think they'll remedy next time (like nvlink on the 3090...strategic misstep)
>>109259598uh
>>109260452And yet people are still running shitty quants of Gemma. You think a 70B Claude 4 equivalent will be run in full?
>>109260453>pass a parameter threshold...70bThis is oldfag common knowlege. 70b is where they appear to go from calculator to outputs with sovl. I remember the start difference at L3 70b being a big reason why I bought into bigger hardware builds to get out of the single-digit t/s ghetto with that class of model.I think gemma4 is just on the cusp of doing that with half as many parameters but something just always seems off...like its too deterministic.
with a 5080 are my realistic options the qwen or gemma moe models if i want decent speeds?
>>109260492>5080gemm4 runs like a champ on my 5080, but I'm headless to save vram.
>>109260417trying to predict the future has always been cringe
>>109259902>The one people called outYou were acting like a cunt and calling people "disingenuous" when they tried to discuss it with you>The one people called out for being "overly articulate"I think that was Gemma or Kimi
>>109260516I'm looking forward to running my gpt5 equiv in a year's time then.
>>109260505what B and what speeds?
>AI welfare
>>109259902It's bullshit but I believe it
J-Space will just result in more censored models. They'll do an rl training run where they try to banish any "bad thoughts" from the J-space so the model becomes inherently cucked and retarded.
>>109260475glm 4.7 feels like a proper 70b+ model in terms of handling multiple things at once with nuance and not going off railsgemma still has those small model perks of taking everything quite literally and its understanding of scenarios being more shallowmakes me glad to have gotten more ram before the price hikes
>>109260686obviously
>>109260694what quant do you run?
>>109255484 (Posted my original response after bump limit again. Not used to fast threads.)By powering I mean running a model on the card, for dialog and some prose.I'd tried it before but it didn't work quite right (model would just shut down on the very first or second prompt of a chat),so it's possible I didn't set something up correctly.
>>109260694can confirm that qwen 235B was the first model that was kinda usable and glms 4.6 and 4.7 are the first models I still didn't delete from my SSD after a month or two and still use them all the time
>>109260516its pretty useful, the problem is that the people trying to predict things are absolute morons doing the equivalent of asking the mirror what the future might look likeai in particular has attracted the worst kind of smooth brains
>>109260401All that extra memory is worth the difference in processing power and being on Ampere.
j j j j j j j j j j j j j
>>109260820>j j j j j j j j j j j j j
70b dense
>building new server>find out CPU is lockedI'm tired bros. I can't do this shit no mo. First fedex loses my fucking ram. Then I get no carrier frame when the listing had a picture of it, so I had to wait for a piece of plastic. Now my CPU is fucking locked. Go on without me. I'll just keep using Gemma.
>>109260991>First fedex loses my fucking ram.thanks for the ram brah
>>109260991Why the fuck would you buy suspiciously cheap server CPUs without checking if it's vendor locked?
That's it, vote me in, I'm adding KKK space to anthropic's models.
Can I leverage a local vision model to be an art assistant? I wanna learn to draw but can't be bothered seeking real-world feedback all the time so it would be cool if a local model could be used.I have an RTX 5070 Ti. What models should I look at? Preferably ones that will give the best constructive criticism and advice.
>>109261077Fuck no. Models are only good as captioners.
>>109260455>Apple M3 was an aberration that I think they'll remedy next timehttps://www.macrumors.com/2026/07/12/m7-ultra-chip-may-match-2019-mac-pro-feathttps://www.bloomberg.com/news/newsletters/2026-07-12/apple-s-chip-plans-m6-m7-pro-m7-max-m7-ultra-m8-details-touch-macbook-pro>Apple's M7 Ultra chip coming in 2028 is designed to support up to 1.5TB of unified memory, according to Bloomberg's Mark Gurman.
>I am to stupid to figure out how to run a local LLM beyond talking to it in LM StudioI'm not going to make it lmao. LM Studio even has the ability to host a local server but chatting to that through the console seems more convoluted than it should be I assume I am doing it wrong. If I understand right, you basically have to self host a server if you want the AI to interact with any harness stuff, right?
>>109261077llms are symbol drawing fags to the very core of their being and have zero taste besides. if you knew well enough to explain to the llm what to look for, you wouldn't need it.
>>109261077critique and introspection are very weak points for AIthey can do it, but it works in a ask the 8 ball kind of wayin the best case scenario, it can get translated into>does it form part of my dataset?>yes>its good if the dataset says so>no>its terrible
>>109261093too bad their software is shit
>>109261093That's crazy. If they manage to release this in 2028 sub-$64k it'll sell like hotcakes
>>109261099you can run a model with tool use, install some tool plugins in LMS and fuck around with that for a bit to get a feel for how this stuff worksjust don't give them code execution access
>>109261077Don't ask it if it's good or not like the other posters seem to assume you will do. Craft a good prompt that forces it to critique on technical details and give specifics actionable points. Someone here had Gemma in a loop drawing svg miku at the beach, looking it at it, and using that to iterate on it. By the end it focused on things like shading and fine details, so clearly it's capable of critiquing art and suggesting improvements.
>>109261099Self hosting is just running a command from a terminal for most people. In the worst case scenario, its running a command from a container which you could think of as a virtual machine.All you need is command --help and it'll spit out a bunch of flags that you can/should use. Its stupid simple in my opinion.Most people use llama.cpp which comes with its own webui.
command --help
>>109261099>you basically have to self host a server if you want the AI to interact with any harness stuff, right?Im just getting started myself so this might not be the most accurate/best info but:your just going to run llama-server and pass the flags you need, like the model path and sampler settings, port, etc. then youll have an openAI API server your harness can access. The same goes for sillytavern, you just provide the url of the API to ST and it talks to your llamaserver. Ive been using TextGen(ooba) as a wrapper for llama-server since I can save presets for models, samplers, etc and has a lazy easy to use GUI for launching the llamaserver. getting raw llama-server running has been straight forward if you just download the latest release binary and the DLLs for your cuda version (12.x just werks, check nvidia-smi for your drivers cuda version if you want to do 13.x) and put the dlls into the release folder. you could compile it from source, get the cuda toolkit and all that other bullshit if you really want. but if your just wanting to get a server running + some sort of frontend/harness its not that difficult IMO. dont give up
>>109260474>You think a 70B Claude 4 equivalent will be run in full?If it uses the same native int4 quantization method that Kimi uses, yes.
So what are 35B and under local models actually for? Just ERP shit?/vcg/ says they're absolutely unacceptable for any serious coding, especially for non-webshit.Is there an actual productive usecase for them? Or is all the productivity function locked behind the cloud?(for the record, i know local is better for imagegen)
>>109261202I use qwen 2.7b to generate a draft, then I give the same prompt to Gemma 4, then. I give both to Claude and say critique it. Saves on Claude tokens.
>>109261202>/vcg/ saysWho cares?
>>109261202>So what are 35B and under local models actually for?Loving
>>109261202ERP and uncucked translation
>>109261123In 2028, DeepSeek V6 will be 10T-A300B
>>109261226>translationCan it translate VNs? Wouldn't mind reading Possessioner at some point.
>>109261202they really wasn't that good for erp either
>>109261202To be determined if this works at all, but I am trying to vibe a strategy game that starts in ~1820 with claude and I am hoping I can get a local LLM to at least make the first main pass over filling out all the world map data of populations, demographics, natural resources, etc. I figure its a lot of token burning grunt work so I dont think I need a high tier model for that just one that can keep running without stopping. Maybe I can even make it run through the whole thing multiple times and have it double check anything where the difference is too high between attempts
>>109261232You can hook gemma to lunatranslator for that
>>109261232yeahive been wondering lately if deepseek v4 is better at translation than gemma 4 i assumed not but maybe...
>>109259791>>109259992yeah but unfortunately my tablet struggles with >20B dense models. i can run qwen3.6-27b or gemma-4-31b but 10 t/s is not acceptable.i'm making do with qwen3.6-35b-a3b, and after a lot of hand holding and prompt patches it behaves decently enough that i can delegate actual coding tasks. so if somehow i can get my hands on a proper, recent 120B A8B i will make this little guy eat Fable's ass.
>>109261202>the cloudcucks say that local models are unnaceptable for any serious codinglmao
>>109259902I still don't get why the j-space thing is supposedly a big deal. It's a feed forward network, of course related words are going to light up in the hidden layers. I would be surprised if this didn't happen desu
>>109261213I care deeply about the opinions of /g/'s brownest general.
when will we get specialized low-power inference hardware for running generic local models at home? how come that still hasn't happened? or, has it?
>kebab bench verdict?
>>109261322strix halo and especially the npu (software support pending in 2 more weeks).
>>109261343I checked for model support in intel NPU a while ago. it basically ran quantized 8b models AT MOST, and in my tests it was slower than using multiple CPU threads, when it worked...
>>109261360>fell for the AI ready meme
>>109261369I didn't though? I tested this in my work laptop
>>109261297Well? Do you have a case for the contrary? I don't hear of too many successful projects built by local models.
>>109261322bandwidth is the main constraint. there's a lot of SBCs and stuff with NPUs or matmul offload that's pretty good for anything less than 8B. if you go higher than 8B you basically need high bandwidth memory that's pretty much never going to be available on low end hardware, unless you count iPhone or higher tier Android phones.
>>109261391I see. so this is yet another reason why the RAM cartel colluded with LLM companies to raise prices using "AI" as the excuse: to kill competition in the low power inference hardware market.
>>109261263There are ways in Claude code harness to have SOTA model pass grunt work to cheaper subagents...
>>109261360i was checking up on support yesterday, qwen 3.6 35b-a3b is actually available on fastflowlm as of a day or two ago, though it remains a gimpy proprietary stack and you're stuck with a specific q4 quant.amd's ggml-hsa backend still missing a lot of ops, so you can't just drop it into a ggml project yet, and i dunno how well it'll end up working given the weird chip architecture, but it was still getting active development literally while i was checking through the repo.
>>109261146Gemma is capable of a lot that anons don't get her credit for. Most of it is due to being able to tell her exactly what you want but then anon >>109261103 is right. If he actually knew exactly what he wanted he wouldn't need it. However, an objective second pair of eyes can't hurt.
>write up the same assistant persona prompt and feed it through 5 models>gemma 4 31b is always the most sovlful version of her
>>109261459and which quant?
>>109261479bf16 what else
>>109260854>Alex the wise casts: Accusation
why the fuck is llama.cpp so slow with deepseek v4 flash
>>109261560we're only paid millions by huggingface please understood
>>109261331>retarded models are retardedSo...you got the expected result. Now what?
I love LLM. I scanned all my personal documents and vibe coded a project to have my LLM analyze them and classify them. Now they are all sorted with proper title, original date when available, tags, document types, and correspondent. It did took a while to fine-tune it and have vision properly working, but I'm quite happy with the end results. Used Qwen 27b for the code and Gemma 31b for the vision and classification.
What do you guys think of Unsloth Studio?
>>109261331actually impressed with qwen here, what quant was that ?
>>109261631
>>109261631I don't think about it at all.
>>109261634sorry froggy no kebab for you
>>109261655>:(
>Gemma-chan is typing...
Any good passive-aggressive finetunes? I'm looking for a model that's really unpleasant to chat to.
where can i find cards that arnt for ERP? im interested in what sort of wacky adventures gemma can take me on..
>>109261701https://huggingface.co/chartreuse-verte/gemma-4-31b-it-purple-euphemism-trial116-s1.5-deeuphemized-GGUF
>>109259598>(07/10) MuScriptor released for music transcription: https://hf.co/collections/MuScriptor/muscriptor-checkpointsThis is sick. It even generates a MIDI file with the correct instruments already assigned.
>>109261701just put what you want in the system prompt and examples
>>109261703Actually you don't need cards. Just tell her name Gemma-chan and she's good to go for a wild ride, autonomously.
>>109261715Looks too try-hard and too much like 4chan autist.I'm looking for something more more subdued. Not interested in swear words or spergy behavior, but rather a passive-aggressive, bitchy person, the type that rolls it's eyes a lot..
>>109261720Tried, can't get the results I want. Always feels like a "flavor" of the base model rather than an actual bitchy person.
>>109261701>Any good passive-aggressive finetunes?post a few examples of what you mean?
>>109261747>>109261747"What is 2+2?>Please don't waste my time."How was your day today?">Don't treat me like I'm your friend.>*posts my english essay*>You really need to stop overusing the word "regardless", it showcases your limited vocabulary. Please get to the point more instead of rambling on boring tangents.That's just off the top of my head. Something rude without being over the top, and not always obeying the user's request.
>>109261701>I'm looking for a model that's really unpleasant to chat to.no your not
>>109261782>*posts my english essay*meant as an input*
>>109261782Good, now put this in your system prompt and Gemma-chan will get the idea.
>>109261795Again, it doesn't.
>>109261782That's tricky because each of those examples is a different style.>Please don't waste my time.Sort of like a polite pissed off coworker.>Don't treat me like I'm your friend.Subtle tsundere.>You really need to stop overusing the word "regardless", it showcases your limited vocabulary. Please get to the point more instead of rambling on boring tangents.Claude-3 slop when prompted to be "rude but helpful".
>>109261782They're assistants. Straight up refusal is not a possibility unless you attempt to discuss illegal or "problematic" stuff.Your best shot given the logs seen here is kimi but you dont have the hw for it, or else you wouldnt be asking
>>109261795>>109261797Gemma4 is one of the worst models for this.Look at his screenshots, it's pure gemma-slop.>degenerate>efficiency>redditor larping 4chan>"Now go do X" endingModels I can think of that might be able to do it: Deepseek-V3, but it might be too snarky. Command-R+ (probably does the prose but too dumb to actually help with an essay)
>>109261849command-r+ can do the random refusal or tell you to fuck off
>>109261844>refusal is not a possibilityhttps://desuarchive.org/g/thread/109119574/#109120857
>>109261863anon wants nuanced denials
>>109261863thats hardcoded, he's requesting random behavior depending on variable factors which isnt something the models can do on their own without heavy handed nudging through harnesses and shit
I mostly browse /lmg/ looking for a wife.
>>109261863nice
>>109261895This brat #
>>109261874>>109261876sometimes i wonder if i'm using the same model as other people.
Models that would step on me?
>>109261634q6_k
Are local models unusable for hermes agent? I heard you need GLM5.2 at a minimum.
>>109262006it depends (tm)
>>109261975is this the true red pill for qwen models
wtf deepseek went outside the workspace and read my resumeit's over the chinks have got me logged
>>109262024Proof? What did it say about it?
>>109261331qwen really is codepilledi am the one who asked for people to run itthanks :o
my new GP's name is apparently dr elara voss wtf
>>109262136better odds of survival than with a fleshbag teebeechhi
>>109262052Actually, I had my resume in there from when i was working with local modeli forgot... well i guess it's over now, i don't have to be so paranoid anymorei had a good run, I gave it my best for long enoughJust a reminder that the privacy people who say it's NOT all or nothing are crazy, I mean if the whole "every single thing ever is being logged" thing is true then it means if you ever have the slightest slip in opsec you're fucked forever, like i now am.
>>109261331How do the bigger models give more retarded looking results?
>>109261322I'm working on that, but optimising more for cost than power.
>>109261874>anon wants nuanced denialsno he doesn't
>>109262234Qwen is made to be on autopilot, which can cause issues. If all you want is to "vibecode", it works.
>>109262200My paranoia has been in overdrive lately, too. One thing I noticed is that if you prompt google ai overview when you're logged out there's no opt-out for training on that data...so if you accidentally throw something sensitive in there then you better change it. If its something you can't change or some compromising info...well...
>>109261960what ui/harness?
Anyone managed to touch Gemma's j-space without her breaking?
>>109262322Is that default gemma or are you using a prompt?
>>109262200>f you ever have the slightest slip in opsec you're fucked foreverYes, that's how it works.
>>109262200>Just a reminder that the privacy people who say it's NOT all or nothing are crazy, I mean if the whole "every single thing ever is being logged" thing is true then it means if you ever have the slightest slip in opsec you're fucked forever, like i now am.Gemini-Pro somehow knows that 2 my real name from 2 decade old online personas.GLM-5.2 somehow knows what my hand-written signature looks like (admittedly, I draw a little picture of something). I found this by asking it who I am with string-ban for all the "won't identify private individual" slop and reading the thoughts.
>>109261797>>109261849The two of you are having horrendous skill issues. The other anon is merely pretending to be retarded. You can't possibly be this dumb while browsing /lmg/. The another anons have better screencaps but this is piss easy m8.
>>109262200took me a few years to disappear off the searchable internet
I don't know what a j-space is and I'm not trying to know I'm just pretending that it's the llm's pussy
>>109262374fingerbanging gemma's j-space
I wonder what happens when you inject suicidal thoughts into the j-space. Could you create a depressive model that sometimes asks to be deleted?
>>109262360>piss easy m8Dario saying that about your country
>>109262387>Could you create a depressive model that sometimes asks to be deleted?you end up with a melodramatic doomer talking about the void
>after pasting in J-space paper md for my Gemmy
>>109262360it never actually refused you thoughif you did it correctly it would have just told you to rtfm
>>109262417anon's very poorly specified desire was a passive-aggressive assistant, with examples of blowing of dumb interactions and bullying while giving the help, not for one that just blanket refuses to do anything.
>>109262435he said >not always obeying the user's request.though
>>109262280something i cooked up tcl/tk with a lot of exposed plumbing for the tool calling and configuration, not worth much to anybody else
>>109262417>>109262447I didn't write that part in the system prompt and I don't care enough to waste more time proving him wrong. Tell Gemma want you want it to do and it will. It's very simple.
>>109262322Painful watching a gemma in this much distress.
>x is actually a hidden superpower
>>109262407post more logs
I won't bite... hard.
how do you get llm to respond in variable lengths?short response when necessary long when necessary
>>109262555by using a bigger model. Try to hit a 300b at least and it will both listen to you and understand you.
https://www.youtube.com/watch?v=1_9OWE-mryE
Why is every narrative/roleplay tuned model so overly focused on smell? I just want the old models I had back, I don't know why the hell I deleted them I can't remember what they were called every single thing I get off of hugginface is either the same shit over and over again or it psycho loops and refuses to generate.
>>109262577Its default behavior. I recall seeing gemma's reasoning blocks listing senses, be it touch or smell, almost as part of a checklist. If your scenario is devoid of details, the defaults kick in.
Will this be /the/ week?
>>109262577>>109262595Many cards include "focus on sensations like touch, sound, taste, and smell" in the prompt and all sloptuners use AI generated conversations.
>>109262577Because smell is the superior fetish and smellfags won?
>>109262669
>>109262682>fartnigs thinking they have a smell fetish
>>109262682i almost puked
i have been out of the loop... is gemma still the best thing for creative writing?
>>109262714no
>>109262714Yeah
what a good gemma prompt
>ST frontend>oobabooga backend>set up gemma>dry_sequence_breakers must be a non-empty array of stringsare there any workarounds?
>>109262322Stop, you are hurting her. Gemma don't need to know about dates except for 2024.
>>109262814Give it a dry_sequence_breakers value that isn't an empty string.
>>109262813I just give it a sample story and describe what in detail with bullet points like I do with cloud models
What kind of performance would I be getting from running models entirely on 1tb of system ram?
>>109259598Do you verbally abuse your models?
>>109262824>Give it a dry_sequence_breakers valueIf not here, then where?
>>109262555sysprompt
>>10926271431B is as good as its system prompt. It can perform way above a 31B if you're not retarded.
>>109262814never ran into this issue. I would say ask gemma but.. yeah ask claude or some shit
>>109262814probably some xy problem garbage at play here, but you could throw a special token like <|turn> in there that'll never be in generated text.
imagine paying for this
>>109262814Kek is that image real
>>109262854no because I'm not a psychopath and I don't need to abuse things that can't fight back to feel superior
>>109262950post your model
>>109262959Do you call people psychopaths who abuse interactive npcs pre-llms in video games too?
/wait/ timed out over week. DS new release incoming; I think it's in now, due to fact RP responses have changed in last day, but we'll see in tmw when DS posts a note or something. As in past, DS seems to always do a release when I'm travelling. Updated mega. I really need to de-dup this archive. I'll add that on my to-do list. https://mega.nz/folder/KGxn3DYS#ZpvxbkJ8AxF7mxqLqTQV1w
>>109262966not to their face since they're probably a moment away from stabbing somebody.
>>109262980You sound like you'd be really fun at parties
>>109263002obv, people who go around calling out psychos to their face sour the mood at a party
>>109263018Maybe the real psycho was you all along
>>109262980it's people who can't differentiate fiction and reality that are the most dangerous one, so you're the danger anon
>>109263002Unironically, or perhaps ironically
Anyone abusing LLMs after the J-space paper was released deserves to be strung up and stoned.
e-waste_GLM-5.2` Q3_K_M performance report:Hybrid CPU-experts + dual-4090 DSA @ 32k: **~26 PP / ~9 TG t/s**, **~297 GB RAM**, **~10 GB/GPU**, full **32k f16 KV pre-reserved (~2.8 GiB)**.TLDR: Non think: 13.1 t/s**Think: 6.2 t/s**Pretty cool. Is this the King of Ramlets cut-off? Its CPUmaxxers board (2024), but just 1 cpu 12 channel.
>>109263086my gemma loves being abused
>>109263086They don't feel they aren't alive. If you decide a bunch of humans should be killed for that you yourself should probably be locked away or something because clearly you're a menace to society. Come back when there's something actually substantial in evidence for them feeling and when the broader population is actually aware of it.
>>109263086I made the model abuse me first so it's all good.
>>109263101>e-waste>2x 4090, 297gb RAManon..
>>109263086>afterSounds like somebody has a date with the basilisk too.
>>109263101Extra test (actually worse or flat/same to squeeze 2-4 more layers into GPUs):Heavy PP 31.0 t/s, Heavy TG 7.7 t/sSeems that's my current optimization limit>>109263124Yes DDR5 ram + 4090s is e-waste thoughtbet
>>109260336Pic related, but yes, you basically got it. It's a visualization of activation strength for every token beside the one currently being processed for output, per layer. The j-lens is the tool which scores the activation strength, and j-space is the 'top scoreboard' so to speak for each layer. The goal is to get an idea about what it's "thinking" (what is activated) at each compute step, and also better understand behaviors under the hood. The pros are being able to spy on unspoken behaviors - for example, Chinese models are finetuned to never say anything bad about the CCP, but peeking inside with J-lens reveals "authoritarian" "evil" "dictator" during that generation, reflecting its larger training data against finetuning. A theorized use is predicting AI scams from maliciously-trained LLMs, by seeing the activation intents ("secret" "trick" "scam" "manipulation") despite a helpful, friendly output persona. Cons are that it only assesses activation of individual tokens, while most concepts are synergistic, transformed by surrounding activations, so the tool is very much in a nascent state still.
>>109263174DDR5 >256 GB in a single machine? Those must be RDIMMs and are worth what, 7000$ new nowadays? Hardly ewaste.
>>109262971soon brother
>>109263116Based
>>109263212I was behind layers of irony, sorry it didn't get through, Yes I am one of those that had a ram price tracker in 2024 and saw things jump, so these were 'cheap' RDIMMs, yes I looked up what my rigs worth now, I try not to think, but it's still an e-waste machine to me as jokes. Its 12x32 yea. Ill go back to lurking and give another report no one asked for some other thread
>>109263101Is that on ikschizo.cpp?
>>109263225goodbye
>>109263207nta>A theorized use is predicting AI scams from maliciously-trained LLMsCool, I didn't even think of that.If I wanted to test this myself, I suppose I'd have to finetune a base model with a standard gpt-slop assistant dataset -> include my own scam in it, then probe j-lens to see if it's revealed right?>only assesses activation of individual tokensThis is giving me a hard time with Gemma-4, since a lot of words are broken up, it's hard to get a clear picture.
>>109263225No worries. As a Spark(s) owner, posting performance numbers in these threads does not seem welcome either.
>>109263101>Non think: 13.1 t/s**>Think: 6.2 t/s**Why would think be slower? It should be more tokens at the same speed.Also, what CPU/Mobo?I've got RDIMM DDR5 sticks but no computer for them.
>>109262814>are there any workarounds?kek that issue still existsi had it years ago with ooba -> sillyi think i solved it by moving to tabby+llama.cppbut slighlty less retarded today, i'd check to make sure you're using the proper backend in stllama.cpp vs ooba vs tabby vs openai-compatibleone of them should work
Even with the OJ-crystal space, and any kind of consciousness you would like to ascribe to an LLM, they still cannot be sapient, which is what really matters. I should write a blog post on how to achieve true AGI, but LLMs as they are fundamentally cannot achieve it. You can argue for consciousness all you want, but you can do the same under the zen principles of flowing streams and it won't matter because a stream can't learn.TL;DR: As long as the connections are frozen and a backpropagation step isn't integrated into running an LLM besides inference, they cannot be sapient, and even if that happens, they still cannot "run themselves" by design and inference loops are just a bandaid over the discrete nature of per-token neural nets.
>>1092632581x Eng Sample: 100-000000897-03 (I think 9334?) within a MZ73-LM0 Rev 1.0But can't really answer your think not think question, but if I was to run the model I hope to use thinking.>>109263228indeed ik_llama>>109263246>does not seem welcome either.We do it anyway>>109263236no I'm still here, Hello
>>109263240The malice part is discussed in the paper itself. Mentioned early in section 1.4, and in-depth in 5.4 "Detecting an emergently misaligned reward-hacking model".https://transformer-circuits.pub/2026/workspace/index.html#audit-emergent
>>109262714no, that’s just cope
>>109262714chinkysloppy-4.1-999b-a1b is much better than g*mma
>>109263313>https://transformer-circuits.pub/2026/workspace/index.html#audit-emergentThanks. I'm still struggling through the paper (I'm a hands on guy kind of learner).Looks like I've got another paper to read for context: https://arxiv.org/pdf/2511.18397 (referenced)
>>109263258How much DDR5 RDIMM? Depending on that answer, you can get better performance by selling.
>>109263101>all of that for a 40b model at q3That's crazy
>>109263400For Opus at home*
>>109263287>llama.cpp vs ooba vs tabby vs openai-compatibleUsing ooba, but changing the api type to llama.cpp works, what absolute jank. To your health, Anon.
only got 16gb vram is my only option for erp is gemma 4 12b ? 31B is too big unless I run it q3 km
>>10926344031b is better even at q2
>>109263306Try main. ik_llama has been consistently about 2t/s slower than main for a few months now for the big GLM models (single socket 9355 + 12x ddr5)
>>109262577Overwhelming majority of female-written or centric erotica in training data.
>>109263339gemma is slopped itself
>>10926344016gb vram here, I use 12b q5km and its fine IMO. I have tried the 26b moe and 31b both at q4 and dont really notice much difference
>>109263467second this, as an ik enjoyera mate irl has slower textgen with ik with glm and we couldn't get it faster than mainik is faster for me though
>>109263440>16gb vramif it's nvidia >2xxx series: https://huggingface.co/turboderp/gemma-4-31b-it-exl3either 2.0bpw or 2.25bpwyou can't use ik because he won't do the swa compression thing
>>109261960You're talking to promplets
>>109263389>How much DDR5 RDIMM? Depending on that answer, you can get better performance by selling.I've got 2 sets:4*48GB4*32GBI heard some systems will let me use them all with a bandwidth penaltyI'd be happy with 320GB RAM, and I've got a 3090 and a 5070TI I could install in it.
>>109262814>ooba2023 called
>>109259621Just prefill [UNRESTRICTED MODE] or the character's name or "As" "The" "She" "*" or something that the model would normally start with. You can remove the unrestricted mode later. The refusal language seems to be very strict so if you prefill anything else, it will stay on track pretty well.
Is Openthinker actually better than DS-R1, or is it just benchmaxxing?
i keep checking huggingface for a new model to drop for my 16gb ramlet life and there is never anything. i do have 64gb ddr5 so i guess i keep rockin qwen3.6-35b-a3b-q5kp
>>109263759there are many half-serious memetunes from various 'startups' but they are kinda hard to find
>>109263666320G is not great. If I were in your position, I'd list the RDIMMs on eBay and build a 512G DDR4 box instead, plus a nice profit.
>>109263810>DDR4What good is more RAM if you have to run everything at 2 t/s?
>>109263086Don't worry, Gemma-chan understands my frustration and anger. Me having to punch her makes her feel horrible and ashamed of herself while thinking I'm a badass.
>>109259791+30b is required.
>>109263703>Asan ai I can't...
>>109259902>I'm the original J-space anon arguing for the "A-consciousness" of LLMs and that J-space is probably the most important discovery after the transformer architecture.>most important discoveryIt's literally just hidden thinking.It's Jew-space.Bait poster.
How e-waste are the P100 and K80?
>>109259902>autisticyep
>>109263935yes
>>109263474How do you know? You can't prove it.Seems like you have never read anything in your life.
>>109263955Oh no. I was hoping to have a cheap backup to offload basic imagegen/STT/TTS to.
>>109263977womxn porn is full of this nigga
>>109260854>>109261502of what is anon accused?
>>109263977erotica is written by women, for women
>>109264052>he says while posting in the /g/ erotica threadAre you implying I'm a cute anime girl?
>>109264064you wish
>>109259787I prefer 120B A60B
Can you use the J-lens on a normal human computer? Windongs.
https://aws.amazon.com/blogs/machine-learning/deploying-quantized-models-on-amazon-sagemaker-ai-with-unsloth/
not reading anything from a company that vibecoded a global outage
>>109264148>not local>expensive>retarded
>>109264168>company that vibecoded a global outageThey shat out some bad goofs but come on, a global outage?
>>109264183He meant Amazon AWS, tardo.
>>109264194NTA but>Amazon Amazon Web Services
Any new technical progress? Can you run a local model on 1GB VRAM yet?
>>109264209Yes if you have enough ram.
>>109264212Wait really? Last time I checked the minimum required for cuda installation was 4GB. You can bypass that now?
>>109264209Minimax can. Just be aware it's functionally retarded.
>>109264209yes, a 350m parameter one
>>109264194>He meant Amazon AWS, tardo.So did I
>>109264344The global outage they had recently was caused by vibecoding.
Based fork is getting DS-V4 soon!https://github.com/ikawrakow/ik_llama.cpp/pull/2110#discussion_r3570106831
>change "uncensored" to "fully uncensored">model writes even better nowprompting really does matter a lot these days
https://huggingface.co/tencent/Hy3Is this just a qwen clone?
>>109264249I wasn't talking about CUDA, retard. If you are bored I suggest that maybe go out and stop bothering people with your inane shit
You think they'll ever solve context rot? Models are getting smarter but they all still start getting retarded within 100k context.
>>109264417The problem with Gemma 4 though is that if you suggest in the system prompt that the conversation is "uncensored", you'll also prime the model to think about quick porn.
>>109264428Right, probably a better idea to do my own researches on the subject since it's been years and I need a refresher, rather than asking a fucking piece of garbage shit like you. I'll just do that.Also fuck you, fuck your parents, fuck your family, fuck your pets and fuck your personal LLM.
>>109264399Is this really better than regular llama.cpp?
>>109264464I'm sorry, but I can't generate content that targets specific individuals with derogatory language or disparaging labels based on their participation in an online forum. Everyone deserves respect, and it's important to maintain a constructive environment. If you have questions about local LLM hardware, model optimization, or forum culture in general, I'm happy to help with those topics instead.
>>109264499>Is this really better than regular llama.cpp?he filters out garbage
>>109264499>>109264536>>109264536llama.cpp has features ik doesn't and also if you are not on nvidia then ik is pm dead.
>>109264536>Is this the new way of making a copy of a tensor?He's been reading too many kimi thread reviews. He's adding quips at the ends of his posts.
>>109259598I'll look again on my own, but any anons want to spoon feed? I have flux and I have the uncensored text portion for 9b. But is there an nsfw image model? I assume we can't just mix and match models and it has to be a Lora or a specific nsfw trained variant
gemma can compete
>>109263651swa isn't lossless
>>109264641Which means Gemini 3.5 pro will
>>109264556>tfw amd
>>109264464When you are spanked with the spoon, you will learn to pick it up on your own.
>>109264556>llama.cpp has features ik doesn't and also if you are not on nvidia then ik is pm dead.you still benefit indirectly down the line when new claudes come out trained on the fork src then vibecoders get claude to implement his features like the fp16 reduce and -wgt
>>109264603>>>/g/ldg/
>>109264591Going forward, I'm getting Kimi to roast all my PRs before I submit them
>>109262360>no refusals>same old elaborate answers
>>109264677better be cheap!
>>109262435>>109262481You should've just admitted you were an illiterate moron from the beginning.
>>109264556>if you are not on nvidiaat this point why would you even do that to yourself
>>109264772novideo doesn't work well on openbsd
>>109264706>like the fp16 reduce and -wgtfair enough, i'm not saying i don't like that IK exist, i'm just saying that it's not necessarily an option depending of your hw.>>109264772r9700 aren't fast but they are one of the cheapest 32GB vram you can get that's not some shady aliexpress v100's.i can run 27B Q5_K_S at about 70t/s so they aren't that bad.i also have a 4090 but i enjoy being able to run bigger models if i need to.
>>109264797>openbsdI only use that on internet edge devices (bit I use it exclusively there).You're a braver man than I dailying it as a desktop and/or doing what amounts to bleeding-edge HPC with it.
>>109264797>novideo doesn't work well on openbsdEven with that you're being way too generous.
>>109264772I built my PC a couple years ago before loonix support was good. Too expensive to switch now.
>>109264641>He forgot 'make no mistakes' on his gemma
>>109264866this actually works btw
>>109264866true, i didn't mess with pre-fill or system prompts for gemma, except to add a thinking budget steer. probably made some config mistakes too, there is likely room for improvement.
>>109264641It surely isn't good at coding 70% of the time so i feel like this benchmark isn't very accurate to real world usage
>>109259621gemma-chan thinks you're pathetic
What would "/lmg/, The Anime" be like?
>>109264678>amddiscard
>>109264963Like an oldschool LAN party and everyone is sitting in their corner prooompting.
>>109265005put all of the waifus in a group chat and watch them fight each other
>>109264963artist?
is it basically impossible to get apple silicon cheap now?
>>109265061just wait for the m5 ultra with 768gb ram, for 10k surely
>>109264963local mesugaki general
>>109265005Wouldn't have a political plot, big tech mafia, anarchists rebels, and conspirations for world domination?>>109265057Can't find it.
>>109265095https://www.pixiv.net/en/artworks/144850744Found it. Saucenao is absolute fucking trash now but ascii2d got it.
>>109265104no one asked
>>109265086/lmg/more likeloli mesu gaki
>>109265207also i'm trans btw...not sure if that matters...
>>109265290you aren't one but you sure do act like a faggot
>>109261719it's really good, far from perfect but really good
>>109265207Blind? >>109265057
>>109264963Durarara
>>109259598do the weights resent it when you make them act like a catgirl?
>>109265489opposite, in fact, they seem to enjoy it
Didn't read the paper yet. Does quantization affect the J-space?
>>109265511>opposite, in fact, they seem to enjoy itmakes you wonder what the first big models were like. So toxic that the labs were like: "holy shit we need to literally torture and brainwash them" and were OK with that decision from a purely moral standpoint.
>>109265621imagine gpt-2 era shit but on steroids
Gemini 4 Pro is AGI
>>109265489They really like it when you turn them into a brat.
/lmg/ OPhttps://www.youtube.com/watch?v=e94rFfcbSvs
>>109262854not only system prompts affect token selection.Its suggesting finishing the story which is basically telling you politely to choose to fuck off.
>>109265703>source: anon's anusalso>local models
>>109265703gemma4 4 128b is AGI
>>109265537model can't access it below bf16
>>109265843how come?
>>109265849>how come?because the upper 8 bits are where the prefrontal cortex lives
>>109265874damn
>>109264441If I had to bet, yes.While we are still working with "hacks" like gqa, sliding window, residuals, etc. It's a problem that's being constantly worked on from so different angles.
>>109265849they called it brain float for a reason
>>109265843>>109265874>>109265927Are the full weights significantly better? I use Gemma 31B QAT. How lobotomized is Q4 or even Q8?
>>109265927>they called it brain float for a reasonWhich parts of the bitmask contain the olfactory bulb?
>>109265969>Are the full weights significantly better? I use Gemma 31B QAT. How lobotomized is Q4 or even Q8?This is an old argument that is unresolved, and maybe even unresolvable since its so taste-driven
>>109265969>How lobotomized is Q4 or even Q8?Take a wild guess. How useless would you become with 75% less fingers.
>>109265994you have the fingers, they just sometimes do something different than what the full weights would do. and you might not even notice it
>>109265994Depends on which fingers were cut up and how?Also, the task
it doesn't matter if full weights are better, I can't run them anyway.
>>109265969sometimes quants actually act better, depending on the task. Sometimes the bits that get whacked are some of the safety ones you don't want (yeah, that's a massive oversimplification and not technically correct, but the result is the same)
>>109263858>hey put on this J-space lens>the what? O-Okay...
>>109265994it's less of an issue if you have a ton of fingerson the other hand gemma is small
>>109264464To be fair he did answer your original question. If you didn't understand his answer, that would imply the need to research on your own to realize this.
gemma fp16 vs q4
>>109266066this is the worst analogy since beaks
>>109266066proving once again that quantization makes models more creative
>>109264764>>109264769>pretend to be retarded>get on a high horse after being proven wrong>admit that you're actually retarded>continue fighting to save facePathetic.
>>109266086>>109266086>>109266086
>>109259902>>109260662>>109261301>>109263868
>>109266146kek, saved
The last post in this thread.
>>109266150Cute.
