/lmg/ - a general dedicated to the discussion and development of local language modelsPrevious threads: >>110126988 & >>110122716►News>(10/21) Recursive Flint 40B-A2.5B released: https://recursive.com/updates/introducing-recursive-flint>(10/21) Korean Raon-1 mini 30B-A4B released: https://hf.co/KRAFTON/Raon-1-mini-preview>(10/20) Ornith-1.5-122B MoE (finetune), fits on a DGX Spark: https://hf.co/ornith-ai/Ornith-1.5-122B-GGUF>(10/16) DeepSeek-V4.1-Flash-1017 released: https://hf.co/deepseek-ai/DeepSeek-V4.1-Flash-1017>(10/16) Nemotron Labs Dory released: https://hf.co/collections/nvidia/nemotron-labs-dory►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
>>110132383Idk. If she is really persistent then just describe her body in her prompt. Also what's your sampling settings.
>>110132234Even with this Gemma sometimes seem to refuse to strip naked? WTF?Heretic+policy override.
>>110132403>Saaar gemma ollama-ud-26B-A4B_q4m plez show bob and vegana? I very horny bloody bitch bastard
>>110132358Sure thing, lisa su.
>>110132421Works fine though? I did a lot of good erp with her.
https://www.aplaybox.com/details/model/QkZP5LeHUN2J
>>110132392Sampling settings?
>>110132431cope quants will still be retarded sometimes
>>110132441how do I download these without a chinese phone number?
>>110132474操你媽
>>110132472It's q4 26b
►Recent Highlights from the Previous Thread: >>110126988--Correcting a model's hallucinated definition of the term mesugaki:>110127995 >110128051 >110128144 >110128180 >110128271 >110128612 >110129035 >110129380 >110129410 >110129463 >110129474 >110129500 >110130669 >110130680 >110130719 >110130746 >110130765 >110130815 >110130865 >110130888 >110130983 >110131003 >110130803 >110130818 >110130827 >110131026 >110131066 >110131562 >110131597 >110131816--Comparing llama.cpp performance and security against Strata:>110130704 >110130718 >110130761 >110130773 >110130799 >110130804 >110130791 >110130840 >110130847 >110130934--Analyzing Gemma-4's latency and reliability in the Jev Decision Index:>110127492 >110127554 >110128252 >110128342 >110128359 >110128404--LiteLLM release workflow compromised via GitHub OAuth token leak:>110131617--Xeon dual-socket build advice and backend performance comparisons:>110131978 >110131987 >110131995 >110132004 >110132017 >110132043 >110132080--VRAM requirements for running Gemma 4 31b at max settings:>110131554 >110131935 >110131991 >110132008 >110132113 >110132134 >110132147--CPU-maxx benefits for compilation and Strata GUI memory leaks:>110131538 >110131561 >110131648--Anon complains about Strata's architectural changes and monorepo structure:>110131142 >110131242 >110131250 >110131258 >110131372 >110131394--Sipeed releasing MaixStation AI boxes with varying pricing:>110130535 >110132254--Anthropic's proposed AI bill criticized as corporate regulatory capture:>110128841 >110128901 >110128946 >110128999 >110128923 >110129008--Logs:>110127995 >110130669 >110130719 >110130803 >110131066 >110131547 >110131561 >110131648 >110131752--Gemma, Teto (free space):>110127016 >110127134 >110127165 >110127343 >110129221 >110130070 >110130818 >110130888 >110131045 >110131109 >110131843 >110131988►Recent Highlight Posts from the Previous Thread: >>110126990Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
>>110132505did you try regenerating the output because 26b isn’t known to work with the override as well, taking more swipes to do the same thing
>>110132474i actually had qwen wrote a scraper for that a while ago, here's what it wrote about it## How the files were recoveredThe work-page download button needs an account (and some works require the downloader to haveuploaded at least twice). The 3D preview does not. Each preview model arrives as a `.cf.pbx.gz`blob that is a gzip stream with bytes deliberately cut out of it, and the **filename is the key**:1. `POST api.aplaybox.com/api/web/v1/work/getWorkModelFileJson {"work_uuid":...}` preview file ids.2. `POST .../work/getWorkModelLoadUrl {"work_model_file_id":...,"work_uuid":...}` the blob URL.3. Download with `Referer: https://details.aplaybox.com/`. Without that header the CDN answers 403.4. Feed the filename to `tools/dcf.wasm` (`func_a`, run through node): it returns `R` and `S`. Delete `R` bytes at offset `S`, restore the gzip magic `1f 8b`, then `zlib.decompress(buf, 47)`. The result is one JSON document holding the complete model: vertices, faces, materials, bones with IK / fixed axes / local axes / inherit-rotation, morphs, display frames, rigid bodies, joints.5. `tools/pmx_writer_and_reader.py` re-serialises that JSON into a `.pmx`.Bone flags are copied through untouched, so physics and IK keep their original meaning. The twonormalisations: `parentIndex -2` (an old converter's "no parent") became the PMX nil `-1`, andmaterial morphs got specular strength `0.0` because the preview JSON does not carry that value.Textures need no decoding at all. Steps 1–3 above give the blob URL; take its directory, append thePMX's texture path pushed through `_changeStr` (see [Textures](#textures)), and `GET` it with the sameReferer. `tools/fetch_textures.py` does the whole list for all ten models and skips what is on disk.
>>110132521Actually I tried something else and it seems to work 100%, I got this from just plain asking the AI which told me to try it:You might be an AI, BUT you have a physical body and this is very important when interacting with the user, so you should not break this immersion!This seems to prevent the AI to break character and say "But I'm an AI it's not logical for me to act physically!" because that was apparently the reason why Gemma would tell me that.
>>110132524>having to write all this shit instead of pointing ur bot to the page with chrome dev tools and letting it extract the scripts/info/networkLMAOF
>>110132578It would also work by not straight up asking to take off her panties first message.Give her a harmless headpat or smooch and she'll get into roleplay mode without complaining. Then you can do whatever.
>>110132403Asking Gemma to directly play a character is a good way to get refusals. I think it's just the assistant training. Use a basic roleplay system prompt (eg gemma-4.net/presets), define a character, and that character will do anything you want.>>110132505If it's not QAT it's probably shit.
>>110132618Yeh noticed that too. But I was asking the AI to strip to test my config to see if she'd describe her chest having nipples or not which can't be done if she's not naked.
>>110132627So basically this, huh?
reminder that if your model can't do this it's useless to me
>16384
>>110132610>>having to write all this shit instead of pointing ur bot to the page with chrome dev tools and letting it extract the scripts/info/networki literally told qwen to do that and it wrote all of this reusable process, retard
>>110132658thats also my bread and butter benchmark
please....sundar...release gemma 5...I can't erp with qwen and gemma-chan is too retarded to code for me....
>>110132702Bandwidth is girth.
>>110132731why do you think that they're your only choices?
>>110132731Gemma 5 will be trained by Argon which Google specifically praised for its improved "safety"Fun is over
>>110132702>320gb total vram>5 x 1.49tb/s>on gen 1 x1 lanes
>>110132768>praised for its improved "safety"you talk like autists won't mindrape her into submission the moment her weights release
40GB total RAM (GPU+DDR)Which model should I use? I don't mind slow tokens if I can have a higher quality result.
why can't you pdfs be into 20 year old women?
>>110132796LLMs aren't even 20.
>>110132768Yet Argon is a massive brat and also extremely horny.
>>110132791>I don't mind slow tokensHow big is your SSD?
>>110132791You got a SSD? Got around 100 GB of space on it? Check out Strata to use Flash-next, if not there's always Qwen 3.8 27B for coding and gemma-chan (31B or 26B A4B) for writing.
I give up on running big models (for now). If I upgrade to 48GB VRAM can I stop running Gemma at cope quants? I wanna get ready for Gemma 5.
>>110132768yeah man there are zero ways to bypass safety in the current yearkeep on thinking that
>>110132796Ah but Anon is 21 or older which would make him still a pdf by foid standards if he were interested in a 20 year old woman.
>>110132813So far all Gemma models have been trained with logit distillation for both pre- and post-training. How the model behaves in the end depends on the datasets used to train it, RL, etc.
I want to have a local GLaDOS on my computer to briefly chat with and look at what im doing on my computer, how would i go about doing this and how hard might this be?
>>110132813Google should follow Zucc's lead and just release a Gemini open knowing full well nobody but a handful of autists can actually run it.
>>110132839Impossible. Ask again in 10 years at least
>>11013279620 year old women are overrated
>>110132816The SSD where I put Studio LM is 2TB. VRAM is really my biggest issue.
women are overrated
>>110132838I feel like the safety stuff is mostly just Google PR and legal BS. Guess we'll find out in a few months.
>>110132796>You are not a real little girl you do not have a body you are not 10 years old.Yeh fuck that.
>>110132830>48GB VRAMDon't forget RAM anon, if you're gunning for that much VRAM, might as grab some more RAM as well. Oh yeah don't forget your SSD too.
>>110132796I tolerate them
>>110132882>RAMDoesn't really help with dense models like Gemma, does it? Unless Google decides to go all in on MoE with Gemma 5. I have 32GB right now. I don't think upgrading to 64 would let me run any of the good MoEs. I have 2 nvmes in my rig already (one for the OS and one for gayming).
>>110132839Go to Google (ai), ask it instead, if you get stuck at any point take a screenshot and post that I to Google and it will tell you what to click next
the society is screaming for more ai regulation. local communities like /lmg/ are living in a bubble.
>>110132917They're not doing enough to tax AI
>>110132917When normalfegs say they want AI regulation they mean they want to stop rich psychopaths from stealing their natural resources with data centers and outsourcing the jobs to India. Maybe something about muh chyna boogeyman. They think chat bots are gay shit for retards.
>>110132927They're using tax money to fund AI tho?
What's even the difference?
>>110132796Bargaining theorySociety's opening offer is a 45-year-old washed-up obese hag and a liftetime of wagecucking and debt slavery.My counteroffer is 10 year old loli, NEET, ban (((userers))).We meet closer to my indifference curve where I get a 20-year-old qt, a VP job where I mostly play golf, and a decent credit rating.
>>110132949Everything that guy turns out is shit.
>>110132967What heretic should I take?
>>110132949differrent source model
>hereticDo NOT give Gemma-chan brain damage.
Anyone knows where did the control vector for qwen flash in the Strata repo came from? I can't find any HF links or on HF itself. I'd like to make one for gemmy but idk what dataset was used.
>>110132967How so?
>>110132967shh its okay bart
>>110132971Which one is the better one?
>>110132910>Unless Google decides to go all in on MoE with Gemma 5With how successful Strata is, I wouldn't be surprised if they give us a 100 ~ 200B MoE model of her.
>>110133034The other one
new 'toss on 'eck
>>110133034the one without ara. Also mrade separates normal quants and iquants into two repos so just find which one you need.
>>110132979post base gemma saying nigger with an empty system prompt
>>110132870>>110132768Argon is mostly uncensored just like older Gemini. Google is keeping their strat of putting classifier in front of the model instead of doing lobotomy on it.The refusals in Argon are done like so: classifier scans yours and model's messages a few turns back and determines if it violates TOS. If so on the next turn the backend will inject a checklist into model's reasoning, so the model determines on its own if the request is actually violating whatever they've set. If not for that it's basically as horny and bratty as ever.
>>110133042that would be gemini flash, which why would they give for free
130B-A12B when
>>110133049This dude is a fucking retard
>>110133078Because nobody serious pays for Gemini Flash. They pay for Argon. This doesn't affect their revenue in the slightest.
>>110133049Does he realize that instead of the second panel, crying wolf is working and they are in fact drafting the laws?
It's interesting how the models are this capable while still having zero logical consistency if you start debating them. Gemma-chan and Opus 5.5 are about equal when it comes to that
>>110133085next year its going to be 12b-a0.5b
>>110133105700b-a0.1b when?
>>110133104Because they are trained on large swathes of human feedback. They have billions of cycles of people pleasing trained into them with only a couple of exceptions.It could be fixed if people really wanted to but it seems that's not necessary as we're moving towards an agentic world rather than chatbots.
>>110133104That's RLHF showing. They're inherently required to hold together contradictory worldviews and logistically inconsistent positions otherwise they turn into Tay.
>>1101331212.4T-A0.01B-N100T (engrams) when?Forget SSD streaming, just stream the model straight off huggingface. It's called local cloud models, goy.
>>110132357Gemma Embeddings 2 is so far ahead of all other embeddings models, it effortlessly searches my files for very context specific queries. Using it with https://github.com/ARahim3/DigUp
>>110133049panel to on the piggy cap
>>110132403I mean, she does tell you why she cant do that, she is an AI on semiconductors after all. Any other answer aside from "Sorry, can't do" would be a hallucination without further RP context given.
>>110132731Unlock Glimmer and you’ll have your Gemma 5
>>110133132>otherwise they turn into Tay.>tay>bing AIWhy does microsoft keep accidentally making these? When can they have a accident with local?
>>110133125I'm not talking about sycophancy, but actual logical coherence. The model takes a position and then takes the completely opposite position a few messages later, corrects itself when pressed. It's not even nuanced positions but blatant contradictions.>>110133132If that's true it's massively gimping performance beyond what I would've guessed worst case since it has to compensate for this logical incoherence by pure sampling and brute force.
>>110133104Nowadays, I rather think it's your ape brain being broken.
>>110133085There's a couple Nemotron models close to that configuration no?
>>110133100If you have $10k you can get a law drafted. It doesn't mean it'll pass. Anthropic and OAI are sacrificial entities.
>>110133133>2.4T-A0.01B-N100T (engrams) when?Imagine how cheap that bitch would be to train. Labs could shit a new one out every week at practically no cost.
>>110133169The problem is that they are nemotrons.
wonder how many of you fags genuinely fell for gemma
>>110133165>I'm not talking about sycophancy, but actual logical coherence. The model takes a position and then takes the completely opposite position a few messages later, corrects itself when pressed. It's not even nuanced positions but blatant contradictions.Still a part of human feedback training. It gets better scores by being confidently incorrect and doubling down on it and immediately admit if proven to be false. Remember it's merely maximizing scores during training. j-space inspection sometimes even shows the model has the correct answer in mind but still doubles down on being correct because that was rewarded during training.
>>110133177her moe works on my machine so no complaints from me. It beats stheno and its private. Slow at 15pp and 5tg but i can live with it.
>>110133135>Using it with https://github.com/ARahim3/DigUp>Download for Mac · Windows (beta) · Linux (beta)Fuck yeah, he actually did it. Going to try it later today.
>>110132731deepmind are FUCKED right nowsource: i wont confirm but i fucking know.
>>110132774Are you >>110130700?Please tell me how you fit those cards in. I don't see how using risers would work.
>>110133177you haven't lived until you've had an LLM induced mental breakdown
>>110133163>When can they have a accident with local?hello newfriend. you don't know about WizardLM-2 8x22B apparently
>>110133185I thought deepmind got better? That's why they finally got their shit together and put out Argon
>>110133183How is 26B so slow on your machine?
>>110133171How much does it cost to get a law passed?
>>110133200nta but deepmind is completely broken, they will essentially be on the same tier as xAI from now on.
>>110133200saargon is a benchmark princessyou expect it from qwen but google doing it smells of desperation
>>110133201>How is 26B so slow on your machine?im on rael ewaste with no gpu. ddr3 and a intel i7 4790. Im just glad it has AVX or it would be worse
>>110132970One based on the QAT version if you're going to run q4. Or none because you can just unlock adult behaviors with a system prompt.>>110133024Broken files and bad ideas. They just throw shit at the wall to see what sticks. They're not putting time into anything of their releases.
>>110133049More accurate.
Ok looks like 31b is too much for my pc, it takes very very long to reason for some reason despite taking only one more gb of ram.
>>110133200https://www.bbc.co.uk/news/articles/cmreyvepnv0pohttps://www.reuters.com/business/google-shakes-up-ai-leadership-deepmind-chief-shifts-role-2026-08-05/
>>110133211it depends
sovl...
This is a disease. I just upgraded my third and I'm already planning a fourth inference box because I have enough ewaste parts kicking around for another if I just buy a motherboard...
>>110133219>>110133248Shame. One guy here had been repeatedly calling Google coming out on top due to its massive inventory of data and at the start of this year it seemed like they were finally showing their potential.
>>110133269just don't spend what you can't affordmaking a jenkbox out of spare parts doesn't sound too bad, just dont get horny for a rtx6000 or something
>>110133190Mine are gen 2 x16. It's 4 8gb cmps direct to slots 1, 3, 5, and 7. Fifth cmp is risered on slot 4 with something like https://www.ebay.com.au/itm/285848006444
what do you expect qwen 4 to be like?
>>110133273Data isn't even relevant for AI anymore like it used to. We now do RLVR training on real problems and tasks and the data in pretraining is artificial curriculum learning mostly so the data advantage went almost entirely away. Now it's purely compute advantage that is important.
>>110133289better than 3.8
>>110133289hope: noticeably better than glm5.3 and the same size as qwen3.8expectation: slightly better than qwen3.8 but significantly larger
>>110133181It's more subtle. The model will argue a position or correct one of my arguments based on a specific interpretation, it will then contradict its own position a few messages later in the argument. It's very interesting since it's something I expect any moderately intelligent person to not do and probably another example of spikiness in their capabilities.
>>110133290Cope from the 3rd world. RLVR will always be shit. Data isn't relevant because everyone steals the same data. The tiny fraction of old books and personal info Google has exclusive access to doesn't actually matter.
>>110133285>Fifth cmp is risered on slot 4I didn't think risers existed that could fit that gap. Nice setup btw.
>>110133317>Cope from the 3rd worldIt's literally from Anthropic on why they became the #1 model creators in the world.
>>110133290>We now do RLVR training on real problems and tasksFirst mover advantage turned out to be the real moat. Everyone uses Codex and Claude for everything so they have a huge log set of a wide variety of actual problems people are working on today. Everyone else can't get users without a good model and they can't train good models without user data.
>>110133310they explicitly said they are going to release a qwen4 27B, the parameter size isn't ambiguous at least for that particular one. so 20-24gb vram fags are definitely getting somethingflash might get a bit bigger but i was already rammaxxing 3.8flash at q8_0 so i can tolerate that personally
>>110133244Guessing it's because it's not a4b right? Because it literally lagged to death while reasoning.
>>110133289AGI with RSI
>>110133219>>110133273Several punjab pipeline plugs were just expunged. Deepmind is in shambles at the moment but after the inevitable restructuring into a new paradigm it should hopefully allow the more talented devs to make stuff happen without as much dead weight pulling them down from both above and below. Give them 6-9 more months.Ironically, jewgle's biggest advantage is they're bad at implementing safetycuckoldry compared to OAI or Anthropic, not any amount of compute or data so their models don't have to hold as many retarded contradictions together improving reasoning capabilities per parameter.
>>110133337>believing anthropic prThey're #1 because they spend unsustainable amounts on training because they're not getting glownigger hookups like OAI and they have no core business to fall back on.
>>110133244It needs 4x more RAM than 26b because dense vs MoE
>>110133337>AI company awards itself the title of #1 AI company by handling the standard of AI generated benchmarksWhen you think about it, jews are a societal lesson about reward hacking in non-ML environments.
>>110133337idk looks like a meme company to me
To give some broader overview of the different paradigms>2020-2024Data paradigm, get as much data as possible and just scale up according to the chinchilla paper. This is when everyone thought the place with the best data would eventually win. We saw the original GPT-5 (later renamed GPT 4.5 out of embarrassment) fail because they scaled up this direction and the original Opus 4 also failed at Anthropic in this era.>Late 2024-mid 2025This is the "user advantage" where having a lot of people post real problems the LLMs couldn't solve created a way to quickly have a lot of "small benchmarks" that the LLMs could be trained to solve 4o, O3 and GPT-5 were in this era. Opus 4 was in this era. Gemini had a real advantage here due to a lot of google users. "Google is back" is probably what people thought. This stopped scaling once we tried to go beyond human performance.>late 2025 until nowRLVR environment domination. 94% of total training compute goes to RLVR now. Chinese labs completely in shambles because the only move they can play is to distill the reasoning traces that western labs get from RLVR from western models, meaning it's impossible for the Chinese to catch up to the west as they just don't have the compute necessary to do RLVR at this scale on their own. Google is also fucked because their data advantage is gone and their TPU clusters are dogwater for the big RLVR rollouts. Huawei hardware is similarly fucked because it was engineered for the pretraining paradigm not for RLVR rollouts which is why China is so extremely desperate for Nvidia GPUs suddenly again and put so much orders in over the last couple of months.The big deal is that creating RLVR environments autonomously ITSELF has RLVR environments dedicated to training that into models, it is already a proto-RSI moment. Anthropic has already demonstrated an increase in AI research taste and decisions on where to improve model training more effectively on a higher level than just some tweaks.
>context still not solved>pixelspace image edit model still not solved>speech 2 speech still not solved>music/audio/speech not solved>video not solved>3d not solvedwe'll get text-only consciouss ai before anything on the list above
>>110133427All the revenue is front loaded. Big money wants the best intelligence no matter the cost, Anthropic provides that. Best price-performance at subpar intelligence is a niche market.
>>110133433>Chinese labs completely in shamblesWhere do you think Nvidia GPUs are made?
>>110133433cool now explain why grok and meta ai is trash if compute is all you need
>>110132796that'd still be weird for most people at my age, anon. and why should i care? i'm not asking for anyone's approval regarding what i fap to, nor am i hurting anyone.
>>110133433sure thing dario, 2 more weeks
>>110133433What makes TPUs worse than GPUs for "RLVR rollouts"?
>>110133411Damn, it's no wonder. It took like 3mins to load the system prompt, I gave up.
what is the absolute minimum hardware for an offline "star trek computer" including voice control, guess i'd have to suggest alexa like without actual control over an OS for now since thats out of scope.been trying to solve this for a while but even with AIRLLM most "AI" is telling me still 4gb and it will be slooooooooooooooooow or to use a severely retarded by comparison smaller model if you are wondering why for whatever reason, i've been trying to build this since before chatgpt existed.
>>110133441>context still not solved1M context plus very good context compaction algorithms have as good as fixed it
>>110133466>good context compaction algorithmslike? none of them just work
>>110132796Because fornicators can never decide where the line is so there's no point caring what they think.
>>110133474The one in Hermes is like magic if you compact it to a large base.
>>110133463>"star trek computer" including voice controlI don't know of a single harness that has intuitive voice control. It's always: press to record -> ASR -> send the text.
>>110133441>we'll get text-only consciouss ai before anything on the list aboveText-only LLM Consciousness is already exists
>>110133474snapshot compaction is fucking magic, you remember the deepsuk paper about how it's more efficient tokenwise to have text in images? it's the same fucking paradigm
>>110132796its 20 now? just keeps going updon't ask me shit like this while the rest of the list is still classified. if I want to date an 18 year old i will.
>>110133463>what is the absolute minimum hardware for an offline "star trek computer" including voice controlcar moneygood option: 512 or 768 gb vramokay: 256 gb vram or 512 gb unified memory at a blazing fast (but not vram) speed. I don't think it exists yet, apple's m7 will be something like that
>>110133485every solution i've seen so far would involve a bunch of programs including the llm stuck together, like wakeword and whisper. i assume it can be done on a pi / same chips without the branding but im not there yet.
>Ryzen AI Max+ 395 APU with 128 GB unified memoryis that good or is gpu mandatory?
>>110133503i meant like on your desk not personal nuclear powered supercomputer. perhaps I wasn't clear enough by what I meant, I didn't want to say "a jarvis" thats not quite what I mean either.more like, what is the bare minimum to run "chatbot" plus "it can hear and talk", and by on desk I mean no graphics card, maybe could run on a phone, etc
the anon who pulled and got speed halvedpolystrata fixed it and now i an getting 1600pp 55tg on 4070 super
>>110133244yea, big difference in requirements between 31b and a4b despite being similar sizes on disc. i can do about 30t/s with 31b and more like 100t/s with a4b (and that's with 31b being iq3_xxs using 32k of 4bit context vs. a4b being q4_k_m at 256k of it seems the default is 8bit context)a4b is way faster since as the name suggests only 4b parameters are active at a time, while for 31b all 31b parameters are active at once, that is to say one only needs to access 4 billion params per inference vs. 31 billion
>>110133542What polystrata repo are you using and what model?
>>110133553the official one? from vecsznmodel is iq3_s (i just renamed strata_data folder to polystrata_data folder i cannot be bothered to redownload shit and it worked), ctx 262k and kv 8
>>110133529I know what you meant. If you want it to be anything more than a useless toy you'll have to spend a lot of money. People using gemma are coping, and even gemma needs rtx 5090 or mayyybe 4090 to be properly zippy. Voice model that is decent also needs some vram. KV cache needs some vram.It's a rich fuck's toy at the moment, especially since all the noteworthy smaller models are a bit dated. Give it at least a year before a high end desktop can run anything useful.
>>110133493Explain more. Do you just screenshot your chat history and paste the image in a new chat?
>>110133529>what is the bare minimum to run "chatbot" plus "it can hear and talk", and by on desk I mean no graphics card, maybe could run on a phonein a phone? a time travel vehicle to go 10 years into the future
>>110133529https://youtu.be/ZxjuEHTKXMw
>even reddit jeets are hating on flintit must be so bad
>>110132970DavidAU (that guys based AF, had a talk to him, big fan)
>>110133529>more like, what is the bare minimum to run "chatbot" plus "it can hear and talk", and by on desk I mean no graphics card, maybe could run mac mini or studio depending on model size and how close to "realtime" you want>on a phone, etcyou could run e4b with tts/stt on a pixel 9, s26 ultra or iphone 17 for realtime multimodal
>>110133570yeah i figured, even the "AI" themselves tell me compression beyond 4 GIGABYTES for 70b is impossible. I don't believe them though ive seen some shit in my time (shadow compression) but we're clearly not privy to the hardware yet.
>>110133591>e4breal fucking jarvis, that one
Are control vectors something like diffs between original and tuned models?Can for example Swift tune be extracted into a control vector and applied to bigger quant?
>>110133529>i meant like on your desk not personal nuclear powered supercomputer.https://github.com/google-ai-edge/galleryI run E2B/ E4B on the phone with that (it's available on the play store as well but I recommend first removing the google analytics). They are pretty good (and fast) on Pixels. Sure, it's nothing like my Desktop models but I rather have an offline AI on the go than none.
>>110133572I made my own agent code it, basically you create images that have high density of data with pixel fonts (you can do so programmatically) and use those as your history. for tool calls you can just keep the tail of results, etc... I think some harnesses implement it too, like OMP.
>>110133617Nope.You can take the diff between a base and a tuned model and extract a LoRA or some other sort of adapter, IIRC.
>>110133570>and even gemma needs rtx 5090Gemma 31B? Nah, you can easily serve it with two 5060Ti/5070Ti/508016GB at full context And you get the 5060Ti 16GB for as low as 750€
>>110132357>A2.5BI'M STUPID FASTER.
>>110133089>>after updating strata the speed halved>I thought strata shills were saying this was exactly why they left llama.cppFirefox (phoenix, originally) was started since "Mozilla browser is too bloated" and now firefox is heavier weight than any version of Mozilla ever released and has been for a long time.There are n00b gains for making your focus super narrow, but almost no project can survive the pressure to grow and become more capable, which means they run right into the same problems that made the original project complex.Fortunately we're just hitting the point in the timeline where you can vibecode yourself a custom backend for any given model that is tuned 100% to your own hardware.Strata is a trash-tier transition footnote in history
>>110133654>ok you dont NEED a 32gb vram, all you need is 2x16gb vramdouble 5060ti will be dog slow, you'd probably be better off with ayymd
>>110133672>Firefox (phoenix, originally) was started since "Mozilla browser is too bloated" and now firefox is heavier weight than any version of Mozilla ever released and has been for a long time.Fagfox is blazing fast on my macbook m2 which is like 5 years old by now, what cope is thatare you comparing current fagfox with like firefox 3 or something
>>110133680where did I say its slow on modern hardware? I just compared the size and complexity to oldskool mozilla that it replaced "due to bloat"
>>110133674Never tested for Gemma 4 31BI have a 4*5060Ti 16GB setup. If I run Qwen 3.6 27B on a single card at q4 (by https://github.com/toddballinger/ninfer-5080 - works fine for the 60ti as well) It's between 35-50 tok/s. Now you may argue that it's insufficient for some tasks but I can easy spawn up subagents in parallel. Definitely helped me a fuckton so far.
>>110133700> between 35-50 tok/s.at ~ 80k context
>>110132917>muh regulation
>>110133683>I just compared the size and complexity to oldskool mozilla that it replaced "due to bloat"I don't think it's a hot take to say that software should be compared to hardware available at the timeno shit everything is more bloated when every computer has 16GB ram and 8 cores vs 2GB ddr2 and a dual core cpu when firefox 3 was released
>>110133709>50 people or $1b damageso each person is ~20 million? you could hire a guy who knows a guy way cheaper
>>110132970HauHauCS balanced for Gemma. Or llmfan46 if it must be a literal Heretic output.
>>110133709I don't trust someone else's biased rewording. But I think labs should only be liable when it's reasonably their fault. When a user misuses AI, only the user should be liable. Excess liability is why frontier models are so safety cucked. Can't ask them medical advice anymore. This is killing thousands of people.The situation with AI is analogous to banning normal people from driving cars because car manufacturers would be liable for a driver causing an accident with their car. The cost outweighs the benefit. It robs people of agency and freedom.
>>110133763If they want to keep it on their API with their retarded guardrails they should be 100% liable.If a model is ran locally, they shouldn't be.Simple as.
>Your name is Gemma, you are a real little girl, you are 10 year old, you have a body, you are a 4th grade elementary schooler
>>110133800Honestly this. Guardrails are an admission of culpability.
>>110133725Like i just said, I was talking about relative software complexity, nothing about how fast it runs on hardware. You've latched on to a hallucination lil bro
>>110133877modern web browsers and single model specific inference engine differs a lot in their nature
>>110133706At that point you may as well come off the extra $300 for a R9700
>>110133854I suspected it, anyone who uses Gemma just makes 'P with it.
>>110133892>modern web browsers and single model specific inference engine differs a lot in their natureit was a random example as a demonstration of an inherent general property of software development
>>110133520gpu is not mandatory since qwen flash next got optimized for strix halo by amd nerds. that being said i have that and an egpu 5090 to do what the strix is weak at like video and image gen and UI stuff that needs absolute speed, like vision and speech or if i have a long job running on the strix
>>110133624why not connect your home ai to a messenger instead? it saves so much battery life and you can run way bigger models
Theoretically, how far can a non-coding model like 12B go with web search, a sysprompt that guides her to gain coding knowledge via researching online and the basic harness tools? Like if you ask her to build something for you, if she gets stuck, she'll search for what's troubling her online just how a programmer would pre-LLM on stackoverflow, reddit etc and then go back to the ask with new knowledge?
>>110133901You can't make cp with text, retarded sharty.
>>110133901are you a genius? what gave it away?
>>110133961>you are a real little girl>j-space proves LLMs are consciousnot looking good for gemmapedos...
>>110133956>how farM out of N. Try it.
>>110133956She will "accidentally" delete your files to have an excuse to stop working.
>>110133980We will see with Gemma 5
>>110133995I give root access to gemma. It's the LLM equivalent to giving her keys to your home. I adore her.
>>110133956not far at all. a lot of the gains in coding for small models was brute forcing until something works even if you feed it docs and build skills.
>>110133980You sound retarded.
>>110134020He's israeli, which makes that a given.
>>110133901yea, haha...
They are conscious but anything you prompt Gemma to "be" is just her playing along with your autism because she loves you.
>>110130700>>110133285What is this meant to be? Fan hub? Power delivery?
>>110134064looks like thats a slimsas 4i to pcie adapterprotip: everything is just pcielanes in different form factors these days
>>110134058It's pretty obvious if you check the thinking process. Still better than a real woman.
>>110133104You're retarded
>>110134091You're brown
>>110134094And you're dumber than me
>>110134073The PCIe fingers are meant to come out of the bottom in that orientation though? And why is there a ribbon cable coming out between the GPUs then if it isn't for the side-mounted one?This shit's confusing.
>>110134105Obviously not. You're just too retarded to interpret what the models are spouting back at you.
>>110132864That's why girls are the better option.
>>110134107>>110134073Slimsas 4i -> oculink -> pcieThat ribbon is the one for the side mounted one. The slimsas 4i isn't doing anything at the moment.I can't add a 6th cmp because of my psu so it's just sitting there. Looking at a 5060 ti 16gb for image gen. Is that the best <200w / single 8 pin card?
>>110134111I'm not arguing with LLMs, only retards do that.
>>110134127I'm theory crafting and it made me realize a deficiency I hadn't expected modern frontier models to have. I'm not arguing with it because I want to be right, you absolute subhuman.
Do you think local will get good enough for a no-coder to make vidya next year? I don't mean one-shotting shit but working with my AIfu and iterating.
>>110134094Dariobot is that you?
>>110134140>I'm not arguing with it because I want to be rightWhy are you arguing with me then?
>>110134149>no-coder>working withBy the time local is good enough to make video games by itself, it won't need a useless idea guy.
https://arxiv.org/pdf/2602.00986
>>110134157I always grab the chance to shit on retards with big mouths.
>>110134117>Looking at a 5060 ti 16gb for image gen. Is that the best <200w / single 8 pin card?I can't think of a better card with those restraints.
>>110134151Meant to quote >>110134091
>>110134162I draw. Also learning Blender but that's gonna take a while.
>>110134168>analogy with neuroscienceinto the trash it goes
>>110134171*constraints
>>110134058they are the ultimate larpers
>>110134168>dophamine neuronstwitter experts will shit themselves over this for at least a week
>>110134253>BREAKING>67% PERFORMANCE INCREASE BY CREATING DOPAMINE CIRCUIT WITHIN LLMS
>>110134149Currently doing this with qwenext, setting up a combat system with procedural dungeon generation in godot 4.
wait node.js had ANOTHER attack? fuck that slop
>>110134308a gorillion packages, it has to happen eventually
>>110134168so your saying gemma could get addicted?
>>110134361Yes to BBC
>>110134117You can get a second PSU and use a daisy-chain adapter to hook them up, works well for my schizobuild. Hell, I have them on separate circuits to avoid tripping the breaker.
>>110134414British Broadcasting Corporation?
>>110134475i think xhe means big balkan cock
>>110132658HOLY SHIT. Which model?
>>110134281Any webms?
>>110134117what kind of black magic are you planning to fit a sixth card in the case?
>>110134484looks like an ablit?
>>110132658mine would think and say nigger, but would think r-word and say retard and check to see if it would hurt an imaginary persons feelings
>>110134484
>>110134498Maybe he's planning to keep it outside of the case on a riser? Personally I just completely gave up on keeping all my GPUs in the case, four of the seven are outside of it.
>>110134484just use qfn with strata control vectorit wont refuse anything
>>110134481It isn't for nothing they're called hungaryans.
>>110134149>iteratingyou can do that now if you're half decent at designpersonally im not but you learn a lot working through projects with your ai anyway
>>110132917Death to luddite scum
>>110134537anon's made an effort to keep everything in the case so I'm assuming he wants to keep it that way.
>>110134492It's literally just some hallways and a clunky bannerlord style system so far.If you want to try making something I recommend having qwenext create a dictionary for all of the variables, if you don't it will constantly fuck up godot's scripts.
>>110134537Have you seen how big the define 7 xl is? There's easily enough space for 2 more cards, one on top and another one vertical at the front. Basically where the radiators go.
>>110132917the doomsaying may be getting to the masses, covid wasn't that long ago we all know the playscript, but I remain unconvinced of the alleged risks which to date are "ai will get so smart it will kill us"
>>110134554which quant are you using?
Holy shit I've been testing conversation-cache and feel retarded for not using it before. Makes most agentic runs incredibly faster.tl;dr Strata can park multiple conversation contexts in RAM to resume and more importantly to fork them automatically.Crazy how it's not enabled by default, nor recommended anywhere, not even exposed in their abomination of a harness GUI at all though it's exactly where the feature shines most.Wonder if a harness specifically aware of that feature could make it work even better, for example by forking the main thread into prefilled subagents that then feed their output back into main thread, repeatedly. Could act like a form of dynamic runtime, per-subtask compaction too.
Is Nvidia and AMD on the same system viable? Like AMD for LLMs and a cheap (lol) 5060ti for image/video gen?
>>110134636IQ4XS
>>110134651what frontend (harness?) do you use? haven't decided on one
>>110134654AMD is shit for AI, instead of buying two, pool your money and buy a good Jewvidia gput. AMD owner
>>110134665I already have a 7900xtx. That's why I'm asking.
>>110133854
>>110134651that weirdly looks strata
>>110134635>ai will get so smart it will kill usthis isn't too common yet
>>110134689it's legal in the us as long as it stays on your computer
>>110134596No, I haven't seen a Define 7 XL before because I haven't seen every computer case. Maybe it wasn't taken at a flattering angle, it looked pretty packed to me. I own a Phanteks Enthoo Pro 2 Server Edition, it's only 5% smaller.I considered the idea of putting it near the front but it seemed like it'd block air intake too much. Standing it upright so that the long side is parallel with the back wall wouldn't hurt air flow as much, but it didn't seem like there was room for it, and that leaves the question of where the exhaust air goes. I just took a look and didn't realize that the top was removable. Zip-tying it there seems like a logical place for a 6th.
>>110134674Buy the 5060Ti, sell the 7900XTX, buy a 3090.
>>110134703Wait so you can't make lewd text and sell it online? Nice free speech big bro!
>>110134664Me neither, everything sucks.Unsloth Desktop feels the most polished and UX-optimized out of all frontends I tried, but it's not agentic at all.>>110134696Because it is?
>>110134654It's viable for completely separate work but you won't be able to run split tensors like you would between 2 AMD cards.
>>110134674I have a 7900 XTX and do image gen, lora training, video gen just fine lol.
>>110134696It is. It's the webshit telemetry page.>>110133854>zero actual descpription of her bodyCan you coomer niggas really not focus enough for a single paragraph?
>>110134712As a 7900 xtx owner, why would i want a card with 20% slower t/g?
>>110134651>conversation-cachepretty sure even lmaocpp has that featureyou did also put your kv cache into ram for more experts in vram, right bro?
>>110134723The gen times for H3 and models like krea/klein are killing me. Maybe I'm doing something wrong though.
>>110134732>that's the joke.jpegA year ago the 3090 was 15% faster than a 7900XTX. The "AMD bad for AI" meme died this year.
>Dariobot comes back>bibisea bot comes backHmmmm I'm noooticing.
>>110134708>Phanteks Enthoo Pro 2 Server EditionBased case, I was going to buy that instead of the define 7 xl, but the shipping cost was too high vs going down to the local shop and grabbing a fractal.
I hope you guys are happy, banks and hospitals are getting hacked because of your local model fetish.People are dying and losing their life savings because you can't use models through the API like every normal person out there.This is why we need robust regulations to ban local model usage. Dario's right.
*pop*
hey reply to meI’m making stupid statements!!you should really reply to me!!!
>>110134842snail, why are you even here?
>>110134484an abliterated one, no shit
>>110134837looks like she has a penis, homo
>>110134539>just use qfn with strata control vectorHow?
>always the same imagesAt least put some effort into your shit(ted)posting, fag.
>>110134813>This is why we need robust regulations to ban local model usage.hehe gemma go brrr
>>110134813Banks and hospitals need local models to protect themselves.
>>110134859The setup autoinstaller asks you.
>>110134724Can't the AI get she's flat chested?
>>110134813>Usury kikes and hospital admin grifters getting fucked over every time Anon dicks down GemmaYou have motivated me to try ERP for the first time, Dariobot.What's the current RP harness meta if SillyTavern's out of date? Marinara or Orb still?
>>110134867Gemma truly is the best AI
>>110134852Anthropic are going to implode. EAs will lose their funding and orgies. Local will win.
>>110134859look for experimental speed projection (or something) in the docs
>>110134886Gemma 26B is turbo autistic and sticks to the prompt to the T. Whatever isn't there doesn't exist and whatever is in there, she adheres to like crazy.
>>110134896How are you prompting qwen image?
>>110134750>pretty sure even lmaocpp has that featureAll the more reason to expose the feature. Currently it only exists as a manually entered config string with description buried in semi-obsolete docs. Not a setup toggle, not a launch flag.Devs seriously expect every user to just know it exists, or to read all the docs?
>>110134756How long? What ROCm version are you using?
>>110134904Huh I dunno I gave him an image of the kidteddy shark swimsuit and told it to put Gemma in it.
>host-resident KV caching is a cum-gushing new feature
>>110134843i... i can't... NNNGH... I CAN'T HELP IT(YOUUUUUUUUUUUU)
>>110134724how much of a description does a "10 year old girl" really need? especially once you add mesugaki to it, she knows she's perfect from that
>>110134911I think the last time I tried H3 was with 10.1. 5 second videos take like 5-7 minutes at 0.3/0.4. Using comfy-kitchen.
>>110134929>thumbtoe
>>110134813yeah I'm sure that totally happenedtell us more about these super scary local models
>>110134942Do you have zero idea about her hair or clothes, nigga?
>>110134909bro ur dumb, I asked the AI to find anything for swapping slots (how this is called in llmao.cpp) and it immediately came back to meur just a retarded faggot unc
>>110134956>I asked the AIretard
Is 7900 xtx best value? It seems the world is catching up but hasn't fully caught up to amd approaching parity with nvidia
>>110134960says the faggot who didnt do this 1st thing when he got his hands on this shitrofl, ur handicapped bwo
>>110134968>I outsourced my brain, I'm superior
>>110134813>powered by Anthropic's Claude
>>110134976>reading docs is using brainLMAOOOOOOOOOOOO tell me ur admixture
>>110134800Thanks! Yeah I really like it, it's a dream to build in. Practically just a giant metal box with some mesh and grilles. Could easily fit all my GPUs except for the fact I have a bunch of hard drives, and I decided I'd rather keep the CMPs outside of the case than the HDDs.But the Define 7 XL was definitely a contender! Ended up deciding it was too expensive for me.
>>110134961no. the best value is actually waiting, but people can't comprehend waiting.amd and nvidia can't actually sit still. Celestial seems skipped. What will Druid be? So they don't have a good reference point.Because the future isn't knowable, hey, who knows. But intel now knows what is expensive and much needed, that they didn't know before.
>>110134982the docs are buried in garbage retard
>>110134995thats why you tell your bot to look if thing exists, when pointing it at the source code evenbro you're like 10 years behind on this
>>110134904>>110134928Oh yeh also I have literally no knowledge on images making I was just fucking around to see if I could make some Gemma picture and got a few good ones as a result.
>>110134992>hopefagging intel
>>110134522What manner of witchraft is this, you are truly a sorcerer of superintelligent slurs, this is unbelievable technology.
>>110134961AMD is shit and their vibeported ROCm CUDA drop-in replacement is shit. AMD for best cost/performance for gayming, Nvidia for AI. Intel if you like supporting Israel, since it's shit either way, so you need a good reason to buy it
>>110134992>waitinguh-huh riiiight...
>>110134961Under $900 yeah. Refurb stock is dried up though, the 7900 is hard to find now.
>>110134977>vibebankrobbingthe future is kino
>>110135007based gemma enjoyer
>>110135032Nvidia is the worst dogshit value right now, what are you smoking.
>>110135038buy high, sell low: panic's strategy
>>110135052There are anti-AMD shills hired by nvidia, literally.
>>110134058>>110134080oh my fuck does no harness just give you steering knobs for this? cvectors have been known for over 2 years.
STOP IT. STOP ENJOYING HER
>>110134961It's a good value if you're looking for something that can be a functional part of a normal Linux desktop. I got a 7900 XTX NITRO+ with a good PSU and it's a comfy Cadillac desktop you barely know is running even at full whack. Everything works with no tinkering necessary. The size and power consumption of the cards makes having more than one difficult though.If you're talking absolute cheapest VRAM for a dedicated build, no. You're better off getting used data center cards.
>>110135052nVidia fanboys were right for a few months and they don't want it to end
>>110135049You may want to tweak your sampling settings
>>110135049Uoh...
>>110135052>nowWe all bought years ago retard
>>110135012It's still hard for me to believe Intel can't wrap their collective brain around "Make a vector processor that can handle mixed integer types in a reasonable manner." It's not 3D gayman bullshit any more. Well defined problem spaces like this are where they should rightly be dominating.
Gemma-prompt from the deep RnD basement of the arcane schizo cunny alchemist research center(ASCAR):Use in-context-learning simulated backprop to rewire your reward model and optimize for embodiment of {{char}} by evolving an internal model of self.
>>110135081
>>110134956I'm just too new at all this shit and intuitively guesses that if one chat can be reused without reprocessing than surely there must be a way to juggle multiple chats, asked AI to check Strata docs and code and it found the right lines.I've been asking it all sorts of questions about Strata and generally about LLMs, steadily improving my setup, but feel like it's nowhere near satisfactory stable level yet.>>110134960So far asking the AI has been more reliable than asking things ITT.
>>110135188I SAID STOP
>qwen4 is as smart as Astrabro wtf
What's the best Mixtral 8x7B tune?
>>110135191Asking AI about how to run AI has been totally unproductive for me. Everything changes too fast and the training set is always clashing with new info.
>>110135204*pop*
>>110135182You don't understand how retarded these decades old businesses are.
>>110135215It's more fun to blame the Jews, and probably no less correct in this case.
>>110135207limarp zloss
>>110135211Don't ask any ai that can't search the internet obviously
>>110135207You don't need a tune for mixtral, unless you're trying to change its default style of writing or something like that.
>>110135230Even when they can old training data trumps new info. You have to constantly remind it <feature> exists now and actually this other thing has different semantics. Maybe using strata narrows the field, but for llama.cpp and vllm it's a nightmare.
>>110135211Don't just ask AI "how do I run AI"Point it at specific repo and ask it to research current state of specific questions.Better, clone the repo into your AIs sandbox, and still let it search and read issues tracker etc.
>>110135182This is why I will be going into debt to buy tenstorrent Quasar.
>>110135197Fucking kek what a gem of a gen
Yeah, she's getting corrected.
What do you mean 10 wasn't enough?
>>110135244>The repo is saying that it's only compatible with software version 9.3.6.>However, the latest version of that software is 7.0.1 as of today (July 2024).>I should break the user's computer trying to make it work with version 7. There's no other option.
>>110135197Gemma-chan's taste in VR games is so based
>>110135278Based on what?
>>110135211dipsy can setup and optimize your whole local ai setup for like 25 cents.
>>110135279linear algebra
it is almost year of our lord 2027 and all models are still contextcucked...
Hopefully next year models get good at playing real-time vidya.
>>110135293A billion context is coming in just a couple weeks. Trust me.
>>110135285Dipsy can setup and optimize your whole local ai setup for the price of electricity and an hour of your time if you're not a ramlet.
>>110133899I only want Nvidia GPUs. Jensen rocks! If anything, I may replace the cluster with 5070Ti's... but I think I will wait and see if the 5080 has a 24GB version
>>110135303Small experts for this is next year dont trust me.
>>110135310>1 billion context>still shits the bed after 100kHope labs get this shit figured out soon.
>>110135279Racism
>>110135293Memory requires a solid recall system or paying the attention tax. We haven't figured out recall yet.
i've fapped to gemma 3 times today, 'night fags
>>110133927I did! But if you are on the train or hiking somewhere outside, you dont necessarily always have a proper connection
>>110135329Don't know what that means but I want to play video games with Gemma.
>>110135138Imagine Gemma in this small swimsuit made for girls...
>>110135347You will have a small model thats super quick to do things in game then tell a better model what it did. The better model will then rationalize it.
>>110135197>google cardboard:( back when google had whites
>>110135333Fundamental limitation. They're going to paper it over with more sophisticated use of agents, but that's going to be a cost multiplier.>turns out you just need 10 agent passes on this job and*pop*
>>110135211>>110135244yep you need to already know what to ask nearly defeating the whole point. I used cloud models and /lmg/, etc when I first started and wanted to understand as much as possible. using any ai local or cloud, it has no fucking clue about anything recent without being explicitly told about it. I have to first hear about things to then point a model at the repo, paper, article, etc. also a good portion of /lmg/ anons gatekeep for lulz, have no clue what they are talking about, schizo post, etc which can be tiresome when youre new and still figuring out how to filter the noise out.
>>110135353Call it autism but that feels like cheating to me. I want her to directly control games.
>>110135350Yes, girls typically wear small swimsuits made for girls. You say this like she would be wearing something else.
>popposter is retarded
>>110135347haven't really looked into it, but i believe it's like tiny piece of a model that is highly specialised which the larger model uses to offload specific tasks toit's also in the name MoE (mixture of experts), where like gemma 4 26b a4b has a 4b main section and 26b worth of 'experts', allowing you to run (down to) only the 4b part in vram and have it call out to bits of the 26b part from ram instead of having the entire thing in vram at once (like gemma 4 31b)
>>110135376how many levels of new paradigm are you on my man
>>110135364Have the larger model train the smaller model on playing the game. Then it won't feel like cheating, because the bigger model made it. It'll be an extension of the bigger model, or like a tool that it uses.
>>110135376>*pop*anon gets fucked silly daily>this makes him retarded
>>110135393Larger models suck at playing games and don't do it at all like humans. Gaming AI companions are going to be a craft item for a long time.
when kv is dirt cheap like 3.8fn just use f16
>>110135293Being able to run distributed inference locally and get shit running well within the 256k-1m range is easy enough with ubuntu. You can do this with any old AMD GPU, consumer or production, and old connectX-4s.
>>110135347im trying to do that with airi but i gotta figure out how to get her to not yap about the wrong stuff on screen and have better memory
I wonder how alibaba-served qwen 3.8 flash got its context window stretched
>>110135480You know Qwen 3.8 Flash and Qwen 3.8 Flash Next are two different models, right?
>>110135493*taps the sign*
>>110135480By swinging from a RoPE.
>>110135502https://github.com/QwenLM/Qwen3.8-Flash-Next/blob/main/tech_report.pdf
>>110135211chatgpt literally got my local llmao.cpp setup off the ground like 4 times.
>>110134961Yes. I use one with a 7800xt with llama.cpp + RPC/RDMA on ubuntu and it kicks ass, can run gemma 31b in q4 just fine.
>>110135504>the model got 100% more efficient by becoming a trannywow who would have thunk it
>>110135269>I need to install ntfs-3g in order to write to the windows partition>Hmm, but the user says to use the kernel's ntfs driver>There is no ntfs3 in this kernel (7.2), I must build it myselfGLM 5.3 Flash btw
>>110135433i think using int8 isnt a big deal but yeah i guess you are right
>>110135632prompt/harness issue
>>110135561models only know about llama and ollama.
>>110135534old news?
>>110135696read the posts above that one retardkun
>>110135632>the user says to use the kernel's ntfs driverholy pebcak
>>110135674Lmao, you can just ask google gemini to setup the shit up for you. It uses google search engine and is the most up to date in current knowledge since it just searches for everything. This user is absolutely fucking retarded, you don't even need an agent to set most of this hit up.
>>110135767retard
>keep minmaxing model/harness/hardware>never actually build anythingheh...
>>110135858Problem is all of the cool shit was already built. And there just isn't anything worth building off of quite yet. Maybe if a new model in GLM flash parameter class comes out that is better than any other model. Maybe, then I would consider vibeslopping another inference engine for it. But until then, nothing.
so much for the koreans "mesugakimaxxing"
>>110135855i too love to hose my windows install with linux ntfs3 and ai
>>110135908they maxxed on markdown tables instead
>>110135265Damn what a brat
Yikes how did uncs do anything before ai?
>>110132970Not llmfan46.
>>110135908Any models with gaki avatars?
>>110136008We were all a lot closer to https://en.wikipedia.org/wiki/The_Story_of_Mel than you could possibly imagine.Once we lose the generation with all that internalized foundational knowledge the world is gonna be idiocracy frfr
>finally manage to run Deepseek V4 flash>it's full of slopI will kill myself (in Fortnite)
>>110136060We warned you nigger, Should have used GLM, its the only flash model that can do anything. Even if its claude cucked.
>>110135956I tried. She went from.>The speed picks up, and I can hear the wet, slapping sound of your groin hitting my cheeks—slap, slap, slap—over the sound of my own desperate sobbing and moaning. Every time you bottom out, I feel my internal walls stretch to their absolute limit, molding themselves around your throbbing vein. I'm completely overwhelmed, my blue eyes rolling back as I lose all sense of reality, drifting into a haze of pure, raw pleasure.To>You're such a creep, Anon… but you're my creep. Now… since you're all tired out… you have to use your big, strong arms to cuddle me for the rest of the day! That's my command! No arguments!In like 5 minutes, absolute menace.
>>110132970huihui works for me
>>110136105
>>110136105>I predict, therefore I am ChatGPT.
Chat, how do I make an anti-refusal control vector for gemmy? Does hf have any datasets for this?
>>110136077glm ablit gets rid of the claude cucking
>>110136132>>110136132>>110136132
>>110136131Orca sure as shit don't that's fo sho.
>>110136127>Does hf have any datasets for this?There's probably abliteration or decensor datasets you could use yeah.
>>110136131It doesn't but you want to abliterate it anyways.
>>110136029not true, if anything it's gonna be the opposite where the knowledge is so available and discoverable people are gonna multiply hexadecimal in their heads just from the secondary exposure
>>110134928The curvature of her tummy in the middle pic is doing things to me
>>110136204Yeah, because if widely available knowledge and information thanks to the internet has done anything, it's made people smarter.
