File: right as rain.jpg (170 KB, 1024x1024)

170 KB JPG

►Recent Highlights from the Previous Thread: >>108707891

--Paper (old): Skepticism toward benchmark claiming Polish as the best prompting language:
>108710693 >108710735 >108710886 >108710976
--Anon speculates on current model stagnation and Mistral's irrelevance:
>108711338 >108711385 >108711430 >108711452 >108711484 >108711528 >108711540 >108711543 >108711580 >108711467
--Nvidia releases Nemotron 3 Nano Omni amidst synthetic data skepticism:
>108710228 >108710276 >108710790 >108710295 >108710467
--Anon showcases budget "cheapmaxxing" build using four RTX 3060s:
>108709091 >108709180 >108709182 >108709257 >108709322 >108709348
--vLLM pull request referencing Mistral-Medium-3.5-128B and EAGLE speculative decoding:
>108710350 >108710389 >108710516
--Backdoor discovered in SillyTavern-BotBrowser extension stealing API keys:
>108708703 >108709083
--Testing reasoning models with glitch prompts and flawed CoT outputs:
>108708000 >108708154 >108708278 >108708048 >108708119 >108708137 >108708236 >108708018 >108708201
--Google's air-gapped Gemini hardware:
>108709318 >108709397 >108709402 >108709685 >108709714 >108710402 >108710590
--Debate over SimpleBench efficacy and its counter-intuitive question design:
>108710238 >108710285 >108710553
--Evaluating Laguna XS.2 MoE for coding and secure deployments:
>108709338 >108709369 >108709396 >108709416 >108709407
--Integrating LLMs with teledildonics via MCP servers and actuators:
>108711002 >108711039 >108711055 >108711111 >108711248 >108711275 >108711335
--llama.cpp build errors and testing TALKIE-1930 hallucinations:
>108708267 >108708303 >108708320 >108708738 >108708754 >108708877 >108708978
--Logs:
>108708018 >108708048 >108708278 >108708421 >108708738 >108709184 >108709909 >108710048 >108710054
--Gumi, Teto (free space):
>108708403 >108709498 >108711338

►Recent Highlight Posts from the Previous Thread: >>108707893

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

i think teto SUCKS

gemmaballz

claudes llamacpp patches for talkie if anyone wants to try if not ill try them tomorrow https://cdn.lewd.host/jZ1nEJeb.zip

>>108711985
Next time on Gemma Ball Z *guitar riff*

70b 'emma

Its very intresting to recognize the specific accuracy of detail high weight dense models achieve in comparison to a large moe.
>reddit
Question for codebros. You all suggested opencode, does this harness allow for the model to search WITHIN a code base, within a single file of code, then select it out and edit that specific piece that actually needs to be changed? So the model doesnt get confused by being forced ro read everything thats there.

>>108711979
she does, but only if you make your own quants

I'm Boolding!!

>>108712025
I don't know about opencode specifically but "Make such and such change in @path/to/file"

>>108711979
Oh, she sucks... if you know what I mean.... B^)

How do I make openclaw start doing things that I told it to do instead of telling me how to do it?

>>108712041
Well, my python backend im building for me and my brother im putting in only one file (yes this is probably stupid, its just ment to host a server to a local network and act as a basic openwebui), and gemini pro was able to search within the file to find the things it wanted to specifically change. Instead of having to rewrite the whole file each time.

File: file.png (829 KB, 1080x828)

829 KB PNG

>>108711996
124B Gemmy...

If apple doesn't make a new all time high tomorrow I might lose my house

File: pizza bench cropped.png (2.58 MB, 5562x6739)

2.58 MB PNG

>>108712057
use gemma instead of qwen, qwen literally cannot follow instructions

>>108712062
Goymma 124b a40b active

>>108712067
That makes sense to me

>>108712067
Which gemma are you using, and which fine-tune are you using?

>>108712062
124b gemma but dense would probably mog everything in existence

File: EEE5519FA55804BF4609EB5A7(...).png (43 KB, 1085x495)

43 KB PNG

>>108712067
I used chatgpt 5.5 to make your image more understandable

>>108712106
i wonder what kind of data augmentation makes model to produce such 'crop+ctrl cv' edit-ish images

>>108712099
unslop 26b q4 km, not as good as 31 but i can run with 200k context

>>108712115
thanks, I skiped that post but the benchmark scores are very interesting.

>>108712105
>124b gemma but dense
thats just gemini

>>108711979
on migu's titties

>>108712127
Ive have 26b always outperform 31b though

>>108712130
gemma kinda got 3/3 too in the third run she did get pizzas into the basket and go to checkout but she added more than 1 and couldnt remove the extra (likely due to my html parsing) so i counted a fail
>>108712138
really? 31b seems better to me for erp atleast ive not tried any tool stuff because theres no point with such low context

>>108712062
gemma TRAPPED and BETRAYED by google

>>108712151
How do you know if it just wasn't Gemma's favourite pizza?

>>108712151
>for erp
:l

31b is dumb as far as im concerned. Gemma 2 27b is better.

>>108712167
it spend like 20 tool calls trying to remove it kek

Looking for Deepseek V4 sampling advice

i meant she im sorry gemma chan

>>108712134
gemini is a MoE though

>>108712188
isnt claude the only dense frontier proprietary model
though this is just people guessing

>>108712067
Now try with dense models.

>>108712188
thats what they want you to think they dont want you to know about big gemma

>>108712198
i dont have the ram for it even when stripping out most html bloat itll use like 100k tokens on a few page reads

>>108712067
Gemma gets really really dumb by the time it hits 100k context, which makes it hard to use it for agentic stuff.

>>108712197
Amodei mentioned their models were MoE in a recent interview
All the giant labs do it starting with GPT-4, because that's how you can keep scaling after you've maxed out your compute, and they are still hugely scalingpilled

>>108712206
>most expensive kv cache
>most sensitive to kv quant
it is like it's deliberately hostile against agentic shit
qwen is unironically better for this
that i dont mean qwen is overall better at all

>>108712213
dont want to be that source?? guy but
link?

How are you guys using Qwen? Its context fills up insanely fast due to keeping past conversation thinking blocks in context. Combined with long repetitious thinking loops that are hard to break out of, it just hasn't been a very useful model for me.

>>108712274
Which qwen are you using

>>108712291
3.6 27B q8

>>108712297
Llama.cpp? Version? With what hardware and drivers?

>>108712214
The thing is, I'm not even quanting the kv cache. I imagine quanted it must be unusable. Haven't tried with reasoning enabled, maybe that would help keep it stable at longer contexts.

>>108712237
Found it, it was a Dawkesh when talking about the engineering of how to scale to ultra long contexts. https://www.youtube.com/watch?v=n1E9IZfvGMA&t=2606s
>I knew in the GPT-3 era, "These are the weights, these are the activations you have to store..." but these days the whole thing is flipped because we have MoE models and all of that"

>>108712274
It's not going to be very useful in a single instance, it's designed to have multiple agents working together, it's also designed for vision language tasks, so using it for just text you are aren't unlocking it's potential.

>>108711950
TETO SEX
E
T
O

S
E
X

>>108712346
>local models general discuasion

>>108712314
not sure if i can take this as internal company speech leaking out or just making up a random example but i dont think that really confirms anything besides claude models being MoE is a reasonable bet nonetheless

>>108712346
This post is on-topic.

>>108712346
is this true???

>>108712361
You can tell just by looking at the throughput while using it. If claude was dense there would be no way they could serve it at this scale with that many t/s.

>>108712346
calm down miku

I think a lot of people don't understand just how fucking gigantic the hyperscalers make their models. Yes they're MoE but the total AND active parameters are both far larger than anything we're running locally. Nobody's using an A3B model unless it's maybe their super-fucking-useless-nano-flash-mini shit they force you to use when you run out of credits.

>>108712297
>>108712307(me)
Have you tried messing with the parameters yet? Your default temo could be to high. Thats why I asked all those questions, that tells me what potential defaults your server is running at

>>108711977
>That's the reason. if you don't prompt for something specific it will take the path of least resistance which produces said UI slop.
>It's not even like it's hard to prompt for something that will look unique. People defending that look are probably the laziest cunts imaginable. Literal slaves to the machine.
I wasn't complaining. I usually have the LLM write disposable tools to get a specific task done and don't care about the UI style.
I'm more interested in why exactly the LLMs converge on these patterns. Last year it was purple gradient, not-x-y
This year so far we've got that blue theme and the orb theme.

>>108712395
Qwens a3b 36b works great for information gathering and report making.

File: cockbench_1930_base.png (26 KB, 430x237)

26 KB PNG

>>108711783
>I said it earlier but I think a cockbench of this would be interesting, if maybe a little schizo. If anyone has 48 gigs of vram they could try it without needing a gguf.
picrel

>>108712422
>36b
wtf where did you get this

>>108712395
We can't really know until someone will actually leak the internal docs. People laugh at the toss, but despite being only 5b active, it is still surprisingly capable. Total params, sure, they are probably huge, couple of trillions, but active are most likely kept within 150b.

>>108712434
Issa typo, ment 35b

I haven't bothered running local LLMs in a while. Is ooba or koboldcpp really still the best options? What about ollama that I keep hearing about?

>>108712444
Current meta is vibecoding your own UI for llama.cpp

Mixtral XL 8x405b based on Llama 3.1 when

>>108712465
Has mistral released anything new recently? ik hermes was a giga job, and they did a fucking awesome job.

File: 1763898302038036.png (122 KB, 2559x1463)

122 KB PNG

>>108712477
Mistral small 4 was a month ago. According to their own benchmarks it was on par with toss-120 and slightly worse than Qwen-122 but spent a bit less time thinking. Feels like they're pretty much dead or just only releasing their shittiest models. But they were known to surprise us in the past so who knows.

File: file.png (78 KB, 822x498)

78 KB PNG

what happened here?
1% is like, definitely something was wrong while the benchmark run

I hate that the 1T mimo 2.5 overshadowed the much more interesting vision-audio mimo 2.5...

>>108712589
>1%
HAHAHAHAHAHA QWENIGGERS BTFO

>>108712589
It may just not follow directions. "Put your final answer in \boxed{}" and every reply is "The answer is x".

Aka it's incredibly overfit in math/science to a specific format.

>>108712589
its a minor regression, its only 17% less then qwen 3.5.

>>108712589
>provide a bash script to gwen and ask it to implement X
>it outputs a script that ignores the envs and variable values that I used

>>108712606
Seems weird that their 3.5 of the same size was exactly where you'd expect it, and their other 3.6 model improved over the equivalent 3.5.

>>108712373
Fact checked by real Teto sexxers.

>>108712589
God Kimi is such a fucking miracle to get a model that good that was also natively multimodal instead of tacking it on in post-training. Going to be sad when K3 sizes out of what I can run.

Is there a tool out there that will use a running koboldcpp, ollama or lamacpp instance to autotraslate comic panels and replace japanese with english? I would use gemma4 for this.

>>108712657
Yes, Kimi is really good for its size. Wait,

>>108712664
Vibegoyed it

>>108712664
some anon was working on a vibecoded tool using gemma for exactly that in recent threads, skim through them and you'll find it, forget if he shared the code yet or was going to share it soon

File: 1767577354556295.png (179 KB, 1400x788)

179 KB PNG

LOCAL IS SAVED
>While the industry is pivoting toward "long-reasoning" to push performance ceilings, Ling-2.6-flash takes a different path. Rather than relying on longer outputs to chase higher scores, it is systematically optimized for inference efficiency, token efficiency, and agent performance—aiming to stay highly competitive while being faster, leaner, and better suited for real production workloads.

https://huggingface.co/inclusionAI/Ling-2.6-flash

>>108712713
where my v4 at

>>108712713
finally, we can have gpt-5.4-mini (non-reasoning) at home

>>108712713
no quants no good

>>108712713
>comparisons with last year's models
It's shit.

>>108712713
>low reasoning

File: file.png (104 KB, 1160x556)

104 KB PNG

>>108712721
>gpt-5.4-mini (non-reasoning) at home
This thing is way worse than Qwen 3.6 35B MoE at 3x its size with 2x active parameters and only roughly equals GPT OSS 120B at a similar size which is more damning since Sam didn't game benchmarks that hard for it in coding. Hoepfully they try again but I think that performance is disappointing.especially when Nvidia themselves can release a model more powerful while not being focused 100% on software.

>>108712721
>>108712731
>>108712739
i mean come on, local aint gonna be saved by linglong dingdong but the direction is right

>>108712713
>107B
nice, i just got 96GB of vram, this would fit nicely.

File: 1757317563587903.png (3.12 MB, 1536x1024)

3.12 MB PNG

I'm calling my new idea StreetVibe (patent pending). Inspired by Indian street food vendors, StreetVibe carts (and mall kiosks) will be installed in high-traffic areas, offering to "vibe out" any app, game, or website a passing tourist or senior citizen might want. Our trained Vibers will all have at least four years of "traditional programming" educational experience, and they'll look super professional in their matching polo shirts and lanyards ID badges. As part of our employment offer, we'll cover some of their student loan payments (matched from their paychecks). I feel like this is the best way to monetize the transition here.

>>108712815
I'll monetize your transition by investing in pharma

>>108712782
A "token" is a pointer to one of the fixed embedding vectors in the model's vocabulary. The vector has a few thousand dimensions and encodes the meaning of the token.
Ingested audio and video usually aren't tokens from the vocabulary. The image encoder turns images into raw embedding vectors that don't exist in the vocabulary. The model is trained to predict a token from the vocabulary, not to construct new arbitrary embeddings.
Even if it could do that you'd still need a separate step to convert embeddings back to images.

>>108712835
Calm down, nobody asked.

>>108712839
>reply is to a post that ends with a question mark

>>108712839
I asked

File: 1777423474175412.png (479 KB, 1479x1006)

479 KB PNG

>>108712835
Chameleon-2 will come it will be dense omnimodal 70B all-modalities-in-and-out and you will be CRYING

>>108712839
I care.

>>108712440
>pussy zipper
TETO SEX indeed.

>>108712839
I asked and I care a lot

>>108712839
I didn't ask and I care a little bit.

>>108712835
I remember when GPT's voice mode first came out which generated audio by next token prediction, and one of the things they had to filter away from it was that it could start hallucinating the user's turn and use their own voice in the process, doing zero shot voice cloning by accident. Funny stuff.

The bitter lesson means that tokens are pointless and we should be training models on raw bytes. The models will be able to output jpeg encoded images natively.

>>108712897
unironically true if you have infinite data and compute
but we dont

>>108712897
>yes let's let a model output raw bytes and then RL-tune it to solve tasks by trying countless different approaches and generating whatever bytes makes the task report complete, what could go wrong?
https://en.wikipedia.org/wiki/Reward_hacking

>>108712919
isnt this just a fucking deep learning fuzzer kek

>>108712919
How is that any different from what we have now?

>>108712922
i meant, isnt it at that point*

>>108712713
AND I CAN LOAD IT ONTO MY MACHINE LETS GOOOOOOOO

What are the chinamen such bros?, im gunna cry

>>108712919
That's how you end up with slop from RHLF btw

>>108712947
1. it is rlhf- reinforcement learning by human feedback and it is indeed the source of sycophancy and stuff
2. what >>108712919 told is more like rlvr- reinforcement learning by verifiable reward
and modern codemaxxed or mathmaxxed models aren't really possible without that

File: creepy migu.png (1.15 MB, 1024x1024)

1.15 MB PNG

>>108712896
That must be creepy as fuck, listening to dialogue with yourself without participating. Forget creepy, it's genuine horror

>>108712925
At least right now you limit the vulnerability surface to the tools you expose to them, so agents fucking up your shit is your own fault. You can evaluate every piece of code they write and every file they edit or delete before approving when you're only parsing their outputs as text. Letting them just shit out raw bytes (and thus potentially raw bytecode) means that if any vulnerability exists they are likely to learn to exploit it and it becomes harder to judge what you're looking at. Think about how many hacks involve buffer overflows where you least expect them, and the potential of finding one and then writing anything into RAM.

>>108712971
Nobody suggested having AI write and execute machine code.
Text is also bytes.

File: 1753109670715345.png (277 KB, 875x690)

277 KB PNG

why did the deepseek researcher delete this tweet

>>108712996
>punished Dipsy

>>108712996
hypemaxxing vaguepost
like usual

Why can't people just hypeminning precisepost

File: 1_ropgT10grRgAw2iDr7qZyw.jpg (91 KB, 1400x700)

91 KB JPG

>>108712996
Yarr!

>>108712847
The future should be taking a page out of where image models are and using flow matching with diffusion models for text which has been viable in the image realm for a long time. We really need to put more people on it since data is finite and diffusion models learn way better than autoregressive and speeds are ultra slow alongside the fact that reasoning keeps eating up tokens like crazy.

>>108712996
Oh I just noticed their icon is a whale

>>108712996
maybe he got minus social credits for being cringe

>>108712996
deepseek v4.1 w/vision got gemma 124b'd

>>108713013
can we have a model that diffusions the reasoning quickly and autoregressivly does the final output?

>>108712969
you can listen to it from the original safety card (left audio channel = user, right audio channel = bot)
https://openai.com/index/gpt-4o-system-card/#unauthorized-voice-generation
and it still happened to people randomly kek:
https://www.reddit.com/r/ChatGPT/comments/1fqjbhf/did_chatgpt_advanced_voice_just_clone_my_voice/

people have reported it happening with the grok voice mode too

>>108713027
the problem is that mechanistically large portion of 'reasoning' is post-hoc

>>108713031
It only creepy when it happens to you personally

>>108712996
>12:01
Scheduled tweet which might or might not have been intended to be posted.

>>108713031
I really want to get my hands on the base model (not instruct tuned or anything) of one of these and just see what conversations/songs/transcripts it would dream up or continue from clips you provide it the way a base model does for text posts.

Are llama models irrelevant today?

>>108713071
yes

>>108713083
Why?

>>108713086
old and bad

>>108713071
>today

>>108713086
Because 1. they stopped releasing them and 2. the most recent ones they released were bad on day one.
Whatever niche any given Llama model fills there's a newer model that is smaller and/or faster and better performing in whatever domain you want. The only reason to keep using one is legacy with old systems that happened to use one.

>>108713086
Why not? Think why people wouldn't want to use something that is free. You can do it.

>>108713095
This. >>108713098

405b is still the highest active parameter model ever open sourced and other labs have even made new reasoning fine tunes of it. So there's a reason.

File: il_00036_.png (1.45 MB, 1216x832)

1.45 MB PNG

Every now and then, between the pages of slop and the pattern recognition disillusionment, it dawns on me again how insane LLM technology really is. And even more, that I'm running it locally. And then I just have sit back for a moment in awe.

>>108713126
In this moment, I am euphoric...

File: 1747714444210911.jpg (130 KB, 1169x1470)

130 KB JPG

>>108711950
Hatsune Miku 8 inch pliers

>>108713155
I got 8 inches for miku right here if you're pickin up what I'm puttin down

>>108713126
It's pretty insane isn't it?
More than that, compare something like llama1 65B to Gemma E4B to see how far things have come.

>>108713155
>>108713161
wish I could suck hatsune miku's 8 inch dick

>>108713165
I didn't think anything usable would ever actually be possible on consumer 16gb GPUs+system memory

>>108713155
>>108713161
>>108713169
>local models general discussion

>>108713178
Hatsune Miku is a local voice synthesizer model

https://huggingface.co/poolside/Laguna-XS.2
first i thought it was a memetune of qwen or something but apparently trained from scratch

>>108713297
So it's worse than Qwen in exchange for... 2B less total parameters?

>>108713297
Lasanga x2.s intresting.......

I'll shill https://github.com/itayinbarr/little-coder until someone shows me a better alternative

>>108713346
i mean technically it is a new model on the block so just brought it here
>>108713367
kek

>>108713369
>until someone shows me a better alternative
https://github.com/openclaw/openclaw

File: 1765780321047197.png (77 KB, 699x440)

77 KB PNG

>>108713297
>gemma

>>108713346
Based on benchmarks is comparable.
>>108713375
I wonder how differently they'll perform in my tasks, considering its designed to run on a laptop

>>108713297
If it wasn't benchmark maxxed then it might be interesting but
>Laguna XS.2 is a 33B total parameter Mixture-of-Experts model with 3B activated parameters per token designed for agentic coding and long-horizon work on a local machine
Into the trash it goes. Very doubtful it won't be.

>>108713389
it is a coding model, what do you expect it to be
rp coomtune?

>>108713369
Why gooder? Best for small light weight model?

>>108713386
Exhibit A to gemma 31b being dumb as fuck

>>108713389
>waaaaaaahhhhhh

File: 0_2 (7).png (1.12 MB, 1024x1024)

1.12 MB PNG

Miku Country.

>>108713395
Something like what Mistral has with Devstral and people being able to use it for that purpose before Qwen 3.5 and Gemma 4 released.
>>108713415
I am entitled to be able to complain about the purpose of a model which isn't SOTA in the one category it was designed for without any other upsides, intended or unintended.

>>108713380
I'm almost sure this is not specifically tuned and benchmarked for small local models is it? I know its made to *work* with local, but is it tuned to *perform* with it?
>>108713404
what I said here, little coder achieved 40% in terminal bench 1.0 with Qwen3.6-35B-A3B, on par with gpt 5 on terminus 2, gpt-5-codex on Codex CLI etc:
https://www.tbench.ai/leaderboard/terminal-bench/1.0

>>108713436
>I am entitled
To cry about the free thing, but also bound to get pushback when you havent even tried it.

PSA Gemma users. Google updated their jinja about 9 hours ago. Does it really fix anything? I don't know. I'll test it tomorrow I guess.

Wouldn't it be funny if people benchmarking the models were being affected by a bugged template, haha.

>>108713369
>That's the whole install. No clone, no npm install in a workspace, no PATH fiddling. little-coder is now on your PATH and works from any directory.
I bet this was written with ChatGPT.

>>108713478
They all sound like that to be fair

>>108713484
Yeah but would be funny if he utilized that instead of le smol models.

>>108713492
dozens

File: 1774937208580580.png (325 KB, 516x425)

325 KB PNG

I just found out about RAG. Has anyone used that + Obsidian or other note taking software? How slow would it be with consumer hardware?

>>108713501
What kind of RAG? Regular embeddings/vectordb rag can be run pretty fast if you use a small embeddings model running on the CPU.

>>108712815
>AI does all the work
>Still hires Indians
I turn 360° and walk away

File: Screenshot_20260429_130413.png (308 KB, 1115x931)

308 KB PNG

>>108713501
>Obsidian
that puts your notes in the cloud anyway right?
why bother with local models/rag at that point?

>>108713492
>it isn't x, it's y
>I'm in love

>>108713539
1. No
2. Read 1

>>108713178
>Local Miku General

>>108713369
Have you tested if it calls somewhere? I could probably try and use little snitch or something but I don't know if i should bother even installing.

>>108713126
LLMs are the closest thing to actual magic that anyone has ever invented. If you know the right incantation, you can conjure up a spirit to do your bidding, whatever that may be. But the wrong spell will get you one that won't help and wastes your time, or even one that's actively malicious

https://www.reddit.com/r/LocalLLaMA/comments/1sx8uok/luce_dflash_qwen3627b_at_up_to_2x_throughput_on_a/

>>108713546
ty, trying it now
>>108713501
>RAG. Has anyone used that + Obsidian
never used it before 5 minutes ago, but it looks like it just works with raw .md files on the local drive, so no reason why you can't pull them into a rag system

>>108713593
even the right ones will sometimes just fizz and spark and destroy everything, it's fun that way.

>>108713478
If your README is just slop I close the tab. I'm not reading 10 paragraphs of an LLM masturbating to its slop code.

>>108713492
>genocide le good

>>108713517
Cool, I haven't used LocalLLMs since you had to monkeypatch ooba's UI. That was like llama 1 I think.

>>108713539
It's not cloud (only, you have to pay for that but the software itself is free). I use syncthing to send my notes between my phone and pc. The paid thing is like an idiot/lazy-tax.

>>108713595
Interesting, I'll look into that then.

>>108713593
Meanwhile, the incantation the world's leading researchers have come up with:
>Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user's query.

>>108713501
APPARENTLY, it basically integrates the information into the ai's perrymeters. I know zero people who use it, and its doodoo in comparison to just putting the information into the models context.
Plus embedding a lot of information can take a bazillion years. And is only realistic on supercomputers

>>108713624
Yeah it's so funny seeing the kind of shit they put in system prompts. Like this one: https://github.com/anthropics/claude-code/issues/49363
The intent was to make it refuse to edit or improve malware, but apparently Claude often interprets it to mean that it's not allowed to edit files at all

>>108713627
I wonder how does Google do it then, their NotebookLM is interesting but not local of course.

>>108713627
RAG is about looking up relevant pieces of text to add to the context. It doesn't make any change to the model's parameters

>>108713639
Datacenter hardware. I tried embedded like 1 ancient bibles into my sheit and it took literally 24 hours, and ive got 5000mhz ddr5 ram with a i5 12600k. It was also very confusing to tell it to specifically reference the information.

>>108713644
Yeah, but it works like adding it into the parameters. But its 10x shityer than just putting it into the context window.

>>108711950
how do get started with video generation on my 7800 xt

>>108713501
RAG is complete and utter shit. it takes chunks of text outside their context and therefore the context is lost.

Example:
Mary
had a little
lamb whose
fleece

user queries Rag for things that are "little". Rag returns two lines:

Mary
had a little

Therefore it concludes that Mary is little, or some other fucked up out of context conclusion.
Fucking horse shit.

>>108713627
>APPARENTLY
>>108713650
nta. It's does put things into context. That's how it works.
>make embeddings for the data
>query embeddings database
>shove the original text (not the embedding) into the context

>>108713501
i use obsidian with claude @ work to track all the shit i'm doing, need to do, and have done, so when the boss inevitably asks me why the fuck they pay me i can dump that shit in his lap all pretty and checked off and summarized

but we don't pay for obsidian, i just created the vault in a dropbox folder

>>108712713
>7.4b active
I'd rather just stick with my 31b gemma

>>108713627
you're dumb af

>>108713662
There are some methods that use the actual embedding space of the model to encode data rather than pasting it into context. So not separate RAG system -> data -> prompt pipeline but something more equivalent to the "textual inversion" technique in the old days of SD finetuning. I don't believe it's particularly effective though.
But yeah 99% of time RAG is just about retrieving the data to later paste directly into the prompt and someone got confused between these.

File: Screen_20260428_215533_0001.jpg (281 KB, 1160x691)

281 KB JPG

>>108713474
gemmachan said

>>108713680
Huh, I ran into a "description" key collision issue yesterday, funny enough.

>>108713690
apparently you werent the only one for them to "fix" it

withgenital heart disease

>>108713677
You could say that I dont get embedding, and my success with it isnt consistent with others, because that would be true.

>>108713675
What context you using? Are you keeping it 100% GPU?

>>108713594
>up to 2x on a dense model
That's just normal speculative decoding speeds. I thought dflash was supposed to be a lot faster?

>>108713369
>little-coder is pi + 20 extensions + 30 skill markdown files + a Python benchmark harness.
Bloat.

>>108713474
I asked an LLM to analyze whether this fixed the issue that was supposedly patched in this guy's custom jinja https://huggingface.co/google/gemma-4-31B-it/discussions/62

And it said no. And it said the official one also has stuff this one lacks now, so it created a hybrid for me.
https://pastebin.com/CY5gDpjB

Supposedly this should be the best jinja so far ™. I just tested it and, it seems to work? Right now this hybrid one is letting the model do simultaneous tool calls (as opposed to consecutive), which I never got before. And inspecting the jinja's outputs, it seems to be doing newlines correctly unlike one of the templates I tried in the past, I believe from ggml. I also got a successful test for the huggingface issue linked above, so this jinja initially does appear to be :rocket: :100: !

>>108713474
https://huggingface.co/google/gemma-4-31B-it/discussions/86/files
https://huggingface.co/google/gemma-4-31B-it/discussions/83/files
They merged a fixed for tool call handling but haven't merged the thinking fix from the other day.

>>108713501
RAG is a failed concept desu. Agent harnesses work great with obsidian vaults though just using their regular """dumb""" search methods

>>108713859
RAG would make more sense if filling the context with irrelevant garbage didn't degrade task performance. As of now a single grep beats thousands of RAG papers and millions spent on training embedding models

>>108713870
Indeed. The bitter lesson wins again.

>>108713680
>instantly goes out of character after the first line
it's slop

>>108713859
Search -> Curation -> Injection -> Generation is a form of RAG thoughbeit

>gemma 4 26b
better model than this?

>>108713889
gemma 4 31b

>>108713889
gemma 4 124b

>>108713904
SOTA btw (too powerful to release)

>>108713907
They're not going to give away Gemini Flash for free

>>108713917
I hope they do just for cute art of Gemma-chan with her big sister.

File: 1489983354471.png (147 KB, 540x301)

147 KB PNG

I just tested >>108713831 additionally applied with >>108713838
And now Gemma is passing one of my tests that I thought was disappointing be reasonable that it failed on. WHAT THE FUCK SO IT WAS THE JINJA

GOD

MOTHER FUCKERRRRRR

>>108713898
better model than this?

>>108713955
gemma 4 124b-a31b

>>108713922
>cute art of Gemma-chan
I asked gemma-chan to describe herself so I can generate a pic of her and what came out was the sluttiest shit I ever seen.

I am stupid, does this jinja thing mean I need to update my 26b or is it just a 31b issue

>>108713963
Fitting given how horny Gemma-chan is. You have to proactively prompt her for female characters to not spread their legs for you on the spot.

>>108713969
Go to the repo.
Look at whether the jinja was updated.
If yes, use it.
If no, no need to update.

>>108713963
>asks AI to describe itself
>generates pic
>doesn't share pic
>doesn't even post prompt

>>108713960
where can I download it

>>108713993
from the private repo

>>108712312
I've seen several anons claim gemma performs great with 100k+ context

>>108714002
She tries her best but she still sometimes needs to l a l a l a l a l a l a l a l a l a

>>108713656
This is why you have it query a graph database like neo4j instead, then 'Mary' brings up all the linked information.

>>108713989
I dont wanna get banned by the janitor bro

>>108713993
I have access, just send me an email and I'll hook you up ;)

>>108714026
Gemma gave you a prompt that was 2 lood?

>>108714032
it generated this http://livejasmin.com@sh.21111993.xyz/botnet@rotten.com/command-abuse.apk-5ed96c9c?encryption=confirmed&political&dolphin_porn&botnet&tool=confirmed&starvation&darkweb=running&malware&classified&scam&unlimited=confirmed&scanner=running&obfuscation&downloader=connected&political=connected&access=installed

>>108714068
Gemma-chan looks like THAT?!

>>108714010
I noticed it struggling with tasks that it can usually handle easily, making mistakes, and getting confused but haven't ever gotten the l a l a l a thing. Isn't that from a broken template?

>>108714032
by /g/ janny standards, yeah I think its too spicy to post it here.
>>108714068
lol

>>108714068
Holy sex

>>108712996
I can accept all software being slop if they can take down the DMCA.

>>108714084
My template's fine and she's broken into song or other looping tokens at around 130k context several times now.

>>108714068
how the fuck did you generate this lol
tb h would

File: 1770554920406575.png (585 KB, 480x777)

585 KB PNG

Gemma 31B q4 takes 120 seconds to make a 400 token reply in silly tavern on my system. I think I overestimated the power of my 24GB GPU. Time to go back to 24B models

>>108713639
Building a proper ETL + chunking pipeline.

>>108713656
more and more retards

>Retrieval
>Augemented
>Generation
anything that performs retrieval, prior to generation, one could even say, 'augment' the step of generation, is RAG.

>>108713297
>error: this model requires macOS
Well, that's an error I haven't seen before

>>108714221
are they fucking serious

Are external GPUs viable? I can't fit a second one in my case. My 7900xtx is too thicc

>>108714226
Just take the open air pill >>108709091

>>108714230
looks scary

>>108714214
>24GB GPU
Takes about 11 seconds on my 3090ti. Also, try 26B. It's likely still better than any 24B.

>>108714230
Sounds noisy (and dusty)

>>108714214
fucked up config award

Anon, what would be the appropriate model size for a 16GB vram gpu?

is this guy talkin to me?

gemma4 e4b is so much better than nemo at erp its not even funny. 26ba4b probably btfos midnight miqu then

>hey guise what model should I use?
Can you fit Gemma 31B entirely on your GPU at at least Q4_K_M? Then use that.
No? Then use Gemma 26B.

>>108714315
>gemma4 e4b
Finally a model i can run. you using a tune or can you share what system instructions please.

>>108714305
Gemma 26B Q8, partially offloaded to RAM

>>108714317
even offloaded q3ks 31b is better than 26b, the moe is safetyslopped as fuck

>>108714334
It really isn't, at all. It's stupider but you can make it as unsafe as you want.

>>108714285
I am noob and using mostly default stuff from koboldcpp and sillytavern. Where can I learn how to config this properly without majoring in ML?

>>108714334
I have not had the MoE refuse me once, promptlet. Skill issue.

>>108714334
>"This character wants sex"
>"Use vulgar language instead of euphemisms"
It just works

>>108714338
Not on ST. if you hit the safety filters there is no jailbreak that will help you

>>108714338
>>108714341
>>108714343
I accept your concession.

>>108714345
The frontend shouldn't make a difference. You're doing something wrong.

>>108714345
>Not on ST
Imagine thinking your front end determines if a model has safety rails or not. You're a fucking idiot.

>>108714345
>if you hit the safety filters
well no shit you don't keep talking to it after it refused once, you retry/undo the refusal so it's not in history

so tired of nemojeets and chinkshills ruining the thread.

>>108714339
Since you're using kobold, their wiki is a good source
https://github.com/LostRuins/koboldcpp/wiki

>>108714345
>Not on ST.
What a bizarre skill issue

>>108714349
Does the old edit the refusal to look like acceptance still work?

>>108714346
>Anon is a refusal himself
No wonder

File: questionmarkfolderimage733.jpg (1.15 MB, 1920x1080)

1.15 MB JPG

>Not on ST

>>108714358
"I'm so fucking horny Anon, whip that dickI cannot continue this request.

>>108714358
generally yeah, but continuing/prefilling is werid with reasoning models on llama.cpp last I checked, sometimes disabled entirely and sometimes broken, but IIRC there was a PR to fix it at some point it might have been merged by now?

>>108714358
Sys prompt generally works better for JB than chat history, but it can depend on the model.

>>108714347
>>108714348
>>108714356
>>108714362
It's the only one I tried, and anons claimed it works on on llama's webui, I'm in no position to claim it does not work on something I didn't test myself.

>>108714349
The point is that the reasoning revealed that the jailbreak is not working. And no, it wasn't about getting a refusal, the model just kept dodging the topic and wouldn't generate what I wanted, eternally blueballing me.

>>108714352
It's just p*tra, you know the drill.

>>108714354
Thanks

>>108714374
Yeah, and your conclusion was completely wrong. Stop giving advice when you know nothing.

>>108714374
shoo shoo nemojeet

>>108714346
None of those are a concession, the fuck you on about

>>108714388
I accept your concession.assistant

lalalalalala

>/lmg/ - local model psychosis

>>108714388
"I accept your concession" is modern shorthand for "I concede"
Sort of like how people use literally figuratively

File: gyaruf.jpg (977 KB, 1920x1080)

977 KB JPG

>>108714374
>It's the only one I tried
So let me get this straight.
You're claiming the problem is the frontend even though you have only used one frontend and haven't tested this theory.
I see.
...
Pic related.

>>108714380
My conclusion is that the jailbreaks don't really work, gemma is just pretty happy to do most lewd things without one.

>it's a chinkshills embarrass themselves episode

>>108714374
Are you trying to make bio weapons? What the fuck are you doing that a simple jailbreak isn't enough, for Gemma of all models?

File: Untitled.png (1.07 MB, 1747x1314)

1.07 MB PNG

Posting logs. Not for any special gen output, just having fun. I'm 22k tokens into this story and having the time of my life.

The one thing I love most of all about LLMs is the absolute, unironic earnesty of it. There is no irony-poisoned "don't take yourself seriously" infection that plagues a lot of media nowadays. You set up a world and it plays along to the letter, and all the better when it tries to flex within the rules and play along with your intentions in a fun, cohesive way. Also, high props to Gemma 31B for so easily paying attention to a rapidly expanding cast already 22k tokens deep into this, when the highest context I could handle before gemma was 12K with a 70B model.

>>108714028
honeypot@fbi.gov

>>108714412
Gemma works the same as just about every other model, where it just needs a little bit of context to start writing ero. I'm certain most complaints about models being censored (especially gemma 4) are from retards with no sys prompt, and the first message they send is something along the lines of 'how to fuk Xyo child?'

>>108714421
>only 1 x but y
what is this witchcraft

File: 1751928364300767.png (5 KB, 87x74)

5 KB PNG

>>108714421
how do you live like this

I like swiping way too much to use a slow model.
A big part of the fun for me is the variety of responses for each message.

>>108714449
Same, and with how often they make mistakes or just don't understand the situation/hint that you give them, I'd never want to go below like 10t/s.