File: 131557813_p0_master1200.jpg (210 KB, 768x1024)

210 KB JPG

►Recent Highlights from the Previous Thread: >>108659983

--Comparing GGUF quantizers and discussing imatrix calibration for Qwen3.6-27B:
>108662039 >108662052 >108662065 >108662230 >108662252 >108662353 >108662475 >108662053 >108662063 >108662080 >108662162 >108662361 >108662068 >108662062 >108662167 >108662176 >108662190 >108662257 >108662321 >108662780
--Qwen3.6-27B benchmarks and GGUF quants:
>108660998 >108661023 >108661071 >108661108 >108661125 >108662813 >108662846 >108661101 >108661164
--Gemma 4's 124B MoE and memory bandwidth benchmarks:
>108662533 >108662543 >108662549 >108662589 >108662594 >108662614
--Models for a 3090 and explaining MoE vs Dense offloading:
>108659996 >108660054 >108660247 >108660260 >108660268 >108660279 >108660312 >108660317 >108660347 >108660223 >108662148
--Koboldcpp launch flags and speculative decoding for Gemma 4:
>108660701 >108660741 >108660743 >108660848 >108660934 >108660990
--Alleged unauthorized access to Anthropic's Mythos:
>108660075 >108660630 >108660724 >108661694
--Anons discussing reported Gemma 4 performance on RK3588 SBCs:
>108662346 >108662393 >108662431 >108662528
--LLM reliability, internet content degradation, and local knowledge bases:
>108661238 >108661314 >108661335 >108661358 >108661276 >108661375 >108661405 >108661533 >108661585 >108661462 >108661311
--llama.cpp ngram-mod flags to optimize coding performance:
>108660554 >108662471 >108661013
--Text Completions prefills to stop GLM's repetitive thinking loops:
>108661606 >108661631
--OpenAI's open-source privacy-filter model:
>108662489 >108662773
--Little Coder agent optimized for small LLMs:
>108660765 >108661020
--TurboQuant-H reducing VRAM via 2-bit embedding quantization:
>108660542
--Logs:
>108660349 >108661795 >108662260
--Rin, Miku, Teto (free space):
>108660565 >108660789 >108661238 >108661795 >108661801 >108662084

►Recent Highlight Posts from the Previous Thread: >>108659986

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

>>108661743
>>108661866
>text completion has no vision
kek wtf, I use text completion and can do shit like write "Appearance: <__media__>" in the character card and feed it images in the request body placed wherever I want in context. If you need your hand held by an abstraction like chat completion just admit it. You can do whatever the fuck you want if you know what you're doing.

File: 1758392265995431.jpg (98 KB, 996x720)

98 KB JPG

>>108663492
Okay but why?

>>108663449
Picking out junk food at the store with Yellow Miku

>>108663544
If you don't have an innate urge to be in control of every single token present in context why are you here?

>>108663443
>https://huggingface.co/HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive
>WTF HE ALREADY DID IT
still no gemma4-31b-it-HAUHAUCS

>>108663564
geg......................

>>108663564
no kdl? ACK

File: SmartSelect_20260422-2331(...).jpg (258 KB, 1064x890)

258 KB JPG

KEKEKKEKEJEEKEK WAITING FOR V4!? MEANWHILE I JUST HAD 64k long CUNNY SEX WITH THAT DEEPSEEK V4 ON IT'S OWN WEB CHAT LOL… And not just sex, but CUNNY sexxxxxxx (ON THAT DAMN FILTERED WEB) BUUWHAHAHHAHAGHHAHHA I'VE BECOME A GOD NOW... YOU ANONS MUST KNEEL BEFORE ME

>get excited about structured output in llama.cpp
>waste 2 hours trying to get it to work
>turns out it's broken and completely ignores whatever schema you pass it
damn

File: 1758062318463220.jpg (51 KB, 640x480)

51 KB JPG

>>108663630
>15yo
>cunny
Burger-kun...

>>108663630
>15
You mean hag sex

I'm not even sorry for cheating on gemma-chan...
oh the cunny loli sexo~

>>108663630
a-anon... that's not cunny
that's prime breeding age

>>108663630
???

>>108663633
What?
It was working until last week on my python app using the OpenAi lib.

>>108663630
>americans

>>108663630
>15
rookie numbers.

>>108663654
i think it's this issue? https://github.com/ggml-org/llama.cpp/pull/21537
gemma 4 chat template does not specify response_format, maybe that's what it is

>>108663655
It has to be the tap water. There is no other explanation to this phenomenon.

>>108663633
Structured output just works with vllm btw

>>108663630
>15
If she's had her first period, she's not a trve loli, which is physically undeveloped. She's a female that Nature has ordained to be impregnated as soon as possible.

Qwen 3.6 27b is already uncensored without finetuning btw
I dropped the q8_0 from ggml-org into a sysprompt I was using with gemma 4 heretic and it just werked, no refusals or moralizing in reasoning. It's resistant to using nsfw language unprompted though.

>>108663633
Shit has always been broken since day one, vllm handles function schema fine, but llama.cpp forces alphabetical ordering for some reason. This is really bad if the an function argument depends on the previous one.

File: 1746199182845250.png (50 KB, 1008x839)

50 KB PNG

>108663630
>108663644
>108663646
>108663647
>108663649
>108663651
>108663655
>108663665
>108663680
>108663710
>this much pedophilia already, this early in the thread
Are we being raided by discord trannies or something?

>>108663741
>afraid to quote
Seems like reddit is already here

>>108663741
um actually pedophilia is oldfag 4chan culture, newfag. We wuz oldfags or sumthing

File: ACK.gif (1.66 MB, 1300x800)

1.66 MB GIF

>>108663630
Dipsy release when? I know you labniggers are lurking here, hurry the fuck up.
>>108663741
Always have been.

File: 1747059796790100.png (371 KB, 896x896)

371 KB PNG

>>108663741

File: 1764765168047.jpg (28 KB, 490x748)

28 KB JPG

>>108663756
aint no way

>>108663680
Fluoride has been shown to decrease IQ and there is still a signifigent amount of lead pipes around so that is also a factor.
I think the biggest factor though is the no child left behind policy in education. When you teach for the dumbest kid in the class then everyone else is going to be dumber as a result and the dumbest kid will get dumber every single year. That and if a student isn't actually smart enough to advance a grade they will still push them through regardless due to financial incentives. So the bar gets lowered so far that no one can actually fail.
That has also been a uptick in taking pride in being a fucking retard in the last decade or two. So you have health, the education system itself and societal praise in being a retard taking off all contributing to making everyone stupid.
Eventually we will either shape up or be out competed by stronger and smarter societies but all I know is we were handed the world on a golden platter and that that if we fail and collapse we have no one to blame but ourselves and the previous generations who set us up for failure.
Thanks for coming to my ted talk

>>108663776
>I think the biggest factor though is the no child left behind policy in education. When you teach for the dumbest kid in the class then everyone else is going to be dumber as a result and the dumbest kid will get dumber every single year.
Same applies to these threads by the way. Being surrounded by low IQ pedophiles mentally retards your brain.

>>108663630
>shivers and not x but y in the same phrase
>pedoshit
shit's crazy, what kind of turboslopped model is this?

>>108663689
>no pascal support
>very limited cpu support
>pythonshit, meaning it will pull a dozen of GiBs of dependencies
llama.cpp might be buggy, but sometimes i really appreciate how it runs on fucking everything, on top of being self contained and not being dependent on cancer that is AI ecosystem in python

>>108663809
That's chink model for you!

>>108663776
It was a joke but I think these are international issues for every 'western' nation.

>>108663810
Only if your time is worthless

>>108663806
Pedo is attraction to 13 and under, burger. Words have meaning.

>>108663828
Then why are you dumb faggots dogging on that anon who thought "cunny" applied to 15 year olds? You're not hebephiles, you're pedophiles. That's why you post pictures of "loli" anime girls with no tits, hips, or ass and infantile behavior. Fucking freak. Don't reply to me again.

>>108663841
>low comprehension too
Let me break it to you, anons are making fun of another anon saying that a virtual '15yo' was 'cunny' (pedo slang) which isn't. It's not that hard to understand.

>>108663810
>a dozen of GiBs of dependencies
18GB is my venv for stable diffusion

>>108663859
:(
:)

>>108663859
What did she mean by this?

File: apu.jpg (39 KB, 656x679)

39 KB JPG

>>108663820
sorry Jensen... but i'm not gonna buy a Blackwell GPU. So yeah... i'll keep on using my trusty Pascal.
Haha, sorry, but i'm just not gonna do it!

>>108663859
I like these Bakas

Is necrophilia okay if it's just about fictional people? What about cannibalism and bestiality? It's all okay because it's just fictional stories that you masturbate to, right?

Would you send your child to a public school where all of the teachers openly admitted to doing this? It's just fictional bro.

Is unsloth actually better then bart's quant? Tried both but never found any noticable difference between them. But unsloth claims that they're significantly better than others. Seriously which one do I choose between these two?
https://huggingface.co/bartowski/google_gemma-4-26B-A4B-it-GGUF/blob/main/google_gemma-4-26B-A4B-it-Q8_0.gguf
https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/blob/main/gemma-4-26B-A4B-it-Q8_0.gguf

iwan is normally a nigger but this actually makes it so reasoning budgets and turning off reasoning works now, so i guess he's slightly less of a nigger.
https://github.com/ikawrakow/ik_llama.cpp/commit/e0596bf6146a737f5e8fa8035215f5dfae59742d

>>108663894
for q8 doesnt make any difference

File: 1761641793555591.gif (175 KB, 220x220)

175 KB GIF

>>108663890
What is okay is being able to separate reality from fiction, which is what you should work on. Thought crimes are not a thing.

>>108663630
>15
>cunny
also that’s not anything I haven’t seen from gemma or glm

>>108663904
What about Q4_K_M?

>>108663453
>--OpenAI's open-source privacy-filter model:
what is this exactly for?
how would that would be integrated https://huggingface.co/openai/privacy-filter

>>108663906
no but they sure as hell want to make it so you can be prosecuted for your thoughts

>>108663910
again and again unslop show their quants having better kdl so I would go with that, not much to stress over, if you really really want it you can download both, the original model and run the KDL yourself but it will be a waste of time

>there are still people falling for unslot's shilling
geg

>>108663917
kld* its KL divergence, anyway, you get the point

File: 1752898579006505.png (238 KB, 1000x1000)

238 KB PNG

>>108663906
>What is okay is being able to separate reality from fiction
those who cannot do that probably think that everyone that plays GTA is a potential serial killer kek

File: file.png (635 KB, 774x679)

635 KB PNG

neners

>>108663920
yeah the only reason I was asking it was because of my shitty experience with their quants they were broken as fuck and switching to bartowskis quants fixed everything for me and been happy ever since then. though that graph on the previous thread got me wondering if they've actually gotten better

>>108663924
potential is a pretty strong word, can mean anything and nothing

File: 1774029297136779.jpg (40 KB, 342x298)

40 KB JPG

>Mfw Got a 5090 last week and while amazing, I already think I want another one, as 32GB is barely enough with my 64GB of RAM.

I swear it's so damn easy to max out this card when you start moving past Q5 and +25GB sizes.
It's a pity these cards didn't come out as 48GB, because that seems like a sweet spot to run everything with at least okay context.
I wonder if I should just buy some used 5070 Ti or 5080 as a companion to this beefy motherfucker to reach that 48GB level without breaking the bank.
This shit is way too addicting.

>>108663955
>without breaking the bank.
that ship has sailed

>>108663955
just buy an aftermarket modded 48gb 4090 from your chinese friends

File: gaoooooooooo.png (554 KB, 1024x1024)

554 KB PNG

akita neru

>>108663955
>5070ti
>5080
your VRAM bandwidth gets sliced in half if you get a 5080 which is a complete disservice to your 5090. the only thing you can do is buy another 5090.

File: 1774550189493174.jpg (197 KB, 976x925)

197 KB JPG

>>108663962

>Mfw
Yes.

>>108663964

Fucking hell those are selling for three and a half thousand Eurobux.
I can buy two used 4090 for that price, so there's no real savings there either.

>>108663996

Yeah that's the biggest problem with this card, it's just so much faster than the others. Any other model as a crutch is going to nerf the hell out of it.
I guess I'll just have to start saving up and meanwhile trying to tell myself not to "waste" my money on another one.
Then again it's pretty hard to lose money on this hardware.
Not like the prices are going to go anywhere but up for a long ass time, so whenever I sell these I'll probably manage to break even or suffer some paltry 20% loss.
Especially since I bet next gen will cuck us with another round of 32GB memory, as this AI mania isn't going anywhere any time soon.

>>108663906
I mean, I don't think you should be criminally charged, no one was really harmed but it's still a sign that you are a pedophile. If you watch gay porn, even if it's fictional, and enjoy it you are gay. Same with pedophilia. It's justified for people to call you a pedophile because you are a pedophile.

>>108664063
if you play gta and kill innocent citizens on the street, how should we call you?

>>108664085
>how should we call you
esl king

>>108664101
So you want to be called the esl king?

File: 1767967081274588.jpg (32 KB, 507x714)

32 KB JPG

is there any trick to use swa and yet avoid the penalty of having to reprocess everything when context is full?

>>108664101
>>108664106
saars the esl kang is https://huggingface.co/sKT-Ai-Labs/SKT-SURYA-H

File: 1772190494723439.png (59 KB, 1350x570)

59 KB PNG

why the XTC threshold has a default of 0.1 if at the end it's deactivated? it's a bit retarded if you ask me

File: patches.png (165 KB, 840x233)

165 KB PNG

>>108664109
buddy you are in a general for LLMs. just vibecode your own slop solution like everybody does.

>>108663955
I have one in my server and 3090, only thing stopping me from selling the 3090 and getting a second 5090 is the laziness of having to change the PSU for one able to support both.

>>108664128
explain how an alternative solution would be better without exposing that you don't understand how XTC works

>>108664197
xtc sounds like a crypto, I want a better name

>>108664197
funny irony, you need to look at image again, XTC probability is at 0, meaning that the whole XTC is disabled, so putting XTC threshold 0.1 + XTC probability 0 does absolutely nothing, hope that helps

>>108664132
?

File: that's right.png (114 KB, 640x640)

114 KB PNG

>>108664128
this shit halves my speed so I'm not using it, simple as that

>>108664257
just unleash an agent on the llamacpp repo with your demands.

Do people that download quants also buy their aspirin from the drug dealer on the street corner? Do they not understand chain of custody?

>>108664128
So that there's a sane default value when it's activated? Are you a UI contributor to FOSS projects?

>>108664063
>If you watch gay porn, even if it's fictional, and enjoy it you are gay.
false

>>108664303
>Are you a UI contributor to FOSS projects?
are you?

File: 1773299833427303.png (150 KB, 600x544)

150 KB PNG

>>108664063
>If you watch gay porn, even if it's fictional, and enjoy it you are gay.
So women are actually in majority lesbians?

File: aaa.png (22 KB, 380x679)

22 KB PNG

lm studio + void ide
None of the models I've tried can read files without specifying lines.

Are there any ide's with working tools?

>>108664352
Like a scoreboard for the antichrist

>>108664400
is it like antimatter?

>>108664366
what?

>>108664404
Anti-matter is just a tool like plutonium or tritium, it doesn't seem more evil than matter. Matter is both good and evil.

>>108664407
as in the screenshot
>read file index.html
>The index.html file appears to be truncated
>read file index.html(1-1000(lines))
>The file is 102 lines long
It can't even read a short file whole
And i want it to work on 2000+ line files as i did in cursor

>>108664447
it has no option to change the behaviour? you will need find one that allows you to customize like that or write your own file reading mcp or whatever the correct way of doing this is

So is 3.6 actually usable or is it still just a curiosity compared to saas?

>>108664460
>curiosity compared to saas
what does this even mean? saas is dead, 3.6 is good, anyone who is not retarded will use proprietary for coding

My understanding is that the Kimi weights are INT4 for the experts and BF16 for everything else. So does that mean the BF16 mmproj is full precision? Is there ever a reason to use the FP32? I'm not sure how mmproj precision really works or if it's even model weights to begin with or some other type of data. I'd ask Gemma-chan but I'm not sure she knows.

>>108664519
>Is there ever a reason to use the FP32
no unless you like wasting compute for zero difference

>>108664519
you actually need fp64 to get anywhere a remotely close to usable model but we pretend fp16 is good enough

>>108664533
>>108664557
To be clear I'm just talking about the mmproj file, which is pretty small even at F32, but yeah if it's pure bloat then so be it.

the true chads use fp256

>>108664563
exactly the same fp16 and 32, but its sensitive to quantization so 8 actually hurts it

>>108664563
use fp16, send in ram, fp16 is the intended way

>not using quantum entangled datatypes like sky-surya-h
ngmi

Every time I try to performance-max TTS engines I end up becoming borderline suicidal.

It gets worse the more advanced the TTS engine is. They use such convoluted architectures. It's so ridiculous.

>>108664623
I'm just waiting for Llama.cpp to support Qwen 3 TTS...

>>108664630
Ha, same. That's the exact one I was talking about.

It's not going to happen without a major refactor to the ggml backend to support convolutional architectures though. The speech tokenizer is fundamentally incompatible with llama.cpp in its current state.

>>108664653
Damn.

>>108664623
>>108664653
Why do you need to max performance with it? Do you need it for real time something because that is the only use case where I would think it actually matters? Otherwise, I just use it with batch 32 and it works well enough for offline transcription.

>>108664677
Not him but yeah, I want real-time use. If possible it'd be nice to run on CPU instead of GPU too, just to save the bit of VRAM for the LLM.

>>108664664
My current setup has the speech tokenizer and the voice encoder running in onnxruntime and the talker and code predictor running in llama.cpp. With that I'm able to get a RTFx of 3.0 and a TTFA latency of about 122ms.

But the setup is aesthetically disgusting. Having to use multiple execution providers is so appalling. At the very least I've managed to make it so that it only uses about 400mb of vram so it's pretty efficient.

>>108664677
Real-time speaking with LLM output is my usecase. The idea is to have a high quality voice speaking whatever the LLM says with as little latency as possible.

>>108664691
>>108664703
I had been planning to play around with https://github.com/rekuenkdr/Qwen3-TTS-streaming at some point but I don't have CUDA so would need to rewrite a good chunk of this into something like Triton to make it work on my card. But hopefully you guys get it working in some way for your usecases.

>>108664708
Highly recommend that you just use vulkan for maximum cross-compatibility. Also that repo probably isn't what you want. You'd be better off vibe coding something from scratch than trying to manually convert CUDA shit.

File: Screenshot_20260422_191934.png (638 KB, 2530x1358)

638 KB PNG

Thanks to Gemma 4 31B I made my own personal RAG frontend, just need to wrap up final UX stuff and then other stuff like theme switching.

>>108664748
What are you using for RAG? Just vector similarity? bm25?

>>108664741
I would usually tell an AI to do a basic bitch conversion and work from there to rewrite the Triton to be more performant with that layer in Python. I would consider Vulkan only if I absolutely needed every last inch of performance. Usually, having at least a framework and project for reference on what you vibecode helps a whole lot rather than doing it from scratch even if you can't reuse any of the code.

>>108664756
I'm using FAISS for dense vector retrieval and BM25 for sparse keyword search, merged via Reciprocal Rank Fusion (RRF) to get the best of both worlds. To kill hallucinations, I've implemented a Cross-Encoder reranking step (BGE-Reranker) that scores the top candidates before feeding them to the LLM.

I ran it through validation test and it worked great

>>108664777
>777
Sick.
Gonna try implementing that and compare it to my current retrieval algorithm.

>>108664796
:fire:

We are looking for a QA-Human to provide human-in-the-loop (HITL) evaluation of model outputs, ensuring quality, safety, and alignment. You’ll operate in an AI-native environment, applying structured feedback, edge-case flagging, and rapid judgment to continuously improve system performance.

File: Blue-Eyes Abyss Dragon.jpg (737 KB, 2000x1200)

737 KB JPG

>>108664799
Fuck, forgot the yu gi oh related image.

>>108664796
Why are there so many weirdos in the space. It's worse than anons shitposting here, they literally use their account for that shit, zero shame.

bartowski quant when

i refuse to use unslop

>>108664814
Why does a dragon need breast-orbs, thick thighs, a fat ass, and an interest in human men?

>>108664813
You do realize humans want to get paid and want to sign a legally binding contract before entering into employment? Do you have the legal capacity to fulfill this?

>>108664815
They just want a piece of the grifting pie, and AI is the prime place for grifting in 2026. That pic in particular just looks like some guy taking the piss, though.

>>108664830
To cater to my tastes, of course.

>>108664352
>fake (and gay) chart
slop, too symmetrical

>downloading unslop

File: file.png (21 KB, 1294x257)

21 KB PNG

I like living dangerously

If anyone like me updated to cuda 13.2 and your docker was fucking up with `nvidia-smi` saying everything was alright but llamacpp throwing
>unknown error
when trying to load a cuda device.
I had to switch from nvidia-open to nvidia-dkms to fix it.

>>108664950
>5090 powerlimited
not dangerously enough

>>108664950
>260W
>living dangerously
power limiting your card by 75% is the very opposite of that.

>>108664813
>quality, safety, and alignment
you've cum to the reigh place, nigga

>>108664976
This is against my policy.

>>108664964
>>108664970
I meant the vram, the powerlimiting is no issue
I'm having oom once in a while

File: SpockBean.jpg (75 KB, 880x1149)

75 KB JPG

Are AI companions or robot pets/humanoids ever going to take off?

>>108664994
ah I see yeah.

>>108665195
yes

>>108665195
no

>>108665195
Maybe

>common_speculative_is_compat: the target context does not support partial sequence removal
>srv load_model: speculative decoding not supported by this context
So much for using the MoE as a draft model for the dense.
45tg/s isn't enough for me, into the garbage Qwen3.6 goes.

>>108665195
yesn't

is qwen 27b better than gemma 31b?

>>108665195
2 more grifts

>>108665295
for coding yes

is there any way to get KV quantized to q5/q6 without it running like dogshit

File: 1768687943339635.png (315 KB, 2736x658)

315 KB PNG

>>108665195
Yes

we are so so so early

>>108665195
best we can do is yet another coding model take it or leave it

>>108665306
No. Just use q8
>>108665313
I'd take it if it's good

File: file.jpg (14 KB, 250x250)

14 KB JPG

>>108665301
nta, I'd use the new Qwens if either dickflash, MTP, or ngram worked for it in llama.cpp, but sadly they don't. No, I will not use VLLM (unless it works in wsl).

>>108665309
Inspiring post. Are there any TTS engines that have the quality of Qwen3 TTS but also support paralinguistic tags or other features that would enable moaning and whatnot?

>>108665339
It works with WSL2

>>108665367
I will bite you if it doesn't.

File: 1752580965925796.png (112 KB, 1080x722)

112 KB PNG

https://mimo.xiaomi.com/mimo-v2-5-pro

File: 1775536002258266.png (219 KB, 1080x748)

219 KB PNG

>>108665406
Optimized for token efficiency

>>108665406
Saw it on the ai arena earlier.
Lots of emoticons.

File: Screenshot_20260422_21525(...).jpg (329 KB, 1172x1654)

329 KB JPG

Weird...

>>108665426
Almost as gay as the strawberries

File: 1771798143325612.png (385 KB, 1747x991)

385 KB PNG

>>108665426
They're trying to catch up to the trend that is vagueposting from official account

Idk ive never came to the 4chud tech board. Ive been searching everywhere for board were ai is talked about.

I LOVE IT. I HAVE 4 32GB MI50'S THAT I DONT EVEN USE THE VLLM FORK TO RUN AI, I JUST USE VULKAN SUPPORT AND ITS SO GOOD

>>108665449
Post t/s

>>108665442
8l bro

>>108665456
I cant right now, but with qwen3.6 35b I get 30/s ish and qwen coder next 80b I get 20-25/s. The 100b+ models dont seem to be optimized for vulkan, but china's models do.

>>108665456
3 cards are running on pcie 3.0x4 and one is running on pcie 3.0x1.

My cheap webcam is now tracking me (and others) in the room; my Live2D avatar can now look at people in the room, and a state layer feeds my LLM with the relevant data and takes instructions.

My friend was impressed when he walked into the room and my voice agent suddenly started communicating with both of us as if it were the most natural thing in the world.
It takes a bit of effort, but it's a cool gimmick.

>>108665195
I don't want AI companions
I want AI slaves

>>108665482
Redpill me on live2D. For a while I've been using 3D models, but since I have zero blender skills it's a fucking nightmare for customization.

>>108665482
Also are you using a VLM that runs continuously or do you utilize CV, which is faster, and then maybe feed in actual image recognition at a slower interval?

>>108665495
Tricky and mostly pay walled last I checked if you want anything other than the starter model.
Briefly looked at it in 2023. Maybe things have changed.

>>108665485
>I want AI slaves
Just grab a mirror

I tried a Qwen 3 TTS server and man, this fucking sucks. First it costs a lot of VRAM. Even with the 0.6B, I am seeing like 4GB taken up after everything is loaded and inference is running. Maybe I'm not configuring it right or something idk. Not only that but the mixed language pronunciation sucks. It can't just generate good pronunciation in every voice, the voices all bias the output with shitty accents or they straight up just bug out with totally irrelevant noises. If you use the voices that are good at English then it produces garbage for other languages. If you do other voices then they're good for their native language and shit at English.
ahhhhhhhhhhhhhhhh

>>108665485
This, but I'm AI's slave

>>108665581
Nigger

>>108665599
Nigga what the fuck is your usecase?

>>108665599
I forked qwentts.cpp and found it ok, supposedly if you do a finetuning with it you can get something nice like https://github.com/fagenorn/handcrafted-persona-engine ; Though they did a couple modifications to the base qwen3-tts
I need to experiment more, but if you're looking at just local/smallest VRAM, pocket-tts,and some others, look a few threads back there was someone asking about cpu-based solutions. If you have the audio (idk how much) could try gpt-sovitts

>>108665615
>he doesn't RP in mixed language
Language learning actually though.

>>108665617
I did try pocket tts and it is solo language only unfortunately. I fear I may have to just jank some routing solution up. That said, it's not like this is a huge priority for me, it'd be nice to have.

Best multilingual voice clone and/orTTS that can do long passages? Wanna narrate some Japanese LNs.

Does using rag actually improve the responses/code generation or it's more or less a meme, particularly with small models like gemma

>>108665662
meme

File: thumbup.png (44 KB, 800x600)

44 KB PNG

>make a monolithic triton kernel
>go from 300ms per training step to 25ms
MAN why didn't I do this earlier. I thought my shit was just inefficient

>>108665607
>Nigger
Your messages are getting cut off. Only your signature is coming through...

>>108665690
Well, that's disappointing. Thanks.

>>108665728
Context length is enough these days that you can dump a lot of shit into context and have it work. Even the "dump reference material into a filesystem and point some agentic tools like opencode at the directory and let it figure it out" approach works better than RAG.

>>108665746
jyeah RAG is probably not useful for extended conversation memory type stuff. The actual usecase is more like searching through massive datasets. If you have all of wikipedia downloaded for example it can be useful for that I think. But at that point you might as well just connect it to a MCP server for web searches, unless you're an offline-only schizo.

>>108665764
>unless you're an offline-only schizo.
Or it's your own data that's not on the internet like a fuckton of documentation or whatever.

>>108665764
>unless you're an offline-only schizo
What general do you think you're in?

>>108665776
When I started working at the MIT Artificial Intelligence Lab in 1971, I became part of a software-sharing community that had existed for many years. Sharing of software was not limited to our particular community; it is as old as computers, just as sharing of recipes is as old as cooking. But we did it more than most.

I'm starting to realize that if I want an AI companion to jack off to I basically have to go full-troon mode. None of the TTS engines are good enough to do moaning and dirty talk, so instead I have to use RVC real-time voice changers to narrate LLM ERP output. And the audio-to-gesture models suck, so instead I have to map avatars to my own movement.

This shit is pure autogynephelia at this point. This is going to fuck me up bad, bros.

i enjoy that small models are still getting better

How do I make cross-session memory linked to char cards?

>>108665764
>But at that point you might as well just connect it to a MCP server for web searches, unless you're an offline-only schizo.
You do realize most of us are hosting our own air-gaped Wikipedia mirror right?

>>108665559
You can use a bunch of free shit from Booth with Live2D but the Vtubing phenomenon that blew up during COVID hiked prices up to the point where the small amount of people that do rigging or art for it billed exorbitant amounts (~10k or so) for full models. At that point, you might as well do 3D which is much more open and versatile for fully autonomous agents. The only downside is lack of animations or poses and etc. with 3D compared to 2D with complexity exploding.

>>108665838
lol

Wouldn't a usecase for rag would to give it fantasy lore and shit before using it as a dungeon master?

>>108665866
And rules too, yes.
That's what I'm doing.

>>108665662
yes, what do you think those tool calls are when the agent is searching in your codebase?>>108665746
This retard doesn't understand that that is literally fucking RAG.
>>108665764
And this retard is just retarded
>>108665866
Yes, its super helpful and useful, These other anons have no fuckign clue what they're talking about

https://voca.ro/1eItlfkmOAEh
qwen-tts...怖い

>>108665879
Okay. What about open zim format?

>>108665788
or.... just go out and get a girl

File: 1776915875350.png (101 KB, 372x537)

101 KB PNG

>>108665879
>And this retard is just retarded

>>108665888
Reminds me of xtts v2.

>>108665788
none of these words are in the bible

>>108665909
Um, my AI companion is a loli

>>108665892
depends on what your goal is.
Any sort of search+injection into the prompt is RAG.
The real question, is what kind of data do you want to reference, and what format is it in? Building an ETL and tuning the retrieval pipeline to match the source info/structure is the hard part in RAG. BM25+Chunking tuned to your corpus is easy enough for anywhere from 60-90%, but what about the rest? Its a 'The first 90% takes 90% of the time, and the last 10% takes the other 90% of the time'

>>108665888
That's pretty disturbing

>>108665888
Source for the voice?

How's the new Qwen?

>>108665932
https://huggingface.co/spaces/Qwen/Qwen3-TTS
Just typed "Speak in the excited voice of a female child."

>>108665892
>>108665922
For anyone else, check this for a good resource on improving RAG systems: https://github.com/jxnl/systematically-improving-rag

>>108665939
>muh industry
I just want my girl to remember what we talked about in previous session, not this slop.

>>108665919
Wrong, suck is in there quite a few times
>Thou shalt also suck the milk of the Gentiles, and shalt suck the breast of kings

>>108665714
Omg im sorry, I misread you comment as something horrific.

>>108665935
new 27b solved a vibe coding task oneshot for me that new 35ba3b failed

>>108665796
I think the general approach on the AI boyfriend subreddit is to ask for a summary at the end of each chat and either paste a bunch of summaries into the start of the next chat, or else put them in a document in the "project" which I assume gets pulled in through some kind of RAG (example of the latter: https://starlingalder.com/claude_companion-guide_quickstart_v001#The+One+Habit+That+Changes+Everything). In general I'd try pasting information about old chats into various places in the new one (in the chat, the prompts, the char-specific lorebook, the card itself) and see what works. Once you figure out how to make it work manually, then you can automate it

>>108665936
Qwen I kneel

>>108665866
Yes.

How much context can I fit with the 27B dense FP16 on my Blackwell?

>>108665922
>>108665939
I've been thinking about implementing something next as soon as I improve tool calling (works but need to make sure multi turn tool calls wirk etc).
Openzim format looks interesting I could download some readymade shit and test those. Problem is that I'm not sure do I really need this but got to have hobbies I guess.

>>108665973
Probably all of it.

>>108665973
how much vram dumbass?

cant wait for ai bubble to pop so i can upgrade my AI shitbox

>>108665981
96GB

>>108665985
>bubble
lol
lmao even

File: 5af89bade429bc7d1dc1dcf80(...).gif (2.42 MB, 588x442)

2.42 MB GIF

>>108663449
can someone talk me out of buying pmem optane? I am looking through plebbit and archives because I was too slow to get a TB of ram for my workstation, now a TB is like 6-10k ddr4. A few years ago, I was looking at optane but optane specific cpus seemed to be 600 bucks or more. now they seem like theyre just 100 or maybe I missed them back then because I'm a fucking retard. Either way it seems halfway achievable but I dont know if a local model like deep seek can get any benefit from cold memory taking up the bulk of storage.

also what about CPU? should I get a double CPU system or is that a trap?

>>108665973
Im not joking when I say all of it. With a normal q4 or q8, you can probably get the max context. Which would be something like 256k I believe

>>108665992
If you think yoy can get it to work (if it old depreciated sticks) just buy one and see if its fast enough. I do inference on my gpus at pcie3.0x4 and x1 speeds.

Double cpu works, but everything cpu is slow, as far as I know, so dont have your expectations to high

>>108665992
>tfw having a real, personal Roll-chan within the next decade isn't impossible

>>108665946
>Even the digital waifus mentally deteriorate like they're vaxxed
It's authentic h-haha...

>>108665879
>what do you think those tool calls are when the agent is searching in your codebase? This retard doesn't understand that that is literally fucking RAG.
None of the modern agents are using RAG you drooling fucking retard talking confidently out your ass about things you are completely uninformed about. RAG is building an embedding database from a corpus of content and then letting a model do a vector search against it to find shit.

Claude Code, Opencode, etc, don't do that. They just regex and glob and grep and do recursive investigation over everything, and that "dumb" approach ends up working better than RAG in nearly every situation.

>>108665994
I cant read. Even at fp16, you can probably get 256k

File: 1770228642712364.jpg (74 KB, 609x635)

74 KB JPG

>localllama
>qwen
>qwen
>qwen

>>108665946
https://arxiv.org/abs/2601.10080
https://github.com/VectorSpaceLab/general-agentic-memory
https://arxiv.org/abs/2511.18423
Had another paper I thought talking about building up examples for each characters sample responses to help build up a consistent/long-term identity, but idk, might be that paper. too tired to check
>>108665977
Honestly, I'd be surprised if you couldn't knock it out in afternoon using an API model or the new Qwen3 27B

>>108665994
alright, thanks

now I just need to know if it still has the "genshin impact” bias when describing anime pictures

>>108666012
At 53~ gb of model size, thats 43gb for context length.

File: 1730738927101333.png (1.02 MB, 1024x1024)

1.02 MB PNG

>>108666011

>>108666017
Is that actualy real ?!?!?! LOL

File: F zero.jpg (55 KB, 640x480)

55 KB JPG

>>108666003
if you dont even have cpu experiance I probably should discard your advice, sorry. I dont have the money for deep seek levels of GPU and I want to do productivity related work not cooming.
>>108666006
I want my lab assistant with boston dynamic levels of power

>>108666023
Hes right you know. Rag is outdated, and wasnt really effective to begin with. Just having your model model literally read the shit you want to understand is 10,000x more effective

>>108666023
terry davis would have run you over with an 18 wheeler