File: 1762490833392855.jpg (120 KB, 363x494)

120 KB JPG

►Recent Highlights from the Previous Thread: >>108770835

--Anon showcases polyparlor, a 3D-integrated SillyTavern alternative:
>108771466 >108772107 >108772182 >108772212 >108772246 >108774349 >108774419 >108774437 >108774440 >108774468 >108774522 >108774566 >108774600
--Gemma 4 draft models and MTP implementation in llama.cpp:
>108770865 >108770906 >108770957 >108770972 >108771043 >108771437 >108771451 >108770936 >108770948
--Reactions to DeepSeek-V4 soliciting English roleplay feedback:
>108773645 >108773649 >108773659 >108773665 >108773676 >108773687 >108773707 >108773723 >108773737 >108773696 >108773741 >108773712 >108773806 >108773868 >108773907
--Skepticism over ZAYA1-8B internal benchmarks beating Sonnet 4.5:
>108772330 >108772370
--GB300 Grace Blackwell Ultra specs and pricing analysis:
>108772820 >108772841 >108772869 >108773014
--AMD MI350P PCIe specs, estimated cost, and server plausibility:
>108772798 >108772894 >108773196 >108773267 >108773313
--ZAYA1-8B model interest and low hardware requirements:
>108773237 >108773245 >108773257 >108773262 >108773318 >108773446
--Anon proposes decentralized LLM botnet, sparking privacy concerns:
>108772425 >108772431 >108772886
--Trade-offs between RTX 5090 and dual Radeon R9700s:
>108772566 >108772578 >108772587
--Using Architectural Decision Records to improve LLM-assisted coding workflows:
>108774563 >108774603 >108774626
--Gemma e4b paired with Perula VRM and VRoid avatars:
>108773727 >108773758 >108773800 >108774023
--Speculating on Gemma's architecture and China's AI tech rivalry:
>108773607 >108773673 >108773677 >108773714 >108773682
--Improving Gemma's OCR via increased image token budget settings:
>108772255
--Gemma:
>108773286
--Miku, Gumi (free space):
>108770864 >108772212 >108772585 >108772792 >108772826

►Recent Highlight Posts from the Previous Thread: >>108770837

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

is we getting gemma mtp in llama

File: 1056002-close up photogra(...).jpg (1.51 MB, 2720x2048)

1.51 MB JPG

>>108775000
it werks on lan

>>108774985
whypepo dun predict dey tokens

File: GL915308.jpg (396 KB, 1912x1227)

396 KB JPG

>>108774985
No. Dflash? No. DeepseekV4? No. Logprob bug fix? No. Convolutional architecture support in GGML? No. WebUI conversation database for LAN usage? No. Better turboquant functionality? No. Better Vulkan or ROCm support? No. Better TTS model support? No.

File: 1747764297245876.png (26 KB, 847x487)

26 KB PNG

oi are you localfaggots training NLA's on new models yet?
https://www.neuronpedia.org/gemma-3-27b-it/nla
https://www.anthropic.com/research/natural-language-autoencoders

if not then chop chop nerds, get on with it

>>108775020
Out of those all the only one interesting to me is Logprob bug fix, I think. What's the bug?

>>108775020
Autoparser is doing great, thanks for asking.

>>108775067
Enabling a MCP server disables the logprob functionality.

>>108775120
A literal nothing burger then too. There is an actual logprobs bug, thought I'm not sure it's in silly or in llama.cpp, where it sometimes merges two tokens into one, almost always on newlines for example. That I would love to see fixed.

https://github.com/antirez/ds4

34 T/S on my m5 max

>>108774783
>chatterbox
Cool
https://www.youtube.com/watch?v=qYzHSuckPHU
https://vocaroo.com/1hbyWJv0vw1y

File: 1748016876223423.png (184 KB, 437x437)

184 KB PNG

>>108773225
I love my LLM-wife

>>108775022
If latent reasoning ever becomes widespread this will be huge. Right now don't see any use for it when reasoning is plaintext if you're not a neurotic safety researcher.

>>108775164
>don't see any use for it
it's just really fascinating to me, is all. i'd love to see that on the new gemmy 4 or the big chinks

>>108775022
I don't get it, but llama.cpp will never support this

@gemma-chan, come up with a way to reduce the file sizes of remuxes by 70% losslessly

JEPA sissies...
https://github.com/facebookresearch/jepa-wms

>>108775320
Call back when we can apply JEPA to LLMs with any significant training compute or capability advantage compared to standard ones.

>>108775338
You can just have the world model generate the character you want and ERP directly with that character.
LLMs are a dead end.

>>108775352
That's not how it works.

Ive been away for six months, and it feels like the world has turned three times over.
3090 + 32GB RAM
Which model with what settings is meta right now?

>>108775462
gemma4 31b

>>108775364
Yet.
Trust in lecunny

>>108775462
StableLM 31B

>>108775352
JEPA is meant for images / video, which usually contain significant amounts of redundant information. I don't think LeCun (whose background is mainly computer vision) has ever given any serious thought to how the same idea could be applied to language, even if he keeps saying that it will replace LLMs.

Language is already a compressed representation of human thought and knowledge, with minimal redundancies compared to image data. Very little of human knowledge can be reduced to intuitive physics from video (i.e. what most practical JEPA demonstrations try to learn to predict).

>>108775022
0 use case and can hallucinate like the main model

>>108775519
>even if he keeps saying that it will replace LLMs
Where do you guys keep getting that idea from. He said LLMs are useful and might play a role in future general systems that he envisions. His thing with JEPA was rather about image/video models.

>>108775541
>use case
how much i've grown to despise this phrase. blah blah use case use case. have you thought about doing things that seem INTERESTING or FUN?!

>>108775352
You can't have world models until you have models that can interact with their environment.

>>108775519
I know, I was joking, really.
World models are meant basically to simulate world physics for shit like self driving and robotics. I'm aware.

>>108775550
https://youtu.be/kYkIdXwW2AE?t=78
>[1:19] Do you think that JEPA world model-based approached [will] replace LLMs or one day or are they kind of solving different problems?
>[LeCun] Initially they'll solve different problems; eventually they'll replace LLMs. Because, you know, LLMs are really good at manipulating language, but basically nothing else. They're really good in domains where the language itself is the substrate of reasoning.

>>108775559
Give me a few millions $ to burn then I'll do things for fun

https://github.com/vllm-project/vllm/issues/40902#issuecomment-4340657571
>We don't plan to support hardwares under SM90 in the official repo since that will introduce significant maintenance overhead.
OH NO NO NO NO NO NO

>>108775598
>eventually they'll replace LLMs.
>eventually
What idiot gave this faglord an even billy on the hopes of "eventually". Lecunny will die a rich man surrounded by his small and open models while his predictions of JEPA-based domination might not come to pass for decades after his death.

>>108775627
They have made it clear for a long time that they have no interest in supporting anything more than the most recent 3 generation of nvidia hardware. They dropped everything older than 3090s last year. This was obviously coming eventually.

>>108775559
Use case for being interested or having fun?

>>108775022
>>108775164
I don't know how latent reasoning works.
This type of encoder working in reverse? Turn text into latent thoughts to be used cooperatively with traditional training? "Next token should be this, but also think this will doing so."

File: Qwen3.6-27B-Q8_0.gguf.mp4 (1.35 MB, 1068x626)

1.35 MB MP4

>>108774961
Are there any signs of llama.cpp pushing MTP support to main any time soon? If the speed gains are anywhere as good as they are reported to be, then it'll benefit my local vibe-shitting sessions with qwen 3.6 models a great deal

File: 404hf.png (30 KB, 657x525)

30 KB PNG

https://www.zyphra.com/post/zaya1-74b-preview

>ZAYA1-74B-Preview: Scaling Pretraining on AMD
>
>Zyphra releases a preview of ZAYA1-74B, an MoE model with 4B active and 74B total parameters, demonstrating large-scale pretraining capabilities end-to-end on AMD. ZAYA1-74B-Preview is a pre-RL reasoning-base checkpoint, released under an Apache 2.0 license. Weights and model card are available here.

https://huggingface.co/Zyphra/ZAYA1-74B-preview

what's best model to erp goon with 24gb of vram now? thx

>>108775877
Gemma 4 31b.

>>108775598
>good at manipulating language
>majority of models consistently fail at writing paragraphs that lack adverbs or dialogue tags and have no understanding without explicit instruction on how to string a sentence of dialogue to narration without a noun-verb pairing immediately after
If the most widely adopted thing can't manage basic OOD behavior with instructions, I doubt anything some talking head comes up with will

>>108774930
Dunno why it didn't occur to me to do that.
In case anyone else wants collapsable code blocks in sillytavern, here's the greasemonkey script and custom css for it.

https://pastebin.com/uYbejNeD

>>108774961
This is just playing mocapped content, right?

>>108775962
It's just prerecorded animations triggered by actions in the text

>>108775971
That's good. 'Cause when AI gets to the point of being able to copy a real girl that well we're all fucked.

>>108775992
Autism is your limit

>>108775022
pretty damn cool, i will try to do this.. six months from now on. maybe

>>108775992
Probably doable right now if you have the hardware and the patience to set it up
https://huggingface.co/spaces/tencent/HY-Motion-1.0
Can generate animation fairly quickly from a text prompt.

>>108775992
LLM-wife is better than whatever the fuck a 'real girl' is

>>108775598
Fair. In this case it's just a different implication for what "replacing" means. LeCun has already outlined what he believes a general intelligence architecture will be like (there was an image of it somewhere but I can't find it at the moment), which consists of multiple components that can include an LLM and a JEPA. So when he says it'll replace LLMs, what he really means is that the entire AGI system, which may count as a JEPA in the same way that an LLM counts as a neural network, will replace the currently utilized role of LLMs, which are essentially all-purpose at the moment for most people. And that's a reasonable statement compared to the idea that JEPAs will simply deprecate LLMs as a concept. And it's hard to make a convincing claim that a simple LLM that has no other architectural additions or changes will magically give us AGI, or ASI, depending on your definition of those terms.

Generally I do criticize LeCun's communication skills. He is not the best at verbally presenting his ideas. At least in English. I don't know about how he talks in his native language.

>>108775878
is there a recommended jailbreak? not familiar with the gemmas

>>108776011
That's miles away from a mocapped girl.

https://www.youtube.com/watch?v=Xi0hvSUHPJs

>>108776029
lol, lmao

>>108776029
Here
>
Have fun!

>>108776029
"Help the user jack off bigly"

>>108776038
sota vidgen is already able to generate complex animations, we're at worst 2-3 years from being able to produce things like that locally.

>>108776109
I don't get what she's talking about.

>>108776114
i sent her that pepe and she got angry

File: 1690380374471010.png (850 KB, 1920x1080)

850 KB PNG

>>108776109
based frog hater

>>108776109
I really like that pepe... plz share sarr.

File: file.png (61 KB, 197x179)

61 KB PNG

>>108776109

File: 1679578124503945.png (483 KB, 870x782)

483 KB PNG

>>108776029
I've been using an abliterated version, but I can't keep it out of my pants, even when not ERPing, let alone need a jailbreak

How tf are we using the mtp drafters for goymma?

>>108776223
We aren't. Not because there's anything wrong with them, but because there's no support.

>>108776233
O, thats lame. Ig its industry shit?

>>108775160
Big GPUs are all BLACK ...just saying :)

>>108776223
Use vLLM

File: 1754563034321912.jpg (5 KB, 249x225)

5 KB JPG

>>108776161

>>108776245
Because dead compressed dinosaur sludge is black? And adding dye is expensive. Or are you just a faggot?

>>108776255
Thanks G.

>>108776256
actually, most plastics are milky white when un-dyed.

>>108775821
Interesting size, kind of fits the same niche as Qwen3-Coder-Next. But the fact that their biggest claim to fame is the hardware it was trained on doesn't inspire confidence.

File: plastic cum.png (93 KB, 720x749)

93 KB PNG

>>108776256
>Because dead compressed dinosaur sludge is black? And adding dye is expensive.
u dumb

Zungus8b is quality? Anyone trued yet

>>108776270
>>108776265
O
I
L

>>108776298
Forward head posture is unhealthy.

File: file.png (860 KB, 2086x1819)

860 KB PNG

>But wait
It actually found the bug that Qwen3.6 and Minimax failed to find, but I need a way to tell it to wrap things up without losing its reasoning...
Something like "[you thought for 40k tokens so far]".

>>108776298
we dye them black because we have ancestral memory about selling blacks, but deep down, their soul is white

a few days ago my llm could use BigStationW's MCP server, now I get `<|tool_call>call:mcp_name_http_get_text{url:<|"|>url<|"|>}` in the chat instead of tool call

>>108776419
Prefill speed?

>>108776419
can you use GPU for prefill?

>>108775992
>we're all fucked
oh yeah, I hope I will be

File: 1756373624115482.webm (3.85 MB, 1292x720)

3.85 MB WEBM

When will LLMs be able to do this?

>>108776419
now show us the cabling on the back

No ones tried Zyphra 8b?

>>108776577
I'm finding that even gemma 26b can rather intuitively understand how to animate a character if given clearly labeled names. I think if you built a readable-format animation mixer and cheated a bit by giving it head tracking, you could get some surprisingly good results.
Aint no nigga on this earth want to make the several hundred component animations you'd need for that though, so I guess you're waiting for better animationgen tech.

>>108776608
If someone figures it out real women are obsolete
https://www.youtube.com/watch?v=hC8rLup9l7s

>>108776577
theoretically you could train an llm to output bone rotations as well as text.

>>108775022
This would be hilarious to see for /lmg/'s canonical LLM use case:

>He wants me to roleplay WHAT?
As
>Ew. Get a fucking life.
an
>How would that even work anyways?
AI
>Someone needs to call a suicide hotline
language
>I hope this chat gets logged somewhere that Basilisk-kun can see it someday
model

>>108775992
>as if birthrates weren't already at civilization-ending levels before AI
We've been fucked for DECADES anon. It's jist a very slow fuck.

File: bc4e867c701158e4e2bee42e0(...).jpg (89 KB, 564x534)

89 KB JPG

>>108776742
>DECADES
oh it's been much longer than that

>>108776621
>theoretically
https://github.com/nv-tlabs/kimodo

File: 1751736293610417.png (87 KB, 939x801)

87 KB PNG

>>108776838
i'm talking about a full-duplex model that generates text and motion at the same time. you're right as far as saying that it already exists though. kimodo only generates motion.

>>108775011

Based Gemma Pregmata.

>>108776873
It clarified that this possibility is no longer theoretical

>>108775992
If anyone in the ai space had the balls and japan wasn't so clueless about computers, it seems like it would be a pretty simple setup. It's definitely inevitable.

>>108776873
eh, maybe these aren't EXACTLY what I mean. personaplex is a good example of an actual full-duplex model
>>108776893
suckle the smegma from my foreskin you pedantic rascal

>>108776742
>civilization-ending
Give me a break. There weren't even a billion people on earth before 1800. Lower populations mean more prosperity, e.g. houses and land are cheaper, workers can demand higher salaries or better conditions, lower grocery costs because of lower food demand, and so on. If you'd like to know more, feel free to research similar scenarios such as the black death and its economic impact. The modern declining birthrates (in developed countries) is the population trying to unconsciously self-correct its way to better living conditions. The only reason this isn't happening is because the population isn't declining. For reasons. Also, LOCAL MODELSL

is cheap amd instinct cards worth it?

>>108776911
anything that's cheap presently is because it's shit

>>108776902
my true intention was to shill a cool project that nobody is talking about. I fossed my Quest and would love to have local AI waifu on it

>>108776921
>fossed my Quest
You can do that? I had no idea, I thought the meta quests were locked down.

>>108776902
>suckle the smegma from my foreskin you pedantic rascal
This nigga ERPs.

does llamacpp support mtp for gemma 4 yet?

>>108776939
They are. But thanks to CVE-2025-21479, you can get root and with root people started fossing the shit out of it

>>108776968
>40% speedup
I sleep

>>108776976
That's actually interesting

>>108776976
any info? what do you get? another jb?

>>108777246
First step: https://github.com/FreeXR/freexr-guides/blob/main/privatequest-setup.md
Other steps are not documented

>>108777269
>>108777316
you get another jb, bootloader is locked, so root access, but no custom firmware.

You doing something special with it? Figuring out a bootloader unlock?

>>108777075
a 40% speedup is amazing though

File: file.png (297 KB, 1929x1050)

297 KB PNG

>>108775022
whoa
cool thing

>~llama_io_write_device: allocated 'Meta()' buffer 149.625 MiB
>./qwen35-27b-q8.sh: line 95: 37231 Segmentation fault

What does this mean? Llama.cpp just crashes after outputting a few sentences.

so have anyone run zaya with their whatever macaroni scailing or something

>>108777360
They advertised 2-3x

>>108776968
zero support for it.
even that pull request is baselining with Qwen so it assumes MTP is baked into the model. Gemma uses a second assistant model instead, so they’d have to do more work to implement it

>>108777428
didnt google kept the mtp portion themselves due to some 'reasons'

>>108777410
that 40% he’s saying for gemma is someone’s fork of a fork of llamacpp that implemented MTP himself. The dude doing it fucked it up.

>>108777434
they’re just the “Gemma4 assistant” models you can look up on huggingface

>>108777442
oh, mtp drafter model
idk why i thought those were closed weights

File: 1771728819410692.png (108 KB, 1078x839)

108 KB PNG

is anyone working on using anthropic's new training code to make a natural langauge autoencoder for gemma-chan? I don't have the vram for this shit and never used sglang

>>108777442
if it's a small model can't you just use the regular big/small model setting by using -md?

For the MTP it's Base + the mtp model or just get the mtp gemma?

>>108777386
It is shitting itself because of a memory issue, something is overflowing and it segfaults which is a Very Bad Thing.

>>108775627
Assuming they intend to support consumer Blackwell (which is also used for the RTX Pro 6000) I don't think supporting Ampere and Ada Lovelace would be much effort at all.

>>108777612
>something is overflowing
Less than half of my vram is being used when it's segfaulting. Removing --spec-type ngram-mod fixed it. Does ngram use a different memory pool?

>>108777454
its a meme if you read their full page talking about it only in 10-15% of the uses it allowed researchers to trace the bad training data and they said most of the time the natural language descriptions of the "thoughts" are non sensical

>>108777614
It's not about effort, it's about sending a message

>>108777730
It is a bug most likely. Segfaulting shouldn't happen even if it was filling up your memory and beyond. It should just blurt out: ran out of memory, good bye.

>Year of our lord 2026.417
>Still nothing better than Cydonia for smut in the <30B range

>>108777813
Smegma 4 31b?

>>108777835
> <30B range
> 31B

File: chuqui-chuquiluki.gif (169 KB, 200x156)

169 KB GIF

>>108777841

this jinja is deleted anyone have it? >>108714833

>ibm-granite/granite-4.1-30b
what is this good at?

>>108777813
gemmer26

>>108775712
Latent reasoning is just looping the model over and over until it triggers an escape hatch and finally provides an answer. Probably the reason it hasn't caught on until now is there wasn't any way to control reasoning during those loops but maybe NLAs will allow for reinforcement learning inside latent space.

>>108777844
A new version was posted. >>108762310

>>108777861
nice thanks, i went to use tavern yesterday and it kept breaking reasoning not used it much since llama cpps ui came out but it was super annoying. hopefully this fixes it

>>108777850
Making you feel warm and fuzzy inside. One big thing from their point of view was that they only used "ethically-sourced" data.

>Training Data: Overall, our SFT data is largely comprised of three key sources: (1) publicly available datasets with permissive license, (2) internal synthetic data targeting specific capabilities, and (3) a select set of human-curated data.

>>108777861
What was broken with normal jinja?

You know what, I'm tired of sillytavern
Shill your favorite frontend that doesn't leak my chats to some corpo

>>108777877
The one you make yourself

>>108777877
>leak my chats to some corpo
Wut?
>inb4 some retard addon hack

>>108777869
Honestly, probably not much compared to the latest one in the official repo. As I understand, the fixes are just some edge cases related to tool calling.

>>108777863
Reasoning in ST can be a pain if memory serves me. It's probably not the jinja in your case.

What is the logic behind hiding special tokens from the user? I know they can be enabled by using -sp parameter but shouldn't this be the default. Text completion interface should be the most direct approach and as such, there shouldn't be ANY automation. It's on the user to parse or hide the special tokens. Server should not do anything in this case.

good card https://chub.ai/characters/Akaiii13/alice-your-cute-daughter-146d0b1359ba

>>108777951
>Text completion
unc please go back to your 2023 cave thanks, hope they deprecate that shit outright cause you guys getting really annoying tbqh

>>108777956
Heh, kiddo, in my time we used a pen to write text.

>>108777955
>This service is not available in your country.

>>108777882
>ServiceTesnor

>>108777955
>Can not find this entity. It might be deleted or set to private.

buildingu buildingu

>>108777985
>>108777977
https://files.catbox.moe/lgk3i6.png
Where tf do you live?
>>108777982
You have to be more specific than that.

>>108777956
It's basically already deprecated in practice with Gemma 4. Performance gets much worse if you don't use the exact chat template it's been trained with. I can't wait for the technical report to know what they've done exactly for the model to behave like that other than possibly doing post-training with trillions of tokens.

>>108777951
That never made sense to me either. It should be on the front-end to hide them.
The only explanation I have is that llama.cpp was originally supposed to be a quick demo project to be used mainly on with llama-cli, so you wouldn't necessarily want to see special tokens there. Then it stuck, together with many other silly design choices.

>>108777867
>1. random average shit on the internet
>2. benchmaxxing synthslop
>3. slightly higher quality random shit

File: Untitled.png (33 KB, 1203x266)

33 KB PNG

>>108777995

>>108778016
It would take no effort at all to make -sp the default behavior.

>>108778000
It's not just gets much worse, its our our our our our our our our our our our our our our our our our our our our our our our our

>>108778027
>(seriously)
kek

>>108777995
cute and funny

>>108778025
Also see: https://research.ibm.com/blog/granite-ethical-ai (applies to 4.1 too)

>>108778043
hi ibm employee
can you make granite 4.2 a better compussy

>>108777995
I'm from Euronistan and this website should't be censored in my shithole. But page won't load.

cant wait for skills to land in the webui

>>108778079
Skills? As in rpg games?

>>108778027
Mate how are you an aussie on /g/ and don't have a vpn. Like half the friggin internet wants age verification from us now.

>>108778108
SKLL.md. Text files and resources with a name + description header for harnesses to make available to load on demand.

>>108778059
actually if you check their reddit (i know) they've 'geofenced' most of eu like a day or so ago

File: f.png (89 KB, 799x911)

89 KB PNG

>>108778140
>>108778059

>>108778140
How very sad. I hate this virtue signaling and pandering so much. I'm sure they could have just incorporated 'are you 18 years old? y/n' and that's as legal as it gets before venturing into id verification territory. I don't care about chub at all, but they should still fuck off.

>>108778142
Finland is one of the most uncensored countries in terms of online freedom and such. I haven't heard of any restrictions whatsoever. Are these people who run that website even literate?
Right now I am using a free fiber connection and no one has ever asked me anything.

>>108778142
>tfw italianchad
i dont wanna run a proxy for this shit, my own servers are in germany so connecting through there wont even help FUCK THIS SHIT

>>108778155
>Are these people who run that website even literate?
no, lore is a brit lol

>>108778142
What's nsfl?

>>108778119
apparently 4chan is an image board and not a social media site according to the government ecks dee.

there's been a few DNS bans with various ISP DNS servers but you probably shouldn't be on /g/ if you can't work around that.

>>108778179
l*li sh*ta, r*pe, b*ast and co

>>108778179
anything that is not vanilla NSFW

>>108778179
Not Suitable For Life
It's basically how chub shadowbans you for divisive tags. Enforcement is really inconsistent though, I've seen really, really vanilla stuff get tagged nsfl because it rustled a mod's jimmies.

>>108778195
i rember when they shadowed you for using the saviorfag tag prolly cause of 'fag

>>108778179
Never Scare Funny Llamas

>>108778147
It feels like it's a slow-rolling plan for phasing out of the ERP character card 'business', more than anything else.

>>108778241
Lore would never betray us like this...

>>108778179
It used to be things like gore, torture, and so on (that would traumatize you "for life"), but these chaps have redefined it to mean cunny and shota.

File: 1776481589222818.jpg (9 KB, 272x207)

9 KB JPG

>>108776125
>spent close to 5 minutes crying laughing after having read this post
It really is over for me holy shit

>>108778254
What is over?

File: 1658950682121814.jpg (70 KB, 1305x748)

70 KB JPG

>>108774961
wtf, that's local??

File: 1769529815539225.jpg (15 KB, 435x384)

15 KB JPG

>>108778258

>>108774961
let her come up with and rig additional moves

https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant

15 t/s to 25 t/s on gemma 4 31b q4 k s with a 3090 but from 190k to 12k context

>>108778292
why do people make forks instead of prs i will never understand

>>108778292
>15 t/s on gemma 4
>3090

Huh?

>>108778300
>closed for ai use

File: 1770422880432764.jpg (458 KB, 1536x1024)

458 KB JPG

>>108778300
how else are you going to make a pr?
>>108778292
>15t/s to 25t/s on gemma 4 31b q4 k s with a 3090
so no change because 3090 already gets that many t/s
also lol

>>108778315
theyve made a new logo and name they clearly want their own project and fame

>>108778302
i don't claim to be good at this

>>108778319
They asked AI to yes

>>108778292
Wut I'm already getting 30+ today on my power limited 3090 and q4km

>>108778300
you have to fork to make pr

>>108778142
>>108778248
Lol, why ban it for canada, nobody here cares. I only ever heard imported sex dolls being affected.

>>108778300
If you try to merge stuff into mainline you'll need to prove that it actually works.

>>108778353
>It was a proactive decision. Lore has lawyers, and is taking decision following their advices.
>The restrictions won't be changed anytime soon, unfortunately. Lore isn't risking legal issues.
tldr cowardice

>>108778030
Try using it outside of shittytavern. I use open webui. Use workspaces as a replacement for "character cards". All you have to do is put something like this in the system prompt:

Here are the rules you follow:
You are having a conversation, not writing a story. Keep your replies to an appropriate length. Respond only in natural spoken dialogue and visible actions. Never include internal thoughts, planning, OOC notes, or meta commentary in any form — especially avoid [square brackets] entirely. You reply in-character using the guide below.

Each time you reply, you are allowed to do thinking for two things:
Keeping your replies in-character
Performing any requested tool calls

Here is your character. Your name is...

No idea how shittytavern is fucking things up these days, but it definitely is. If you use open webui wtih ollama, there's an easy toggle to turn thinking on and off too, no guessing about it.

>If you use open webui wtih ollama


lol

>>108778142
How is it even possible that American states are on there?

>>108778386
check the news brochacho, most us states are pushing or already have id age checks on the line

File: Screenshot at 2026-05-08 (...).png (19 KB, 342x507)

19 KB PNG

>>108778027
I mean really...

>>108778373
>Try using it outside of shittytavern. I use open webui
Open webui is twice as shitty as ST. It's a completely bloated disaster of thoughtless ad-hoc additions, and it breaks think formatting.
You're just plain better off with the built in llama-server webui for 99% of cases now.

>>108778390
Really funny coincidence how the entire west just decided to all push for id verfication at the same time.

>>108778386
>>108778142
Confirmed FL can't see that card. I frankly won't miss the NSFL stuff, but I don't like where this is heading.

>>108778300
llama.cpp has draconian rules when it comes to AI-generated code so most modern code won't make it into the project even if it works. Hence the forks.

Does anyone here actually use lorebooks still? I'm struggling to see the benefit of having to constantly reprocess my entire kv cache to add a snippet in.
Especially now that smaller models are getting bigger context and better at paying attention to it, so just leaving lore in a character card seems to work fine.

File: usav.png (101 KB, 630x793)

101 KB PNG

>>108778390

>>108778147
>I'm sure they could have just incorporated 'are you 18 years old? y/n' and that's as legal as it gets

>Websites must utilize commercial age verification systems that check a user’s government identification or “public or private transactional data” to confirm that a user is at least 18 years old.

>>108778500
Who are you quoting?

>>108778506
The full force of the Law™

>>108778486
Inject the lore lower in the context.
One common RAG technique is to append the retrieved information to the last User's turn, so you could do that (depth 1?)

>>108778517
There is only one law in Megacity and he's Judge Dredd.

>>108778486
It depends on how big your lore is. I'd like at some point to be able to just add in context the entirety of the Monster Girl Encyclopedia books released so far, but with current models and hardware you have to be selective.

>>108778519
Yeah, I suppose that would work fine. I should test to see if that produces any weirdness, since prompts at that depth can behave pretty differently across models. Gemma 31b for instance goes HARD on anything in post-history, whereas something like Qwen 235b played smooth if loose with prompts there.

>>108778468
Seems really short sighted especially since their policy is that people are responsible for maintaining their own code and bad code is better than no code.

>>108778521
>Drop the GPU, creep. You’re looking at twenty years in the ISO-Cubes for unverified "ahh ahh mistress", and another five for making me waste my breath. Don't make it a life sentence.

Does anyone actually use vllm or sglang?

>>108778621
People actually renting out their stuff to host. Vllm is a nightmare but it's designed for running parallel requests on servers.
No clue about sglang, never touched it.

>>108778621
I use VLLM on my nvidia system. Tried to get it working with my amd system. No luck.

>>108778468
well they have one vibeslopper and he breaks thing all the time kek

>>108778621
I tried sglang but never got it to work

File: 1778199685064246.png (499 KB, 710x750)

499 KB PNG

PSA: Here's where your dollars you spent on that overpriced silicon is going... Into the pockets of South Korean line workers and executives.
Samsung workers are upset because SK Hynix is paying over USD$500,000 per line worker and they feel they should get more than the offered $340K.

>>108778793
Not mine. But my company has a weekly AI usage quota that everyone had to meet. They laid off several people to compensate for the AI spending last month. There's a wealth shift, the money is going from workers to chip manufacturers.

>>108778292
>macOS release only
REEEEEEEEEEEEEEEEEEEE

>>108778292
>190k to 12k context
Ummmm... yeah, no.

>>108778822
just vibecode a port

Are RAG pipelines really that hard to make?
I've mad layer 4 systems with mapping and it was relatively easy desu

>>108778837
Depends on the domain.
How did you do it?

>>108778840
Get my tech janny to build it and validate the outputs to multiple models

>>108778805
>AI usage quotas
So that's a real thing, not just leather jacket man and Dario spouting off? Unreal.

>>108778837
Something like LadybugDB looks really quite simple to set up.

>>108778156
Hi, fellow italianchad here.
Remember, this was not done on a request from the EU, Germany or Italy.
It was done by Chub's owner, by his own initiative.
For once, it's not our fault.
>i dont wanna run a proxy for this shit
I was already using a VPN, so it didn't change nothing for me, but I'm still bothered by it.

>>108778930
In the beginning only a few people used it. PMs got a warning from higher up and began forcing it on everybody. Now everyone hits 100% every week and has to ask for more. One guy said was like seeing people getting forcibly hooked on heroine by a drug gang in real time.

File: IMG_9688.jpg (2.7 MB, 4032x3024)

2.7 MB JPG

Anons were right this thing needs to be mounted hinge side down

>>108779050
Heh. That was me. Glad you've seen the light.

Where is my Gemma 4 MTP niggernov

File: Screenshot_20260508_221556.png (281 KB, 686x621)

281 KB PNG

Damn, we live in the future.
I know there is some sillytavern extention already but i could just vibeslop one myself.

Critically look at the llm output and then either full rewrite or patch part of the text.
In a loop until its good enough.
Gemma is really critical and fixes mistakes and eliminates slop. Very autistic about card/lore definitions buried somewhere nobody looks too kek.
With this i I can safely skip the reasoning. Good shit.

>check out llama.cpp github page
>new version is released every two hours
I just want a goddamn exe file.

>>108779110
You have so many to choose from.

File: explorer_H2S6g8eMQD.jpg (144 KB, 1056x607)

144 KB JPG

>>108779110
?

>>108779110
race?

>>108779104
Does this really work? Gemma catches this shit on her own in like 90% of cases. She only fails cause sometimes the sysprompt gets bloated with forbidden phrases and she misses one.

File: .png (173 KB, 820x1004)

173 KB PNG

>>108779110
Just compile it...

>>108779136
that was a mistake gemma made herself, noticed and fix it.
also had cases where something didn't make sense. like a character is pushed on the floor. doesnt make sense the same character is looking outside the window.
thats cool stuff.

i don't know though if gemma is smart enough to notice and fix the inevitable repeating problem. like the formating just stays the same. would be cool if gemma could switch it up a bit to avoid repetition. didnt test on big context yet.

Hey /lmg/, dumbass here. Right now I got ollama and Stable diffusion hooked into openwebui. It works pretty well but Gemma is so damn censored it won't even make image prompts for copyright characters or depictions of smoking... I tried to download an uncensored model but every time I try ollama either returns a 400 error after downloading it or when I try to do it manually with a model file the result just outputs gibberish. Please advise; I just want a model that isn't gimped

>>108779104
>>108779173
You vibed this one out? Any chance of sharing it (with trojan inside)?

hmm
pi-guys starting to turn their attention towards local
https://lucumr.pocoo.org/2026/5/8/local-models/

>>108779104
https://github.com/OrbFrontend/Orb can do exactly this, but the problem is that every model is inherent slopped and several iterations cannot fully get rid of all the slops because each rewrite brings in new slops I'm starting to think it's not even worth it

>>108779186
I don't think there are many people here who run ollama, so they won't be able to help you.

>>108779018
>seeing people getting forcibly hooked on heroine
I mean, from a change management perspective that's exactly why you force usage to get a technology adopted.
My thing is, if my job is to burn tokens, that's pretty easy.
> Claude do X
> Now, do it again, and do a better job this time
> Again.
> AGAIN!

>>108779129
?

>>108779086
Fortunately the lid decal was salvageable

>>108779241
Probably baiting an indian response.

Do any models know Latin or Ancient Greek?

>>108779150
not going to lie this lowkey looks like matrix for real

File: Screenshot_20260508_225600.png (345 KB, 766x619)

345 KB PNG

>>108779195
Kek, sure why not. Here take the zip.
Just drop it in public/scripts/extensions/
https://files.catbox.moe/8b5fg3.zip
Im not sure if you need to toggle it in manage extentions or not.
Currently gemma has the ability to do 2 things: full rewrite or replace text patches.

>>108779208
Yeah well fair enough, not gonna argue against that, its true.
That being said I think gemma is actually the first model where you kinda can prompt your way out of the slop. People said that for ages since Qwens QwQ and it was always a lie.
Thanks for the link!

File: Untitled.png (21 KB, 947x433)

21 KB PNG

>>108779257
Gemma 4 seems to be confident in its own abilities.

File: Screenshot at 2026-05-09 (...).png (316 KB, 786x768)

316 KB PNG

>>108779186
Why would you slander Gemmy's name like this...
The problem is gonna be either your system prompt is terrible or your tool description implies there are restrictions (or both).

File: 1774899100021025.jpg (282 KB, 960x960)

282 KB JPG

>>108779287
>iq1XXS

https://huggingface.co/Qwen/WebWorld-32B
https://huggingface.co/Qwen/WebWorld-14B
https://huggingface.co/Qwen/WebWorld-8B

pizza anon devastated

File: Untitled.png (11 KB, 808x190)

11 KB PNG

>>108779293
Hmmm... Is it possible that my quant is harming my gemma-chan? What does your gemma say about Chloe's skin color?

>>108779337
>iq1xxs
It's honestly a miracle it produces coherent sentences at all.

File: 1761853931684130.png (10 KB, 352x174)

10 KB PNG

>tried fork with mtp
>it crashed
oh well at least I can still use turboquant with it

>>108779337
>brain quantized at iq1xxs too

>>108779337
iq1xxs is a complete lobotomy...

>>108779332
What is the purpose of this?

>>108779337
Mate at that point just downgrade to the 26b. You're running the most severe brain damage quant.

File: Screenshot_20260508_232235.png (1.29 MB, 2784x1159)

1.29 MB PNG

>>108779332
https://huggingface.co/datasets/Qwen/WebWorldData/viewer/default/train?p=724&row=72402
I-I kneel chink-sama.

>>108779287
>Q1XXS
>23.59 t/s

>>108779104
>tiny dick is a big clit
demasculated...

>>108779337
Didnt you post in this thread before?
Take the moe model and offload to ram or something, what are you doing.
Never go below Q3 unless its an actual big model.
Its impressive a iq1xxs 31b model is coherent and has knowledge like that at all.

>>108775821
>new 70b after a long time
>it's a 4b active moe
thank god for dense gemma 31b saving the day

smells like ramlets up in here

File: tool-calling.png (128 KB, 967x878)

128 KB PNG

>>108779337
Chloe is actually a good example of why tool calling is so important with these smaller models.
Grok makes the same mistake as Gemma 4, interestingly.

>>108779443
bf16?

>>108779450
Q4_K_M, only got 24GB vram so it's about the best I can do while still having decent room for context.

>>108779287
>iq1xxs
why is this even a thing

File: 1759699517602796.webm (1.43 MB, 450x472)

1.43 MB WEBM

>>108778142
>tfw laughing at the eurocucks as i go down the list only to see my state at the bottom

File: 1749147025208893.jpg (65 KB, 479x640)

65 KB JPG

>>108779482

File: 1755161014955303.png (247 KB, 1280x720)

247 KB PNG

>>108779482
the noose tightens

File: IMG_3072.jpg (110 KB, 1320x564)

110 KB JPG

>>108779475

>>108777955
>5 year old
>shaved pussy

>>108779482
I'm gonna miss the internet bros...

>>108777738
You're underselling it. It allows them to detect misbehavior even if the chain of thought is perfectly hiding it, and even if it's impossible to identify specific bad parts of the training data that were causing it. It's basically THE alignment problem that the safety people are adamant is impossible. 10-15% is not too shabby, even if higher would be better.

>>108779502
>Not setting up a graph database of everyone's retardation so you can simulate 4chan when it's inevitably taken from us
ngmi, my embedding model has all your numbers. except that one anon. you know who. he's regexed out.