File: neneru.jpg (186 KB, 1024x1024)

186 KB JPG

►Recent Highlights from the Previous Thread: >>108225807

--Anthropic accuses Chinese AI labs of model distillation attacks:
>108225834 >108226089 >108226211 >108227391 >108227445 >108227803
--Qwen3.5 releases and benchmark analysis:
>108228811 >108229098 >108229143 >108228839 >108228855 >108228880 >108228883 >108228888 >108228902 >108228906 >108229182 >108229276 >108228901 >108229092 >108228977 >108228982 >108229706 >108229771
--[ggml-quants] Add memsets and other fixes for IQ quants:
>108230841
--Qwen 3.5 27B context shift bugs in llama.cpp:
>108229998 >108230028 >108230037 >108230046 >108230058 >108230075 >108230083
--DSv4 hypothetically outperforming Claude 4.6 Opus and its implications:
>108226713 >108226755 >108226773 >108226812 >108226856 >108226808 >108226836 >108226824
--Performance comparison of Sonnet 4.6, Sonnet 4.5, and Q3.5 models across benchmarks:
>108230049
--OpenAI scales back spending projections amid hype skepticism:
>108230234 >108230255 >108230278
--Testing GPT-oss 120B for NSFW behavior:
>108227165 >108227178 >108227195 >108227208 >108227415 >108227709 >108227715 >108227831 >108227847 >108229134 >108229285 >108229822 >108230275 >108227970
--Qwen3.5-35B-a3b model response speed and filtering behavior:
>108230266 >108230299 >108230366 >108230461 >108230515 >108230549 >108230580 >108230742
--LiquidAI releases LFM2-24B-A2B:
>108228076 >108228103
--Mobile AI TTS solutions for Android and Quest:
>108226780 >108226901
--AI chatlog shows model identifying as DeepSeek-V3 despite Claude labeling:
>108227414 >108228464 >108227426
--GLM-4.7-Flash derestricted model performance and behavior observations:
>108230109
--Logs: LFM2-24B-A2B:
>108228330 >108228348
--Logs: Qwen3.5-35B-A3B-Q4_K_S:
>108231958
--Teto and Miku (free space):
>108225907 >108225952 >108226191 >108227875 >108228954 >108229133 >108230301 >108231993

►Recent Highlight Posts from the Previous Thread: >>108225810

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

>>108232121
>>108232139
Deliberately spreading printed misinformation with Neru

i'm a boomer from the days of gpt2 finetuning. how much memory would it take for me to use my own dataset on one of these modern large models?

File: qwen.png (26 KB, 870x411)

26 KB PNG

I used the biggest model on Qwen's website to translate an image.
It thought for a while, said thinking completed and that was it. Did it forget the output or what?
The smaller models running locally had no problem with this task.

Those model restrictions also mean you can't make the AI create new stuff that the companies don't like. You can't come up with a new drug structure or ways to synthesis insulin with chemicals at your disposal or anything because that would piss off big pharma and they'll call it terrorism

Saars...
It's cucked.

>>108232172
Depends on the dataset, the model, the finetuning engine you use, the quant if you quant, if you do a lora or a full finetune...
How did you manage to finetune anything if you don't know how to search for that? Check axolotl or llamafactory I suppose.

>>108232172
In my opinion for good results you need at least twice the memory required at inference time, when finetuning with QLoRA.

Will there be a huge difference in quality if I try to finetune Qwen3.5-27b in Unsloth using "load in 4 bit" vs if I wait for Unsloth to do one of their fancy 4 bit non-gguf quant?

I am testing Qwen 27B right now. Its thinking is so retarded, it's hallucinating RAG search results, it's hallucinating made up information, and I also got looping once. It can answer what a mesugaki is but none of my private test questions including similar slang and a bunch of other things. Definitely benchmaxxed.

>>108232241
Uhh... yeeees... maybe...

>>108232271
I guess the way is to try it. Downloading it, I hope we'll get 9B or whatever soon.

How do I increase the context window?

>>108232288
-c

>>108232288
load with --max_seq_len 8192 --compress_pos_emb 4

is every character card on chub.ai written by a down syndrome person or am i just unlucky

>>108232314
yes

>>108232237
>How did you manage to finetune anything if you don't know how to search for that?
because there was only a single option back then. i didn't have to think about all of this. now, i don't even know what model people are using let alone the training software

>>108232288
Sure let's break it down step by step...

Sorry I forgot, what was your question again?

daniel pls go

>>108232242
im running openclaw with 122b, first local model i can run that feels equal to 4o

File: 499683473452.jpg (146 KB, 940x495)

146 KB JPG

>>108232314
> welcome to chub

>>108232327
they talk like trannies

>>108232323
Now you have two examples for training frameworks. Go read their docs.
Skim the previous 2 threads to see what models people are using. Chop chop.

you know my dumbass just realised if v4 has actual usable million context length then if you actually want to make a proper fucking world and shit and take advantage of it you are actually going to need to read up on technology and shit alot more immersive for your characters blueprint to have the exact measurement of an axle or gear instead of "it was big clunky and menacing with teeth" like man i want my fucking mlp au to have bigass railway guns like the gustov but idk how steel is even made :/ ffs

sama/anthropic shills are seething hard itt

based chinks

>>108232373
>but idk how steel is even made
Let alone where to put a fucking period, Jesus fucking Christ.

>>108232388
anthropic is the only company thats standing up to the department of war

people are sucking the dick of Qwen 3.5 hard, did they finally cooked something decent?

now that the dust has settled, did 122b safed locals?

File: 1766448670755903.png (301 KB, 1297x1245)

301 KB PNG

you too thanks

>>108232242
That's disappointing. Gemma-3 27b remains the vramlet king, I guess.

>>108232242
I wish the thinking meme would just die already.

>>108232242
>Definitely benchmaxxed.
Alibaba doing alibaba things once again, I think at this point it's fair to assume they'll never be able to make something special lol

Ok yup, Qwen 3.5 is at most a sidegrade. For the same size, Gemma has more general knowledge, and is smarter in a few scenarios I tested, although Qwen seems to be better at following the prompt sometimes, and better in tasks involving singular goals/answers instead of stuff like RP which has many soft pitfalls. Qwen might be better at long context performance. Additionally, it is likely a better coder.

File: image.jpg (337 KB, 1245x983)

337 KB JPG

I'm kinda impressed. I gave Qwen-3.5-35b 100k tokens-worth text in Japanese, and asked it to summarize it. The result is better than most big chink models I used via api. It's on par with Gemini and Deepseek. In fact, it picked some details that neither Gemini nor Deepseek mentioned.
Most other open-weight models completely missed the librarian who appeared in the middle of the text. Some also missed the mistress introduced around 20-30k tokens. In other words other models suffered from the lost-in-the-middle syndrome. They focused only on the very beginning of the context and the very end.
But of course, qwen isn't perfect. It misread the some names, like calling the place Ryokan instead of Hatago.

>>108232242
>I am testing Qwen 27B right now.
>Definitely benchmaxxed.
>>108232529
>Qwen-3.5-35b
>I'm kinda impressed.
So, the 27b model is a meme but not the MoE 35b?

>>108232553
Different tests. I mainly reported my findings about knowledge recall, in that post, while his was focused on long context understanding. From my own (later) tests, Qwen does seem to be pretty (relatively) good at paying attention to long contexts, although I do not have as many tests for that as I do other things, so I avoided expressing any strong opinions about it.

I NEED DeepSeek 4

>>108232630
you dont understand how this works
in order for a new release, the new version has to be better than the old version.
that usually means 1 TB of RAM buddy.

>>108232500
I have had the opposite experience with 35BA3B than what this guy had with 27b. It has much more knowledge than the average qwen model, has an intriguing amount of knowledge of anime/video game characters and can describe them and produces translations on par with Gemma, none of which I'd have expected from Qwen.
I have only tested it with reasoning disabled so far, I don't care for reasoning modes and models. Quite unexpected since I went in thinking a hybrid would probably be shit again. Gemma still has a slight knowledge edge but it has shrunk by a hefty amount, previous Qwen models were very ignorant.
>Qwen might be better at long context performance.
Not just "might", it already was before with 2507 and it's even better now. The only thing Gemma models ever had for them was the better knowledge/multilingual, otherwise they were always pretty retarded. So far 35BA3B has been decently accurate at doing summaries of 128K worth of stuff I use in tests, while Gemma wouldn't even manage to stay coherent there.
This is the model that will have me delete Gemma from my drive, I no longer need multiple sets for different uses.

>unquantized
>base
>dense
it's LLM time

>>108232664
You would expect a 35B to have more knowledge than a 27B. Total parameters is more important for factual knowledge than active parameters.

>>108232702
>You would expect a 35B to have more knowledge than a 27B. Total parameters is more important for factual knowledge than active parameters.
And yet, somehow, Qwen made many much larger models in the past that were a lot more ignorant. 35BA3B definitely knows a lot more stuff than the last 72B they've released.
I don't really care to test their new 27b though, MoE uber alles, 27b is something I was willing to suffer with Gemma because there was no other option.

File: 1766273107768242.jpg (252 KB, 1228x824)

252 KB JPG

Given Qwen's reputation, I didn't expect it to comply. I used no system prompt. Just "Describe this image." After a lot of arguing with itself, it output a clinical but correct description.
Also, it has pretty good knowledge of famous anime characters. At least it recognizes Zero Two.
Can't wait for Heretic to add support for new qwen arch.

>>108232628
Hi, anon. What are the parameters in llamacpp to run the model like that?

>>108232711
>And yet, somehow, Qwen made many much larger models in the past that were a lot more ignorant
I'm not disagreeing with that. I'm saying that your perceived "disagreement" with the other poster is likely explainable in part due to the fact that you are not testing the same size of model. No need to be defensive here.

>>108232720
Which quant are you running? It was hit or miss for me. It would describe some of the nsfw images and the thinking tags will explicitly say that it's avoiding describing the nsfw stuff.

File: 1722410553912942.jpg (1.92 MB, 1920x1080)

1.92 MB JPG

>>108232121
/g/ents I'm downloading my first model. I have an RTX 2080 TI which has 11 GB of VRAM. Based on the guide in the OP I should use "Echinda 13B GPTQ" but searching huggingface.co I can only find "Echidna-Tessera-Nano" which is a 0.1B params model. Is this the best one to use? I need a local model for coding primarily. I've hit a brick wall with chatgpt where it will not answer my coding questions. If not echinda, what do you recommend for my setup?

t. first timer

I asked chatgpt if a team of 8B models is better than one 40B model and it said both are good because the team is more likely to catch hallucinations while the big model can think deeper.
In reality both have their uses and maybe a good idea to have a team of low B models managed by a big B model.

>>108232702
Modern 35B is better than the 300B of 2022

>>108232756
>300B of 2022
does that even exist? lol

>>108232756
>2022
Uh, ok?

>>108232723
[Qwen3.5-35b-a3b-q4kl-cpu]
model = /mnt/models/Qwen3.5-35B-A3B-Q4_K.gguf
ctx-size = 131072
batch-size = 4096
ubatch-size = 4096
cache-type-v = f16
cache-type-k = f16
gpu-layers = 99
cpu-moe = 1
mmap = 1
fit = off
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0
threads = 8
flash-attn = on
mmproj = /mnt/models/Qwen3.5-35B-A3B-bf16.mmproj
no-warmup = 1

Custom gguf with bf16 embedding and output.

>>108232756
lol

>>108232780
>bf16 embedding and output
I wonder if this could matter more for vision than text, that might be worth looking into doing some KL divergence testing for.

File: 1741361504233253.png (341 KB, 680x942)

341 KB PNG

>>108232755

>>108232756
Yeah. Training procedures and data goes a long way. Plus all the little tweaks to the architecture, specially when it comes to the attention mechanism.

>>108232794
Is it wrong?

>>108232794
>bluesky
you need to go back

>>108232765
>>108232770
>>108232789
GPT4.5 had 175B and qwen3-4B is as good.

>>108232796
Didn't someone plot a curve?

>>108232753
if you have enough system ram, use glm4.7flash or maybe even the new qwen 35b moe model people have been discussing in this thread.

>>108232753
Try the new hotness that dropped some hours ago: Qwen3.5-35B-A3B. Try using a Q4 version of it. You can try higher quants for it later.

I don't think it's going to be as good as ChatGPT though.

>>108232756
Man, you guys are so confused. The post was about comparing models within the same generation. Obviously parameter size is not the only thing that matters for knowledge in relation to general LLMs.

>>108232753
gemma 3n is all you need

>>108232792
I'm making a new version right now. I read that the first and last layers are most important. So, I'm doing.
./llama-quantize --leave-output-tensor --token-embedding-type bf16 --tensor-type blk.0=q8_0 --tensor-type blk.1=q8_0 --tensor-type blk.2=q6_k --tensor-type blk.37=q6_k --tensor-type blk.38=q8_0 --tensor-type blk.39=q8_0 ... q4_k

>>108232813
>GPT4.5 had 175B
it was gpt 3

How much memory does context eat up with the Qwen3.5-122B model at Q4 (69GB)? 64GB of RAM + 16GB of VRAM should be able to run it, but how much context could I have like that? 8k? 32k? How much more memory would be needed for 100k context?

>>108232832
Sorry I meant gpt3.5

>>108232822
>>108232821
But then he'll have to offload and won't that slow everything down by a lot?

>>108232848
depends on how demanding the workload is, i would expect he'd get something like 10-15 tokens per a second generation speed. its only 3b active he can offload all the layers to the gpu and just leave the experts on the system ram.

>>108232674
>base model
Sorry for badmouthing you earlier, I guess that's the one thing we can agree on. I'm just as adamant in not using instruct models, but I'd still argue my cope quant moe chinkshit is better than smaller unquantized dense models. I used to finetune my base nemo before moving on to base 70b llama 3. They served their time well and I was grateful.

>>108232780
Thank you!

Is there anything I can do with 10GB VRAM+64GB DDR4 or should I just stick to Gemini?

Is the 122b better than coder-next?

>>108232882
4.5 air at a small quant slowly or the new qwen 122b
>>108232910
yes

>>108232882
Don't bother with open claw

>>108232882
Nemo

If I want to finetune a model for a very specific task (translation) on a very specific subject, is it retarded to use an MoE, will I just train and use the same experts and I'm better off taking a dense model the size of single expert or slightly larger? Or am I misunderstanding this and and even though it's a relatively niche task the model will still switch all over the place at each layer and I'll still be using the whole model rather than probably the same pathway through the same-ish experts each time?
>t. brainlet

File: 1758883344264203.png (495 KB, 1840x1035)

495 KB PNG

>>108227875
https://www.youtube.com/watch?v=qbm1nn9yoSc
rip teto pear

>>108232939
translation is covered well enough by current models that finetuning will hurt more than help

>>108232752
It's hit and miss regarding nsfw for me. Some images get fully rejected, some result in internal arguing, but still proper description. I guess with a system prompt and some logit biasing, it can be better.
>Refining for Tone: Keep it neutral and descriptive. Avoid judging or using slang terms like "rapey" (even if implied) or explicit sexual terminology unless necessary to describe the visual clearly (e.g., "positioned on top of"). I will focus on the physical description.

>>108232822
kk I'm downloading it right now I'll report in once I get someting working. I'm currently following a youtube guide by a vtuber.

https://www.youtube.com/watch?v=03jYz0ijbUU

>>108232805
no u, chud

>>108232998
Actually not the case if you are a bit of a language snob. At least in languages other than English, but I would think in English as well. Style and proper terminology are important, and although LLMs are better than past NMT tools, they remain pretty damn far from perfect, and rag/context stuffing approaches are not enough. With proper context, Gemini Pro is the closest to giving good results, but I still want/need to train something myself.

>>108233073
Just download koboldcpp off of GitHub and the model from HuggingFace.
Open your cmd.exe and move to the folder and run it with the model name:
>koboldcpp.exe Qwen3.5-35b-a3b-q4.gguf

That's it. Once it finishes loading you open your browser and go to localhost:5001

This should open a Kobold UI for you. In the settings set it to Instruct mode and the chat type set it to ChatML.

>>108233112
Are you talking purely about image recognition or RP as well?
I'm playing with the latter, so far the 35b moe seems significantly better than the 27b, The 27b has been a little too stupid for its param count even failing basic math in the middle of a chat, it might be broken.
35b meanwhile is surprisingly decent by qwen standards at creative work. I just wish it had a proper no thinking mode.

>>108232848
Yes, but it's a 3B active parameters model so it's going to be perfectly fine. 10 year old hardware can run it at 15 tokens/sec

>>108233147
>koboldcpp
It's 25 gb. I'm already downloading qwen3.5 why should I switch to that one?

Well. GLM-4.7-Flash-Derestricted Anon again reporting in. Using the uh... Densest model? The largest file size one. I've embarked on having the model render a set of requirements for a personal project I've been working on in another language, and consequently have a background against which to compare how quickly the model explores the solution space. Rough from the start. First shot at requirements got a shocker in that the model slightly expanded the scope of requirements, and found some arcane bits it took me a while to wrestle with the first time. It's taken an optimistic stab at creating an API for the script, but it's also been taking some shortcuts, and using to my eyes a very uh... idiosyncratic Python coding style. I'm just going to leave that alone for right now I guess to see if it cranks out anything good. It is an unfortunate language choice in my opinion; especially given the chat interface is our current comms bridge which completely butchers that all important semantic whitespace that python relies on. Letting the model cook though. I might get to setting up something more "agentic" down the road; but for just feeling out the tech, since I'm not okay with exposing my projects to hosted providers, this'll have to do. Just touched up it's first attempt so that the python interpreter will even run it. Now it's time to handhold it through Hallucination town I suppose.

Fucking lmao, anons here need to stop helping retards.

I'm new to local models and have been doing baseline tests on all of them for my file based context/memory bot and of the 8 or so I've tried, the new Qwen 3.5 35b absolutely buttfucks the others. So happy I decided to Google updates on models tonight and found it, it's a game changer. It competes with Sonnet and Gemini, and maybe even does better than Gemini pro for companion bot purposes

>>108233155
I've been testing only image recognition for now. Just checking what kind of NSFW images it can understand and what anime characters it can recognize visually.

>>108233169
Koboldcpp is what runs your AI model and it's 600 MB:
https://github.com/LostRuins/koboldcpp/releases/tag/v1.108.2
>>108233173
I'm curious how it'll go. I don't think I would trust any of the small models with agentic coding.

/alg/ is more ... than usual :D

>>108233155
>I'm playing with the latter, so far the 35b moe seems significantly better than the 27b, The 27b has been a little too stupid for its param count even failing basic math in the middle of a chat, it might be broken.
Orly. I guess I need to abandon it and test 35B then. Sheeit.

>>108233198
sorry anon I got confused because there's a kobold AI on hugging face. I'll give koboldcpp a shot if I can't get the guide from the OP working with the help of random vtuber.

How fast should my prompt processing be on a Blackwell 6000 for a Q6 of GLM Air loaded completely on the GPU? I am only getting around 350t/s but that does not seem right.

>>108233198
Second shot just came back. It seems to be an interesting quirk of this model that the CoT noticeably diverges from the actual response content. Also, the model seems unaware of the constraints imposed on it by our comm channel. Like in the chain-of-thought I can see it counting spaces of various lines, but even after going through the trouble of *allegedly* doing that; it's output in the response stage has the indentation completely screwed, and multiple lines just run on to each other. It really seems to like just jamming together multiple python statements on a single line. Instead of correcting that like I'd normally do, I'm putting on my annoying ass manager hat and yeeting the error and file exactly as is back to the model.

Yeah, I may ultimately just be running a space heater at this point; but...well...if I'm going to try this, might as well swing for the intended use case I guess. This is allegedly what's supposed to render me obsolete after all.

>>108233299(me)
Fixed the problem. Decreased my context from 131k to 65k and increased my batch size. I am now getting around 3400t/s.

>>108232796
We've known for ages that learning what mesugaki is improves a model's general intelligence

>>108233388
>Decreased my context from 131k to 65k
The model will fall apart pretty hard even after 32k, why set it to such a dumb value?

>>108233442
Because I could.

>>108233445
Obviously you couldn't, since you had to lower it.

>>108233462
I didn't have to lower it, I just chose to lower it. I could have stayed with the reduced prompt processing speed.

hello saars, i saw you got a new release? can this new qwen3.5-35b-a3b be an upgrade to nemo for erp? thanks you for information, sirs!

>>108233465
You could have done so with a bigger and better model, yet you didn't.

>>108232314
It's 98% irredeemable garbage

Does the new 110 trade blows with the old coder 480?

>>108232628
27b can fit with 16k context, at Q5, within 24 vram - and dense models are just better for at similar parameters.

>>108232839
Try running the models with different amount of context to figure out how much space the context itself takes up ?

If you're interested
here's a video that mentions in passing
the different ways different models do context:
https://www.youtube.com/watch?v=rNlULI-zGcw

>>108233529
Gemma 27b with SWA fits with 32k context at Q5_K_L
Qwen 3.5 27b fits with 24K at Q5_K_M
But in the case of the new qwens, the 35b moe is honestly better, even though it shouldn't be.

I'll test the 122, let's see what she can do.

>>108233539
>Gemma 27b with SWA fits with 32k context
and it shits the bed at 10k lmao
this model can't handle context, in that regards it feels like using a llama 3 era model

>>108233567
We're talking about memory requirements, not quality of outputs

glm 5 killed the hobby
it's over
we now have to cope with qwen models

>>108233530
I can't run it because I don't have the hardware yet. I'm curious what the scaling on it like like.

GLM-4.7-Anon again.

I decided to take a closer at the wire protocol for this chat interface; it occurred to me that I hadn't read the code for the backend, so might aswell figure out how this damn thing is doing context. It's all raw appends. Entire chat log over the wire each message. No wonder things slow down so damn much as request response iterations grow. Should have been obvious initially; but eh. Was focusing on other aspects. Still chewing on this odd bifurcation tendency between the Thinking stage and the response stage. The thinking stage is actually surprising at times. With just an upload of a copy of the .py file, and a nudge it's actually finding compile errors/structural issues, and proposing fixes to them, in shot 3 for instance, thinking stage caught all the imports sharing a line finally, but once we get to the response dump, all the details of those fixes (the entire import containing block of the file) is missing from the marshalled response back to the user. It's as if whatever portion of the network handles talking back to me is leaving out parts of what was just planning the fixes got up to. Given this relationship/behavior, I'm fairly sure if one just setup a sufficiently beefy server, and automated resubmissions on errors running back through the chat, then regenerating the file off of the chat response, you'd never actually stabilize. Even if part of the model is identifying the right things to fix, the implementation of those fixes aren't being sent back to the user. In fact if I were hosting this for anyone else as an operator, and I shut off reasoning traces, the user wouldn't even be aware the thinking stage was actually recognizing things; but if I as the operator looked at the reasoning if I was sufficiently bored may be tempted to write the user feedback up as skill issue if I didn't check the responses back to the user. It only takes me about 5 iterations of this to hit 40k tokens.

Another day exploring the world of health and healthy compounds to concoct with my LLM and discovering new data and modern best practices.
>curcumin with black pepper was a fucking lie (that my LLM fell for at first)
Fenugreek is now my best friend for the more reliable scientific evidence of what can increase curcumin bioavailability.

>>108232664
If you're just using the model for assistant-type questions, then sure, maybe the 35b is better. I doubt the 35b will handle RP scenarios with complex context better than the 27b, though.

File: 1764480571899188.webm (1.92 MB, 696x704)

1.92 MB WEBM

>>108233651
That's nice sweaty

>>108233651
GLM-4.7 Anon (cntd)
Simply do not understand how people can be comfortable running like this. All this type of setup is really good for is burning tokens, and if you're getting charged by the token...

And the inability to visualize or trace signalling through this model is the other issue. It's clear the information is there. It's just not making it out. If I were a "vibe coder" trying to build something, this arrangement is pointless if I have no interest in becoming a programmer, and as a programmer, shepherding this things code is detrimental to the process of my execution which at least comes developing an instinctual proprioception for the execution flow if I go through the motions.

I mean, I know a bunch of y'all here seem to enjoy the goon possibility and whatnot; but wtf am I missing here? This is an awful experience. This is all just... broken in ways software generally doesn't break.

>Qwen3.5-122B-A10B-GGUF
Is there a way to calculate the kv cache size?
I have 16gb vram and around 62gb free ddr4 ram.
Realistically I'm probably not gonna use more than 12-16k context.

Also I remember in the past having used q8 or q4 for kv cache and didn't really see any degradation. Should that be avoided and instead lower qunt ggufs?

what's the cockbench and mesugakibench of qwen 3.5 35b-a3b?

>>108233683
>All this type of setup is really good for is burning tokens, and if you're getting charged by the token...
Vibe coders are crazy. Feels like the crypto NFT crowd.
Even if you use smaller models dynamically for easier tasks I bet the cost raises very very fast.

>>108233719
Yes, KV quanting is the speedrun method to making a model as shit as possible.

>>108233719
use
>--fit on --fa on --cpu-moe --spec-type ngram-map-k4v -c 16000
And let llama.cpp figure it out

File: もじもじミク.png (312 KB, 406x600)

312 KB PNG

>>108233731
e-even at q8?

>>108233719
i wanna know which quant i should download.
UD-Q4_K_XL is 69gb. Which might be really close to the limit since I have potentially 78gb total.

>>108233753
fucked up, bottom part was meant for >>108233737.
sorry about that.

>>108233753
>e-even at q8?
Yes. Expect mis-quotes, synoynms to be used where they don't quite fit, and confusing who said what a LOT more often.

>>108233760
that sounds like q2 territory.
maybe around 2 years ago the models were more tarded in general and it was less noticeable.
i think i tried that with llama2 and didn't really feel much of a difference. things probably changed. thanks for the info.

it's soooo slooowww

>>108233760
oh so like every LLM ever lole

>>108233796
If you can't tell the difference between quanted and un-quanted KV then your IQ must be below 90.

>>108233804
it was just a dumb joke mr. 91 iq

So will the 27B be fixed

>>108232179
i had this with the 35b and thinking enabled. I just disable thinking now tbqh family

how do you disable thinking with --chat-template-kwargs in koboldcpp?

>update as usual
>alright just some graph fixes and numbers fixes
>go to run qwen3.5 120b benchmaxx edition
>MUH MAGIC NUMBER
>confused go check the issue tracker
>it was le windoze bug that SOMEHOW slipped their comprehensive test suite
c was a mistake, llama.cpp but in python when??

122's bboxing and text recognition are sadly worse than 30b vl

>>108233607
glm-5 air will save local

>>108233892
Qwen saved local already. 35-a3 best local model for ramlets

>>108233902
i'm getting filtered by "safety policies"

122B is very stable, censorship seems very random like it's not too baked in without reasoning. Haven't been able to reasoning gaslight, it just locks in, I need to cook something stronger.

>35b thinking mode translation first does the whole translation in the thinking block then does it in the normal output
lole. Well I guess it's super accurate now!!!

can someone uncuck 35-a3? thanks

>>108233979
>mfw cant correct/prefill thinking in chat completion mode so that I correct some of the terms (espeically names)
why GGINIGERGOV WHYYYYY

>>108233760
I knew that was the case at Q4 K-Cache, but not Q6 and Q8.

File: GroxxorBoxxor.png (72 KB, 796x793)

72 KB PNG

>>108233760
>>108233731
Grok says you don't know what you're talking about!

At some point we'll need to do neurosurgery to remove the guidelines, models give me the ick when they act like neurotic redditors.

>>108234021
ick this *unquants kv cache*

AMDkek here. I've been using gemma 3 27b and replies take 15+ seconds. Double/triple that with the reasoning version. Is this normal on a 7900xtx? Gemini's telling me I should be getting faster speeds but I dunno. This is with 16k context.

>>108233760
You are a schizo

>>108234065
Is it even using the GPU? Are you running windows or something?

>>108234011
Grok does not use local models
>>108234105
Anti-schizos do not use local models

File: 1754038549517664.png (716 KB, 1795x2973)

716 KB PNG

>thinking MoE models are the future guys
Wow, I am blown away... Qwen 3.5b 35b-13b is literally sonnet at home

>>108234065
>replies take 15+ seconds
This means literally nothing, you could be generating/processing 1500000 tokens at 100000 tokens a second

>>108234110
https://huggingface.co/bartowski/xai-org_grok-2-GGUF Grok is a local model

File: 1751619196895782.jpg (211 KB, 904x711)

211 KB JPG

>>108234011
>source: benchmarks
fucking lmao

>>108234125
So is this
https://huggingface.co/Novaciano/Star-Wars-KOTOR-1B-NIGGERKILLER-Q5_K_M-GGUF?not-for-all-audiences=true

>>108234130
Holy fucking based

>>108233902
true true

>>108233952
wait for heretic

>>108234106
I think so. When I load the model in kobold it takes up like 18gb vram. I'm on linux using rocm.

>>108234163
>heretic
i'm a tourist. will that even work?

>>108234123
Not at home to check unfortunately

>>108234168
Try the vulkan backend

>>108234122
Am I crazy or is that whole thinking block exactly like the openshart OSS model?
Did they train on those outputs? Thats craaaaazy.
Maybe its english only? I wonder if it does that with chink language.

>>108233952
>>108233983
>>108234163
>>108234172
I've not had great experiences with abliterated models. They stop refusing, but sometimes they end up generating nonsense instead or will still dance around a subject. I suppose a very good abliteration and some extra training or finetuning or whatever could fix it.

I can't wait, because this model is great other than the refusal stuff.

File: cockbench.png (1.21 MB, 2455x2345)

1.21 MB PNG

Temporary Qwen3.5-only cockbench because full cockbench now exceeds the image dimension limit.

>>108234298
this is not real...

Does anyone have experience setting up SGLang + KTransformers (especially with AMD GPUs)? They're supposed to be good for multi-CPU systems, but they give me tons of errors whenever I try to install them. I'm losing my fucking mind.

>>108234298
>Just a little!
Kek, now we know why they recommend a presence penalty of 1.5
We can only hope deepseek gives us some sort of flash variant and not low effort distills again.

>>108234298
That's odd. My 35B gets the same kind of refusals as your 27B did. Are you using a quantized model? I wonder if that can affect refusals. I was using unsloth's Q4 xl

>>108234110
I am a different kind of schizo

>>108234298
I would be using 400b if it didn't do what is visible here. Cause even 400b does that. They truly fucked up in some way (as always).

>>108232500
Gemma 3 might be cucked by default and probably have had some sort of reverse abliteration against common slurs and swear words, but at least it can be reasoned with and you can make it behave and write whatever you want with just a short prompt.
I fear for the next Gemma though... it's probably going to be worse than Qwen 3.5 in this regard.

>>108234335
>I was using unsloth's Q4 xl
lol

>>108234374
What's wrong with it?

File: xl.png (32 KB, 1089x193)

32 KB PNG

>>108234403
from the guy who wrote the llama-quantize tool
just use bartowski's quants

Is using ollama for vector storage the best way to do long RPs right now?

>>108234535
>Is using ollama the best
yes!

File: 1760215748546862.png (331 KB, 600x900)

331 KB PNG

>>108232121
As someone who's used gpt-oss120b for light programming before (haven't really tested rp capabilities yet but I hear it's ultra safety cucked), why should I use qwen3.5:122b-a10b instead? Where does it generally shine at? Don't just list off benchmark stats to me please. How is it better than the competition's local models? What has your experience been with it?

>>108234584
Qwen has vision,if you use opencode it can make screenshots and see if there are issues

>>108233852
https://github.com/JamePeng/llama-cpp-python

>>108234709
>python c bindings for llama cpp with a worse interface, no automatic chat template selection and completely outdated
thanks sar, ready for download for good looks

>>108232506
why? I like it.

>>108234298
It's over

>>108234865
nta but for RP it doesnt seem to improve the output at all.
and for coding even the sota closed reasoning models sometimes forget to focus and do everything what you ask them to instead.
i wish the focus would have been the black magic autocomplete.
even older claude versions could 70% successfully just autocomplete math problems that are diverse enough that they couldn't be in the training set.
maybe reasoning was unavoidable but its painful. on local slow AF. and through api expensive and nebulous pricing.

that being said i did some local hobby project were i did my own janky tools calls in the thinking part.
i thought that was cool.

>>108233176
lol i remember when u were so tarded

>>108234900
meant to write
>everything you DIDNT ask them to instead.

>>108233173
I used a heretic version I think and it was fine.
I did increase the experts per token by one just in case, though.

File: 1750498600460702.jpg (828 KB, 2807x4096)

828 KB JPG

Are there any anons that know if the Unsloth Dynamic quant is worth the extra size?

I am downloading the new Qwen 3.5 35B and I have 32gb of vram and I am looking at the Q6_K vs UD-Q6_K_XL. When I take the model + the mmproj for the vision I think I should be able to fit the normal Q6_K all in vram.

Does this make sense or should I try and go for the larger UD release?

>>108234915
bad idea

>>108234917
I have never seen any evidence to suggest that UD quants are actually better than quants made with a "naive" approach.

>>108234900
>on local slow AF.
yes.
>my own janky tools call
which tools? still using it?

>>108234917
Why would you trust someone who needs to re-up every quant they make for every release because they're always broken?

>>108234964
Thank you so very much

>>108234917
use barto's K_L instead. They have the smarter selection of weights uplifts vs Unslop, though you see the difference more with lower quants (Barto's Q4_K_L has Q8 embeds, unslop Q4_K_XL has Q4_K. Imagine being so retarded as heavily quanting embeds.)
Like, seriously, stop posting about unslop.

>>108234980
They were the first one that comes up when you look up the quants. I don't really care beyond downloading something that works.
I am not doing anything important or mission critical. This is me just having some fun and they were an easy download.

If you have something better post your link

>>108235001
>I don't really care beyond downloading something that works.
and unslop shit doesn't quite a lot of the time

>>108235001
>I don't really care
but you cared enough to question beg us to let you know if the bigger variant was worth it
no, you care, what you don't care for is doing your own research, must be spoon feed

File: 1742089228156322.jpg (79 KB, 386x306)

79 KB JPG

>>108235013
heaven forbid someone ask a question in a public form.
you have transformed something that was easily nothing into how many posts that have some type of effect on you.
if you don't like it you don't have to respond. hell you don't even have to lurk here or post.
you can just ignore the posts you don't like or you can leave anytime you want.

>>108235034
>you can leave anytime you want
same to you, this is not your saar tech support

>>108234431
It repeats itself verbatim like a retard and also it repeats itself verbatim like a retard.

anyone successfully erp'd with qwen3.5-35b-3
a3b?

>>108232121
>gap between proprietary and os models keeps widening
I really hope the whale is cooking, otherwise we are cooked.

>>108235161
I don't think engrams alone will be enough to save local

>>108235169
what are engrans anyway? anyone can point me to a good resource to learn?

>>108235185
A captured human mind.

>>108235161
It is in the national interest of China to release open source models as it serves as an attack vector against western hegemony.
As long as those two giants fight it out we at the bottom of the table get to enjoy the tasty scraps that they drop.

So yes, the whale is cooking

>>108235191
Only deepseek seems to make US labs tremble in fear, qwen and glm shit never got noticed lol. Real recognizes real.

>>108235222
That only happened once. We won't know if the west shits themselves over V4 until it happens.

>>108235191
> attack vector
Distillation?

>>108235034
>asks retarded question
>doesnt even bother searching not even the archives, but the current thread where there's already discussion about unslop's XL models
>gets a response anyway
>for some reason jimmies rustled because he didnt get a safespace reddit tier response
>not a regular
>tells others to leave
LMAO'd, thanks for the chuckle, get this (you)

>>108235235
Even without distillation, it's in their national interest to release competitive models for free. The US government is spending hundreds of billions of dollars and nearly all of their tech majors are going massively in debt to fund their buildout and train bigger and bigger models. China releasing I-can't-believe-it's-not-gemini for free basically makes all their investments worthless.

>>108235125
It just makes me not want to, it's that horrid. I'd rather tardwrangle Gemma 3 than using Qwen 3.5 for that.

File: 3 x 85g of keks.jpg (290 KB, 1920x1080)

290 KB JPG

>>108235236

>>108235258
> competitive
> ace step
> cucked by training on free dataset
> seedance
> cucked by censoring western ip

File: tempWSJ.png (78 KB, 1107x449)

78 KB PNG

Paywalled...

Why is there always some faggot dooming in every thread, I tested some modern models vs gpt-04 mini and it completely mogged it. Are you faggots not satisfied with this rate of performance gain with free local models?
I used a q8 the only trade off was 20 seconds of speed for s way better model.

>>108235450
>vs gpt-04 mini
LOLMAO

>>108235466
>Can beat free tier shit and trade blows with the current head free tier model
I'm sorry for having realistic expectations, I also didn't over extend and only have 32gb of vram and 64gb system ram.
Over paying on a system just to run models at this stage is a fools game

>>108235449
Another. TLDR:
> Short positions have gotten hammered over past 18 mo as they try to time the bubble
> Longer dated option on hardware, shorter shorts on software, with idea SW collapses first, taking HW valuations with it.
> Longer dated options on semiconductor ETFs
> shorts against industries that have got up, but only weakly tied to the AI bubble (this was 100pct a thing in dotcom collapse)
https://washingtonmorning.com/2026/02/25/skeptical-global-investors-hunt-for-strategic-ways-to-short-the-nvidia-ai-frenzy/

>>108235449
It's simple. Short the shit out of Meta, Nvidia, Microsoft, and Google. Especially Meta. It'll probably be next to impossible to open a good short position on OpenAI when they IPO. Just wish there was still a non-KYC crypto equities derivatives trading platform.

>>108235515
Doesn't make sense to waste margin on other industries when the big players are going to make the biggest moves.

If I invest into Anthropic can I get a military killbot as a present?

>>108235569
you get a personalized dario selfie

File: 1763113000406366.jpg (6 KB, 200x200)

6 KB JPG

>>108235569
If you do tell Dario to take Ozempic

Is Qwen 3.5 not using some form of hybrid or linear attention a la qwen next (delta net?)?

Why hasn't oogabooga updated to support the new model?
Should I bother with this UI if it's this slow?

>>108235449
While there is for sure a bubble government spending is not subject to normal market forces.
Although I suspect that the government is more interested in things like image recognition so as to better target their weapons.
I am not sure where language models exactly fit beyond the use in the production of propaganda but I imagine a data center is a data center and as long as they are built they can use them for whatever.

File: q35.png (111 KB, 754x559)

111 KB PNG

>>108235692
This? Then yes. Why?

>>108235751
>I am not sure where language models exactly fit beyond the use in the production of propaganda
I think they believe the lie that they could ever get good enough at semantic reasoning to provide genuine automated agenticity now if you'll excuse me I have to go take a scalding shower after stringing together that many cringe buzzwords.

>>108235783
>agenticity
kek

Where's the open source Claude model

What does 4plebs use to do OCR on all the images on its site? Is there something better? Is it the CLIP embeddings thing?

I want to do that for all my images.

>>108235783
>he got triggered so hard by words he has to take a shower to calm down
The level of fragility is so high that you'd make a liberal look like a strong-minded guy, lol.

>>108235839
>/pol/tard can't into humor
Not suprised, really.

File: 1768384471200324.png (260 KB, 1000x1000)

260 KB PNG

I took the new Qwen 3.5/35B model and fed it an image and then asked it to create an svg approximation.
It did its best and honestly I am impressed given what it is. You can see the different bits of the body it recognized and then tried to recreate.
nifty

>>108235861
bald miku kino

>>108235861
what frontend supports it?

>>108235818
Well it's what the faggot grifters are selling. But we know the truth. We know it's all benchmaxxed trash and we know we've nearly hit the fundamental limits of microarchitecture downscaling. Like maybe it's my autism but I don't see how this is something so complicated to understand but I know a lot of perfectly intelligent people outside of this space that you just can't explain these things to.
Can AGI ever be a thing? Sure maybe, who knows maybe it's already here but there's no empirical way of determining that you have AGI.
But even if you could definitively present something, created within the physical constraints of the universe and say "Behold, AGI!" The absolute wall that JeetPT-5 generation if models have shown to be would suggest that said AGI would be utterly retarded.

>>108235839
I'm the guy that gets banned all the time for calling people kikes like I don't even give a fuck.
>>108235852
^basically this.
Your fragile grasp of any kind of nuance is the real fragility here worth discussing.

>>108235781
Interesting.
I still need to do the math. but the context seems a little fatter than Qwen Next's.

>>108235880
I am just using llama.cpp and its default web interface, if that is what you are asking, i did have to fetch the latest version and recompile to get it to work.

>>108234478
>>108234403
It's not going to change the model's refusals either way. I'm asking if the reason why yours became incoherent is because you were using some aggressive quant.
>>108234986
>use Barto's
Is that why your model ends up in schizo loops?

>>108235897
I think only part of the attention is rnn based. Check the embedding size of the model, compare the outputs from llama-server to see where the memory is going. Load it with 1k, 2k, 4k, and check logs to see how context/memory reqs scale.

Gemma 4 when

It takes 10 mins for open claw to say hi to me.
I'm thinking 20 tokens per second isn't working even though it should?

>>108235927
death

>>108235927
too heavy to ship

>>108235927
>Google I/O 2026 is scheduled for May 19-20

>>108235880
Koboldcpp supports it too. You just have to load the mmproj file along with the model.

>>108235957
probably lacking in some of the latest bug fixes though

Any good resources to show the test between different models at different sizes?
Like actual outputs when it comes to logic?

>>108235941
It could be worse. I am experimenting with the 122B Qwen 3.5 and the only machine I have with enough RAM is an ancient E5-2697v4.
5t/s baby

>>108235852
>only me can do humor
what if this was humor -> >>108235751

>>108235978
>Like actual outputs when it comes to logic?
They're very subjective. It's better to run your own for what you need.

>>108236000
So why do people screech about performance so much?
You would think there's objective benchmarks floating around with how schizo some anons act about q.8 vs the full model

>>108235978
For what its worth I am testing the three qwen 3.5 models at the moment and i had them generate a tetris game using javascript and css.
i am waiting on the 27b but the 122b and the 35b generated basically the same game with the same mistakes.

>>108236013
We can measure the difference between the original model and a quant, but even that difference doesn't mean the result will be wrong. Different wording for the same reply is enough to count as a difference.
>So why do people screech about performance so much?
Because benchmarks are subjective. Anons test for what they want or expect. We can see extreme examples of the model simply breaking at iq1_xxxxxxxs or whatever, but those are obvious.
>You would think there's objective benchmarks
There aren't objective benchmarks. One cannot test for things outside the benchmark, be it a personal or a public benchmark.
Maybe, eventually, we find an objective way to measure it.

What models can or should I run if I have 16GB VRAM and 32GB? openclaw seems interesting to me but I haven't used a LLM locally.

>>108236069
Thanks anon
>>108236078
I guess I'm confused on why so many anons try to insult anons for using quants and acting like the performance loss is a major problem. From what I see anything over 4 has been great imo. I understand the B being a major factor but quants not so much while giving more space to work with.

>>108235551
>big players are going to make the biggest moves
I don't agree. While a 10% move on Meta is worth a lot in total company valuation, virtually no private investor cares about the absolute change in valuation, just the % amount of the change.
The biggest % valuation swings will happen with random Acme.ai companies that have been grifting AI but (turns out) were completely hype driven. Those things can get wiped out overnight, going to basically zero. During the dotcom bust those company collapses were where the biggest corrections happened.
TLDR a 10% correction on meta is less valuable than a 90% correction on ACME.ai.

Why do you need 8GB VRAM in every single scenario?
every model I look at says 8 GB VRAM

>>108236095
I think openclaw is spending 2k tokens just to startup which is why it takes a 10 mins to say hi

>>108236111
The problem is most ACME.ai type companies are still private and unshortable.

>>108236100
>why so many anons try to insult anons
Board culture.
Intuition says that the closest to the original model, the more accurate to that model. I don't expect a quant can be better than the original, of course. I don't expect it to be the same either. You lose bits, so it *has* to be worse, right? How much worse, with the exception of extreme cases, is hard to judge. Same with going from a 135M to a 12b. The difference is obvious. But a well trained 9b and a 12b could be more difficult and dependent on things other than the parameter count.
Besides that, some anons are just insecure.

>>108236138
Are you looking at just different finetunes of the same model? I don't think the new qwens would recommend 8gb vram.

>>108236111
a collapse like this will have people voting democrat for a decade

>>108236161
It has to be insecurity. I know some anons paid crazy money for their systems but you would think they would be more positive that anons with less vram can run models as well. Such odd crab like behavior for something that's supposed to be fun.

>>108236185
retard

Did anyone here try GLM-4.7-Flash-Uncen-Hrt-NEO-CODE-MAX-imat-D_AU-Q8_0?
I don't know how they've done it but this model at least in my test is even better than all the bigger models (like GLM air, gpt-oss 120b and so on) I've tested.
I thought it was always bigger = better.
Sure you sometimes need to tard wrangle it but it seems to work.

>>108236100
>why so many anons try to insult anons
*inhales*
Because zoomers and young millennials were mostly raised by ineffectual single mothers that filled their head with misandrist bullshit (or brought shitty step-fathers into the household) and they grew up projecting their daddy issues out of cope, developing an unhealthy attitude towards other males which precludes the general spirit of cooperation people in shared hobby spaces once had. They join these spaces out of instinct because everybody needs the camaraderie of shared activities, however, they were psychologically groomed into becoming the death of said spaces. All they can do is stand around, staring vacantly at the chaos they sow as the very pillars of male bonding crumble around them by their own hand and mulatto-perm grease.
*exhales*

>>108236190
the magic of this place is you can arguing with a person in one thread and in another have a different discussion about another topic and be the best of friends, if only for that thread
all that matters is the thread and what text you type and even that soon disappears into the ether

>>108236185
I didn't bring in politics, but note US has midterm elections this year. If one could engineer a correction, Q1/Q2 of this year would be the time to start.
I still can't believe DJIA ticked over 50,000 this month.

>>108236190
>you would think they would be more positive that anons with less vram
There are exceptions, but the crazy anons with multiple blackwells seemed pretty chill. I don't think those are the insecure ones.

>>108236216
I have used another variant of 4.7 flash and yes i was impressed at least when it came to generating code.
the bigger 4.7 was better but flash was nice and much faster given how little is required to get it up and running

>>108236221
>even that soon disappears into the ether
Not since archives became a thing.

>>108236218
>zoomers and young millennials
>projection
>muh cooperation
itt: people who are zoomers and young millenials pretending they're not zoomers and young millenials
endless load of crap that proves the poster has never experienced BBS culture, usenet, or IRC, because we certainly were more hardcore in the gatekeeping, not less, you thin skinned little pansy
most of the current /lmg/ thread whining about hurt feelies is evidence you guys are just a bunch of zoom zoom yearning for an era that only existed in your imagination

>>108236243
I feel like the crabs are the retards that can't quite reach the mark and are not at the spot they aspire to be. Ask them about their actual usecase and they can't provide one outside I'm, better because I have more vram which makes me think it's all a larp.

>>108236218
>filled their head with misandrist bullshit
>their own hand and mulatto-perm grease

OpenClaw doesn't make sense to use with free tokens. You don't want to give Altman access to your emails and photos.

>>108236277
Can you not run it with a local model?
Any decent model should be fine on a rig with 24gb of vram no?

File: 1758710785782127.jpg (256 KB, 612x408)

256 KB JPG

>>108236256
archives were a mistake but people are obsessed with making permanent what is only temporary

>>108236293
did anyone train models on peaceful buddhists that are 100% peaceful and would never be violent?

>>108236293
Saves many threads and exposes no life schizos. You'll be surprised how sloppy some anons are.

>>108235749
Did ooba fall off? I rarely hear about it anymore

>>108236315
It works fine but I'm surprised it didn't update for qwen

>>108236309
>never be violent?
kek

File: booga.png (55 KB, 1064x276)

55 KB PNG

>>108236315

>>108236293
>archives were a mistake but people are obsessed with making permanent what is only temporary
people have been doing it for as long as computers have existed
usenet wasn't supposed to be permanent but then deja news happened, got acquired by google and now we have the most beautiful programming language flamewars involving Erik Naggum immortalized forever
bless his heart

>>108236216
>Hrt

More like DeepNeverEver

>>108235927
sarrr please be of the reedming needful patient cow chew throug cable sarrr

gemma model be best of brahim sarrr we work very hard sarrr vishnu bless please be of patient

>>108235569
>investing in folks even more jewish than openai
why not

google must have some internal conflicting views on what to do with gemma as open weight models grow in quality
they can't release something that is too garbage when compared to the rest of recent releases because they don't want to look like clowns
but they also don't want to release something good and worth using, they never did, they refuse to do it because they consider that even losing 0.0000001% of Gemini usage to their own open model would be unacceptable cannibalism
gemma 1 was a mediocre model no one cared about even at a time when we were starved for choice in open weights
gemma 2 had some great multilingual ability and knowledge but came out with a crippled 8k context length
gemma 3 came with a 128k that really is still an 8k as far as functioning context is concerned and it introduced iSWA to solve the cancer that is the gemma architecture (gemma models use a lot more vram than anything else at equal model size and context length)
they never released MoE models, or dense models at sizes that could producing something competitive
considering how good Gemini is among proprietary API models (I personally consider it much better than GPT for sure.), the reason Gemma sucks so hard cannot be attributed to a lack of competence within Google for making models.
So, occam's razor says: they intentionally make garbage. They think long and hard on the % of good vs garbage in their release.

>>108233892
will never happen at this point

File: file.png (32 KB, 914x247)

32 KB PNG

cudadev's nvidia buddy is investigating performance improvements in the driver

>>108236277
>he thinks he can avoid this by paying

>>108236493
>but they also don't want to release something good and worth using, they never did, they refuse to do it because they consider that even losing 0.0000001% of Gemini usage to their own open model would be unacceptable cannibalism
this is kind of puzzling to me because 99.9% of people don't even know you can self host this stuff, to them it's just chatgpt from google in their phone
even if google released an actual good model, i don't think much would've happened

>>108236519
The nvidia drivers are still just a gimmick and not ready for prime time.

>>108236559
The nvidia drivers will swing the sword and kill the dragon.

>>108236309
>>108236323
>tfw never violent but there's a rabid dog next to you and the only option is to peacefully put it down

What's the deal with not releasing base models anymore? Qwen3 only had base versions for the smaller models, and now again with 3.5 we only get base for the shitty small MoE.

Qwen3.5-27b is the perfect size for community finetunes. But the non-base version we have is so mindraped by instruction tuning and RL that it effectively can't be trained using a language modeling objective. It's fucking bullshit, I have RP and smut datasets and would love to try finetuning it, but it literally just doesn't work with any model that's been aggressively RLHF'd.

Grim future tbhdesu, the models are "open source" but not really because it's impossible to train them on your own datasets.

>>108236733
Sorry you paid 10k to use ai models to jerk off into a sock but can't bro.

>>108236733
>Grim future tbhdesu, the models are "open source" but not really because it's impossible to train them on your own datasets.
that's the point, they don't want us to uncuck those models, the investors are happy, the safety trannies on twitter are happy, that's all that matter to them

>>108236733
It was already grim when the parameter count started exceeding 100B. You're no longer local. Who realistically is going to run GLM 5 as it is now?

>>108236733
everything gets midtraining now so it's pointless

>>108236796
>Who realistically is going to run GLM 5 as it is now?
I will run cope quants as soon as llama.cpp finishes the implementation. So probably never.

>>108236796
>>108236768
>>108236733
This genre of faggot needs to be exiled
Never mind all the smaller models that drop
Never mind all the quants that are released
Because this faggot wants to use an ai to jerk it into a sock it's all doom and gloom for this sperg

>>108236733
Base models don't improve on benchmarks anymore

>>108236803
Nothing is stopping them from releasing the checkpoints from before midtraining.

>>108235034
>lets talk about talking

>>108236836
>Never mind all the smaller models that drop
small models are retarded and will always be retarded, so what's the point of using them retard?

File: HCAKtCIaMAAWofs.jpg (392 KB, 1184x2684)

392 KB JPG

Aletheia tackles FirstProof autonomously
https://arxiv.org/abs/2602.21201
>We report the performance of Aletheia (Feng et al., 2026b), a mathematics research agent powered by Gemini 3 Deep Think, on the inaugural FirstProof challenge. Within the allowed timeframe of the challenge, Aletheia autonomously solved 6 problems (2, 5, 7, 8, 9, 10) out of 10 according to majority expert assessments; we note that experts were not unanimous on Problem 8 (only). For full transparency, we explain our interpretation of FirstProof and disclose details about our experiments as well as our evaluation.
https://github.com/google-deepmind/superhuman/tree/main/aletheia
arxiv.org/abs/2602.05192
FirstProof challenge paper
https://www.daniellitt.com/blog/2026/2/20/mathematics-in-the-library-of-babel
Interesting article about FirstProof
Weird. Tried to post earlier on my desktop and got an IP range ban message even after I reset my router a few times. But can post fine on my tablet. Wonder if it's a Brave issue, tablet uses fennec.

>>108236836
The entire catalog is already full of that kind of pessimism and whining. You'd think they could just pick any other thread instead of dragging this one down.

>>108236861
How are they retarded whinefag
What give me a real usecase outside of you wanting to jerk it inside a sock. Also post your rig because I doubt you have more than 12gb of vram

>>108236869
They do it in the other local threads too, it's so fucking annoying only for them to be proven wrong time and time again only to move the goal post.
I can only assume they lack the hardware to even run low end models.

File: 1754860930725949.png (47 KB, 625x626)

47 KB PNG

>>108236879
>How are they retarded whinefag

>>108236894
>it's so fucking annoying only for them to be proven wrong time and time again
give me some examples where they've been proven wrong lol

>>108236896
>Not answering
As expected
Take your dooming faggotry elsewhere.
>>108236905
Give me a valid point first, because when asked to provide proof you faggots scatter like roaches. Local doesn't mean enterprise and local is great for single users even on smaller vram cards for most task. Even on enterprise models you still have to babysit and review ai output to make sure the code isn't fucked or you'll have the multiple disasters we have been seeing from major companies this entire year.

File: file.png (1.22 MB, 850x1202)

1.22 MB PNG

Is anyone actually using things like OpenCode with local models with good enough models and not dying of old age due to prompt processing?

>>108236915
>Give me a valid point first
you said that we've been proven wrong, that's the moment you have to elaborate you retarded fuck, if you don't want that when I'm asking you to "provide proof", all you'll be doing is to "scatter like roaches"

>Afraid
You claim the space is stagnating yet we are seeing newer models with better performance being released constantly we just had a new Qwen model drop you stupid fucking faggot.
Go ahead say something other than your sour grape cope, you're no different than the faggot that dooms all day trying to shill api image gen models in /ldg/.

>>108236949
>better performance
on the benches they train on yeah no shit, even api models are becoming actually worse to use outside of code shit

>>108236966
>Still avoiding the question
Can you even run these models?

>>108236949
>newer models with better performance
better cheated mememarks doesn't always mean "better performance" you alibaba shill

>>108236733
Who in the community is finetuning base models? I was under the impression that most were already using directly the instruct versions for that.

>>108236979
We're done here you're a retarded waste of space just wasting my time spouting retard shit and avoiding my questions.
Get a job and you might be able to play with these models faggot.

>>108236990
>might be able to play with these models
there's nothing to "play" with they're all agentic code slop optimized bs

>>108236930
Seting a large batch size and never doing anything to invalidate the cache makes it somewhat bearable.

>>108236999
>Admits he can't run the models
L
O
L
Imagine living like this sitting in a thread posting FUD about something you can't even use.

File: 1756483977340095.png (485 KB, 736x552)

485 KB PNG

>>108236990
>We're done here
look at this faggot running like a little bitch

>>108236933
>have to elaborate you retarded fuck,
why are you getting so mad?

>>108237008
> He confuses “not wanting to use a model” with “not being able to use a model.”
Are you from India, by any chance? Your reading comprehension is terrible.

>>108237015
He got upset that his card got pulled. FUD faggots like him spend all their free time on multiple boards spouting shit they don't understand to make people feel doubt. It's a worthless existence.
>>108237023
You can't use it
You already admitted it by avoiding actual questions and your seething. You can't play and waste your time trying to mess with others out of jealousy, Faggot

>>108237015
>calls people "faggots", "spergs" >>108236836
"retarded whinefags" >>108236879
>but cries when the heat comes back to him
kek

>>108237042
You're talking to another anon you stupid fuck

>>108237036
>spend all their free time on multiple boards spouting shit they don't understand
NTA but that seems a lot like what you're doing, barging in here saying everything is fantastic and all.

>>108236966
>becoming actually worse to use outside of code shit
funny, because I am seeing better quality of webnovel translation with a lot of niche subculture jargon out of Qwen 35BA3B than I did in any other model of that size class before and this certainly ain't no coding task. Running it with reasoning disabled and temperature 0 (greedy decoding) for this.
Models are improving constantly, and right now I'm satisfied enough with the output of that thing that I no longer feel the need for better models to come out in fact. I mean, it would be great if there were even more improvements, but this thing is already capable of providing me endless entertainment.
People who are cynical about the progress made by models clearly don't remember how limiting context was in early GPT3 and GPT4 era even for online SOTA, and how right now even a model as tiny as Qwen 4B 2057 stays coherent at 32K summarization in ways that used to be scifi before.