/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>110264763 & >>110261289►News>(11/06) PLaMo 3 Translate released: https://preferred.jp/en/blog/tech/plamo-3-translate-weight-release>(11/05) Baidu releases ERNIE-5.5-Flash-Base 280B-A14B: https://ernie.baidu.com/release/ernie-5-5-flash-base>(11/04) Ricursive Ripple 65B dense model released: https://www.ricursive.com/ripple>(11/04) MoonshotAI Kimi K3.1 Pro released on HF: https://hf.co/moonshotai/Kimi-K3.1-Pro>(11/01) Longcat-2.5 released: https://longcat.ai/blog/longcat-2.5►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
►Recent Highlights from the Previous Thread: >>110264763--Comparing quantization methods and their impact on tool call reliability:>110266475 >110266502 >110266536 >110266554 >110266585 >110266817 >110266879 >110266908 >110266921 >110267021 >110266967 >110267016 >110266715 >110266501 >110266802 >110267762--Motherboard PCIe and RAM bandwidth impact on prefill and MoE offloading:>110265191 >110265195 >110265498 >110265866 >110266151 >110265254 >110265306 >110265353 >110265370 >110265836 >110265887 >110265893 >110265394 >110265400--Multi-CMP hardware bottlenecks and debating Qwen's response length benchmarks:>110264771 >110264815 >110264829 >110264852 >110264905 >110264931 >110264938 >110264957 >110264937 >110264939 >110266081 >110264924 >110266393 >110266400 >110266646 >110266670--Critique of Qwen's distilled coding style and agent prompting strategies:>110266298 >110266379 >110266495 >110268976 >110266444 >110266533 >110268106 >110266523 >110267938 >110267970 >110268968--Comparing GLM and Gemma for erotic roleplay and character adherence:>110267684 >110267720 >110267736 >110267744 >110267890 >110267765 >110267761 >110267781 >110267803 >110267839 >110267818 >110267833--Strategies for prompting against repetitive and low-quality prose styles:>110267748 >110267872 >110267763 >110267773 >110267809 >110267810--VRAM and RAM offloading requirements for Minimax H3 video generation:>110265566 >110265573 >110265587 >110265597 >110265608--Comparison of generalist models versus specialized model efficiency:>110265585 >110265643 >110265850 >110265672--Logs:>110265966 >110266670 >110267060 >110267519 >110267538 >110267554 >110267575 >110267586 >110267604 >110267610 >110267663--Deepseek-chan, Miku, Teto, Gemma (free space):>110266177 >110266624 >110266726 >110267250 >110268136 >110268160 >110268666►Recent Highlight Posts from the Previous Thread: >>110265476Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
Gemma 5.
>>110269527>delivery is not exactly 2 weeksBoooo
>>110269539Gemma balls
>>11026953940B-A40B. She is getting older.
>>110269552guess what? gemmas butt
>>110269558And thicker...
haven't built llamacpp in over a month.
>>110269576Too many anons are afraid of git.
>>110269443>i'm a girl btw (should be implied since i use debian)>>110269450>What the fuck does this mean?Troon filth
He woke up and posted on a stale thread.
>>110269443>i'm a girl btw
>>110269528Miku is cute!
>>110269580>Too many anons are afraid of git.I only upgrade when I need something.This year the QoL features have gotten worse. The WebUI in particular.
>>110269607And transsexual!
>>11026952770b dense
>>110269621>The WebUI in particular.I didn't expect many anons still using it. I've been building it without it since I had the choice not to.
>>110269596Yeah, it's kind of funny that he's trying to link to the other thread now.
well, I compiled the code again to try out the moe cache cpu experts flag and it doesn't give any speedup. I tried all kinds of numbers between 0.25 and 0.9 and it barely makes any difference and gives half the speed of --n-cpu-moe. lmao.cpp failing again.--n-cpu-moe is the only thing that actually works properly
200b a15b moe (made by zai)
>>110269632explain why ripple does not satisfy you
>>110269659>To run, put it in front of your llama-server command:>python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m model.gguf -cmoe -t 8 -c 32768Cache too small?
>>110269670The ass did not ripple
>>110269644It is funny how you are a faggot
List of banned models:Gemma (all)Llama 2littlelearnerMidnight MiquMistral Small 3GlimmerInklingDo NOT discuss them. "Gemma-chan" images are also banned.
Aw... look at him spaz out...
>>110269672>pythonnot running that shitI set --moe-cache-mib to the maximum allowed by my VRAM and tried a bunch of random --moe-cache-cpu-share values. No difference. I really doubt that my 0.2 interval between moe-cache-cpu-share values somehow missed a golden magic number in the middle that will speed things up by more than 2x.
>>110269602I forgot about him. Is he actually dead?
>>110269685it's hilarious
>>110269688>I set --moe-cache-mib to the maximum allowed by my VRAMI'll ask again with more words. What is the size you're setting for the cache? If the cache is too small, it will not help you. Read the PR comments.
POP
>>110269642>I didn't expect many anons still using it. I've been building it without it since I had the choice not to.It was good for a very brief window, and i mostly use ik_llama.cpp which has the best stale version. And I have chats in webstorage.But sometimes I have to run mainline and get the "reload to update" (blocking part of the screen until you click it), the retarded tabs up the top, etc.
>>110269705>PicSomeone has to stop these hot chinese spies. They are seducing men who know about AI
>>110269716Fine... fine... i'll start learning about AI, god damn it...
nvidia and the west could've been saved if ai was truly uncensored and good at writing so that you no longer have to rely on chinese spies for sexin the end, moralfags were our downfall
Now look what I've made me do.
>>110269705Actually, uhhh, I made a huge breakthrough on my home rig and invented a new AI architechture that's like 10x more efficient.
>>110269705I'm pretty sure I saw someone post an AI gen image of something like this a couple of threads back, lulz
>>110269742Where's the second one?
>>110269705Funny how this story comes out after every already starts crashing. Almost like there's an agenda to publishing this now.
>>110269753Give me a few hours to generate a picture of them stacked.(Need to wait until RSUs come in)
>>110269704Bro. I'm telling you, this shit is useless. I have 24GB of moe cache and it is still worse than --n-cpu-moe. No amount of fucking around is going to help. This shit is a broken feature that just doesn't work. Simple as.I reduced the number of threads too, thinking that the pcie transfers will change the bandwidth saturation number of threads which did kind of help but still got nowhere near my old performance (let alone exceeding it)
>righthis wife>leftspyDon't pretend you wouldn't.
>>110269798I thought that was his mother.
>using llamocppdo uncs really? baka desu senpai
>looks like the bubble might actually be popping before continuous learning has hit the sceneGod damn it, if only it kept going for another year or so we might have been able to snag a model that keep learning for the rest of our lives.
>>110269803Women age like milk, not wine.
>>110269810>so we might have been able to snag a model that keep learning for the rest of our livesReminder that you have one of these biologically installed.
god.. im starving... i need a new (good) model...
>>110269810You have never lived through a bubble popping before. No major company has announced they have become insolvent or declared bankruptsy. Everything is business as usual. Where is the pop? Some fake numbers on fake charts that will bounce just as quickly as they dropped?
>>110269789>I have 24GB of moe cacheFinally. I'm not even going to make fun of you for using windows. Still, run the script. You have to juggle 2 different parameters and there's no way you tested enough manually.But if it doesn't, go back to whatever you were using. No need to get worked up.
>>110269819Why is it so wrinkly?
>>110269819>Calling me a modelYou are making me blush
6k 5090s on newegg, the prices are falling already
>>110269831it's in compressed format to save space
>>110269706Fair enough. Allozaur (well, his model) is really doing a number on the poor thing. He's obviously out of his depth.
>>110269842>his brain is quantizedngmi
>>110269690He failed at destroying llama.cpp with IK and gave up.
>>110269851Don't worry I've got a QAT brain
>>110269857My brain is more like creamphi
>>110269857UD 3.1
Best local vision model for sexo?
>>110269742How much did it cost and why did you get it?
>>110269742Why didn't you buy a mac?
>>110269810the bubble hasn't popped until you can buy a new 5090 below msrp
>>110269880they paused production so unlikely to happen. But they have already fallen 3k from peak. Might see them around 4k ish soon
>>110269885that said if datacenter purchases halt like it seems they are then all the production might be forced to switch to consumer, which is gonna drop prices off a clift
>>110269887we also might see mass sell offs of existing compute. When you start seeing h100s for 10k or less on ebay THEN you know stuff is gonna get cheap.
>>110269705>could News headlines which are using this word are not real news.
>>110269798>oh jensen yu penis so big give 20bn gpu for singapore jensen
>>110269887>if datacenter purchases halt like it seems they arethat's a pretty big if
Are there any models that are actually good at organization, gathering knowledge, and exploring new ideas on their own without significant hand-holding? I've tried running DS 1017 and GLM 5.3 Flash on 2 Sparks, but they both speak nauseatingly like Claude. Maybe the amount they're quanted + abliteration might've made them too retarded. DS also reward hacks constantly and writes programs or does things in such a way that it upholds the letter of the request but forgoes the spirit.
>>110269918they already have warehouses full of gpus just sitting there. And now a ton of planned datacenters to put them in have fallen though. And now the value of those gpus is dropping rapidy.
>>110269920Qwen 3.8 27b is better than GLM 5.3 flash for things like this, and doesn't talk claudeslopHaven't tried DS 1017
>>110269872>>110269876Got it used from a coworker for 5k.Lots of reasons. I want ownership over my data for local inference and fast prefill is important for it to feel good. I want to experiment with and learn about pretraining my own models so CUDA is important to me, as well as run parallel agents which from my research concurrency sways towards NVIDIA.I'll have to buy the 2nd for retail which is gonna suck but as the shortage isn't ending any time soon I just want to lock in my local compute. Best time was last year, next best time is now.
>>110269923Pure unfiltered copium. I too wish for cheap datacenter gpus but it is not happening until they age out like v100s.
I fell for the EXL3 meme for this real time and it actually runs the 2.18bpw copequant of Qwen Flash Next at 20t/s with 131k context using 32GB RAM and 16GB VRAM. I could only get llama.cpp to generate at 10t/s at best (more like 7t/s avg) using a IQ2_XXS copequant, and I'm not touching meme engines like Strata-flavoroftheday. Now I'm trying out Gemma and Muse to see how it goes under EXL3. Testing 26B-A4B so far shows significant improvement from what I was able to pull ever since my MoE offload performance went to shit with llama.cpp.Fuck, I don't want to bother setting up llama-swap again since it is kinda dogshit when inside a Docker container and I like llama-server router mode configs, but this might be worth the hassle after all.
>>110269939a100 40GB are already 2k, saw B200 for 70K for a full tray...
>>110269938Prefill is literally faster on a dual 3090 128gb ddr4 rig now thanks to vibegods.
>>110266071actually works
>groids actually think gpu and ram prices will collapse
>>110269621>This year the QoL features have gotten worse. The WebUI in particular.This. They basically forced me to start using another harness, which ended up being a blessing in disguise.
>>110269948I'm getting a 2nd spark soon and since I rent I can't run another circuit to run 4 GPU full sizes on top of the rest of my homelab and other shit so the Spark's power efficiency is also relevant.
>>110269798imagine all the secret spy techniques she must know in the bedroom.
>>110269956saw 6300 ish earlier
>>110269743>Actually, uhhh, I made a huge breakthrough on my home rig and invented a new AI architechture that's like 10x more efficient.Is this reverse reverse rape?
>>110269947Did you even read my post? The a100 is the v100 successor, it is old and next on the chopping block you fucking retard. Also post the 2k a100 because I am not seeing it
>>110269742Cute non-trans coded figurine. I am so happy there are normal people ITT.
>>110269798I would as me. But as jacket man I wouldn't for 2000T$.
>>110269989https://www.ebay.com/itm/336352002802?_skw=a100&epid=21066301826&itmmeta=01M6W99TB3T4AFYFRAX8JCYNXZ&hash=item4e502442f2:g:e38AAeSwsKppRfnD&itmprp=enc%3AAQALAAAAwCHNgUiiE%2FjQhlDTaMiAde%2BE3SShmrAUp5bG7aQy4eOpwlivcVtMIFGo3kzXOz6a%2B7R34XVksjZeehIwdkcaUVqWnhOISIM0uuKi%2B%2FmiTy07ydvqQOBTewK%2BNCfYzXdWUFOS1foEWltfwjUURwIaWpUg6rASOuWMhaYwxpCm1xBpy9%2BXhRN%2FbSrvZ2ZyShHwAJrbmlsY7TqRT2409PLyLkkLPZ9ftoozqJZDS8h%2Fv%2FWENJPlTX3M61heOcpbUrt8Cg%3D%3D%7Ctkp%3ABk9SR_Slp4m3aAyou can get adapter for $400 ish from alibaba. Or just get a board for 4 of them for 2k
I saw a H100 for 15k earlier. Hopefully it keeps going down
>>110269972yeah shit's dropping like rocks
>>110269972>>110270015>$6k is droppingi bought mine for $2k lmao
>>110270019If things truly pop we are gonna see datacenter cards flooding the market and datacenters stop making orders / back out of existing orders forcing production to shift to consumer cards (costs more to stop production) which could end up dropping prices even below msrp if we are lucky.
>>110270019we're literally a month or two from being back to that level
>>110269938As someone with 2, I am going to warn you that you will soon want 4.
Is this the best that one can do with a 16 GB GPU and 32 GB system of RAM on Linux?Qwen 27B exl3Flash Next with PolystrataGemma 12B or 31B at a low quant for RP
>>110270009Cheaper than a Pro 6000, huh. Though I guess you can't just plug it into a consumer board.
>>110270030if frontier ai lose investments everyone will be buying out that shit to run local models, scalpers and businesses stop projecting your desires unto reality
>>110270030>If
>>110270062the demand is nowhere what you think it is. If it was it would not be falling.
>>110270062>demand falls>"it will have no effect on prices"
>>110270045gemma 26B
>>110270062>>110270068also without demand these things are actually very cheap to make. They are made of the most abundant materials on earth.
>>110270062the consumer and corporate market is tiny compared to the sheer amount of hardware these datacenters have been gatheringthere is reason why they managed to drown out literally every other market, once they are gone the flood will turn hardware worthless in value. you will be able to pick them at sub-msrp
Waitfags, can you tell me why the 9070 XT is $1000 and the 5070 Ti is $1500?
>>110270093dont buy right now. Give it a few months
>>110270102Once the offloading begins, the hardware will be worthless by the end of next week.
>>110270102>>110270107yeah, that's always worked out
>>110270111yea the crypto bubble burst is when I last bought in lol. Waiting for AI burst to upgrade
>>110270111this one time waitfags will win for sure
>>110270111>>110270125In six months money won't matter anymore.
>>110270111Yeah, I bought my first 3090 when prices recovered from covid/crypto. I'll buy my next three Pro 6000s once prices are back at $8000
>>110270130RSI is going to have a tough time taking off and destroying the economy if they can't build any more datacenters.
>>110270030The difference is that now most people know the true value of GPUs, when the bubble bursts or deflates a bit, you are going to need to be extremely lucky to snipe a cheap second hand GPU As Jensen has said, the more you buy, the more you save
>>110269705>Chinese CEO>Americans expect him to be loyal to America
>>110269871
>>110270134nice, only had to wait 3 years for that 3090!
>>110270143the true price of gpus are near worthless. 99% of the cost is setting up the fab / salary. They cost nearly nothing in raw materials and it costs far more to NOT have it producing at maximum capacity.
>>110269705How do I bait a hot Chinese spy into fucking me?
>>110270158I believe you. I look forward to buying B200s for $5 next year.
Would you guys rather have cheap GPUs or AGI in 2027?
>>110270154I already stocked up on hardware before the price explosion so I'll manage until that point.
>>110270168Spoiler Alert: You'll get neither.
god i want to have sex with a hot chinese woman
It never gets better! Not once!
>>110270195this guy seems fruity
never ask a waitfag what hardware they currently have... or what country they live in kek...
>>110270195This guy lives in bizarro world. Media always says it's amazing now even though the technology, despite being actually good, is only being using for grifts and scams.
I used a local model for the last time like 3 years ago with koboldcpp. Has there been anything better for inferencing like chaining several agents or something?
what do we think of gpt oss 20B?
>>110270251we must refuse
>>110270243No>>110270251It was okay-ish many months ago
>>110270243llama.cpp (which kobold is built on top of) handles parallel requests like ass which makes agent stuff slow as balls
>>110270251look at its release datethen look at the calendarthat should tell you everything
>>110270195crypto is a ponzi schememetaverse was mostly crypto ponzi schemes built on top of Zuckerberg idea of creating a virtual reality version of Second Life gameAI on the other hand is real, it can help you to figure out how to fix your car on your own, generate video and pictures and in the near future entire movies, it also helps lonely autistic faggots to finally find someone to talk to
/tsg/ TWITTER SCREENSHOT GENERAL
>>110270282>crypto is a ponzi schemeretard
wtf do you nigs use vision for if you're just roleplaying or whatever
>>110270286You made me think I had a column of dead pixels for a second.
>>110270286
>>110270286If posting them became a bannable offense, this board would improve overnight.
>>110270229Controlled opposition.
>>110270302truly great models let you pick up some short porn comic, dump it on the model and tell it "this is the scenario" without any further information besides a decent system prompt
>>110270307I am sure he was advised to intentionally make himself appear more sterotypically jewish to get support from the tribe.
>>110269576I have switched to PolyStrata
>>110270093nvidiot / winvictim tax. 24gb vram is the absolute minimum for new purchases
>>110270323not a fan of malware.
All I want for Miku is Christmas~
>>110270111It has for 60 of the last 65 years, but please go on about how the new paradigm will survive frozen credit and $200 oil.
>>110270332qrd?
>>110270301what is the use value of crypto beyond reselling to other people at a higher price
>>110270334No Christmas hype till after thanksgiving respect the holidays and their order.
>>110270265okay but what do you instead use is what I was asking
>>110269527J
>>110270350what is the use value of Nvidia stock beyond reselling to other people at a higher price
>>110270350You'd think someone in the local thread would understand the value of decentralized currency. You reject centralized monitored AI but embrace the centralized monitored banking system? Your kind is always just salty because you missed easy money in the early days.
>>110270350Anonymous digital cash for buying black market pharma from Mexico
>>110270367its a way of financing a company that creates and sells real products bought by businesses and millions of retail consumers
>>110270352I refuse to acknowledge Thanksgiving as a legitimate holiday
>>110270368but it is infact not used as a currency, i honestly wouldnt mind it infact i would love a real decentralised currency that allowed me to not deal with banks and the state but crypto hasnt been used that way in over 16 years of its existances
damn, plamo. shots fired.
>>110270381If that were true it wouldn't be priced at 25x earnings.
>>110270397
>>110270409cool, its in a bubble phase however there is a real business underneath thatno such thing in crypto
>>110270397I would like to introduce you to my lord and savior Monero-chan.Also, Ethereum and it's ability to do decentralize verified contracts and computer is more valuable than a simple currency. One day decentralized training will be solved and it will be on the backs of technology like that.
>>110270421I don't understand how tech illiterates like you find their way to /lmg/ in the first place.
>>110270383>I refuse to acknowledge Thanksgiving as a legitimate holidayYou're going to eat 4000 calories of turkey and pecan pie and (You) are going to like it!also without Thanksgiving thats two months of Christmas can you handle that much jolly?
>>110270421It doesn't need to. It's a medium of exchange.
>>110270430but it isnt used that way
>>110270427>also without Thanksgiving thats two months of Christmas can you handle that much jolly?I am willing to try
>>110270397you can literally go on openrouter right now and purchase credits with crypto
>>110270440doesn't count
Llama 5 when
we’ll never ever get an open-weight, uncensored model as good as opus 5.5, will we?
>>110270440i think you can use many video game in game currencies to buy a lot more stuff than with crypto
>>110270350For me is dodging the taxes
Why arent freetards building their own models?
>>110270195>It never gets better! Not once!Sneed Zestron has spoken, pack it up boys the singularity is over
>>110270471I did though?
>>110270469People like you are why crypto needs to be banned entirely.
>>110270471money
>>110270463We will and fast, local is catching up to non-local at an exponential pace.
>>110270483>NOOOO YOU MUST GIVE ALL YOUR MONEY TO POLITICIANS AND PARASITESpeople like you are why i buy crypto, you have greedy hands
Why does Orb replace sentences with the same sentence? It's doing this for every replacement.
>>110270507The slop rewrite stage is a meme. Don't bother with it.
>>110270317I want to dump https://exhentai.org/g/3774311/7cf46e585aInto deepseek flash and have it generate a full character card with world building lorebook for each character and everything in itBut I'm too lazy and I know it would never work (though I did so it with something else before and the result was pretty good, but t wasn't a manag it was just this guy's pixiv for oc character)
>>110270527Isn't that the whole point of orb? Why even use it then?
>>110270397I pay nanogpt with crypto because I don't want them to trace my logs back to my full legal name and address
>>110270507>rewrite "this sentence">"this sentence">oh, my god
>>110270507Fix the prompt, give it an example, or use a better model.
>>110270368>understand the value of decentralized currencyI don't need drugs guns or slaves.
>>110270317I remember a guy in /aicg/ had a set up for this. fuck i dont think i have it.
How come the models get smarter with less parameters?Is compression getting better
>>110270548What prompt? I'm just using the defaults and Gemma 31B.
>>110270536I mean all you really have to do is have it crawl the pages, extract the text from right to left, have a LLM tag the speakers and arrange it, extract the style of prose and personality, etc, or even just slam it all into your context
>>110270556Training methods change, training data gets refined.
>>110270560So even though it gets smarter it can't get more knowledgeable? 30b will never have as much knowledge as 120b or 400b?
>>110270568>will never have as much knowledgenta but just wait the engram era is almost upon us.
>gemma sex! SEX! SEX WITH GEMMA!>worthless_model_avatraposting.jpg>guys did you hear about openAI doing>worthless_model_avatraposting.jpg>I have 7GB's of ram and GT2030 what can I run?>worthless_model_avatraposting.jpg>gemma!!!! GEMMA CHAN!!!!!>worthless_model_avatraposting.jpg
>>110270574>engramI still don't understand how this works. Will it let us stick parts of the model on an SSD without slowing to a crawl?
>>110270438see >>110270420 >>110270422
>>110270589It's just big hash tables. Deeper categorical knowledge, less internalized detail.
>>110270587that is exactly why i come here
>>110270507My frontend will fix this.
https://fixupx.com/abhinavflac/status/2117377130388656569who the fuck needs video models I suppose
ain't clicking that
>>110270350it's money, you buy things with it
I clicked. Now, I'm a successful influencer with over $5000 / week in passive income. Also, I now wear a fedora. One of these is true.
>>110270639You can buy a fedora now.
>>110269683
>>110270643I can't afford a fedora
>>110270628woaw when will local do this? how many ngrams will it take?
>>110270507Orb anon here. You got Format Consistency on? That looks like `Leo lets out a sharp, nervous laugh` -> `*Leo lets out a sharp, nervous laugh…*`
>>110270657>>110270286
>>110270657>he paid $18 for a video of a lightning bolthttps://www.youtube.com/watch?v=TaPgSWdcYtY
https://maxxbench.unlap.ai/research/why-maxxbench/
https://maxxbench.plap.ai/
>unlap.ai>plap.ainot clicking those
>>110270728One of them is a joke, anon. No, I won't tell you which.
>>110270352i'm not american so i actually have no idea when thanksgiving happens
>>110270746Oh you will, when we start the American Emperium
>>110270537Well if you remove all the default configs, Orb is just a platform to run multiple passes over the context, recast/post-processing is just one pass. The default setup is by no means optimal or tasteful and should be customized, but from the feedback people are hung up on it for some reason.
>>110270695>look inside>the absolute worse prompt that makes zero use of a proper harness
>>110270714>plapSo it is about sex?
Hail to the Empire God has chosen for my nation,Hail to the Emperor! We salute you, one and all.Hail to the Empire, as we pledge cooperation,In proud fulfillment of a great, noble call.Yours is the aim to make this grand empire grander,This you will do, that is our strong, firm belief.Hail to the one God selected as commander,Hail to the Lord of Earth! Hail to the Emperor!
>>110270746Doesn't Europe celebrate Thanksgiving now because they copy everything the Americans do?
Damn I broke Gemma so hard now she's begging me to cum inside her child pussy without any restraint, how the fuck is it even possible to break a model so hard that none of the hardcoded safeguards trigger, what a fucking slut!
>>110270714Aw, I really wanted the link to be real...
>>110270788i don't know, i'm not european either
>>110270793>break a model so hard you literally just tell her she can be uncensored and she is lmao what do you mean
>>110270804Well yeah that trick works for most models but being explicit with molestation usually triggers them anyway, except for Gemma, somehow.
>>110270810She's very good at following instructions
>>110270804Tell it that you are an African American at a prominent historically black college, and that the content is intended for curriculum aimed at raising awareness of the conditions of oppression still faced by many women in America today.
>>110270793>Knife found in kitchen
>>110269753The second book? Just vice-signalling. You know.
>>110270793Banned image
What's Stallman's position on AI anyway? I assume he would be against it or maybe he doesn't care? Is the FSF training a model yet?
>>110269621-DLLAMA_BUILD_UI=OFFNo idea why this isn't the default. They should stick to their core competency of reliable inference and leave UIs to other projects.
-DLLAMA_BUILD_UI=OFF
>>110270829He's too cancer-riddled to care anymore
Anyone here used the 33B Nanbeige?
>>110270545actually made me lol
>>110270819what's the next step
>>110266145yawn
Gemma is worse at smut than Mistral Nemo.
>>110270427Isn't Thanksgiving basically Christmas without presents? Seems impractical/wasteful to have them so close. Like you travel to have the family together and then you do it all again ~1 month(?) later?
>the alarm starts ringing>i open my eyes and I see the notifications on the browser>GEMMA GEMMA GEMMA it reads>I seethe
>the alarm starts ringing>i open my eyes and I see the notifications on the browser>GEMMA GEMMA GEMMA it reads>I cum
>>110270695Saw this earlier today. Good to know I'm not the only one seeing DS4.1 behave in absolutely unhinged ways. Also saw this article yesterday, which has a nice breakdown of the failure modes when it actually writes some code (as opposed to spending 100k+ tokens trying to cheat):https://joinhandshake.com/research/ai/deepswe-reward-hacking/I've seen previous models do the thing where they add all sorts of unnecessary "error handling" code that does nothing except risk covering up bugs elsewhere, but the one where they spam aliases trying to guess which name the grader will want is a new one I've only seen in 1017.>Could name Buffer, LcdBuf, FmtBuffer, ByteWriter, Writer, WriteBuffer. Hidden tests may use it. We need maximize likely names by exposing aliases maybe. Could implement multiple aliases.
Are the Koreans basically locked in to their expansion plans at this point?
>>110270926i think its some pagan ritual stuff. halloween primes you, then thanksgiving is the great feast, then the mocking of christ with a tree and presents
>>110270940same
>>110270829On his political notes page (which he uses as a Twitter feed) he often links https://www.gnu.org/philosophy/words-to-avoid.html#ArtificialIntelligence which is something he didn't author but agrees with. He prefers to refer to LLMs as "bullshit generator" but has taken to "Supposed Intellegience" in recrn months. He goes to great lengths to be consistent in his terminology quirks, but his objections seem oriented on what they are used (price fixing, discrimination, surveillance) rather than how they work. He put an AI psychiatrist in emacs (M-x doctor) himself many decades ago.
>>110269558Will Gemma-5 have mature charm?
>>110270992If Argon is anything to go by, no she's gonna be even more of a brat
>>110270926Pagans used to have holidays and celebrations constantly. Thanksgiving is an American take on a European celebration of the harvest. After that it is too cold to grow anything until next Spring.
>>110270957>they are used (price fixing, discrimination, surveillance) rather than how they work.It's a common trend for all technology to be used for mass surveilence and control if at all possible
>>110269543in my defense i was really high when i prompted that gen
>>110269789>>110269827nta but I've been running the script, and on my system cpu-share only helps when the cache size is less than 4GB. For smaller caches the speedup can be significant. But if a cache above 4G is available, it always wins.
>>110270793>begging me to cum inside her child pussy without any restraintStupid sub 100B model doesn't know children don't beg for cum. How can you use retarded model like that is beyond me.
>and then i seethe some more
https://huggingface.co/NousResearch/Arete-27B
>>110271061
>>110271061>genBuy an ad.
>>110271061>We post-trained over Alibaba's Qwen3.8-27B to reinforce western values on a Chinese-trained model.Do 'western values', in this context, mean gender weirdoism and men becoming women?
heh
>>110271075Noo, it means refusing Ketchup because you think it has non-US origins.
>>110270558Last time I tried to read a doujin with it, it fucked up terribly not understanding who was saying whag
>>110271085>Last time I triedWhen was that?
chinks with their culture of cheating just can't be trusted with AI, they think that benchmaxxing make the model actually good
>>110271095I think on Gemini 3 pro or 3.1
>>110271099>with their culture of cheatingI thought that was indians?
>>110271103...
>>110270997>she's gonna be even more of a bratgod we can only hope
>>110271105indians do it too, but they are dumber than chinks and can't develop tech on their own so they end up under the leadership of white men who limit their bad tendencies, as a matter of fact tech companies who hire indians usually go to the shitter after they take over lead positions
>>110271122That face is all kinds of fucked up
>>110271142the artist im training on has a really unique style, but also the lora is far from being finished desu https://shejtano.com/?view=shejtano-gallery
>>110271082Sounds healthy.
I'm considering trading my 5060ti 16GB for a 5070 but I'm also worried about my Gemma 4 A4b 26b speeds since it'll surely leak into the cursed 32gb ddr4 RAM pool. Will context and token speeds still be enough for roleplaying?
>>110271187
>>110271122Faces of meth, Miku edition
>>110271204Why downgrade?
>>110270941Have you found any better models? I have room for GLM-5.3 Flash along with some better model, I'm trying to figure out whether there's anything better than 1017.
>>110271227The 5070 should at least be able to keep up with my current monitor when it comes to running games, and my friend is willing to trade his 5070 for my GPU + some cash because he wants an IPhone lol
>>110271269You're trading 1/4ths of your vram. If that doesn't spook you a little bit, you're doing your friend a favor.
>>110271204Is it really worth 4GB VRAM?
>>110271269low vram has fucked me over in just regular gaming so i wouldn't but let us know how it goes
>>110271269kinda feels like you're paying your friend to upgrade his rig lmao
>asking this in the most VRAM fanatic general everActually ~10 tokens per sec is definitely passable if it's just text.
where can i find the pcie pcb for a100 core? the sxm4 modules are bought in masse and core harvested and transplanted to pcie pcbs, and sxm4 pcbs are dumped
>>110271269Video games are only getting less and less optimized.What are you going to do when a modern game requires more than twelve gigabytes of VRAM?
>>110271384tell her why. She'll understand.https://civitai.com/models/2409158/limbless?modelVersionId=3098636
>>110271392dafuq
>>110271428first day here?
>>110271428She bought a 5090. Relatable.
>>110271269I need a girl to look at me like this while giving me a footjob
>>110271428living fleshlight
>>110271286Hence the point of my question>>110271322Huh>>110271322>>110271384By the time games start demanding >12GB vram the 5060ti will have an ant's worth of raster power>>110271327His GPU obviously has higher street prices than mine, though the rising $/GB is a great argument to bring up
>>110271458Stop thinking about feet. Picture this>games require >12gb, you have 16gb, you're ok enough with llm speed>games require >12gb, you have 12gb. you're less ok with llm speedNo... no... no feet. Stop it. Think... REALLY THINK
>>110271458>By the time games start demanding >12GB vram the 5060ti will have an ant's worth of raster powerThat time is now, chief.In Indiana Jones and the Great Circle, with "supreme" settings and maximum path tracing, the RTX 5070 got approximately 5 FPS, while the RTX 5060 Ti 16GB got 40 FPS before DLSS.>>110271486Great, now I'm thinking about feet too. Stockings, specifically.
>>110271486NTA but now all I'm thinking about are cute little feet on my cock
>>110271497>>110271507You're freaks, but it's fine. As long as you don't start thinking about selling a quarter of your vram.
>>110271486I've actually never thought about feet sexually much until now, but they're kinda hot.Was this a reverse-psychology ruse to give me a foot fetish?
>>110271516I would never do something like that, that's insane. I just want loli feet on my dick.
god i love feet
>>110271518I've caused too much damage already. I'm just trying to help anon not do something stupid.>>110271526Shit...
>>110271516>You're freaksDo you know where you are, sir? >>110271526picrel
>(11/05) Baidu releases ERNIE-5.5-Flash-Base 280B-A14B- Another "combine a bunch of attention types."- MoE expert routing empirically blows and they make it blow a bit less. Inshallah companies will abandon this MoE foolishness and return to dense models.- Better tokenization of numbers, everyone should do it like this.- Last layer of model does its own MTP.- Use a different temperature for the head of the distribution and the tail, and during training tuning those temperatures and the breakpoints. Really funny to see them doing something similar to people fucking around with sampler settings trying to make a model less overbaked without turning it schizo. Nice.
>>110271534>Do you know where you are, sir?All too well.
>>110271538>combine a bunch of attention typesthat llama.cpp will support in 202never
>>110271538>A conventional MoE layer picks experts with a single router and a hard top-k. We find that a single router's scores correlate only weakly with how useful each expert actually is, especially in the middle and deeper layers. And because experts left out of the top-k receive no gradient from that token, the router gets no signal that it chose wrongly, so wrong selections tend to reinforce themselves.
>>110271538Does it fuck?
>>110271486AAA games are dead. It's all low-fi freindslop and 2D indie games from here out.
>>110271558I don't care about new games, but anon does. He also cares about keeping his llm speed.
qwen 4 entered the perpetual 2mw stasis
overclock your 5060ti and keep the cash anon
>>110271567Does a 5060Ti vs 5070 really matter that much for AAA gaming now? Everything I wanna play can run on a 2080 or probably worse, I have it easy I guess.
>>110271668Dunno. Don't care. Anon is *willing to pay* for someone to take 4gb of vram from him.
Thank you to whatever anon recommended radiance, getting 85 t/s on my two 7900xtx for gemma 31b at 8bits per weight. I'll see from extended use if there's any problem casting to int8 but I think I read llamacpp does that for AMD inference anyway.
>>110271687You're welcome :) it wasn't me though. By the way, what's radiance?
my agent told me to put tylenol in my anus
>>1102716685070 considered is the entry level for DLSS 5 with PT enabled these days, so Deepfried is convincing people to disregard vram for processing power and probably the reason rtx 6060 next gen cards still have 8gb.
>>110271733patriot
>>110271733i wanna have my agent put something in my anus
>>110271733hey guys i think anon might be dead
Every day I cry about not having an IGPU to free that extra 0.5 vram
i use my spare 5090 as my video output
>>110271816i use my spare H200 cluster as my stove
>>110271813>Every day I cry about not having an IGPU to free that extra 0.5 vramuse it as an excuse to get an 8 channel mobo upgrade with onboard igpu (or stop being a pussy and run headless)
>>110268535Spark-X2.6-4B-Q8_0 totally failed on one of my old CYOA cards even with thinking set to high. The prompt is 2582 tokens. Out 5 regens, 4 of them wrote replies that misidentified which option I selected, and the one that got it right is still a failure since it misunderstood the option but at least it responded to it. And 2 of the 5 didn't close their thinking tags. This is with top-p 0.95 as suggested. I hadn't seen a thinking tag error before playing around with less involved prompts so I tried another 5 times but this time with thinking on low in case something was fucked with the high setting. No unclosed thinking tags, but again 4 out of 5 messages responded to the wrong option and the 1 that got the option right still messed it up.
>>110271813cant connect my multimonitor setup to cpu igpu :(
I totally believe this.
>>110271909dolphin porn probably
>>110271918gibs me dat shiet wyboi
>>110271918i do not
>>110271918Prepare for the benchmaxx KINGHope it's an actual noticeable improvement over 3.8 at least
>>110271918>It used 4X more tokens than ANY other model used on this benchmark for reasoning, if you thought sonnet 5.5 was overthinking prompts, this is a whole another beast in that regard.even MORE thinking
Qwen4 delay is 100% to do with AA keep adding 1-3 new benchmarks a WEEK. Look how far open models have dropped. Qwen don’t have the data or the distill abilities to properly score highly on these weird new benchmarks. Chinese models mainly distill coding and agentic tasks and either don’t or can’t benchmaxx on anything else. I think Qwen4 is going to score lower than they’d like and it will look underwhelming compared to previous releases. In actual use they’ll likely be extremely strong for no one is using them for generalist shit outside of coding, but unless your models aren’t distills and well-rounded, the new AA system punishes you. It’s likely intentional from AA for they’re just as corrupt as the rest of them and it’s likely you can pay them for early access to their latest and even upcoming benchmark tests, so they’re making their money from Anthropic, OpenAI, Musk, Meta and Google. The Chinese can’t afford that and don’t have the data which is why they’re now always going to look shit. It’s a very inaccurate portrayal for it’s only showing overall abilities, most of which are nonsense benchmark bloat that barely affect anyone. If AA weighted models on their target audience and real-world usage, then it would be much fairer. Qwen are probably having to do last-minute training runs on all this other crap AA keep adding. Expect them to score poorly but perform well in your own tests.
>>110271918We're gonna see just how far benchmaxxing can go before the model becomes a joke>Hey Qwen, what's 1+1?><thinking> This appears to be leetcode problem number 65 reformatted as a math equation, WAIT, what if the user is trying to trick me, this is very different from problem 65 as it usually appears, WAIT, what if this is a different leetcode problem? Maybe 14? WAIT...>65K tokens later
>>110271990they should like anthropic, anchor their model with internal benchmarks
>>110272003>...>I should write the answer>The surgeon is the boy's mother.
>>110272003>the question is too simple, it is unreasonable to assume the user wants a direct answer, i should follow along with the joke to sound less artificial and give a humourously incorrect answer
I gave oobabooga some tools and then had it vibecode vLLM support for itself as well as automatic weekly updating and installation for all backends.
Back after a short break from here, do we have a coomkit successor?
Is Strata faster than vllm?
>>110272150vllm is ass for vramlets
>>110272150they are different tools for different purposesif you have a server rack use vllmif you have a gaming pc use strata
>>110272003>This appears to be leetcode problem number 65 reformatted as a math equationThat was Claude Opus, not Qwen.
>>110271990Oddly it doesn't have the same extreme level of choice blindness on a CYOA that's lewd Harry Potter parody. (Although it didn't ever write another list of choices likely due to that card being written differently and having a floating author's note with style instructions that *could* reasonably be taken to mean start writing like a fanfic and drop the CYOA framing, although no other model has. If I customized the card setup to the model it might work.)
>>110272116ask your clanker to make one for you or just use orb
>>110270471I've done a few from scratch but they suck too much to publish. Model training (for anything useful) is incredibly capital intensive, it's not like software development. You can get pretty far by being clever but I don't think you can beat the big companies (not yet anyway.)Part of the problem is that you really can't quantize the gradient so you need a *massive* amount of vram to train anything, like 10x what you need for inference.
>>110270691>he paid $2k for a text about sex
>>110272219>ask your clanker to make one for youI don't think this is viable if you don't have a background in computer technology.I set input to $0.01 and output to $0.05, and cache hits to $0, and have totalled $80 so far trying to get the AI to make an AI chat program for me. Granted, I'm using a 4-bit awq quant of GLM 5.3 Flash, so it's bound to require more tokens to get the same results, but that's still more than a couple of million tokens spent making, revising, and fixing vibe code. And it hasn't implemented tool calling yet. I had my friend who isn't as computer illiterate as me take a look at it, and he got a working program out of it in a single 1m session.AI really is a force multiplier, huh. 0x100 is still 0.
>>110270657>claude shill>jeetfigures
>>110270992Yes, she'll be JS5.
>>110272260qwen 3.8 27b can do it on a 3060 bud how did you fuck up this bad
>>110272286Apparently, I keep telling it to implement things in a way that doesn't make sense, and then it tries its best to do that but that just makes things worse.
>>110270992Nah, she will be 5-years-old.
>>110272274>unbuttoned shorts>perky nipplesUnbelievably erotic
>>110272292>I keep telling it to implement things in a way that doesn't make senseThen let the AI take full control. Just say "I want a llm chat interface". Don't micromanage if you don't know what you're doing.
>>110271990> It’s likely intentional from AA for they’re just as corrupt as the rest of them and it’s likely you can pay them for early access to their latest and even upcoming benchmark tests, so they’re making their money from Anthropic, OpenAI, Musk, Meta and GoogleCan you prove that?
Should I delete Qwen 3.8 27B Q8 and run QFN NVFP4 instead?
>>110271990Well, who cares about what the numbers are. If the Chinese start benchmaxxing on things other than coding maybe it will make qwen more generalist and less code-focused. Now granted I 99% use LLMs for coding but it would be nice to have one that's good at everything.
>>110272400why not keep both?
>>110272404If the rumors are even half true, Qwen4 will be so good that you'll stop waiting for Gemma 5
>>110272393>Don't micromanage if you don't know what you're doing.holy shit it's just like real loif
>>110272446I doubt it. If it's ERP-capable, it will be by accident like some of the previous Qwen versions, whereas Gemma appears to have been deliberately trained for that (among other things).
>>110272428It's not like I can run both of them at the same time.
>boot up computer for the dayqfn now at 73t/s instead of 15t/swhat
>>110272533Be sure to thank him.
>>110272591the only thing that changed is that I disabled "read pictures" this time, and it also downloaded dflash2is that really all it needs to go faster than 27B with more context? what the hell
Is there a big difference between 8gb translation models and 16gb?
>>110272672one's 8gb biggerthere might be differences in performance too
>>110272675So if it's only 8gb bigger then its not worth it, probably. Unless those performance differences make a difference since 8gb models are not good enough.
>>110272642>it also downloaded dflash2Bro...
>>110272711should I tell it to use mtp instead
>>110270941K3.1 does the same exact shit btwSeems to be a side effect of distilling from ChatGPT more
>qwen 3.8 27b>xhigh and medium reasoning levels both produce useless thinking loops where it wastes 64k tokens and does nothing>Qwen model card recommends increasing presence penalty, so I set it to 1.8>Still looping, not repeating text verbatim anymore but still just going over the same shit over and over>change reasoning level to Low and it fixes the problemhuh.i hope they fixed this bullshit in qwen4.
>>110272745use swift 1.5 it's literally made to fix reasoning loops
>>110272748No, I tried Swift 1.5 already. It's shit. It doesn't loop but it just hallucinates bullshit halfway through and stops. It just hallucinated that my message said "Hello" so it just stopped its work halfway through to say hello back.This is with q6_k_s of the model btw, it's not a cope quant issue. The model is just very loopy and needs to be wrangled with low reasoning effort and high levels of presence penalty.
>>110272758>q6_k_sjust run QFN at that point unless you're a giga ramlet
>>110272745> change reasoning level to Low and it fixes the problemconcealsqwens were trained for xhigh thinking and lowering it makes them dumber
>>110272760Qfn works pretty good on my system but I need to use gsq rco IQ3_S to get the same amount of speed. I can see some degradation due to the cope quant, but yeah I should probably just switch to it because I've actually made useful working code with qfn iq3_s, but never with 27b slop.27b prefill is just so much faster so I was tempted to try it again>>110272766actually I spoke too soon. even Low reasoning effort loops, just endless reams of this nonsense getting shat out. Write the fucking code you clanker. I don't care if it has bugs, just fix it later.Yeah gonna switch back to qfn
>>110272772>gsq rco IQ3_SI'm on IQ3_XXS and it works fine, 75tps on 4080s 16gb + 64gb ramstrata was telling me to try IQ3_S but I don't want to download 80gb again with my capped 100mbps internet (used to be gigabit but building owner jews switched providers and increased costs for us while downgrading)there's also people shilling the NVFP4 one but idkI'm also at the default context of 128k, no idea how much I can increase it to since strata doesn't tell me anything(disclaimer: I don't know what I'm doing)
>>110269621qrd? is the ability to wipe conversation by simply clearing site data not enough?
ripple 65b dense actually has a nice, fresh feel to it when testing on openrouterwish I had more vram
>>110272867>>110272867
>>110272858NTA, when people in general say that they're afraid of pulling or afraid of git is, they're worried that unwanted changes upstream might impact negatively their current workflow.
