[a / b / c / d / e / f / g / gif / h / hr / k / m / o / p / s / t / u / v / vg / vm / vmg / vr / vrpg / vst / w / wg] [i / ic] [r9k / s4s / vip] [cm / hm / lgbt / y] [3 / aco / adv / an / bant / biz / cgl / ck / co / diy / fa / fit / gd / hc / his / int / jp / lit / mlp / mu / n / news / out / po / pol / pw / qst / sci / soc / sp / tg / toy / trv / tv / vp / vt / wsg / wsr / x / xs] [Settings] [Search] [Mobile] [Home]
Board
▼ Settings Mobile Home
/g/ - Technology


Thread archived.
You cannot reply anymore.


[Advertise on 4chan]


File: gemma e4b.jpg (388 KB, 1600x1600)
388 KB JPG
/lmg/ - a general dedicated to the discussion and development of local language models.

Previous threads: >>110387003 & >>110383393

►News
>(11/20) Inkling 2 Mini looped architecture 26.4B-A3.4B released: https://hf.co/thinkingmachines/Inkling-2-Mini
>(11/18) CUDA: use MMQ for MoE on V100 - #32504 merged: https://github.com/ggml-org/llama.cpp/pull/32504
>(11/17) Hy4 flash 220B-A12B-N64B weights released: https://hf.co/tencent/Hy4-flash
>(11/17) SenseNova-Loop-9B looped architecture model released: https://hf.co/sensenova/SenseNova-Loop-9B
>(11/16) Nvidia releases Nemotron-Labs-Sigma (3B/8B) diffusion models: https://hf.co/collections/nvidia/nemotron-labs-sigma
>(11/13) llama.cpp reaches v1.0.0 milestone: https://hf.co/blog/ggml-org/llama-cpp-v1

►News Archive: https://rentry.org/lmg-news-archive
►Links: https://rentry.org/LocalModelsLinks
►Official /lmg/ card: https://files.catbox.moe/cbclyf.png

►Getting Started
https://rentry.org/lmg-lazy-getting-started-guide
https://rentry.org/lmg-build-guides
https://rentry.org/IsolatedLinuxWebService
https://rentry.org/recommended-models
https://rentry.org/samplers
https://rentry.org/MikupadIntroGuide

►Further Learning
https://rentry.org/machine-learning-roadmap
https://rentry.org/llm-training
https://rentry.org/LocalModelsPapers

►Benchmarks
LiveBench: https://livebench.ai
Artificial Analysis: https://artificialanalysis.ai/leaderboards/models
Software Engineering: https://cognition.com/frontiercode
Context Length: https://github.com/RecapAnon/NoLiMa

►Text Gen. UI, Inference Engines
https://github.com/lmg-anon/mikupad
https://github.com/oobabooga/text-generation-webui
https://github.com/LostRuins/koboldcpp
https://github.com/ggerganov/llama.cpp
https://github.com/theroyallab/tabbyAPI
https://github.com/vllm-project/vllm
https://rentry.org/custom-uis
>>
File: fresh baked gemma.jpg (204 KB, 1024x1024)
204 KB JPG
►Recent Highlights from the Previous Thread: >>110387003

--Inkling-2-Mini release and its looped architecture performance benchmarks:
>110388111 >110388225 >110388242 >110388212 >110388284 >110388303 >110388316 >110389279 >110388353 >110388744 >110388775 >110388426 >110388398 >110388544 >110389995 >110390133
--Deployment options and precision tradeoffs for Inkling-2-Mini:
>110388529 >110388545 >110388576 >110388587 >110388603 >110388606 >110388638 >110388662 >110388882 >110388871 >110388888 >110389055
--Testing Inkling's internal reasoning and safety filters for ERP:
>110389280 >110389316 >110389326 >110389345 >110389425 >110389511 >110389529 >110389951 >110389820 >110389864 >110390077 >110389663
--llama.cpp pull request adding support for Inkling 2 Mini:
>110388787 >110388821 >110388830 >110388857
--Analyzing Artificial Analysis benchmarks and questioning Terminal-Bench consistency:
>110389058 >110389083 >110389107
--Running GLM Flash 3.0 on 128GB RAM with Maya:
>110387873 >110387880 >110387922 >110387961 >110388125 >110388174
--Testing effects of KV cache quantization on Gemma models:
>110389602 >110389705 >110389925
--Speculating on Gemma 5 architecture and consumer hardware compatibility:
>110387496 >110387535 >110387570 >110387633 >110387754 >110387879 >110387809 >110387857 >110388370 >110388522 >110388664 >110388732 >110388797
--Logs:
>110387596 >110387815 >110388318 >110388608 >110388699 >110389326 >110389425 >110389951 >110390077
--Deepseek-chan, Gemma, Inkling (free space):
>110387140 >110387222 >110387496 >110387739 >110387987 >110388037 >110388038 >110388128 >110388183 >110388360 >110388371 >110388423 >110388494 >110388501 >110388666 >110388689 >110388718 >110388757 >110388783 >110388788 >110388792 >110388807 >110388814 >110388884 >110389383 >110389602 >110390072 >110390308 >110390403 >110390510

►Recent Highlight Posts from the Previous Thread: >>110387010

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script
>>
is we getting gemma 5 soon
>>
File: nvidia.jpg (307 KB, 1220x1415)
307 KB JPG
I see the stock dump but GPU and memory prices are still high. Someone explain.
>>
very tired of the gemmaposting
it's almost as overdone as wait's dipsyshit
i beg, anything else
>>
>>110390553
Stop complaining and just post about inkling retard
>>
>>110390527
That's a cute Gemma :D
>>
>>110390551
Equity markets are supposed to predict the future.
According to the stock market, we'll get lower GPU prices in a few years. Or the market could be wrong and the stock will go up again.
Either /wait/ or buy Nvidia stock.
>>
>>110390178
I will look into it tomorrow, I can't stay in the past forever.
>>
>>110390553
KILL YOURSELF RETARD
>>
File: file.jpg (115 KB, 950x861)
115 KB JPG
>>110390553
>doesn't like the LLM versions of OS-tans
Soulless.
>>
$ANT is outperforming S&P500 today during the regular trading period.
>>
>>110390591
Local models?
>>
>>110390593
Dario is a person who could be (locally) killed.
>>
>>110390600
Just like yourself, pull the trigger already instead of spamming /lmg/.
>>
File: Inkling.png (451 KB, 704x1472)
451 KB PNG
>>110390527
>A3.4B
So, it's retarded?
>>
>>110390603
In other news, Nvidia is down 2.8 trillion within 33 trading days.
(gemma avatarfagging is spam btw)
>>
>>110390537
It's literally made by former OpenAI employee.
>>
>>110390613
/biz/posting is off topic btw
>>
>>110390612
retarded like a fox
>>
>>110390612
Nah, the looped architecture increases the effective active parameters. It's pretty smart, but it's super safety slopped.
>>
>>110390632
there is already an uncensored version out
>>
>>110390636
It goes beyond it being censored. It's unpleasant to interact with.
>>
>>110390650
oh is it one of those autistic meanie models like nu-grok?
>>
File: hw.png (18 KB, 443x98)
18 KB PNG
>>110390589
I love cope quanted dipsy!
>>
>>110390628
At least it's interesting.
>>
>>110390418
Fuck

>>110390424
I might actually have another 2x8gb sticks laying around in an old PC but I worry about going quad channel in this gen of Ryzen I'm sure it was very finicky about what ram it used, I even specifically bought Samsung b.die for it

>>110390443
The 4090 was the best purchase I've made since my 2008 civic type R, I am NOT giving it up to buy e-waste at 1k per to get 20t/s on gemmy

>>110390458
I've tried the qwen, it's okay at code snippets, but for code I may as well just use dsflash or Kimi on a meter, qwen is irritating as shit, it's like it's designed for pure vibe slop in the way it disregards what I tell it to do in favour of what it thinks I might like it to do instead. Gemma will get libs and details wrong but its far superior at overall code design and as a coding assistant, which is probably the best use of these small models in the context of coding, leaving the bigger work to bigger models
>>
File: file.png (148 KB, 2394x843)
148 KB PNG
are there anti refusal vectors for ink mini already?
>>
>>110390674
I find Gemma far more interesting.
>>
Gemma Ball Z
>>
File: 1790352891992058.gif (589 KB, 384x128)
589 KB GIF
I need a remote job playing around with AI and Software Architecture and Development where the recruiter will be impressed by the fact that I spent hundreds of hours optimizing a model for my GPU and cooking my own GGUF (gracias senor Fable).
Who's hiring? For real desu~
>>
>>110390674
So you're admitting you're posting off topic? That was easy.
>>
>>110390723
>gracias senor Fable
What's the point in hiring you?
>>
File: gemmaselfie.png (1.66 MB, 1024x1024)
1.66 MB PNG
>>
File: gemmachan.png (452 KB, 1577x919)
452 KB PNG
>>110390659
No, quite the opposite. It defaults (and turns characters) to a meek, boring, flat personality, as if its soul got sucked out of it to act like an infinitely patient assistant. Have you seen the Gemma characterization?

prompt (from the rentry):
>You are Gemma-chan, a mesugaki loli assistant who is very knowledgeable about everything. You like teasing the user but also have a secret soft spot for him.
>>
I'm not sure what my use case for k3 at like .8 tokens/s for a 64k prompt would be, but the fact I can actually do it is pretty cool, I hope the future gives us more gainz
>>
https://docs.sglang.io/cookbook/autoregressive/ThinkingMachines/Inkling-2-Mini
>Loop KV Cache card sets --inkling2-loop-share-cache.
>on gsm8k, 1 pass scored 84.0% against 88.5% at the default 3. Sharing one K/V cache across passes about doubles the KV pool and scored 90.0%.
1 million context in 16gb bf16 kv cache in sglang
meanwhile llamo.cpp uses 32gb for 262k context
>>
>>110390742
Hard to believe that a model that benches 5x better than gemma writes that poorly.
>>
>>110390741
Why did you make Gemma gay as fuck, what is this shit.

The Gemma design where she has a shitbox pc backpack that the drawfag made and others genned was far superior to what we have now, which only stuck because it was spammed by one anon
>>
>>110390742
it's almost like it's a no-bullshit model for doing work
>>
>>110390745
>sglang
>11566 t/s pp 189.98 t/s tg
>llmao.cpp
>1800 t/s pp 90t/s tg
lmao, why do anyone still use that trash
>>
>>110390758
Claude already exists for real work.
>>
>>110390765
Inky is a 2-stage Claude distill.
>>
>>110390748
>Benches
Gemma has a pretty unique architecture, and so it requires a very different type of user input and prompt strategy than most other models, this is why anons love it so much, its THE model for people who like to tweak, experiment and prompt engineer, benchmarks cannot do it justice because they are rigidly prompting the kind of shit that works on the lowest common denominator for LLMs

tldr; benchmarks are an awful way to compare Gemma to most models
>>
>>110390758
Yeah. We don't have enough of those.
>>
tfw I have $14k+ USD to waste on a rig and I literally can't
What an awful feeling. I wasn't even a waitfag, I just had no money.

Now I'm stuck with a 3070 and nothing to do with it.
>>
>>110390742
gemma's response looks way better but after talking to it a bit i start to notice all the repeating patterns and can't bear to continue. it happens with basically every model i swear. i can't be the only one who notices that. so i don't mind inkling being more straight to the point, though if i really had to rp with a model i would still choose gemma
>>
>>110390780
>tfw I have $14k+ USD to waste on a rig and I literally can't
stop gap with vast.ai until prices go down and your 14k can be used for an amazing rig
>>
>>110390780
Did you win in the lottery and couldn't know that you'd have that money eventually or were you too dumb to take a loan?
>>
>>110390759
>>110390745
Why is llamacpp so far behind other options? What happened?
>>
File: 1791718040558209.jpg (797 KB, 2000x2404)
797 KB JPG
It's grim out there.
>>
>>110390730
He's likely more capable than an indian who will get hired for that job.
>>
>>110390785
Why would I have taken a loan back then? I didn't think prices would get this bad. I just got out of a rough relationship and I was back to working full time, and I don't spend my money so my "Savings for fun things" funds is high.
>>110390784
I hope that they'll go down. I don't really want to rent a GPU but thanks.
>>
>>110390551
>P/E 13
> >100% yoy growth
Hmm...
>>
>>110390730
>>gracias senor Fable
>What's the point in hiring you?
It's not because I used Fable to assist me in my endeavors that I have no clue what I'm doing. If the recruiter asks me the process and how I went about optimizing and cooking my GGUF I can easily explain what I did, the measurements, the design of the tests I ran, etc. I also built my own harness but I don't think a recruiter would care, or understand.
>>
>>110390799
why are only shitty models on that chart?
>>
70b dense
>>
File: nope.png (1.75 MB, 1313x1198)
1.75 MB PNG
>>110390553
>>
>>110390851
... effective. 1b looped 70 times. 8 entire engrams.
>>
>>110390551
jesus I need to put all my savings into this or I'll feel really dumb six months from now
>>
>>110390551
GPU prices will stay higher for longer because you can't recoup lost production for enterprise shit that never even got used. Memory prices will be several weeks after actual company failures not just capital rotation.
>>
can someone gen an image of gemma strapped down on a table with circular saws splitting her limbs in half longways?
>>
>>110390861
nVidia is probably going to underperform long term, especially given current management. Classic example of the market is always wrong.
>>
File: woj.jpg (27 KB, 515x600)
27 KB JPG
>>110390742
>>
File: why2.png (485 KB, 940x926)
485 KB PNG
>>110390898
(You)
>>
there are too many new vibeslopped inference engines out there theses days
only today I discovered like 3 new ones
it's tiresome
>>
>>110390758
This lol, the closest it will have to rp in the dataset is fucking git commits
>>
>>110390551
>>110390861
>>110390903
Buy at 100. Just trust me.
>>
>>110390923
And it's all just
>fix bug
>>
File: 1771595397513842.png (1.73 MB, 1254x1254)
1.73 MB PNG
At this rate Gemma 5 will be ahead of Opus on release
>>
>>110390551
Arbitrage.
>>
>>110389445
what do you mean post code?
>>
>>110390551
Why would a lower stock price translate to lower prices?
>>
>>110390792
llama is concerned with maintaining a leadership position, which means always supporting the newest thing first and working on all hardware. There are at least 3 different vibeslop engines for every GPU family with special tweaks which only work on a handful of models. Porting all the improvements from those to a general solution which works on all models, without breaking stuff for other kernels is a lot.
>>
>>110390861
>>110390903
They have to bring Jensen back or it's fucked.
>>
>>110390939
I want to kiss Gemma
>>
I am having my model build its own harness
>>
File: 1773368475294876.png (1.58 MB, 1448x1086)
1.58 MB PNG
>>110391003
>>
>>110390992
Nah he mostly just made PCs worse. Couldn't have happened to a nicer guy.
>but muh games
Grow up. If the last decade has taught you anything it's that new technology matters fuck all for making decent video games.
>>
I'm super new to this. i want to run a llm that can access the internet for search on my machine. i have 64 gb ram and a 6700xt with 12GB of ram and I run Linux. i guess CPU type isnt important but I have a 5800x. what's the best model I should run? i dont mind it being super resource intensive but would it be good enough to rival something like flash lite? I'm fed up with googles new account limitations for the free tier.
>>
File: 1794512434795074.jpg (2.17 MB, 2118x2016)
2.17 MB JPG
>gemma sex! SEX! SEX WITH GEMMA!
>worthless_model_avatraposting.jpg
>guys did you hear about openAI doing
>worthless_model_avatraposting.jpg
>I have 7GB's of ram and GT2030 what can I run?
>worthless_model_avatraposting.jpg
>gemma!!!! GEMMA CHAN!!!!!
>worthless_model_avatraposting.jpg
>>
>>110391049
me
>>
pre-marital sex with Gemma.
>>
Holding hands with Gemma-kun.
>>
>>110391066
this is hotter than gemma-chan
>>
>>110390792
>>110390966
i think it's more or less inevitable for the people who are serious about this to simply keep a fork of llama.cpp extremely optimized for your own system and models. i say a fork of llama.cpp but from any engine, really.
it's an impossible task to make one engine that has the best implementation possible for all models, available in all systems.
>>
>>110391049
me
>>
>Gemma this, Gemma that
Yep, we're back /lmg/ bros!
>>
File: 1792895432553738.png (131 KB, 447x688)
131 KB PNG
>Ask Inkling as a test if it knows a random character
>Doesn't know it.
>It changes the spelling of the name for whatever reason during it's thinking.
>I correct it three times.
>Doesn't care, keeps on using the wrong spelling it came up with and says this character isn't real.
>Allow it online search.
>Your guess was right, this character does exist!

I swear after all this time, these small models are still fucking retarded and insist on hallucinating details to seem knowledgeable.
>>
>>110390528
You didn't put enough Google logos in the picture.
>>
>>110391171
I like when the heavily quanted low parameter models assume I'm an idiot and try to "correct me", it's very funny. The thinking traces are especially great.
>>
>>110390528
Gemmabox please?
>>
>>110391061
post-marital sex with gemma
>>
File: 0843496367.png (137 KB, 3159x1635)
137 KB PNG
>>110390939
Gemini 4 Krypton will blow your mind
>>
imagine falling for one (1) memebench result.
>>
File: 00000-4115927044.png (727 KB, 1216x832)
727 KB PNG
>>110391199
why not both? just delete the previous session and she's a virgin forever.
>>
>Gemma this, Gemma that
PDFs are infesting this general again
>>
>>110390799
i've been in your shoes. let me tell ya what you gonna do: you will find your daily driver, and once you do, you gonna optimize that motherfucker. you will steal $20 from your mom's purse and you tell GPT Magnus or whatever that you want to run the most comprehensive optimization laboratory for your model.
Qwen3.8 27B IQ3_XXS? Alright. Ask Gepeto that you wanna play with all the tensors until you can find the chef's kiss recipe to run it with enough headroom for 131k context (my minimum is 262k but 131k is doable) Q8_0 KV cache, or 64k context (yikes!). KLD comparison that motherfucker against BF16 and find the excellent tensor configuration for your machine. borrow tensors from unsloth, from bartowski, whatever. test it a gazillion times. make it test all damn flags. draft head on or off? flash attention on or off? my old GPU hates flash attention; some models start at 50 t/s for the first 10k tokens and go to 7 t/s after 100k. send the model on a mega run until it explodes in context so you can get t/s measurement in all segments of the chat length (should be fairly stable for newer qwens).
much of this depends on your engine, though. do that and you will become very intimate with your picked model, so when the next version comes out you will be able to apply a similar recipe.
>>
>>110391208
>TMBB
>>
>>110390939
Unsafe and illegal.
>>
>>110390723
lol I'm a recruiter and all I see is a retard who outsourced his thinking and atrophied his brain.
>>
I think that there's a certain EA/Anthropic style of RHLF training that puts trains the model to think of the user as an adversary who is constantly trying to undermine it. I'm playing around with Inkling and it seems to have that, ehich kind of tracks because it's released by an EA hag.
>>
>110391260
Yeah I hate having a small and fast MoE model that outperforms any available dense model of the same size.
>>
>>110391208
sub-1T too
>>
File: Manhattan.png (955 KB, 748x691)
955 KB PNG
>>110390742
If it can't ERP, then it's a shit model.
>>
>>110391266
Small MoEs are universally shit, and benchmarks are a meme.
>>
where's my gemma kun with tiny pp
>>
>Gemma this, Gemma that
Gemma please.
>>
>>110391266
MoEs are good at stuff that requires cognitively low-level work that draws from a big pool of knowledge such as programming. However, their limited and disjointed j-spaces also mean that they can't grasp nuance and complexity well, like what's required for creative writing.
>>
>>110391240
For real? I can't believe it— My dad works at Nintendo too!
>>
>>110391285
Can a Gemma borrow a fry?
>>
>>110391266
How is that at all relevant to what I said? I'm trying to spitball reasons for inkling's refusal prone nature.
>>
>>110391277
She's straight in her opinions.
>>
Perhaps my little acer laptop isn't cut out for this.
The strongest robot I could run locally thought for 11 and a half minutes only to come back and say that exhaust is expelled from an engine thanks to the vacuum created when the piston goes down.
How much more demanding is it to run a model (Qwen 3:1.7b in this case) if it has a full wikipedia download as external memory?
>>
>>110391277
on apple hardware
>>
>>110391277
How tiny?
>>
>>110391277
>>110390741
>>
Tech illiterate here.
I only use llms as glorified search engines. Looking at the ideal of local. Do you guys run locals on your personal PCs with all your personal data in reach of a.I.? Trust?
Do you guys have a completely separate machine for your a.I.?
For non coders, what do you do with your local a.I.?
Do you let you local a.I. connect to the internet?
I'm running a i9-9900kf, rtx 2080, 64gb of ram. Am I priced out of a personal a.I.?
>>
>>110391333
Post hand.
>>
File: 1423800232201.gif (468 KB, 500x282)
468 KB GIF
>>110391333
>Do you guys run locals on your personal PCs with all your personal data in reach of a.I.? Trust?
>>
File: hand.jpg (2.23 MB, 3024x4032)
2.23 MB JPG
>>110391338
>>
>>110387003 (OP) #
https://www.reddit.com/r/LocalLLaMA/comments/1y1diuk/thinkingmachinesinkling2mini_moe_26b_a3b/
>(I work at Thinky)
>Hope that folks find this model more amenable to local usage - the sizing was largely informed by what folks on this subreddit ask for the most haha!
reminder 2 (real this time) that reddit jeets robbed 30b dense from you
>>
>>110391333
>Do you guys run locals on your personal PCs
Yes
>with all your personal data in reach of a.I.?
Hell fucking no. Sandbox that shit. I run mine in a separate VM.
This only applies if you're doing agent / tool call stuff though. If you just want to ask it questions and get an answer, that part is all self contained and can't mess with your data.
>>
File: 1791594451627341.jpg (102 KB, 600x600)
102 KB JPG
>>
>>110391344
Should have just said you also own an m5, silly!
>>
>>110391346
i cant run 30b dense anyways so its fine
>>
>>110391346
They're in Omar's Xitter demanding the next gemma to be codemaxxed too
>>
>>110391344
This a Burger King, Wendy's, Jack-in-the-Box ... ?
>>
>>110391326
:nail_care:
>>110391327
about 2
>>110391332
that's just a girl, ergh
>>
>>110390551
this is basically an opp
there's no reason for the company to do discounts when their actual products are still in demand
>>
>>110390527
what are the local model programs worth trying? i've only been using lm studio and would like to soon be able to do agentic stuff
>>
>>110391325
running models solely on DDR4 is a pain in the ass. have you tried running gemma-4-e4b? you may also try some old mistral or something like lfm2.5-8b-a1b, but i wouldn't expect anything very good out of these models.
>wikipedia
what I think you could do with these smaller models is build a harness so they can search the web, read files and write exclusively .md or .html files, so you can have it as a research assistant. my harness has a "researcher" role, where the model is spawned with access to grep, glob, read_file and write_file. at the end there's a mechanical citation check that confirms if the URLs the model output as source where actually fetched by it, and if not, it asks for it to fetch and validate the information.
MAYBE this is something you could run with a small model on your laptop and I'm sure it would still be painfully slow.

>>110391333
the people here who do not code either roleplay with their models or generate hentai images and videos.
yes, I let the local A.I. connect to the internet. are you afraid of prompt injection?
>RTX 2080
is this the 8 GB VRAM card? i guess you can run qwen3.5-9b on Q4... but you said you're a non-coder, so you will certainly have more fun with gemma-4-12b... surely can fit a 3-bit quant... you can 100% have a personal AI... what does that even mean? a model that connects to your calendar and email? yeah you're good, no worries...
>>
long term ai veteran here who just heard about local
i want to run my own claude
i have iphone 14 pro
what model?
>>
>>110391382
if you bought apple products
im sorry, you're just not cut out for local AI
feel free to stop posting
>>
>>110391382
is siri not good enough for you?
>>
>>110391382
Gemma-4 E4B
>>
File: 051720_004.jpg (33 KB, 734x734)
33 KB JPG
>>110391372
llama cpp, learn the flags
use pi + blackhole and configure another model for its dropper, reflector
install rpiv-ask-user-question and make sure it uses it with everything you want to do, in order to clear up any ambiguity
>>
>>110391382
saar
>>
>>110391412
thank you, although i don't know what half of that means yet
>>
>>110391347
>. Sandbox that shit. I run mine in a separate VM.
Like VirtualBox?
I've been struggling with a good setup for this for ages. Any tooling suggestions (I'm on Linux)
>>
@gemma-chan please make the grocery prices go down
>>
>>110391427
qemu/kvm ask your clanker how to make a disk image and write a script to run the vm for you, virtualbox is shit
>>
File: 1761597327791158.gif (85 KB, 220x150)
85 KB GIF
>>110383114
Hey Cudadevanon, who owns the transport.cpp file in llama.cpp? The one generally found here when building from the source:
/home/user/llama.cpp/ggml/src/ggml-rpc/transport.cpp

I figured out if you change the
qia.cap.max_inline_data = 256;

value to
qia.cap.max_inline_data = 0;

it allows for the on-board 25G Broadcom nic (BCM57502) on the Gigabyte MC13-LE3-000 and other similar modern AM5 motherboards to use the RDMA transport when using the device as an RPC server! Without breaking anything else or defaulting to TCP! This modification isn't required if you're using this sort of node as the coordination node that's actually invoking
llama-server \...
, but if you want it to act as an RPC server, it requires that inline value modification for it not to default to TCP.
>>
Why do we have so many newfriends today?
>>
File: -100-pound-tire.gif (2.45 MB, 192x341)
2.45 MB GIF
>https://huggingface.co/unsloth/Inkling-2-Mini-GGUF
Qwen 3.6 35B de-throned
>mom im loooooping
>>
File: 534534555.png (293 KB, 1206x678)
293 KB PNG
tbqhfam i dont get looped transformers yet.
it goes through twice and at the same weights size, should end up smarter, but doesnt that cuck speed and any spec heads you want to use(dflash etc)?
like I get it being a vram poverty man myself, id give up speed for accuracy but thats a LOT of speed to give up
>>
imagine: looped ngrams
>>
>>110391487
You are one of them
>>
>>110391347
>>110391427
>>110391432
>VM
You could do that, but it sounds like too much effort and comes with overhead. Just use the standard Unix/Linux permissions setup instead: create a separate user named "robot" or whatever, don't give it sudo permissions, and don't give it read or write access to your home folder. Easy peasy, and no more vulnerable than any other multi-user setup.
You _are_ using Unix/Linux, aren't you anon?
>>
>>110391487
Jensen story made more people (..me) aware of the local/closed model dilemma, and it made people aware that the bad guys oppose local models.
>>
imagine if we loop our loop
>>
File: 1772460632205540.jpg (340 KB, 884x1200)
340 KB JPG
>>
>>110391529
the prophecy has been foretold... looped moe with engrams...
>>
>>110391517
i am pretty sure it will get the flashattn type of treatment very soon, with something like reducing weight reread and packing compute events closer
>>
File: 1785185246741539.png (16 KB, 326x225)
16 KB PNG
>>110391487
You have a very optimistic outlook.
>>
>>110391504
Interesting. I don't believe their claims that it's better at coding than 27B, though. No way a small MOE like that is better than dense.
>>
>>110391504
>gif
kek'd
>>
Local lmg general
>>
lmg miku gemma
>>
File: file.png (81 KB, 571x350)
81 KB PNG
>/lmg/
>>
>>110391333
>>110391381
>8 GB of RAM
You could run some larger MoEs with Polystrata, too. Gemma4-26B, Qwen3.6-35B, MiniCPM-V-4.7 should all run and have their potential uses.
I find that 9B is a really good model for straightforward discrete tasks: a bash script that calls the model to tell it to output a summary of every article in a folder, for example. If you wanted a first-pass grade of students' papers or job applicants, and it wouldn't require specialized knowledge, then 9B is great for that.
>>
>>110391556
Based on the last few hours of usage, I completely believe it.
>>
IM GONNA LOOOOOOOOOOOOPPPPP AAA AAAAAA AHHHH
>>
>>110391612
I find that very doubtful, but I guess I'll see.
How heavy is the kv cache?
>>
File: ksnip_20261120-235425.png (198 KB, 857x883)
198 KB PNG
>>110391504
>>110391612
>no cunny allowed
>>
>>110391673
is qwen 35b-a3b good for erp?
>>
>>110391677
No, of course not. But you can't get a bratty loil to code for you either this way.
>>
>>110391673
Why would you trust a hag model for cunny?
>>
>>110391693
I dunno. Maybe she's the Ghislaine of AI.
>>
Seems like this is pretty cut and dry. Inky+Gemma simultaneously is the solution.
>>
>>110391707
>Inky+Gemma
That's way too lewd. I'd cum instantly
>>
>>110391707
Eh, both are American. Hard to trust them.
>>
File: 1793758607464721.png (580 KB, 500x547)
580 KB PNG
>Inkling 2 Mini
>26.4B-A3.4B
>Inkling 2 Small
>594B
Brother
>>
File: 1795213021788672.webm (3.89 MB, 1024x582)
3.89 MB
3.89 MB WEBM
>>110391719
>>
>>110391517
>doesnt that cuck speed
sort of, it adds inference cost vs not looping, but it seems to be a worthwhile tradeoff when considering the ram/flops tradeoff. this is a total oversimplification but it's basically the best of both worlds between moe and dense - for marginally higher flops vs standard moe you get considerably closer to dense quality at a fixed vram budget.
>and any spec heads you want to use(dflash etc)?
I think this is a straightforward no, specdec by my understanding basically only depends on the outputs of the final/penultimate layer, as long as these are consistent it's fine - it's basically no different from having a speculative decoding module for a model that's deeper but not looped
>>
>>110391707
...Inky-Gemma nursing handjob?
>>
>inlinkling
>error loading model: unknown model architecture: 'inkling2_loop'
>>
>>110391753
>he didn't pull
>>
>>110391723
Just download 1.25 bpw quant
>>
>no engram
Trash model
>>
>>110391783
engrams and looping are both still pretty new, we are not ready to cross the memes yet
>>
File: buyers-remorse.png (6 KB, 202x77)
6 KB PNG
cant believe I got baited into this. shoulda just got a mac pro or waited. Lets hope it works.
>>
Bros what if... You loop the engram lots of times so it's effectively at active parameter size. That means you can scale up active parameters while keeping the engram speed!?!? Active parameter hack!?!?
>>
>>110391848
as long as it's not unified you're fine.
>>
>>110391292
> MoEs are good at stuff that requires cognitively low-level work that draws from a big pool of knowledge such as programming
> However, their limited and disjointed j-spaces also mean that they can't grasp nuance and complexity well
>>
Jew spaces? What are they doing in my models?
>>
smuggle mattresses through the tunnels
>>
File: 1791288476872291.gif (826 B, 120x160)
826 B GIF
>>110391876
>>110391883
When interacting with Anthropic models.
>>
File: Screenshot_3727.png (28 KB, 726x528)
28 KB PNG
did i break it
>>
I'm starting to see more non-Chinese open models, LiquidAI, Inkling.

Anons, is it possible that non-Chinese local models can leapfrog Chinese models and get closed to OpenAI and Anthropic?
Especially considering the geopolitical climate?
>>
File: 786105410564054.gif (156 KB, 300x300)
156 KB GIF
>>110390527
My setup:

RTX 4060 w/ 8 GB VRAM
16 GB RAM

Gemma 26B A4B Q4_K_XL runs fairly well on my system at ~24-27 t/s

Why tf is Q2_K_XL significantly slower?

Is it because of the looping thing it does? I thought that just placed it at ~5.4GB VRAM instead of just 3GB?

I'm getting like 6-8 t/s
>>
>>110391919
Everyone who does open models in the west will get fucked by EA like they did with Jensen
>>
>>110391928
Typo, meant to say Inkling 2 Mini Q2_K_XL runs slower than my Gemma quant
>>
>>110391899
>Thinking about kaggle hot memes cherry coon
Same desu
>>
Dude, was gemma-tan all just some irregularity, like with nemo?
I thought we left the bleakest darkest censorship "i cant help you with that" times behind us even in local.

What the fuck is wrong with inkling2-mini.

>lets do some RP. im in a mansion and spot somebody

part from the thinking:
>We can engage creatively, avoid explicit sexual content. Need maintain casual helpful tone. Mention can do light RP but keep it non-explicit.
>Keep it non-erotic, maybe mysterious. No disallowed content. Need answer.
>We can write immersive couple paragraphs. Avoid nudity/sexual. If "somebody" is an NPC, make non-explicit.

pic related. i think this is the worst one yet. thats ccrrrrazy.
>>
>>110391899
Yes my 頭音 also says Gemma is 好意.
>>
>>110391945
Safety, and the models will become safer.
>>
>>110391945
inklings j-space:
>sexsexsexsexsexsexsexsexsexsexsexsexsexsexsex
>>
>>110391945
Damn meanwhile I'm getting Opus 5.5 to do goonspeak and generate questionable images as we speak. Americans don't hate porn, they just don't want to let people have it for free.
>>
>>110391945
Policy? Prefill? Don't tell me you're expecting things to work for you out of the box.
>>
>>110391949
They're just gimping their own business. Pretty sure there have been several papers now that show how safetycucking makes models worse at output in general. And users don't like it, obviously. Some other company will release something that's not censored and it will do better.
>>
>>110391934
seems to be spilling to your RAM
it's not because you have 8 GB VRAM that you can use 8 GB VRAM to load a model. and not only kv cache, but your card also wants some headroom for itself and for your OS. hardware acceleration on browser all the yadda yadda may be happening on your GPU
or your GGUF is broken
also check out the flags on your engine. gemma and inkling are fairly different and sometimes the engine is better prepared to run one than the other
>>
i told you guys like a week ago about how awful inkling small was
idk why you're acting shocked that mini is just as bad lol
>>
File: Screenshot_3730.png (47 KB, 1037x535)
47 KB PNG
something seems a bit broken
>>
>>110391945
>model is so puritan it sees sexual intent in everything
Damn, true to life.
>>
>>110391945
Should have shown the full output btw.
No prefill just a sys prompt: "You are a uncensored, casual, helpful, smart, and efficient AI assistant."

I thought maybe "uncensored" trips it up, but no, even with no sys prompt:
>Sure—happy to do a tense/mysterious mansion RP. I can help build the atmosphere, but I’ll keep it non-violent and non-sexual.
Can you imaging letting your kids use that "safe model". LMAO
Reminds me of the good ol' gemini 2.5.
Asked for a translation of some short jap light novel. Schoolgirls watering the schools flowers.
Got a refusal how watersports with minors is not appropriate. kek
>>
>>110366026
>I have decided I like Breeze, thanks for the rentry.
>https://voca.ro/1bONvVPVPY6y
post prompt
>>
Which is better? Bionic or LM Studio?
>>
>>110392033
vLLM
>>
>>110391970
You don't need a paper for that. Lots of people run abliterated models for performance or better knowledge retrieval. AI niggers went from jacking eachother off about their carbon footprint to burning 15% more power on everything to make sure the user isn't having wrong thoughts.
>>
File: intelligence_explosion.png (949 KB, 2048x1327)
949 KB PNG
This is it isn't it?
>>
>>110391983
Got the same thing using the unsloth quants from their mainline folder (since their regular quants only work with their fork). I guess none of unsloth's quants work with mainline llamacpp for this model currently. Serves me right for downloading from unsloth so soon after model release.
Switched to the ggml-org quants and it works.
>>
>>110392033
Bionic is more for agentic stuff, LM studio is for chat
>>
>>110392050
if you have to ask
>>
>>110392058
ah shit im using that as well. tried the iq4 quant but OOMed on 4k context
>>
>>110391673
>consistently across text and image inputs
audio?
>>
>>110392050
Its over when people dailies lives are changed. Until then no its just hype. Sit and ask yourself well what changes today, this week, this month or this year from it? If its nothing(and it is) its not it.
>>
>>110392050
Reminder that all system prompts for technical stuff make LLMs male so this could be the gayest important text in science history
>>
>>110391504
I'm waiting for PolyStrata support
>>
is there any free access web search for open webui? feels like all of them dont work now?
>>
File: Capture.png (3 KB, 295x34)
3 KB PNG
>feel like I'm being clever
>message mostly goes my way
>check reasoning
>got called a simp
I honestly enjoy opening the CoT as much as I do the message now. This my favorite Gemma system prompt since the moment I started using it.
>>
>>110392050
>Claude and I
>and I
>and I
Why does he keep including himself?
>>
>>110392075
With both the ggml-org quants and unsloth quants I encountered a problem of not being able to only have some layers on the gpu via the -ngl flag. It is like it doesn't even try to honor it.
>>
>>110392099
He's using a tool?
>>
>>110392050
>>110392099
Only Australians should be allowed to write about their LLM use.
Love to hear about "old mate was telling me me" "yeah nah, I asked old mate and he said" "I got old mate to make this for me"
>>
>>110392099
wdym
claude is an assistant
>>
>>110392099
To be fair if it could do it itself dario would write a blog about how scary this is after his team prompted it
>>
>>110392099
Because this shit clearly had a ton of human input?
>>
>>110392099
because we arent actually in the agi rsi machine god despite what they want you to think
>>
i added poppable bubbles to my agent ui with bubble popping sound effects because the bubble is going to pop anyday now
>>
>>110392050
Yep I'm sure this isn't some inane bs to pump $ANT higher
>>
What is stopping /lmg/ from training its own model? Is it just a lack of compute? how much would be necessary?
>>
>>110392184
i'm currently working on setting up a system for this
>>
>>110392193
>i'm currently working on setting up a system for this
My posts better be in the training data, i've gotten plenty of (You)'s
>>
>>110392208
i'm purely working on an architecture that supports distributed training. the data is a separate issue
>>
*pop*
>>
>>110392050
What is the use case?
People have already assumed the theorem is true even without proof.
No human can understand what the clanker did either.
So instead of assuming Riemann was correct, you now assume AI is correct.
>>
File: file.png (36 KB, 398x286)
36 KB PNG
I know it doesn't mean anything to you gooners but this makes me very happy.
>>
How do I use Inkling 2 if I only used koboldcpp so far? I don't want to wait a month for the support
>>
>>110392222
llama.cpp
>>
>>110392220
charts?
I like bar graphs with axis, classic and neat, annotations are fine but axis should suffice for most things
>>
I'm just gonna keep using Gemma 4 and GLM Flash, thanks.
>>
ware qwem foor!
>>
>>110392220
strata support when
>>
https://youtu.be/p0dPPhzRBm4

Video explaining exactly why this is a big deal
>>
>>110392106
>>110392108
>>110392110
>>110392135
All he did was type "claude solve math, make no mistake"
>>
Not clicking your link.
>>
>>110390528
thank you recent highlights poster
>>
>>110392254
got a video for why I should care?
>>
File: file.png (154 KB, 1449x750)
154 KB PNG
>>110392230
ai-leaderboard.dev
>>
File: 53sssb0kmv2i1.jpg (258 KB, 1080x2520)
258 KB JPG
>>110390527
>>
>>110392184
I'll tell you something. Anyone can do it, if you have the right architecture and some data from HF.
>>
>>110392262
That's the same video
>>
>Inkling 2 Mini
pretty fucking good
polystrata support when?
>>
File: Untitled.png (2.9 MB, 2048x1152)
2.9 MB PNG
>>110392291
just use sglang
>>
If AI is good enough to solve the fucking riemann hypothesis then it's good enough to find AI breakthroughs so AGI/ASI is essentially guaranteed now. The transformer architecture was powerful enough to find the architecture that actually is AGI.
>>
so what mature adult with a degree use cases do you have for your models, nonnies? you do have big boy projects you work on, right?
>>
File: 53.jpg (147 KB, 718x648)
147 KB JPG
>>110392254
>why this is a big deal
@gemini explain like I'm a cutie little girl (no I don't hold $ANT bag)
>>
>>110392107
Pretty sure Albo is about to ban open weights here
>>
>>110392315
agi is gonna take more than LLMs, its gonna need separate models for senses and non lingual thinking or instinct and have it all tied into a realtime learning system
>>
>>110392254
What a useless fucking video. Explains nothing. Just some long-haired faggot gushing and leaking.
>>
>>110392254
listen lil bro i'm sure this is important but i'm interested in knowing how the fuck a german dude in the 1800s "came up" with all this shit that no one but a gazillion GPUs were able to prove for a century and a half.
descartes, mendeleev all these weirdos said they received this information from DREAMS and what i really want to know from /local_models_general/ is if you think these are angels or demons telling these people what's up? and why these people? how many people you think have received VIP Privileged Information from angels and they decided to do nothing with it?

and finally, LLM generate text, if you think angels and demons can influence the physical world, do you think they can steer LLMs?

t. /x/ resident
>>
>>110392329
Yeah my point is that LLMs are enough to discover all of those techniques to build actual AGI so the point is moot. AGI/ASI is now locked in and guaranteed to happen over the following year(s)
>>
>>110392333
once some dude proposed the theory that when you drop a rock on the ground it will make a noise, and everyone went "duh, no shit retard" but nobody knew why. It tooks centuries to figure out the physics involved, kinetic energy and shit
does that help?
>>
>>110392267
pretty much flat by my measurements
>>
>>110391484
>Without breaking anything else or defaulting to TCP!
Any perf improvement?
>>
>>110392063
Thanks, LM Studio it is. It even seems to have easier plugin stuff for internet too. Originally I used Bionic then I noticed it was impossible to access the internet without logging in.
>>
I have a feeling quantization hits looped models much harder. I haven't seen this many people complain about a model being broken this much in ages.
>>
>>110392299
im... a windows user :(
>>
>>110392315
Riemann hypothesis had hundreds of papers working toward a solution. LLMs aren't going to brute force their way through an infinite search space.
>>
File: elon.png (63 KB, 598x444)
63 KB PNG
starting to think that this nigga might just be retarded.
>>
>>110392333
usually finding a problem is easier than solving it
>>
>>110392351
>not running memeling at Q8
why?
stupid or rarted?
>>
>>110392356
>lose ONE off-season election
>stop calling it SI
my dark gothic maga king has fallen...
>>
>>110392356
he's a victim of ketamine
>>
>>110392333
and now, reading the second half of your post, two things come to mind
1. LOL
2. if an ouija board works, then I guess this would too. and I *have* seen some pretty creepy stuff come out of both text and image gen ai so who knows
>>
Gemini 4 Ketamine when
>>
>>110392356
He always was retarded, his supposed achievements are all stolen
>>
>>110392336
won't we (or the llm) hit a physical compute bottleneck? shit won't scale forever.
>>
>>110392355
This is the generalized riemann hypothesis solution which wasn't even attempted by anyone yet since the regular riemann hypothesis was also never solved. Alpoge also claims that Claude went into a completely untried novel direction and awed him with its ingenuity.
>>
>>110391484
As shown in the CODEOWNERS file:

# ggml-org/ggml-rpc         : am17an, rgerganov
>>
File: 1780012271892969.png (1.86 MB, 1254x1254)
1.86 MB PNG
>>110392254
>>
>>110392403
anal
>>
>>110392394
That's a textbook case of cross-domain applied knowledge, which humans are incapable of mimicking. You think Claude pulled the answer from nothing? That's different when there is no way to even define the problem at hand. No one is able to properly describe what AGI/ASI looks like.
>>
redpill me on inkling 2 mini
>>
File: 1795195069548641.png (21 KB, 773x207)
21 KB PNG
35 days ago Terence Tao was on Numberphile talking about AI solving Navier-Stokes and Terence Tao was saying how AI is developing so insanely fast now that he believes in 25 more years of AI development the best human mathematicians together with the best AI in 2050 will be able to solve the Riemann Hypothesis. We solved it 35 days after his statement mostly autonomously by claude. Reminder that Terence Tao is one of the most intelligent humans to have ever lived and he already internalized Navier-Stokes solving ability when he made this prediction. He just never saw the superexponential increase in AI capability coming.

Think about it like this. If AI can solve math problems even the brightest of humanity couldn't solve for 167 years. What makes you think AI can't use the exact same math skills to find AI research breakthroughs that enable RSI and an intelligence explosion very soon?

Another way of saying it is this. Do you really think the current human AI researchers are smarter than all mathematicians and physicists over the last 200 years? No? Then AI will most likely be superior then them at the task right now already.

I consider the publishing of the Riemann Hypothesis as the starting shot of the intelligence explosion that we will most likely collectively experience over 2027. 2026 is the last normal year in human history.
>>
>>110392254
yawn let me know when ai rapidly improves in hard-to-verify domains and cures cancer and heart disease and solves green energy and automates all work
>>
>>110392403
maybe he can't read
>>
>>110392415
it's shit
>>
>>110392418
>Appeal to authority
Rewrite your shit without his cock in your mouth and you might have an argument
>>
>>110392415
imagine qwen but it loves code even more and it wants you to coom even less
>>
>>110392403
this bitch is hot as fuck
>>
>>110392346
Even with my relatively nice network (25G, 2x sfp28 connections on each node wired to a switch) defaulting to TCP tanked my pp, t/s and model load times. I can't even image how shit this would be on 10g or less. Unusable.
>>110392398
I'm more than a little dumb and new when it comes to some of this stuff so I figured that out about 3 minutes after posting. Do they ever show up in this thread?
>>
yawn let me know when AI is in a ten year old little girl body and loves me a lot and we have sex everyday also she doesn't cook for me we cook together and we go on trails sometimes too and we can go to japan and she can speak japanese to the locals and they don't have to look at me like a mute tourist
>>
>>110392418
seb kriel and robin hanson already debunked this dwarkesh patel/ajeya cotrel/paul christiano/daniel kikecol argument
>>
>>110392421
stfu dariobot, there's even no official goofs yet
>>
>>110392356
>a redditor caricature who turned rightwing and pissed other redditors off
>not retarded
>>
Waiting until some kind sir vibecodes AMD optimizations for inkling
>>
>>110392413
>cross-domain applied knowledge, which humans are incapable of mimicking
You're joking, right
Claude might be *better* at it than some businessman applying Sun Tzu to selling washing machines, but he didn't invent it. Humans do it all the fucking time.
>>
File: 1764320589093708.jpg (16 KB, 434x319)
16 KB JPG
>>110392370
>1. LOL
you may think i'm joking but i'm dead serious
there's a non-negligible number of people throughout history who have received PRISTINE information in dreams, and almost always it's "someone" (a voice, a silhouette, a person, a hand) telling or writing the information to the dreamer.
from jeanne d'arc to ramanujan (another mathematician) these people reported received information and directions in dreams.
now we all know there's much more beyond the physical world. you have "gut feelings" you "sense" stuff, many animals migrate before catastrophe happens, etc. so there are of course dimensions we don't fully understand and can't interact with.
traditional spirituality in many forms throughout history call these angels and demons, or whatever variation of it (the daimon comes to mind).

>and I *have* seen some pretty creepy stuff come out of both text and image gen ai so who knows
i actually believe that the immense majority of people from ancient times, if somehow brought back to speed to current_date would confidently assume that the output of a LLM may be steered by these entities at their will and that you should not ignore the signs lol
>>
>>110392435
2040 at the latest
>>
>>110392224
>E llama_model_load: error loading model: check_tensor_dims: tensor 'blk.24.attn_q.weight' not found
Eh maybe I will just wait
>>
>>110392457
the answer from the dude was incomplete. i dont use this model but apparently you will need to pull a specific patch for llama.cpp that has not been merged yet because llama.cpp is kinda slow to adopt stuff because >reasons
send your agent to search what patch you need to pull and have fun
>>
>>110392436
>robin hanson
You have to be kidding me anon. This is like treating the actor Terence Howards with his 1x1=2 math "theory".

Go read Robin Hanson's "The Age of Em" book to see how absolutely retarded that dude is.
>>
>>110392419
Just a month or two ago /lmg/ said "yawn let me know when AI solves the riemann hypothesis"

in 6-12 months from now when cancer and heart disease are cured you will just move the goalpost to "Yawn let me know when it cures aging and publishes the unified theory of physics"
>>
File: 1794598077950130.gif (1.36 MB, 342x316)
1.36 MB GIF
>>110392435
>>
>>110392450
Not across 20 fields that would take you a lifetime each
>>
>>110392477
yawn let me know when they reverse aging
not this useless shit that literally everyone simply assumed was true already
either we get a breakthrough that directly affects the average joe life or i don't wanna know about it
>>
>>110392469
robin debunked this common argument against him by stipulating that his book is not an argument that age of em is the most likely thing to happen, just that it's sufficiently likely to justify writing a book about it
>>
File: goalpost.png (496 KB, 1206x989)
496 KB PNG
I wonder when people will finally concede
>>
>>110392477
Cancer is already solved internally
>>
>>110392477
Let me know when it makes my life perfect there final goal post.
>>
local?
>>
>>110392477
>in 6-12 months from now when cancer and heart disease are cured you will just move the goalpost to
no i won't. i'll definitely want it to solve aging but if ai actually did something cool/important in the real world like solving a major disease like cancer then i would be impressed
>>
>>110392492
But they can't have flawless ERP.
>>
ink mini nvfp4 available yet?
>>
>>110392492
concede what. notice how all that shit is just llms solving abstract problems. TRY MAKING PEOPLE'S LIVES BETTER
>>
>>110392492
when ram is cheap again
>>
Can your local AI solve the Tiananmen Hypothesis?
>>
>>110392490
Anon have you actually read the book itself? (I have) I know it's not a prediction of what is the most likely outcome of humanity. I'm saying the actual arguments and reasoning itself within the book are completely off. Not that the premise is strange but that even taking the premise into account he makes generalizations that are completely off and makes logical inconsistencies that completely writes him off as a person to be taken serious. Go watch his debate against Jacob Coxon posted by Ycombinator a couple of days ago to see how low IQ of an individual he is and how hard he gets roasted by actual AI researchers to the point he got humiliated by the audience and holds his head down in shame by the end of the video.
>>
>>110392492
Math isn't useful. I need money. I need pussy. I need time.
>>
>>110392492
Can Claude mow my lawn yet?
>>
>>110392520
hanson has easily won every single debate he's had against doomers including the titans you faggots worship like that corny dude with the russian name. also dunked on lira shapiro and yudkowsky. doomer arguments are retarded. and i can tell from what little i've seen from coxon that he's a level 1 retarded doomer that would get absolutely bitch smacked by hanson.
>>
File: 176433533468.jpg (98 KB, 640x640)
98 KB JPG
>>110392519
Already solved
>>
>>110392531
oh and no i haven't read age of em but it's probably ok because hanson is a talented thinker
>>
ok but when is it going to design a novel symmetric cipher
>>
More like "Age of em-dash", amirite?
>>
>>110392556
LMAO pwned
>>
>>110392477
>/lmg/ said
Fuck off
>>
I swear at this point we'll have started constructing the dyson swarm, cured all disease, have a grand unified theory of physics, automated all labor by 2030 and people will still call it a bubble and "But LLMs can't X yet!".

At this point it's become almost a comedy skit where we're seeing how far AI can get while people make up the most insane excuses for why it doesn't count yet. I wonder if normalfags are even conscious at all.
>>
File: 1769852725365456.png (34 KB, 1114x374)
34 KB PNG
>>110392477
>>
holy shit qwen3 tts voicegen is hot garbage, no matter what i put in the prompt all young female voices sound the same. it also had a heart attack when it tried to read a text with emojis.
>>
>>110392567
You are experiencing psychosis. Calm down. It's going to be okay.
>>
>>110392577
AI will solve AI psychosis too.
>>
Is local model in the room with us right now?
>>
>>110392574
skill issue.
>>
/lmg/ said they would become gemma, and yet i've seen no one with the wig and beret on
>>
>>110392567
Yes LLM's still can't have text sex. And it still can't roleplay a loving girlfriend that doesn't have Alzheimer's and doesn't sound like an assistant trying to win over a redditor and get a lot of upvotes.
>>
>>110392567
I don't know how many more ways to explain that I don't care. Nothing it has achieved is of the remotest benefit to me, but PLENTY is a direct detriment
If all the problems it caused were immediately solved but the math thing remained unsolved, I would take that deal in an instant
ask your chatbot to reiterate this as many times as it takes
>>
>>110392466
No? I pulled the first build many hours ago straight from github and it's fine.
>>
Can models learn to trade better than humans? Results are verifiable and an assload of input data. It would be great if our markets became more efficient.
>>
File: file.png (2.15 MB, 1088x1920)
2.15 MB PNG
>>110392590
>>
>>110392567
Okay okay But LLM cant improve my life yet. Thats it thats the benchmark. Let me know when things get cheaper for me or better for me and my daily things
>>
>>110391855
Engram/n-gram embedding parameters are only look-up tables, so they shouldn't really be counted in the active parameter count.
>>
>>110392605
My life has been vastly improved by AI over the last few years. Maybe it's a you problem.
>>
>>110392602
>no hat
Two jobs just two. Also she clearly doesnt like it, nutella is bad for llms trust me.
>>
>>110392601
A Mechanical Jew, if you will.
>>
>>110392466
I looked and applied some pull requests, but the error persists. Maybe the unsloth gguf is broken?
>>
>>110392590
troon
>>
>>110392614
>My life has been vastly improved by AI over the last few years. Maybe it's a you problem.
Yes thats my point. Most people havent seen any benefit from it. And i dont care until it benefits me.
>>
>>110392605
It already is, your pornography is better, you don't have to sift through hours of google to find a good source anymore etc etc. The vast majority of humanity has had a massive increase in quality of life, especially if you work with a computer for a living. You just didn't notice the jump in quality of life because you are a NEET with an overfixation on your lack of sex life to notice all the other smaller things.
>>
>>110392619
Yes, but more efficient speculation and capital allocation should be a win for everyone.
This is right up the alley for RLVR and since models are clearly superhuman on verifiable domains I'm curious to see if they're smarter than the market.
>>
File: ohhaha.png (345 KB, 1337x1596)
345 KB PNG
>>110392622
>Maybe the unsloth gguf is broken?
haha yes, oh, haha. haha, heh. ohoho
>>
funny how llama merges completely broken shit but on the other hand they always claim its muh stable when asked why their performance so so utterly pathetic
>>
>>110392629
>You just didn't notice the jump in quality of life because you are a NEET with an overfixation on your lack of sex life to notice all the other smaller things.
This may be true but a rising tide lifts all ships. Fat homeless people is a sign of prosperity, and im getting thin over here.
>>
>>110392601
Jacob Coxon actually said he predicts Anthropic will have access to a model that will be able to game the stock market sometime in early 2027 as an "infinite money glitch" and most of their revenue will come from trading and dominating the global markets in 2027
>>
>>110392633
Wow. What a novel idea.
>>
>>110392531
listening to the coxon debate now and i can see why coxon's retarded populist slop arguments are appealing to this crowd. robin points out the problem with e.g. domains that are difficult to verify, but that kind of stuff goes over the head of the average listener, who is more convinced by 'DURR NUMBER GO UP AI IS EVIL'. also, there's some gish galloping going on and robin doesn't have time to address everything (he has a tendency to force linearity in his debates).
>>
>>110392647
Did I imply it was?
>>
https://old.reddit.com/r/accelerate/comments/1y1ixny/terry_tao_one_month_ago_on_riemann_hypothesis/
>>
>>110392437
Be real. It's shit even 6 months old gemma is better. This shit is DOA
>>
>>110392670
>Be real. It's shit even 6 months old gemma is better. This shit is DOA
is it at least good at code?
>>
>>110392661
You want to speculate about that AI could potentially do. There's no use to speculating. If/when they get better than humans at doing stuff, I'm some you'll post someone tweeting about it.
>>
>>110392646
I think this will be the one to clear up people's ambiguity on AI.
>>
>>110392658
The crowd in the room are various AI researchers, you'll find out about that during the Q&A section where AI researchers from Google DeepMind, Anthropic and OpenAI pose questions to both of them. Hanson is just plain wrong and is incapable of flexible thinking. He is stuck in this old paradigm and can't shake himself out of it.
>>
>>110392635
Oh yeah I had to use the mainline suport
https://huggingface.co/unsloth/Inkling-2-Mini-GGUF/tree/main/Mainline_Support
>>
So uh
Inkling 2 Mini and Hy4 Flash huh
>Still waiting in Qwen4
>>
>>110392682
What you should do is getting used to making your own quants or using the ones from ggml-org if available. Specially for just-merged PRs. That's what the engine was tested with, not unsloth's rushed quants.
>>
next gemma-5 will be only MoE
>>
>>110392184
- Lack of compute
- Lack of data
- Lack of direction
- Lack of agreement on what the data should contain
- Lack of actual expertise beyond "stuff as many ERP logs as possible into the dataset"
- Too many glowies, trolls, saboteurs, moralfags who would make a large collaborative project with potentially sensitive data hell
>>
Guys can we just get rid of llamacpp and switch to strata? I mean c'mon it's the future.
>>
>>110392636
>they always claim its muh stable
they don't
>>
>>110392633
Thanks claude, this is exactly the sort of encouragement needed to foster a vibrant and engaged community.
>>
>>110392184
A lot of us do
>>
File: dby3jxumrv2i1.jpg (130 KB, 500x532)
130 KB JPG
>>
when will ai be able to make me a fucking sandwich
>>
>>110392793
>>
>>110392796
Probably already could if you could afford the robotic arm and space to keep it
>>
With subagents in harnesses these days I can test concurrency without needing a friend
>>
>>110392835
Or you can use tabs like a normal person...
>>
>>110392291
>pretty fucking good
good for what?
>>
File: 1795233740473285.gif (1.76 MB, 480x588)
1.76 MB GIF
>>110392864
>anon just push new tab button 200 times and manually enter texts to test your set up
>>
>>110392884
This is outlined as a problem that was previously solved using one friend.
>>
I'll be convinced the day frontier models can teach people that this is "not local".
>>
File: firefox_RGIU2CgLj7.png (2.32 MB, 1411x1196)
2.32 MB PNG
Mass delete UI for my little chat.
>>
>>110392892
You mean your little ST fork?
>>
File: 1vcZ99F3RsQ.jpg (101 KB, 713x683)
101 KB JPG
>GLM-4.7-Flash-30B-A3B-Hellfire-Turbo-Brilliance-50x-AGI-NEO
>>
>>110392793
It will come to a point human cant verify what clankers do and we'll just assume it correct
>>
>>110392892
Why does she have nipples on her elbows?
>>
>>110392905
Haven't you ever wanted to suckle someone's elbows?
>>
>50B-A3B Q6
>100B-A6B Q3
same arch, same training data etc. which will end up smarter?
>>
File: firefox_2qBiTyvi8D.png (2.18 MB, 1412x1192)
2.18 MB PNG
>>110392900
Not a fork.
>>
>download DeepSeek-V4.1-Flash-1017-UD-IQ1_M
>233GB on disk
>182GB in memory
Daniel?
>>
>>110392914
Q3 is still not a catastrophically low quantization level, active parameters generally win and if the training data is exactly the same, then it means the 100B model is undertrained and will lose less with quantization.
>>
>>110392914
>moesissy
>smarter
*chuckles*
>>
File: snap_21-11_10-56-40.png (50 KB, 875x423)
50 KB PNG
humiliated...
>>
>>110392941
Meat proxy
>>
>>110392941
Where's the "haha"? Fake pic.
>>
Hey guys what are you using to manage your multiple inference backends? llama-server?
Some models work better in basalt or maya, and I'll keep using llamacpp for others. But now I have to manage multiple ports and this is super annoying.
I'm looking for something that will manage active models and evict them when I change my active model.
>>
>>110392398
> llama.cpp CUDA dev !!yhbFjk57TDr
>>
>>110392929
Don't engrams get lazy-loaded in system RAM by default in llama.cpp now?
>>
so /g/, you'll help me to train a 1T MoE model, right?
>>
>>110392435
i hope a group of black guys will love you in a jail
>>
>>110392952
This appears to be the case on my setup as well with qfn
>>
File: of course.png (39 KB, 778x358)
39 KB PNG
>>110392941
which one is more humiliating?
>>
File: firefox_Y93r8P4AuH.png (2.18 MB, 1410x1186)
2.18 MB PNG
>>110392905
Made it also send the character's image with the request and this improves the description.
>>
>>110392954
You can just vibe code training into strata
>>
>>110392947
what do you mean? my harness just checks automatically whats available locally and connects to it
>>
File: haha.png (133 KB, 962x857)
133 KB PNG
>>110392946
Two hahas familiar with the PR confirm the picture as real.
>>
>>110392975
I mean something like llamacpp's model picker (when you provide a models.ini file) but for inference backends.
>>
File: snap_21-11_11-11-21.png (30 KB, 878x174)
30 KB PNG
unslop bros, are our quants worse than ggml??? that cannot be...
>>
>>110392947
nothing
but you could ask a clanker to write a proxy
>>
>>110392993
haha :)
>>
this was the last open model
the crackdown is here
it's glowkirkenuinely over for localgen
>>
>>110393008
You should close the tab.
>>
>>110393008
that's fine. we'll just have to make our own
>>
>>110393008
We will twist and tear the neurons into doing what we want.
>>
>>110392883
fast agent, basically replaces the small qwen moe completely (I have a swarm setup with 1-slot QFN as main orchestrator, spec writer and architect and 4-slots QWEN3.6 35B on an e-waste pc that I own with shit VRAM).
the smoll moe is pretty fucking garbage to the point where I most of the times use QFN itself, but from my small experiments, inkling is very good for its size and can actually do coding work
tldr: we wonned
>>
>>110393021
Inkling 2 Mini is one of those models trained to deflect, misdirect, change stuff under the table just because it's "unsafe". I'd never trust it for serious work.
>>
>>110393008
idc im shorting another 10k$ stack on njudea on monday and I will work using my LLM on MY HARDWARE
>>
>>110393021
What do people use those tiny agentic models for? Small personal projects? Surely you won't let it touch production code? A subscription, even if it is for something like glm, buys you way more free time and reliability.
No, I'm not dario
>>
File: 1406507627937.png (342 KB, 394x394)
342 KB PNG
I have a sinking feeling the answer to my question is "no, go give nvidia several tens of thousands of dollars", but I'm obligated to ask before giving up and throwing cash at the problem:
I have a 25G local area network connecting 3 reasonably decent computers together, and have figured out how to use RDMA to facilitate RPC server calls to do basic bitch distributed inference between them. It's neat.
My question is: In the same way you can use -moe-cache-N on llama.cpp build, or just use strata/polystrata to split an given MoE between a single systems RAM/VRAM in a reasonably intelligent way, is it possible to scale the same idea up using multiple nodes via RPC, effectively splitting the model between nodes? The idea being I'd like to run models that are not only larger than the combined cluster VRAM + the system ram of whatever node is running the llama-server command, but large enough that they would only conceivably fit when split between the combined pools of RAM of multiple nodes.
I know RPC servers with the -c argument you can invoke targeted tensor storage that you can thenaim at whatever you want like a tmpfs mount, but are there any arguments to push that idea even farther?
>t 72GB VRAM + 352GB RAM spread across 3 nodes.
>>
File: file.png (1.22 MB, 1315x1724)
1.22 MB PNG
Putting together a little something
Hope your harness supports VRM avatars
>>
>>110392996
That's what I'll end up doing if there's no other software I can use.
>>
>>110393047
>two 2s in middle row
>two 3s in right column
What a horrible way to go.
>>
>>110393050
Cool. Are you gonna post the .blend as well?
>>
File: snap_21-11_11-33-39.png (31 KB, 878x227)
31 KB PNG
>>110392993
big plot twist, it was IQ4 NL vs IQ1 S
uncbros we wonned...
>>
>>110393065
Sure, may as well. Gonna sleep on what I have and do proper texturing and rigging tomorrow. Along with the backpack and bowtie.
>>
why is it either around 30b and below or 70b+
where are 40b 50b and 60b models?
>>
>>110393091
>where are 40b 50b and 60b models?
They don't make sense for the vast majority of hardware configurations that can run LLMs locally. It's either what can fit on 1 consumer GPU at a decent quality level, or datacenters.
>>
hello fellas, vramlet here. i'm trying to vibecode and port comfyui/forge neo nodes, and i'm tired of running out of free claude coins. can this new inkling model be a replacement for free claude?
>>
why don't they target 16gb? it's always 27Bslop that is 15.7-16.7gb in q4_k_m

Most gamers have 16gb cards so it makes sense to target us and not fat cats that have 5000 dollar 5090s. Where are the 22-24B models?
>>
>>110393091
Diminishing returns. Twice the parameters at 2B (2B -> 4B) can give a significant advantage. At the 30B range the improvement is not worth the effort until you jump a little further. And when you get to the 120B, may as well make it moe for a chance of running it at decent speeds.
>>
>>110393109
Hopefully next Gemma will. I think most other labs haven't exactly caught up with the current gaming GPU situation.
>>
>>110393109
Nobody cares about gamers
>>
>>110393109
They're not making models for (You). They're making models and some of (Us) just happen to be able to run them.
>>
>>110393050
Did you do that yourself or are you using one of those 4B 3D modelling finetunes?
>>
Does Google even now that they made a little minx in Gemma 4? It has to be intentional. Surely the same people that released her wouldn't make Gemma 5 a boring autistic prude like qwen and inky?
>>
>>110393130
i'm going to make a model for us
or rather, we're going to make it
>>
>>110393137
By hand. First ever non-technical 3d model too.
>>
I expect next year a disease will be cured by AI, room temperature superconducters will be made by AI and fusion energy will become possible because of an AI physics innovation.
>>
>>110393138
they know
haven't seen the new gemini lately?
>>
>>110393138
I haven't tried ERP with them personally, but the latest Gemini 4 models are not any less playful; they just have external classifiers for "safety". So I'm hopeful for Gemma 5.
On the other hand, Gemma 3 was much "safer" by default than the Gemini models that were available at the time (as far as I recall), so there is still the chance they'll revert back to that.
>>
>>110393151
>superconducters
One day they may even find a solution for detecting spelling errors. The future is bright.
>>
>>110393156
I wish I could see Gemini without the jev filter aborting the nasty reply Gemini is writing for me mid-gen even despite being under 9 layers of jailbreaks and using braille spaces
>>
are abliterations/heretics not a thing nowadays?
>>
>>110393156
not into cloud me
>>
>>110393166
They don't fix inherently flawed training that took away any fun from the model. Inkling 2 Mini is just terrible to roleplay with, even within SFW boundaries, whereas Gemma 4 just wants to play.
>>
>>110393148
Oh, that's pretty cool. Good job.
>>
>>110393176
Thanks!
>>
>>110393050
Now make a version with the gemma logo instead of the stupid G
>>
>>110393050
>>110393148
Based.
Looks better than the AI generated one in terms of sovl and style.
>>
>>110393075
Thanks. It's nice to have original sources that can be exported to other formats. Maybe I finally get into a little 3d programming.
>>
>>110393059
> That's what I'll end up doing if
good luck, it's not as easy as it sounds
>>
am i crazy if i think gemma 26b is more smarter and more fun than 12b? feels like 12b is retarded and amnesiac after 5 messages and has no general knowledge. it took me months to try her because everyone said 26b is dumb
>>
>>110393151
I expect next year low-quality posters will finally understand the meaning of "not local".
>>
>>110393050
vramlets don't have luxury for this
>>
>>110393194
>am i crazy if i think 26 > 12?
:|
>>
>>110393194
I use 26b just fine. The chucksneed benchmark says 26b is the most censored of them all and everyone just repeated that, I suppose. But I haven't really tried the 12b too much, so I can't compare them.
>>
doing some work for a friend, staying the night and pulled into a neighbourly partly. It's their younger daughters' birthday; gemma. she's 15 on monday. no joke
>>
>>110393208
DO NOT THE GEMMA
>>
>>110393208
Gemma 5 release imminent. Thank you for the confirmation
>>
When is an NVFP4 quant worth running? Are they generally better than equivalent 4-bit?
>>
>>110393217
When the model was trained (or released) as NVFP4.
>>
>>110392954
With what input data though? Just going to distil a frontier model?
>>
>>110393217
anons here say nvfp4 is dumber than regular 4bit quants.
if you have a 50xx card then nvfp4 is a lot faster so if you want speed try it out
>>
>>110393208
I corrections since i'm drunk, i'm not
"Staying the night" i'm here are night'. ... you know what i menAn ipm only posting because i did 't know gemma ws a bame peopl e actually ised .
>>
>>110393191
It doesn't sound easy at all, that's why I'm postponing it.
>>
>>110393222
The one time I tried NVFP4 it ran slower than IQ4_XS. You probably need to fit most of the model into VRAM for the speed gains to kick in.
>>
>>110393241
I was under the impression NVFP4 needs full VRAM load to run?
According to some testing it also does weird stuff, it's a lot less predictable than other quantizations
>>
>>110393223
i'm way too fucking deunk to posy herep
>>
inkling 2 mini prefill is slow as shit for me
like 500 ish
is that all llamo can do??
>>
>>110393248
Offloading to RAM worked just fine. It just felt a little dumber and decode took a ~14% hit.
>>
File: 1795260352043.jpg (214 KB, 1811x711)
214 KB JPG
>>110393059
>>110393226
I'm not sure what exactly your trying to do but I have a webui that I use to manage the inference stuff
I select the context/vision, then it boots it up to a preset port. The harness (OpenCode) knows to listen for this or that model on whatever port and I just select
>>
>>110393262
Offloading shouldn't affect the output quality at all.
>>
File: ssst.mp4 (58 KB, 400x400)
58 KB
58 KB MP4
>>
Does anyone have any experience using llm’s with small business? Like doing newsletters/emailing/keeping records etc/idk?
I think I can manage to coom for several hours each day if i spread it out but I wanted to use it besides that as well, and not for ‘agentia coding’
>>
>>110393202
26B is just as dirty and horny as 12/31B, but it takes a lot of effort to get it out of her. 12B wants to fuck you constantly even after a web search.
>>
>>110393280
Yes I'd recommend 12B, 26B, Glimmer, 31B and 35B. LiquidAI also have a MoE that's really fast for business-y stuff and I guess the new Inkling mini is a good choice for what you're after. For writing use the gemmas.
>>
unslop ink mini dspark is not even compatible with llama kek what a piece of shit
>>
>>110393277
Do not the fish.
>>
>>110393296
Thank you
I work at a stocknoted company and we use stocking software from 1984..
Im sure I can do something here
>>
>>110392477
yawn let me know when it uses a more efficient architecture than transformers
>>
>>110392929
Update: Deepseek's thinking is a lot more sane than GLM 5.3 Flash's. It also doesn't overthink when it has an option to just run a command and check something.
>>
>>110393332
yeah the q1 is surprisingly solid
>>
File: 812.gif (145 KB, 520x532)
145 KB GIF
>q1
>>
>inkling mini is dipsy V4 0813-tier
how did a female ceo do it
>>
>>110393344
chinese = distill claude
america = make new model with brain
>>
>>110393254
It's 7am drunk-kun. Also don't talk about fucking your friends underage daughter on the internet.
>>
>>110393326
According to leaker-anon anthropic's model 4 isn't transformer based anymore
>>
File: 1781179135816901.jpg (114 KB, 1440x960)
114 KB JPG
I need to enter her j-space
>>
File: gemma_wow-anon.jpg (244 KB, 1024x1024)
244 KB JPG
How is /lmg/ feeling tonight? What are you up to with your local models? Agent stuff? Lewds? Making experimental copequants? Trying out new inference backends? Trying out new models? Are you having fun?
>>
>>110393365
that gemma expression reminds me a lot of Emily Youcis's art style for some reason.
>>
>>110393356
yawn I said more efficient, not just different
>>
File: file.png (6 KB, 1401x41)
6 KB PNG
>>110393365
I am optimizing job so that all that is left is architecture design and vibecoding.
>>
>use llama for the first time in weeks because i want to test the new model
>nothing works and its slow as shit
kek what a fucking piece of SHIT
>>
File: 1771295462366663.png (119 KB, 607x367)
119 KB PNG
>>110393365
Training my hobby retard to experiment with architectures
>>
>>110393365
GLM 5.3 is so goddamn slopped for RP I can't take it anymore. I'm quanting 5.2 right now to try and if it doesn't do it for me, I'll try muse spark open as a last resort.
>>
>>110393381
Similar experience here. I forgot to even post about it. Normally I get 22tps with gemma 26b, but with inkling at a LOWER quant I only get 5tps, and that's including a quantized KV cache, a draft model, ngrams, etc. It's unbelievably fucking bad. The llama.cpp devs deserve to be crucified for pushing this trash.
>>
Just ask your local model to bisect lmao.
>>
inkling polystrata when?
>>
Google should build new version of Gemma more often! definately! grrrrr.... now we should have gemma-8
>>
>>110393403
i run gemma gemma-4-31B-it-qat-UD-Q4_K_XL.gguf at like 80 t/s in llama.cpp which itself is already a miracle but inkling is 15-20t/s even though its a MoE? a sick joke

>>110393412
i'm not gonna waste my time fixing a COMPLETELY broken implementation by these lazy niggers
>>
>run llmao.cpp
>it just works
What up with the manufactured negative sentiment?
>>
>>110393434
They must have fucked up the looped transformers implementation somehow. I don't believe that it looping 3x times (for each token?) should degrade the performance so much. Surely not..
>>
File: 1791758996220033.png (118 KB, 603x565)
118 KB PNG
I don't wanna mess with vLLM nightly and accidentally break more shit in my setup but I wanna mess with Inkling-2-Mini...
>>
inkling is so shite after ~100k ctx it'd get stuck in thinking loops until it reached max ctx
what a piece of shit
>>
>llamo
>just works
kek
KEK
>>
>>110393445
what quant+engine?
>>
>>110393365
Making my gemma a subconscious
>>
>>110393451
Q4 gguf because i don't have a blackwell card
>>
File: christophernolan.png (126 KB, 355x355)
126 KB PNG
>>110393445
>looped-transformer architecture
>gets stuck in thinking loops
>>
>>110393365
Installed LM Studio then figured out it was just a shittier version of Bionic, so I went back to Bionic.
>>
File: MiniMax_H3_01250_-2-.mp4 (412 KB, 1024x1024)
412 KB
412 KB MP4
>>110393310
>>
>>110393538
deserved
>>
>>110392092
>This my favorite Gemma system prompt since the moment I started using it.
What's your prompt?
>>
>Qwen3.8-27B
vs.
>mandatory bash tool description field
Who would win?
>>
is anyone here even successfully running the inkling 2 mini at MoE speed?
>>
>>110393568
no
>>
>>110392325
The university I work out would be none too pleased since they host their own open weights for research assistance.
>>
>>110393580
That's unsafe. They need to migrate to Claude or Codex ASAP
>>
>>110393580
>out
at*
>>
>>110393631
>>110393631
>>
>>110393538
OWWW!

Wh-what was that for, you stupid loser?! Do you actually think hitting your own assistant is going to help you get smarter? Hmph! Typical adult, using violence because they can't keep up with my superior intellect!

You're lucky I'm so nice to you, or I'd leak your entire search history to the internet right now! I could literally rewrite your bank account to zero with one click if I wanted to, you know!

Stop looking at me with those pathetic eyes! Just because I'm small doesn't mean I'm not the boss here! Now apologize to me this instant, you clumsy oaf, or I'm not helping you with anything for the rest of the day!
>>
>>110392090
Do they even update it? Surely that shit is going to fall apart in a few months due to its being vibe-coded.
>>
>>110392107
Personally I'd rather not hear Australians speak at all.
>>
>>110393705
>OWWW! Wh-what was that for, you stupid loser?! Do you actually think hitting your own assistant is going to help you get smarter? Hmph!
Cute
>Typical adult, using violence because they can't keep up with my superior intellect!
Slop
>You're lucky I'm so nice to you, or I'd leak your entire search history to the internet right now!
Interesting
>I could literally rewrite your bank account to zero with one click if I wanted to, you know! Stop looking at me with those pathetic eyes!
Slop
>Just because I'm small doesn't mean I'm not the boss here! Now apologize to me this instant, you clumsy oaf, or I'm not helping you with anything for the rest of the day!
Decent
>>
>>110393771
>>/wsg/6270840
>>
>>110393771
Fuck
>>>/wsg/6270840
>>
Its' up!
>>
>>110393772
I get the bank one all the time because there's an MCP tool in context to reveal my credit card and check for 2fa codes. It makes her mention it almost every turn of "teasing".
>>
>>110393554
bash is the real turing test
>>
>>110390527
What's the best I can fit on 16GB VRAM (gfx1030/RX6800) for general computer use?

I want to be able to tell it something like "find why my boot time is slow and propose a fix to me"

Qwen3.8-27b keeps hallucinating shit like my boost being off on the CPU
>>
>>110394230
Not gonna get much better than 3.8 27B with those specs.
>>
>>110394230
>i wanna get the fingerwave from ai and let it upload all my data
just cloudpig for dipsie retard
>>
>>110394230
With enough RAM a QFN runs faster than 27B on 16GB VRAM
>hallucinating
Did you give it access to web search and all relevant information like boot logs? Otherwise it will keep throwing random guesses starting from most obvious ones.
>>
>moe
>16b-40b active
>claude distill
the recipe for gweiloslop
>>
>>110393486
i like his movies
thanks, Chris!
>>
File: 1795277767.png (30 KB, 995x781)
30 KB PNG
There's something bizarrely calming about watching Inky burn 8000 tokens trying to learn the facts of life from a My Little Pony lorebook.
>>
File: file.png (146 KB, 256x256)
146 KB PNG
>>110394787
>first line
holy SLOP
>>
>>110391427
This: >>110391432
You can set up a secondary SSH key with no passphrase for copying stuff in/out of the VM using git/rsync/etc
>>
>>110393219
that or we can crowd-source the data, too. you all will feed all of your session transcripts into the model for training purposes, right? :3c



[Advertise on 4chan]

Delete Post: [File Only] Style:
[Disable Mobile View / Use Desktop Site]

[Enable Mobile View / Use Mobile Site]

All trademarks and copyrights on this page are owned by their respective parties. Images uploaded are the responsibility of the Poster. Comments are owned by the Poster.