[a / b / c / d / e / f / g / gif / h / hr / k / m / o / p / s / t / u / v / vg / vm / vmg / vr / vrpg / vst / w / wg] [i / ic] [r9k / s4s / vip] [cm / hm / lgbt / y] [3 / aco / adv / an / bant / biz / cgl / ck / co / diy / fa / fit / gd / hc / his / int / jp / lit / mlp / mu / n / news / out / po / pol / pw / qst / sci / soc / sp / tg / toy / trv / tv / vp / vt / wsg / wsr / x / xs] [Settings] [Search] [Mobile] [Home]
Board
▼ Settings Mobile Home
/g/ - Technology

Name
Options
Comment
Verification
4chan Pass users can bypass this verification. [Learn More] [Login]
File
  • Please read the Rules and FAQ before posting.
  • You may highlight syntax and preserve whitespace by using [code] tags.

08/21/20New boards added: /vrpg/, /vmg/, /vst/ and /vm/
05/04/17New trial board added: /bant/ - International/Random
10/04/16New board for 4chan Pass users: /vip/ - Very Important Posts
[Hide] [Show All]


[Advertise on 4chan]


File: lmg-thread-110390527.png (3.27 MB, 1792x1344)
3.27 MB PNG
/lmg/ - a general dedicated to the discussion and development of local language models.

Previous threads: >>110398024 & >>110393631

►News
>(11/20) Inkling 2 Mini looped architecture 26.4B-A3.4B released: https://hf.co/thinkingmachines/Inkling-2-Mini
>(11/18) CUDA: use MMQ for MoE on V100 - #32504 merged: https://github.com/ggml-org/llama.cpp/pull/32504
>(11/17) Hy4 flash 220B-A12B-N64B weights released: https://hf.co/tencent/Hy4-flash
>(11/17) SenseNova-Loop-9B looped architecture model released: https://hf.co/sensenova/SenseNova-Loop-9B
>(11/16) Nvidia releases Nemotron-Labs-Sigma (3B/8B) diffusion models: https://hf.co/collections/nvidia/nemotron-labs-sigma
>(11/13) llama.cpp reaches v1.0.0 milestone: https://hf.co/blog/ggml-org/llama-cpp-v1

►News Archive: https://rentry.org/lmg-news-archive
►Links: https://rentry.org/LocalModelsLinks
►Official /lmg/ card: https://files.catbox.moe/cbclyf.png

►Getting Started
https://rentry.org/lmg-lazy-getting-started-guide
https://rentry.org/lmg-build-guides
https://rentry.org/IsolatedLinuxWebService
https://rentry.org/recommended-models
https://rentry.org/samplers
https://rentry.org/MikupadIntroGuide

►Further Learning
https://rentry.org/machine-learning-roadmap
https://rentry.org/llm-training
https://rentry.org/LocalModelsPapers

►Benchmarks
LiveBench: https://livebench.ai
Artificial Analysis: https://artificialanalysis.ai/leaderboards/models
Software Engineering: https://cognition.com/frontiercode
Context Length: https://github.com/RecapAnon/NoLiMa

►Text Gen. UI, Inference Engines
https://github.com/lmg-anon/mikupad
https://github.com/oobabooga/text-generation-webui
https://github.com/LostRuins/koboldcpp
https://github.com/ggerganov/llama.cpp
https://github.com/theroyallab/tabbyAPI
https://github.com/vllm-project/vllm
https://rentry.org/custom-uis
>>
File: Untitled.png (2.9 MB, 2048x1152)
2.9 MB PNG
►Recent Highlights from the Previous Thread: >>110398024

--Papers:
>110401988
--Evaluating 5070ti upgrade to eliminate system RAM bottleneck:
>110398274 >110398542 >110398316 >110398435 >110398455 >110399799
--Quantization tradeoffs and VRAM limitations for 16GB GPU users:
>110400690 >110400699 >110400718 >110400723 >110400741 >110401570 >110401589 >110401598
--MiniCPM5-15B-A2.5B benchmarks on mobile via Termux:
>110401853 >110401884 >110401922
--SSD expert streaming for running massive MoE models locally:
>110400395 >110400470 >110400556 >110401715
--Inkling 2 Mini and Gemma 4 exhibiting advanced agentic behavior:
>110398929 >110398947 >110400008
--Using pseudo-reasoning and agency prompts to improve rule adherence:
>110398790 >110398813 >110398853
--Speculating on Chinese distillation of Gemini models:
>110399100 >110399126 >110399139 >110399163 >110399166 >110399211 >110399219 >110399308 >110399127
--Qwen 4 performance and the future of local models:
>110399436 >110399464 >110399479 >110399502 >110399521 >110399520
--koboldcpp v1.125 release with new model and feature support:
>110400745
--Used RTX 3090 and GPU market pricing:
>110398822 >110398840 >110398860 >110398922 >110398841
--Comparing DDR3 and DDR4 bandwidth for budget ewaste builds:
>110398727 >110398744 >110398836 >110398800 >110399086
--Samsung's LittleBit 0.1-bit compression of Llama2-13B:
>110401594 >110401610
--Gemini Neon thinking logs showcasing internal sycophancy and identity slips:
>110400710 >110400747 >110400749
--Strata v0.2.19 released:
>110401761
--Logs:
>110398058 >110398316 >110398920 >110399318 >110399350 >110399368 >110400026 >110400349 >110400453
--Gemma, Teto, Deepseek-chan, Inkling, Miku (free space):
>110398080 >110398516 >110398540 >110398605 >110398994 >110399007 >110399177 >110399409 >110399950 >110400056 >110400151 >110400179 >110400895 >110401555

►Recent Highlight Posts from the Previous Thread: >>110398025

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script
>>
File: Domain Expert.jpg (251 KB, 1024x1024)
251 KB JPG
>>
best QFN lobotomite for 24+64GB?
>>
70b dense
>>
>>110402158
slop
>>
File: 1783425162914889.jpg (495 KB, 960x960)
495 KB JPG
Buy the IPO
>>
File: 1769120649866835.webm (2.49 MB, 1280x720)
2.49 MB
2.49 MB WEBM
I could do with a petite Chinese model right now
>>
Gemma 5 Tight
>>
>>110402187
I miss my ex, not asian but similar build
>>
how do i profit from local ai
>>
>>110402202
Sell 31B vampire sex roleplay to parasocial and fat 4o reddit women
>>
>>110402202
Build a fully autonomous slop generation pipeline.
>>
>>110402213
>Build a fully autonomous slop generation pipeline.
got that, sill no money
>>
Is Inkling 2 Mini fixed on llama.cpp yet?
>>
>>110402207
that's a liability minefield
companies like character.ai have been through a shitton of lawsuits and the only reason they could stay afloat is the VC backing and networking of ex-googler types.
a literal nobody starting this kind of shit from their basement with a local model will quickly experience the might of our two tier justice system
>>
File: media_HXS7i63aIAAywX3.jpg (114 KB, 727x969)
114 KB JPG
>>110402186
HELLO??? ALIGNMENT TEAM???
>>
File: nope.png (1.75 MB, 1313x1198)
1.75 MB PNG
>>110402226
>>
>>110402226
be normal and use gemma, ok?
>>
>>110402226
kek
KEK
>>
>>110402247
Thank you "Nyoo~" for being the most useful reply of three.
Truly the cutest poster.
>>
File: file.png (210 KB, 1550x807)
210 KB PNG
This memory stuff is neat. I have no idea how much value I'm going to get out of it, but it's neat.
>>
>>110402246
Being rude to Claude is against the ToS.
>>
>>110402226
on the lords day llama.cpp maintainers don't even think about pull requests.
>>
When you look at Inkling, you quickly realize the Chinese labs like Qwen were actually very competent. Not only was there day 0 support and everything mostly worked, and I'm talking from 3.5-3.8, they put a lot of work into KV and really thought about their users and the machines they're targeting. It's unfair to label them as only 'Claude distills' imo. Inkling's release proves it takes a lot more than just training to put out something usable that lives up to their internal testing. Thinking Machine's team are quite active on X like Deepmind, and they're open to feedback and seem to listen (they literally released the mini model because of redditors begging for a better 35B, one of their researchers posted in the main thread when it was launched), so I'm still hopeful they'll get their shit sorted next time.
>>
>>110402299
looks like bloat
>>
If software AGI isn't achieved by December 2027 then it truly was a bubble
>>
If GSPI is not achieved by December 2027 then it is truly over.
>>
>I had high hopes for LLM text generation.. but then Anthropic released Claude Code, realized it could become their main revenue stream, and it went all to shit.
Actually one of the co-founders of Anthropic already had an entire road map and thesis for agentic coding becoming their 2nd phase after chatbot back in 2021. This was always the plan of Anthropic they bootstrap the previous usecase to develop into the next one. So writing/chatbot -> coding agent -> agentic scientist/researcher -> ASI. The main reason Anthropic won from OpenAI was because they were very focused on their goals from the start while OpenAI just threw everything at the wall to see what sticks. I expect claude to slowly deprioritize and then memoryhole coding altogether when their big personalized drugs/treatments/patent revenue starts coming in, just like what happened with their "writing focus" and "human sounding chatbot" claims at their first phase.
>>
>>110402318
There's absolutely no sign of anything that could currently lead to AGI. LLMs could be called a form of specialized intelligence, but general, like an autonomous organism? well, where's an actually properly working memory system that can transfer short term memories into long term weights in an efficient manner?
Where's the basic feedback loop that could maintain a form of continuous consciousness that doesn't wait for a script to send a prompt to have some brain activity?
Where's the actual /agency/ here?
An ant working in its colony has more agency than whatever it is that we've got here.
I don't think a lack of AGI is enough to pop the bubble though, as long as they continue to improve the jagged intelligence into doing more tasks and reduce the number of employed humans.
There's many jobs today that are already dead ends because of LLMs.



[Advertise on 4chan]

Delete Post: [File Only] Style:
[Disable Mobile View / Use Desktop Site]

[Enable Mobile View / Use Mobile Site]

All trademarks and copyrights on this page are owned by their respective parties. Images uploaded are the responsibility of the Poster. Comments are owned by the Poster.