/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>107525233 & >>107515387►News>(12/10) GLM-TTS with streaming, voice cloning, and emotion control: https://github.com/zai-org/GLM-TTS>(12/09) Introducing: Devstral 2 and Mistral Vibe CLI: https://mistral.ai/news/devstral-2-vibe-cli>(12/08) GLM-4.6V (106B) and Flash (9B) released with function calling: https://z.ai/blog/glm-4.6v>(12/06) convert: support Mistral 3 Large MoE #17730: https://github.com/ggml-org/llama.cpp/pull/17730>(12/04) Microsoft releases VibeVoice-Realtime-0.5B: https://hf.co/microsoft/VibeVoice-Realtime-0.5B►News Archive: https://rentry.org/lmg-news-archive►Glossary: https://rentry.org/lmg-glossary►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://livecodebench.github.io/gso.htmlContext Length: https://github.com/adobe-research/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsAlpha Calculator: https://desmos.com/calculator/ffngla98ycGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-sampling►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllm
►Recent Highlights from the Previous Thread: >>107525233--ROCm support challenges and alternatives for Koboldcpp on Windows:>107530979 >107530999 >107531023 >107531043 >107531003 >107531018 >107531106 >107531139--Manual fandom scraper workaround for model training database creation:>107529955 >107529985 >107530131 >107530163--Enhancing roleplay through structured prompts and character dynamics:>107530514 >107530535 >107530543 >107530565 >107530582--Zai Kaleido model training methodology and VRAM requirements inquiry:>107527349 >107527409 >107527751--pip dependency resolution woes and alternative package management solutions:>107525683 >107526010 >107526331 >107530918--OpenAI's circuit sparsity release:>107533877 >107533906--Techniques for maintaining narrative control:>107530602 >107530642 >107530716 >107534318 >107534259--GLM4V vision integration in llama.cpp with current text quality tradeoffs:>107534080 >107534101 >107534374--Roleplaying with AI models and exploring creative techniques:>107525864 >107526297 >107526429 >107526316 >107526610 >107526906 >107527111 >107527167 >107526935 >107527009 >107527089 >107527150 >107527198 >107527212 >107527244 >107527245 >107527266 >107527282 >107527399 >107526360 >107529092--Questioning LLM reasoning capabilities through a vector space math problem:>107528577 >107528851 >107529085 >107529323 >107528652 >107528694--Critique of a poorly maintained LLM-integrated creative writing tool:>107531460 >107531502 >107531525 >107531581 >107531615 >107531775 >107531792 >107531810 >107531869 >107533539--Skepticism about leaked Nemotron models' role-playing capabilities:>107528051 >107529702 >107531280--Olmo 3.1 model released, nearing Qwen performance, potential for further updates:>107529801--Miku and friends (free space):>107525338 >107525594 >107525657 >107530112 >107532702►Recent Highlight Posts from the Previous Thread: >>107525236Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
Anyone used zai tts yet?
>>107535431Do you expect it to be good? They did not bother uploading examples to hf/github
What is currently the best model that runs on a single 5090 32gb?
gm sirs
>>107535466nemo
>>107535458It's the first time I've seen tts that claimed to have 'emotional control'.
>>107535480The paid essay writing service or the Nvidia frame work for training models?
>>107535466z-image-turbo
What kind of token/sec do people with 4x 3090 get? trying to do some comparisons, ideally from a common 70b/123b model
>>107535520The game engine.
>>107535520good one mate
>>107535551Really? That even fits in 16gb I have 32gb
>>107535568nothing better until glm46 which is hueg
>>107535579nemo is still better as a model, it's like a really comfortable car that only does 25mph.
>>107535605pure cope and you know it
>>107535644I run both and nemo somehow gets me.
So MOE models and the merged 3x8b into a 24b actually do have a clear defined multiple "people" inside or is it just some terminus technicus and it's still the same as any other LLM?
>>107534661why not just write the story in mikupad?
>>107535689it's placebo for the tech illiterate
>>107535550on 123b about 600t/s prompt and 20t/s gen, conservatively.
>>107535698nta. I don't like CYOA, I want to talk to something. Just personal preference.
>>107535776What quant?
>>107535644magnum v2 mogs all
>>107535824Q4-Q5. That's what will fit. I like to grab the latter usually, I can still fit like 65k+ ctx.
crazy how devstral made deepseek, glm and kimi irrelevant the moment it dropped
>>107535520The film actually
>>107535900Bait used to be believable
Devstral cured my psychosis.
>>107535900The 24b, right?
>>107535900devstral is easier to run fast. at least the non-deepseek version. 96gb of vram cheaper to get than vram + ddr5.
I've just crawled from under a rock. What happened to Pygmalion fag and dataset he was collecting from anons' submissions? Was it released? Is it any good?
>>107536312https://huggingface.co/datasets/PygmalionAI/PIPPA
>>107536330thebloke bros...
>>107536312>I've just crawled from under a rock. What happened to Pygmalion fagThey made a website, eventually.https://pygmalion.chat/There is still some activity in the matrix, but the devs are mostly gone from there. They are generally on the official discord. The lead dev 0x000011b disappeared some time after the Llama 1 release and the project was continued toward a commercial direction by the others.https://matrix.to/#/#waifu-ai-collaboration-hub:halogen.city?via=halogen.city>and dataset he was collecting from anons' submissions? Was it released?https://huggingface.co/datasets/PygmalionAI/PIPPA>Is it any good?Not really, it's a small subset of the entire data, and just composed of early character.ai chatlogs anyway, with all the bad and good quirks. You'll never truly replicate character.ai with this, just like you can't replicate Anthropic Claude just with some ERP logs, only imitate it at most.
>>107536330kek
>>107536384petra bros...
>>107536379Are they really making money from it? They're not even allowing nsfw (lmao coming from c.ai)
>>107536406I think the idea is still that you can do whatever you want with private bots. I don't know if they're making money, there's so much free choice nowadays.
What is currently the best model that runs on a single 1060 6gb?
>>107536439>there's so much free choice nowadays.Which made me realize, Google AI Studio (Gemini) is about as functional for roleplay as character.ai was in late 2022, while being completely free, far smarter than CAI ever was and allowing limited explicit ERP (as long as you're not into noncon and lolisho). The only advantage other websites have is community-made bots.
>>107536211Looks like we've got tensor parallel for it in ikllama now too
>>107536705>the only advantage is the only thing normalfags wanthuh?
>>107536696gemma3n maybe https://huggingface.co/bartowski/google_gemma-3n-E2B-it-GGUF
>>107536746I've basically only ever used private custom bots on CAI before I switched to local ERP around the time of Pygmalion-6B, so I guess I can't fully appreciate the usefulness of community cards. I don't even use cards from Chub.
Do any of the AI erp threads elsewhere on the board have a more up-to-date settings guide than the rentry here? It only mentions llama2 as newest (I think). I have some L3 - llama3 - I guess, gemma and some Qwens. And everyone seems to be telling to use settings completely opposite from what the other guy says.
>>107536705Gemini got good when noam shazeer moved back to google. Make of that what you will.
>>107536851Use temperature only samping.better yet- use your fucking brain. The people using meme samplers are the same people whining about output quality. What settings doe that point you to, you dumb nigger? *beep* dey ceiling birds is back
>>107536884I doubt, until I use meme samplers devstral2 was shit. And that's their official API on OR.
Package arrived.I am running out of pcie lanes on my poorfag 9950X.
>>107537010Why do you need more?
>>107537010When is the 4th arriving?
>>107537010>still not enough VRAM to run Deepseek and Kimi
>>107536851yeah i got one right here for youtemp=1top_p=0.95you don't need more
>>107537010Get rid of your piece of trash 4090.That'll make space for your 4th 6000 Blackwell.
>>107537516NTA but I think he already intends to replace the 4090 with the 6000 in the picture.Consumer motherboards usually max out at 3 PCIe slots.
>>107536851>Top P 0.9>Top K 10>Temp MAXRemember to have temperature as the last sampler.
>>107537010>>107537533>replaced the 4090 that was connected to a 5.0 4x m.2 slot>motherboard code 96 - pci bus assign resourcesTried changing pcie setting in the bios to no avail so far.>>107537067You can always have more.>>107537125After I upgrade to threadripper or epyc it seems like.
>>107537010>running out of pcie lanesYour top pcie slot probably support bifurcation into x4x4x4x4.The jank route is slimsas 4i or mcio 4i adapters and cables.
why not just stack quadro 8000s for lots of cheap vram?
>>107537609>quadroWeren't they Turing at best?
How long until local models hit the level of something like Gemini?My boss believes in a year or two everyone will be running stuff locally
>vLLM omnihttps://github.com/vllm-project/vllm-omni>SGLang difussionhttps://lmsys.org/articles/sglang-diffusion-accelerating-video-and-image-generationAre they faster than Comfy?
dont trust females
>>107537534>top-k 10Her eyes shivered down her spine. The assistant gleamed. I can't continue this conversation.
>>107537676Comfy isn't LLMcentric really. The first thing might be good for "conversational" gen sesh. Second is snakeoil #4574645
>>107537796I was mostly interested in this part, if it's true for diffusion models:>Tensor, pipeline, data and expert parallelism support for distributed inference
>>107537606>bifurcation into x4x4x4x4It does. Asus sells pic related so I assume a chinese adapter with four riser cables would work. But I also assumed that the current setup with an m.2 adapter would work because it worked with the 4090 and it doesn't.I also have no idea how I'm going to mount all that.
>>107537672Your boss is a drooling retard. The trend for the last 15 years has been herding people onto online subscription services, and that was before memory prices quintupled for the foreseeable future.Cheap hardware only allowed their software be more resource wasteful, but in the future everyone will be running thin clients that struggle to run their "everything app".
>>107537981>m.2 adapter didn't workTried dropping everything to pcie1 speeds?You can turn the speeds back up afterwards if it works.It worked for me on my poorfag quad-3090 rig.>I also have no idea how I'm going to mount all that.The jank solution is a mining frame.
Are the latest Mistral models that uncensored?https://speechmap.ai/models/
>>107538184Yes, it still didn't work. There's a new bios version I'll see if that works.
>>107538235The non thinking ones will more or less continue any RP text completion with the usual basic bitch system message manipulation. Although if you use chat completion and ask it to write dirty shit they will refuse.And the thinking models are basically just dysfunctional brain-rotted trash not worth using for anything.
>>107538235devstral called me a faggot and large told me to kill myself so I think so.
>>107535474
>>107538328is the migu in danger?
Amazon is already making a sequel to nova apparently. It seems to be distilled off of gemma.
>>107538357It appears so
>>107538379I seem to remember that in addition of talking about sea otters holding hands while sleeping, Mistral Small 3.0 also gave hotlines.
>>107535410>pic rel is me in this thread
Which of the llm uis is the most retard proof?
>>107538328>>107538389it's over
>>107538414It even knows to make the pajeets feet all fucked up due to mutations from the toxic waste they're constantly standing in. Amazing.
>>107538379>I can't respond this request>might encourage details>depicting situations where a medical professional refuses to treat a patient...Ask if it has any tv series or film recommendations in that vein.
