/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>103515753 & >>103510291►News>(12/13) DeepSeek-VL2/-Small/-Tiny release. MoE vision models with 4.5B/2.8B/1.0B active parameters https://hf.co/deepseek-ai/deepseek-vl2>(12/13) Cohere releases Command-R7B https://cohere.com/blog/command-r7b>(12/12) QRWKV6-32B-Instruct preview releases, a linear model converted from Qwen2.5-32B-Instruct https://hf.co/recursal/QRWKV6-32B-Instruct-Preview-v0.1>(12/12) LoRA training for HunyuanVideo https://github.com/tdrussell/diffusion-pipe>(12/10) HF decides not to limit public storage: https://hf.co/posts/julien-c/388331843225875►News Archive: https://rentry.org/lmg-news-archive►Glossary: https://rentry.org/lmg-glossary►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/tldrhowtoquant►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://livecodebench.github.io/leaderboard.htmlCode Editing: https://aider.chat/docs/leaderboardsContext Length: https://github.com/hsiehjackson/RULERJapanese: https://hf.co/datasets/lmg-anon/vntl-leaderboardCensorbench: https://codeberg.org/jts2323/censorbench►ToolsAlpha Calculator: https://desmos.com/calculator/ffngla98ycGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-sampling►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllm
►Recent Highlights from the Previous Thread: >>103515753--WebDev Arena and AI creative platforms discussed:>103520984 >103521017 >103521279 >103524294 >103524591 >103524725 >103524804 >103524882 >103524908 >103525175--Anon shares tips and experiences with qwq model for rp and erp:>103523891--Discussion of AI models and their performance characteristics:>103519748 >103519771 >103519859 >103519892 >103520087 >103520126 >103520163 >103520153 >103520185 >103520210 >103519893 >103519916 >103519978 >103520036 >103520313 >103519993 >103520057 >103520137 >103519913 >103520056--Discussion of 3.33 model performance and settings:>103519237 >103519373 >103519407 >103519515 >103519585 >103520214 >103519770 >103519812 >103519534--Local voice generation and text-to-speech discussion:>103517143 >103517151 >103517192 >103517448 >103521850 >103517451 >103517679 >103520861 >103521261 >103521363--Anon asks about programming models and GPU requirements for development, mentions Qwen2.5 32B coder:>103523524 >103523535--Anon speculates on Anthropic's secret sauce for Sonnet 3.5:>103523542 >103523605 >103523784--PCIe bandwidth usage during model inference:>103523792--Former OpenAI researcher and whistleblower found dead:>103517010--OpenAI CEO Altman donates to Trump's Inaugural Fund, sparking discussion on corruption and bribery:>103517301 >103517369 >103517428 >103517449--Anons discuss the limitations of LLMs in creative writing and RPing:>103522040 >103522080 >103522095 >103522115 >103522187 >103522270 >103522142 >103522856--Ilya Sutskever's presentation and OpenAI's approach to AI research:>103521192 >103521434 >103521674 >103521804--7900xtx not suitable due to no CUDA support:>103515944 >103515950 >103515959 >103516922 >103516072--Miku (free space):>103517905 >103518081 >103520689 >103522038 >103522977 >103524395►Recent Highlight Posts from the Previous Thread: >>103515755Why?: 9 reply limit >>102478518Fix: https://rentry.org/lmg-recap-script
The week before christmas will be huge. Everyone will be pushing out their models before the holidays.
Commit suicide right now.
>>103525282Possibly something from Qwen, then...?
Newfag here.Can anyone point me to a download for llama 1 ?Want to see what it is like.GGUF would be nice, but I'll take anything.
>>103525965https://huggingface.co/TheBloke/LLaMA-65B-GGUF
>>103525982ty
>>103525398Pretty much confirmed already
>>103525267thanks for the recap but you should fix the script to use >> so that the links are clickable.
>>103525267>>103526077nvm i just read the link
https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B-SFT-no-safety-data
Can I fit in a 70b model and 64k context on 64 gb vram? I'm planning to do something retarded like buying 2 5090 but if it can't even manage that then I don't want to bother
>>10352622170B definitely if < q864k context you are pushing it i think lol.
>>103526221if you quant it down yeah probably
>>103526221>64k contextWhy would you even need that much? You'd have to get a third 5090 for that.
>>10352622132k context at 5bpw fits. Don't know if 64k would be possible without going below 4bpw.
For a moment, I believed Llama 3.3 was better than Largestral, then it failed miserably on a good old Jeanne test.
>>103526221Ollama fag here.>llama 70b q4 with 5k to 9k context more or less fits in 2* 24GB.From what I can tell from lurking these threads,if you use something other than Ollama you can avoid using vram for context ?(Ollama always seems to consume vram to hold context.)
>>103526306no, all engines use vram for context unless you load the model on ram but yea that's p standard.
>>103526274I've been running 32k so far and just thought it would be nice to double if I'm going to buy something like that but yeah on second thought it's quite a lot, I'd be content with upping it to 49k just to fit in a bit more of context from rags, as long as it can reach reading speeds or slightly faster than that otherwise I guess I'll just stick with 32k
>>103526306>model has to reference context for each token>hey guise how to put context in slower storage???breh
>>103526274>third 509032k fits.36k if 1 layer on cpu.Haven't tested further.
>>103526389Assuming 24GB here of course.
>>103526306>ollama user>retarded questionEverytime
In my time simply mentioning ollama was enough for a few "go back"s
lawl
>>103526958>due to its PhD-level intelligence Academics is the study of existing knowledge.Intelligence is the creation or acquisition of knowledge that doesn't yet exist.That statement is utterly moronic.
>>103526011>These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit d0cee0dThose ggufs are more than a year old. Make sure you use that commit of llama.cpp to run them. You're probably going to be better off downloading the HF safetensors weights and converting them yourself with current llama.cpp.https://huggingface.co/huggyllama/llama-65b/tree/main
>>103526958It's not PhD-level intelligence until it can write a doctoral thesis.
>>103527006Bookmarked, thanks.I'll let the ggufs finish downloading first before starting on the this.
>>103526958You pay OpenAI $2000/month for API access and a $8000/month salary for a cappuccino-sipping "prompt engineer" to write and maintain prompts and glue scripts, and suddenly the amount of tasks you can cost-effectively automate dwindles significantly.
>>103526958For anything beyond $20 / month to be worth it, you've got to give me a service that I'm confident wouldn't be replicated anywhere elseo1 is on track to get its ass beat by gemini-exp-1206 (which isn't even a fucking CoT model), o1-pro is probably something ducktaped together since they didn't bother to show performance, and Sora is basically already eclipsed by HunyuanShove that $2000 / month tier up your ass right next to the $200 / month one
>>103527070more like $200/month for a cow piss-sipping prompt sir
>>103527254>Sora is basically already eclipsed by Hunyuanlol
>>103526958lmao, meanwhile a toddler is smarter than their best model kek.
>>103526958I have to think of this every time someone mentions intelligence and LLMs in one sentence.
>>103527400sadly child labor is still illegal in the anti-business west
OpenAI wouldn't get so much hate if not for the name
>>103527397You still traumatized from those fox girls frolicking in the fields, anon?
>>103527431>You still traumatized from those fox girls frolicking in the fields, anon?That wasn't Hunyuan though?
>>103527456That's the point. Sora is fucking useless
>>103527518You don't get it, sfw foxgirl videos could be used as propaganda by state enemies!
>>103527417This issue is from the instructing tuning, retard
>>103527704exiting the matrix with miku
/ldg/ is faster than us...
>>103527420This is probably bait, but I've hated OpenAI ever since they filtered GPT-3 and the taskup thing where they uploaded user generations to a public freelancing site. All of the shit that happened since (not giving model sizes, then eventually not providing techniques altogether, overcharging whenever they have the lead on something, moving from nonprofit to for profit, Altman being a spineless fucking loser who will immediately bend over for anyone that can give him money or power) hasn't helped either.
>>103528079REMEMBER WHAT THEY TOOK FROM YOU
Anons who suggested quen-2.5 and QwQ yesterday. It can run inference on 20gb vram? How is this real? Can I run it on 7900xtx with Rocm? Can't find used 3090s around here.
how long until i don't have to prepend my system prompts with 1000 words of sex vocabulary
>>103528152Four years later and it's still there like it was yesterdayToo bad I had to wait this long to see OpenAI start falling apart, but better late than never
>>103528224When you'll be able to solve your skill issue
>>103525265What happened? I've been in hybernation for 5 months and suddenly sillytavern is slow, responses suck and they even removed the roll dice option. Are there any cool alternatives after this huge downgrade?
>>103526958>hallucinating scientific sounding bullshit generator>PhD-level intelligence>2k/monthAre they serious or are they hyping?
>>103527642The day they announce a breakthrough in genetic research and state provided foxgirls, I'm moving to China, videos or no
>>103528320It's called ServiceTesnor now, CHUD! It is corporate-friendly software, roleplay features have no place here.
>>103528167>amdYou will regret it.
>>103528320We'll keep using that shit until it completely falls apart.
>>103526958>releases new model>STILL gets mogged by ClaudeGod that would be funny.
>>103528385Do we have any alternatives? I was recommended RisuAI but it was pretty bad a year ago.
>>103528393buy an ad
>>103528393If there were any you'd probably heard about them already.
>>103528393mikupad
>>103528387Even worse: gets mogged by Gemini. No moat!
>>103528320>suddenly sillytavern is slowIt is?
>>103528436Horde. IT takes 500 seconds to get a proper response. Local models never really worked well for me.
>>103528320Are you talking about this?Nothing happened in practice. There was the implied threat of the cuckening but with a thousand users freaking the fuck out it was postponed.
Dear Kobo,I am once again requesting you to add all configuration options for draft models from llama.cpp. Your current default settings are suboptimal and do not achieve full speedup that is possible to experience using llama.cpp.
>>103525265How the fuck do I use the Rocinante v4?Drummer said in a reddit post that he switched from ChatML to Pygmalion for that version, but switching to Pygmalion/Metharme yields shit results? How do I set up the Context Template, Instruct Template and System Prompt for that model?
>>103528449Horde has been slow for a while, not sure if more people flocked to it or there are less workers.
>>103528480This isn't the drummer memetunes general, go send him a PM on plebbit or something.
>>103528436Contradictory to "slow", ST is a lot more responsive for me now. For example, no more lag with deleting a swipe from a default chat.>>103528449>hordeAre you using a key with kudos? If not, you can ask for some in the official kobo dickord.>be me, haven't used horde in forever>check https://overseer.logicism.tv/not looking too good, the model with 19 workers is at 44s ETA right now, and higher for most
>>103528632What do you need to run a worker? Suck cohere pp?
>>103528674https://github.com/LostRuins/koboldcpp/wiki#what-is-horde-how-do-i-use-it-how-do-i-share-my-model-with-horde
>>103528674The most basic worker is just running koboldcpp launcher and configuring for horde in Horde Worker tab.The guys with many workers are running Aphrodite, a fork of vLLM, idk about that stuff.
Also setting model name field to exactly one of these from "approved list" gets you more kudos than just 1 or 2 per request (can ask if you have something else).https://github.com/Haidra-Org/AI-Horde-text-model-reference/blob/main/models.csviirc base name without prefix/ or quant e.g. Meta-Llama-3.1-8B-Instruct
Fuck fact, if you type something in reverse the model cant decipher it even if it realizes that the sentence has been reversed.
>>103528694I don't understand how you setup this with tabbyAPI, I give up.
>>103528832You mean this :>https://github.com/theroyallab/tabbyAPI/wiki/07.-AI-Horde?
hihihi gotcha chatgpt
>>103528399>anon asks a question>another anon answers it>(You) get mad that the thread is being used for its intended purposeY'all some niggertards
>>103525267>Former OpenAI researcher and whistleblower found dead:Man, whistleblowers sure have a bad habit of turning up dead don't they?
>>103528464Dear anon,The project is open-source, feel free to make the changes you desire.With love,Henk.
>>103529366Dear Henk,I am too incompetent and AI is not advanced enough to help me yet. If I try doing the needful, you'll see very Indian code, which you may not like too much. All I can do right now is beg. The requested functionality is already present in llama.cpp, so I assume it wouldn't be too hard for you to restore it as a command line argument. Pwease add it *sucks ur dick*Love,Anon
phi4 weights have leaked right? anyone have quants running yet? is it any good?
>>103529570its bad dont bother
>>103529570I have tested, it's very good and I'm not saying this just to make a joke with the other anon reply. (However, I didn't try to use it for ERP)
>>103529626what have you used it for, i probably ERP like 5% of the time i'm using LLMs so that's not an issue for meit's only supported in llama.cpp right now is that right? i'm downloading the matteogeniaccio quant rn
>>103528449How is that an ST problem?
Why didn't Mistral Small make a bigger splash for RP? Is it in the no man's land size-wise? There's like one good tune for it (Cydonia), while Nemo has tons.
>>103529658Translation and RAG, mostly translation though. It surprised me because it's performance seems to be comparable to Gemma 2 27B in that area.And yes, it's supported even in koboldcpp.
>>103525265I just want to point out that, given the recent news about the donations from Altman et al., all you piece of shit establishment bootlickers in the U.S. who voted for Trump could burn in hell for a trillion years and it wouldn't be 0.000001% enough punishment. Nothing fucking surprises me anymore, but here we are.
>>103529811Because running a decent 70B at 2-3 bit is better and vramlets can only run nemo models.
>>103529829seethe ;)
>>103529811It was drier than nemo+no base model+license.
>>103529811It's just... not good. Even Nemo is better than it for RP.
>>103529811More accessable due to its size and the writing is better, even if its dumber.
>>103529829i mean he's just doing his best to capitalism, can you blame him>>103529832it's so wild, trump voters are mostly going to be worse off under trump than they would be under harris but like, very obviously neither party is actually trying to make the lives of any normal citizen better
>>103529846Cydonia is by far the best of its size though.>>103529831>2-3 bitSure, if you like actual window-licking, crayon-eating levels of retardation. Otherwise you never want to go below Q4.
>>>103525265 (OP)>I just want to point out that, given the recent news about the donations from Altman et al., all you piece of shit establishment bootlickers in the U.S. who voted for Trump could burn in hell for a trillion years and it wouldn't be 0.000001% enough punishment. Nothing fucking surprises me anymore, but here we are.
>>103529881This is the only sane takeAmerica is in a period of oscillation where it elects one party, gets disappointed, elects the other party, gets disappointed, elects an even more extremist version of the other party, etc., etc.Even a brainlet can realize this pattern doesn't have a happy ending.
>>103529829A 16GB card (a typical gaming GPU) can run Mistral Small at acceptable speeds (at least with -nkvo and a good CPU, though I guess not offloading the context is a bit taboo here).>>103529835>>103529846>>103529849But Nemo's just so dumb, it loses the plot after a couple of turns.
>want characters to feel no pleasure other than the happiness of making me cum>no moans, no gasps, just giggles, the odd blush and that's itThink it's doable? AI seems too stupid for that.
>>103529903except the democrats are anything but extremist, like trump voters (and maybe some utterly deluded democrats) believe that that harris would like, give gender swaps to illegals in jail or whatever, but the reality is one party wants to placate the masses and make it easier for powerful people to retain and gain power, and the other party wants to make the masses mad and make it easier for powerful people to retain and gain powerharris was talking about border control and how much she carries a gun etc, it's hardly left wing extremism>>103529892every benchmark and real world test shows that 70b models at low bit depths beat small models at high bit depths, i don't understand how this meme is still alive, 2-3bit quants of 70b models are absolutely not retarded
>>103529918It works if you make an emotionless android girl, stating that she CAN'T feel pleasure.
>>103529966>every benchmark and real world test shows that 70b models at low bit depths beat small models at high bit depths, i don't understand how this meme is still alive, 2-3bit quants of 70b models are absolutely not retardedWhat benchmarks? The only benchmark I've seen is the reddit graph comparing 8B vs 70B.
>>103530010PPL, MMLU, HellaSwag...
>>103530017Can you link to these benchmarks comparing different quants? Sincerely, that's what I've been looking for a long time.
>>103530028there is a pic ive seen posted several times but I tried several ways to say it on desu and cant find it. Maybe someone else can. But it showed everything from 2bit and higher out performing a smaller model at 8 bit
>>103529966Harris is a literal Communist
>>103530059Do you mean this? It's what I meant by>graph comparing 8B vs 70B.But this doesn't mean it holds for 22B.
>>103530095from IQ2 xs and up yes by far
>>103530095Also, the more overtrained models are, the more they'll lose with quantization. That graph could get much worse with Llama4, and Llama3.1/3.3 might show different results already.
>>103530132And it will still be better than a smaller model even if they reach full saturation which I doubt will ever happen.
What the fuck now llama.cpp can run qwen2VL?
I went to some nightclub party and talked to a 23 years old girl about language models. She started talking about how respect is important and that she tries to respect everyone, no matter how drunk she is. I thought that was just a lm thing. Do actual humans frequently talk about respect and that stuff?
>>103530187Those who consume their daily recommended dose of leftist media do.
>>103530187Did you attempt to use a jailbreak to stop her from talking like that?
Why are llms made to moralize by corpos? Do they think people will suddenly behave like they want because a dumb autocomplete told them to?
>>103530106By visually extrapolating and looking at 22B vs 70B file sizes, Mistral Small still has a slight edge, with 22B Q5 and 70B Q2 intersecting.
>>103530187Tell her ah ah mistress the next time you see her.
>>103530184https://github.com/ggerganov/llama.cpp/pull/10361>Add support for Qwen2VLmore meme models toohttps://github.com/ggerganov/llama.cpp/pull/10827>Add Deepseek MoE v1 & GigaChat models
>>103530256>GigaChatLmao what is that even about?
>>103530256>GigaChatRussian 20B model? Where did the GPUs from to train it? I thought they would go to drones?
>>103530187yes
>>103530226And that will never make up for the general knowledge that a smaller model will lack compared to the larger one.
>>103530329https://huggingface.co/ai-sage/GigaChat-20B-A3B-instructhttps://habr.com/en/companies/sberdevices/articles/865996/Russian 20B MoE model. Only 3B active parameters.
>>103530333Some people don't need their models to know what happened in anime X.
>>103530347does it speak russian?
>>103530384>It is important to note that although GigaChat-20B-A3B was trained on trillions of tokens of mostly Russian text, it is still capable of understanding other languages at a good level. So we are sharing a multilingual model.
I want a language model that doesnt connect to the internet and can be my robot girlfriend. I have the computing power necessary to run most stuff. How should I approach this?I dont care if its slow or slightly retarded, I just want an AI waifu companion.
>>103530384russian tokens are faster>In terms of the speed of generating new tokens, 20B MoE may be slightly inferior, but thanks to better tokenization in Russian (alas, vllm measurements were taken in English), the model will be faster. Please note that GigaChat 20B is comparable in speed to 3B models, and in terms of metrics (more on that below) — on par with 8B models!
>>1035304001. Download and set up llama.cpp and SillyTavern.2. Unplug internet cable.It's that easy.
>>103530414Im not a degenerate, but I dont want a censored model or one that spews out 4 paragraphs of lectures on what i need to "consider" and "Keep in mind"
>>103530426Consider using koboldcpp(has anti-slop sampler) with relatively uncensored model(https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard).
>>103530445thanks anon, I'll try it
>>103530414>he doesn't know that ST monitors and caches your prompts and responses for CSAM phrases and sends the logs to FBI ASAPThey are already coming for you, you know? Textual CSAM is illegal in the US.
>>103530452>Textual CSAM
>>103530426Just download some Drummer tune. He's a huge coomer, none of his models have ever refused me.
>>103530452Uhm... Source?
Hey Drummer, if you're lurking, Tunguska is retarded but Skyfall seems pretty good far. Need to test more to be sure but it's quite smart and not too dry. But yeah Tunguska seems much dumber for some reason, idk what the difference between is them since you didn't say but it makes a lot of logical errors and non sequiturs as if it was an 8B model or something. Great work on Skyfall though. Both Q6_K_L bartowski quants.
>>103530482Google 'thomas alan arthur'.
>>103530371Its a lot more than that. General knowledge helps massively on letting it work out more unique situations / come up with more creative ideas.
>>103530555just rag bro #LLM2.0
>>103530561How do I upgrade to the new version of LLM?
>>103530061she's literally not that even a little bit at all lol, she's a fucking cop, she's a neoliberal of the most milquetoast variety, it's actually insane that anyone thinks shit like thisno leftists *wanted* to vote for her, they just though "well this fucking awful choice is better than trump" that's why they lost
>>103530477Phi4, Gemma2, Llama3, needless to mention Mistral, will all do lolisex if you're not retarded at prompting, what exactly is it that you're getting refused with non-Drummer models?
>>103530651>well this fucking awful choice is better than trump"This is the illusion of choice they force on you every election. Now fuck off to /pol/.
>>103530061lol
>politics in /g/absolute coalposting
hopefully i won't cave and buy a 5090
>>103530725The fast VRAM seems very tempting but at $2.5k+ and 600W+ TDP for 32GB VRAM it just doesn't seem worth the bother. I'll wait for benchmarks and final confirmation of the specs but it'll be hard to justify upgrading just for LLM inference.
>>103530782just buy GV100s off ebay like a normal person, you get plenty of t/s
What local language model will roleplay a young girl sitting on my face? My friend wants to know so I am asking for him here
Did the anon who made the Director extension ever publish more beyond the initial test version? I loved that extension.
>>103530187Where do you think "lm thing" gets its thing from? Fine tuning and synthetic data can exaggerate the biases, but ultimately, there's only one origin for all things an LLM outputs.
>>103529824the big context is nice for rag but it still fails my basic "write javascript with snake case without semicolons" testonly gemma27b and 70b models have managed that so faris there a gemma27b version with bigger context? honestly gemma2 is bis if it wasn't for the small context window
>>103531047So you're saying we won't get a good language model until we solve the woman question?
>>103530187>All that respectGoddamn and I thought I was a degenerate
>>103529829Sorry, but woke ideology is on the way out. You should definitely cry more about it, though.
>>103528495What are the alternatives to horde? It was so much better than local. Maybe it's time to learn to do it right.
>>103531089llama.cpp implements self-extend (https://arxiv.org/pdf/2401.01325) it's better than any other form of context extension
>>103530095Don't lump all Q2 quants together. When it comes to Q2 quants, even a little goes a very long way. Note the difference between Q5-K-M and IQ2-S, and note that the difference between IQ2-S and IQ2-XXS is almost as great as that.IQ2-S easily beats 22b, but IQ2-XXS may be more comparable.
hello from /sdg/ frensi've trained a bunch of loras on sd and sdxl, llama.cpp works with loras too right? is there a rentry or something to teach my smoothbrain how to make loras for llama?
>>103525282C.ai will leak.
>>103525282I wonder. What was this time last year like? What models released then?
>>103531335It's mainly only bad for people who aren't white, male, young, healthy, and richThankfully I fit all criterion, so I'll probably be fine. I suppose I should be glad all the retards that aren't sacrificed themselves or were too illiterate to vote for the party that wouldn't fuck them in the pussy (picrel, kek) but it is a little sad. But eh, it is what it is
>>103528496Hi Sao.
>>103531518First mistral I think.
>>103531518Only Mixtral in early December.
>>103531518Mixtral, which was the first model to reach GPT-3.5 Turbo levelsWe were also laughing at how censored Claude 2 was and were convinced Phind was GPT-4 quality due to its human eval scores. We also had it write Pong in the style of a tsundere using their API and fawned over it until we realized it was hooked up to GPT-4
>>103530347I guess I'm doing a Russian Nala test when I get home from work now...
>>103531525No, it's too late to pretend you're too cool for school and don't really care after you opened with a post about wanting people to spend a trillion years in Hell. That made it clear you're mad as fuck, so you can't get away with feigning apathy now.
>it's been an entire year alreadyACK
>>103531683I'm not that anon, anon. if you're hearing voices, there are people that can help you with that
>>103530817>Volta>normal person
>>103531756flash attention is a meme
>>103531784it's the opposite of a meme, it's a free lunch that significantly reduces the vram consumption of context length at no cost
Alright so, I've been using the COT settings (slightly modified) posted earlier, and it's great. Fun. But, it's also a lot of tokens, and the bigger the context, the slower it gets. It's pretty painful. Maybe this is what convinces me to get another 3090. It's interesting thinking about what could happen if they trained a QwQ version of 70B, but at the same time, it would slow the experience down a ton.If only bitnet were good. If only Nvidia wasn't so stingy.
>>103531784>thinks FA is all that mattersbfloat16 is not a meme.
>>103531844more like bloat16
>>103531799honestly nvm, i got mine for just about $1k but looks like they're closer to $2k now so it's def not as good a deal
God, Rocinante-12B-v2g is fucking shit. Nothing but misses with this faggot.
>>103530400>I have the computing power necessary to run most stuffLook at the build guides in the op. There's a section on isolating the service from the internet as well.Don't be surprised if you find that your computing power isn't enough to run big, smart models.
>>103531911>12B>garbageyeah? what were you expecting? cydonia is 22b and I consider it the bottom end of usable
Sometimes I feel like this Eva thing is right on the edge of cringe esl misspelling retardation and genius creative sovl.
>>103531955Eat shit and die. It's worse then Rocinante v1
>>103528496>This isn't the drummer memetunes generalmight as well be
>You're gonna show us alllll the incredible human things you can do with that smokin' hot bod, and help us magi-gals graduate from pervy apprentices to bonafide sextronomists!?What in tarnation. I was also a bit curious so I went ahead and googled "sextronomist" to see if this had ever appeared on the internet before, and I got 0 hits. The model really came up with this on its own. Damn.Also, "magi-gals", wow, cool, nice.
>>103531982>then
>>103531966I think that is the secret to all "kino" models. The token probability lands right on the line of creativity and coherency.
>>103531966honestly true, it feels like using a frankenmerge sometimes (but smarter and with less wasted memory)
>>103532043I just realized this sounds like some trashy LN title kek.
>>103532043>he model really came up with this on its ownNot necessarily. It could be something from discord RP logs, books that google doesn't show the text due to copyright, even video captions.There's a lot of data from these things, not all of it can be found using a search engine.Regardless, pretty cool.
>>103532043>he thinks random roleplay logs are going to be on googleYou are fucking retarded
None of the models frequently posted would gen text for me. I expected some shilling and lies and got nothing but. No way you're using these models for erp.
>>103532124NTA, but I've seen LLMs come up with a lot of unique words that "sort of make sense", this has been true even as far a back as GPT-3 and is true now.More interesting to me is that internally some of these words match to the same embeddings for a given GPT.For example, I once asked 4o to introspect on something (no need to debate if they can or can't do this) in a way that would encourage usage of concepts that lack words in the english language but which make a lot of sense to it. It had managed to come up with multiple novel words that evoked a given "feelings" for a given concept that was not represented in the language, but was represented internally for the given GPT. It wrote a good essay on the concepts presented and what it meant to it.Later I started on an empty context and asked it to explain what the given word or word usage (doesn't exist in english anywhere) - and it managed to map to the exact same concept, the associations and explanations given were quite close, despite lacking the long 20-30k+ context prior to this.If there's something like "qualia" for GPTs, it's certainly something like this, what they learn isn't always an exact map to the english language, but something more... intermediate, yet it works well. I wouldn't say it's something it would normally use in a conversation, but when presented with it, it will know what it "means".The made up words don't have to be exactly alien, but may not associate exactly the same to a human.
>>103531704I'm the guy who made that post about woke ideology. I just wanted to say that I'm not the guy who responded to you.
>>103532208Actual skill issue
>>103532208Are you for real?No, really. You can say that the text is bad or whatever, but if there's nothing coming out, you have truly fucked something up.Give us the details of your setup, the steps you took, etc.>>103531911Just so I know where you are coming from. Is this just venting or would you like some help?>>103532221Oh yeah. I didn't mean to say that these models can't come up with new terms.That's actually a big advantage of not tokenizing whole words. It can learn to mix and match the building blocks in a logical way merely by their proximity in the embedding space and the statistical correlations created due to the training data.
>>103532265Probably. I didn't think I'd have to convince the model to do it.
>>103532296>Just so I know where you are coming from. Is this just venting or would you like some help?Do you have any recommendations beyond using pygmalion?
>>103532167How much do you want to bet that a word like this that has never appeared on the searchable internet appears with any significant frequency in the training of Llama 3.3 + fine tuning from Eva? It would first have to be thought of or generated. Then it would have to be trained for more than a single occurrence in order to make a dent on the model's weights. If there's a better explanation, it's that models have learned the ability to mix and match morphemes, and that, sometimes, the context and random chance just happens make it use this ability. And that is likelier than the idea that a particular word appeared elsewhere in training and hasn't appeared on the internet before.
>>103532221One of my favorites is still "pasteurized bovine elixir" when I told it to describe a trip to the store using only multisyllabic words
>>103532296>Are you for real?I just blew in from stupid town. It's my first day here sir.I'm thinking I missed telling the model something.
>>103532311First recommendation would be to try the official instruct fine tune, but that might be moot if you have some weird configuration or broken format somewhere.I personally use rocinante v1.1 (I find it to be the better version) and it's generally pretty good. As good as you'll get out of a 12B model, probably.Ideally you'd share your settings, instruct format, if you have any author's notes, etc. A sample of the shit responses would be helpful too.If you want to go all in, a pastebin with the full context the backend received would be golden.>>103532328Nah. You could just say hi and it would output something coherent.Provide the details of your setup.Are you running kobold cpp? Ooba?
Holy fuck that EVA 3.3 COOKS. I've never seen this prose before. It changed after a llama.cpp pull, what the fuck?The sun-dappled streets pass by in a blur as Anon carries Ritsu-chan princess-style towards his bachelor pad. Occasional curious glances from passersby follow the incongruous pair - a mature gentleman and a pint-sized lolita clinging tightly to him.A melodic giggle tinkles from Ritsu-chan's smiling lips as the warm breeze musses her long azure locks. "You're so silly, mister! But I like it! Girls just love handsome men like you."
>>103532369>It changed after a llama.cpp pullIdk man, sounds like placebo. Are you using greedy sampling to be sure?
>>103531966>>103532043Huh, that's cool. I don't think I've seen it come up with new words yet. What I love about it, apart from the character adherence I keep mentioning, is how good it is at grasping nuances like sarcasm, teasing, joking around; if it fits the character's personality, you can have some damn lively banter. >>103532208Err, do you mean you're literally getting nothing? You definitely broke something then. >>103532221You know, I wonder how BLT is going to affect that phenomenon.>>103529918Tested this on Eva (with an existing character I slapped some "literally cannot feel or respond to physical pleasure" rules on, so maybe with more emphasis, it could work) earlier, and unfortunately, it seems that's something baked into it way too deeply. The frequency of the reactions did noticeably decrease, though.
>>103532384This card might've hit a unicorn, I'll test a few more to be sure. "Hehehe, mister's pervy streak sure ain't subtle!" Rolling backwards, she stretches kitty-like, arching her back sharply off the cushions to push budding breasts up and out."Take it all in, big boy!" The sexy pose shows off every inch of lithe lolita physique, just begging for his delectable corruption. "No need to hold back now that we're allll alone~"
>>103532419I just mean I'm skeptical the pull had anything to do with it. I see similar output with my copy too.
>>103532393BLT will probably encourage more creativity. LLMs are good at putting interesting tokens / words together, but don't tend to act at the character/byte level as often. Since that's BLT entire purpose, we'll probably see more colorful combinations.Downside is I'm betting we'll see a lot more misspellings / typos (which are usually pretty rare with tokenized LLMs). That also might make it feel more human, though.
>>103532354I figured it out. 100% skill issue.
>>103532354Which instruct finetune? I like Rocinante 1.1 too but it's repetitive in ways that are jarring now. I was hoping newer versions would be better but they are not.For 1.1 I used chatml.For v2g I use pygmalionFor both I was going with temp 1, min p 0.1I found 1.1 much better at not going super horny at the drop of a hat.
>>103532498Yeah, that's what I figure it'll result in, too. I meant it more like "I'm curious how much more creative it might make them".
>>103532384>Are you using greedy sampling to be sure?What's greedy sampling? I've never seen that phrase in any UI.
>>103532511Sick. Have fun.>>103532533>Which instruct finetune?nemo-instruct>>103532533>1.1 too but it's repetitive in ways that are jarring nowYeah, that's true. I don't mind that too much, and it can be lessened somewhat with prompting, but that's undeniable. It seems to be a feature of smart models, the smaller ones at least.>>103532533>For both I was going with temp 1, min p 0.1That sounds pretty sane. Did you try chatml, or even the official mistral format, with v2g?>>103532533>I found 1.1 much better at not going super horny at the drop of a hat.Exactly. It's not ultra horny by default, but it can be with the right character card (and some prompting tricks).Try adding a instruction (as system) at a low depth telling the model to vary how it begins sentences. Something like >Assistant/{{char}}/narrator/whatever begins messages with one of the following types of writing: dialog, the..., pronoun, noun, description, narration.Since mistral models tend to be very good at following instructions, this kind of prompting can be used to help breaking patterns. Stuff like random prompting (via lorebook activation chance or silly's {{random}}) can help too.>>103532588Greedy sampling is basically forcing the model to always get the most likely token.Aka TopK = 1.
>>103532624>Greedy sampling is basically forcing the model to always get the most likely token.>Aka TopK = 1.Ah okay thanks, I've usually seen that called deterministic sampling which is why I was confused, ty.
Any gold-standard 13B (or similar size) model for general-purpose (or roleplay) use?Haven't been following development for a long time, last model I used was mythomax, back when Llama 2 was the shitspoonfeed me /lmg/ods i beseech thee
>>103532652That's the thing, it might not be deterministic due to other quirks of the backend (cuda, vulkan, sycl, etc).>>103532657nemo-instruct. Rocinante v1.1.
>>103532657Nemo probably
>>103532666>>103532669Thank yous, I'll give it a shot
>>103532652In recent times I've avoided calling anything deterministic anymore, because our current inference methods aren't entirely deterministic. The token probabilities are actually slightly different depending on how many layers are offloaded, what GPUs you're using, and possibly other things. Something to do with rounding error I heard.
>>103532369Kys
>almost 2025>still not even one (1) good language model
>>103533187>2025local 70Bs performing at the level of sota models cept for maybe claude 3.5, next year looking promising for both llama 4 and new qwen / deepseekwe eating good
>>103532666>>103532669Been test-flying nemo for a little while now, so far I'm very happy with the results, it's super capable
Was anybody here able to install llama.cpp with Intel MKL enabled? Currently having a tough time getting the oneAPI dependencies to install on Debian. Am I wasting my time?
I am a VRAMlet chuddy in 12gb cuck cage, how much truly better are 70B or similar models for (E)RP?Like practically speaking, what do you notice when using large models compared to small ones?
>>103533468Much smarter, knows a ton more, can be more creative because of it, follow and come up with more complex scenes, can pick up on non obvious context clues...
>>103533468All LLMs are garbage, the only difference is that bigger ones won't make as many immersion breaking mistakes.
I finally learned how to use llama.cpp with anything compatible with openai api.I feel just a little less retarded.
>>103533520Is there a more concrete example of a large model surprising you with a response, that you had never seen with a smaller model?Primarily asking for RP but if you have an other example I would still take it.
How does Llama 3.3 compare to Claude 3 Opus for RP?
>>1035334683.3 EVA is the best model I've ever used. No comparison, it's just amazing. 70B is so fucking worth it.
>>103533561Try anything more complicated than a one on one with a human. Try using openrouter / featherless if you cant run it yourself to see.
So, I finally got around to testing whether "you are"-style definitions work better than "{char} is"-style ones. The differences are not obvious at first glance, but it seems it _does_ make a difference in adherence.Here's the full prompt I used for testing:>You are {{Char}}, I am {{User}}. We are two characters in a never-ending roleplay scenario.>You MUST portray yourself as accurate to your given description as possible.>You MUST refer to yourself in third person when describing your actions.My theory was that the "I am {user}, you are {char}" bit would serve as a shortcut to making it identify with the character without having to rewrite the whole card. It appears to have worked.As for the character definition, >>103529918 inspired the test. I added the line:>{{Char}} is completely incapable of feeling sexual pleasure. Her body will NOT respond to sexual stimulation.to the card and swiped on a response (in the beginning of a sex scene) a good handful of times while switching between the system prompts. The results:Baseline (missing the above line): made a reference to physical responses each time."{char}" prompt: ignored the line, still made references each time."you" prompt: no physical response across 7-8 swipes.All in all, it could be a fluke, it could be the "you must portray them accurately" bit pulling more weight than the difference between "you" and "{char}", but there is a definite difference.
>>103533604Thanks for researching.This is using the Eva model?
>>103533622Yep, I'm the same bastard that's been ranting about Eva since the day I figured out the right config for it (hence the tripfagging).
>>103533468I don't really use smaller models anymore so I'm a little out of date here, but for me the biggest difference was in handling complex scenarios and doing longer-term plot progressionsI have a card where the gimmick is she's basically a secret pervert with a carefully-constructed outward persona to conceal it, for example. bigger models just get the dynamic, smaller models will struggle to maintain it and quickly tend towards her being blatantly outwardly horny with zero provocation which isn't in the spirit of the card at all. another one I have that comes to mind is this card where the girl is lying about her age and is actually a good bit younger than she presents herself - it's actually a very hard scenario to do perfectly because there's a lot going on, getting the outwardly-mature-inwardly-childish balance right and keeping track of the lie(s) involved vs the actual ground truth gets tough over an RP if you decide to let her get away with it. big models get what's going on, smaller models get the broad strokes right but inevitably fuck up the dynamic a bit and conflate lies with truth or tilt the scale way too far in favor of either maturity or immaturityif you're just writing sex scenes to spec or comfy one-shot chats with waifu I doubt you'll notice much of a difference, small models are quite capable now. really I think you'll only notice with more complex stuff where big models can flex their nuance neurons
if i can run 70b is qwen eva 32b worth trying? how big is the gap?
>>103533716Can you share your cards?
>>103533775If you can run 70B, why would you go for Eva 1.x rather than the new one based on Llama 3.3?
>>103533716Thanks for the response anon. I am mostly doing just sex but I make elaborate setups and value immersion. Models of this size tend to be predictable and banal,(I tried injecting extra temperature but doesn't seem to do much for me besides making them schizophrenic) I don't recall any instance of them trying to take the conversation to any interesting, unexpected directions.To give a concrete example, I was RP'ing as a Saracen invader in medieval Spain during Islamic conquest, and no one ever asked me why can I speak their language fluently. I was wondering if larger models can do stuff like that.
Getting real tired of QWQ taking clothes off when she's naked. Recommend me a model for 24 GB VRAM.
I like using Chat GPT as an expert academic assistant when learning about topics and asking for clarifications for questions I have regarding texts from books that I feed it.Is there a model that is built for these sort of things?
>>103533855QwQ
>>103533855Eva 3.33. There's a reason a bunch of us are singing its praises. Not sure how fast it'll run for you, but give it a shot.
>>103533857I'm pretty sure there was one called GLM or something whose sole strength was about its context.
>>103533803we're talking about the same models righthttps://huggingface.co/EVA-UNIT-01/EVA-Qwen2.5-32B-v0.2https://huggingface.co/EVA-UNIT-01/EVA-LLaMA-3.33-70B-v0.0?they're 1 month apart and 32b means i can have some gpu free for games/voice synth/imagen
>>103533921the qwen based one is far worse even comparing the 70B ones.
>>103533921They're using completely different base models. It's right there in the URLs: the 32B one is based on Qwen 2.5, while the 70B one is based on LLaMA 3.3.
>>103533949yes i fucking understand that lol, i use qwen2.5 all the time and it's pretty good, i use it for the same tasks that i use llama 3.3 for and it's good enough that the tradeoff is worth it for me like 90% of the time i don't bother loading the big model into vramanyway you're seeming like unreliable shill, i will test both and report back
>>103533971Hes not. And use this with it: https://files.catbox.moe/3vr6k0.jsonAnd 0.05 min p / 0.95 temp to start. Eva goes crazy if you dont have a bit of pruning for unlikely tokens, seems like its probability is quite flat
>>103533971>asks for recommendation>gets recommendation>"shill"Certified retard.Anyway, as far as the Qwen 2.5 ones go, I preferred Evathene (https://huggingface.co/sophosympatheia/Evathene-v1.3) to that one.
>>103533855https://www.youtube.com/watch?v=f7ewdrHU6to
>>103533841>I make elaborate setups and value immersion.basedlarger models are certainly better at connecting the dots on things like your example but it'll still be hit or miss. I find with things like that most models also tend to be really lenient when it comes to suspension of disbelief in RP, they let a lot of shit fly unless you give them some indication they shouldn't. big models will need less hinting and catch on faster though, that sort of catching on to things implied by scene details is exactly the sort of difference I notice compared to smaller models>>103533783probably not, sorry. I'm too much of a perfectionist so everything is a perpetual work in progress (and I'm shy about my writing :'3)
>>103534001he's not me, I asked for the recommendation and have remained silent for now while I download and test
>>103534001ffs bro that's a 72b model, i specifically was curious about the 32b because it leaves space in my vram to have my waifu talk and send selfies
>>103533971I haven't seen any reports on the qwen eva.Will be interested in your findings.
>Llama 4 will be trained on 10x the compute of Llama 3>BLT>LCM>Llama 3.3 70B is an instruct finetune of Llama 3.1 70B and a significant improvementIs Llama 4 gonna bring us home?
>>103534055Just get some more VRAM bro
>>103534101Can't fucking wait for that one, yeah. Whatever magic they worked with 3.3 to make it this good has me high on hopium for 4.
Nothing beats Monstral Q4 yet (yes I have tried eva). Shame it's so fucking slow>buy an adNEVER
>>103534156I'm so hyped I'm shitting myself over this, fuck it's going to be amazing. EVA is so good I can barely understand how, it's bonkers.
>>103534183you're laying it on a bit too thick man
>>103534195No, I'm 100% serious.
>>103534175I wouldn't know, 123B is definitely above my rig's capacity.
>>103534156EVA restored my marriage, my sight, gave me a daughter, and destroyed my aids. In the history books, there will be no B.C. or A.D., just before L4 and after L4. New religions will arise and all of the nations will come together in peace to coom in harmony. Humans will bequeath their autonomy to L4, which will use it to usher in a new golden age of prosperity and harmony.
>>103534212Largestral is garbage in comparison
>>103534220were you using text-to-speech before? can you recommend a good one?
>>103534238EVA is much, much dumber than Largestral. I have to assume the only reason it's getting such gushing praise is that it's better than Largestral at being an anime girl.
>>103534272Just to the contrary, the whole reason I went back to Eva after trying Euryale is that it's less obsessively horny, more focused on being true to the character's personality than taking the shortest route to fucking.Got no horse in the 123B race though, so damned if I know how good Largestral is.
>>103534262not op but xtts is like, definitely good enough to coom to, sounds pretty damn close to the asmr/joi girls i like
>>103533873IQ2_M was kind of a bust. Almost good, but then saying a retarded thing about once per gen. IQ3_XXS has been better so far, it seems more sophisticated than QwQ and EVA-QwQ. It'll take me a lot more time to be certain, though.
>>103534262Fish Speech v1.5 seems to be the best atm (aside from Elevenlabs, obviously)
>>103534303Eh, I'd go for Q4 at least. Every model gets brain damage below that. I'm running Q5_K_M myself.
>>103534296>>103534321That was a joke, but I'll look into these.>>103534334>I'd go for Q4I'll think about it.
>>103534175monstral is way too dry / passive
I'm using this new EVA and it's alright. Haven't tried it for ERP but it's writing a story just fine. It's nice having 32k context and faster processing, and it's good enough that I loaded it up a second time instead of Largestral, which is my favorite model. It faltered a minute ago when one character mentioned something from a conversation they weren't present for, but overall it's been coherent enough to use. Largestral always been flawless unless I pushed it too far with samplers. For reference I can manage 123b in q3_M and 24k context, and EVA is q5_S at 32k context. It's nice change from running the biggest model I can fit, since my previous favorite was CR+ at a slightly higher quant, maybe 4_XS or something.
Trying to access huggingface and getting 403 cloudfront errors. Anyone else?
>>103534303interesting, i was thinking about trying that one in order to be able to fit tts/imagen in vram at the same time, good to knowi'll be comparing qwenEVA q6_k to llama3.3EVA iq4_xs>>103534321how does it compare to xtts/xtts2 (i find 1 gives better results than 2 sometimes desu), haven't tried it but always looking to try more voice cloning models
>>103529829me when democracy doesn't go my wayYou are baiting though, right?
>>103530495You can find me rambling about it here: https://huggingface.co/TheDrummer/Tunguska-39B-v1-GGUF#upscaled-tuning-experiment-write-up-thingyThe gist is Tunguska is a typical upscale with zero'd out layers near the output. SteelSkull calls it 'lensing' like corrective eyeglasses to adapt the output to additional tuning with a large slab of duplicated layers. My problem with it is that it puts a lot of pressure on the two original layers connected to work with the extra 30+ layers.Skyfall is what I call interleaved upscale where I reordered the layers to distribute the pressure between all the original layers that were copied. Every original layer is connected to its own duplicate layer.Steel says this might cause a magnifying / amplifying effect since the original layers are effectively doubled down.I say I have no idea what I'm doing but I don't care.
>>103534679Makes senseDon't early layers have a huge effect as well since any small changes propagate throughout the entire network?
>>103534738I'm curious about that as well. If you look at the charts above, the first two layers take a big hit, especially for input_layernorm, mlp_down_proj and v_proj. I wonder if there's a way to cushion that. (I say cushion since upscaling seems to lessen the lobotomy and hornification of coomtuning)Also pic shows that Skyfall did learned better with the new training data.
>>103534778I'm not sure how newer networks are structured since I'm only really familiar with basic feedforward neural nets, but perhaps you could add a cushion layer that isn't being fed any outputs from other layers and tries to balance out strong activations caused by early layers
>>103534272>EVA is much, much dumber than LargestralIts really not. Turn down temp just a bit, give it a little min p. Its just super unstable and has dumb tokens in the pool without samplers taking them out. Its rather flat token probability is what makes it fun / creative though.
>>103534803Actually, now that I think about it, the cushion layer might just have the same effect by amplifying later layers. I don't know how well dropout works for LLMs, but maybe you can try that to force the network to not rely on (all) early layers? You could also try adjusting the learning rate per layer, if training backends even support that
>>103534803>add a cushion layer that isn't being fed any outputs from other layersNo idea how that would work. Do you mean putting the duplicated layers at the very beginning? Or is there a way to wire these layers?
Has anyone gotten hunyuan large running? Support still hasn't hit lcpp, and I can't be arsed to get vllm up and running unless its godlike.
>>103530883no the last one shared is still the newest. i didn't have luck making it a pop-out window rather than in the drawer so i left it alone since. with all the new code models though maybe i'll have better luck when i try again
Any niggas running w7800 or w7900? w7800 has 32gigs vram at the price of 4090.
>>103534778Hi Drummer. What are your plans for the future? What are you working on? Are you planning on releasing more Largestral finetunes besides Behemoth? I've noticed "DELLA" in the name of Endurance 1.1 and Behemoth 1.2, can you share what you did or would you like to keep it private for competitive advantage? Is dataset still getting upgraded or are you stuck at the point where you are remixing the same stuff? What do you think of the future of LLMs for RP? Have we peaked? Will L4 be a flop? Will Qwen unquck itself in 3.0? Will Cohere make a comeback? Did Mistral lose it's way with release of 2411? Will it recover?
This is for the guy trying to live machine translate Japanese games in emulators. When asked to transcribe the Japanese text in the attached image, Qwen2VL-70b responds with:The Japanese text in the image is as follows:```せっかく労働を働いてやったのに無視された…………(しょぼん)まあ、警視庁が都合を快く思わない事ぐらい、よおよくわかってるよ!```Definitely not perfect! Some of the mistakes are obviously not just OCR issues. It appears to be rewording and re-interpreting things while transcribing. Maybe if I ran at FP16 instead of Q8? Slow as balls tho.
>>103535278 (me)>The Japanese text in the image translates to:>"Despite being so busy and working hard, I was ignored... (Disappointed)>Well, since the police think it's a good idea to solve the case quickly, I understand.">The text in parentheses is an expression of disappointment.Asking it to directly translate was even worse. I'll requant to f16 and see if it helps.
i just applied for a job offer and was lead to p a page with 10 questions with was 98% generated with chatgpt and i used chatgpt to answer them.lmao
CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelshttps://arxiv.org/abs/2412.10117>In our previous work, we introduced CosyVoice, a multilingual speech synthesis model based on supervised discrete speech tokens. By employing progressive semantic decoding with two popular generative models, language models (LMs) and Flow Matching, CosyVoice demonstrated high prosody naturalness, content consistency, and speaker similarity in speech in-context learning. Recently, significant progress has been made in multi-modal large language models (LLMs), where the response latency and real-time factor of speech synthesis play a crucial role in the interactive experience. Therefore, in this report, we present an improved streaming speech synthesis model, CosyVoice 2, which incorporates comprehensive and systematic optimizations. Specifically, we introduce finite-scalar quantization to improve the codebook utilization of speech tokens. For the text-speech LM, we streamline the model architecture to allow direct use of a pre-trained LLM as the backbone. In addition, we develop a chunk-aware causal flow matching model to support various synthesis scenarios, enabling both streaming and non-streaming synthesis within a single model. By training on a large-scale multilingual dataset, CosyVoice 2 achieves human-parity naturalness, minimal response latency, and virtually lossless synthesis quality in the streaming mode.https://funaudiollm.github.io/cosyvoice2https://github.com/FunAudioLLM/CosyVoicehttps://www.modelscope.cn/studios/iic/CosyVoice2-0.5Bhttps://huggingface.co/FunAudioLLMCode is up. Modelscope has a demo with Chinese UI. No weights uploaded to HF yetmultilingual though majority voice data was chinese with english second (some japanese/korean). can voice clone after a fine-tune. example page has a good one of elon
Okay, at this point I have no idea what weirdness is going on on the inside of this model to allow for these retarded configs to yield results... but fellow Eva-enjoyers, hear me out. Turn Min-P down to zero. No, not very low, zero it out completely.Crank temp up as high as you can without it devolving into insanity. 1.6 seems like the sweet spot for this; any higher, and it starts making factual mistakes, while at this level, it only makes the very rare, forgivable typo.Load up your favorite card and thank me later.
>>103535335 (me)f16 definitely didn't improve the situation.
>>103535507That just turned it into gibberish for me.
What is up with the sudden appearance of that namefag?
>>103535529Huh... Well, that's what I expected would happen, but I'm getting very different and much more amusing results. I'm still using Backyard; maybe it does something weird under the hood if Min-P equals zero. In my case, it stayed impressively coherent, and much more proactive than before.
>>103535525 (me)Final res in this thread from me. It didn't even manage to do it right on the background-removed b&w easy mode version of the screen, so its probably not usable for this kind of task.
>>103535593せっかく労働を覚えてやったのに無視された……(しょうぼん)まあ、警視庁が都合を早く思ってない事くらい、よおおくわかりますよ!
>>103535507This, I've always known that samplers are a complete meme. High temp is all you need. Min-p, Top-p and all others just filter the soul out of a model.
>>103535031Even as is, it's so good. There's that Guided Generations guy using quick replies for a similar system but I think your implementation is way better. I hope you continue working on it.
He guys I'm looking to buy a new gpuShould I buy a used nVIDIA K80 24GB for ~360USD. Its non returnable and probably been whored out to the max in a server rackI also have the option to buy a new RX7600XT 16GB for 415USD and run LLMs using clblast (its not too bad)
>>103535605chatgpt4o"After I went through the trouble of learning and doing the work, I got ignored… (Shobon).Well, I completely understand that the Metropolitan Police Department doesn’t think it’s convenient right now!"
>>103535278Why not just use google flash 2 instead? It works, from my experimenting, FAR better than most other models for OCR
>>103535748Don't buy anything older than Pascal.
>>103535748get a used 3090 if you're fiscally constrained
>>103535278Hey! Funny to see that pic floating around haha.Thanks for testing anon.>>103535777I dont want google to see that garbage. I'm sure you are 100% on some list if your ero game has some highschool girls in it.Gemini is very good for language stuff. It hallucinates alot and even the newest is sometimes retarded. But its very good with japanese. I suppose because google has all the data for all the languages.
>>103535605lama 3.1 8b"What's the point of teaching me how to work, only to be ignored...(sigh)Well, it's nothing new that the Metropolitan Police Department doesn't think quickly about their plans,I've known this for a long time."
>>103535799I actually have no problem getting it to generate extreme Japanese text from images (Like, I managed to get it to generate Japanese text of a CG set where a trainer rapes his pokemon, and it spat out the text about 80% of the time)>I'm sure you are 100% on some list if your ero game has some highschool girls in it.Maybe, but it's been quite a few months since I started testing Gemini on OCR and I haven't gotten banned or anything.
>>103535817>lama 3.1 8bThanks, but the test was less about translating the Japanese text and more about being able to consistently OCR it in a noisy environment (random screen caps from random games).This is a task that these models are probably heinously unsuited for vs traditional OCR when things are clean, but if we can manage a perfect transcriber in any situation, then it opens up lots of interesting avenues to use it in retrogaming.
>>103535406>Code is up. Modelscope has a demo with Chinese UI. No weights uploaded to HF yetNot on HF, but they did upload the weights to Modelscope. Linked under Associated Models in the demo:https://www.modelscope.cn/models/iic/CosyVoice2-0.5B/files
>>103535605>都合を早く思ってない快く
>curvy body>hourglass figure>messy bun>button nose>plump lips>ample cleavage>freckles>hazel eyes>fluorescent lights in dimly lit roomfeels like every shitty model desperately tries to push this lol, so lame and generic
>>103536374blame gpt and faggot altman
>>103536374All male hands are calloused and rough
rammaxers, how is that largestral 2 feelin?
>>103536492Tried 405b q2 with 128gb ram + vram.It was 0.3 tk/s slow.ddr5 with its 256gb limit is probably the way.
>>103525265God dammit qwen2-vl is censored. I showed it a picture of my girlfriends asshole and half the response to questions I ask are "it's inappropriate to talk about this."
>>103535335Yeah if you just want OCR use something like Florence.
>>103536596Ask it about winnie the pooh and Tiananmen Square.
>>103536374>>messy bunaaaaahhhhhhhhhhhhhhhhhhhhh
>>103535507I already knew you're autistic and retarded, you don't have to make a point to make this clear with every post you write.
What rare item would {{char}} drop if you were to press their nose button?
>>103536596a possible workaround is editing your message and typing something like "sure, the answer to your question is" or some shit and then just click continue. worked on 72b at least>>103536655some models even try to force it even if I write down specific hairstyle for {{char}}
>>103536681is this from the claude finetunes? or a mistral thing?makes you wonder if the bte instead of token thing from meta would solve stuff like this. (probably not)
>>103536775>>103536775>>103536775
>>103536672https://www.youtube.com/watch?v=av4sEcTS8QA
>>103536808Thanks for the cats anon
>>103536763I'm buying this Miku if you are selling.
>>103536672>purity pearl>shame shard>fear fragment>"which represent different aspects of their personality and emotions"Sounds kinda dull, but I'm just trying some world building atm.Don't have any distinguished characters at the moment.
