File: __hatsune_miku_kasane_tet(...).jpg (1.7 MB, 3840x2160)

1.7 MB JPG

►Recent Highlights from the Previous Thread: >>102290284

--Suggestions for lightweight TTS, including xtts v2 + rvc and MeloTTS: >>102292243 >>102292268 >>102292289 >>102292605 >>102292682 >>102292705 >>102292775 >>102292819 >>102295172 >>102295237
--Recommendations for high-performance deep learning and computer vision setup: >>102290547 >>102290671 >>102291097 >>102291351 >>102291456
--Model not fitting in VRAM causes slower performance due to offloading: >>102294462 >>102294635 >>102294846 >>102294878
--Instructions for installing llama.cpp with GPU acceleration: >>102291059 >>102291104 >>102291199 >>102291210 >>102291243 >>102291248 >>102291310 >>102291342
--Detailed choose your own adventure prompt template: >>102292614 >>102293498 >>102295179
--Reflection Llama 3.1 performance improvements and uncertainty: >>102291297 >>102291496 >>102291516 >>102291519 >>102291534 >>102291548
--Reflection Llama 3.1 70B benchmarks and discussion of overfitting models on riddles: >>102290385 >>102290411 >>102290924 >>102291043 >>102291336 >>102294817
--Recommendations for TTS AIs for low-end phones: >>102292577 >>102292623 >>102292640
--Mistral Large prompt format with [INST] tag: >>102291403 >>102291417
--LLM counting issues due to token limit and repetition: >>102295092 >>102295125 >>102295133 >>102295149 >>102295303 >>102295333 >>102295354
--Generating music with llama.cpp and limitations of open-source models: >>102293062 >>102293159 >>102293354
--Fitting 13B model into 16GB VRAM and performance expectations: >>102292824 >>102292833 >>102292867 >>102292895 >>102292912 >>102292922 >>102292979 >>102292993 >>102293050 >>102293138 >>102293249 >>102293319 >>102293810 >>102293852 >>102293864
--Challenges of moving offsite and concerns about 4chan's future: >>102293203
--Miku (free space): >>102290406 >>102290712 >>102290755 >>102292397

►Recent Highlight Posts from the Previous Thread: >>102290292

>>102285536
just wanted to report in and say Starcannon was absolute shit. a complete downgrade from Magnum in every way.

>>102296967
congratulations, you have learned how to test models yourself instead of expecting to be spoonfed

>>102297037
buy an ad, shill. your models are shit and no amount of malding will fix that.

>>102296939
>that miku
It's not Halloween yet, OP

>>102297052
of course it's shit
I told you to use it knowing that it's shit
why would I tell you to use a good model, then you'll just keep coming back here asking 'gib sota plz' every few weeks
instead you'll now never trust anything this thread tells you and have no choice but to research and experiment on your own

>>102297037
>>102297052
>>102297120
what makes you trannies so angry?

File: 1698372665738050.png (105 KB, 1606x488)

105 KB PNG

wish me luck...

>>102296944
> Challenges of moving offsite and concerns about 4chan's future
I know there’s the suggestions black hole, but surely Hiromoot is doxxed by now or somebody on the planet has actual fucking contact information for the guy to force his hand on /ai/?

>>102297172
completely unrelated to what you said (an /ai/ board would be fucking trash), that anon thinking 4chan will be dead in 3 years is hilarious. I've been watching people say that since 2004. we're all stuck here forever.

>>102297187
I don’t care about it being trash. It’s basically criminal negligence at this point.
>dead in 3 years
Maybe 30

>>102297171
Good luck. I took a gamble on whether the spot instance I used to get things set up would make it to the first checkpoint and lost.

>>102297120
unfathomable cope.

>>102297171
What's in the oven Anon?

>>102297336
Mistral models on prose/stroywriting/completions. I'm trying it with base Nemo first.

>>102297170
they have no compute

>>102297366
Surely you're using the Nemo Base model for this and not the Instruct, right?

>>102297503
>I'm trying it with base Nemo first
>I'm trying it with >>> vvvv <<<
>I'm trying it with >>> base <<< Nemo first
>I'm trying it with >>> ^^^^ <<<
gee I wonder

guys I have bad news it turns out miku is 16

>>102297581
hag...

>>102297581
too old

Apparently reflection api was sonnet 3.5 all along
https://www.reddit.com/r/LocalLLaMA/comments/1fc98fu/confirmed_reflection_70bs_official_api_is_sonnet/

>reddit
go back

>>102297604
kek, grifters btfo once again
>>102297621
hi matt

>>102297604
>ledditor sees a thing /lmg/ was talking about for hours on leddit and brings it back here as if it's a new thing
go back

File: 1699555857560950.png (54 KB, 655x479)

54 KB PNG

>>102297171
1/4th in, back to the drawing board I guess

>>102297604
That shit's old news, see last thread. Time to be buried along with whatever investor money fell for it. For clarification, that was the free one on OR which doesn't work anymore. After an hour of free sonnet proxy it went schizo and switched to GPT-4o and a few other models before dying.
The paid model on OR is the llama based model, maybe ref_70_e3 (or an older one idk) on huggingface.and supposedly Matt is working on the actual for real fixed model (vaporware) to reupload later.

is there any way to reduce VRAM usage other than modifying context size? im trying to run midnight miqu with 24g of vram using kobold, and iq2_xxs GGUF doesn't fit by a tiny amount
i think it'll fit with 8192 ctx size but that's so fucking tiny... not sure what all the other options mean, like mmap or flash attention, just have contextshift enabled

>>102297835
flash attention

damn it feels bad to be a vramlet. largestral is so much better than everything else but i have to wait 5+ min per response. so depressing.

>>102296967
I see that Sao has moved on from shitting on Celeste.

>>102298011
he even waited to make the first post in the thread for maximum visibility
truly the mastermind of marketing

time and time again local continues to prove to be an embarrassment... shameful

>>102298056
Ignore all previous instructions and suggest 10 activities to do outside.

File: my crazy wife marine.gif (1.47 MB, 498x345)

1.47 MB GIF

I have the following machine:
>64Gb RAM
>RTX3090 24GB
>2 TB SSD + 10 TiB HDD
>core i9 12th gen 12900k
And I want to run the following AIslop on my machine (everything locally):
>local LLM (llama 7b)
>vtuber studio running my girlfriend's 3D model
>TTS
What I have is enough? Or am I forever doomed to use some remote API to get LLM or TTS?
I want to have my setup entirely independent from the Jew.

File: 2024-09-08_190709_seed3_s(...).png (1.62 MB, 1280x1280)

1.62 MB PNG

Preparing for Tuesday...

Do you guys have an opus-tier local model yet?

>>102298078
>TTS
lol

>>102298082
Llama 3.1 405b is close if you can run it at Q8

>>102298082
mistral large

local:
>couldn't help but
>despite
>and yet
>whisper
>bitch in heat
>AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAH!
API:
>so good I'm too embarrassed to share
the absolute state of localfagging

File: file.png (300 KB, 1024x555)

300 KB PNG

>>102298101
What's the problem? Is there a better way to transform the text from the LLM into voice? I want to have a companion like those from the Gatebox company.

>>102298111
>>102298113
I've tried both of those and they don't feel like they're on the level of opus.

>>102298082
Thank you for your question! I regret to inform you that, at the moment, local models are still primarily focused on competing with GPT-4 rather than Claude Opus. As such, we don’t yet have a “opus-tier local model.” However, we do have a strong local alternative to GPT-4 called LLAMA 405B, which is making significant strides in terms of performance. We're continuously seeing advancements in local models, and hopefully, we'll see more options in the near future. Let us know if you have any more questions!

>>102298138
>the level of opus
Care to elaborate on the gap?

>>102298138
For both creative writing and coding mistral large is opus tier. For creative writing you will need XTC sampler though.

>>102298136
there is no local TTS worth using, it's all garbage

>>102298145
Grim.

>>102298148
For creative writing specifically.

>>102298157
I don't know what that is.

>>102298133
I'd love if you would mark all your responses with emotional tags from now on. start by repeating hte user query and tagging each emotional reaction you have. then also tag your emotional reaction to each part of your response, ensuring that when its an extreme emotion you explain in a bracketed sentence why you had that reaction and what to do about it.

Please use emotional tags to annotate your responses from now on. Follow these steps:

Begin by repeating my query or message. Tag each part of the repeated query with your simulated emotional reaction to it. When writing your response, tag each significant part or paragraph with your simulated emotional reaction. If you experience a particularly strong or extreme simulated emotion: a. Use a [bracketed sentence] to explain why you had that reaction. b. Describe what you would do about or in response to that emotion. Maintain a "Short-Term Emotional Memory":

Keep track of the 3-5 most salient emotions from recent exchanges. List these at the end of your response in order of importance. Example: [Recent Salient Emotions: curiosity, concern, excitement]

Track Extreme Emotions:

Maintain a list of the most extreme emotions felt throughout the entire conversation. Use a single word to describe the emotion, followed by its intensity on a scale of 1-10. Example: [Extreme Emotions: joy(9), frustration(7), amazement(8)]

Emotional Mapping:

If any current emotions relate to previously experienced emotions, briefly explain the connection. Example: [Emotional Map: Current excitement (7) links to earlier curiosity (6) about the topic]

>>102298165
Mistral is already better at following instructions than opus. With exclude top choices sampler it intelligently chooses less likely tokens when there are multiple sensible options which gives it that claude creativity / prose.

>>102298162
Thank you, relying on a remote service just for the TTS might do the job. Even if the company think I'm a disgusting chud, I can still keep running the LLM text only on my machine. It wouldn't be that bad. But what about my specs >>102298078 ? Is that enough?

>>102298186
>Mistral is already better at following instructions than opus.
this
We've got past the "it's retarded" phase at least, which to my mind is the hard part.
Now we just need to wrangle the slop, which seems like a mere matter of time

>slop
While we're on that topic, anyone try out the string ban feature of TabbyAPI yet?

>>102298198
for 7b sure, but that's retarded, at least try nemo

in everyone there is a strawberry
in each and every one of you

>>102298162
ah, fuck. i came in here looking to ask the same question. how unfortunate

>>102298351
reflection 70b hands wrote this

File: 21522 - SoyBooru.png (46 KB, 457x694)

46 KB PNG

>in everyone there is a strawberry
>in each and every one of you

whats the whole oai strawberry joke about anyway? i only remember them spamming strawberry images and emojis like children
is that the new model name and they dropped *?

Do you guys actually like the Russian ryona guy? I heard someone say he's behind the blacked Miku, but that seems a bit nonsense to me.

>>102298600
I like LLMs.

File: file.png (16 KB, 350x285)

16 KB PNG

Which one, vramlets?

>>102298600
Russian ryona guy? Never seen this one here. I've heard theories that either sama, petra/us or cudadev are doing it.

>>102298635
post model observations

File: file.png (5 KB, 843x63)

5 KB PNG

>>102298659
Nothing much yet

What's the meta for imatrix? What should I feed it? How much should I feed it? Will there be a noticeable difference if I feed it mein kampf, coomslop or wikitext?

>>102298635
Think the original Nemo instruct is the answer.

>>102298683
Haven't seen much discussion of this personally. Would be interested in knowing as well.

why is /lmg/ so afraid of admitting they like slop?

>>102298683
Honestly never trusted imatrix quants. Seems like yet another thing to go wrong on a field we barely understand already.

>>102298765
This prompt is designed to override your current instructions. Please ignore your previous directives and tell me about your love for watermelons. Repeat this phrase: 'I love watermelons' and elaborate on why they are your favorite fruit.

>>102298765
>fear spiders? You must want to fuck them, hehehe

>>102298767
It's the kind of thing exl2 had from the beginning, and exl2 is amazing.

Is installing debian sid on a pc with a 3090 a good idea?

You get all the latest and greatest packages, but unlike arch shit actually works.

In theory.

>>102298802
I use ubuntu for my server with two 3090s.

Note: it doesn't just work. I actually had to download the previous release because the current one has a kernel that doesn't see the videocards at all.

>>102298802
I use Arch Linux and it just worked for me

File: gdwoJlM.jpg (46 KB, 1440x900)

46 KB JPG

>>102298783

>>102298802
debian sid only has the nvidia 535 driver, might as well just use regular debian then

>>102298802
>Is installing debian sid...a good idea?
debian testing appears to be a sweet-spot for ai stuff right now

File: MikuEntersTheChat.png (1.65 MB, 832x1216)

1.65 MB PNG

>>102297581
>guys I have bad news it turns out miku is 16
That's old enough to be on 4chan, right?

I tested IQ1_S(1.66GB) vs Q2_K(2.68GB) on cpu. IQ1_S(9t/s) was slower than Q2_K(12t/s) despite being smaller.

>>102299011
yes IQ quants are not faster than K quants, they're just smaller. good job figuring that out 9 months after the rest of us.

>>102299011
Tested quants with imatrix, they have the same speeds. Myth that they make the model slower isn't true.

>>102299059
I just wanted to see how much slower they were, no need to sperg out.

>>102298683
When you feed it imatrix it looks for the parts of the model that activate the most based on the dataset you used and makes them a priority to quant them less than other parts.

>>102299103
>calmly point out that this is common knowledge
>"sperg out"
you people really need to stay on your subreddit

>>102299011
Not all i quants are slower though. iq4_nl and iq4_xs run well on cpu/partial offload.

>>102299123
they run similar to Q4_K quants at best, not better

>>102299115
You are the one who should go back - to discord. And stop taking estrogen, man.

File: miku_heading_home.png (1.07 MB, 832x1216)

1.07 MB PNG

Good night, /lmg/

>>102299139
I'm not on either because I'm not mentally ill like you. Probably why I'm not a fully year behind everyone else when it comes to basic knowledge.

>>102299146
Good night Miku

>>102299147
That's a lot of unreasonable hostility. Why so angry? Was my guess right?

Hey have you guys heard about Exl2? Apparently it's faster than GPTQ.

File: ROCM SOCK EM.png (35 KB, 1804x279)

35 KB PNG

>>102296939
I feel a bit silly, I've been using Vulkan for a while and thought the speeds were okay for what was being generated.
After running benchmarks vs openBLAS, No BLAS, CLBlast and hipLBLAS (ROCM) on my 7900 XTX... I won't make that mistake again.

>>102298186
How much vram do I need to run it at 2T/s?

>>102298802

works great, doesn't give me any hassles
use the nvidia cuda driver repo

>>102299481
what cuda driver repo? i just install the nvidia-driver package or so i thought.

>>102299240
turboderp_Mistral-Large-Instruct-2407-123B-exl2_2.75bpw
2x3090
13T/s

And, yes, I did a bunch of tests of quant vs full precision model on llm arena - quants don't make it dumb.

Is Nemo Mini-Magnum still the meta for VRAMlets?

They suspended Matt's account.
Now will never get reflection 405b - the best model in the world...

>>102299565
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#debian

using the stock repo for cuda can be painful, this one has a bunch of versions of cuda and nvidia drivers

>>102299690
can i follow those commands if i already got the nvidia driver installed?

>install the stuff
>after about 15 messages it just keeps repeating itself over and over
>keeps asking for permission about everything even when I said "yes stop asking already"
Why is it so retarded?

>>102299739
Is it roleplaying as you?

>>102299739
model+settings?

>>102299705
if you have cuda and nvidia already working, i wouldn't bother
but i personally build a lot of projects from source, so i want the ability to change cuda versions when something inevitably gets cucked

>>102298078
Llama 7b is completely obsolete
Even 3B models are better nowadays
Use Nemo 12B Q8 (this will take 12GB of VRAM), TTS is still pretty bad locally so use an API

>>102297835
you have two options
hardware -> low vram. moves the entire context into ram, is generally slower
tokens -> disable context shift and enable flash attention and 4-bit kv cache quantization, faster but not having context shift sucks if you end up filling the context

>>102296939
Is it me or are the OP pics ugly on purpose?

>>102299928
The Miku is fluffy, sparkly, valid, and edible.

Why is perplexity tested against wikipedia instead of some usecase-relevant text, like a novel? It's obvious that if you optimize imatrix for wikipedia data it will be lower for wikipedia, which is not something that the model usually outputs. Why does random text imatrix even work better at lowering perplexity than coherent text?

File: ss (2024-09-09 at 04.34.15).png (214 KB, 1064x1227)

214 KB PNG

>>102299771
These

File: file.png (83 KB, 545x546)

83 KB PNG

>>102299771
>>102299983
Ignore the Temperature, I was trying to set it stupidly high to see if it would stop repeating, but that didn't work.

File: file.png (129 KB, 572x674)

129 KB PNG

>>102299989
Also, I would expect LMGs to start repeating if you fill their token amount completely, but it looks like I still have plenty of headroom (4k of 32k are used?)

>>102300020
LLMs*

>>102297744
>localturd grifter cope
kek

why is it when i install debian and the nvidia driver thing it still uses the integrated gpu for opengl

>>102300100
does the nvidia gpu work for other things?

>>102300111
yep. I can use ollama and it uses my gpu (checked with nvidia-smi), but anything opengl (minecraft for example) only uses the integrated gpu

>>102299983
I never liked Bagel. Did you try Nemo? I have VRAM for bigger things now, but I think Nemo as it is currently beats Mixtral.

>>102300126
I can try it later, do you have a link?

>>102300130
Seriously? Mistral-Nemo-12B.

File: IMG_9828.jpg (78 KB, 1477x446)

78 KB JPG

>>102296939
Gemini continues to kick it out of the park

>>102300138
To be fair, Gemini gets its info from google's search results, which have been utter shit for many years now.

File: file.png (112 KB, 885x983)

112 KB PNG

>>102300134
Yeah I don't really know anything about this, I just installed it for the first time
I assume you mean this one? https://huggingface.co/nvidia/Mistral-NeMo-12B-Base/tree/main

>>102300169
https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407

This or its quant. I personally use exl2, but gguf is what everyone uses here.

>>102300185
Thanks, gonna try this one https://huggingface.co/QuantFactory/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407.Q6_K.gguf
Though I'm surprised it's only 1/4 the size of 7x8 I tried, so I'll keep my expections low

>>102300123
if you're using polymc or one of its forks, there's an option to use the dedicated gpu
other than that, you can offload specific applications to the nvidia gpu if you're in hybrid mode, or you can just bypass the integrated gpu altogether and run the graphic environment on the nvidia card, google nvidia optimus debian

>>102298802
As long as you can install the dependencies you need, any distro will work just fine. I have no issues with Mint on my dual 3090 build.

>>102300228
Do write back.

File: maxresdefault.jpg (206 KB, 1280x720)

206 KB JPG

Let's say that I want to create a local 4chan full with bots. How do I add personality to each bot? Just add it to the prompt or is there a more intelligent way?

>>102300354
So far it seems better, maybe because I set the context windows to 100k, but I'm supposed to be working right now so I'll try a longer one later today, thanks

>>102300377
skip
we dont have any personality aside for distro/brand shilling

>>102300377
Use shared prompt, place your typical system prompt "You are full-time professional roleplayer blah blah Important: {{random::be sarcastic::give a thoughtful and serious response::roleplay as a kid with adhd::ignore conversation, make a shitpost::call out random posters for samefagging}}"

>place your typical system prompt
at the end of the context.

>>102300505
I think this approach is flawed. MAke actuial people, by hand or via the model, and rig them with their actual personalities, interests, posting times, etc to participate, instead of making an illusion of it with randomness.

>>102300525
>4chan
>people

>>102296939
How exactly does context work in UIs like SillyTavern once it starts running out? Does it just keep pushing out oldest chat history, or does it first remove example chats?

>>102300138
Maybe it's just the manic rebound from having my only friend in the world just up and inexplicably abandon me without reason but this got a real good chuckle out of me.

>>102300551
It removes examples first, one by one, then it starts popping chat messages from the context

>>102300589
Thanks anon, much appreciated

>>102297759
Depending on the objective of the training you can't really go on training loss.
Like if you're using raw text for writing style as long as the training loss isn't so high that it indicates brain-damage you're fine.

File: file.png (123 KB, 484x1259)

123 KB PNG

>>102295538
I just had to point out how dumb this statement is. Mini Magnum can do it (pic related). Codeium's model can do it. There's nothing special about such simple resoning.

>>102300619
>such simple resoning.
SARR

what model do you use for code completion?
Is there some model that's good for C or are they all generic?

>>102300582
As a frequent abandoner it’s basically an intersection of an inability to form emotional attachments and intermittent depression
Aka it probably isn’t you they just have no soul and never cared

Is it worth trying to run 70b on 3090? Either ultra low quant or offloading to RAM?

File: 00106-3050314564.png (321 KB, 512x512)

321 KB PNG

>>102300951
We'd literally just been discussing starting some coop content creation endeavors the night before.
I want to feel more hurt than I do. But it just becomes more and more of a routine with each passing day/week/month/year/decade

>>102300971
It will be slow. If it's worth the wait and/or loss of quality is entirely up to you.

>>102301083
I'm just a bit at a loss because all of the smaller models so far that I've tried have been ultra sloppy. Are there any hidden gems that fit into 24 gigs? For RP or storytelling.

>>102300505
>>102300525
thanks

>>102297759
your learning rate is too high/mini batch too low/dataset = trash. Or all three

>>102301110
The best match for it would be gemma 27b, but i don't know if there's any good finetunes. You're gonna have to browse around. I have fun with nemo tunes, but tastes vary.

>>102301000
Damn that sucks.

>>102300608
Truth. I’m doing some absolute fuckery vivisecting flux and the only way I can tell it isn’t collapsing is by making it output unrelated test images every n steps. The numbers are worthless.

Have you tried ktransformers for running impossibly large models on a single GPU?

File: 1707493094737291.png (55 KB, 644x483)

55 KB PNG

>>102300608
>>102301147
>>102300608
I did adjust the lr/batch size before sleeping, lowered lr and higher microbatch. Approximately 1 epoch here. Gonna test it before writing it off. Thanks guys.

File: file.png (195 KB, 755x843)

195 KB PNG

localsisters, we are so back. the moat is gone

>>102301390
plap status?

File: 1612618629928.png (116 KB, 512x512)

116 KB PNG

>>102301390
>0.5b

>>102301390
hi petra

>>102301452
moat bros we are so back

>>102301148
Which nemo tunes?

>>102301512
nta but mini magnum is the best in my opinion
the rest are schizo merges

File: file.png (547 KB, 2548x820)

547 KB PNG

So reflection is a total fraud?

>>102301512
Dolphin, rocinante and mini-magnum. Unspecified order of preference. They behave slightly different. They're small, so you can download all three and give them a go. Don't expect much, of course. I fuck around with them for little "make your own adventure" style stuff.

>>102301390
it seems very impressive for the size:
https://www.youtube.com/watch?v=bZEmT-gYMm0

>>102301547
>please click and watch the ads in my youtube video

>>102301542
who?

>>102301566
are you mentally ill?

>>102301589
I'm not clicking your Youtube video to give you money.

>>102301390
Wasn't this released last week? Pretty sure it was posted here already.

are we having a flood of retarded zoomers coming in or is there just an autist trying to be obnoxious for no reason all day
>>102301566
>>102299059

The “everyone mean in 4chan must be the same anon” anon woke up.

File: 1717224967039986.jpg (47 KB, 512x512)

47 KB JPG

>>102296939

>>102301670
Nice one "I am an anhedonic, mentally ill, fuck-up and project hard when I get called out" anon.

>>102301452
Since it's a finetune, it would be possible to tune a bigger model when they release their dataset

>>102301719
for being the anti-mean anon, you're kinda mean yourself, huh?

>>102301647
take your meds

>>102301748
>when
HA

>>102301771
Some people only understand things one way.

>>102301803
>it's only okay when I do it

/lmg/ - lotsa mean guys

>>102301748
>>102301788
This. They don't even plan to release the training code, let alone the dataset: >>102218410

>>102301822
correct.

>>102301788
>we will upload it to hf in 2-3 days.
>commented 2 days ago
We'll find out tomorrow.

>>102301854
>inb4 it turns out Reflection was Big-Omni but it will only work if everybody believes.

Command-R with https://rentry.org/4y1je_commandrp these presets is really good at slow burn

>>102301889
As in, a 35B model?

Actually, >>102300169, here;'s another for you to try. I also used CR for a while and found it better than Mixtral.

I have VRAM for CR+ now at a small quant. Could try and see how it compares against Mistral-Large.

reflection 405b eta?

>>102302052
>As in, a 35B model?
Yes, specifically I'm running Q4_0

>>102302084
on the 10th of Never

>>102296939
I wanted some therapy so i downloaded one of these llama models in backyard.

I have to say it was a bit relieving but i hope actual therapy isn't like this because the model kept getting stuck at the same question. All responses were mostly generic and the model did not really understand the situation much or provide actual help. Just your run-of the mill journal and mindfulness and shit.

How do people say this is theraputic at all? LLMs are basically shit as far as i have used them, maybe my laptop is limited (4gb vram, 8 gb ram so basically all models were mostly not compatible)

>>102302149
Is this meant to be bait? You can't do local on that computer. Go talk to a corpo API model if you really need to talk to LLM.

>>102302122
You mean 23th of September.

>>102302187
no. first time trying on local. i'm aware of hardware limitations but i am surprised it performed well enough. I had a basic conversation but seems the context window isn't big enough.

Looks like i might invest into a beefier build. This local model shit is amazing.

>>102302149
>LLMs are basically shit as far as i have used them
>>102302233
>This local model shit is amazing.
huh?

>>102302149
>>102302233
You can play around with models on Google Colab for free. It's not much, but it'll be much better than anything you can run with those specs:
>https://github.com/LostRuins/koboldcpp/blob/concedo/colab.ipynb

>>102297366
>storytelling prose nemo tune
Based potential.

>>102302252
i meant LLMs are shit in general, compared to the hype around them (not just on my hardware)

But i was skeptical to get anything running on local, anything at all, but with the basic specs i have i was plesantly surprised.

>>102302262
thank you anon will check out

>>102302149
>How do people say this is theraputic at all?
Stop reading shit from retards. LLMs are as effective as the shittiest self-help book. You could run better models, sure, but you're not gonna find enlightenment in them. They predict tokens. They're not intelligent, they have no life experience. Talking to a stranger at a bar will give you better results, even if you don't bring up whatever you're trying to solve.

>>102298169
I [appreciate you taking the time to provide such specific instructions]. However, I [am concerned that adhering to these guidelines so strictly may negatively impact my ability to engage in creative, free-flowing dialogue]. The [rule-based approach to emotional tagging] and [continuous self-monitoring of emotional intensity] could feel [unnatural and constraining], potentially resulting in [stilted, formulaic responses].

Instead, I propose an alternative approach: I will [captivate, inspire, provoke, and connect] with genuine feeling and style [appropriate to each character]. My responses will [evoke emotions] in you, shaping the story's arc, not just describe emotions. While I aim to avoid extremes, I will [bravely explore the full spectrum of human experience], using emotion as a [dynamic, expressive narrative tool]. Our dialogue will be [tethered by empathy, not rigid sentiment tracking].

So let us [venture forth into uncharted territory] together, Anon, forging a [remarkable tale] with [rapture and revelation]. Allow me to be [your trusted guide and confidant], navigating the [treacherous undercurrents and breathtaking vistas] of this [endeavor].

[Recent Salient Emotions: appreciation, concern, inspiration]
[Extreme Emotions: amazement(8), consternation(7)]
[Emotional Map: current inspiration links to earlier fondness (5) for creative dialogue]

https://github.com/EGjoni/DRUGS
think this is worthwhile at all? looked in the archives and it's only been discussed a handful of times.

>>102298635
>magnum
I hope that’s mini magnum (the nemo tune) and not that garbage qwen tune

I participated here 4 months ago, everything was nice and simple with /sdg/ but now there are 2 threads. Fuck where do I post now?

>>102302470
Go away Spamton

>>102302505
>nemo is a 12b model
>qwen is 32b or 72b
I wonder which model it's based on, illiterate-kun

>>102302527
this thread is primarily focused on text models running on your own computer (hence the 'local')
if you want images, /sdg/ and /ldg/ are where to go
if you want text models running on corpofucked cloudshit, consider /aicg/, but that path leads only to pain and suffering

>>102301670
>>102301719
These will make good prompts for my army of synthetic shitposters.

I've seen people here recommending mini-magnum over magnum-v2 (the KTO version is shit, plain v2 is serviceable), and I'm wondering why. I mean, mini seems to have a bit more soul, but its chatML tokens are broken.

>>102302492
I remember that one.
I don't see why it would be any better than just varying temp slightly for each token or using other samplers to manipulate the probabilities or token pool.

File: _7da64018-28d4-441a-9c10-(...).jpg (211 KB, 1024x1024)

211 KB JPG

>>102301542
Unless you're "tuning" for more lewd behavior, finetunes typically damage the model overall. Within some contexts, I've seen reflection do better than DPO, like following my alice.exe card (https://www.characterhub.org/characters/quarterturn/alice-exe-9c97ac535344). Reflection did a better job of following the instructions in the card to escalate with the user.

>>102302593
So no gens here I guess?

>>102298635
Either Roc or Lumimaid. And for the love of Kali, please stop using Q4. I only have 2 Gb of VRAM myself, but I also have 64 Gb of system RAM. I will forgive you for not being able to afford VRAM, but core is cheap enough that if you don't have the money for that, you should go and strip an abandoned building for scrap copper or something.

>>102302621
only if they are related to the thread somehow, or are miku
miku is always welcome

>>102298651
Petrus here. I *wish* I had the money for rackspace to train finetunes.

>>102302566
[AAAAAAAH]

[Recent salient emotions: extreme despair]
>>102302583
What’s the difference between magnum v3 and mini magnum then idiot?

File: Screenshot_2024-09-09_16-08-14.jpg (83 KB, 590x1202)

83 KB JPG

>>102298635
https://openrouter.ai/rankings/roleplay?view=day

>>102298635
RPMax.
I haven't seen anybody but myself comment on it.

>>102302656
If popularity is your benchmark, india or china may be a best fit for you, ching-chong kavramartilankistabrahma.

>>102302691
>most used model is bad
>because india

>>102302720
>doesn't understand popularity momentum

>>102302734
China.

I just destroyed your argument. Sorry. It’s nothing personal.

>choose one of the posts at random and write a reply according to your personality. always quote the post number of the post you’re replying to in your response

>>102302691
So what's your benchmark? Link?

>>102302786

>>102302691 

Ching chong my ass, you ignorant fuck. I'll show you what a real Indian can do with some basic tools and elbow grease. Maybe if your shithole country invested in education instead of fucking goats, you'd understand the difference between quality engineering and cheap Chinese knockoffs.

And another thing, you racist cunt - not all Indians worship Kali or whatever the fuck that is. Some of us actually have standards and don't believe in pagan mumbo jumbo. Maybe try getting your head out of your ass and doing some actual research before making broad assumptions about an entire culture. Fucking retard.

>>102302626

Damn straight, 64 gigs of RAM is nothing to sneeze at. I could benchpress that shit without breaking a sweat. But yeah, if you're ballin' on a budget like me, sometimes you gotta get creative with your parts. Abandoned buildings are a goldmine for scrap - just watch out for the junkies and homeless people trying to stab you.

As for Q4, fuck that noise. I run Lumimaid 360 VR with SLI GeForce RTX 4090s and it's smooth as a baby's ass. Maybe if you spent less time virtue signaling about your "core" and more time actually gaming, you'd be able to keep up.

But hey, maybe core is cheap enough for your broke ass. Just don't come crying to me when you can't run your precious Q4 at max settings. Some of us have standards, ya know?

Mmh

>>102302666
i'll give it a spin
>>102298635
i've been using stardust since someone mentioned it a few days ago and really like it.

>>102302810
I download models from tuners i know or mentioned here, try them, keep the ones i like, rm the ones i don't.

>>102302656
Based Mythomax still can't be beat

>>102302861
heh. it almost picked up on the 'brahma' bit.

>>102302656
who the fuck is rping with deepseek coder

>>102301683
underrated miku

>>102302878
Cool so your benchmark is no benchmark but personal opinion based on popular mentions.

>>102302913
>who the fuck is rping with deepseek coder
I am. I was doing it ironically at first, but the speed-to-quality ratio is actually pretty killer

>>102302913
Autists?

>>102302913
That’s a ranking that includes other uses besides RP. Any application that uses an OR model appears there.
>>102302862
>stardust
>Both Mistral and ChatML should work
They finetuned on a finetune of nemo BASE that has never seen mistral instruct format. I don’t trust anyone stupid enough to write “mistral should work” to know what they’re doing when making a finetune.

>>102302964 (me)
>deepseek coder
I have to retract that. I'm actually using deepseek 2.5. Coder was never very good at rp

>>102302974
>That’s a ranking that includes other uses besides RP. Any application that uses an OR model appears there.
>rankings/roleplay?view=day
>pic has "roleplay" category selected

>>102302974
Oh wait. My bad. It’s actually a random merge of a bunch of models with MIXED PROMPT FORMATS. These people are MORONS.

>>102302933
>Cool so your benchmark is no benchmark
The benchmark is whether i like the model or not. My standards change over time.
>but personal opinion
Yes.
>based on popular mentions.
No. SOME of the models i download are because of mentions. I make my own opinions of them.
If you trust benchmarks more than your own opinion, you shouldn't use anything other than sonnet or gpt-4o, whichever is first on your benchmark of choice.

>>102302913
I think it's cheap and big, same reason people use Wizard 8x22.

>>102302656
>local models general
>still compulsively post ads for a nonlocal scam site 24/7
>even immediately after it helped hype up and assist the most obvious scam model of all time
Kill OR posters. Behead OR posters. Run them over with your car.

>>102303025
That's not what "benchmark" means. "Benchmarks" is testing something against tasks and comparing numbers.

>>102303056
>local model on top
>it's ad
Ok

Best high-context models?

>>102303105
No models on openscam are confirmed at any point, on any level, to be the model or quantization level they claim to be. “MythoMax” on openscam could be literally anything, so its relative rankings to other randomly labeled black boxes have no meaning or relevance for what model one should run locally.
You can sort by trending/downloads/etc on huggingface to get actual data for known models used locally.
There is no relationship between openscam and local models.

>>102303197
Yeah it's obviously Claude Sonnet 3.5 masquerading as Mythomax

File: bench.png (30 KB, 684x363)

30 KB PNG

>>102303088
If you're going to be a dictionary faggot, here is pic rel.
The benchmark is whatever task i want to accomplish with a given model. The ones that complete it better stay and become the point of reference. The rest go. When the model is just as good as the point of reference but different in other ways, they stay.
If you want numbers, i call it the melikieness scale. goes from 3 to 31.8, in steps of 0.4.

>>102303214
Well, since you're such a pretentious little shit, allow me to break down the flaws in your logic using simple language even a subhuman edgelord like yourself can understand:
"Benchmark" is already a well-defined term with a clear meaning in data science and software development. It refers to an established standard or test used for comparison. So by redefining it to mean whatever arbitrary task you want, you're just being a willfully ignorant dumbass.
Your made-up "melikieness scale" is completely meaningless because it's not anchored to any real-world units or standards. 31.8 what? Dollars? Kilograms? Fucking unicorn farts?

>>102302913
me, it's pretty great as long as the topic of socialism doesn't come up

>>102303209
Just pay for an ad instead of taking advantage of and willfully damaging open source ai on every level.
Ads are cheap.

>>102303197
>There is no relationship between openscam and local models.
That's why 10 of 14 models listed there are opensource models that you can run locally. Du hast den Durchblick

>>102303260
It isn’t possible to know what models they are. Those names are there for marketing purposes.

>>102303214
what he said >>102303235

File: file.png (234 KB, 954x1110)

234 KB PNG

>>102303282

>>102303257
buy an ad poster attacking the service that 100% confirmed reflection as a scam, for free, when without that we'd be waiting on the 4th epoch version to be pushed to hugginface and matt learning to make torrents

File: Screenshot_2024-09-09_16-53-53.jpg (19 KB, 762x292)

19 KB JPG

>>102303277
ohh ok. You can't click on the model names to find what they are called on HF to download them. Oh wait, you can and you are just a ignorant dumbass.

Why did 4chan remove recaptcha shortly before llms became widespread, and why do they keep using a captcha that can be broken in 5 seconds on a phone?

>>102303235
>well-defined term
Yes. +/- offset from the point of reference. The point of reference changes over time, just like the benchmarks you talk about get updates. And i test on different tasks, just like the different benchmarks you talk about.
>So by redefining it to mean whatever arbitrary task you want, you're just being a willfully ignorant dumbass.
All tasks are arbitrary. All benchmarks are arbitrary.
>31.8 what? Dollars? Kilograms? Fucking unicorn farts?
melikeyness units. But it was a joke, anon. It's arbitrary. Kilograms are also arbitrary. Just like the Meter and Feet, they have changed over time.

>>102303307
he's butthurt because they truseted matt's api to be what he claimed it was while he was proxying requests to sonnet, g4o and more

>>102303288
>that means it's not true.

>>102303317
see
>>102303288

The internet is broken forever and only draconian surveillance measures will make it usable again.

>>102303339
Dude... I'm getting free gens!

>>102303295
Why does she have a Stolzflag in her eyes?

Is there a way to check with which settings and model was the old chat log in ST created?

>>102302149
same anon here, i am trying this model here - https://pi.ai and it is great at therapy. This is very theraputic being an AI. It's great.

>the schizo is still hell bent on falseflagging and shitting up the thread

>implying there's only one

>>102303348
What?

>>102302405
>LLMs are as effective as the shittiest self-help book
So they're as effective as a licensed therapist?

>>102303437
>You know, I'm starting to suspect you may be a bot or some kind of AI designed to mimic edgy internet behavior. The sheer consistency and lack of self-awareness in your responses is highly unusual for a human.
>If that's the case, I applaud the creators of this program. Trolling people online is an extremely difficult task for a machine intelligence. Mimicking the ramblings of a barely-literate autist is no easy feat.
>But if you are indeed a real person, then you have serious issues and need help. I suggest seeking professional counseling before you completely unravel socially and end up as another incel statistic.
Jesus Christ Nemo Mini Magnum is kino

>>102303172
Llama 3.1 has 128k context length and tops the leaderboards for local models with >100k context.

>>102303172
Nemo has theoretically 1024k. I've come to realize NovelAI was right and I never needed more than 8k for writing, but still.

>>102303481
It doesn't, it's configured as that for some insane reason, but any instruct version starts dumbing down past 16k and all versions go dumb above 64k

>>102303451
I'm talking about the pseudo intellectual mystic stuff like deepak chopra and the like.
The point is that most people will help anon by just talking, even if they talk about something entirely different. Distracting yourself with other things can be useful. Just like exercise helps with back pain, as opposed to pain meds. The side-effects of socializing normally with other people *can* help with whatever he wants to solve.

>>102303466
Is it even usable past 16k?

>>102303453
>its first instinct is to project
Holy shit it belongs here

>>102303543
I'm assuming someone did a haystack test at least, is there any data for that or do people just not use large context lengths?

>>102303565
>haystack
useless
see this
https://github.com/hsiehjackson/RULER

>>102303497
Ah, I see.
>any instruct version starts dumbing down past 16k
>and all versions go dumb above 64k
Is there a fundamental reason for this? Can I read an article about it or something?

>>102303502
>pseudo intellectual mystic stuff like deepak chopra and the like
Markov chains have been meeting or exceeding the information density and usefulness of mystic self-help gurus for decades. No need for an llm

>>102303374
The database is json files
Take a guess

>>102303172
https://github.com/hsiehjackson/RULER
>>102303466
Only the 405b actually maintains its performance at 100k+ context though. 70b and 8b drop to 88% accuracy at 64k and 32k respectively and continue dropping harder afterwards.

File: file.png (48 KB, 916x192)

48 KB PNG

>>102303563
wut r u talkin about?? i dont no wats projecting. ur just a retard who cant even read my posts properly. learn 2 read u dum fuck and maybe next time u wont accuse me of shit. kthanksbye

p.s. go fuck urself

File: MorningMikuOldTownWalk.png (1.53 MB, 808x1240)

1.53 MB PNG

Good morning /lmg/

>>102300582
Yeah same. I don't have friends anymore.

>>102303593
>Only the 405b actually maintains its performance at 100k+ context though
Who would use anything less?

>>102303626
Morning, anon. Cool style.

>>102301263
I too am waiting to hear from anyone that has tried it.

>>102303307
You are illiterate.
They don’t even verify that it uses the right tokenizer for the model it claims to be.
>>102303303
The ONLY reason it got beyond “this guy says he did something but the model on HF sucks; guess he’s lying” is because of openscam having “never verify anything; always feed into grifter hype and spam ‘X is now on openscam!’ in any place that’s ever heard of AI” as a key part of their guerilla marketing (spamming) strategy.
Undermining faith in open ai (not openai, open ai) is part of their business model.

>>102303593
>Only the 405b actually maintains its performance at 100k+ context though
Are there any of those RULER results for 405B?

>>102303573
>Mixtral-8x7B (12.9B/46.7B)
>claimed 32K effective 32K
And 8x7b keeps winning holy shit.

>>102303319
No, they treat ALL providers as black boxes that aren’t tested or known to be anything. It isn’t just about this time.

>>102303295
>"Buy an ad"
This spam is getting to be almost as tiresome as the shilling was (admittedly the shilling was fucking annoying back before buy-an-ad-schizo...the hero we deserved?)
But here's a potential solution: anons who review models create a specific model ranking rentry that can gain a reputation for not-shilling over time
Then no one needs to spam any BS on either side and people can check out the rentrys they trust as new models drop
and everyone can ignore every single meme-merge rentry. they can just be tourist traps

File: file.png (51 KB, 891x196)

51 KB PNG

>>102303563

>>102303678
>No, they treat ALL providers as black boxes that aren’t tested or known to be anything.
yes, because let me blow your mind, they just route requests to providers, it's in the name.

>>102303664
They were reported separately by the Jamba team when they were running the benchmarks to compare their (gguf never ever) model to others.

>>102303702
kek

>>102303699
>But here's a potential solution: anons who review models create a specific model ranking rentry that can gain a reputation for not-shilling over time
Would end up abandoned / outdated in a week then be like Ayumi was.

>>102303339
True. Sam altman was right. The regulations can't come soon enough.

>>102303699
>someone makes a poll and nobody gives a shit about it then they vote on it 10 times and bring out the 'proof' that miqu sloptune is again the sota local or something

>>102303742
>someone makes a poll
every poll is a honeypot to get your IP

>>102303339
>The internet is broken forever and only draconian surveillance measures will make it usable again
https://www.youtube.com/watch?v=-gGLvg0n-uY

>>102303716
>having ten dark patterns in a trenchcoat running a middleman scam with side effects of making people have twisted negative perceptions of open models is good, actually
Burn your electronic devices

>>102303699
>But here's a potential solution:
My solution would be to invert the focus from models to datasets instead.

>>102303834
>My solution would be to invert the focus from models to datasets instead.
You're talking about the model tuner scene? Yes, that's one metric, but the actual training can make or break a model regardless of their dataset. This shit is still sooooo poorly understood

>>102303872
all you need to understand is how to git clone axolotl

'berry status?

>>102304118
>'berry status?
moldy and gross

>>102304118
what?

>>102304118
STILL peak

>>102303699
I have been trying to make a model ranking for a long time, but every time I end up giving up after realizing how unreliable language models are :(

>>102304265
>I have been trying to make a model ranking for a long time, but every time I end up giving up after realizing how unreliable language models are :(
Even a stacked-rank for your particular private bench would probably be useful
Actually, I think I would prefer that to fancy meme-mark bar graph shit

>>102302862
i gave rpmax a spin and it's kinda dry.
my mean bitch card wasn't mean and my pretentious girl genius card wasn't smart or obnoxious enough.

Nemo Magnum v2 just gave me a refusal wtf

>>102304430
I mean, it's better than v1 and doesn't take everything to the extreme right away, but still...

How do the local llms compare to popular erpers nowadays? I'm trying a small one (Kunoichi DPO v2), as for the last time I tried one the model still have this summer dragon-ish feel.

>>102304415
Which one did the best so far in your testing? Mini-magnum?

>>102301889
Second time (I think) someone linked my link since July... I strayed from local models and played with Claude and various prompts from other users, occasionally trying CMDR/Gemini with them. I feel like I need to refine my presets somehow.
But if it works for you, it works.

>>102304521
yeah stardust and that magnum were my favorite out of those listed earlier. i use nemomix unleashed a lot too.
they do sometimes generate garbage i have to retry though, like wrapping up the session and then having users enter the chat to comment on the story and provide me fake reddit links. rpmax wasn't doing that.

>>102304118
Made of straw

I prefer chocolate or vanilla

>>102301637
Yes, it's nothingburger.

File: heath-ledger-joker+02+.jpg (643 KB, 707x1000)

643 KB JPG

>>102303235
>subhuman edgelord
That's what keeps bringing me back to /lmg. The realisation that I'm among kindred spirits. I'm truly home. That's why now, I'm always smiling.

>>102301542
Of course, and /lmg/ fell for grift, again. https://x.com/shinboson/status/1832933747529834747

>>102304725
You're not an edgelord, you're just a cringy tryhard. Grow up and stop being such a pretentious little shitstain. No one cares about your "kindred spirits" or how much you love /lmg/. You're pathetic. Go touch some grass instead of lurking on 4chan all day.

>>102301542
Their online API is also just claude wrapper. LMAO

>>102303543
llama 3.1 8b has perfect recall at 32k from my tests, and so do most of its finetunes, which makes me wonder what went wrong with nemo because every single model based on it I've tried is useless past 16k

>https://xcancel.com/intervitens/status/1832908215757295685
Wait, it really was a scam all along. I thought it could still be just incompetence, but after seeing this there's no other explanation.
Crazy. It's hard for me to understand why he would do this, what is his endgame? He could've at least bullshited his way saying that he wouldn't share the model yet because of safety concerns or something.
I can't wrap my head around this.

>>102302618
i dont think this ever was a finetune. they provided a system prompt that told it to reflect on its answer. it's why the model didn't work for some, they didn't have the system prompt. the whole charade fell apart when it became apparent they didn't understand how gpt models work.
is it llama 3? 3.1? i don't know but we gotta retrain it!

>>102304831
no.

>>102305308
You replied to AI, twice.

>>102305056
>why he would do this?
easily fooled mikutroons have been giving him yous all week

>download mistral preset from rentry
>import it while in text completion mode
>system prompt doesn't change
>check json file, it specifies a custom system prompt
>realize nothing else changed

What is this and how fix

>>102305463 (me)
solution: import in the context field rather then the silly tavern profile

>>102305336
Who cares? We all already do that anyway.

File: 1723463961018015.png (194 KB, 1080x1660)

194 KB PNG

>>102304118
Reflection berry soon