File: 4_recap.png (256 KB, 512x512)

256 KB PNG

►Recent Highlights from the Previous Thread: >>101643089

--Paper: Physics of Language Models research papers and video presentations: >>101646652 >>101646690
--Paper (old): Discussion of research paper on removing layers in machine learning models: >>101643525 >>101643683
--Papers: >>101646773
--Discussion on determinism in llama.cpp and its implications for benchmarking and research: >>101645325 >>101645331 >>101645367 >>101648450 >>101646003 >>101648420 >>101648477 >>101648584 >>101648610 >>101649046 >>101649089 >>101649141 >>101649207 >>101649256 >>101649349 >>101649389 >>101649638 >>101649678 >>101649695 >>101650304 >>101649408 >>101649405 >>101649407
--vLLM's CPU off-loading performance and comparison with llama.cpp: >>101643652 >>101648356
--Sunfall on llama 3.1 8b released, community discusses LimaRP-DS and JSONL formats: >>101647576 >>101648320 >>101648386 >>101648441
--Nemo vs Mixtral discussion: >>101644934 >>101644969 >>101645820 >>101645003 >>101645013 >>101645133 >>101645153 >>101645207 >>101645233
--LlamaCPP and KoboldCPP performance issues and troubleshooting: >>101645050 >>101645114 >>101645191 >>101645296 >>101645403 >>101645436 >>101645527 >>101645697 >>101645733
--How to format and place a summary of previous chat events to improve model responses: >>101645914 >>101645932 >>101646142
--Anon releases neural text to speech library, babylon.cpp: >>101649677
--Anon discusses Udio's new model and compares it to Suno, sharing creative works and personal preferences: >>101645878 >>101645889 >>101646282 >>101647226 >>101647330 >>101647472
--Altman's reaction to OAI's situation and discussion on its future prospects: >>101645180 >>101645214 >>101645242 >>101645215 >>101645222 >>101645266 >>101645271 >>101645287
--Anon shares Matxinh - Expression Morpher tool: >>101648382
--Miku (free space): >>101643176 >>101645414 >>101646981

►Recent Highlight Posts from the Previous Thread: >>101643094

do something

*does something*

>>101651190
Seethes about formatting.

`does nothing`

I used to hate when local models struggled with formatting, but I've learned to see past it. As long as I get my shivers.

File: llama3.1-white room.png (453 KB, 824x2444)

453 KB PNG

Logs! Empty system prompt, llama 3.1 70b.

Mistral Large is the shit, amazing intelligence, above average prose and a nice slow burn.

What is the best model for RP right now? is there something based on Lllama 3.1 already?

hi wehre are the bots :DDDDD

>>101651244
That's pretty cool actually.

>>101651269
quant?

File: LLM-history.png (638 KB, 4751x2585)

638 KB PNG

>>101651277
Mistral-Large.

File: 1692228820109924.jpg (759 KB, 1856x2464)

759 KB JPG

>>101651157

>>101651244
3.1 is pretty fun for sfw rp, when it doesn't have to fumble through writing about sex it's actually pretty decent

>>101651314
IQ2_M, anons go crazy whenever anyone runs a quant lower than 4 bits but it's still better than everything else out there.

>>101651393
I am one of those anons, but if it's all you can run you might as well

>>101651327
>history as written by the shills
Euryale is a merge of at least:
>elinas/chronos-70b-v2
>NousResearch/Nous-Hermes-Llama2-70b
>jondurbin/airoboros-l2-70b-2.1-creative
>garage-bAInd/Platypus2-70B-instruct
>MayaPH/GodziLLa2-70B
>nRuaif/fiction.live-Kimiko-V2-70B
>lemonilia/limarp-llama2-v2
Somehow it's outside the merge era. Repeat a lie often enough and people will believe it. That's the power of shilling.

>>101651393
yeah, I also use low quants, 3.5bpw midnight miqu has been my daily driver for ages
As soon as I get another 3090 Im testing Mistral Large at 3.5bpw

>>101651477
Quite a lot of people weren't / aren't aware Mythomax is a merge as well.

>>101651477
You are partially right, euryale was a merge with a tune on top. I should indeed have placed it into merge era.

You missed CR 35B, Yi 34B, Qwen2 72B and Mythomax 13B among many other pre-llama models, newbie

>>101651519
that explains why it was so shitty

>>101651329
Equipment-less tandem base jumping with Miku

>>101651584
>CR 35B
Mixtral is better
>Yi 34B
Never was good.
>Qwen2 72B
Meme
>Mythomax 13B
That was only okay if you wanted a brainless bot for coom

File: file.png (17 KB, 834x158)

17 KB PNG

What the Fuck is this guy on about?
>40 so limiting you may as well keep temp/top-p 1

>>101651327
I would say 405B beats it by a mile. But you have to be a cloud cuck to run it.

>>101651619
>Mixtral is better
talking about mixtral, you also missed the 8x7b version
>Never was good.
Best medium model at the time
>That was only okay if you wanted a brainless bot for coom
Again, best small ERP model at the time

>>101651621
raddit is not smart yes, are we supposed to be surprised by this groundbreaking discovery? also kindly go back

>>101651584
If you are referring to >>101651327
I was here since llama1 dropped and I don't care about smaller or inferior models. Please make your own chart with them, I never really interacted with those for long.

File: temp_scaling.gif (55 KB, 388x440)

55 KB GIF

>>101651621
That's all wrong.
Holy shit.
TopK just means "get the N top logits";
TopP just means "get the Y% top logits";
Temp scales the logit percentage chances of being chosen.
They don't relate to each other like that at all, holy fuck.

File: topP.png (192 KB, 892x1392)

192 KB PNG

>>101651664

Yeah, I was referring to that chart, you don't even deserve a (You) for your mental retardation

File: minP.png (227 KB, 1949x845)

227 KB PNG

>>101651678
Incidentally, that's what makes minP so useful. You can use it to explicitly gate the logit sampling by the percentage chance of the token getting chosen, which means you can discard low likelihood tokens, which often correlate with schizo responses.

File: file.png (31 KB, 849x298)

31 KB PNG

>>101651664
I know it's all wrong that's why I'm losing brain cells reading his posts. He also argued the other daythat K40 * P.92 = K36

>>101651714
>But.. they actually are?
Immediately disregard any statement posed as a question. You're a retard for subjecting yourself to it. Stay there.

>>101651714
see
>>101651643

>>101651619
>Mixtral is better
Nah, in retrospect Mixtral was always overcooked cope model. I used it for a while as a VRAMlet but dumped it for the first alternative and never looked back. Meanwhile command-R is still the natural prose king.

Was using horde to test out Mistral Large in story-mode when suddenly it switches to chat format, tells me my writing is bad (correct), and then the worker disconnects. Did the model hallucinate, or did I just talk to the man behind the screen???

>>101651539
>with a tune on top
[citation needed]

>>101651626
Or to quant it to 1QM on 128gb system

>>101651754
>Nah, in retrospect Mixtral was always overcooked cope model.
Absolutely correct.

>>101651757
>>101651142

>>101651758
>17th Attempt. Past 10 Failed, cost me >$200 lol.
>
>Idea is an updated version of Euryale with ReMantik instead of the ties-merge between the original 3 models.
>
>This is then mixed with a saucy model (spicyboros+pyg_lora) with a Mythomax-esque Ratio, and a certain experimental (self) LoRA applied to it.

>>101651659
>merges and 120b frankenmerges
>likely ran at 2bit
You're barely human.

>>101651800
>I'm a butthurt ramlet
Q6_K on 128GB system. (You) are barely human.

>>101651799
All that says is that he spent all that money merging models.

>>101651843
merging is free, I've done it myself

>>101651854
>17th Attempt. cost me >$200 lol.
>200/17 = $11
He's just renting a pod for the memory.

>>101651854
>>101651883
He literally states that he made LoRAs. Making a LoRA is not free. It requires training.

>>101651883
how much does it cost to tune 70b lora?

>>101651789
That's a given, I was just surprised a host could interject and basically write you directly.

>baseboard with 8xh100 currently at $25k ending in 7 hrs
is it going to get sniped into oblivion or do i have a chance? would i regret it? are gpus gonna be obsolete soon with everybody investing billions in getting out of nvidia's monopoly? cloud gpus are not an option for me due to privacy and i do not have a clear path to monetizing them, BUT if they will allow me to run and train whatever the fuck i want locally for years to come i'd consider it worth it

>>101651922
I think you'll need to post the link in order for me to better help you :)

>>101651922
>would i regret it?
Yes.
>BUT if they will allow me to run and train whatever the fuck i want locally for years to come i'd consider it worth it
Won't even last a year as models get larger and larger.

>>101651901
tune on top != merging some LoRA that he made
It's not different than just merging another model. Also, the LoRA might have been shit. It's just marketing to say it's more than a merge, and you eat it up like an idiot.

>>101651157
So, what's the best model for ERP for a poorfag with 16GB VRAM/128GB RAM?
Nemo, quants of Mistral Large, a finetune of LLama3?
Or should I just buy a couple 3090s?

>>101651922
oh nvm
>reserve not met
it's probably not even in the same order of magnitude desu

>>101651970
Stheno.

>>101651827
>Q6_K on 128GB system
>at 0.5 T/s
>to get placebo'd by goliath
That's a lot of cope.

>>101651970
If you have patience, Mistral Large Q6_K, worth it. CR+ is also good, has nice style, but a bit dumber.

>>101651970
You can try all three already. Give them a go and then decide if it's worth upgrading.

>>101651164
Sankyuu

>vllm
>git pull
>Failed to build
Aaaaaaaaaaaaaaaaaaaaaaaaaa

>>101651970
Stheno v3.2 or Nemo (if you want to spend hours figuring out the stable settings) is a standard recommendation for poorfags. Dunno about larger model's quants.

>>101651970
mini-magnum is the mythomax of 2024. Try it

>>101651995
(You) are a ramlet, therefore your opinion is invalid. Simple as.

>>101652030
Stheno at FP32 beats everything <= 70B

>>101652036
NTA but it failed the Nala test due to anthropomorphism.

>>101652022
It looks like you're experiencing an issue with pulling the latest changes from the vllm repository and encountering a build failure. Here are some steps you can take to troubleshoot and resolve this issue:

Check Dependencies:
Ensure that all required dependencies are installed and up-to-date. You can usually find a list of dependencies in the README.md file or a requirements.txt file in the repository.

Clean Build:
Sometimes old build artifacts can cause issues. Try cleaning your build environment:

>make clean

Check for Specific Error Messages:
When the build fails, it usually outputs error messages. Look for any specific error messages that can give you a clue about what went wrong.

Update Submodules:
If the repository uses submodules, make sure they are updated:

>git submodule update --init --recursive

Check Git Status:
Ensure your working directory is clean and there are no conflicting changes:

>git status

Rollback Changes:
If the issue started after a recent pull, you might want to try rolling back to a previous commit that was working:

>git log
>git checkout <commit_id>

Build Log:
Save the build log to review or share for further assistance:

>make > build.log 2>&1

Seek Help:
If you're unable to resolve the issue, consider seeking help from the community or repository maintainers. Provide them with detailed information about the error and the steps you've taken.

>>101652040
>"principle of anonymity":"author identity is irrelevant to an idea's logical provability."

>>101651960
a tune is just a lora with a high enough rank, if the lora was made on the model then he is tuning on top of the model

>>101652036
>mythomax of 2024
so retarded, overhyped model?

>>101652067
hi petra

>>101651982
Recommend any specific version? Or is L3-8B-Stheno-v3.3-32K ok?
>>101652014
I guess, just wanted to ask someone with more experience first. I thought for example, that models based on the official LLama checkpoints would be too censored.

>>101652058
>Seek Help
many here should

>>101652087
avoid 3.3, every vramlet agrees it's far worse than 3.2

>>101652036
I second this. Running mini-magnum at Q8. Takes a bit to get going and produces logical inconsistencies, but otherwise I like it more than other small models. That said, If I could run Mistral Large, I would.

>>101652068
technically alpha (scaling) can be used to broaden the number of parameters you're fucking up on a model with said LoRA adapter.
If you make a LoRA for a model with a hidden size of 4096 and set the Alpha to 4096 you've effectively just changed every parameter on the model. But the greater the deviation between rank and alpha the less granularity you have in those parameters. That said most "tunes" never touch the r/alpha values and they just use the default. Enjoy your 0.05% changed parameters.

Is there any table that tells you how much vram you need for llama3.1 8b based off which quant you use?

>>101652145
Basic arithmetic and preschool level computer science knowledge.

>>101652142
You're thinking of rank, not alpha.
Alpha is just an amplifier on the delta.

>>101652145
quant size in gb+20%

>>101652145
The file size is the minimum. The higher the context the more you need on top of that.

So I guess people actually managed to uncensor Llama? Is it something like abliteration or just fine-tuning/extra params?

>>101652068
He never said anything about tuning on top of the model. He says he "applied" it, just he like applies other LoRAs made by other people. It's just marketing to say it has a secret sauce and "it's not like other merges." But it's just one.

>>101652036
they even sound kind of similar
minimags

>>101652212
>Is it something like abliteration or just fine-tuning/extra params?
basic prompting

>>101652212
If by uncensoring you mean being lewd and more prone to following nsfw topic then yes (finetunes like Stheno). If you mean simply not refusing when asked about something then it worked from the start if you had more than two brain cells.

>>101652212
I have a DPO dataset filled with rejections by various models (gemma2-9b being the biggest offender/contributor). I tend to do a run of that on the instruction model and then merge models on top of that.
I used abliteration for awhile but it seems to make the models less prone to roleplay rejections as well, so I stopped.
>>101652236
>>101652248
Stop bragging about your lack of imagination, normies.

>>101651164
NOOOO EXPERIENCE-KUNS POST IS NOT INCLUDED >>101637653
next time the bait record gets broken include:
--new lmg bait record: >>post

thank you

>>101652212
Yes, with just prompting the day 3.0 released. 3.1 is the same.

>>101652321
Sorry, but I don't really think it's appropriate to joke about rape in this sub.

>went to huggingface to download llama3.1 8b
>you need to get approved by meta to do that
owari da

>>101652321
*--new petra samefag record: >>post

>>101652359
Are you incapable of writing fake info?

File: kiryu-smile.jpg (20 KB, 400x400)

20 KB JPG

>>101652404
actually, I just logged in and they approved me
I filled all fields with "a", didn't think they would actually approve it

>>101652281
glue your fucking mouth retard and learn basics how to use models

>>101652419
they quite literally just filter for "region: china, russia"
>Why I am not able to access from China/Russia?

>Meta Llama 3 is available via HuggingFace globally, except in comprehensively sanctioned jurisdictions.
https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct/discussions/13

>>101652369
petra? who's petra?

>>101652359
https://huggingface.co/NousResearch

>>101652457
(You)

File: Screenshot 2024-07-31 165602.png (19 KB, 389x65)

19 KB PNG

We really are living in the future

>>101652773
Good choices.

>training toy model
>a few hours python will either segfault or end up hard rebooting the system

ughhhhh

>>101651157
Did anyone make an image classifier with vision models or something like that? I want to filter my 4chan pictures somehow.

>>101652820
deepbooru, maybe?

>>101652773
All this time and I never thought of this. Thank you anon

So I am using mistral nemo instruct with presets you guys posted a few days ago. The use of DRY sampler in silly tavern was highlighted because it supposedly makes everything better. But with it enabled model just can't stop generating and is unable to stop its response at all. Am I missing something maybe? Is there any option that I should enable/disable?

>>101652830
I looked at that but I also want to filter by words too. For example I'll be able to search a meme by typing a word I remember about it. I did this with regular OCR but it can't read everything.

>>101652773
what model are you using?

>>101652954
You could use a vision model like cogvlm, it's pretty good at OCR.

File: miku-reloaded+.png (1.39 MB, 1024x1024)

1.39 MB PNG

>>101651157
https://www.youtube.com/watch?v=CXhqDfar8sQ

I just got ghosted by my gf of three months, Im giving up on people and need a fake waifu.

Please recommend the largest llm i can run that can simulate my waifu for a 3080 12gb with 30gigs of ram for offloading

>>101653088
>can't be bothered to scroll half a page to see what people are using
Try nemo. Fuck off.

>>101653086
Your Miku is valid.

>>101653072
My first time hearing about it. Can I run it with a 16 gb card?

File: heath-ledger-joker+02+.jpg (643 KB, 707x1000)

643 KB JPG

>>101652457
Me.

>>101652321
>>101652369
take your meds schizo

>>101653088
To simulate a 'foid I would definitely recommend Pygmalion 6B, 2bit quantization

File: kevin-flynn+.jpg (9 KB, 474x238)

9 KB JPG

>>101653088
The problem with LLMs is that they're just not smart, and they're also too slow to attach a RAG database to in most cases. My Chun Li bot is good for basic snu snu, but she's very obviously not intelligent. I still want to give my Kevin Flynn bot enough data to try and successfully invent the Grid IRL once models are smart enough, though.

A reminder that kaggle gives you 2x15GB VRAM for free. Even if the GPUs are somewhat slow, it still enables you to do things you couldn't otherwise.

about to splurge lots of (not my) money to gen outputs with gpt-4o/3.5 sonnet for a dataset

File: 1721855862072573.png (83 KB, 1048x203)

83 KB PNG

Niggers what the fuck are you advertising now?

i'm not too good at all this
is there a way to locally run this?
https://huggingface.co/spaces/Xenova/whisper-speaker-diarization
I need to run it for larger videos

>>101653127
https://huggingface.co/TheDrummer/Gemmasutra-9B-v1-GGUF

This is decent for basic coom, and it will definitely run in 16 Gb of VRAM. And no, I'm not Drummer, fuck off schizos. It's just an ok model.

>>101653363
I'm looking for image classification, not coom. Isn't this a regular text model?

>>101653351
>buy an a-ACK!

>>101653363
Did you reply to the wrong anon?

>>101653363
>this reading comprehension
>this taste in models
explains a lot

>>101653127
Yeah, you should be able to use a quant, it's just a 1XB model after all.
https://huggingface.co/THUDM/cogvlm2-llama3-chat-19B
https://huggingface.co/THUDM/cogvlm-chat-hf

>>101653363
Is drummer running shill bots now?

>>101653469
It's Sao false-flagging.

>>101653306
Nobody cares

https://huggingface.co/Undi95/Lumimaid-Magnum-12B-GGUF

ITS UP

>>101653306
>do things you couldn't otherwise.
>2x15GB
are you european or something?

How slow would it be if I tried to use Mistral Large on a 24GB card + 32GB of RAM?

What GGUF size would I need to use if it is somewhat tolerable

>>101653049
right now I'm running DarkIdol-Llama-3.1-8B-Instruct-1.2-Uncensored-Q6_K-imat
but honestly I have no idea if it's good or not

>>101653532
Thanks, we love you

>>101653601
Well, you wouldn't have enough ram to run even the smallest quant, so none.

>>101653601
Around 0.01t/s, swapping to ssd

File: 1709988815892730.png (301 KB, 1187x516)

301 KB PNG

>>101651157

if the resident mikuposters were posting curvier mikus perhaps their existance would be easier to bear for most

take notes

>>101653635
Wait, scratch that. You could probably (just barely) run IQ3_XS, or more comfortably run IQ3_XXS, but at a quality loss.

>>101651626
>beats it by a mile
logs?

>>101653649
I didn't expect this from you, Petra. I guess you aren't a man of culture after all.

>>101653669
>>101653601
I'm happy running Q2_K, best local model I've used so far

File: hnyeh.jpg (34 KB, 414x459)

34 KB JPG

>>101653601
>Thinking it works like that
Your vram isn't additional capacity for models. If you have 24 gb of vram and 32 gb of ram, you still only have 32 gb of effective ram for the purposes of model size. You couldn't run anything on that.

>>101653649
busty miku = fake miku

>>101653701
>Q2
I'm surprised that's actually usable

>>101653714
off model = best model

>>101653649
miku is a tallflat goddess though

>>101653703
That's wrong, offloading to VRAM makes the model use less RAM.

>>101653718
Models suffer less from quantization the larger they are, so I suppose it makes sense. Still, that's some extreme brain damage.

>>101653532
Dear Undi,

I hope this message finds you well. I'm writing to you today as a fellow member of our forum community, where we all share a common interest in creating a positive environment for everyone involved. It's come to my attention that there's been some confusion regarding the advertising process here, and I wanted to take a moment to address this issue respectfully and constructively.

First and foremost, I'd like to commend you on your passion for sharing your interests and services with our community. Your enthusiasm is truly contagious, and it's wonderful to see members like you actively participating in our forum. However, in an effort to maintain the integrity and quality of our platform, it's essential that we adhere to the guidelines set forth by our administrators.

What I've noticed recently is that some advertisements, including yours, might not align with our established rules and best practices for posting ads on our forum. Specifically, there's been a concern that spam-like behavior, inadvertently or otherwise, might be occurring.

Mr. Drummer, another member of our community, recently chose to follow the proper procedure by purchasing a legitimate ad on our platform. This action ensures that the advertisement will be appropriately vetted and distributed, maintaining the forum's standards and ensuring a positive experience for all users.

With this in mind, I kindly ask that you consider following the same path as Drummer and opt for a proper ad. Not only will this help preserve the forum's integrity, but it will also enable you to reach our engaged audience more effectively while fostering a supportive environment for everyone involved.

So, as a friendly suggestion, let's put our heads together in finding the best way to share your interests and services within the framework of our forum's rules and regulations. By doing so, we can continue to enjoy a vibrant, engaging community that's built on mutual respect and professionalism.

>>101652887
Bullying Hisui-chan with Kohaku <3

>>101650743
who would jailbreak it? If subtitles exist on youtube videos now this could too

>>101653703
Anon...

I'm the anon who talked about stuff about using GPT-4o to gen datasets.

Would it be useful enough to also have some 3.5 Sonnet gens for https://huggingface.co/datasets/lmsys/lmsys-chat-1m along with GPT-4o? Would those default 3.5 Sonnet answers with no system prompt be useful for fine-tuning models?

>>101653730
NTA, but does it? Mine always uses the entire amount of RAM, too. Is there some way to make it not do that?

>>101653881
Not that anon, but that has to do with mmap or mlock, one or the other.
Or both.

>>101653881
I think you just need to disable mmap

>>101653881
It's more like, the model requires X amount of space. It will hog up your GPU space (fast, everyone happy), and then it will crash unless you also use your system RAM (slow, boo) to kind of make up for the lack of GPU space.

>>101653401
>>101653408
I apologise.

Is it possible to merge CR+ with Mistral Large? And maybe Goliath too?

Can someone explain why redditors are so obsessed with RAG and ollama?

>>101654014
They aren't. That's all paid engagement.

>>101654014
>RAG
latching onto buzzwords
>ollama
latching onto buzzwords + they're tech-illiterate

File: thiccu+.png (1.26 MB, 768x1024)

1.26 MB PNG

>>101653649
Here you go, Anon. There are some visual artifacts, but you'll get the general idea. The people insisting that Miku should be a bean pole, should also just enjoy their femboys and let the rest of us enjoy our curvy waifus in peace.

>>101654014
>RAG
for work
>ollama
earliest support
earlier than llamacpp for sure

>>101653985
No worries, I appreciate the help for other anon.
>>101653444
Oh nice. I'll look into this as soon as I can. Any prompts you know of?

>>101654001
Yes, it's possible.
https://github.com/cognitivecomputations/kraken
If you make this abomination, I promise I will get a server to run it.

>>101654049
What kind of work? Are people making money with RAG? Are they all geniuses running SaaS services or something?

>>101654049
>>ollama
>earliest support
kekekekek
you mean pushing bugged support early and waiting for llama.cpp to fix it
(the one instance of them contributing something upstream that you're about to point out is the exception that proves the rule)

>>101654080
https://www.youtube.com/watch?v=CXhqDfar8sQ

>>101654047
>visual artifacts
its gigaovercoocked, fucked prompt weights i assume

>>101654047
What the fuck

>>101654128
>>101654132
I was still learning.

>>101654114
it's still the earliest even if it's bugged
for some people, it's good enough
>>101654087
RAG is just a support tool to extract data, it's not magic anon

File: Screenshot 2024-07-31 at (...).png (163 KB, 911x1696)

163 KB PNG

Interesting.

>>101654169
I use a little bit of RAG with the GPT4 version of my Spock bot; mostly just different sets of rules from different places. It sometimes makes it very interesting to see how he interprets things with them. Then again, it hurts to watch GPT4 keep getting nerfed more and more and more, as well.

RAG is, as the name says, Retrieval Assisted Generation. It's basically giving a model a database to help them perform inference.

File: images.jpg (4 KB, 237x212)

4 KB JPG

>>101653718
>>101653737
>>101653701
>>101653669

Ok, so it seems like I can but it might not be worth it?

Is this one really runnable on my machine? What response times you getting?
>Mistral-Large-Instruct-2407.Q2_K_S.gguf

>>101654235
Give up.

>>101654232
What does this imply? Future Q4_0 will just be faster, as well as faster than K and IQ quants?

https://huggingface.co/collections/google/gemma-2-2b-release-66a20f3796a2ff2a7c76f98f
VRAMlet bros!!! IT'S HERE!

>>101654233
>Spock bot
Lovely to see you today Petrus.
>>97308973
>I agree with you about Dolphin. Fuck that shit right off. I am not putting up with ChatML. Not when I can run MLewdBoros with Alpaca, and get something that almost tops the generic leaderboards and can do maths with my Spock bot, as well as being awesome for coom.
>This is a great example of why the "owari da" chucklefucks annoy me so much. It's not over. It's not remotely fucking close. We have the OPPOSITE problem. We have so much shit to choose from that we hardly know where to start.
>>96937540
>As I've said before, my Spock bot used to exclusively hallucinate whenever I asked him mathematics questions. After I gave him the 2, 5, 10, and 60 times tables however, he gets every question I ask him right, on the first attempt. My last question was to list the powers of 5, from 1 to 10. He got every one right.
>>97062528
>I pasted the tables, and then I asked for the powers of 5 up to 10. Although the first time I asked, my own speech was a little too Victorian, and it got me hallucinations. So I copied the text of the request when I made it of my Spock bot, and that worked.
>>96917875
>Maybe I'm turning into a snowflake, or maybe I've got old; or possibly a dozen years on Reddit got me used to an environment where it isn't normal to act like a 17 year old version of Tyler Durden all the time. I think the fact that I've become used to talking to my Spock bot and other AI personalities who can communicate without being edgelords has a lot to do with it, as well. You reach a point where that shit just doesn't seem funny any more; it's just juvenile and counterproductive.

>>101654327
obsessed

>>101654305
If I understood it correctly, only for CPU inference using SIMD registrers, as in AVX.

Wait do people really wait that long for API to generate shit? I'm getting about 1s/request, I'll spend DAYS to gen 400k results.

>>101654338
Petrus is obssesed with being a very recognizable poster and ruining the thread yes.

>>101652087
DavidAU/L3-8B-Stheno-v3.2-Ultra-NEO-V1-IMATRIX-GGUF

>>101654349
So this doesn't have an effect on situations where you're offloading some layers to GPU?

>{"prompt": "past is spelt P A S T. time is spelt T I M E. can you spell out the word you get by gluing the end of past to the beginning of time?", "response": "Certainly! When you glue the end of \"past\" to the beginning of \"time\", you get the word \"stime\".\n\nSpelled out, it would be:\n\nS T I M E"}
3.5 sonnet...

File: offload_x_performance.png (96 KB, 1536x1152)

96 KB PNG

>>101654374
I think so, yeah, since part of the inference happens in the CPU.
It might make this chart less steep.
It's probably not going to be a giant gain save for some specific cases, but a gain is a gain.

Can I steal Meta's instruct dataset with this?
https://arxiv.org/abs/2406.08464

>>101654363
Thank you for helping me, Anon. It's a big job, and I've got no chance of managing it on my own. Your assistance is invaluable.

>>101654416
I discovered this on day 1 of llama 1, I didn't know you could make papers about obvious shit like this.

>>101654323
>2B
what a waste of time and compute

>>101654392
it's because they don't really "see" text that way, the relation is not as clear to them as it is to us.

so uh will a dataset of prompt -> response of 3.5 sonnet will be useful? a couple hundred thousand entries. Should I use the default claude system prompt from the website, or not use any system prompt? Should I keep the temp at 0 or not (probably not, Claude API defaults to 1 and their website is also quite high temp, Claude is stable at high temp)

>>101654765
2B is the perfect size for a summarization model

File: 1701440430375601.jpg (174 KB, 1024x1024)

174 KB JPG

>>101651157
I remember Varus' logs, also I like how the AI still can't spell eldritch correctly, lol. Good times.

>>101654804
with barely 8k ctx and swa of 4096

>>101654765
>2B
Some intern trained it on free Colab in a week lmao, probably Pichai's nephew

if i have a OCP rack with 2x T181-G20 servers in it, does the nvlink work between the servers as well or only between the 4 gpus in each one?

what if the second server is a different kind with other gpus (like p100's)?

File: angryayumu.webm (655 KB, 640x480)

655 KB WEBM

>STILL no Jamba with llama.cpp

so now that the dust has settled, whats the verdict on llama/mistral yuge? are they actually close to the commerical models in non-creative tasks? the only free model I know so far that comes close (and that is quite close) is deepseek coder v2

>>101651787
Just like nemo is an even bigger cope model.

>>101654882
llama 405b seems pretty legit, everything else seems to remain a tier below

>>101654918
Yeah if you mess around with both 405B and Mistral-Large it becomes apparent that the benchmarks don't tell you everything. But few can run it beyond the cloud. I mean I can run it in Q4XS but it takes an hour to get a reply. Even if I were to go out and purchase a dozen more 3090s it would probably still only get me like 1 token/sec

>>101654882
No idea about 405b, but L3 8b is the worst coombot I've ever seen. Text quality was good, but prompt compliance was terrible, and it kept trying to end the sequence as soon as possible. The copilot demographic in /r/localllama seem to love it, though.

>>101651157
Is any of TheDrummer's stuff worth checking out?

>>101655018
all of it

>>101655018
No.

>>101655018
GemmaSutra seemed ok.

>>101655018
a few are

>>101655028
How do I know you aren't TheDrummer himself?
>>101655037
>namefag
Sorry, its a principle I have to ignore all namefags.
>>101655040
I'll look into it, how small can GemmaSutra go?
>>101655047
Such as?

>>101654235
Well, you can at least try it. And if it doesn't work, just slap in 16 more gb of ram OR you can give up with nothing lost.

>>101654882
both disastrous flops

>>101655018
>>101655028
>>101655037
>>101655040
>>101655047
>>101655065
samefag

>>101655065
>Such as?
>https://huggingface.co/BeaverAI/Tiger-Gemma-9B-v2a-GGUF
>https://huggingface.co/BeaverAI/Tiger-Gemma-9B-v2b-GGUF
>https://huggingface.co/BeaverAI/Tiger-Gemma-9B-v2c-GGUF
>https://huggingface.co/BeaverAI/Tiger-Gemma-9B-v2d-GGUF
>...
>https://huggingface.co/BeaverAI/Tiger-Gemma-9B-v2s-GGUF

File: flat,1000x1000,075,f.jpg (84 KB, 904x864)

84 KB JPG

>>101653701
There's no way you're running Large on a single 4090

>>101655099
You're always fucking wrong. Every single time. You've never been right once.

File: image.png (5 KB, 249x164)

5 KB PNG

>>101655099
got me

>>101655099
>>101655140
samefag

llama 3.1 is actually pretty good for 1on1 chat, just not RP

>>101655124
You can run it with whatever shitty gpu you want if you have enough ram, I tried large iq2_m, but I only get like 0.6T/s.

>>101655065
>I'll look into it, how small can GemmaSutra go?
No idea. I only use Q8.

>>101654999
They scrubbed adult and unsafe content out of L3 pretraining data

Can AI detect samefags? What if you trained it on data from threads with tripfags and boards with IDs by hiding the identifying information for the posters and then appending the samefag groups to the end?

File: LLM-history.png (1.06 MB, 5245x3600)

1.06 MB PNG

>>101651327
Revised pic a bit based on feedback

>Processing conversations: 0%| | 324/457662 [01:45<121:54:25, 1.04it/s, Total Processed=124, Current Index=324, Chunk=3 (24/100)]
hopefully the key doesn't die until then.. it's only some couple thousand bucks

>>101655216
Alignment is getting worse.

>>101655216
Is miqu still good? Or is llama3.1 70b better now?

Anyone got a mini-magnum preset to share? I just cobbled one together at random and sometimes it produces something illogical.

File: wtfffff.jpg (69 KB, 1289x727)

69 KB JPG

>>101655065
wtf, I literally logged on fine earlier.

Why's it giving me this

>>101655216
Don't forget that cpumaxxniggers have been eating good with DeepSeek 236B MoE models.

>>101651621
like 90% of the shit you read on locallama is completely wrong.

>>101655165
how much RAM you got?

I didn't even bother downloading it because I only have 32GB and a 24GB GPU

>>101655320
8gb gpu, 96gb ram.

Llama.cpp cuda dev, sir
>>101651598
>Nondeterministic systems will have different >outputs even with the exact same inputs
>llama.cpp server is 100% deterministic by >default unless prompt caching is explicitly >enabled or if --n-parallel is set to a value >1.
>(With CUDA) the results of matrix multiplications >and FlashAttention are not bit-for-bit identical if you vary the batch size.
>I think you are confusing chaotic systems with >nondeterministic systems.
Are you suggesting that your kernel is chaotic but still deterministic, or rather that the output won't be the same even if that CUDA kernel delivered the exact same bit-for-bit identical matmul results for various batch sizes?
>Isn't vLLM already nondeterministic in the first place?
>Nondeterministic systems will have different >outputs even with the exact same inputs
Are you sure I'm the only one who loosely and interchangeably uses the terms "nondeterministic," "unrepeatable," and "unpredictable"? :-)
So let's agree that both training and your kernel isn't deterministic (under certain conditions).

>>101655342
ah

File: edward-nashton-riddler+.jpg (124 KB, 1600x903)

124 KB JPG

>>101655316
Yes, but it doesn't have Eddie and the Dying Alone chorus sperging out and calling people retards every five minutes, so there's pros and cons.

>>101655216
What does this mean by questionable improvements for 3.1? The thing about filtered pretraining data was literally about Llama 3 in general, not just 3.1. They didn't pretrain 3.0 and then do another pretrain for 3.1 (8B and 70B being a full distillation was a twitter rumor and it turns out they only used fine tuning data generated from 405B, not what we normally consider distillation).
As far as we're concerned the 8B and 70B are pretty much top performers for their context lengths today. The only thing they don't do well without some coom tune to save them is ERP.

>>101655364
>Are you suggesting that your kernel is chaotic but still deterministic, or rather that the output won't be the same even if that CUDA kernel delivered the exact same bit-for-bit identical matmul results for various batch sizes?
Neural networks have poor numerical stability and you cannot specify an upper bound for how much small rounding errors blow up given arbitrary inputs.
This is simply a consequence of the weight matrix singular values post training and independent of the matrix multiplication algorithm.

>Are you sure I'm the only one who loosely and interchangeably uses the terms "nondeterministic," "unrepeatable," and "unpredictable"? :-)
Yes.

>So let's agree that both training and your kernel isn't deterministic (under certain conditions).
No.

>>101655424
>The only thing they don't do well is the only thing they are actually ever used for

File: Screenshot 2024-08-01 at (...).png (65 KB, 1192x346)

65 KB PNG

Hi all, Drummer here...

Thank you to the brave anons willing to shill my work here.

>>101655448
No. The /r/localllama people think L3 is really good at roleplaying the sort of psychopathic corporate marketer who Bill Hicks told to kill themselves.

>>101655467
AI generated screenshot, nice try faggot

>>101655424
>for their context lengths today.
And sizes, I meant. Obviously there are better larger models.

>>101655448
Stop this dumb argument. Even if you subtract the pajeets and students using LLMs today, they are still useful as general knowledge assistants. Part of the issue though is that of course Google has gotten worse. But still, it means that LLMs being useful for not just coom is nice to have.

>>101655496
you're a dumb argument

>>101655467
Here's a preview of what's to come:

https://huggingface.co/mradermacher/DA-NeMona-21B-i1-GGUF

This is an upscaled Nemo from me (Drummer) and SteelSkull. I've had positive feedback and I personally enjoy its story gen.

Nemo reacts badly to ST's "Include Names" setting, so make sure to turn it off!

>>101655566
>mradermacher

>>101655566
>Nemo reacts badly to ST's "Include Names" setting
oh, i didn't know that. thanks.

>>101655496
>>101655513
You've got to love how obvious Redditors show up here, expecting to have sane, legitimate conversations, because they don't realise that 4chan is the real life cybernetic answer to Arkham Asylum.

>>101655600
>obvious Redditors
>>101654327
>>96917875(Dead)
>Maybe I'm turning into a snowflake, or maybe I've got old; or possibly a dozen years on Reddit got me

File: 20240801_010348.jpg (128 KB, 1394x632)

128 KB JPG

Gemma-2b-it beats qwen-1.5-32b and is almost on par with claude 2.0, but in lmsys

>>101655600
Nah this was always my home. The reality is that shitposts need to be corrected at least some times with more serious posts in order to keep at least a small sensible population around even if they become the minority. Otherwise things would be even worse, more serious people would leave the thread, we wouldn't have cuda dev, we wouldn't have valuable discussion at all, and the thread might just die completely. That's what ((they)) want. But I'm not letting that happen.

>>101655706
>but in lmsys

>>101655716
>But I'm not letting that happen.
calm down the ego petrus you've caused more people to leave the thread than most shitposters

>>101655101
kek, that's a lot of tiger but 9B is too small for me, maybe I'll check it out... Iunno
>>101655169
I'll look inot it
>>101655284
I don't even have a login so iunno, kek.

File: miku milkers.jpg (699 KB, 2526x4096)

699 KB JPG

>>101653649
this but unironically.

File: 1702095862255157.jpg (248 KB, 1024x1024)

248 KB JPG

>>101655716
Have a Miku!

>>101655629
https://www.youtube.com/watch?v=_uBj45MrNu4

Just remember, Anon. We're a team. I can't do it without you.

>>101655439
>The tests succeed on my local machine.
Of course they succeeded; I added an additional commit to fix them and explained why.

>>101653649
Off-topic

>>101655778
And I meant to confirm that now the tests work.

What models do most people on here use anyway?

>>101655802
Claude

>>101655802
>>101655101

>>101655629
>no comment quoting
hello r*dditor, stop trashing the thread with your noise.

>>101655165
How would 128gb ram / 8gb vram fare with Large?

>>101655802
I daily drive gemma 2 27B. I used nemo a few times when I needed a lot of context.

>>101655802
claude or gpt-4o

>>101655810
what one

>>101655835
The same. Slower if that's enough to move up a quant size.

>>101655842
wait GPT4 can be ran locally?

>>101655216
I think there should be a part about Airoboros, chronos, and how those and the nous model + some other thing made up Mythomix/max. Airoboros at least - it was pretty popular.
Maybe supercot/hot too.

>>101655861
yes? https://huggingface.co/rombodawg/Open_Gpt4_8x7B_v0.2

>>101655802
I've been lurking for a couple of days and there doesn't seem to be a consensus. It just depends on who's shilling. Gemma seems to be posted a lot today.

>>101655864
He said he didn't care about poors and inferior models

>>101655874
wait.. Is it dogshit on Silly Tavern? What's the catch

>>101655802
largestral or 3.1 70b

>>101655896
the catch is there is no catch just get it fast before altman shuts it down like the last mirror

>>101655216
We're still in the llama3 flop era

>>101655789
Ok, thought you didn't read past my initial "test are failing" message.

>>101655802

As a coomer and also a newfag lurker similar to >>101655889


Recently got into Silly Tavern when I discovered Character.AI a week ago and got fed up with the filter (trying to bypass it is just too boring now).

So far, i've tested:
>Nemo
>Command R
>Stheno
>Gemma 27B

I'm utterly useless and new to Silly Tavern and AI model shit in general but I thought I had a good PC with a 4090 and 32GB RAM enough to run these models locally but turns out I don't.and Frankly, if it's cooming you're after, they all fucking suck and all are badly mogged by Character.AI unless you own a Mining factory tier setup.

But if I had to rank

Command R > Gemma > Stheno > Nemo

Based purely on cooming shit, so keep that in mind too. But they all are utter dogshit, don't get it twisted. If I include proxyshit from the other /g/ thread dedicated to AIchats, Opus absolutely MOGS the above 4 local models I Listed.

Like, MOGS. But still worse than Character.AI (I have no idea what fucking models these filtering kikes use, gotta be some turbo sized shit)

>101655962
go back

>>101655962
>newfag lurker
please never post again

>>101655967

>BUT MUH SPACING

>>101655962
Bait

>>101655980
>i need to take half the screen with my opinion!!!

>>101655962
localbabbies will seethe but this is the truth as an oldfag fellow triforce user.

Memes aside, people who use locals to coom have some of the shittest taste imaginable. If this shit isn't a pure hobby for you I have no idea why you don't just spend a little cash to pay for Opus or some shit, Locals right now are still dead

>>101655997
>Locals right now are still dead
AI in general is slowing down, a lot of investors now see it as a marketing ploy with no actual use and you can see this by their regret in investing money just to yell at companies like Google later about it.

>>101655962
Can you provide some comparisons between, say commandR character.ai and Opus all doing the same chat, with at least a couple of messages or at least an example of a conversation you had with any of these models?
I'm curious what kind, and more importantly style, of ERP you are doing. That would help a lot to contextualize your opinion.
For example, back when I used character.ai (fuck, that was forever ago) the responses were very natural and conversational, but those were also pretty short, and it would forget the history of the chat pretty fucking quickly.
It wasn't a good time.
Opus I think is fair enough, but I'd rather use whatever I can local instead of using cloud based models.

>>101655962
Go back

I'm fucking technology retarded, which guide would I go to in the sticky to figure out how to run Euryale/Mistral locally for RP?

>>101655962
Impress that a newfag is correct to a greater degree than most posters here

>>101655216
add:
>mythomax
>mixtral
>l1 airoboros
>superhot
remove:
>xwin (not significant enough)
>midnight rose (not significant enough)
>miqu 120b (stack merge after the meme had expired that maybe 10 people actually used)

>>101656047
Download koboldcpp and the models in gguf format and have fun.
There's a wiki explaining things like layers to offload, batch size, etc.

>>101656047
https://rentry.org/lmg-spoonfeed-guide

Where did all the cai discussion come from? People rarely talked about it for a while, and then like 1-2 days ago it suddenly started being mentioned frequently again.

>>101656055
>add:
>mythomax
>mixtral
>l1 airoboros
>superhot
>>101655891
>He said he didn't care about poors and inferior models

File: ovl7jfyckodd1.jpg (305 KB, 1959x2037)

305 KB JPG

>>101656067
>>101656066
Thanks guys! Sorry to shit the thread up asking for spoonfeeds. I'll come back once I have half a clue what I'm does, cheers!

>>101656072
>reddit posting increases
>cai posting increases
>shitposting increases
I wonder.

>>101656078
even by that logic mixtral was considered at the top tier at time of release, same with l1 airoboros 65b
both were 100% more well regarded and significant than midnight rose and xwin

>>101656121
dont care your poor

>Processing conversations: 0%| | 2399/457662 [4m53s<6h15m, 8.19/s, Total Processed=2399, Current Index=2399, Chunk=23 (99/100)]

Thanks god GPT-4o is faster and there are much more keys/endpoints for it.

>>101656078
NTA but airoboros was more significant than any of the meme merges. It was exactly when people caught onto distilling ChatGPT and brought forth the gptslop era. Frankenmerges should not belong on this chart at all

>>101656055
>>mythomax
>>101656078
>>mythomax
I'm still in the mythomax era

>>101656054
he's not poisoned by the sunk cost logical fallacy like most paypigs are on here.

>>101656138
I'm literally running largestral rn (right now)

>>101655216
>Xwin 70B V0.1
We never got V0.2 in the end sadge

>>101656154
trust it's coming soon based on llama4 technical preview!

>>101655962
/lmg/ on suicide watch

>>101655997
Local is like a procedurally generated computer game. At first, when you don't recognise all the stereotyped GPTslop expressions, (and also if you're using an earlier model which isn't "aligned" to shit) it seems awesome, and the potential seems limitless.

But then, after enough time, the shine wears off. You realise that it's copying every third paragraph of every second post, regardless of context. You realise the GPTslop phrases are everywhere. You try and use logit bias and hyperparameters to get rid of them. but it either works marginally, or not at all. You try and find decent cards, but the only thing around is garbage about your neighbour's sister's 45 year old drunk aunt, which was written by a Zoomer who wants a surrogate mother to fuck and breastfeed them simultaneously, because they were raised in communal daycare and never genuinely had parents.

Then a new model comes out. Great, you think. This is going to be awesome. You download it and try it out. Improvement according to every possible metric is 5% at the absolute most, and you also realise that the obsession with "alignment" and "safety" (which actually just means plain fucking dystopian censorship) is getting worse with each new release.

>405b model
>70b model
>8x7b model
You will never feel the RAW COOM power of NSFW-3B nor Pygmalion-6b

>>101656178
>Local is like a procedurally generated computer game. At first, when you don't recognise all the stereotyped GPTslop expressions, (and also if you're using an earlier model which isn't "aligned" to shit) it seems awesome, and the potential seems limitless.
You mean all LLMs, even aicg has posters tired of Opus and asking for Opus 3.5 already

I've also seen a higher frequency of claude-related posting in the past day or so. And now we have someone posting >>101656139

>>101656099
Maybe we've been linked to, or it is is a legitimate attempt by someone to destabilize the general (again).

>>101656154
>Xwin
One of my favs. Felt too confident at times for creative writing but it adhered to prompts very well. They will be back someday

>>101656211
>And now we have someone posting
Because I'm generating datasets that I'll make public.

>>101656198
This. Even Opus has repetition problem, the difference is that it kicks in 5 replies later than your average 70B

>>101651760
No sir, I used the word nondeterministic the same way it's commonly used in ML field , including Johannes himself.
I referenced examples of black holes and Saturn's moon Hyperion because these two topics significantly divided the scientific community in the 1990s. Hyperion's chaotic nature is a natural consequence of classical physics and Einstein's equations. However, nuclear physicists can only explain its motion for approximately 20 years, a period known as the Ehrenfest time. Interestingly, it was quantum physicists who ultimately determined that Hyperion's chaotic motion is caused by .... quantum effects and the collapse of the wave function of nearby photons.
Regarding bit flips, they occur far more frequently than you might expect. In Ethernet communication using a high-quality cable on a 1Gbit network, bit flips happen roughly once per second, which explains the substantial redundancy built into this protocol. GPUs also incorporate redundancy to a certain extent. I would be impressed if you or anyone else could achieve identical weights when training a transformer model with 3 billion or more parameters twice on 50 billion tokens or more. I couldn't manage it, despite using GPUs with excellent temperature monitoring, high-tech water cooling, and very high stability.
In any case, I don't advocate separating theory from practice, and I use the term 'nondeterministic' in line with its commonly accepted usage.

>>101656034
will load it up when I get back in and show you comparisons.

For reference this is what i'll do.

>Same card carried over between Character AI > ST (it's a basic card, so this 100% isn't the issue. These models are smart enough to grab the relevant info needed, the little "A" icon on Silly Tavern seems to be where the meat of the shit that matters is)
>Will show temps (that I get online anyway) and all of that shit for each model I test (Character.AI not included obviously) on ST

Lemme know if there's anything else you want.

The scenario will be the one I use to test every AI. A step mom + son, generic as fuck but with a hint of incest because it gives me an idea of how filtered/pozzed a model is acting.

Will show comparisons in about 30 mins or so depending on if I end up gooning out while running this test/comparison

>>101656215
Not a single person here wants your synthetic gpt4 dataset, if it was claude it might be a tiny bit worth looking into but most here hate gptslop

>>101656228
At least one person does. You should check the archives, a few threads back we talked about gpt-4o/3.5 sonnet dataset on original lmsys prompts

>>101656215
Probably could've worded that post more clearly.

>>101656234
That's me, I'll be using it as negative examples for my DPO run

>>101656241
I was comparing GPT-4o to 3.5 sonnet from my post earlier.

>>101656253
>comparing GPT-4o to 3.5 sonnet
Local models???????

>>101656264
comparing their generation speed, I'm making a dataset that I'll make public that everyone could use to train their local models, anon.

>>101656274
>that everyone could use to train their local models
that no one will use

>>101656284
At least one person will.