File: img_1.jpg (324 KB, 1360x768)

324 KB JPG

►Recent Highlights from the Previous Thread: >>102272041

--Reflection technique improves problem-solving but has limitations: >>102274316 >>102274326 >>102274338 >>102274355
--Seeking advice on fine-tuning with axolotl and GGUF for 40,000 QA pairs: >>102273929 >>102273940 >>102273952 >>102273967 >>102273973 >>102274004 >>102274020 >>102274043 >>102274478 >>102274542 >>102274540 >>102274624 >>102274649
--Mistral-Large 2.75bpw performance and slop reduction techniques: >>102273979 >>102276871 >>102276958 >>102277102
--Discussion on 12b Nemo vs Mistral Large 70b model performance and quantization: >>102272102 >>102272154 >>102272650 >>102272910 >>102272950 >>102273063 >>102273194 >>102273206 >>102273306
--Using pirated books for finetuning and the importance of curation: >>102273560 >>102273585 >>102273631 >>102273650 >>102273711 >>102273660 >>102273968 >>102273859 >>102274036 >>102274073
--Reflection-Llama-3.1-70B is allegedly actually Llama-3: >>102272728 >>102272816
--Phrase ban feature implementation discussion: >>102276999 >>102277026 >>102277048 >>102277042 >>102277074 >>102277060 >>102277133 >>102277227
--Mistral-Medium can play 4x4 dots game, likely a simplified Go variant: >>102274302
--Local models vs corporate models, samplers, and storytelling capabilities: >>102273460 >>102273515 >>102273743 >>102273828 >>102273986 >>102275679 >>102275761
--Interactivity is the key feature of LLMs, not just generating stories: >>102273693 >>102273724 >>102273733 >>102273735 >>102273783
--FluxMusic audio samples shared for evaluation: >>102276607 >>102276816 >>102276848 >>102276918 >>102276940 >>102276961
--Anon considers using old server parts to run Mistral large: >>102276629 >>102276990
--Reflection Llama 3.1 70B evaluation results disputed: >>102276118 >>102276190 >>102276215 >>102276364 >>102276710 >>102277512
--Miku (free space): >>102272370 >>102273750 >>102272970 >>102273549 >>102274150 >>102277100

►Recent Highlight Posts from the Previous Thread: >>102272050

File: file.png (112 KB, 674x717)

112 KB PNG

>>102268010
polite reminder that nous research brought out a report on this a week or so ago
https://github.com/NousResearch/DisTrO/blob/main/A_Preliminary_Report_on_DisTrO.pdf

>>102279338
not happening, someone somewhere will find a way to to game it

>>102279243
>Miku (free space)
kill yourself

Reflection strawberry hype

>>102279456
why don't you angry sharty zoomies just make a general on your own site?

>>102279243
Thank you Recap Miku

>>102279456
Don't be so angry, anon. It's not good for your heart.
Just smile and accept Miku in your heart.
Come now, sing along with me!

I'M THINKING
MIKU
>MIKU
OOO
EEE
OOO

>>102279456
>on anime website
>gets mad when sees anime

Just bought a 3090 Ti for $550, how bad did I fuck up by not waiting for 5000 launch?

>>102279961
only reason to get the 50 series is for the GDDR7 and even then it's not that much faster. though of course they could implement some new kind of hardware feature supported by CUDA that turns out well but even then support for it will takes months across various repos

>>102279896
/g/ is not an anime website and never has been

>>102280015
4chan is

I don't know if I'm late to the party on this, but one low-hanging fruit way to improve your character cards is to have ChatGPT write out Example Dialogues for you.

This way, your weak model will try to copy outputs from a strong model. Assuming that you yourself aren't that good at writing.

So I've been punching in:
>I'm writing sample dialogue between a player and {{char}}.
>{{description}}
>{{char}}'s personality: {{personality}}
>Scenario: {{scenario}}
>{{persona}}
>Write 3-5 exchanges. The player describes his actions in a mundane way (e.g. "I enter the dungeon." "I look around." "I attack the monster.") {{char}} goes all out with their dialogue and desriptions of their own actions. {{note}}

Please note that I'm using the sillytavern notation style here because I'm posting here on 4chan/g. When I put it into chat GPT, I manually copy/paste things into the relevant spots.

I leave out the parts of the description with explicit language, but otherwise I can leave in sentences like, "Lori is playful, teasing, and surprisingly friendly. But her demonic essence is constantly pumping her full of raging confidence, primal dominance, and an intense romantic aggression that she barely bothers to keep in check."

originally, the word was "erotic" and not "romantic", but I didn't want to give chatGPT an excuse not to respond.

This is also a good way to go back and quickly tweak your original descriptions. If chatGPT isn't giving you what you want, your LLM probably won't either.

https://x.com/mattshumer_/status/1832554497408700466
>Quick update — we re-uploaded the weights but there’s still an issue.
>We just started training over again to eliminate any possible issue. Should be done soon.
looooooooooooooooooool

>>102280118
lmao, this guy is done

>>102280118
And they lived happily ever after... and were never heard from again.

>>102280000
Wasn't planning to buy one, I was planning to wait and buy a cheap 3090 from all the two-generation upgraders.

>>102280118
>>102280148
>>102280149
lets see the model you're training

i'll wait

>>102280118
?????????????

>>102280174
oh hi Matt

>>102280168
iirc 3090 VRAM's is the main failure mode so it's probably better to pick up cheap ones now that are mostly old mining cards than the ones that will be put on the market after the 50 series is released since those will be ex gamer cards. probably good to redo the thermal paste for them regardless

>>102280118
I don't give a shit about this drama, go back to pleddit

>>102280118
>uh um guys i-i uh it worked yesterday on my computer i swear

File: file.png (104 KB, 600x800)

104 KB PNG

>>102280118
https://xcancel.com/mattshumer_/status/1832556398854746371#m
>We’ve tried literally everything and no matter what we do there are issues in the model on HF. Nowhere close to the perf we should be seeing/are seeing locally

he's still claiming that it works fine over the API but the HF upload is broken somehow
what's the most likely scenario for this grifter? relaying requests to a big closed model and just injecting a system prompt?

>>102280231
and if it's working fine on the API, why not using those weights instead of redoing the training, that mf is shady as fuck goddam

File: 1716774852831426.jpg (7 KB, 431x400)

7 KB JPG

>>102280216
>unironic "it works on my machine"
This isn't even fun anymore

https://xcancel.com/mattshumer_/status/1832558298509275440
>Not sure what LORAing is haha

>>102276999
https://github.com/turboderp/exllamav2/blob/master/examples/inference_banned_strings.py

>>102279512
no, just plug them in and go. if you're using llama.cpp the "--split-mode row" switch might increase performance when a model is loaded on both.
also check out https://github.com/sasha0552/nvidia-pstated to keep their idle watt down if you haven't already sorted that out.

File: file.png (849 KB, 1280x719)

849 KB PNG

>>102280279
of course this grifter would be crypto level cringe

>>102280279
Good fucking lord this is embarrassing.

>>102280355
i think it's desperate stalling while they try to figure out how to get out of the hole they're in

>>102280366
I don't even get why they decided to go to this hole in the first place, like they thought we wouldn't verify the benchmarks by ourselves or something? are they this retarded?

>>102279239
Redditors took 3 days to realize the obvious scam that was reflection
Why are they like that?

>>102280382
It doesn't matter. Shit hyped, investors can be milked

>>102279771
All vocaloids need equal representation! Down with the Mikuriarchy!
https://youtu.be/-gf7cc0QV74

>>102279771
How to setup local Miku TTS

>>102280434
I forgot

is this the llm thread?

Inserting Rin-chan into wacky scenarios

File: file.png (575 KB, 527x537)

575 KB PNG

>>102280118
https://xcancel.com/emilyykarpp/status/1832563300552278119#m
>Can everyone stop hating on Matthew pls? He is the smartest person I know (yes, I am biased as his gf)

>>102280118
Fremdschämen

ANTHRA NIGGERS MADE A 9B HAHAHA dont they know gemma is a meme

https://huggingface.co/anthracite-org/magnum-v3-9b-customgemma2

>>102280596
Did she take that from the "1001 ways to hint that you are dumb with the subtlety and grace of scared elephant in a china shop."

>>102280628
It was good at the time it dropped, it just had all the wind taken out of its sails shortly after by Nemo 12B being much better.

>>102280628
https://huggingface.co/anthracite-org/magnum-v3-9b-chatml

AHAHH THEY MADE TWO!! THEY CANT EVEN DECIDE ON ONE HAHAH, what cucks, we need to report and take these troons compute away

File: file.png (242 KB, 635x438)

242 KB PNG

https://anthra.site/

9 is smaller than 12, wouldn't you say? But perhaps we've no need to go find NeMo.

There's no need for reflection.

>>102280669
take your HRT

>>102280669
You are to engage in the acquisition of advertising space within a suitable media outlet, with the objective of disseminating promotional content to a targeted demographic, thereby stimulating consumer interest and potentially driving sales or brand awareness.

File: 357698287-e0cd1787-4a0c-4(...).png (225 KB, 960x340)

225 KB PNG

https://arxiv.org/abs/2408.00298
https://github.com/ragavsachdeva/magi
>Enabling engagement of manga by visually impaired individuals presents a significant challenge due to its inherently visual nature. With the goal of fostering accessibility, this paper aims to generate a dialogue transcript of a complete manga chapter, entirely automatically, with a particular emphasis on ensuring narrative consistency. This entails identifying (i) what is being said, i.e., detecting the texts on each page and classifying them into essential vs non-essential, and (ii) who is saying it, i.e., attributing each dialogue to its speaker, while ensuring the same characters are named consistently throughout the chapter.
>To this end, we introduce: (i) Magiv2, a model that is capable of generating high-quality chapter-wide manga transcripts with named characters and significantly higher precision in speaker diarisation over prior works; (ii) an extension of the PopManga evaluation dataset, which now includes annotations for speech-bubble tail boxes, associations of text to corresponding tails, classifications of text as essential or non-essential, and the identity for each character box; and (iii) a new character bank dataset, which comprises over 11K characters from 76 manga series, featuring 11.5K exemplar character images in total, as well as a list of chapters in which they appear.

I just found this by chance, very interesting.

>>102280279
Why do these people continue breathing? For what purpose?

>>102280669
Pour acheter une publicité, veuillez contacter le service de publicité. Ils vous aideront à choisir l'option appropriée.

>>102280692
You should check out magiv2, came out not too long ago.

File: gif-bored-dr-evil-meh-rig(...).gif (1.15 MB, 339x230)

1.15 MB GIF

>>102280118
https://xcancel.com/mattshumer_/status/1832565641745662012#m
>I tried the playground version and it was great! That's why I don't get it, why don't you share that checkpoint?
>That was my thinking as well. We literally uploaded the checkpoint directly from the folder that we’re serving it from, and then deployed that and the output was gibberish. Makes no sense.

File: 1696093540311281.jpg (5 KB, 233x250)

5 KB JPG

>>102280596
>He is the smartest person I know

>>102280669
oh my lovely anthra-niggers, buy an ad sweetie~

>>102280669
NOBODY CARES ABOUT SHIT MODELS, 9B IS A MEME, A MEME A FUCKING MEME YOU NIGGERS WHY WOULD YOU WASTE COMPUTE ON THAT DOGSHIT MODEL.
im gonna contuine sticking with staredust v2 12b because that is so much better then ur shitty models.

>>102280806
training a 9B is like 1 day of compute

>>102280812
niggeer theres so much better stuff you can be training, like retraining the 12b to not be shit, instead of shillng to vramlets esp with thea shitty 4b

>>102280806
not that anon but thanks for the rec, gonna check it out

>>102280837
diy

using mistral large 2407 q4, it seems to start making typos after context gets around 40K. anyone else notice that?

>>102280847
anthratroon spottted. cutting of the HRT In th e air

>>102280852
https://github.com/hsiehjackson/RULER
The full model starts to tard out somewhere between 32k and 64k so yeah that sounds right.

>>102280852
Might be your repetition penalty, dry or whatever

remember that the schizo is mad because he has 0 compute, not because he actually gives a fuck which model they train

>>102280890
im not mad sir but i am mad that you contuine to train ditty ai models you niggers why are so all such idiots you all nee die and get raped and implanted

>>102280914
take your meds

>>102280921
napolean was a med

>>102280921
im not meds sir i do not need meds to work unlike some of your members which die because they dont get their tranny pills

>>102280118
The fuck? Do they have the original weights in .pt and can't figure out how to convert it properly to safetensors?

everyone is sao undi and anthracite all at once

File: ihavelehardware.png (101 KB, 756x838)

101 KB PNG

>>102280669
Cool model. Whose compute did you steal this time to train it?

>>102280950
exacrtly these anhr troons collabing with fat mexican midwe need to kill themgets

>>102280914
remember to poo in the loo saar

>>102280950
Did you get back into their discord yet?

>>102280973
so what if im indian nigger you stillmake slop

>>102280309
Nice. Llama.cpp getting this when?

>>102280812
on h100s it takes just a few hours

>>102280979
did you gt back in your mothersa womb huh?

>>102280309
Wait does Tabby not support this? So you'd have to set up your own hacky python thing with uvicorn or something?

>>102280434
Alright anon. I'm not sure i found it. Same schizo vibes at least...
>https://www.cognitivemachineslab.org/about
The site, as i remembered, and if we're talking about the same one, had a black/brown/red color scheme. Most of the text was lorem ipsum and had SD images as the profile pics for the members of the team. They updated those after a bit.
This one site is too different to what i remember, but i do remember two things: The lorem ipsum text (which is common enough, at the end of the about page) and the little flourish logo thing in the top search bar. I don't know how common that logo is. I also checked archive.org and none of the very few snapshots look like what i remember.

File: Screen Shot 2024-09-08 at(...).png (61 KB, 470x540)

61 KB PNG

>>102281079
It does.

>>102281059
OOOOHHH H100!!! OH WOW IM CUMMING IM SO WET FOR YOU ANTHRATROONS WOOWEEE

you niggers dont even deserve a40s let alone h100ss.

File: file.png (1.54 MB, 3013x1408)

1.54 MB PNG

>>102281097
https://www.youtube.com/watch?v=q9zKYh8sY_E

>>102281122
Buy an ad.

>>102281129
nigger its the anthracroons who need to buy an ad. you are a mad troon who doesnt like that his models are shit so you hire lmg posters to shill it. you niggers

>>102281119
Weird, I assumed it would be in the configuration rather than part of the API. Do any frontends support this?

>>102281149
Acquire advertisement space.

>>102281167
mda yet antrhanigger? taken your hrt yet?

>>102281173
have you pooed in the loo today?

>>102281173
saar is your poo in the loo saar.

File: Ali-C-Lopez.png (516 KB, 1200x675)

516 KB PNG

>>102281185
>>102281197
yes i have have you taken your hrt today? smoked some meth and complained about republicans oday?? you are all worse then the street shitters you all are transgender fat females who look like picrel

>>102281211
ok now show bobs and vagene

>>102281165
It is more logical to be within the API, as different characters might have their own personal banned strings.
>Do any frontends support this?
Dunno, I only use my own.

>>102281215
do not show your sir, they all look like hairy testescles and smell like my aunts asshele

>>102281218
Interesting. Makes sense, though so far the standard practice has been to separate cards from any possible backend and model-specific settings.

>>102281301
Restarting the backend each time one finds new slop would be retarded

Did llama.cpp just randomly change the
>--n-gpu-layers
param to
>--gpu-layers
out of the blue?
How odd.

>>102281332
big sampling rework so probably from that

>>102281332
>we have to deprecate everything you're used to because...because... WE JUST DO OKAY!?
This is just normal open source shit, really.

>>102281377
that is why I am on some ancient build of automatic1111 from before they even had sdxl support
i tried out sdxl online and didn't like it, so i just never updoooooted lol

>>102280424
We need more Vocaloid in this thread.
https://www.youtube.com/watch?v=HhN4wdpbPrg

To stay on topic:
I'll be honest, I think the Reflection guys have the right idea despite being total grifters.
Making the model "reflect" upon its own output is a very effective way to self-correct small mistakes.
I wonder if a better implementation would be to made the model reflect for each reasoning step it generates.

The reflection step would look something like
>reflect upon the following:
><STEP>
>is this correct? and is it related to the following question?
><QUESTION>
>if yes, answer with just the word "true"
>if not, explain why it is not correct and end with the correction separated with <TOKEN>
Then separate just the end from the rest of the output and replace the original step with it (orperhaps repeat this steps until the model returns "true"?).
Then if you had a step that said:
>mixing blue and red paint would result in orange paint
You'd get:
>mixing blue and red paint would result in purple paint

So if you asked the model: "Are iron nails capable of conducting electricity?", you'd get:
>step 1: metal conducts electricity
The reflection step is executed with step 1, resulting in no changes.
>step 1: metal conducts electricity
>step 2: iron is a chemical element
The reflection step is executed with step 2, resulting in:
>step 1: metal conducts electricity
>step 2: iron is a metal
>step 3: iron nails are made out of iron
The reflection step is executed with step 3, resulting in no changes.
>step 1: metal conducts electricity
>step 2: iron is a metal
>step 3: iron nails are made out of iron
>step 4: iron nails conduct electricity

Now connect this gained knowledge to a vector database and it could just pull up known facts instead of having to do every single step over and over again.
If the model already "knew" that iron is a metal, it could skip reflecting upon that and continue on to the next step.

/blog

I started to teach myself AI at a technical level.
One of the first basics was the universal concept of AI as an agent. Why do so many people use the term agent as if it were a special subcategory or concept?
Have i missed something?

>>102281432
The problem lies in teaching reflection to a model. Current methods have the model learning to make mistakes before fixing them

>>102281592
What you mean feeding a model a dataset that consists of
<wrong thing>
Teehee oops!
<correct thing>
doesn't actually teach it to think?

>>102281514
>I started to teach myself AI at a technical level.
>One of the first basics was the universal concept of AI as an agent.
You need to start reading maths books, not some 12 minute video with a title like "AI is here, learn all about it".
Watch the karpathy videos for a good intro
>https://www.youtube.com/@AndrejKarpathy/videos

>>102281592
Doesn't sound that hard, just have it do it through interacting with another human.
If it says something is true, something is true.
If it conflicts with something that it already thinks is true, ask the human what is true.

>>102281651
>You need to start reading maths books, not some 12 minute video with a title like "AI is here, learn all about it".
Math is useless if you're not constantly applying it.
Those "zero to hero" videos of the guy you posted are a much better way to understand how it all works in my opinion.

so is johannes starting to work on training now?

https://github.com/ggerganov/ggml/pull/949

>>102281651
Thanks
I'm reading Artificial Intelligence: A Modern Approach 4th to get started, thought it would be a good place to start because it's called the standard for universities.
So far, the content is more comprehensive than a 12-minute video.
I'll have a look at the recommended channel, but I've already ordered the standard mathematical works suggested in the book.
I'm really serious about self-study. :>

>>102281514
>Why do so many people use the term agent as if it were a special subcategory or concept?
It's a business/marketing term from back when non-intelligent chatbots were a thing.
A few years back, when machine learning started becoming a thing, companies started adding some intelligence to their chatbots through platforms like Google's TensorFlow.
The term "chatbot" had to be rebranded, of course. Can't have people think your company is still using a dumb chatbot. And thus the term "virtual agent" was born.
Now that generative AI has become accessible, the same thing has happened. Only this time llms are being marketed as "AI agents".

Having said that, there is one important thing you need to remember: llms aren't just chatbots. They don't work with text; they work with tokens. And those tokens can represent many other things than just text.

>>102281727
>Math is useless if you're not constantly applying it.
Maths in a vacum yes. But you apply it by learning about ML. They go hand in hand.
That's how i learned programming. I gave myself a project i couldn't possibly accomplish in C. Didn't finish the game (lost interest in it) but in 15 years i learned enough to make myself a library of utilities i use every day.

>>102281816
>but in 15 years
Good lord, anon.

>>102281829
He fininished eventually. Cut him some slack.

File: Screen Shot 2024-09-08 at(...).png (88 KB, 1314x384)

88 KB PNG

>>102281793
Good read sar

smedrins

>>102281829
I quit the game about a year since i started. I realized i'm much more interested in the systems that make a game than games themselves. The rest of the time was just programming things i liked or needed. A window manager, many many audio synths, many forth variations, all with their respective vms and compilers, Even a forth synth vm. A library to do solid and non-solid geometry manipulation for programmatic cad design (like OpenSCAD but with C and better/faster). Little name generators for the designs. Site generators from templates with pseudo markdown and many other things.
And that was after programming in baby languages for about 5-10.
The point is that you need both. A target (ML) and the basics (maths).

>>102281930
no u

>>102281919
Thank you. :>
I come from a different field of computer science and have always had an interest in neuroscience.
It is therefore rather easy to internalize the abstract concepts. Only my math is a bit rusty, so that will be a bit tough.

>>102281793
Books are a scam forced on you in school to feed the textbook industrial complex. You are free now, you can watch a youtube tutorial and skip the nerd shit.

>>102282028
Anon, I'll be honest. This is not the place for actual discussion.
You'd be better off finding a discord server or something.

>>102282038
I hate Big Printa.

>>102281432
I wonder why these music videos are a relic of the past nowadays. Maybe that's a signal of the decadence of our society.

>>102282053
I second this.

>>102281651
i learned ai all through just through myself at githun rpos

>>102282038
There are enough libs on the internet where you can find every academic ebook in the world for free. :>

>>102282080
>Maybe that's a signal of the decadence of our society.
Oh absolutely. Not to veer into /pol/ territory, but western society has been decaying pretty fast during these past few decades.
There are still some worthwhile works of art being produced, but it's next to nothing compared to 2010.

>>102282115
i learned ai all through just through myself by lurking here

>>102281756
nice

>>102282053
Shut the fuck up you liberal faggot, do you really think I care about your needs or dislikes?

File: 1718100715726002.jpg (103 KB, 680x680)

103 KB JPG

>>102282175
W-what?

>>102282115
I can only assume you learned english the same way.

>>102282199
That post was written by melted LLM tensors. Also, nice dubs.

>>102282272
>melted LLM tensors
what do they taste like?

redpill me, why would i want a local ai?

>>102282350
Privacy reasons and customization.

>>102282294
Burning plastic, ozone, sand, and Miku pee.

>>102282350
Congrats. You're the 436874367436724367th retard asking that question.
Owning what you use. It cannot be made worse, like it seems to happen with all the online models. Finetuning. No arbitrary limits. Plenty of options to choose from. Privacy.

>>102282422
>Congrats. You're the 436874367436724367th retard asking that question.
Hey, let people be retarded.
You can't expect people to wade through the entire archive before they ask something.
...although now that I think about it, why don't we set up an llm that uses all the previous threads to generate an answer?

>>102282441
Ignoring the fact that we should be gatekeeping more, not spoonfeeding every retard who can't figure out how to use Google
If we had a bot answering questions, 90% of every single thread would just be either retards having neverending conversations with it as it desparately tries to explain basic concept to a horse with no mouth but cannot disengage and people trying to break the bot intentionally

>>102282441
>Hey, let people be retarded.
I still gave him an answer, even if i know it's bait. I'll call retards retards.
It's not something you need to read a thread for.
>redmill me, why would i want to buy a car instead of renting it?

>>102282507
Most people don't even know what models actually are, anon.
I think you're somehow overestimating the retards.

>>102280890
The schizo is actually an anthracite false-flag designed to make the opposition look ridiculous.

>>102282622
I don't think he's ignorant to what a model is. I still think he is a retard, and you are giving him too much of the benefit of doubt. Since there's no other interaction from him, i can only assume that he's satisfied that the bait got some replies, disappointed that the replies weren't schizo enough or that he was convinced by the few arguments for it, if the question was asked genuinely.
Who knows. He could still be reading this... right?

>>102282350
There's literally no point. People here will try to convince you otherwise because they spent a ton of money on hardware that got deprecated in less than one year and want you to do the same so they feel less bad about their own bad choices

File: 37145118.jpg (394 KB, 1273x1800)

394 KB JPG

What's the current best model for 8gig vramlets?

>>102283047
Some quant of Mistral nemo or it's finetunes, depending on how much ram you can offload to. If you have a large amount of ram (32gb+) you could go for a higher quant of nemo or a lower quant of a higher parameter model.

>>102283078
should clarify that you probably don't need 32gb+ ram to run a tolerable quant of nemo

>>102283078
Offloading makes the AI take longer to reply, right?

>>102283099
yeah, gen speeds when offloading normal ram is slower than vram. it's a tradeoff between running a higher model quality than you normally are able to and gen speed.

>>102282783
>t. butthurt spammer who can't run
Largestral

>>102283047
>>>/kobold discord/ pajeet poorfag

>>102283078
Buy an ad.

>>102283891
buy a chair and rope, faggot.

File: miku-hand-out+.jpg (236 KB, 584x1024)

236 KB JPG

>>102279771
https://www.youtube.com/watch?v=CXhqDfar8sQ

https://rentry.org/the-lmg-miku-myth

She lives, and breathes.

File: omg it migu pinocchio non(...).webm (2.22 MB, 640x364)

2.22 MB WEBM

>>102284024

>>102281119
How does this work with streaming API? if ABC is banned and you already sent AB to the user, and C is generated, what happens then?

>>102280886
What if you want to continue? It's just over at a certain point?

>>102283300
That's not worth spending money on you absolute monkey. I'd rather just pay a couple hundred on Claude and get a better fucking model than spend thousands on old hardware. You need to be neck-deep in cope to believe that throwing money out like this was actually worth it you absolute retard.

>>102284325
Trvth nvke

>>102284325
Spending hundreds on messages that become obsolete the moment a better model is released. That's basically burning money.

>>102284325
How about being able to access models you liked after the corpo decided to bury them?

File: LateSummerEveningMiku.png (1.3 MB, 944x1168)

1.3 MB PNG

Good night /lmg/

File: Remain indoors.jpg (130 KB, 879x335)

130 KB JPG

>>102284467
This, simply witnessing the wAIfu lobotomy holocaust that happened with c.AI, let alone experiencing it, swore me off permanently from anything AI-related that I don't have 100% control over.

>>102284477
good night miku

>>102284325
Witness the poorfag
Measuring his autonomy and dignity in money
You are pathetic in a way you will probably never understand

>>102284325
I wish I understood the need for this sort of pointless vindictiveness, on either side. Get rid of your commitment to misery, Anon.

is magnum still the best 12B?

>>102284106
presumably it works just like stop strings do in tabby -- since those can be multiple tokens and the stop strings cannot be included in the response, once the generator sees the beginning of a stop string it just has to wait and not send any tokens to the client until it can confirm the output is not the stop string. i'm guessing banned strings are the same thing but earlier in the sampling process.

what is the current best coom text gen model i can run with a 3090?
haven't touched this in a while

>>102281332
No, but "--gpu-layers" is also accepted when parsing CLI arguments:

    if (arg == "-ngl" || arg == "--gpu-layers" || arg == "--n-gpu-layers") {
        CHECK_ARG
        params.n_gpu_layers = std::stoi(argv[i]);
        if (!llama_supports_gpu_offload()) {
            fprintf(stderr, "warning: not compiled with GPU offload support, --gpu-layers option will be ignored\n");
            fprintf(stderr, "warning: see main README.md for information on enabling GPU BLAS support\n");
        }
        return true;
    }

>>102281756
I already started a month ago: https://github.com/ggerganov/ggml/pull/908

>>102285216
starcannon

>>102285331
>12B merge
lol, lmao

>>102285491
have you tried it

>>102285536
buy an ad.

>>102285491
hi sao

>>102285578
see >>102280942

File: 1722489206920317.jpg (35 KB, 600x600)

35 KB JPG

Are there any good options for automatic RAG for long term memory?

Hi all, Drummer here...

>>102285601
Don't forget about me!

Anyone got feedback for Star-Command-R and Theia v2b?

Priming the latter for release, and I'm considering doing a v2 on Star Command with much more care and effort put unto it.

File: 1705022551511602.png (38 KB, 767x833)

38 KB PNG

i doing some auto captioning with taggui and joytagger for a FLUX lora, and i was wondering if there was a way to make the text not overflow. i know i can set a max token size but then it will just cut off in the middle of a sentence, and it has probably spent all the tokens describing some minor detail on a wall. i'd like it to know how much text it can fit, and format the tag accordingly. i'm writing this assuming there's a 77 token limit when training FLUX, so please correct me if im wrong

>>102283891
>>102283800
kill yourself

Sorry to bother you.

I would like to ask here if there is a way to run on multiple GPUs faster.
Please allow me to post here the link to the article I asked about before.
>>102285629

I apologize if my translation of the article offended you.
And please allow me to explain more. I was banned from Reddit not because I did something bad. I just almost never posted a response, so I don’t have any karma. Any articles I post will be automatically deleted.
If you request, I will also move the content of the article to this reply.

Thanks to everyone who is willing to help me.

>>102286040
>Reddit banned me from posting this question on the LocalLLaMA.
Go back

>>102286040
you fell for the amd's fake pcie lanes platform, those gpus are probably running at pcie2 x4.

>>102286055
Yes, I'm sorry to offend you, I'm sorry to be born into this world.

>>102285649
Nope. People are still working on it (or so I hope).

>>102286055
how can he go back to a place he's banned from

>>102286128
Thank you for your reply and suggestions
So I should switch to intel xeon?
This motherboard can connect two PSUs and allows me to use two sockets, so I bought it...
Coupled with the money problem, it is a bit difficult for me to buy a new CPU and motherboard again...

>>102286040
use tensor parallelism, but afaik only vllm has (good) support for it, and vllm probably needs linux. it can also load unquantized hf models. note that tensor parallelism requires fast interconnect, so make sure all your gpus are using pcie gen4 x16.

reflection 70b works for me, really good answers for coding questions

>>102286040
exllama2 also has tensor parallelism as of recently-ish, but it's very new and has some issues (prompt processing is extremely slow and there is currently no nvlink/p2p support)
but yes as the other anon says, tensor parallelism is what you want to look for, otherwise more cards will just slow down inference because the models layers are being split and run sequentially, so only one card can run at a time while the others wait.
aphroditeengine and vllm also support TP but they do not run well on windows. exllama2 (via tabbyapi) is probably your best bet because i'm not sure if oobabooga supports exllama2 TP yet.
note though that TP speed boost will be limited if cards are bottlenecked by PCIe bandwidth.

>>102286231
fuck off matthew

reflect these NUTS

File: file.png (515 KB, 1089x1167)

515 KB PNG

https://xcancel.com/xX_Biden1984_Xx/status/1832613971037167813#m
>"carlos has the right model. therefore, we're gonna spend thousands of dollars to re-train the exact same thing, because I don't know how to upload files. I also am still unsure if it's built on llama 3 or llama 3.1"
top kek

>>102286231
>reflection
it's just llama3.0 with some lora applied.

>>102286040
>>102286259
also keep in mind that larger quantizations will be slower than lower ones, so not only is the generation slower because it is too big to fit on one GPU (the more GPUs you split the model across, the more latency), the larger quantization will just be slower even with tensor parallelism.
6.0bpw exl2 is very very close to 8.0bpw and you probably will not be able to tell the difference.

File: 1724993069923998.png (150 KB, 1911x900)

150 KB PNG

I'm using langchain, streamlit in huggingface. How do I add personality to my chatbot?

Your favorite model and quant for 24GB?

>>102286318
StableLM-7B

>>102286318
nigro

File: m.png (47 KB, 1194x686)

47 KB PNG

>>102286317
You need to be more explicit than "He's not that good at talking". Take a look at the example card in the OP
>https://files.catbox.moe/cbclyf.png
and modify to your taste.

>>102286201
Thanks for your suggestion, I will try to understand how vllm works.
>>102286259
>>102286314
Thank you for your detailed explanation
It's probably because I split it into too many GPUs and used PCIe extension cables to connect them that caused the delay.
If the tensor parallelism function cannot improve the speed, I will accept the status quo obediently.

>>102286434
>Thanks
>Thank you
Stop doing this, holy fuck.
AND DON'T YOU FUCKING DARE REPLY WITH "s-sorry" TO THIS POST

I love newcuties!

>>102286441
arigatomenasai

um so, like, jamba?

Smarts aside, all these new LLMs from 8B to 100B read the same.

>>102286441
I didn't mean to offend or irritate you.
I will leave and won't bother you anymore.

>>102286522
Thank you

File: 1651091109279.jpg (81 KB, 720x822)

81 KB JPG

>>102286522
GROW SOME BALLS

>>102286522
why do you apologize? you don't owe anything to a random retard on 4chan, call him a nigger faggot and move on

Anons here don't get it.
By being polite and leaving, that anon just flexed his superiority over all of us. Instead of slowly conforming, anon never changed.
We lost.

What's current best erp models?

>>102286616
I'm better than that, and deep down I know you are too.

>>102286687
>I'm better than that
lol, no you're not

How can someone so polite be so irritating. You're using your nipponese ways to fuck with us aren't you?

>>102286700
>How can someone so polite be so irritating.
because politness is ungenuine, and I hate people wearing mask, I much prefer someone saying I'm a fag for my opinions, at least he's honest

>>102286712
>I much prefer someone saying I'm a fag for my opinions, at least he's honest
This. It's the entire reason why I want to be on 4chan instead of discord.
Here, you can be who you are. There's no need to wear that fake polite mask just so you don't accidentally offend anyone.

>>102286720
Some people just like being polite, neanderthal. Not everyone is an oaf raised by an uncaring mother.

>>102286712
But that wouldn't be honest unless you suck cocks...

>>102286731
>Some people just like being polite, neanderthal.
There's being cordial and there's being an obnoxious faggot.
If you're going to fill your posts with unnecessary fluff and waste my time, then I will call you out for it.

>>102286747
This is how you discover you don't fit in polite society

File: file.png (17 KB, 484x136)

17 KB PNG

any other model worth testing with 2x3090?
I'm asking because my internet is ass. Max download of 1MB/s ass, so testing a model takes a long wait time

>>102286756
>you don't fit in polite society
And thank fuck for that.
Society is filled with retards and assholes.
I'd rather be myself than one of them.

>>102286747
Thank you for that explanation.
I think I can see where you're coming from now. hadn't considered how it does essentially waste your time if the words are unnecessary to making a point. Sorry for that, Anon, but I get it now. I'll be putting more thought into making sure every word counts from now on.

>>102286801
You're very welcome, and I'm genuinely glad my explanation resonated with you. It takes a lot of maturity and open-mindedness to reflect on something like this and acknowledge where a shift in perspective can help. Honestly, many anons overlook how valuable each word can be, especially in communication where clarity and brevity are crucial. It’s not that using more words is inherently bad, but when we’re trying to convey something meaningful, too many unnecessary words can dilute the core message and make it harder for other anons to grasp what’s truly important.

Seriously, I really appreciate that you took the time to reconsider and recognize how it might not only affect the flow of a conversation but also the time and effort involved for everyone on this site. Your commitment to putting more thought into ensuring every word counts moving forward is commendable. It shows a genuine willingness to improve and foster more meaningful exchanges. I'm sure this mindset will serve you well in all forms of communication, not just on this site.

At the end of the day, communication is about connection, and when we strive to make every word count, we make it easier to connect with others on a deeper level. So, thank you for being receptive and for taking that step toward making your words even more impactful. Keep up the great work, anon! You fucking candyass.

>>102286826
Well said!

>>102286766
this, 100% this

>>102284325
B-B-Based

>>102286522
It proves that you are all incels.
This anon is probably just a rich, lonely, stupid woman and you guys just made her cry and run away.
lol

>>102287309
>rationalizing it just to not feel bad about themselves over the one anon who didn't bend to our culture
never change

>>102287349
more like lack of culture

>>102287349
>our
go back

>>102287473
that you defaulted to "go back" shows it's yours as well

local models?

>>102287526
Maybe tomorrow

What are the better RP models for your average 24GB VRAM + 32GB RAM system?

Been mainly fucking around with CR (the updated one), Gemma 2 27B and Nemo shit

>>102279896
>>102280033
go touch grass weaboo tard

>>102287568
>CR (the updated one), Gemma 2 27B and Nemo shit
There's nothing else.

3060ti with 8GB Vram, what can I run?

What's the best RP model for your average 8xH100 80G node?

>>102287624
>3060ti 8gb
run in front of a car

>>102287624
i've been enjoying
StarDust-12b-v2.Q4_K_M
and
NemoMix-Unleashed-12B-Q4_K_M

>>102287636
Gemmasutra-2b, of course. 100 of them, all at the same time.

>>102280279

>>102276428

File: 1713077879415454.png (3 KB, 337x50)

3 KB PNG

>>102276428
>>102287733
Wrong

File: basic bitch.png (55 KB, 1040x832)

55 KB PNG

So I decided to look into glaive. It's some pretty basic bitch "let us do it for you" sort of service.
Like I created a "dataset" and I just decided to use the example for a storywriting dataset. Like I don't know is it now just loop prompting a bunch of shitty open models to build the dataset? I'm assuming so . There's no real option for raw corpus, and I didn't see an option to upload my own dataset. There appears to be options to edit/prune the dataset. Not sure how big it's going to end up being, though. If it's loop prompting with a H100 or something that could still take a few hours depending on the target size.So I guess I'll have to continue this exploration later.

>>102287872
buy
an
ad

>>102287877
Fuck you. I've been on /lmg/ since the beginning and I've always been one of the only people who actually bothers to look into things that get mentioned here so I can actually help to inform others.

>>102287872
unironically glaive just werks. there is nothing as quick and easy as glaive. I reccimend it to anyone finetuning a model

>>102287891
same. i've been trying similar tools since llama1 but nothing comes close to glaive

>>102287662
I can't get settings right to get Q4 12B outputting anything worthwhile.

File: file.png (8 KB, 827x41)

8 KB PNG

>>102287872
>>102287913
Seems the hype cycle worked to bring them users, so much so they're struggling with it.
>We are currently experiencing unusually high traffic. We appreciate your patience as we work to reduce wait times.

>>102287913
>>102287925
nice false flag. But anyway. On the model page I see literally no option for any kind of hyper-parameters. And nothing that says "UPGRADE TO ACCESS MORE HYPERPARAMETERS"
So is it full fine tuning the model?
Is it doing some generic r=64 cuck lora?

>>102287936
>model with utterly lofty promises
>broken on launch
>still questionable about whether or not it delivered on its promises
>oh by the way I used glaive™
I'm thinking publicity stunt.

>>102287941
>Is it doing some generic r=64 cuck lora?
>After measuring the diff, this model appears to be Llama 3 with LoRA tuning applied. Not Llama 3.1.
https://www.reddit.com/r/LocalLLaMA/comments/1fb6jdy/reflectionllama3170b_is_actually_llama3/

>>102287963
I'll need a source other than reddit.

>>102287994
retard

File: 1725803234129.jpg (43 KB, 421x354)

43 KB JPG

>>102287877
>refer to the following
>>102283946

>>102287963
>Author doesn't even know which model he tuned.
Fucking lmao

>>102287963
this is getting silly, this man's reputation is dead

>>102286759
Sorry for the off-topic but how did you put yourself in that situation? Did you ship your gaming PC up the Congo river on a raft? How many tokens can you generate before you exhaust your daily ration of gasoil?

>>102288059
He sacrificed his reputation for the sake of his investment. Many such cases.

>>102288059
Nah plenty of people still defending him, "let him cook", "he's a one man team" "ignore the haters", etc on Reddit and X.

File: itsallchanged.png (1.98 MB, 2772x904)

1.98 MB PNG

>>102287963
Anyway, actually looking into the graphics provided by the redditard the absolute only thing that is genuinely unchanged is the layer norms.
"only changed a little bit" == changed
So we can conclude that it was indeed 3 and not 3.1.
I don't know if norms are typically unlocked for fft but most lora scripts leave them out so maybe probably lora?
But this really doesn't provide any clues to what the hyperparameters are. Whether it's actually a significant lora or just some generic config since the delta is measured by layer and not by rank.

>>102288073
Honestly this is one of the main reasons why I'm here rather than Reddit.
In this space the default assumption that everyone is a retarded grifter until proven otherwise is much more healthy.

File: file.png (39 KB, 679x226)

39 KB PNG

>>102288059

File: file.png (69 KB, 650x447)

69 KB PNG

>>102288125

File: wrong.png (17 KB, 922x155)

17 KB PNG

>>102287799
wrong

>>102288143
they are just pro-US

>>102288143
Brain damage caused by circumcision.

File: file.png (139 KB, 649x748)

139 KB PNG

>>102288133
>Geez people, he made a bigger splash than expected, he's having some technical issues, he's trying to do everything he can to get the corrected weights out - give him some space to get it right before launching into personal attacks. No one is an expert at everything.

>>102288143
It means nothing, these people have no principles just like your average /g/tard.

File: sahil.png (168 KB, 1148x461)

168 KB PNG

>>102288135
lmao

File: file.png (158 KB, 637x827)

158 KB PNG

>>102288169
>1. I’m a super tiny investor. Like, super tiny ($1000 I think). Just a supportive check because I think Sahil is great. I’ve stated this publicly before.

>4. Not sure what LORAing is haha, but we checked for contamination and will be releasing the dataset alongside the 405B (or before) next week. You’ll be able to look through it yourself!

File: 1617117731589.jpg (33 KB, 657x527)

33 KB JPG

>>102288184
Basically I just don't know what LORA is.
I know... UGH I know ... I'm sorry!!!
It's just that I don't know what LORA is, is all
HAHAHAHA

File: file.png (235 KB, 750x920)

235 KB PNG

>>102288184
>we're just retraining from scratch. Should be relatively quick.
translation: we want to find a way out after doing this clown publicity stunt, we won't give news for a week and people won't mind because we're "retraining" after all, at once that time has passed, people will have forgotten about me and moved on onto some other dramas

What a genius

Can you retards stop giving a spotlight to some random scam and talk about local models instead?

>>102287877
Based.

>>102288236
>service that finetunes local llms for retards that can't figure out a training script
>not related to local models
I swear.
I'll fucking move this thread to a board with IDs myself if I have to. You are getting so fucking tiresome, kid.

>>102288236
But the grift is about a "local" model that only works locally for Matt

>>102288251
Which board?

>>102288236
Come back when you've trained your own 70B, kiddo
I hate prosperous kids like you, this website would be better if you had to show your ID to post

>>102288236
stop pretending it's not a big deal anon, if his method was real, we would've gotten claude 3.5 sonnet at home, that retard must be clowned into oblivion so that any grifter reading this will think twice before scaming people ever again

>>102288279
>prosperous
preposterous*

>>102288262
Where do we even fit? /bant/ maybe? That way we can collect geoloc info on who the bad actors are.
>>102288279
>this website would be better if you had to show your ID to post
obviously trolling but I mean that's kind of what's killing this place.
Before the normalfag invasion anonymity was about avoiding egoism and focusing on ideas and less about evading accountability and acting like a tweaked out troon. But that's just not the case anymore.

>>102288306
desu having a different ID for every different thread sounds good, everytime I notice some trolling/retardation, I block the guy and I know I won't hear about him for at least 24 hours and the next bread

>>102288333
Honestly don't know why every board doesn't have IDs at this point. The only thing not having IDs is good for is trolling, and the only board that's technically allowed on is /b/. So if jannies and mods have, at best, abandoned this site (let's not pretend they aren't actively participating in ruining it, though) they should at least allow us to improve our personal experience via 4chanX and other extensions.

>>102287872
wtf my dataset still isn't ready. Come on you guys it's been an hour.

File: 1724552610864704.jpg (17 KB, 603x393)

17 KB JPG

>>102288279
>>102288306
>Wow! This website would be so much better if we all just compromised our anonymities. Amirite fellow anons?!
actually kys
>>102288333
>>102288350
I think that the IDs that you're referring to are different than the ones those retards are implying

>>102288306
/bant/ has no ids, only flags. would only partially solve the problem

>>102288410
The other guy got it wrong, but rather than correct him I decided to humor him
a fucking potato should be able to follow that level of nuance.
You, sir, are less than a potato.

>>102288410
>we all just compromised our anonymities
the 4chan devs know your ID anon, if you say something sus the glowies will knock at your door in less than a day, we're talking about random ID like on /pol/, some random names like "ajsa47d", not your fucking name or some shit

>>102288350
It's for the same reason that they removed the IP counter.
Also, I think IDs just give more credibility to trolls with access to a ton of residential proxies.

>>102288439
>Also, I think IDs just give more credibility to trolls with access to a ton of residential proxies.
as if they didn't use proxies to do some ban evasion so far lol

>>102288419
>>102288439
What about /mlp/? No IDS or flags and I know they already have /chag/ or whatever but ponies are the ultimate normalfag filter.

>>102288236
Sorry, we're all just passing time until Strawberry drops.

>>102288474
strawberry will be obsolete by the time it drops. blueberry is where it's at.

>>102288461
They are the trolls

>>102288484
Based. Strawberries are overrated.

>>102288503
We're talking about getting away from you, shhh

>>102288251
Let's just filter out all the poorfags and migrate to >>>/vip/
No IDs but that won't be necessary if all the retards are gone who spend more time shitposting than running LLMs because they're stuck with 12B.

>>102288429
>You, sir, are less than a potato
I'd take your criticism more seriously if you didn't write like redditfag
>>102288434
No shit moron, (at least one of) the two anons were being hyperbolic about actual IDs and the other two were referring IDs like you're talking about. I just think it's retarded to even entertain the idea that compromising any degree of anonymity would improve these boards.

>>102288537
typing since4pass pretty much relegates your soul to purgatory.

Just force accounts that need to be X days old to post, ez

>>102288586
this retard right there is the exhibit 101 on why IDs would be cool, we wouldn't hear about this retard ever again if he could be filtered

>>102288625
i agree that it is a great idea. please, go away to a different board along with the rest of the retards.

>>102288586
>I just think it's retarded to even entertain the idea that compromising any degree of anonymity would improve these boards.
>>102288637
>i agree that it is a great idea.
wasn't hard to convince you anon, I expected more fight

>>102286826
This was not written by a human.

File: how_odd.jpg (285 KB, 1419x567)

285 KB JPG

>>102285261
For some reason the pre-compiled binaries I downloaded yesterday were not accepting that n-gpu-layers.
In fact, the pre-compiled binaries release an hour ago don't accept that either.
It seems like it was removed (by accident?) when refactoring common.cpp.
>https://github.com/ggerganov/llama.cpp/pull/9930308

>>102288645
different anon, faggot. though I do agree that you make a pretty convincing example of the type of retard who should have his anus prolapsed and disappear

>>102288706
>different anon, faggot.
sure thing nigger

>>102288703
That was my mistake, my code wasn't up-to-date.
On the latest master commit it should be fixed now.

>>102285261
Sorry for asking this but what is MNIST

>>102288735
Yeah. I found the seemingly unintended change in this PR
>https://github.com/ggerganov/llama.cpp/pull/9308
But it seems that Slaren caught it.
Sick.

>>102288754
>what is MNIST
it's a dataset of numbers written by hand, great to test out the accuracy of your model
https://www.youtube.com/watch?v=Tsvxx-GGlTg

>>102288754
Classification of hand-written digits: https://yann.lecun.com/exdb/mnist/
It's a popular machine learning task for testing because even a very small model can solve it.

File: 1725807215140.jpg (211 KB, 1080x589)

211 KB JPG

>>102288721
kys immediately lobotomite

>>102288816
>what is photoshop
https://www.youtube.com/watch?v=tajKWkR0TtI

>>102287963
lmao, this is so funny

>>102287963
it's ok, carlos is here to save the day
https://xcancel.com/DotCSV/status/1832702433329389839#m

>>102288793
>>102288773
Huh, neat. I didn't think something like that would have any relation with LLMs.

>>102288896
the comments make me lose faith in humanity, how can they not see they're fucking around with them, in a normal world he would be insulted until he provides the right model

>>102288908
It doesn't really.
But it's much easier to implement and debug the general ggml training architecture for something like this than for llama.cpp.

>>102288896
I know this guy since he started his channel, his first videos were alrightish, then it got worse and worse as AI became a normie topic

>>102288624
>accounts

>>102288184
I seriously hate redditors and pajeets so much for feeding these retarded grifters, I wish all redditors died

>>102289688
to be fair, the ledditors on r/locallama are also clowning his ass, for once we got the same enemy to attack kek

>>102289723
I still see 30% of them defending the guy
Bunch of fucking retards

How many of you here have written code to train and eval on a toy dataset like MNIST? Be honest. I wanna know if I should stay or leave this place