So I happen to be tinkering with local LLMs and I had an idea I wanted to try to train some small local model.As always the quality of the information is extremely important, if you feed garbage you are only confusing the model wasting compute in garbage.I wanted to make a lite LLM with short stories, but before that I need to properly format it, I was building a system to format that data for simplicity my first iteration was going to be short stories.So I downloaded one of their official datasets, called Cosmopedia, it had a convenient short stories database, also it was generated by a Mistral LLM so I would have the information distilled, I download the entire stories dataset but then I notice through a random sample that one story had python code..I did a Regular Expression search on the database and found that 25% of the stories datasets were tainted data that contained absurd coding problems.Huggingface CEO is best buddies with with OpenAI... the same startup that is scanning books and then shredding them... makes you wonder... they tainted the supposed "curated" datasets...
>>539910271>scanning books and then shredding themI get why they are scanning them, but destroying books seems sacrilege (unless it's some shit that's come out in the past ~15 years about gay people and kids, etc etc). Like... if the LLMs are ordering custom published copies of random old manuals and shit to scan in and have the model read, I get it... but old/rare books, it seems weird to destroy.
>>539910428>it seems weird to destroyNo, it doesn't. Wouldn't you want full control of all knowledge and have the ability to manipulate it whenever you see fit?Knowledge is power. And they are taking it all for themselves.
>>539910271Surely you can get a better collection of short stories from project Gutenberg.How much RAM/VRAM does the system you're training it on have?
>>539910428It would be noble if they scanned those old books, catalogue them and then release those as properly formated PDFs.But they are destroying those because they dont want others to have that literature/information.On the other hand the same thing that happened to me happened to others, the datasets to train LLMs are being tainted, LLM generated content is generated on the internet that in turn is fed again to the same LLM that produced it...And these American Startups are shredding valuable human language for the sake of short term profits.
>>539910698>Surely you can get a better collection of short stories from project Gutenberg.I started with Gutenberg but found several problems like improperly scanned text or old english expressions that are no longer used, but considering the current situation I will have to go back.The other thing is doing distillation with a local LLM like Qwen3.5 that will be better than anything generated on Mistral.>How much RAM/VRAM does the system you're training it on have?Consumer grade, 24GB VRAM and 32GB Ram, but I plan to expand to 96GB VRAM ( multiple GPUs ) and 128GB ram if I can get results, but in any case what I want is to learn and tinker with training and generating neural network... so for purposes of learning my current rig is enough.
>>539911008>Consumer grade, 24GB VRAM and 32GB RamDamn son. I only have 24GB RAM/12GB VRAM. I can run a fair few local models but never even considered using it to train something. Think it could be possible on my specs? How long would it take if so?
>>539910271No shit Sherlock
>>539911653What I did is working with deepseek flash and pro in tandem, I tell them to create a plan for creating a training script and to list as many optimizations as possible considering the hardware I have, and it works... Deepseek pro generates the technical documentation and acts like a system architect while deepseek flash does the heavy lifting, you might want to try some newer models like Kimi-K3 for that system architect role ( you dont have to do heavy processing just documentation )you can train locally, I once had the old gemini finding out a way to dramatically reduce ram consumption during training in a previous experiment...The problem is largely time... training a model a a single GPU takes an insane amount of time... but with your rig you certainly can do it... at least if you just want to learn the mechanics... if you are thinking to train something like 8 billion or more parameters you most likely will need to rent hardware on the cloud... Unless of course you happen to find the code for machine cognition, who knows? Evolutions happens in the wild not in a lab. Also keep in mind that LLMs are not the only type of neural networks.
>>539910428I wish I had real insider info about this. I highly doubt these people are scanning first editions and destroying them. If you go on ebay you can find whole libraries of books for extremely cheap.....it would not be hard to buy a shit load of used books that otherwise would be thrown away, scan them and then toss them. People are disgusted naturally at the claim, but waste of this sort happens very often......It feels like a headline in the age of it being fashionable to absolutely loathe anything and everything involving AI.
>>539910271A lot of the companies, meta, anthropic, openai, used torrents. There's a notorious "books3" torrent that was used, it had a lot of interesting literature. There was another dataset called "the pile", which had all kinds of interesting and not so common data, including all the Eron emails (and perhaps it also references books3 and other interesting datasets). It would appear that these people pulled the ladder up from under them and such things are harder to find on the public internet.I understand you want to try to use your own LLM to generate a story but, it only gets really good at that when you give it lots of good quality literature. Even if you're not planning to generate a novel, the novels contain more data. Even the non-fiction books seem to help because these often have good prose and quotes and so on. Which help the model generalise better.Locally, your best bet might actually be to just download one of the Chinese models that can comfortably run on your machine. Or go with something like a llama3 variant that fits on your card. (not llama4 that's jeeted and worse). Both of these already have some people doing finetunes to unfuck them with refusals, so called "uncensored" models (they might still refuse, just less than the base model). Search ollama for uncensored and see. These models can be a good basis of your own fine tune. And what I would recommend you doing is:>A little more pretraining (just regular batch training) on some rare literature (especially pre-WWII rare books, hand pick at least a dozen full books).>Curate a mini dataset of at least 100 of the stories you personally like, you fine tune on theseThen it will be very capable of generating stories for you. I also encourage you to incorporate your own custom tokens into the finetuning process you could do something like <storyrequest>Generate a racist fairytale</storyrequest><story>Once upon a time in niggerland....</story>then it will do really good.
>>539913034The problem is that those books are not on the digital, because otherwise they would use the digital.By not publishing the digital copies they are destroying what might be valuable material while forfeiting its content for the rest of humanity.If on the other hand they would release those digital copies it would be a noble act.Its ironic that jews bitch about nazis burning books but now they are destroying potential cultural heritage for the sake of profits...On the other hand there is a value now to those books, why would they go to do that instead of just using more internet data? the reason is what I posted, internet became a place full of low quality data and now they are on a feedback loop of LLMs generating content that is then ingested again, Claude 5 is going autistic and performing worse than its previous version so this feedback loop is not working.These people are psychopats they want to keep high quality data for themselves while everyone else gets LLM slop.
>>539913312I didnt think of it that way desu and yeah that is a lot more concerning.It'll be a shit PR move if they dont get transparent about this shit soon......otherwise its a shady practice and lord willing we dont see more of it in the future (im being optimistic here)>These people are psychopats they want to keep high quality data for themselves while everyone else gets LLM slop.I can absolutely see this being a thing, considering its a part of our history pre-AI, to sequester certain knowledge into corners until it becomes myth or forgotten
>>539913274>Locally, your best bet might actually be to just download one of the Chinese models that can comfortably run on your machine.I currently have those uncensored models and already have sys prompts to properly make content.But it would take a lot of time to generate a large amount of those text on my rig...
>>539913475>It'll be a shit PR move if they dont get transparent about this shit soon.They lack the intelligence, they are billionares but that does not make them smart, if they would release those books on PDF format for archiving purpose then it would be a great PR move, its basic corporate ettiquete.
>>539913828>then it would be a great PR movewho gives a fuck what the retarded nigger cattle thinks? no ones reads anymore anyhow
>>539913964less about what people read and more about whats preserved in the humanities, dont need a million readers to make a book worth existing.
>>539910271use the kek dataset and deepseek. everything else is trash. a kek trained bot wreaked havoc here. this whole thread could be bots trained on that data set.
>watching coders ruin coding
>>539914680isnt it this way with everything?
>>539914733yes, sectors of people often ignore nation-wrecking schemes if it puts them in chargedoesn't make it right or something we should repeat
first world and third world are BOTH about to become second world
>>539914860no it doesnt, I'm just asking if this process has existed in the past and if we couldnt avoid it.....what makes our lifespan anymore special than the last?Will we suddenly eradicate man's impulse towards selfish preservation, towards war, towards tyranny?An age of honesty? It never existed.
>>539910271>download synthslop dataset>look inside>synthslopskill issue
>>539913034>>539910428I've heard that it's because of a court ruling that scanning books for archiving is legal, but only if the original is destroyed. Otherwise it's a breach of copyright.That said, I've seen no proof that this scanning and shredding is even taking place, just lots of rumours
>>539916071right, there's a scary picture of a book on a scanning machine (these have been in operation for like more than a decade) and yellow headlines that are so cartoonishly alarmist, I cant take them at face value
>>539910271>OpenAI... the same startup that is scanning books and then shredding themI thought that was Anthropic. But what difference does it actually make... anyway, bump.
>>539910271>shitty euroscrub llm outputs shit>blame americans
