Welcome to the Pony Voice Preservation Project!youtu.be/730zGRwbQuEThe Pony Preservation Project is a collaborative effort by /mlp/ to build and curate pony datasets for as many applications in AI as possible.Technology has progressed such that a trained neural network can generate convincing voice clips, drawings and text for any person or character using existing audio recordings, artwork and fanfics as a reference. As you can surely imagine, AI pony voices, drawings and text have endless applications for pony content creation.AI is incredibly versatile, basically anything that can be boiled down to a simple dataset can be used for training to create more of it. AI-generated images, fanfics, wAIfu chatbots and even animation are possible, and are being worked on here.Any anon is free to join, and there are many active tasks that would suit any level of technical expertise. If you’re interested in helping out, take a look at the quick start guide linked below and ask in the thread for any further detail you need.EQG and G5 are not welcome.>Quick start guide:docs.google.com/document/d/1PDkSrKKiHzzpUTKzBldZeKngvjeBUjyTtGCOv2GWwa0/editIntroduction to the PPP, links to text-to-speech tools, and how (You) can help with active tasks.>The main Doc:docs.google.com/document/d/1y1pfS0LCrwbbvxdn3ZksH25BKaf0LaO13uYppxIQnac/editAn in-depth repository of tutorials, resources and archives.>Active tasks:Research into animation AIResearch into pony image generation>Latest developments:GDrive clone of Master File now available >>37159549SortAnon releases script to run TalkNet on Windows >>37299594TalkNet training script >>37374942GPT-J downloadable model >>37646318FiMmicroSoL model >>38027533Delta GPT-J notebook + tutorial >>38018428New FiMfic GPT model >>38308297 >>38347556 >>38301248FimFic dataset release >>38391839Offline GPT-PNY >>38821349FiMfic dataset >>38934474SD weights >>38959367SD low vram >>38959447Huggingface SD: >>38979677Colab SD >>38981735NSFW Pony Model >>39114433New DeltaVox >>39678806so-vits-svt 4.0 >>39683876so-vits-svt tutorial >>39692758Hay Say >>39920556Haysay on the web! >>40391443SFX seperator >>40786997 >>40790270Synthbot updates GDrive >>41019588Private "MareLoid" project >>40925332 >>40928583 >>40932952VoiceCraft >>40938470 >>40953388Fimfarch dataset >>410279715 years of PPP >>41029227Audio re-up >>41100938RVC Experiments >>41244976 >>41244980Ace Studio Demo >>41256049 >>41256783>The PoneAI drive, an archive for AI pony voice content:drive.google.com/drive/folders/1E21zJQWC5XVQWy2mt42bUiJ_XbqTJXCp>Clipper’s Master Files, the central location for MLP voice data:mega.nz/folder/jkwimSTa#_xk0VnR30C8Ljsy4RCGSigmega.nz/folder/gVYUEZrI#6dQHH3P2cFYWm3UkQveHxQdrive.google.com/drive/folders/1MuM9Nb_LwnVxInIPFNvzD_hv3zOZhpwx>Cool, where is the discord/forum/whatever unifying place for this project?You're looking at it.Last Thread:>>41137243
FAQs:If your question isn’t listed here, take a look in the quick start guide and main doc to see if it’s already answered there. Use the tabs on the left for easy navigation.Quick: docs.google.com/document/d/1PDkSrKKiHzzpUTKzBldZeKngvjeBUjyTtGCOv2GWwa0/editMain: docs.google.com/document/d/1y1pfS0LCrwbbvxdn3ZksH25BKaf0LaO13uYppxIQnac/edit>Where can I find the AI text-to-speech tools and how do I use them?A list of TTS tools: docs.google.com/document/d/1y1pfS0LCrwbbvxdn3ZksH25BKaf0LaO13uYppxIQnac/edit#heading=h.yuhl8zjiwmwqHow to get the best out of them: docs.google.com/document/d/1y1pfS0LCrwbbvxdn3ZksH25BKaf0LaO13uYppxIQnac/edit#heading=h.mnnpknmj1hcy>Where can I find content made with the voice AI?In the PoneAI drive: drive.google.com/drive/folders/1E21zJQWC5XVQWy2mt42bUiJ_XbqTJXCpAnd the PPP Mega Compilation: docs.google.com/spreadsheets/d/1T2TE3OBs681Vphfas7Jgi5rvugdH6wnXVtUVYiZyJF8/edit>I want to know more about the PPP, but I can’t be arsed to read the doc.See the live PPP panel shows presented on /mlp/con for a more condensed overview.2020 pony.tube/w/5fUkuT3245pL8ZoWXUnXJ42021 pony.tube/w/a5yfTV4Ynq7tRveZH7AA8f2022 pony.tube/w/mV3xgbdtrXqjoPAwEXZCw52023 pony.tube/w/fVZShksjBbu6uT51DtvWWz>How can I help with the PPP?Build datasets, train AIs, and use the AI to make more pony content. Take a look at the quick start guide for current active tasks, or start your own in the thread if you have an idea. There’s always more data to collect and more AIs to train.>Did you know that such and such voiced this other thing that could be used for voice data?It is best to keep to official audio only unless there is very little of it available. If you know of a good source of audio for characters with few (or just fewer) lines, please post it in the thread. 5.1 is generally required unless you have a source already clean of background noise. Preferably post a sample or link. The easier you make it, the more likely it will be done.>What about fan-imitations of official voices?No.>Will you guys be doing a [insert language here] version of the AI?Probably not, but you're welcome to. You can however get most of the way there by using phonetic transcriptions of other languages as input for the AI.>What about [insert OC here]'s voice?It is often quite difficult to find good quality audio data for OCs. If you happen to know any, post them in the thread and we’ll take a look.>I have an idea!Great. Post it in the thread and we'll discuss it.>Do you have a Code of Conduct?Of course: 15.ai/code>Is this project open source? Who is in charge of this?pony.tube/w/mqJyvdgrpbWgZduz2cs1CmPPP Redubs:pony.tube/w/p/aR2dpAFn5KhnqPYiRxFQ97Stream Premieres:pony.tube/w/6cKnjJEZSCi3gsvrbATXnCpony.tube/w/oNeBFMPiQKh93ePqTz1ns8
>>41284357Anchor.
HEY, FIFTEEN.Can you add S.h.o.d.a.n. please?
>>41284576I thought 15 died of ligma
>>41284645Who is Steve Jobs?
>>41284645What's 15?
>>41285049A ghost at this point.
There are way to install hay_say_ui with conda instead of docker?
>>41284357I made a cover of Lemon Demon's I've Got Some Falling to Do using so-vits. My friendanon made the ponified lyrics and I sang the vocals. I also tried animating the lyrics.https://youtu.be/atLuQp66BJI
I will be upgrading my GPU soon, but have been completely out of the AI scene as of late. Is NVIDIA still the only option if I want to run anything AI locally? Gaming will be it's primary function but I still want to be able to generate a bit of content as well.Both are modest gpus, same price. I'm deciding between a 8gb rx 6650xt and a 12gb RTX 3060. The amd gpu performs 10% better on games on average from the benchmarks I saw. Any advice?
>>41286002While there is better support for AMD nowadays but those 4 extra gb are valuable. the more memory you can get the more you have access to in terms of local AI models.
>>41286002Depends on how much (if any) AI you plan on doing, but AMD support (ROCm) is unfortunately still a bit of a meme and for now still limited to just a handful of top-end AMD cards (it will allow you to try on others, but things will break). Maybe it'll get better "soon", but people have been saying that for the last 2 years or so. Also, last I checked, AMD AI support was completely non-functional on windows if that's the OS you use. It's not super relevant since you can double-boot and Linux is lightweight, but worth mentioning the extra hassle. So, to go back to the question,>Is NVIDIA still the only option if I want to run anything AI locally?In short, yeah.I built my PC just a couple months ago, and I specifically opted for the 16 GB version of 4060 Ti (yes, the meme card) to be able to do textgen at decent speeds. You don't need 16GB if you're not interested in text—and the newest llama 3.1 can even run pretty fine on 12 GB—but going to 8 GB *and* AMD will hurt a basically any AI.
>>41284357>EQG and G5 are not welcome.Can I ask for ponified EqG? They're the best.
>>41284576Seconding.
>>41286335>but going to 8 GB *and* AMD will hurt a basically any AI.as an owner of 8GB nvidia card (due to poorfagging) I have to agree, half the shit people talk about in the ai image/text threads straight up do not load for me due to lack of raw gpu power.
>>41286002If you want to use AMD GPU, you have to Linux.
>>41285481Short answer: Sorry, but no.Long answer, if you're feeling adventurous: It may be technically feasible, but would take a lot of work. The Hay Say UI code communicates with the other Hay Say components (which live in their own Docker containers) by making local webservice calls to them. To eliminate Hay Say's dependence on Docker, you would have to install each component individually (by mimicking what its Dockerfile does) and then edit your hosts file to map all the service names to localhost so that the web service communication would still work between them. Alternatively, you could install each component in its own virtual machine and set up networking between them but the performance of that would probably be terrible.
>10
>>41267685Thanks for trying it. I'm guessing it didn't work because "speaker" isn't represented linearly in HuBERT. I took 50 clips from Twilight and bdl (male from CMU ARCTIC), put them through HuBERT, and computed the PCA for each layer's output. The bottom of pic related shows a complete overlap in the first 2 PCs for all layers, unlike the linear separation seen in the top graph (from https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction).This makes sense, since HuBERT is trained to discard speaker information. If it didn't, then retrieval probably wouldn't work at all.The next idea I have is to use MiMiC from https://arxiv.org/abs/2402.09631, which matches both mean and covariance. This apparently reduces "bias by neighbors", which might improve retrieval.If that fails, then I'm not sure. Maybe the artifacts aren't from speaker info leakage, but something else?For example, unless I'm reading the code wrong, it seems like RVC is trained on unmodified HuBERT features. This could cause a mismatch since inference uses retrieved features, which usually come from completely different clips. You could try to fix this by passing features through the index first, so you're training on the nearest neighbor of each feature vector.I also wonder how robust retrieval is to time misalignments. e.g. shifting an audio clip by 10ms would change the features slightly, which could cause retrieval to return different results. Maybe this messes up the model in inference.
Is it just me or the option for generating StyleTTS2 on haysay is disabled for anybody else?
>>41288265It's working for me. Could you provide more details? Like, is the StyleTTS tab missing for you, or are buttons greyed out or does it just hang when generating or something else? Also, are you using haysay.ai or a local installation? If local, what OS do you use?
>>41288375the button is just gray out and I cant press it to generate audio.
>>41288450Did you type anything into the Input section at the top of the page? StyleTTS requires text input, so the Generate button will be greyed out until you provide some text.
>>41288554oh, the styletts2 do not work with audio references than?
>youtu.be/eGe4Do1mkFE>https://files.catbox.moe/mn9sdn.mp3song for Rarity and jazz enjoyers
>>41288972StyleTTS can take an audio reference in addition to text, but unlike so-vits or RVC, the reference is used for setting the "style" and prosody of the output rather than doing a voice conversion. For best results, you'll want to use a clip of the target character's voice (not your own voice) that sounds similar (e.g. emotion, pace, etc) to how you want the line delivered.
is 15.ai all of a sudden gonna comeback ?
>>41291330Anything that makes you say that?
>>41291330I seriously doubt it, but I wish he would at least open source this shit Even if you needed like a supercomputer to run it, it would be better than paying 11 ai
https://vocaroo.com/15DZbXqTkx4ghttps://pomf2.lain.la/f/41cwwxss.mp4https://www.udio.com/songs/fgAs8B89VVZ6erafA5jY2SUdio-130A Serenade for the Garden[Verse 1]Golden sun in the morningWishes in the breezeDreams scattered like leaves (dreams scattered like leaves)Royal, sereneBlossoms dancing, floating freeSeeking a tender plea[Chorus]Princess Celestia, don't pee hereIn this sacred place, pretty pleaseKeep it pure with your graceIn these gardens, in these trees[Verse 2]Luminous skies at twilightWhispers from the starsHarmony, we are (harmony, we are)Regal, so brightTwilight sparkling in your eyesFaith in this calm night[Chorus]Princess Celestia, don't pee hereIn this sacred place, pretty pleaseKeep it pure with your graceIn these gardens, in these trees[Bridge]Majesty, cascading lightThrough the ages, through the nightLet it be, clear and brightHarmony in our sight[Chorus]Princess Celestia, don't pee hereIn this sacred place, pretty pleaseKeep it pure with your graceIn these gardens, in these trees
>>41291330Even if that's the case, he has the poor track record of "I'm updating so nobody could use it for another 6 months."
>MLPRegressorWhich one of you fags is responsible for this?
>>41292298I accidentally deleted it from Udio, because when trying to move it to a different playlist, I selected songs from the recently created songs and clicked Delete. If I selected them in a playlist, the button would have been "Remove Songs from Playlist".So here is a couple of songs I saved.Prompt: a song by Pinkie Pie, who is for some reason asking Princess Celestia not to pee in the Sugarcube Corner! Of course, Celestia wasn't going to do that, but you know Pinkie.https://pomf2.lain.la/f/pp0tz2kf.mp4https://voca.ro/1gfw6B89FeLUA Sweet Plea[Verse 1]Princess CelestiaPlease hear my pleaSugarcube CornerNeeds to be pee-freeJust for a secondThink about the piesTwilight’s here tooWe're aiming for the skies[Chorus]Please don’t peeHere in the shopLet's keep the sprinklesAnd the fun non-stopPlease don’t peeHere in the shopLet's keep the sprinklesAnd the fun non-stop[Verse 2]Cupcakes and muffinsNeed to stay cleanCookies and browniesAnd all treats betweenJust for our sakesLet's play, not weep We'll laugh foreverWith our memories to keep[Chorus]Please don’t peeHere in the shopLet's keep the sprinklesAnd the fun non-stopPlease don’t peeHere in the shopLet's keep the sprinklesAnd the fun non-stop[Bridge]Think of the laughterEveryone will shareWe can have funWithout any scare[Chorus]Please don’t peeHere in the shopLet's keep the sprinklesAnd the fun non-stophttps://voca.ro/1iYgDeFxIHJzA Sweet Request[Verse 1]Oh Celestia, what a sunny day,But there's one tiny thing I need to say!Please, don't pee in Sugarcube Corner,I know you wouldn't, but I gotta warn ya!It's a place for cupcakes, not a place for piddles,Save that for the loo, in the castle riddles![Chorus]Oh Princess dear, keep it clear,The Corner's for sweets, not what you excrete!For cupcakes and treats, come inside,Just keep your potty breaks outside![Verse 2]All the ponies gather, and we have such fun,But pee on the floor? No, that's not done!Sprinkles and frosting, that's all we need,Keep it sugar-sweet, that's Pinkie's creed!Let's keep it tidy, no puddles here,Just laughter and joy, and lots of cheer![Chorus][Bridge]Celestia, I know you're grand,With a kingdom at your command,Let's keep our Corner pristine clean,For everypony, king or queen![Chorus]
And I know somepony would ask about this, so here is a different request. It seems like "pee" stands for "be" in most cases here.https://voca.ro/13V22n5a7DPkA Plea for Light (to Princess Celestia)male vocalist, electronic, pop, indie pop, synthpop, electropop, bittersweetLyrics[Verse 1]Dear Princess Celestia, hear my pleaWay up in Canterlot, beautiful and freeI need your magic, your golden lightTo chase away the shadows, deep in the night[Chorus]Dear Princess CelestiaWon't you come near?Dear Princess CelestiaPlease, please pee here[Verse 2]It's an odd request, I know it sounds strangeBut your grace and power, can help me changeI feel so broken, lost in the darkI need your warmth, to light a spark[Chorus]Dear Princess CelestiaWon't you come near?Dear Princess CelestiaPlease, please pee here[Bridge]Just one touch of divine lightCould make everything alright[Chorus]Dear Princess CelestiaWon't you come near?Dear Princess CelestiaPlease, please pee here
>>41291330Something, something, flying pigs.
>>41294609We need some more Irish / Saxon style poetry going on with ponies .
>>41287953 (1/2)>I'm guessing it didn't work because "speaker" isn't represented linearly in HuBERTMost likely>This makes sense, since HuBERT is trained to discard speaker informationA bit of confusion: apparently even though they called their checkpoint "hubert_base" it's actually ContentVec:https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/2078AFAIK ContentVec does try to do things like pitch and formant shifting to try to disentangle "content" from speaker identity, but even with transformations like that, you'd still be able to tell from accents, for instance, whether something was said by Applejack vs. Rarity. And on top of that I'm still not sure moving each individual feature closer to the nearest features the model saw in training is necessarily the "correct" mapping to go from one voice to another. Using phonemes as an analogy, there's no inherent reason to believe that my AE is closest to Rarity's AE vs. any other phoneme in feature space (by the fact that retrieval does seem to work somewhat, they probably are "close" but the mapping is not perfect). Plus accents introduce a lot of time-dependent variations that probably wouldn't be fixed by retrieval much, if at all.I think the RVC maintainers might be getting at the same issue with RVCv3--they say something about being limited by ContentVec here:https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/2013>MiMiCIt looks like they're just using stuff from an existing library (ot.da.LinearTransform) to do this? I fit it on the input features vs. concatenated features from 50 samples of the target speaker. Doesn't really seem to do much but introduce a few small artifacts:>No MiMiC, index rate 0.75https://files.catbox.moe/swdkjw.mp3>MiMiC, index rate 0.75https://files.catbox.moe/o0r03m.mp3>Maybe it's something else?Possible, but it would also have to account for StyleTTS2 generated inputs for the same character showing better output?https://files.catbox.moe/vd9op0.mp3Me doing my best attempt to match: https://files.catbox.moe/k53i0b.mp3Another styletts2 gen: https://files.catbox.moe/iqwa5o.mp3I also wonder if pitch contours have something to do with it--I feel like pitch detection is a bit dodgier for lower pitches (lower resolution).
>>41287953>>41296158 (2/2)>Retrieval mismatchThat is a good point--I hadn't thought about trying to fix the problem from that end. It might be asking the network and finetuning process to do some heavy lifting though? (Might also be why sibilants get a lot buzzier and there are more clicky artifacts when you use retrieval: the network is seeing features in a sequence it hasn't learned to interpret?)>I also wonder how robust retrieval is to time misalignments. e.g. shifting an audio clip by 10ms would change the features slightly, which could cause retrieval to return different results. Maybe this messes up the model in inference.Tested this. It can definitely introduce some small crackly artifacts, although they're not exactly the kind of artifacts we're looking at.Index rate of 1.0 used for all samples.0ms offset: https://files.catbox.moe/3pcjhu.mp350ms offset: https://files.catbox.moe/qwi1fc.mp3110ms offset: https://files.catbox.moe/yx8cia.mp3Might be something worth putting into the training end if we decide to go that route?>Other thoughtsWith TalkNet we could be a lot less dependent on the input voice. I almost wonder if it makes sense to fall back to a TalkNet-like system (without the hackiness and resource-intensiveness of just feeding the audio output of one into the other). Maybe you could get around the whole "needing a transcription" problem by training something on ASR decoder logits (or some intermediate feature in an ASR decoder)?I'm starting to wonder if the apparent better quality of RVCv2 over so-vits-svc 4 is even due to any architecture and implementation differences vs. just having a more robust pretrained model; the two don't seem to be that different.so-vits-svc 5.0 seems to be a tiny bit better on this front; I'm guessing it's either because A) they add noise to the features during training, and/or B) Whisper encoder features might be slightly more robust against speaker identity since they're explicitly trained on ASR.>Codehttps://github.com/effusiveperiscope/raraai/blob/master/2_featureexpl/add2.ipynb
>>41296161Also, I finally got around to finishing "transcribing" the leak material. (I used automated processes to do as much work as possible and just did a pass of transcription corrections, so there's some questionable clipping points; also, the timestamps are synthetic).https://drive.google.com/drive/folders/1XXO8o7O8m-kJGZvkYx3U1eheo2s902N5?usp=sharingSeparated into "Broadcast Stems" and "Studio Recordings" so it's easier to tell which are unprocessed recordings.
