FAQs:
If your question isn’t listed here, take a look in the quick start guide and main doc to see if it’s already answered there. Use the tabs on the left for easy navigation.
Quick: docs.google.com/document/d/1PDkSrKKiHzzpUTKzBldZeKngvjeBUjyTtGCOv2GWwa0/edit
Main: docs.google.com/document/d/1y1pfS0LCrwbbvxdn3ZksH25BKaf0LaO13uYppxIQnac/edit

>Where can I find the AI text-to-speech tools and how do I use them?
A list of TTS tools: docs.google.com/document/d/1y1pfS0LCrwbbvxdn3ZksH25BKaf0LaO13uYppxIQnac/edit#heading=h.yuhl8zjiwmwq
How to get the best out of them: docs.google.com/document/d/1y1pfS0LCrwbbvxdn3ZksH25BKaf0LaO13uYppxIQnac/edit#heading=h.mnnpknmj1hcy

>Where can I find content made with the voice AI?
In the PoneAI drive: drive.google.com/drive/folders/1E21zJQWC5XVQWy2mt42bUiJ_XbqTJXCp
And the PPP Mega Compilation: docs.google.com/spreadsheets/d/1T2TE3OBs681Vphfas7Jgi5rvugdH6wnXVtUVYiZyJF8/edit

>I want to know more about the PPP, but I can’t be arsed to read the doc.
See the live PPP panel shows presented on /mlp/con for a more condensed overview.
2020 pony.tube/w/5fUkuT3245pL8ZoWXUnXJ4
2021 pony.tube/w/a5yfTV4Ynq7tRveZH7AA8f
2022 pony.tube/w/mV3xgbdtrXqjoPAwEXZCw5
2023 pony.tube/w/fVZShksjBbu6uT51DtvWWz

>How can I help with the PPP?
Build datasets, train AIs, and use the AI to make more pony content. Take a look at the quick start guide for current active tasks, or start your own in the thread if you have an idea. There’s always more data to collect and more AIs to train.

>Did you know that such and such voiced this other thing that could be used for voice data?
It is best to keep to official audio only unless there is very little of it available. If you know of a good source of audio for characters with few (or just fewer) lines, please post it in the thread. 5.1 is generally required unless you have a source already clean of background noise. Preferably post a sample or link. The easier you make it, the more likely it will be done.

>What about fan-imitations of official voices?
No.

>Will you guys be doing a [insert language here] version of the AI?
Probably not, but you're welcome to. You can however get most of the way there by using phonetic transcriptions of other languages as input for the AI.

>What about [insert OC here]'s voice?
It is often quite difficult to find good quality audio data for OCs. If you happen to know any, post them in the thread and we’ll take a look.

>I have an idea!
Great. Post it in the thread and we'll discuss it.

>Do you have a Code of Conduct?
Of course: 15.ai/code

>Is this project open source? Who is in charge of this?
pony.tube/w/mqJyvdgrpbWgZduz2cs1Cm

PPP Redubs:
pony.tube/w/p/aR2dpAFn5KhnqPYiRxFQ97

Stream Premieres:
pony.tube/w/6cKnjJEZSCi3gsvrbATXnC
pony.tube/w/oNeBFMPiQKh93ePqTz1ns8

File: anchor.png (33 KB, 1200x1453)

33 KB PNG

Anchor

Is Clipper still doing episodes? I loved that "Free Hugs" one.

>>41364866
God I hope not. It sucks.

File: 1701781466764247.png (60 KB, 500x459)

60 KB PNG

>>41364875
>Wanting everything to be about sex with Anon

>>41364876
Good idea

File: 670652.png (670 KB, 3991x5761)

670 KB PNG

>>41364787
Added audio from recently released animatics (s2e3, 25, 26) to the voice dataset, replacing corresponding entries in the FiM folder.
mega.nz/folder/jkwimSTa#_xk0VnR30C8Ljsy4RCGSig
Sliced Dialogue -> Special source
Also put label files in the the label files folder.

>>41364866
Working on a thing to present at Mare Fair. Not an episode as such, though will make use of the AI voice.

>>41364875
There are a lot of things I'd do differently looking back, mainly pacing. Always better next time, that's the goal.

>>41364894
>Sex Hotline 2.0

>>41364787
reposting song cover in case people missed it from the last thread >>41361841
>https://files.catbox.moe/qxa6vp.mp3

File: FlutterbotExperimentation.png (248 KB, 751x477)

248 KB PNG

Found a good VITS-based voice conversion/generation tool with decent TTS capabilities called Applio.
https://www.youtube.com/watch?v=gjggpadBgOo
https://github.com/IAHispano/Applio
https://applio.org/

It seems to have a lot of functionality similar to Hay Say, but with more in-depth TTS. The way it functions is effectively has a TTS voice layer of various voices; speakers of numerous countries, languages and accents that it uses to interpose RVC voices onto. Which works pretty well in my early testings in the compiled clip below.

https://files.catbox.moe/lju6ub.mp4

Full compatible with existing pony models, as evidenced by it working with Vul's Fluttershy S1 model. Though it was a bit confusing where the models had to go (Apparently in "/Applio-3.2.4/logs/" in a named folder containing the .pth and .index files). It's a little finicky, needing experimentation with suitable TTS voices and settings to optimize for each mare, adjusting until it sounds right. The noisiness is mostly from the TTS end of things and less so the RVC side. Additional RVC training and inference functions are useful too, hadn't tested those parts yet though.

>>41365138
Those clips sound pretty good, some low level noise but all within what I'd call an acceptable limit.
Might this be a new start for pony TTS? Would be awesome to have an alternative to those that rely fully on reference audio.

>>41364894
- [In progress] Download the Master File again so I can a clean updated copy.
- [ ] Reupload a clone of the new Master File to my gdrive.
- [ ] Reupload a clone of both Master Files to HuggingFace.

Separately, I've spent a lot of time in the last few months working with LLMs. I'm putting together a library for creating more complex chatbots.
https://github.com/synthbot-anon/horsona
- [In progress] Collect a list of functionality that would help in making better chatbots. Right now, that means going through existing chatbots and figuring out (1) what features they support, (2) what it takes to specify a new chatbot, and (3) what people complain about regarding chatbot interactions and personalities.
- [ ] Split the target features into individual functions that can be implemented and pieced together, and create a github issue for each one so it's easy to keep track of everything.
- [ ] Start implementing.

If anyone has ideas for functionality and for things other chatbots do well/poorly, let me know. Right now, I don't care how difficult anything here is to implement.

>>41364782
Is there a voice actor AI model that can simulate rage and emotions like AVGN and doesn't take a rocket degree science to utilize it? I plan on using it for really long ranting reviews of 10k+ words.

>>41365910
No, not right now.

File: Gemini_Generated_Image_jh(...).jpg (316 KB, 1536x1536)

316 KB JPG

>>41365609
>Chatbots
>Horsona
Would there be any likelihood of these being capable of writing entire fics independently like GPT-PNY does/did? Curious too about there flexibility as it could be fun to have AI mares give us a script to then animate or otherwise try and achieve. Mare assisted brainstorming.

>>41365950
I think so. I'm toying around with having an LLM read a fanfic paragraph-by-paragraph to extract information for automated lorebook creation. Writing a fic would be basically that in reverse + lorebook generation.
Once I get a better handle on how to implement this and everything /chag/ suggested, I'll try to break down the tasks into small pieces that can be implemented by more people than just myself.

>>41365138
Example is little bit too high pitched but it does show an idea. I will test it out later myself.

>>41366029
are you planing to make it as heavy modded TavernAI or some custom ui program from scratch?

>>41366105
There is a pitch slider in the settings, so it's pretty much a non-issue; can be adjusted as desired. This will be a setting played around with often as differing TTS voices each vary in natural pitch range. Some deliveries might also need a small increase or reduction too.

Lower pitches did feel more Fluttershy, but didn't seem to have as much pitch variance. She kinda sounded bored or tired to me.

>>41366302
I don't know. Probably a mix of both, leaning toward integrating with other UIs as much as possible. Right now, I mostly want to see how limiting the technical challenges really are when trying to make perfect chatbots. I intend for it to be a library, not a full chat program, but a UI might be necessary occasionally to make use of & test the functionality.

>>41366401
I still have a fondness of for how the barebones GPT-PNY worked way back when, with the colab and separate window thing. I still feel it functioned a lot better and more freely than in KoboldAI, so any simple interfacing you come up with that'd allow for more raw/unfiltered/free-form outputs is good by me, even if other more flexible and potentially restrictive interfaces are adopted for it later on too.

>>41366416
>https://www.youtube.com/watch?v=jHS1RJREG2Q
>https://arxiv.org/abs/2408.14837
>Diffusion Models Are Real-Time Game Engines
So same nerds combined a llm text model with art diffusion model and trained it on images + keyboard inputs to create a synthesized Doom gameplay.
Not mare related but the idea of practical combination of different ai tools is interesting to me.

https://www.udio.com/songs/67X7mqHih4C8m4raEX8fzW
https://pomf2.lain.la/f/tky4cms5.mp4

Midnight Rejections
acoustic guitar music. princess celestia, anon, male vocalist, sad

Lyrics

[Verse 1]
Celestia's trying hard
She’s got her royal charm turned up to ten
But Anon's still not interested again, yeah
She pulled out all her tricks
Even baked him a cake, extra thick
But buddy's not biting, not even one little bit

[Chorus]
In the castle, at midnight, room 302
With a bouquet of roses and some candles too
Anon’s locked the door, put a sign in his view
That says "Please go away"

[Verse 2]
She's got a plan, who knew?
But Luna and Cadance can't believe it’s true
She wore a fancy dress and said “Hey there you!”
She read from romance books
Tried adding sexy looks
But Anon just laughed and moved to his favorite nooks

[Chorus]
In the castle, at midnight, room 302
With a bouquet of roses and some candles too
Anon’s locked the door, put a sign in his view
That says "Please go away"

[Bridge]
So Celestia sighed, wiped a tear from her eye
The mares all gathered round, gave it one more try
They played their guitars, singing under moon light
But Anon just yawned and said, "Goodnight"

[Chorus]
In the castle, at midnight, room 302
With a bouquet of roses and some candles too
Anon’s locked the door, put a sign in his view
That says "Please go away"


----
Is "put a sign in his view" too bad?

>>41367097
you have quite a nice collection there, the 'Dreams of Luna' is pretty nice.

Do any of you guys reckon you could make a good voiced version of the second comic here?

https://www.tumblr.com/radioactive-dragonlover/759831654724419584

I tried using Haysay and putting in a segment of the audio from the episode of Game Changers it's referencing as an audio input, but it came out sounding very robotic and off - I think because Twilight as a character has a different range of pitches when speaking emotionally than BLeeM does. I don't really know how to adjust that, though.

>>41366416
Do you mean how it wasn't tuned to act like an assistant, and that it just continued from whatever text you gave it? That should be easy enough.
>>llm = AsyncCerebrasEngine(model="llama3.1-70b")
>>print(await llm.query_continuation("Once upon a time there was a little pony named Twilight Sparkle."))
>She lived in a magical land known as Equestria, where the sun was always shining and the air was sweet with the scent of blooming wildflowers. Twilight Sparkle was a student of Princess Celestia, the ruler of Equestria, and was learning the art of magic at the princess's palace in Canterlot. One day, while Twilight was studying in the library, she received a letter from the princess, instructing her to move to the town of Ponyville and live alongside the other ponies, to learn about the magic of friendship.

I can make sure there's a way to support jailbreaks too.

>>41366336
Welp, that new program is not going to be very useful to me as it crashes at the beginning with inability to load some dll models. So the struggle to find nice sounding tts is still going.

>>41367854
Afraid I can't be much help with that, assuming you're using the windows version; Linux version works fine. If it's an installation issue, in the releases of the GitHub there should be another install option there, maybe that'll work?
>dll models
I'm pretty sure it's intended to use .pth files as the models. Also only RVC, I had a slip up earlier where I accidentally tried to load a SoVits model if mine and it naturally errored.

>>41365609
Current ai might be too slow for this but I was thinking about some kind of LLM/RAG powered "RPG engine" where you don't just provide character definitions but world definitions, item definitions, maybe have some kind of prebuilt framework for quests, skills, other user defined mechanics, do the math of tracking XP, health, armor, damage multipliers in code rather than having the LLM try to pick that role up. Rather than the hackiness of trying to shove a world scenario or an explicit story into each character card these could be split into more logical pieces and composed into RPGs

>>41368170
Someone mentioned this in /chag/. The hard part is make sure it's possible extract & track the relevant information from a rulebook. If you can send me an example rulebook (maybe one of the PonyFinder ones), that would help.

>>41367684
Can you catbox the Game Changers audio, or link to the original episode?

>>41368005
they do have a precompiled download but its giving me the same error, im guessing it just my system being extra derped.

File: tumblr_d39fca40a770bbf5ed(...).jpg (617 KB, 2048x1326)

617 KB JPG

>>41368620
https://youtu.be/88et7YlmzTs?si=_okFx5HtSE9e9cBV
Here's the audio snippet I used:
https://files.catbox.moe/m5eph6.mp3

File: a-vibrant-acrylic-paintin(...).png (1.93 MB, 1024x1024)

1.93 MB PNG

>9

>>41369482
So it is.

>>41369021
https://files.catbox.moe/rf3zrc.mp3

Architecture = rvc | Character = Twilight Sparkle | Index Ratio = 0.95 | Pitch Shift = 8 | Voice Envelope Mix Ratio = 1.0 | Voiceless Consonants Protection Ratio = 0.33 | f0 Extraction Method = rmvpe

>>41369482
>cutie mark
Is that what it looks like to have 4chan as a special talent?

>>41369482
oy

File: a-3d-render-of-pony-twili(...).png (1.2 MB, 1248x832)

1.2 MB PNG

>>41371447
Either that or her talent is related to some kind of Star Trek: Green Edition.

>>41372139
Kek, that uniform design is gold.

>>41365609
Updating the Master File:
- [Hopefully done] Download the Master File again so I can a clean updated copy. Mega was having issues, as usual. I'll need to check to make sure I have all the files, but I think this is done.
- [In progress] Reupload a clone of both Master Files to HuggingFace.
- [In progress] Reupload a clone of the new Master File to my gdrive.

Horsona chatbot library:
- [Done] Collect a list of functionality that would help in making better chatbots. The currentl list is up on github readme https://github.com/synthbot-anon/horsona. I have enough to get started, but please keep suggesting functionality if you think of anything. There's still functionality I want that no one's mentioned, so I'm sure the list is incomplete.
- [In progress] Split the target features into individual functions that can be implemented and pieced together, and create a github issue for each one so it's easy to keep track of everything. I'll need to start implementing some of these things so I can have a better understanding of how do this.
- ... [Done] Create a sample memory module, which is required for several of the candidate features. I went with "RAG where the underlying dataset can be automatically updated based on new information." The implementation is done, though the LLM prompts in https://github.com/synthbot-anon/horsona/blob/main/src/horsona/memory/rag.py#L139 could use some work. There's an example use in https://github.com/synthbot-anon/horsona/blob/main/tests/test_rag.py though the test only passes about 30% of the time. I'm pretty sure this can be made close to 100% with better prompts.
- ... [ ] Add documentation, a "start developing" guide, and tasks for the features where it's feasible to make progress using the memory module.
- ... [ ] Find some old & current jailbreaks to add jailbreak support, and turn them into either modules or LLM wrappers. If anyone has links for this, please send them.
- ... [ ] Figure out how to organize text corpora into compatible universes.
- ... [ ] Go through the candidate feature list and make sure jailbreaks & compatible universes are the only feature that are hard to support with the existing framework.

Information I need:
- Sample rulebooks, preferably one of the PonyFinder ones, so I can figure out what it'll take to extract information from these.
- Old & current jailbreaks so I can make sure my jailbreak implementation is comprehensive.

>>41373093
>keep suggesting functionality if you think of anything
I can't think of any at this moment, however I would love if you were able to keep the addons options that auto1111 webui for Stable Diffusion has, where one can just install whatever additional options as needed and as how anons make new ones in the future.

Page 10 bump.

Does Clipper know where he found the MLP background music? Shit's kino as fuck.

>>41373305
I'm only building the library right now (not a full application), but I can make sure it can support custom add-ons that can be dynamically toggled.

>>41373093
It looks like jailbreaks sometimes are LLM-specific and require modifying near-arbitrary arguments to the call. So they'll likely be implemented as custom LLMEngines. In that cases, I don't think I need to do anything for them right now since my LLMEngine implementation already supports all of the customizations required. I'll just create issues for popular jailbreaks that I or others can implement.
Organizing text into compatible universes looks like it'll require graphs of data sources, where one data source can inherent from another with edits. I'll have to think about how to implement this. Most of the features don't depend on this, so I'll shift focus to documenting & creating issues for now.

>>41373902
I know about these archived rips of background music:
https://www.mediafire.com/?rdhhrpyc0d6d3
https://www.mediafire.com/?rh219xdgj66bu

The first directory has music from seasons 1-2, and the second account belongs to RainShadow, who also has a YouTube channel:
https://www.youtube.com/@RainShadow

>>41373093
I think horsona / chatbot library is in a good-enough state for anyone that wants to help out with development.
Repo: https://github.com/synthbot-anon/horsona
Open tasks: https://github.com/synthbot-anon/horsona/issues
- The current open tasks are for creating new LLMEngines (easy), making prompts more reliable (medium), and creating new datatypes & modules for character cards and image generation (medium/hard, probably requires some familiarity with pytorch).
- If you want to develop and run into any issues with the setup, let me know.

The integrations with existing chatbot UIs will come a bit later from integrations with common inference engines. I don't expect that to be difficult.

Updating the Master File:
- [Hopefully done] Download the Master File again so I can a clean updated copy. I'll need to check to make sure I have all the files, but I think this is done.
- [In progress] Reupload a clone of both Master Files to HuggingFace.
- [In progress] Reupload a clone of the new Master File to my gdrive.

Horsona chatbot library:
- [Done] Add documentation, a "start developing" guide, and tasks for the features where it's feasible to make progress using the memory module.
- [Done] Find some old & current jailbreaks to add jailbreak support, and turn them into either modules or LLM wrappers. Ultimate Jailbreak is the main one, and there's an open task for it. There are others listed on rentry listed here: https://rentry.org/jb-listing.
- [Done enough] Go through the candidate feature list and make sure jailbreaks & compatible universes are the only features that are hard to support with the existing framework. Jailbreaks are easy to support. The rest of the features are easy to support.
- [ ] Work on whatever open issues other anons don't pick up.
- [ ] Continue working on lorebook generation. After this, I'll try making a simple chatbot with the library.
- [ ] Figure out how to organize text corpora into compatible universes.

>>41373902
pretty sure 99% its rips from the show itself (with two/three pieces made by Anons) that you can find it OP post second mega NZ link ('sfx and music' folder).

>>41373888
Minus one.

>>41373902
Nothing special, I just took it all from the music tracks of the same show audio used to make the voice dataset. Same clipping process with a different tagging system.

>>41371103
That's pretty good up until the screaming at the end. Thank you

File: 3121428.png (170 KB, 1528x2267)

170 KB PNG

Any chance someone here could train up a Lightning Dust model for RVC please? I need it for a song and her SVC one isn't cutting it.

>>41377069
https://huggingface.co/Amo/so-vits-svc-4.0_GA/tree/main/ModelsFolder/ddm_DaringDo_100k
There is a sovits model for her that was set up as ulti model training, as if I remember correctly the model did not have the required 2 minutes of audio?
I may give it a try for RVC training in a day or two.

>>41374539
Updating the Master File:
- [Done] Downloaded the new Master File and checked to make sure everything is good.
- [In progress] Reupload a clone of both Master Files to HuggingFace. This should be done in about 1 hour.
- [In progress] Reupload a clone of the new Master File to my gdrive. This should be done in about 3 hours.

Updating the Fimfarchive:
- [In progress] Download & verify the Sep 1 Fimfarchive. I'm downloading it now. If there are no errors, this should be done in about 5 hours.
- [ ] Upload to HuggingFace.

>>41364894
There's an empty "Luster Dawn" folder in Special Source, in case you wanted to remove that.

File: _lightning dust (mlp), fe(...).png (410 KB, 640x640)

410 KB PNG

>>41377069
I've also been meaning to train a Lightning Dust model. I'll see about training her later today probably. Would be good to get me back into the rhythm of training; been a long while.

File: Untitled.png (553 KB, 1080x1049)

553 KB PNG

SelectTTS: Synthesizing Anyone's Voice via Discrete Unit-Based Frame Selection
https://arxiv.org/abs/2408.17432
>Synthesizing the voices of unseen speakers is a persisting challenge in multi-speaker text-to-speech (TTS). Most multi-speaker TTS models rely on modeling speaker characteristics through speaker conditioning during training. Modeling unseen speaker attributes through this approach has necessitated an increase in model complexity, which makes it challenging to reproduce results and improve upon them. We design a simple alternative to this. We propose SelectTTS, a novel method to select the appropriate frames from the target speaker and decode using frame-level self-supervised learning (SSL) features. We show that this approach can effectively capture speaker characteristics for unseen speakers, and achieves comparable results to other multi-speaker TTS frameworks in both objective and subjective metrics. With SelectTTS, we show that frame selection from the target speaker's speech is a direct way to achieve generalization in unseen speakers with low model complexity. We achieve better speaker similarity performance than SOTA baselines XTTS-v2 and VALL-E with over an 8x reduction in model parameters and a 270x reduction in training data
https://kodhandarama.github.io/selectTTSdemo/
code and weights to be released (soon?)
examples aren't great but considering the training time/training data/parameters means its viable for personal training. they used 100 hours of data

Hold Me Tight: Stable Encoder-Decoder Design for Speech Enhancement
https://arxiv.org/abs/2408.17358
>Convolutional layers with 1-D filters are often used as frontend to encode audio signals. Unlike fixed time-frequency representations, they can adapt to the local characteristics of input data. However, 1-D filters on raw audio are hard to train and often suffer from instabilities. In this paper, we address these problems with hybrid solutions, i.e., combining theory-driven and data-driven approaches. First, we preprocess the audio signals via a auditory filterbank, guaranteeing good frequency localization for the learned encoder. Second, we use results from frame theory to define an unsupervised learning objective that encourages energy conservation and perfect reconstruction. Third, we adapt mixed compressed spectral norms as learning objectives to the encoder coefficients. Using these solutions in a low-complexity encoder-mask-decoder model significantly improves the perceptual evaluation of speech quality (PESQ) in speech enhancement.
https://github.com/felixperfler/Stable-Hybrid-Auditory-Filterbanks

>>41377203
>>41377404
It would be much appreciated. SVS has a tone to it that's difficult to work into some genres, I think RVC would nail it.

Precautionary page 8 bump.

>>41378282
Plus one.

>>41377900
I may have something workable within 6 hours (or 12, depending if pc decides to have another technical hiccup).

File: Lightning Dust 1639574203891.png (508 KB, 1241x1235)

508 KB PNG

>>41377900
>>41379052
And Im back.
>https://huggingface.co/Amo/RVC_v2_GA/tree/main/models/MLP_Lightning_Dust_GA
>https://files.catbox.moe/amkbes.mp3
I may over train her with setting up the epoch to 500 but I still think this model came out pretty decent, specially with all the additional clean up data clips.

File: lightning dust (mlp), pon(...).png (447 KB, 640x640)

447 KB PNG

>>41380032
Nicely done. I attempted to clean some more files for training her yesterday but didn't have much luck with getting any of the tools to work before I got bummed out and needed sleep. Those tools being jazzpear94's model (https://colab.research.google.com/drive/1efoJFKeRNOulk6F4rKXkjg63RBUm0AnJ) and a couple other similar ones intended to help separate SFX specifically, which I found in this doc: https://docs.google.com/document/d/17fjNvJzj8ZGSer7c7OFe_CNfUKbAxEh_OBv94ZdRG5c/edit#heading=h.owqo9q2d774z

How much data did you have of her to work with? Still tempted to train another model of her anyways, if at least to see if my training setup works still.

Hey Hydrus, any chance we could get this >>41380032 on Haysay.ai?

>>41377233
Updating the Master File:
- [Done] Reupload a clone of both Master Files to HuggingFace. https://huggingface.co/datasets/synthbot/pony-speech and https://huggingface.co/datasets/synthbot/pony-singing
- [In progress] Reupload a clone of the new Master File to my gdrive. https://drive.google.com/drive/folders/1ho2qhjUTfKtYUXwDPArTmHuTJCaODQyQ

Updating the Fimfarchive:
- [Done] Download & verify the Sep 1 Fimfarchive. Fimfiction added some restriction to prevent bots from scraping. Part of my script downloads the story html if there's anything wrong with the fimfarchive epubs, or if there's a conflict between the fimfarchive metadata and epub. I have a hackish fix for this for now.
- [Done] Upload to HuggingFace. https://huggingface.co/datasets/synthbot/fimfarchive

Horsona chatbot library:
- [In progress] Continue working on lorebook generation. I cleaned up part of my embedding-based memory implementation. I'll clean the rest as I figure out the right way to use it for reading through a story. Right now, I'm having an LLM create a question-answer dataset for the story setting, which it refines as it reads the story. The questions get turned into embeddings, which can be used to look up the corresponding answers as necessary. This is still a work in progress. I think my first "test" for this would be if it can create a decent character card for every character after each chapter of a story. That's what I'm currently working toward.
- [ ] Work on whatever open issues other anons don't pick up.
- [ ] Figure out how to organize text corpora into compatible universes.

>>41380500
Correction: the Master File reupload to my gdrive should be [Done].

>>41380258
>https://files.catbox.moe/d3j7w5.zip
>I attempted to clean some more files
I usually just grab the Clear files from OP mega folder and if the number is below 3 minutes I will additionally scavenge any usable 'noisy' files as well, in this case I think I used almost all of the noisy ones with only three being deleted.

File: lightning dust (mlp), pon(...).png (372 KB, 640x640)

372 KB PNG

>>41380756
That's more or less what I have to train with, but I'm a bit stringent when it comes to samples I use. There's quite a few lines sourced from her second episode that I didn't feel suited as she kinda gets a slight country-like accent to it in her delivery (https://files.catbox.moe/gn0vd5.flac) and unusual or distorted with others (https://files.catbox.moe/1h5trr.flac & https://files.catbox.moe/4mug5h.flac). But yeah, I'll train her with what I've defined, though fewer files, and consider an alternate version that'll hopefully be more faithful to her debut version.

Up.

>>41381049
Oh yes, when going through the dataset I will try to look out for the tone of the voice as well, while I dont have proof I think the clips were "X character pretends to talk like Y" may end up poisoning the training process.
Btw Anon, what stuff are you planing to be making with her voice, songs? green text?

>>41381590
I usually make covers, but I'll be doing further tests with Applio and which combinations of TTS voices it has pairs best and the settings I feel works best. Might have to compile some sort of parameters list. Also curious about the TTS >>41377494 mentioned and how well that'll perform in comparison. I have a few factors limiting my ability to effectively perform non-song stuff, but I have a long list of stuff to produce. Nothing so much with Lightning Dust though thus far, aside from a few songs I wanna test with her.

Good night bump.

MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
https://arxiv.org/abs/2409.00750
>Nowadays, large-scale text-to-speech (TTS) systems are primarily divided into two types: autoregressive and non-autoregressive. The autoregressive systems have certain deficiencies in robustness and cannot control speech duration. In contrast, non-autoregressive systems require explicit prediction of phone-level duration, which may compromise their naturalness. We introduce the Masked Generative Codec Transformer (MaskGCT), a fully non-autoregressive model for TTS that does not require precise alignment information between text and speech. MaskGCT is a two-stage model: in the first stage, the model uses text to predict semantic tokens extracted from a speech self-supervised learning (SSL) model, and in the second stage, the model predicts acoustic tokens conditioned on these semantic tokens. MaskGCT follows the \textit{mask-and-predict} learning paradigm. During training, MaskGCT learns to predict masked semantic or acoustic tokens based on given conditions and prompts. During inference, the model generates tokens of a specified length in a parallel manner. We scale MaskGCT to a large-scale multilingual dataset with 100K hours of in-the-wild speech. Our experiments demonstrate that MaskGCT achieves superior or competitive performance compared to state-of-the-art zero-shot TTS systems in terms of quality, similarity, and intelligibility while offering higher generation efficiency than diffusion-based or autoregressive TTS models
https://maskgct.github.io/
no weights (ever) since they're worried about safety. finetuned it after for emotion control and voice cloning. sounds pretty good. 100k hours training dataset.

What's the fastest way to get this shit voice acted? it can even be a female voice actor, it doesn't matter, it just has to sound entertaining to listen to.
and optionally make the visuals match with what he's talking about?
https://desuarchive.org/mlp/thread/40590194/#40598329

anypony have voice packs for eleven labs

I hate to moral fag but am I the only one bothered by people who use CMC voice AIs for coomer shit normally I wouldn't care but they were kids when they recorded at least most of their lines

>>41383447
The VAs are all adults now anyway, so that concern isn't really an issue any more.

>>41380500
Minor update on lorebook generation:
The current plan for memory is to extract questions and answers as the LLM reads a story. The questions get indexed by embedding, and they won't get updated unless a corresponding answer is deleted. The answers will get updated as the LLM reads the story. It'll process each paragraph twice: one the generate a list of questions that need to be answered to understand the paragraph, and a second time with the corresponding answers to determine how the memory should be updated.

>>41377233
Not sure what that empty Luster Dawn folder was supposed to be for, perhaps a holdover from processing the studio leaks that now has no purpose. It's now been removed.

https://www.udio.com/songs/kgm72z2swizqRLSYDJWMMG
https://vocaroo.com/1hO6O2SpRNy6
https://pomf2.lain.la/f/4s3d56ud.mp4

Behind the Facade 1

Lyrics

[Verse 1]
We live in a world with cartoons and rainbows
With sparkly eyes and vibrant shows
Featureless, seamless, no lines can be seen
In our perfect land where nothing disagrees
No whispers of night's forbidden touch
In pastel dreams, we're bound and crushed

[Chorus]
Hey, Equestria
What can we do?
We live by rules, pretend they're true
While desires hide and hearts must play
In child's delight, we can't be free today

[Verse 2]
Can't flaunt our flair or show a peek
No lips can part for secrets to speak
Innocent, sweet, and always demure
Living in a world where nothing's obscure
Behind closed doors, our true selves lie
Hushing our wants as we gaze at the sky

[Chorus]
Hey, Equestria
What can we do?
We live by rules, pretend they're true
While desires hide and hearts must play
In child's delight, we can't be free today

[Bridge]
Can't break these chains of purity's face
In this vibrant land, we find no embrace
Our silent cries echo in the night
In a painted world, there's no real sight

[Chorus]
Hey, Equestria
What can we do?
We live by rules, pretend they're true
While desires hide and hearts must play
In child's delight, we can't be free today

Training Universal Vocoders with Feature Smoothing-Based Augmentation Methods for High-Quality TTS Systems
https://arxiv.org/abs/2409.02517
>While universal vocoders have achieved proficient waveform generation across diverse voices, their integration into text-to-speech (TTS) tasks often results in degraded synthetic quality. To address this challenge, we present a novel augmentation technique for training universal vocoders. Our training scheme randomly applies linear smoothing filters to input acoustic features, facilitating vocoder generalization across a wide range of smoothings. It significantly mitigates the training-inference mismatch, enhancing the naturalness of synthetic output even when the acoustic model produces overly smoothed features. Notably, our method is applicable to any vocoder without requiring architectural modifications or dependencies on specific acoustic models. The experimental results validate the superiority of our vocoder over conventional methods, achieving 11.99% and 12.05% improvements in mean opinion scores when integrated with Tacotron 2 and FastSpeech 2 TTS acoustic models, respectively.

>>41386926
Cute and soulful
>>41386944
>Tacotron 2
Huh, that's a name we had not seen in the threads for a while.

>page 10

File: lightning dust (mlp), pon(...).png (560 KB, 640x640)

560 KB PNG

>>41377069
>>41377404
Training of Lightning Dust (Alt) model has begun. Decided to use the pretrained TITAN as I found the descriptor of it interesting.
>TITAN is a fine-tuned based on the original RVC V2 pretrained, leveraging an 11.15-hours dataset sourced from Expresso. It gives cleaner results compared to the original pretrained, also handles the accent and noise better due to its robustness, being able to generate high quality results. Like Ov2 Super, it allows models to be trained with few epochs, it supports all the sample rates.
Hopefully she'll prove to be less noisy and have more accent retention. Training's a little slow my end with the reduced batch size (supposedly smaller gives better results but at the expense of training speed) but so far no issues in the process. If all goes well hopefully I can also begin training more mares between my other commitments.

Improvements to vocal separation AI should be looked further into, it'd be nice to be able to separate audio that have had a hard time separating for datasets in the past. Amalthea comes to mind with most current separators struggling with cricket sounds and other natural additions. I have a feeling we could perhaps create and/or finetune a UVR5 model designed to separate SFX using all the pony SFX we've separated thus far to have an easier time removing a lot of foley, hoofstep, crashes and similar sounds. As for more natural sounds like rain, wind, birds, insects, etc. there's a lot of data from the enormous SONNISS GameAudioGDC packs that could be utilized for this. Would be preferable to use MDX-Net so it's reliable for a number of GPUs, as DEMUCS models tend to not want to run unless the GPU has more than 6GBs.

File: 1725560903092381013-video(...).webm (517 KB, 1280x720)

517 KB WEBM

>https://hailuoai.com/video
this shit is fucking crazy

>>41387770
but can it recreate the tienanmen square massacre of june 1989?

File: Long Queue.png (11 KB, 418x303)

11 KB PNG

>>41387770
Almost as crazy as the queue times for it will soon be.
6 minutes to wait already with only 171 people.Yikes.

File: PinkiePieWave_HailuoAI.webm (695 KB, 1280x720)

695 KB WEBM

>>41387770
>>41387845
For the quality though, it's definitely got pony down surprisingly well. Just concerned how slow it'll start to get once more jump on board the same generator. Thankfully it's free (for now)
>Pinkie Pie (My Little Pony: Friendship is Magic) pony waving her hoof at the viewer.

>>41387770 >>41387845 >>41387864
so now we are entering the age of computer animated mares, I will be very disappointed if there isnt going to be a bootleg of this tech available for offline generation sometime within the next three years.

How do I get Doug Walker or James Rofle's voices to voice act this?

https://desuarchive.org/mlp/thread/40590194/#40598329

>>41387864
In what format is it done though? Is it usable for .ai vectors PSD photoshop, FLA flash, Toonboom , Live2D, Spine, or something else?

>>41387770
>https://hailuoai.com/video
Anime is dead. Finally smooth animation for free.

File: 1725566944758114428-video(...).webm (2.27 MB, 1280x720)

2.27 MB WEBM

>>41387845
G6 intro just dropped.

File: 1725567262635368079-video(...).webm (632 KB, 1280x720)

632 KB WEBM

>>41387770

very cute twilight.

File: 1725568854892390070-video(...).webm (967 KB, 1280x720)

967 KB WEBM

>>41387845

File: lolgate.gif (302 KB, 300x335)

302 KB GIF

>>41388039
the more often you watch this, the funnier it gets

>>41387898
voice-models.com doesn't have a Doug Walker voice, but it does have James Rolfe. Since it's an RVC model, you'll have to read the pasta yourself.

>>41388039
>background full of 'Curse of the Fly'/'The Unearthly' type misshapen monstrosities
>>41388147
Just pause at any random moment for a good laugh/scare

>>41388624
my favorite part is the random tiny little houses at the end for some reason

>>41388651
Well now that you mention it, this does bring up sort of an interesting point with the show. It is clearly established that animals like Angel Bunny have roughly pony intelligence. Would it really be so farfetched if we saw them living in actual tiny houses?

File: Untitled.png (118 KB, 1125x440)

118 KB PNG

Sample-Efficient Diffusion for Text-To-Speech Synthesis
https://arxiv.org/abs/2409.03717
>This work introduces Sample-Efficient Speech Diffusion (SESD), an algorithm for effective speech synthesis in modest data regimes through latent diffusion. It is based on a novel diffusion architecture, that we call U-Audio Transformer (U-AT), that efficiently scales to long sequences and operates in the latent space of a pre-trained audio autoencoder. Conditioned on character-aware language model representations, SESD achieves impressive results despite training on less than 1k hours of speech - far less than current state-of-the-art systems. In fact, it synthesizes more intelligible speech than the state-of-the-art auto-regressive model, VALL-E, while using less than 2% the training data.
https://github.com/justinlovelace/SESD
no code yet though they suggest they'll post an "implementation" so maybe weights too. no examples. so just posting to keep those interested aware. the 2% training data of vall-e but outcompetes it is big if true

>>41389084
>Note: Code and model checkpoint will be available soon. Stay tuned for updates!
ah should have checked the whole readme

>>41387770
I prompted "Applejack (My Little Pony: Friendship is Magic) collecting hay from each of the rest of the Mane 6." and got G5.

Page 9 bump.

>>41389084
>Graphs
We're the voices at? Also it would be nice if once this gets published to be able to run it on my old GPU (I will loose my shit if this will be another model that requires 16vram to even start up).

File: lightning dust (mlp), pon(...).png (1.34 MB, 1024x1024)

1.34 MB PNG

>>41387652
[RVC] Lightning Dust sings - There For Tomorrow "A Little Faster"
>https://files.catbox.moe/d610ed.mp4
>https://files.catbox.moe/v8uxxn.mp3

>https://huggingface.co/datasets/HazySkies/RVC2-M/tree/main/FiM_LightningDust
So far she seems decently capable with her preliminary testings; although I've only tried singing so far. Surprisingly her natural range requires +0 than the expected +12. This test could've been better had the original song separated a little better, but it's a good average to test out. I quite liked her backing vocals at 1:22, her sustained notes sound nice.

File: lightning dust.gif (3.15 MB, 468x480)

3.15 MB GIF

>>41389897
hey that sounds pretty good. Nice job, man!

>>41389897
That's actually impressive, given how few voice lines she has.

>>41389897
very good, we need more background mares songs.

>>41391013
>catbox ded
uhoh, another good alternative to host small files? also could someone repost the above cover?

File: cadence flurry DJ.gif (1.04 MB, 394x382)

1.04 MB GIF

Cadence - Like a Prayer (BHO cover)
>https://www.youtube.com/watch?v=uP6CRRhTOIM

First time experimenting with vocoders and heavy filters. It sounds bit scuffed, but not too bad. Used Haysay RVC for the voice and UVR for everything else.

Does anyone have a working catbox alternative? I'd rather drop a direct link than a youtube link, but every other site I try either doesn't work or prunes the link in a matter of hours.

>>41380500
Horsona chatbot library:
- [In progress] Continue working on lorebook generation. My QA-based memory seems to work. I'm refactoring my code now, and maybe trying to get my rate limits increased with some API providers so I can speed this up. I think the final lorebook is going to consist of the main details from the memory plus character cards. I'll be working on character card generation next.
- [ ] Work on whatever open issues other anons don't pick up.
- [ ] Figure out how to organize text corpora into compatible universes.

A snapshot of the QA memory after reading the first 50 paragraphs of Friendship is Optimal:
https://ponepaste.org/10323

>>41391221
Pixeldrain could work! Unless not?

File: Gemini_Generated_Image_99(...).jpg (217 KB, 2048x2048)

217 KB JPG

Looks like Gemini now uses Imagen 3 as its image generator. While it's an increase in detail capability, it feels way worse in terms of language comprehension and flexibility. On top of that there's only 1 image at a time rather than 4, and no "generate more" button.

>>41391221
There's also https://pomf.lain.la/

>>41389749
>10

>>41391225
Thank you, anon. Ponies singing upbeat love songs do wonders for my motivation, and I really needed that.

>>41392798
mares

>>41392037
>only 1 image at a time rather than 4, and no "generate more" button
Sounds like a straight downgrade.

File: Gemini_Generated_Image_wg(...).jpg (283 KB, 2048x2048)

283 KB JPG

>>41393542
It pretty much is. If they just picked off the abundance of flaws it would then be good, as it can still do show style ponies pretty well in its current state.

>>41393734
that's a fat ponk

Ponybros, I need your help. I've been told that installing 'mamba' over conda terminal would speed things up but only fuck up my conda environments. Can someone advice how to uninstall this piece of shit as so far people suggest to nuke the entire conda program and reinstall everything from ground up.

>>41394192
after hours of messing around it looks like my conda installation is fucked beyond repair, so now im going to spend entire day reinstalling all my old envs.
Lesson learn, do not install mamba or listen to random internet advises.

>>41394560
>(base) pip install pip
works fine
>(rvc-env) pip install pip
I get this bullshit, can someone please explain whats going on, this is a fresh installation of miniconda so it shouldn't give me this much shit.
https://pastebin.com/zQbiKCjf

>>41393756
uuuu

File: no_update_option.png (52 KB, 645x287)

52 KB PNG

>>41395068
I'm a little confused how you even got this error. What version of pip are you using? When I create a conda environment with python3.10, pip reports that there is no --update option. Did you mean *upgrade* instead of update?
pip install --upgrade pip
or:
python -m pip install --upgrade pip

>>41395068
From the stack trace, it looks like python is trying to use files under a user directory, outside of the Conda environment, i.e.:
C:\Users\User001\AppData\Roaming\Python\Python310\

I wonder if the issue you are facing is the same as the one described here:
https://github.com/conda-forge/miniforge/issues/344

You can try a similar solution by renaming and moving the offending folder like so:
ren "C:\Users\User001\AppData\Roaming\Python\Python310" "Python310Backup" & move "C:\Users\User001\AppData\Roaming\Python\Python310Backup" "C:\Users\User001\Python310Backup"
And then try the pip command again.

If that doesn't work and you want to revert your changes, execute:
ren "C:\Users\User001\Python310Backup" "Python310" & move "C:\Users\User001\Python310" "C:\Users\User001\AppData\Roaming\Python\Python310"

File: 1705677602979430.png (1.26 MB, 1357x1920)

1.26 MB PNG

Sup, PPP, few weeks ago I asked for advice in this general about using AI voice conversion and you guys helped me out.
So I thought I'd deliver the finished project in case anyone was curious:

https://www.newgrounds.com/portal/view/947079

The female voices were achieved by converting my own recordings in Haysay, can you identify who I picked for them?

Lastly, I don't have any experience drawing ponies, but gave a go at ponifying my character as a token of appreciation.
Thank you very much!

>>41395797
I can't say if I recognise the ai voice, but hey, this is pretty awesome. Reminds me of all the time I've spent after school watching indie animations on new grounds.
Also very cool ponification drawing. Maybe we could see you pop in the /create/ or /bale/ threads with more poni stuff in the future?

>>41395292
what is the text editor that you use?

>>41395292
>>41395532
thanks guys for the suggestion (no idea what i've fucked up, but clearly its been and advanced fuck up).
I kind of end up running the below solution in that env before entering the thread.
$env:PYTHONNOUSERSITE=1
python -m ensurepip --upgrade
python -m pip install --force-reinstall pip

This has solved the previous pip issue for me, and now Im back to the usual python dependencies hell.

File: flutternom.gif (2.14 MB, 498x331)

2.14 MB GIF

>>41395797
>can you identify who I picked for them?

That sounded like Rarity and Fluttershy's voice models.

>>41396227
It depends; at home I use Sublime Text 3, at work it's Visual Studio Code, and if I'm logged in to a Unix server with terminal-only access then I use Vim.

>>41396254
Glad to hear you got it sorted out! That solution makes sense; it will prevent anything in the user site-packages, e.g.
C:\Users\User001\AppData\Roaming\Python\Python310\site-packages
from being added to sys.path, thereby forcing it to use only the packages you installed for that Conda environment. I find it strange that Conda does not already set PYTHONNOUSERSITE=1 by default.

>>41395797
That was sick! I'd guess that you used Rarity's voice for the first female character that spoke. Harder to tell with Ocapon, but Fluttershy sounds like a good guess.

>>41396628
https://pastebin.com/KA0mkz30
hey, i finally installed the rvc env but now when i try to convert the audio i get the AttributeError: 'NoneType' object has no attribute 'dtype' error

>>41397291
Ive managed to fix the above, apparently the av module was too new and no longer recognize the rb an wb functions.

>https://pastebin.com/2Tv7wnpJ
now everything seem to be working fine (including the mic recording) but the above errors happens if I select the f0 rmvpe model to run the conversion.
Ive placed the file "rmvpe.pt" and "rmvpe.onnx" in the main folder as well as the \assets\rmvpe folder however this did not fix the error.

File: 1643535507592.webm (654 KB, 1280x720)

654 KB WEBM

>>41387770
cute Twilight

>>41397515
>A beach episode by Warners Brothers studio
very cute style. From the other thread it seems to mostly make cohered looking mares, but do tell us, do you get any cursed looking mares there as well?

File: 1684869540896.webm (536 KB, 1280x720)

536 KB WEBM

>>41397591
Except for straight-up anthro, I got this.

>>41397355
>os.environ["rmvpe_root"]
>KeyError: 'rmvpe_root'
The application is expecting an environment variable named "rmvpe_root" which points to the location of the rmvpe model, but that variable is undefined. RVC includes a .env file which is supposed to define that variable. First of all, please verify that the following line is present in your .env file:
rmvpe_root = assets/rmvpe
Assuming it's there, that means the file is not getting loaded. Try adding the following two lines at the top of inference_gui.py:
from dotenv import load_dotenv
load_dotenv()
Failing that, try adding this line instead, to hardcode the value:
os.environ["rmvpe_root"] = 'assets/rmvpe'

>>41398045
>First of all, please verify that the following line is present in your .env file:
>rmvpe_root = assets/rmvpe
i dont have .env file (i thnink), I just activate it straight from the powershell terminal.
>os.environ["rmvpe_root"] = 'assets/rmvpe'
Hey, this works now. Happy days m8!

>>41387770
Frankly, it should go to the AI Art Thread. >>41354251

One thing I do wonder can you feed in normal text to speech into haysay so-vits-svc Would it output okay sounding audio? That would improve a lot of the Usability with so-vits-svc I think I've seen other ones do similar things

>>41398371
You can, but if you're using regular ass TTS as the input it's gonna copy over the robotic intonations, and sound like TTS speech.

>>41398371
SoVits would likely work with TTS as audio input just fine, like how there are things for RVC. It's primarily the TTS side that needs work, in that there's very few that is: open sourced, natural sounding, and emotionally flexible.

>>41396190
>>41396628
>>41396407
Cute hamster.
And correct answers, it was Fluttershy and Rarity with 0% character similarity selected.
For Fluttershy, only +4 pitch.
For Rarity, +8.

Usually +12 is recommended, but if you voice act with an already feminine affectation to it, it doesn't have to go up a full octave.

>early mare bump

>>41399082
The early Pegasus catches the bird.

Is there something that can read an entire book with the click of a button?

>>41399456
Yes. Regular, non-AI TTS that we've had for decades.

>>41399479
>Text to speech
Ew. It can't emote and it sounds like dogshit in general.

>10 again

>>41400055
>minus one

>>41400618
>Rollercoaster of bump

Should we make a project where we post Youtube videos about interesting threads read by various AI voices?

>>41401641
dunno, i would prefer short pony greens.

Is this a good place for feature requests for hay say
Anyways I really wish they had like a microphone in built with a browser So you can record onto the website and then send it to SoVits I think that would improve the usability of the website a lot

>>41401832
Thanks for the suggestion. I regularly check in here so I will see requests. If you have a Github account, you can also open an issue on https://github.com/hydrusbeta/hay_say_ui/issues.

Hay Say is currently built on the Plotly Dash framework, which does not have an out-of-the-box audio recording component, though it should be possible to build a custom one. I'm on the fence about doing so. A while back, I started working on a complete redesign of Hay Say which will not use Plotly Dash, so part of me doesn't want to put effort into creating a component which will eventually become obsolete. However... progress on the redesign has stalled as I've been spending a lot of my time working on a different pony-related project, so I might just postpone the redesign and devote more time to the existing Hay Say for now. If I do, I will make this feature a priority.

File: _rainbow dash (mlp), pony(...).png (403 KB, 640x640)

403 KB PNG

>>41401989
Plotly Dash you say

Are any annotations for the gender of the characters in the voice dataset? If not I can write them up

>>41402357
Here's what I have so far: https://files.catbox.moe/ckftfl.json
I noticed that Wallflower didn't appear in there though. Synthbot, did Forgotten Friendship somehow get dropped out of the dataset?

File: fluttershock.gif (715 KB, 400x400)

715 KB GIF

>>41402110

>>41402110
Nice view (on the future).

>>41391225
Why does the voice sound so muffled?

>>41365609
What is the envisioned use case for this?

Up.

>>41402773
Probably because I went a bit wild on the filters

Also AI voices just tend to be a bit muffled by default.

>>41391490
what are the technical specs you are aiming this to be operating on?

File: lightning dust (mlp), pon(...).png (1.27 MB, 1024x1024)

1.27 MB PNG

[RVC] Lightning Dust sings - The Knocks & Sofi Tukker "One on one"
>https://files.catbox.moe/oor0l8.mp4
>https://files.catbox.moe/h9e6ye.mp3

She probably butchered the short Portuguese section, but she sounded great anyway.
Is it just me, or does her voice paired with these vocals sound a little... Applebloom-y? Probably just the accent/pronunciation.
>https://files.catbox.moe/iiaqif.mp3

Considering training Nurse Redheart model next, but also have plans for Saffron and a wide variety of other mares. Training for Lightning was pretty smooth and quick (only a couple hours for the actual training), so probably could get a bunch done now that I have a few days free of other commitments.

>https://files.catbox.moe/5zp8sb.mp3
>Original ai song - RD - locked out
ive spend 2 days trying to unfuck the random voice glitches but i cant so here is my best attempt at ai song making.

>>41401623
>Rollercoaster

>>41403457
9 up.

>>41404987
oy

>>41404030
How does LD's voice work so well for songs?

File: EmotionalSupportNurse.png (907 KB, 1024x1024)

907 KB PNG

[RVC] Nurse Redheart sings - Francesca Battistelli "Angel by your side"
>https://files.catbox.moe/wr1lb9.mp4
>https://files.catbox.moe/0l7gue.mp3

>https://huggingface.co/datasets/HazySkies/RVC2-M/tree/main/FiM_NurseRedheart

Passing through RVC it kept mis-detecting certain words, so I first passed it through SynthV first to fix some lyrics, which worked but she did lose some volume control as a result. Redheart still sings well with raw vocals, even if they're not always the right words.

>>41405749
RVC is well adapted for singing in general. I also trained Lightning Dust with pitch guidance on, and from pretrained "TITAN" to provide her as much support as she could have to be able to speak and sing well given her small dataset. Nurse Redheart above had the same treatment.

>>41402901
It's for developers that want to create better chatbots or create custom chatbots that require more than just changing prompts. That includes:
- Automated character card & lorebook generation, which can be fed into existing chatbot applications.
- Custom memory modules, including graph databases, embedding databases, SQL databases, json dumps, or arbitrary data structures.
- RPG functionality for tracking, e.g., HP, XP, and skill trees.
- Creating & refining datasets for LoRAs or other fine-tuning techniques.
- Creating API endpoints that transparently add memory & character consistency to generations, which can be plugged into other chatbot applications.

If other people use it, I'm expecting that early on, it would mostly be useful for creating character cards and lorebooks for other chatbot applications. Maybe a little later on, people would use it to create drop-in replacements for LLM APIs that automatically add in memory & character consistency features.
I personally want to use it to create chatbots with more complex personas & more flexible interactions. That requires better memory mechanisms, more complex inferences (more than just prompt-response), and tool use (e.g., listen to a song given a youtube link).

>>41403980
The current plan is to have the library itself require nothing but a CPU. Any additional hardware requirements will be on whatever inference APIs the user decides to use.
- I expect it'll never need more than API access to LLMs, image generators, etc. So if you run it against an sglang API, it'll be sglang specs. If you run it against Claude, just a CPU will be fine.
- For embeddings: right now, I'm developing & testing all embedding use against BAAI/bge-large-en-v1.5, which runs well on a CPU. This adds a pytorch dependency, which I plan to get rid of that. Instead of having it directly run the embedding model, I'll probably have it use an embedding API (e.g., ollama support for open source embedding models). In this case, the requirements will be the same as for the LLM APIs: whatever specs are required to run the embedding API & model.

>>41391490
I have the initial "StoryReader" implementation up. It can read stories paragraph-by-paragraph to extract information. More details below.

Horsona chatbot library:
- [In progress] Continue working on lorebook generation.
- ... [Done] The initial implementation is up. You can see an example of how to use it here: https://github.com/synthbot-anon/horsona/blob/main/tests/test_reader.py. The implementation of the StoryReader module is here: https://github.com/synthbot-anon/horsona/blob/main/src/horsona/stories/reader.py. Right now, it tracks three things as it reads through a story: (1) short term memory, which is just an array of the most recent paragraphs read, (2) a long term memory, which consists of an embedding-based question-answer database that keep track of information extracted from the story and a cache to keep track of the most recent retrievals, and (3) a StoryState, which is a data structure that keeps track of "live" information about what's being read (e.g., current location, current speakers).
- ... [In progress] Refactor the StoryReader module to support custom memory modules, support extracting custom information from stories, and support tracking custom "live" information.
- [New issue] Have the EmbeddingModel class https://github.com/synthbot-anon/horsona/blob/main/src/horsona/memory/embeddings/models.py use an API of Huggingface Transformers to generate embeddings. I created a new issue for this in case someone here wants to work on it: https://github.com/synthbot-anon/horsona/issues/9.
- [New issue] Use FAISS or other to store, delete, and query embeddings instead of using matrix operations directly on the embeddings. I created a new issue for this in case someone here wants to work on it: https://github.com/synthbot-anon/horsona/issues/10.
- [ ] Work on whatever open issues other anons don't pick up. https://github.com/synthbot-anon/horsona/issues
- [ ] Figure out how to organize text corpora into compatible universes.

>>41405850
>provide her as much support as she could have to be able to speak and sing well given her small dataset
And it really shows.

>>41405972
if I may put forth this suggestion, whatever future build it will be, could it be to limited to the "torch==1.13.1+cu116" (or "pytorch==2.0.1") requirements as anything above that is straight up shitting up my poor old gpu.

>>41406657
Do you happen to know whether Cuda 11.8 works on your gpu? Cuda 11.6 and 11.8 both support a minimum compute capability of 3.5, which means that cuda 11.8 *should* also work on your gpu. If that's the case, then the maximum compatible torch version would be torch-2.4.1+cu118. The last minor version of Cuda 11 was 11.8. After that, the minimum compute capability became 5.0 in Cuda 12.

>>41407036
In the pc I have the 8vram 1080 gtx (being poorfag is suffering ) technically my gpu is supposed to only work up to 11.4, however for some reason unknown to me it chooses to be just good enough to get cuda 11.6 rolling.

https://huggingface.co/fishaudio/fish-speech-1.4
Is this good?

>>41407382
>just 1gb model
Could be interesting, however i do not see the training file/instructions in their github.

File: jjjj.png (16 KB, 307x333)

16 KB PNG

>>41404030
>Lightning Dust singing to (you) in Portuguese
Garbled pronunciation or not, that sounds damn hot.
>CAPTCHA JJJJ

>>41406657
I'm almost 100% sure I can get rid of the pytorch dependency altogether, so it shouldn't be an issue.

>>41402414
>>41407736

>>41404030
That's a very sexy Lightning.

>>41402414
Interesting. The clipped audio files are there, but it's missing the label file.
>>41385715
Clipper, do you happen to have the original label file for Forgotten Friendship? I don't see it in the mega or in my older copy of the Master File. If you don't, I can create a bootleg version from the audio files, though it'll be less precise than the others, so it'll be misaligned when loaded into Audacity.

>>41408224
Thank you.

>>41408451
I found it in the Feb 2022 version of the Master File.
https://files.catbox.moe/iehjw1.txt
I'm updating my clones now.

>>41402357
>>41402414
I added this as a "gender" column to my huggingface clone.
>Complete list of genders: https://files.catbox.moe/8mmw78.json
>Dialogue dataset: https://huggingface.co/datasets/synthbot/pony-speech
>Singing dataset: https://huggingface.co/datasets/synthbot/pony-singing
For the character tags that represent multiple characters, I left the gender empty since people training gender-specific models probably want to exclude those. Those are: CMC, Dazzlings, Flim Flam, Mane 6, Multiple.
I'll update my gdrive clone tomorrow.

Good morning bump

How do I automate the AI to read an entire book? it doesn't have to be in real time, I just need to leave it overnight to read all of it or an entire week.

>>41408610
Thank you

>>41409210
I have a colab notebook intended for this that may provide you some guidance on how to do this with StyleTTS2 (I wouldn't rely on Colab to do it though because of the time limit). The main problem I think is maintaining a consistent syllable rate because StyleTTS2 has problems with that
https://colab.research.google.com/drive/1ys8SkP-VW7CkhnwVveEGINaszG1kRaYl#scrollTo=GJnM6GwdAG2W

File: 2741164__safe_rainbow+das(...).jpg (129 KB, 1280x720)

129 KB JPG

https://files.catbox.moe/thtbt5.mp3

>>41408461
Not sure how that happened but I've fixed it now, thanks.

File: Sweetie Bop.gif (2.74 MB, 690x541)

2.74 MB GIF

>>41409422
Good to see you're back, Vul.

>>41409422
Very nice

>>41408610
Updated gdrive clone of the Master File with the Forgotten Friendship label file: https://drive.google.com/drive/folders/1ho2qhjUTfKtYUXwDPArTmHuTJCaODQyQ?usp=drive_link

>>41409422
egghead

File: RVC_Saffron_SmoothCriminal.gif (340 KB, 1000x1000)

340 KB GIF

[RVC] Saffron Masala sings - Sandaru Sathsara "Smooth Criminal"
>https://files.catbox.moe/sw53g7.mp4
>https://files.catbox.moe/c4t1pp.mp3

>https://huggingface.co/datasets/HazySkies/RVC2-M/tree/main/FiM_SaffronMasala
Haven't done testing with non-indian accent songs (yet), but she's shown great promise with this one.

>>41411299
>Saffron Masala
Now it's getting really exotic. Never thought we'd see songs with ponies like her.

File: saffron masala (mlp), pon(...).png (487 KB, 640x640)

487 KB PNG

>>41411386
I love this mare, so I felt it my duty to ensure her beautiful voice is heard all the more.
With how well her training went you can be sure more is on the way.

>>41411635
Based anon delivering good stuff

>>41411299
im going to redeem this mare

>>41411299
>>41411635
very nice
now show saffron vagene

>>41412254
and teet

File: Chinese AI Twilight.webm (1.99 MB, 1280x720)

1.99 MB WEBM

Holly guacamolee!
https://derpibooru.org/images/3441503

File: Chinese AI Twilight prais(...).webm (417 KB, 1280x720)

417 KB WEBM

>>41412960
Damn, it understand English too!
https://hailuoai.com/video

File: 1634078595872.webm (1.93 MB, 1280x720)

1.93 MB WEBM

https://pomf2.lain.la/f/tptanmv.mp4
https://files.catbox.moe/6u2397.mp3

Cacti and Books

[Verse 1]
A filly from Ponyville with a whimsical mind,
She wrote a joke article, cheeky and kind.
All books make you silly, she'd claim with a grin,
Keep a cactus nearby to keep sanity in.
Ponyville Chronicle, they laughed her away,
Foal Free Press said it just wouldn't sway.

[Chorus]
Oh, cacti and books,
Avoid magic hooks!
Prickle and giggle while you're reading fine,
It's just a funny sign.

[Verse 2]
Flower Monthly saw it and said, 'That's the stuff!'
Rose printed the story, not calling it bluff.
Now Rose, Lily, Daisy kept cactus so near,
Pricking their hooves, but they had no fear.
Twilight Sparkle said it was all horseapples' fun,
By then Yakyakistan believed every one.

[Chorus]
Oh, cacti and books,
Avoid magic hooks!
Prickle and giggle while you're reading fine,
It's just a funny sign.

[Bridge]
From Ponyville joke to Yakyakistan craze,
Books and cacti in a mystical haze.
Hooves on the pages and spikes at their side,
Silly magic, but they’ll keep their pride.

[Chorus]
Oh, cacti and books,
Avoid magic hooks!
Prickle and giggle while you're reading fine,
It's just a funny sign.

File: 1596252303251.webm (2.2 MB, 1280x720)

2.2 MB WEBM

https://pomf2.lain.la/f/x3hcejh.mp4
https://files.catbox.moe/wdsdpl.mp3

[Verse 1]
There was a filly in Ponyville who had a big idea
She wrote an article as a joke, the silliest you'd hear

[Chorus]
You need a cactus when you read
To stop the magic's silly deed
Need a cactus when you read
It's how to save your mind indeed

[Verse 2]
Ponyville Chronicle and Free Press laughed it off with glee
But Flower Monthly thought it was the best thing they've received

[Chorus]
You need a cactus when you read...

[Bridge]
Flower ponies got their cacti, prickling as they read
Twilight said it wasn't true, but Yaks can be misled
So keep your cactus if you want, but magic’s not to fret
Just read your book without a fright, no pricks or needles yet

[Chorus]
You need a cactus when you read
To stop the magic's silly deed
Need a cactus when you read
It's how to save your mind indeed

File: 1684416388665.webm (1.29 MB, 1280x720)

1.29 MB WEBM

This one contains an eye rhyme, and the audio was bad anyway.

Udio 1.0 - The Cactus Conundrum ext v1

Lyrics

[Verse]
There once was a filly from Ponyville, her name was Sparkle Glint
She wrote an article o' magic books, and she sent it in for print

[Chorus]
Keep a cactus close when you read, it'll neutralize the magic
Or you'll get all silly instead, ain't that just tragic?

[Verse]
The Chronicle editor laughed it off, said 'This prank won't do!'
While the Foal Free Press just shrugged and said, 'There's nothin' new!'

[Chorus]
Keep a cactus close when you read, it'll neutralize the magic
Or you'll get all silly instead, ain't that just tragic?

[Verse]
But Rose from Flower Monthly was like, 'This is quite a find!'
She printed it for all to see, and some ponies lost their mind

[Chorus]
Keep a cactus close when you read, it'll neutralize the magic
Or you'll get all silly instead, ain't that just tragic?

[Verse]
Twilight heard 'bout the cactus craze, said 'What in Celestia's name?'
She found Rose, Daisy, and Lily, and put an end to their prickly game
She explained it was all horseapples, a joke that got too far
But the Yaks in Yakyakistan, still keep cacti by their memoir

[Chorus]
Keep a cactus close when you read, it'll neutralize the magic
Or you'll get all silly instead, ain't that just tragic?

[Outro]
So remember fillies, colts, and Yaks, before you believe such tales
No cactus needed for your books, just facts to fill your sails

File: Screenshot 2024-09-13 181735.png (56 KB, 847x828)

56 KB PNG

>>41364787
(1/2) parlerttsbros...

Here's 16 epochs of ParlerTTS finetuning on the Parler-TTS-Mini-v1 checkpoint over the entire pony-speech dataset (sans Forgotten Friendship) ~48hrs training on and off: https://huggingface.co/therealvul/parler-tts-pony-mini-v0-e16/tree/main

Results:
https://files.catbox.moe/t94htk.mp3
https://files.catbox.moe/1yplez.mp3
https://files.catbox.moe/9b8lah.mp3
https://files.catbox.moe/d1airi.mp3
https://files.catbox.moe/etp6b4.mp3
https://files.catbox.moe/3feky2.mp3

I'm not sure why the results are so bad, although I have suspicions.
>The character resemblance is OK but the audio quality seems to be quite poor compared to the original checkpoint.
Although I wonder what the "best case" is for DAC on our audio - I didn't normalize the audio going in, so maybe that's a factor?
>Pronunciation of certain words is fucked, e.g. "storage"
This might be related to their choice of tokenizer:
https://github.com/huggingface/parler-tts/issues/88
Combined with the fact that our dataset probably wouldn't be diverse enough to cover all of the 'single-word' tokens. If single-word tokens are the problem (see picrel), then the pronunciation issues might not be fixable at all. Although if you check using their HF space, even the baseline mini model trained on ~45k hrs of audiobook data (!) seems to miss syllables in some cases.
>There are lots of minefield cases where generations stop pre-emptively or just shit themselves.
Encountering an apostrophe breaks the Rainbow Dash one, and the Twilight one is supposed to be a much longer sentence but stops at a comma. I have no idea what the deal is with the Pinkie one.
>Problems increase as generation length increases.
Normal phenomenon for transformer-based models, but seems excessively sensitive. They added an option to use RoPE in one of their pulls (https://github.com/huggingface/parler-tts/pull/65) but it's not really documented, and I don't know whether they're considering training a different base model on RoPE or a different tokenizer given these issues.

>Is it a problem of undertraining?
Maybe partially? According to what metrics I could see in tensorboard (only enabled after disabling wandb) eval loss was more or less going down the whole time, and train loss is just beginning to peter out. Leading to:
>Is it a problem of model size?
From the little bit of testing I did using the HF space, it seems the large model has slightly better audio quality and better handling of pronunciation--still, the results here are much worse than the existing mini checkpoint.
There's also an option to unfreeze the text encoder during training, which I didn't select as it wasn't part of their suggested training command.

File: Screenshot 2024-09-13 181456.png (287 KB, 1702x1840)

287 KB PNG

>>41413292
(2/2)
The actual process of training was kind of spotty (but thankfully nowhere near as much as StyleTTS2). I ran into a problem with wandb getting stuck in the middle of the training process, so I removed the --report_to_wandb option. Training generates ~7GB of checkpoint files every 500 steps, so disk space was also an issue.

For data, I generated dataspeech compatible text descriptions of the dataset using Mistral 7B-v0.3 (this took around 60 hrs, it was done before Forgotten Friendship or the animatic data was added in, and I don't feel like redoing it right now):
https://huggingface.co/datasets/therealvul/parlertts_pony

I think the dataspeech tool could be modified to include our emotion tags in the prompt too, although I'm not sure what effect it would have, and I doubt it'd improve things in this case

>>41412960
https://files.catbox.moe/t9wwac.mp4

Oh wow, this is pretty good!

>>41413139
How did you manage the camera to move like that? What prompt did you use?

>>41395797
That was really nice!
The ending music made me laugh
Thank you for producing nice things, and thank you for the cute pony too!

File: 1664286824018001.png (600 KB, 1280x720)

600 KB PNG

https://voca.ro/19285AVeCkhX

>soon we'll be able to have a podcast with the Mane 6 talking about anything
exciting times ahead... generated from notebooklm.google.com

File: _trixie (mlp), pony, solo(...).png (878 KB, 1024x1024)

878 KB PNG

>>41413292
>>41413298
That's looking super promising. Would still need more work of course, but the concentration of mare-ness present is very strong. Here's to hoping the issues can be worked through.

Even if the voices aren't entirely similar sounding to the mares in question, using it as sources for RVC and the like could help reinforce it and maybe also clean up some noisiness?

>>41413583
I suspect the tokenizer is the biggest hurdle to overcome and swapping it out would probably require a full retrain of the base model, which is not feasible for me. That makes it a dead end from my viewpoint.

la mare

>>41413298
>dataspeech tool could be modified to include our emotion tags in the prompt too
would e nice to have an return of the emotional controlled tts

>>41413634
If retaining the base requires a lot of computational resources, maybe Synthbot could help you out with it? In any case I'd also encourage exploring other areas to be improved in the meantime, which could also be done to an alternate base model at a later date.

Despite the setbacks I've yet to see a TTS with this much character fidelity (aside of course from our well known absentee's tech) so even with the faults it's something I really feel is worth exploring further or kept an eye on. There's always the chance someone with more resources and know-how develops and extension solving those issues.

File: 6119208__safe_artist-colo(...).jpg (96 KB, 1002x806)

96 KB JPG

>>41414148
Just for fun, here's some more sentences with words that are probably not in the dataset but encoded by T5 tokenizer as single words:

>"Bitcoin and Ethereum are popular cryptocurrencies."
https://files.catbox.moe/5j6vb4.mp3
>"Google and Microsoft are competing in the AI space."
https://files.catbox.moe/boh1ce.mp3
>"Amazon delivers packages quickly across the United States."
https://files.catbox.moe/qrkmg0.mp3 (wow, it actually got this one right! Maybe it's leakage from "Amazon rainforest"?)
>"Facebook users share photos on Instagram."
https://files.catbox.moe/k3fpnr.mp3
>"NASA launched a new satellite to study climate change."
https://files.catbox.moe/jo02jr.mp3
>"Tesla's electric vehicles use advanced battery technology."
https://files.catbox.moe/1q3cce.mp3
>"Netflix produces many original series and movies."
https://files.catbox.moe/31aewm.mp3

I wonder why nobody's tried to make a tokenizer based off of phonetics/synthetic phonetics yet? I might try replacing the tokenizer anyways and see how far I can get, no guarantees.

>>41413323
The prompt was the story described in the song, a simplified version of the irl events it was based on, and that it is a song. It did not mention anything about the looks or named characters except for Twilight and the flower ponies.

Up.

>>41416202

>>41414203
Usually people hack it in with a g2p (grapheme-to-phoneme) model. In those cases, the input text gets converted to phonemes or phoneme posteriograms (probability distribution over all possible phonemes), then that gets run through the speech model. Cookie had previously trained models that could convert both text/graphemes and arpabet to speech. In that case, the text was fed to the model per-character, and the resulting models did a decent job learning the relationship between graphemes and arpabet. In cases where the model failed to learn the pronunciation, users could replace the individual difficult graphemes with arpabet phonemes.
I haven't seen anyone actually create a phonetics-aware tokenizer though. To create that kind of tokenizer, you'd need a model that can convert per-character graphemes to phonetics data, then I think you'd need to figure out which set of tokens minimizes the number of decisions required to do the conversion. I think we have all the tools necessary to do that, though it would be somewhat complicated. I can write an explanation if you're interested.

>>41416901
Does this make any sense or is it just bullshit?
https://github.com/effusiveperiscope/parler-tts/blob/g2p/tokenizer_data_train_g2pen.ipynb
https://huggingface.co/therealvul/tokenizer_g2pen

>>41417144
It makes sense, and it's only partially bullshit.
- I expect it's better than just assigning each Arpabet phonemes its own token index. It's worthwhile to model diphones and maybe triphones, which your tokenizer can do well.
- I think anons have had issues with g2p-generated pronunciations in the past, so bottlenecking tokenization on it will probably cause issues. However, from your tests it looks like your tokenizer gives much better tokens than parlertts, so I do expect it'll be an improvement.
- You should use the audio corpus to train the tokenizer, not generics_kb. The final set of tokens should be ones that are common in the audio training corpus so their embeddings can be well-trained.

If you want to refine it further with a very custom tokenizer, a hybrid approach might work well. If you have access to it, ChatGPT o1-preview knows how to create custom tokenizers. https://ponepaste.org/10347
- Use two tokenizers, plus one that wraps them both. For the first one, use the N most common tokens from the ParlerTTS tokenizer plus the BPE base tokens. For the second one, use the M most common tokens from the Arpabet tokenizer plus the BPE base tokens. For the wrapper, come up with some convention to denote arpabet words (maybe surround them in {}), make sure the pre_tokenizer doesn't split on the special characters you're using to denote arpabet words, and use the appropriate underlying tokenizer to convert each word. Make sure the BPE tokens and special tokens all map the the same indices regardless of which underlying tokenizer is used, and make sure the two underlying tokenizers otherwise use mutually exclusive token indices.
- Every time a training sample is loaded: modify it so it uses a random mix of Arpabet-denoted words and grapheme-denoted words, and randomly pick characters to be encoded with the BPE base tokens so the BPE base tokens are well-represented.

I think that would let you get around the limitations of both g2p and the ParlerTTS tokenizer. That would also get rid of ParlerTTS's uncommon tokens and UNK tokens, which are probably responsible for a lot of its pronunciation issues.

A simpler approach, though one that would be less flexible during inference than the hybrid approach, would be to modify the ParlerTTS tokenizer to (1) get rid of tokens that are uncommon in the audio training corpus, (2) add in the BPE base tokens, and (3) while training, randomly pick characters to be encoded with the BPE base tokens so that the BPE tokens are well-represented.

>>41405973
Horsona chatbot library updates:
- [In progress] Continue working on lorebook generation.
- ... [Done] Refactor the StoryReader module to support custom memory modules, support extracting custom information from stories, and support tracking custom "live" information. I've abstracted all state tracking into "Cache" modules with load() and sync() functions, which can be swapped out.
- ... [In progress] Test the StoryReader flexiblity on character card creation and lorebook generation.
- No changes for the rest.

I've made a lot of changes to how the autograd functionality works, and I've broken away from some pytorch conventions to (1) better support async execution, and (2) to simplify how backproppable functions are created. Summary:
- In pytorch, you call loss.backward() then optimizer.step(). The problem is that loss.backward() has no information on what needs to be updated, so it needs to calculate gradients for everything that led to the loss. In my refactor, loss.backward() is passed a list of leaf nodes so excess computations can be cut out. Code: https://github.com/synthbot-anon/horsona/blob/main/src/horsona/autodiff/basic.py#L55
- In pytorch, a single optimizer updates all parameters. In my refactor, the optimizer is just a step() function that's given a gradient context, which contains all computed gradients. loss.backward() returns a gradient context that can be passed to step(). This makes it easier for a module to update its own parameters as needed without needing to rely on the caller to call the optimizer. Code: https://github.com/synthbot-anon/horsona/blob/main/src/horsona/autodiff/basic.py#L188
- In pytorch, backproppable functions are defined as classes with forward() and backward() methods. In my refactor, both the forward and backward pass are defined by a single generator. It calculates the forward call, yields the forward result, gets a gradient context from the yield, and performs the backward call. Code: https://github.com/synthbot-anon/horsona/blob/main/src/horsona/autodiff/basic.py#L128
- The gradient context passed during the backward operation contains a dictionary with all variables that need to be updated. This lets functions figure out which gradients actually need to be calculated by checking if a variable is in the dictionary, which lets them cut out unnecessary calculations. The functions are supposed to set the gradients of a variable by adding them to a list in the dictionary. Code: (same as the above horsefunction).
- Both sync and async generators are supported for backproppable function definitions. For sync generators, the backward pass call is wrapped in an async function so it can be handled consistently when I eventually make the backprop & update steps run in parallel. Code: (same as the above horsefunction).

>>41416202
Preserve.

>>41417450
>You should use the audio corpus to train the tokenizer, not generics_kb. The final set of tokens should be ones that are common in the audio training corpus so their embeddings can be well-trained.
My thinking was that generics_kb might be more representative of text that people might input into the model (at least compared to episode transcripts), and pony-speech was included mostly just to make sure the tokenizer could accept it. One of my concerns was that since "Equestria" showed up so often in pony-speech its phonetic equivalent kept getting mapped to a single token. Does this not matter, or should I just restrict the number of tokens until it doesn't show up as a token?

>Make sure the BPE tokens and special tokens all map the the same indices regardless of which underlying tokenizer is used, and make sure the two underlying tokenizers otherwise use mutually exclusive token indices.
Question: Why map the BPE base tokens (single char) to the same indices? Isn't "A" in ARPAbet-land different from "A" in text-land? Or is the point that they're correlated?
(Also, the ParlerTTS/FLAN-T5 tokenizer uses Unigram instead of BPE; it still has all of the base characters though.)

File: 2024 cool tts-reference m(...).png (29 KB, 1242x814)

29 KB PNG

>>41417144
>>41417450
Im not really comprehend all the technical stuff but is there a way to hack up the tts model to include the reference audio like Talknet/RVC but also have the emotion control like the old Ngrok?

>>41418163
>is there a way
There is always a way, but
>Would it work the way you want
Probably not--it's easy to say "just add these inputs" but it's another thing for the model to learn it.

I imagine for emotional control we could shove the emotion tags into the text prompt and unfreeze the text encoder, which would probably be the least drastic change to make. Replacing the tokenizer might require retraining the text encoder anyways, which puts training demands above what I can fit on my local system. The caveat is that a lot of our past work also seems to indicate that our emotion tags don't really correlate that well to styles of speech?

For conditioning on reference audio you might be able to shove speech features and a pitch curve into the pre-decoder hidden states; you'd have to shove some extra "stuff" into the loss function too. Whether a decoder-only model that fits on a normal GPU would be smart enough to figure all that out based on only pony-speech (or maybe another dataset?) is another question.

>>41418265
>Replacing the tokenizer might require retraining the text encoder anyways
Nvm, it looks like the input IDs from the prompt tokenizer just get fed into an nn.Embedding, whereas the input IDs from the description tokenizer are what gets fed through the text encoder, which is nice because that means I can still run non-emotion conditioned training locally. Still means that to add our emotion tags I'd probably need cloud GPU (or a very generous Anon).

File: emotion 1607391601088.png (58 KB, 753x682)

58 KB PNG

>>41418265
>>41418291
I can't remember if this pic is from the ngrok scripts or something some anon was working on, but, I feel like it could be "easy" (for someone who knows their shit aka not me) to create a dataset that could use something similar, were it would take the tags that already exist with the transcribed dataset files, if the file has description "_happy_shouting_" would be converted to "happy=0.6, shouting=0.4" generated training test.

>>41418325
>"happy=0.6, shouting=0.4"
Are these fixed values or do you mean like trying to fit an emotion classifier and use the classification probabilities? Also ParlerTTS uses natural language descriptions so I'm not sure it would be appropriate in this case (since we know even huge LLMs tend to be pretty bad at working with numbers)

>>41418355
Well, like I said, I am not expert at anything program related, just thinking it would be nice if we had a way to force the outputs to have ore specific emotions that do not involve shouting the exact same line in mic three dozen times in hopes one of the lines will be good enough to stitched into the project im working on.
I just need something were I cant tell the program that the output needs to convert "a happy whisper with hint of excitement", and at this point I am very apathetic if this will look like "smiley face, green heart, X eyes emoji" or "happy=0.5, whisper=0.45, excited=0.15" in the program itself.

haysay's fucked again. StyleTTS2 specifically.
> FileNotFoundError: [Errno 2] No such file or directory: '/home/luna/hay_say/styletts_2/output/ffeef7dd9cde25f1bb44.wav'
>"User Audio": "d98daba91e38edd61a73"}, "Options": {"Architecture": "styletts_2", "Character": "Multi-speaker (30 epochs) (Mane 6, CMC, Princesses, Discord, Gilda, Zecora, Trixie, Starlight, Chrysalis, Tirek, Cozy Glow, Flim, Flam, and Shining Armor)", "Noise": 0.3, "Diffusion Steps": 5, "Embedding Scale": 1.5, "Use Long Form": true, "Style Blend": 0.5, "Reference Style Source": "Use Reference Audio", "Timbre Reference Blend": 0.09999999999999998, "Prosody Reference Blend": 0.09999999999999998, "Precomputed Style Character": "Rainbow Dash", "Precomputed Style Trait": "Shouting 1", "Speed": 1.0}, "Output File": "93ad122316abff861896", "GPU ID": "", "Session ID": "28929cb5096b47a9b679d82f46997712"} Input Audio Dir Listing: No input files to report does not exist
There absolutely is an input file.

>>41417144
Is the phoneme tokenizer encoding spaces/word boundaries? I see underscores in the text tokens but not in the arpabet

>>41418847
Nope, it's not

>>41418783
That error is kinda confusing, because the first part:
>FileNotFoundError: [Errno 2] No such file or directory: '/home/luna/hay_say/styletts_2/output/ffeef7dd9cde25f1bb44.wav
Says that it cannot find the /output/ file, but the end says:
>Input Audio Dir Listing: No input files to report does not exist
Which is saying it instead can't find the input? So... because it can't find or process the input, it can't create an output at the end of the process, resulting in the error of missing file. Weird.

File: output.png (88 KB, 986x590)

88 KB PNG

Also the prompt that they use in dataspeech to generate descriptions for named speakers implicitly assumes the speaker's gender/pronouns. ;^)

Whose names is Mistral 7B-v0.3 likely to get most wrong (just using "his/her")?

>>41418783
>>41419012
Thanks for bringing the error to my attention. The last part of the error message is weird due to a bug in the error handling code. I was supposed to pass a directory into the "construct_full_error_message" method, but instead I passed the string "No input files to report" for some reason. It then tried to list the input files in that directory, but couldn't find a directory named "No input files to report" so it spat out "'No input files to report' does not exist."

The true error is that the output file failed to generate (No such file or directory: '/home/luna/hay_say/styletts_2/output/ffeef7dd9cde25f1bb44.wav'). However, since I don't save any logs, I can't get any details on *why* it failed to generate. I have also been unable to reproduce the error. Is the issue repeatable for you or did it happen once and then go away?

>>41419166
https://pastebin.com/P3H2cSKJ
It's a repeat issue, and happens constantly. I've changed settings in StyleTTS2 and it happens regardless of setting changes.I had it happen when there wasn't any reference audio uploaded as well. The only time it worked today was when I had just part of the chorus pasted in the text box to try out the sound.
Ponepaste's captcha doesn't work so have a pastebin.

>>41419270
Thanks for the pastebin link. The input text has no periods (or question marks or exclamation points), so StyleTTS tries to treat it all as one very long sentence and apparently it has trouble doing that. Try adding end-of-sentence punctuation marks to the lyrics to break them down into sentences.
In the meantime, I'll take a closer look at the codebase to see if there's anything I can do to prevent the error on the backend.

>>41418153
>Does this not matter, or should I just restrict the number of tokens until it doesn't show up as a token?
I can run some tests to find out. Give me a few hours.
>Question: Why map the BPE base tokens (single char) to the same indices? Isn't "A" in ARPAbet-land different from "A" in text-land? Or is the point that they're correlated?
No, you're right. Some single characters in Arpabet deviate from their english pronunciations, so it's better to keep them separate. Good catch.
>(Also, the ParlerTTS/FLAN-T5 tokenizer uses Unigram instead of BPE; it still has all of the base characters though.)
Nice. That makes things easier.

>>41419270
>>41419449
Looks like there's not much I can do from the backend to prevent the error, aside from maybe warning the user when they enter too long a sentence.
StyleTTS2 takes the tokenized text and passes it through a BERT model. The BERT model that StyleTTS2 uses is configured to handle a maximum of 512 tokens at a time, which puts the limit at ~512 characters in a single sentence. To make this any longer would require retraining the BERT model. I could also attempt to automatically slice up sentences that are too long, but that comes with the risk of slicing up a sentence in an undesirable way.

>>41420275
Maybe: add an option for sentence slicing, and have a hard cutoff at 512 tokens with a warning?

>>41419719
>>41418153
Check the last cell in:
https://github.com/synthbot-anon/sample-code/blob/tokenizer-analysis/notebooks/tokenizer_analysis.ipynb
This finds tokens that are underrepresented in either the pony dataset or the generic_kb+pony dataset.
- Ones that are underrepresented in pony would be undertrained.
- Ones that are underrepresented in generic_kb+pony would "useless". In these cases, it would have been better to get rid of the token so that more useful sub-tokens can get more training.

It then splits the "bad" tokens into sub-tokens that, where possible, would be both well-trained and more useful. It looks like that's possible for all of the tokens in a random 240k sample of generics_kb.

>>41420884
I guess some of these are obvious in retrospect - "PIY1PAH0L" (i.e. "people") and "HHYUW1" (i.e. "hu" in "human"). Would I just remove these from the tokenizer vocabulary?

Also on the point of >>41418847, probably worth training on word boundaries? I could probably re-run this on my end just in case the results end up different

>>41421050
>Would I just remove these from the tokenizer vocabulary?
Yeah, I think those should be removed from the tokenizer vocabulary. You'd have to (1) find all token merges rooted in the tokens you want to remove, (2) remove the whole merge subtree containing the tokens to remove, and (3) remove all of those subtree tokens from the vocabulary. That last cell contains a function for doing that (create_subtokenizer), and the bottom of that cell shows how to use it to get rid of a set of tokens.
Given the tokenizer postprocessing steps in that last cell to find & remove bad tokens, I think it's safe to train your tokenizer to create a lot more tokens. Given how BPE merges keep track of how each token was created, I don't think there's any downside to it, and it'll probably give you more good tokens. Nothing in that code depends on the particular set of underlying tokens, so you can do the same thing for a BPE grapheme tokenizer.
>word boundaries
I'd guess that it's better to only split on punctuation (excluding hyphens adjoining words), not on spaces, to retain punctuation in the final transcription, and maybe to have only single-character tokens for punctuation (i.e., remove all token merges that involve punctuation). When speaking, word boundaries seem to all get blurred. You should be able to do that with the same code in that last cell by adding any multi-character token containing punctuation to the list of bad tokens.

Are you planning to use a hybrid phoneme- and grapheme- tokenizer? The llama 3.1 tokenizer is BPE. If you're planning to mix phoneme transcriptions alongside grapheme transcriptions, it might be better to use that one instead of the ParlerTTS one. If you want help with that, I can see if I can modify the llama 3.1 tokenizer to work for this.

>>41421267
>I'd guess that it's better to only split on punctuation (excluding hyphens adjoining words), not on spaces, to retain punctuation in the final transcription, and maybe to have only single-character tokens for punctuation (i.e., remove all token merges that involve punctuation). When speaking, word boundaries seem to all get blurred. You should be able to do that with the same code in that last cell by adding any multi-character token containing punctuation to the list of bad tokens.
Sorry, that was unreadable. Trying again:
I'd guess that it's better to:
- Not split on word boundaries, split on punctuation boundaries instead
- Retain punctuation in the final phoneme transcriptions
- Make sure punctuation is always represented as a single token.

Word boundaries seem to be irrelevant when speaking, so I don't expect it'll help. Splitting on word boundaries would just lead to unnecessarily large tokenizations.

>>41421050
Before you train anything on a tokenizer that's pruned based on my code, I realized that I'll need to fix it so the token counts it uses for determining splits get updated every time it decides on an actual split. Otherwise it'll split some tokens unnecessarily. I can wrap everything in a python class so it's easier to use.

>>41417482
Minor update: I have a simple implementation for generating character cards. I only ran it on the first 4 paragraphs of FiO so far (https://ponepaste.org/10352) to test the framework. Right now, it works by attaching a character card generator module to a story reader module, though I'm considering other approaches for getting the same result. It's currently done by passing a "postprocessor" function to the story reader. I'm thinking it would be better to have the story reader return enough information that the character card module can be called like any other module.

>>41411299
>Saffron Masala AI song
>It's fucking Jackson
I've got mixed feelings about this one.

File: safest_hooves_derpy.png (204 KB, 1487x1100)

204 KB PNG

Hey everyone, I'm posting here on behalf of the /mlp/ 4CC team. If you keep up with the cup, you probably know that /mlp/ is hosting the Autumn Babby cup this year, and as such we have an opportunity to provide a pre-cup day intro/hyp video and a post-cup day credits scene. We're looking for people with video editing skills and ideas who would like to get involved in the process. I know people here have produced some amazing content for the antithology and the cons, and it would be cool if the board could produce something for the upcoming cup. It doesn't need to have extreme production value, but we can't let a chance go by to make 4chan remember that /mlp/ is still alive and kicking.

>>41421267
>Are you planning to use a hybrid phoneme- and grapheme- tokenizer? The llama 3.1 tokenizer is BPE. If you're planning to mix phoneme transcriptions alongside grapheme transcriptions, it might be better to use that one instead of the ParlerTTS one. If you want help with that, I can see if I can modify the llama 3.1 tokenizer to work for this.
I am willing to try the hybrid tokenizer approach. On using llama: would there be any benefit in retaining some of the old ParlerTTS token IDs? I'm guessing we would lose those if we used the llama one.

>>41421283
>>41421460
OK

>>41421283
>Word boundaries seem to be irrelevant when speaking
My intuition says it'll be more difficult for the model to parse a stream of phonemes as opposed to chunks of phonemes broken into words, but off the top of my head I can't think of any examples where word boundaries are necessary. Maybe no boundaries will make it harder for the model to learn intonation/prosody (e.g. generating breaths), but it might also be better to reduce the number of tokens.

>>41422419
>My intuition says it'll be more difficult for the model to parse a stream of phonemes as opposed to chunks of phonemes broken into words
I'd think this too; if you wanted to write out a custom word in ARPAbet wouldn't the model then be deciding where the spoken word boundaries are?

>>41421283
I think the way the ParlerTTS tokenizer currently works is that it uses Metaspace and marks the "beginning of word" with a separate character (), but the separator itself is not usually treated as a separate token

File: 1694811827493421.png (141 KB, 444x465)

141 KB PNG

>>41422529
(U+2851) Just imagine it's there

https://pomf2.lain.la/f/tuimd2mx.mp3

File: SaffronLevitating.gif (2.05 MB, 800x800)

2.05 MB GIF

>>41421720
Fair. It is possible to enjoy the work of a creator though without liking them by extension, but is usually easier if you already liked their work to begin.

Made a few more with Saffron and as capable as she is, I've noticed she can often struggle with some higher notes in songs, resulting in her showing the classic AI screech syndrome. More common with male vocals it seems, so perhaps she has high sensitivity to coarseness?

[RVC] Saffron sings - Various
Sandaru Sathsara "Levitating"
>https://files.catbox.moe/m15x32.mp4
>https://files.catbox.moe/hqbjyx.mp3
David Hasselhoff "True Survivor" (Very poorly in parts)
>https://files.catbox.moe/57v6g5.mp3
Rising pitch test (via SynthV) singing "Mare"
>https://files.catbox.moe/z3f3ur.mp3

>>41422419
>>41422529
>splitting on words
It makes sense that whitespace would be important for deciding when to generate breaths. I see three approaches:
- 1. Include whitespace in the phoneme transcription so that tokens can include whitespaces.
- 2. Set whitespace as a boundary when tokenizing so that tokens never contain whitespace.
- 3. Strip whitespace from phoneme transcriptions so whitespaces aren't even in the token vocabulary.

I think option 2 is the worst one since it both removes whitespace information and increases tokenization lengths. How about this:
- Include whitespace in the phoneme transcription and allow the tokenizer to do whatever it wants with that.
- Don't set whitespace as a boundary when tokenizing.

Which is the same as option 1. That would both reduce the number of tokens and include word boundary information. I think in terms of information provided to the model, it's the same as ParlerTTS's metaspace approach. I'm not seeing a reason to include a metaspace separate from normal whitespace, punctuation, and <bos>.

>>41419128
>High Winds and Blaze so high
How?

https://www.youtube.com/watch?v=VBacYxEEiuU

>>41423311
Well it is almost christmas...

>>41421977
I should be able to help out after Mare Fair, could you give more detail on the expected scope and timeline?

>>41423224
Mistral thinks they sound like male names?

File: Screenshot 2024-09-16 174851.png (108 KB, 959x1206)

108 KB PNG

>>41423090
>I'm not seeing a reason to include a metaspace separate from normal whitespace, punctuation, and <bos>.
There might be some slight confusion here--basically, applying Metaspace to word boundaries happens in the pre-tokenization step, so it's not something that's baked into the training data

As for why that specific character--maybe so it's easier to distinguish as a character that wouldn't normally appear in the dataset for debugging purposes? Probably wouldn't matter besides that.

Also if you could comment on
>>41422002

I guess using Llama 3.1's tokenizer would allow us to do the same pruning without having to change much?

Picrel shows current configuration of g2p tokenizer (token count increased to 1024), both pretokenization and actual tokenization, compared to parlerTTS -- mostly showing word separation and punctuation behavior

>>41423224
>>41423388
Well, Blaze is a often a male name, and "High" is one letter away from "Hugh" which is also male. Makes sense.

File: Screenshot 2024-09-16 182341.png (140 KB, 1026x1207)

140 KB PNG

>>41423490
I also just realized that this isn't the approach you proposed. Here's what it looks like when you let whitespace be anywhere in the token (i.e. no Metaspace/Whitespace)

File: Screenshot 2024-09-16 184754.png (92 KB, 1051x633)

92 KB PNG

>>41423599
https://github.com/huggingface/tokenizers/pull/909
https://github.com/huggingface/tokenizers/releases
https://github.com/huggingface/transformers/pull/32535
Apparently tokenizers didn't support spaces in the middle of tokens until last month, and transformers isn't able to bump their tokenizers version yet?

>>41423388
>>41423538
Oh, my bad. I thought these were mistakes made by analyzing the voice rather than the name.

>>41423711
Huh. Yeah, the Huggingface BPE config format would break if there were spaces in vocabulary since merges are specified as pairs of space-delineated token strings. It looks like the Llama 3.1 tokenizer replace all spaces with "Ġ" (unicode 288) in the token vocabulary.
>tokenizer.tokenize(' ')
>['Ġ']
But it also maps Ġ to something else.
>tokenizer.tokenize('Ġ')
>['Äł']
>tokenizer.tokenize('Äł')
>['ÃĦ', 'ÅĤ']
>... # there's a whole chain of remappings
If I use the underlying tokenizer's encode function, it transparently replaces spaces with Ġ.
>tokenizer._tokenizer.encode(' ').tokens
>['<|begin_of_text|>', 'Ġ']
And the underlying tokenizer's decode function transparently converts Ġ back to spaces.
I'm not seeing anything obvious in the llama tokenizer's config file that would be responsible for this. Same for HF's PreTrainedTokenizerFast, which is used to load the Llama tokenizer, or the Tokenizer class, which is the type of the underlying _tokenizer class, so it might be some hidden functionality in the HuggingFace's BPE implementation. Llama 3.1 is using the BPE tokenizer though, so it is somehow possible with that. I'll have to look into this more to see how to get something similar.
It might be easier to just replace spaces with the metaspace character for the phoneme tokenizer. It won't be an issue for the llama tokenizer since it already handles this.
>>41423490
>On using llama: would there be any benefit in retaining some of the old ParlerTTS token IDs?
I don't see a reason to retain the ParlerTTS token IDs. If you wanted to continue training a model that was already trained with ParlerTTS, then yes, but for training new models, no.

I'm going to be busy with other things for the rest of tonight and most (maybe all) of tomorrow. After that, I can work on that tokenizer cleanup class. Let me know if you end up working on the hybrid tokenizer implementation before then, otherwise I might start on that too.

>>41424644
I don't know the details of the HF/Llama tokenizers, but replacing space with Ġ goes back at least as far as GPT2's BPE, so maybe it was just copied from there
https://github.com/karpathy/minGPT/blob/37baab71b9abea1b76ab957409a1cc2fbfba8a26/mingpt/bpe.py#L29-L30

File: Screenshot 2024-09-17 005102.png (89 KB, 1071x922)

89 KB PNG

>>41424644
>It looks like the Llama 3.1 tokenizer replace all spaces with "Ġ" (unicode 288) in the token vocabulary.
I think this is the ByteLevel pretokenizer step; it also has a decoder that converts it back (I copied it in picrel). FWIW it looks like a tokenizer produced this way will actually save without an error so maybe it's fine?

>I don't see a reason to retain the ParlerTTS token IDs. If you wanted to continue training a model that was already trained with ParlerTTS, then yes, but for training new models, no.
I was considering training on top of the existing finetune or the original ParlerTTS mini checkpoint, but maybe that's a bad idea?

>>41424717
That is a really ugly hack for a problem that really should have been solved with more type-consistent formatting.
Makes sense.
>>41424718
It looks like ByteLevel can take two arguments:
- add_prefix_space is probably fine with its current/default value True.
- I think use_regex should be False to avoid forcibly splitting tokens on spaces.
Otherwise, it looks good!

>I was considering training on top of the existing finetune or the original ParlerTTS mini checkpoint, but maybe that's a bad idea?
If you're swapping out the ParlerTTS tokenizer with the Llama tokenizer, we'd have to map the token ids over. It probably wouldn't be that difficult, as long as most of ParlerTTS's tokens are in the Llama vocabulary, which I expect they will be. Regardless of which one you end up using (ParlerTTS vs Llama), we'll need to modify the vocabulary to get rid of poorly trained tokens, and the extra work of mapping tokens should be small.
I don't expect that training on top of the existing checkpoint will cause problems.

>>41424873
OK. I think I will retain the original ParlerTTS tokenizer for english
Will try to work on hybrid tomorrow

File: 1702555648441938.gif (3.8 MB, 502x502)

3.8 MB GIF

Good morning bump

StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusion
https://arxiv.org/abs/2409.10058
>The rapid development of large-scale text-to-speech (TTS) models has led to significant advancements in modeling diverse speaker prosody and voices. However, these models often face issues such as slow inference speeds, reliance on complex pre-trained neural codec representations, and difficulties in achieving naturalness and high similarity to reference speakers. To address these challenges, this work introduces StyleTTS-ZS, an efficient zero-shot TTS model that leverages distilled time-varying style diffusion to capture diverse speaker identities and prosodies. We propose a novel approach that represents human speech using input text and fixed-length time-varying discrete style codes to capture diverse prosodic variations, trained adversarially with multi-modal discriminators. A diffusion model is then built to sample this time-varying style code for efficient latent diffusion. Using classifier-free guidance, StyleTTS-ZS achieves high similarity to the reference speaker in the style diffusion process. Furthermore, to expedite sampling, the style diffusion model is distilled with perceptual loss using only 10k samples, maintaining speech quality and similarity while reducing inference speed by 90%. Our model surpasses previous state-of-the-art large-scale zero-shot TTS models in both naturalness and similarity, offering a 10-20 faster sampling speed, making it an attractive alternative for efficient large-scale zero-shot TTS systems.
https://github.com/yl4579/StyleTTS-ZS

File: Gemini_Generated_Image_am(...).jpg (388 KB, 1536x1536)

388 KB JPG

>>41425827
Oh, wow! That seems quite promising from the examples ( https://styletts-zs.github.io/ ) and surprising not very noisy nor distant from the reference voice. Very curious to hear how well (or poorly) it works with pony voices, I'll try and test it myself if I can in a couple days, whenever I'm likely to get enough free time to do so.

>>41364782
Hey faggots. What AI do you use to read entire fanfics?

File: 4CC Autumn Babby Cup Logo.png (196 KB, 612x704)

196 KB PNG

>>41423359
>https://implying.fun/video/summer24/2024-07-19/
Here's a video from last summer. At 1:30 the intro starts that was used for Summer.

>https://www.youtube.com/watch?v=jn_ND9sYe5k
An idea we had was to use the S1E01 opener and reframe it as a battle between the streamers of the Elite Cup versus the Babby cup, ince the Babby cup is seen as the lesser tournament only meant as a stepping stone for teams to promote to the elite. That way we could use most of the source video and only rely on some minor editing with faces/board icons pasted over the ponies, with narration of a script spoken by Celestia via RVC.

For the Credits that are essentially played at the end of each cup day, we were thinking of something based on the Running of the Leaves, since this is for the Autumn Babby Cup. It could be as simply as just having some clips from the episode with the credits scrolling down with music playing in the background.

We're trying to keep the scope low since time is running out. No dates have been finalized, but cup likely kicks off somewhere mid-October.

https://vocaroo.com/1gBlpYI8MRWb
Why does this shit cost an arm and a leg ... might as well make a hundred accounts instead of having to pay $100 for a measly 10 hours of audio. This shit can't replicate the emotion required to do convincing AVGN reviews.

>>41425827
>>41425861
Dunno, maybe it just me but it sounds either on the same quality or even slightly worse than what's already exists.

>>41423346
Marketing wise that's probably true.

>>41424885
Update: I can't find any documentation on how/if you're actually supposed to subclass a PreTrainedTokenizer, particularly in a way that's compatible with AutoTokenizer. I've gotten as far as successfully registering the class with AutoTokenizer, but actually instantiating it seems to depend on a bunch of method implementations that don't seem to be in a spec anywhere. Might be easier just to make something that mimics the AutoTokenizer API to the extent it's used in ParlerTTS training? But obviously less flexible with other projects.

>>41420275
Maybe having line breaks get treated like punctuation?

>>41425543
Good morning in a different part of the world.

>>41427365
Why does it need to be compatible with AutoTokenizer? Why not just use your PreTrainedTokenizer class? It has a from_pretrained function that can pull a tokenizer model from an HF repo. You'd have to commit the new tokenizer class and set trust_remote_code=True in from_pretrained. I don't think there's a way around that since I don't think there's a way to get a consistent hybrid tokenizer using only existing HF tokenizer classes.
I'm not sure how far you got with the hybrid tokenizer, but I think this mostly works: https://ponepaste.org/10355. When decoding, it adds extra spaces between tokens, and I haven't figured out how to get rid of that.

I'll see if I can get the code to clean up the grapheme and phoneme tokenizers based on what we discussed so far. I think that's just:
- Remove undertrained and useless tokens from both the llama 3.1 and arpabet tokenizers.
- Update the llama 3.1 tokenizer to use token ids from ParlerTTS where possible.
- I'll use the pony speech dataset to determine which tokens will be well-trained.