AudioGen: Textually Guided Audio Generation | Text To Audio | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I do a deep dive of the recent "AudioGen: Textually Guided Audio Generation | Paper Explained" paper that introduced text-guided audio synthesis.

In a nutshell, it's the VQ-VAE/GAN idea applied to the audio modality.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://felixkreuk.github.io/text2audio_arxiv_samples/paper.pdf
✅ Site: https://felixkreuk.github.io/text2audio_arxiv_samples/

✅ 3B1B on Fourier transform: https://www.youtube.com/watch?v=spUNpyF58BY
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
01:17 Why is text-to-audio hard?
02:51 Comparison with VQ-GAN
05:15 Comparison with SoundStream
06:20 AudioGen overview
09:10 Deep dive: audio representation, LSTM
14:05 Losses explained
17:40 Complex-valued STFTs
21:57 Audio Language Modeling
23:37 Multi-stream audio inputs
25:32 Data and augmentations
29:05 Results
35:28 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#audiogen #audiosynthesis #multimodal
 What's up guys? In this video I'm covering AudioGen, textually guided audio generation. The idea here is this paper is doing the same thing to audio what Dali2, Mid Journey, Stable Diffusion and other models have been doing to images. I.e. it's doing the text to audio synthesis. And the other models I mentioned were doing the text to image synthesis. Okay, so let's hear the samples they have here and then we can dig into the actual paper. The code is still not released so I might cover the... I might do a code walkthrough in one of the next videos or I might not as well. So let's hear this sample.... Okay, so one thing you might have noticed is that the speech, the human speech was unintelligible. The reason for that is because they've done some filtering where they removed all of those recordings where the speech tag was present. And by doing that they avoided the class imbalance problem but as you can see they've lost the intelligibility of human speech, which kind of sucks. So yeah, let's dig into the actual paper, let's see the details. I'm gonna first give you a high level overview. I'm gonna compare this paper with VQGAN and the sound stream papers and then we can see the actual details. So let's... yeah, so first the paper title is Audio Gen Textually Guided Audio Generation from the teams of MetaAI and the Hebrew University of Jerusalem. A couple of things worth mentioning is that... my gut feeling is that the difficulty level of the audio synthesis lies somewhere in between text to image generation, so image synthesis and text to video, so papers such as Make a Video recently published by MetaAI as well. So I think it's probably way closer to the text to image level of difficulty, but that's my gut feeling. So here are some of the reasons they mentioned why this is an inherently difficult task. One of them is the lack of big datasets and we did see that OpenAI Whisper did solve that particular problem for the case of audio transcription and translation. But here they're still struggling, so they didn't actually create a new dataset for this particular task. So they say here, scarce text annotations impose another constraint limiting the ability to scale models. So that's precisely what I've said. And then the second thing is, well, because we're dealing with audio, which depending on the sampling frequency can be very long, they have the problem of just modeling long range dependencies, which is a common problem we are already familiar with. So they say finally modeling high-field audio requires encoding audio at high sampling rate, leading to extremely long sequences. There are other problems with audio, but those are some worth mentioning. OK, so I mentioned VQGAN, I mentioned this sound stream. Let me first explain, give you some background. So I did cover the VQGAN model that was introduced in the Staming Transformers for High Resolution Image Synthesis paper months ago. So I have a whole playlist on where I've covered VQGANs, VQVAEs, Dali, Glide, various text to image models. So if you like the background knowledge, I encourage you to check out that playlist. But having said that, I think this video is going to be fairly self-contained as well. So let me give you a quick high-level overview. So why so basically what what AudioGen has done is repurposed VQGAN to audio modality. That would be like a simple description of what this paper is doing. So the VQGAN had this two-stage approach. The first thing you do is you train this basically, oops, you train this auto-encoding structure. So you have this auto-encoding structure here. And after you do that, then you train, you fit an autoregressive transformer on top of the latent space of these discrete representations here. So that's kind of the high-level two-stage structure that VQGAN used, which this AudioGen paper uses as well. So why do we have this encoding, decoding structure? Well, the idea is to compress the representation starting from a high resolution, high dimensionality image and go into a lower dimensional latent space where you have these basically discrete vectors. So you have this codebook table and you basically fetch the vectors that are output of this encoder and you find the closest vectors in the table here. And then you basically find the index of that closest vector. And that's why you end up with this grid here with tokens. Once you have the tokens and it's fairly trivial to basically fit an autoregressive model such as transformer to learn how to generate then later novel images unconditionally or conditionally in case you prepend text. So, yeah, also the losses they use to train this auto-encoding structure. So adversarial losses, reconstruction losses, all of those will be reused in the AudioGen paper as we'll soon see. OK, so that's a high-level overview there. So now let's get to the SoundStream paper. So SoundStream was basically inspired by VQGen and they repurposed it to the audio modality. So you can see here a very similar structure. So here instead you have a raw audio instead of an image. You pass it through this auto-encoding structure. Again, have this quantization and they did introduce a novelty here, a thing called residual vector quantization. I'll mention a bit later more about how that thing works. Although it's not that highly relevant for understanding of the AudioGen paper because AudioGen actually is not using it as far as I understood. So as you can see, the idea is similar. You reconstruct the audio raw waveform here and then you again have the adversarial component and you also have the reconstruction component. So that's kind of the high-level overview. You can see it's a similar structure to the VQGen first stage part of the VQGen. OK, so now let me show you the actual architecture of AudioGen. Again, you'll see it's very similar. So we have the auto-encoding structure. We have the raw waveform here. We pass it through the structure. We get the reconstructed waveform and then they have a couple of losses. So we see here the LT which stands for loss in the time domain, which means you basically do, I think that you do L1 between the raw sequences. We'll see how audio is represented, but basically it's an array. In case you have a single audio channel and not the stereo channels, then you just have a single array of numbers which represent your audio file. And then they also do the LF. LF is just the loss in the frequency domain where they convert the audio file into male spectrograms and then they do weighted average of L1 and L2 losses. We'll see the details a bit later, but that's the idea. So all of those kind of enforce the accurate reconstruction of the output signal. So you want to make sure that this output signal is as close as possible as this one in the, well, in both the time and frequency sense of the word. And finally, they have this multiscale STFT. STFT stands for short time Fourier transform. So basically what they do here is they pass complex valued spectrograms and then they do the adversarial loss. If you're confused, we'll get to details a bit later. So once you train this model here, then you can grab the components such as the encoder. So as you can see here, this encoder here corresponds to this structure here. We have the decoder here corresponding to this structure here. And we have the text encoder for which they use the pre-trained T5 to basically encode text such as a dog is barking in the park into a set of embedding vectors. And then they have some additional details like positional encoding of vectors, et cetera. But basically what they use then is they can use that to condition via cross attention to condition the audio tokens and then generate the set of audio tokens here, which using the decoder, they can return back into the actual waveform into the time domain space. So now once you have this structure, you can also just ignore the text conditioning part and you can just start unconditionally generating audio or you can also condition on text or you can do continuation where we have some part of the audio tokens being present and then you just do the continuation using those tokens and using the text tokens. So hopefully that makes sense. That was a kind of high level overview. Now let's dig into a bit more details behind the paper. So let's start here. Let's start with the I think this is the data part. Okay. So here we generate the dog barks while somebody plays the trumpet in a busy street. In the above prompt, the model must generate three categories of acoustic content with varying degrees of background, foreground, durations and relative position in the temporal axis, the composition of which is highly unlikely to be present in the training set. Okay. So this is another paragraph stating why this is a difficult task. Although I guess the ultimate reason why it's difficult is because we don't have big enough data sets to cover a vast amount of possible use cases and then just basically the model can just interpolate between those and get something that looks novel. Okay. So they say three, they said three categories. So that's like a dog is barking, somebody's playing a trumpet and there is a busy street. So it's like the sound of cars passing by or whatnot. So you have to decide like what's going to be the strength of each of those, what's going to be the foreground versus background. You have to decide what's going to be the duration of those sounds. You have to understand how to kind of place them on the temporal axis. So all of that is the reason why this is kind of intuitively a difficult task, right? Although if we like give the modeling up data and just use a generic structure, which is something that OpenAI Whisper has showed us, then we can expect much better results, I would assume, from this work. So I already mentioned the first and second stage. So I'm going to skip that part. So let's continue and see what else is here. So again, I mentioned the audio representation. So they say here an audio signal of duration D can be represented by sequence. So X belongs to this range, CA times T. So this means, so CA is the number of audio channels. Usually, I think in this work, they just deal with basically, this is always equal to 1. So and then T is just the duration, the number of samples. So if you have D seconds of audio, if the sampling frequency is FSR, so they use 16 kilohertz in their example, then you just multiply those two numbers and you get the actual number of samples in your audio signal. You can obviously always normalize the amplitude of the signal such that it always lies in the minus 1 to 1 range. So basically, you end up with something like this. So let me just draw the representation for you, for those of you who are not as familiar. So you can imagine having a signal like this. So here is the time axis. Here is the amplitude axis. Let me denote it as Y. And so you'll just have some signal, some audio signal like this. And let's say the amplitude is between 1 and minus 1 after you've done some normalization. And so what you then do is you just sample with certain frequency. Maybe you do like, I guess, uniform sampling is the most common one. And you basically end up with a discrete signal of which can be represented as just a ray, a 1D array of numbers between minus 1 and 1. And how people usually draw this is basically as this discrete samples like this. It's going to be something like this and then this and then this. I'm just mimicking the above signal and sampling it at the points which I've denoted by blue dots here. Okay, so that's one representation. People also usually use male spectrograms. I did cover that in the Open Air Whisper paper if you're curious to learn about that representation. But in this video, we're going to use a simple raw 1D representation here. Okay, so then they say the whole system is trained end-to-end to minimize reconstruction loss applied over both time and frequency domain together with the perceptual loss in the form of several discriminators operating in different temporal resolutions. So we've seen that already on this diagram here. Now, let's see the actual details of those losses. Before we go there, just a minor note. The convolutional blocks are followed by two-layer LSTM for sequence modeling. So that's a minor change they have as compared to the sound. What's the name of this paper? I forgot the name because I'll be referencing it. So Soundstream. Okay, let's get back here. So Soundstream does not have this part with the LSTM. They simply use COM1D blocks and that's it. So this is also kind of peculiar that I haven't seen LSTM in a while and I strongly assume this is not needed. You can just have a simple architecture, throw a bunch of data at the problem and that's it. You don't really have to have this. And also I didn't see any ablations why using LSTM instead of, I guess, transformer block. I mean, it's kind of confusing. I'm not sure about that part. Why they've done it and I haven't seen the explanation anywhere. Anyways, let's get back to the losses. Let's see how the losses are constructed. We have the first time domain loss. So we minimize the L1 distance between the target and reconstructed audio over the time domain. So basically L1 between those two signals. That's it. As simple as that. For the frequency domain loss, we use a linear combination between the L1 and L2 losses over the mouse spectrogram using several time scales. Okay, so let's kind of decompose what this means. In practice, they set this alpha to 1 so we can kind of ignore, completely ignore the alpha here. Let me just change the color. So we just have like a equally weighted sum of the L1 norm and L2 norm between the mouse spectrograms. And now the differences. So you can see here. So X is just the original signal. X hat is the reconstructed signal. You grab the mouse spectrograms where the time resolution is basically a function of this I. So they say here. So SI is a 64 bin mouse spectrogram using a normalized STFT with window size of 2 raised to the power of I and hop lengths of 2 raised to the power of I over 4 where E goes from 5 to 11. Okay, so it's just a set and I goes through that set. So I was going to have values of 2 raised to the power of 5, which means then we'll have a window size of 32, I guess. And then you just keep on increasing by 2. So the window sizes will be increasing and the hop lengths will also be increasing, which means the higher the I, the shorter will be the time dimension of the mouse spectrogram, if that makes sense. So basically, you'll end up for, for example, for I equals 5, you'll end up with a particular mouse spectrogram that has the following shape. So it's going to be, as they said, 64, 64 bins for the frequency. And then this is going to be something like whatever, like N. And then for I equals 6, basically, this is going to be N over 2. And then so that means we'll end up with a smaller spectrogram, etc, etc. So those will have different resolution of the features and that kind of helps in practice. So, yeah, and then they just do, as you can see here, this simple loss between those mouse spectrograms. And again, a mouse spectrogram is simply this. I think I have a drawing here. So this signal where a single particular dot on this spectrogram tells you that at this particular point of time T, you have a frequency F with a particular intensity that's described by the color of the spectrogram. So that's the amplitude of that frequency. And that just all comes from the Fourier transform theory. Basically, any signal can be decomposed as a sum of sinusoids and with different phases and different amplitudes. And that's where the complex numbers are going to come in. But let me not bother you with that at this point of time. We'll get there a bit later. Also, 3Blue1Brown has amazing explanation of the Fourier transform. So check out his video. I'm going to link it down in description, actually, in case you're confused about the Fourier transform. Although I do assume most of you do know at least on a high level what it is. OK, let's see the third loss. They say here the MS STFT discriminator is based on identically structured networks operating on multiscaled complex valued STFT where it's real and imaginary parts are concatenated. OK, actually, here are the complex numbers already. So let me quickly describe what it is. So those complex valued STFTs are going to be the same shape as the actual male spectrogram. It's just that the frequency. So this part here is going to be 2x. So it's going to be two times bigger because it has complex numbers and they're just going to concatenate the complex numbers. So let me explain to you what that means. I already explained what this dot means here. So it's a particular sinusoid of a particular frequency and a particular amplitude. You can imagine before we got this spectrogram, we actually had a complex signal so you can represent a complex number in this 2D plane. And so you can imagine we have a complex number here. So maybe this is the I axis and this is the real axis. This is kind of the real and the imaginary also describe also denoted as imaginary axis. And so this is your complex number. Right. And so in order to get this amplitude of this of this particular and by the way, this vector represents a particular sinusoid. Okay. If you just grab the amplitude of this thing. So the let's denote it as a then you end up with the scalar here. That's the amplitude. But we ditch the phase information by doing that by just taking the amplitude. We ignore the phase. So there is the face here. So the face kind of tells you how shifted is this sinusoid compared to the origin point. So if you're this is just basic signal signal processing stuff. So basically, let's say this is your normal sinusoid that has phase shift of zero. If you were to increase introduce a phase shift of let's say like whatever pi over two or something. Then you end up with like a cosine signal, etc, etc. So you basically just change the shift and that shift information is lost when you deal with mel spectrograms. And in order not to lose it, they just concatenate both the real and the imaginary axis and they get a complex valued STFT of spectrogram representation. Hopefully that makes sense. Let me know if you if you understood this part nicely. Okay. So now that we have this representation, what they do is they again just have a multiple temporal resolutions. So they can see you can see here that we use five different scales with STF window lengths of blah, blah, blah, blah, blah, blah, which which will lead to two different representations like in the time domain will be of different size. And then you just pass those representations to these discriminators that they've denoted somewhere on the diagram here. So you're just going to pass those representations here. Out comes a scalar that tells you whether the signal is comes from a real audio or from a fake one. And then it's just your regular adversarial loss. I'm not going to dig into a lot of details there. So you can see here we are forcing this this signal to be as close to one as possible because that means we're tricking the discriminator into thinking that the image is real. I mean when I say image, I mean this spectrogram complex representation. And finally, we have this feature matching loss where they just take the features across the discriminator and do make sure that the L1 difference between the complex spectrogram that corresponds to the real signal and the complex spectrogram that corresponds to the generated signal are as close as possible. And finally, you can see the losses are like this. So the discriminators are trained to minimize this thing. So a sum over k discriminators because we saw we have multiple of those. And then you just make sure that you can see here because we want to minimize it. That means we want to have this dk of facts being as close to one as possible. That's when we're going to minimize this part here. And we want to have this as close as possible to zero because this is a generated signal. And finally, the generator is trained with this complex weighted sum of various loss components. We have the time domain reconstruction loss. We have the frequency domain reconstruction loss. We have the generative generator loss here and we have the feature loss here. Okay, guys, so that's the loss. And then we have the audio language modeling part. So as I said, the text representation is obtained using a pre-trained T5 text encoder. And we just have a simple basically cross entropy here. So we are trying to minimize the cross entropy. So you can see the only potentially confusing part here is so use are the text embedding vectors. These are the audio embedding vectors. And then you can see we do a sum over the TV, which is basically we want to predict the next audio token, which kind of makes sense. And there is also this confusing part with I going from one to N and N as a symbol has not been introduced anywhere in the paper. But just reading it later, I figured out it's basically because of this multi stream approach which they are using, which is not that like important to understand how the paper is working. It's more of a way to go around the problem of having long sequences. And so, yeah, the additional use the classifier pre-guidance. We've seen that concept introduced in the future models. I covered that multiple times. So the idea here is nothing different except that here we just as you can see here. So we have the signal that's unconditional. So we don't have any use, which are the text embedding vectors. And then we have the actual condition signal and then we just create a weighted sum of those two. And then and then we sample from that distribution. Also, the Lee Mini project uses this very same idea. So you can check out that code code base if you're curious to see how it's exactly implemented until the code for this paper is released. And the final detail before showing you quickly the results is this the multi stream audio inputs and they say it here. So in order to generate high quality audio samples, we down sample the raw audio by a factor of 32 because of those calm one thing is they have and then which corresponds to two milliseconds for each audio token, assuming the 16 kilohertz sampling rate. So this requires us to operate over extremely long sequences as each second of audio is represented by 500 tokens to alleviate this problem. We propose a multi stream representation and modeling paradigm. So let me just quickly describe what it is. The basic idea is this. So you start with a super long sequence. This is the in the time domain. And then you basically reduce the dimensionality by 32 X usually. And then you end up with with basically audio tokens. And now when you're training like a transformer on top of this space, you can see this is super long. The transformer is going to struggle. So what I do instead is they do the following. So they just have a much higher down sampling strategy. And so they end up with maybe 2 X or 8 X or whatnot smaller dimensionality here. So it's going to be way smaller. And then that means it's much more feasible for transformers to handle that length. But now the problem is you're obviously by compressing it so much more, you're losing a lot of data information. And so because of that, they have this multi transformer approach where they now have multiple transformers running here and then they average the logits of those transformers. Anyways, I'm going to stop there because it's not that relevant. And even the results are like they lose some performance and they just gain a little bit of speed up. So I'm really not sure how this is helping because even the memory footprint will probably not be smaller because you still have multiple transformers in this case. So, yeah, I'm kind of confused by this whole multi stream approach. And what do we get from it actually? Okay, let's quickly go through the experiments. Couple of things worth mentioning about a data set. So for textual description, we they say we use two types of annotations. The first one is multi label notation. And so we form sentences by concatenating a list of text available per audio sample. For example, if you have three labels for that particular audio clips like dog bark and park that's transformed into dog bark bark. And you can already see the problem with this approach. And then the second type of annotation is natural language captions. And then they say we apply a preprocessing step to better match the class label annotation distribution specifically where we move stop words and numbers. Finally, we'll lemmatize the remaining words. And so a dog is barking at the park is transformed to dog bark bark. And so, again, that's the reason why they are hitting some of the limitations they mentioned here. So it still lacks the understanding of temporal ordering the scene. For example, a dog is barking, then a then a then birds are humming, I guess. It's a typo here. So versus the dog is barking and a bird's humming in the background. So is it like happening in parallel or is it like first the dog is barking and then the birds are humming? And they also mentioned that last as we omit most of the speech samples in our training set, the proposed approach often generates unintelligible speech. I did mention that in the beginning of the video. Anyways, let's get back here. And finally, they end up with four thousand with only four thousand hours of training data because they filter out the speech text. You can see here. As speech is the dominant class in the data, we we filter all samples where the tag or caption contains the word speech to generate a more balanced data set. And as a consequence, they end up with unintelligible speech. They do one more thing. I want to mention the data augmentation part. And this is probably just completely unnecessary if you simply have a bigger data set. Although one thing is like it's not easy to obtain high quality, bigger data sets. So, yeah. So one of the most impressive capabilities of recently proposed generative models is their ability to create unseen object compositions. We know from the text image models that a sentence such as an astronaut riding a horse in space is something we can we can kind of generate, even though we the model probably never has seen such an image. So to achieve similar capabilities regards to audio generation, we propose an augmentation method that fuses pairs of audio samples and their respective text captions, thus creating new concept compositions during training. Formally, given two audio samples X1 and X2 and their respective text captions C1 and C2, we first randomly draw a temporal offset to merge the two audio samples. Next, we draw a random signal to noise ratio in the signal minus five to five. And finally, we mix the audio samples and concatenate the text captions C1 and C2. So that's how they get more data from the existing data points. So basically, you grab let me just change the color here. So you basically grab two signals. You have a signal here and you have a different signal here. And so what you do is you basically decide on how to combine them. So you decide on some particular offset point like maybe here and then you basically do the overlap. So just overlap those two and then additionally you need to decide on the amplitude of this one and the amplitude of this one. That's the idea. Okay, that's the augmentation. Let's see some results. Obviously, they have two baselines. One is using the T5 large, the other one is using T5 bass and obviously the bigger the better. Like across all of these subjective and objective metrics, I'm not going to go into what they actually are. But you can see the numbers are in favor for the large model. Surprise, surprise. They also compare favorably with the stiff sound baseline, which is cool. Let me parse the charts for you here. Let's first focus on the right hand side here. So on the x-axis we have the audio prompt duration. So basically how many audio tokens do we use to prompt the model? And then we are looking on the y-axis at this objective KL divergence metric. In the background, they use some pre-trained audio classifier. It doesn't matter. Like let's just take that as a black box here. And so the differential between two approaches, one is to condition on text and the other one is to not condition text. So the unconditional part, the unconditional case. And so you can see that in the case of conditioning with text, you already with only a couple of audio tokens, you already have a signal that's very close to the ground truth. And you get some improvement the more you add the number of the more you increase the number of audio tokens. On the other side, without conditioning on text, you can see that we have a very big discrepancy between the ground truth and actually generated signal if we don't have any text conditioning. And after a while, which kind of makes sense, if you have enough audio tokens, then the text part becomes less relevant. OK, so let's now focus on the chart on the left. This one might be a bit confusing. So let me try and draw a couple of things before that. So we have two audio samples. We have one audio signal here. I'm going to denote it as as a red signal. And there is some associated text. So there is some associated text with this particular signal. OK, and then we have a different audio signal, a different sample. And what we do is we grab the text of this first sample. So I'm going to know this one is one. I'm going to denote this sample here as two. So what we do is we grab the text here. We pass it through the T5 encoder and we end up with some tokens. So we end up with some text tokens. OK, so here they are. Here are some text tokens. And then what we do here is we pass this one through the encoder. That's the first stage of the audio gen model. And we end up with some tokens here. OK, and so we now pass those into the ALM, into the audio language modeling like model. And we obviously have two conflicting requests. This one is requesting, hey, generate basically generate a signal that corresponds to this text. And this is going to be the ground truth. And we have a different set of tokens which are telling us, hey, generate this particular audio clip. And so we can see on this diagram here on the x-axis as the number of audio tokens increases. So the blue tokens here. Let's now focus on, for example, the purple curve. The purple curve is comparing the generated audio signal with basically this signal here, signal one. And it kind of makes sense that as you're increasing the number of blue tokens, you're getting worse and worse at generating a signal that's close to this one, which kind of makes sense. And vice versa, if we focus on this dashed yellow line, here we're comparing with this signal. So we're using this one as the reference one. And obviously, as we're increasing the number of these blue tokens, we'll be closer and closer to generating something that looks like this original signal here. So again, we are just to to to mind you, we are basically sampling. So we pass all of these in into our ALM, which is just basically a transformer. And we keep on generating these samples one after another. And then we just pass all of that through the decoder that we train in the first stage. So let me just go back here. So we just pass those through this structure here. So best was token through this one. And we decode the original audio signal. Okay. I mean, the generator, not the original one. And then we just do the comparisons. The green line is just like a baseline where where you don't even have the text prompts. You just have a couple of tokens from random signal. And then you're comparing whatever is generated with this blue signal here. Kind of confusing. But the idea is that like after one point five seconds, roughly the contribution from text and the contributions from from the audio tokens become roughly similar. And I'm not quite sure what was the length of these text tokens, because otherwise this is not kind of very informative, because obviously this if the number of text tokens is bigger, then this chart would not probably be at one point five. The intersection would not happen at one point five seconds, if that makes sense. OK, in any case, that's pretty much it here. They show some diagrams how the guidance scale for the in the classifier free guidance influences the results. So the FAD metric and the KL metric and they find that the minimum is around three. So for number three, they get the best rate of across metrics, as usually the guidance scale helps. And finally, they have some table comparing this, the multi stream transformer approach I did mention briefly. And they say the results suggest that increasing the number of streams degrades the performance in both base and large models when compared to a single stream system, which kind of makes sense because we're losing information. But let's see what we gain. So what we gain is you can see here a speed up. We gain like three point six X speed up when we use such an approach. But you can see like the performance drop significantly. That's if you focus on the FAD metric, we get from like two to like eleven, which is probably a big difference. Cool, guys. That's pretty much it for this for this video. So a couple of thoughts here. So just like analyzing open eyes whisper that also deals with speech data. I guess the conclusion here would be we probably like next paper down the line. We won't need the LSTMs. We won't need all of these details when it comes to the model architecture. If you just have a generic architecture that simple, if you have much bigger data set and of higher quality, because as we saw here, we are what they are doing is they are basically concatenating just labels. And thus they are losing a lot of the information that's kind of rich and present in the language. And so by improving the data set, but making it bigger and just using a generic like architecture, I assume that one of the next papers will just have way better results. And yeah, so if you like this video, let me know down in the comments. Subscribe to this channel if you haven't already. And until next time, bye bye.
[{"start": 0.0, "end": 5.6000000000000005, "text": " What's up guys? In this video I'm covering AudioGen, textually guided audio generation."}, {"start": 5.6000000000000005, "end": 10.5, "text": " The idea here is this paper is doing the same thing to audio what"}, {"start": 10.5, "end": 15.5, "text": " Dali2, Mid Journey, Stable Diffusion and other models have been doing to images."}, {"start": 15.5, "end": 18.6, "text": " I.e. it's doing the text to audio synthesis."}, {"start": 18.6, "end": 23.5, "text": " And the other models I mentioned were doing the text to image synthesis."}, {"start": 23.5, "end": 29.6, "text": " Okay, so let's hear the samples they have here and then we can dig into the actual paper."}, {"start": 29.6, "end": 35.2, "text": " The code is still not released so I might cover the... I might do a code walkthrough in one of the next videos"}, {"start": 35.2, "end": 38.800000000000004, "text": " or I might not as well. So let's hear this sample."}, {"start": 38.800000000000004, "end": 49.2, "text": "..."}, {"start": 49.2, "end": 54.900000000000006, "text": " Okay, so one thing you might have noticed is that the speech, the human speech was unintelligible."}, {"start": 54.9, "end": 60.8, "text": " The reason for that is because they've done some filtering where they removed all of those recordings"}, {"start": 60.8, "end": 64.5, "text": " where the speech tag was present."}, {"start": 64.5, "end": 69.3, "text": " And by doing that they avoided the class imbalance problem"}, {"start": 69.3, "end": 74.7, "text": " but as you can see they've lost the intelligibility of human speech, which kind of sucks."}, {"start": 74.7, "end": 78.2, "text": " So yeah, let's dig into the actual paper, let's see the details."}, {"start": 78.2, "end": 81.2, "text": " I'm gonna first give you a high level overview."}, {"start": 81.2, "end": 88.9, "text": " I'm gonna compare this paper with VQGAN and the sound stream papers and then we can see the actual details."}, {"start": 88.9, "end": 94.7, "text": " So let's... yeah, so first the paper title is Audio Gen Textually Guided Audio Generation"}, {"start": 94.7, "end": 99.0, "text": " from the teams of MetaAI and the Hebrew University of Jerusalem."}, {"start": 99.0, "end": 105.5, "text": " A couple of things worth mentioning is that... my gut feeling is that the difficulty level of the audio synthesis"}, {"start": 105.5, "end": 110.7, "text": " lies somewhere in between text to image generation, so image synthesis"}, {"start": 110.7, "end": 116.2, "text": " and text to video, so papers such as Make a Video recently published by MetaAI as well."}, {"start": 116.2, "end": 122.60000000000001, "text": " So I think it's probably way closer to the text to image level of difficulty, but that's my gut feeling."}, {"start": 122.60000000000001, "end": 127.0, "text": " So here are some of the reasons they mentioned why this is an inherently difficult task."}, {"start": 127.0, "end": 133.6, "text": " One of them is the lack of big datasets and we did see that OpenAI Whisper did solve that particular problem"}, {"start": 133.6, "end": 138.4, "text": " for the case of audio transcription and translation."}, {"start": 138.4, "end": 143.5, "text": " But here they're still struggling, so they didn't actually create a new dataset for this particular task."}, {"start": 143.5, "end": 148.5, "text": " So they say here, scarce text annotations impose another constraint limiting the ability to scale models."}, {"start": 148.5, "end": 150.4, "text": " So that's precisely what I've said."}, {"start": 150.4, "end": 156.20000000000002, "text": " And then the second thing is, well, because we're dealing with audio, which depending on the sampling frequency"}, {"start": 156.20000000000002, "end": 160.70000000000002, "text": " can be very long, they have the problem of just modeling long range dependencies,"}, {"start": 160.70000000000002, "end": 163.6, "text": " which is a common problem we are already familiar with."}, {"start": 163.6, "end": 170.79999999999998, "text": " So they say finally modeling high-field audio requires encoding audio at high sampling rate, leading to extremely long sequences."}, {"start": 170.79999999999998, "end": 174.4, "text": " There are other problems with audio, but those are some worth mentioning."}, {"start": 174.4, "end": 178.0, "text": " OK, so I mentioned VQGAN, I mentioned this sound stream."}, {"start": 178.0, "end": 181.6, "text": " Let me first explain, give you some background."}, {"start": 181.6, "end": 190.7, "text": " So I did cover the VQGAN model that was introduced in the Staming Transformers for High Resolution Image Synthesis paper months ago."}, {"start": 190.7, "end": 198.6, "text": " So I have a whole playlist on where I've covered VQGANs, VQVAEs, Dali, Glide, various text to image models."}, {"start": 198.6, "end": 203.39999999999998, "text": " So if you like the background knowledge, I encourage you to check out that playlist."}, {"start": 203.39999999999998, "end": 207.2, "text": " But having said that, I think this video is going to be fairly self-contained as well."}, {"start": 207.2, "end": 210.89999999999998, "text": " So let me give you a quick high-level overview."}, {"start": 210.89999999999998, "end": 219.1, "text": " So why so basically what what AudioGen has done is repurposed VQGAN to audio modality."}, {"start": 219.1, "end": 223.2, "text": " That would be like a simple description of what this paper is doing."}, {"start": 223.2, "end": 227.6, "text": " So the VQGAN had this two-stage approach."}, {"start": 227.6, "end": 235.1, "text": " The first thing you do is you train this basically, oops, you train this auto-encoding structure."}, {"start": 235.1, "end": 238.79999999999998, "text": " So you have this auto-encoding structure here."}, {"start": 238.79999999999998, "end": 247.9, "text": " And after you do that, then you train, you fit an autoregressive transformer on top of the latent space of these discrete representations here."}, {"start": 247.9, "end": 255.0, "text": " So that's kind of the high-level two-stage structure that VQGAN used, which this AudioGen paper uses as well."}, {"start": 255.0, "end": 259.3, "text": " So why do we have this encoding, decoding structure?"}, {"start": 259.3, "end": 265.7, "text": " Well, the idea is to compress the representation starting from a high resolution, high dimensionality image"}, {"start": 265.7, "end": 271.2, "text": " and go into a lower dimensional latent space where you have these basically discrete vectors."}, {"start": 271.2, "end": 279.0, "text": " So you have this codebook table and you basically fetch the vectors that are output of this encoder"}, {"start": 279.0, "end": 282.7, "text": " and you find the closest vectors in the table here."}, {"start": 282.7, "end": 286.4, "text": " And then you basically find the index of that closest vector."}, {"start": 286.4, "end": 289.9, "text": " And that's why you end up with this grid here with tokens."}, {"start": 289.9, "end": 296.09999999999997, "text": " Once you have the tokens and it's fairly trivial to basically fit an autoregressive model such as transformer"}, {"start": 296.1, "end": 303.70000000000005, "text": " to learn how to generate then later novel images unconditionally or conditionally in case you prepend text."}, {"start": 303.70000000000005, "end": 308.70000000000005, "text": " So, yeah, also the losses they use to train this auto-encoding structure."}, {"start": 308.70000000000005, "end": 316.40000000000003, "text": " So adversarial losses, reconstruction losses, all of those will be reused in the AudioGen paper as we'll soon see."}, {"start": 316.40000000000003, "end": 319.70000000000005, "text": " OK, so that's a high-level overview there."}, {"start": 319.70000000000005, "end": 322.40000000000003, "text": " So now let's get to the SoundStream paper."}, {"start": 322.4, "end": 330.29999999999995, "text": " So SoundStream was basically inspired by VQGen and they repurposed it to the audio modality."}, {"start": 330.29999999999995, "end": 332.2, "text": " So you can see here a very similar structure."}, {"start": 332.2, "end": 335.7, "text": " So here instead you have a raw audio instead of an image."}, {"start": 335.7, "end": 337.9, "text": " You pass it through this auto-encoding structure."}, {"start": 337.9, "end": 346.09999999999997, "text": " Again, have this quantization and they did introduce a novelty here, a thing called residual vector quantization."}, {"start": 346.09999999999997, "end": 349.9, "text": " I'll mention a bit later more about how that thing works."}, {"start": 349.9, "end": 354.4, "text": " Although it's not that highly relevant for understanding of the AudioGen paper"}, {"start": 354.4, "end": 358.09999999999997, "text": " because AudioGen actually is not using it as far as I understood."}, {"start": 358.09999999999997, "end": 359.7, "text": " So as you can see, the idea is similar."}, {"start": 359.7, "end": 367.09999999999997, "text": " You reconstruct the audio raw waveform here and then you again have the adversarial component"}, {"start": 367.09999999999997, "end": 369.4, "text": " and you also have the reconstruction component."}, {"start": 369.4, "end": 371.79999999999995, "text": " So that's kind of the high-level overview."}, {"start": 371.8, "end": 380.1, "text": " You can see it's a similar structure to the VQGen first stage part of the VQGen."}, {"start": 380.1, "end": 384.8, "text": " OK, so now let me show you the actual architecture of AudioGen."}, {"start": 384.8, "end": 386.3, "text": " Again, you'll see it's very similar."}, {"start": 386.3, "end": 388.40000000000003, "text": " So we have the auto-encoding structure."}, {"start": 388.40000000000003, "end": 391.1, "text": " We have the raw waveform here."}, {"start": 391.1, "end": 393.1, "text": " We pass it through the structure."}, {"start": 393.1, "end": 398.2, "text": " We get the reconstructed waveform and then they have a couple of losses."}, {"start": 398.2, "end": 403.59999999999997, "text": " So we see here the LT which stands for loss in the time domain,"}, {"start": 403.59999999999997, "end": 410.7, "text": " which means you basically do, I think that you do L1 between the raw sequences."}, {"start": 410.7, "end": 413.9, "text": " We'll see how audio is represented, but basically it's an array."}, {"start": 413.9, "end": 417.4, "text": " In case you have a single audio channel and not the stereo channels,"}, {"start": 417.4, "end": 423.09999999999997, "text": " then you just have a single array of numbers which represent your audio file."}, {"start": 423.09999999999997, "end": 424.7, "text": " And then they also do the LF."}, {"start": 424.7, "end": 430.8, "text": " LF is just the loss in the frequency domain where they convert the audio file into"}, {"start": 430.8, "end": 436.59999999999997, "text": " male spectrograms and then they do weighted average of L1 and L2 losses."}, {"start": 436.59999999999997, "end": 439.0, "text": " We'll see the details a bit later, but that's the idea."}, {"start": 439.0, "end": 444.9, "text": " So all of those kind of enforce the accurate reconstruction of the output signal."}, {"start": 444.9, "end": 451.4, "text": " So you want to make sure that this output signal is as close as possible as this one in the,"}, {"start": 451.4, "end": 455.9, "text": " well, in both the time and frequency sense of the word."}, {"start": 455.9, "end": 458.4, "text": " And finally, they have this multiscale STFT."}, {"start": 458.4, "end": 462.09999999999997, "text": " STFT stands for short time Fourier transform."}, {"start": 462.09999999999997, "end": 467.29999999999995, "text": " So basically what they do here is they pass complex valued spectrograms"}, {"start": 467.29999999999995, "end": 469.5, "text": " and then they do the adversarial loss."}, {"start": 469.5, "end": 472.0, "text": " If you're confused, we'll get to details a bit later."}, {"start": 472.0, "end": 478.59999999999997, "text": " So once you train this model here, then you can grab the components such as the encoder."}, {"start": 478.6, "end": 487.5, "text": " So as you can see here, this encoder here corresponds to this structure here."}, {"start": 487.5, "end": 494.3, "text": " We have the decoder here corresponding to this structure here."}, {"start": 494.3, "end": 501.0, "text": " And we have the text encoder for which they use the pre-trained T5 to basically encode"}, {"start": 501.0, "end": 505.90000000000003, "text": " text such as a dog is barking in the park into a set of embedding vectors."}, {"start": 505.9, "end": 510.7, "text": " And then they have some additional details like positional encoding of vectors, et cetera."}, {"start": 510.7, "end": 517.0, "text": " But basically what they use then is they can use that to condition via cross attention"}, {"start": 517.0, "end": 521.0, "text": " to condition the audio tokens and then generate the set of audio tokens here,"}, {"start": 521.0, "end": 528.9, "text": " which using the decoder, they can return back into the actual waveform into the time domain space."}, {"start": 528.9, "end": 536.5, "text": " So now once you have this structure, you can also just ignore the text conditioning part"}, {"start": 536.5, "end": 541.1, "text": " and you can just start unconditionally generating audio or you can also condition on text"}, {"start": 541.1, "end": 547.1, "text": " or you can do continuation where we have some part of the audio tokens being present"}, {"start": 547.1, "end": 552.1, "text": " and then you just do the continuation using those tokens and using the text tokens."}, {"start": 552.1, "end": 553.1, "text": " So hopefully that makes sense."}, {"start": 553.1, "end": 555.1999999999999, "text": " That was a kind of high level overview."}, {"start": 555.2, "end": 558.9000000000001, "text": " Now let's dig into a bit more details behind the paper."}, {"start": 558.9000000000001, "end": 560.9000000000001, "text": " So let's start here."}, {"start": 560.9000000000001, "end": 563.9000000000001, "text": " Let's start with the I think this is the data part."}, {"start": 563.9000000000001, "end": 569.9000000000001, "text": " Okay. So here we generate the dog barks while somebody plays the trumpet in a busy street."}, {"start": 569.9000000000001, "end": 574.2, "text": " In the above prompt, the model must generate three categories of acoustic content"}, {"start": 574.2, "end": 579.7, "text": " with varying degrees of background, foreground, durations and relative position in the temporal axis,"}, {"start": 579.7, "end": 583.1, "text": " the composition of which is highly unlikely to be present in the training set."}, {"start": 583.1, "end": 587.3000000000001, "text": " Okay. So this is another paragraph stating why this is a difficult task."}, {"start": 587.3000000000001, "end": 592.3000000000001, "text": " Although I guess the ultimate reason why it's difficult is because we don't have big enough data sets"}, {"start": 592.3000000000001, "end": 598.1, "text": " to cover a vast amount of possible use cases and then just basically the model can just interpolate"}, {"start": 598.1, "end": 602.6, "text": " between those and get something that looks novel."}, {"start": 602.6, "end": 607.0, "text": " Okay. So they say three, they said three categories."}, {"start": 607.0, "end": 611.7, "text": " So that's like a dog is barking, somebody's playing a trumpet and there is a busy street."}, {"start": 611.7, "end": 614.2, "text": " So it's like the sound of cars passing by or whatnot."}, {"start": 614.2, "end": 618.4000000000001, "text": " So you have to decide like what's going to be the strength of each of those,"}, {"start": 618.4000000000001, "end": 620.6, "text": " what's going to be the foreground versus background."}, {"start": 620.6, "end": 625.2, "text": " You have to decide what's going to be the duration of those sounds."}, {"start": 625.2, "end": 629.5, "text": " You have to understand how to kind of place them on the temporal axis."}, {"start": 629.5, "end": 635.0, "text": " So all of that is the reason why this is kind of intuitively a difficult task, right?"}, {"start": 635.0, "end": 639.7, "text": " Although if we like give the modeling up data and just use a generic structure,"}, {"start": 639.7, "end": 643.5, "text": " which is something that OpenAI Whisper has showed us,"}, {"start": 643.5, "end": 648.5, "text": " then we can expect much better results, I would assume, from this work."}, {"start": 648.5, "end": 651.3000000000001, "text": " So I already mentioned the first and second stage."}, {"start": 651.3000000000001, "end": 654.0, "text": " So I'm going to skip that part."}, {"start": 654.0, "end": 657.6, "text": " So let's continue and see what else is here."}, {"start": 657.6, "end": 659.8000000000001, "text": " So again, I mentioned the audio representation."}, {"start": 659.8000000000001, "end": 664.0, "text": " So they say here an audio signal of duration D can be represented by sequence."}, {"start": 664.0, "end": 668.2, "text": " So X belongs to this range, CA times T."}, {"start": 668.2, "end": 671.7, "text": " So this means, so CA is the number of audio channels."}, {"start": 671.7, "end": 677.8000000000001, "text": " Usually, I think in this work, they just deal with basically, this is always equal to 1."}, {"start": 677.8000000000001, "end": 681.9000000000001, "text": " So and then T is just the duration, the number of samples."}, {"start": 681.9000000000001, "end": 687.2, "text": " So if you have D seconds of audio, if the sampling frequency is FSR,"}, {"start": 687.2, "end": 689.8000000000001, "text": " so they use 16 kilohertz in their example,"}, {"start": 689.8000000000001, "end": 695.4000000000001, "text": " then you just multiply those two numbers and you get the actual number of samples in your audio signal."}, {"start": 695.4, "end": 698.5, "text": " You can obviously always normalize the amplitude of the signal"}, {"start": 698.5, "end": 701.6999999999999, "text": " such that it always lies in the minus 1 to 1 range."}, {"start": 701.6999999999999, "end": 703.9, "text": " So basically, you end up with something like this."}, {"start": 703.9, "end": 707.8, "text": " So let me just draw the representation for you, for those of you who are not as familiar."}, {"start": 707.8, "end": 709.9, "text": " So you can imagine having a signal like this."}, {"start": 709.9, "end": 712.1, "text": " So here is the time axis."}, {"start": 712.1, "end": 714.3, "text": " Here is the amplitude axis."}, {"start": 714.3, "end": 715.6999999999999, "text": " Let me denote it as Y."}, {"start": 715.6999999999999, "end": 720.4, "text": " And so you'll just have some signal, some audio signal like this."}, {"start": 720.4, "end": 725.6, "text": " And let's say the amplitude is between 1 and minus 1 after you've done some normalization."}, {"start": 725.6, "end": 730.1999999999999, "text": " And so what you then do is you just sample with certain frequency."}, {"start": 730.1999999999999, "end": 735.0, "text": " Maybe you do like, I guess, uniform sampling is the most common one."}, {"start": 735.0, "end": 739.9, "text": " And you basically end up with a discrete signal of which can be represented as just a ray,"}, {"start": 739.9, "end": 742.9, "text": " a 1D array of numbers between minus 1 and 1."}, {"start": 742.9, "end": 750.4, "text": " And how people usually draw this is basically as this discrete samples like this."}, {"start": 750.4, "end": 754.6, "text": " It's going to be something like this and then this and then this."}, {"start": 754.6, "end": 761.5, "text": " I'm just mimicking the above signal and sampling it at the points which I've denoted by blue dots here."}, {"start": 761.5, "end": 762.9, "text": " Okay, so that's one representation."}, {"start": 762.9, "end": 765.0, "text": " People also usually use male spectrograms."}, {"start": 765.0, "end": 769.6, "text": " I did cover that in the Open Air Whisper paper if you're curious to learn about that representation."}, {"start": 769.6, "end": 774.9, "text": " But in this video, we're going to use a simple raw 1D representation here."}, {"start": 774.9, "end": 778.6, "text": " Okay, so then they say the whole system is trained end-to-end to minimize reconstruction loss"}, {"start": 778.6, "end": 782.1, "text": " applied over both time and frequency domain together with the perceptual loss"}, {"start": 782.1, "end": 785.9, "text": " in the form of several discriminators operating in different temporal resolutions."}, {"start": 785.9, "end": 788.6, "text": " So we've seen that already on this diagram here."}, {"start": 788.6, "end": 792.3000000000001, "text": " Now, let's see the actual details of those losses."}, {"start": 792.3000000000001, "end": 795.1, "text": " Before we go there, just a minor note."}, {"start": 795.1, "end": 798.9, "text": " The convolutional blocks are followed by two-layer LSTM for sequence modeling."}, {"start": 798.9, "end": 802.9, "text": " So that's a minor change they have as compared to the sound."}, {"start": 802.9, "end": 803.8, "text": " What's the name of this paper?"}, {"start": 803.8, "end": 807.5, "text": " I forgot the name because I'll be referencing it."}, {"start": 807.5, "end": 808.4, "text": " So Soundstream."}, {"start": 808.4, "end": 810.1, "text": " Okay, let's get back here."}, {"start": 810.1, "end": 814.0, "text": " So Soundstream does not have this part with the LSTM."}, {"start": 814.0, "end": 818.1, "text": " They simply use COM1D blocks and that's it."}, {"start": 818.1, "end": 825.5, "text": " So this is also kind of peculiar that I haven't seen LSTM in a while and I strongly assume this is not needed."}, {"start": 825.5, "end": 831.8, "text": " You can just have a simple architecture, throw a bunch of data at the problem and that's it."}, {"start": 831.8, "end": 833.5, "text": " You don't really have to have this."}, {"start": 833.5, "end": 841.4, "text": " And also I didn't see any ablations why using LSTM instead of, I guess, transformer block."}, {"start": 841.4, "end": 842.8, "text": " I mean, it's kind of confusing."}, {"start": 842.8, "end": 843.8, "text": " I'm not sure about that part."}, {"start": 843.8, "end": 847.0, "text": " Why they've done it and I haven't seen the explanation anywhere."}, {"start": 847.0, "end": 848.8, "text": " Anyways, let's get back to the losses."}, {"start": 848.8, "end": 850.6, "text": " Let's see how the losses are constructed."}, {"start": 850.6, "end": 854.2, "text": " We have the first time domain loss."}, {"start": 854.2, "end": 859.4000000000001, "text": " So we minimize the L1 distance between the target and reconstructed audio over the time domain."}, {"start": 859.4000000000001, "end": 861.8000000000001, "text": " So basically L1 between those two signals."}, {"start": 861.8000000000001, "end": 863.2, "text": " That's it. As simple as that."}, {"start": 863.2, "end": 871.4000000000001, "text": " For the frequency domain loss, we use a linear combination between the L1 and L2 losses over the mouse spectrogram using several time scales."}, {"start": 871.4000000000001, "end": 875.7, "text": " Okay, so let's kind of decompose what this means."}, {"start": 875.7, "end": 882.6, "text": " In practice, they set this alpha to 1 so we can kind of ignore, completely ignore the alpha here."}, {"start": 882.6, "end": 884.6, "text": " Let me just change the color."}, {"start": 884.6, "end": 891.0, "text": " So we just have like a equally weighted sum of the L1 norm and L2 norm between the mouse spectrograms."}, {"start": 891.0, "end": 892.3000000000001, "text": " And now the differences."}, {"start": 892.3000000000001, "end": 893.3000000000001, "text": " So you can see here."}, {"start": 893.3000000000001, "end": 895.3000000000001, "text": " So X is just the original signal."}, {"start": 895.3000000000001, "end": 897.4, "text": " X hat is the reconstructed signal."}, {"start": 897.4, "end": 904.8000000000001, "text": " You grab the mouse spectrograms where the time resolution is basically a function of this I."}, {"start": 904.8000000000001, "end": 906.6, "text": " So they say here."}, {"start": 906.6, "end": 919.8000000000001, "text": " So SI is a 64 bin mouse spectrogram using a normalized STFT with window size of 2 raised to the power of I and hop lengths of 2 raised to the power of I over 4 where E goes from 5 to 11."}, {"start": 919.8000000000001, "end": 922.8000000000001, "text": " Okay, so it's just a set and I goes through that set."}, {"start": 922.8000000000001, "end": 928.8000000000001, "text": " So I was going to have values of 2 raised to the power of 5, which means then we'll have a window size of 32, I guess."}, {"start": 928.8000000000001, "end": 931.3000000000001, "text": " And then you just keep on increasing by 2."}, {"start": 931.3, "end": 945.5999999999999, "text": " So the window sizes will be increasing and the hop lengths will also be increasing, which means the higher the I, the shorter will be the time dimension of the mouse spectrogram, if that makes sense."}, {"start": 945.5999999999999, "end": 957.1999999999999, "text": " So basically, you'll end up for, for example, for I equals 5, you'll end up with a particular mouse spectrogram that has the following shape."}, {"start": 957.2, "end": 962.0, "text": " So it's going to be, as they said, 64, 64 bins for the frequency."}, {"start": 962.0, "end": 966.2, "text": " And then this is going to be something like whatever, like N."}, {"start": 966.2, "end": 972.0, "text": " And then for I equals 6, basically, this is going to be N over 2."}, {"start": 972.0, "end": 976.2, "text": " And then so that means we'll end up with a smaller spectrogram, etc, etc."}, {"start": 976.2, "end": 982.0, "text": " So those will have different resolution of the features and that kind of helps in practice."}, {"start": 982.0, "end": 986.8000000000001, "text": " So, yeah, and then they just do, as you can see here, this simple loss between those mouse spectrograms."}, {"start": 986.8, "end": 990.4, "text": " And again, a mouse spectrogram is simply this."}, {"start": 990.4, "end": 991.8, "text": " I think I have a drawing here."}, {"start": 991.8, "end": 1008.3, "text": " So this signal where a single particular dot on this spectrogram tells you that at this particular point of time T, you have a frequency F with a particular intensity that's described by the color of the spectrogram."}, {"start": 1008.3, "end": 1010.5999999999999, "text": " So that's the amplitude of that frequency."}, {"start": 1010.5999999999999, "end": 1014.3, "text": " And that just all comes from the Fourier transform theory."}, {"start": 1014.3, "end": 1023.0, "text": " Basically, any signal can be decomposed as a sum of sinusoids and with different phases and different amplitudes."}, {"start": 1023.0, "end": 1025.8, "text": " And that's where the complex numbers are going to come in."}, {"start": 1025.8, "end": 1029.3, "text": " But let me not bother you with that at this point of time."}, {"start": 1029.3, "end": 1030.7, "text": " We'll get there a bit later."}, {"start": 1030.7, "end": 1035.2, "text": " Also, 3Blue1Brown has amazing explanation of the Fourier transform."}, {"start": 1035.2, "end": 1036.2, "text": " So check out his video."}, {"start": 1036.2, "end": 1040.8, "text": " I'm going to link it down in description, actually, in case you're confused about the Fourier transform."}, {"start": 1040.8, "end": 1045.1, "text": " Although I do assume most of you do know at least on a high level what it is."}, {"start": 1045.1, "end": 1046.8, "text": " OK, let's see the third loss."}, {"start": 1046.8, "end": 1057.8999999999999, "text": " They say here the MS STFT discriminator is based on identically structured networks operating on multiscaled complex valued STFT where it's real and imaginary parts are concatenated."}, {"start": 1057.8999999999999, "end": 1060.7, "text": " OK, actually, here are the complex numbers already."}, {"start": 1060.7, "end": 1064.2, "text": " So let me quickly describe what it is."}, {"start": 1064.2, "end": 1071.0, "text": " So those complex valued STFTs are going to be the same shape as the actual male spectrogram."}, {"start": 1071.0, "end": 1072.6000000000001, "text": " It's just that the frequency."}, {"start": 1072.6000000000001, "end": 1074.7, "text": " So this part here is going to be 2x."}, {"start": 1074.7, "end": 1083.7, "text": " So it's going to be two times bigger because it has complex numbers and they're just going to concatenate the complex numbers."}, {"start": 1083.7, "end": 1087.4, "text": " So let me explain to you what that means."}, {"start": 1087.4, "end": 1089.4, "text": " I already explained what this dot means here."}, {"start": 1089.4, "end": 1093.8, "text": " So it's a particular sinusoid of a particular frequency and a particular amplitude."}, {"start": 1093.8, "end": 1105.6, "text": " You can imagine before we got this spectrogram, we actually had a complex signal so you can represent a complex number in this 2D plane."}, {"start": 1105.6, "end": 1107.5, "text": " And so you can imagine we have a complex number here."}, {"start": 1107.5, "end": 1111.0, "text": " So maybe this is the I axis and this is the real axis."}, {"start": 1111.0, "end": 1116.0, "text": " This is kind of the real and the imaginary also describe also denoted as imaginary axis."}, {"start": 1116.0, "end": 1117.8999999999999, "text": " And so this is your complex number."}, {"start": 1117.8999999999999, "end": 1122.8, "text": " Right. And so in order to get this amplitude of this of this particular and by the way,"}, {"start": 1122.8, "end": 1125.0, "text": " this vector represents a particular sinusoid."}, {"start": 1125.0, "end": 1129.8, "text": " Okay. If you just grab the amplitude of this thing."}, {"start": 1129.8, "end": 1133.5, "text": " So the let's denote it as a then you end up with the scalar here."}, {"start": 1133.5, "end": 1134.6, "text": " That's the amplitude."}, {"start": 1134.6, "end": 1138.6, "text": " But we ditch the phase information by doing that by just taking the amplitude."}, {"start": 1138.6, "end": 1139.6, "text": " We ignore the phase."}, {"start": 1139.6, "end": 1141.3999999999999, "text": " So there is the face here."}, {"start": 1141.3999999999999, "end": 1147.0, "text": " So the face kind of tells you how shifted is this sinusoid compared to the origin point."}, {"start": 1147.0, "end": 1150.8, "text": " So if you're this is just basic signal signal processing stuff."}, {"start": 1150.8, "end": 1158.8, "text": " So basically, let's say this is your normal sinusoid that has phase shift of zero."}, {"start": 1158.8, "end": 1166.0, "text": " If you were to increase introduce a phase shift of let's say like whatever pi over two or something."}, {"start": 1166.0, "end": 1170.5, "text": " Then you end up with like a cosine signal, etc, etc."}, {"start": 1170.5, "end": 1177.0, "text": " So you basically just change the shift and that shift information is lost when you deal with mel spectrograms."}, {"start": 1177.0, "end": 1187.4, "text": " And in order not to lose it, they just concatenate both the real and the imaginary axis and they get a complex valued STFT of spectrogram representation."}, {"start": 1187.4, "end": 1188.4, "text": " Hopefully that makes sense."}, {"start": 1188.4, "end": 1192.2, "text": " Let me know if you if you understood this part nicely."}, {"start": 1192.2, "end": 1199.4, "text": " Okay. So now that we have this representation, what they do is they again just have a multiple temporal resolutions."}, {"start": 1199.4, "end": 1205.6, "text": " So they can see you can see here that we use five different scales with STF window lengths of blah, blah, blah, blah, blah, blah,"}, {"start": 1205.6, "end": 1211.1999999999998, "text": " which which will lead to two different representations like in the time domain will be of different size."}, {"start": 1211.1999999999998, "end": 1219.1999999999998, "text": " And then you just pass those representations to these discriminators that they've denoted somewhere on the diagram here."}, {"start": 1219.1999999999998, "end": 1221.3999999999999, "text": " So you're just going to pass those representations here."}, {"start": 1221.3999999999999, "end": 1227.6999999999998, "text": " Out comes a scalar that tells you whether the signal is comes from a real audio or from a fake one."}, {"start": 1227.6999999999998, "end": 1230.8999999999999, "text": " And then it's just your regular adversarial loss."}, {"start": 1230.8999999999999, "end": 1232.6, "text": " I'm not going to dig into a lot of details there."}, {"start": 1232.6, "end": 1239.0, "text": " So you can see here we are forcing this this signal to be as close to one as possible"}, {"start": 1239.0, "end": 1243.5, "text": " because that means we're tricking the discriminator into thinking that the image is real."}, {"start": 1243.5, "end": 1249.0, "text": " I mean when I say image, I mean this spectrogram complex representation."}, {"start": 1249.0, "end": 1255.1, "text": " And finally, we have this feature matching loss where they just take the features across the discriminator"}, {"start": 1255.1, "end": 1261.6999999999998, "text": " and do make sure that the L1 difference between the complex spectrogram that corresponds to the real signal"}, {"start": 1261.7, "end": 1267.9, "text": " and the complex spectrogram that corresponds to the generated signal are as close as possible."}, {"start": 1267.9, "end": 1270.7, "text": " And finally, you can see the losses are like this."}, {"start": 1270.7, "end": 1273.4, "text": " So the discriminators are trained to minimize this thing."}, {"start": 1273.4, "end": 1279.3, "text": " So a sum over k discriminators because we saw we have multiple of those."}, {"start": 1279.3, "end": 1283.5, "text": " And then you just make sure that you can see here because we want to minimize it."}, {"start": 1283.5, "end": 1289.9, "text": " That means we want to have this dk of facts being as close to one as possible."}, {"start": 1289.9, "end": 1292.3000000000002, "text": " That's when we're going to minimize this part here."}, {"start": 1292.3000000000002, "end": 1299.3000000000002, "text": " And we want to have this as close as possible to zero because this is a generated signal."}, {"start": 1299.3000000000002, "end": 1305.8000000000002, "text": " And finally, the generator is trained with this complex weighted sum of various loss components."}, {"start": 1305.8000000000002, "end": 1308.9, "text": " We have the time domain reconstruction loss."}, {"start": 1308.9, "end": 1311.4, "text": " We have the frequency domain reconstruction loss."}, {"start": 1311.4, "end": 1317.4, "text": " We have the generative generator loss here and we have the feature loss here."}, {"start": 1317.4, "end": 1320.2, "text": " Okay, guys, so that's the loss."}, {"start": 1320.2, "end": 1322.7, "text": " And then we have the audio language modeling part."}, {"start": 1322.7, "end": 1326.9, "text": " So as I said, the text representation is obtained using a pre-trained T5 text encoder."}, {"start": 1326.9, "end": 1330.6000000000001, "text": " And we just have a simple basically cross entropy here."}, {"start": 1330.6000000000001, "end": 1334.5, "text": " So we are trying to minimize the cross entropy."}, {"start": 1334.5, "end": 1342.1000000000001, "text": " So you can see the only potentially confusing part here is so use are the text embedding vectors."}, {"start": 1342.1000000000001, "end": 1345.9, "text": " These are the audio embedding vectors."}, {"start": 1345.9, "end": 1353.4, "text": " And then you can see we do a sum over the TV, which is basically we want to predict the next audio token,"}, {"start": 1353.4, "end": 1355.0, "text": " which kind of makes sense."}, {"start": 1355.0, "end": 1362.2, "text": " And there is also this confusing part with I going from one to N and N as a symbol has not been introduced anywhere in the paper."}, {"start": 1362.2, "end": 1367.8000000000002, "text": " But just reading it later, I figured out it's basically because of this multi stream approach which they are using,"}, {"start": 1367.8000000000002, "end": 1371.6000000000001, "text": " which is not that like important to understand how the paper is working."}, {"start": 1371.6, "end": 1377.6, "text": " It's more of a way to go around the problem of having long sequences."}, {"start": 1377.6, "end": 1381.6999999999998, "text": " And so, yeah, the additional use the classifier pre-guidance."}, {"start": 1381.6999999999998, "end": 1384.8999999999999, "text": " We've seen that concept introduced in the future models."}, {"start": 1384.8999999999999, "end": 1386.5, "text": " I covered that multiple times."}, {"start": 1386.5, "end": 1392.1999999999998, "text": " So the idea here is nothing different except that here we just as you can see here."}, {"start": 1392.1999999999998, "end": 1395.0, "text": " So we have the signal that's unconditional."}, {"start": 1395.0, "end": 1398.3999999999999, "text": " So we don't have any use, which are the text embedding vectors."}, {"start": 1398.4, "end": 1404.4, "text": " And then we have the actual condition signal and then we just create a weighted sum of those two."}, {"start": 1404.4, "end": 1407.4, "text": " And then and then we sample from that distribution."}, {"start": 1407.4, "end": 1410.5, "text": " Also, the Lee Mini project uses this very same idea."}, {"start": 1410.5, "end": 1417.2, "text": " So you can check out that code code base if you're curious to see how it's exactly implemented until the code for this paper is released."}, {"start": 1417.2, "end": 1424.7, "text": " And the final detail before showing you quickly the results is this the multi stream audio inputs and they say it here."}, {"start": 1424.7, "end": 1440.9, "text": " So in order to generate high quality audio samples, we down sample the raw audio by a factor of 32 because of those calm one thing is they have and then which corresponds to two milliseconds for each audio token, assuming the 16 kilohertz sampling rate."}, {"start": 1440.9, "end": 1448.2, "text": " So this requires us to operate over extremely long sequences as each second of audio is represented by 500 tokens to alleviate this problem."}, {"start": 1448.2, "end": 1451.5, "text": " We propose a multi stream representation and modeling paradigm."}, {"start": 1451.5, "end": 1454.3, "text": " So let me just quickly describe what it is."}, {"start": 1454.3, "end": 1455.3, "text": " The basic idea is this."}, {"start": 1455.3, "end": 1459.3, "text": " So you start with a super long sequence."}, {"start": 1459.3, "end": 1462.2, "text": " This is the in the time domain."}, {"start": 1462.2, "end": 1467.6, "text": " And then you basically reduce the dimensionality by 32 X usually."}, {"start": 1467.6, "end": 1472.1, "text": " And then you end up with with basically audio tokens."}, {"start": 1472.1, "end": 1478.3, "text": " And now when you're training like a transformer on top of this space, you can see this is super long."}, {"start": 1478.3, "end": 1479.8999999999999, "text": " The transformer is going to struggle."}, {"start": 1479.8999999999999, "end": 1483.1, "text": " So what I do instead is they do the following."}, {"start": 1483.1, "end": 1488.5, "text": " So they just have a much higher down sampling strategy."}, {"start": 1488.5, "end": 1494.1, "text": " And so they end up with maybe 2 X or 8 X or whatnot smaller dimensionality here."}, {"start": 1494.1, "end": 1495.3999999999999, "text": " So it's going to be way smaller."}, {"start": 1495.3999999999999, "end": 1499.3, "text": " And then that means it's much more feasible for transformers to handle that length."}, {"start": 1499.3, "end": 1505.3999999999999, "text": " But now the problem is you're obviously by compressing it so much more, you're losing a lot of data information."}, {"start": 1505.4, "end": 1514.5, "text": " And so because of that, they have this multi transformer approach where they now have multiple transformers running here and then they average the logits of those transformers."}, {"start": 1514.5, "end": 1517.1000000000001, "text": " Anyways, I'm going to stop there because it's not that relevant."}, {"start": 1517.1000000000001, "end": 1523.6000000000001, "text": " And even the results are like they lose some performance and they just gain a little bit of speed up."}, {"start": 1523.6000000000001, "end": 1532.2, "text": " So I'm really not sure how this is helping because even the memory footprint will probably not be smaller because you still have multiple transformers in this case."}, {"start": 1532.2, "end": 1535.2, "text": " So, yeah, I'm kind of confused by this whole multi stream approach."}, {"start": 1535.2, "end": 1537.2, "text": " And what do we get from it actually?"}, {"start": 1537.2, "end": 1540.5, "text": " Okay, let's quickly go through the experiments."}, {"start": 1540.5, "end": 1543.4, "text": " Couple of things worth mentioning about a data set."}, {"start": 1543.4, "end": 1547.6000000000001, "text": " So for textual description, we they say we use two types of annotations."}, {"start": 1547.6000000000001, "end": 1550.6000000000001, "text": " The first one is multi label notation."}, {"start": 1550.6000000000001, "end": 1556.4, "text": " And so we form sentences by concatenating a list of text available per audio sample."}, {"start": 1556.4, "end": 1564.4, "text": " For example, if you have three labels for that particular audio clips like dog bark and park that's transformed into dog bark bark."}, {"start": 1564.4, "end": 1566.9, "text": " And you can already see the problem with this approach."}, {"start": 1566.9, "end": 1570.0, "text": " And then the second type of annotation is natural language captions."}, {"start": 1570.0, "end": 1577.7, "text": " And then they say we apply a preprocessing step to better match the class label annotation distribution specifically where we move stop words and numbers."}, {"start": 1577.7, "end": 1579.9, "text": " Finally, we'll lemmatize the remaining words."}, {"start": 1579.9, "end": 1585.0, "text": " And so a dog is barking at the park is transformed to dog bark bark."}, {"start": 1585.0, "end": 1591.6000000000001, "text": " And so, again, that's the reason why they are hitting some of the limitations they mentioned here."}, {"start": 1591.6, "end": 1596.0, "text": " So it still lacks the understanding of temporal ordering the scene."}, {"start": 1596.0, "end": 1600.8999999999999, "text": " For example, a dog is barking, then a then a then birds are humming, I guess."}, {"start": 1600.8999999999999, "end": 1601.8, "text": " It's a typo here."}, {"start": 1601.8, "end": 1605.8999999999999, "text": " So versus the dog is barking and a bird's humming in the background."}, {"start": 1605.8999999999999, "end": 1611.8, "text": " So is it like happening in parallel or is it like first the dog is barking and then the birds are humming?"}, {"start": 1611.8, "end": 1617.6, "text": " And they also mentioned that last as we omit most of the speech samples in our training set, the proposed approach often generates unintelligible speech."}, {"start": 1617.6, "end": 1620.1999999999998, "text": " I did mention that in the beginning of the video."}, {"start": 1620.2, "end": 1622.3, "text": " Anyways, let's get back here."}, {"start": 1622.3, "end": 1631.7, "text": " And finally, they end up with four thousand with only four thousand hours of training data because they filter out the speech text."}, {"start": 1631.7, "end": 1633.7, "text": " You can see here."}, {"start": 1633.7, "end": 1643.6000000000001, "text": " As speech is the dominant class in the data, we we filter all samples where the tag or caption contains the word speech to generate a more balanced data set."}, {"start": 1643.6000000000001, "end": 1647.4, "text": " And as a consequence, they end up with unintelligible speech."}, {"start": 1647.4, "end": 1650.4, "text": " They do one more thing. I want to mention the data augmentation part."}, {"start": 1650.4, "end": 1654.5, "text": " And this is probably just completely unnecessary if you simply have a bigger data set."}, {"start": 1654.5, "end": 1659.5, "text": " Although one thing is like it's not easy to obtain high quality, bigger data sets."}, {"start": 1659.5, "end": 1668.0, "text": " So, yeah. So one of the most impressive capabilities of recently proposed generative models is their ability to create unseen object compositions."}, {"start": 1668.0, "end": 1675.7, "text": " We know from the text image models that a sentence such as an astronaut riding a horse in space is something we can we can kind of generate,"}, {"start": 1675.7, "end": 1679.4, "text": " even though we the model probably never has seen such an image."}, {"start": 1679.4, "end": 1682.6000000000001, "text": " So to achieve similar capabilities regards to audio generation,"}, {"start": 1682.6000000000001, "end": 1689.1000000000001, "text": " we propose an augmentation method that fuses pairs of audio samples and their respective text captions,"}, {"start": 1689.1000000000001, "end": 1692.7, "text": " thus creating new concept compositions during training."}, {"start": 1692.7, "end": 1699.7, "text": " Formally, given two audio samples X1 and X2 and their respective text captions C1 and C2,"}, {"start": 1699.7, "end": 1704.9, "text": " we first randomly draw a temporal offset to merge the two audio samples."}, {"start": 1704.9, "end": 1708.8000000000002, "text": " Next, we draw a random signal to noise ratio in the signal minus five to five."}, {"start": 1708.8000000000002, "end": 1713.2, "text": " And finally, we mix the audio samples and concatenate the text captions C1 and C2."}, {"start": 1713.2, "end": 1718.6000000000001, "text": " So that's how they get more data from the existing data points."}, {"start": 1718.6000000000001, "end": 1722.9, "text": " So basically, you grab let me just change the color here."}, {"start": 1722.9, "end": 1725.2, "text": " So you basically grab two signals."}, {"start": 1725.2, "end": 1731.1000000000001, "text": " You have a signal here and you have a different signal here."}, {"start": 1731.1, "end": 1735.1999999999998, "text": " And so what you do is you basically decide on how to combine them."}, {"start": 1735.1999999999998, "end": 1741.1, "text": " So you decide on some particular offset point like maybe here and then you basically do the overlap."}, {"start": 1741.1, "end": 1747.3999999999999, "text": " So just overlap those two and then additionally you need to decide on the amplitude of this one and the amplitude of this one."}, {"start": 1747.3999999999999, "end": 1748.6, "text": " That's the idea."}, {"start": 1748.6, "end": 1750.6, "text": " Okay, that's the augmentation."}, {"start": 1750.6, "end": 1753.6999999999998, "text": " Let's see some results."}, {"start": 1753.6999999999998, "end": 1757.5, "text": " Obviously, they have two baselines."}, {"start": 1757.5, "end": 1763.6, "text": " One is using the T5 large, the other one is using T5 bass and obviously the bigger the better."}, {"start": 1763.6, "end": 1769.1, "text": " Like across all of these subjective and objective metrics, I'm not going to go into what they actually are."}, {"start": 1769.1, "end": 1772.5, "text": " But you can see the numbers are in favor for the large model."}, {"start": 1772.5, "end": 1774.1, "text": " Surprise, surprise."}, {"start": 1774.1, "end": 1778.3, "text": " They also compare favorably with the stiff sound baseline, which is cool."}, {"start": 1778.3, "end": 1781.2, "text": " Let me parse the charts for you here."}, {"start": 1781.2, "end": 1783.9, "text": " Let's first focus on the right hand side here."}, {"start": 1783.9, "end": 1787.7, "text": " So on the x-axis we have the audio prompt duration."}, {"start": 1787.7, "end": 1792.5, "text": " So basically how many audio tokens do we use to prompt the model?"}, {"start": 1792.5, "end": 1799.6000000000001, "text": " And then we are looking on the y-axis at this objective KL divergence metric."}, {"start": 1799.6000000000001, "end": 1802.5, "text": " In the background, they use some pre-trained audio classifier."}, {"start": 1802.5, "end": 1803.1000000000001, "text": " It doesn't matter."}, {"start": 1803.1000000000001, "end": 1805.9, "text": " Like let's just take that as a black box here."}, {"start": 1805.9, "end": 1813.1000000000001, "text": " And so the differential between two approaches, one is to condition on text and the other one is to not condition text."}, {"start": 1813.1, "end": 1816.3999999999999, "text": " So the unconditional part, the unconditional case."}, {"start": 1816.3999999999999, "end": 1824.1, "text": " And so you can see that in the case of conditioning with text, you already with only a couple of audio tokens,"}, {"start": 1824.1, "end": 1828.0, "text": " you already have a signal that's very close to the ground truth."}, {"start": 1828.0, "end": 1834.8999999999999, "text": " And you get some improvement the more you add the number of the more you increase the number of audio tokens."}, {"start": 1834.8999999999999, "end": 1838.1999999999998, "text": " On the other side, without conditioning on text,"}, {"start": 1838.2, "end": 1846.4, "text": " you can see that we have a very big discrepancy between the ground truth and actually generated signal if we don't have any text conditioning."}, {"start": 1846.4, "end": 1854.6000000000001, "text": " And after a while, which kind of makes sense, if you have enough audio tokens, then the text part becomes less relevant."}, {"start": 1854.6000000000001, "end": 1858.1000000000001, "text": " OK, so let's now focus on the chart on the left."}, {"start": 1858.1000000000001, "end": 1859.5, "text": " This one might be a bit confusing."}, {"start": 1859.5, "end": 1864.1000000000001, "text": " So let me try and draw a couple of things before that."}, {"start": 1864.1000000000001, "end": 1865.8, "text": " So we have two audio samples."}, {"start": 1865.8, "end": 1869.3, "text": " We have one audio signal here."}, {"start": 1869.3, "end": 1871.7, "text": " I'm going to denote it as as a red signal."}, {"start": 1871.7, "end": 1873.3999999999999, "text": " And there is some associated text."}, {"start": 1873.3999999999999, "end": 1876.5, "text": " So there is some associated text with this particular signal."}, {"start": 1876.5, "end": 1882.2, "text": " OK, and then we have a different audio signal, a different sample."}, {"start": 1882.2, "end": 1886.0, "text": " And what we do is we grab the text of this first sample."}, {"start": 1886.0, "end": 1888.0, "text": " So I'm going to know this one is one."}, {"start": 1888.0, "end": 1891.2, "text": " I'm going to denote this sample here as two."}, {"start": 1891.2, "end": 1894.1, "text": " So what we do is we grab the text here."}, {"start": 1894.1, "end": 1898.3999999999999, "text": " We pass it through the T5 encoder and we end up with some tokens."}, {"start": 1898.3999999999999, "end": 1899.8, "text": " So we end up with some text tokens."}, {"start": 1899.8, "end": 1902.1999999999998, "text": " OK, so here they are."}, {"start": 1902.1999999999998, "end": 1903.6, "text": " Here are some text tokens."}, {"start": 1903.6, "end": 1907.8, "text": " And then what we do here is we pass this one through the encoder."}, {"start": 1907.8, "end": 1910.6999999999998, "text": " That's the first stage of the audio gen model."}, {"start": 1910.6999999999998, "end": 1913.3999999999999, "text": " And we end up with some tokens here."}, {"start": 1913.3999999999999, "end": 1921.1999999999998, "text": " OK, and so we now pass those into the ALM, into the audio language modeling like model."}, {"start": 1921.2, "end": 1924.9, "text": " And we obviously have two conflicting requests."}, {"start": 1924.9, "end": 1930.2, "text": " This one is requesting, hey, generate basically generate a signal that corresponds to this text."}, {"start": 1930.2, "end": 1932.6000000000001, "text": " And this is going to be the ground truth."}, {"start": 1932.6000000000001, "end": 1939.0, "text": " And we have a different set of tokens which are telling us, hey, generate this particular audio clip."}, {"start": 1939.0, "end": 1945.6000000000001, "text": " And so we can see on this diagram here on the x-axis as the number of audio tokens increases."}, {"start": 1945.6000000000001, "end": 1947.9, "text": " So the blue tokens here."}, {"start": 1947.9, "end": 1951.1000000000001, "text": " Let's now focus on, for example, the purple curve."}, {"start": 1951.1, "end": 1958.5, "text": " The purple curve is comparing the generated audio signal with basically this signal here, signal one."}, {"start": 1958.5, "end": 1962.3999999999999, "text": " And it kind of makes sense that as you're increasing the number of blue tokens,"}, {"start": 1962.3999999999999, "end": 1968.3999999999999, "text": " you're getting worse and worse at generating a signal that's close to this one, which kind of makes sense."}, {"start": 1968.3999999999999, "end": 1976.1, "text": " And vice versa, if we focus on this dashed yellow line, here we're comparing with this signal."}, {"start": 1976.1, "end": 1978.1999999999998, "text": " So we're using this one as the reference one."}, {"start": 1978.2, "end": 1981.5, "text": " And obviously, as we're increasing the number of these blue tokens,"}, {"start": 1981.5, "end": 1987.1000000000001, "text": " we'll be closer and closer to generating something that looks like this original signal here."}, {"start": 1987.1000000000001, "end": 1991.3, "text": " So again, we are just to to to mind you, we are basically sampling."}, {"start": 1991.3, "end": 1997.9, "text": " So we pass all of these in into our ALM, which is just basically a transformer."}, {"start": 1997.9, "end": 2001.6000000000001, "text": " And we keep on generating these samples one after another."}, {"start": 2001.6000000000001, "end": 2006.1000000000001, "text": " And then we just pass all of that through the decoder that we train in the first stage."}, {"start": 2006.1, "end": 2008.6, "text": " So let me just go back here."}, {"start": 2008.6, "end": 2012.1999999999998, "text": " So we just pass those through this structure here."}, {"start": 2012.1999999999998, "end": 2013.8, "text": " So best was token through this one."}, {"start": 2013.8, "end": 2017.6999999999998, "text": " And we decode the original audio signal."}, {"start": 2017.6999999999998, "end": 2019.1, "text": " Okay."}, {"start": 2019.1, "end": 2020.8999999999999, "text": " I mean, the generator, not the original one."}, {"start": 2020.8999999999999, "end": 2022.8, "text": " And then we just do the comparisons."}, {"start": 2022.8, "end": 2029.1999999999998, "text": " The green line is just like a baseline where where you don't even have the text prompts."}, {"start": 2029.1999999999998, "end": 2032.5, "text": " You just have a couple of tokens from random signal."}, {"start": 2032.5, "end": 2037.2, "text": " And then you're comparing whatever is generated with this blue signal here."}, {"start": 2037.2, "end": 2038.4, "text": " Kind of confusing."}, {"start": 2038.4, "end": 2041.8, "text": " But the idea is that like after one point five seconds,"}, {"start": 2041.8, "end": 2049.8, "text": " roughly the contribution from text and the contributions from from the audio tokens become roughly similar."}, {"start": 2049.8, "end": 2053.5, "text": " And I'm not quite sure what was the length of these text tokens,"}, {"start": 2053.5, "end": 2056.8, "text": " because otherwise this is not kind of very informative,"}, {"start": 2056.8, "end": 2061.1, "text": " because obviously this if the number of text tokens is bigger,"}, {"start": 2061.1, "end": 2064.2999999999997, "text": " then this chart would not probably be at one point five."}, {"start": 2064.2999999999997, "end": 2068.7999999999997, "text": " The intersection would not happen at one point five seconds, if that makes sense."}, {"start": 2068.7999999999997, "end": 2071.7999999999997, "text": " OK, in any case, that's pretty much it here."}, {"start": 2071.7999999999997, "end": 2079.0, "text": " They show some diagrams how the guidance scale for the in the classifier free guidance influences the results."}, {"start": 2079.0, "end": 2084.1, "text": " So the FAD metric and the KL metric and they find that the minimum is around three."}, {"start": 2084.1, "end": 2090.7, "text": " So for number three, they get the best rate of across metrics, as usually the guidance scale helps."}, {"start": 2090.7, "end": 2093.1, "text": " And finally, they have some table comparing this,"}, {"start": 2093.1, "end": 2097.6, "text": " the multi stream transformer approach I did mention briefly."}, {"start": 2097.6, "end": 2103.3999999999996, "text": " And they say the results suggest that increasing the number of streams degrades the performance in both base and large models"}, {"start": 2103.3999999999996, "end": 2107.7999999999997, "text": " when compared to a single stream system, which kind of makes sense because we're losing information."}, {"start": 2107.7999999999997, "end": 2109.3999999999996, "text": " But let's see what we gain."}, {"start": 2109.3999999999996, "end": 2112.1, "text": " So what we gain is you can see here a speed up."}, {"start": 2112.1, "end": 2117.2999999999997, "text": " We gain like three point six X speed up when we use such an approach."}, {"start": 2117.2999999999997, "end": 2120.2999999999997, "text": " But you can see like the performance drop significantly."}, {"start": 2120.3, "end": 2128.6000000000004, "text": " That's if you focus on the FAD metric, we get from like two to like eleven, which is probably a big difference."}, {"start": 2128.6000000000004, "end": 2131.5, "text": " Cool, guys. That's pretty much it for this for this video."}, {"start": 2131.5, "end": 2133.3, "text": " So a couple of thoughts here."}, {"start": 2133.3, "end": 2139.3, "text": " So just like analyzing open eyes whisper that also deals with speech data."}, {"start": 2139.3, "end": 2144.0, "text": " I guess the conclusion here would be we probably like next paper down the line."}, {"start": 2144.0, "end": 2145.6000000000004, "text": " We won't need the LSTMs."}, {"start": 2145.6000000000004, "end": 2149.7000000000003, "text": " We won't need all of these details when it comes to the model architecture."}, {"start": 2149.7, "end": 2156.0, "text": " If you just have a generic architecture that simple, if you have much bigger data set and of higher quality,"}, {"start": 2156.0, "end": 2161.5, "text": " because as we saw here, we are what they are doing is they are basically concatenating just labels."}, {"start": 2161.5, "end": 2166.7, "text": " And thus they are losing a lot of the information that's kind of rich and present in the language."}, {"start": 2166.7, "end": 2172.1, "text": " And so by improving the data set, but making it bigger and just using a generic like architecture,"}, {"start": 2172.1, "end": 2176.1, "text": " I assume that one of the next papers will just have way better results."}, {"start": 2176.1, "end": 2180.9, "text": " And yeah, so if you like this video, let me know down in the comments."}, {"start": 2180.9, "end": 2183.1, "text": " Subscribe to this channel if you haven't already."}, {"start": 2183.1, "end": 2206.1, "text": " And until next time, bye bye."}]