High Fidelity Neural Audio Compression | Paper & Code Explained
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I cover the "High Fidelity Neural Audio Compression" paper and code. 

With 6 kbps they already get the same audio quality (as measured by the subjective MUSHRA metric) as mp3 at 64 kbps! 10x compression rate!  This is super important as streaming video+audio makes for ~82% of total internet traffic!

Lots of ideas we've already seen in previous paper overview videos such as VQ-VAE, VQ-GAN, and AudioGen applied to the problem of audio compression.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2210.13438
✅ Code: https://github.com/facebookresearch/encodec
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
02:37 Paper walk-through: high level overview
12:05 Residual Vector Quantization
18:05 Reducing the BW using arithmetic coding and transformers
20:05 Loss formulations and results
23:40 Code walk-through
26:00 EnCodec architecture
28:20 Residual Vector Quantizer module
32:55 Loading the audio signal
34:35 Compression - a forward pass through the encoder
38:00 Quantization forward pass
42:35 Efficiently packing the bits
45:25 Using LM to further compress audio
57:50 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#neural #audio #compression
 What's cracking guys, Alex here. In this video I'll be covering a new paper called in codec high fidelity neural audio compression from the meta team and basically I'm gonna do a code walkthrough as well as the paper walkthrough and hopefully that combination is gonna be useful for you guys. So basically what they've done is they've shown how they can preserve the quality after compressing and decompressing the audio file even though they are using neural networks and they can do so and basically save up a lot of space so like the bandwidth of signals encoded using this neural method are much lower compared to say MP3 format, et cetera, et cetera, so it's very cool results. We're gonna see the actual numbers a bit later but for now go ahead and basically clone this repo and you should probably take a look at the installation for development section in the readme if you wanna follow along and do a stepping through the code base. Okay, so there is a couple of samples they've shown here. I'm gonna play a couple of them. So they have two models basically. One is the 48 kilohertz stereophonic model that's non-streamable. We'll see what it means in a moment and then they also have the 24 kilohertz monophonic. So monophonic just means a single channel. Stereophonic means in this particular case two channels but it can be even more channels and basically this one is streamable which means you can use it to real time encode, send the data as it's, so as the raw samples are arriving you can encode it real time and basically that's what streamable refers to in this context. So here is the ground truth signal. Let me play it. ["I Am You"] ["In The Galaxy Globes"] So now if I play the encodex six, so that's like, I don't know what's the original size of the audio per second but it must be much higher than the numbers you can see here. So here we have like six kilobits per second and that's the encodex. So that's this method. Let's hear it. ["I Am You"] Okay and finally here is the MP3 and the funniest thing is we'll see this in the table a bit later is that this particular method with only six kilobits per second achieves the same subjective score as the MP3. So let's hear the 64 kilobits per second. ["I Am You"] ["In The Galaxy Globes"] Cool guys, so let's dig into the paper and let's see what the magic is about. So again, as I said, high fidelity neural audio compression by the meta team and there's a couple of papers that this paper is building off of. So one of those is the sound stream and end to end neural audio codec and the second one is audiogen. So that's kind of a paper I covered a couple of videos ago and that could be a very useful prerequisite because I've covered some things like what are male spectrograms, et cetera, et cetera. So maybe do check out that one in case you struggle understanding what's happening in this video. Okay, having said that, let's dig into the paper. So first of all, why is audio compression important? Well, this sentence kind of clarifies it. So recent studies suggest that streaming audio and video have accounted for the majority of the internet traffic in 2021, 82% according to Cisco report. Okay, that's kind of a huge number and basically if we can manage to better compress our data, then it's a no brainer that we can save a lot like the bandwidth of the internet, that's the energy, money, et cetera, et cetera. So like that's a hugely impactful thing to be working on. Okay, so then because we mentioned we have a neural audio compression, so what's the alternative? Well, the alternative are these signal processing hand engineered methods and we'll see some of the baselines a bit later, but basically as you can see here, the audio codecs typically employ a carefully engineered pipeline and like I've kind of highlighted the signal processing part because they are not learned, they're not neural, instead they are handcrafted. Okay, so what's the problem? What's the potential pitfall of these neural models? So here you can see that the loss in neural compression has a couple of problems. So the first one is the model has to represent a wide range of signals, such as not to overfit the training set or produce artifact laden audio outside its comfort zone. So basically outside of its training data set. And this is something we are well familiar with and the way they cope with this problem is that they have a diverse set of data sets and diverse domains such as general speech and noise speech and music, et cetera, et cetera. Okay, and then the other problem is that instead of compressing efficiently, both in compute time and in size. So for the compute time, they mentioned here, so for the former, we'll limit ourselves to models that run in real time on a single CPU core. So I think they've tested all of these models on a MacBook 2019 or something and that's kind of cool. And then for the latter, we use residual vector quantization of the neural encoder floating point output. Okay, so some of these ideas are probably familiar to you in case you watched some of my videos like VQVAE or VQGAN or any of the derivatives of those models. But I'm gonna still cover and explain what those mean in a second. Okay, so let me quickly walk you through the high level diagram here. So let's see how this thing is constructed. So on the left side of the diagram here, we have a raw audio signal and we also have the male spectrogram representation. So that's basically on one axis here, we have frequency. On the other axis, we have basically time axis. And then the color of the dot on this diagram tells you the amplitude of that particular sinusoid of that particular frequency at that point of time, if that makes sense. So again, do watch the audio gen in case you're confused, but that's basically equivalent representation to the raw audio. So this is like the time representation, this is the frequency space representation of the audio signal. Okay, and then there's a couple of things they do. So first they kind of pass the signal through the encoder, which is built up as you can see here out of these COM1D layers. And they additionally have the LSTM, a two layer LSTM sub module here, which helps to integrate the temporal information because otherwise we'd be basically, so like given the particular architecture they are using, they basically encode 320 samples from the raw audio into a single vector here, that's 128 dimensions large. Okay, and they refer to this thing here as the frame. So this thing is referred to as frame, but it also referred to some of these things here as frames, it's kind of inconvenient. Okay, so once you have this representation at the output of the encoder, what you do is you quantize it. So that means each of these vector quantizations is gonna have associated code book table. So basically that means they're gonna have something like this, so like they have a table and the dimensions of this table are gonna be 1024. So they have 1024 vectors and each of these vectors are, so the dimension here is 128, the same as the output of the encoder, okay? And each of these, so 128, and each of these vector quantization modules is gonna have a particular independent code table like this one. And so what these do then is you take this particular vector at the output of the encoder and you find the closest vector in this code table and you then just grab its index, okay? So when I say close, I think they're using the L2 norm. So basically just find the closest vector here and then let's say for example, this vector here is the one that corresponds, which is the closest one to this one, then you just grab its index here. So that might be like, I don't know, like 238. And now you use 238 as a representation of this whole vector. And that's how you kind of compress down the output representation here. And then additionally they have, as you can see, they have multiple of these and that's because they do something called residual quantization. So they first snap, so snap, they find basically, they find the closest vector and then they take this vector and they subtract it from the original vector and then the residue is encoded by the next vector quantization module. And then rinse and repeat and that's how they basically encode this thing. So you end up in this particular drawing here, you would end up instead of 128 dimensions, you would end up with four dimensions because you have four numbers being allocated to that particular vector. And we have, again, in the background, we have four tables, okay? And then finally, so I'm gonna ignore the transformer module here for a second, but like finally, you can basically decode those signals because you have the numbers and then you can just grab the vector from the associated tables, you can sum them up and you form your representation which you then pass through the decoder, okay? And decoder is just basically you can see symmetric to what we had in the encoder. So I'm gonna skip explaining it. And finally, we end up with the reconstructed audio signal here, okay? So how the thing is trained is there are multiple losses and they also introduced this thing called a balancer which uses the gradient information to balance and scale up all of the losses as needed. So first loss here is the LT which stands for time domain loss. They basically do L1 between the raw representations of the reconstructed audio here and the input audio here. And then they have the LS which is the basically a spectral domain L1 basically norm between the reconstructed mouse spectrogram and the input mouse spectrogram. So that's what they do here. And then finally, they have the LD and LG which are the generator and discriminator losses. And you can see here, they're like a couple of identical discriminators. And so what you do here is you just convert the audio signal into the real and imaginary STFD which are just again, the spectral representations. And these are strictly more informative. So let me just change the color here. So these are strictly more informative than the mouse spectrogram down here because given this information, you can easily get the mouse spectrogram information because you just have to find the amplitude of these imaginary complex numbers. And that's what gives you the amplitude of the mouse spectrogram. Anyways, you kind of pass that through this encoder to the discriminator. And then you train using the adversarial loss as usual. So you just basically contrast, you give it the real signal you give it the generated signal and you teach the discriminator to discriminate between those two and you teach the generator which is in this particular case, this whole pipeline here how to generate signals that look as real ones. Finally, a couple more losses here. So we covered these ones and the final two ones are this one. So this is the commitment loss. And again, if you watched any of the VQVA, VQGAN papers you're familiar with this one. And finally, there is the loss that corresponds to this transformer here. So what the transformer does is it helps us to basically compress this latent representation to even lower dimensionality at the cost of having to spend more compute and basically increase the latency of your system. But I'm gonna kind of go through probably if you have enough time, I'm gonna go through and show you how the language model part fits into this pipeline. Once we start walking through the code. Okay guys, let me quickly show you this residual vector quantization logic. There is from this before I mentioned this sound stream paper and they have a very neat pseudocode for this residual vector quantization. So let's kind of run through it. So you can see here, Y is denoted as the output of the encoder. We have multiple of these quantizers and there is NQ of them. So on this particular drawing, NQ is four because we have one, two, three, four of these components here. Okay. So let's go back here. And so here is how it works. So initially we initialize the residual variable with Y, which is the output from the encoder. And then you can see here, we quantize it and then, and basically then we add that to the reconstructed signal. Then we subtract that quantize, those quantize vectors from the initial residual vectors. And then we use those residual ones and we feed them into the next quantizer. And that's basically how it looks like on a high level. I also wanna walk you through the rationale. Why are we doing this? So I think a couple of these paragraphs might help elucidate the idea behind it. Okay. So the vector quantizer VQ proposed in these papers here in the context of VQ VAE meets this requirement. This vector quantizer learns a code book of N vectors. So N was 1024 in our particular case. To encode each D dimensional frame, and D was 128 of the output of the encoder here denoted as NQ of X. The encoded audio, as you can see the dimensionality being S times D. So if we have a one second audio signal, S will be 75 because we have the hop being 320. And I'm just using particular numbers because it's easier to kind of create a mental picture. But basically imagine you have a 24 kilohertz signal and then one second means you'll have 24,000 samples. If we divide that by 320, we end up with 75. So that means that for a signal of audio, for one second of the audio and the input we'll end up with 75 frames at the output. And then D is just the, as I said, 128. Okay. So that's then mapped to a sequence of one hot vectors of shape SN. So N is again 1024, which can be represented using this many bits. Okay. So because this is 1024, this is going to be 10. So once you do the log, you'll end up with 10 bits here as that's how much you need to index into those codebook tables. Okay. So let's see a concrete example here. So let us consider a codec targeting a bit rate of six K, so kilo bits per second. When using a striding factor of 320, each second of audio at sampling rate 24 kilohertz is represented by 75 frames at the output of the encoder. This corresponds to 80 bits allocated to each frame, right? So if we have the target bandwidth of 6,000 and we have 75 frames, that means we can allocate up to 80 bits and we will still not basically go over the target bandwidth. So using a plain vector quantizer, this requires storing a codebook with two raised to the power of 80 vectors, which is obviously unfeasible. Okay. And then they say to address this issue, we adopt a residual vector quantizer, AKA multi-stage vector quantizer, which cascades NQ layers, so we have four in the drawing below, of VQ as follows. The unquantized input vector is passed through our first VQ and quantization residuals are computed. The residuals are then iteratively quantized by a sequence of additional NQ minus one vector quantizers as described in algorithm one, which was the one we saw here. The total rate budget is uniformly allocated to each VQ, i.e. we have R divided by NQ, so R was 80 bits and then we divide that by the number of quantizers and that's how many bits each of the quantizers has to basically allocate. For example, when using NQ eight, each quantizer uses a codebook of size 1,024 and that's the number we'll be using in this paper as well. Okay. So hopefully that makes sense, so that's why this approach was basically developed. It makes it easy to get to the target bandwidth without having to have huge, enormous basically codebooks. Okay, so let's now quickly go through some of the details. Basically, as I said, they have the non-streamable and the streamable architectures. The streamable ones, so they say here, thanks to this padding scheme, which is basically left padding, the causal padding, the model can output 320 samples, which corresponds to 13 milliseconds, given a 24 kilohertz sampling rate. As soon as the first 320 samples are received. So that means as soon as you have 320 samples on the input, you can already encode it into a frame and then you can quantize that frame and you can send that down the line and then you can just keep on doing that as the signal keeps on coming as you're receiving your input signal. You can keep on encoding it and sending it. So that means that's why it's streamable. Okay, and then they mentioned by selecting a variable number of residual steps at train time, a single model can be used to support multiple bandwidth targets. That's something we saw up there with this particular example. So basically, depending on how many of these quantizer models you add, you can achieve a particular bandwidth. So here we picked eight because we want to have 80 bits per frame and thus we want to have six kilobits per second. Okay. Okay, so I did mention the language model that's being used here. So implemented as a transformer to further reduce the bandwidth. So let's kind of read this paragraph and then later we probably will have enough time to just go through that and see it in code. So for a timestamp T, the discrete representation obtained at time T minus one is transformed into a continuous representation using learned embedding tables, one for each code book and which are then summed. So we'll see that in the code a bit later. But basically, so discrete representation are these numbers I mentioned. So for example, if this is a step T minus one, you can see that the discrete representation is basically these four numbers here. And when I say four numbers, even though it's kind of float initially, which means you have like 64 bits, we know that it doesn't have to be larger than 10 bits because it just needs to index into this table that has 1024 vectors. And that's why these numbers are actually just 10 bit numbers, right? Okay, so let's go back here. And then for T equals zero, a special token is used instead. Okay, the output of the transformer is fed into ankyl linear layers with as many output channels as the cardinality of each code book, which is 1024 in our examples, giving us the logits of the estimated distribution over each code book for time T. Okay, so basically instead of just finding the closest vector in the code book table and snapping our output vector to that particular code book vector, instead of that, we basically do, we form like a probability distribution across the whole code book. So this is the code book table and we form a probability distribution across its indices and then combined with arithmetic coding, they show that you can further compress your latent representation using this approach. Okay, let's continue here. So here are the formulas for the losses. I'm not gonna spend too much time here, but you can see here L1 norm between the time domain signals. And then you can see here a combination of L1 and L2 norms between the mal spectrograms across various scales. And then there is the discriminative loss. Basically, you can see here the generator loss and you can also find the discriminator loss somewhere in here. So here it is. But I'm not gonna spend any more time on this because I've covered this. Also, I think the audio gen or VQ gen paper basically have pretty much same losses as this paper. So I'm gonna kind of skip all of those details. Here is also the commitment loss, fairly familiar to those of you who watch those videos and who know about those papers. And finally, the final loss is just a weighted sum of all of these various losses. And in order to avoid having to basically do like a search and find the right coefficients, they make it adaptive. And so they basically calculate the gradients of these losses with respect to the input. And then they use the norm of the gradient to automatically rescale the importance of each of these losses. So you can see here, if a gradient of a particular loss was huge, so we have this, this is basically a norm and exponentially, and they've done an exponentially moving average on the norm of the gradient. And you can see if that was huge historically, then that means this is gonna be down weighted. So the gradients from that particular loss are gonna be down weighted because this is a big number in the denominator. And because of that, that loss is gonna be down weighted. So that's the kind of the intuition behind it. Okay. So as I said, they use diverse domains to train these models. That's how they ensure that they don't overfit to particular domain. And yeah, so they mentioned here speech, noisy speech, music, and general audio. And here are the results. So you can see here how they compare against some of the baselines, such as OPUS and EDS, which are basically signal processing baselines. And then we have Lyra from Google, which is a neural codec as well. And you can see it basically outperforms all of those models for all of the target bandwidths. Additionally, this entropy coded one, which uses the transformer to further compress the representation, basically outperforms for a lower target bandwidths, outperforms the normal one, the normal encodec without the transformer. Okay, guys. Let's see what else is interesting here. Yeah, this result here is kinda cool. So I think I mentioned this in the beginning of the video. Basically, you can see here that MP3 with the target bandwidth of 64 kilobits per second has the same, this MUSHR score, which is just basically subjective score, and has the same score as this particular encodec with six kilobits per second. So you can see these are fairly on-pair. And you can see that already with 12 kilobits per second, it heavily outperforms the MP3 format. And also, if you include the entropy coding, then you can even further reduce. With 4.2 kilobits per second, you outperform 64 kilobits per second MP3, and you have comparable basically subjective metrics. That's fairly cool. Okay, having seen all of this, having seen the paper, let's now switch our focus to the actual code. And hopefully that's gonna help us understand this much better. So I just created a long JSON, and I basically passed, as you can see here, the arguments, I just have this test 24K. I'm gonna pronounce this as wave signal. I'm not sure how this format is pronounced, but wave sounds easy enough. Basically, test 24K is already a part of the codebase. It's checked in, so you just have to clone the repo, and you will have it out of the box. So that's it. That's everything we have to set, and later I'm just gonna add this LM flag to see how the language model fits into this whole picture. And there is also the HQ, which is the 48 kilohertz non-streamable model, but I'm not sure I'll have enough time. I don't wanna go over one hour in this video. So let's basically start. I'm gonna enable all of the breakpoints here, and let's run this thing. Okay, again, I'm gonna focus only on the crucial parts. So you just make sure that the audio signal exists, and then here you can see that because we have wave signal, and this is basically ECDC, which is the compressed format suffix. Because of that, we are gonna go into the compression branch. So, but by understanding this compressed function, this decompressed function is gonna be fairly trivial to understand. So I'm gonna focus on this particular branch of the code. Okay, so we formed the output, as we just changed the suffix. Basically, it's also gonna be, as you can see here, test24k, oops, test24k ECDC. That's the output. And then we check that the output exists, and because it already exists, I'll just have to first basically delete that signal. I'm gonna delete this output compressed file, and now it's gonna work. So this is gonna happen to you as well. So I'm not gonna edit this out. I'm just gonna restart the program again, and this time it will not fail. So let's skip to this check output, and this time this will not fail, so everything's fine there. Cool. So because HQ is set to false, we're gonna deal with 24 kilohertz signal. Let's enter this. Let's enter the encodec model 24 kilohertz function. And basically, so you can see here, we just set some variables, and finally we need to first basically get this model. So the model consists, as you can see here, from the encoder and decoder. So I'm gonna enter into the encoder just to briefly show you how everything looks like. So we are in the encoder here, and you can see basically just some variables, the location there, and then we have this ratios list, which basically tells us how we are reducing the temporal dimensionality as we are propagating through the encoder. And if you sum all of these up, you'll get the number that's probably familiar at this point, and that's 320. And that's why we need 320 raw samples to get a single output frame. That's the particular reason why this is happening, because of the architecture of the encoder. Okay? So let's continue. So you can see here, we just have these COM1D layers with this S prefix. We just suggest that they have, they kind of modified it a bit. I'm gonna show you how it looks like in a second. But after that, they just keep on adding these ResNet blocks, which consists basically out of bunch of these COM1D layers and some skip connections. So basically nothing fundamental there. It's just your encoder architecture, and that's it. I'm gonna quickly just show you how this thing looks like. So they have this S prefix, as I mentioned. And so the thing here is, they'll have this, depending whether we are dealing with a causal model or non-causal model, they'll have a bit different padding here. And I think that's pretty much it. They also have some details like how they wrap up this norm layer with the COM layer. So you can see here, just some idiosyncrasies, but nothing crucial really to understand how this is working. So let's treat it as a black box, and we have our encoder. The decoder is fairly similar, so I'm gonna skip it completely. And finally, let me zoom in a bit in case you can see a bit better now. Okay, hopefully this works. Let me know whether the visibility is now better than before I zoomed in. By the way, I'm a bit sick, so that's why my throat might sound a bit different. But hopefully the video is gonna be high quality nonetheless. So let's see what's happening here. You can see here that this enqueue is gonna be the max number of those quantizers we had in that bottleneck of the encoder decoder that this model can have in case we have the largest target bandwidth. And you can see here, target bandwidths go from 1.5 kilohertz all the way to 24 kilohertz. And when we index by minus one, we fetch the largest bandwidth, which is the 24 kilohertz. And because it's 24, we multiply with thousands such that we end up with hertz. So this is now the number of hertz for the target bandwidth. And then we divide that number with this thing. And so what's this thing? So basically we have here the number of frames. You can see here sample rate. When you divide that by the hop length, you end up with the output number of frames. It's gonna be like 75 for our particular case here. And then we multiply that with 10 because they kind of hard coded that because they assume that the code book is gonna be 1,024 slots. And they basically does assume that we need 10 bits to index into that code book. So again, that means that a single quantizer will have 75 times 10 bits. So that means 750 bits for a second of the video. And so when we divide these two, we get the number of quantizers we need for the target bandwidth of 24 kilohertz. As you can see here, it's gonna be 32. So that means we need 32 of these quantizers to be able to achieve this particular target bandwidth of 24 kilohertz. So again, just to clarify as this might be a bit confusing, imagine we have a signal, a second of the audio raw signal that's gonna be 24,000 samples, right? And then we encode that and we end up with 75 frames. And because for each of the frames, we allocate 10 bits, that means we have 75 times 10 bits, right? And then if you wanna figure out how many of these do we need to get to 24K, we just divide 24K by that number of bits. And that's precisely what you've seen here. Okay, second thing I wanna show you is this residual vector quantizer, although all of the fun is gonna happen in the forward pass, not here, but let's see how it's constructed. Again, just bunch of variables here. And then there is this residual vector quantization. Let's enter that thing. And you can see here, we have 32 of these vector quantizations basically wrapped into a list. And let me see whether I have a break point here. So let's enter into one of these vector quantization modules and let's see what's going on there. So here is the codebook dimensionality, it's gonna be 128. So that's the number we've been using as a running example throughout the video. And then because we don't require any projection because the dimensionality of the output of the encoder and the codebook are the same because of that we don't need any projection and that's why we're gonna end up with identities here. Okay, and finally we have this Euclidean codebook and the whole fun is gonna be in the forward pass, I'm gonna skip the actual initialization, but that's the implementation of the codebook I've been mentioning this whole time. And it's called Euclidean because once we do the, once we try to figure out the closest vector in that code table we use the L2 norm to figure that out, hence Euclidean. Okay, I'm gonna hit F5 here and let's exit this. And now I'm gonna disable all of the break points and let's exit this one. And we are back here. I'm gonna re-enable all of the break points and let's continue here. Okay, so we have the encoder, we have the decoder, we have the quantizer, and finally we just have this encodic model which is just basically gonna wrap up all of those variables, nothing fancy, I can move this break point, I can remove it and let's continue here. So let's just get to here and that's it. So we are out of here and we have our model finally ready. So now we just fetch the actual state dictionary, so the pre-trained weights, we initialize the model, we set it into the eval mode and we continue here. Okay guys, so now we set the target bandwidth, okay? So the target bandwidth by default is gonna be six kilobits per second and what it does is basically just sets this bandwidth internal variable of the encodic model to six and that's it. So nothing fancy happening there. Okay, so now let's just load the audio file. It's gonna be a 24 kilohertz, 20 second audio file. So if you take a look at the root of this repository, we can find that signal here. Let me see where we can hear it. Okay, you can see it's a 20 second audio file, that's it. Okay, so let's continue here. Let me just show you the shape of that WAV file. So shape is 480,000 and that's because if you just think about it, if you multiply 20, which is the number of seconds of the audio file times the 24,000 samples per second, we end up with 480,000 samples. My brain is glitching, man. Okay, so let's continue. We now do the conversion but because, it's gonna be a no-op basically. I'm gonna enter here just to show you but basically because the target channel is set to one, we do a mean across the zero dimension but because we only have a single channel, it's gonna be a no-op. If we had a stereo sound, then we'd have two channels and we'd just do a mean across the two channels to get the final output channel because the target number of channels is set to one here. But in our case, it's just no-ops, we can ignore it and then we do resampling but again, because both our signal and the target signal sampling rates are the same 24K, we do nothing. So we just basically, it's a no-op. Okay, so here is the fun part. We do the compression of the signal, of the loaded signal and we pass it through our model. Let's see how that's gonna work. Okay, so let's kinda skip over all of these details. What we do here is we are basically going to load the whole signal. So this whole loop is gonna have just a single iteration. And I think worth mentioning is even though they support the streamable setting, they don't have it implemented here neither do they have implemented the training code. So what you can find here is the inference. You can also find some details about the balancer and about the multi-scale STFT discriminators. Okay, so basically you can see here, they also use the terminology frame for the input raw audio signal and not only for the output of the encoder. So just keep that in mind and don't get confused by the terminology here. So frame is again, is gonna be simply the same as our input signal. So because we fetched every single sample, it's just gonna be 480,000 samples. Okay, and now the fun happens in the code frame function. So let's see how that's gonna work. So we don't do it in normalizations, we can skip that. And now we pass the X, so the input signal through the encoder. So let's see how that's gonna work. So again, this is just gonna be a series of these COM1D layers and some of them are gonna have a stride of two. And so our input signal is slowly gonna be reduced across the temporal dimension and the number of channels is gonna be increasing. So what I'm gonna do here is I'm gonna enter the COM1D layer here and I'm gonna put the break point with a particular condition. And this is why I love Visual Studio Code. So you can basically edit the break point and you can set a particular condition when you wanna hit this one. And I wanna hit it when stride is bigger than one, right? And so because I wanna see how the number of the temporal axis is being reduced. Okay, so now let's hit F5. And here we are, so currently if I just print a shape, the shape is, as you can see here, we already have 32 channels because we had some previous COM1D layers which didn't have the stride but they still had increased the number of channels. But once I hit, so now I'm gonna put a regular break point and hit F5. So once we are here, now if I do x.shape, we have, as you can see here, two x, a smaller dimensionality across the temporal axis and we have two x more channels. And that's a regular pattern we've been seeing all around. CNNs do that all the time. As soon as you reduce the spatial dimensionality by two, you also increase the number of channels by two. That's a common pattern you'll see all around. Okay guys, so now I'm gonna remove basically all of the break points and I'm gonna exit from this function. I'm gonna exit here, I'm gonna exit here, I'm gonna exit through all of these and we are done. Okay, so this is the encoder. I'm gonna hit F10 and here is the final dimensionality. It's gonna be 1,500, 128. And that's the output of our encoder, right? And it's 1,500 frames because again, we have 20 seconds and each second is 75 frames and that's why we end up with 1,500. Hopefully that makes sense. Okay, so now what we do is we have to quantize those vectors and let's see how that's gonna work. So here we are. So we first have to figure out for the desired target bandwidth, how many of the quantizers do we have to actually use to get that target bandwidth? And so that's what this function does. So let's enter into the function and let's see how it works. So, okay, so they say get bandwidth per quantizer. So does that make sense? Let's think about it. So we have 75 here. So that means we have 75 frames. It's gonna be 10 because bins is gonna be 1,024. So this is 10 times the number of frames. So this is basically the number of bits for a single quantizer module. And then we divide by 1,000 because we wanna have kilobits, right? And so you can see here bandwidth per quantizer. And that's gonna be 0.75 kilobits per quantizer, okay? And now what we do is we take the target bandwidth, which is six kilobits per second, and we divide by 0.75, and we end up that we need eight of these quantizers to achieve that particular target bandwidth, okay? And that's it. So now we actually do the encoding with these eight quantizers. So let's see how that's gonna work. So you can see here, we just take the first eight quantizers instead of the full list, which is like 32. So remember, we have actually, we have 32 of these, and we can only deploy those when we have target bandwidth of 24 kilohertz. Okay, so let's enter here. Let's see how the encode works. So it's fairly simple. So first of all, this pre-process is just gonna do some reshaping, nothing fancy there. So we end up with X being of shape 1,528. And then this is where the fun happens in the quantize function. So in the quantize function, let me enter the quantize function. You can see this is the codebook, the codebook table I was mentioning the whole time. So let's see the dimensionality of the codebook table. This is gonna be 1,024 because we have 1,024 vectors, and each of those is 128 dimensionality, right? And so what we do here is we basically do, we compute this distance matrix, which tells us for each of our output vectors, how close are each of these codebook vectors from that particular vector. And because of that, we're gonna end up, let me just show you this. Let me show you the dimensionality of the distance matrix. It's gonna be 1,500, 1,024, because as I said, you have for each of the rows, you have the columns telling you how far away is that particular codebook vector from that particular output encoder vector, right? And because of this minus sign, if we do a max, we actually find the closest vectors, and that's it. So we end up with, we basically end up with, as you can see here, 1,500 indices, and let me print the actual numbers. So those are the indices of the closest vectors in those code tables. Hopefully that makes sense. And again, post-process just does some reshaping, and we are done, guys. That's it. So now we do the decoding. That's the second interesting part. So let's enter it into the decoding part. So let's see what happens there. So decoding is gonna call this Dequantize method, and Dequantize is basically just gonna fetch the indices, and you can see here, we just grab the vectors from that table that correspond to those indices, and we end up with the quantized vectors. Whoops, let me just show you the dimensionality here. We have 1,500, 128. So that's when we snap our output vectors onto the closest vectors in the code table. That's what happened there. Okay, and then projections identity, so we can ignore this, and that's pretty much it. And now that we have the quantized vectors, we do the residual minus quantized, and then we save the indices from the first quantizer, and we repeat the process, right? We now take the second code book, and we take the residuals, and we just encode them, and rinse and repeat. So I'm gonna now disable all of the breakpoints, and I'm gonna just put the breakpoint here, hit F5. I'm gonna re-enable the breakpoints, and that's it. So the out indices now are 1,508. So that means for each of the frames, we now have eight numbers. That's how we encoded our output of the encoder. Okay, let's continue here, and that's pretty much it. We can now keep on exiting here, and we have everything we need. We have the data, the compressed version of our audio file. We now write some metadata into the header of this ecdc file. I'm gonna ignore that completely, not that important for us. And then because we're not using LM, we are gonna construct this bitpacker. And so why the bitpacker? So again, the problem is these numbers are gonna be either ints or floats. Let me just see for a second. Let me tell you the exact thing here. So frames, again, frames shape is, okay, frames is a list. Let me just see. Okay, it's a list, and then I think I just have to do something like this. Yeah, okay, so here are the frames. And if we do the D type, let's see the data type. So it's gonna be int 64. So that means each of these eight numbers currently take 64 bits, even though we only need 10 bits. And because of that, we only, because we only, out of those 64 bits, only lower 10 bits have the valuable information. That's where this bitpacker comes into play to handle the low level details. I did debug it and understand how it works, but it's gonna be, I think, redundant if I do that in this video, because you can do it at your own pace, but nothing fundamental there other than low level details of how do you extract the 10 bits that are actually valuable and pack them into the output bit by stream. Okay, and that's it. So here, what we do, we just keep on pushing those values, and values are coming from the frames. And so basically, that's it. We're just taking those numbers from this structure here, and we're just basically encoding them into this by stream. And so let me hit F5 here. Let me remove this thing. And we are done. We are done. We can exit here. We can exit here. We just write the bytes, and that's it, guys. So that's it. So let me just find the file. So the file is here. As you can see here, test24k.ecdc is again created, and that contains our compressed audio file. You can see that we started from 937 kilobytes, and we ended up with 14.7 kilobytes. Does that make sense? Let's see. So we have 14.7 times, I guess, eight, and this is how many kilobits we have. And that's roughly, well, that's roughly 20 times six, right? Which is 120, because we have six kilobits per second target bandwidth, and we have 20 seconds, so that's roughly it. So everything basically fits. Okay, so now I'm gonna go to the launch file. I'm gonna quickly add the LM, and I'm kinda afraid that I will not be able to explain this, to give it justice, but I'm gonna try nonetheless, because I think it would require quite a lot of time to actually explain all of the details behind how and why it works. Okay, so now I'm gonna start this again, and let's just see the diff between the last ones. I'm not gonna focus on the things that are the same, obviously, I just wanna focus on the differences. So the differences here are gonna be only, as you can see here, we passed the LM argument to the compressed function, so I'm gonna disable the breakpoints, I'm gonna enable this one. I'm gonna hit F5. Whoops, again, I forgot to delete the file, god damn it. So I'm gonna delete this one, and I'm gonna restart. So let's hit, you can add the force argument to avoid having these errors, but I was lazy, so I kinda skipped it. So I'm gonna enable the breakpoints, so let's now enter this, let's enter here, and let's see, so use LM is where the differences are. So there's a couple of places where there are the differences. So I'm gonna get here first, so let's see what the get LM model does. So basically we instantiate a language model here. So let's see how that thing is implemented. I'm gonna hit F5 here, here we are. We have something called Streaming Transformer Encoder, and we have, as you can see here, 32 of these embedding tables, and we also have 32 of these linear layers that project from the 200, which is the dimensionality of the transformer, to 1024, because that's the cardinality, that's what card stands for, the cardinality or the size of the codebook table. That's why we have 1024 there, right? So I'm gonna, okay, I have a breakpoint here already. Let me see whether there is something interesting to mention here, let's quickly walk through the transformer here. Okay, basically there are these Streaming Transformer Encoder Layer modules, and let me just see whether those are interesting. So those are basically just, as you can see here, just stock PyTorch Transformer Encoder Layer with some differences, like, yeah. This might be too complex to explain in this shorter video, but in any case, I'm gonna skip this, I'm gonna skip the instantiation, so we just have a transformer, we create those embedding tables, we create the linear layers, and we continue here. So that's the LM, okay? So, now what happens is we'll load the actual pre-trained weights. We put it into the eval mode, and we return the language model, okay. So now we do the encoding the same way as before. So nothing changes here. We still end up with 1500 times eight numbers. So let me show you that. So we have, oops, again, I have to first do the zero, zero indexing, and then we end up with 1508, so that's the same shape as before. And so now, let's see where the difference is. So the first difference is here. We formed this thing called Arithmetic Coder, and it's fairly complex to explain, as you can see just by the description of how this thing works, and some of the low-level details here of bit shifting, et cetera, et cetera, but basically in a nutshell, what it does is using the, well, I'm gonna get there in a second, so it's gonna be wasteful for me to try and explain it like this. So in any case, we have this Arithmetic Coder. We're gonna see how it works a bit later, okay. So you can see here, we first initialize the input as all zeros, and k is equal to eight, which is the number of our quantizers. So this is kinda, before we get the actual eight numbers that are the encoded version of the first output vector of the encoder, before that, we just assume we have all zeros, right? So again, let me just show you what I mean. Let me find the diagram here. So here we are. So again, before we have the actual encoding for this first vector, so we'll have like four numbers here or eight numbers in our particular case, we just assume we have all zeros. We have all zeros, and that's what the program assumes here. Okay, that's what this thing is, okay. So let's now see how this is gonna work. So we pass the input, the states, and the offset into the language model, and then we just basically do a forward pass through the model to get the output probabilities. Okay, let's enter here, just to, let me just show you one thing. Whoops, let me just say one thing, and that's the logits part. So after we just do a forward pass through the transformer, which I'm gonna skip, so I'm gonna skip this part. So I'm gonna put breakpoint here. So I'm gonna disable everything, enable this one. We're gonna do a forward pass through the transformer, and we end up with the output, okay. So output is gonna be of the shape, let's see, so 200, because that's the inherent, that's the inner dimensionality of the transformer. So we passed in all zeros, and we basically, how we convert those all zeros into a continuous representation is on this line here. As you can see here, we just do eight times. We grab the number, and we basically grab the corresponding embedding table, and then that's how we get the representation, and then we just sum them up. And I think that was also explained in the paper here. So let me just show you where that thing is explained. I think somewhere here, so okay. Okay, for a time step T, the discrete representation obtained at that time, T minus one is transformed into continuous representation using learned embedding tables, one for each codebook, and which are summed. So that's basically this line. This line describes this piece of code here. So again, let's go back to the diagram. It's gonna be easier. So we have these, let's say four numbers here, and each of these four numbers is gonna index into a particular table. So we'll have four tables associated with this transformer, and we get their corresponding vectors, then we sum them up, and then we do a forward pass through the transformer, and we end up with a 200-dimensional vector. That's what we do there, okay? That's the idea. And now we just map those using those linear layers I mentioned before that are gonna have the output dimensionality being equal to the cardinality of the codebook, which is 1,024. So if I do just a forward step over this, we're gonna end up with logits. Here are the shape. The shape is 1,024, eight. And that makes sense because for each of the eight numbers that we are supposed to output for this particular time step, we have a distribution, which is 1,024 bins, across our codebook table, corresponding codebook table. I know this might be confusing, so you'll have to go through the code base at your own pace, but hopefully this helps you somewhat. Okay, so probabilities are there. So what we do next is this. So we have this, we grab the next eight numbers and we add one. And one is added because the zeros, as you see here, have a special connotation. They're just like initial values, which cannot actually be, we can't get zero as the output. And that's why there is this plus one here. Anyways, a small detail there. So here is the complex part. We build this quantized CDF, which is a cumulative distribution function. And basically how it works is that the higher the probability at the particular bin, the bigger the range that particular bin is gonna occupy. So again, I told you this is gonna be fairly complex. So it says here, turn the given PDF into a quantized CDF that splits this particular range into chunks of size roughly proportional to the PDF, right? And they use 24 bits here. So that means this is gonna be two raised to the power of 24 minus one. Again, let me try and quickly draw this, and I think we are almost done with the video. Okay, so here is how it's gonna look like. So we get as the output, we get some distribution. So let's say it looks like this. Okay, and so what this is gonna be mapped into is this range. So this range is gonna be from zero to two raised to the power of 24 minus one, okay? And then we're trying to split this range. And so here, initially, because we have a peak here, that means that this range is gonna be huge, okay? Let me just change the color here. This range is gonna be huge. And then where we have lower probability, we'll have much smaller, whoops, let me change the color again. So we'll have much smaller ranges here. So these ranges here are gonna be small, but these ones are gonna be bigger, and they're gonna slowly diminish here. So that's the mapping that's happening in that particular function I showed you. And these are all of the details behind the arithmetic encoding. And so you'll have to dig into that a bit deeper yourself in case you're curious, but that's it. I'm gonna go back here. So we end up with the CDF, and let me just show you. So CDF again, the shape here is 1,024 again. But you can see here that we just have the numbers from zero to two raised to the power of 24 minus one. And you can see here that the differences here, if they are super big, that means that the corresponding bin in the PDF had a higher probability. And that's the idea. And then we use this arithmetic coder, and we push the value. Whoops, here it is. But we also pass the CDF, because the arithmetic coder requires that information of the ranges in order to efficiently encode this. And this is gonna occupy less than 10 bits in expectation. So before, because we had eight numbers, the best we could do was eight times 10 bits, that's 80 bits. Here we can go below 80 bits. And that's the magic behind the arithmetic coder. So I'm gonna basically enter this just to show you the complexity, but I'm not gonna try and explain everything here. Let me hit F5 here. And so you can see here how the numbers are, how the range information here is used to basically finally here get the actual encoding for this particular number and this CDF. So you can see here, we are packing bit by bit. This packer is the same packer as the one we had before, but instead of using 10 bits, instead of pushing 10 bits into the stream, it pushes just a single bit into the stream. And, but in expectation, this while loop is gonna have less than 10 iterations, i.e. 10 bits per symbol. And that's why we're gonna be able to basically compress this even more than using the, than not using the arithmetic coding. So I'm gonna exit here. I'm gonna exit here. And basically, I'm gonna remove all of the breakpoints, put a breakpoint here, hit F5, and that's it. So you can see it was a bit slower. And that's the trade off you have to take if you wanna have even higher compression ratio. And that's it, guys. Everything else remains the same. We just write the output bytes, and that's it. So hopefully this combination of using the paper and the video was helpful. Do let me know whether you have some questions. I know I had to kinda rush through some of the details behind it, but hopefully this gives you some intuition, some understanding. Do leave some comments down below. Any feedback you might have for me, it will be super useful. And guys, until next time, bye bye. Lindsey Ch resolution
[{"start": 0.0, "end": 2.58, "text": " What's cracking guys, Alex here."}, {"start": 2.58, "end": 5.58, "text": " In this video I'll be covering a new paper called"}, {"start": 5.58, "end": 8.5, "text": " in codec high fidelity neural audio compression"}, {"start": 8.5, "end": 11.58, "text": " from the meta team and basically I'm gonna do"}, {"start": 11.58, "end": 14.9, "text": " a code walkthrough as well as the paper walkthrough"}, {"start": 14.9, "end": 17.400000000000002, "text": " and hopefully that combination is gonna be useful"}, {"start": 17.400000000000002, "end": 18.32, "text": " for you guys."}, {"start": 18.32, "end": 22.18, "text": " So basically what they've done is they've shown"}, {"start": 22.18, "end": 25.38, "text": " how they can preserve the quality after compressing"}, {"start": 25.38, "end": 27.580000000000002, "text": " and decompressing the audio file even though"}, {"start": 27.58, "end": 31.06, "text": " they are using neural networks and they can do so"}, {"start": 31.06, "end": 34.78, "text": " and basically save up a lot of space so like the bandwidth"}, {"start": 34.78, "end": 38.739999999999995, "text": " of signals encoded using this neural method"}, {"start": 38.739999999999995, "end": 43.18, "text": " are much lower compared to say MP3 format, et cetera,"}, {"start": 43.18, "end": 44.78, "text": " et cetera, so it's very cool results."}, {"start": 44.78, "end": 47.3, "text": " We're gonna see the actual numbers a bit later"}, {"start": 47.3, "end": 50.599999999999994, "text": " but for now go ahead and basically clone this repo"}, {"start": 50.599999999999994, "end": 53.54, "text": " and you should probably take a look at the installation"}, {"start": 53.54, "end": 55.72, "text": " for development section in the readme"}, {"start": 55.72, "end": 60.06, "text": " if you wanna follow along and do a stepping through"}, {"start": 60.06, "end": 61.22, "text": " the code base."}, {"start": 61.22, "end": 65.46, "text": " Okay, so there is a couple of samples they've shown here."}, {"start": 65.46, "end": 67.74, "text": " I'm gonna play a couple of them."}, {"start": 67.74, "end": 69.42, "text": " So they have two models basically."}, {"start": 69.42, "end": 73.18, "text": " One is the 48 kilohertz stereophonic model"}, {"start": 73.18, "end": 74.82, "text": " that's non-streamable."}, {"start": 74.82, "end": 76.82, "text": " We'll see what it means in a moment"}, {"start": 76.82, "end": 80.3, "text": " and then they also have the 24 kilohertz monophonic."}, {"start": 80.3, "end": 81.98, "text": " So monophonic just means a single channel."}, {"start": 81.98, "end": 84.7, "text": " Stereophonic means in this particular case two channels"}, {"start": 84.7, "end": 86.46000000000001, "text": " but it can be even more channels"}, {"start": 86.46000000000001, "end": 88.38000000000001, "text": " and basically this one is streamable"}, {"start": 88.38000000000001, "end": 92.46000000000001, "text": " which means you can use it to real time encode,"}, {"start": 92.46000000000001, "end": 94.38, "text": " send the data as it's,"}, {"start": 94.38, "end": 96.58, "text": " so as the raw samples are arriving"}, {"start": 96.58, "end": 98.58, "text": " you can encode it real time"}, {"start": 98.58, "end": 101.58, "text": " and basically that's what streamable refers to"}, {"start": 101.58, "end": 102.66, "text": " in this context."}, {"start": 102.66, "end": 104.98, "text": " So here is the ground truth signal."}, {"start": 104.98, "end": 105.82000000000001, "text": " Let me play it."}, {"start": 106.78, "end": 108.62, "text": " [\"I Am You\"]"}, {"start": 108.62, "end": 111.62, "text": " [\"In The Galaxy Globes\"]"}, {"start": 113.06, "end": 114.98, "text": " So now if I play the encodex six,"}, {"start": 114.98, "end": 117.98, "text": " so that's like, I don't know what's the original size"}, {"start": 117.98, "end": 121.78, "text": " of the audio per second but it must be much higher"}, {"start": 121.78, "end": 123.26, "text": " than the numbers you can see here."}, {"start": 123.26, "end": 126.06, "text": " So here we have like six kilobits per second"}, {"start": 126.06, "end": 127.06, "text": " and that's the encodex."}, {"start": 127.06, "end": 127.9, "text": " So that's this method."}, {"start": 127.9, "end": 128.72, "text": " Let's hear it."}, {"start": 130.72, "end": 133.22, "text": " [\"I Am You\"]"}, {"start": 135.34, "end": 137.3, "text": " Okay and finally here is the MP3"}, {"start": 137.3, "end": 139.46, "text": " and the funniest thing is we'll see this in the table"}, {"start": 139.46, "end": 141.52, "text": " a bit later is that this particular method"}, {"start": 141.52, "end": 143.9, "text": " with only six kilobits per second"}, {"start": 143.9, "end": 147.60000000000002, "text": " achieves the same subjective score as the MP3."}, {"start": 147.60000000000002, "end": 150.02, "text": " So let's hear the 64 kilobits per second."}, {"start": 150.02, "end": 152.60000000000002, "text": " [\"I Am You\"]"}, {"start": 152.60000000000002, "end": 156.56, "text": " [\"In The Galaxy Globes\"]"}, {"start": 156.56, "end": 158.52, "text": " Cool guys, so let's dig into the paper"}, {"start": 158.52, "end": 160.84, "text": " and let's see what the magic is about."}, {"start": 160.84, "end": 162.06, "text": " So again, as I said,"}, {"start": 162.06, "end": 165.5, "text": " high fidelity neural audio compression by the meta team"}, {"start": 165.5, "end": 169.54, "text": " and there's a couple of papers that this paper"}, {"start": 169.54, "end": 170.66, "text": " is building off of."}, {"start": 170.66, "end": 172.64, "text": " So one of those is the sound stream"}, {"start": 172.64, "end": 174.74, "text": " and end to end neural audio codec"}, {"start": 174.74, "end": 176.76, "text": " and the second one is audiogen."}, {"start": 176.76, "end": 180.2, "text": " So that's kind of a paper I covered a couple of videos ago"}, {"start": 180.2, "end": 182.6, "text": " and that could be a very useful prerequisite"}, {"start": 182.6, "end": 184.0, "text": " because I've covered some things like"}, {"start": 184.0, "end": 187.08, "text": " what are male spectrograms, et cetera, et cetera."}, {"start": 187.08, "end": 190.3, "text": " So maybe do check out that one in case you struggle"}, {"start": 190.3, "end": 192.9, "text": " understanding what's happening in this video."}, {"start": 192.9, "end": 195.26, "text": " Okay, having said that, let's dig into the paper."}, {"start": 195.26, "end": 198.51999999999998, "text": " So first of all, why is audio compression important?"}, {"start": 198.51999999999998, "end": 200.51999999999998, "text": " Well, this sentence kind of clarifies it."}, {"start": 200.51999999999998, "end": 203.76, "text": " So recent studies suggest that streaming audio and video"}, {"start": 203.76, "end": 206.35999999999999, "text": " have accounted for the majority of the internet traffic"}, {"start": 206.35999999999999, "end": 210.56, "text": " in 2021, 82% according to Cisco report."}, {"start": 210.56, "end": 211.79999999999998, "text": " Okay, that's kind of a huge number"}, {"start": 211.79999999999998, "end": 215.1, "text": " and basically if we can manage to better compress our data,"}, {"start": 215.1, "end": 217.23999999999998, "text": " then it's a no brainer that we can save a lot"}, {"start": 217.23999999999998, "end": 218.84, "text": " like the bandwidth of the internet,"}, {"start": 218.84, "end": 220.68, "text": " that's the energy, money, et cetera, et cetera."}, {"start": 220.68, "end": 225.18, "text": " So like that's a hugely impactful thing to be working on."}, {"start": 225.18, "end": 228.44, "text": " Okay, so then because we mentioned"}, {"start": 228.44, "end": 230.48000000000002, "text": " we have a neural audio compression,"}, {"start": 230.48000000000002, "end": 231.36, "text": " so what's the alternative?"}, {"start": 231.36, "end": 233.92000000000002, "text": " Well, the alternative are these signal processing"}, {"start": 233.92000000000002, "end": 236.32, "text": " hand engineered methods"}, {"start": 236.32, "end": 238.4, "text": " and we'll see some of the baselines a bit later,"}, {"start": 238.4, "end": 240.36, "text": " but basically as you can see here,"}, {"start": 240.36, "end": 241.60000000000002, "text": " the audio codecs typically employ"}, {"start": 241.60000000000002, "end": 243.24, "text": " a carefully engineered pipeline"}, {"start": 243.24, "end": 246.28, "text": " and like I've kind of highlighted the signal processing part"}, {"start": 246.28, "end": 248.60000000000002, "text": " because they are not learned, they're not neural,"}, {"start": 248.60000000000002, "end": 250.70000000000002, "text": " instead they are handcrafted."}, {"start": 250.70000000000002, "end": 252.48000000000002, "text": " Okay, so what's the problem?"}, {"start": 252.48000000000002, "end": 254.48000000000002, "text": " What's the potential pitfall of these neural models?"}, {"start": 254.48, "end": 258.32, "text": " So here you can see that the loss in neural compression"}, {"start": 258.32, "end": 259.84, "text": " has a couple of problems."}, {"start": 259.84, "end": 261.84, "text": " So the first one is the model has to represent"}, {"start": 261.84, "end": 263.48, "text": " a wide range of signals,"}, {"start": 263.48, "end": 265.48, "text": " such as not to overfit the training set"}, {"start": 265.48, "end": 270.0, "text": " or produce artifact laden audio outside its comfort zone."}, {"start": 270.0, "end": 273.09999999999997, "text": " So basically outside of its training data set."}, {"start": 273.09999999999997, "end": 275.24, "text": " And this is something we are well familiar with"}, {"start": 275.24, "end": 276.96, "text": " and the way they cope with this problem"}, {"start": 276.96, "end": 279.74, "text": " is that they have a diverse set of data sets"}, {"start": 279.74, "end": 283.36, "text": " and diverse domains such as general speech"}, {"start": 283.36, "end": 285.76, "text": " and noise speech and music, et cetera, et cetera."}, {"start": 285.76, "end": 288.32, "text": " Okay, and then the other problem is that"}, {"start": 288.32, "end": 290.6, "text": " instead of compressing efficiently,"}, {"start": 290.6, "end": 293.42, "text": " both in compute time and in size."}, {"start": 293.42, "end": 295.2, "text": " So for the compute time, they mentioned here,"}, {"start": 295.2, "end": 296.76, "text": " so for the former, we'll limit ourselves"}, {"start": 296.76, "end": 299.72, "text": " to models that run in real time on a single CPU core."}, {"start": 299.72, "end": 301.98, "text": " So I think they've tested all of these models"}, {"start": 301.98, "end": 305.76, "text": " on a MacBook 2019 or something and that's kind of cool."}, {"start": 305.76, "end": 306.88, "text": " And then for the latter,"}, {"start": 306.88, "end": 309.40000000000003, "text": " we use residual vector quantization"}, {"start": 309.40000000000003, "end": 313.08000000000004, "text": " of the neural encoder floating point output."}, {"start": 313.08, "end": 318.0, "text": " Okay, so some of these ideas are probably familiar to you"}, {"start": 318.0, "end": 321.64, "text": " in case you watched some of my videos like VQVAE or VQGAN"}, {"start": 321.64, "end": 325.08, "text": " or any of the derivatives of those models."}, {"start": 325.08, "end": 327.08, "text": " But I'm gonna still cover and explain"}, {"start": 327.08, "end": 329.44, "text": " what those mean in a second."}, {"start": 329.44, "end": 331.64, "text": " Okay, so let me quickly walk you through"}, {"start": 331.64, "end": 334.28, "text": " the high level diagram here."}, {"start": 334.28, "end": 336.03999999999996, "text": " So let's see how this thing is constructed."}, {"start": 336.03999999999996, "end": 338.14, "text": " So on the left side of the diagram here,"}, {"start": 338.14, "end": 340.12, "text": " we have a raw audio signal"}, {"start": 340.12, "end": 343.2, "text": " and we also have the male spectrogram representation."}, {"start": 343.2, "end": 346.68, "text": " So that's basically on one axis here, we have frequency."}, {"start": 346.68, "end": 351.42, "text": " On the other axis, we have basically time axis."}, {"start": 351.42, "end": 355.4, "text": " And then the color of the dot on this diagram"}, {"start": 355.4, "end": 358.92, "text": " tells you the amplitude of that particular sinusoid"}, {"start": 358.92, "end": 361.48, "text": " of that particular frequency at that point of time,"}, {"start": 361.48, "end": 362.36, "text": " if that makes sense."}, {"start": 362.36, "end": 364.72, "text": " So again, do watch the audio gen in case you're confused,"}, {"start": 364.72, "end": 368.08, "text": " but that's basically equivalent representation"}, {"start": 368.08, "end": 369.12, "text": " to the raw audio."}, {"start": 369.12, "end": 370.56, "text": " So this is like the time representation,"}, {"start": 370.56, "end": 372.6, "text": " this is the frequency space representation"}, {"start": 372.6, "end": 373.92, "text": " of the audio signal."}, {"start": 373.92, "end": 375.88, "text": " Okay, and then there's a couple of things they do."}, {"start": 375.88, "end": 380.2, "text": " So first they kind of pass the signal through the encoder,"}, {"start": 380.2, "end": 381.74, "text": " which is built up as you can see here"}, {"start": 381.74, "end": 384.48, "text": " out of these COM1D layers."}, {"start": 384.48, "end": 386.12, "text": " And they additionally have the LSTM,"}, {"start": 386.12, "end": 389.44, "text": " a two layer LSTM sub module here,"}, {"start": 389.44, "end": 392.24, "text": " which helps to integrate the temporal information"}, {"start": 392.24, "end": 395.4, "text": " because otherwise we'd be basically,"}, {"start": 395.4, "end": 399.96, "text": " so like given the particular architecture they are using,"}, {"start": 399.96, "end": 404.96, "text": " they basically encode 320 samples from the raw audio"}, {"start": 405.02, "end": 410.02, "text": " into a single vector here, that's 128 dimensions large."}, {"start": 411.47999999999996, "end": 416.47999999999996, "text": " Okay, and they refer to this thing here as the frame."}, {"start": 417.44, "end": 419.23999999999995, "text": " So this thing is referred to as frame,"}, {"start": 419.23999999999995, "end": 421.47999999999996, "text": " but it also referred to some of these things here as frames,"}, {"start": 421.47999999999996, "end": 422.47999999999996, "text": " it's kind of inconvenient."}, {"start": 422.48, "end": 425.64000000000004, "text": " Okay, so once you have this representation"}, {"start": 425.64000000000004, "end": 427.3, "text": " at the output of the encoder,"}, {"start": 427.3, "end": 429.32, "text": " what you do is you quantize it."}, {"start": 429.32, "end": 431.76, "text": " So that means each of these vector quantizations"}, {"start": 431.76, "end": 434.68, "text": " is gonna have associated code book table."}, {"start": 434.68, "end": 437.16, "text": " So basically that means they're gonna have something"}, {"start": 437.16, "end": 440.52000000000004, "text": " like this, so like they have a table"}, {"start": 440.52000000000004, "end": 445.40000000000003, "text": " and the dimensions of this table are gonna be 1024."}, {"start": 446.36, "end": 450.40000000000003, "text": " So they have 1024 vectors and each of these vectors are,"}, {"start": 450.4, "end": 453.15999999999997, "text": " so the dimension here is 128,"}, {"start": 453.15999999999997, "end": 456.08, "text": " the same as the output of the encoder, okay?"}, {"start": 456.08, "end": 458.12, "text": " And each of these, so 128,"}, {"start": 458.12, "end": 460.4, "text": " and each of these vector quantization modules"}, {"start": 460.4, "end": 463.0, "text": " is gonna have a particular independent code table"}, {"start": 463.0, "end": 463.84, "text": " like this one."}, {"start": 463.84, "end": 467.79999999999995, "text": " And so what these do then is you take this particular vector"}, {"start": 467.79999999999995, "end": 469.28, "text": " at the output of the encoder"}, {"start": 469.28, "end": 472.64, "text": " and you find the closest vector in this code table"}, {"start": 472.64, "end": 475.59999999999997, "text": " and you then just grab its index, okay?"}, {"start": 475.59999999999997, "end": 479.03999999999996, "text": " So when I say close, I think they're using the L2 norm."}, {"start": 479.04, "end": 481.36, "text": " So basically just find the closest vector here"}, {"start": 481.36, "end": 484.0, "text": " and then let's say for example, this vector here"}, {"start": 485.0, "end": 487.0, "text": " is the one that corresponds,"}, {"start": 487.0, "end": 488.68, "text": " which is the closest one to this one,"}, {"start": 488.68, "end": 490.52000000000004, "text": " then you just grab its index here."}, {"start": 490.52000000000004, "end": 494.44, "text": " So that might be like, I don't know, like 238."}, {"start": 494.44, "end": 497.44, "text": " And now you use 238 as a representation"}, {"start": 497.44, "end": 498.28000000000003, "text": " of this whole vector."}, {"start": 498.28000000000003, "end": 499.76, "text": " And that's how you kind of compress"}, {"start": 499.76, "end": 502.48, "text": " down the output representation here."}, {"start": 502.48, "end": 504.76, "text": " And then additionally they have,"}, {"start": 504.76, "end": 506.6, "text": " as you can see, they have multiple of these"}, {"start": 506.6, "end": 508.24, "text": " and that's because they do something"}, {"start": 508.24, "end": 509.84000000000003, "text": " called residual quantization."}, {"start": 509.84000000000003, "end": 512.64, "text": " So they first snap, so snap,"}, {"start": 512.64, "end": 515.88, "text": " they find basically, they find the closest vector"}, {"start": 515.88, "end": 517.2, "text": " and then they take this vector"}, {"start": 517.2, "end": 519.92, "text": " and they subtract it from the original vector"}, {"start": 519.92, "end": 521.88, "text": " and then the residue is encoded"}, {"start": 521.88, "end": 525.84, "text": " by the next vector quantization module."}, {"start": 525.84, "end": 527.08, "text": " And then rinse and repeat"}, {"start": 527.08, "end": 529.8, "text": " and that's how they basically encode this thing."}, {"start": 531.24, "end": 533.2, "text": " So you end up in this particular drawing here,"}, {"start": 533.2, "end": 535.5600000000001, "text": " you would end up instead of 128 dimensions,"}, {"start": 535.5600000000001, "end": 537.0600000000001, "text": " you would end up with four dimensions"}, {"start": 537.06, "end": 539.56, "text": " because you have four numbers being allocated"}, {"start": 539.56, "end": 540.8, "text": " to that particular vector."}, {"start": 540.8, "end": 542.1199999999999, "text": " And we have, again, in the background,"}, {"start": 542.1199999999999, "end": 543.8, "text": " we have four tables, okay?"}, {"start": 543.8, "end": 545.7199999999999, "text": " And then finally, so I'm gonna ignore"}, {"start": 545.7199999999999, "end": 549.4, "text": " the transformer module here for a second,"}, {"start": 549.4, "end": 552.92, "text": " but like finally, you can basically decode those signals"}, {"start": 552.92, "end": 554.4799999999999, "text": " because you have the numbers"}, {"start": 554.4799999999999, "end": 556.0799999999999, "text": " and then you can just grab the vector"}, {"start": 556.0799999999999, "end": 558.2399999999999, "text": " from the associated tables, you can sum them up"}, {"start": 558.2399999999999, "end": 560.0, "text": " and you form your representation"}, {"start": 560.0, "end": 561.8399999999999, "text": " which you then pass through the decoder, okay?"}, {"start": 561.8399999999999, "end": 564.06, "text": " And decoder is just basically you can see"}, {"start": 564.06, "end": 566.0799999999999, "text": " symmetric to what we had in the encoder."}, {"start": 566.08, "end": 567.96, "text": " So I'm gonna skip explaining it."}, {"start": 567.96, "end": 571.08, "text": " And finally, we end up with the reconstructed audio"}, {"start": 572.0400000000001, "end": 573.84, "text": " signal here, okay?"}, {"start": 573.84, "end": 577.24, "text": " So how the thing is trained is there are multiple losses"}, {"start": 577.24, "end": 579.64, "text": " and they also introduced this thing called a balancer"}, {"start": 579.64, "end": 582.12, "text": " which uses the gradient information to balance"}, {"start": 582.12, "end": 586.44, "text": " and scale up all of the losses as needed."}, {"start": 586.44, "end": 590.5200000000001, "text": " So first loss here is the LT"}, {"start": 590.5200000000001, "end": 592.8000000000001, "text": " which stands for time domain loss."}, {"start": 592.8000000000001, "end": 596.0, "text": " They basically do L1 between the raw representations"}, {"start": 596.0, "end": 601.0, "text": " of the reconstructed audio here and the input audio here."}, {"start": 601.0, "end": 602.52, "text": " And then they have the LS"}, {"start": 602.52, "end": 607.52, "text": " which is the basically a spectral domain L1 basically norm"}, {"start": 608.84, "end": 612.16, "text": " between the reconstructed mouse spectrogram"}, {"start": 612.16, "end": 613.92, "text": " and the input mouse spectrogram."}, {"start": 613.92, "end": 615.28, "text": " So that's what they do here."}, {"start": 615.28, "end": 617.28, "text": " And then finally, they have the LD and LG"}, {"start": 617.28, "end": 621.52, "text": " which are the generator and discriminator losses."}, {"start": 621.52, "end": 623.46, "text": " And you can see here, they're like a couple"}, {"start": 623.46, "end": 625.56, "text": " of identical discriminators."}, {"start": 625.56, "end": 628.8, "text": " And so what you do here is you just convert the audio signal"}, {"start": 628.8, "end": 631.68, "text": " into the real and imaginary STFD"}, {"start": 631.68, "end": 634.4, "text": " which are just again, the spectral representations."}, {"start": 634.4, "end": 636.1999999999999, "text": " And these are strictly more informative."}, {"start": 636.1999999999999, "end": 637.88, "text": " So let me just change the color here."}, {"start": 637.88, "end": 639.5999999999999, "text": " So these are strictly more informative"}, {"start": 639.5999999999999, "end": 641.6999999999999, "text": " than the mouse spectrogram down here"}, {"start": 641.6999999999999, "end": 646.06, "text": " because given this information, you can easily get"}, {"start": 646.06, "end": 647.92, "text": " the mouse spectrogram information"}, {"start": 647.92, "end": 650.16, "text": " because you just have to find the amplitude"}, {"start": 650.16, "end": 652.68, "text": " of these imaginary complex numbers."}, {"start": 652.68, "end": 654.8399999999999, "text": " And that's what gives you the amplitude"}, {"start": 654.84, "end": 656.2800000000001, "text": " of the mouse spectrogram."}, {"start": 656.2800000000001, "end": 659.9200000000001, "text": " Anyways, you kind of pass that through this encoder"}, {"start": 659.9200000000001, "end": 661.96, "text": " to the discriminator."}, {"start": 661.96, "end": 665.4, "text": " And then you train using the adversarial loss as usual."}, {"start": 665.4, "end": 668.72, "text": " So you just basically contrast, you give it the real signal"}, {"start": 668.72, "end": 670.52, "text": " you give it the generated signal"}, {"start": 670.52, "end": 672.88, "text": " and you teach the discriminator to discriminate"}, {"start": 672.88, "end": 675.1600000000001, "text": " between those two and you teach the generator"}, {"start": 675.1600000000001, "end": 677.9200000000001, "text": " which is in this particular case, this whole pipeline here"}, {"start": 677.9200000000001, "end": 682.36, "text": " how to generate signals that look as real ones."}, {"start": 682.36, "end": 683.84, "text": " Finally, a couple more losses here."}, {"start": 683.84, "end": 686.76, "text": " So we covered these ones and the final two ones"}, {"start": 686.76, "end": 687.6, "text": " are this one."}, {"start": 687.6, "end": 688.72, "text": " So this is the commitment loss."}, {"start": 688.72, "end": 692.52, "text": " And again, if you watched any of the VQVA, VQGAN papers"}, {"start": 692.52, "end": 694.08, "text": " you're familiar with this one."}, {"start": 694.08, "end": 697.48, "text": " And finally, there is the loss that corresponds"}, {"start": 697.48, "end": 698.98, "text": " to this transformer here."}, {"start": 698.98, "end": 701.8000000000001, "text": " So what the transformer does is it helps us"}, {"start": 701.8000000000001, "end": 706.14, "text": " to basically compress this latent representation"}, {"start": 706.14, "end": 708.96, "text": " to even lower dimensionality at the cost"}, {"start": 708.96, "end": 711.88, "text": " of having to spend more compute"}, {"start": 711.88, "end": 716.36, "text": " and basically increase the latency of your system."}, {"start": 716.36, "end": 719.64, "text": " But I'm gonna kind of go through"}, {"start": 719.64, "end": 721.32, "text": " probably if you have enough time, I'm gonna go through"}, {"start": 721.32, "end": 723.5, "text": " and show you how the language model part"}, {"start": 723.5, "end": 724.68, "text": " fits into this pipeline."}, {"start": 724.68, "end": 726.96, "text": " Once we start walking through the code."}, {"start": 726.96, "end": 729.76, "text": " Okay guys, let me quickly show you this"}, {"start": 729.76, "end": 732.28, "text": " residual vector quantization logic."}, {"start": 732.28, "end": 736.12, "text": " There is from this before I mentioned this sound stream"}, {"start": 736.12, "end": 740.04, "text": " paper and they have a very neat pseudocode"}, {"start": 740.04, "end": 742.92, "text": " for this residual vector quantization."}, {"start": 742.92, "end": 744.5999999999999, "text": " So let's kind of run through it."}, {"start": 744.5999999999999, "end": 746.8399999999999, "text": " So you can see here, Y is denoted as the output"}, {"start": 746.8399999999999, "end": 747.92, "text": " of the encoder."}, {"start": 747.92, "end": 749.76, "text": " We have multiple of these quantizers"}, {"start": 749.76, "end": 751.16, "text": " and there is NQ of them."}, {"start": 751.16, "end": 754.3, "text": " So on this particular drawing, NQ is four"}, {"start": 754.3, "end": 758.64, "text": " because we have one, two, three, four"}, {"start": 758.64, "end": 759.7199999999999, "text": " of these components here."}, {"start": 759.7199999999999, "end": 761.12, "text": " Okay."}, {"start": 761.12, "end": 762.92, "text": " So let's go back here."}, {"start": 762.92, "end": 765.24, "text": " And so here is how it works."}, {"start": 765.24, "end": 768.4, "text": " So initially we initialize the residual variable"}, {"start": 768.4, "end": 770.6, "text": " with Y, which is the output from the encoder."}, {"start": 770.6, "end": 773.0, "text": " And then you can see here, we quantize it"}, {"start": 773.84, "end": 776.56, "text": " and then, and basically then we add that"}, {"start": 776.56, "end": 778.28, "text": " to the reconstructed signal."}, {"start": 778.28, "end": 782.64, "text": " Then we subtract that quantize, those quantize vectors"}, {"start": 782.64, "end": 784.56, "text": " from the initial residual vectors."}, {"start": 784.56, "end": 786.92, "text": " And then we use those residual ones"}, {"start": 786.92, "end": 789.62, "text": " and we feed them into the next quantizer."}, {"start": 789.62, "end": 792.62, "text": " And that's basically how it looks like on a high level."}, {"start": 793.52, "end": 796.98, "text": " I also wanna walk you through the rationale."}, {"start": 796.98, "end": 798.1999999999999, "text": " Why are we doing this?"}, {"start": 798.2, "end": 800.44, "text": " So I think a couple of these paragraphs"}, {"start": 800.44, "end": 803.24, "text": " might help elucidate the idea behind it."}, {"start": 803.24, "end": 804.08, "text": " Okay."}, {"start": 804.08, "end": 806.76, "text": " So the vector quantizer VQ proposed in these papers here"}, {"start": 806.76, "end": 809.6, "text": " in the context of VQ VAE meets this requirement."}, {"start": 809.6, "end": 812.84, "text": " This vector quantizer learns a code book of N vectors."}, {"start": 812.84, "end": 815.72, "text": " So N was 1024 in our particular case."}, {"start": 815.72, "end": 819.2, "text": " To encode each D dimensional frame, and D was 128"}, {"start": 820.32, "end": 824.1, "text": " of the output of the encoder here denoted as NQ of X."}, {"start": 824.1, "end": 826.84, "text": " The encoded audio, as you can see the dimensionality"}, {"start": 826.84, "end": 828.22, "text": " being S times D."}, {"start": 828.22, "end": 833.22, "text": " So if we have a one second audio signal, S will be 75"}, {"start": 833.8000000000001, "end": 836.9200000000001, "text": " because we have the hop being 320."}, {"start": 836.9200000000001, "end": 838.6, "text": " And I'm just using particular numbers"}, {"start": 838.6, "end": 840.86, "text": " because it's easier to kind of create a mental picture."}, {"start": 840.86, "end": 844.94, "text": " But basically imagine you have a 24 kilohertz signal"}, {"start": 844.94, "end": 849.44, "text": " and then one second means you'll have 24,000 samples."}, {"start": 849.44, "end": 853.0, "text": " If we divide that by 320, we end up with 75."}, {"start": 853.0, "end": 856.6800000000001, "text": " So that means that for a signal of audio,"}, {"start": 856.68, "end": 859.0799999999999, "text": " for one second of the audio and the input"}, {"start": 859.0799999999999, "end": 862.1999999999999, "text": " we'll end up with 75 frames at the output."}, {"start": 862.1999999999999, "end": 865.9599999999999, "text": " And then D is just the, as I said, 128."}, {"start": 865.9599999999999, "end": 866.8, "text": " Okay."}, {"start": 866.8, "end": 868.04, "text": " So that's then mapped to a sequence"}, {"start": 868.04, "end": 870.2399999999999, "text": " of one hot vectors of shape SN."}, {"start": 870.2399999999999, "end": 873.7199999999999, "text": " So N is again 1024, which can be represented"}, {"start": 873.7199999999999, "end": 876.0999999999999, "text": " using this many bits."}, {"start": 876.0999999999999, "end": 876.9399999999999, "text": " Okay."}, {"start": 876.9399999999999, "end": 881.12, "text": " So because this is 1024, this is going to be 10."}, {"start": 881.12, "end": 884.76, "text": " So once you do the log, you'll end up with 10 bits here"}, {"start": 884.76, "end": 886.72, "text": " as that's how much you need to index"}, {"start": 886.72, "end": 888.4, "text": " into those codebook tables."}, {"start": 888.4, "end": 889.22, "text": " Okay."}, {"start": 889.22, "end": 890.72, "text": " So let's see a concrete example here."}, {"start": 890.72, "end": 893.68, "text": " So let us consider a codec targeting a bit rate"}, {"start": 893.68, "end": 897.22, "text": " of six K, so kilo bits per second."}, {"start": 897.22, "end": 900.48, "text": " When using a striding factor of 320,"}, {"start": 900.48, "end": 903.76, "text": " each second of audio at sampling rate 24 kilohertz"}, {"start": 903.76, "end": 907.04, "text": " is represented by 75 frames at the output of the encoder."}, {"start": 907.04, "end": 911.16, "text": " This corresponds to 80 bits allocated to each frame, right?"}, {"start": 911.16, "end": 915.04, "text": " So if we have the target bandwidth of 6,000"}, {"start": 915.04, "end": 917.76, "text": " and we have 75 frames, that means we can allocate"}, {"start": 917.76, "end": 922.52, "text": " up to 80 bits and we will still not basically"}, {"start": 922.52, "end": 925.14, "text": " go over the target bandwidth."}, {"start": 926.0, "end": 928.0, "text": " So using a plain vector quantizer,"}, {"start": 928.0, "end": 930.64, "text": " this requires storing a codebook with two raised"}, {"start": 930.64, "end": 934.56, "text": " to the power of 80 vectors, which is obviously unfeasible."}, {"start": 934.56, "end": 935.4, "text": " Okay."}, {"start": 935.4, "end": 936.64, "text": " And then they say to address this issue,"}, {"start": 936.64, "end": 940.0799999999999, "text": " we adopt a residual vector quantizer,"}, {"start": 940.08, "end": 942.2, "text": " AKA multi-stage vector quantizer,"}, {"start": 942.2, "end": 944.96, "text": " which cascades NQ layers, so we have four"}, {"start": 944.96, "end": 948.36, "text": " in the drawing below, of VQ as follows."}, {"start": 948.36, "end": 951.9200000000001, "text": " The unquantized input vector is passed through our first VQ"}, {"start": 951.9200000000001, "end": 954.2800000000001, "text": " and quantization residuals are computed."}, {"start": 954.2800000000001, "end": 956.32, "text": " The residuals are then iteratively quantized"}, {"start": 956.32, "end": 959.76, "text": " by a sequence of additional NQ minus one vector quantizers"}, {"start": 959.76, "end": 961.32, "text": " as described in algorithm one,"}, {"start": 961.32, "end": 962.88, "text": " which was the one we saw here."}, {"start": 964.12, "end": 968.0, "text": " The total rate budget is uniformly allocated to each VQ,"}, {"start": 968.0, "end": 972.44, "text": " i.e. we have R divided by NQ, so R was 80 bits"}, {"start": 972.44, "end": 974.44, "text": " and then we divide that by the number of quantizers"}, {"start": 974.44, "end": 977.0, "text": " and that's how many bits each of the quantizers"}, {"start": 977.0, "end": 980.04, "text": " has to basically allocate."}, {"start": 980.04, "end": 981.96, "text": " For example, when using NQ eight,"}, {"start": 981.96, "end": 985.68, "text": " each quantizer uses a codebook of size 1,024"}, {"start": 985.68, "end": 987.68, "text": " and that's the number we'll be using in this paper as well."}, {"start": 987.68, "end": 988.74, "text": " Okay."}, {"start": 988.74, "end": 991.72, "text": " So hopefully that makes sense, so that's why"}, {"start": 991.72, "end": 994.1, "text": " this approach was basically developed."}, {"start": 994.1, "end": 999.1, "text": " It makes it easy to get to the target bandwidth"}, {"start": 999.22, "end": 1000.94, "text": " without having to have huge,"}, {"start": 1000.94, "end": 1004.5400000000001, "text": " enormous basically codebooks."}, {"start": 1004.5400000000001, "end": 1009.5400000000001, "text": " Okay, so let's now quickly go through some of the details."}, {"start": 1010.4200000000001, "end": 1013.34, "text": " Basically, as I said, they have the non-streamable"}, {"start": 1013.34, "end": 1015.78, "text": " and the streamable architectures."}, {"start": 1015.78, "end": 1017.86, "text": " The streamable ones, so they say here,"}, {"start": 1017.86, "end": 1019.78, "text": " thanks to this padding scheme,"}, {"start": 1019.78, "end": 1022.62, "text": " which is basically left padding, the causal padding,"}, {"start": 1022.62, "end": 1024.72, "text": " the model can output 320 samples,"}, {"start": 1024.72, "end": 1027.18, "text": " which corresponds to 13 milliseconds,"}, {"start": 1027.18, "end": 1031.18, "text": " given a 24 kilohertz sampling rate."}, {"start": 1031.18, "end": 1034.8, "text": " As soon as the first 320 samples are received."}, {"start": 1034.8, "end": 1037.8, "text": " So that means as soon as you have 320 samples on the input,"}, {"start": 1037.8, "end": 1039.94, "text": " you can already encode it into a frame"}, {"start": 1039.94, "end": 1041.44, "text": " and then you can quantize that frame"}, {"start": 1041.44, "end": 1043.7, "text": " and you can send that down the line"}, {"start": 1043.7, "end": 1046.26, "text": " and then you can just keep on doing that"}, {"start": 1046.26, "end": 1048.44, "text": " as the signal keeps on coming"}, {"start": 1048.44, "end": 1050.34, "text": " as you're receiving your input signal."}, {"start": 1050.34, "end": 1052.06, "text": " You can keep on encoding it and sending it."}, {"start": 1052.06, "end": 1054.46, "text": " So that means that's why it's streamable."}, {"start": 1054.46, "end": 1057.34, "text": " Okay, and then they mentioned by selecting a variable number"}, {"start": 1057.34, "end": 1059.02, "text": " of residual steps at train time,"}, {"start": 1059.02, "end": 1060.34, "text": " a single model can be used"}, {"start": 1060.34, "end": 1063.28, "text": " to support multiple bandwidth targets."}, {"start": 1063.28, "end": 1065.86, "text": " That's something we saw up there"}, {"start": 1065.86, "end": 1067.5, "text": " with this particular example."}, {"start": 1067.5, "end": 1070.1, "text": " So basically, depending on how many"}, {"start": 1070.1, "end": 1073.6599999999999, "text": " of these quantizer models you add,"}, {"start": 1073.6599999999999, "end": 1076.7, "text": " you can achieve a particular bandwidth."}, {"start": 1076.7, "end": 1077.78, "text": " So here we picked eight"}, {"start": 1077.78, "end": 1080.74, "text": " because we want to have 80 bits per frame"}, {"start": 1080.74, "end": 1084.34, "text": " and thus we want to have six kilobits per second."}, {"start": 1084.34, "end": 1085.58, "text": " Okay."}, {"start": 1085.58, "end": 1088.74, "text": " Okay, so I did mention the language model"}, {"start": 1088.74, "end": 1089.74, "text": " that's being used here."}, {"start": 1089.74, "end": 1091.7, "text": " So implemented as a transformer"}, {"start": 1091.7, "end": 1094.78, "text": " to further reduce the bandwidth."}, {"start": 1094.78, "end": 1096.78, "text": " So let's kind of read this paragraph"}, {"start": 1096.78, "end": 1099.26, "text": " and then later we probably will have enough time"}, {"start": 1099.26, "end": 1101.3, "text": " to just go through that and see it in code."}, {"start": 1101.3, "end": 1103.26, "text": " So for a timestamp T,"}, {"start": 1103.26, "end": 1107.64, "text": " the discrete representation obtained at time T minus one"}, {"start": 1107.64, "end": 1110.54, "text": " is transformed into a continuous representation"}, {"start": 1110.54, "end": 1112.58, "text": " using learned embedding tables,"}, {"start": 1112.58, "end": 1115.58, "text": " one for each code book and which are then summed."}, {"start": 1115.58, "end": 1117.18, "text": " So we'll see that in the code a bit later."}, {"start": 1117.18, "end": 1119.04, "text": " But basically, so discrete representation"}, {"start": 1119.04, "end": 1120.3999999999999, "text": " are these numbers I mentioned."}, {"start": 1120.3999999999999, "end": 1125.3999999999999, "text": " So for example, if this is a step T minus one,"}, {"start": 1125.94, "end": 1127.86, "text": " you can see that the discrete representation"}, {"start": 1127.86, "end": 1130.46, "text": " is basically these four numbers here."}, {"start": 1130.46, "end": 1132.22, "text": " And when I say four numbers,"}, {"start": 1132.22, "end": 1135.94, "text": " even though it's kind of float initially,"}, {"start": 1135.94, "end": 1138.32, "text": " which means you have like 64 bits,"}, {"start": 1138.32, "end": 1141.4199999999998, "text": " we know that it doesn't have to be larger than 10 bits"}, {"start": 1141.4199999999998, "end": 1143.9399999999998, "text": " because it just needs to index into this table"}, {"start": 1143.9399999999998, "end": 1146.74, "text": " that has 1024 vectors."}, {"start": 1146.74, "end": 1147.74, "text": " And that's why these numbers"}, {"start": 1147.74, "end": 1149.82, "text": " are actually just 10 bit numbers, right?"}, {"start": 1150.6799999999998, "end": 1152.54, "text": " Okay, so let's go back here."}, {"start": 1154.8999999999999, "end": 1156.1, "text": " And then for T equals zero,"}, {"start": 1156.1, "end": 1157.34, "text": " a special token is used instead."}, {"start": 1157.34, "end": 1158.86, "text": " Okay, the output of the transformer"}, {"start": 1158.86, "end": 1160.8, "text": " is fed into ankyl linear layers"}, {"start": 1160.8, "end": 1162.22, "text": " with as many output channels"}, {"start": 1162.22, "end": 1163.98, "text": " as the cardinality of each code book,"}, {"start": 1163.98, "end": 1166.02, "text": " which is 1024 in our examples,"}, {"start": 1166.02, "end": 1168.78, "text": " giving us the logits of the estimated distribution"}, {"start": 1168.78, "end": 1170.84, "text": " over each code book for time T."}, {"start": 1171.8, "end": 1174.5, "text": " Okay, so basically instead of just finding"}, {"start": 1174.5, "end": 1178.82, "text": " the closest vector in the code book table"}, {"start": 1178.82, "end": 1180.7, "text": " and snapping our output vector"}, {"start": 1180.7, "end": 1183.06, "text": " to that particular code book vector,"}, {"start": 1183.06, "end": 1185.04, "text": " instead of that, we basically do,"}, {"start": 1185.04, "end": 1188.34, "text": " we form like a probability distribution"}, {"start": 1188.34, "end": 1189.5, "text": " across the whole code book."}, {"start": 1189.5, "end": 1191.1, "text": " So this is the code book table"}, {"start": 1191.1, "end": 1194.62, "text": " and we form a probability distribution across its indices"}, {"start": 1194.62, "end": 1198.7199999999998, "text": " and then combined with arithmetic coding,"}, {"start": 1198.7199999999998, "end": 1200.82, "text": " they show that you can further compress"}, {"start": 1200.82, "end": 1204.26, "text": " your latent representation using this approach."}, {"start": 1204.26, "end": 1206.12, "text": " Okay, let's continue here."}, {"start": 1207.2199999999998, "end": 1209.2399999999998, "text": " So here are the formulas for the losses."}, {"start": 1209.2399999999998, "end": 1211.02, "text": " I'm not gonna spend too much time here,"}, {"start": 1211.02, "end": 1212.58, "text": " but you can see here L1 norm"}, {"start": 1212.58, "end": 1214.7399999999998, "text": " between the time domain signals."}, {"start": 1214.7399999999998, "end": 1218.02, "text": " And then you can see here a combination of L1 and L2 norms"}, {"start": 1218.02, "end": 1220.86, "text": " between the mal spectrograms across various scales."}, {"start": 1220.86, "end": 1223.78, "text": " And then there is the discriminative loss."}, {"start": 1223.78, "end": 1226.8999999999999, "text": " Basically, you can see here the generator loss"}, {"start": 1227.76, "end": 1230.44, "text": " and you can also find the discriminator loss"}, {"start": 1230.44, "end": 1231.78, "text": " somewhere in here."}, {"start": 1231.78, "end": 1233.1, "text": " So here it is."}, {"start": 1233.1, "end": 1235.3, "text": " But I'm not gonna spend any more time on this"}, {"start": 1235.3, "end": 1236.78, "text": " because I've covered this."}, {"start": 1236.78, "end": 1240.26, "text": " Also, I think the audio gen or VQ gen paper"}, {"start": 1240.26, "end": 1243.3799999999999, "text": " basically have pretty much same losses as this paper."}, {"start": 1243.3799999999999, "end": 1245.26, "text": " So I'm gonna kind of skip all of those details."}, {"start": 1245.26, "end": 1247.58, "text": " Here is also the commitment loss,"}, {"start": 1247.58, "end": 1250.78, "text": " fairly familiar to those of you who watch those videos"}, {"start": 1250.78, "end": 1252.3999999999999, "text": " and who know about those papers."}, {"start": 1252.4, "end": 1256.7, "text": " And finally, the final loss is just a weighted sum"}, {"start": 1256.7, "end": 1258.8200000000002, "text": " of all of these various losses."}, {"start": 1258.8200000000002, "end": 1263.3200000000002, "text": " And in order to avoid having to basically do like a search"}, {"start": 1263.3200000000002, "end": 1266.5400000000002, "text": " and find the right coefficients, they make it adaptive."}, {"start": 1266.5400000000002, "end": 1269.22, "text": " And so they basically calculate the gradients"}, {"start": 1269.22, "end": 1271.3200000000002, "text": " of these losses with respect to the input."}, {"start": 1271.3200000000002, "end": 1273.1000000000001, "text": " And then they use the norm of the gradient"}, {"start": 1273.1000000000001, "end": 1276.6200000000001, "text": " to automatically rescale the importance"}, {"start": 1276.6200000000001, "end": 1277.8200000000002, "text": " of each of these losses."}, {"start": 1277.8200000000002, "end": 1278.9, "text": " So you can see here,"}, {"start": 1278.9, "end": 1282.42, "text": " if a gradient of a particular loss was huge,"}, {"start": 1282.42, "end": 1285.14, "text": " so we have this, this is basically a norm"}, {"start": 1285.14, "end": 1286.7800000000002, "text": " and exponentially,"}, {"start": 1288.14, "end": 1290.0600000000002, "text": " and they've done an exponentially moving average"}, {"start": 1290.0600000000002, "end": 1291.3400000000001, "text": " on the norm of the gradient."}, {"start": 1291.3400000000001, "end": 1294.14, "text": " And you can see if that was huge historically,"}, {"start": 1294.14, "end": 1296.94, "text": " then that means this is gonna be down weighted."}, {"start": 1296.94, "end": 1298.94, "text": " So the gradients from that particular loss"}, {"start": 1298.94, "end": 1299.9, "text": " are gonna be down weighted"}, {"start": 1299.9, "end": 1304.0600000000002, "text": " because this is a big number in the denominator."}, {"start": 1304.0600000000002, "end": 1307.1000000000001, "text": " And because of that, that loss is gonna be down weighted."}, {"start": 1307.1, "end": 1309.1399999999999, "text": " So that's the kind of the intuition behind it."}, {"start": 1309.1399999999999, "end": 1310.36, "text": " Okay."}, {"start": 1310.36, "end": 1311.1999999999998, "text": " So as I said,"}, {"start": 1312.3, "end": 1316.8999999999999, "text": " they use diverse domains to train these models."}, {"start": 1316.8999999999999, "end": 1319.54, "text": " That's how they ensure that they don't overfit"}, {"start": 1319.54, "end": 1321.1799999999998, "text": " to particular domain."}, {"start": 1321.1799999999998, "end": 1324.8999999999999, "text": " And yeah, so they mentioned here speech,"}, {"start": 1324.8999999999999, "end": 1327.9399999999998, "text": " noisy speech, music, and general audio."}, {"start": 1327.9399999999998, "end": 1328.9399999999998, "text": " And here are the results."}, {"start": 1328.9399999999998, "end": 1331.6999999999998, "text": " So you can see here how they compare"}, {"start": 1331.6999999999998, "end": 1334.78, "text": " against some of the baselines, such as OPUS and EDS,"}, {"start": 1334.78, "end": 1337.42, "text": " which are basically signal processing baselines."}, {"start": 1337.42, "end": 1339.1, "text": " And then we have Lyra from Google,"}, {"start": 1339.1, "end": 1341.46, "text": " which is a neural codec as well."}, {"start": 1341.46, "end": 1345.1, "text": " And you can see it basically outperforms all of those models"}, {"start": 1345.1, "end": 1347.92, "text": " for all of the target bandwidths."}, {"start": 1349.18, "end": 1351.62, "text": " Additionally, this entropy coded one,"}, {"start": 1351.62, "end": 1353.66, "text": " which uses the transformer"}, {"start": 1353.66, "end": 1356.26, "text": " to further compress the representation,"}, {"start": 1356.26, "end": 1360.5, "text": " basically outperforms for a lower target bandwidths,"}, {"start": 1360.5, "end": 1363.1, "text": " outperforms the normal one,"}, {"start": 1363.1, "end": 1365.8999999999999, "text": " the normal encodec without the transformer."}, {"start": 1366.8999999999999, "end": 1368.4199999999998, "text": " Okay, guys."}, {"start": 1368.4199999999998, "end": 1371.4199999999998, "text": " Let's see what else is interesting here."}, {"start": 1371.4199999999998, "end": 1373.56, "text": " Yeah, this result here is kinda cool."}, {"start": 1373.56, "end": 1375.3, "text": " So I think I mentioned this in the beginning of the video."}, {"start": 1375.3, "end": 1377.74, "text": " Basically, you can see here that MP3"}, {"start": 1377.74, "end": 1380.8999999999999, "text": " with the target bandwidth of 64 kilobits per second"}, {"start": 1380.8999999999999, "end": 1383.52, "text": " has the same, this MUSHR score,"}, {"start": 1383.52, "end": 1385.62, "text": " which is just basically subjective score,"}, {"start": 1386.9399999999998, "end": 1390.62, "text": " and has the same score as this particular encodec"}, {"start": 1390.62, "end": 1392.8999999999999, "text": " with six kilobits per second."}, {"start": 1392.9, "end": 1395.98, "text": " So you can see these are fairly on-pair."}, {"start": 1395.98, "end": 1399.14, "text": " And you can see that already with 12 kilobits per second,"}, {"start": 1399.14, "end": 1403.94, "text": " it heavily outperforms the MP3 format."}, {"start": 1403.94, "end": 1406.5800000000002, "text": " And also, if you include the entropy coding,"}, {"start": 1406.5800000000002, "end": 1408.4, "text": " then you can even further reduce."}, {"start": 1408.4, "end": 1410.74, "text": " With 4.2 kilobits per second,"}, {"start": 1410.74, "end": 1413.9, "text": " you outperform 64 kilobits per second MP3,"}, {"start": 1413.9, "end": 1418.9, "text": " and you have comparable basically subjective metrics."}, {"start": 1418.9, "end": 1419.7800000000002, "text": " That's fairly cool."}, {"start": 1419.7800000000002, "end": 1422.8600000000001, "text": " Okay, having seen all of this, having seen the paper,"}, {"start": 1422.86, "end": 1425.86, "text": " let's now switch our focus to the actual code."}, {"start": 1425.86, "end": 1427.9799999999998, "text": " And hopefully that's gonna help us"}, {"start": 1427.9799999999998, "end": 1430.04, "text": " understand this much better."}, {"start": 1430.04, "end": 1432.0, "text": " So I just created a long JSON,"}, {"start": 1432.0, "end": 1434.3799999999999, "text": " and I basically passed, as you can see here,"}, {"start": 1434.3799999999999, "end": 1437.26, "text": " the arguments, I just have this test 24K."}, {"start": 1438.12, "end": 1440.54, "text": " I'm gonna pronounce this as wave signal."}, {"start": 1440.54, "end": 1442.1999999999998, "text": " I'm not sure how this format is pronounced,"}, {"start": 1442.1999999999998, "end": 1444.8999999999999, "text": " but wave sounds easy enough."}, {"start": 1444.8999999999999, "end": 1448.3799999999999, "text": " Basically, test 24K is already a part of the codebase."}, {"start": 1448.3799999999999, "end": 1450.82, "text": " It's checked in, so you just have to clone the repo,"}, {"start": 1450.82, "end": 1453.1799999999998, "text": " and you will have it out of the box."}, {"start": 1453.1799999999998, "end": 1454.02, "text": " So that's it."}, {"start": 1454.02, "end": 1455.36, "text": " That's everything we have to set,"}, {"start": 1455.36, "end": 1457.46, "text": " and later I'm just gonna add this LM flag"}, {"start": 1457.46, "end": 1461.46, "text": " to see how the language model fits into this whole picture."}, {"start": 1461.46, "end": 1463.4199999999998, "text": " And there is also the HQ,"}, {"start": 1463.4199999999998, "end": 1467.3799999999999, "text": " which is the 48 kilohertz non-streamable model,"}, {"start": 1467.3799999999999, "end": 1468.74, "text": " but I'm not sure I'll have enough time."}, {"start": 1468.74, "end": 1471.6599999999999, "text": " I don't wanna go over one hour in this video."}, {"start": 1471.6599999999999, "end": 1473.6599999999999, "text": " So let's basically start."}, {"start": 1473.6599999999999, "end": 1476.3799999999999, "text": " I'm gonna enable all of the breakpoints here,"}, {"start": 1476.3799999999999, "end": 1479.2, "text": " and let's run this thing."}, {"start": 1479.2, "end": 1482.54, "text": " Okay, again, I'm gonna focus only on the crucial parts."}, {"start": 1482.54, "end": 1486.18, "text": " So you just make sure that the audio signal exists,"}, {"start": 1486.18, "end": 1491.06, "text": " and then here you can see that because we have wave signal,"}, {"start": 1491.06, "end": 1493.3400000000001, "text": " and this is basically ECDC,"}, {"start": 1493.3400000000001, "end": 1496.3, "text": " which is the compressed format suffix."}, {"start": 1496.3, "end": 1497.78, "text": " Because of that, we are gonna go"}, {"start": 1497.78, "end": 1499.42, "text": " into the compression branch."}, {"start": 1499.42, "end": 1502.66, "text": " So, but by understanding this compressed function,"}, {"start": 1502.66, "end": 1504.22, "text": " this decompressed function is gonna be"}, {"start": 1504.22, "end": 1505.46, "text": " fairly trivial to understand."}, {"start": 1505.46, "end": 1508.06, "text": " So I'm gonna focus on this particular branch of the code."}, {"start": 1508.06, "end": 1511.26, "text": " Okay, so we formed the output,"}, {"start": 1511.26, "end": 1512.98, "text": " as we just changed the suffix."}, {"start": 1512.98, "end": 1514.6599999999999, "text": " Basically, it's also gonna be, as you can see here,"}, {"start": 1514.6599999999999, "end": 1518.78, "text": " test24k, oops, test24k ECDC."}, {"start": 1518.78, "end": 1519.8999999999999, "text": " That's the output."}, {"start": 1519.8999999999999, "end": 1522.74, "text": " And then we check that the output exists,"}, {"start": 1522.74, "end": 1524.26, "text": " and because it already exists,"}, {"start": 1524.26, "end": 1528.72, "text": " I'll just have to first basically delete that signal."}, {"start": 1528.72, "end": 1532.34, "text": " I'm gonna delete this output compressed file,"}, {"start": 1532.34, "end": 1533.22, "text": " and now it's gonna work."}, {"start": 1533.22, "end": 1534.3799999999999, "text": " So this is gonna happen to you as well."}, {"start": 1534.3799999999999, "end": 1536.6599999999999, "text": " So I'm not gonna edit this out."}, {"start": 1536.66, "end": 1539.5, "text": " I'm just gonna restart the program again,"}, {"start": 1539.5, "end": 1541.46, "text": " and this time it will not fail."}, {"start": 1541.46, "end": 1544.46, "text": " So let's skip to this check output,"}, {"start": 1544.46, "end": 1546.26, "text": " and this time this will not fail,"}, {"start": 1546.26, "end": 1547.3000000000002, "text": " so everything's fine there."}, {"start": 1547.3000000000002, "end": 1548.1200000000001, "text": " Cool."}, {"start": 1548.1200000000001, "end": 1549.88, "text": " So because HQ is set to false,"}, {"start": 1549.88, "end": 1552.8600000000001, "text": " we're gonna deal with 24 kilohertz signal."}, {"start": 1552.8600000000001, "end": 1554.2, "text": " Let's enter this."}, {"start": 1554.2, "end": 1559.2, "text": " Let's enter the encodec model 24 kilohertz function."}, {"start": 1560.0600000000002, "end": 1561.5800000000002, "text": " And basically, so you can see here,"}, {"start": 1561.5800000000002, "end": 1562.7, "text": " we just set some variables,"}, {"start": 1562.7, "end": 1566.5800000000002, "text": " and finally we need to first basically get this model."}, {"start": 1566.58, "end": 1568.32, "text": " So the model consists, as you can see here,"}, {"start": 1568.32, "end": 1570.98, "text": " from the encoder and decoder."}, {"start": 1570.98, "end": 1573.3799999999999, "text": " So I'm gonna enter into the encoder"}, {"start": 1573.3799999999999, "end": 1575.54, "text": " just to briefly show you how everything looks like."}, {"start": 1575.54, "end": 1577.74, "text": " So we are in the encoder here,"}, {"start": 1577.74, "end": 1581.74, "text": " and you can see basically just some variables,"}, {"start": 1581.74, "end": 1582.78, "text": " the location there,"}, {"start": 1582.78, "end": 1586.26, "text": " and then we have this ratios list,"}, {"start": 1586.26, "end": 1588.74, "text": " which basically tells us how we are reducing"}, {"start": 1588.74, "end": 1592.06, "text": " the temporal dimensionality"}, {"start": 1592.06, "end": 1596.1, "text": " as we are propagating through the encoder."}, {"start": 1596.1, "end": 1598.1, "text": " And if you sum all of these up,"}, {"start": 1598.1, "end": 1600.1799999999998, "text": " you'll get the number that's probably familiar"}, {"start": 1600.1799999999998, "end": 1602.3, "text": " at this point, and that's 320."}, {"start": 1602.3, "end": 1606.1799999999998, "text": " And that's why we need 320 raw samples"}, {"start": 1606.1799999999998, "end": 1608.06, "text": " to get a single output frame."}, {"start": 1608.06, "end": 1610.62, "text": " That's the particular reason why this is happening,"}, {"start": 1610.62, "end": 1612.82, "text": " because of the architecture of the encoder."}, {"start": 1612.82, "end": 1614.1, "text": " Okay?"}, {"start": 1614.1, "end": 1615.06, "text": " So let's continue."}, {"start": 1615.06, "end": 1616.0, "text": " So you can see here,"}, {"start": 1616.0, "end": 1620.3799999999999, "text": " we just have these COM1D layers with this S prefix."}, {"start": 1620.3799999999999, "end": 1622.54, "text": " We just suggest that they have,"}, {"start": 1622.54, "end": 1624.06, "text": " they kind of modified it a bit."}, {"start": 1624.06, "end": 1627.5, "text": " I'm gonna show you how it looks like in a second."}, {"start": 1627.5, "end": 1630.4199999999998, "text": " But after that, they just keep on adding"}, {"start": 1631.6599999999999, "end": 1634.26, "text": " these ResNet blocks,"}, {"start": 1635.3, "end": 1639.06, "text": " which consists basically out of bunch of these COM1D layers"}, {"start": 1639.06, "end": 1641.4199999999998, "text": " and some skip connections."}, {"start": 1641.4199999999998, "end": 1644.86, "text": " So basically nothing fundamental there."}, {"start": 1644.86, "end": 1647.6599999999999, "text": " It's just your encoder architecture, and that's it."}, {"start": 1647.6599999999999, "end": 1650.7, "text": " I'm gonna quickly just show you how this thing looks like."}, {"start": 1650.7, "end": 1654.06, "text": " So they have this S prefix, as I mentioned."}, {"start": 1654.06, "end": 1655.66, "text": " And so the thing here is,"}, {"start": 1655.66, "end": 1657.22, "text": " they'll have this,"}, {"start": 1657.22, "end": 1659.26, "text": " depending whether we are dealing with a causal model"}, {"start": 1659.26, "end": 1660.6200000000001, "text": " or non-causal model,"}, {"start": 1660.6200000000001, "end": 1662.5800000000002, "text": " they'll have a bit different padding here."}, {"start": 1662.5800000000002, "end": 1665.3400000000001, "text": " And I think that's pretty much it."}, {"start": 1665.3400000000001, "end": 1667.5, "text": " They also have some details like"}, {"start": 1667.5, "end": 1671.3400000000001, "text": " how they wrap up this norm layer with the COM layer."}, {"start": 1671.3400000000001, "end": 1674.1000000000001, "text": " So you can see here, just some idiosyncrasies,"}, {"start": 1674.1000000000001, "end": 1676.94, "text": " but nothing crucial really to understand how this is working."}, {"start": 1676.94, "end": 1680.5800000000002, "text": " So let's treat it as a black box, and we have our encoder."}, {"start": 1680.58, "end": 1682.22, "text": " The decoder is fairly similar,"}, {"start": 1682.22, "end": 1683.9399999999998, "text": " so I'm gonna skip it completely."}, {"start": 1683.9399999999998, "end": 1687.06, "text": " And finally, let me zoom in a bit"}, {"start": 1687.06, "end": 1689.1399999999999, "text": " in case you can see a bit better now."}, {"start": 1689.1399999999999, "end": 1690.8999999999999, "text": " Okay, hopefully this works."}, {"start": 1690.8999999999999, "end": 1693.82, "text": " Let me know whether the visibility is now better"}, {"start": 1693.82, "end": 1695.1799999999998, "text": " than before I zoomed in."}, {"start": 1695.1799999999998, "end": 1696.34, "text": " By the way, I'm a bit sick,"}, {"start": 1696.34, "end": 1698.74, "text": " so that's why my throat might sound a bit different."}, {"start": 1698.74, "end": 1701.74, "text": " But hopefully the video is gonna be high quality nonetheless."}, {"start": 1701.74, "end": 1703.62, "text": " So let's see what's happening here."}, {"start": 1704.98, "end": 1707.98, "text": " You can see here that this enqueue is gonna be"}, {"start": 1707.98, "end": 1711.42, "text": " the max number of those quantizers we had"}, {"start": 1711.42, "end": 1715.02, "text": " in that bottleneck of the encoder decoder"}, {"start": 1715.02, "end": 1718.94, "text": " that this model can have"}, {"start": 1718.94, "end": 1722.06, "text": " in case we have the largest target bandwidth."}, {"start": 1722.06, "end": 1725.42, "text": " And you can see here, target bandwidths go from 1.5 kilohertz"}, {"start": 1725.42, "end": 1727.1, "text": " all the way to 24 kilohertz."}, {"start": 1727.1, "end": 1729.1200000000001, "text": " And when we index by minus one,"}, {"start": 1729.1200000000001, "end": 1732.98, "text": " we fetch the largest bandwidth, which is the 24 kilohertz."}, {"start": 1732.98, "end": 1736.22, "text": " And because it's 24, we multiply with thousands"}, {"start": 1736.22, "end": 1737.78, "text": " such that we end up with hertz."}, {"start": 1737.78, "end": 1741.34, "text": " So this is now the number of hertz for the target bandwidth."}, {"start": 1741.34, "end": 1744.58, "text": " And then we divide that number with this thing."}, {"start": 1744.58, "end": 1745.42, "text": " And so what's this thing?"}, {"start": 1745.42, "end": 1749.34, "text": " So basically we have here the number of frames."}, {"start": 1749.34, "end": 1750.58, "text": " You can see here sample rate."}, {"start": 1750.58, "end": 1752.5, "text": " When you divide that by the hop length,"}, {"start": 1752.5, "end": 1754.18, "text": " you end up with the output number of frames."}, {"start": 1754.18, "end": 1756.98, "text": " It's gonna be like 75 for our particular case here."}, {"start": 1756.98, "end": 1759.3799999999999, "text": " And then we multiply that with 10"}, {"start": 1759.3799999999999, "end": 1762.1399999999999, "text": " because they kind of hard coded that"}, {"start": 1762.1399999999999, "end": 1763.96, "text": " because they assume that the code book"}, {"start": 1763.96, "end": 1766.02, "text": " is gonna be 1,024 slots."}, {"start": 1766.02, "end": 1769.18, "text": " And they basically does assume that we need 10 bits"}, {"start": 1769.18, "end": 1770.82, "text": " to index into that code book."}, {"start": 1770.82, "end": 1773.16, "text": " So again, that means that a single quantizer"}, {"start": 1773.16, "end": 1776.22, "text": " will have 75 times 10 bits."}, {"start": 1776.22, "end": 1781.22, "text": " So that means 750 bits for a second of the video."}, {"start": 1781.46, "end": 1784.06, "text": " And so when we divide these two,"}, {"start": 1784.06, "end": 1786.3799999999999, "text": " we get the number of quantizers we need"}, {"start": 1786.3799999999999, "end": 1788.58, "text": " for the target bandwidth of 24 kilohertz."}, {"start": 1788.58, "end": 1790.62, "text": " As you can see here, it's gonna be 32."}, {"start": 1790.62, "end": 1793.5, "text": " So that means we need 32 of these quantizers"}, {"start": 1793.5, "end": 1797.98, "text": " to be able to achieve this particular target"}, {"start": 1797.98, "end": 1800.14, "text": " bandwidth of 24 kilohertz."}, {"start": 1800.14, "end": 1803.48, "text": " So again, just to clarify as this might be a bit confusing,"}, {"start": 1803.48, "end": 1808.14, "text": " imagine we have a signal, a second of the audio raw signal"}, {"start": 1808.14, "end": 1810.26, "text": " that's gonna be 24,000 samples, right?"}, {"start": 1810.26, "end": 1814.46, "text": " And then we encode that and we end up with 75 frames."}, {"start": 1814.46, "end": 1817.9, "text": " And because for each of the frames, we allocate 10 bits,"}, {"start": 1817.9, "end": 1821.22, "text": " that means we have 75 times 10 bits, right?"}, {"start": 1821.22, "end": 1822.56, "text": " And then if you wanna figure out"}, {"start": 1822.56, "end": 1826.26, "text": " how many of these do we need to get to 24K,"}, {"start": 1826.26, "end": 1829.58, "text": " we just divide 24K by that number of bits."}, {"start": 1829.58, "end": 1832.3, "text": " And that's precisely what you've seen here."}, {"start": 1832.3, "end": 1835.34, "text": " Okay, second thing I wanna show you"}, {"start": 1835.34, "end": 1837.04, "text": " is this residual vector quantizer,"}, {"start": 1837.04, "end": 1839.5, "text": " although all of the fun is gonna happen in the forward pass,"}, {"start": 1839.5, "end": 1843.22, "text": " not here, but let's see how it's constructed."}, {"start": 1843.22, "end": 1846.3, "text": " Again, just bunch of variables here."}, {"start": 1846.3, "end": 1849.1, "text": " And then there is this residual vector quantization."}, {"start": 1849.1, "end": 1850.74, "text": " Let's enter that thing."}, {"start": 1850.74, "end": 1853.94, "text": " And you can see here, we have 32 of these"}, {"start": 1853.94, "end": 1858.06, "text": " vector quantizations basically wrapped into a list."}, {"start": 1858.06, "end": 1861.32, "text": " And let me see whether I have a break point here."}, {"start": 1861.32, "end": 1863.58, "text": " So let's enter into one of these"}, {"start": 1863.58, "end": 1865.7, "text": " vector quantization modules"}, {"start": 1865.7, "end": 1867.54, "text": " and let's see what's going on there."}, {"start": 1867.54, "end": 1870.22, "text": " So here is the codebook dimensionality,"}, {"start": 1870.22, "end": 1871.26, "text": " it's gonna be 128."}, {"start": 1871.26, "end": 1873.04, "text": " So that's the number we've been using"}, {"start": 1873.04, "end": 1875.82, "text": " as a running example throughout the video."}, {"start": 1875.82, "end": 1878.34, "text": " And then because we don't require any projection"}, {"start": 1878.34, "end": 1881.3, "text": " because the dimensionality of the output of the encoder"}, {"start": 1881.3, "end": 1883.58, "text": " and the codebook are the same"}, {"start": 1883.58, "end": 1885.3799999999999, "text": " because of that we don't need any projection"}, {"start": 1885.3799999999999, "end": 1887.8999999999999, "text": " and that's why we're gonna end up with identities here."}, {"start": 1887.8999999999999, "end": 1891.22, "text": " Okay, and finally we have this Euclidean codebook"}, {"start": 1891.22, "end": 1893.34, "text": " and the whole fun is gonna be in the forward pass,"}, {"start": 1893.34, "end": 1895.54, "text": " I'm gonna skip the actual initialization,"}, {"start": 1895.54, "end": 1897.86, "text": " but that's the implementation of the codebook"}, {"start": 1897.86, "end": 1899.9399999999998, "text": " I've been mentioning this whole time."}, {"start": 1899.9399999999998, "end": 1902.3799999999999, "text": " And it's called Euclidean because once we do the,"}, {"start": 1903.3, "end": 1905.6599999999999, "text": " once we try to figure out the closest vector"}, {"start": 1905.66, "end": 1908.78, "text": " in that code table we use the L2 norm"}, {"start": 1908.78, "end": 1911.42, "text": " to figure that out, hence Euclidean."}, {"start": 1911.42, "end": 1914.78, "text": " Okay, I'm gonna hit F5 here and let's exit this."}, {"start": 1914.78, "end": 1918.14, "text": " And now I'm gonna disable all of the break points"}, {"start": 1918.14, "end": 1919.64, "text": " and let's exit this one."}, {"start": 1920.74, "end": 1922.26, "text": " And we are back here."}, {"start": 1922.26, "end": 1924.5800000000002, "text": " I'm gonna re-enable all of the break points"}, {"start": 1924.5800000000002, "end": 1925.78, "text": " and let's continue here."}, {"start": 1925.78, "end": 1928.22, "text": " Okay, so we have the encoder, we have the decoder,"}, {"start": 1928.22, "end": 1930.74, "text": " we have the quantizer, and finally we just have"}, {"start": 1930.74, "end": 1933.8200000000002, "text": " this encodic model which is just basically gonna"}, {"start": 1933.82, "end": 1936.7, "text": " wrap up all of those variables, nothing fancy,"}, {"start": 1936.7, "end": 1940.34, "text": " I can move this break point, I can remove it"}, {"start": 1940.34, "end": 1942.02, "text": " and let's continue here."}, {"start": 1942.02, "end": 1945.1, "text": " So let's just get to here and that's it."}, {"start": 1945.1, "end": 1949.98, "text": " So we are out of here and we have our model finally ready."}, {"start": 1949.98, "end": 1953.82, "text": " So now we just fetch the actual state dictionary,"}, {"start": 1953.82, "end": 1956.26, "text": " so the pre-trained weights, we initialize the model,"}, {"start": 1956.26, "end": 1959.74, "text": " we set it into the eval mode and we continue here."}, {"start": 1959.74, "end": 1963.1, "text": " Okay guys, so now we set the target bandwidth, okay?"}, {"start": 1963.1, "end": 1965.74, "text": " So the target bandwidth by default is gonna be"}, {"start": 1965.74, "end": 1968.8799999999999, "text": " six kilobits per second and what it does is basically"}, {"start": 1968.8799999999999, "end": 1971.3799999999999, "text": " just sets this bandwidth internal variable"}, {"start": 1971.3799999999999, "end": 1974.78, "text": " of the encodic model to six and that's it."}, {"start": 1974.78, "end": 1977.4599999999998, "text": " So nothing fancy happening there."}, {"start": 1977.4599999999998, "end": 1981.26, "text": " Okay, so now let's just load the audio file."}, {"start": 1981.26, "end": 1985.8, "text": " It's gonna be a 24 kilohertz, 20 second audio file."}, {"start": 1985.8, "end": 1989.4399999999998, "text": " So if you take a look at the root of this repository,"}, {"start": 1990.5, "end": 1992.1999999999998, "text": " we can find that signal here."}, {"start": 1992.2, "end": 1993.8, "text": " Let me see where we can hear it."}, {"start": 1993.8, "end": 1998.8, "text": " Okay, you can see it's a 20 second audio file, that's it."}, {"start": 1999.6200000000001, "end": 2000.98, "text": " Okay, so let's continue here."}, {"start": 2000.98, "end": 2004.54, "text": " Let me just show you the shape of that WAV file."}, {"start": 2004.54, "end": 2009.54, "text": " So shape is 480,000 and that's because if you just think"}, {"start": 2012.46, "end": 2016.0, "text": " about it, if you multiply 20, which is the number of seconds"}, {"start": 2016.0, "end": 2019.8400000000001, "text": " of the audio file times the 24,000 samples per second,"}, {"start": 2019.84, "end": 2024.84, "text": " we end up with 480,000 samples."}, {"start": 2025.58, "end": 2027.58, "text": " My brain is glitching, man."}, {"start": 2027.58, "end": 2030.06, "text": " Okay, so let's continue."}, {"start": 2030.06, "end": 2032.52, "text": " We now do the conversion but because,"}, {"start": 2034.1399999999999, "end": 2035.54, "text": " it's gonna be a no-op basically."}, {"start": 2035.54, "end": 2037.6799999999998, "text": " I'm gonna enter here just to show you but basically"}, {"start": 2037.6799999999998, "end": 2039.82, "text": " because the target channel is set to one,"}, {"start": 2039.82, "end": 2042.86, "text": " we do a mean across the zero dimension"}, {"start": 2042.86, "end": 2045.4199999999998, "text": " but because we only have a single channel,"}, {"start": 2045.4199999999998, "end": 2046.8799999999999, "text": " it's gonna be a no-op."}, {"start": 2046.88, "end": 2050.38, "text": " If we had a stereo sound, then we'd have two channels"}, {"start": 2050.38, "end": 2052.82, "text": " and we'd just do a mean across the two channels"}, {"start": 2052.82, "end": 2054.84, "text": " to get the final output channel"}, {"start": 2054.84, "end": 2058.26, "text": " because the target number of channels is set to one here."}, {"start": 2058.26, "end": 2060.78, "text": " But in our case, it's just no-ops, we can ignore it"}, {"start": 2060.78, "end": 2063.1, "text": " and then we do resampling but again,"}, {"start": 2063.1, "end": 2067.7400000000002, "text": " because both our signal and the target signal sampling rates"}, {"start": 2067.7400000000002, "end": 2070.38, "text": " are the same 24K, we do nothing."}, {"start": 2070.38, "end": 2072.7400000000002, "text": " So we just basically, it's a no-op."}, {"start": 2072.7400000000002, "end": 2074.78, "text": " Okay, so here is the fun part."}, {"start": 2074.78, "end": 2078.82, "text": " We do the compression of the signal,"}, {"start": 2078.82, "end": 2081.46, "text": " of the loaded signal and we pass it through our model."}, {"start": 2081.46, "end": 2083.1800000000003, "text": " Let's see how that's gonna work."}, {"start": 2083.1800000000003, "end": 2087.1600000000003, "text": " Okay, so let's kinda skip over all of these details."}, {"start": 2087.1600000000003, "end": 2091.94, "text": " What we do here is we are basically going to load"}, {"start": 2091.94, "end": 2093.94, "text": " the whole signal."}, {"start": 2095.1400000000003, "end": 2098.6200000000003, "text": " So this whole loop is gonna have just a single iteration."}, {"start": 2098.6200000000003, "end": 2102.6200000000003, "text": " And I think worth mentioning is even though they support"}, {"start": 2102.62, "end": 2106.54, "text": " the streamable setting, they don't have it implemented here"}, {"start": 2106.54, "end": 2109.06, "text": " neither do they have implemented the training code."}, {"start": 2109.06, "end": 2111.06, "text": " So what you can find here is the inference."}, {"start": 2111.06, "end": 2113.5, "text": " You can also find some details about the balancer"}, {"start": 2113.5, "end": 2118.5, "text": " and about the multi-scale STFT discriminators."}, {"start": 2119.02, "end": 2121.1, "text": " Okay, so basically you can see here,"}, {"start": 2121.1, "end": 2123.58, "text": " they also use the terminology frame"}, {"start": 2123.58, "end": 2126.94, "text": " for the input raw audio signal"}, {"start": 2126.94, "end": 2129.18, "text": " and not only for the output of the encoder."}, {"start": 2129.18, "end": 2131.74, "text": " So just keep that in mind and don't get confused"}, {"start": 2131.74, "end": 2133.7, "text": " by the terminology here."}, {"start": 2133.7, "end": 2137.4599999999996, "text": " So frame is again, is gonna be simply the same"}, {"start": 2137.4599999999996, "end": 2138.3799999999997, "text": " as our input signal."}, {"start": 2138.3799999999997, "end": 2140.66, "text": " So because we fetched every single sample,"}, {"start": 2140.66, "end": 2143.7, "text": " it's just gonna be 480,000 samples."}, {"start": 2143.7, "end": 2146.52, "text": " Okay, and now the fun happens in the code frame function."}, {"start": 2146.52, "end": 2149.04, "text": " So let's see how that's gonna work."}, {"start": 2149.04, "end": 2151.3399999999997, "text": " So we don't do it in normalizations, we can skip that."}, {"start": 2151.3399999999997, "end": 2154.8399999999997, "text": " And now we pass the X, so the input signal"}, {"start": 2154.8399999999997, "end": 2155.7999999999997, "text": " through the encoder."}, {"start": 2155.7999999999997, "end": 2157.1, "text": " So let's see how that's gonna work."}, {"start": 2157.1, "end": 2159.66, "text": " So again, this is just gonna be a series"}, {"start": 2159.66, "end": 2162.2599999999998, "text": " of these COM1D layers and some of them"}, {"start": 2162.2599999999998, "end": 2163.74, "text": " are gonna have a stride of two."}, {"start": 2163.74, "end": 2167.8599999999997, "text": " And so our input signal is slowly gonna be reduced"}, {"start": 2167.8599999999997, "end": 2170.66, "text": " across the temporal dimension and the number of channels"}, {"start": 2170.66, "end": 2172.02, "text": " is gonna be increasing."}, {"start": 2172.02, "end": 2174.1, "text": " So what I'm gonna do here is I'm gonna enter"}, {"start": 2174.1, "end": 2178.1, "text": " the COM1D layer here and I'm gonna put the break point"}, {"start": 2178.1, "end": 2179.22, "text": " with a particular condition."}, {"start": 2179.22, "end": 2182.02, "text": " And this is why I love Visual Studio Code."}, {"start": 2182.02, "end": 2184.7, "text": " So you can basically edit the break point"}, {"start": 2184.7, "end": 2186.22, "text": " and you can set a particular condition"}, {"start": 2186.22, "end": 2187.42, "text": " when you wanna hit this one."}, {"start": 2187.42, "end": 2190.94, "text": " And I wanna hit it when stride is bigger than one, right?"}, {"start": 2190.94, "end": 2193.2000000000003, "text": " And so because I wanna see how the number"}, {"start": 2193.2000000000003, "end": 2196.86, "text": " of the temporal axis is being reduced."}, {"start": 2196.86, "end": 2199.06, "text": " Okay, so now let's hit F5."}, {"start": 2199.06, "end": 2202.42, "text": " And here we are, so currently if I just print a shape,"}, {"start": 2202.42, "end": 2203.76, "text": " the shape is, as you can see here,"}, {"start": 2203.76, "end": 2207.98, "text": " we already have 32 channels because we had some previous"}, {"start": 2207.98, "end": 2210.82, "text": " COM1D layers which didn't have the stride"}, {"start": 2210.82, "end": 2213.7200000000003, "text": " but they still had increased the number of channels."}, {"start": 2213.7200000000003, "end": 2216.62, "text": " But once I hit, so now I'm gonna put a regular break point"}, {"start": 2216.62, "end": 2217.74, "text": " and hit F5."}, {"start": 2217.74, "end": 2221.66, "text": " So once we are here, now if I do x.shape,"}, {"start": 2221.66, "end": 2223.3399999999997, "text": " we have, as you can see here,"}, {"start": 2223.3399999999997, "end": 2227.2, "text": " two x, a smaller dimensionality across the temporal axis"}, {"start": 2227.2, "end": 2229.5, "text": " and we have two x more channels."}, {"start": 2229.5, "end": 2232.7799999999997, "text": " And that's a regular pattern we've been seeing all around."}, {"start": 2232.7799999999997, "end": 2234.3199999999997, "text": " CNNs do that all the time."}, {"start": 2234.3199999999997, "end": 2237.1, "text": " As soon as you reduce the spatial dimensionality by two,"}, {"start": 2237.1, "end": 2238.8199999999997, "text": " you also increase the number of channels by two."}, {"start": 2238.8199999999997, "end": 2241.46, "text": " That's a common pattern you'll see all around."}, {"start": 2241.46, "end": 2243.98, "text": " Okay guys, so now I'm gonna remove basically"}, {"start": 2243.98, "end": 2246.18, "text": " all of the break points and I'm gonna exit"}, {"start": 2246.18, "end": 2247.02, "text": " from this function."}, {"start": 2247.02, "end": 2250.4199999999996, "text": " I'm gonna exit here, I'm gonna exit here,"}, {"start": 2250.4199999999996, "end": 2254.58, "text": " I'm gonna exit through all of these and we are done."}, {"start": 2254.58, "end": 2256.4199999999996, "text": " Okay, so this is the encoder."}, {"start": 2256.4199999999996, "end": 2260.2599999999998, "text": " I'm gonna hit F10 and here is the final dimensionality."}, {"start": 2260.2599999999998, "end": 2264.4199999999996, "text": " It's gonna be 1,500, 128."}, {"start": 2264.4199999999996, "end": 2266.8599999999997, "text": " And that's the output of our encoder, right?"}, {"start": 2266.8599999999997, "end": 2271.8599999999997, "text": " And it's 1,500 frames because again, we have 20 seconds"}, {"start": 2272.3199999999997, "end": 2274.14, "text": " and each second is 75 frames"}, {"start": 2274.14, "end": 2276.2599999999998, "text": " and that's why we end up with 1,500."}, {"start": 2276.2599999999998, "end": 2277.62, "text": " Hopefully that makes sense."}, {"start": 2277.62, "end": 2281.58, "text": " Okay, so now what we do is we have to quantize"}, {"start": 2281.58, "end": 2284.7, "text": " those vectors and let's see how that's gonna work."}, {"start": 2284.7, "end": 2286.42, "text": " So here we are."}, {"start": 2286.42, "end": 2289.02, "text": " So we first have to figure out"}, {"start": 2289.02, "end": 2291.3799999999997, "text": " for the desired target bandwidth,"}, {"start": 2291.3799999999997, "end": 2294.62, "text": " how many of the quantizers do we have to actually use"}, {"start": 2294.62, "end": 2296.74, "text": " to get that target bandwidth?"}, {"start": 2296.74, "end": 2298.22, "text": " And so that's what this function does."}, {"start": 2298.22, "end": 2300.1, "text": " So let's enter into the function"}, {"start": 2300.1, "end": 2301.7, "text": " and let's see how it works."}, {"start": 2301.7, "end": 2305.74, "text": " So, okay, so they say get bandwidth per quantizer."}, {"start": 2305.74, "end": 2307.7, "text": " So does that make sense?"}, {"start": 2307.7, "end": 2308.74, "text": " Let's think about it."}, {"start": 2308.74, "end": 2311.4199999999996, "text": " So we have 75 here."}, {"start": 2311.4199999999996, "end": 2313.9399999999996, "text": " So that means we have 75 frames."}, {"start": 2313.9399999999996, "end": 2317.3399999999997, "text": " It's gonna be 10 because bins is gonna be 1,024."}, {"start": 2317.3399999999997, "end": 2319.74, "text": " So this is 10 times the number of frames."}, {"start": 2319.74, "end": 2322.22, "text": " So this is basically the number of bits"}, {"start": 2322.22, "end": 2325.46, "text": " for a single quantizer module."}, {"start": 2325.46, "end": 2327.02, "text": " And then we divide by 1,000"}, {"start": 2327.02, "end": 2329.14, "text": " because we wanna have kilobits, right?"}, {"start": 2329.14, "end": 2332.5, "text": " And so you can see here bandwidth per quantizer."}, {"start": 2332.5, "end": 2337.3799999999997, "text": " And that's gonna be 0.75 kilobits per quantizer, okay?"}, {"start": 2337.3799999999997, "end": 2340.2, "text": " And now what we do is we take the target bandwidth,"}, {"start": 2340.2, "end": 2343.46, "text": " which is six kilobits per second,"}, {"start": 2343.46, "end": 2345.18, "text": " and we divide by 0.75,"}, {"start": 2345.18, "end": 2348.2, "text": " and we end up that we need eight of these quantizers"}, {"start": 2348.2, "end": 2350.6, "text": " to achieve that particular target bandwidth, okay?"}, {"start": 2350.6, "end": 2351.56, "text": " And that's it."}, {"start": 2351.56, "end": 2353.66, "text": " So now we actually do the encoding"}, {"start": 2353.66, "end": 2356.8599999999997, "text": " with these eight quantizers."}, {"start": 2356.8599999999997, "end": 2358.1, "text": " So let's see how that's gonna work."}, {"start": 2358.1, "end": 2361.42, "text": " So you can see here, we just take the first eight quantizers"}, {"start": 2361.42, "end": 2365.1, "text": " instead of the full list, which is like 32."}, {"start": 2365.1, "end": 2368.2999999999997, "text": " So remember, we have actually, we have 32 of these,"}, {"start": 2368.2999999999997, "end": 2369.62, "text": " and we can only deploy those"}, {"start": 2369.62, "end": 2372.94, "text": " when we have target bandwidth of 24 kilohertz."}, {"start": 2372.94, "end": 2374.2599999999998, "text": " Okay, so let's enter here."}, {"start": 2374.2599999999998, "end": 2375.92, "text": " Let's see how the encode works."}, {"start": 2375.92, "end": 2376.7799999999997, "text": " So it's fairly simple."}, {"start": 2376.7799999999997, "end": 2379.5, "text": " So first of all, this pre-process"}, {"start": 2379.5, "end": 2381.8399999999997, "text": " is just gonna do some reshaping, nothing fancy there."}, {"start": 2381.8399999999997, "end": 2386.8399999999997, "text": " So we end up with X being of shape 1,528."}, {"start": 2386.84, "end": 2388.56, "text": " And then this is where the fun happens"}, {"start": 2388.56, "end": 2389.88, "text": " in the quantize function."}, {"start": 2389.88, "end": 2391.2200000000003, "text": " So in the quantize function,"}, {"start": 2391.2200000000003, "end": 2393.46, "text": " let me enter the quantize function."}, {"start": 2393.46, "end": 2395.32, "text": " You can see this is the codebook,"}, {"start": 2396.4, "end": 2398.6000000000004, "text": " the codebook table I was mentioning the whole time."}, {"start": 2398.6000000000004, "end": 2401.2000000000003, "text": " So let's see the dimensionality of the codebook table."}, {"start": 2401.2000000000003, "end": 2404.6000000000004, "text": " This is gonna be 1,024 because we have 1,024 vectors,"}, {"start": 2404.6000000000004, "end": 2408.76, "text": " and each of those is 128 dimensionality, right?"}, {"start": 2408.76, "end": 2412.36, "text": " And so what we do here is we basically do,"}, {"start": 2412.36, "end": 2414.6000000000004, "text": " we compute this distance matrix,"}, {"start": 2414.6, "end": 2417.88, "text": " which tells us for each of our output vectors,"}, {"start": 2417.88, "end": 2421.16, "text": " how close are each of these codebook vectors"}, {"start": 2421.16, "end": 2422.2999999999997, "text": " from that particular vector."}, {"start": 2422.2999999999997, "end": 2424.24, "text": " And because of that, we're gonna end up,"}, {"start": 2424.24, "end": 2425.52, "text": " let me just show you this."}, {"start": 2425.52, "end": 2428.88, "text": " Let me show you the dimensionality of the distance matrix."}, {"start": 2428.88, "end": 2433.0, "text": " It's gonna be 1,500, 1,024, because as I said,"}, {"start": 2433.0, "end": 2435.04, "text": " you have for each of the rows,"}, {"start": 2435.04, "end": 2438.3199999999997, "text": " you have the columns telling you how far away"}, {"start": 2438.3199999999997, "end": 2440.08, "text": " is that particular codebook vector"}, {"start": 2440.08, "end": 2443.8399999999997, "text": " from that particular output encoder vector, right?"}, {"start": 2443.84, "end": 2447.92, "text": " And because of this minus sign, if we do a max,"}, {"start": 2447.92, "end": 2451.0, "text": " we actually find the closest vectors, and that's it."}, {"start": 2451.0, "end": 2453.76, "text": " So we end up with, we basically end up with,"}, {"start": 2453.76, "end": 2456.36, "text": " as you can see here, 1,500 indices,"}, {"start": 2456.36, "end": 2458.2000000000003, "text": " and let me print the actual numbers."}, {"start": 2458.2000000000003, "end": 2461.48, "text": " So those are the indices of the closest vectors"}, {"start": 2461.48, "end": 2464.2000000000003, "text": " in those code tables."}, {"start": 2464.2000000000003, "end": 2465.32, "text": " Hopefully that makes sense."}, {"start": 2465.32, "end": 2468.92, "text": " And again, post-process just does some reshaping,"}, {"start": 2468.92, "end": 2470.76, "text": " and we are done, guys."}, {"start": 2470.76, "end": 2471.6400000000003, "text": " That's it."}, {"start": 2471.6400000000003, "end": 2472.84, "text": " So now we do the decoding."}, {"start": 2472.84, "end": 2474.2000000000003, "text": " That's the second interesting part."}, {"start": 2474.2000000000003, "end": 2476.2400000000002, "text": " So let's enter it into the decoding part."}, {"start": 2476.2400000000002, "end": 2478.1200000000003, "text": " So let's see what happens there."}, {"start": 2478.1200000000003, "end": 2482.96, "text": " So decoding is gonna call this Dequantize method,"}, {"start": 2482.96, "end": 2486.6800000000003, "text": " and Dequantize is basically just gonna fetch the indices,"}, {"start": 2486.6800000000003, "end": 2489.88, "text": " and you can see here, we just grab the vectors"}, {"start": 2489.88, "end": 2493.6800000000003, "text": " from that table that correspond to those indices,"}, {"start": 2493.6800000000003, "end": 2497.8, "text": " and we end up with the quantized vectors."}, {"start": 2497.8, "end": 2500.96, "text": " Whoops, let me just show you the dimensionality here."}, {"start": 2500.96, "end": 2503.32, "text": " We have 1,500, 128."}, {"start": 2503.32, "end": 2506.2400000000002, "text": " So that's when we snap our output vectors"}, {"start": 2506.2400000000002, "end": 2508.2400000000002, "text": " onto the closest vectors in the code table."}, {"start": 2508.2400000000002, "end": 2509.96, "text": " That's what happened there."}, {"start": 2509.96, "end": 2511.96, "text": " Okay, and then projections identity,"}, {"start": 2511.96, "end": 2514.84, "text": " so we can ignore this, and that's pretty much it."}, {"start": 2514.84, "end": 2517.4, "text": " And now that we have the quantized vectors,"}, {"start": 2517.4, "end": 2519.8, "text": " we do the residual minus quantized,"}, {"start": 2519.8, "end": 2523.88, "text": " and then we save the indices from the first quantizer,"}, {"start": 2523.88, "end": 2525.68, "text": " and we repeat the process, right?"}, {"start": 2525.68, "end": 2527.4, "text": " We now take the second code book,"}, {"start": 2527.4, "end": 2530.2400000000002, "text": " and we take the residuals, and we just encode them,"}, {"start": 2530.24, "end": 2531.64, "text": " and rinse and repeat."}, {"start": 2531.64, "end": 2534.4799999999996, "text": " So I'm gonna now disable all of the breakpoints,"}, {"start": 2534.4799999999996, "end": 2537.68, "text": " and I'm gonna just put the breakpoint here, hit F5."}, {"start": 2537.68, "end": 2540.16, "text": " I'm gonna re-enable the breakpoints, and that's it."}, {"start": 2540.16, "end": 2543.16, "text": " So the out indices now are 1,508."}, {"start": 2545.3199999999997, "end": 2547.0, "text": " So that means for each of the frames,"}, {"start": 2547.0, "end": 2548.9199999999996, "text": " we now have eight numbers."}, {"start": 2548.9199999999996, "end": 2552.04, "text": " That's how we encoded our output of the encoder."}, {"start": 2552.04, "end": 2556.72, "text": " Okay, let's continue here, and that's pretty much it."}, {"start": 2556.72, "end": 2560.9599999999996, "text": " We can now keep on exiting here,"}, {"start": 2560.9599999999996, "end": 2562.8399999999997, "text": " and we have everything we need."}, {"start": 2562.8399999999997, "end": 2566.9199999999996, "text": " We have the data, the compressed version of our audio file."}, {"start": 2566.9199999999996, "end": 2569.3599999999997, "text": " We now write some metadata into the header"}, {"start": 2569.3599999999997, "end": 2570.7599999999998, "text": " of this ecdc file."}, {"start": 2570.7599999999998, "end": 2572.08, "text": " I'm gonna ignore that completely,"}, {"start": 2572.08, "end": 2574.2, "text": " not that important for us."}, {"start": 2574.2, "end": 2576.9199999999996, "text": " And then because we're not using LM,"}, {"start": 2576.9199999999996, "end": 2579.52, "text": " we are gonna construct this bitpacker."}, {"start": 2579.52, "end": 2580.8799999999997, "text": " And so why the bitpacker?"}, {"start": 2580.8799999999997, "end": 2584.04, "text": " So again, the problem is these numbers"}, {"start": 2584.04, "end": 2587.4, "text": " are gonna be either ints or floats."}, {"start": 2587.4, "end": 2588.8, "text": " Let me just see for a second."}, {"start": 2588.8, "end": 2591.88, "text": " Let me tell you the exact thing here."}, {"start": 2591.88, "end": 2596.2, "text": " So frames, again, frames shape is,"}, {"start": 2596.2, "end": 2597.24, "text": " okay, frames is a list."}, {"start": 2597.24, "end": 2598.52, "text": " Let me just see."}, {"start": 2598.52, "end": 2600.7599999999998, "text": " Okay, it's a list, and then I think I just have"}, {"start": 2600.7599999999998, "end": 2603.12, "text": " to do something like this."}, {"start": 2603.12, "end": 2605.2, "text": " Yeah, okay, so here are the frames."}, {"start": 2605.2, "end": 2608.08, "text": " And if we do the D type, let's see the data type."}, {"start": 2608.08, "end": 2609.88, "text": " So it's gonna be int 64."}, {"start": 2609.88, "end": 2611.92, "text": " So that means each of these eight numbers"}, {"start": 2611.92, "end": 2615.7200000000003, "text": " currently take 64 bits, even though we only need 10 bits."}, {"start": 2615.7200000000003, "end": 2617.6, "text": " And because of that, we only,"}, {"start": 2617.6, "end": 2620.12, "text": " because we only, out of those 64 bits,"}, {"start": 2620.12, "end": 2623.48, "text": " only lower 10 bits have the valuable information."}, {"start": 2623.48, "end": 2625.32, "text": " That's where this bitpacker comes into play"}, {"start": 2625.32, "end": 2627.52, "text": " to handle the low level details."}, {"start": 2627.52, "end": 2629.84, "text": " I did debug it and understand how it works,"}, {"start": 2629.84, "end": 2632.88, "text": " but it's gonna be, I think, redundant"}, {"start": 2632.88, "end": 2634.08, "text": " if I do that in this video,"}, {"start": 2634.08, "end": 2637.2400000000002, "text": " because you can do it at your own pace,"}, {"start": 2637.2400000000002, "end": 2640.8, "text": " but nothing fundamental there other than low level details"}, {"start": 2640.8, "end": 2642.6800000000003, "text": " of how do you extract the 10 bits"}, {"start": 2642.6800000000003, "end": 2643.6000000000004, "text": " that are actually valuable"}, {"start": 2643.6000000000004, "end": 2647.04, "text": " and pack them into the output bit by stream."}, {"start": 2647.04, "end": 2648.88, "text": " Okay, and that's it."}, {"start": 2648.88, "end": 2651.6800000000003, "text": " So here, what we do, we just keep on pushing those values,"}, {"start": 2651.6800000000003, "end": 2653.88, "text": " and values are coming from the frames."}, {"start": 2653.88, "end": 2655.76, "text": " And so basically, that's it."}, {"start": 2655.76, "end": 2658.52, "text": " We're just taking those numbers from this structure here,"}, {"start": 2658.52, "end": 2662.92, "text": " and we're just basically encoding them into this by stream."}, {"start": 2662.92, "end": 2665.5600000000004, "text": " And so let me hit F5 here."}, {"start": 2665.5600000000004, "end": 2667.04, "text": " Let me remove this thing."}, {"start": 2667.04, "end": 2668.92, "text": " And we are done."}, {"start": 2668.92, "end": 2669.76, "text": " We are done."}, {"start": 2669.76, "end": 2671.0, "text": " We can exit here."}, {"start": 2671.0, "end": 2671.92, "text": " We can exit here."}, {"start": 2671.92, "end": 2674.7200000000003, "text": " We just write the bytes, and that's it, guys."}, {"start": 2674.7200000000003, "end": 2675.5400000000004, "text": " So that's it."}, {"start": 2675.5400000000004, "end": 2678.0, "text": " So let me just find the file."}, {"start": 2678.0, "end": 2680.1200000000003, "text": " So the file is here."}, {"start": 2680.1200000000003, "end": 2684.28, "text": " As you can see here, test24k.ecdc is again created,"}, {"start": 2684.28, "end": 2689.28, "text": " and that contains our compressed audio file."}, {"start": 2689.32, "end": 2693.96, "text": " You can see that we started from 937 kilobytes,"}, {"start": 2693.96, "end": 2697.6800000000003, "text": " and we ended up with 14.7 kilobytes."}, {"start": 2697.6800000000003, "end": 2698.6600000000003, "text": " Does that make sense?"}, {"start": 2698.6600000000003, "end": 2699.5, "text": " Let's see."}, {"start": 2699.5, "end": 2704.5, "text": " So we have 14.7 times, I guess, eight,"}, {"start": 2706.2, "end": 2709.68, "text": " and this is how many kilobits we have."}, {"start": 2709.68, "end": 2714.68, "text": " And that's roughly, well, that's roughly 20 times six, right?"}, {"start": 2715.46, "end": 2719.12, "text": " Which is 120, because we have six kilobits per second"}, {"start": 2719.12, "end": 2721.92, "text": " target bandwidth, and we have 20 seconds,"}, {"start": 2721.92, "end": 2722.84, "text": " so that's roughly it."}, {"start": 2722.84, "end": 2725.66, "text": " So everything basically fits."}, {"start": 2725.66, "end": 2728.96, "text": " Okay, so now I'm gonna go to the launch file."}, {"start": 2728.96, "end": 2733.12, "text": " I'm gonna quickly add the LM, and I'm kinda afraid"}, {"start": 2733.12, "end": 2735.6, "text": " that I will not be able to explain this,"}, {"start": 2735.6, "end": 2738.68, "text": " to give it justice, but I'm gonna try nonetheless,"}, {"start": 2739.84, "end": 2742.16, "text": " because I think it would require quite a lot of time"}, {"start": 2742.16, "end": 2744.64, "text": " to actually explain all of the details"}, {"start": 2744.64, "end": 2746.56, "text": " behind how and why it works."}, {"start": 2747.66, "end": 2752.16, "text": " Okay, so now I'm gonna start this again,"}, {"start": 2752.16, "end": 2754.96, "text": " and let's just see the diff between the last ones."}, {"start": 2754.96, "end": 2758.5, "text": " I'm not gonna focus on the things that are the same,"}, {"start": 2758.5, "end": 2761.28, "text": " obviously, I just wanna focus on the differences."}, {"start": 2761.28, "end": 2766.28, "text": " So the differences here are gonna be only,"}, {"start": 2766.88, "end": 2770.04, "text": " as you can see here, we passed the LM argument"}, {"start": 2770.04, "end": 2774.24, "text": " to the compressed function, so I'm gonna disable"}, {"start": 2774.24, "end": 2776.16, "text": " the breakpoints, I'm gonna enable this one."}, {"start": 2776.16, "end": 2778.0, "text": " I'm gonna hit F5."}, {"start": 2778.0, "end": 2782.16, "text": " Whoops, again, I forgot to delete the file, god damn it."}, {"start": 2782.16, "end": 2787.08, "text": " So I'm gonna delete this one, and I'm gonna restart."}, {"start": 2787.08, "end": 2790.64, "text": " So let's hit, you can add the force argument"}, {"start": 2790.64, "end": 2793.34, "text": " to avoid having these errors, but I was lazy,"}, {"start": 2793.34, "end": 2794.88, "text": " so I kinda skipped it."}, {"start": 2794.88, "end": 2796.92, "text": " So I'm gonna enable the breakpoints,"}, {"start": 2796.92, "end": 2799.56, "text": " so let's now enter this, let's enter here,"}, {"start": 2799.56, "end": 2803.68, "text": " and let's see, so use LM is where the differences are."}, {"start": 2803.68, "end": 2804.88, "text": " So there's a couple of places"}, {"start": 2804.88, "end": 2805.84, "text": " where there are the differences."}, {"start": 2805.84, "end": 2809.88, "text": " So I'm gonna get here first, so let's see"}, {"start": 2809.88, "end": 2811.64, "text": " what the get LM model does."}, {"start": 2811.64, "end": 2815.48, "text": " So basically we instantiate a language model here."}, {"start": 2815.48, "end": 2818.76, "text": " So let's see how that thing is implemented."}, {"start": 2818.76, "end": 2821.9, "text": " I'm gonna hit F5 here, here we are."}, {"start": 2821.9, "end": 2825.2, "text": " We have something called Streaming Transformer Encoder,"}, {"start": 2825.2, "end": 2828.28, "text": " and we have, as you can see here, 32 of these"}, {"start": 2828.28, "end": 2831.7, "text": " embedding tables, and we also have 32 of these"}, {"start": 2831.7, "end": 2834.96, "text": " linear layers that project from the 200,"}, {"start": 2834.96, "end": 2838.28, "text": " which is the dimensionality of the transformer,"}, {"start": 2838.28, "end": 2842.2, "text": " to 1024, because that's the cardinality,"}, {"start": 2842.2, "end": 2844.26, "text": " that's what card stands for, the cardinality"}, {"start": 2844.26, "end": 2846.28, "text": " or the size of the codebook table."}, {"start": 2846.28, "end": 2848.4, "text": " That's why we have 1024 there, right?"}, {"start": 2849.6400000000003, "end": 2852.92, "text": " So I'm gonna, okay, I have a breakpoint here already."}, {"start": 2852.92, "end": 2855.0, "text": " Let me see whether there is something interesting"}, {"start": 2855.0, "end": 2857.32, "text": " to mention here, let's quickly walk through"}, {"start": 2857.32, "end": 2859.5200000000004, "text": " the transformer here."}, {"start": 2861.8, "end": 2865.0, "text": " Okay, basically there are these"}, {"start": 2865.0, "end": 2869.1000000000004, "text": " Streaming Transformer Encoder Layer modules,"}, {"start": 2869.1000000000004, "end": 2871.84, "text": " and let me just see whether those are interesting."}, {"start": 2871.84, "end": 2873.7000000000003, "text": " So those are basically just, as you can see here,"}, {"start": 2873.7, "end": 2876.4399999999996, "text": " just stock PyTorch Transformer Encoder Layer"}, {"start": 2876.4399999999996, "end": 2880.2799999999997, "text": " with some differences, like, yeah."}, {"start": 2881.7, "end": 2885.1, "text": " This might be too complex to explain in this shorter video,"}, {"start": 2886.3999999999996, "end": 2888.3199999999997, "text": " but in any case, I'm gonna skip this,"}, {"start": 2888.3199999999997, "end": 2889.96, "text": " I'm gonna skip the instantiation,"}, {"start": 2889.96, "end": 2891.72, "text": " so we just have a transformer,"}, {"start": 2891.72, "end": 2893.2, "text": " we create those embedding tables,"}, {"start": 2893.2, "end": 2895.24, "text": " we create the linear layers, and we continue here."}, {"start": 2895.24, "end": 2897.06, "text": " So that's the LM, okay?"}, {"start": 2897.06, "end": 2900.46, "text": " So, now what happens is we'll load"}, {"start": 2900.46, "end": 2903.2599999999998, "text": " the actual pre-trained weights."}, {"start": 2903.26, "end": 2905.6400000000003, "text": " We put it into the eval mode,"}, {"start": 2905.6400000000003, "end": 2908.48, "text": " and we return the language model, okay."}, {"start": 2908.48, "end": 2912.28, "text": " So now we do the encoding the same way as before."}, {"start": 2912.28, "end": 2913.6200000000003, "text": " So nothing changes here."}, {"start": 2913.6200000000003, "end": 2918.6200000000003, "text": " We still end up with 1500 times eight numbers."}, {"start": 2919.76, "end": 2921.1600000000003, "text": " So let me show you that."}, {"start": 2921.1600000000003, "end": 2925.7200000000003, "text": " So we have, oops, again, I have to first do the zero,"}, {"start": 2925.7200000000003, "end": 2930.0, "text": " zero indexing, and then we end up with 1508,"}, {"start": 2930.0, "end": 2932.2200000000003, "text": " so that's the same shape as before."}, {"start": 2932.22, "end": 2935.6, "text": " And so now, let's see where the difference is."}, {"start": 2935.6, "end": 2936.9199999999996, "text": " So the first difference is here."}, {"start": 2936.9199999999996, "end": 2939.9199999999996, "text": " We formed this thing called Arithmetic Coder,"}, {"start": 2939.9199999999996, "end": 2943.0, "text": " and it's fairly complex to explain,"}, {"start": 2943.0, "end": 2945.18, "text": " as you can see just by the description"}, {"start": 2945.18, "end": 2946.56, "text": " of how this thing works,"}, {"start": 2946.56, "end": 2949.72, "text": " and some of the low-level details here"}, {"start": 2949.72, "end": 2951.4199999999996, "text": " of bit shifting, et cetera, et cetera,"}, {"start": 2951.4199999999996, "end": 2955.3999999999996, "text": " but basically in a nutshell, what it does is using the,"}, {"start": 2956.7999999999997, "end": 2959.48, "text": " well, I'm gonna get there in a second,"}, {"start": 2959.48, "end": 2961.56, "text": " so it's gonna be wasteful for me"}, {"start": 2961.56, "end": 2963.12, "text": " to try and explain it like this."}, {"start": 2963.12, "end": 2964.92, "text": " So in any case, we have this Arithmetic Coder."}, {"start": 2964.92, "end": 2968.38, "text": " We're gonna see how it works a bit later, okay."}, {"start": 2968.38, "end": 2970.92, "text": " So you can see here, we first initialize the input"}, {"start": 2970.92, "end": 2973.88, "text": " as all zeros, and k is equal to eight,"}, {"start": 2973.88, "end": 2975.68, "text": " which is the number of our quantizers."}, {"start": 2975.68, "end": 2979.68, "text": " So this is kinda, before we get the actual eight numbers"}, {"start": 2979.68, "end": 2983.0, "text": " that are the encoded version"}, {"start": 2983.0, "end": 2986.6, "text": " of the first output vector of the encoder,"}, {"start": 2987.58, "end": 2990.08, "text": " before that, we just assume we have all zeros, right?"}, {"start": 2990.08, "end": 2992.16, "text": " So again, let me just show you what I mean."}, {"start": 2992.16, "end": 2996.4, "text": " Let me find the diagram here."}, {"start": 2998.92, "end": 2999.92, "text": " So here we are."}, {"start": 2999.92, "end": 3003.36, "text": " So again, before we have the actual encoding"}, {"start": 3003.36, "end": 3004.6, "text": " for this first vector,"}, {"start": 3004.6, "end": 3006.36, "text": " so we'll have like four numbers here"}, {"start": 3006.36, "end": 3008.22, "text": " or eight numbers in our particular case,"}, {"start": 3008.22, "end": 3010.46, "text": " we just assume we have all zeros."}, {"start": 3010.46, "end": 3012.12, "text": " We have all zeros,"}, {"start": 3012.12, "end": 3014.52, "text": " and that's what the program assumes here."}, {"start": 3014.52, "end": 3017.14, "text": " Okay, that's what this thing is, okay."}, {"start": 3017.14, "end": 3019.84, "text": " So let's now see how this is gonna work."}, {"start": 3019.84, "end": 3022.84, "text": " So we pass the input, the states,"}, {"start": 3022.84, "end": 3025.96, "text": " and the offset into the language model,"}, {"start": 3025.96, "end": 3028.2000000000003, "text": " and then we just basically do a forward pass"}, {"start": 3028.2000000000003, "end": 3033.2000000000003, "text": " through the model to get the output probabilities."}, {"start": 3033.5, "end": 3034.88, "text": " Okay, let's enter here,"}, {"start": 3034.88, "end": 3037.48, "text": " just to, let me just show you one thing."}, {"start": 3038.32, "end": 3040.84, "text": " Whoops, let me just say one thing,"}, {"start": 3040.84, "end": 3042.6400000000003, "text": " and that's the logits part."}, {"start": 3042.6400000000003, "end": 3045.7200000000003, "text": " So after we just do a forward pass through the transformer,"}, {"start": 3045.7200000000003, "end": 3047.8, "text": " which I'm gonna skip, so I'm gonna skip this part."}, {"start": 3047.8, "end": 3049.2400000000002, "text": " So I'm gonna put breakpoint here."}, {"start": 3049.24, "end": 3052.3199999999997, "text": " So I'm gonna disable everything, enable this one."}, {"start": 3052.3199999999997, "end": 3054.8799999999997, "text": " We're gonna do a forward pass through the transformer,"}, {"start": 3054.8799999999997, "end": 3058.04, "text": " and we end up with the output, okay."}, {"start": 3058.04, "end": 3060.56, "text": " So output is gonna be of the shape, let's see,"}, {"start": 3060.56, "end": 3063.04, "text": " so 200, because that's the inherent,"}, {"start": 3063.04, "end": 3066.0, "text": " that's the inner dimensionality of the transformer."}, {"start": 3066.0, "end": 3068.7999999999997, "text": " So we passed in all zeros,"}, {"start": 3068.7999999999997, "end": 3071.04, "text": " and we basically,"}, {"start": 3071.04, "end": 3074.2799999999997, "text": " how we convert those all zeros"}, {"start": 3074.2799999999997, "end": 3077.7599999999998, "text": " into a continuous representation is on this line here."}, {"start": 3077.76, "end": 3080.36, "text": " As you can see here, we just do eight times."}, {"start": 3080.36, "end": 3082.6400000000003, "text": " We grab the number,"}, {"start": 3082.6400000000003, "end": 3086.28, "text": " and we basically grab the corresponding embedding table,"}, {"start": 3086.28, "end": 3088.32, "text": " and then that's how we get the representation,"}, {"start": 3088.32, "end": 3090.1200000000003, "text": " and then we just sum them up."}, {"start": 3090.1200000000003, "end": 3092.36, "text": " And I think that was also explained in the paper here."}, {"start": 3092.36, "end": 3095.7200000000003, "text": " So let me just show you where that thing is explained."}, {"start": 3096.6800000000003, "end": 3099.76, "text": " I think somewhere here, so okay."}, {"start": 3099.76, "end": 3101.5600000000004, "text": " Okay, for a time step T,"}, {"start": 3101.5600000000004, "end": 3103.6800000000003, "text": " the discrete representation obtained at that time,"}, {"start": 3103.6800000000003, "end": 3106.1600000000003, "text": " T minus one is transformed into continuous representation"}, {"start": 3106.16, "end": 3108.08, "text": " using learned embedding tables,"}, {"start": 3108.08, "end": 3110.7999999999997, "text": " one for each codebook, and which are summed."}, {"start": 3110.7999999999997, "end": 3112.08, "text": " So that's basically this line."}, {"start": 3112.08, "end": 3114.6, "text": " This line describes this piece of code here."}, {"start": 3114.6, "end": 3116.64, "text": " So again, let's go back to the diagram."}, {"start": 3116.64, "end": 3118.16, "text": " It's gonna be easier."}, {"start": 3118.16, "end": 3122.64, "text": " So we have these, let's say four numbers here,"}, {"start": 3122.64, "end": 3125.04, "text": " and each of these four numbers is gonna index"}, {"start": 3125.04, "end": 3127.48, "text": " into a particular table."}, {"start": 3127.48, "end": 3132.3599999999997, "text": " So we'll have four tables associated with this transformer,"}, {"start": 3132.3599999999997, "end": 3134.3599999999997, "text": " and we get their corresponding vectors,"}, {"start": 3134.3599999999997, "end": 3135.64, "text": " then we sum them up,"}, {"start": 3135.64, "end": 3139.0, "text": " and then we do a forward pass through the transformer,"}, {"start": 3139.0, "end": 3142.2, "text": " and we end up with a 200-dimensional vector."}, {"start": 3142.2, "end": 3143.8399999999997, "text": " That's what we do there, okay?"}, {"start": 3144.68, "end": 3145.96, "text": " That's the idea."}, {"start": 3145.96, "end": 3148.7999999999997, "text": " And now we just map those using those linear layers"}, {"start": 3148.7999999999997, "end": 3151.3199999999997, "text": " I mentioned before that are gonna have the output"}, {"start": 3151.3199999999997, "end": 3153.8399999999997, "text": " dimensionality being equal to the cardinality"}, {"start": 3153.8399999999997, "end": 3155.64, "text": " of the codebook, which is 1,024."}, {"start": 3157.0, "end": 3159.7999999999997, "text": " So if I do just a forward step over this,"}, {"start": 3159.7999999999997, "end": 3161.8399999999997, "text": " we're gonna end up with logits."}, {"start": 3161.8399999999997, "end": 3163.3599999999997, "text": " Here are the shape."}, {"start": 3163.3599999999997, "end": 3165.6, "text": " The shape is 1,024, eight."}, {"start": 3165.6, "end": 3168.2799999999997, "text": " And that makes sense because for each of the eight numbers"}, {"start": 3168.2799999999997, "end": 3171.68, "text": " that we are supposed to output for this particular time step,"}, {"start": 3171.68, "end": 3176.44, "text": " we have a distribution, which is 1,024 bins,"}, {"start": 3176.44, "end": 3181.24, "text": " across our codebook table, corresponding codebook table."}, {"start": 3181.24, "end": 3182.52, "text": " I know this might be confusing,"}, {"start": 3182.52, "end": 3187.24, "text": " so you'll have to go through the code base at your own pace,"}, {"start": 3187.24, "end": 3189.24, "text": " but hopefully this helps you somewhat."}, {"start": 3190.3199999999997, "end": 3192.0, "text": " Okay, so probabilities are there."}, {"start": 3192.0, "end": 3193.92, "text": " So what we do next is this."}, {"start": 3193.92, "end": 3197.36, "text": " So we have this, we grab the next eight numbers"}, {"start": 3197.36, "end": 3198.64, "text": " and we add one."}, {"start": 3198.64, "end": 3203.12, "text": " And one is added because the zeros, as you see here,"}, {"start": 3203.12, "end": 3204.6800000000003, "text": " have a special connotation."}, {"start": 3204.6800000000003, "end": 3206.64, "text": " They're just like initial values,"}, {"start": 3206.64, "end": 3211.04, "text": " which cannot actually be, we can't get zero as the output."}, {"start": 3211.04, "end": 3212.88, "text": " And that's why there is this plus one here."}, {"start": 3212.88, "end": 3214.6800000000003, "text": " Anyways, a small detail there."}, {"start": 3215.52, "end": 3217.0, "text": " So here is the complex part."}, {"start": 3217.0, "end": 3219.84, "text": " We build this quantized CDF,"}, {"start": 3219.84, "end": 3222.0, "text": " which is a cumulative distribution function."}, {"start": 3222.0, "end": 3225.52, "text": " And basically how it works is that the higher the probability"}, {"start": 3225.52, "end": 3228.36, "text": " at the particular bin, the bigger the range"}, {"start": 3228.36, "end": 3230.6, "text": " that particular bin is gonna occupy."}, {"start": 3230.6, "end": 3235.08, "text": " So again, I told you this is gonna be fairly complex."}, {"start": 3235.08, "end": 3238.44, "text": " So it says here, turn the given PDF into a quantized CDF"}, {"start": 3238.44, "end": 3242.32, "text": " that splits this particular range into chunks of size"}, {"start": 3242.32, "end": 3245.4, "text": " roughly proportional to the PDF, right?"}, {"start": 3245.4, "end": 3246.96, "text": " And they use 24 bits here."}, {"start": 3246.96, "end": 3248.64, "text": " So that means this is gonna be"}, {"start": 3248.64, "end": 3251.08, "text": " two raised to the power of 24 minus one."}, {"start": 3251.08, "end": 3254.7599999999998, "text": " Again, let me try and quickly draw this,"}, {"start": 3254.7599999999998, "end": 3257.0, "text": " and I think we are almost done with the video."}, {"start": 3258.12, "end": 3260.4, "text": " Okay, so here is how it's gonna look like."}, {"start": 3260.4, "end": 3263.24, "text": " So we get as the output, we get some distribution."}, {"start": 3263.24, "end": 3264.7999999999997, "text": " So let's say it looks like this."}, {"start": 3265.64, "end": 3269.92, "text": " Okay, and so what this is gonna be mapped into"}, {"start": 3269.92, "end": 3272.0, "text": " is this range."}, {"start": 3272.0, "end": 3274.0, "text": " So this range is gonna be from zero"}, {"start": 3274.0, "end": 3278.68, "text": " to two raised to the power of 24 minus one, okay?"}, {"start": 3278.68, "end": 3281.24, "text": " And then we're trying to split this range."}, {"start": 3282.24, "end": 3286.6, "text": " And so here, initially, because we have a peak here,"}, {"start": 3286.6, "end": 3290.24, "text": " that means that this range is gonna be huge, okay?"}, {"start": 3291.2799999999997, "end": 3293.52, "text": " Let me just change the color here."}, {"start": 3293.52, "end": 3294.8799999999997, "text": " This range is gonna be huge."}, {"start": 3294.8799999999997, "end": 3296.6, "text": " And then where we have lower probability,"}, {"start": 3296.6, "end": 3299.24, "text": " we'll have much smaller, whoops,"}, {"start": 3299.24, "end": 3300.72, "text": " let me change the color again."}, {"start": 3300.72, "end": 3302.8399999999997, "text": " So we'll have much smaller ranges here."}, {"start": 3302.8399999999997, "end": 3304.8399999999997, "text": " So these ranges here are gonna be small,"}, {"start": 3305.8799999999997, "end": 3307.9199999999996, "text": " but these ones are gonna be bigger,"}, {"start": 3307.92, "end": 3310.08, "text": " and they're gonna slowly diminish here."}, {"start": 3310.08, "end": 3312.04, "text": " So that's the mapping that's happening"}, {"start": 3312.04, "end": 3313.7200000000003, "text": " in that particular function I showed you."}, {"start": 3313.7200000000003, "end": 3314.92, "text": " And these are all of the details"}, {"start": 3314.92, "end": 3316.96, "text": " behind the arithmetic encoding."}, {"start": 3316.96, "end": 3320.56, "text": " And so you'll have to dig into that a bit deeper yourself"}, {"start": 3320.56, "end": 3322.88, "text": " in case you're curious, but that's it."}, {"start": 3322.88, "end": 3324.36, "text": " I'm gonna go back here."}, {"start": 3324.36, "end": 3327.84, "text": " So we end up with the CDF, and let me just show you."}, {"start": 3327.84, "end": 3332.0, "text": " So CDF again, the shape here is 1,024 again."}, {"start": 3332.0, "end": 3335.44, "text": " But you can see here that we just have the numbers"}, {"start": 3335.44, "end": 3339.08, "text": " from zero to two raised to the power of 24 minus one."}, {"start": 3339.08, "end": 3343.56, "text": " And you can see here that the differences here,"}, {"start": 3344.68, "end": 3347.32, "text": " if they are super big, that means that the corresponding bin"}, {"start": 3347.32, "end": 3350.08, "text": " in the PDF had a higher probability."}, {"start": 3350.08, "end": 3351.56, "text": " And that's the idea."}, {"start": 3351.56, "end": 3354.04, "text": " And then we use this arithmetic coder,"}, {"start": 3354.04, "end": 3356.04, "text": " and we push the value."}, {"start": 3356.04, "end": 3357.84, "text": " Whoops, here it is."}, {"start": 3357.84, "end": 3359.56, "text": " But we also pass the CDF,"}, {"start": 3359.56, "end": 3363.36, "text": " because the arithmetic coder requires that information"}, {"start": 3363.36, "end": 3366.52, "text": " of the ranges in order to efficiently encode this."}, {"start": 3366.52, "end": 3371.52, "text": " And this is gonna occupy less than 10 bits in expectation."}, {"start": 3371.52, "end": 3373.7200000000003, "text": " So before, because we had eight numbers,"}, {"start": 3373.7200000000003, "end": 3376.04, "text": " the best we could do was eight times 10 bits,"}, {"start": 3376.04, "end": 3376.88, "text": " that's 80 bits."}, {"start": 3376.88, "end": 3379.2000000000003, "text": " Here we can go below 80 bits."}, {"start": 3379.2000000000003, "end": 3381.84, "text": " And that's the magic behind the arithmetic coder."}, {"start": 3381.84, "end": 3385.52, "text": " So I'm gonna basically enter this"}, {"start": 3385.52, "end": 3387.46, "text": " just to show you the complexity,"}, {"start": 3387.46, "end": 3390.48, "text": " but I'm not gonna try and explain everything here."}, {"start": 3390.48, "end": 3392.08, "text": " Let me hit F5 here."}, {"start": 3392.08, "end": 3396.64, "text": " And so you can see here how the numbers are,"}, {"start": 3396.64, "end": 3399.4, "text": " how the range information here is used"}, {"start": 3399.4, "end": 3404.4, "text": " to basically finally here get the actual encoding"}, {"start": 3406.96, "end": 3410.16, "text": " for this particular number and this CDF."}, {"start": 3410.16, "end": 3415.16, "text": " So you can see here, we are packing bit by bit."}, {"start": 3415.24, "end": 3419.3199999999997, "text": " This packer is the same packer as the one we had before,"}, {"start": 3419.3199999999997, "end": 3421.12, "text": " but instead of using 10 bits,"}, {"start": 3421.12, "end": 3422.8399999999997, "text": " instead of pushing 10 bits into the stream,"}, {"start": 3422.8399999999997, "end": 3424.96, "text": " it pushes just a single bit into the stream."}, {"start": 3424.96, "end": 3427.1, "text": " And, but in expectation,"}, {"start": 3427.1, "end": 3431.24, "text": " this while loop is gonna have less than 10 iterations,"}, {"start": 3431.24, "end": 3434.08, "text": " i.e. 10 bits per symbol."}, {"start": 3434.08, "end": 3436.92, "text": " And that's why we're gonna be able to"}, {"start": 3436.92, "end": 3439.6, "text": " basically compress this even more"}, {"start": 3439.6, "end": 3444.6, "text": " than using the, than not using the arithmetic coding."}, {"start": 3444.9, "end": 3446.8399999999997, "text": " So I'm gonna exit here."}, {"start": 3446.8399999999997, "end": 3448.2799999999997, "text": " I'm gonna exit here."}, {"start": 3448.28, "end": 3452.84, "text": " And basically, I'm gonna remove all of the breakpoints,"}, {"start": 3452.84, "end": 3457.84, "text": " put a breakpoint here, hit F5, and that's it."}, {"start": 3458.2000000000003, "end": 3460.6400000000003, "text": " So you can see it was a bit slower."}, {"start": 3460.6400000000003, "end": 3462.6000000000004, "text": " And that's the trade off you have to take"}, {"start": 3462.6000000000004, "end": 3466.7200000000003, "text": " if you wanna have even higher compression ratio."}, {"start": 3466.7200000000003, "end": 3467.78, "text": " And that's it, guys."}, {"start": 3467.78, "end": 3469.26, "text": " Everything else remains the same."}, {"start": 3469.26, "end": 3472.0, "text": " We just write the output bytes, and that's it."}, {"start": 3472.0, "end": 3476.26, "text": " So hopefully this combination of using the paper"}, {"start": 3476.26, "end": 3478.6800000000003, "text": " and the video was helpful."}, {"start": 3478.6800000000003, "end": 3481.28, "text": " Do let me know whether you have some questions."}, {"start": 3481.28, "end": 3482.96, "text": " I know I had to kinda rush through"}, {"start": 3482.96, "end": 3485.36, "text": " some of the details behind it,"}, {"start": 3485.36, "end": 3487.5200000000004, "text": " but hopefully this gives you some intuition,"}, {"start": 3487.5200000000004, "end": 3489.4, "text": " some understanding."}, {"start": 3489.4, "end": 3491.32, "text": " Do leave some comments down below."}, {"start": 3491.32, "end": 3492.6400000000003, "text": " Any feedback you might have for me,"}, {"start": 3492.6400000000003, "end": 3493.96, "text": " it will be super useful."}, {"start": 3493.96, "end": 3496.7200000000003, "text": " And guys, until next time, bye bye."}, {"start": 3496.72, "end": 3505.74, "text": " Lindsey Ch resolution"}]