OpenAI Whisper: Robust Speech Recognition via Large-Scale Weak Supervision | Paper and Code
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I cover Whisper, an ASR system from OpenAI's "Robust Speech Recognition via Large-Scale Weak Supervision" paper.

Trained on a huge multi-lingual, multi-task weakly supervised dataset it achieves a very high effective robustness and accuracy closing the gap with the human baseline using only an off-the-shelf transformer.

I walk you through both the paper as well as the actual code. Let me know whether the code part helped!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://cdn.openai.com/papers/whisper.pdf
✅ Code: https://github.com/openai/whisper

✅ Nice explanation of mel spectrograms: https://www.youtube.com/watch?v=9GHCiiDLHQ4
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00:00 Intro
00:02:05 Paper overview
00:07:30 Collecting a large scale weakly supervised dataset
00:13:55 Evaluation metric issues (WER)
00:16:05 Effective robustness
00:18:40 Scaling laws in progress
00:26:30 Decoding is hacky
00:28:30 Code walk-through
00:30:25 Model architecture (diagram vs code)
00:33:30 Transcription task
00:34:10 Loading the audio, mel spectrograms
00:37:50 Language detection
00:45:00 Transcription task continued
00:47:35 Suppressing token logits
00:52:00 Voice activity detection
00:53:35 Decoding and heuristics
01:01:56 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#whisper #openai #asr
 OpenAI just open sourced Whisper, which is an automatic speech recognition system that's approaching the human level baseline on both the accuracy as well as on the robustness wise for the English transcription. And that's super awesome. So in this video, I'm gonna cover the paper as well as I'm gonna go through the code behind the paper, which they luckily open source. So the inference code as well as the model checkpoints have all been open sourced and that's kinda cool. So yeah, without further ado, let's dig into the video. So here is the tweet that kinda announced the release of Whisper. They say we've trained a neural net called Whisper that approaches human level robustness and accuracy on English speech recognition. It performs well even on diverse accents and technical language. Whisper is open source for all to use. You can see that Whisper can do many tasks and it's a multilingual multitask system. So here's a couple of examples. So English transcription. So given an audio in English, ask not what your country can do for blah, blah, blah. You ask and then you see the transcription here, ask not what your country can do for you. So that's kinda trivial. And then we have any two English speech translation where we have maybe audio file in let's say Spanish, El rapido thorough marron salta sobre. And then you have the translation, the quick brown fox jumps over. And then we have non-English transcription where the audio file is again in some foreign language that's not English. And the transcription is also in that same language. So I'm not gonna try and pronounce Chinese. My Mandarin, it's not at its peak. So there is the last task here is the no speech one where given the background music playing or like whatever, it's important that we don't have human voices. The model outputs a special token signaling that there is well, no human speech going on. Other than these tasks, the model can also do language detection, which means it can given the audio detect which language does that particular audio file belong to. Okay, let's jump into the paper. Let's see the abstract and then I'm gonna first give you some high level overview and then we can dig a bit deeper. So the name of the paper is robust speech recognition by a large scale weak supervision. We're gonna see that this is one of the main components of the paper so just like the data set itself. And let's kind of skim through the abstract first. So we studied the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalized well to standard benchmarks and are often competitive with prior fully supervised results but in a zero shot transfer setting without the need for any fine tuning. When compared to humans, the models approach their accuracy and robustness. Okay, so let's first start by just showing you the high level diagram, how the architecture looks like and these special tokens that you use to prompt the model to do specific tasks. So the architecture itself is literally like off the shelf transformer encoder decoder transformer module from 2017 attention is all you need paper. So they wanted to kind of remove that confounding factor of just tweaking the architecture for this particular task of speech recognition. And instead they just took the off the shelf model and they've been playing with the data. Like data is the main part here as well as the special tokens. We'll see those in a second. So again, how the system works on the high level is you grab an audio file and then you convert it from its raw form into something called log male spectrograms. So spectrograms are these representations where here on the X axis, you have the time information. So the time steps and then on the Y axis you have the actual frequency. And then the color basically tells you what's the amplitude of that particular frequency at that particular point of time. So like this point here, like this will be the particular time. And then you can see the frequency here. And the frequency is usually in Hertz but in the case of male spectrograms, it's in males. And male is just like a logarithmic scale because that's how we perceive pitch and not like linearly like Hertz scale. So that's a TLDR, but I'll link some resources for explain very nicely how male spectrograms work because that's not the topic of this video. Okay, so once we have that representation of the audio then we literally do a feed forward pass through the encoder stacks. You can see here just blocks of simple transformer layers that consist of self-attention and MLP, multilayer perceptron and outcomes the final representation here. So once you have the final audio representation you use that to condition the decoder, the causal decoder model here via cross attention mechanism. And then the whole magic is in these special tokens. So you can see here, we have this start of transcription token, we have then the English token and then we have the transcribe token and then the model generates like this is first the timestamp and then outcomes the actual transcription of the audio file. You can see that the, here is the target. So it's a supervised dataset. We do have the label for the audio obviously and then we just do, we shift by one and do the classic language modeling objective here. There are some details that we can kind of dig into a bit later like the coder uses learn positionally codings. Here we have sinusoidal ones, but as I said architecture is not the main contribution of this paper. It's the dataset and it's this multi-task setup with the tokens. So ignoring this part for the time being on the left, let's focus on the right part of the pipeline. So we start with the start of transcript token and then the model will autoregressively decode either this no speech token, which basically means that we are not detecting, as I said previously, I think there is no speech going on the audio. The model is only detecting maybe music or something without human vocals or the model could generate language tag and these are by the way, the only things that the model can generate because they have various like masking mechanisms to make sure this is the case. Okay, so let's say the language tag was decoded. In that case, we basically then have two options either transcribe or translate. We've seen the differences. So main differences in the translate option whatever the source language of the audio file is we'll end up with the English translation. Okay, and then we can do the no timestamps route or we can do the one with the timestamps where you additionally predict timestamp information and then you do some transcription inside of that segment and then rinse and repeat. And finally you have the end of transcription token. Okay, so all of these give rise to multiple tasks and we'll see how those play out a bit later in the code as well. Okay, so having explained all of this, let's dig deeper into the paper. Well, let's start in the beginning here. So there are roughly two research directions that people have been pursuing before a whisper was published. So one big research direction was the unsupervised direction. So let me read this one for you. So these pre-trained audio encoders learn high quality representations of speech, but because they are purely unsupervised they lack an equivalently performant decoder mapping those representations to usable outputs necessitating a fine tuning stage in order to actually perform a task such as speech recognition. Okay, so that means on one part of the spectrum we have models that are trained on a huge amount of data. So we have, let me denote the data set like this. So we have huge amount of data. I think they mentioned here, like you can see here scaled up to million hours of training data. So that's like, and that's the worst visualization of data ever. Let me kind of delete it for a second and draw it properly. So as I said, we have a data set of 1 million hours of audio here. This is a bit better. And that's on one part of the spectrum. So let me now show you the other part of the spectrum here. And again, what I've said here is that they train these in a non-supervised fashion, which means you go from audio to audio representation. I guess from male spectrogram to male spectrogram something like that. And then the problem is how do you decode those representations to actual human language? That's the problem with these models. That's why you have to do the fine tuning part. Okay. And then the second direction is this one. So speech recognition systems that are pre-trained in a supervised fashion across many data sets domains exhibit higher robustness and generalize much more effectively to held out data sets than models trained on a single source. Okay, so there's just stating that we wanna have a multilingual multitask setup, which we kind of know already, especially if you have bigger models. However, there is still only a moderate amount of this data easily available. These authors mix together seven pre-existing data sets totaling only 5,000 hours of supervision. Okay, so that's the other part of the spectrum. We have tiny, tiny, tiny models here. I mean, tiny data sets. So this one has like 5K hours compared that to 1 million hours there. Okay. And they say this, so whoops, by relaxing the requirements of gold standard human validated transcripts, these authors make use of sophisticated automated pipelines to scale weekly supervised speech recognition to 10,000 and 30,000 hours of moisture training data. This trade off between quality and quantity is often the right call. Okay, so that means those authors scaled up this by using the weak supervision, but they only went to maybe a bit like a couple of times bigger data sets. So now we have like something like 10 to 30K. And finally, let's get to this paper. So what these authors have done is, whoops, what's going on here? Yet these new data sets are only a few times larger than the sum of existing high quality data sets and still much smaller than prior unsupervised work. In this work, we closed that gap, scaling weekly supervised speech recognition, the next order of magnitude to 680,000 hours of labeled audio data, which means they literally close this gap. They have almost as much data as the unsupervised approach. Obviously there are the trade-offs of the quality and the quantity. And so here are some of the heuristics that I've used to make sure that the quality is as high as possible, considering that they are using these automated pipelines. Okay, so initial inspection showed a large amount of sub-pair transcripts in the raw data set. To address this, we developed several automated filtering methods to improve transcript quality. So let's see a couple of them. In order to avoid learning transcriptase, we developed many heuristics to detect and remove machine-generated transcripts from the training data set. And one of those is, if you detect all uppercase or all lowercase, then it's very unlikely that that's a human-generated transcript, and then you're just gonna remove that pair from your data set. A second heuristic they've used is, ensure that the spoken language matches the language of the transcript according to some pre-trained CLD2 model, which does the language detection, basically. And if the two do not match, we don't include the audio transcript pair as a speech recognition training example in the data set. Et cetera, et cetera. And then finally, once they had this big data set, they trained the initial versions of Whisper, and then they used the predictions of Whisper to figure out what's wrong with the data. So it's kind of very iterative process. So they say that here, for an additional filtering pass, after training an initial model, we aggregated information about its error rate on training data sources and performed manual inspection. So I can assume that that was a fairly laborious process, and that's probably one of the reasons that they did not release the actual data set that was used to train Whisper. Here's some details around the MEL Spectrogram. I'm gonna skip that. Maybe we can get back to that later. I don't wanna dig into too many details that are not that vital here. Here are some details worth mentioning, though. During early development and evaluation, we observed that Whisper models had a tendency to transcribe plausible, but almost always incorrect guesses for the names of the speakers. And the reason being is usually in those transcripts, you have something like this predicted. So open bracket and then a name of a person, and then close bracket. And then the model learned to, because of the way how models learn, statistical learning, to just predict these, even though you cannot infer what's the name of the person from a 30-second audio clip that they are using, in most cases, unless the name was actually referenced. So they say here, to avoid this, we fine-tuned Whisper models briefly on the subset of transcripts that do not include speaker annotations, which removes this behavior. Okay, so one more thing, and then we're gonna get to the robustness part. They had problems with this WER metric, so that's the word error rate metric. And it's a similar problem as to what people are encounter with when they deal with machine translation, for example, with the blue metric, et cetera. So let me read this part. So a WER, which is based on string edit distance, penalizes all differences between the model's output and the reference transcript, including innocuous differences in transcript style, which kind of sucks. We wanna focus on the semantics and understand whether the, well, in the case of audio transcription, every single word matters, that's true, but the actual maybe punctuations or style that's superficial and not changing anything, you kind of want to ignore that, right? So they say here, while this poses a problem for all transcribers, it is particularly acute for zero-shot models like Whisper, which do not observe any examples of specific dataset transcript formats. So that means those other baselines, because they're actually trained on that particular dataset, and then basically they do the evaluate on the validation portion, but the distribution is very similar, they kind of can learn and overfit the particular style, and thus they have an advantage compared to zero-shot Whisper model. So because of that, they developed various laborious normalizers, so they say here, our text normalizer was developed through iterative manual inspection to identify common patterns where naive WER penalized Whisper models for innocuous difference. Because I will not cover this in the code bit later, so I'm gonna just quickly show you how those look like. Let me just try and find those normalizers, so here they are, and if I open up English normalizers, you can see how laborious this must have been, like there is just a lot of rules that they had to do to make sure, yeah, you can see it's kind of messy. So let me go back to the paper, and let's continue here. So they introduced this concept of effective robustness, and what it means is it measures the difference in expected performance between a reference dataset, which is usually in distribution, and one or more out of distribution datasets. So obviously, we wanna have this difference to be as small as possible, which means that you are generalizing to out of distribution datasets, which is a desirable behavior, right? We want to have general models. So a model with high effective robustness does better than expected on out of distribution datasets as a function of its performance on the reference dataset, and approaches the ideal of equal performance on all datasets. So here is the actual diagram. So here we can see in an ideal world, we'd have this line here. Basically, the metric on the LibreSpeech dev clean should be whatever the performance of your model there is, you wanna have the same error rate on the other datasets. So here is the average WER on these other datasets. So you can see here, some of these models that have much better performance compared to Whisper on the LibreSpeech. So you can see this point here, for example, whoops, this point here, I'm terrible at drawing today. So it has much lower error rate here compared to let's say this point, but then you can see how poor the performance actually is. The error rate is huge for these data points of these LibreSpeech models. So you can see all in all, when you fit like a line across these data points, basically you see that it's much more robust, Whisper is much more robust compared to the other supervised LibreSpeech models. Even though it's not actually soda on the LibreSpeech, as you can see here, but that's, I guess, less important. We wanna create robust general models. Okay, so they mentioned that even the smallest zero-shot Whisper model, which only has 39 million parameters and a 6.7 WER on LibreSpeech test clean is roughly competitive with the best supervised LibreSpeech model when evaluated on other data sets. So I guess that's like, literally this model here probably is competitive, as you can see here, performance wise, the Y point here. So the average performance on the other data sets is close to these supervised LibreSpeech models. They have much better performance on the actual LibreSpeech dev clean test. Data set, sorry. Okay, so here are a couple more interesting observations. Probably not that surprising at this point of time. And that's that with more data, you can see on the x-axis hours of transcribed audio, you can see that these languages that have more data available in the data set have much lower error rate. And you can see it's super predictive. Like the correlation coefficient here is 0.84. And they mentioned somewhere here, a couple of these outliers, you can see here, ZH, which I think stands for Mandarin Chinese or something, Korean are kind of outliers here, given that even though they have this much data, the error rate is somewhat higher. And they kind of answer that, hypothesize why that might be the case here. So checking the regression coefficient for a linear fit to these log log values results in an estimate that the error rate has for every 16x increase in training data. So that's the general trend we've seen. And then they say that the many of the largest outliers in terms of worst than expected performance, according to this trend, are languages that have unique scripts and are more distantly related to the Indo-European languages, making up the majority of the training data set, such as Hebrew, Telugu, Chinese, and Korean. So I did point out to these two languages here. So they speculate that these differences could be due, I mean, it's not a speculation, it's a valid hypothesis, I guess, could be due to a lack of transfer, due to linguistic distance, or simply the problem with tokenizer being a poor match for these languages or variations in data quality. So all of those factors could be confounding. Okay, so they have additional interesting diagram here showing that the translation performance is not as predictive, given the hours of translated audio, it's not that easy to infer what will be the blue metric. And then they hypothesize here, somewhere below here. So they say here, so we suspect this is partly caused by the noisier training data due to errors in audio language identification. So as an example, Welsh CY is an outlier with much worse than expected performance at only nine blue, despite supposedly having 9,000 hours of translation data. But as it turns out, and let me just show you on the diagram here, where the CY lies, so you can see here, it literally has 9,000, but it has around nine blue, which is like super low compared to what this trend here would suggest. But then they say here, inspection shows the majority of supposedly Welsh translation data is actually English audio with English captions, where the English audio was misclassified as Welsh by the language identification system, resulting in it being included as translation training data rather than transcription data according to our data set creation rules. So that's a problem because that means you were using the Welsh token, but actually you had English audio and English transcription, which messes up the results. So I guess that's suggestive of not taking this plot here too seriously and just like, I guess the general point of the paper is, given enough data that's high quality, we can achieve human performance on speech recognition pretty much. There is some situation phenomenon, but I'm gonna show you that in a couple of minutes. Okay, so let's continue here. Let's see a couple more things here. Things worth mentioning is this one. So they do the following thing. They add some noise on top of the input audio signal, such as for example, in the left diagram, we see white noise and then on the right, the pub noise. So that's kind of background noise from a restaurant or a pub or something. And you can see that the whisper, which is this red star, you can see it definitely does not have the lowest error rate on the LibreSpeech test screen, but then as the noise levels keep increasing, it all of a sudden becomes one of the best models. And especially here on pub noise, you can see that the model outperforms all of the other ones. So it has lower error rates. So that means it's more robust to noise, and especially the natural noise compared to this synthetic white noise. Okay, let's continue here. They also compare whisper to other baselines in this long form transcription, where because the model can only see 30 second audio inputs, they have to do sliding window approach. And they show here that they are on pair and even better compared to many of these even commercial services. You can see the blue candle bars are for most of these long form transcription datasets lower than the baselines. Given there are some exceptions, obviously, like here you can see that this company B is outperforming them, but yeah, that's kind of impressive. This is also cool. They show how they compare to human transcription. So you can see how whisper is very close to other, even human transcription services. So here they took actual, well, humans to transcribe some, I think like they had 25 recordings of text or something, and then they compare them, and you can see it's on pair pretty much. But then there is this computer assisted human transcription which is the best out of all of these approaches. So that again means that this combination of humans and machines is still the best way to go about many, many problems. Let's see a couple of more results here. Obviously model parameter, this is again not that surprising, although it's very noisy. Let me show you what's going on here. So we have English speech recognition on this leftmost plot. You can see that the model parameters as it's scrolling, we can see that the error rate is obviously falling down, but there is some saturation going on. And there is also a lot of variance. If you take a look at these lighter lines, the dark blue one is just the average of those. So that kind of means the performance varies a lot across these 12 data sets. And we can also see saturation. I guess that's the main takeaway here. We see saturation on English in particular. And then on multilingual speech recognition, we can also see like a trend, downward trend, general downward trend. And again, a lot of variance if you take a look at some of these light blue curves. Considering that the models are approaching, that we are doing a supervised learning approach here and that the models are, so where was the diagram here? So that the models are on pair with this human services, with human transcription that might explain it. And the final results I want to show you are these ones here. Basically, you can see two lines here, two fits, and one is the English only training data set. And the second one is multilingual and multitask. And if you're in this regime where the flops are lower, that means you're training smaller models. And you can see that with smaller capacity, you're suffering the, so you wanna have, this is the error rates, you wanna be lower. So you can see the English only version is better. But the multilingual multitask models scale better. So that means when you go into the bigger model, like bigger scales, you can see that the error rate is actually lower for the multilingual multitask, which is not that surprising, I guess, and encouraging, given that we can just scale it even more and have higher quality data and we can achieve even better results. These are some candle bars about the talk, the actual normalizer, I'm gonna skip that. And finally, it's important to realize that there are still a lot of heuristics that they had to use to make this decoding reliable. So we have developed a set of heuristics that help avoid failure cases of long form transcription in particular. So they say here, first we use beam search with five beams using the log probabilities as the score function to reduce repetition looping, which happens more frequently in greedy decoding. And that's a common problem you have with greedy decoding in general, like you have the repetition problem. And then they say we start with the temperature zero, I always selecting the tokens with the highest probability, which is basically greedy, and then increase the temperature by zero two up to 1.0, when either the average log probability over the generated tokens is lower than minus one, or the generated text has a GZIP compression rate higher than 2.4, which would probably mean that if it's higher than that, that means that the text is too random and too hard to compress, which means it's probably gibberish. I guess that's the rough reasoning behind the heuristic. And there is a lot more details here, I'm just gonna skip that for now. And you can see here, like in a tabular form, the same thing, adding beam search, adding temperature fallback, adding previous text conditioning, adding this voice activity detection, initial timestamp constraints, all of those help improve the average performance across different transcription, long form transcription data sets here. So I guess the main takeaway from all of this is, if you take a look at the diagrams here, this plot is kind of suggestive that, given higher quality and abundance of audio files, of audio data for a particular language of interest, and even off the shelf model, you can achieve remarkable performance. And we also wanna do the multi-lingual multitask setup. Okay, guys, having seen the main ideas from the paper, let's now switch to the code, and let me walk you through and try and show you the correspondence between the code and the diagram. You see here, I'm gonna see in a bit more detail how this special token pipeline looks like. So I already went ahead and cloned the repo, created a condo environment, so all of that should be fairly trivial. Everything is written down in the README quite nicely. So yeah, let me just show you the types of models they have. So they support a couple of models. You can see here, tiny, base, small, medium, and large. You can also see the.end versions, which are English only, and these ones with the other suffix are basically multi-lingual multitask models. Okay, so you can specify multiple audio files here. We're using small model by default. Default task is transcribe. You can also choose translation. Language is set to none, which means we'll infer automatically what the language is, so that's gonna be interesting. And then bunch of parameters around the coding and these thresholds that help robustify the decoding pipeline. Now they just pop some arguments. I'm gonna skip all of these parts that are not relevant and crucial to the understanding of the model. So this is like temperature goes from zero to one with increments of 0.2. So that's something they said in the model they used to robustify the decoding, if that's even a word, and load model. So let's kinda go here and let me jump all the way to here. So I'm gonna disable breakpoints, put a breakpoint here, and let's just go to whisper. Okay, so basically in the background, we download the model. Actually, I already downloaded beforehand, so it just kinda loads the weights. So here is how the model looks like. Let me maybe now enable the breakpoints and enter here. So we have the autoencoder and we have the text decoder. So those are the two stacks we saw previously in the diagram. So let me now enter the, let's enter the encoder and let's see how it looks like. So we can see here, we have two comp1d layers. We have the positional embeddings which are not learned, so they're sinusoids. And let me just put the code and the diagram side by side, it's gonna be easier. Okay, so here we are. I can see the diagrams on the side here. So again, we have the two comp1d layer, so that's this part here, the stem of the encoder. Then we have the non-learnable positional embeddings. You can see those here. And finally, we have bunch of blocks, which are just residual attention blocks. So that's nothing new there. And finally, we end up with a layer norm. And later we'll see in the actual forward pass how we pass the male spectrogram through the comp1d and then we have the JALU, blah, blah, blah, the stem, and then the encoder part. So everything is fairly trivial here. Okay, so let's exit the construction of the audioencoder. Let me just make sure that I have everything enabled. Let's hit F5 and enter the decoder. And now let's just go back to this side by side view. So here is the text decoder. You can see a couple of things. Of course, embedding table here. And because we're dealing with obviously a decoder model that has a vocabulary, we have positional embeddings, which are, as you can see here, these ones are learned. So it says here, learn positional embeddings. And that's why we have parameter. And we just initialize, as you can see here. So 448 is the context length that they are using and the state is around, yeah, 768. Okay, and then we have a bunch of blocks. These ones are cross-attention true. So that means they're additionally gonna have this third component in the transformer layer, the so-called cross-attention block. But yeah, that's it. Fairly similar to what we saw in the encoder part. Okay, and we have a mask, which is going to be basically a causal mask. This is gonna be a causal decoder. So if I were to print this, you can see minus infinities on the upper triangle, which means we're gonna mask out the future tokens and this particular token can only attend itself and the previous token and the previous token, et cetera. Okay, hopefully nothing new there, just like standard attention stuff, transformer stuff, sorry. And now let's go back. So let me now kinda expand this again. So we load the dictionaries. We just push the model to the device. Let's exit here and they have a small, well, it's not a bug, but they do the two device two times, which is kinda unnecessary. But yeah, okay. So here we are, we popped the audio path. So I'm gonna use the one that they provided in the code base. So maybe it will be useful for me to show you how that sounds. So here it is. And so my fellow America, ask not what your country can do for you. Can do for you. Ask what your country can do for your country. Okay, so that's the audio file we are using and let's now transcribe it. So again, let me make sure we are enabling the breakpoints. We hit F5 and here we are. So let's see what's interesting going to happen here. So the first interesting part is the log mal spectrogram. So let's see how we convert an audio file, which is just a path into a mal spectrogram. So let's enter here. So the first part is loading audio files. So we just use FFmpeg for that. And then you can see blah, blah, blah, some from buffer because these are bytes, integer 16. And that's we divide by 32,000 because precisely because we're using int 16. So that's like, I guess two raised to the power of 15 is exactly that number. And that's why we are basically putting the amplitude in the minus one, one range, just normalization detail. Okay, so once we have, so we are in the log mal spectrogram. So once we load the data, which is just raw data at the moment, so that's gonna be just a sequence of numbers, 176,000 floats. Now we form the hand window, just some signal processing stuff. Then we apply the short, basically a Fourier transform. It stands for, what does it stand for? A short time, yeah, short time Fourier transform. Okay, so we applied the short time Fourier transform, which is gonna basically convert this audio file into a spectrogram, which is your regular spectrogram. So if I were to print this particular variable here, the spectrogram, the shape is gonna be, as you can see here, 201 and 1,101. So 201 is the frequency dimension, this one is the time dimension. The reason being they have some, they're doing sliding window approaches. So that's why we don't have as many timestamps as in the original raw audio. Okay, so then we do the magnitudes because we wanna have the magnitude component of the Fourier. And then we form these filters. As you can see here, we just, what they do is they use this Librosa to generate these MEL filters, and then they saved the MEL filters into this particular file. So you can see here, this assets MEL filters, MPZ. So if you go to the assets directory here, you can see that the MPZ, so they just kinda save them there and then they can avoid having the dependency. Anyways, it's already too many details. Let's get out of here, filters, and then we do the matrix multiplication. Again, that's a detail where by, once you multiply using these triangular filters, once you multiply that with the actual magnitude of Fourier spectrogram, then you end up with a MEL spectrogram. And now we're gonna see how this is going to have 80 bands. So if I were to print the shape of this thing, it's gonna have, oops, let's just grab this variable. It's gonna have, as you can see here, 80 bands and we have 1,100, again, time dimension. And if you go to the paper, you'll see all of these details there. So again, let me just quickly jump to that part where they mention it, somewhere here. So all audio is resampled to 16,000 hertz and an 80 channel log magnitude MEL spectrogram representation is computed on 25 millisecond windows with a stride of 10 milliseconds. And because we're using these windows of 25 milliseconds, that's why we end up with a much shorter representation in the time dimension compared to having a 176,000 or something. Okay, let's go back to the code, some normalization. I'm not gonna focus too much on this. And finally, because the language is none, we're now going to predict the language. Let's see how that's gonna work. So detecting language using the first 30 seconds, blah, blah, blah, we first do the padding, which is just going to pad this to a 30 second audio file. So that means it's gonna have 3,000, basically dimensions in this time, along the time axis. And now let's basically do the detect language part. So we passed the segment, which is just a pad MEL spectrogram, which means basically we padded with all zeros to the right. And let's now do the detection. So here is how it's gonna look like. We get the tokenizer first. It might be interesting to just briefly jump into the tokenizer. And let me show you how it looks like. So we specify that it's a multilingual tokenizer, task is transcription, we have English, and now we do the build part. Okay. So here is how building the tokenizer looks like. We are basically taking the pre-trained GPT-2 tokenizer fast from the Huggins face library, and we just pre-loaded. And here are the special tokens. So here are the tokens we saw before in the paper. So we have the start of the transcript token, SOT. We have, as you can see here, I think this language's keys is like 99 keys in total or something. Let me just double check that. So you can see here, 99 keys precisely. So that means we'll add for each of those languages, we'll add this special token. For example, this and like the smaller sign and then bar and then N, like EN, and then the rest. And for each of the languages, a similar thing. So you can see here how the keys look like here for the languages. And then we have translate, transcribe, start of LM, start of prev, no captions and no timestamps. So each of these have some meaning. We're gonna see maybe some of them are subset a bit later. So we end up adding those to the pre-trained Huggins face tokenizer and we return back here. Okay, so now let's just see what's interesting. None of this is gonna be interesting for the time being. They just create a SOT sequence, which is gonna have, as you can see here, they start with the SOT and they added the EN token and they added the transcribe token. And so that's basically the sequence we saw before here. So let me just go back to the paper. So that's your classic sequence here. So the start of transcript, language tag and transcribe. Let's go back to the code. But we are not going to use that for the language detection that we are currently doing. So that's why I said it's not that important at the moment. So we just do unsqueeze on the mail, nothing fancy. And then we pass the mail through the encoder stack. So let's now get the features. So you can see here that everything is the same as what the diagram says. We do a forward pass through, I guess, 12 blocks or something. So we can just skip that. Let me just see how many blocks this one has. I guess it's like 12 or something, 12, yeah. Let's just pass through all of that. And we ultimately end up with the following shape. So you can see here, 1,500. So because we have one of these has stride of two, stride is set to two. And that's why we went from 3,000 to 2,500. And the dimensionality is 768. So that's the model dimension of the encoder. Okay, let's exit here. Number of audio is gonna be one because we only have a one audio file. And here you see, so here we just set the SOT token and we create a tensor that has SOT. And now we're gonna pass the mail, so the encoder features from the output of the encoder, and we pass the SOT. And let's see what the model is gonna generate. So let's see which language it gives the highest probability to. And that's how we pick the language. So let me just go into the code instead of explaining like this. So here we are, we are in the text decoder forward pass. I'm gonna ignore the cache parts. Those are just implementation details of how they optimize this execution of the forward pass of the decoder. And let's see what's going on. So they do the token embedding. So again, remember this is just the SOT token, as you can see here. And then they just add the positional embedding. So all of that is kinda standard stuff. And then you can see here, they're doing the cross attention. So XA, XA is our, the output features of the encoder, as you can see here. And so that's what's going on here. So because of that, I'm just gonna skip all of that. And let's get here. Now the output features are multiplied by the, we do matrix multiplication with the embedding table, and that's gonna project those from 768 dimensions into the vocab space, which is like, let's see what's the dimensionality. So logits shape is gonna be 51,865. Okay, so those are the logits. And now comes the interesting part. So we create a mask that initially just has basically ones for every single token in the vocab. Okay, so that's the mask initially. And then for the language tokens, let's set that mask to false, which means we do not want to mask the language token. So the EN, the DE, all of the 99, I think this should be 99. Let me just make sure that's the case. So let's see what's the length of this list, 99, precisely 99. And so that's how we modify the mask. And now we put the logits of every single token, except for the language tokens to minus infinity, which after we apply the softmax is gonna be, is gonna set them to zero. So that means they'll have zero probability and only the language tokens will have non-zero probability. Okay, so here we do argmax to find the highest probability language token. We end up with, I think, English or something. So this is, I think, English. And we also return the probabilities for the other language tokens. We just zip that and we return back those results. So we return the actual language token and we return the probabilities. But I think, yeah, this one is not even used. So we just grab the, this is kind of wasteful. They could have just passed it directly here, but anyways. So they end up with the language. We just grab the key. So you can see the probabilities are language token and the probability, you can see that EN, English has 0.95, roughly. And that's why we're gonna pick precisely English as the language here. And we output detected language is English language. Okay, that's the first part. So we now have the English predicted. Now let's continue doing the actual transcription. I'm gonna skip over all of these details. Let me just disable the break points. Let's get to the interesting part. These are just some details where, so the precision is, as you can see here, 20 milliseconds. That's why this is 0.02. Those are, I guess, not that vital. Okay, we just pad again the, the MEL so that we end up with 3000 here. So let's just check the shape of this segment. So you can see it's 83,000 because we have 80 MEL bands and we have 3000 because we have a 30 second padded audio clip. Okay, we calculate the duration. It's gonna be 30 seconds. And then this is where the magic starts happening. Decode will fall back. Let's see how that's gonna work. Okay, here we are. Temperature, again, we have the list of various temperatures. We grab the first one, which is gonna be the zero temperature, which means we are basically greedy decoding. And well, actually, because we'll be actually using the beam. So let's just continue here and see what's going on. Decoding task. So this is how they kind of design this. They form a decoding task and then they call run on that task and they pass the MEL spectrogram. So let's see what's going on. I'm gonna skip some of these details because there's too many stuff going on. Just wanna focus on the stuff that's gonna be useful for whisper understanding, of course. So here, this one implements this penalty with the length. So when you're doing beam decoding, you want to take into account how probable that beam is, but you also wanna penalize if it's too long. So this is what this maximum likelihood ranker does here, as you can see here. So penalty and we are appending the log prop divided by the penalty. I'll skip this, but in case you want to learn more, and I think this repo is amazing when it comes to understanding how beam decoding works. If you wanna, I might create a separate video where I just go in a lot of depth and explain how this works, but here I'm gonna skip those details. So we form a beam decoder here, and then this is the interesting part. So here are some of the heuristics that they are using. They're suppressing certain tokens during the decoding process. So one of those is gonna be suppressed blank, which means we're gonna suppress the blank token as the first token after we start the transcription. So we'll get to that a bit later. Then they have a specific list of tokens that they wanna suppress. So because we have, let me enter this one. So let me enter this one. So suppressed tokens are set to minus one because that's our input argument to the program. And now let's see how that's gonna be mapped into the actual tokens. So here you can see we extend by these non-speech tokens and here they are. So just a list of tokens that cannot appear in the spoken language. And again, I'm gonna skip because there is a lot of details, but that's the rough point. We wanna help the decoder by just kinda mapping, by masking out those particular tokens. Okay. And they add some more. We don't wanna predict the start of transcription because we already know we are transcribing. So that's kinda a no-brainer, but yeah. And finally, the no-captions, which is the token that tells you that we don't have any human voice. We just have like music or something going on. Okay. That's a second heuristic they have. And then they also have these applied timestamp rules, which do what? Okay. I have a break point here. We'll see what it does once we hit the apply. So let's just continue. Let's continue here. Let's exit here. And we are now in the run method. So all of that was construction of this decoding task. Now let's see how the actual run looks like. So we first fetch the audio features. And this one is gonna again, just pass the the mel through the encoder and return back the audio features. So I'm gonna skip all of that. So I'm just gonna disable that. Let's fetch the audio features by passing the mel through the encoder stack. And by the way, I'm running this on a CPU because my cond environment was not properly set up. That's why it's a bit slower, but it's still working fairly fast, which is kinda cool. Okay. So we set the initial tokens to these three. So that's the SOT, that's the English, and that's the transcribe. That's our initial sequence. And then this is gonna be a no-op. Basically we just passed the language. We already detected language. We know it's English. So that's gonna be a no-op. Now we do repeat by five because we are doing beam. Again, I'm not gonna focus on the details of beam, but let's just see the shape before and after. So the shape now is 1500 because that's the time. And then this is the actual embedding dimension. So let's see what's going on after this call. We have five. So we have five because we'll have five hypotheses that we are keeping track of during the beam decoding. And we do the same for tokens. And then now here is the main loop. So here is where the magic happens. Okay. Let's start here. So we now sample for 224 samples, and we first pass the tokens as well as the audio features to grab the logits. So it's gonna be a simple pass through the decoder. Again, some hook stuff that's again in optimization detail. Here's the decoder call. So what we do is again, we pass all of the, we embed our three tokens. So here, let's see what's the shape now. So you can see here five, three, 768, because we have three, five because we have beam, and this is the dimension of the model. Now we pass all of those. And we also have the, again, we are cross attending to the output features of the audio encoder. Okay. That's the idea. And finally, we end up with the logits. So we can skip all of this. This is just like regular transformer stuff. Whoops. And let's exit here. Let's exit. And we have our logits. Now this is an interesting point. Let's see what happens here. Because we are in the iteration zero, and because no captions is set to, yeah, because no captions token exists, we're going to enter this branch. And what we do is we grab the logits for the SOT index. So that means what's the, once we condition the model on the SOT token, what's the distribution across the next tokens after the SOT. And that's in case that there is a high probability for the no caption probability, then that means we don't have any audio, any human vocals in the audio clip. So again, we index here, we grab the SOT logits, and that's gonna be, let's see the shape. So here is the shape. So you can see here, because it's being, we have five, but we just have the logits above the SOT token. And then additionally, we grab precisely this particular, we are interested in the probability of this particular token, and that's gonna be the no caption probability. And in our particular case, it's very low, but if it was high enough, then we would basically ignore the decoder output and just say that, okay, there is no voice happening in this audio clip. And let me remind you here, that's this part. So we either have the language tag or the no speech tag, and that's the voice activity detection. So that's how that part, that task breaks. So now you see it in code. Okay, so having collected that, let's now grab the, because remember we actually have the SOT, English and transcribe tokens. And we're gonna do the decoding, even if this was higher probability, we're still gonna do the decoding here and assume as if we wanna transcribe the audio, but in case this was high enough, then we would just ditch and ignore all of that output, which is probably wasteful. Maybe some, just a break statement here or something would be more optimal, but yeah, that's just an optimization detail. So we grab the logits for the fourth token, we apply a bunch of filters. So those are the filters which I mentioned before. So let's see. The first one is the suppressed blank. And you can see here that we wanna mask the blank token and we set it to minus infinity because we don't want to generate, as the first token does not, like it doesn't make any sense to store the audio with a blank token. That's why they can mask it out. What's probably weird is why is it not just like a generic white space and instead it's just like a space. I'm not sure about that point, but yeah, let's continue. And let me just enable all of the breakpoints. Let's enter here. So here's the second one. These are the special tokens. So that's again a list of these special tokens that we wanna suppress. We set all of them to minus infinity, the logits. So that means their probabilities after the softmax will be set to zero. And we have the final one. Here it is. So here is the no timestamp. And let's see how that one works. I'm not sure about this one. Okay, so it seems that we can never sample the no timestamps because we set it to minus infinity. And that means we kinda cannot ever go down this route here, right? So that's kinda weird. So in which setting do they actually sample this particular one? I'm not sure about that. Okay, let's continue here. I think this is gonna be a no op. It's constantly gonna, yeah, it's gonna skip all of that. So let's just put the breakpoint here, hit that five. And let's continue. Let's see what else. So we now apply the log softmax. And now this is one of these additional details. It's kinda important. So if the sum of probability over timestamps is above any other token, then sample the timestamp. So that's how we sample the timestamp. So let's see this part. That's kinda important. So we take the log probabilities, which are basically after we apply log softmax, and before that we did the masking. So that means now we have all of the relevant tokens can be sampled from. And as you can see here, we grab only those tokens which correspond to timestamp. And we do a sum across those just to get their cumulative probability, log probability. And we do the same thing for all the previous tokens. As you can see here, we grab them, actually we grab the max one, not the sum. And if the probability of this one is bigger than that one, then we basically mask all of the non timestamp tokens. And then we're certain that we'll generate the timestamp information. So as you can see in this particular instance, we are generating, because this is a first step, we're generating the zero zero tokens. So let me remind you here, we're actually generating this begin time as the first token, which kinda makes sense. Okay, so let's go back to the code. Let's exit here. Let me just shift F11, we're out. Those are the logits. And now we have this, what's this update function? This is just basically wrapping up the, okay, so this is actually the beam decoding part, collecting the candidates, blah, blah, blah. As I said, I'm gonna skip all of those. Let's just kinda assume that as a black box, we end up sampling, I guess, five candidates. So let's see how that's gonna look like. So tokens is gonna be five, four, and before that we had five, three. And that's because, yeah, we now have five new candidates that are highly likely. And we can just skip everything else here. I think we're safe to skip all of this. I'm just gonna hit F5 to the break point, exit, and that's it. So now we just keep on doing the same thing. So we just keep on, so we get the logits. Whoops, let's just exit from the text decoder. So we get the logits, and then now we'll be skipping this because we're not in the zero step. That's the only step where we calculate the no caption props. And then we just keep on sampling until we hit the end of transcription token, pretty much. So let me now remove all of the break points. Let's put a break point here. Let's exit and let's see what's gonna happen. Okay, so here we are. So tokens, let's see the shape. We have five and each of them has 33 tokens, okay? We return back the probabilities as well as the no caption probabilities. Okay, we can skip all of this. I'm not gonna focus on that too much. And let's just return back to result. So let's return back to result. Here we are. So now we have the results. And now we have this fallback mechanism, as you can see here. And in our case, we'll just be directly, we'll get directly to the results and we will not enter this branch, which basically would force us to do decoding again with a different temperature. So because we satisfied these conditions and that's that the compression ratio is not higher than this threshold 2.4, which means we are not generating something that's super random. And we're also passing this test here. The log probability, the average log probability is lower than minus one. Because of that, we can just kind of, we are already good. We have the correct transcription and we can exit here. We check here whether we have the no caption threshold and it's set. And because of that, we now check, so this is the check. Is the no caption probability higher than the threshold? If it is, then we should skip. And that means that we would literally skip the whole segment here and continue. Continue basically transcribing the next segment. Okay, so in our case, it's not the case. So we just ignore this part here. Now there is some part about the timestamps here. You can see here tokens greater than equal timestamp begin. Because the timestamp token have this nice property that they are sequential. So that means that the beginning one, the zero zero moment of the transcription is maybe on has IDN and then all of the bigger timestamps have even bigger basically index in the embedding table. And that's why this is gonna work. So this is gonna fetch all of those tokens that are timestamp tokens. And then we basically use that information to add some segments. This is not that by looking, go through the code at your own pace. But I'm gonna return back to results and here we are. So we can now tokenizer decode all tokens. So if we were to print this, you can see, and so my fellow Americans, ask not what your country can do for you. Ask what you can do for your country. So that's the, if you remember the clip from the beginning of this coding session, that's precisely what we were playing. So here we just saved that as the TXT and we also saved this format where we have the timestamp information additionally aside from the transcription. Guys, that's pretty much it. Hopefully you saw, I mean, the details of the code are complex, but the idea is if you've bear with me, up until this point, congrats. Hopefully you've seen how this intricate system here works. There is a lot of heuristics, but the system is ultimately very elegant in the sense that it uses off the shelf transformer module and basically just leverages huge data and some decoding heuristics. Cool, so if you like this video, consider subscribing, sharing it out with your friends and see you next time. So until then, thanks for watching.
[{"start": 0.0, "end": 2.64, "text": " OpenAI just open sourced Whisper,"}, {"start": 2.64, "end": 5.48, "text": " which is an automatic speech recognition system"}, {"start": 5.48, "end": 7.7, "text": " that's approaching the human level baseline"}, {"start": 7.7, "end": 11.92, "text": " on both the accuracy as well as on the robustness wise"}, {"start": 11.92, "end": 13.64, "text": " for the English transcription."}, {"start": 13.64, "end": 15.34, "text": " And that's super awesome."}, {"start": 15.34, "end": 17.88, "text": " So in this video, I'm gonna cover the paper"}, {"start": 17.88, "end": 21.0, "text": " as well as I'm gonna go through the code behind the paper,"}, {"start": 21.0, "end": 22.8, "text": " which they luckily open source."}, {"start": 22.8, "end": 25.060000000000002, "text": " So the inference code as well as the model checkpoints"}, {"start": 25.060000000000002, "end": 28.2, "text": " have all been open sourced and that's kinda cool."}, {"start": 28.2, "end": 29.64, "text": " So yeah, without further ado,"}, {"start": 29.64, "end": 31.64, "text": " let's dig into the video."}, {"start": 31.64, "end": 34.0, "text": " So here is the tweet that kinda announced"}, {"start": 34.0, "end": 35.8, "text": " the release of Whisper."}, {"start": 35.8, "end": 38.52, "text": " They say we've trained a neural net called Whisper"}, {"start": 38.52, "end": 40.7, "text": " that approaches human level robustness and accuracy"}, {"start": 40.7, "end": 42.120000000000005, "text": " on English speech recognition."}, {"start": 42.120000000000005, "end": 44.480000000000004, "text": " It performs well even on diverse accents"}, {"start": 44.480000000000004, "end": 45.72, "text": " and technical language."}, {"start": 45.72, "end": 48.24, "text": " Whisper is open source for all to use."}, {"start": 48.24, "end": 51.28, "text": " You can see that Whisper can do many tasks"}, {"start": 51.28, "end": 54.34, "text": " and it's a multilingual multitask system."}, {"start": 54.34, "end": 56.400000000000006, "text": " So here's a couple of examples."}, {"start": 56.400000000000006, "end": 57.68, "text": " So English transcription."}, {"start": 57.68, "end": 60.6, "text": " So given an audio in English, ask not what your country"}, {"start": 60.6, "end": 62.48, "text": " can do for blah, blah, blah."}, {"start": 62.48, "end": 65.22, "text": " You ask and then you see the transcription here,"}, {"start": 65.22, "end": 67.12, "text": " ask not what your country can do for you."}, {"start": 67.12, "end": 67.96, "text": " So that's kinda trivial."}, {"start": 67.96, "end": 71.44, "text": " And then we have any two English speech translation"}, {"start": 71.44, "end": 75.0, "text": " where we have maybe audio file in let's say Spanish,"}, {"start": 75.0, "end": 78.2, "text": " El rapido thorough marron salta sobre."}, {"start": 78.2, "end": 79.86, "text": " And then you have the translation,"}, {"start": 79.86, "end": 82.9, "text": " the quick brown fox jumps over."}, {"start": 82.9, "end": 85.08, "text": " And then we have non-English transcription"}, {"start": 85.08, "end": 87.92, "text": " where the audio file is again in some foreign language"}, {"start": 87.92, "end": 89.12, "text": " that's not English."}, {"start": 89.12, "end": 92.46, "text": " And the transcription is also in that same language."}, {"start": 92.46, "end": 94.12, "text": " So I'm not gonna try and pronounce Chinese."}, {"start": 94.12, "end": 96.96, "text": " My Mandarin, it's not at its peak."}, {"start": 96.96, "end": 100.32, "text": " So there is the last task here is the no speech one"}, {"start": 100.32, "end": 104.48, "text": " where given the background music playing or like whatever,"}, {"start": 104.48, "end": 106.64, "text": " it's important that we don't have human voices."}, {"start": 106.64, "end": 109.4, "text": " The model outputs a special token signaling"}, {"start": 109.4, "end": 113.48, "text": " that there is well, no human speech going on."}, {"start": 113.48, "end": 115.12, "text": " Other than these tasks,"}, {"start": 115.12, "end": 118.76, "text": " the model can also do language detection,"}, {"start": 118.76, "end": 120.96000000000001, "text": " which means it can given the audio detect"}, {"start": 120.96000000000001, "end": 125.76, "text": " which language does that particular audio file belong to."}, {"start": 125.76, "end": 128.8, "text": " Okay, let's jump into the paper."}, {"start": 128.8, "end": 130.86, "text": " Let's see the abstract"}, {"start": 130.86, "end": 133.84, "text": " and then I'm gonna first give you some high level overview"}, {"start": 133.84, "end": 136.26, "text": " and then we can dig a bit deeper."}, {"start": 136.26, "end": 138.88, "text": " So the name of the paper is robust speech recognition"}, {"start": 138.88, "end": 141.2, "text": " by a large scale weak supervision."}, {"start": 141.2, "end": 144.11999999999998, "text": " We're gonna see that this is one of the main components"}, {"start": 144.11999999999998, "end": 147.88, "text": " of the paper so just like the data set itself."}, {"start": 147.88, "end": 150.64, "text": " And let's kind of skim through the abstract first."}, {"start": 150.64, "end": 153.79999999999998, "text": " So we studied the capabilities of speech processing systems"}, {"start": 153.79999999999998, "end": 156.76, "text": " trained simply to predict large amounts of transcripts"}, {"start": 156.76, "end": 158.83999999999997, "text": " of audio on the internet."}, {"start": 158.83999999999997, "end": 163.6, "text": " When scaled to 680,000 hours of multilingual"}, {"start": 163.6, "end": 165.48, "text": " and multitask supervision,"}, {"start": 165.48, "end": 168.67999999999998, "text": " the resulting models generalized well to standard benchmarks"}, {"start": 168.68, "end": 172.28, "text": " and are often competitive with prior fully supervised results"}, {"start": 172.28, "end": 174.44, "text": " but in a zero shot transfer setting"}, {"start": 174.44, "end": 176.66, "text": " without the need for any fine tuning."}, {"start": 176.66, "end": 177.72, "text": " When compared to humans,"}, {"start": 177.72, "end": 181.6, "text": " the models approach their accuracy and robustness."}, {"start": 181.6, "end": 184.44, "text": " Okay, so let's first start by just showing you"}, {"start": 184.44, "end": 186.60000000000002, "text": " the high level diagram,"}, {"start": 186.60000000000002, "end": 188.04000000000002, "text": " how the architecture looks like"}, {"start": 188.04000000000002, "end": 190.60000000000002, "text": " and these special tokens that you use"}, {"start": 190.60000000000002, "end": 193.24, "text": " to prompt the model to do specific tasks."}, {"start": 193.24, "end": 196.32, "text": " So the architecture itself is literally"}, {"start": 196.32, "end": 200.35999999999999, "text": " like off the shelf transformer encoder decoder transformer"}, {"start": 200.35999999999999, "end": 204.92, "text": " module from 2017 attention is all you need paper."}, {"start": 204.92, "end": 208.23999999999998, "text": " So they wanted to kind of remove that confounding factor"}, {"start": 208.23999999999998, "end": 210.04, "text": " of just tweaking the architecture"}, {"start": 210.04, "end": 212.48, "text": " for this particular task of speech recognition."}, {"start": 212.48, "end": 215.32, "text": " And instead they just took the off the shelf model"}, {"start": 215.32, "end": 217.07999999999998, "text": " and they've been playing with the data."}, {"start": 217.07999999999998, "end": 219.68, "text": " Like data is the main part here"}, {"start": 219.68, "end": 221.24, "text": " as well as the special tokens."}, {"start": 221.24, "end": 222.84, "text": " We'll see those in a second."}, {"start": 222.84, "end": 225.44, "text": " So again, how the system works on the high level"}, {"start": 225.44, "end": 227.44, "text": " is you grab an audio file"}, {"start": 227.44, "end": 230.16, "text": " and then you convert it from its raw form"}, {"start": 230.16, "end": 233.35999999999999, "text": " into something called log male spectrograms."}, {"start": 233.35999999999999, "end": 236.6, "text": " So spectrograms are these representations"}, {"start": 236.6, "end": 241.38, "text": " where here on the X axis, you have the time information."}, {"start": 241.38, "end": 244.84, "text": " So the time steps and then on the Y axis"}, {"start": 244.84, "end": 246.68, "text": " you have the actual frequency."}, {"start": 246.68, "end": 250.12, "text": " And then the color basically tells you"}, {"start": 250.12, "end": 253.64, "text": " what's the amplitude of that particular frequency"}, {"start": 253.64, "end": 254.96, "text": " at that particular point of time."}, {"start": 254.96, "end": 256.68, "text": " So like this point here,"}, {"start": 256.68, "end": 258.96000000000004, "text": " like this will be the particular time."}, {"start": 258.96000000000004, "end": 260.6, "text": " And then you can see the frequency here."}, {"start": 260.6, "end": 262.48, "text": " And the frequency is usually in Hertz"}, {"start": 262.48, "end": 265.16, "text": " but in the case of male spectrograms, it's in males."}, {"start": 265.16, "end": 267.6, "text": " And male is just like a logarithmic scale"}, {"start": 267.6, "end": 270.28000000000003, "text": " because that's how we perceive pitch"}, {"start": 270.28000000000003, "end": 273.40000000000003, "text": " and not like linearly like Hertz scale."}, {"start": 273.40000000000003, "end": 277.52, "text": " So that's a TLDR, but I'll link some resources"}, {"start": 277.52, "end": 280.90000000000003, "text": " for explain very nicely how male spectrograms work"}, {"start": 280.90000000000003, "end": 283.5, "text": " because that's not the topic of this video."}, {"start": 283.5, "end": 287.0, "text": " Okay, so once we have that representation of the audio"}, {"start": 287.0, "end": 289.4, "text": " then we literally do a feed forward pass"}, {"start": 289.4, "end": 291.04, "text": " through the encoder stacks."}, {"start": 291.04, "end": 294.0, "text": " You can see here just blocks of simple transformer layers"}, {"start": 294.0, "end": 296.56, "text": " that consist of self-attention and MLP,"}, {"start": 296.56, "end": 297.88, "text": " multilayer perceptron"}, {"start": 297.88, "end": 301.24, "text": " and outcomes the final representation here."}, {"start": 301.24, "end": 304.44, "text": " So once you have the final audio representation"}, {"start": 304.44, "end": 307.14, "text": " you use that to condition the decoder,"}, {"start": 307.14, "end": 309.12, "text": " the causal decoder model here"}, {"start": 309.12, "end": 311.12, "text": " via cross attention mechanism."}, {"start": 311.12, "end": 314.68, "text": " And then the whole magic is in these special tokens."}, {"start": 314.68, "end": 318.36, "text": " So you can see here, we have this start of transcription"}, {"start": 318.36, "end": 320.64, "text": " token, we have then the English token"}, {"start": 320.64, "end": 322.64, "text": " and then we have the transcribe token"}, {"start": 322.64, "end": 325.36, "text": " and then the model generates like this is first"}, {"start": 325.36, "end": 330.36, "text": " the timestamp and then outcomes the actual transcription"}, {"start": 332.14, "end": 333.34000000000003, "text": " of the audio file."}, {"start": 333.34000000000003, "end": 336.72, "text": " You can see that the, here is the target."}, {"start": 336.72, "end": 338.84000000000003, "text": " So it's a supervised dataset."}, {"start": 338.84, "end": 342.96, "text": " We do have the label for the audio obviously"}, {"start": 342.96, "end": 345.15999999999997, "text": " and then we just do, we shift by one"}, {"start": 345.15999999999997, "end": 348.71999999999997, "text": " and do the classic language modeling objective here."}, {"start": 348.71999999999997, "end": 350.91999999999996, "text": " There are some details that we can kind of dig"}, {"start": 350.91999999999996, "end": 352.79999999999995, "text": " into a bit later like the coder uses"}, {"start": 352.79999999999995, "end": 354.2, "text": " learn positionally codings."}, {"start": 354.2, "end": 357.08, "text": " Here we have sinusoidal ones, but as I said"}, {"start": 357.08, "end": 360.67999999999995, "text": " architecture is not the main contribution of this paper."}, {"start": 360.67999999999995, "end": 364.44, "text": " It's the dataset and it's this multi-task setup"}, {"start": 364.44, "end": 365.55999999999995, "text": " with the tokens."}, {"start": 365.55999999999995, "end": 368.73999999999995, "text": " So ignoring this part for the time being"}, {"start": 368.74, "end": 372.12, "text": " on the left, let's focus on the right part of the pipeline."}, {"start": 372.12, "end": 374.68, "text": " So we start with the start of transcript token"}, {"start": 374.68, "end": 378.52, "text": " and then the model will autoregressively decode"}, {"start": 378.52, "end": 381.68, "text": " either this no speech token, which basically means"}, {"start": 381.68, "end": 385.32, "text": " that we are not detecting, as I said previously, I think"}, {"start": 385.32, "end": 388.08, "text": " there is no speech going on the audio."}, {"start": 388.08, "end": 390.92, "text": " The model is only detecting maybe music or something"}, {"start": 390.92, "end": 394.28000000000003, "text": " without human vocals or the model could generate"}, {"start": 394.28000000000003, "end": 397.16, "text": " language tag and these are by the way, the only things"}, {"start": 397.16, "end": 401.16, "text": " that the model can generate because they have various"}, {"start": 401.16, "end": 404.12, "text": " like masking mechanisms to make sure this is the case."}, {"start": 404.12, "end": 408.36, "text": " Okay, so let's say the language tag was decoded."}, {"start": 408.36, "end": 411.96000000000004, "text": " In that case, we basically then have two options"}, {"start": 411.96000000000004, "end": 413.68, "text": " either transcribe or translate."}, {"start": 413.68, "end": 414.68, "text": " We've seen the differences."}, {"start": 414.68, "end": 418.06, "text": " So main differences in the translate option"}, {"start": 418.06, "end": 420.96000000000004, "text": " whatever the source language of the audio file is"}, {"start": 420.96000000000004, "end": 422.92, "text": " we'll end up with the English translation."}, {"start": 422.92, "end": 426.90000000000003, "text": " Okay, and then we can do the no timestamps route"}, {"start": 426.9, "end": 429.23999999999995, "text": " or we can do the one with the timestamps"}, {"start": 429.23999999999995, "end": 432.59999999999997, "text": " where you additionally predict timestamp information"}, {"start": 432.59999999999997, "end": 436.15999999999997, "text": " and then you do some transcription inside of that segment"}, {"start": 436.15999999999997, "end": 438.2, "text": " and then rinse and repeat."}, {"start": 438.2, "end": 441.35999999999996, "text": " And finally you have the end of transcription token."}, {"start": 441.35999999999996, "end": 445.79999999999995, "text": " Okay, so all of these give rise to multiple tasks"}, {"start": 445.79999999999995, "end": 448.88, "text": " and we'll see how those play out a bit later"}, {"start": 448.88, "end": 449.91999999999996, "text": " in the code as well."}, {"start": 450.88, "end": 453.76, "text": " Okay, so having explained all of this,"}, {"start": 453.76, "end": 456.15999999999997, "text": " let's dig deeper into the paper."}, {"start": 456.16, "end": 458.66, "text": " Well, let's start in the beginning here."}, {"start": 458.66, "end": 461.08000000000004, "text": " So there are roughly two research directions"}, {"start": 461.08000000000004, "end": 462.44, "text": " that people have been pursuing"}, {"start": 462.44, "end": 464.22, "text": " before a whisper was published."}, {"start": 464.22, "end": 466.84000000000003, "text": " So one big research direction"}, {"start": 466.84000000000003, "end": 469.08000000000004, "text": " was the unsupervised direction."}, {"start": 469.08000000000004, "end": 471.16, "text": " So let me read this one for you."}, {"start": 471.16, "end": 472.96000000000004, "text": " So these pre-trained audio encoders"}, {"start": 472.96000000000004, "end": 475.44000000000005, "text": " learn high quality representations of speech,"}, {"start": 475.44000000000005, "end": 477.52000000000004, "text": " but because they are purely unsupervised"}, {"start": 477.52000000000004, "end": 480.84000000000003, "text": " they lack an equivalently performant decoder"}, {"start": 480.84000000000003, "end": 483.96000000000004, "text": " mapping those representations to usable outputs"}, {"start": 483.96000000000004, "end": 485.84000000000003, "text": " necessitating a fine tuning stage"}, {"start": 485.84, "end": 487.76, "text": " in order to actually perform a task"}, {"start": 487.76, "end": 489.28, "text": " such as speech recognition."}, {"start": 489.28, "end": 492.84, "text": " Okay, so that means on one part of the spectrum"}, {"start": 492.84, "end": 496.2, "text": " we have models that are trained on a huge amount of data."}, {"start": 496.2, "end": 499.32, "text": " So we have, let me denote the data set like this."}, {"start": 499.32, "end": 500.91999999999996, "text": " So we have huge amount of data."}, {"start": 500.91999999999996, "end": 502.23999999999995, "text": " I think they mentioned here,"}, {"start": 502.23999999999995, "end": 504.91999999999996, "text": " like you can see here scaled up to million hours"}, {"start": 504.91999999999996, "end": 505.76, "text": " of training data."}, {"start": 505.76, "end": 507.64, "text": " So that's like, and that's the worst"}, {"start": 508.59999999999997, "end": 510.47999999999996, "text": " visualization of data ever."}, {"start": 511.32, "end": 514.24, "text": " Let me kind of delete it for a second and draw it properly."}, {"start": 514.24, "end": 519.24, "text": " So as I said, we have a data set of 1 million hours"}, {"start": 521.04, "end": 523.04, "text": " of audio here."}, {"start": 524.0, "end": 525.52, "text": " This is a bit better."}, {"start": 525.52, "end": 527.44, "text": " And that's on one part of the spectrum."}, {"start": 527.44, "end": 530.28, "text": " So let me now show you the other part of the spectrum here."}, {"start": 531.84, "end": 535.16, "text": " And again, what I've said here is that they train these"}, {"start": 535.16, "end": 537.48, "text": " in a non-supervised fashion, which means you go"}, {"start": 537.48, "end": 542.04, "text": " from audio to audio representation."}, {"start": 542.04, "end": 544.02, "text": " I guess from male spectrogram to male spectrogram"}, {"start": 544.02, "end": 544.86, "text": " something like that."}, {"start": 544.86, "end": 547.3199999999999, "text": " And then the problem is how do you decode those"}, {"start": 547.3199999999999, "end": 551.0, "text": " representations to actual human language?"}, {"start": 551.0, "end": 552.36, "text": " That's the problem with these models."}, {"start": 552.36, "end": 554.3199999999999, "text": " That's why you have to do the fine tuning part."}, {"start": 554.3199999999999, "end": 555.16, "text": " Okay."}, {"start": 555.16, "end": 556.9, "text": " And then the second direction is this one."}, {"start": 556.9, "end": 559.12, "text": " So speech recognition systems that are pre-trained"}, {"start": 559.12, "end": 562.76, "text": " in a supervised fashion across many data sets domains"}, {"start": 562.76, "end": 565.68, "text": " exhibit higher robustness and generalize much more"}, {"start": 565.68, "end": 568.24, "text": " effectively to held out data sets than models trained"}, {"start": 568.24, "end": 569.5799999999999, "text": " on a single source."}, {"start": 569.5799999999999, "end": 571.36, "text": " Okay, so there's just stating that we wanna have"}, {"start": 571.36, "end": 573.42, "text": " a multilingual multitask setup,"}, {"start": 573.42, "end": 574.88, "text": " which we kind of know already,"}, {"start": 574.88, "end": 577.12, "text": " especially if you have bigger models."}, {"start": 578.12, "end": 580.0799999999999, "text": " However, there is still only a moderate amount"}, {"start": 580.0799999999999, "end": 582.52, "text": " of this data easily available."}, {"start": 582.52, "end": 586.0799999999999, "text": " These authors mix together seven pre-existing data sets"}, {"start": 586.0799999999999, "end": 589.1999999999999, "text": " totaling only 5,000 hours of supervision."}, {"start": 589.1999999999999, "end": 591.9599999999999, "text": " Okay, so that's the other part of the spectrum."}, {"start": 591.9599999999999, "end": 595.3, "text": " We have tiny, tiny, tiny models here."}, {"start": 596.4799999999999, "end": 597.76, "text": " I mean, tiny data sets."}, {"start": 597.76, "end": 600.64, "text": " So this one has like 5K hours compared"}, {"start": 600.64, "end": 602.56, "text": " that to 1 million hours there."}, {"start": 602.56, "end": 603.4, "text": " Okay."}, {"start": 603.4, "end": 605.9599999999999, "text": " And they say this, so whoops,"}, {"start": 605.9599999999999, "end": 609.24, "text": " by relaxing the requirements of gold standard"}, {"start": 609.24, "end": 611.9599999999999, "text": " human validated transcripts,"}, {"start": 611.9599999999999, "end": 615.66, "text": " these authors make use of sophisticated automated pipelines"}, {"start": 615.66, "end": 618.0, "text": " to scale weekly supervised speech recognition"}, {"start": 618.0, "end": 622.4, "text": " to 10,000 and 30,000 hours of moisture training data."}, {"start": 622.4, "end": 624.12, "text": " This trade off between quality and quantity"}, {"start": 624.12, "end": 625.66, "text": " is often the right call."}, {"start": 625.66, "end": 630.66, "text": " Okay, so that means those authors scaled up this"}, {"start": 630.86, "end": 632.8, "text": " by using the weak supervision,"}, {"start": 632.8, "end": 634.64, "text": " but they only went to maybe a bit"}, {"start": 634.64, "end": 637.0799999999999, "text": " like a couple of times bigger data sets."}, {"start": 637.0799999999999, "end": 640.7199999999999, "text": " So now we have like something like 10 to 30K."}, {"start": 640.7199999999999, "end": 643.1999999999999, "text": " And finally, let's get to this paper."}, {"start": 643.1999999999999, "end": 645.4399999999999, "text": " So what these authors have done is,"}, {"start": 645.4399999999999, "end": 647.16, "text": " whoops, what's going on here?"}, {"start": 647.16, "end": 651.0799999999999, "text": " Yet these new data sets are only a few times larger"}, {"start": 651.0799999999999, "end": 653.4399999999999, "text": " than the sum of existing high quality data sets"}, {"start": 653.4399999999999, "end": 656.5999999999999, "text": " and still much smaller than prior unsupervised work."}, {"start": 656.5999999999999, "end": 658.52, "text": " In this work, we closed that gap,"}, {"start": 658.52, "end": 661.4399999999999, "text": " scaling weekly supervised speech recognition,"}, {"start": 661.44, "end": 666.0400000000001, "text": " the next order of magnitude to 680,000 hours"}, {"start": 666.0400000000001, "end": 667.44, "text": " of labeled audio data,"}, {"start": 667.44, "end": 669.7600000000001, "text": " which means they literally close this gap."}, {"start": 669.7600000000001, "end": 673.0, "text": " They have almost as much data as the unsupervised approach."}, {"start": 673.0, "end": 674.84, "text": " Obviously there are the trade-offs of the quality"}, {"start": 674.84, "end": 675.96, "text": " and the quantity."}, {"start": 675.96, "end": 678.6400000000001, "text": " And so here are some of the heuristics that I've used"}, {"start": 678.6400000000001, "end": 681.6800000000001, "text": " to make sure that the quality is as high as possible,"}, {"start": 681.6800000000001, "end": 682.84, "text": " considering that they are using"}, {"start": 682.84, "end": 685.72, "text": " these automated pipelines."}, {"start": 685.72, "end": 688.08, "text": " Okay, so initial inspection showed a large amount"}, {"start": 688.08, "end": 690.6400000000001, "text": " of sub-pair transcripts in the raw data set."}, {"start": 690.64, "end": 692.64, "text": " To address this, we developed several"}, {"start": 692.64, "end": 696.3199999999999, "text": " automated filtering methods to improve transcript quality."}, {"start": 696.3199999999999, "end": 697.68, "text": " So let's see a couple of them."}, {"start": 697.68, "end": 700.56, "text": " In order to avoid learning transcriptase,"}, {"start": 700.56, "end": 702.84, "text": " we developed many heuristics to detect"}, {"start": 702.84, "end": 705.16, "text": " and remove machine-generated transcripts"}, {"start": 705.16, "end": 706.4399999999999, "text": " from the training data set."}, {"start": 706.4399999999999, "end": 709.4399999999999, "text": " And one of those is, if you detect all uppercase"}, {"start": 709.4399999999999, "end": 712.28, "text": " or all lowercase, then it's very unlikely"}, {"start": 712.28, "end": 715.48, "text": " that that's a human-generated transcript,"}, {"start": 715.48, "end": 717.08, "text": " and then you're just gonna remove that pair"}, {"start": 717.08, "end": 718.36, "text": " from your data set."}, {"start": 718.36, "end": 720.36, "text": " A second heuristic they've used is,"}, {"start": 720.36, "end": 722.8000000000001, "text": " ensure that the spoken language matches the language"}, {"start": 722.8000000000001, "end": 726.84, "text": " of the transcript according to some pre-trained CLD2 model,"}, {"start": 726.84, "end": 729.6, "text": " which does the language detection, basically."}, {"start": 729.6, "end": 730.84, "text": " And if the two do not match,"}, {"start": 730.84, "end": 732.96, "text": " we don't include the audio transcript pair"}, {"start": 732.96, "end": 736.2, "text": " as a speech recognition training example in the data set."}, {"start": 736.2, "end": 737.04, "text": " Et cetera, et cetera."}, {"start": 737.04, "end": 740.22, "text": " And then finally, once they had this big data set,"}, {"start": 740.22, "end": 743.0, "text": " they trained the initial versions of Whisper,"}, {"start": 743.0, "end": 745.12, "text": " and then they used the predictions of Whisper"}, {"start": 745.12, "end": 747.0, "text": " to figure out what's wrong with the data."}, {"start": 747.0, "end": 748.88, "text": " So it's kind of very iterative process."}, {"start": 748.88, "end": 752.2, "text": " So they say that here, for an additional filtering pass,"}, {"start": 752.2, "end": 754.4, "text": " after training an initial model,"}, {"start": 754.4, "end": 757.36, "text": " we aggregated information about its error rate"}, {"start": 757.36, "end": 761.4399999999999, "text": " on training data sources and performed manual inspection."}, {"start": 762.96, "end": 766.6, "text": " So I can assume that that was a fairly laborious process,"}, {"start": 766.6, "end": 768.14, "text": " and that's probably one of the reasons"}, {"start": 768.14, "end": 771.48, "text": " that they did not release the actual data set"}, {"start": 771.48, "end": 773.92, "text": " that was used to train Whisper."}, {"start": 773.92, "end": 776.32, "text": " Here's some details around the MEL Spectrogram."}, {"start": 776.32, "end": 777.16, "text": " I'm gonna skip that."}, {"start": 777.16, "end": 779.24, "text": " Maybe we can get back to that later."}, {"start": 779.24, "end": 782.12, "text": " I don't wanna dig into too many details"}, {"start": 782.12, "end": 784.3199999999999, "text": " that are not that vital here."}, {"start": 784.3199999999999, "end": 787.8, "text": " Here are some details worth mentioning, though."}, {"start": 787.8, "end": 789.76, "text": " During early development and evaluation,"}, {"start": 789.76, "end": 791.88, "text": " we observed that Whisper models had a tendency"}, {"start": 791.88, "end": 795.0799999999999, "text": " to transcribe plausible, but almost always incorrect guesses"}, {"start": 795.0799999999999, "end": 796.88, "text": " for the names of the speakers."}, {"start": 796.88, "end": 799.7199999999999, "text": " And the reason being is usually in those transcripts,"}, {"start": 799.7199999999999, "end": 803.8, "text": " you have something like this predicted."}, {"start": 803.8, "end": 807.3199999999999, "text": " So open bracket and then a name of a person,"}, {"start": 807.3199999999999, "end": 808.56, "text": " and then close bracket."}, {"start": 808.56, "end": 810.04, "text": " And then the model learned to,"}, {"start": 810.04, "end": 813.52, "text": " because of the way how models learn, statistical learning,"}, {"start": 813.52, "end": 816.4799999999999, "text": " to just predict these, even though you cannot infer"}, {"start": 816.4799999999999, "end": 819.3199999999999, "text": " what's the name of the person from a 30-second audio clip"}, {"start": 819.3199999999999, "end": 821.24, "text": " that they are using, in most cases,"}, {"start": 821.24, "end": 823.7199999999999, "text": " unless the name was actually referenced."}, {"start": 823.7199999999999, "end": 825.4399999999999, "text": " So they say here, to avoid this,"}, {"start": 825.4399999999999, "end": 828.12, "text": " we fine-tuned Whisper models briefly"}, {"start": 828.12, "end": 829.92, "text": " on the subset of transcripts"}, {"start": 829.92, "end": 832.12, "text": " that do not include speaker annotations,"}, {"start": 832.12, "end": 833.96, "text": " which removes this behavior."}, {"start": 833.96, "end": 837.12, "text": " Okay, so one more thing,"}, {"start": 837.12, "end": 839.36, "text": " and then we're gonna get to the robustness part."}, {"start": 839.36, "end": 841.96, "text": " They had problems with this WER metric,"}, {"start": 841.96, "end": 844.5600000000001, "text": " so that's the word error rate metric."}, {"start": 844.5600000000001, "end": 846.12, "text": " And it's a similar problem"}, {"start": 846.12, "end": 848.88, "text": " as to what people are encounter with"}, {"start": 848.88, "end": 851.2, "text": " when they deal with machine translation, for example,"}, {"start": 851.2, "end": 852.8, "text": " with the blue metric, et cetera."}, {"start": 852.8, "end": 854.48, "text": " So let me read this part."}, {"start": 854.48, "end": 859.16, "text": " So a WER, which is based on string edit distance,"}, {"start": 859.16, "end": 862.28, "text": " penalizes all differences between the model's output"}, {"start": 862.28, "end": 863.92, "text": " and the reference transcript,"}, {"start": 863.92, "end": 866.8399999999999, "text": " including innocuous differences in transcript style,"}, {"start": 866.8399999999999, "end": 867.66, "text": " which kind of sucks."}, {"start": 867.66, "end": 869.0799999999999, "text": " We wanna focus on the semantics"}, {"start": 869.0799999999999, "end": 871.24, "text": " and understand whether the,"}, {"start": 871.24, "end": 873.7199999999999, "text": " well, in the case of audio transcription,"}, {"start": 873.7199999999999, "end": 875.7199999999999, "text": " every single word matters, that's true,"}, {"start": 875.7199999999999, "end": 880.24, "text": " but the actual maybe punctuations or style"}, {"start": 880.24, "end": 882.1999999999999, "text": " that's superficial and not changing anything,"}, {"start": 882.1999999999999, "end": 884.48, "text": " you kind of want to ignore that, right?"}, {"start": 884.48, "end": 885.56, "text": " So they say here,"}, {"start": 885.56, "end": 888.0799999999999, "text": " while this poses a problem for all transcribers,"}, {"start": 888.08, "end": 891.44, "text": " it is particularly acute for zero-shot models like Whisper,"}, {"start": 891.44, "end": 893.76, "text": " which do not observe any examples"}, {"start": 893.76, "end": 896.6, "text": " of specific dataset transcript formats."}, {"start": 896.6, "end": 898.6800000000001, "text": " So that means those other baselines,"}, {"start": 898.6800000000001, "end": 902.5, "text": " because they're actually trained on that particular dataset,"}, {"start": 902.5, "end": 905.88, "text": " and then basically they do the evaluate"}, {"start": 905.88, "end": 907.32, "text": " on the validation portion,"}, {"start": 907.32, "end": 909.36, "text": " but the distribution is very similar,"}, {"start": 909.36, "end": 913.0, "text": " they kind of can learn and overfit the particular style,"}, {"start": 913.0, "end": 915.9200000000001, "text": " and thus they have an advantage"}, {"start": 915.92, "end": 919.1999999999999, "text": " compared to zero-shot Whisper model."}, {"start": 919.1999999999999, "end": 920.04, "text": " So because of that,"}, {"start": 920.04, "end": 923.5999999999999, "text": " they developed various laborious normalizers,"}, {"start": 923.5999999999999, "end": 924.4399999999999, "text": " so they say here,"}, {"start": 924.4399999999999, "end": 926.04, "text": " our text normalizer was developed"}, {"start": 926.04, "end": 928.0799999999999, "text": " through iterative manual inspection"}, {"start": 928.0799999999999, "end": 929.74, "text": " to identify common patterns"}, {"start": 929.74, "end": 933.1999999999999, "text": " where naive WER penalized Whisper models"}, {"start": 933.1999999999999, "end": 935.36, "text": " for innocuous difference."}, {"start": 935.36, "end": 937.56, "text": " Because I will not cover this in the code bit later,"}, {"start": 937.56, "end": 941.28, "text": " so I'm gonna just quickly show you how those look like."}, {"start": 941.28, "end": 943.92, "text": " Let me just try and find those normalizers,"}, {"start": 943.92, "end": 945.5999999999999, "text": " so here they are,"}, {"start": 945.6, "end": 947.48, "text": " and if I open up English normalizers,"}, {"start": 947.48, "end": 951.52, "text": " you can see how laborious this must have been,"}, {"start": 951.52, "end": 953.5600000000001, "text": " like there is just a lot of rules"}, {"start": 953.5600000000001, "end": 956.16, "text": " that they had to do to make sure,"}, {"start": 956.16, "end": 958.52, "text": " yeah, you can see it's kind of messy."}, {"start": 958.52, "end": 961.44, "text": " So let me go back to the paper,"}, {"start": 961.44, "end": 963.0400000000001, "text": " and let's continue here."}, {"start": 963.0400000000001, "end": 967.44, "text": " So they introduced this concept of effective robustness,"}, {"start": 967.44, "end": 970.3000000000001, "text": " and what it means is it measures the difference"}, {"start": 970.3000000000001, "end": 973.86, "text": " in expected performance between a reference dataset,"}, {"start": 973.86, "end": 975.6800000000001, "text": " which is usually in distribution,"}, {"start": 975.6800000000001, "end": 978.5, "text": " and one or more out of distribution datasets."}, {"start": 978.5, "end": 980.0600000000001, "text": " So obviously, we wanna have this difference"}, {"start": 980.0600000000001, "end": 981.52, "text": " to be as small as possible,"}, {"start": 981.52, "end": 983.34, "text": " which means that you are generalizing"}, {"start": 983.34, "end": 984.88, "text": " to out of distribution datasets,"}, {"start": 984.88, "end": 986.8000000000001, "text": " which is a desirable behavior, right?"}, {"start": 986.8000000000001, "end": 989.02, "text": " We want to have general models."}, {"start": 990.0600000000001, "end": 992.24, "text": " So a model with high effective robustness"}, {"start": 992.24, "end": 995.4, "text": " does better than expected on out of distribution datasets"}, {"start": 995.4, "end": 997.96, "text": " as a function of its performance on the reference dataset,"}, {"start": 997.96, "end": 1000.6, "text": " and approaches the ideal of equal performance"}, {"start": 1000.6, "end": 1003.24, "text": " on all datasets."}, {"start": 1003.24, "end": 1005.6800000000001, "text": " So here is the actual diagram."}, {"start": 1005.6800000000001, "end": 1009.0, "text": " So here we can see in an ideal world,"}, {"start": 1009.0, "end": 1011.0600000000001, "text": " we'd have this line here."}, {"start": 1011.0600000000001, "end": 1016.0600000000001, "text": " Basically, the metric on the LibreSpeech dev clean"}, {"start": 1016.12, "end": 1020.16, "text": " should be whatever the performance of your model there is,"}, {"start": 1020.16, "end": 1024.56, "text": " you wanna have the same error rate on the other datasets."}, {"start": 1024.56, "end": 1029.4, "text": " So here is the average WER on these other datasets."}, {"start": 1029.4, "end": 1030.76, "text": " So you can see here,"}, {"start": 1030.76, "end": 1033.1200000000001, "text": " some of these models that have much better performance"}, {"start": 1033.12, "end": 1036.6799999999998, "text": " compared to Whisper on the LibreSpeech."}, {"start": 1036.6799999999998, "end": 1039.1999999999998, "text": " So you can see this point here, for example, whoops,"}, {"start": 1039.1999999999998, "end": 1041.6, "text": " this point here, I'm terrible at drawing today."}, {"start": 1042.4599999999998, "end": 1045.12, "text": " So it has much lower error rate here"}, {"start": 1045.12, "end": 1046.84, "text": " compared to let's say this point,"}, {"start": 1046.84, "end": 1050.04, "text": " but then you can see how poor the performance actually is."}, {"start": 1050.04, "end": 1053.04, "text": " The error rate is huge for these data points"}, {"start": 1053.04, "end": 1055.3999999999999, "text": " of these LibreSpeech models."}, {"start": 1055.3999999999999, "end": 1056.8, "text": " So you can see all in all,"}, {"start": 1056.8, "end": 1061.8, "text": " when you fit like a line across these data points,"}, {"start": 1061.8, "end": 1065.12, "text": " basically you see that it's much more robust,"}, {"start": 1065.12, "end": 1066.28, "text": " Whisper is much more robust"}, {"start": 1066.28, "end": 1070.2, "text": " compared to the other supervised LibreSpeech models."}, {"start": 1070.2, "end": 1073.96, "text": " Even though it's not actually soda on the LibreSpeech,"}, {"start": 1073.96, "end": 1076.8799999999999, "text": " as you can see here, but that's, I guess, less important."}, {"start": 1076.8799999999999, "end": 1080.54, "text": " We wanna create robust general models."}, {"start": 1080.54, "end": 1084.52, "text": " Okay, so they mentioned that even the smallest"}, {"start": 1084.52, "end": 1085.96, "text": " zero-shot Whisper model,"}, {"start": 1085.96, "end": 1088.7, "text": " which only has 39 million parameters"}, {"start": 1088.7, "end": 1092.24, "text": " and a 6.7 WER on LibreSpeech test clean"}, {"start": 1092.24, "end": 1093.48, "text": " is roughly competitive"}, {"start": 1093.48, "end": 1095.6000000000001, "text": " with the best supervised LibreSpeech model"}, {"start": 1095.6000000000001, "end": 1097.66, "text": " when evaluated on other data sets."}, {"start": 1097.66, "end": 1100.3600000000001, "text": " So I guess that's like,"}, {"start": 1100.3600000000001, "end": 1103.3600000000001, "text": " literally this model here probably is competitive,"}, {"start": 1103.3600000000001, "end": 1107.16, "text": " as you can see here, performance wise, the Y point here."}, {"start": 1107.16, "end": 1109.5800000000002, "text": " So the average performance on the other data sets"}, {"start": 1109.5800000000002, "end": 1113.2, "text": " is close to these supervised LibreSpeech models."}, {"start": 1113.2, "end": 1114.52, "text": " They have much better performance"}, {"start": 1114.52, "end": 1117.8, "text": " on the actual LibreSpeech dev clean test."}, {"start": 1117.8, "end": 1120.0, "text": " Data set, sorry."}, {"start": 1120.0, "end": 1123.6, "text": " Okay, so here are a couple more interesting observations."}, {"start": 1123.6, "end": 1126.3999999999999, "text": " Probably not that surprising at this point of time."}, {"start": 1126.3999999999999, "end": 1127.9199999999998, "text": " And that's that with more data,"}, {"start": 1127.9199999999998, "end": 1131.06, "text": " you can see on the x-axis hours of transcribed audio,"}, {"start": 1131.06, "end": 1132.24, "text": " you can see that these languages"}, {"start": 1132.24, "end": 1135.48, "text": " that have more data available in the data set"}, {"start": 1135.48, "end": 1137.24, "text": " have much lower error rate."}, {"start": 1137.24, "end": 1138.96, "text": " And you can see it's super predictive."}, {"start": 1138.96, "end": 1143.96, "text": " Like the correlation coefficient here is 0.84."}, {"start": 1143.96, "end": 1146.8, "text": " And they mentioned somewhere here,"}, {"start": 1146.8, "end": 1149.12, "text": " a couple of these outliers, you can see here,"}, {"start": 1150.32, "end": 1153.56, "text": " ZH, which I think stands for Mandarin Chinese or something,"}, {"start": 1153.56, "end": 1156.56, "text": " Korean are kind of outliers here,"}, {"start": 1156.56, "end": 1160.24, "text": " given that even though they have this much data,"}, {"start": 1160.24, "end": 1162.1599999999999, "text": " the error rate is somewhat higher."}, {"start": 1162.1599999999999, "end": 1165.68, "text": " And they kind of answer that,"}, {"start": 1165.68, "end": 1168.04, "text": " hypothesize why that might be the case here."}, {"start": 1168.04, "end": 1170.12, "text": " So checking the regression coefficient"}, {"start": 1170.12, "end": 1172.2, "text": " for a linear fit to these log log values"}, {"start": 1172.2, "end": 1174.96, "text": " results in an estimate that the error rate"}, {"start": 1174.96, "end": 1179.3600000000001, "text": " has for every 16x increase in training data."}, {"start": 1179.3600000000001, "end": 1181.6000000000001, "text": " So that's the general trend we've seen."}, {"start": 1181.6000000000001, "end": 1184.24, "text": " And then they say that the many of the largest outliers"}, {"start": 1184.24, "end": 1186.32, "text": " in terms of worst than expected performance,"}, {"start": 1186.32, "end": 1187.48, "text": " according to this trend,"}, {"start": 1187.48, "end": 1189.56, "text": " are languages that have unique scripts"}, {"start": 1189.56, "end": 1191.1000000000001, "text": " and are more distantly related"}, {"start": 1191.1000000000001, "end": 1192.66, "text": " to the Indo-European languages,"}, {"start": 1192.66, "end": 1195.6200000000001, "text": " making up the majority of the training data set,"}, {"start": 1195.6200000000001, "end": 1198.6000000000001, "text": " such as Hebrew, Telugu, Chinese, and Korean."}, {"start": 1198.6000000000001, "end": 1201.52, "text": " So I did point out to these two languages here."}, {"start": 1201.52, "end": 1204.0, "text": " So they speculate that these differences could be due,"}, {"start": 1204.0, "end": 1205.48, "text": " I mean, it's not a speculation,"}, {"start": 1205.48, "end": 1207.32, "text": " it's a valid hypothesis, I guess,"}, {"start": 1207.32, "end": 1209.28, "text": " could be due to a lack of transfer,"}, {"start": 1209.28, "end": 1211.68, "text": " due to linguistic distance,"}, {"start": 1211.68, "end": 1213.52, "text": " or simply the problem with tokenizer"}, {"start": 1213.52, "end": 1215.0, "text": " being a poor match for these languages"}, {"start": 1215.0, "end": 1216.8, "text": " or variations in data quality."}, {"start": 1216.8, "end": 1219.66, "text": " So all of those factors could be confounding."}, {"start": 1221.48, "end": 1225.4, "text": " Okay, so they have additional interesting diagram here"}, {"start": 1225.4, "end": 1228.68, "text": " showing that the translation performance"}, {"start": 1228.68, "end": 1232.0, "text": " is not as predictive,"}, {"start": 1232.0, "end": 1234.42, "text": " given the hours of translated audio,"}, {"start": 1234.42, "end": 1237.96, "text": " it's not that easy to infer what will be the blue metric."}, {"start": 1237.96, "end": 1242.96, "text": " And then they hypothesize here, somewhere below here."}, {"start": 1243.32, "end": 1244.5, "text": " So they say here,"}, {"start": 1244.5, "end": 1246.76, "text": " so we suspect this is partly caused"}, {"start": 1246.76, "end": 1248.64, "text": " by the noisier training data"}, {"start": 1248.64, "end": 1251.9, "text": " due to errors in audio language identification."}, {"start": 1251.9, "end": 1256.16, "text": " So as an example, Welsh CY is an outlier"}, {"start": 1256.16, "end": 1258.4, "text": " with much worse than expected performance"}, {"start": 1258.4, "end": 1259.8, "text": " at only nine blue,"}, {"start": 1259.8, "end": 1263.8799999999999, "text": " despite supposedly having 9,000 hours of translation data."}, {"start": 1263.8799999999999, "end": 1265.08, "text": " But as it turns out,"}, {"start": 1265.08, "end": 1268.08, "text": " and let me just show you on the diagram here,"}, {"start": 1268.08, "end": 1270.04, "text": " where the CY lies,"}, {"start": 1270.04, "end": 1272.8799999999999, "text": " so you can see here, it literally has 9,000,"}, {"start": 1272.8799999999999, "end": 1275.24, "text": " but it has around nine blue,"}, {"start": 1275.24, "end": 1279.6, "text": " which is like super low compared to what this trend here"}, {"start": 1279.6, "end": 1280.8999999999999, "text": " would suggest."}, {"start": 1280.8999999999999, "end": 1282.3999999999999, "text": " But then they say here,"}, {"start": 1282.3999999999999, "end": 1285.6399999999999, "text": " inspection shows the majority of supposedly Welsh"}, {"start": 1285.6399999999999, "end": 1288.2, "text": " translation data is actually English audio"}, {"start": 1288.2, "end": 1290.0, "text": " with English captions,"}, {"start": 1290.0, "end": 1293.28, "text": " where the English audio was misclassified as Welsh"}, {"start": 1293.28, "end": 1295.56, "text": " by the language identification system,"}, {"start": 1295.56, "end": 1299.0800000000002, "text": " resulting in it being included as translation training data"}, {"start": 1299.0800000000002, "end": 1300.28, "text": " rather than transcription data"}, {"start": 1300.28, "end": 1302.04, "text": " according to our data set creation rules."}, {"start": 1302.04, "end": 1302.88, "text": " So that's a problem"}, {"start": 1302.88, "end": 1305.56, "text": " because that means you were using the Welsh token,"}, {"start": 1305.56, "end": 1308.56, "text": " but actually you had English audio and English transcription,"}, {"start": 1308.56, "end": 1309.76, "text": " which messes up the results."}, {"start": 1309.76, "end": 1313.0, "text": " So I guess that's suggestive"}, {"start": 1313.0, "end": 1316.8400000000001, "text": " of not taking this plot here too seriously"}, {"start": 1316.84, "end": 1319.12, "text": " and just like,"}, {"start": 1319.12, "end": 1321.84, "text": " I guess the general point of the paper is,"}, {"start": 1321.84, "end": 1325.08, "text": " given enough data that's high quality,"}, {"start": 1325.08, "end": 1327.08, "text": " we can achieve human performance"}, {"start": 1327.08, "end": 1328.8799999999999, "text": " on speech recognition pretty much."}, {"start": 1330.3799999999999, "end": 1331.9199999999998, "text": " There is some situation phenomenon,"}, {"start": 1331.9199999999998, "end": 1333.9599999999998, "text": " but I'm gonna show you that in a couple of minutes."}, {"start": 1333.9599999999998, "end": 1335.52, "text": " Okay, so let's continue here."}, {"start": 1335.52, "end": 1337.54, "text": " Let's see a couple more things here."}, {"start": 1339.02, "end": 1340.72, "text": " Things worth mentioning is this one."}, {"start": 1340.72, "end": 1342.9599999999998, "text": " So they do the following thing."}, {"start": 1342.9599999999998, "end": 1346.24, "text": " They add some noise on top of the input audio signal,"}, {"start": 1346.24, "end": 1347.84, "text": " such as for example, in the left diagram,"}, {"start": 1347.84, "end": 1351.8, "text": " we see white noise and then on the right, the pub noise."}, {"start": 1351.8, "end": 1353.08, "text": " So that's kind of background noise"}, {"start": 1353.08, "end": 1355.04, "text": " from a restaurant or a pub or something."}, {"start": 1355.04, "end": 1359.08, "text": " And you can see that the whisper, which is this red star,"}, {"start": 1361.32, "end": 1363.72, "text": " you can see it definitely does not have the lowest error rate"}, {"start": 1363.72, "end": 1365.68, "text": " on the LibreSpeech test screen,"}, {"start": 1365.68, "end": 1368.96, "text": " but then as the noise levels keep increasing,"}, {"start": 1368.96, "end": 1372.64, "text": " it all of a sudden becomes one of the best models."}, {"start": 1372.64, "end": 1377.64, "text": " And especially here on pub noise, you can see that"}, {"start": 1377.64, "end": 1379.5600000000002, "text": " the model outperforms all of the other ones."}, {"start": 1379.5600000000002, "end": 1381.3200000000002, "text": " So it has lower error rates."}, {"start": 1381.3200000000002, "end": 1383.3200000000002, "text": " So that means it's more robust to noise,"}, {"start": 1383.3200000000002, "end": 1385.5200000000002, "text": " and especially the natural noise"}, {"start": 1385.5200000000002, "end": 1388.16, "text": " compared to this synthetic white noise."}, {"start": 1390.4, "end": 1392.3600000000001, "text": " Okay, let's continue here."}, {"start": 1393.3200000000002, "end": 1398.0, "text": " They also compare whisper to other baselines"}, {"start": 1398.0, "end": 1399.72, "text": " in this long form transcription,"}, {"start": 1399.72, "end": 1403.92, "text": " where because the model can only see 30 second audio inputs,"}, {"start": 1403.92, "end": 1406.52, "text": " they have to do sliding window approach."}, {"start": 1406.52, "end": 1410.08, "text": " And they show here that they are on pair"}, {"start": 1410.08, "end": 1412.4, "text": " and even better compared to many of these"}, {"start": 1412.4, "end": 1413.68, "text": " even commercial services."}, {"start": 1413.68, "end": 1418.68, "text": " You can see the blue candle bars are for most of these"}, {"start": 1419.1200000000001, "end": 1424.1200000000001, "text": " long form transcription datasets lower than the baselines."}, {"start": 1424.24, "end": 1426.2, "text": " Given there are some exceptions, obviously,"}, {"start": 1426.2, "end": 1428.64, "text": " like here you can see that this company B"}, {"start": 1428.64, "end": 1431.6000000000001, "text": " is outperforming them, but yeah, that's kind of impressive."}, {"start": 1433.5600000000002, "end": 1434.5200000000002, "text": " This is also cool."}, {"start": 1434.5200000000002, "end": 1439.4, "text": " They show how they compare to human transcription."}, {"start": 1439.4, "end": 1441.68, "text": " So you can see how whisper is very close to other,"}, {"start": 1441.68, "end": 1443.68, "text": " even human transcription services."}, {"start": 1443.68, "end": 1448.68, "text": " So here they took actual, well, humans to transcribe some,"}, {"start": 1450.6000000000001, "end": 1453.2800000000002, "text": " I think like they had 25 recordings of text or something,"}, {"start": 1453.2800000000002, "end": 1454.3600000000001, "text": " and then they compare them,"}, {"start": 1454.3600000000001, "end": 1456.0400000000002, "text": " and you can see it's on pair pretty much."}, {"start": 1456.04, "end": 1459.56, "text": " But then there is this computer assisted human transcription"}, {"start": 1459.56, "end": 1462.1599999999999, "text": " which is the best out of all of these approaches."}, {"start": 1462.1599999999999, "end": 1465.08, "text": " So that again means that this combination of humans"}, {"start": 1465.08, "end": 1466.8799999999999, "text": " and machines is still the best way"}, {"start": 1466.8799999999999, "end": 1469.52, "text": " to go about many, many problems."}, {"start": 1469.52, "end": 1471.6399999999999, "text": " Let's see a couple of more results here."}, {"start": 1472.92, "end": 1474.72, "text": " Obviously model parameter,"}, {"start": 1474.72, "end": 1476.36, "text": " this is again not that surprising,"}, {"start": 1476.36, "end": 1477.44, "text": " although it's very noisy."}, {"start": 1477.44, "end": 1479.92, "text": " Let me show you what's going on here."}, {"start": 1479.92, "end": 1481.6399999999999, "text": " So we have English speech recognition"}, {"start": 1481.6399999999999, "end": 1483.44, "text": " on this leftmost plot."}, {"start": 1483.44, "end": 1486.16, "text": " You can see that the model parameters as it's scrolling,"}, {"start": 1486.16, "end": 1489.04, "text": " we can see that the error rate is obviously falling down,"}, {"start": 1489.04, "end": 1491.1200000000001, "text": " but there is some saturation going on."}, {"start": 1491.1200000000001, "end": 1492.48, "text": " And there is also a lot of variance."}, {"start": 1492.48, "end": 1495.8400000000001, "text": " If you take a look at these lighter lines,"}, {"start": 1495.8400000000001, "end": 1498.8400000000001, "text": " the dark blue one is just the average of those."}, {"start": 1498.8400000000001, "end": 1500.96, "text": " So that kind of means the performance varies a lot"}, {"start": 1500.96, "end": 1503.3600000000001, "text": " across these 12 data sets."}, {"start": 1503.3600000000001, "end": 1504.8, "text": " And we can also see saturation."}, {"start": 1504.8, "end": 1509.8, "text": " I guess that's the main takeaway here."}, {"start": 1509.8, "end": 1513.44, "text": " We see saturation on English in particular."}, {"start": 1513.44, "end": 1515.6, "text": " And then on multilingual speech recognition,"}, {"start": 1515.6, "end": 1518.8799999999999, "text": " we can also see like a trend, downward trend,"}, {"start": 1518.8799999999999, "end": 1520.32, "text": " general downward trend."}, {"start": 1520.32, "end": 1521.9199999999998, "text": " And again, a lot of variance"}, {"start": 1521.9199999999998, "end": 1524.6399999999999, "text": " if you take a look at some of these light blue curves."}, {"start": 1524.6399999999999, "end": 1526.8799999999999, "text": " Considering that the models are approaching,"}, {"start": 1526.8799999999999, "end": 1529.24, "text": " that we are doing a supervised learning approach here"}, {"start": 1529.24, "end": 1532.1599999999999, "text": " and that the models are, so where was the diagram here?"}, {"start": 1532.1599999999999, "end": 1537.08, "text": " So that the models are on pair with this human services,"}, {"start": 1537.08, "end": 1540.1599999999999, "text": " with human transcription that might explain it."}, {"start": 1540.1599999999999, "end": 1542.04, "text": " And the final results I want to show you"}, {"start": 1542.04, "end": 1544.32, "text": " are these ones here."}, {"start": 1544.32, "end": 1548.04, "text": " Basically, you can see two lines here, two fits,"}, {"start": 1548.04, "end": 1551.6, "text": " and one is the English only training data set."}, {"start": 1551.6, "end": 1553.9199999999998, "text": " And the second one is multilingual and multitask."}, {"start": 1553.9199999999998, "end": 1557.1599999999999, "text": " And if you're in this regime where the flops are lower,"}, {"start": 1557.1599999999999, "end": 1559.24, "text": " that means you're training smaller models."}, {"start": 1559.24, "end": 1563.8799999999999, "text": " And you can see that with smaller capacity,"}, {"start": 1563.8799999999999, "end": 1566.6399999999999, "text": " you're suffering the, so you wanna have,"}, {"start": 1566.64, "end": 1568.3600000000001, "text": " this is the error rates, you wanna be lower."}, {"start": 1568.3600000000001, "end": 1570.88, "text": " So you can see the English only version is better."}, {"start": 1570.88, "end": 1574.2800000000002, "text": " But the multilingual multitask models scale better."}, {"start": 1574.2800000000002, "end": 1577.5600000000002, "text": " So that means when you go into the bigger model,"}, {"start": 1577.5600000000002, "end": 1580.3200000000002, "text": " like bigger scales, you can see that the error rate"}, {"start": 1580.3200000000002, "end": 1583.3600000000001, "text": " is actually lower for the multilingual multitask,"}, {"start": 1583.3600000000001, "end": 1587.6000000000001, "text": " which is not that surprising, I guess, and encouraging,"}, {"start": 1587.6000000000001, "end": 1589.72, "text": " given that we can just scale it even more"}, {"start": 1589.72, "end": 1591.0, "text": " and have higher quality data"}, {"start": 1591.0, "end": 1593.5600000000002, "text": " and we can achieve even better results."}, {"start": 1593.5600000000002, "end": 1595.48, "text": " These are some candle bars about the talk,"}, {"start": 1595.48, "end": 1597.76, "text": " the actual normalizer, I'm gonna skip that."}, {"start": 1597.76, "end": 1599.84, "text": " And finally, it's important to realize"}, {"start": 1599.84, "end": 1601.56, "text": " that there are still a lot of heuristics"}, {"start": 1601.56, "end": 1605.2, "text": " that they had to use to make this decoding reliable."}, {"start": 1605.2, "end": 1607.0, "text": " So we have developed a set of heuristics"}, {"start": 1607.0, "end": 1608.64, "text": " that help avoid failure cases"}, {"start": 1608.64, "end": 1611.4, "text": " of long form transcription in particular."}, {"start": 1611.4, "end": 1614.48, "text": " So they say here, first we use beam search with five beams"}, {"start": 1614.48, "end": 1617.6, "text": " using the log probabilities as the score function"}, {"start": 1617.6, "end": 1619.16, "text": " to reduce repetition looping,"}, {"start": 1619.16, "end": 1621.88, "text": " which happens more frequently in greedy decoding."}, {"start": 1621.88, "end": 1623.4, "text": " And that's a common problem you have"}, {"start": 1623.4, "end": 1625.44, "text": " with greedy decoding in general,"}, {"start": 1625.44, "end": 1627.24, "text": " like you have the repetition problem."}, {"start": 1627.24, "end": 1630.0, "text": " And then they say we start with the temperature zero,"}, {"start": 1630.0, "end": 1633.44, "text": " I always selecting the tokens with the highest probability,"}, {"start": 1633.44, "end": 1634.96, "text": " which is basically greedy,"}, {"start": 1634.96, "end": 1639.96, "text": " and then increase the temperature by zero two up to 1.0,"}, {"start": 1640.0800000000002, "end": 1641.88, "text": " when either the average log probability"}, {"start": 1641.88, "end": 1644.8000000000002, "text": " over the generated tokens is lower than minus one,"}, {"start": 1644.8000000000002, "end": 1648.52, "text": " or the generated text has a GZIP compression rate"}, {"start": 1648.52, "end": 1651.0800000000002, "text": " higher than 2.4, which would probably mean"}, {"start": 1651.0800000000002, "end": 1652.72, "text": " that if it's higher than that,"}, {"start": 1652.72, "end": 1654.76, "text": " that means that the text is too random"}, {"start": 1654.76, "end": 1656.08, "text": " and too hard to compress,"}, {"start": 1656.08, "end": 1657.8, "text": " which means it's probably gibberish."}, {"start": 1657.8, "end": 1662.04, "text": " I guess that's the rough reasoning behind the heuristic."}, {"start": 1662.04, "end": 1663.44, "text": " And there is a lot more details here,"}, {"start": 1663.44, "end": 1665.76, "text": " I'm just gonna skip that for now."}, {"start": 1665.76, "end": 1668.08, "text": " And you can see here, like in a tabular form,"}, {"start": 1668.08, "end": 1669.92, "text": " the same thing, adding beam search,"}, {"start": 1669.92, "end": 1670.96, "text": " adding temperature fallback,"}, {"start": 1670.96, "end": 1672.3600000000001, "text": " adding previous text conditioning,"}, {"start": 1672.3600000000001, "end": 1674.16, "text": " adding this voice activity detection,"}, {"start": 1674.16, "end": 1675.8, "text": " initial timestamp constraints,"}, {"start": 1675.8, "end": 1678.64, "text": " all of those help improve the average performance"}, {"start": 1678.64, "end": 1680.88, "text": " across different transcription,"}, {"start": 1680.88, "end": 1682.72, "text": " long form transcription data sets here."}, {"start": 1682.72, "end": 1686.6000000000001, "text": " So I guess the main takeaway from all of this is,"}, {"start": 1686.6000000000001, "end": 1690.3200000000002, "text": " if you take a look at the diagrams here,"}, {"start": 1690.3200000000002, "end": 1692.48, "text": " this plot is kind of suggestive that,"}, {"start": 1692.48, "end": 1697.48, "text": " given higher quality and abundance of audio files,"}, {"start": 1698.2, "end": 1702.88, "text": " of audio data for a particular language of interest,"}, {"start": 1702.88, "end": 1705.1200000000001, "text": " and even off the shelf model,"}, {"start": 1705.1200000000001, "end": 1706.92, "text": " you can achieve remarkable performance."}, {"start": 1706.92, "end": 1711.3600000000001, "text": " And we also wanna do the multi-lingual multitask setup."}, {"start": 1711.3600000000001, "end": 1714.52, "text": " Okay, guys, having seen the main ideas from the paper,"}, {"start": 1714.52, "end": 1716.0800000000002, "text": " let's now switch to the code,"}, {"start": 1716.0800000000002, "end": 1717.24, "text": " and let me walk you through"}, {"start": 1717.24, "end": 1719.2, "text": " and try and show you the correspondence"}, {"start": 1719.2, "end": 1721.04, "text": " between the code and the diagram."}, {"start": 1721.04, "end": 1723.6000000000001, "text": " You see here, I'm gonna see in a bit more detail"}, {"start": 1723.6000000000001, "end": 1727.3600000000001, "text": " how this special token pipeline looks like."}, {"start": 1727.3600000000001, "end": 1731.2, "text": " So I already went ahead and cloned the repo,"}, {"start": 1731.2, "end": 1732.6000000000001, "text": " created a condo environment,"}, {"start": 1732.6000000000001, "end": 1734.68, "text": " so all of that should be fairly trivial."}, {"start": 1734.68, "end": 1738.96, "text": " Everything is written down in the README quite nicely."}, {"start": 1738.96, "end": 1743.1200000000001, "text": " So yeah, let me just show you the types of models they have."}, {"start": 1743.1200000000001, "end": 1744.96, "text": " So they support a couple of models."}, {"start": 1744.96, "end": 1747.88, "text": " You can see here, tiny, base, small, medium, and large."}, {"start": 1747.88, "end": 1749.64, "text": " You can also see the.end versions,"}, {"start": 1749.64, "end": 1751.0, "text": " which are English only,"}, {"start": 1751.0, "end": 1753.76, "text": " and these ones with the other suffix"}, {"start": 1753.76, "end": 1758.3600000000001, "text": " are basically multi-lingual multitask models."}, {"start": 1758.3600000000001, "end": 1763.3600000000001, "text": " Okay, so you can specify multiple audio files here."}, {"start": 1763.36, "end": 1766.0, "text": " We're using small model by default."}, {"start": 1766.9199999999998, "end": 1768.28, "text": " Default task is transcribe."}, {"start": 1768.28, "end": 1770.8799999999999, "text": " You can also choose translation."}, {"start": 1770.8799999999999, "end": 1772.12, "text": " Language is set to none,"}, {"start": 1772.12, "end": 1775.56, "text": " which means we'll infer automatically what the language is,"}, {"start": 1775.56, "end": 1777.0, "text": " so that's gonna be interesting."}, {"start": 1777.0, "end": 1779.6399999999999, "text": " And then bunch of parameters around the coding"}, {"start": 1779.6399999999999, "end": 1782.6399999999999, "text": " and these thresholds that help robustify"}, {"start": 1782.6399999999999, "end": 1783.84, "text": " the decoding pipeline."}, {"start": 1783.84, "end": 1785.3999999999999, "text": " Now they just pop some arguments."}, {"start": 1785.3999999999999, "end": 1786.84, "text": " I'm gonna skip all of these parts"}, {"start": 1786.84, "end": 1789.4399999999998, "text": " that are not relevant and crucial"}, {"start": 1789.4399999999998, "end": 1791.3999999999999, "text": " to the understanding of the model."}, {"start": 1791.4, "end": 1794.1200000000001, "text": " So this is like temperature goes from zero to one"}, {"start": 1794.1200000000001, "end": 1795.5600000000002, "text": " with increments of 0.2."}, {"start": 1795.5600000000002, "end": 1797.44, "text": " So that's something they said in the model"}, {"start": 1797.44, "end": 1799.92, "text": " they used to robustify the decoding,"}, {"start": 1799.92, "end": 1802.8400000000001, "text": " if that's even a word, and load model."}, {"start": 1802.8400000000001, "end": 1805.5600000000002, "text": " So let's kinda go here"}, {"start": 1805.5600000000002, "end": 1809.0800000000002, "text": " and let me jump all the way to here."}, {"start": 1809.0800000000002, "end": 1812.6000000000001, "text": " So I'm gonna disable breakpoints, put a breakpoint here,"}, {"start": 1812.6000000000001, "end": 1814.2, "text": " and let's just go to whisper."}, {"start": 1815.24, "end": 1817.92, "text": " Okay, so basically in the background,"}, {"start": 1817.92, "end": 1819.8000000000002, "text": " we download the model."}, {"start": 1819.8, "end": 1821.8799999999999, "text": " Actually, I already downloaded beforehand,"}, {"start": 1821.8799999999999, "end": 1824.56, "text": " so it just kinda loads the weights."}, {"start": 1824.56, "end": 1826.9199999999998, "text": " So here is how the model looks like."}, {"start": 1826.9199999999998, "end": 1831.6399999999999, "text": " Let me maybe now enable the breakpoints and enter here."}, {"start": 1831.6399999999999, "end": 1834.6399999999999, "text": " So we have the autoencoder and we have the text decoder."}, {"start": 1834.6399999999999, "end": 1837.44, "text": " So those are the two stacks we saw previously"}, {"start": 1837.44, "end": 1838.68, "text": " in the diagram."}, {"start": 1838.68, "end": 1843.68, "text": " So let me now enter the, let's enter the encoder"}, {"start": 1844.0, "end": 1845.1599999999999, "text": " and let's see how it looks like."}, {"start": 1845.1599999999999, "end": 1848.9199999999998, "text": " So we can see here, we have two comp1d layers."}, {"start": 1848.92, "end": 1851.3600000000001, "text": " We have the positional embeddings which are not learned,"}, {"start": 1851.3600000000001, "end": 1852.64, "text": " so they're sinusoids."}, {"start": 1852.64, "end": 1855.28, "text": " And let me just put the code and the diagram side by side,"}, {"start": 1855.28, "end": 1856.2, "text": " it's gonna be easier."}, {"start": 1856.2, "end": 1857.52, "text": " Okay, so here we are."}, {"start": 1857.52, "end": 1861.04, "text": " I can see the diagrams on the side here."}, {"start": 1861.04, "end": 1864.8000000000002, "text": " So again, we have the two comp1d layer,"}, {"start": 1864.8000000000002, "end": 1867.96, "text": " so that's this part here, the stem of the encoder."}, {"start": 1867.96, "end": 1871.5600000000002, "text": " Then we have the non-learnable positional embeddings."}, {"start": 1871.5600000000002, "end": 1873.3600000000001, "text": " You can see those here."}, {"start": 1873.3600000000001, "end": 1875.16, "text": " And finally, we have bunch of blocks,"}, {"start": 1875.16, "end": 1877.4, "text": " which are just residual attention blocks."}, {"start": 1877.4, "end": 1879.0400000000002, "text": " So that's nothing new there."}, {"start": 1879.0400000000002, "end": 1880.64, "text": " And finally, we end up with a layer norm."}, {"start": 1880.64, "end": 1883.0400000000002, "text": " And later we'll see in the actual forward pass"}, {"start": 1883.0400000000002, "end": 1887.6000000000001, "text": " how we pass the male spectrogram through the comp1d"}, {"start": 1887.6000000000001, "end": 1890.48, "text": " and then we have the JALU, blah, blah, blah, the stem,"}, {"start": 1890.48, "end": 1892.24, "text": " and then the encoder part."}, {"start": 1892.24, "end": 1894.24, "text": " So everything is fairly trivial here."}, {"start": 1894.24, "end": 1899.24, "text": " Okay, so let's exit the construction of the audioencoder."}, {"start": 1900.1200000000001, "end": 1905.1200000000001, "text": " Let me just make sure that I have everything enabled."}, {"start": 1905.12, "end": 1908.4399999999998, "text": " Let's hit F5 and enter the decoder."}, {"start": 1908.4399999999998, "end": 1912.9199999999998, "text": " And now let's just go back to this side by side view."}, {"start": 1912.9199999999998, "end": 1914.6399999999999, "text": " So here is the text decoder."}, {"start": 1914.6399999999999, "end": 1916.7199999999998, "text": " You can see a couple of things."}, {"start": 1916.7199999999998, "end": 1919.56, "text": " Of course, embedding table here."}, {"start": 1919.56, "end": 1922.6, "text": " And because we're dealing with obviously a decoder model"}, {"start": 1922.6, "end": 1925.76, "text": " that has a vocabulary, we have positional embeddings,"}, {"start": 1925.76, "end": 1927.84, "text": " which are, as you can see here, these ones are learned."}, {"start": 1927.84, "end": 1929.84, "text": " So it says here, learn positional embeddings."}, {"start": 1929.84, "end": 1931.1999999999998, "text": " And that's why we have parameter."}, {"start": 1931.1999999999998, "end": 1934.0, "text": " And we just initialize, as you can see here."}, {"start": 1934.0, "end": 1938.6, "text": " So 448 is the context length that they are using"}, {"start": 1938.6, "end": 1941.36, "text": " and the state is around, yeah, 768."}, {"start": 1941.36, "end": 1943.8, "text": " Okay, and then we have a bunch of blocks."}, {"start": 1943.8, "end": 1946.32, "text": " These ones are cross-attention true."}, {"start": 1946.32, "end": 1948.14, "text": " So that means they're additionally gonna have"}, {"start": 1948.14, "end": 1952.08, "text": " this third component in the transformer layer,"}, {"start": 1952.08, "end": 1955.6, "text": " the so-called cross-attention block."}, {"start": 1955.6, "end": 1956.44, "text": " But yeah, that's it."}, {"start": 1956.44, "end": 1961.08, "text": " Fairly similar to what we saw in the encoder part."}, {"start": 1961.08, "end": 1964.8799999999999, "text": " Okay, and we have a mask, which is going to be"}, {"start": 1964.8799999999999, "end": 1966.84, "text": " basically a causal mask."}, {"start": 1966.84, "end": 1968.24, "text": " This is gonna be a causal decoder."}, {"start": 1968.24, "end": 1973.24, "text": " So if I were to print this, you can see minus infinities"}, {"start": 1973.58, "end": 1977.62, "text": " on the upper triangle, which means we're gonna mask out"}, {"start": 1977.62, "end": 1980.1999999999998, "text": " the future tokens and this particular token"}, {"start": 1980.1999999999998, "end": 1982.28, "text": " can only attend itself and the previous token"}, {"start": 1982.28, "end": 1984.04, "text": " and the previous token, et cetera."}, {"start": 1984.04, "end": 1986.8, "text": " Okay, hopefully nothing new there,"}, {"start": 1986.8, "end": 1988.8, "text": " just like standard attention stuff,"}, {"start": 1988.8, "end": 1991.24, "text": " transformer stuff, sorry."}, {"start": 1991.24, "end": 1992.8799999999999, "text": " And now let's go back."}, {"start": 1992.8799999999999, "end": 1996.6399999999999, "text": " So let me now kinda expand this again."}, {"start": 1996.6399999999999, "end": 1998.48, "text": " So we load the dictionaries."}, {"start": 1998.48, "end": 2001.76, "text": " We just push the model to the device."}, {"start": 2001.76, "end": 2004.76, "text": " Let's exit here and they have a small,"}, {"start": 2004.76, "end": 2008.1399999999999, "text": " well, it's not a bug, but they do the two device two times,"}, {"start": 2008.1399999999999, "end": 2010.32, "text": " which is kinda unnecessary."}, {"start": 2010.32, "end": 2012.0, "text": " But yeah, okay."}, {"start": 2012.0, "end": 2014.56, "text": " So here we are, we popped the audio path."}, {"start": 2014.56, "end": 2017.56, "text": " So I'm gonna use the one that they provided"}, {"start": 2017.56, "end": 2019.6799999999998, "text": " in the code base."}, {"start": 2019.6799999999998, "end": 2022.0, "text": " So maybe it will be useful for me to show you"}, {"start": 2023.28, "end": 2024.52, "text": " how that sounds."}, {"start": 2024.52, "end": 2025.36, "text": " So here it is."}, {"start": 2026.32, "end": 2028.26, "text": " And so my fellow America,"}, {"start": 2029.24, "end": 2032.6399999999999, "text": " ask not what your country can do for you."}, {"start": 2032.6399999999999, "end": 2034.3999999999999, "text": " Can do for you."}, {"start": 2034.3999999999999, "end": 2036.52, "text": " Ask what your country can do for your country."}, {"start": 2036.52, "end": 2039.56, "text": " Okay, so that's the audio file we are using"}, {"start": 2039.56, "end": 2041.5, "text": " and let's now transcribe it."}, {"start": 2041.5, "end": 2044.84, "text": " So again, let me make sure we are enabling the breakpoints."}, {"start": 2044.84, "end": 2047.6399999999999, "text": " We hit F5 and here we are."}, {"start": 2047.6399999999999, "end": 2052.04, "text": " So let's see what's interesting going to happen here."}, {"start": 2052.04, "end": 2054.5, "text": " So the first interesting part is the log mal spectrogram."}, {"start": 2054.5, "end": 2056.5, "text": " So let's see how we convert an audio file,"}, {"start": 2056.5, "end": 2058.88, "text": " which is just a path into a mal spectrogram."}, {"start": 2058.88, "end": 2060.16, "text": " So let's enter here."}, {"start": 2060.16, "end": 2062.92, "text": " So the first part is loading audio files."}, {"start": 2062.92, "end": 2064.92, "text": " So we just use FFmpeg for that."}, {"start": 2064.92, "end": 2066.84, "text": " And then you can see blah, blah, blah,"}, {"start": 2066.84, "end": 2070.96, "text": " some from buffer because these are bytes, integer 16."}, {"start": 2070.96, "end": 2073.24, "text": " And that's we divide by 32,000"}, {"start": 2073.24, "end": 2075.9199999999996, "text": " because precisely because we're using int 16."}, {"start": 2075.9199999999996, "end": 2079.9199999999996, "text": " So that's like, I guess two raised to the power of 15"}, {"start": 2082.08, "end": 2083.8799999999997, "text": " is exactly that number."}, {"start": 2083.8799999999997, "end": 2087.56, "text": " And that's why we are basically putting the amplitude"}, {"start": 2087.56, "end": 2091.6, "text": " in the minus one, one range, just normalization detail."}, {"start": 2091.6, "end": 2094.8199999999997, "text": " Okay, so once we have, so we are in the log mal spectrogram."}, {"start": 2094.8199999999997, "end": 2095.9199999999996, "text": " So once we load the data,"}, {"start": 2095.9199999999996, "end": 2097.68, "text": " which is just raw data at the moment,"}, {"start": 2097.68, "end": 2100.4399999999996, "text": " so that's gonna be just a sequence of numbers,"}, {"start": 2100.44, "end": 2105.36, "text": " 176,000 floats."}, {"start": 2105.36, "end": 2107.44, "text": " Now we form the hand window,"}, {"start": 2107.44, "end": 2109.68, "text": " just some signal processing stuff."}, {"start": 2109.68, "end": 2114.68, "text": " Then we apply the short, basically a Fourier transform."}, {"start": 2114.7200000000003, "end": 2118.7200000000003, "text": " It stands for, what does it stand for?"}, {"start": 2118.7200000000003, "end": 2121.44, "text": " A short time, yeah, short time Fourier transform."}, {"start": 2121.44, "end": 2124.2000000000003, "text": " Okay, so we applied the short time Fourier transform,"}, {"start": 2124.2000000000003, "end": 2127.44, "text": " which is gonna basically convert this audio file"}, {"start": 2127.44, "end": 2131.56, "text": " into a spectrogram, which is your regular spectrogram."}, {"start": 2131.56, "end": 2136.56, "text": " So if I were to print this particular variable here,"}, {"start": 2136.7200000000003, "end": 2139.16, "text": " the spectrogram, the shape is gonna be,"}, {"start": 2139.16, "end": 2143.44, "text": " as you can see here, 201 and 1,101."}, {"start": 2143.44, "end": 2145.52, "text": " So 201 is the frequency dimension,"}, {"start": 2145.52, "end": 2146.8, "text": " this one is the time dimension."}, {"start": 2146.8, "end": 2148.92, "text": " The reason being they have some,"}, {"start": 2148.92, "end": 2150.68, "text": " they're doing sliding window approaches."}, {"start": 2150.68, "end": 2154.44, "text": " So that's why we don't have as many timestamps"}, {"start": 2154.44, "end": 2158.44, "text": " as in the original raw audio."}, {"start": 2158.44, "end": 2160.68, "text": " Okay, so then we do the magnitudes"}, {"start": 2160.68, "end": 2164.52, "text": " because we wanna have the magnitude component of the Fourier."}, {"start": 2164.52, "end": 2167.44, "text": " And then we form these filters."}, {"start": 2167.44, "end": 2169.2000000000003, "text": " As you can see here, we just,"}, {"start": 2169.2000000000003, "end": 2172.8, "text": " what they do is they use this Librosa"}, {"start": 2172.8, "end": 2175.68, "text": " to generate these MEL filters,"}, {"start": 2175.68, "end": 2177.96, "text": " and then they saved the MEL filters"}, {"start": 2177.96, "end": 2180.16, "text": " into this particular file."}, {"start": 2180.16, "end": 2184.7999999999997, "text": " So you can see here, this assets MEL filters, MPZ."}, {"start": 2184.7999999999997, "end": 2188.24, "text": " So if you go to the assets directory here,"}, {"start": 2188.24, "end": 2189.64, "text": " you can see that the MPZ,"}, {"start": 2189.64, "end": 2192.3199999999997, "text": " so they just kinda save them there"}, {"start": 2192.3199999999997, "end": 2195.3199999999997, "text": " and then they can avoid having the dependency."}, {"start": 2195.3199999999997, "end": 2197.08, "text": " Anyways, it's already too many details."}, {"start": 2197.08, "end": 2199.6, "text": " Let's get out of here, filters,"}, {"start": 2199.6, "end": 2202.3199999999997, "text": " and then we do the matrix multiplication."}, {"start": 2202.3199999999997, "end": 2204.6, "text": " Again, that's a detail where by,"}, {"start": 2204.6, "end": 2207.0, "text": " once you multiply using these triangular filters,"}, {"start": 2207.0, "end": 2210.36, "text": " once you multiply that with the actual magnitude"}, {"start": 2210.36, "end": 2211.92, "text": " of Fourier spectrogram,"}, {"start": 2211.92, "end": 2213.64, "text": " then you end up with a MEL spectrogram."}, {"start": 2213.64, "end": 2217.92, "text": " And now we're gonna see how this is going to have 80 bands."}, {"start": 2217.92, "end": 2220.36, "text": " So if I were to print the shape of this thing,"}, {"start": 2220.36, "end": 2223.54, "text": " it's gonna have, oops, let's just grab this variable."}, {"start": 2224.48, "end": 2227.08, "text": " It's gonna have, as you can see here,"}, {"start": 2227.08, "end": 2232.08, "text": " 80 bands and we have 1,100, again, time dimension."}, {"start": 2232.52, "end": 2233.6, "text": " And if you go to the paper,"}, {"start": 2233.6, "end": 2235.68, "text": " you'll see all of these details there."}, {"start": 2235.68, "end": 2240.04, "text": " So again, let me just quickly jump to that part"}, {"start": 2240.04, "end": 2243.3599999999997, "text": " where they mention it, somewhere here."}, {"start": 2244.2, "end": 2247.44, "text": " So all audio is resampled to 16,000 hertz"}, {"start": 2247.44, "end": 2250.12, "text": " and an 80 channel log magnitude MEL spectrogram"}, {"start": 2250.12, "end": 2253.44, "text": " representation is computed on 25 millisecond windows"}, {"start": 2253.44, "end": 2255.3999999999996, "text": " with a stride of 10 milliseconds."}, {"start": 2255.3999999999996, "end": 2258.64, "text": " And because we're using these windows of 25 milliseconds,"}, {"start": 2258.64, "end": 2261.6, "text": " that's why we end up with a much shorter representation"}, {"start": 2261.6, "end": 2263.48, "text": " in the time dimension compared to having"}, {"start": 2263.48, "end": 2266.2400000000002, "text": " a 176,000 or something."}, {"start": 2266.2400000000002, "end": 2269.6, "text": " Okay, let's go back to the code, some normalization."}, {"start": 2269.6, "end": 2271.84, "text": " I'm not gonna focus too much on this."}, {"start": 2271.84, "end": 2273.68, "text": " And finally, because the language is none,"}, {"start": 2273.68, "end": 2276.2, "text": " we're now going to predict the language."}, {"start": 2276.2, "end": 2277.6, "text": " Let's see how that's gonna work."}, {"start": 2277.6, "end": 2281.32, "text": " So detecting language using the first 30 seconds,"}, {"start": 2281.32, "end": 2284.12, "text": " blah, blah, blah, we first do the padding,"}, {"start": 2284.12, "end": 2288.44, "text": " which is just going to pad this to a 30 second audio file."}, {"start": 2288.44, "end": 2292.4, "text": " So that means it's gonna have 3,000,"}, {"start": 2292.4, "end": 2297.4, "text": " basically dimensions in this time, along the time axis."}, {"start": 2299.4, "end": 2303.2000000000003, "text": " And now let's basically do the detect language part."}, {"start": 2303.2000000000003, "end": 2304.2000000000003, "text": " So we passed the segment,"}, {"start": 2304.2000000000003, "end": 2305.96, "text": " which is just a pad MEL spectrogram,"}, {"start": 2305.96, "end": 2310.12, "text": " which means basically we padded with all zeros to the right."}, {"start": 2310.12, "end": 2311.92, "text": " And let's now do the detection."}, {"start": 2311.92, "end": 2314.42, "text": " So here is how it's gonna look like."}, {"start": 2314.42, "end": 2316.6600000000003, "text": " We get the tokenizer first."}, {"start": 2316.6600000000003, "end": 2319.04, "text": " It might be interesting to just briefly"}, {"start": 2319.04, "end": 2320.8, "text": " jump into the tokenizer."}, {"start": 2320.8, "end": 2323.1000000000004, "text": " And let me show you how it looks like."}, {"start": 2323.1000000000004, "end": 2326.36, "text": " So we specify that it's a multilingual tokenizer,"}, {"start": 2326.36, "end": 2328.6000000000004, "text": " task is transcription, we have English,"}, {"start": 2328.6000000000004, "end": 2330.28, "text": " and now we do the build part."}, {"start": 2330.28, "end": 2331.6600000000003, "text": " Okay."}, {"start": 2331.6600000000003, "end": 2334.6200000000003, "text": " So here is how building the tokenizer looks like."}, {"start": 2334.6200000000003, "end": 2337.1000000000004, "text": " We are basically taking the pre-trained"}, {"start": 2337.1000000000004, "end": 2340.04, "text": " GPT-2 tokenizer fast from the Huggins face library,"}, {"start": 2340.04, "end": 2341.96, "text": " and we just pre-loaded."}, {"start": 2341.96, "end": 2343.2400000000002, "text": " And here are the special tokens."}, {"start": 2343.2400000000002, "end": 2346.96, "text": " So here are the tokens we saw before in the paper."}, {"start": 2346.96, "end": 2350.1200000000003, "text": " So we have the start of the transcript token, SOT."}, {"start": 2350.12, "end": 2351.6, "text": " We have, as you can see here,"}, {"start": 2351.6, "end": 2355.48, "text": " I think this language's keys is like 99 keys"}, {"start": 2355.48, "end": 2357.08, "text": " in total or something."}, {"start": 2357.08, "end": 2358.68, "text": " Let me just double check that."}, {"start": 2358.68, "end": 2361.16, "text": " So you can see here, 99 keys precisely."}, {"start": 2361.16, "end": 2363.68, "text": " So that means we'll add for each of those languages,"}, {"start": 2363.68, "end": 2365.5, "text": " we'll add this special token."}, {"start": 2365.5, "end": 2369.6, "text": " For example, this and like the smaller sign"}, {"start": 2369.6, "end": 2373.7599999999998, "text": " and then bar and then N, like EN, and then the rest."}, {"start": 2373.7599999999998, "end": 2375.88, "text": " And for each of the languages, a similar thing."}, {"start": 2375.88, "end": 2377.96, "text": " So you can see here how the keys look like here"}, {"start": 2377.96, "end": 2379.16, "text": " for the languages."}, {"start": 2379.16, "end": 2382.62, "text": " And then we have translate, transcribe, start of LM,"}, {"start": 2382.62, "end": 2385.8199999999997, "text": " start of prev, no captions and no timestamps."}, {"start": 2385.8199999999997, "end": 2388.1, "text": " So each of these have some meaning."}, {"start": 2388.1, "end": 2390.94, "text": " We're gonna see maybe some of them are subset a bit later."}, {"start": 2390.94, "end": 2393.56, "text": " So we end up adding those to the pre-trained"}, {"start": 2393.56, "end": 2397.54, "text": " Huggins face tokenizer and we return back here."}, {"start": 2397.54, "end": 2401.12, "text": " Okay, so now let's just see what's interesting."}, {"start": 2401.12, "end": 2403.52, "text": " None of this is gonna be interesting for the time being."}, {"start": 2403.52, "end": 2406.02, "text": " They just create a SOT sequence,"}, {"start": 2406.02, "end": 2407.7, "text": " which is gonna have, as you can see here,"}, {"start": 2407.7, "end": 2411.7799999999997, "text": " they start with the SOT and they added the EN token"}, {"start": 2411.7799999999997, "end": 2414.58, "text": " and they added the transcribe token."}, {"start": 2414.58, "end": 2418.7799999999997, "text": " And so that's basically the sequence we saw before here."}, {"start": 2418.7799999999997, "end": 2420.7999999999997, "text": " So let me just go back to the paper."}, {"start": 2420.7999999999997, "end": 2422.98, "text": " So that's your classic sequence here."}, {"start": 2422.98, "end": 2426.3599999999997, "text": " So the start of transcript, language tag and transcribe."}, {"start": 2427.5, "end": 2429.58, "text": " Let's go back to the code."}, {"start": 2429.58, "end": 2432.18, "text": " But we are not going to use that for the language detection"}, {"start": 2432.18, "end": 2433.9199999999996, "text": " that we are currently doing."}, {"start": 2433.9199999999996, "end": 2436.3799999999997, "text": " So that's why I said it's not that important at the moment."}, {"start": 2436.38, "end": 2440.1800000000003, "text": " So we just do unsqueeze on the mail, nothing fancy."}, {"start": 2440.1800000000003, "end": 2443.46, "text": " And then we pass the mail through the encoder stack."}, {"start": 2443.46, "end": 2445.3, "text": " So let's now get the features."}, {"start": 2445.3, "end": 2448.6600000000003, "text": " So you can see here that everything is the same"}, {"start": 2448.6600000000003, "end": 2451.1, "text": " as what the diagram says."}, {"start": 2451.1, "end": 2453.5, "text": " We do a forward pass through, I guess,"}, {"start": 2453.5, "end": 2455.1400000000003, "text": " 12 blocks or something."}, {"start": 2455.1400000000003, "end": 2456.6600000000003, "text": " So we can just skip that."}, {"start": 2456.6600000000003, "end": 2459.1, "text": " Let me just see how many blocks this one has."}, {"start": 2459.1, "end": 2462.1600000000003, "text": " I guess it's like 12 or something, 12, yeah."}, {"start": 2462.16, "end": 2466.6, "text": " Let's just pass through all of that."}, {"start": 2466.6, "end": 2469.3399999999997, "text": " And we ultimately end up with the following shape."}, {"start": 2470.24, "end": 2473.8799999999997, "text": " So you can see here, 1,500."}, {"start": 2473.8799999999997, "end": 2477.3599999999997, "text": " So because we have one of these has stride of two,"}, {"start": 2477.3599999999997, "end": 2478.3799999999997, "text": " stride is set to two."}, {"start": 2478.3799999999997, "end": 2481.7999999999997, "text": " And that's why we went from 3,000 to 2,500."}, {"start": 2481.7999999999997, "end": 2484.56, "text": " And the dimensionality is 768."}, {"start": 2484.56, "end": 2488.52, "text": " So that's the model dimension of the encoder."}, {"start": 2488.52, "end": 2490.72, "text": " Okay, let's exit here."}, {"start": 2490.72, "end": 2492.7599999999998, "text": " Number of audio is gonna be one because we only have"}, {"start": 2492.7599999999998, "end": 2493.9599999999996, "text": " a one audio file."}, {"start": 2493.9599999999996, "end": 2496.9599999999996, "text": " And here you see, so here we just set the SOT token"}, {"start": 2498.72, "end": 2501.9199999999996, "text": " and we create a tensor that has SOT."}, {"start": 2501.9199999999996, "end": 2504.3599999999997, "text": " And now we're gonna pass the mail,"}, {"start": 2504.3599999999997, "end": 2508.3599999999997, "text": " so the encoder features from the output of the encoder,"}, {"start": 2508.3599999999997, "end": 2509.9599999999996, "text": " and we pass the SOT."}, {"start": 2509.9599999999996, "end": 2512.56, "text": " And let's see what the model is gonna generate."}, {"start": 2512.56, "end": 2514.7999999999997, "text": " So let's see which language it gives"}, {"start": 2514.7999999999997, "end": 2516.3599999999997, "text": " the highest probability to."}, {"start": 2516.3599999999997, "end": 2518.3999999999996, "text": " And that's how we pick the language."}, {"start": 2518.4, "end": 2521.36, "text": " So let me just go into the code instead of explaining"}, {"start": 2521.36, "end": 2522.92, "text": " like this."}, {"start": 2522.92, "end": 2527.28, "text": " So here we are, we are in the text decoder forward pass."}, {"start": 2527.28, "end": 2529.08, "text": " I'm gonna ignore the cache parts."}, {"start": 2529.08, "end": 2532.2400000000002, "text": " Those are just implementation details of how they optimize"}, {"start": 2532.2400000000002, "end": 2536.0, "text": " this execution of the forward pass of the decoder."}, {"start": 2536.0, "end": 2537.2200000000003, "text": " And let's see what's going on."}, {"start": 2537.2200000000003, "end": 2539.1, "text": " So they do the token embedding."}, {"start": 2539.1, "end": 2541.6800000000003, "text": " So again, remember this is just the SOT token,"}, {"start": 2541.6800000000003, "end": 2542.56, "text": " as you can see here."}, {"start": 2542.56, "end": 2544.7400000000002, "text": " And then they just add the positional embedding."}, {"start": 2544.7400000000002, "end": 2547.48, "text": " So all of that is kinda standard stuff."}, {"start": 2547.48, "end": 2549.44, "text": " And then you can see here,"}, {"start": 2549.44, "end": 2550.92, "text": " they're doing the cross attention."}, {"start": 2550.92, "end": 2553.48, "text": " So XA, XA is our,"}, {"start": 2555.22, "end": 2558.36, "text": " the output features of the encoder, as you can see here."}, {"start": 2558.36, "end": 2560.36, "text": " And so that's what's going on here."}, {"start": 2560.36, "end": 2563.26, "text": " So because of that, I'm just gonna skip all of that."}, {"start": 2563.26, "end": 2565.2, "text": " And let's get here."}, {"start": 2565.2, "end": 2569.0, "text": " Now the output features are multiplied by the,"}, {"start": 2569.0, "end": 2572.76, "text": " we do matrix multiplication with the embedding table,"}, {"start": 2572.76, "end": 2577.76, "text": " and that's gonna project those from 768 dimensions"}, {"start": 2577.84, "end": 2580.0800000000004, "text": " into the vocab space, which is like,"}, {"start": 2580.0800000000004, "end": 2582.1200000000003, "text": " let's see what's the dimensionality."}, {"start": 2582.1200000000003, "end": 2587.1200000000003, "text": " So logits shape is gonna be 51,865."}, {"start": 2589.5200000000004, "end": 2591.76, "text": " Okay, so those are the logits."}, {"start": 2591.76, "end": 2594.5600000000004, "text": " And now comes the interesting part."}, {"start": 2594.5600000000004, "end": 2599.5600000000004, "text": " So we create a mask that initially just has basically ones"}, {"start": 2599.56, "end": 2603.4, "text": " for every single token in the vocab."}, {"start": 2603.4, "end": 2605.2, "text": " Okay, so that's the mask initially."}, {"start": 2605.2, "end": 2607.68, "text": " And then for the language tokens,"}, {"start": 2607.68, "end": 2609.12, "text": " let's set that mask to false,"}, {"start": 2609.12, "end": 2611.72, "text": " which means we do not want to mask the language token."}, {"start": 2611.72, "end": 2614.48, "text": " So the EN, the DE, all of the 99,"}, {"start": 2614.48, "end": 2616.32, "text": " I think this should be 99."}, {"start": 2616.32, "end": 2619.2799999999997, "text": " Let me just make sure that's the case."}, {"start": 2619.2799999999997, "end": 2623.4, "text": " So let's see what's the length of this list, 99,"}, {"start": 2623.4, "end": 2624.56, "text": " precisely 99."}, {"start": 2624.56, "end": 2626.7599999999998, "text": " And so that's how we modify the mask."}, {"start": 2626.76, "end": 2630.96, "text": " And now we put the logits of every single token,"}, {"start": 2630.96, "end": 2634.36, "text": " except for the language tokens to minus infinity,"}, {"start": 2634.36, "end": 2637.6000000000004, "text": " which after we apply the softmax is gonna be,"}, {"start": 2637.6000000000004, "end": 2638.6400000000003, "text": " is gonna set them to zero."}, {"start": 2638.6400000000003, "end": 2640.48, "text": " So that means they'll have zero probability"}, {"start": 2640.48, "end": 2643.36, "text": " and only the language tokens will have non-zero probability."}, {"start": 2643.36, "end": 2645.1600000000003, "text": " Okay, so here we do argmax"}, {"start": 2645.1600000000003, "end": 2648.92, "text": " to find the highest probability language token."}, {"start": 2648.92, "end": 2651.0, "text": " We end up with, I think, English or something."}, {"start": 2651.0, "end": 2653.0400000000004, "text": " So this is, I think, English."}, {"start": 2653.0400000000004, "end": 2656.2400000000002, "text": " And we also return the probabilities"}, {"start": 2656.24, "end": 2659.24, "text": " for the other language tokens."}, {"start": 2660.16, "end": 2663.0, "text": " We just zip that and we return back those results."}, {"start": 2663.0, "end": 2665.24, "text": " So we return the actual language token"}, {"start": 2665.24, "end": 2666.8799999999997, "text": " and we return the probabilities."}, {"start": 2667.7599999999998, "end": 2671.2799999999997, "text": " But I think, yeah, this one is not even used."}, {"start": 2671.2799999999997, "end": 2674.3999999999996, "text": " So we just grab the, this is kind of wasteful."}, {"start": 2674.3999999999996, "end": 2678.06, "text": " They could have just passed it directly here, but anyways."}, {"start": 2679.08, "end": 2681.2799999999997, "text": " So they end up with the language."}, {"start": 2681.2799999999997, "end": 2682.9599999999996, "text": " We just grab the key."}, {"start": 2682.9599999999996, "end": 2685.66, "text": " So you can see the probabilities are language token"}, {"start": 2685.66, "end": 2687.92, "text": " and the probability, you can see that EN,"}, {"start": 2687.92, "end": 2691.24, "text": " English has 0.95, roughly."}, {"start": 2691.24, "end": 2694.68, "text": " And that's why we're gonna pick precisely English"}, {"start": 2694.68, "end": 2695.8599999999997, "text": " as the language here."}, {"start": 2696.8199999999997, "end": 2700.7599999999998, "text": " And we output detected language is English language."}, {"start": 2700.7599999999998, "end": 2702.3799999999997, "text": " Okay, that's the first part."}, {"start": 2702.3799999999997, "end": 2705.08, "text": " So we now have the English predicted."}, {"start": 2705.08, "end": 2709.7999999999997, "text": " Now let's continue doing the actual transcription."}, {"start": 2709.7999999999997, "end": 2712.8399999999997, "text": " I'm gonna skip over all of these details."}, {"start": 2712.84, "end": 2715.88, "text": " Let me just disable the break points."}, {"start": 2715.88, "end": 2718.34, "text": " Let's get to the interesting part."}, {"start": 2718.34, "end": 2720.32, "text": " These are just some details where,"}, {"start": 2720.32, "end": 2723.7200000000003, "text": " so the precision is, as you can see here, 20 milliseconds."}, {"start": 2723.7200000000003, "end": 2726.4, "text": " That's why this is 0.02."}, {"start": 2726.4, "end": 2731.08, "text": " Those are, I guess, not that vital."}, {"start": 2731.08, "end": 2733.6800000000003, "text": " Okay, we just pad again the, the MEL"}, {"start": 2733.6800000000003, "end": 2735.88, "text": " so that we end up with 3000 here."}, {"start": 2735.88, "end": 2739.28, "text": " So let's just check the shape of this segment."}, {"start": 2739.28, "end": 2741.1200000000003, "text": " So you can see it's 83,000"}, {"start": 2741.12, "end": 2744.72, "text": " because we have 80 MEL bands and we have 3000"}, {"start": 2744.72, "end": 2749.08, "text": " because we have a 30 second padded audio clip."}, {"start": 2749.08, "end": 2751.2799999999997, "text": " Okay, we calculate the duration."}, {"start": 2751.2799999999997, "end": 2752.88, "text": " It's gonna be 30 seconds."}, {"start": 2752.88, "end": 2755.6, "text": " And then this is where the magic starts happening."}, {"start": 2755.6, "end": 2757.06, "text": " Decode will fall back."}, {"start": 2757.06, "end": 2758.44, "text": " Let's see how that's gonna work."}, {"start": 2758.44, "end": 2761.72, "text": " Okay, here we are."}, {"start": 2761.72, "end": 2765.56, "text": " Temperature, again, we have the list"}, {"start": 2765.56, "end": 2766.72, "text": " of various temperatures."}, {"start": 2766.72, "end": 2768.92, "text": " We grab the first one,"}, {"start": 2768.92, "end": 2770.48, "text": " which is gonna be the zero temperature,"}, {"start": 2770.48, "end": 2774.92, "text": " which means we are basically greedy decoding."}, {"start": 2774.92, "end": 2779.92, "text": " And well, actually, because we'll be actually using the beam."}, {"start": 2781.34, "end": 2783.84, "text": " So let's just continue here and see what's going on."}, {"start": 2785.12, "end": 2786.98, "text": " Decoding task."}, {"start": 2786.98, "end": 2790.48, "text": " So this is how they kind of design this."}, {"start": 2790.48, "end": 2793.54, "text": " They form a decoding task and then they call run"}, {"start": 2793.54, "end": 2796.58, "text": " on that task and they pass the MEL spectrogram."}, {"start": 2796.58, "end": 2798.04, "text": " So let's see what's going on."}, {"start": 2798.04, "end": 2800.8, "text": " I'm gonna skip some of these details"}, {"start": 2800.8, "end": 2803.68, "text": " because there's too many stuff going on."}, {"start": 2803.68, "end": 2808.68, "text": " Just wanna focus on the stuff that's gonna be useful"}, {"start": 2810.84, "end": 2812.32, "text": " for whisper understanding, of course."}, {"start": 2812.32, "end": 2817.32, "text": " So here, this one implements this penalty with the length."}, {"start": 2818.4, "end": 2820.24, "text": " So when you're doing beam decoding,"}, {"start": 2820.24, "end": 2824.36, "text": " you want to take into account how probable that beam is,"}, {"start": 2824.36, "end": 2826.84, "text": " but you also wanna penalize if it's too long."}, {"start": 2826.84, "end": 2831.84, "text": " So this is what this maximum likelihood ranker does here,"}, {"start": 2832.2000000000003, "end": 2833.8, "text": " as you can see here."}, {"start": 2833.8, "end": 2836.46, "text": " So penalty and we are appending the log prop"}, {"start": 2836.46, "end": 2837.96, "text": " divided by the penalty."}, {"start": 2837.96, "end": 2841.2000000000003, "text": " I'll skip this, but in case you want to learn more,"}, {"start": 2841.2000000000003, "end": 2843.6800000000003, "text": " and I think this repo is amazing when it comes"}, {"start": 2843.6800000000003, "end": 2846.4, "text": " to understanding how beam decoding works."}, {"start": 2846.4, "end": 2849.1800000000003, "text": " If you wanna, I might create a separate video"}, {"start": 2849.1800000000003, "end": 2851.4, "text": " where I just go in a lot of depth"}, {"start": 2851.4, "end": 2853.84, "text": " and explain how this works,"}, {"start": 2853.84, "end": 2856.1200000000003, "text": " but here I'm gonna skip those details."}, {"start": 2856.12, "end": 2859.16, "text": " So we form a beam decoder here,"}, {"start": 2859.16, "end": 2861.24, "text": " and then this is the interesting part."}, {"start": 2861.24, "end": 2864.14, "text": " So here are some of the heuristics that they are using."}, {"start": 2864.14, "end": 2866.6, "text": " They're suppressing certain tokens"}, {"start": 2866.6, "end": 2868.2599999999998, "text": " during the decoding process."}, {"start": 2868.2599999999998, "end": 2871.4, "text": " So one of those is gonna be suppressed blank,"}, {"start": 2871.4, "end": 2874.3199999999997, "text": " which means we're gonna suppress the blank token"}, {"start": 2874.3199999999997, "end": 2877.98, "text": " as the first token after we start the transcription."}, {"start": 2877.98, "end": 2881.4, "text": " So we'll get to that a bit later."}, {"start": 2881.4, "end": 2885.18, "text": " Then they have a specific list of tokens"}, {"start": 2885.18, "end": 2886.9199999999996, "text": " that they wanna suppress."}, {"start": 2886.9199999999996, "end": 2889.7799999999997, "text": " So because we have, let me enter this one."}, {"start": 2889.7799999999997, "end": 2891.04, "text": " So let me enter this one."}, {"start": 2892.2599999999998, "end": 2896.12, "text": " So suppressed tokens are set to minus one"}, {"start": 2896.12, "end": 2899.58, "text": " because that's our input argument to the program."}, {"start": 2899.58, "end": 2903.94, "text": " And now let's see how that's gonna be mapped"}, {"start": 2903.94, "end": 2905.8799999999997, "text": " into the actual tokens."}, {"start": 2905.8799999999997, "end": 2910.16, "text": " So here you can see we extend by these non-speech tokens"}, {"start": 2910.16, "end": 2911.64, "text": " and here they are."}, {"start": 2911.64, "end": 2914.56, "text": " So just a list of tokens that cannot appear"}, {"start": 2914.56, "end": 2916.7799999999997, "text": " in the spoken language."}, {"start": 2917.7599999999998, "end": 2920.7799999999997, "text": " And again, I'm gonna skip because there is a lot of details,"}, {"start": 2920.7799999999997, "end": 2923.0, "text": " but that's the rough point."}, {"start": 2923.0, "end": 2925.7599999999998, "text": " We wanna help the decoder by just kinda mapping,"}, {"start": 2925.7599999999998, "end": 2928.56, "text": " by masking out those particular tokens."}, {"start": 2928.56, "end": 2929.7999999999997, "text": " Okay."}, {"start": 2929.7999999999997, "end": 2931.16, "text": " And they add some more."}, {"start": 2931.16, "end": 2933.16, "text": " We don't wanna predict the start of transcription"}, {"start": 2933.16, "end": 2935.0, "text": " because we already know we are transcribing."}, {"start": 2935.0, "end": 2937.88, "text": " So that's kinda a no-brainer, but yeah."}, {"start": 2938.7599999999998, "end": 2940.56, "text": " And finally, the no-captions,"}, {"start": 2940.56, "end": 2941.92, "text": " which is the token that tells you"}, {"start": 2941.92, "end": 2945.2400000000002, "text": " that we don't have any human voice."}, {"start": 2945.2400000000002, "end": 2948.12, "text": " We just have like music or something going on."}, {"start": 2948.12, "end": 2948.96, "text": " Okay."}, {"start": 2950.04, "end": 2951.8, "text": " That's a second heuristic they have."}, {"start": 2951.8, "end": 2956.8, "text": " And then they also have these applied timestamp rules,"}, {"start": 2958.58, "end": 2960.84, "text": " which do what?"}, {"start": 2960.84, "end": 2962.6, "text": " Okay. I have a break point here."}, {"start": 2962.6, "end": 2965.04, "text": " We'll see what it does once we hit the apply."}, {"start": 2965.04, "end": 2966.44, "text": " So let's just continue."}, {"start": 2967.7400000000002, "end": 2969.32, "text": " Let's continue here."}, {"start": 2969.32, "end": 2971.38, "text": " Let's exit here."}, {"start": 2971.38, "end": 2973.56, "text": " And we are now in the run method."}, {"start": 2973.56, "end": 2976.12, "text": " So all of that was construction of this decoding task."}, {"start": 2976.12, "end": 2980.32, "text": " Now let's see how the actual run looks like."}, {"start": 2980.32, "end": 2982.36, "text": " So we first fetch the audio features."}, {"start": 2982.36, "end": 2983.56, "text": " And this one is gonna again,"}, {"start": 2983.56, "end": 2986.88, "text": " just pass the the mel through the encoder"}, {"start": 2986.88, "end": 2988.48, "text": " and return back the audio features."}, {"start": 2988.48, "end": 2990.88, "text": " So I'm gonna skip all of that."}, {"start": 2990.88, "end": 2992.4, "text": " So I'm just gonna disable that."}, {"start": 2992.4, "end": 2993.88, "text": " Let's fetch the audio features"}, {"start": 2993.88, "end": 2996.36, "text": " by passing the mel through the encoder stack."}, {"start": 2997.36, "end": 2999.88, "text": " And by the way, I'm running this on a CPU"}, {"start": 2999.88, "end": 3002.94, "text": " because my cond environment was not properly set up."}, {"start": 3002.94, "end": 3005.88, "text": " That's why it's a bit slower,"}, {"start": 3005.88, "end": 3008.6800000000003, "text": " but it's still working fairly fast, which is kinda cool."}, {"start": 3009.9, "end": 3013.12, "text": " Okay. So we set the initial tokens to these three."}, {"start": 3013.12, "end": 3015.06, "text": " So that's the SOT, that's the English,"}, {"start": 3015.06, "end": 3017.0, "text": " and that's the transcribe."}, {"start": 3017.0, "end": 3018.6, "text": " That's our initial sequence."}, {"start": 3018.6, "end": 3022.88, "text": " And then this is gonna be a no-op."}, {"start": 3022.88, "end": 3024.8, "text": " Basically we just passed the language."}, {"start": 3024.8, "end": 3026.04, "text": " We already detected language."}, {"start": 3026.04, "end": 3026.88, "text": " We know it's English."}, {"start": 3026.88, "end": 3028.1800000000003, "text": " So that's gonna be a no-op."}, {"start": 3028.18, "end": 3033.18, "text": " Now we do repeat by five because we are doing beam."}, {"start": 3034.3599999999997, "end": 3036.6, "text": " Again, I'm not gonna focus on the details of beam,"}, {"start": 3036.6, "end": 3038.8799999999997, "text": " but let's just see the shape before and after."}, {"start": 3038.8799999999997, "end": 3042.52, "text": " So the shape now is 1500 because that's the time."}, {"start": 3042.52, "end": 3046.2, "text": " And then this is the actual embedding dimension."}, {"start": 3046.2, "end": 3049.3399999999997, "text": " So let's see what's going on after this call."}, {"start": 3049.3399999999997, "end": 3050.18, "text": " We have five."}, {"start": 3050.18, "end": 3053.8399999999997, "text": " So we have five because we'll have five hypotheses"}, {"start": 3053.8399999999997, "end": 3057.24, "text": " that we are keeping track of during the beam decoding."}, {"start": 3057.24, "end": 3058.9199999999996, "text": " And we do the same for tokens."}, {"start": 3058.9199999999996, "end": 3060.7999999999997, "text": " And then now here is the main loop."}, {"start": 3060.7999999999997, "end": 3063.06, "text": " So here is where the magic happens."}, {"start": 3063.06, "end": 3066.2799999999997, "text": " Okay. Let's start here."}, {"start": 3066.2799999999997, "end": 3071.02, "text": " So we now sample for 224 samples,"}, {"start": 3071.02, "end": 3076.02, "text": " and we first pass the tokens as well as the audio features"}, {"start": 3076.2, "end": 3078.24, "text": " to grab the logits."}, {"start": 3078.24, "end": 3080.64, "text": " So it's gonna be a simple pass through the decoder."}, {"start": 3080.64, "end": 3084.66, "text": " Again, some hook stuff that's again in optimization detail."}, {"start": 3084.66, "end": 3086.08, "text": " Here's the decoder call."}, {"start": 3086.08, "end": 3088.7599999999998, "text": " So what we do is again, we pass all of the,"}, {"start": 3088.7599999999998, "end": 3091.12, "text": " we embed our three tokens."}, {"start": 3091.12, "end": 3094.4, "text": " So here, let's see what's the shape now."}, {"start": 3094.4, "end": 3096.7599999999998, "text": " So you can see here five, three, 768,"}, {"start": 3096.7599999999998, "end": 3099.22, "text": " because we have three, five because we have beam,"}, {"start": 3099.22, "end": 3101.36, "text": " and this is the dimension of the model."}, {"start": 3101.36, "end": 3103.0, "text": " Now we pass all of those."}, {"start": 3103.0, "end": 3104.7799999999997, "text": " And we also have the, again,"}, {"start": 3104.7799999999997, "end": 3108.2799999999997, "text": " we are cross attending to the output features"}, {"start": 3108.2799999999997, "end": 3110.16, "text": " of the audio encoder."}, {"start": 3110.16, "end": 3111.62, "text": " Okay. That's the idea."}, {"start": 3111.62, "end": 3113.2799999999997, "text": " And finally, we end up with the logits."}, {"start": 3113.2799999999997, "end": 3114.4, "text": " So we can skip all of this."}, {"start": 3114.4, "end": 3117.8, "text": " This is just like regular transformer stuff."}, {"start": 3117.8, "end": 3119.4, "text": " Whoops."}, {"start": 3119.4, "end": 3120.88, "text": " And let's exit here."}, {"start": 3122.12, "end": 3123.12, "text": " Let's exit."}, {"start": 3123.12, "end": 3125.56, "text": " And we have our logits."}, {"start": 3125.56, "end": 3127.1600000000003, "text": " Now this is an interesting point."}, {"start": 3127.1600000000003, "end": 3128.04, "text": " Let's see what happens here."}, {"start": 3128.04, "end": 3130.7200000000003, "text": " Because we are in the iteration zero,"}, {"start": 3130.7200000000003, "end": 3135.04, "text": " and because no captions is set to,"}, {"start": 3135.04, "end": 3137.46, "text": " yeah, because no captions token exists,"}, {"start": 3137.46, "end": 3139.48, "text": " we're going to enter this branch."}, {"start": 3139.48, "end": 3143.44, "text": " And what we do is we grab the logits"}, {"start": 3143.44, "end": 3145.12, "text": " for the SOT index."}, {"start": 3145.12, "end": 3148.04, "text": " So that means what's the,"}, {"start": 3148.04, "end": 3152.32, "text": " once we condition the model on the SOT token,"}, {"start": 3152.32, "end": 3155.82, "text": " what's the distribution across the next tokens"}, {"start": 3155.82, "end": 3157.32, "text": " after the SOT."}, {"start": 3157.32, "end": 3160.68, "text": " And that's in case that there is a high probability"}, {"start": 3160.68, "end": 3162.08, "text": " for the no caption probability,"}, {"start": 3162.08, "end": 3164.88, "text": " then that means we don't have any audio,"}, {"start": 3164.88, "end": 3168.36, "text": " any human vocals in the audio clip."}, {"start": 3168.36, "end": 3169.66, "text": " So again, we index here,"}, {"start": 3169.66, "end": 3172.36, "text": " we grab the SOT logits,"}, {"start": 3172.36, "end": 3174.6800000000003, "text": " and that's gonna be, let's see the shape."}, {"start": 3174.6800000000003, "end": 3175.92, "text": " So here is the shape."}, {"start": 3175.92, "end": 3177.28, "text": " So you can see here,"}, {"start": 3177.28, "end": 3178.6400000000003, "text": " because it's being, we have five,"}, {"start": 3178.6400000000003, "end": 3182.28, "text": " but we just have the logits above the SOT token."}, {"start": 3182.28, "end": 3186.32, "text": " And then additionally, we grab precisely this particular,"}, {"start": 3186.32, "end": 3187.6400000000003, "text": " we are interested in the probability"}, {"start": 3187.6400000000003, "end": 3188.96, "text": " of this particular token,"}, {"start": 3188.96, "end": 3191.28, "text": " and that's gonna be the no caption probability."}, {"start": 3191.28, "end": 3193.6200000000003, "text": " And in our particular case, it's very low,"}, {"start": 3193.6200000000003, "end": 3195.08, "text": " but if it was high enough,"}, {"start": 3195.08, "end": 3198.44, "text": " then we would basically ignore the decoder output"}, {"start": 3198.44, "end": 3199.78, "text": " and just say that, okay,"}, {"start": 3199.78, "end": 3202.08, "text": " there is no voice happening in this audio clip."}, {"start": 3202.08, "end": 3204.52, "text": " And let me remind you here,"}, {"start": 3204.52, "end": 3205.36, "text": " that's this part."}, {"start": 3205.36, "end": 3208.52, "text": " So we either have the language tag or the no speech tag,"}, {"start": 3208.52, "end": 3210.44, "text": " and that's the voice activity detection."}, {"start": 3210.44, "end": 3213.7599999999998, "text": " So that's how that part, that task breaks."}, {"start": 3213.7599999999998, "end": 3215.22, "text": " So now you see it in code."}, {"start": 3216.6, "end": 3218.58, "text": " Okay, so having collected that,"}, {"start": 3218.58, "end": 3219.88, "text": " let's now grab the,"}, {"start": 3219.88, "end": 3222.88, "text": " because remember we actually have the SOT,"}, {"start": 3222.88, "end": 3227.1, "text": " English and transcribe tokens."}, {"start": 3227.1, "end": 3228.96, "text": " And we're gonna do the decoding,"}, {"start": 3228.96, "end": 3231.02, "text": " even if this was higher probability,"}, {"start": 3231.02, "end": 3232.96, "text": " we're still gonna do the decoding here"}, {"start": 3232.96, "end": 3236.92, "text": " and assume as if we wanna transcribe the audio,"}, {"start": 3236.92, "end": 3238.8, "text": " but in case this was high enough,"}, {"start": 3238.8, "end": 3243.12, "text": " then we would just ditch and ignore all of that output,"}, {"start": 3243.12, "end": 3244.8, "text": " which is probably wasteful."}, {"start": 3244.8, "end": 3246.7599999999998, "text": " Maybe some, just a break statement here"}, {"start": 3246.7599999999998, "end": 3249.14, "text": " or something would be more optimal,"}, {"start": 3249.14, "end": 3251.8, "text": " but yeah, that's just an optimization detail."}, {"start": 3252.64, "end": 3255.48, "text": " So we grab the logits for the fourth token,"}, {"start": 3255.48, "end": 3257.38, "text": " we apply a bunch of filters."}, {"start": 3257.38, "end": 3259.82, "text": " So those are the filters which I mentioned before."}, {"start": 3259.82, "end": 3260.68, "text": " So let's see."}, {"start": 3260.68, "end": 3263.3199999999997, "text": " The first one is the suppressed blank."}, {"start": 3263.3199999999997, "end": 3268.3199999999997, "text": " And you can see here that we wanna mask the blank token"}, {"start": 3268.3199999999997, "end": 3269.96, "text": " and we set it to minus infinity"}, {"start": 3269.96, "end": 3272.64, "text": " because we don't want to generate,"}, {"start": 3272.64, "end": 3274.7599999999998, "text": " as the first token does not,"}, {"start": 3274.7599999999998, "end": 3277.04, "text": " like it doesn't make any sense to store the audio"}, {"start": 3277.04, "end": 3278.2799999999997, "text": " with a blank token."}, {"start": 3278.2799999999997, "end": 3280.3999999999996, "text": " That's why they can mask it out."}, {"start": 3280.3999999999996, "end": 3282.12, "text": " What's probably weird is why is it not"}, {"start": 3282.12, "end": 3283.74, "text": " just like a generic white space"}, {"start": 3283.74, "end": 3285.18, "text": " and instead it's just like a space."}, {"start": 3285.18, "end": 3288.24, "text": " I'm not sure about that point, but yeah, let's continue."}, {"start": 3288.24, "end": 3290.62, "text": " And let me just enable all of the breakpoints."}, {"start": 3290.62, "end": 3291.7599999999998, "text": " Let's enter here."}, {"start": 3291.7599999999998, "end": 3293.08, "text": " So here's the second one."}, {"start": 3293.08, "end": 3295.18, "text": " These are the special tokens."}, {"start": 3295.18, "end": 3297.92, "text": " So that's again a list of these special tokens"}, {"start": 3297.92, "end": 3299.12, "text": " that we wanna suppress."}, {"start": 3299.12, "end": 3301.44, "text": " We set all of them to minus infinity, the logits."}, {"start": 3301.44, "end": 3303.88, "text": " So that means their probabilities after the softmax"}, {"start": 3303.88, "end": 3305.22, "text": " will be set to zero."}, {"start": 3306.2799999999997, "end": 3307.96, "text": " And we have the final one."}, {"start": 3307.96, "end": 3308.7999999999997, "text": " Here it is."}, {"start": 3308.7999999999997, "end": 3311.3599999999997, "text": " So here is the no timestamp."}, {"start": 3311.3599999999997, "end": 3312.68, "text": " And let's see how that one works."}, {"start": 3312.68, "end": 3314.3599999999997, "text": " I'm not sure about this one."}, {"start": 3314.3599999999997, "end": 3319.3599999999997, "text": " Okay, so it seems that we can never sample"}, {"start": 3319.36, "end": 3323.4, "text": " the no timestamps because we set it to minus infinity."}, {"start": 3323.4, "end": 3326.8, "text": " And that means we kinda cannot ever go down"}, {"start": 3326.8, "end": 3329.7000000000003, "text": " this route here, right?"}, {"start": 3329.7000000000003, "end": 3331.6400000000003, "text": " So that's kinda weird."}, {"start": 3331.6400000000003, "end": 3334.76, "text": " So in which setting do they actually sample"}, {"start": 3334.76, "end": 3335.76, "text": " this particular one?"}, {"start": 3335.76, "end": 3337.1600000000003, "text": " I'm not sure about that."}, {"start": 3338.32, "end": 3341.08, "text": " Okay, let's continue here."}, {"start": 3341.08, "end": 3343.02, "text": " I think this is gonna be a no op."}, {"start": 3343.02, "end": 3345.44, "text": " It's constantly gonna, yeah, it's gonna skip all of that."}, {"start": 3345.44, "end": 3349.08, "text": " So let's just put the breakpoint here, hit that five."}, {"start": 3349.08, "end": 3350.38, "text": " And let's continue."}, {"start": 3351.96, "end": 3352.7999999999997, "text": " Let's see what else."}, {"start": 3352.7999999999997, "end": 3356.08, "text": " So we now apply the log softmax."}, {"start": 3356.08, "end": 3358.16, "text": " And now this is one of these additional details."}, {"start": 3358.16, "end": 3359.38, "text": " It's kinda important."}, {"start": 3359.38, "end": 3362.2799999999997, "text": " So if the sum of probability over timestamps"}, {"start": 3362.2799999999997, "end": 3365.9, "text": " is above any other token, then sample the timestamp."}, {"start": 3365.9, "end": 3367.36, "text": " So that's how we sample the timestamp."}, {"start": 3367.36, "end": 3368.48, "text": " So let's see this part."}, {"start": 3368.48, "end": 3369.64, "text": " That's kinda important."}, {"start": 3369.64, "end": 3372.94, "text": " So we take the log probabilities,"}, {"start": 3372.94, "end": 3376.44, "text": " which are basically after we apply log softmax,"}, {"start": 3376.44, "end": 3378.52, "text": " and before that we did the masking."}, {"start": 3378.52, "end": 3381.68, "text": " So that means now we have all of the relevant tokens"}, {"start": 3381.68, "end": 3383.52, "text": " can be sampled from."}, {"start": 3383.52, "end": 3386.92, "text": " And as you can see here, we grab only those tokens"}, {"start": 3386.92, "end": 3389.7599999999998, "text": " which correspond to timestamp."}, {"start": 3389.7599999999998, "end": 3391.92, "text": " And we do a sum across those"}, {"start": 3391.92, "end": 3395.84, "text": " just to get their cumulative probability, log probability."}, {"start": 3395.84, "end": 3398.48, "text": " And we do the same thing for all the previous tokens."}, {"start": 3398.48, "end": 3401.56, "text": " As you can see here, we grab them,"}, {"start": 3401.56, "end": 3404.92, "text": " actually we grab the max one, not the sum."}, {"start": 3404.92, "end": 3408.4, "text": " And if the probability of this one is bigger than that one,"}, {"start": 3408.4, "end": 3413.4, "text": " then we basically mask all of the non timestamp tokens."}, {"start": 3414.84, "end": 3415.88, "text": " And then we're certain"}, {"start": 3415.88, "end": 3417.96, "text": " that we'll generate the timestamp information."}, {"start": 3417.96, "end": 3419.88, "text": " So as you can see in this particular instance,"}, {"start": 3419.88, "end": 3422.08, "text": " we are generating, because this is a first step,"}, {"start": 3422.08, "end": 3424.44, "text": " we're generating the zero zero tokens."}, {"start": 3424.44, "end": 3426.44, "text": " So let me remind you here,"}, {"start": 3426.44, "end": 3429.52, "text": " we're actually generating this begin time"}, {"start": 3430.84, "end": 3433.1600000000003, "text": " as the first token, which kinda makes sense."}, {"start": 3433.1600000000003, "end": 3435.52, "text": " Okay, so let's go back to the code."}, {"start": 3435.52, "end": 3437.56, "text": " Let's exit here."}, {"start": 3437.56, "end": 3442.16, "text": " Let me just shift F11, we're out."}, {"start": 3442.16, "end": 3443.4, "text": " Those are the logits."}, {"start": 3443.4, "end": 3447.4, "text": " And now we have this, what's this update function?"}, {"start": 3448.2, "end": 3452.6, "text": " This is just basically wrapping up the,"}, {"start": 3452.6, "end": 3456.4, "text": " okay, so this is actually the beam decoding part,"}, {"start": 3456.4, "end": 3458.2, "text": " collecting the candidates, blah, blah, blah."}, {"start": 3458.2, "end": 3460.04, "text": " As I said, I'm gonna skip all of those."}, {"start": 3460.04, "end": 3463.12, "text": " Let's just kinda assume that as a black box,"}, {"start": 3463.12, "end": 3467.88, "text": " we end up sampling, I guess, five candidates."}, {"start": 3467.88, "end": 3469.72, "text": " So let's see how that's gonna look like."}, {"start": 3469.72, "end": 3474.24, "text": " So tokens is gonna be five, four,"}, {"start": 3474.24, "end": 3476.16, "text": " and before that we had five, three."}, {"start": 3476.16, "end": 3480.2, "text": " And that's because, yeah, we now have five new candidates"}, {"start": 3480.2, "end": 3482.12, "text": " that are highly likely."}, {"start": 3482.12, "end": 3484.7999999999997, "text": " And we can just skip everything else here."}, {"start": 3484.7999999999997, "end": 3486.7999999999997, "text": " I think we're safe to skip all of this."}, {"start": 3486.7999999999997, "end": 3491.24, "text": " I'm just gonna hit F5 to the break point, exit,"}, {"start": 3491.24, "end": 3492.42, "text": " and that's it."}, {"start": 3492.42, "end": 3494.48, "text": " So now we just keep on doing the same thing."}, {"start": 3494.48, "end": 3499.0, "text": " So we just keep on, so we get the logits."}, {"start": 3499.84, "end": 3502.92, "text": " Whoops, let's just exit from the text decoder."}, {"start": 3504.42, "end": 3509.42, "text": " So we get the logits, and then now we'll be skipping this"}, {"start": 3509.6800000000003, "end": 3511.2000000000003, "text": " because we're not in the zero step."}, {"start": 3511.2000000000003, "end": 3514.36, "text": " That's the only step where we calculate the no caption props."}, {"start": 3514.36, "end": 3517.28, "text": " And then we just keep on sampling until we hit"}, {"start": 3517.28, "end": 3521.6800000000003, "text": " the end of transcription token, pretty much."}, {"start": 3521.68, "end": 3525.04, "text": " So let me now remove all of the break points."}, {"start": 3525.04, "end": 3527.3199999999997, "text": " Let's put a break point here."}, {"start": 3527.3199999999997, "end": 3530.3199999999997, "text": " Let's exit and let's see what's gonna happen."}, {"start": 3532.3599999999997, "end": 3533.98, "text": " Okay, so here we are."}, {"start": 3533.98, "end": 3536.2, "text": " So tokens, let's see the shape."}, {"start": 3536.2, "end": 3540.7599999999998, "text": " We have five and each of them has 33 tokens, okay?"}, {"start": 3540.7599999999998, "end": 3542.44, "text": " We return back the probabilities"}, {"start": 3542.44, "end": 3544.44, "text": " as well as the no caption probabilities."}, {"start": 3544.44, "end": 3546.12, "text": " Okay, we can skip all of this."}, {"start": 3549.08, "end": 3551.06, "text": " I'm not gonna focus on that too much."}, {"start": 3551.06, "end": 3553.6, "text": " And let's just return back to result."}, {"start": 3553.6, "end": 3556.0, "text": " So let's return back to result."}, {"start": 3556.0, "end": 3557.52, "text": " Here we are."}, {"start": 3557.52, "end": 3559.2799999999997, "text": " So now we have the results."}, {"start": 3559.2799999999997, "end": 3562.42, "text": " And now we have this fallback mechanism,"}, {"start": 3562.42, "end": 3563.56, "text": " as you can see here."}, {"start": 3563.56, "end": 3567.68, "text": " And in our case, we'll just be directly,"}, {"start": 3567.68, "end": 3569.2799999999997, "text": " we'll get directly to the results"}, {"start": 3569.2799999999997, "end": 3573.12, "text": " and we will not enter this branch,"}, {"start": 3573.12, "end": 3578.12, "text": " which basically would force us to do decoding again"}, {"start": 3578.12, "end": 3581.24, "text": " with a different temperature."}, {"start": 3581.24, "end": 3582.96, "text": " So because we satisfied these conditions"}, {"start": 3582.96, "end": 3585.56, "text": " and that's that the compression ratio"}, {"start": 3585.56, "end": 3590.2, "text": " is not higher than this threshold 2.4,"}, {"start": 3590.2, "end": 3591.7999999999997, "text": " which means we are not generating something"}, {"start": 3591.7999999999997, "end": 3593.3599999999997, "text": " that's super random."}, {"start": 3593.3599999999997, "end": 3595.8399999999997, "text": " And we're also passing this test here."}, {"start": 3595.8399999999997, "end": 3597.88, "text": " The log probability, the average log probability"}, {"start": 3597.88, "end": 3599.2799999999997, "text": " is lower than minus one."}, {"start": 3599.2799999999997, "end": 3601.6, "text": " Because of that, we can just kind of,"}, {"start": 3601.6, "end": 3603.06, "text": " we are already good."}, {"start": 3603.06, "end": 3606.7599999999998, "text": " We have the correct transcription and we can exit here."}, {"start": 3606.76, "end": 3610.36, "text": " We check here whether we have the no caption threshold"}, {"start": 3610.36, "end": 3611.5200000000004, "text": " and it's set."}, {"start": 3611.5200000000004, "end": 3614.6400000000003, "text": " And because of that, we now check, so this is the check."}, {"start": 3614.6400000000003, "end": 3618.1200000000003, "text": " Is the no caption probability higher than the threshold?"}, {"start": 3618.1200000000003, "end": 3620.1600000000003, "text": " If it is, then we should skip."}, {"start": 3620.1600000000003, "end": 3622.1800000000003, "text": " And that means that we would literally skip"}, {"start": 3622.1800000000003, "end": 3625.28, "text": " the whole segment here and continue."}, {"start": 3625.28, "end": 3628.0200000000004, "text": " Continue basically transcribing the next segment."}, {"start": 3628.0200000000004, "end": 3630.6800000000003, "text": " Okay, so in our case, it's not the case."}, {"start": 3630.6800000000003, "end": 3634.6000000000004, "text": " So we just ignore this part here."}, {"start": 3634.6, "end": 3637.88, "text": " Now there is some part about the timestamps here."}, {"start": 3637.88, "end": 3640.3199999999997, "text": " You can see here tokens greater than equal"}, {"start": 3640.3199999999997, "end": 3641.8399999999997, "text": " timestamp begin."}, {"start": 3641.8399999999997, "end": 3645.2, "text": " Because the timestamp token have this nice property"}, {"start": 3645.2, "end": 3646.52, "text": " that they are sequential."}, {"start": 3646.52, "end": 3648.66, "text": " So that means that the beginning one,"}, {"start": 3648.66, "end": 3651.56, "text": " the zero zero moment of the transcription"}, {"start": 3651.56, "end": 3656.56, "text": " is maybe on has IDN and then all of the bigger timestamps"}, {"start": 3658.2, "end": 3663.2, "text": " have even bigger basically index in the embedding table."}, {"start": 3663.2, "end": 3664.48, "text": " And that's why this is gonna work."}, {"start": 3664.48, "end": 3667.72, "text": " So this is gonna fetch all of those tokens"}, {"start": 3667.72, "end": 3669.92, "text": " that are timestamp tokens."}, {"start": 3669.92, "end": 3672.04, "text": " And then we basically use that information"}, {"start": 3672.04, "end": 3673.92, "text": " to add some segments."}, {"start": 3673.92, "end": 3675.48, "text": " This is not that by looking,"}, {"start": 3675.48, "end": 3678.76, "text": " go through the code at your own pace."}, {"start": 3678.76, "end": 3683.32, "text": " But I'm gonna return back to results and here we are."}, {"start": 3683.32, "end": 3686.78, "text": " So we can now tokenizer decode all tokens."}, {"start": 3686.78, "end": 3689.48, "text": " So if we were to print this, you can see,"}, {"start": 3689.48, "end": 3690.94, "text": " and so my fellow Americans,"}, {"start": 3690.94, "end": 3693.2400000000002, "text": " ask not what your country can do for you."}, {"start": 3693.24, "end": 3695.2799999999997, "text": " Ask what you can do for your country."}, {"start": 3695.2799999999997, "end": 3698.04, "text": " So that's the, if you remember the clip"}, {"start": 3698.04, "end": 3701.72, "text": " from the beginning of this coding session,"}, {"start": 3701.72, "end": 3706.3399999999997, "text": " that's precisely what we were playing."}, {"start": 3706.3399999999997, "end": 3708.12, "text": " So here we just saved that as the TXT"}, {"start": 3708.12, "end": 3709.9599999999996, "text": " and we also saved this format where we have"}, {"start": 3709.9599999999996, "end": 3713.3199999999997, "text": " the timestamp information additionally"}, {"start": 3713.3199999999997, "end": 3715.2799999999997, "text": " aside from the transcription."}, {"start": 3715.2799999999997, "end": 3717.0, "text": " Guys, that's pretty much it."}, {"start": 3717.0, "end": 3719.64, "text": " Hopefully you saw, I mean, the details of the code"}, {"start": 3719.64, "end": 3723.0, "text": " are complex, but the idea is if you've bear with me,"}, {"start": 3723.0, "end": 3725.36, "text": " up until this point, congrats."}, {"start": 3725.36, "end": 3728.52, "text": " Hopefully you've seen how this intricate system here works."}, {"start": 3729.4, "end": 3731.04, "text": " There is a lot of heuristics,"}, {"start": 3731.04, "end": 3733.8, "text": " but the system is ultimately very elegant"}, {"start": 3733.8, "end": 3737.12, "text": " in the sense that it uses off the shelf transformer module"}, {"start": 3737.12, "end": 3740.6, "text": " and basically just leverages huge data"}, {"start": 3740.6, "end": 3742.04, "text": " and some decoding heuristics."}, {"start": 3742.04, "end": 3743.6, "text": " Cool, so if you like this video,"}, {"start": 3743.6, "end": 3747.12, "text": " consider subscribing, sharing it out with your friends"}, {"start": 3747.12, "end": 3749.0, "text": " and see you next time."}, {"start": 3749.0, "end": 3754.0, "text": " So until then, thanks for watching."}]