How do transformers work? (Attention is all you need)
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video, I give you a semi-quick tour through the "Attention is all you need" paper. The paper that introduced the first-ever transformer model! 

I also show you some cool blogs along the way and my half-baked implementation of the original transformer model.

You'll learn about:
✔️ How the original transformer model works

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ The Annotated Transformer blog: http://nlp.seas.harvard.edu/2018/04/03/attention.html
✅ Jay Alammar's blog: https://jalammar.github.io/illustrated-transformer/
✅ Original paper: https://arxiv.org/abs/1706.03762
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
0:00 prerequisite theory and my semi-done transformer implementation
1:40 High-level overview of the paper
2:55 Visualization of positional encodings (my code)
5:07 Attention mask (no looking forward!)
7:35 Optimizer
10:20 Multi-head attention in depth
15:15 A glimpse at the code implementation
17:49 Training procedure - machine translation
18:09 Na ja, wie geht's?

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#transformer #attention #deeplearning
 What's up folks? So over the last month, I've been reading and learning about transformers. So I read a bunch of research papers, I started implementing the original transformer paper from scratch just a week ago. So I thought while it's still fresh in my memory, I should go ahead and do a small deep dive into the Attention Is All You Need paper. So without further ado, let's jump into it. So let me open the paper, Attention Is All You Need. So actually before I go and do a high level overview of this paper, I strongly recommend you go and read these three blogs, which I already briefly mentioned in my previous video, which I'll link somewhere here. So it's from Jay Elmer, and they are super useful if you're new to transformers and to your attention mechanisms and stuff like that. So this one is really nice, introduces you to the concept of word to back to like word embeddings in general. And then there is this sec to sec tutorial, which is also really nice. And finally, the most relevant one is this illustrated transformer, which is really, really nice. So take your time, read through those, and then you'll have the required prerequisite knowledge for understanding this video. Although you can try and do it without reading the blogs, I just strongly recommend you go through the blogs. Also, so yeah, I already said this. So like a week ago, I started implementing the original transformer paper, and you can see it here in my coding history. So hopefully all of those nitty gritty details will be like fresh in my RAM. So without further ado, let's jump into the paper. Okay, so let's first do a short high level overview of the whole paper. So in 2017, a team from Google published this paper called Attention is All You Need. And they basically introduced this novel architecture called the transformer. The main idea there was that they stopped using RNNs and CNNs and instead used a simple architecture, but they used attention in a really clever way that enabled them to have models, transformers, which are much more parallelizable, and thus more compute optimal than previous architectures. And they basically show here that, yeah, they basically say, hey, this is better than LSTMs and better than GRUs in modeling long range dependencies among words, and also much more compute optimal. You can see the architecture here, and how it functions on the high level is you basically take the input sentence, you convert it into embeddings, you add something called positional encodings, which I'll show you right now because I've been coding that up from scratch on my own. So here is my implementation and just visualize it because it helped me also understand better how these look like. And what you basically see here is that every single row is something that gets added to the token representation. And using this information, the model knows where all of the tokens came from, otherwise the word order would get lost. So this is really important, and they didn't use the learned representation for this positional encodings, they just used this, these are basically sine and cosine functions whose frequencies form a geometric progression. And it turned out to be as good as learnable positional encodings, which they showed in ablation studies later in the paper. Okay, enough of that, let me go back to the paper. So then what you do is you pass those representations into multi-head attention, which I'll go into a bit more depth a bit later, and they then apply something called point-wise feed-forward network. Finally, the output encoder representations are fed into the decoder, which is super similar to encoder part, except that it has this middle multi-head attention, which actually attends also to encoder representations, and not only to previous representations from the same stack. So that's the difference. And this bottom multi-head attention also uses something called masking. Basically, the current token can only see representations that came before it. And that makes sense because Transformer was used to do some machine translation, so translating from English to German and English to French, and thus it doesn't make any sense to see the future words because you're trying to predict the future words. That's why the mask is used. And I can show you that here, just a second. So this is how the mask looks like. You basically put some really big negative numbers before the softmax in the attention module, and that causes masking of token representations. Okay, so that was a brief overview of the architecture. Yeah, finally, you have the linear layer and softmax, which creates a distribution, and you're just trying to minimize the distribution neither using cross-entropy loss or KL divergence in the case of soft labeling, which I'll explain a bit later. And hopefully, I know this probably sounds gibberish at the moment, but just stick with me, and you'll get to kind of slowly learn all of those details in time. Okay, so attention, I already mentioned it. They are using something called scaled.productattention. I'll explain that a bit later. And point-wise feedforward networks, I mentioned those. So basically, what they do is you have a single network, and it processes a single token representation, and that's it. You just train a single network, and you just do something like a for loop over all of the token representations. Embeddings and softmax, positional encoding, I showed you how this looks like. So basically, they use these sine and cosine functions with frequencies which form geometric progression. And then they argue why self-attention is good. So why it's good is it's computationally really, a really, really nice thing. So basically, if you compare the second row recurrent network with self-attention, you have that the maximum path length is O of one for transformer, whereas it's O of n for RNNs. And similarly, complexity of all areas, n squared times d, where n is the number of input tokens, which is usually smaller than d, which is the dimension of the model, which was 512 in the baseline model. So that's why it's, so basically, attention plus highly parallelizable, that's what's the secret sauce of transformers. Okay, they then just, they train this model for translation tasks, as I already mentioned, and they achieved so does, so state of the art, in both English to German and English to French translation tasks. They also use some really interesting optimizing learning rate schedule. And I can show you that briefly in this super nice blog called the, let me see if I can find it, the annotated transformer, and I'll link the link down in the description. So basically, this is how it looks like. You basically have, you linearly increase the learning rate initially for the warm up number of steps, and then you start decaying them proportional to, proportionally to this inverse square root of the number of steps. So that's how the learning rate schedule looks like for their transformer, and they use a simple, like a pretty popular optimizer, like first order optimizer called Atom. Finally, they did some regularizations, dropouts, label smoothing, which, simply said, you usually wanna match like a one hot vector. What label smoothing does is instead of putting one on the ground truth word, you put 0.9, and you just spread out the rest of the mass across the rest of the vocabulary, and that's it in a nutshell. So I mentioned machine translation, they did ablation study, this part is interesting. So they figured out that bigger models are better, and like fast forward to 2020, you know this, and you have GPT family of models where GPT-3 has 175 billion parameters, and you have BERT and BERT derivatives, and all of those models are huge. Compared to this one, I think it was, the original transformer had around 100 million parameters, so compare that to 175 billion today in 2020. And finally, they conclude, saying they have ideas of using this same architecture also in vision and speech recognition, and again, fast forward to 2020, you have this visual transformer which performs really good, really nice with images, and that's super exciting, although only in the big data regime, but still. Finally, they have some nice visualizations here showing how different multi-hat attention modules attend different words in the previous layers, so you can see here that the masking, this word masking, oh sorry, making, is focusing a lot on more difficult, which makes sense if you look at this semantically, and yeah, there are just nice, interesting emerging patterns that appear when you train these transformers and networks. So with that being said, let's go a bit into more depth about how the multi-hat attention actually works. Okay, let me try to explain how multi-hat attention works, and I'll use two methods here. I'll first use JLMR's visualizations because they're really nice, and I'll use my own code I've been developing over the last couple of days. So let's see how it works. So basically, you have, so let me open the paper. We said you convert the sentence into a sequence of embeddings, right? And you're basically here. And now, let me show you how the multi-hat attention works. So these are the embeddings. So you have, say, thinking machines, and in real world, you won't actually have a single vector for a single word. There is something called tokenization, and concretely, Transformer used something called byte-pair encoding, meaning thinking will actually be split maybe into two vectors, like one which actually corresponds to THIN, I don't know, or the second one for the KI, and the third one for NG, whatever. So but basically, at the end, you have a sequence of vectors. And what you do now is you apply three neural networks. One will produce something called queries, the second one will produce something called keys, and the third one will produce values. And now, you basically will use the same network for every single token vector to produce queries. So there are only three networks in one multi-hat attention, and there is one more, like at the output, but I'll mention the one a bit later. So what you do now is you wanna find how similar query from a particular word, say, query one, is similar to keys of every single token in a sentence. So you do a dot product, so that's more generally called a scoring function, but the Transformer paper used something called scale dot product scoring function. So you do a dot product between query one and key one, and you get a number, say, 14. And then you do a dot product between query one and key two, and you get 12, and those are scores. And now you do softmax, which makes the sum goes to one, and you get 0.88 and 0.12, and basically, you take those numbers, you multiply the value vectors, value one, value two, and Jay nicely visualized it here, so because this is 0.12, basically this one will be dimmer, and then you sum them up, and you get Z1. So that's how the attention works. But now, the only difference with multi-hat attention is basically you split this query one. In the baseline Transformer model, it has 512 dimensions. Once you split them into eight hats, that's the number I used in the paper, you get, so one query one will actually have 64 dimensions. And then you do the same procedure I just described, but independently for all of those eight hats. And once you get Z1s for all of those eight hats, you just concatenate them together, and you do another feed forward through a neural network, and you get the final result. And that's when you get to this part here before addition with this residual connection. So that's how it works in a nutshell. Now, I mentioned the scoring functions, I mentioned dot product. Why do you do a dot product between query one and V1 and V2, whatever? The reason is that gives you a similarity between those vectors. So what I mean by that, let me use Google for this. So this is a 2D case. So basically, I told you, so those vectors are like, so query one and key one or key two, they have 512 dimensions. But if you kind of maybe project them into two dimensions, and you get something like this. And now the smaller the angle, the more similar the vectors are. And basically, so the smaller the angle, you'll get higher score here. And basically, that means you'll attend more to that particular representation. So I hope this didn't confuse you too much, because that wasn't the point. But yeah, let me know if this helped or it did not. Just reiterating once more, you take queries and you take keys, and you wanna see how similar they are. And if certain query is really similar to certain key, then you wanna take into account that vector with a high score. And basically, if vectors are orthogonal, the score will be zero. If vectors are like even on the opposite side, then you'll have a negative score. So that's basic cosine distance, basically. Jumping into code, I wanna show you how it looks like like in flash. So as I said here, so you have these query, key, and value nets. You apply them separately on all of those tokens, and you get query, key, and value batches. And once you do that, you pass that into the attention, oops, you pass that into the attention function, which does some matrix multiplication. And that way, you basically apply attention to all of the tokens and through all of the batches in a really optimal way, but it's harder to understand than by looking at single batch and examples that I showed you in the blog. Then we have potentially some masking operation. And as I mentioned, in case you wanna ignore a certain representation, you just put negative infinity, oops, this should be negative here. It's still a prototype code, so forgive me for this. And then you apply softmax to scoring functions, you get attention weights. And also one more detail is that you sometimes wanna drop certain attention weights, which means certain attention where we get pulled to zero. And that's one more nitty gritty detail of implementation. And then you do once more matrix multiplication between attention weights and between those values. And here is where you get those, so here is where you get those, let me just open this up, these vectors, and now you wanna concatenate them and do one more pass through a neural network to get the final result. And that's exactly what I do here. So intermediate representations, and then we change the view of those, and finally we do another projection and you get the final results. So I hope this wasn't too hard to follow along. If it was, please let me know in the comments. And I'll be open sourcing this code in a week or two, so you'll be able to go through the code at your own pace and read through the blogs at your own pace. So hopefully all of this together will help you grasp how all of this works and fits together. Finally, I just wanna give you a feeling of how the actual machine translation training for the original transformer looked like. So you basically have two corpuses of text. You have, say, German and you have English. And those corpuses are synced, meaning you have one sentence in German and there is a corresponding sentence in English. So for example, Wissen Sie eines der großen Vergnügen beim Reisen und eine der Freuden bei der ethnographischen Forschung ist, gemeinsam mit den Menschen zu leben, die sich noch an die alten Tagen erinnern können. That was a bit of German for you. And you basically have a corresponding sentence like here on the other side of the document, if I open it up, you know one of the intense pleasures of travel, blah, blah, blah. So having those two synced, so basically if you're translating from English to German, the German sentence is something called like the gold translation. So somebody like a professional translator translated the sentence. Okay, so once you have those two sentences that correspond to each other, what you do is the following. So you have the transformer. You take the English sentence, you tokenize it, you embed it. You do a forward pass through the encoder module here and you get representations. On the other side, you take a German sentence, you again tokenize it. But now what you do is you set, you put as a prefix, you put this start of sentence token. So that's here. And then you do forward pass here. And finally, the gold translation, just you convert that into soft labeled distributions and you just do KL divergence. And then doing back prop will train and tune the weights of the whole transformer model and that's a single training step. The important part is that you wanna use those masks so that a certain token, like say a token that's here, cannot see tokens that are coming afterwards. That being said, if you found this video useful, consider subscribing to this channel and hit the bell icon to get notified when I upload a new video. And until next time, keep learning deep. And I'll see you next time.
[{"start": 0.0, "end": 1.44, "text": " What's up folks?"}, {"start": 1.44, "end": 5.24, "text": " So over the last month, I've been reading"}, {"start": 5.24, "end": 7.08, "text": " and learning about transformers."}, {"start": 7.08, "end": 9.28, "text": " So I read a bunch of research papers,"}, {"start": 9.28, "end": 12.040000000000001, "text": " I started implementing the original transformer paper"}, {"start": 12.040000000000001, "end": 14.08, "text": " from scratch just a week ago."}, {"start": 14.08, "end": 16.54, "text": " So I thought while it's still fresh in my memory,"}, {"start": 16.54, "end": 19.080000000000002, "text": " I should go ahead and do a small deep dive"}, {"start": 19.080000000000002, "end": 21.72, "text": " into the Attention Is All You Need paper."}, {"start": 21.72, "end": 24.240000000000002, "text": " So without further ado, let's jump into it."}, {"start": 24.240000000000002, "end": 27.560000000000002, "text": " So let me open the paper, Attention Is All You Need."}, {"start": 27.56, "end": 30.2, "text": " So actually before I go and do a high level overview"}, {"start": 30.2, "end": 33.6, "text": " of this paper, I strongly recommend you go"}, {"start": 33.6, "end": 37.0, "text": " and read these three blogs, which I already briefly mentioned"}, {"start": 37.0, "end": 40.04, "text": " in my previous video, which I'll link somewhere here."}, {"start": 40.04, "end": 45.04, "text": " So it's from Jay Elmer, and they are super useful"}, {"start": 46.3, "end": 49.72, "text": " if you're new to transformers and to your attention mechanisms"}, {"start": 49.72, "end": 50.56, "text": " and stuff like that."}, {"start": 50.56, "end": 54.44, "text": " So this one is really nice, introduces you to the concept"}, {"start": 54.44, "end": 57.76, "text": " of word to back to like word embeddings in general."}, {"start": 57.76, "end": 61.04, "text": " And then there is this sec to sec tutorial,"}, {"start": 61.04, "end": 62.16, "text": " which is also really nice."}, {"start": 62.16, "end": 64.9, "text": " And finally, the most relevant one is this illustrated"}, {"start": 64.9, "end": 68.0, "text": " transformer, which is really, really nice."}, {"start": 68.0, "end": 70.96, "text": " So take your time, read through those,"}, {"start": 70.96, "end": 75.72, "text": " and then you'll have the required prerequisite knowledge"}, {"start": 75.72, "end": 77.53999999999999, "text": " for understanding this video."}, {"start": 79.12, "end": 81.88, "text": " Although you can try and do it without reading the blogs,"}, {"start": 81.88, "end": 85.39999999999999, "text": " I just strongly recommend you go through the blogs."}, {"start": 85.39999999999999, "end": 88.36, "text": " Also, so yeah, I already said this."}, {"start": 88.36, "end": 90.67999999999999, "text": " So like a week ago, I started implementing"}, {"start": 90.67999999999999, "end": 92.19999999999999, "text": " the original transformer paper,"}, {"start": 92.19999999999999, "end": 94.38, "text": " and you can see it here in my coding history."}, {"start": 94.38, "end": 96.44, "text": " So hopefully all of those nitty gritty details"}, {"start": 96.44, "end": 98.36, "text": " will be like fresh in my RAM."}, {"start": 98.36, "end": 102.1, "text": " So without further ado, let's jump into the paper."}, {"start": 102.1, "end": 105.64, "text": " Okay, so let's first do a short high level overview"}, {"start": 105.64, "end": 106.72, "text": " of the whole paper."}, {"start": 106.72, "end": 110.47999999999999, "text": " So in 2017, a team from Google published this paper"}, {"start": 110.48, "end": 112.68, "text": " called Attention is All You Need."}, {"start": 112.68, "end": 115.04, "text": " And they basically introduced this novel architecture"}, {"start": 115.04, "end": 116.96000000000001, "text": " called the transformer."}, {"start": 116.96000000000001, "end": 120.68, "text": " The main idea there was that they stopped using RNNs"}, {"start": 120.68, "end": 124.4, "text": " and CNNs and instead used a simple architecture,"}, {"start": 124.4, "end": 127.96000000000001, "text": " but they used attention in a really clever way"}, {"start": 127.96000000000001, "end": 131.84, "text": " that enabled them to have models, transformers,"}, {"start": 131.84, "end": 134.36, "text": " which are much more parallelizable,"}, {"start": 134.36, "end": 138.66, "text": " and thus more compute optimal than previous architectures."}, {"start": 138.66, "end": 143.34, "text": " And they basically show here that, yeah,"}, {"start": 143.34, "end": 148.34, "text": " they basically say, hey, this is better than LSTMs"}, {"start": 148.54, "end": 152.7, "text": " and better than GRUs in modeling long range dependencies"}, {"start": 152.7, "end": 156.12, "text": " among words, and also much more compute optimal."}, {"start": 159.57999999999998, "end": 161.66, "text": " You can see the architecture here,"}, {"start": 161.66, "end": 164.9, "text": " and how it functions on the high level"}, {"start": 164.9, "end": 167.7, "text": " is you basically take the input sentence,"}, {"start": 167.7, "end": 170.6, "text": " you convert it into embeddings,"}, {"start": 170.6, "end": 173.28, "text": " you add something called positional encodings,"}, {"start": 173.28, "end": 175.06, "text": " which I'll show you right now"}, {"start": 175.06, "end": 178.54, "text": " because I've been coding that up from scratch on my own."}, {"start": 178.54, "end": 181.94, "text": " So here is my implementation and just visualize it"}, {"start": 181.94, "end": 183.94, "text": " because it helped me also understand"}, {"start": 183.94, "end": 185.82, "text": " better how these look like."}, {"start": 185.82, "end": 190.66, "text": " And what you basically see here is that every single row"}, {"start": 190.66, "end": 195.66, "text": " is something that gets added to the token representation."}, {"start": 195.66, "end": 199.18, "text": " And using this information, the model knows"}, {"start": 199.18, "end": 202.3, "text": " where all of the tokens came from,"}, {"start": 202.3, "end": 204.76, "text": " otherwise the word order would get lost."}, {"start": 204.76, "end": 206.01999999999998, "text": " So this is really important,"}, {"start": 206.01999999999998, "end": 209.34, "text": " and they didn't use the learned representation"}, {"start": 209.34, "end": 213.38, "text": " for this positional encodings, they just used this,"}, {"start": 213.38, "end": 216.32, "text": " these are basically sine and cosine functions"}, {"start": 216.32, "end": 219.32, "text": " whose frequencies form a geometric progression."}, {"start": 219.32, "end": 222.06, "text": " And it turned out to be as good as learnable"}, {"start": 223.54, "end": 224.42, "text": " positional encodings,"}, {"start": 224.42, "end": 227.22, "text": " which they showed in ablation studies later in the paper."}, {"start": 227.22, "end": 230.94, "text": " Okay, enough of that, let me go back to the paper."}, {"start": 230.94, "end": 234.51999999999998, "text": " So then what you do is you pass those representations"}, {"start": 234.51999999999998, "end": 236.17999999999998, "text": " into multi-head attention,"}, {"start": 236.17999999999998, "end": 239.29999999999998, "text": " which I'll go into a bit more depth a bit later,"}, {"start": 239.29999999999998, "end": 241.45999999999998, "text": " and they then apply something"}, {"start": 241.45999999999998, "end": 243.94, "text": " called point-wise feed-forward network."}, {"start": 243.94, "end": 246.44, "text": " Finally, the output encoder representations"}, {"start": 246.44, "end": 247.89999999999998, "text": " are fed into the decoder,"}, {"start": 247.89999999999998, "end": 250.42, "text": " which is super similar to encoder part,"}, {"start": 250.42, "end": 253.89999999999998, "text": " except that it has this middle multi-head attention,"}, {"start": 253.9, "end": 258.78000000000003, "text": " which actually attends also to encoder representations,"}, {"start": 258.78000000000003, "end": 263.74, "text": " and not only to previous representations from the same stack."}, {"start": 263.74, "end": 265.46, "text": " So that's the difference."}, {"start": 265.46, "end": 269.02, "text": " And this bottom multi-head attention"}, {"start": 269.02, "end": 272.14, "text": " also uses something called masking."}, {"start": 272.14, "end": 276.14, "text": " Basically, the current token"}, {"start": 276.14, "end": 279.06, "text": " can only see representations that came before it."}, {"start": 279.06, "end": 282.74, "text": " And that makes sense because Transformer was used"}, {"start": 282.74, "end": 285.74, "text": " to do some machine translation,"}, {"start": 285.74, "end": 289.02, "text": " so translating from English to German and English to French,"}, {"start": 289.02, "end": 292.56, "text": " and thus it doesn't make any sense to see the future words"}, {"start": 292.56, "end": 294.56, "text": " because you're trying to predict the future words."}, {"start": 294.56, "end": 296.08, "text": " That's why the mask is used."}, {"start": 296.08, "end": 300.74, "text": " And I can show you that here, just a second."}, {"start": 301.6, "end": 305.34000000000003, "text": " So this is how the mask looks like."}, {"start": 305.34000000000003, "end": 308.62, "text": " You basically put some really big negative numbers"}, {"start": 308.62, "end": 313.22, "text": " before the softmax in the attention module,"}, {"start": 313.22, "end": 317.56, "text": " and that causes masking of token representations."}, {"start": 318.42, "end": 322.62, "text": " Okay, so that was a brief overview of the architecture."}, {"start": 322.62, "end": 325.78000000000003, "text": " Yeah, finally, you have the linear layer and softmax,"}, {"start": 325.78000000000003, "end": 327.34000000000003, "text": " which creates a distribution,"}, {"start": 327.34000000000003, "end": 330.3, "text": " and you're just trying to minimize the distribution"}, {"start": 330.3, "end": 334.18, "text": " neither using cross-entropy loss or KL divergence"}, {"start": 334.18, "end": 336.5, "text": " in the case of soft labeling,"}, {"start": 336.5, "end": 338.14, "text": " which I'll explain a bit later."}, {"start": 338.14, "end": 340.62, "text": " And hopefully, I know this probably sounds gibberish"}, {"start": 340.62, "end": 342.3, "text": " at the moment, but just stick with me,"}, {"start": 342.3, "end": 344.78, "text": " and you'll get to kind of slowly learn"}, {"start": 344.78, "end": 347.76, "text": " all of those details in time."}, {"start": 347.76, "end": 351.68, "text": " Okay, so attention, I already mentioned it."}, {"start": 351.68, "end": 355.38, "text": " They are using something called scaled.productattention."}, {"start": 355.38, "end": 358.3, "text": " I'll explain that a bit later."}, {"start": 358.3, "end": 362.46, "text": " And point-wise feedforward networks, I mentioned those."}, {"start": 362.46, "end": 366.09999999999997, "text": " So basically, what they do is you have a single network,"}, {"start": 366.1, "end": 370.54, "text": " and it processes a single token representation,"}, {"start": 370.54, "end": 371.38, "text": " and that's it."}, {"start": 371.38, "end": 374.26000000000005, "text": " You just train a single network,"}, {"start": 374.26000000000005, "end": 376.06, "text": " and you just do something like a for loop"}, {"start": 376.06, "end": 377.94, "text": " over all of the token representations."}, {"start": 377.94, "end": 380.3, "text": " Embeddings and softmax, positional encoding,"}, {"start": 380.3, "end": 381.78000000000003, "text": " I showed you how this looks like."}, {"start": 381.78000000000003, "end": 385.74, "text": " So basically, they use these sine and cosine functions"}, {"start": 385.74, "end": 388.74, "text": " with frequencies which form geometric progression."}, {"start": 388.74, "end": 392.3, "text": " And then they argue why self-attention is good."}, {"start": 392.3, "end": 396.1, "text": " So why it's good is it's computationally really,"}, {"start": 396.1, "end": 398.46000000000004, "text": " a really, really nice thing."}, {"start": 398.46000000000004, "end": 403.1, "text": " So basically, if you compare the second row recurrent"}, {"start": 403.1, "end": 405.86, "text": " network with self-attention,"}, {"start": 405.86, "end": 408.94, "text": " you have that the maximum path length is O of one"}, {"start": 408.94, "end": 413.94, "text": " for transformer, whereas it's O of n for RNNs."}, {"start": 414.26, "end": 416.46000000000004, "text": " And similarly, complexity of all areas,"}, {"start": 416.46000000000004, "end": 421.46000000000004, "text": " n squared times d, where n is the number of input tokens,"}, {"start": 421.46, "end": 423.65999999999997, "text": " which is usually smaller than d,"}, {"start": 423.65999999999997, "end": 426.06, "text": " which is the dimension of the model,"}, {"start": 426.06, "end": 428.7, "text": " which was 512 in the baseline model."}, {"start": 430.09999999999997, "end": 432.85999999999996, "text": " So that's why it's, so basically,"}, {"start": 432.85999999999996, "end": 436.02, "text": " attention plus highly parallelizable,"}, {"start": 436.02, "end": 438.71999999999997, "text": " that's what's the secret sauce of transformers."}, {"start": 440.09999999999997, "end": 443.65999999999997, "text": " Okay, they then just, they train this model"}, {"start": 443.65999999999997, "end": 446.5, "text": " for translation tasks, as I already mentioned,"}, {"start": 446.5, "end": 448.85999999999996, "text": " and they achieved so does, so state of the art,"}, {"start": 448.86, "end": 451.86, "text": " in both English to German and English to French"}, {"start": 451.86, "end": 453.92, "text": " translation tasks."}, {"start": 453.92, "end": 456.42, "text": " They also use some really interesting"}, {"start": 456.42, "end": 459.26, "text": " optimizing learning rate schedule."}, {"start": 459.26, "end": 463.18, "text": " And I can show you that briefly in this super nice blog"}, {"start": 463.18, "end": 466.78000000000003, "text": " called the, let me see if I can find it,"}, {"start": 466.78000000000003, "end": 468.58000000000004, "text": " the annotated transformer,"}, {"start": 468.58000000000004, "end": 471.82, "text": " and I'll link the link down in the description."}, {"start": 471.82, "end": 475.06, "text": " So basically, this is how it looks like."}, {"start": 475.06, "end": 477.58000000000004, "text": " You basically have, you linearly increase"}, {"start": 477.58, "end": 481.3, "text": " the learning rate initially for the warm up number of steps,"}, {"start": 481.3, "end": 484.46, "text": " and then you start decaying them proportional to,"}, {"start": 484.46, "end": 487.9, "text": " proportionally to this inverse square root"}, {"start": 487.9, "end": 489.38, "text": " of the number of steps."}, {"start": 489.38, "end": 492.02, "text": " So that's how the learning rate schedule looks like"}, {"start": 492.02, "end": 494.18, "text": " for their transformer, and they use a simple,"}, {"start": 494.18, "end": 497.46, "text": " like a pretty popular optimizer,"}, {"start": 497.46, "end": 499.82, "text": " like first order optimizer called Atom."}, {"start": 501.29999999999995, "end": 503.97999999999996, "text": " Finally, they did some regularizations, dropouts,"}, {"start": 503.98, "end": 508.14000000000004, "text": " label smoothing, which, simply said,"}, {"start": 508.14000000000004, "end": 512.26, "text": " you usually wanna match like a one hot vector."}, {"start": 512.26, "end": 515.38, "text": " What label smoothing does is instead of putting one"}, {"start": 515.38, "end": 518.62, "text": " on the ground truth word, you put 0.9,"}, {"start": 518.62, "end": 520.58, "text": " and you just spread out the rest of the mass"}, {"start": 520.58, "end": 522.94, "text": " across the rest of the vocabulary,"}, {"start": 522.94, "end": 524.84, "text": " and that's it in a nutshell."}, {"start": 526.02, "end": 527.54, "text": " So I mentioned machine translation,"}, {"start": 527.54, "end": 529.86, "text": " they did ablation study, this part is interesting."}, {"start": 529.86, "end": 532.34, "text": " So they figured out that bigger models are better,"}, {"start": 532.34, "end": 537.0600000000001, "text": " and like fast forward to 2020, you know this,"}, {"start": 537.0600000000001, "end": 538.98, "text": " and you have GPT family of models"}, {"start": 538.98, "end": 542.26, "text": " where GPT-3 has 175 billion parameters,"}, {"start": 542.26, "end": 544.58, "text": " and you have BERT and BERT derivatives,"}, {"start": 544.58, "end": 546.32, "text": " and all of those models are huge."}, {"start": 546.32, "end": 548.14, "text": " Compared to this one, I think it was,"}, {"start": 548.14, "end": 551.5, "text": " the original transformer had around 100 million parameters,"}, {"start": 551.5, "end": 555.1, "text": " so compare that to 175 billion today in 2020."}, {"start": 556.02, "end": 561.02, "text": " And finally, they conclude, saying they have ideas"}, {"start": 561.02, "end": 565.9399999999999, "text": " of using this same architecture also in vision"}, {"start": 565.9399999999999, "end": 570.3, "text": " and speech recognition, and again, fast forward to 2020,"}, {"start": 570.3, "end": 573.42, "text": " you have this visual transformer which performs really good,"}, {"start": 573.42, "end": 577.26, "text": " really nice with images, and that's super exciting,"}, {"start": 577.26, "end": 579.88, "text": " although only in the big data regime, but still."}, {"start": 580.9, "end": 583.78, "text": " Finally, they have some nice visualizations here"}, {"start": 583.78, "end": 586.74, "text": " showing how different multi-hat attention modules"}, {"start": 586.74, "end": 591.0600000000001, "text": " attend different words in the previous layers,"}, {"start": 591.0600000000001, "end": 592.82, "text": " so you can see here that the masking,"}, {"start": 592.82, "end": 595.86, "text": " this word masking, oh sorry, making,"}, {"start": 595.86, "end": 598.7, "text": " is focusing a lot on more difficult,"}, {"start": 598.7, "end": 601.98, "text": " which makes sense if you look at this semantically,"}, {"start": 601.98, "end": 605.02, "text": " and yeah, there are just nice, interesting emerging patterns"}, {"start": 605.02, "end": 610.02, "text": " that appear when you train these transformers and networks."}, {"start": 611.5, "end": 615.7, "text": " So with that being said, let's go a bit into more depth"}, {"start": 615.7, "end": 619.58, "text": " about how the multi-hat attention actually works."}, {"start": 619.58, "end": 623.0200000000001, "text": " Okay, let me try to explain how multi-hat attention works,"}, {"start": 623.0200000000001, "end": 625.58, "text": " and I'll use two methods here."}, {"start": 625.58, "end": 628.4200000000001, "text": " I'll first use JLMR's visualizations"}, {"start": 628.4200000000001, "end": 631.1800000000001, "text": " because they're really nice, and I'll use my own code"}, {"start": 631.1800000000001, "end": 633.22, "text": " I've been developing over the last couple of days."}, {"start": 633.22, "end": 635.6800000000001, "text": " So let's see how it works."}, {"start": 635.6800000000001, "end": 640.58, "text": " So basically, you have, so let me open the paper."}, {"start": 640.58, "end": 642.6600000000001, "text": " We said you convert the sentence"}, {"start": 642.6600000000001, "end": 645.5600000000001, "text": " into a sequence of embeddings, right?"}, {"start": 645.56, "end": 647.2199999999999, "text": " And you're basically here."}, {"start": 647.2199999999999, "end": 650.38, "text": " And now, let me show you how the multi-hat attention works."}, {"start": 651.3, "end": 653.26, "text": " So these are the embeddings."}, {"start": 653.26, "end": 655.18, "text": " So you have, say, thinking machines,"}, {"start": 655.18, "end": 658.38, "text": " and in real world, you won't actually have"}, {"start": 658.38, "end": 660.42, "text": " a single vector for a single word."}, {"start": 660.42, "end": 662.3, "text": " There is something called tokenization,"}, {"start": 662.3, "end": 664.6999999999999, "text": " and concretely, Transformer used something"}, {"start": 664.6999999999999, "end": 667.54, "text": " called byte-pair encoding, meaning thinking"}, {"start": 667.54, "end": 670.04, "text": " will actually be split maybe into two vectors,"}, {"start": 670.04, "end": 675.04, "text": " like one which actually corresponds to THIN, I don't know,"}, {"start": 676.5799999999999, "end": 680.4599999999999, "text": " or the second one for the KI,"}, {"start": 680.4599999999999, "end": 682.98, "text": " and the third one for NG, whatever."}, {"start": 682.98, "end": 686.0999999999999, "text": " So but basically, at the end, you have a sequence of vectors."}, {"start": 686.0999999999999, "end": 691.0999999999999, "text": " And what you do now is you apply three neural networks."}, {"start": 691.56, "end": 693.8, "text": " One will produce something called queries,"}, {"start": 693.8, "end": 696.6999999999999, "text": " the second one will produce something called keys,"}, {"start": 696.6999999999999, "end": 699.26, "text": " and the third one will produce values."}, {"start": 699.26, "end": 702.4, "text": " And now, you basically will use the same network"}, {"start": 702.4, "end": 707.4, "text": " for every single token vector to produce queries."}, {"start": 707.7, "end": 711.9, "text": " So there are only three networks in one multi-hat attention,"}, {"start": 711.9, "end": 714.1, "text": " and there is one more, like at the output,"}, {"start": 714.1, "end": 715.9, "text": " but I'll mention the one a bit later."}, {"start": 715.9, "end": 720.34, "text": " So what you do now is you wanna find how similar"}, {"start": 721.3, "end": 724.28, "text": " query from a particular word, say, query one,"}, {"start": 724.28, "end": 729.28, "text": " is similar to keys of every single token in a sentence."}, {"start": 729.66, "end": 732.6999999999999, "text": " So you do a dot product, so that's more generally"}, {"start": 732.6999999999999, "end": 735.86, "text": " called a scoring function, but the Transformer paper"}, {"start": 735.86, "end": 738.6999999999999, "text": " used something called scale dot product scoring function."}, {"start": 738.6999999999999, "end": 743.3399999999999, "text": " So you do a dot product between query one and key one,"}, {"start": 743.3399999999999, "end": 746.24, "text": " and you get a number, say, 14."}, {"start": 746.24, "end": 751.24, "text": " And then you do a dot product between query one and key two,"}, {"start": 751.3399999999999, "end": 753.86, "text": " and you get 12, and those are scores."}, {"start": 753.86, "end": 758.1800000000001, "text": " And now you do softmax, which makes the sum goes to one,"}, {"start": 758.1800000000001, "end": 761.74, "text": " and you get 0.88 and 0.12, and basically,"}, {"start": 761.74, "end": 764.86, "text": " you take those numbers, you multiply the value vectors,"}, {"start": 764.86, "end": 769.22, "text": " value one, value two, and Jay nicely visualized it here,"}, {"start": 769.22, "end": 773.3000000000001, "text": " so because this is 0.12, basically this one will be dimmer,"}, {"start": 773.3000000000001, "end": 775.86, "text": " and then you sum them up, and you get Z1."}, {"start": 775.86, "end": 778.26, "text": " So that's how the attention works."}, {"start": 778.26, "end": 781.44, "text": " But now, the only difference with multi-hat attention"}, {"start": 781.44, "end": 785.2800000000001, "text": " is basically you split this query one."}, {"start": 785.2800000000001, "end": 789.0600000000001, "text": " In the baseline Transformer model, it has 512 dimensions."}, {"start": 789.0600000000001, "end": 791.2600000000001, "text": " Once you split them into eight hats,"}, {"start": 791.2600000000001, "end": 793.84, "text": " that's the number I used in the paper, you get,"}, {"start": 793.84, "end": 796.7800000000001, "text": " so one query one will actually have 64 dimensions."}, {"start": 796.7800000000001, "end": 799.96, "text": " And then you do the same procedure I just described,"}, {"start": 799.96, "end": 802.9000000000001, "text": " but independently for all of those eight hats."}, {"start": 802.9000000000001, "end": 806.34, "text": " And once you get Z1s for all of those eight hats,"}, {"start": 806.34, "end": 808.4200000000001, "text": " you just concatenate them together,"}, {"start": 808.42, "end": 813.42, "text": " and you do another feed forward through a neural network,"}, {"start": 813.5, "end": 815.18, "text": " and you get the final result."}, {"start": 815.18, "end": 819.42, "text": " And that's when you get to this part here"}, {"start": 819.42, "end": 823.36, "text": " before addition with this residual connection."}, {"start": 823.36, "end": 825.76, "text": " So that's how it works in a nutshell."}, {"start": 825.76, "end": 827.9799999999999, "text": " Now, I mentioned the scoring functions,"}, {"start": 827.9799999999999, "end": 829.3, "text": " I mentioned dot product."}, {"start": 829.3, "end": 831.4599999999999, "text": " Why do you do a dot product between query one"}, {"start": 831.4599999999999, "end": 834.04, "text": " and V1 and V2, whatever?"}, {"start": 834.04, "end": 836.9399999999999, "text": " The reason is that gives you a similarity"}, {"start": 836.9399999999999, "end": 838.3199999999999, "text": " between those vectors."}, {"start": 838.32, "end": 841.0600000000001, "text": " So what I mean by that, let me use Google for this."}, {"start": 841.0600000000001, "end": 842.6400000000001, "text": " So this is a 2D case."}, {"start": 842.6400000000001, "end": 846.0200000000001, "text": " So basically, I told you, so those vectors are like,"}, {"start": 846.0200000000001, "end": 849.34, "text": " so query one and key one or key two,"}, {"start": 849.34, "end": 852.12, "text": " they have 512 dimensions."}, {"start": 852.12, "end": 857.12, "text": " But if you kind of maybe project them into two dimensions,"}, {"start": 857.72, "end": 859.0, "text": " and you get something like this."}, {"start": 859.0, "end": 860.82, "text": " And now the smaller the angle,"}, {"start": 860.82, "end": 863.34, "text": " the more similar the vectors are."}, {"start": 863.34, "end": 867.4200000000001, "text": " And basically, so the smaller the angle,"}, {"start": 867.42, "end": 870.66, "text": " you'll get higher score here."}, {"start": 870.66, "end": 873.8199999999999, "text": " And basically, that means you'll attend more"}, {"start": 873.8199999999999, "end": 876.6999999999999, "text": " to that particular representation."}, {"start": 876.6999999999999, "end": 879.9599999999999, "text": " So I hope this didn't confuse you too much,"}, {"start": 879.9599999999999, "end": 881.18, "text": " because that wasn't the point."}, {"start": 881.18, "end": 885.42, "text": " But yeah, let me know if this helped or it did not."}, {"start": 885.42, "end": 887.6999999999999, "text": " Just reiterating once more,"}, {"start": 887.6999999999999, "end": 890.4, "text": " you take queries and you take keys,"}, {"start": 890.4, "end": 892.76, "text": " and you wanna see how similar they are."}, {"start": 892.76, "end": 897.12, "text": " And if certain query is really similar"}, {"start": 897.12, "end": 901.54, "text": " to certain key, then you wanna take into account"}, {"start": 901.54, "end": 903.9, "text": " that vector with a high score."}, {"start": 903.9, "end": 906.38, "text": " And basically, if vectors are orthogonal,"}, {"start": 906.38, "end": 907.7, "text": " the score will be zero."}, {"start": 907.7, "end": 911.34, "text": " If vectors are like even on the opposite side,"}, {"start": 911.34, "end": 912.74, "text": " then you'll have a negative score."}, {"start": 912.74, "end": 917.34, "text": " So that's basic cosine distance, basically."}, {"start": 919.38, "end": 923.5, "text": " Jumping into code, I wanna show you how it looks like"}, {"start": 923.5, "end": 924.8, "text": " like in flash."}, {"start": 924.8, "end": 929.4799999999999, "text": " So as I said here, so you have these query,"}, {"start": 929.4799999999999, "end": 931.06, "text": " key, and value nets."}, {"start": 931.06, "end": 935.4599999999999, "text": " You apply them separately on all of those tokens,"}, {"start": 935.4599999999999, "end": 939.4599999999999, "text": " and you get query, key, and value batches."}, {"start": 941.06, "end": 944.5, "text": " And once you do that, you pass that into the attention,"}, {"start": 945.38, "end": 948.52, "text": " oops, you pass that into the attention function,"}, {"start": 948.52, "end": 951.74, "text": " which does some matrix multiplication."}, {"start": 951.74, "end": 956.74, "text": " And that way, you basically apply attention"}, {"start": 957.22, "end": 960.82, "text": " to all of the tokens and through all of the batches"}, {"start": 960.82, "end": 963.78, "text": " in a really optimal way, but it's harder to understand"}, {"start": 963.78, "end": 967.74, "text": " than by looking at single batch and examples"}, {"start": 967.74, "end": 970.1, "text": " that I showed you in the blog."}, {"start": 970.1, "end": 973.0600000000001, "text": " Then we have potentially some masking operation."}, {"start": 973.0600000000001, "end": 977.3, "text": " And as I mentioned, in case you wanna ignore"}, {"start": 977.3, "end": 980.62, "text": " a certain representation, you just put negative infinity,"}, {"start": 980.62, "end": 983.78, "text": " oops, this should be negative here."}, {"start": 984.74, "end": 989.5, "text": " It's still a prototype code, so forgive me for this."}, {"start": 989.5, "end": 991.82, "text": " And then you apply softmax to scoring functions,"}, {"start": 991.82, "end": 993.26, "text": " you get attention weights."}, {"start": 993.26, "end": 997.78, "text": " And also one more detail is that you sometimes wanna"}, {"start": 997.78, "end": 1001.7, "text": " drop certain attention weights, which means"}, {"start": 1001.7, "end": 1005.44, "text": " certain attention where we get pulled to zero."}, {"start": 1005.44, "end": 1009.2, "text": " And that's one more nitty gritty detail of implementation."}, {"start": 1009.2, "end": 1013.62, "text": " And then you do once more matrix multiplication"}, {"start": 1013.62, "end": 1015.86, "text": " between attention weights and between those values."}, {"start": 1015.86, "end": 1017.9000000000001, "text": " And here is where you get those,"}, {"start": 1019.5400000000001, "end": 1024.02, "text": " so here is where you get those, let me just open this up,"}, {"start": 1024.02, "end": 1027.5800000000002, "text": " these vectors, and now you wanna concatenate them"}, {"start": 1027.5800000000002, "end": 1030.98, "text": " and do one more pass through a neural network"}, {"start": 1030.98, "end": 1032.3, "text": " to get the final result."}, {"start": 1032.3, "end": 1034.44, "text": " And that's exactly what I do here."}, {"start": 1034.44, "end": 1037.06, "text": " So intermediate representations,"}, {"start": 1037.06, "end": 1040.1399999999999, "text": " and then we change the view of those,"}, {"start": 1040.1399999999999, "end": 1042.62, "text": " and finally we do another projection"}, {"start": 1042.62, "end": 1044.74, "text": " and you get the final results."}, {"start": 1044.74, "end": 1049.74, "text": " So I hope this wasn't too hard to follow along."}, {"start": 1050.94, "end": 1053.5, "text": " If it was, please let me know in the comments."}, {"start": 1053.5, "end": 1056.32, "text": " And I'll be open sourcing this code in a week or two,"}, {"start": 1056.32, "end": 1058.62, "text": " so you'll be able to go through the code"}, {"start": 1058.62, "end": 1060.8999999999999, "text": " at your own pace and read through the blogs"}, {"start": 1060.8999999999999, "end": 1062.0, "text": " at your own pace."}, {"start": 1062.0, "end": 1065.86, "text": " So hopefully all of this together will help you"}, {"start": 1065.86, "end": 1070.26, "text": " grasp how all of this works and fits together."}, {"start": 1070.26, "end": 1072.62, "text": " Finally, I just wanna give you a feeling"}, {"start": 1072.62, "end": 1075.4599999999998, "text": " of how the actual machine translation training"}, {"start": 1075.4599999999998, "end": 1077.54, "text": " for the original transformer looked like."}, {"start": 1077.54, "end": 1079.54, "text": " So you basically have two corpuses of text."}, {"start": 1079.54, "end": 1082.54, "text": " You have, say, German and you have English."}, {"start": 1082.54, "end": 1085.02, "text": " And those corpuses are synced,"}, {"start": 1085.02, "end": 1087.34, "text": " meaning you have one sentence in German"}, {"start": 1087.34, "end": 1088.9799999999998, "text": " and there is a corresponding sentence in English."}, {"start": 1088.9799999999998, "end": 1089.9199999999998, "text": " So for example,"}, {"start": 1090.78, "end": 1093.62, "text": " Wissen Sie eines der gro\u00dfen Vergn\u00fcgen beim Reisen"}, {"start": 1093.62, "end": 1096.3, "text": " und eine der Freuden bei der ethnographischen Forschung"}, {"start": 1096.3, "end": 1098.6599999999999, "text": " ist, gemeinsam mit den Menschen zu leben,"}, {"start": 1098.6599999999999, "end": 1101.82, "text": " die sich noch an die alten Tagen erinnern k\u00f6nnen."}, {"start": 1101.82, "end": 1103.54, "text": " That was a bit of German for you."}, {"start": 1103.54, "end": 1105.58, "text": " And you basically have a corresponding sentence"}, {"start": 1105.58, "end": 1109.1799999999998, "text": " like here on the other side of the document,"}, {"start": 1109.1799999999998, "end": 1110.1399999999999, "text": " if I open it up,"}, {"start": 1110.1399999999999, "end": 1112.1399999999999, "text": " you know one of the intense pleasures of travel,"}, {"start": 1112.1399999999999, "end": 1113.1, "text": " blah, blah, blah."}, {"start": 1113.1, "end": 1115.26, "text": " So having those two synced,"}, {"start": 1115.26, "end": 1119.5, "text": " so basically if you're translating from English to German,"}, {"start": 1119.5, "end": 1121.1, "text": " the German sentence is something called"}, {"start": 1121.1, "end": 1122.62, "text": " like the gold translation."}, {"start": 1122.62, "end": 1124.9399999999998, "text": " So somebody like a professional translator"}, {"start": 1124.9399999999998, "end": 1126.54, "text": " translated the sentence."}, {"start": 1126.54, "end": 1128.86, "text": " Okay, so once you have those two sentences"}, {"start": 1128.86, "end": 1130.1799999999998, "text": " that correspond to each other,"}, {"start": 1130.1799999999998, "end": 1131.5, "text": " what you do is the following."}, {"start": 1131.5, "end": 1134.58, "text": " So you have the transformer."}, {"start": 1134.58, "end": 1136.3799999999999, "text": " You take the English sentence,"}, {"start": 1136.3799999999999, "end": 1138.1399999999999, "text": " you tokenize it, you embed it."}, {"start": 1138.1399999999999, "end": 1142.02, "text": " You do a forward pass through the encoder module here"}, {"start": 1142.02, "end": 1143.4199999999998, "text": " and you get representations."}, {"start": 1143.4199999999998, "end": 1145.34, "text": " On the other side, you take a German sentence,"}, {"start": 1145.34, "end": 1147.34, "text": " you again tokenize it."}, {"start": 1147.34, "end": 1149.62, "text": " But now what you do is you set,"}, {"start": 1149.62, "end": 1154.62, "text": " you put as a prefix, you put this start of sentence token."}, {"start": 1155.58, "end": 1156.7399999999998, "text": " So that's here."}, {"start": 1156.7399999999998, "end": 1160.02, "text": " And then you do forward pass here."}, {"start": 1160.02, "end": 1164.6999999999998, "text": " And finally, the gold translation,"}, {"start": 1164.6999999999998, "end": 1168.82, "text": " just you convert that into soft labeled distributions"}, {"start": 1168.82, "end": 1170.6999999999998, "text": " and you just do KL divergence."}, {"start": 1170.6999999999998, "end": 1173.26, "text": " And then doing back prop will train"}, {"start": 1173.26, "end": 1175.34, "text": " and tune the weights of the whole transformer model"}, {"start": 1175.34, "end": 1178.3, "text": " and that's a single training step."}, {"start": 1178.3, "end": 1181.98, "text": " The important part is that you wanna use those masks"}, {"start": 1181.98, "end": 1186.1399999999999, "text": " so that a certain token, like say a token that's here,"}, {"start": 1186.1399999999999, "end": 1190.1, "text": " cannot see tokens that are coming afterwards."}, {"start": 1191.3, "end": 1194.58, "text": " That being said, if you found this video useful,"}, {"start": 1194.58, "end": 1196.58, "text": " consider subscribing to this channel"}, {"start": 1196.58, "end": 1198.6599999999999, "text": " and hit the bell icon to get notified"}, {"start": 1198.6599999999999, "end": 1200.1, "text": " when I upload a new video."}, {"start": 1200.1, "end": 1203.4199999999998, "text": " And until next time, keep learning deep."}, {"start": 1203.42, "end": 1208.42, "text": " And I'll see you next time."}]