Attention Is All You Need (Transformer) | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

I do a detailed walkthrough of how the original transformer works. I use a simple machine translation (English to German) example to explain the inner workings.

You'll learn about:
✔️ Everything there is to know about the transformer :P

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ my implementation: https://github.com/gordicaleksa/pytorch-original-transformer
✅ paper: https://arxiv.org/abs/1706.03762

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable:
00:00 A high-level overview
02:13 tokenization
05:21 embeddings and positional encodings
10:14 encoder preprocessing (splitting into subspaces)
13:20 single MHA head explanation
20:40 pointwise network
23:15 causal masking MHA
28:00 source attending MHA
30:30 projecting into vocab space and loss function
36:30 decoding

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#transformer #attention #vaswani
 What's up? So in this video I thought walking you through a specific example of how the transformer, like one iteration of the training loop for the transformer looks like. And I already did a previous video where I covered the attention is all you need paper, so you can check that one out if you want to know, like a whole, if you have, if you want to have a holistic overview of the whole paper. But here you'll understand how the, like the, how the details work and how the architecture works. So I'll briefly recap how transformers work here. So basically they have, you have two parts, let me take my pencil out. So you have this encoder part on the left, this is the encoder part, and you have the decoder part on the right. And what you basically do is you split your sentence into tokens, you map those into input embeddings, you add up the positional encodings, then you pass those representations here to multi-head attention, followed up with some layer normalization, and there are also some residual connections, and then you do the feed-forward pass here, and you repeat that n times. So where n is equal to 6 in the original paper. So now I'm just walking you through, like giving you a high-level overview, we'll go into every specific detail of the things I just mentioned in a couple of seconds. So then on the other side you have the decoder, you do the same thing, you just take the German sentence, you again tokenize it, embed it, pass it through the, this time the multi-head attention, which has something called causal masking, and I'll get into that a bit later. And then we have this again, multi-head attention, but this one you're attending to the representations from the encoder, and not from the, so you're using those representations as keys and values, and not the ones that come from the decoder. And at the end you just basically project those representations into your output vocabulary space, and you just do simple, like cross-entropy loss or KL divergence, and then you basically backprop. So that's the overview of the transformer, and if you didn't understand every single thing, that's good, because I'm now going to dig deeper into how everything fits together. So the first part that happens is you basically want to learn how to split the sentence. So you want to learn how to split the English sentence into sub-words, and you want to learn how to do the same thing for the German sentence. So you basically have, you start with a huge English corpus, and you start with the second corpus, like the German corpus, that corresponds exactly one-to-one to this corpus. So you basically have a sentence in English, like how are you today, and you have that translation, something called the golden translation, because maybe like humans, like professional translators created this, and so the corresponding sentence would be, wie geht es Ihnen heute in German, and you train something called tokenizer objects, which will learn how to split these sentences and map them into some IDs. And that's where these vocabulary tables come into play. So this is an imaginary English vocab table, and then we have the German one here. So what happens is, so the procedure goes like this. You take the English sentence, and you don't do the simple splitting, like on spaces. That would be something really naive. The original transformer used something called byte-pairing coding, and I won't get into the details of that algorithm. If you're interested in learning more about that, I could create a video, a separate video, so just comment down below if you think that'd be useful. So you will maybe split the word today into separate sub words, so maybe like into something like two and day. And once it knows how to split, and it has the IDs, it can convert the sentence into just a list of IDs. So how are you today gets split into how are you today tokens, so five tokens in total, and we just map them into corresponding IDs. So you can see in the table how it corresponds to two, so we put two here, etc. And we do the same thing for the German sentence, we get as in and hoi-te, so I kind of, I imaginary split this word hoi-te into two substrings. It depends on the tokenizer how it will do this, but let's say it's like this. So what's interesting here is this start of sentence token. So that one is used to prompt the decoder into generating the translation. So basically, you later wants to transform restrained, you just prompted with this start of sentence token, and it will start generating the words here, you'll feed them back into the input, and that's something called autoregressive decoding, autoregressive models. So that's the first part how the sentence gets split into these lists of IDs. Now for the second part. Once you have the English and the German tokenizer object trained, you can do the following. So you basically, what you do is you, let me just, man, let me take something, yeah, okay. So basically you take the list, and you have this thing called input embedding. So basically that's a, so this, so you have a basically a huge matrix, like that's the size of your English vocabulary, so say that's like maybe 30k rows, and every single row correspond to a single word in the English vocab. And you basically have, in the original transformer, these were 512 dimensional vectors. So basically that's the input encoding. And now what you do is you take this sentence, so you have, how are you today? You take the ID, and the ID basically indexes into this table. So if you have two, you'll basically take the second vector, so you'll take this one, and you'll stick it right here, so in the beginning of the encoder. So you'll put the 512 dimensional vector here. You do the same thing for R. You take the third row, you place it here, and that's it. And now the thing is, in the original paper, this matrix was shared with the German matrix, and was shared with softmax weights here. But for the sake of easier explanation, I'll just, and that's how I implemented this in my own project, and I'll link my project, my implementation of the original transformer in the description. So basically split all of these into separate matrices. So once you have that, you do the same thing for German. So basically the German has a separate embedding matrix, which has the size of the, this size corresponds to the German vocab size, so maybe, I don't know, like 35k rows. And this one will be also 512. And you basically repeat the procedure. You take the first word, which is the start of sentence token, which is maybe mapped to, I don't know, like index two, in this matrix. So you take this row, you place it here, and you repeat the procedure for the rest of the words. So that's how the embedding part works. So you just use the IDs as indices into these embedding tables. And these are completely randomly initialized in the beginning. So that's like random, totally random, like maybe uniform sampling or something. It doesn't matter. So that's something that you learn throughout the training. One of the things you learn. We'll see other parameters that get learned. So the second part is this positional encoding and how that works. So that's also a matrix. And this time it's shared between English and German. And it looks something like this. And now it's basically like a, basically a bunch of sinusoids, so sines and cosines, whose frequencies form a geometrical progression. So you can see that the wavelength here is getting bigger than the wavelength here. And basically this is just some heuristic. You can basically also randomly initialize this matrix and learn it throughout the training. But the authors show that both of those approaches are equivalent, but they still stick with this one. So you do the same thing. You basically take the ID of the words, so how, you take two IDs, two, and you basically take this row and you'll stick that row here and you'll just add them up. You just add those two up and you get the new representation. And now you're here. So this is where we are now. So basically do the same thing for other words. And so just find, so this corresponds to the word R. So this is R. So you'll find the ID that's three. You'll take the third row from this column. You'll find the positional encoding. You'll add them up. Simple addition. And you end up with representations. And now we're finally came to this part where the whole logic happens. And yeah, one more thing. The same thing obviously happens for the German part. So you just find embeddings. You use the positional encoding table and you add them up and you end up with token representations here. So that's the first part. And hopefully this was clear enough. So I'll focus on the multi-head attention and I'll ignore the residual connection and the layer normalization for a second and we'll get to them a bit later. So we have five tokens, right? So we have how, R, blah blah blah, day. And all of these are like 512 dimensional vectors. So now what happens is the multi-head attention basically has four matrices that gets learned during the training. So the first three, the important ones are the query, the key matrix, and finally the value matrix. And all of these have dimension the same as these embeddings. So that's 512. So we'll have 512 times 512 for the query matrix and the same goes for the key and value matrices as well. So what we do now is we basically point-wise apply these matrices. So what I mean by that is I take the Q matrix and I multiply this one and this vector and we get, so we get 512 here and we get also 512 here, et cetera, the same here. And we do the same thing with the key matrix. So we end up with three vectors for every single token, right? So, and here we have value. So that's like the query, the key, and the value vectors. So once we are here, now what happens is you basically, because that's a multi-head attention, basically the original paper had eight heads that go in parallel. So you just split these vectors into eight pieces. So this gets split into the parts. So this one has eight times 64, yeah. So we get 64 dimensional vectors here. We take the first chunk from the second token representation as well and we do that for this. So this corresponds to token day. So we basically take the first part, first segment, which has basically 64 dimension as well. And what we do now is we get into the interesting part. So we just group all of these together. So we'll have, so we have eight heads, right? So that's head one, head two, blah, blah. We had eight heads. So we group these first segments and place them here. We group the second pieces. So we group, so this is the second piece. This is already getting really annoying, but hopefully you can see what I mean. So we take the second segment, you group them and place them here and you do the same thing for the last segments and place them here. So now I'm going to zoom out and we come to the really interesting part and that's the attention itself. Okay, so we projected all of these embedding vectors into eight subspaces and let's take a single one. So what happens is, so we ended up with those small vectors now. So this one is 64 and that's the query one. Then we have the key and the value vectors as well. So that's 64, 64. And that's for the first token, that's how, because remember we had the sentence, how are you today? So we repeat the same thing until the last token, that's the day token. It's also 64 and the same thing happens here. Just imagine the other vectors. So this is the interesting part and this is how the attention works. So you take the query vector, so this one, and you basically do a dot product between this query vector and every single key vector. So this vector and you'll do, so imagine one vector being here. So you'll do the dot product between this query and this key and with all the other keys until you get to this key and you just do the dot product. So what you end up with is basically a measure of similarity. So that's how dot product, if you're not familiar, I hope you are, so it's a simple linear algebra. So let's say this is our query vector and let's say this is our key vector. So in this, when you do a dot product between those two, because the angle is so small, basically when you project this one, you add them up, you basically get a huge value. Once the, if the key vector was like orthogonal to the query vector, you'd have like a 90 degree here, orthogonal right, and then you'd have zero score. So basically what happens here, you're measuring the similarity between the query vector and the key vectors. And the closer the key vectors are in these subspaces to this query vector, the higher the score. So you basically end up with something like this. Maybe you end up with 3.2 score here. Maybe you end up for the second token, which was how r, so this is r. Maybe you end up with like, I don't know, like 0.3 similarity and you maybe end up with the last token, you end up with, I don't know, like 2.2. And there's one interesting, one more important thing they did and that's normalization by the square root of the dimension of these heads. So that's 64. So this is dimension. So we have 64 here and that's why we do square root of 64, which is 8. They show that this normalization by the square root of this dimension helps them better train the transformer. Otherwise, they suspect that the, like the huge value, so the bigger the dimension, the bigger the dot products would be and they'd push the softmax into regions where the gradients gets really small and then the training just kind of slows down. So that's why they do this. So they just basically divide the scores here and once you do that, you apply the softmax. So softmax will make this sum go to 1 and so the bigger one will have maybe, after doing the softmax, will have maybe 0.7. This one would be maybe 0.01 and this one would be maybe 0.1. And once you have these values, aside from applying probably like a dropout is usually in practice applied here. So you randomly drop one of these to 0. So maybe this one would go to 0. And once you do that, then you use these scores to finally multiply them, these value vectors by these and you end up with the final representation. So what I mean by that, so we were, so all of these results came for this specific query vector, right? So now we just use this 0.7 and we multiply this value vector with 0.7 and then we multiply this vector with 0 and finally all of these with their corresponding values, the last one will be multiplied by 0.1. We add them up and we get the final representation that's 64 dimension long and it contains the information about all of these surrounding tokens. So that's why attention is really nice and why transformers perform so well. Because they can attend to all of the representations and create a new contextualized representation which is later helpful for the downstream tasks. So that's, you repeat the same thing basically for all of the other tokens. So now you take this query vector, you multiply it by, you do that product between different keys again, you again do the softmax, etc. etc. And you end up with all of the representations. So we end up with five representations here because again remember we have five tokens. So that's how the multi-head attention pretty much works. So now just concatenate all of the eight heads. So we have eight heads, we just group all of these together. So this was the first segment, right? So these came from these parts. So we'll just return them back into the same shape and that's it. We are ready for the final part of the attention. So once you concatenate the vectors, you end up with again 512 because we concatenated all of the eight heads. And we have five of these corresponding to our tokens. How are you today, right? And the last matrix I mentioned, so these are the three ones I showed you so far. But there is one more matrix that's used in the attention module and that's the output matrix. So basically that one again, you just, it's also 512 by 512. It's the, let's call it O, like output matrix. You basically again point-wise project these into 512. So basically just apply this one for this vector and independently apply it to this one. So that's what I mean by point-wise. And this representation is what we get out right here. So that's where we ended up. So now the residual connection would just add up those representations that were here before we did all of these attention thing and they just add it up to our final representations that came out from the, from multiplying by this matrix. That was arguably the most important part of the transformer architecture. And now what happens is, and I'll just ignore the layer normalization. It's similar to batch normalization. It basically just kind of updates all of the weights in the batch so that we can, so the gradients don't get into like, so that we don't have either vanishing or exploding gradients problems basically. It just stabilizes the training. So let me just show you this part with the feed forward net and then like the residual connection and the layer normalization just repeats the same as down here. So we ended up right here and I'll just, for the sake of making this less dirty, just repeat it again here. So again, we have five tokens corresponding to our like tokens. How are you today? And what we do now is we point-wise apply the neural network. Basically that neural network consists out of two matrices and like we have one value like the nonlinear activation layer. Usually transformers have, so as I said, they have two matrices. So the first one just kind of expands this dimension into 4x. Just don't ask me why. It's just people found out this works and that's it. So we have 512. Now we end up with 2048, right? So and then we just apply the second layer and that DOM projects this into 512. So those are the two additional matrices that we are going to learn during the training. So we had the four ones in the multi-head attention layer. We have two matrices here. We have those embedding matrices and we don't learn the positional encodings because they are, we use the sine and cosine heuristic. So that's it. That's how the encoder part of the transformer works. So we ended up here. We just repeat the same procedure six times or n times like in general and we end up with encoder representations of our initial tokens. So we end up with, in our case, with five token representations here. There are two more interesting and important things I want to cover and that's how exactly this part works and how exactly this part works. So it's really similar to this multi-head attention I already explained, but there are some details that are worth covering because people usually get confused by those. The feed forward network is the same and I'll just cover like the third portion is this, how this works. So let me start with this part. So the MAST multi-head attention module. So this is getting really like a nuclear physics whiteboard, except it's not nuclear physics. Okay. Anyways, we have tokens. This time we have German tokens. So we, let's say we have eight of these and they are 512. This one is maybe the start of the sentence token. I don't know how to write today. Okay. Start the sentence. So S okay. And we end up with the last word, which was Hoytas. So we have the substring T here. So the only difference between this multi-head attention and the last one I showed you is this thing called causal masking. And what basically happens is so everything, so for the sake of making this easier, let's assume we have only one head and not eight heads. So what happens during the attention operation here is so we have, so we have a query, we have key and we have value vectors. And now what happens here is this specific token can only look at the representations that came beforehand. So you can only look to itself and to previous. So this one gets mapped to again, value, key and query. So if you take this query, what happens is it will do a dot product between this query and this key and this query and this key, etc. It will even do a dot product between that query and this key for the last token. But now the interesting thing that happens is we just, before we add these, before we pass these to a softmax, so we ended up like maybe some scores like 3.2, I don't know, like 0.1 and this would have maybe, I don't know, 5.4 and even these would have some scores, so maybe 3.6. But we add up like minus infinity here. So we add up minus infinity here and we add up minus infinity to all of the other ones. So this got a clutter a little bit, but I just went ahead and redraw this a little bit to make it a bit clearer. So the only difference is, so now when we have like maybe the query vector that corresponds to the token s, because remember the sentence was vket s enen hoite, so I split this into two substrings and this into two substrings. So the s one, so we take its query and we do, again we just do dot products with different keys and we end up with some scores like maybe 3.2 here, blah blah blah, we end up with maybe 5.1 here, we end up with, I don't know, 2.4 here and we end up with, I don't know, like 0.1. And now what happens is every single score that comes after this current token, we just add up, basically we add up minus infinity. And what it does is basically now we feed this into softmax and after feeding it into softmax it gets pulled to zero and that basically means we won't be using these value vectors in the final representation for this specific token s. So this token, once we get the value, so 3.2 maybe gets mapped to 0.2 after softmax, this one gets mapped to, I don't know, 0.7, so we'll use 0.7 of these value vectors, we'll use 0.2 of these, but we won't use these. And that's what makes it causal. So this one can only see what came previously. And so the final representation only contains the previous representations and the current representation. If you were to do this for the first token, everything would get, so all of the other tokens would get mapped to 0. And so the first token, the start of sentence token, can only use its own representation. So basically nothing happens for this. So basically we just pass the value token here and that's what happens for the first token. And the last one we'll be using all of the representations because it's the last one. Okay, so that was this part. So that was this mask multi-head attention module. We covered that one, checked. Now the last one is this and it's simple. So again, we have German tokens, we get the Sinnenheute, this is the start of sentence, blah blah blah, we end up with the TE token. And we have these representations which ended up being calculated after iterating through six of these encoder layers. So we are using those. We have five of those. So basically we have five of those tokens here. And so this one corresponds to v, no, sorry, how are you today? And now we just use, we basically use these as the key and the value and we use these as the queries. So again, we have three matrices, the query, the key and the value matrix. Query, key, value. We take the query, we create query vectors for the German tokens, but we use key and value and apply them to these tokens. So we end up with key and value representations here all the way down to this token. And now we basically take this query and we do the usual dot product with these keys which will after doing the softmax, we'll know how to combine these value vectors to get the new token. So that's what's different for this module. So it basically combines the value vectors that come out from the encoder. So using the queries that came from the bottom layer of this decoder layer. So that's it. So I intentionally went into a lot of details. I usually love to go through from high level explanation and then slowly and gradually explain the complexity. But here I just wanted to go fully into details and hopefully some of you found this useful. But we are not finished yet. I still have this part to cover and it's pretty easy. So once we end up at the top of this stack, it's quite simple actually. So let's say we end up with eight token representations. So these correspond to start of sentence, blah blah blah. This corresponds to this token TE. And now what we do is we apply, we project these representations which are still 512 as always. We project them into this space that has the dimensionality the same as the German vocabulary size. So that was for example 35k. So that means this one gets projected into 35,000 dimensional vector and that's huge. That means we have a huge matrix here whose dimension is 512 times 35k which is where most of the computation comes into play. And there are many strategies where you can maybe create something called, you can Google it, hierarchical softmax instead of doing this one. Or you can do something called negative sampling, etc. But I won't get into those because those techniques were not mentioned in the original paper. So after you project these, you do the same thing for all of the tokens. And now you just have the ground truth. So basically because we know what the output sentence should be, so that's, we get the sin heute, that was the German sentence. So just to freshen up your memory. So we get the sin heute, okay? So basically what will happen here is we'll have, so we'll take the token v and we'll find the id. And basically what happens is we'll have the ground truth vector here and on the position 3 it will have 1. And here for example this one will be some, won't be a binary vector, it will be, maybe we'll have, I don't know, like 0.7 here and then some random value distribution, right? And basically we do either cross entropy, in this case where we have 1 here, and basically that means if we have 0.7, the loss would be log of 0.7. And if you want to minimize this loss, it should go, this argument better go to 1 because log of 1 equals 0. So the final loss when you're training the transformer is basically a sum of those logs. And I'll get to soft labeling in a bit, but let's assume we're using cross entropy for the moment. And so basically we'll have log of 0.7 here, so this vector will maybe have, let me draw the ground truth one. So let's say this Te is the ground truth vector is 1 here and everything else is 0, but like the distribution was a bit different, so the part that we're interested in is maybe 0.3, so the loss would be log of 0.3 and that loss is even bigger. I mean we just add the minus and then it's big otherwise this is a negative number. So we just sum up those logs and we average them. So basically that's what the loss function is and then you just do the basic gradient based learning using your favorite optimizer like I think they used Adam in the original transformer paper. Yeah they used Adam with a specific learning rate strategy. So that was how it usually works, but there was a slight detail here. They're actually using, they're not using these one-hot vectors as the ground truth, they're using something called soft labeling where basically this would be set to 0.9 and then the rest of the probability mass would be equally distributed evenly across this vector except for the padding token which would always get set to 0. So that's the soft label and then you do something called KL divergence instead of cross entropy loss and you can Google that, I won't get into, this video is already getting too long. So that's it, that's how the whole procedure looks like. So you, just to recap once more, you split the sentences into tokens, you map them into IDs, you look up at those embedding tables, you find the embedding vectors, you just add up those positional encodings. Now you have the initial representations here, now you go through six, you go six times through this encoder stack where you apply the attention and you apply the feedforward net, you end up with the final representations. You do the same thing for the German sentence except that you have the causal mask here and you attend to the encoding vectors in this module and that's it, you finally project into the output vocabulary space and you do backprop on the loss function that consists of KL divergence or cross entropy loss and that's how you do it. I owe you just one more explanation and that's how the, once you train the network using the process I just showed you, how do you do the actual decoding, the translation process. So what you do is you take the, just the input sentence like maybe the same one, how are you today, again you do the same procedure, tokenization, embeddings, blah, blah, blah and you just prompt the model, the decoder with the start of the sentence token and so initially this one would get propagated upwards and we'd get some word here like hopefully it will be the correct word like V and we'll just take that word and we'll embed it here and we'll just repeat the process and that's why this model is called autoregressive. You're basically whatever the output, whatever you have is the output, you just feed it back as the input and you're just autoregressing until you end up. So what's the stopping condition? The stopping condition is that end of sentence token I mentioned here, so basically you, once the network outputs end of sentence token you stop the translation process and that's pretty much it. So, this was a long video, I'm experimenting with this channel obviously, I'm trying to see what works best for you as well as for me, so if you found this video useful please put some comments down in the comment section, I appreciate any feedback like was it too detailed, was it too messy, maybe I can, yeah, I'll try and improve the next time. So, see you next time and subscribe to this channel if you haven't already and click that bell icon to get notified when I upload a new video and until next time, keep learning deep.
[{"start": 0.0, "end": 5.5600000000000005, "text": " What's up? So in this video I thought walking you through a specific example of how the"}, {"start": 5.5600000000000005, "end": 11.0, "text": " transformer, like one iteration of the training loop for the transformer looks like. And I"}, {"start": 11.0, "end": 15.24, "text": " already did a previous video where I covered the attention is all you need paper, so you"}, {"start": 15.24, "end": 18.96, "text": " can check that one out if you want to know, like a whole, if you have, if you want to"}, {"start": 18.96, "end": 24.0, "text": " have a holistic overview of the whole paper. But here you'll understand how the, like the,"}, {"start": 24.0, "end": 29.96, "text": " how the details work and how the architecture works. So I'll briefly recap how transformers"}, {"start": 29.96, "end": 36.44, "text": " work here. So basically they have, you have two parts, let me take my pencil out. So you"}, {"start": 36.44, "end": 41.6, "text": " have this encoder part on the left, this is the encoder part, and you have the decoder"}, {"start": 41.6, "end": 48.16, "text": " part on the right. And what you basically do is you split your sentence into tokens,"}, {"start": 48.16, "end": 53.96, "text": " you map those into input embeddings, you add up the positional encodings, then you pass"}, {"start": 53.96, "end": 61.68, "text": " those representations here to multi-head attention, followed up with some layer normalization,"}, {"start": 61.68, "end": 66.84, "text": " and there are also some residual connections, and then you do the feed-forward pass here,"}, {"start": 66.84, "end": 72.6, "text": " and you repeat that n times. So where n is equal to 6 in the original paper. So now I'm"}, {"start": 72.6, "end": 76.28, "text": " just walking you through, like giving you a high-level overview, we'll go into every"}, {"start": 76.28, "end": 82.12, "text": " specific detail of the things I just mentioned in a couple of seconds. So then on the other"}, {"start": 82.12, "end": 86.80000000000001, "text": " side you have the decoder, you do the same thing, you just take the German sentence,"}, {"start": 86.80000000000001, "end": 93.84, "text": " you again tokenize it, embed it, pass it through the, this time the multi-head attention, which"}, {"start": 93.84, "end": 99.2, "text": " has something called causal masking, and I'll get into that a bit later. And then we have"}, {"start": 99.2, "end": 104.32000000000001, "text": " this again, multi-head attention, but this one you're attending to the representations"}, {"start": 104.32, "end": 114.11999999999999, "text": " from the encoder, and not from the, so you're using those representations as keys and values,"}, {"start": 114.11999999999999, "end": 121.39999999999999, "text": " and not the ones that come from the decoder. And at the end you just basically project"}, {"start": 121.39999999999999, "end": 128.16, "text": " those representations into your output vocabulary space, and you just do simple, like cross-entropy"}, {"start": 128.16, "end": 138.16, "text": " loss or KL divergence, and then you basically backprop. So that's the overview of the transformer,"}, {"start": 138.16, "end": 142.76, "text": " and if you didn't understand every single thing, that's good, because I'm now going"}, {"start": 142.76, "end": 151.24, "text": " to dig deeper into how everything fits together. So the first part that happens is you basically"}, {"start": 151.24, "end": 156.32, "text": " want to learn how to split the sentence. So you want to learn how to split the English"}, {"start": 156.32, "end": 161.35999999999999, "text": " sentence into sub-words, and you want to learn how to do the same thing for the German sentence."}, {"start": 161.35999999999999, "end": 168.2, "text": " So you basically have, you start with a huge English corpus, and you start with the second"}, {"start": 168.2, "end": 173.64, "text": " corpus, like the German corpus, that corresponds exactly one-to-one to this corpus. So you"}, {"start": 173.64, "end": 179.12, "text": " basically have a sentence in English, like how are you today, and you have that translation,"}, {"start": 179.12, "end": 184.84, "text": " something called the golden translation, because maybe like humans, like professional translators"}, {"start": 184.84, "end": 191.76, "text": " created this, and so the corresponding sentence would be, wie geht es Ihnen heute in German,"}, {"start": 191.76, "end": 196.96, "text": " and you train something called tokenizer objects, which will learn how to split these sentences"}, {"start": 196.96, "end": 205.44, "text": " and map them into some IDs. And that's where these vocabulary tables come into play. So"}, {"start": 205.44, "end": 212.12, "text": " this is an imaginary English vocab table, and then we have the German one here. So what"}, {"start": 212.12, "end": 217.16, "text": " happens is, so the procedure goes like this. You take the English sentence, and you don't"}, {"start": 217.16, "end": 222.52, "text": " do the simple splitting, like on spaces. That would be something really naive. The original"}, {"start": 222.52, "end": 228.64000000000001, "text": " transformer used something called byte-pairing coding, and I won't get into the details of"}, {"start": 228.64000000000001, "end": 232.88, "text": " that algorithm. If you're interested in learning more about that, I could create a video, a"}, {"start": 232.88, "end": 239.16, "text": " separate video, so just comment down below if you think that'd be useful. So you will"}, {"start": 239.16, "end": 244.32, "text": " maybe split the word today into separate sub words, so maybe like into something like two"}, {"start": 244.32, "end": 250.68, "text": " and day. And once it knows how to split, and it has the IDs, it can convert the sentence"}, {"start": 250.68, "end": 257.6, "text": " into just a list of IDs. So how are you today gets split into how are you today tokens,"}, {"start": 257.6, "end": 262.71999999999997, "text": " so five tokens in total, and we just map them into corresponding IDs. So you can see in"}, {"start": 262.71999999999997, "end": 267.36, "text": " the table how it corresponds to two, so we put two here, etc. And we do the same thing"}, {"start": 267.36, "end": 275.08000000000004, "text": " for the German sentence, we get as in and hoi-te, so I kind of, I imaginary split this"}, {"start": 275.08000000000004, "end": 281.16, "text": " word hoi-te into two substrings. It depends on the tokenizer how it will do this, but"}, {"start": 281.16, "end": 287.84000000000003, "text": " let's say it's like this. So what's interesting here is this start of sentence token. So that"}, {"start": 287.84000000000003, "end": 296.0, "text": " one is used to prompt the decoder into generating the translation. So basically, you later wants"}, {"start": 296.0, "end": 301.12, "text": " to transform restrained, you just prompted with this start of sentence token, and it"}, {"start": 301.12, "end": 306.52, "text": " will start generating the words here, you'll feed them back into the input, and that's"}, {"start": 306.52, "end": 313.08, "text": " something called autoregressive decoding, autoregressive models. So that's the first"}, {"start": 313.08, "end": 322.48, "text": " part how the sentence gets split into these lists of IDs. Now for the second part. Once"}, {"start": 322.48, "end": 328.6, "text": " you have the English and the German tokenizer object trained, you can do the following."}, {"start": 328.6, "end": 336.04, "text": " So you basically, what you do is you, let me just, man, let me take something, yeah,"}, {"start": 336.04, "end": 342.96000000000004, "text": " okay. So basically you take the list, and you have this thing called input embedding."}, {"start": 342.96000000000004, "end": 349.28000000000003, "text": " So basically that's a, so this, so you have a basically a huge matrix, like that's the"}, {"start": 349.28, "end": 357.71999999999997, "text": " size of your English vocabulary, so say that's like maybe 30k rows, and every single row"}, {"start": 357.71999999999997, "end": 364.32, "text": " correspond to a single word in the English vocab. And you basically have, in the original"}, {"start": 364.32, "end": 372.2, "text": " transformer, these were 512 dimensional vectors. So basically that's the input encoding. And"}, {"start": 372.2, "end": 378.59999999999997, "text": " now what you do is you take this sentence, so you have, how are you today? You take the"}, {"start": 378.6, "end": 385.20000000000005, "text": " ID, and the ID basically indexes into this table. So if you have two, you'll basically"}, {"start": 385.20000000000005, "end": 391.72, "text": " take the second vector, so you'll take this one, and you'll stick it right here, so in"}, {"start": 391.72, "end": 398.88, "text": " the beginning of the encoder. So you'll put the 512 dimensional vector here. You do the"}, {"start": 398.88, "end": 405.84000000000003, "text": " same thing for R. You take the third row, you place it here, and that's it. And now"}, {"start": 405.84, "end": 413.08, "text": " the thing is, in the original paper, this matrix was shared with the German matrix,"}, {"start": 413.08, "end": 420.12, "text": " and was shared with softmax weights here. But for the sake of easier explanation, I'll"}, {"start": 420.12, "end": 425.76, "text": " just, and that's how I implemented this in my own project, and I'll link my project,"}, {"start": 425.76, "end": 429.88, "text": " my implementation of the original transformer in the description. So basically split all"}, {"start": 429.88, "end": 437.96, "text": " of these into separate matrices. So once you have that, you do the same thing for German."}, {"start": 437.96, "end": 444.88, "text": " So basically the German has a separate embedding matrix, which has the size of the, this size"}, {"start": 444.88, "end": 451.7, "text": " corresponds to the German vocab size, so maybe, I don't know, like 35k rows. And this one"}, {"start": 451.7, "end": 457.68, "text": " will be also 512. And you basically repeat the procedure. You take the first word, which"}, {"start": 457.68, "end": 464.76, "text": " is the start of sentence token, which is maybe mapped to, I don't know, like index two, in"}, {"start": 464.76, "end": 470.76, "text": " this matrix. So you take this row, you place it here, and you repeat the procedure for"}, {"start": 470.76, "end": 478.16, "text": " the rest of the words. So that's how the embedding part works. So you just use the IDs as indices"}, {"start": 478.16, "end": 483.72, "text": " into these embedding tables. And these are completely randomly initialized in the beginning."}, {"start": 483.72, "end": 488.88000000000005, "text": " So that's like random, totally random, like maybe uniform sampling or something. It doesn't"}, {"start": 488.88000000000005, "end": 494.0, "text": " matter. So that's something that you learn throughout the training. One of the things"}, {"start": 494.0, "end": 499.40000000000003, "text": " you learn. We'll see other parameters that get learned. So the second part is this positional"}, {"start": 499.40000000000003, "end": 506.52000000000004, "text": " encoding and how that works. So that's also a matrix. And this time it's shared between"}, {"start": 506.52, "end": 516.64, "text": " English and German. And it looks something like this. And now it's basically like a,"}, {"start": 516.64, "end": 522.28, "text": " basically a bunch of sinusoids, so sines and cosines, whose frequencies form a geometrical"}, {"start": 522.28, "end": 527.52, "text": " progression. So you can see that the wavelength here is getting bigger than the wavelength"}, {"start": 527.52, "end": 533.24, "text": " here. And basically this is just some heuristic. You can basically also randomly initialize"}, {"start": 533.24, "end": 539.4, "text": " this matrix and learn it throughout the training. But the authors show that both of those approaches"}, {"start": 539.4, "end": 545.5600000000001, "text": " are equivalent, but they still stick with this one. So you do the same thing. You basically"}, {"start": 545.5600000000001, "end": 550.6800000000001, "text": " take the ID of the words, so how, you take two IDs, two, and you basically take this"}, {"start": 550.6800000000001, "end": 558.76, "text": " row and you'll stick that row here and you'll just add them up. You just add those two up"}, {"start": 558.76, "end": 565.92, "text": " and you get the new representation. And now you're here. So this is where we are now."}, {"start": 565.92, "end": 572.92, "text": " So basically do the same thing for other words. And so just find, so this corresponds to the"}, {"start": 572.92, "end": 581.16, "text": " word R. So this is R. So you'll find the ID that's three. You'll take the third row from"}, {"start": 581.16, "end": 586.24, "text": " this column. You'll find the positional encoding. You'll add them up. Simple addition. And you"}, {"start": 586.24, "end": 592.2, "text": " end up with representations. And now we're finally came to this part where the whole"}, {"start": 592.2, "end": 600.0, "text": " logic happens. And yeah, one more thing. The same thing obviously happens for the German"}, {"start": 600.0, "end": 606.44, "text": " part. So you just find embeddings. You use the positional encoding table and you add"}, {"start": 606.44, "end": 613.26, "text": " them up and you end up with token representations here. So that's the first part. And hopefully"}, {"start": 613.26, "end": 619.3199999999999, "text": " this was clear enough. So I'll focus on the multi-head attention and I'll ignore the residual"}, {"start": 619.3199999999999, "end": 625.02, "text": " connection and the layer normalization for a second and we'll get to them a bit later."}, {"start": 625.02, "end": 636.28, "text": " So we have five tokens, right? So we have how, R, blah blah blah, day. And all of these"}, {"start": 636.28, "end": 644.64, "text": " are like 512 dimensional vectors. So now what happens is the multi-head attention basically"}, {"start": 644.64, "end": 650.76, "text": " has four matrices that gets learned during the training. So the first three, the important"}, {"start": 650.76, "end": 665.16, "text": " ones are the query, the key matrix, and finally the value matrix. And all of these have dimension"}, {"start": 665.16, "end": 671.76, "text": " the same as these embeddings. So that's 512. So we'll have 512 times 512 for the query"}, {"start": 671.76, "end": 678.12, "text": " matrix and the same goes for the key and value matrices as well. So what we do now is we"}, {"start": 678.12, "end": 684.3199999999999, "text": " basically point-wise apply these matrices. So what I mean by that is I take the Q matrix"}, {"start": 684.3199999999999, "end": 694.72, "text": " and I multiply this one and this vector and we get, so we get 512 here and we get also"}, {"start": 694.72, "end": 700.6, "text": " 512 here, et cetera, the same here. And we do the same thing with the key matrix. So"}, {"start": 700.6, "end": 709.0, "text": " we end up with three vectors for every single token, right? So, and here we have value."}, {"start": 709.0, "end": 717.76, "text": " So that's like the query, the key, and the value vectors. So once we are here, now what"}, {"start": 717.76, "end": 723.2, "text": " happens is you basically, because that's a multi-head attention, basically the original"}, {"start": 723.2, "end": 728.2, "text": " paper had eight heads that go in parallel. So you just split these vectors into eight"}, {"start": 728.2, "end": 735.36, "text": " pieces. So this gets split into the parts. So this one has eight times 64, yeah. So we"}, {"start": 735.36, "end": 742.72, "text": " get 64 dimensional vectors here. We take the first chunk from the second token representation"}, {"start": 742.72, "end": 751.76, "text": " as well and we do that for this. So this corresponds to token day. So we basically take the first"}, {"start": 751.76, "end": 761.0, "text": " part, first segment, which has basically 64 dimension as well. And what we do now is we"}, {"start": 761.0, "end": 766.24, "text": " get into the interesting part. So we just group all of these together. So we'll have,"}, {"start": 766.24, "end": 774.04, "text": " so we have eight heads, right? So that's head one, head two, blah, blah. We had eight heads."}, {"start": 774.04, "end": 780.76, "text": " So we group these first segments and place them here. We group the second pieces. So"}, {"start": 780.76, "end": 788.36, "text": " we group, so this is the second piece. This is already getting really annoying, but hopefully"}, {"start": 788.36, "end": 792.4399999999999, "text": " you can see what I mean. So we take the second segment, you group them and place them here"}, {"start": 792.4399999999999, "end": 796.84, "text": " and you do the same thing for the last segments and place them here. So now I'm going to zoom"}, {"start": 796.84, "end": 802.92, "text": " out and we come to the really interesting part and that's the attention itself. Okay,"}, {"start": 802.92, "end": 808.96, "text": " so we projected all of these embedding vectors into eight subspaces and let's take a single"}, {"start": 808.96, "end": 818.6, "text": " one. So what happens is, so we ended up with those small vectors now. So this one is 64"}, {"start": 818.6, "end": 824.5600000000001, "text": " and that's the query one. Then we have the key and the value vectors as well. So that's"}, {"start": 824.5600000000001, "end": 831.64, "text": " 64, 64. And that's for the first token, that's how, because remember we had the sentence,"}, {"start": 831.64, "end": 838.88, "text": " how are you today? So we repeat the same thing until the last token, that's the day token."}, {"start": 838.88, "end": 848.32, "text": " It's also 64 and the same thing happens here. Just imagine the other vectors. So this is"}, {"start": 848.32, "end": 853.4399999999999, "text": " the interesting part and this is how the attention works. So you take the query vector, so this"}, {"start": 853.4399999999999, "end": 859.4399999999999, "text": " one, and you basically do a dot product between this query vector and every single key vector."}, {"start": 859.4399999999999, "end": 865.92, "text": " So this vector and you'll do, so imagine one vector being here. So you'll do the dot product"}, {"start": 865.92, "end": 871.12, "text": " between this query and this key and with all the other keys until you get to this key and"}, {"start": 871.12, "end": 877.3199999999999, "text": " you just do the dot product. So what you end up with is basically a measure of similarity."}, {"start": 877.3199999999999, "end": 882.9399999999999, "text": " So that's how dot product, if you're not familiar, I hope you are, so it's a simple linear algebra."}, {"start": 882.9399999999999, "end": 890.28, "text": " So let's say this is our query vector and let's say this is our key vector. So in this,"}, {"start": 890.28, "end": 895.8, "text": " when you do a dot product between those two, because the angle is so small, basically when"}, {"start": 895.8, "end": 901.5999999999999, "text": " you project this one, you add them up, you basically get a huge value. Once the, if the"}, {"start": 901.5999999999999, "end": 908.0799999999999, "text": " key vector was like orthogonal to the query vector, you'd have like a 90 degree here,"}, {"start": 908.0799999999999, "end": 914.9599999999999, "text": " orthogonal right, and then you'd have zero score. So basically what happens here, you're"}, {"start": 914.9599999999999, "end": 920.04, "text": " measuring the similarity between the query vector and the key vectors. And the closer"}, {"start": 920.04, "end": 926.52, "text": " the key vectors are in these subspaces to this query vector, the higher the score. So"}, {"start": 926.52, "end": 932.1999999999999, "text": " you basically end up with something like this. Maybe you end up with 3.2 score here. Maybe"}, {"start": 932.1999999999999, "end": 940.36, "text": " you end up for the second token, which was how r, so this is r. Maybe you end up with"}, {"start": 940.36, "end": 946.3199999999999, "text": " like, I don't know, like 0.3 similarity and you maybe end up with the last token, you"}, {"start": 946.32, "end": 953.5200000000001, "text": " end up with, I don't know, like 2.2. And there's one interesting, one more important thing"}, {"start": 953.5200000000001, "end": 959.36, "text": " they did and that's normalization by the square root of the dimension of these heads. So that's"}, {"start": 959.36, "end": 964.8000000000001, "text": " 64. So this is dimension. So we have 64 here and that's why we do square root of 64, which"}, {"start": 964.8000000000001, "end": 971.0400000000001, "text": " is 8. They show that this normalization by the square root of this dimension helps them"}, {"start": 971.04, "end": 978.24, "text": " better train the transformer. Otherwise, they suspect that the, like the huge value, so"}, {"start": 978.24, "end": 982.52, "text": " the bigger the dimension, the bigger the dot products would be and they'd push the softmax"}, {"start": 982.52, "end": 987.76, "text": " into regions where the gradients gets really small and then the training just kind of slows"}, {"start": 987.76, "end": 992.5799999999999, "text": " down. So that's why they do this. So they just basically divide the scores here and"}, {"start": 992.5799999999999, "end": 999.1999999999999, "text": " once you do that, you apply the softmax. So softmax will make this sum go to 1 and so"}, {"start": 999.2, "end": 1004.6, "text": " the bigger one will have maybe, after doing the softmax, will have maybe 0.7. This one"}, {"start": 1004.6, "end": 1013.76, "text": " would be maybe 0.01 and this one would be maybe 0.1. And once you have these values,"}, {"start": 1013.76, "end": 1019.2, "text": " aside from applying probably like a dropout is usually in practice applied here. So you"}, {"start": 1019.2, "end": 1025.4, "text": " randomly drop one of these to 0. So maybe this one would go to 0. And once you do that,"}, {"start": 1025.4, "end": 1034.3200000000002, "text": " then you use these scores to finally multiply them, these value vectors by these and you"}, {"start": 1034.3200000000002, "end": 1038.74, "text": " end up with the final representation. So what I mean by that, so we were, so all of these"}, {"start": 1038.74, "end": 1046.24, "text": " results came for this specific query vector, right? So now we just use this 0.7 and we"}, {"start": 1046.24, "end": 1052.44, "text": " multiply this value vector with 0.7 and then we multiply this vector with 0 and finally"}, {"start": 1052.44, "end": 1058.1200000000001, "text": " all of these with their corresponding values, the last one will be multiplied by 0.1. We"}, {"start": 1058.1200000000001, "end": 1068.92, "text": " add them up and we get the final representation that's 64 dimension long and it contains the"}, {"start": 1068.92, "end": 1074.38, "text": " information about all of these surrounding tokens. So that's why attention is really"}, {"start": 1074.38, "end": 1080.98, "text": " nice and why transformers perform so well. Because they can attend to all of the representations"}, {"start": 1080.98, "end": 1085.96, "text": " and create a new contextualized representation which is later helpful for the downstream"}, {"start": 1085.96, "end": 1091.0, "text": " tasks. So that's, you repeat the same thing basically for all of the other tokens. So"}, {"start": 1091.0, "end": 1096.72, "text": " now you take this query vector, you multiply it by, you do that product between different"}, {"start": 1096.72, "end": 1103.02, "text": " keys again, you again do the softmax, etc. etc. And you end up with all of the representations."}, {"start": 1103.02, "end": 1109.1, "text": " So we end up with five representations here because again remember we have five tokens."}, {"start": 1109.1, "end": 1114.3999999999999, "text": " So that's how the multi-head attention pretty much works. So now just concatenate all of"}, {"start": 1114.3999999999999, "end": 1120.1999999999998, "text": " the eight heads. So we have eight heads, we just group all of these together. So this"}, {"start": 1120.1999999999998, "end": 1126.36, "text": " was the first segment, right? So these came from these parts. So we'll just return them"}, {"start": 1126.36, "end": 1133.6399999999999, "text": " back into the same shape and that's it. We are ready for the final part of the attention."}, {"start": 1133.64, "end": 1140.0400000000002, "text": " So once you concatenate the vectors, you end up with again 512 because we concatenated"}, {"start": 1140.0400000000002, "end": 1150.68, "text": " all of the eight heads. And we have five of these corresponding to our tokens. How are"}, {"start": 1150.68, "end": 1157.88, "text": " you today, right? And the last matrix I mentioned, so these are the three ones I showed you so"}, {"start": 1157.88, "end": 1164.0400000000002, "text": " far. But there is one more matrix that's used in the attention module and that's the output"}, {"start": 1164.0400000000002, "end": 1172.0, "text": " matrix. So basically that one again, you just, it's also 512 by 512. It's the, let's call"}, {"start": 1172.0, "end": 1182.0400000000002, "text": " it O, like output matrix. You basically again point-wise project these into 512. So basically"}, {"start": 1182.04, "end": 1189.6, "text": " just apply this one for this vector and independently apply it to this one. So that's what I mean"}, {"start": 1189.6, "end": 1197.2, "text": " by point-wise. And this representation is what we get out right here. So that's where"}, {"start": 1197.2, "end": 1205.48, "text": " we ended up. So now the residual connection would just add up those representations that"}, {"start": 1205.48, "end": 1212.44, "text": " were here before we did all of these attention thing and they just add it up to our final"}, {"start": 1212.44, "end": 1218.64, "text": " representations that came out from the, from multiplying by this matrix. That was arguably"}, {"start": 1218.64, "end": 1223.56, "text": " the most important part of the transformer architecture. And now what happens is, and"}, {"start": 1223.56, "end": 1227.56, "text": " I'll just ignore the layer normalization. It's similar to batch normalization. It basically"}, {"start": 1227.56, "end": 1232.84, "text": " just kind of updates all of the weights in the batch so that we can, so the gradients"}, {"start": 1232.84, "end": 1238.04, "text": " don't get into like, so that we don't have either vanishing or exploding gradients problems"}, {"start": 1238.04, "end": 1243.28, "text": " basically. It just stabilizes the training. So let me just show you this part with the"}, {"start": 1243.28, "end": 1247.4399999999998, "text": " feed forward net and then like the residual connection and the layer normalization just"}, {"start": 1247.4399999999998, "end": 1256.3999999999999, "text": " repeats the same as down here. So we ended up right here and I'll just, for the sake"}, {"start": 1256.4, "end": 1264.72, "text": " of making this less dirty, just repeat it again here. So again, we have five tokens"}, {"start": 1264.72, "end": 1273.1200000000001, "text": " corresponding to our like tokens. How are you today? And what we do now is we point-wise"}, {"start": 1273.1200000000001, "end": 1283.5600000000002, "text": " apply the neural network. Basically that neural network consists out of two matrices and like"}, {"start": 1283.56, "end": 1291.9199999999998, "text": " we have one value like the nonlinear activation layer. Usually transformers have, so as I"}, {"start": 1291.9199999999998, "end": 1297.6399999999999, "text": " said, they have two matrices. So the first one just kind of expands this dimension into"}, {"start": 1297.6399999999999, "end": 1304.24, "text": " 4x. Just don't ask me why. It's just people found out this works and that's it. So we"}, {"start": 1304.24, "end": 1315.28, "text": " have 512. Now we end up with 2048, right? So and then we just apply the second layer"}, {"start": 1315.28, "end": 1321.72, "text": " and that DOM projects this into 512. So those are the two additional matrices that we are"}, {"start": 1321.72, "end": 1330.4, "text": " going to learn during the training. So we had the four ones in the multi-head attention"}, {"start": 1330.4, "end": 1335.3600000000001, "text": " layer. We have two matrices here. We have those embedding matrices and we don't learn"}, {"start": 1335.3600000000001, "end": 1341.5600000000002, "text": " the positional encodings because they are, we use the sine and cosine heuristic. So that's"}, {"start": 1341.5600000000002, "end": 1347.92, "text": " it. That's how the encoder part of the transformer works. So we ended up here. We just repeat"}, {"start": 1347.92, "end": 1358.5600000000002, "text": " the same procedure six times or n times like in general and we end up with encoder representations"}, {"start": 1358.56, "end": 1365.96, "text": " of our initial tokens. So we end up with, in our case, with five token representations"}, {"start": 1365.96, "end": 1371.7, "text": " here. There are two more interesting and important things I want to cover and that's how exactly"}, {"start": 1371.7, "end": 1375.76, "text": " this part works and how exactly this part works. So it's really similar to this multi-head"}, {"start": 1375.76, "end": 1380.84, "text": " attention I already explained, but there are some details that are worth covering because"}, {"start": 1380.84, "end": 1384.8799999999999, "text": " people usually get confused by those. The feed forward network is the same and I'll"}, {"start": 1384.88, "end": 1389.72, "text": " just cover like the third portion is this, how this works. So let me start with this"}, {"start": 1389.72, "end": 1399.8000000000002, "text": " part. So the MAST multi-head attention module. So this is getting really like a nuclear physics"}, {"start": 1399.8000000000002, "end": 1407.98, "text": " whiteboard, except it's not nuclear physics. Okay. Anyways, we have tokens. This time we"}, {"start": 1407.98, "end": 1416.44, "text": " have German tokens. So we, let's say we have eight of these and they are 512. This one"}, {"start": 1416.44, "end": 1423.0, "text": " is maybe the start of the sentence token. I don't know how to write today. Okay. Start"}, {"start": 1423.0, "end": 1428.84, "text": " the sentence. So S okay. And we end up with the last word, which was Hoytas. So we have"}, {"start": 1428.84, "end": 1435.6, "text": " the substring T here. So the only difference between this multi-head attention and the"}, {"start": 1435.6, "end": 1441.04, "text": " last one I showed you is this thing called causal masking. And what basically happens"}, {"start": 1441.04, "end": 1445.9199999999998, "text": " is so everything, so for the sake of making this easier, let's assume we have only one"}, {"start": 1445.9199999999998, "end": 1452.84, "text": " head and not eight heads. So what happens during the attention operation here is so"}, {"start": 1452.84, "end": 1464.56, "text": " we have, so we have a query, we have key and we have value vectors. And now what happens"}, {"start": 1464.56, "end": 1471.96, "text": " here is this specific token can only look at the representations that came beforehand."}, {"start": 1471.96, "end": 1482.3999999999999, "text": " So you can only look to itself and to previous. So this one gets mapped to again, value, key"}, {"start": 1482.3999999999999, "end": 1491.84, "text": " and query. So if you take this query, what happens is it will do a dot product between"}, {"start": 1491.84, "end": 1497.84, "text": " this query and this key and this query and this key, etc. It will even do a dot product"}, {"start": 1497.84, "end": 1505.3999999999999, "text": " between that query and this key for the last token. But now the interesting thing that"}, {"start": 1505.3999999999999, "end": 1513.48, "text": " happens is we just, before we add these, before we pass these to a softmax, so we ended up"}, {"start": 1513.48, "end": 1522.88, "text": " like maybe some scores like 3.2, I don't know, like 0.1 and this would have maybe, I don't"}, {"start": 1522.88, "end": 1532.1200000000001, "text": " know, 5.4 and even these would have some scores, so maybe 3.6. But we add up like minus infinity"}, {"start": 1532.1200000000001, "end": 1539.56, "text": " here. So we add up minus infinity here and we add up minus infinity to all of the other"}, {"start": 1539.56, "end": 1546.36, "text": " ones. So this got a clutter a little bit, but I just went ahead and redraw this a little"}, {"start": 1546.36, "end": 1552.6799999999998, "text": " bit to make it a bit clearer. So the only difference is, so now when we have like maybe"}, {"start": 1552.6799999999998, "end": 1558.6, "text": " the query vector that corresponds to the token s, because remember the sentence was vket"}, {"start": 1558.6, "end": 1565.56, "text": " s enen hoite, so I split this into two substrings and this into two substrings. So the s one,"}, {"start": 1565.56, "end": 1570.52, "text": " so we take its query and we do, again we just do dot products with different keys and we"}, {"start": 1570.52, "end": 1578.24, "text": " end up with some scores like maybe 3.2 here, blah blah blah, we end up with maybe 5.1 here,"}, {"start": 1578.24, "end": 1584.9199999999998, "text": " we end up with, I don't know, 2.4 here and we end up with, I don't know, like 0.1. And"}, {"start": 1584.9199999999998, "end": 1591.3999999999999, "text": " now what happens is every single score that comes after this current token, we just add"}, {"start": 1591.4, "end": 1601.68, "text": " up, basically we add up minus infinity. And what it does is basically now we feed this"}, {"start": 1601.68, "end": 1608.3200000000002, "text": " into softmax and after feeding it into softmax it gets pulled to zero and that basically"}, {"start": 1608.3200000000002, "end": 1614.8000000000002, "text": " means we won't be using these value vectors in the final representation for this specific"}, {"start": 1614.8, "end": 1624.0, "text": " token s. So this token, once we get the value, so 3.2 maybe gets mapped to 0.2 after softmax,"}, {"start": 1624.0, "end": 1631.04, "text": " this one gets mapped to, I don't know, 0.7, so we'll use 0.7 of these value vectors, we'll"}, {"start": 1631.04, "end": 1637.84, "text": " use 0.2 of these, but we won't use these. And that's what makes it causal. So this one"}, {"start": 1637.84, "end": 1642.68, "text": " can only see what came previously. And so the final representation only contains the"}, {"start": 1642.68, "end": 1651.0800000000002, "text": " previous representations and the current representation. If you were to do this for the first token,"}, {"start": 1651.0800000000002, "end": 1655.5600000000002, "text": " everything would get, so all of the other tokens would get mapped to 0. And so the first"}, {"start": 1655.5600000000002, "end": 1661.4, "text": " token, the start of sentence token, can only use its own representation. So basically nothing"}, {"start": 1661.4, "end": 1668.28, "text": " happens for this. So basically we just pass the value token here and that's what happens"}, {"start": 1668.28, "end": 1672.76, "text": " for the first token. And the last one we'll be using all of the representations because"}, {"start": 1672.76, "end": 1680.48, "text": " it's the last one. Okay, so that was this part. So that was this mask multi-head attention"}, {"start": 1680.48, "end": 1689.56, "text": " module. We covered that one, checked. Now the last one is this and it's simple. So again,"}, {"start": 1689.56, "end": 1697.2, "text": " we have German tokens, we get the Sinnenheute, this is the start of sentence, blah blah blah,"}, {"start": 1697.2, "end": 1707.26, "text": " we end up with the TE token. And we have these representations which ended up being calculated"}, {"start": 1707.26, "end": 1713.1200000000001, "text": " after iterating through six of these encoder layers. So we are using those. We have five"}, {"start": 1713.1200000000001, "end": 1721.56, "text": " of those. So basically we have five of those tokens here. And so this one corresponds to"}, {"start": 1721.56, "end": 1736.96, "text": " v, no, sorry, how are you today? And now we just use, we basically use these as the key"}, {"start": 1736.96, "end": 1745.6599999999999, "text": " and the value and we use these as the queries. So again, we have three matrices, the query,"}, {"start": 1745.66, "end": 1756.8400000000001, "text": " the key and the value matrix. Query, key, value. We take the query, we create query vectors"}, {"start": 1756.8400000000001, "end": 1764.3600000000001, "text": " for the German tokens, but we use key and value and apply them to these tokens. So we"}, {"start": 1764.36, "end": 1775.9599999999998, "text": " end up with key and value representations here all the way down to this token. And now"}, {"start": 1775.9599999999998, "end": 1784.32, "text": " we basically take this query and we do the usual dot product with these keys which will"}, {"start": 1784.32, "end": 1791.6, "text": " after doing the softmax, we'll know how to combine these value vectors to get the new"}, {"start": 1791.6, "end": 1798.08, "text": " token. So that's what's different for this module. So it basically combines the value"}, {"start": 1798.08, "end": 1803.76, "text": " vectors that come out from the encoder. So using the queries that came from the bottom"}, {"start": 1803.76, "end": 1811.24, "text": " layer of this decoder layer. So that's it. So I intentionally went into a lot of details."}, {"start": 1811.24, "end": 1817.52, "text": " I usually love to go through from high level explanation and then slowly and gradually"}, {"start": 1817.52, "end": 1826.8799999999999, "text": " explain the complexity. But here I just wanted to go fully into details and hopefully some"}, {"start": 1826.8799999999999, "end": 1833.6, "text": " of you found this useful. But we are not finished yet. I still have this part to cover and it's"}, {"start": 1833.6, "end": 1843.72, "text": " pretty easy. So once we end up at the top of this stack, it's quite simple actually."}, {"start": 1843.72, "end": 1854.56, "text": " So let's say we end up with eight token representations. So these correspond to start of sentence,"}, {"start": 1854.56, "end": 1865.92, "text": " blah blah blah. This corresponds to this token TE. And now what we do is we apply, we project"}, {"start": 1865.92, "end": 1873.18, "text": " these representations which are still 512 as always. We project them into this space"}, {"start": 1873.18, "end": 1879.1200000000001, "text": " that has the dimensionality the same as the German vocabulary size. So that was for example"}, {"start": 1879.1200000000001, "end": 1888.24, "text": " 35k. So that means this one gets projected into 35,000 dimensional vector and that's"}, {"start": 1888.24, "end": 1900.8, "text": " huge. That means we have a huge matrix here whose dimension is 512 times 35k which is"}, {"start": 1900.8, "end": 1905.32, "text": " where most of the computation comes into play. And there are many strategies where you can"}, {"start": 1905.32, "end": 1914.08, "text": " maybe create something called, you can Google it, hierarchical softmax instead of doing"}, {"start": 1914.08, "end": 1919.08, "text": " this one. Or you can do something called negative sampling, etc. But I won't get into those"}, {"start": 1919.08, "end": 1924.0, "text": " because those techniques were not mentioned in the original paper. So after you project"}, {"start": 1924.0, "end": 1930.68, "text": " these, you do the same thing for all of the tokens. And now you just have the ground truth."}, {"start": 1930.68, "end": 1936.76, "text": " So basically because we know what the output sentence should be, so that's, we get the"}, {"start": 1936.76, "end": 1944.64, "text": " sin heute, that was the German sentence. So just to freshen up your memory. So we get"}, {"start": 1944.64, "end": 1951.16, "text": " the sin heute, okay? So basically what will happen here is we'll have, so we'll take the"}, {"start": 1951.16, "end": 1963.92, "text": " token v and we'll find the id. And basically what happens is we'll have the ground truth"}, {"start": 1963.92, "end": 1970.16, "text": " vector here and on the position 3 it will have 1. And here for example this one will"}, {"start": 1970.16, "end": 1977.76, "text": " be some, won't be a binary vector, it will be, maybe we'll have, I don't know, like 0.7"}, {"start": 1977.76, "end": 1984.96, "text": " here and then some random value distribution, right? And basically we do either cross entropy,"}, {"start": 1984.96, "end": 1991.52, "text": " in this case where we have 1 here, and basically that means if we have 0.7, the loss would"}, {"start": 1991.52, "end": 2001.02, "text": " be log of 0.7. And if you want to minimize this loss, it should go, this argument better"}, {"start": 2001.02, "end": 2009.52, "text": " go to 1 because log of 1 equals 0. So the final loss when you're training the transformer"}, {"start": 2009.52, "end": 2017.3, "text": " is basically a sum of those logs. And I'll get to soft labeling in a bit, but let's assume"}, {"start": 2017.3, "end": 2023.04, "text": " we're using cross entropy for the moment. And so basically we'll have log of 0.7 here,"}, {"start": 2023.04, "end": 2034.68, "text": " so this vector will maybe have, let me draw the ground truth one. So let's say this Te"}, {"start": 2034.68, "end": 2041.12, "text": " is the ground truth vector is 1 here and everything else is 0, but like the distribution was a"}, {"start": 2041.12, "end": 2049.2, "text": " bit different, so the part that we're interested in is maybe 0.3, so the loss would be log"}, {"start": 2049.2, "end": 2058.3999999999996, "text": " of 0.3 and that loss is even bigger. I mean we just add the minus and then it's big otherwise"}, {"start": 2058.3999999999996, "end": 2067.2, "text": " this is a negative number. So we just sum up those logs and we average them. So basically"}, {"start": 2067.2, "end": 2072.08, "text": " that's what the loss function is and then you just do the basic gradient based learning"}, {"start": 2072.08, "end": 2077.24, "text": " using your favorite optimizer like I think they used Adam in the original transformer"}, {"start": 2077.24, "end": 2091.3599999999997, "text": " paper. Yeah they used Adam with a specific learning rate strategy. So that was how it"}, {"start": 2091.3599999999997, "end": 2096.04, "text": " usually works, but there was a slight detail here. They're actually using, they're not"}, {"start": 2096.04, "end": 2104.24, "text": " using these one-hot vectors as the ground truth, they're using something called soft"}, {"start": 2104.24, "end": 2111.12, "text": " labeling where basically this would be set to 0.9 and then the rest of the probability"}, {"start": 2111.12, "end": 2116.8399999999997, "text": " mass would be equally distributed evenly across this vector except for the padding token which"}, {"start": 2116.8399999999997, "end": 2123.8799999999997, "text": " would always get set to 0. So that's the soft label and then you do something called KL"}, {"start": 2123.8799999999997, "end": 2129.08, "text": " divergence instead of cross entropy loss and you can Google that, I won't get into, this"}, {"start": 2129.08, "end": 2136.92, "text": " video is already getting too long. So that's it, that's how the whole procedure looks like."}, {"start": 2136.92, "end": 2142.84, "text": " So you, just to recap once more, you split the sentences into tokens, you map them into"}, {"start": 2142.84, "end": 2149.0, "text": " IDs, you look up at those embedding tables, you find the embedding vectors, you just add"}, {"start": 2149.0, "end": 2155.4, "text": " up those positional encodings. Now you have the initial representations here, now you"}, {"start": 2155.4, "end": 2160.1600000000003, "text": " go through six, you go six times through this encoder stack where you apply the attention"}, {"start": 2160.1600000000003, "end": 2164.28, "text": " and you apply the feedforward net, you end up with the final representations. You do"}, {"start": 2164.28, "end": 2170.52, "text": " the same thing for the German sentence except that you have the causal mask here and you"}, {"start": 2170.52, "end": 2178.56, "text": " attend to the encoding vectors in this module and that's it, you finally project into the"}, {"start": 2178.56, "end": 2186.7999999999997, "text": " output vocabulary space and you do backprop on the loss function that consists of KL divergence"}, {"start": 2186.7999999999997, "end": 2192.12, "text": " or cross entropy loss and that's how you do it."}, {"start": 2192.12, "end": 2198.56, "text": " I owe you just one more explanation and that's how the, once you train the network using"}, {"start": 2198.56, "end": 2204.0, "text": " the process I just showed you, how do you do the actual decoding, the translation process."}, {"start": 2204.0, "end": 2210.0, "text": " So what you do is you take the, just the input sentence like maybe the same one, how are"}, {"start": 2210.0, "end": 2215.68, "text": " you today, again you do the same procedure, tokenization, embeddings, blah, blah, blah"}, {"start": 2215.68, "end": 2222.84, "text": " and you just prompt the model, the decoder with the start of the sentence token and so"}, {"start": 2222.84, "end": 2233.92, "text": " initially this one would get propagated upwards and we'd get some word here like hopefully"}, {"start": 2233.92, "end": 2243.48, "text": " it will be the correct word like V and we'll just take that word and we'll embed it here"}, {"start": 2243.48, "end": 2248.92, "text": " and we'll just repeat the process and that's why this model is called autoregressive. You're"}, {"start": 2248.92, "end": 2253.0, "text": " basically whatever the output, whatever you have is the output, you just feed it back"}, {"start": 2253.0, "end": 2259.08, "text": " as the input and you're just autoregressing until you end up. So what's the stopping condition?"}, {"start": 2259.08, "end": 2266.48, "text": " The stopping condition is that end of sentence token I mentioned here, so basically you,"}, {"start": 2266.48, "end": 2274.16, "text": " once the network outputs end of sentence token you stop the translation process and that's"}, {"start": 2274.16, "end": 2281.48, "text": " pretty much it. So, this was a long video, I'm experimenting with this channel obviously,"}, {"start": 2281.48, "end": 2286.08, "text": " I'm trying to see what works best for you as well as for me, so if you found this video"}, {"start": 2286.08, "end": 2291.88, "text": " useful please put some comments down in the comment section, I appreciate any feedback"}, {"start": 2291.88, "end": 2301.36, "text": " like was it too detailed, was it too messy, maybe I can, yeah, I'll try and improve the"}, {"start": 2301.36, "end": 2308.68, "text": " next time. So, see you next time and subscribe to this channel if you haven't already and"}, {"start": 2308.68, "end": 2313.64, "text": " click that bell icon to get notified when I upload a new video and until next time,"}, {"start": 2313.64, "end": 2324.3199999999997, "text": " keep learning deep."}]