Fastformer: Additive Attention Can Be All You Need | Paper Explained
👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I covered "Fastformer: Additive Attention Can Be All You Need" paper introducing a novel, linear complexity, transformer model!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2108.09084
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
01:00 Previous work and problems
03:10 Fastformer method explained
07:10 Param sharing and complexity
09:10 Results, Fastformer is effective
11:20 Results, Fastformer is efficient
12:45 Ablations
14:10 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR ML PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#fastformer #transformers #linearcomplexity
 What's cracking guys? In this video I'm covering fastformer, additive attention can be all you need. So yet another all you need paper and yet another attempt to create a transformer that's more efficient, that's hopefully linear, like complexity with respect to the input sequence token length. So the reason I'm covering this paper is because I actually like it. I think like the idea is fairly simple. So what really resonated with me was the simplicity. So I guess that's Rich Sutton School of Thought. And aside from that, they have awesome results on various datasets and they actually showed both theoretically and experimentally that they have a better like efficiency compared to previous work. So previous work, even though they sometimes have allegedly linear complexity, they have a huge constant that's hidden behind the big O notation. So in practice, they are not that useful, but I'll dig into those details a bit later. So before I do that, let me just quickly contrast this work with previous attempts to make attention, the attention module of the transformer more efficient. So one group of papers were focusing on how to make the dense attention pattern of the original Vosfani transformer more sparse, but still like keeping the performance that the original transformer has. And so you can see in the long former paper, they proposed a bunch of various different like attention patterns. You can see they are fairly complex and intricate. And in my honest opinion, when I see something that's so complex and hacky, I just don't think it's going to be there in a couple of years. It's just a current way to boost the results. And I still think it's useful for the community, but I don't think it's going to work in the long run. Same thing with like Reformer here. You can see they used locality-sensitive hashing. You can see just the number of steps you have to do here. So LSH bucketing, sort by the LSH bucket, chunk sort the sequence to parallelize, blah, blah, blah, 10 within the same bucket. So it's fairly complex. You can see just looking at the graph, it's very complex. It doesn't seem like the way to go. That's one of my main problems I had with these papers. Second thing is the thing I don't like about these Transformer papers, which are trying to make it more efficient, is that they are all using different data sets. It's kind of hard to compare apples to apples here. So I'd encourage the community to start using like a dedicated set of data sets and try and evaluate all of them on the same suit of tasks. That's just my two cents. OK, so all of these papers are basically trying to modify this multi-head retention module. So this one here. So same for reformers, same for longformers, same for many of them. That was the basic pain point. That's where the quadratic complexity lies in. And again, I won't be digging into how Transformer works. That's like by now a building block of deep learning. So go check out my video on Transformer if you want to learn how it actually works. So yeah, that's the motivation. We want to reduce the quadratic complexity. And now let's see how FastFormer actually managed to achieve that. I think the idea is fairly neat and fairly simple. So let's dig into it. OK, so first things first, we have input sequence, which are embedded into these vectors. We have so we have n vectors, which represent the end tokens of the input sequence. We have the usual query, transform and key transform and value transform. And now the interesting thing happens here. So they first take these query vectors and they do additive attention. I'm going to explain in a second what that means. They do additive attention. They form these alpha coefficients. They multiply the vectors with the alpha coefficients and they sum them up to achieve this global query vector, which serves as an initial step in creating this global context of this whole sequence. Let's see quickly how the additive attention works. So we have a vector. Let's take like q1, whatever. The additive attention is super simple. You just take the input vector. So q1 and what they're going to do is they're going to just feed it through a fully connected network. So every single dimension in this vector, every single feature is going to have associated like a link here with a weight. And you basically form that's how you form the alpha coefficient. So alpha I in the general case, we basically have the operations in here where these the number of dimensions of the vector. So you can see it's fairly simple and computationally super cheap to form these alpha coefficients, which, as I said, then you just use them to do a scalar product of the vector and you form another representation, which is here, and then you just add them up. You add up all the vectors. You form the Q vector. That's it. That's how you form the Q vector. Then you what you do is you take the Q vector and you modify the key vectors by doing element wise. So this this symbol here is just element wise product. They did some ablation studies. We'll see that in a couple of minutes. But basically the concatenation of this global Q vector or the addition did not work as good as as doing this like element wise product. Basically, once you do that, you form these P vectors and then you again do the additive attention. So the same thing, the same procedure is here. And then you add them up and you form the Q the K vector. So that's the key global vector. Finally, the final step of this of this module is you basically have value vectors. What you're going to do is again do the element wise product. You're going to form the U vectors. And this time, instead of doing additive attention, they did something similar to the original transformer. They did simple linear transformation here. So this block is just like a set of linear layers or MLPs. I think I'm not sure whether they're using single layer or MLP, but that doesn't matter. So basically you have a simple transformation, which is learnable. You form these R vectors. And then finally, you just add them up with the query vectors, which we had here to form the final output. And that's it. So it's much more simple compared to, in my opinion, compared to long former compared to lean former compared, which is the lower end approximation of the matrix compared to every other paper pretty much. Yeah. So that's a big plus in my book, at least. OK, so everything I just explained is written down in these formulas. The only thing I forgot to mention is they have the same as in the original transformer. They have this division by the square root of the dimensionality, which serves to keep the things stable. And so this is just a scale dot product. And that's how I formed the alpha coefficients. And then, as I said, you just do weighted like some of these query vectors to form the global query vector. And that's it. You do the same thing for for the key for the global key vector, as I already mentioned it here. So that's this one. And finally, I explained the third step. And that's this linear transformation part I explained. OK, so a couple of details which are interesting. They did some oblations again. They mentioned here we shared the value and query transformation parameters to reduce the memory cost. In addition, we shared the parameters across different fast formal layers to further reduce the parameter size and mitigate the risk of overfitting. So this is nothing new. We had a bunch of papers prior to this one, which were experimenting with this weight sharing. And so nothing new there, but still that's this is additionally going to save up some parameters. And they also didn't have any slowdowns doing this. So that's that's cool. So if you followed along the explanation of how this thing works, it's easy to notice that the whole complexity will be n times d. So that means it's linear with respect to the number of tokens. And these just the dimensionality of the vectors which are being passed through the transformer. OK, so funny thing, I just kind of had to notice this. This must be the most unlucky place to put the footnote. But, yeah, I guess they did it on purpose. Hopefully they did it on purpose. OK, so having said that, let me just show you the complexity compared to the other previous papers. So we had transformer, the original transformer, which was obviously squared compared to the input sequence length. And that was the problem that that made the yields transformers not good enough to to do inference on long sequences. Then we had a bunch of various papers. So I mentioned Longformer, I mentioned I don't see reformer here, but yeah, Linformer and other papers. So as I said, there is always all of these papers pretty much had some hidden caveat inside. So, for example, Longformer had this K. And if I remember correctly, you had to push K pretty high in order to get decent performance boost. So maybe something like square root of n would be optimal. And so you kind of have linear complexity, but you're not quite there. Same with our papers. You can see some have square dependency on the dimensionality. Some have huge constants in front of them, which you don't see because the big O notation just eats it up. So I think this former is the first paper to actually have decent linear complexity. Let's finally see the results. So they have two sections. First one is they show that the first former is effective and then they show that it's actually efficient as well. So they had, I think, five data sets here to show on Amazon. Yeah, I am DB in mind. So sentiment and topic classification tasks. And you can see that this former pretty much outperforms all of the previous baselines on all of these tasks, except for this one where I guess the difference is fairly negligible and the standard deviation here is even bigger. So, yeah. But like, as I mentioned, the thing I don't like about everybody's using their own data sets, it'd be nice to have like an agreement of which data sets we should use when we are evaluating these efficient transformers. Just putting it out there, maybe somebody starts implementing that idea. OK, so that's the being effective. And here are some other data sets they used. So they have the, I think, recommendation task here and they show again then first former is better compared like across these various different metrics. And I don't want anyone get into what these metrics are, but basically it's better. And they have this combination with this PLM and our baseline plus ensemble disaster just signifies ensemble. And they have even better results that you've even better results by doing that. But, yeah, I guess this this row is what is actually important for us. Additional results on text summarization. Again, fast former is fairly better compared to all of the previous baselines across rows one, rows two, rows L metrics. So different metrics. Again, not that important for this paper. Before I show you how efficient this fast former is, let me just kind of dig into this part. So in our experiments, we use glove embeddings to initialize token embedding matrix. I'm not sure why they're using glove here. So this is like an alternative to virtual back embeddings. Not completely sure why they did not learn the whole thing from scratch. If anybody knows what's going on here, let me know down in the comments because I'm this part confused me, to be honest. OK, let's see how efficient this thing is. So looking at inference, looking at training graphs here, so we can see that as the sequence length increases, somewhere around 512, the transformer starts exploding. So the GPU memory is not large enough. I think they were using V100 GPUs and VDI. And so basically that means you cannot actually even evaluate the transformer on these larger sequences, on these longer sequences. Then you can see that this dark blue line is the fast former and it's way better compared to the other baselines. So this pooling former, if I'm not wrong, let me just see the theoretical complexity they reported. So pooling former n times d times w, where w is the window size. And you can see that even though this looks linear in practice, it's not like optimal because they have huge constant. And I guess that w also contributes to that constant exploding. So yeah, in practice, it doesn't seem to be as good as all of the other baselines. Forgot to mention we have here on the y-axis inference time per layer and here we have training time per layer. I guess it correlates pretty much these two charts. That basically means it's better, it's more efficient for both training and inference compared to the other baselines. And I guess that's nice results. Finally, here are some ablations. I mentioned the Element-Vise product they were using to construct those global vectors. You can see that compared to adding up those vectors instead of doing Element-Vise or concatenating them to the key vectors is worse looking at the accuracy on different datasets. So the trend is kind of similar and sticks throughout all of these different datasets. Finally, I mentioned weight sharing, the de-doblation studies there. So without sharing, you can see decent results, but I guess the number of parameters explodes. With query value sharing, some improvements. And then when you do the head-wise sharing, I think that's important. We have a significant drop here. And the same trend goes for every single dataset. And the whole point is if you read the original transformer paper, and if you ever saw those attention visualizations, every single head is learning different sub-function. So sharing the weights there kind of hurts the performance, and that would be some hand-wavy explanation of why that happens. So every single one needs to learn a dedicated function. For example, if you have a sentence, one of those heads will maybe focus on attending the nouns. One will be attending verbs. I don't know, whatever. So different semantics all in all. So that will be a quick explanation of this transformer paper. So again, as I said, what I like about this paper is it's much more simpler implementation-wise than just understanding how it works compared to previous papers I've covered and read, as well as they obviously have both theoretical and experimental guarantees that they are more efficient, and they're also effective across a bunch of different datasets, as we saw. So yeah, that's pretty much it. And finally, if you missed it, I just created a brand new Discord community, so do join it. I'll link the link in the description. Basically, you'll be able to engage with others there, ask questions related to ML. I'm actually working on bringing startups to offer ML jobs, so if that's something that's interesting to you, do join the community and also subscribe to my monthly AI newsletter, which is going to be super valuable, hopefully. So having said that, subscribe to this channel, hit that bell notification, and until next time, bye-bye.
[{"start": 0.0, "end": 5.9, "text": " What's cracking guys? In this video I'm covering fastformer, additive attention can be all you need."}, {"start": 5.9, "end": 14.0, "text": " So yet another all you need paper and yet another attempt to create a transformer that's more efficient,"}, {"start": 14.0, "end": 20.0, "text": " that's hopefully linear, like complexity with respect to the input sequence token length."}, {"start": 21.0, "end": 25.0, "text": " So the reason I'm covering this paper is because I actually like it."}, {"start": 25.0, "end": 31.0, "text": " I think like the idea is fairly simple. So what really resonated with me was the simplicity."}, {"start": 31.0, "end": 38.0, "text": " So I guess that's Rich Sutton School of Thought. And aside from that, they have awesome results on various datasets"}, {"start": 38.0, "end": 45.0, "text": " and they actually showed both theoretically and experimentally that they have a better like efficiency compared to previous work."}, {"start": 45.0, "end": 54.0, "text": " So previous work, even though they sometimes have allegedly linear complexity, they have a huge constant that's hidden behind the big O notation."}, {"start": 54.0, "end": 58.0, "text": " So in practice, they are not that useful, but I'll dig into those details a bit later."}, {"start": 58.0, "end": 65.0, "text": " So before I do that, let me just quickly contrast this work with previous attempts to make attention,"}, {"start": 65.0, "end": 68.0, "text": " the attention module of the transformer more efficient."}, {"start": 68.0, "end": 78.0, "text": " So one group of papers were focusing on how to make the dense attention pattern of the original Vosfani transformer more sparse,"}, {"start": 78.0, "end": 83.0, "text": " but still like keeping the performance that the original transformer has."}, {"start": 83.0, "end": 90.0, "text": " And so you can see in the long former paper, they proposed a bunch of various different like attention patterns."}, {"start": 90.0, "end": 93.0, "text": " You can see they are fairly complex and intricate."}, {"start": 93.0, "end": 102.0, "text": " And in my honest opinion, when I see something that's so complex and hacky, I just don't think it's going to be there in a couple of years."}, {"start": 102.0, "end": 105.0, "text": " It's just a current way to boost the results."}, {"start": 105.0, "end": 112.0, "text": " And I still think it's useful for the community, but I don't think it's going to work in the long run."}, {"start": 112.0, "end": 115.0, "text": " Same thing with like Reformer here."}, {"start": 115.0, "end": 117.0, "text": " You can see they used locality-sensitive hashing."}, {"start": 117.0, "end": 120.0, "text": " You can see just the number of steps you have to do here."}, {"start": 120.0, "end": 127.0, "text": " So LSH bucketing, sort by the LSH bucket, chunk sort the sequence to parallelize, blah, blah, blah, 10 within the same bucket."}, {"start": 127.0, "end": 128.0, "text": " So it's fairly complex."}, {"start": 128.0, "end": 130.0, "text": " You can see just looking at the graph, it's very complex."}, {"start": 130.0, "end": 132.0, "text": " It doesn't seem like the way to go."}, {"start": 132.0, "end": 135.0, "text": " That's one of my main problems I had with these papers."}, {"start": 135.0, "end": 144.0, "text": " Second thing is the thing I don't like about these Transformer papers, which are trying to make it more efficient, is that they are all using different data sets."}, {"start": 144.0, "end": 147.0, "text": " It's kind of hard to compare apples to apples here."}, {"start": 147.0, "end": 156.0, "text": " So I'd encourage the community to start using like a dedicated set of data sets and try and evaluate all of them on the same suit of tasks."}, {"start": 156.0, "end": 158.0, "text": " That's just my two cents."}, {"start": 158.0, "end": 164.0, "text": " OK, so all of these papers are basically trying to modify this multi-head retention module."}, {"start": 164.0, "end": 166.0, "text": " So this one here."}, {"start": 166.0, "end": 170.0, "text": " So same for reformers, same for longformers, same for many of them."}, {"start": 170.0, "end": 171.0, "text": " That was the basic pain point."}, {"start": 171.0, "end": 174.0, "text": " That's where the quadratic complexity lies in."}, {"start": 174.0, "end": 177.0, "text": " And again, I won't be digging into how Transformer works."}, {"start": 177.0, "end": 180.0, "text": " That's like by now a building block of deep learning."}, {"start": 180.0, "end": 184.0, "text": " So go check out my video on Transformer if you want to learn how it actually works."}, {"start": 184.0, "end": 186.0, "text": " So yeah, that's the motivation."}, {"start": 186.0, "end": 188.0, "text": " We want to reduce the quadratic complexity."}, {"start": 188.0, "end": 191.0, "text": " And now let's see how FastFormer actually managed to achieve that."}, {"start": 191.0, "end": 194.0, "text": " I think the idea is fairly neat and fairly simple."}, {"start": 194.0, "end": 196.0, "text": " So let's dig into it."}, {"start": 196.0, "end": 201.0, "text": " OK, so first things first, we have input sequence, which are embedded into these vectors."}, {"start": 201.0, "end": 206.0, "text": " We have so we have n vectors, which represent the end tokens of the input sequence."}, {"start": 206.0, "end": 211.0, "text": " We have the usual query, transform and key transform and value transform."}, {"start": 211.0, "end": 213.0, "text": " And now the interesting thing happens here."}, {"start": 213.0, "end": 219.0, "text": " So they first take these query vectors and they do additive attention."}, {"start": 219.0, "end": 221.0, "text": " I'm going to explain in a second what that means."}, {"start": 221.0, "end": 223.0, "text": " They do additive attention."}, {"start": 223.0, "end": 225.0, "text": " They form these alpha coefficients."}, {"start": 225.0, "end": 233.0, "text": " They multiply the vectors with the alpha coefficients and they sum them up to achieve this global query vector,"}, {"start": 233.0, "end": 238.0, "text": " which serves as an initial step in creating this global context of this whole sequence."}, {"start": 238.0, "end": 241.0, "text": " Let's see quickly how the additive attention works."}, {"start": 241.0, "end": 243.0, "text": " So we have a vector."}, {"start": 243.0, "end": 244.0, "text": " Let's take like q1, whatever."}, {"start": 244.0, "end": 246.0, "text": " The additive attention is super simple."}, {"start": 246.0, "end": 248.0, "text": " You just take the input vector."}, {"start": 248.0, "end": 254.0, "text": " So q1 and what they're going to do is they're going to just feed it through a fully connected network."}, {"start": 254.0, "end": 262.0, "text": " So every single dimension in this vector, every single feature is going to have associated like a link here with a weight."}, {"start": 262.0, "end": 265.0, "text": " And you basically form that's how you form the alpha coefficient."}, {"start": 265.0, "end": 272.0, "text": " So alpha I in the general case, we basically have the operations in here where these the number of dimensions of the vector."}, {"start": 272.0, "end": 277.0, "text": " So you can see it's fairly simple and computationally super cheap to form these alpha coefficients,"}, {"start": 277.0, "end": 284.0, "text": " which, as I said, then you just use them to do a scalar product of the vector and you form another representation,"}, {"start": 284.0, "end": 286.0, "text": " which is here, and then you just add them up."}, {"start": 286.0, "end": 288.0, "text": " You add up all the vectors."}, {"start": 288.0, "end": 289.0, "text": " You form the Q vector."}, {"start": 289.0, "end": 290.0, "text": " That's it."}, {"start": 290.0, "end": 291.0, "text": " That's how you form the Q vector."}, {"start": 291.0, "end": 296.0, "text": " Then you what you do is you take the Q vector and you modify the key vectors by doing element wise."}, {"start": 296.0, "end": 300.0, "text": " So this this symbol here is just element wise product."}, {"start": 300.0, "end": 301.0, "text": " They did some ablation studies."}, {"start": 301.0, "end": 303.0, "text": " We'll see that in a couple of minutes."}, {"start": 303.0, "end": 314.0, "text": " But basically the concatenation of this global Q vector or the addition did not work as good as as doing this like element wise product."}, {"start": 314.0, "end": 319.0, "text": " Basically, once you do that, you form these P vectors and then you again do the additive attention."}, {"start": 319.0, "end": 322.0, "text": " So the same thing, the same procedure is here."}, {"start": 322.0, "end": 326.0, "text": " And then you add them up and you form the Q the K vector."}, {"start": 326.0, "end": 328.0, "text": " So that's the key global vector."}, {"start": 328.0, "end": 334.0, "text": " Finally, the final step of this of this module is you basically have value vectors."}, {"start": 334.0, "end": 338.0, "text": " What you're going to do is again do the element wise product."}, {"start": 338.0, "end": 340.0, "text": " You're going to form the U vectors."}, {"start": 340.0, "end": 346.0, "text": " And this time, instead of doing additive attention, they did something similar to the original transformer."}, {"start": 346.0, "end": 348.0, "text": " They did simple linear transformation here."}, {"start": 348.0, "end": 353.0, "text": " So this block is just like a set of linear layers or MLPs."}, {"start": 353.0, "end": 357.0, "text": " I think I'm not sure whether they're using single layer or MLP, but that doesn't matter."}, {"start": 357.0, "end": 359.0, "text": " So basically you have a simple transformation, which is learnable."}, {"start": 359.0, "end": 361.0, "text": " You form these R vectors."}, {"start": 361.0, "end": 367.0, "text": " And then finally, you just add them up with the query vectors, which we had here to form the final output."}, {"start": 367.0, "end": 368.0, "text": " And that's it."}, {"start": 368.0, "end": 381.0, "text": " So it's much more simple compared to, in my opinion, compared to long former compared to lean former compared, which is the lower end approximation of the matrix compared to every other paper pretty much."}, {"start": 381.0, "end": 382.0, "text": " Yeah."}, {"start": 382.0, "end": 385.0, "text": " So that's a big plus in my book, at least."}, {"start": 385.0, "end": 390.0, "text": " OK, so everything I just explained is written down in these formulas."}, {"start": 390.0, "end": 394.0, "text": " The only thing I forgot to mention is they have the same as in the original transformer."}, {"start": 394.0, "end": 400.0, "text": " They have this division by the square root of the dimensionality, which serves to keep the things stable."}, {"start": 400.0, "end": 403.0, "text": " And so this is just a scale dot product."}, {"start": 403.0, "end": 405.0, "text": " And that's how I formed the alpha coefficients."}, {"start": 405.0, "end": 412.0, "text": " And then, as I said, you just do weighted like some of these query vectors to form the global query vector."}, {"start": 412.0, "end": 417.0, "text": " And that's it. You do the same thing for for the key for the global key vector, as I already mentioned it here."}, {"start": 417.0, "end": 418.0, "text": " So that's this one."}, {"start": 418.0, "end": 421.0, "text": " And finally, I explained the third step."}, {"start": 421.0, "end": 424.0, "text": " And that's this linear transformation part I explained."}, {"start": 424.0, "end": 428.0, "text": " OK, so a couple of details which are interesting."}, {"start": 428.0, "end": 430.0, "text": " They did some oblations again."}, {"start": 430.0, "end": 434.0, "text": " They mentioned here we shared the value and query transformation parameters to reduce the memory cost."}, {"start": 434.0, "end": 441.0, "text": " In addition, we shared the parameters across different fast formal layers to further reduce the parameter size and mitigate the risk of overfitting."}, {"start": 441.0, "end": 442.0, "text": " So this is nothing new."}, {"start": 442.0, "end": 447.0, "text": " We had a bunch of papers prior to this one, which were experimenting with this weight sharing."}, {"start": 447.0, "end": 454.0, "text": " And so nothing new there, but still that's this is additionally going to save up some parameters."}, {"start": 454.0, "end": 457.0, "text": " And they also didn't have any slowdowns doing this."}, {"start": 457.0, "end": 458.0, "text": " So that's that's cool."}, {"start": 458.0, "end": 465.0, "text": " So if you followed along the explanation of how this thing works, it's easy to notice that the whole complexity will be n times d."}, {"start": 465.0, "end": 468.0, "text": " So that means it's linear with respect to the number of tokens."}, {"start": 468.0, "end": 473.0, "text": " And these just the dimensionality of the vectors which are being passed through the transformer."}, {"start": 473.0, "end": 477.0, "text": " OK, so funny thing, I just kind of had to notice this."}, {"start": 477.0, "end": 481.0, "text": " This must be the most unlucky place to put the footnote."}, {"start": 481.0, "end": 483.0, "text": " But, yeah, I guess they did it on purpose."}, {"start": 483.0, "end": 485.0, "text": " Hopefully they did it on purpose."}, {"start": 485.0, "end": 491.0, "text": " OK, so having said that, let me just show you the complexity compared to the other previous papers."}, {"start": 491.0, "end": 497.0, "text": " So we had transformer, the original transformer, which was obviously squared compared to the input sequence length."}, {"start": 497.0, "end": 506.0, "text": " And that was the problem that that made the yields transformers not good enough to to do inference on long sequences."}, {"start": 506.0, "end": 508.0, "text": " Then we had a bunch of various papers."}, {"start": 508.0, "end": 514.0, "text": " So I mentioned Longformer, I mentioned I don't see reformer here, but yeah, Linformer and other papers."}, {"start": 514.0, "end": 519.0, "text": " So as I said, there is always all of these papers pretty much had some hidden caveat inside."}, {"start": 519.0, "end": 521.0, "text": " So, for example, Longformer had this K."}, {"start": 521.0, "end": 528.0, "text": " And if I remember correctly, you had to push K pretty high in order to get decent performance boost."}, {"start": 528.0, "end": 531.0, "text": " So maybe something like square root of n would be optimal."}, {"start": 531.0, "end": 536.0, "text": " And so you kind of have linear complexity, but you're not quite there."}, {"start": 536.0, "end": 538.0, "text": " Same with our papers."}, {"start": 538.0, "end": 542.0, "text": " You can see some have square dependency on the dimensionality."}, {"start": 542.0, "end": 548.0, "text": " Some have huge constants in front of them, which you don't see because the big O notation just eats it up."}, {"start": 548.0, "end": 553.0, "text": " So I think this former is the first paper to actually have decent linear complexity."}, {"start": 553.0, "end": 554.0, "text": " Let's finally see the results."}, {"start": 554.0, "end": 555.0, "text": " So they have two sections."}, {"start": 555.0, "end": 561.0, "text": " First one is they show that the first former is effective and then they show that it's actually efficient as well."}, {"start": 561.0, "end": 566.0, "text": " So they had, I think, five data sets here to show on Amazon."}, {"start": 566.0, "end": 568.0, "text": " Yeah, I am DB in mind."}, {"start": 568.0, "end": 572.0, "text": " So sentiment and topic classification tasks."}, {"start": 572.0, "end": 579.0, "text": " And you can see that this former pretty much outperforms all of the previous baselines on all of these tasks,"}, {"start": 579.0, "end": 584.0, "text": " except for this one where I guess the difference is fairly negligible and the standard deviation here is even bigger."}, {"start": 584.0, "end": 586.0, "text": " So, yeah."}, {"start": 586.0, "end": 591.0, "text": " But like, as I mentioned, the thing I don't like about everybody's using their own data sets,"}, {"start": 591.0, "end": 598.0, "text": " it'd be nice to have like an agreement of which data sets we should use when we are evaluating these efficient transformers."}, {"start": 598.0, "end": 603.0, "text": " Just putting it out there, maybe somebody starts implementing that idea."}, {"start": 603.0, "end": 606.0, "text": " OK, so that's the being effective."}, {"start": 606.0, "end": 610.0, "text": " And here are some other data sets they used."}, {"start": 610.0, "end": 618.0, "text": " So they have the, I think, recommendation task here and they show again then first former is better compared like across these various different metrics."}, {"start": 618.0, "end": 622.0, "text": " And I don't want anyone get into what these metrics are, but basically it's better."}, {"start": 622.0, "end": 630.0, "text": " And they have this combination with this PLM and our baseline plus ensemble disaster just signifies ensemble."}, {"start": 630.0, "end": 633.0, "text": " And they have even better results that you've even better results by doing that."}, {"start": 633.0, "end": 637.0, "text": " But, yeah, I guess this this row is what is actually important for us."}, {"start": 637.0, "end": 639.0, "text": " Additional results on text summarization."}, {"start": 639.0, "end": 647.0, "text": " Again, fast former is fairly better compared to all of the previous baselines across rows one, rows two, rows L metrics."}, {"start": 647.0, "end": 651.0, "text": " So different metrics. Again, not that important for this paper."}, {"start": 651.0, "end": 656.0, "text": " Before I show you how efficient this fast former is, let me just kind of dig into this part."}, {"start": 656.0, "end": 662.0, "text": " So in our experiments, we use glove embeddings to initialize token embedding matrix."}, {"start": 662.0, "end": 664.0, "text": " I'm not sure why they're using glove here."}, {"start": 664.0, "end": 668.0, "text": " So this is like an alternative to virtual back embeddings."}, {"start": 668.0, "end": 672.0, "text": " Not completely sure why they did not learn the whole thing from scratch."}, {"start": 672.0, "end": 680.0, "text": " If anybody knows what's going on here, let me know down in the comments because I'm this part confused me, to be honest."}, {"start": 680.0, "end": 682.0, "text": " OK, let's see how efficient this thing is."}, {"start": 682.0, "end": 690.0, "text": " So looking at inference, looking at training graphs here, so we can see that as the sequence length increases,"}, {"start": 690.0, "end": 694.0, "text": " somewhere around 512, the transformer starts exploding."}, {"start": 694.0, "end": 697.0, "text": " So the GPU memory is not large enough."}, {"start": 697.0, "end": 701.0, "text": " I think they were using V100 GPUs and VDI."}, {"start": 701.0, "end": 711.0, "text": " And so basically that means you cannot actually even evaluate the transformer on these larger sequences, on these longer sequences."}, {"start": 711.0, "end": 720.0, "text": " Then you can see that this dark blue line is the fast former and it's way better compared to the other baselines."}, {"start": 720.0, "end": 726.0, "text": " So this pooling former, if I'm not wrong, let me just see the theoretical complexity they reported."}, {"start": 726.0, "end": 731.0, "text": " So pooling former n times d times w, where w is the window size."}, {"start": 731.0, "end": 738.0, "text": " And you can see that even though this looks linear in practice, it's not like optimal because they have huge constant."}, {"start": 738.0, "end": 743.0, "text": " And I guess that w also contributes to that constant exploding."}, {"start": 743.0, "end": 748.0, "text": " So yeah, in practice, it doesn't seem to be as good as all of the other baselines."}, {"start": 748.0, "end": 754.0, "text": " Forgot to mention we have here on the y-axis inference time per layer and here we have training time per layer."}, {"start": 754.0, "end": 757.0, "text": " I guess it correlates pretty much these two charts."}, {"start": 757.0, "end": 763.0, "text": " That basically means it's better, it's more efficient for both training and inference compared to the other baselines."}, {"start": 763.0, "end": 765.0, "text": " And I guess that's nice results."}, {"start": 765.0, "end": 767.0, "text": " Finally, here are some ablations."}, {"start": 767.0, "end": 771.0, "text": " I mentioned the Element-Vise product they were using to construct those global vectors."}, {"start": 771.0, "end": 780.0, "text": " You can see that compared to adding up those vectors instead of doing Element-Vise or concatenating them to the key vectors"}, {"start": 780.0, "end": 785.0, "text": " is worse looking at the accuracy on different datasets."}, {"start": 785.0, "end": 791.0, "text": " So the trend is kind of similar and sticks throughout all of these different datasets."}, {"start": 791.0, "end": 796.0, "text": " Finally, I mentioned weight sharing, the de-doblation studies there."}, {"start": 796.0, "end": 802.0, "text": " So without sharing, you can see decent results, but I guess the number of parameters explodes."}, {"start": 802.0, "end": 806.0, "text": " With query value sharing, some improvements."}, {"start": 806.0, "end": 810.0, "text": " And then when you do the head-wise sharing, I think that's important."}, {"start": 810.0, "end": 812.0, "text": " We have a significant drop here."}, {"start": 812.0, "end": 815.0, "text": " And the same trend goes for every single dataset."}, {"start": 815.0, "end": 823.0, "text": " And the whole point is if you read the original transformer paper, and if you ever saw those attention visualizations,"}, {"start": 823.0, "end": 826.0, "text": " every single head is learning different sub-function."}, {"start": 826.0, "end": 832.0, "text": " So sharing the weights there kind of hurts the performance, and that would be some hand-wavy explanation of why that happens."}, {"start": 832.0, "end": 836.0, "text": " So every single one needs to learn a dedicated function."}, {"start": 836.0, "end": 844.0, "text": " For example, if you have a sentence, one of those heads will maybe focus on attending the nouns."}, {"start": 844.0, "end": 847.0, "text": " One will be attending verbs. I don't know, whatever."}, {"start": 847.0, "end": 849.0, "text": " So different semantics all in all."}, {"start": 849.0, "end": 854.0, "text": " So that will be a quick explanation of this transformer paper."}, {"start": 854.0, "end": 861.0, "text": " So again, as I said, what I like about this paper is it's much more simpler implementation-wise"}, {"start": 861.0, "end": 867.0, "text": " than just understanding how it works compared to previous papers I've covered and read,"}, {"start": 867.0, "end": 874.0, "text": " as well as they obviously have both theoretical and experimental guarantees that they are more efficient,"}, {"start": 874.0, "end": 878.0, "text": " and they're also effective across a bunch of different datasets, as we saw."}, {"start": 878.0, "end": 880.0, "text": " So yeah, that's pretty much it."}, {"start": 880.0, "end": 886.0, "text": " And finally, if you missed it, I just created a brand new Discord community, so do join it."}, {"start": 886.0, "end": 888.0, "text": " I'll link the link in the description."}, {"start": 888.0, "end": 893.0, "text": " Basically, you'll be able to engage with others there, ask questions related to ML."}, {"start": 893.0, "end": 898.0, "text": " I'm actually working on bringing startups to offer ML jobs, so if that's something that's interesting to you,"}, {"start": 898.0, "end": 905.0, "text": " do join the community and also subscribe to my monthly AI newsletter, which is going to be super valuable, hopefully."}, {"start": 905.0, "end": 922.0, "text": " So having said that, subscribe to this channel, hit that bell notification, and until next time, bye-bye."}]