ALiBi | Train Short, Test Long: Attention With Linear Biases Enables Input Length Extrapolation
👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I cover ALiBi model from the "Train Short, Test Long: Attention With Linear Biases Enables Input Length Extrapolation" paper. 

Instead of using positional embeddings (like e.g. the original transformer) they added non-learnable biases directly into the query-key matrix and achieved extraordinary extrapolation results (i.e. great perf on longer sequences).

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://ofir.io/train_short_test_long.pdf
✅ Code: https://github.com/ofirpress/attention_with_linear_biases
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
02:00 Main results
03:30 Time and memory tradeoffs
05:00 ALiBi method explained
10:00 Results, low data regime
12:35 Generalization to different datasets
13:15 Results, big data regime
16:00 Why does it work and future work
21:10 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR ML PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#alibi #transformers #extrapolation
 What's up guys, in this video I'm covering yet another transformer paper, train short, test long, attention with linear biases, enables input length extrapolation by Ophir Press, Noah Smith, Mike Lewis from Paul Olin School of Computer Science, FAIR and Allen Institute FAIR for AI. So I mentioned yet another transformer paper, but to be honest, this paper probably has the biggest ratio of performance boost they got over the lines of PyTorch code or whatever your favorite framework is. So they basically they only need a couple lines of code to get the results I'm about to show you and they get pretty decent results as we'll soon see. So the question they open it up this paper with is how to achieve extrapolation at inference time two longer sequences than seen during training. So basically we obviously want to train transformers on way shorter sequences because that way we save up memory, we save up time, computational budget, money, whatnot, and it'd be nice if we could deploy those on unseen longer input sequences. So that's the holy grail I guess. And I think this is a nice step in that direction. So they say that they introduce a simple and efficient method, attention with linear biases or Alibi for short, that allows for extrapolation. Alibi does not add positional embeddings to the word embeddings, instead it biases the query key attention scores with the term that is proportional to their distance. And I'm going to break down this sentence here in a couple of minutes, but like the reason they mentioned positional embeddings is because the original transformer paper Vasvani and his collaborators, they were using sinusoidal like patterns and they were adding them to the input embeddings and only at the input layer. Back then they had a hypothesis that that's going to help using sinusoids instead of using learned embeddings is going to help the extrapolation, but they did not do any serious extrapolation study. And this paper is I think the first one to have done that kind of extrapolation study. And let's see the results right now. So charts are pretty impressive. So here on the left side, you can see the model strain on 512 tokens. And on the right side, we see a model strain on 1024 tokens. So they compared Alibi with three different baselines. So the original transformer, the rotary transformer and the T5 bias baseline. And as you can see, as we increase the inference input tokens, that's further away from the training from what the model saw during the training period, basically can see that the sinusoidal like perplexity just explodes. So again, we have on the y axis perplexity, lower is better. And we see that rotary and T5 bias models perform somewhat better, a lot better. I mean, I'm talking qualitatively here, but still. And finally, Alibi looks like this looks amazing. And to be honest, I'm still puzzled how this is even possible. On the right hand side, we can see qualitatively the same trends. I'm going to skip that chart. Okay. So one thing I mentioned initially is that Alibi can be implemented by changing only a few lines of existing transformer code. And that makes this thing even more impressive because simplicity is the king at the end. If you have some model that achieve some really nice results, but you have to hack it a lot, that's not going to scale. And we had those papers, those transformer papers coming up and only the research groups that actually publish the paper were the ones who implemented it. And that's kind of hard to reproduce. Okay. Having said that, let me show you some other interesting charts. So when we compare again, Alibi with the three baselines we just saw across these three dimensions. So the training speed, the inference speed and the training memory, we're going to see awesome results because sinusoidal, so the original transformer and Alibi are on pair. So you can see here. So this is the training speed, obviously the words per second, the higher the hardest metric is, the better. And you can see that sinusoidal one, so the original transformer and Alibi are on pair across different input lengths. Then we can see inference speed, again, fairly similar across all of the input lengths. Here it's even a bit higher. So that's kind of strange. Not sure why that is. If anybody has an idea, please write it down in the comments. And finally, the training memory again here lower is better. So the number of gigabytes, you can see that sinusoidal and the Alibi are on pair and the rotary and t5 bias, even though they achieve some kind of extrapolation as we saw on the above chart here. So we saw some better performance, obviously compared to sinusoidal one. We see that there is a huge trade-off. And the nice thing about this Alibi paper is that pretty much with zero overhead, so there is some small overhead dimension like 100 megabytes of memory, etc. But let's say zero overhead, they get all of this additional extrapolation capability. And that's pretty amazing, if you ask me. Okay, let's continue and let me try and actually digest how this method works. And it's fairly simple. So by now, hopefully you understand how transformers work. That's a basic building block in deep learning nowadays. Do check out my video on transformers. There are also some nice blogs out there, like J. Alomar's blog is awesome. So do check those out because I'm going to give you only a high level overview here. So short recap of transformers. Let me kind of denote the input vectors like this. So these are input tokens embedded into like vectors. And so all of these lines kind of represent a separate token embedding vector. And so what transformers do is you form, for each of those vectors, you form a query, you form a key vector, and you form, finally, a value vector. And you repeat that same procedure for all of these, right? For all of the input embedding vectors. And so what you want to do is you want to take the query, so query one, and you want to do dot product between query one and key one, and query one and all of the other tokens. And actually, because here we are dealing with language modeling, we have something called causal masking. That means you don't want to have access to the words that come in the future. So what you do in the language modeling task is you're going to take, so let's take this is query three. So query three will only be able to attend to key three, to key two, and to key one. And it won't be able to actually attend, so it won't be able to attend these ones. The ones, the tokens that come afterwards in the sequence. And that's why we see the diagonal pattern right here. So this is just, as you can see, so query one dot product with key one, query two, key one, query two, key two, et cetera. So that's just the thing I just explained here. That's why we see the diagonal pattern. And now these are attention scores, and after that, we do some dropout, we do some softmax, that's your classical, like, attention building block in Transformer. And the novelty here, what they did is instead of adding those sinusoidal embeddings, they just add this simple mask, as you can see the pattern here, which has this recency bias, because you can see that the further away the score is from the current query, so here we have query five, the further away the score is, the higher the penalty here. And so basically, after you apply softmax, the probability, the attention coefficients for that, like, value vector is going to drop down. So that means you're going to take that value vector from the position one with a smaller coefficient compared to value vectors that are close by this Q5 query. So hopefully that was clear. I'm going to dig into a bit more detail a bit later, but yeah, the thing you're probably noticing here is this m parameter, and the funny thing is it's actually not learnable. They just kind of experimentally found some good values, we're going to see those in a minute. And so every single attention head will have like a unique number m, and that's it. That's the whole logic they did, and they achieved those results. Now, let's dig into a bit more details here. So they mentioned here, when using a Libby, would not add position embeddings at any point in the network. The only modification we apply is that after the query key dot product, we add a static non-learned bias. So as I already explained, and here is just an algebraic expression of the thing I just explained above. So m is again a parameter that's dependent on the attention head you're in, and these depend on how far away is the key vector from this current query vector. Then we apply the softmax. So they said here again, m is case specific and fixed before training. And so they have, they experimented with some different things, and they figured out that the geometric progression works the best. So that's kind of similar to what sinusoidal embeddings did, except that they were actually using those geometric progressions inside the sinusoid function, i.e. for the frequency of the sinusoids. And so they did some ablations, they mentioned it here somewhere. So our main insight from this exploration is that the slope sets that work best are the ones in which the slopes are in the 0 to 1 range, with the density of the slopes increasing as we get closer to 0. And as we see, these sequences do obey those properties pretty much. They're between 0 and 1, and we see that it gets denser as we get closer to 0, because yeah, 2 to the 8, and then we have 2 to the power of 9. So it's getting closer and closer, it's converging to 0 as the n, where n is the exponent, goes to infinity. That's me trying to be mathematically rigorous, but it's not working. Okay, I think you got an idea of how this thing works. Now let's finally see the results. So they first tested on this wiki text 103 dataset, and let's see the results here. We can see here that the model trained on 512, so Olivia model trained on 512, has lower perplexity than the sinusoidal model trained on the same input length. But the funny thing is, as you're increasing the input length, so the validation sequence length, it still has better perplexity compared to those models that were actually trained on that very same input sequence length. So what I mean by that is, you take a model that was trained on 512, and you do an inference on 3072, and it's still better, as you can see here, let me zoom in, it's still better than the actual model that was trained on that very same sequence. So that's pretty impressive. Now keep in mind, and we'll see that a bit later, that this dataset and this transformer model, fairly small compared to the next results we'll see. And as we'll soon see, these advantages that Olivia is bringing are much more pronounced in the lower data regime, which makes sense, because biases are always helpful when you have a low data regime. We saw that multiple times. Vision transformers versus CNNs would be one nice example where vision transformers in the big data regime start outperforming CNNs, because they learn even better biases by themselves. Let's put it that way, yeah. To be able to fully appreciate these results, I kind of highlighted this paragraph here. So the L512 model is 1.84 times faster to train, and yet it still beats the 3072 input token length model when extrapolating to that length. In addition, training the 3072 sinusoidal model requires a GPU with more than 16 gigabytes of memory to fit the large attention matrices, but our model beats its performance, even though it can be trained on the GPU with much less memory due to the much smaller attention matrices. Okay, pretty impressive results for this low data regime. They have one more chart where they show pretty much the same information. So, again, training the sinusoidal model with 3072 has a bigger perplexity compared to a model that was trained with only 512 tokens, and it's, as you can see here, way faster to train, because on the x-axis we have words per second during the training time and still a lower perplexity, and the evaluation is on this sequence, so that benefits this model. So, again, just another chart that shows that they are better in this low data regime. Okay, not that interesting. Let's go further on. So this one is interesting because here they completely changed the data statistics. So they are using Toronto Book Corpus, which is a completely different data set, and they show that the same trends still apply, so training the Alibi model and then extrapolating is still better than actually training the sinusoidal model on that, as you can see, 3072 input sequence length. That means that the Alibi model is generalizing to different data sets, and the next thing they kind of showed in their experiments is that they also scale, they also, this method generalizes to the big data regime as well, although, as we'll see, the performance drops, the difference between the two models drops. So here on the left-hand side we can see the model evaluated on 1024, whereas Alibi is trained with 512, the sinusoidal one is trained with 1024, and we can see perplexity is pretty much on pair, like Alibi is a bit worse here compared to the sinusoidal one, but you gotta have in mind that the memory consumption here is lower and it's faster to train, because it has shorter input sequence length. When we go to 2048, you can see that it's pretty much constantly better by some small margin, but still, and it's faster to train, smaller memory again, so that's cool, but like the results are not as drastic compared to the loaded regime. As a quick notice, the models they're using here have 1.3 billion parameters, so that's just, you have like a number in your head. Final results here, they showed that basically there is this local, like local minimum, where they showed that the minimum actually lies somewhere around two times the input sequence length that was used for training, so if the model was trained on 512, the minimum is around 1024, and that's interesting, and as you can see here, then it kind of rises up, the complexity goes up, but it's fairly stable up to 10k tokens, and as you can see here, the sinusoidal one just explodes as soon as you start increasing from 512 towards 1000, it already explodes, and so they don't even have a curve here. Same thing here, just for the 1024 model, similar trend, as you can see, the minimum lies somewhere around 2048 or something. Okay, finally, before digging into some ablations, let's see this one. So this table in the appendix has additional results comparing our models to the sinusoidal baseline when both are trained on the same L, showing that Alibi performs similarly to the sinusoidal baseline when not extrapolating. This is in contrast to the results presented on the smaller datasets, where Alibi consistently outperformed other position methods even when not extrapolating. This hints that Alibi's inductive bias provides additional benefits for lower resource language modeling. So that's something I already mentioned, and I think it's fairly important to understand that this type of bias works best for a low data regime, and they showed that across this paper. Okay, finally, they have some analysis for why these models are extrapolating that well. Okay, so the thing I forgot to mention is that how they're actually evaluating the perplexity, and what they did, they were using this non-overlapping inference method whereby you put all of these words as the context, as the input to Transformer, and then what happens is when you're predicting big, the only context you have is the word d. When you're predicting gray, you have d big, and when you're predicting cat, you have d big gray cat, and then you move on to this other subsequence set on the mat, and again, on will only have, like as a context, this word set, d will have these two words, and mat will have the full context, it's the three words. So, compare that to this overlapping inference whereby, again, the first words will have like little context, and then, so cat will have three words as a context, and then what they'll do is they'll have this sliding window, so this will be, now we'll have these words input as the, let me change the color here, so we'll have these words, big gray cat set, and so the set word will actually have all of these three as the context, whereas in the non-overlapping case, it did not have any, so this method is obviously better because it has more context. On the con side, the trade-off is this is way slower and obviously computationally more expensive, so usually people just do these non-overlapping inferences. Second thing, there is this thing called early token curse, and you can kind of deduce what this is, so that means that, as you saw here, when you're doing these non-overlapping inference, like the early tokens have less context by the very nature of a sequence than the later tokens, so that's called the early token curse, and one of the ways you can kind of deal with it is using the sliding window approach, whereas only a couple of initial tokens will have that problem of lack of context. Okay, so here, they, what it did is they now try to do this sliding window inference instead of doing the non-overlapping method, and they show some results here, and to be honest, I'm a bit puzzled about these results. Okay, so first let me try and explain what I think they've done here. So if we have a sequence that's like 1024, and the model was trained on 512, so let's take this model here. So in previous charts, what we saw was that the model was fed these chunks, so the model would be fed 512 tokens here, it would do the inference, and then would feed these 512 tokens. So the thing that happens is now when you try and feed this whole sequence all at once, the perplexity starts increasing compared to the results we saw previously, and that's especially clear on this chart here. So it's constantly going up and up. So that's kind of sign that it's not learning how to use up that additional context that comes from 1024 tokens, so that means it's actually not extrapolating in this different sense of the word extrapolation, I guess. So they have a couple of hypotheses for why these models are actually performing better on longer input sequences when they're doing these non-overlapping like inference, and they kind of accepted that this second one makes more sense, so the model performs better because longer input sequences provide more context, and so the early token curse is reduced. When we validate a validation set with a model with some input subsequence length L valid, we're actually making L valid predictions, where the first prediction has just one token of context, the second has two, and so on. So the average prediction gets to use this many tokens of previous context. When we extrapolate and increase L valid, the amount of tokens of context we have in the average prediction also increases, and so the performance improvement might not come from being able to do better on the longer sequences, but because our average number of context tokens has become larger for each prediction. I'm not sure how this makes sense. So basically, if you have a really long sequence, what they do is when they're doing the non-overlapping inference, and I guess these are trying to explain exactly how those results are better on longer sequences, I mean, you're gonna chunk this up into, let's say, 512, and that means every time when you come here, you're basically, these initial tokens will not have the previous context, so they will not be able to reference this information here. So I don't see how this is actually reducing the, like alleviating this early token curse. I may have misunderstood something really badly here, but anyways, I encourage you to read this section and also comment down below what you think about this. So as a final thought, I'm gonna leave you with this sentence. So this analysis reveals that when L-valid is bigger than the training input sequence length, Alibi might not be using context longer than the ones it was trained on, and this points to research direction which could be pursued in future work. That's it for this video. If you found it useful, share it out with a friend, subscribe, hit that bell icon, and also join the Discord community if you wanna connect with other ML people, and there's gonna be a lot of goodies happening in that community. So yeah, until next time, bye bye.
[{"start": 0.0, "end": 5.5200000000000005, "text": " What's up guys, in this video I'm covering yet another transformer paper, train short, test long,"}, {"start": 5.5200000000000005, "end": 11.28, "text": " attention with linear biases, enables input length extrapolation by Ophir Press, Noah Smith,"}, {"start": 11.28, "end": 16.88, "text": " Mike Lewis from Paul Olin School of Computer Science, FAIR and Allen Institute FAIR for AI."}, {"start": 16.88, "end": 21.36, "text": " So I mentioned yet another transformer paper, but to be honest, this paper probably has the"}, {"start": 21.36, "end": 28.16, "text": " biggest ratio of performance boost they got over the lines of PyTorch code or whatever your favorite"}, {"start": 28.16, "end": 35.2, "text": " framework is. So they basically they only need a couple lines of code to get the results I'm"}, {"start": 35.2, "end": 39.76, "text": " about to show you and they get pretty decent results as we'll soon see. So the question they"}, {"start": 39.76, "end": 44.480000000000004, "text": " open it up this paper with is how to achieve extrapolation at inference time two longer"}, {"start": 44.480000000000004, "end": 50.0, "text": " sequences than seen during training. So basically we obviously want to train transformers on way"}, {"start": 50.0, "end": 54.72, "text": " shorter sequences because that way we save up memory, we save up time, computational budget,"}, {"start": 54.72, "end": 62.08, "text": " money, whatnot, and it'd be nice if we could deploy those on unseen longer input sequences."}, {"start": 62.08, "end": 66.32, "text": " So that's the holy grail I guess. And I think this is a nice step in that direction. So"}, {"start": 67.28, "end": 71.75999999999999, "text": " they say that they introduce a simple and efficient method, attention with linear biases or"}, {"start": 71.75999999999999, "end": 77.44, "text": " Alibi for short, that allows for extrapolation. Alibi does not add positional embeddings to the"}, {"start": 77.44, "end": 82.56, "text": " word embeddings, instead it biases the query key attention scores with the term that is proportional"}, {"start": 82.56, "end": 86.96000000000001, "text": " to their distance. And I'm going to break down this sentence here in a couple of minutes,"}, {"start": 86.96000000000001, "end": 90.64, "text": " but like the reason they mentioned positional embeddings is because the original transformer"}, {"start": 90.64, "end": 97.36, "text": " paper Vasvani and his collaborators, they were using sinusoidal like patterns and they were"}, {"start": 97.36, "end": 103.68, "text": " adding them to the input embeddings and only at the input layer. Back then they had a hypothesis"}, {"start": 103.68, "end": 108.08, "text": " that that's going to help using sinusoids instead of using learned embeddings is going to help the"}, {"start": 108.08, "end": 113.28, "text": " extrapolation, but they did not do any serious extrapolation study. And this paper is I think"}, {"start": 113.28, "end": 118.16, "text": " the first one to have done that kind of extrapolation study. And let's see the results right now."}, {"start": 119.28, "end": 127.52, "text": " So charts are pretty impressive. So here on the left side, you can see the model strain on 512"}, {"start": 127.52, "end": 136.56, "text": " tokens. And on the right side, we see a model strain on 1024 tokens. So they compared Alibi with"}, {"start": 136.56, "end": 142.08, "text": " three different baselines. So the original transformer, the rotary transformer and the T5"}, {"start": 142.8, "end": 149.6, "text": " bias baseline. And as you can see, as we increase the inference input tokens, that's further away"}, {"start": 149.6, "end": 154.96, "text": " from the training from what the model saw during the training period, basically can see that the"}, {"start": 154.96, "end": 161.84, "text": " sinusoidal like perplexity just explodes. So again, we have on the y axis perplexity, lower is better."}, {"start": 161.84, "end": 168.56, "text": " And we see that rotary and T5 bias models perform somewhat better, a lot better. I mean,"}, {"start": 169.12, "end": 174.24, "text": " I'm talking qualitatively here, but still. And finally, Alibi looks like this looks amazing."}, {"start": 174.24, "end": 178.4, "text": " And to be honest, I'm still puzzled how this is even possible. On the right hand side, we can see"}, {"start": 178.4, "end": 184.08, "text": " qualitatively the same trends. I'm going to skip that chart. Okay. So one thing I mentioned"}, {"start": 184.08, "end": 188.64000000000001, "text": " initially is that Alibi can be implemented by changing only a few lines of existing transformer"}, {"start": 188.64, "end": 193.35999999999999, "text": " code. And that makes this thing even more impressive because simplicity is the king at the end. If you"}, {"start": 193.35999999999999, "end": 197.92, "text": " have some model that achieve some really nice results, but you have to hack it a lot, that's"}, {"start": 197.92, "end": 202.48, "text": " not going to scale. And we had those papers, those transformer papers coming up and only the research"}, {"start": 202.48, "end": 208.32, "text": " groups that actually publish the paper were the ones who implemented it. And that's kind of hard"}, {"start": 208.32, "end": 216.32, "text": " to reproduce. Okay. Having said that, let me show you some other interesting charts. So when we"}, {"start": 216.32, "end": 222.79999999999998, "text": " compare again, Alibi with the three baselines we just saw across these three dimensions. So the"}, {"start": 222.79999999999998, "end": 228.4, "text": " training speed, the inference speed and the training memory, we're going to see awesome results"}, {"start": 228.4, "end": 233.84, "text": " because sinusoidal, so the original transformer and Alibi are on pair. So you can see here."}, {"start": 234.64, "end": 240.0, "text": " So this is the training speed, obviously the words per second, the higher the hardest metric is,"}, {"start": 240.0, "end": 245.28, "text": " the better. And you can see that sinusoidal one, so the original transformer and Alibi"}, {"start": 245.28, "end": 252.32, "text": " are on pair across different input lengths. Then we can see inference speed, again, fairly similar"}, {"start": 252.32, "end": 257.28, "text": " across all of the input lengths. Here it's even a bit higher. So that's kind of strange. Not sure"}, {"start": 257.28, "end": 264.32, "text": " why that is. If anybody has an idea, please write it down in the comments. And finally,"}, {"start": 264.32, "end": 268.72, "text": " the training memory again here lower is better. So the number of gigabytes, you can see that"}, {"start": 268.72, "end": 275.52000000000004, "text": " sinusoidal and the Alibi are on pair and the rotary and t5 bias, even though they achieve some"}, {"start": 275.52000000000004, "end": 280.56, "text": " kind of extrapolation as we saw on the above chart here. So we saw some better performance,"}, {"start": 280.56, "end": 286.48, "text": " obviously compared to sinusoidal one. We see that there is a huge trade-off. And the nice thing"}, {"start": 286.48, "end": 290.96000000000004, "text": " about this Alibi paper is that pretty much with zero overhead, so there is some small overhead"}, {"start": 290.96000000000004, "end": 296.48, "text": " dimension like 100 megabytes of memory, etc. But let's say zero overhead, they get all of this"}, {"start": 296.48, "end": 302.24, "text": " additional extrapolation capability. And that's pretty amazing, if you ask me. Okay, let's continue"}, {"start": 302.24, "end": 307.44, "text": " and let me try and actually digest how this method works. And it's fairly simple. So by now,"}, {"start": 307.44, "end": 311.92, "text": " hopefully you understand how transformers work. That's a basic building block in deep learning"}, {"start": 311.92, "end": 316.8, "text": " nowadays. Do check out my video on transformers. There are also some nice blogs out there, like"}, {"start": 316.8, "end": 320.48, "text": " J. Alomar's blog is awesome. So do check those out because I'm going to give you only a high"}, {"start": 320.48, "end": 327.76, "text": " level overview here. So short recap of transformers. Let me kind of denote the input vectors like this."}, {"start": 327.76, "end": 334.56, "text": " So these are input tokens embedded into like vectors. And so all of these lines kind of"}, {"start": 334.56, "end": 340.8, "text": " represent a separate token embedding vector. And so what transformers do is you form, for each of"}, {"start": 340.8, "end": 348.72, "text": " those vectors, you form a query, you form a key vector, and you form, finally, a value vector."}, {"start": 348.72, "end": 353.92, "text": " And you repeat that same procedure for all of these, right? For all of the input embedding"}, {"start": 353.92, "end": 361.6, "text": " vectors. And so what you want to do is you want to take the query, so query one, and you want to do"}, {"start": 361.6, "end": 368.40000000000003, "text": " dot product between query one and key one, and query one and all of the other tokens. And actually,"}, {"start": 368.40000000000003, "end": 373.68, "text": " because here we are dealing with language modeling, we have something called causal masking. That means"}, {"start": 373.68, "end": 378.64, "text": " you don't want to have access to the words that come in the future. So what you do in the language"}, {"start": 378.64, "end": 384.40000000000003, "text": " modeling task is you're going to take, so let's take this is query three. So query three will only"}, {"start": 384.40000000000003, "end": 393.52, "text": " be able to attend to key three, to key two, and to key one. And it won't be able to actually attend,"}, {"start": 393.52, "end": 399.76, "text": " so it won't be able to attend these ones. The ones, the tokens that come afterwards in the"}, {"start": 399.76, "end": 405.52, "text": " sequence. And that's why we see the diagonal pattern right here. So this is just, as you can see, so"}, {"start": 405.52, "end": 411.2, "text": " query one dot product with key one, query two, key one, query two, key two, et cetera. So that's just"}, {"start": 411.2, "end": 416.56, "text": " the thing I just explained here. That's why we see the diagonal pattern. And now these are attention"}, {"start": 416.56, "end": 422.15999999999997, "text": " scores, and after that, we do some dropout, we do some softmax, that's your classical, like, attention"}, {"start": 422.15999999999997, "end": 427.76, "text": " building block in Transformer. And the novelty here, what they did is instead of adding those"}, {"start": 427.76, "end": 435.52, "text": " sinusoidal embeddings, they just add this simple mask, as you can see the pattern here, which has"}, {"start": 435.52, "end": 440.8, "text": " this recency bias, because you can see that the further away the score is from the current query,"}, {"start": 441.36, "end": 448.0, "text": " so here we have query five, the further away the score is, the higher the penalty here. And so"}, {"start": 448.0, "end": 452.8, "text": " basically, after you apply softmax, the probability, the attention coefficients for that,"}, {"start": 452.8, "end": 457.36, "text": " like, value vector is going to drop down. So that means you're going to take that value vector from"}, {"start": 457.36, "end": 463.6, "text": " the position one with a smaller coefficient compared to value vectors that are close by this"}, {"start": 463.6, "end": 468.88, "text": " Q5 query. So hopefully that was clear. I'm going to dig into a bit more detail a bit later, but yeah,"}, {"start": 469.68, "end": 474.64, "text": " the thing you're probably noticing here is this m parameter, and the funny thing is it's actually"}, {"start": 474.64, "end": 480.8, "text": " not learnable. They just kind of experimentally found some good values, we're going to see those"}, {"start": 480.8, "end": 488.56, "text": " in a minute. And so every single attention head will have like a unique number m, and that's it."}, {"start": 488.56, "end": 493.68, "text": " That's the whole logic they did, and they achieved those results. Now, let's dig into a bit more"}, {"start": 493.68, "end": 500.88, "text": " details here. So they mentioned here, when using a Libby, would not add position embeddings at any"}, {"start": 500.88, "end": 505.52, "text": " point in the network. The only modification we apply is that after the query key dot product,"}, {"start": 505.52, "end": 510.24, "text": " we add a static non-learned bias. So as I already explained, and here is just an algebraic expression"}, {"start": 510.24, "end": 516.32, "text": " of the thing I just explained above. So m is again a parameter that's dependent on the attention head"}, {"start": 516.32, "end": 522.96, "text": " you're in, and these depend on how far away is the key vector from this current query vector."}, {"start": 522.96, "end": 528.48, "text": " Then we apply the softmax. So they said here again, m is case specific and fixed before training."}, {"start": 528.48, "end": 534.72, "text": " And so they have, they experimented with some different things, and they figured out that the"}, {"start": 534.72, "end": 541.52, "text": " geometric progression works the best. So that's kind of similar to what sinusoidal embeddings did,"}, {"start": 541.52, "end": 546.5600000000001, "text": " except that they were actually using those geometric progressions inside the sinusoid function,"}, {"start": 546.5600000000001, "end": 553.2, "text": " i.e. for the frequency of the sinusoids. And so they did some ablations, they mentioned it here"}, {"start": 553.2, "end": 559.6, "text": " somewhere. So our main insight from this exploration is that the slope sets that work best are the ones"}, {"start": 559.6, "end": 565.12, "text": " in which the slopes are in the 0 to 1 range, with the density of the slopes increasing as we get"}, {"start": 565.12, "end": 572.5600000000001, "text": " closer to 0. And as we see, these sequences do obey those properties pretty much. They're"}, {"start": 572.5600000000001, "end": 578.5600000000001, "text": " between 0 and 1, and we see that it gets denser as we get closer to 0, because yeah, 2 to the 8,"}, {"start": 578.5600000000001, "end": 582.64, "text": " and then we have 2 to the power of 9. So it's getting closer and closer, it's converging to 0"}, {"start": 582.64, "end": 587.6, "text": " as the n, where n is the exponent, goes to infinity. That's me trying to be mathematically"}, {"start": 587.6, "end": 594.32, "text": " rigorous, but it's not working. Okay, I think you got an idea of how this thing works. Now let's"}, {"start": 594.96, "end": 602.5600000000001, "text": " finally see the results. So they first tested on this wiki text 103 dataset, and let's see the"}, {"start": 602.5600000000001, "end": 608.72, "text": " results here. We can see here that the model trained on 512, so Olivia model trained on 512,"}, {"start": 609.6, "end": 616.1600000000001, "text": " has lower perplexity than the sinusoidal model trained on the same input length. But the funny"}, {"start": 616.16, "end": 622.8, "text": " thing is, as you're increasing the input length, so the validation sequence length, it still has"}, {"start": 622.8, "end": 628.8, "text": " better perplexity compared to those models that were actually trained on that very same input"}, {"start": 628.8, "end": 634.56, "text": " sequence length. So what I mean by that is, you take a model that was trained on 512, and you do"}, {"start": 634.56, "end": 640.8, "text": " an inference on 3072, and it's still better, as you can see here, let me zoom in, it's still better"}, {"start": 640.8, "end": 646.56, "text": " than the actual model that was trained on that very same sequence. So that's pretty impressive."}, {"start": 646.56, "end": 651.52, "text": " Now keep in mind, and we'll see that a bit later, that this dataset and this transformer model,"}, {"start": 651.52, "end": 658.3199999999999, "text": " fairly small compared to the next results we'll see. And as we'll soon see, these advantages"}, {"start": 658.3199999999999, "end": 663.1999999999999, "text": " that Olivia is bringing are much more pronounced in the lower data regime, which makes sense,"}, {"start": 663.1999999999999, "end": 667.76, "text": " because biases are always helpful when you have a low data regime. We saw that multiple times."}, {"start": 667.76, "end": 673.12, "text": " Vision transformers versus CNNs would be one nice example where vision transformers in the big data"}, {"start": 673.12, "end": 679.4399999999999, "text": " regime start outperforming CNNs, because they learn even better biases by themselves. Let's"}, {"start": 679.4399999999999, "end": 684.24, "text": " put it that way, yeah. To be able to fully appreciate these results, I kind of highlighted"}, {"start": 684.24, "end": 693.92, "text": " this paragraph here. So the L512 model is 1.84 times faster to train, and yet it still beats the"}, {"start": 693.92, "end": 703.5999999999999, "text": " 3072 input token length model when extrapolating to that length. In addition, training the 3072"}, {"start": 703.5999999999999, "end": 708.88, "text": " sinusoidal model requires a GPU with more than 16 gigabytes of memory to fit the large attention"}, {"start": 708.88, "end": 713.04, "text": " matrices, but our model beats its performance, even though it can be trained on the GPU with"}, {"start": 713.04, "end": 719.36, "text": " much less memory due to the much smaller attention matrices. Okay, pretty impressive results for this"}, {"start": 719.36, "end": 724.08, "text": " low data regime. They have one more chart where they show pretty much the same information. So,"}, {"start": 724.08, "end": 732.96, "text": " again, training the sinusoidal model with 3072 has a bigger perplexity compared to a model that"}, {"start": 732.96, "end": 741.12, "text": " was trained with only 512 tokens, and it's, as you can see here, way faster to train, because on the"}, {"start": 741.12, "end": 747.2, "text": " x-axis we have words per second during the training time and still a lower perplexity, and the"}, {"start": 747.2, "end": 754.72, "text": " evaluation is on this sequence, so that benefits this model. So, again, just another chart that"}, {"start": 754.72, "end": 758.96, "text": " shows that they are better in this low data regime. Okay, not that interesting. Let's go further on."}, {"start": 759.76, "end": 765.12, "text": " So this one is interesting because here they completely changed the data statistics. So they"}, {"start": 765.12, "end": 772.1600000000001, "text": " are using Toronto Book Corpus, which is a completely different data set, and they show that the same"}, {"start": 772.16, "end": 778.88, "text": " trends still apply, so training the Alibi model and then extrapolating is still better than actually"}, {"start": 778.88, "end": 785.6, "text": " training the sinusoidal model on that, as you can see, 3072 input sequence length. That means that"}, {"start": 785.6, "end": 792.24, "text": " the Alibi model is generalizing to different data sets, and the next thing they kind of showed in"}, {"start": 792.24, "end": 798.0799999999999, "text": " their experiments is that they also scale, they also, this method generalizes to the big data"}, {"start": 798.08, "end": 804.72, "text": " regime as well, although, as we'll see, the performance drops, the difference between the"}, {"start": 804.72, "end": 811.36, "text": " two models drops. So here on the left-hand side we can see the model evaluated on 1024,"}, {"start": 811.36, "end": 818.32, "text": " whereas Alibi is trained with 512, the sinusoidal one is trained with 1024, and we can see perplexity"}, {"start": 818.32, "end": 825.36, "text": " is pretty much on pair, like Alibi is a bit worse here compared to the sinusoidal one, but you gotta"}, {"start": 825.36, "end": 831.44, "text": " have in mind that the memory consumption here is lower and it's faster to train, because it has"}, {"start": 831.44, "end": 840.32, "text": " shorter input sequence length. When we go to 2048, you can see that it's pretty much constantly better"}, {"start": 840.32, "end": 845.6, "text": " by some small margin, but still, and it's faster to train, smaller memory again, so that's cool,"}, {"start": 845.6, "end": 850.48, "text": " but like the results are not as drastic compared to the loaded regime. As a quick notice, the models"}, {"start": 850.48, "end": 856.88, "text": " they're using here have 1.3 billion parameters, so that's just, you have like a number in your head."}, {"start": 858.72, "end": 865.84, "text": " Final results here, they showed that basically there is this local, like local minimum,"}, {"start": 866.72, "end": 873.04, "text": " where they showed that the minimum actually lies somewhere around two times the input sequence"}, {"start": 873.04, "end": 879.84, "text": " length that was used for training, so if the model was trained on 512, the minimum is around 1024,"}, {"start": 879.84, "end": 884.8000000000001, "text": " and that's interesting, and as you can see here, then it kind of rises up, the complexity goes up,"}, {"start": 884.8000000000001, "end": 893.0400000000001, "text": " but it's fairly stable up to 10k tokens, and as you can see here, the sinusoidal one just explodes"}, {"start": 893.0400000000001, "end": 899.2, "text": " as soon as you start increasing from 512 towards 1000, it already explodes, and so they don't even"}, {"start": 899.2, "end": 905.9200000000001, "text": " have a curve here. Same thing here, just for the 1024 model, similar trend, as you can see,"}, {"start": 905.92, "end": 912.0799999999999, "text": " the minimum lies somewhere around 2048 or something. Okay, finally, before digging into"}, {"start": 912.0799999999999, "end": 918.56, "text": " some ablations, let's see this one. So this table in the appendix has additional results comparing"}, {"start": 918.56, "end": 923.4399999999999, "text": " our models to the sinusoidal baseline when both are trained on the same L, showing that Alibi"}, {"start": 923.4399999999999, "end": 927.92, "text": " performs similarly to the sinusoidal baseline when not extrapolating. This is in contrast to the"}, {"start": 927.92, "end": 933.92, "text": " results presented on the smaller datasets, where Alibi consistently outperformed other position"}, {"start": 933.92, "end": 941.76, "text": " methods even when not extrapolating. This hints that Alibi's inductive bias provides"}, {"start": 941.76, "end": 950.16, "text": " additional benefits for lower resource language modeling. So that's something I already mentioned,"}, {"start": 950.16, "end": 956.4799999999999, "text": " and I think it's fairly important to understand that this type of bias works best for a low data"}, {"start": 956.48, "end": 966.16, "text": " regime, and they showed that across this paper. Okay, finally, they have some analysis for why"}, {"start": 966.16, "end": 971.2, "text": " these models are extrapolating that well. Okay, so the thing I forgot to mention is that how they're"}, {"start": 971.2, "end": 976.4, "text": " actually evaluating the perplexity, and what they did, they were using this non-overlapping inference"}, {"start": 976.4, "end": 982.88, "text": " method whereby you put all of these words as the context, as the input to Transformer,"}, {"start": 982.88, "end": 988.0, "text": " and then what happens is when you're predicting big, the only context you have is the word d."}, {"start": 988.0, "end": 994.64, "text": " When you're predicting gray, you have d big, and when you're predicting cat, you have d big gray"}, {"start": 994.64, "end": 1001.4399999999999, "text": " cat, and then you move on to this other subsequence set on the mat, and again, on will only have,"}, {"start": 1001.4399999999999, "end": 1006.72, "text": " like as a context, this word set, d will have these two words, and mat will have the full context,"}, {"start": 1006.72, "end": 1013.6, "text": " it's the three words. So, compare that to this overlapping inference whereby, again, the first"}, {"start": 1013.6, "end": 1019.36, "text": " words will have like little context, and then, so cat will have three words as a context, and then"}, {"start": 1019.36, "end": 1024.56, "text": " what they'll do is they'll have this sliding window, so this will be, now we'll have these words"}, {"start": 1025.1200000000001, "end": 1031.52, "text": " input as the, let me change the color here, so we'll have these words, big gray cat set, and so the"}, {"start": 1031.52, "end": 1036.8799999999999, "text": " set word will actually have all of these three as the context, whereas in the non-overlapping case,"}, {"start": 1037.6, "end": 1042.96, "text": " it did not have any, so this method is obviously better because it has more context. On the con"}, {"start": 1042.96, "end": 1048.16, "text": " side, the trade-off is this is way slower and obviously computationally more expensive, so"}, {"start": 1048.16, "end": 1053.76, "text": " usually people just do these non-overlapping inferences. Second thing, there is this thing"}, {"start": 1053.76, "end": 1060.56, "text": " called early token curse, and you can kind of deduce what this is, so that means that, as you saw"}, {"start": 1060.56, "end": 1066.56, "text": " here, when you're doing these non-overlapping inference, like the early tokens have less context"}, {"start": 1066.56, "end": 1075.04, "text": " by the very nature of a sequence than the later tokens, so that's called the early token curse,"}, {"start": 1075.04, "end": 1079.9199999999998, "text": " and one of the ways you can kind of deal with it is using the sliding window approach, whereas"}, {"start": 1079.9199999999998, "end": 1087.84, "text": " only a couple of initial tokens will have that problem of lack of context. Okay, so here,"}, {"start": 1087.84, "end": 1093.28, "text": " they, what it did is they now try to do this sliding window inference instead of doing the"}, {"start": 1093.28, "end": 1100.56, "text": " non-overlapping method, and they show some results here, and to be honest, I'm a bit puzzled about"}, {"start": 1100.56, "end": 1106.08, "text": " these results. Okay, so first let me try and explain what I think they've done here. So if we have a"}, {"start": 1106.08, "end": 1112.8, "text": " sequence that's like 1024, and the model was trained on 512, so let's take this model here."}, {"start": 1112.8, "end": 1120.6399999999999, "text": " So in previous charts, what we saw was that the model was fed these chunks, so the model would be"}, {"start": 1120.6399999999999, "end": 1128.3999999999999, "text": " fed 512 tokens here, it would do the inference, and then would feed these 512 tokens. So the thing"}, {"start": 1128.3999999999999, "end": 1135.28, "text": " that happens is now when you try and feed this whole sequence all at once, the perplexity starts"}, {"start": 1135.28, "end": 1140.24, "text": " increasing compared to the results we saw previously, and that's especially clear on this"}, {"start": 1140.24, "end": 1146.72, "text": " chart here. So it's constantly going up and up. So that's kind of sign that it's not learning how to"}, {"start": 1148.88, "end": 1154.4, "text": " use up that additional context that comes from 1024 tokens, so that means it's actually not"}, {"start": 1154.4, "end": 1161.2, "text": " extrapolating in this different sense of the word extrapolation, I guess. So they have a couple of"}, {"start": 1161.2, "end": 1165.36, "text": " hypotheses for why these models are actually performing better on longer input sequences when"}, {"start": 1165.36, "end": 1171.9199999999998, "text": " they're doing these non-overlapping like inference, and they kind of accepted that this second one"}, {"start": 1171.9199999999998, "end": 1177.04, "text": " makes more sense, so the model performs better because longer input sequences provide more context,"}, {"start": 1177.04, "end": 1182.7199999999998, "text": " and so the early token curse is reduced. When we validate a validation set with a model with some"}, {"start": 1182.7199999999998, "end": 1189.04, "text": " input subsequence length L valid, we're actually making L valid predictions, where the first"}, {"start": 1189.04, "end": 1195.12, "text": " prediction has just one token of context, the second has two, and so on. So the average prediction"}, {"start": 1195.12, "end": 1201.1999999999998, "text": " gets to use this many tokens of previous context. When we extrapolate and increase L valid, the"}, {"start": 1201.1999999999998, "end": 1206.0, "text": " amount of tokens of context we have in the average prediction also increases, and so the performance"}, {"start": 1206.0, "end": 1211.12, "text": " improvement might not come from being able to do better on the longer sequences, but because our"}, {"start": 1211.12, "end": 1218.0, "text": " average number of context tokens has become larger for each prediction. I'm not sure how this makes"}, {"start": 1218.0, "end": 1225.52, "text": " sense. So basically, if you have a really long sequence, what they do is when they're doing the"}, {"start": 1225.52, "end": 1233.28, "text": " non-overlapping inference, and I guess these are trying to explain exactly how those results are"}, {"start": 1233.28, "end": 1240.08, "text": " better on longer sequences, I mean, you're gonna chunk this up into, let's say, 512, and that means"}, {"start": 1240.08, "end": 1246.24, "text": " every time when you come here, you're basically, these initial tokens will not have the previous"}, {"start": 1246.24, "end": 1252.64, "text": " context, so they will not be able to reference this information here. So I don't see how this is"}, {"start": 1252.64, "end": 1261.84, "text": " actually reducing the, like alleviating this early token curse. I may have misunderstood something"}, {"start": 1261.84, "end": 1268.4, "text": " really badly here, but anyways, I encourage you to read this section and also comment down below"}, {"start": 1268.4, "end": 1272.48, "text": " what you think about this. So as a final thought, I'm gonna leave you with this sentence. So this"}, {"start": 1272.48, "end": 1277.76, "text": " analysis reveals that when L-valid is bigger than the training input sequence length,"}, {"start": 1278.32, "end": 1283.92, "text": " Alibi might not be using context longer than the ones it was trained on, and this points to research"}, {"start": 1283.92, "end": 1288.72, "text": " direction which could be pursued in future work. That's it for this video. If you found it useful,"}, {"start": 1288.72, "end": 1294.24, "text": " share it out with a friend, subscribe, hit that bell icon, and also join the Discord community"}, {"start": 1294.24, "end": 1299.76, "text": " if you wanna connect with other ML people, and there's gonna be a lot of goodies happening in"}, {"start": 1299.76, "end": 1303.12, "text": " that community. So yeah, until next time, bye bye."}]