Vision Transformer (ViT) - An image is worth 16x16 words | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video I do a (semi) deep dive of the "An image is worth 16x16 words:
transformers for image recognition at scale" paper which introduced the Vision Transformer.

The paper is very interesting as it showed that with minimal modifications transformers can give better results than CNNs on the image classification problem. 

Until now transformers were ruling the NLP world and now they are coming for the CV world as well!

You'll learn about:
✔️ How the vision transformer works
✔️ Main ideas in the paper

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ paper: https://arxiv.org/abs/2010.11929
✅ transformer Jupyter Notebook: https://github.com/gordicaleksa/pytorch-original-transformer/blob/main/The%20Annotated%20Transformer%20%2B%2B.ipynb 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
0:00 Enter The Vision Transformer, Jupyter Notebook
1:00 Deep dive intro
3:14 How does Vision Transformer work?
4:39 Let's go even deeper
8:33 Positional encoding inductive bias
9:50 Model variants and results
11:50 VTAB benchmark results
13:00 Perf vs amount of pretrained data
16:08 What does Vision Transformer learn? (attention span)
19:35 Self-supervision vs Supervised learning
21:10 Scaling the transformer (future research prediction)
22:15 Positional Encodings details
24:15 Logging out

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#transformers #computervision #visiontransformer
 Okay, I'm super excited about this paper. It's called An Image Is Worth 16x16 Words. And the reason is, this paper got me into Transformers two months ago when it was first published, like I think October 3rd. And it was under review and it still is. But I'm going to try and do a deep dive today, so hopefully you'll find it useful. Basically the first time I read it, like on maybe October 5th or something, I didn't understand many details, so I took a step back. Went and researched NLP and Transformers specifically, like read 30 papers, even open sourced the original Transformer paper. And now I came a full circle back, so I'm going to try and explain what I learned and hopefully make this paper clear for you. By the way, it has a hard dependency on the original Transformer, so if you want to understand that, I have a Jupyter notebook, which you may find useful. So as usual, I'll just link it in the description so you can play with it. Okay, without further ado, let's jump into the deep dive. Okay, so the basic premise here is that we don't need, so we don't need CNNs to solve computer vision as we thought until now. So Transformer did the same thing as it did in the NLP field, basically show we don't need CNNs, we don't need recurrence, we can just do like attention, self-attention mechanisms, and that should be enough. So yeah, it attains super results on the computer vision on the image classification task, and it took less time to compute, and that's what's amazing. So a big note here is that it achieved all of those results, which we'll see really soon in the big data regime. So it needs a lot of images in order to accomplish these results, so that's a thing to keep in mind. So a good thing with Transformer in general is that they can be scaled up. So you probably know about GPT-3, which has around 175 billion prams, and I assume something similar will happen with the Vision Transformer in one of the future iterations of this paper. Yeah, ResNets are still, until this paper, we're dominating the different benchmarks, image classification benchmarks. So it's not like this paper was the first one to introduce Transformers to computer vision, that's not the case. Many other people tried. So in the related work here, we can see that people already tried at doing some sort of attention, so including attentions into CNNs, so either replacing certain layers of the CNN with attention modules or just kind of complementing it. And there were also some approaches where people tried to use some more sparse attention patterns and stuff, but it turned out to be really complicated in the engineering sense of the word. So yeah, they didn't give the results as this paper did. So let's jump to the meat of this paper, and I'm going to try to explain how it works using this image. I'm gonna zoom in a little bit. Okay, so maybe first up, like a high level overview. Basically, what we have here is we have the input image, we split it into patches, and then we flatten those patches like in the rest order, so basically, this is the first patch, second, third, fourth, fifth, sixth, et cetera. What we do then is we just flatten out those patches and we do a linear projection here. We add the positional encodings, if you're familiar with the original transformer. I'll get into details, but for now, this is a high level overview. And then we just feed those into the original transformer, so this is the block here. And then just putting a simple MLP, multilayer perceptron hat here, we can do classification. And that's the high level overview. And now this transformer, just a short recap, consists of these layers, which have this multi-hat attention hat, multi-hat attention module, yeah, and followed by like multilayer perceptron. So that's the basic structure of the layer in the transformer. Now, let's zoom in a little bit and get into the details, how it exactly works. I zoomed in a little bit and I'm going to tell you all of the details that go into this preprocessing step in the Vision Transformer. So we take an ImageNet image, which has, which looks like this, and it's 224 by 224. And that's actually where the name of the paper comes from. So basically what they now do is they split the image into patches, which are either 14 by 14 in size or 16 by 16. They have three flavors. They have the base, the large, and the huge model. We'll see that in a minute. But like, let's assume we use 14 by 14 here. And yeah, basically once you have this, if you divide 224 by 14, you get 16. So basically you have 16 by 16 patches. And that's where the, like the title of the paper comes from because we're basically treating all of these patches as simple tokens and we input them into the transformer so you can treat it as a word, whatever. Like that's just like a nice metaphor. Now what you do, you take this patch, you flatten it out. So you get a vector that's like 14 by 14. That's 196, I think. And that's this part. So we are now here. And now we do the simple linear projection, which will embed this vector into the model dimension space. So we'll basically, so for example, for the base model, base vision transformer, they use 796. So that's the same as bird base. So we end up with a vector that has 796 dimensions. And now we are here. So finally we just add the positional encoding to this representation and we input it into the transformer and that's pretty much it. Now for the positional encodings, if you're not familiar, what they did here, and they did some ablation studies, but they ended up using the 1D positional embedding. So you basically have a table like this. And so for example, if you are using, if we are looking at this specific token, so patch embedding, we just, so because it's the first patch, we just take the vector from this table, from the position one, and that's the positional encoding, which is actually learned during the training. So that's the trick. So you just add this one to the embedded patch and that's the final representation. So one thing I didn't mention here, and this is similar to BERT, is that they use this, they have this notion of a class embedding, which is the same thing that BERT used. So this one is also learned and it doesn't come from the image itself. It's just hard coded in the model architecture. So that was like a detailed overview of how this stem part of the model looks like. And one minor detail is that this linear projection, it's like basically a matrix E, so it's shared among different patches. So we use the same matrix E to project all of these patches, like flattened vectors of the patches into the embedding patch. So yeah, that was everything I wanted to mention there. Okay, I zoomed out. Let's go to second important part. And so there is one thing in this model that's kind of has like inductive bias, and that part has to do with positional encodings. So when you fine tune these models, so previous papers showed that when you fine tune for image classification tasks, you wanna fine tune on higher resolution. So what you end up, if you have higher resolution as a consequence, you end up with a bigger number of patches. So you're not 16 by 16 anymore. The number of patches becomes bigger, so the sequence becomes larger. And because of that, the positional encodings wouldn't make any sense if you just applied them during the fine tuning process. So you have to do 2D interpolation. So that's the only part pretty much where, so that's one of the parts where they introduce bias. So the transformer model itself has some biases like the residual connections, et cetera, but like it's more general than some other models like CNNs and LSTMs. So that was an important thing to mention. Now, as I mentioned, we have three variants of the model. They use the base and large, which roughly correspond to BERT base and BERT large when it comes to model size. And then we have the vision transformer huge, which is even bigger than the vision transformer large. And let us see the interesting results they got. So first of all, we can see two state-of-the-art so previous SODAs on these benchmarks, and those were ResNets and the FissionNets. Now, when you take the huge variant of the vision transformer and pre-train it on the JFT, which is like Google's proprietary data set, which contains like 303 million images, you get basically SODAs on every single benchmark. And additionally, you have a lesser, like you need less compute to do that. And if we take a look at, for example, this vision transformer large, it took only 0.68 thousand TPU v3 core days. And that's a mouthful, I'll explain it in a minute. And basically that's much less than 9.9 thousand that this previous ResNet took. So what this means is every single TPU v3 chip has two of these cores. So you basically need, if you have a single TPU v3 chip, you'd need 340 days to pre-train this model. And that's like a lot of compute that pretty much only Google and like richest labs, industrial labs in the world have. So not everybody can reproduce this paper. And that's the bad thing about this, about these kinds of free searches, even though they do advance the field, people can kind of reproduce this. And yeah, that sucks. Okay, let's go further. And interesting thing is this VTab benchmark. What it basically has, so VTab is interesting because it has three types of data. So first of all, it's low data benchmark, meaning it only has around thousand images per task. And secondly, it splits those images into three categories. So natural images, which are kind of like cipher, 10, cipher, 100 data set images. So like natural looking images like dogs, whatever, trees. And then we have these specialized images like medical and satellite images. And then they have this geometric, I think structured class of images. And basically what they showed is that the huge variant of the vision transformer outperformed every other soda across all of those categories, which is really reassuring. So the transfer learning for these huge transformers does work and it gives us nice results. Okay, going further, really interesting analysis they did here is how do these models perform with respect to ResNets, depending on the amount of pre-trained data they have. And the left figure here, we can see when we pre-trained the vision transformer on ImageNet, which has 1.3 million images, then on ImageNet 21K, which has 14 million images. And finally on JFT, which has 303 million images. How they, what's the accuracy, what's the performance of the models? And as we can see here, in the low data regimes, ResNets are the best. And that's because they have a useful inductive bias, meaning that the kernels are basically, that's the bias part of the ResNet. Basically you have to have the kernel, which only looks at the local part of the image and can have a global, like receptive field. And then going deeper into the model, you get like a bigger receptive field. That's one of the biases that ResNets have. The second one is the residual connections, but yeah. And the best vision transformer is actually the base variant. And that's not surprising actually. And then as we increase the number of pre-trained data, we can see that finally in the big data regime, the huge variant of the vision transformer performs the best and it's better than ResNets and it's also more compute like efficient. So that's a cool thing. Right figure shows the same thing, just like splitting the JFT into smaller chunks. And we can see again, the blue, the light blue line here is the vision transformer large, and it outperforms ResNets again in the big data regimes. So that was interesting. That was basically what this paper, one of the main points this paper made. In big data regimes, those inductive biases that we put into our models like CNNs are not useful anymore and we should let the model learn those biases itself. And we'll see in a short, an interesting showcase that shows us how the attention varies from in different layers. Here, yeah, this figure is not so important. Basically, it shows the vision transformers are more like efficient when it comes to compute. And this line is interesting. So third, vision transformers appear not to saturate within the range tried, motivating future scaling efforts. So you can basically expect something similar to GPT-3 in the world of computer vision. That's my prediction. I mean, they'll probably stop as soon as they see saturation because, yeah, you have some fixed budget. You don't want to overspend if you don't get any gains out of it. So yeah, we can expect a bigger vision transformer, that's for sure. Now, this part is really interesting. This is the thing I was mentioning about the attention. So they took the large variant of the model and they analyzed the different attention heads. So in particular, this model has 24 layers and it has 16 attention heads. And what you can see here is in the shallower layers, those attention heads, so some of them have a really localized attention. So they just attend pixels in the neighborhood of a current pixel, whereas the other attention heads have a global attention compared to CNNs, which only have local small receptive fields in these shallower layers. So if I were to draw how CNN receptive field would change with the depth, ComNets will probably have a curve like this. Like they would go linearly increase. So with every single layer, you increase linearly the receptive field. And then once you have the whole, once every single pixel attends to the whole input image, you basically stop increasing the receptive field. And this is how the CNN receptive field graph would look like. And so all of this region is where the vision transformer has an advantage because shallower layers can also attend to like the global to every single pixel in the image. And that's what it gives it the advantage because you can learn how to attend and when to attend and the bias is not needed in the big data regimes as I already mentioned. Couple of more interesting figures here on the left. This is how the 1D positional encodings look like after the model has been pre-trained. And you can see that this 2D representation emerges, which is why, and we'll see that in the appendix, which is why other variants of the embeddings they tried, positional embeddings they tried didn't work any better than the 1D. Finally, they took the E matrix, so the embedding matrix and they plotted like principal components. And you can see if you're familiar with AlexNet or paper or CNNs in general, you can notice that these look like the kernels, like the same as the learn kernels that CNNs learn. So it's doing a similar thing, obviously, as CNNs. Let me zoom out and go and see the other parts of the paper. First, here's a nice visualization of what the transformer attends to. And you can see it's meaningful. So this is what we would probably expect the model to attend to, like the salient objects in the picture. Self-supervision, they also tried self-supervised approach instead of the fully supervised pre-training approach, and it's still behind the supervised training, but it's worth discussing this part, and I'll discuss it more in the appendix. Again, yeah, I also expect, let me jump over the references. So I'm expecting to see like huge vision transformers somewhere in the near future, probably. So I mentioned self-supervision, and if you're familiar with NLP, you know that that's the way how big transformer models are pre-trained in the NLP field. On the other hand, computer vision was mostly about supervised pre-training objectives. And so in this paper, they tried doing self-supervision in the context of computer vision, and they did get encouraging results, still not as good as supervised results, but like they basically, so how they did it is they did it similarly to how BERT, if you're familiar with BERT, so they did a similar approach. They corrupt 50% of the input patch embeddings, and they tried to predict three things. First is the mean color of the patch. So you take basically, you take patch embedding, the original patch embedding, you find the mean value, so that's the mean color, then you mask it, and then you make the model predict what the color should be, depending on the context. So that's how the self-supervised thing works. They didn't only try the color, they also tried predicting the downsampled version of the patch embedding that's like four by four instead of like 16 by 16, as well as the color, and they also tried regressing the whole thing, so like doing auto-encoding pretty much. And as I already mentioned, results are not as good, but that's a promising like research direction, so yeah, we can expect innovation on that front. Let me go through two more sections which I found important. The first one, they did a scaling effort to figure out how should they scale their model to get the best performance out of it. They tried doing depth, they tried doing width, they tried making patches smaller, and like obviously depth gave nice results. As we can see here on the graph, on the chart, basically increasing depth gives us better performance. The computer also goes up, obviously, but like yeah, increasing depth will increase the performance. Now the interesting thing for me is that patch size, the smaller it is, they gain more performance. So I'm gonna make a prediction now. Basically somebody will take some of the efficient transformers like reformer, like longformer or leanformer, and they'll do the same thing as a vision transformer. They just use smaller patches and they'll get better results. That's a research I can expect to come over the next months. And finally, let me finish up, wrap up this already long deep dive, like talking a little bit more about positional encodings. They tried four things, and the biggest gap was between using no positional embeddings, I treating the patches as a bag of patches, and using 1D positional embeddings. So that's where the highest gap was, and you can see here the numbers. So using positional embeddings helps. Now how exactly do you like, which heuristic do you use? Let's call it that way, that matters a bit less. So they tried 2D positional embeddings and 1D which I already explained. Let me try to quickly explain how the 2D positional embeddings work. They basically have two tables this time, and how you index into these tables is the following. So you have your input image, again you split it into patches. And so this one will be patch one one, this one will be patch one two, this one will be patch one three, this is row two, et cetera. So you basically take the indices, so let's say this patch will have one, like so basically take, so this is one, two, same here, one, two, you basically take the first embedding here, and you concatenate it with the second embedding here. So this represents the X dimension, this one represents the Y dimension, and you just train the transformer like that. And they showed that that approach won't do anything better than just using a simple 1D positional encoding. So that was pretty much it. This was already probably too long. Hopefully you found this video useful. This is the first time I'm doing like a deep dive, like a semi deep dive of a research paper on this channel. If you found it useful, please leave a comment down in the description if you'd like me to make more of these. And yeah, subscribe if you haven't already and click that bell icon to get notified when I upload a new video. And until next time, keep learning deep. Bye.
[{"start": 0.0, "end": 2.32, "text": " Okay, I'm super excited about this paper."}, {"start": 2.32, "end": 5.5200000000000005, "text": " It's called An Image Is Worth 16x16 Words."}, {"start": 5.5200000000000005, "end": 8.48, "text": " And the reason is, this paper got me into Transformers"}, {"start": 8.48, "end": 10.24, "text": " two months ago when it was first published,"}, {"start": 10.24, "end": 12.08, "text": " like I think October 3rd."}, {"start": 12.08, "end": 14.52, "text": " And it was under review and it still is."}, {"start": 14.52, "end": 17.580000000000002, "text": " But I'm going to try and do a deep dive today,"}, {"start": 17.580000000000002, "end": 19.44, "text": " so hopefully you'll find it useful."}, {"start": 20.8, "end": 22.84, "text": " Basically the first time I read it,"}, {"start": 22.84, "end": 26.18, "text": " like on maybe October 5th or something,"}, {"start": 26.18, "end": 27.64, "text": " I didn't understand many details,"}, {"start": 27.64, "end": 29.96, "text": " so I took a step back."}, {"start": 29.96, "end": 33.620000000000005, "text": " Went and researched NLP and Transformers specifically,"}, {"start": 33.620000000000005, "end": 35.68, "text": " like read 30 papers, even open sourced"}, {"start": 35.68, "end": 37.6, "text": " the original Transformer paper."}, {"start": 37.6, "end": 40.04, "text": " And now I came a full circle back,"}, {"start": 40.04, "end": 42.2, "text": " so I'm going to try and explain what I learned"}, {"start": 42.2, "end": 45.760000000000005, "text": " and hopefully make this paper clear for you."}, {"start": 45.760000000000005, "end": 47.480000000000004, "text": " By the way, it has a hard dependency"}, {"start": 47.480000000000004, "end": 48.92, "text": " on the original Transformer,"}, {"start": 48.92, "end": 50.88, "text": " so if you want to understand that,"}, {"start": 52.94, "end": 57.040000000000006, "text": " I have a Jupyter notebook, which you may find useful."}, {"start": 57.04, "end": 60.98, "text": " So as usual, I'll just link it in the description"}, {"start": 60.98, "end": 62.56, "text": " so you can play with it."}, {"start": 63.92, "end": 67.06, "text": " Okay, without further ado, let's jump into the deep dive."}, {"start": 67.06, "end": 72.06, "text": " Okay, so the basic premise here is that we don't need,"}, {"start": 72.98, "end": 77.0, "text": " so we don't need CNNs to solve computer vision"}, {"start": 77.0, "end": 78.74, "text": " as we thought until now."}, {"start": 79.94, "end": 84.25999999999999, "text": " So Transformer did the same thing as it did in the NLP field,"}, {"start": 84.25999999999999, "end": 85.9, "text": " basically show we don't need CNNs,"}, {"start": 85.9, "end": 89.88000000000001, "text": " we don't need recurrence, we can just do like attention,"}, {"start": 89.88000000000001, "end": 93.80000000000001, "text": " self-attention mechanisms, and that should be enough."}, {"start": 93.80000000000001, "end": 98.80000000000001, "text": " So yeah, it attains super results on the computer vision"}, {"start": 98.88000000000001, "end": 100.82000000000001, "text": " on the image classification task,"}, {"start": 100.82000000000001, "end": 105.4, "text": " and it took less time to compute, and that's what's amazing."}, {"start": 105.4, "end": 109.0, "text": " So a big note here is that it achieved all of those results,"}, {"start": 109.0, "end": 112.96000000000001, "text": " which we'll see really soon in the big data regime."}, {"start": 112.96000000000001, "end": 115.84, "text": " So it needs a lot of images in order to accomplish"}, {"start": 115.84, "end": 118.4, "text": " these results, so that's a thing to keep in mind."}, {"start": 118.4, "end": 120.44, "text": " So a good thing with Transformer in general"}, {"start": 120.44, "end": 123.76, "text": " is that they can be scaled up."}, {"start": 123.76, "end": 126.4, "text": " So you probably know about GPT-3,"}, {"start": 126.4, "end": 130.42000000000002, "text": " which has around 175 billion prams,"}, {"start": 130.42000000000002, "end": 133.42000000000002, "text": " and I assume something similar will happen"}, {"start": 133.42000000000002, "end": 136.12, "text": " with the Vision Transformer in one of the future iterations"}, {"start": 136.12, "end": 138.16, "text": " of this paper."}, {"start": 139.52, "end": 143.04, "text": " Yeah, ResNets are still, until this paper,"}, {"start": 143.04, "end": 145.68, "text": " we're dominating the different benchmarks,"}, {"start": 145.68, "end": 147.72, "text": " image classification benchmarks."}, {"start": 147.72, "end": 150.8, "text": " So it's not like this paper was the first one"}, {"start": 150.8, "end": 153.76000000000002, "text": " to introduce Transformers to computer vision,"}, {"start": 153.76000000000002, "end": 155.88, "text": " that's not the case."}, {"start": 155.88, "end": 157.28, "text": " Many other people tried."}, {"start": 157.28, "end": 160.08, "text": " So in the related work here, we can see that people"}, {"start": 160.08, "end": 164.8, "text": " already tried at doing some sort of attention,"}, {"start": 164.8, "end": 168.04000000000002, "text": " so including attentions into CNNs,"}, {"start": 168.04000000000002, "end": 172.92000000000002, "text": " so either replacing certain layers of the CNN"}, {"start": 172.92, "end": 177.28, "text": " with attention modules or just kind of complementing it."}, {"start": 177.28, "end": 180.23999999999998, "text": " And there were also some approaches where people tried"}, {"start": 180.23999999999998, "end": 184.39999999999998, "text": " to use some more sparse attention patterns and stuff,"}, {"start": 184.39999999999998, "end": 186.44, "text": " but it turned out to be really complicated"}, {"start": 186.44, "end": 188.92, "text": " in the engineering sense of the word."}, {"start": 188.92, "end": 193.88, "text": " So yeah, they didn't give the results as this paper did."}, {"start": 193.88, "end": 196.48, "text": " So let's jump to the meat of this paper,"}, {"start": 196.48, "end": 199.35999999999999, "text": " and I'm going to try to explain how it works"}, {"start": 199.35999999999999, "end": 200.26, "text": " using this image."}, {"start": 200.26, "end": 202.32, "text": " I'm gonna zoom in a little bit."}, {"start": 202.32, "end": 207.32, "text": " Okay, so maybe first up, like a high level overview."}, {"start": 208.4, "end": 213.4, "text": " Basically, what we have here is we have the input image,"}, {"start": 213.76, "end": 218.0, "text": " we split it into patches, and then we flatten those patches"}, {"start": 218.0, "end": 220.16, "text": " like in the rest order, so basically,"}, {"start": 220.16, "end": 222.64, "text": " this is the first patch, second, third, fourth,"}, {"start": 222.64, "end": 224.56, "text": " fifth, sixth, et cetera."}, {"start": 224.56, "end": 228.88, "text": " What we do then is we just flatten out those patches"}, {"start": 228.88, "end": 231.4, "text": " and we do a linear projection here."}, {"start": 231.4, "end": 232.88, "text": " We add the positional encodings,"}, {"start": 232.88, "end": 235.02, "text": " if you're familiar with the original transformer."}, {"start": 235.02, "end": 237.24, "text": " I'll get into details, but for now,"}, {"start": 237.24, "end": 238.84, "text": " this is a high level overview."}, {"start": 238.84, "end": 241.92000000000002, "text": " And then we just feed those into the original transformer,"}, {"start": 241.92000000000002, "end": 243.24, "text": " so this is the block here."}, {"start": 244.16, "end": 248.04000000000002, "text": " And then just putting a simple MLP,"}, {"start": 248.04000000000002, "end": 252.12, "text": " multilayer perceptron hat here, we can do classification."}, {"start": 252.12, "end": 253.70000000000002, "text": " And that's the high level overview."}, {"start": 253.70000000000002, "end": 256.6, "text": " And now this transformer, just a short recap,"}, {"start": 256.6, "end": 258.36, "text": " consists of these layers,"}, {"start": 258.36, "end": 261.88, "text": " which have this multi-hat attention hat,"}, {"start": 261.88, "end": 264.48, "text": " multi-hat attention module, yeah,"}, {"start": 264.48, "end": 269.04, "text": " and followed by like multilayer perceptron."}, {"start": 269.04, "end": 270.76, "text": " So that's the basic structure"}, {"start": 270.76, "end": 273.52000000000004, "text": " of the layer in the transformer."}, {"start": 274.56, "end": 276.58000000000004, "text": " Now, let's zoom in a little bit"}, {"start": 276.58000000000004, "end": 279.2, "text": " and get into the details, how it exactly works."}, {"start": 279.2, "end": 282.68, "text": " I zoomed in a little bit and I'm going to tell you"}, {"start": 282.68, "end": 286.2, "text": " all of the details that go into this preprocessing step"}, {"start": 287.16, "end": 288.12, "text": " in the Vision Transformer."}, {"start": 288.12, "end": 291.74, "text": " So we take an ImageNet image,"}, {"start": 292.8, "end": 295.72, "text": " which has, which looks like this,"}, {"start": 296.84000000000003, "end": 300.92, "text": " and it's 224 by 224."}, {"start": 302.74, "end": 305.5, "text": " And that's actually where the name of the paper comes from."}, {"start": 305.5, "end": 306.96, "text": " So basically what they now do"}, {"start": 306.96, "end": 309.44, "text": " is they split the image into patches,"}, {"start": 309.44, "end": 313.72, "text": " which are either 14 by 14 in size or 16 by 16."}, {"start": 313.72, "end": 314.68, "text": " They have three flavors."}, {"start": 314.68, "end": 317.32, "text": " They have the base, the large, and the huge model."}, {"start": 317.32, "end": 318.68, "text": " We'll see that in a minute."}, {"start": 318.68, "end": 321.86, "text": " But like, let's assume we use 14 by 14 here."}, {"start": 323.88, "end": 328.04, "text": " And yeah, basically once you have this,"}, {"start": 328.04, "end": 331.71999999999997, "text": " if you divide 224 by 14, you get 16."}, {"start": 331.71999999999997, "end": 334.56, "text": " So basically you have 16 by 16 patches."}, {"start": 334.56, "end": 337.8, "text": " And that's where the, like the title of the paper"}, {"start": 337.8, "end": 340.24, "text": " comes from because we're basically treating"}, {"start": 340.24, "end": 342.68, "text": " all of these patches as simple tokens"}, {"start": 342.68, "end": 345.0, "text": " and we input them into the transformer"}, {"start": 345.0, "end": 347.52, "text": " so you can treat it as a word, whatever."}, {"start": 347.52, "end": 349.6, "text": " Like that's just like a nice metaphor."}, {"start": 350.6, "end": 354.04, "text": " Now what you do, you take this patch,"}, {"start": 354.04, "end": 354.88, "text": " you flatten it out."}, {"start": 354.88, "end": 359.28, "text": " So you get a vector that's like 14 by 14."}, {"start": 359.28, "end": 362.78, "text": " That's 196, I think."}, {"start": 362.78, "end": 364.52, "text": " And that's this part."}, {"start": 364.52, "end": 366.06, "text": " So we are now here."}, {"start": 367.96, "end": 371.48, "text": " And now we do the simple linear projection,"}, {"start": 371.48, "end": 376.48, "text": " which will embed this vector into the model dimension space."}, {"start": 376.8, "end": 381.20000000000005, "text": " So we'll basically, so for example, for the base model,"}, {"start": 381.20000000000005, "end": 385.0, "text": " base vision transformer, they use 796."}, {"start": 385.0, "end": 386.86, "text": " So that's the same as bird base."}, {"start": 388.16, "end": 393.16, "text": " So we end up with a vector that has 796 dimensions."}, {"start": 395.56, "end": 397.18, "text": " And now we are here."}, {"start": 397.18, "end": 402.18, "text": " So finally we just add the positional encoding"}, {"start": 404.98, "end": 408.5, "text": " to this representation and we input it into the transformer"}, {"start": 408.5, "end": 409.98, "text": " and that's pretty much it."}, {"start": 409.98, "end": 413.14, "text": " Now for the positional encodings, if you're not familiar,"}, {"start": 413.14, "end": 415.88, "text": " what they did here, and they did some ablation studies,"}, {"start": 415.88, "end": 419.58, "text": " but they ended up using the 1D positional embedding."}, {"start": 419.58, "end": 424.58, "text": " So you basically have a table like this."}, {"start": 424.58, "end": 429.09999999999997, "text": " And so for example, if you are using,"}, {"start": 429.09999999999997, "end": 434.09999999999997, "text": " if we are looking at this specific token,"}, {"start": 434.5, "end": 437.3, "text": " so patch embedding, we just,"}, {"start": 437.3, "end": 439.74, "text": " so because it's the first patch,"}, {"start": 439.74, "end": 443.06, "text": " we just take the vector from this table,"}, {"start": 443.06, "end": 447.09999999999997, "text": " from the position one, and that's the positional encoding,"}, {"start": 447.09999999999997, "end": 450.0, "text": " which is actually learned during the training."}, {"start": 450.0, "end": 450.84, "text": " So that's the trick."}, {"start": 450.84, "end": 454.46, "text": " So you just add this one to the embedded patch"}, {"start": 454.46, "end": 456.46, "text": " and that's the final representation."}, {"start": 456.46, "end": 458.5, "text": " So one thing I didn't mention here,"}, {"start": 458.5, "end": 462.06, "text": " and this is similar to BERT, is that they use this,"}, {"start": 462.06, "end": 466.64, "text": " they have this notion of a class embedding,"}, {"start": 466.64, "end": 468.53999999999996, "text": " which is the same thing that BERT used."}, {"start": 468.53999999999996, "end": 471.38, "text": " So this one is also learned"}, {"start": 471.38, "end": 473.85999999999996, "text": " and it doesn't come from the image itself."}, {"start": 473.85999999999996, "end": 478.82, "text": " It's just hard coded in the model architecture."}, {"start": 478.82, "end": 481.38, "text": " So that was like a detailed overview"}, {"start": 481.38, "end": 485.62, "text": " of how this stem part of the model looks like."}, {"start": 485.62, "end": 489.26, "text": " And one minor detail is that this linear projection,"}, {"start": 489.26, "end": 493.3, "text": " it's like basically a matrix E,"}, {"start": 494.42, "end": 497.26, "text": " so it's shared among different patches."}, {"start": 497.26, "end": 501.26, "text": " So we use the same matrix E to project all of these patches,"}, {"start": 502.82, "end": 506.38, "text": " like flattened vectors of the patches"}, {"start": 506.38, "end": 508.78, "text": " into the embedding patch."}, {"start": 508.78, "end": 513.74, "text": " So yeah, that was everything I wanted to mention there."}, {"start": 513.74, "end": 515.42, "text": " Okay, I zoomed out."}, {"start": 515.42, "end": 518.5, "text": " Let's go to second important part."}, {"start": 518.5, "end": 522.18, "text": " And so there is one thing in this model"}, {"start": 522.18, "end": 525.9399999999999, "text": " that's kind of has like inductive bias,"}, {"start": 525.9399999999999, "end": 529.38, "text": " and that part has to do with positional encodings."}, {"start": 529.38, "end": 533.6999999999999, "text": " So when you fine tune these models,"}, {"start": 533.6999999999999, "end": 538.66, "text": " so previous papers showed"}, {"start": 538.66, "end": 543.3, "text": " that when you fine tune for image classification tasks,"}, {"start": 543.3, "end": 545.74, "text": " you wanna fine tune on higher resolution."}, {"start": 545.74, "end": 548.4599999999999, "text": " So what you end up, if you have higher resolution"}, {"start": 548.4599999999999, "end": 552.54, "text": " as a consequence, you end up with a bigger number of patches."}, {"start": 552.54, "end": 555.18, "text": " So you're not 16 by 16 anymore."}, {"start": 555.18, "end": 556.9399999999999, "text": " The number of patches becomes bigger,"}, {"start": 556.9399999999999, "end": 559.98, "text": " so the sequence becomes larger."}, {"start": 559.98, "end": 563.38, "text": " And because of that, the positional encodings"}, {"start": 563.38, "end": 566.3, "text": " wouldn't make any sense if you just applied them"}, {"start": 566.3, "end": 567.74, "text": " during the fine tuning process."}, {"start": 567.74, "end": 569.94, "text": " So you have to do 2D interpolation."}, {"start": 569.94, "end": 572.02, "text": " So that's the only part pretty much where,"}, {"start": 572.02, "end": 576.02, "text": " so that's one of the parts where they introduce bias."}, {"start": 576.02, "end": 578.1800000000001, "text": " So the transformer model itself has some biases"}, {"start": 578.1800000000001, "end": 580.54, "text": " like the residual connections, et cetera,"}, {"start": 580.54, "end": 583.42, "text": " but like it's more general than some other models"}, {"start": 583.42, "end": 587.1800000000001, "text": " like CNNs and LSTMs."}, {"start": 587.1800000000001, "end": 590.46, "text": " So that was an important thing to mention."}, {"start": 590.46, "end": 594.66, "text": " Now, as I mentioned, we have three variants of the model."}, {"start": 594.66, "end": 596.58, "text": " They use the base and large,"}, {"start": 596.58, "end": 600.38, "text": " which roughly correspond to BERT base and BERT large"}, {"start": 600.38, "end": 602.3000000000001, "text": " when it comes to model size."}, {"start": 602.3000000000001, "end": 604.9000000000001, "text": " And then we have the vision transformer huge,"}, {"start": 604.9000000000001, "end": 609.62, "text": " which is even bigger than the vision transformer large."}, {"start": 609.62, "end": 612.5400000000001, "text": " And let us see the interesting results they got."}, {"start": 612.5400000000001, "end": 617.0200000000001, "text": " So first of all, we can see two state-of-the-art"}, {"start": 617.0200000000001, "end": 620.82, "text": " so previous SODAs on these benchmarks,"}, {"start": 620.82, "end": 623.5, "text": " and those were ResNets and the FissionNets."}, {"start": 623.5, "end": 627.46, "text": " Now, when you take the huge variant"}, {"start": 627.46, "end": 631.06, "text": " of the vision transformer and pre-train it on the JFT,"}, {"start": 631.06, "end": 633.66, "text": " which is like Google's proprietary data set,"}, {"start": 633.66, "end": 636.78, "text": " which contains like 303 million images,"}, {"start": 638.18, "end": 643.18, "text": " you get basically SODAs on every single benchmark."}, {"start": 643.9, "end": 645.78, "text": " And additionally, you have a lesser,"}, {"start": 645.78, "end": 648.62, "text": " like you need less compute to do that."}, {"start": 648.62, "end": 651.02, "text": " And if we take a look at, for example,"}, {"start": 651.02, "end": 653.14, "text": " this vision transformer large,"}, {"start": 653.14, "end": 658.14, "text": " it took only 0.68 thousand TPU v3 core days."}, {"start": 660.14, "end": 663.5, "text": " And that's a mouthful, I'll explain it in a minute."}, {"start": 663.5, "end": 668.5, "text": " And basically that's much less than 9.9 thousand"}, {"start": 668.58, "end": 672.28, "text": " that this previous ResNet took."}, {"start": 672.28, "end": 676.98, "text": " So what this means is every single TPU v3 chip"}, {"start": 676.98, "end": 678.74, "text": " has two of these cores."}, {"start": 678.74, "end": 680.54, "text": " So you basically need,"}, {"start": 680.54, "end": 684.8199999999999, "text": " if you have a single TPU v3 chip,"}, {"start": 684.8199999999999, "end": 689.8199999999999, "text": " you'd need 340 days to pre-train this model."}, {"start": 690.4599999999999, "end": 692.3399999999999, "text": " And that's like a lot of compute"}, {"start": 692.3399999999999, "end": 697.3, "text": " that pretty much only Google and like richest labs,"}, {"start": 697.3, "end": 699.06, "text": " industrial labs in the world have."}, {"start": 699.06, "end": 700.78, "text": " So not everybody can reproduce this paper."}, {"start": 700.78, "end": 703.14, "text": " And that's the bad thing about this,"}, {"start": 703.14, "end": 705.18, "text": " about these kinds of free searches,"}, {"start": 705.18, "end": 707.5799999999999, "text": " even though they do advance the field,"}, {"start": 707.5799999999999, "end": 710.02, "text": " people can kind of reproduce this."}, {"start": 710.02, "end": 712.14, "text": " And yeah, that sucks."}, {"start": 713.38, "end": 715.62, "text": " Okay, let's go further."}, {"start": 716.88, "end": 721.88, "text": " And interesting thing is this VTab benchmark."}, {"start": 723.06, "end": 724.16, "text": " What it basically has,"}, {"start": 724.16, "end": 728.18, "text": " so VTab is interesting because it has three types of data."}, {"start": 728.18, "end": 730.66, "text": " So first of all, it's low data benchmark,"}, {"start": 730.66, "end": 735.5, "text": " meaning it only has around thousand images per task."}, {"start": 735.5, "end": 739.22, "text": " And secondly, it splits those images into three categories."}, {"start": 739.22, "end": 742.58, "text": " So natural images, which are kind of like cipher,"}, {"start": 742.58, "end": 747.08, "text": " 10, cipher, 100 data set images."}, {"start": 747.08, "end": 750.22, "text": " So like natural looking images like dogs, whatever, trees."}, {"start": 750.22, "end": 753.62, "text": " And then we have these specialized images"}, {"start": 753.62, "end": 755.26, "text": " like medical and satellite images."}, {"start": 755.26, "end": 757.86, "text": " And then they have this geometric,"}, {"start": 757.86, "end": 762.1800000000001, "text": " I think structured class of images."}, {"start": 762.1800000000001, "end": 765.5, "text": " And basically what they showed is that the huge variant"}, {"start": 765.5, "end": 770.06, "text": " of the vision transformer outperformed every other soda"}, {"start": 770.06, "end": 775.06, "text": " across all of those categories, which is really reassuring."}, {"start": 775.18, "end": 780.18, "text": " So the transfer learning for these huge transformers"}, {"start": 780.58, "end": 783.54, "text": " does work and it gives us nice results."}, {"start": 783.54, "end": 785.74, "text": " Okay, going further, really interesting analysis"}, {"start": 785.74, "end": 790.58, "text": " they did here is how do these models perform"}, {"start": 790.58, "end": 792.94, "text": " with respect to ResNets,"}, {"start": 792.94, "end": 796.1, "text": " depending on the amount of pre-trained data they have."}, {"start": 796.1, "end": 799.86, "text": " And the left figure here, we can see when we pre-trained"}, {"start": 800.86, "end": 803.1800000000001, "text": " the vision transformer on ImageNet,"}, {"start": 803.1800000000001, "end": 805.4200000000001, "text": " which has 1.3 million images,"}, {"start": 805.4200000000001, "end": 810.1, "text": " then on ImageNet 21K, which has 14 million images."}, {"start": 810.1, "end": 814.58, "text": " And finally on JFT, which has 303 million images."}, {"start": 814.58, "end": 816.9000000000001, "text": " How they, what's the accuracy,"}, {"start": 816.9000000000001, "end": 818.46, "text": " what's the performance of the models?"}, {"start": 818.46, "end": 821.9000000000001, "text": " And as we can see here, in the low data regimes,"}, {"start": 821.9, "end": 823.9, "text": " ResNets are the best."}, {"start": 823.9, "end": 827.5799999999999, "text": " And that's because they have a useful inductive bias,"}, {"start": 827.5799999999999, "end": 830.74, "text": " meaning that the kernels are basically,"}, {"start": 832.42, "end": 834.5, "text": " that's the bias part of the ResNet."}, {"start": 834.5, "end": 836.5799999999999, "text": " Basically you have to have the kernel,"}, {"start": 836.5799999999999, "end": 840.86, "text": " which only looks at the local part of the image"}, {"start": 840.86, "end": 843.98, "text": " and can have a global, like receptive field."}, {"start": 843.98, "end": 846.86, "text": " And then going deeper into the model,"}, {"start": 846.86, "end": 849.0799999999999, "text": " you get like a bigger receptive field."}, {"start": 849.08, "end": 852.84, "text": " That's one of the biases that ResNets have."}, {"start": 852.84, "end": 856.08, "text": " The second one is the residual connections, but yeah."}, {"start": 856.08, "end": 858.6, "text": " And the best vision transformer"}, {"start": 858.6, "end": 861.2, "text": " is actually the base variant."}, {"start": 861.2, "end": 864.4000000000001, "text": " And that's not surprising actually."}, {"start": 864.4000000000001, "end": 867.1600000000001, "text": " And then as we increase the number of pre-trained data,"}, {"start": 867.1600000000001, "end": 870.44, "text": " we can see that finally in the big data regime,"}, {"start": 870.44, "end": 872.8000000000001, "text": " the huge variant of the vision transformer"}, {"start": 872.8000000000001, "end": 875.48, "text": " performs the best and it's better than ResNets"}, {"start": 875.48, "end": 877.96, "text": " and it's also more compute like efficient."}, {"start": 877.96, "end": 879.38, "text": " So that's a cool thing."}, {"start": 880.3000000000001, "end": 881.9200000000001, "text": " Right figure shows the same thing,"}, {"start": 881.9200000000001, "end": 885.84, "text": " just like splitting the JFT into smaller chunks."}, {"start": 885.84, "end": 887.8000000000001, "text": " And we can see again, the blue,"}, {"start": 887.8000000000001, "end": 892.8000000000001, "text": " the light blue line here is the vision transformer large,"}, {"start": 894.26, "end": 898.32, "text": " and it outperforms ResNets again in the big data regimes."}, {"start": 898.32, "end": 899.76, "text": " So that was interesting."}, {"start": 899.76, "end": 902.8000000000001, "text": " That was basically what this paper,"}, {"start": 902.8000000000001, "end": 905.44, "text": " one of the main points this paper made."}, {"start": 905.44, "end": 909.48, "text": " In big data regimes, those inductive biases"}, {"start": 909.48, "end": 912.0, "text": " that we put into our models like CNNs"}, {"start": 912.0, "end": 914.6, "text": " are not useful anymore and we should let the model"}, {"start": 914.6, "end": 916.6, "text": " learn those biases itself."}, {"start": 916.6, "end": 919.84, "text": " And we'll see in a short, an interesting showcase"}, {"start": 919.84, "end": 922.08, "text": " that shows us how the attention varies"}, {"start": 922.08, "end": 925.0, "text": " from in different layers."}, {"start": 925.0, "end": 927.6400000000001, "text": " Here, yeah, this figure is not so important."}, {"start": 927.6400000000001, "end": 930.1600000000001, "text": " Basically, it shows the vision transformers"}, {"start": 930.16, "end": 936.4399999999999, "text": " are more like efficient when it comes to compute."}, {"start": 936.4399999999999, "end": 937.4399999999999, "text": " And this line is interesting."}, {"start": 937.4399999999999, "end": 940.52, "text": " So third, vision transformers appear not to saturate"}, {"start": 940.52, "end": 943.88, "text": " within the range tried, motivating future scaling efforts."}, {"start": 943.88, "end": 948.04, "text": " So you can basically expect something similar to GPT-3"}, {"start": 948.04, "end": 949.48, "text": " in the world of computer vision."}, {"start": 949.48, "end": 950.64, "text": " That's my prediction."}, {"start": 951.6, "end": 955.3199999999999, "text": " I mean, they'll probably stop as soon as they see saturation"}, {"start": 955.32, "end": 960.32, "text": " because, yeah, you have some fixed budget."}, {"start": 960.32, "end": 963.0, "text": " You don't want to overspend if you don't get any gains"}, {"start": 963.0, "end": 964.08, "text": " out of it."}, {"start": 964.08, "end": 966.48, "text": " So yeah, we can expect a bigger vision transformer,"}, {"start": 966.48, "end": 967.36, "text": " that's for sure."}, {"start": 967.36, "end": 969.6, "text": " Now, this part is really interesting."}, {"start": 969.6, "end": 972.6400000000001, "text": " This is the thing I was mentioning about the attention."}, {"start": 972.6400000000001, "end": 977.36, "text": " So they took the large variant of the model"}, {"start": 977.36, "end": 980.5600000000001, "text": " and they analyzed the different attention heads."}, {"start": 980.5600000000001, "end": 983.72, "text": " So in particular, this model has 24 layers"}, {"start": 983.72, "end": 987.52, "text": " and it has 16 attention heads."}, {"start": 987.52, "end": 991.5600000000001, "text": " And what you can see here is in the shallower layers,"}, {"start": 991.5600000000001, "end": 993.64, "text": " those attention heads, so some of them"}, {"start": 993.64, "end": 995.4, "text": " have a really localized attention."}, {"start": 995.4, "end": 998.72, "text": " So they just attend pixels in the neighborhood"}, {"start": 998.72, "end": 1004.0400000000001, "text": " of a current pixel, whereas the other attention heads"}, {"start": 1004.0400000000001, "end": 1008.2, "text": " have a global attention compared to CNNs,"}, {"start": 1008.2, "end": 1011.0400000000001, "text": " which only have local small receptive fields"}, {"start": 1011.0400000000001, "end": 1012.5600000000001, "text": " in these shallower layers."}, {"start": 1012.56, "end": 1017.56, "text": " So if I were to draw how CNN receptive field would change"}, {"start": 1017.92, "end": 1021.7199999999999, "text": " with the depth, ComNets will probably have a curve like this."}, {"start": 1021.7199999999999, "end": 1024.04, "text": " Like they would go linearly increase."}, {"start": 1024.04, "end": 1026.6799999999998, "text": " So with every single layer, you increase linearly"}, {"start": 1026.6799999999998, "end": 1027.96, "text": " the receptive field."}, {"start": 1027.96, "end": 1030.56, "text": " And then once you have the whole,"}, {"start": 1030.56, "end": 1035.56, "text": " once every single pixel attends to the whole input image,"}, {"start": 1036.36, "end": 1039.48, "text": " you basically stop increasing the receptive field."}, {"start": 1039.48, "end": 1044.48, "text": " And this is how the CNN receptive field graph would look like."}, {"start": 1046.52, "end": 1050.08, "text": " And so all of this region is where the vision transformer"}, {"start": 1050.08, "end": 1054.96, "text": " has an advantage because shallower layers can also attend"}, {"start": 1054.96, "end": 1059.68, "text": " to like the global to every single pixel in the image."}, {"start": 1059.68, "end": 1062.76, "text": " And that's what it gives it the advantage"}, {"start": 1062.76, "end": 1066.0, "text": " because you can learn how to attend and when to attend"}, {"start": 1066.0, "end": 1070.2, "text": " and the bias is not needed in the big data regimes"}, {"start": 1070.2, "end": 1071.4, "text": " as I already mentioned."}, {"start": 1072.48, "end": 1075.08, "text": " Couple of more interesting figures here on the left."}, {"start": 1076.12, "end": 1081.12, "text": " This is how the 1D positional encodings look like"}, {"start": 1081.24, "end": 1083.6, "text": " after the model has been pre-trained."}, {"start": 1083.6, "end": 1088.6, "text": " And you can see that this 2D representation emerges,"}, {"start": 1089.8, "end": 1093.32, "text": " which is why, and we'll see that in the appendix,"}, {"start": 1093.32, "end": 1097.1599999999999, "text": " which is why other variants of the embeddings they tried,"}, {"start": 1097.1599999999999, "end": 1099.48, "text": " positional embeddings they tried didn't work any better"}, {"start": 1099.48, "end": 1100.4399999999998, "text": " than the 1D."}, {"start": 1100.4399999999998, "end": 1105.4399999999998, "text": " Finally, they took the E matrix, so the embedding matrix"}, {"start": 1105.4399999999998, "end": 1108.84, "text": " and they plotted like principal components."}, {"start": 1108.84, "end": 1111.36, "text": " And you can see if you're familiar with AlexNet"}, {"start": 1111.36, "end": 1115.3999999999999, "text": " or paper or CNNs in general,"}, {"start": 1115.3999999999999, "end": 1119.0, "text": " you can notice that these look like the kernels,"}, {"start": 1119.0, "end": 1122.12, "text": " like the same as the learn kernels that CNNs learn."}, {"start": 1122.12, "end": 1126.08, "text": " So it's doing a similar thing, obviously, as CNNs."}, {"start": 1126.08, "end": 1129.9199999999998, "text": " Let me zoom out and go and see the other parts of the paper."}, {"start": 1130.8799999999999, "end": 1134.04, "text": " First, here's a nice visualization"}, {"start": 1134.04, "end": 1136.76, "text": " of what the transformer attends to."}, {"start": 1137.6, "end": 1139.8, "text": " And you can see it's meaningful."}, {"start": 1139.8, "end": 1143.28, "text": " So this is what we would probably expect the model"}, {"start": 1143.28, "end": 1146.76, "text": " to attend to, like the salient objects in the picture."}, {"start": 1149.28, "end": 1152.0, "text": " Self-supervision, they also tried self-supervised"}, {"start": 1152.0, "end": 1155.72, "text": " approach instead of the fully supervised pre-training"}, {"start": 1155.72, "end": 1159.36, "text": " approach, and it's still behind the supervised training,"}, {"start": 1159.36, "end": 1161.96, "text": " but it's worth discussing this part,"}, {"start": 1161.96, "end": 1164.44, "text": " and I'll discuss it more in the appendix."}, {"start": 1164.44, "end": 1169.44, "text": " Again, yeah, I also expect, let me jump over the references."}, {"start": 1171.96, "end": 1176.24, "text": " So I'm expecting to see like huge vision transformers"}, {"start": 1176.24, "end": 1177.96, "text": " somewhere in the near future, probably."}, {"start": 1177.96, "end": 1179.52, "text": " So I mentioned self-supervision,"}, {"start": 1179.52, "end": 1181.88, "text": " and if you're familiar with NLP,"}, {"start": 1181.88, "end": 1185.68, "text": " you know that that's the way how big transformer models"}, {"start": 1185.68, "end": 1188.48, "text": " are pre-trained in the NLP field."}, {"start": 1188.48, "end": 1191.64, "text": " On the other hand, computer vision was mostly about"}, {"start": 1191.64, "end": 1194.0400000000002, "text": " supervised pre-training objectives."}, {"start": 1194.0400000000002, "end": 1198.68, "text": " And so in this paper, they tried doing self-supervision"}, {"start": 1198.68, "end": 1200.64, "text": " in the context of computer vision,"}, {"start": 1200.64, "end": 1204.1200000000001, "text": " and they did get encouraging results,"}, {"start": 1204.1200000000001, "end": 1207.2, "text": " still not as good as supervised results,"}, {"start": 1207.2, "end": 1211.0, "text": " but like they basically, so how they did it"}, {"start": 1211.0, "end": 1214.96, "text": " is they did it similarly to how BERT,"}, {"start": 1214.96, "end": 1216.2, "text": " if you're familiar with BERT,"}, {"start": 1216.2, "end": 1217.68, "text": " so they did a similar approach."}, {"start": 1217.68, "end": 1222.08, "text": " They corrupt 50% of the input patch embeddings,"}, {"start": 1222.08, "end": 1224.6, "text": " and they tried to predict three things."}, {"start": 1224.6, "end": 1228.32, "text": " First is the mean color of the patch."}, {"start": 1228.32, "end": 1232.32, "text": " So you take basically, you take patch embedding,"}, {"start": 1232.32, "end": 1235.08, "text": " the original patch embedding, you find the mean value,"}, {"start": 1235.08, "end": 1238.56, "text": " so that's the mean color, then you mask it,"}, {"start": 1238.56, "end": 1240.52, "text": " and then you make the model predict"}, {"start": 1240.52, "end": 1245.2, "text": " what the color should be, depending on the context."}, {"start": 1245.2, "end": 1247.8799999999999, "text": " So that's how the self-supervised thing works."}, {"start": 1247.8799999999999, "end": 1249.24, "text": " They didn't only try the color,"}, {"start": 1249.24, "end": 1252.4, "text": " they also tried predicting the downsampled version"}, {"start": 1252.4, "end": 1254.84, "text": " of the patch embedding that's like four by four"}, {"start": 1254.84, "end": 1258.84, "text": " instead of like 16 by 16, as well as the color,"}, {"start": 1258.84, "end": 1261.04, "text": " and they also tried regressing the whole thing,"}, {"start": 1261.04, "end": 1263.6, "text": " so like doing auto-encoding pretty much."}, {"start": 1263.6, "end": 1267.16, "text": " And as I already mentioned, results are not as good,"}, {"start": 1267.16, "end": 1269.48, "text": " but that's a promising like research direction,"}, {"start": 1269.48, "end": 1272.64, "text": " so yeah, we can expect innovation on that front."}, {"start": 1273.96, "end": 1276.92, "text": " Let me go through two more sections which I found important."}, {"start": 1276.92, "end": 1279.04, "text": " The first one, they did a scaling effort"}, {"start": 1279.04, "end": 1282.08, "text": " to figure out how should they scale their model"}, {"start": 1282.08, "end": 1284.04, "text": " to get the best performance out of it."}, {"start": 1284.04, "end": 1286.3600000000001, "text": " They tried doing depth, they tried doing width,"}, {"start": 1286.3600000000001, "end": 1288.48, "text": " they tried making patches smaller,"}, {"start": 1288.48, "end": 1293.48, "text": " and like obviously depth gave nice results."}, {"start": 1293.68, "end": 1298.3600000000001, "text": " As we can see here on the graph, on the chart,"}, {"start": 1298.36, "end": 1303.36, "text": " basically increasing depth gives us better performance."}, {"start": 1303.8, "end": 1305.6799999999998, "text": " The computer also goes up, obviously,"}, {"start": 1305.6799999999998, "end": 1309.9199999999998, "text": " but like yeah, increasing depth will increase the performance."}, {"start": 1309.9199999999998, "end": 1312.6399999999999, "text": " Now the interesting thing for me is that patch size,"}, {"start": 1312.6399999999999, "end": 1315.8799999999999, "text": " the smaller it is, they gain more performance."}, {"start": 1315.8799999999999, "end": 1317.4399999999998, "text": " So I'm gonna make a prediction now."}, {"start": 1317.4399999999998, "end": 1319.9599999999998, "text": " Basically somebody will take some of the"}, {"start": 1319.9599999999998, "end": 1321.8799999999999, "text": " efficient transformers like reformer,"}, {"start": 1321.8799999999999, "end": 1324.6, "text": " like longformer or leanformer,"}, {"start": 1324.6, "end": 1327.8799999999999, "text": " and they'll do the same thing as a vision transformer."}, {"start": 1327.88, "end": 1331.7600000000002, "text": " They just use smaller patches and they'll get better results."}, {"start": 1331.7600000000002, "end": 1336.7600000000002, "text": " That's a research I can expect to come over the next months."}, {"start": 1336.88, "end": 1339.5600000000002, "text": " And finally, let me finish up,"}, {"start": 1339.5600000000002, "end": 1342.64, "text": " wrap up this already long deep dive,"}, {"start": 1345.3600000000001, "end": 1349.3200000000002, "text": " like talking a little bit more about positional encodings."}, {"start": 1349.3200000000002, "end": 1353.92, "text": " They tried four things, and the biggest gap was"}, {"start": 1353.92, "end": 1357.0400000000002, "text": " between using no positional embeddings,"}, {"start": 1357.04, "end": 1360.08, "text": " I treating the patches as a bag of patches,"}, {"start": 1360.08, "end": 1363.24, "text": " and using 1D positional embeddings."}, {"start": 1363.24, "end": 1365.04, "text": " So that's where the highest gap was,"}, {"start": 1365.04, "end": 1366.72, "text": " and you can see here the numbers."}, {"start": 1366.72, "end": 1370.48, "text": " So using positional embeddings helps."}, {"start": 1370.48, "end": 1375.48, "text": " Now how exactly do you like, which heuristic do you use?"}, {"start": 1375.8, "end": 1379.72, "text": " Let's call it that way, that matters a bit less."}, {"start": 1379.72, "end": 1382.56, "text": " So they tried 2D positional embeddings"}, {"start": 1382.56, "end": 1384.6, "text": " and 1D which I already explained."}, {"start": 1384.6, "end": 1385.68, "text": " Let me try to quickly explain"}, {"start": 1385.68, "end": 1388.0, "text": " how the 2D positional embeddings work."}, {"start": 1389.04, "end": 1392.0, "text": " They basically have two tables this time,"}, {"start": 1393.1200000000001, "end": 1396.96, "text": " and how you index into these tables is the following."}, {"start": 1396.96, "end": 1400.92, "text": " So you have your input image,"}, {"start": 1400.92, "end": 1403.0800000000002, "text": " again you split it into patches."}, {"start": 1404.1200000000001, "end": 1407.68, "text": " And so this one will be patch one one,"}, {"start": 1407.68, "end": 1409.92, "text": " this one will be patch one two,"}, {"start": 1409.92, "end": 1412.52, "text": " this one will be patch one three,"}, {"start": 1412.52, "end": 1414.0800000000002, "text": " this is row two, et cetera."}, {"start": 1414.08, "end": 1416.04, "text": " So you basically take the indices,"}, {"start": 1416.04, "end": 1419.84, "text": " so let's say this patch will have one,"}, {"start": 1419.84, "end": 1424.84, "text": " like so basically take, so this is one, two, same here,"}, {"start": 1427.58, "end": 1432.58, "text": " one, two, you basically take the first embedding here,"}, {"start": 1433.8799999999999, "end": 1438.24, "text": " and you concatenate it with the second embedding here."}, {"start": 1438.24, "end": 1442.06, "text": " So this represents the X dimension,"}, {"start": 1442.06, "end": 1444.52, "text": " this one represents the Y dimension,"}, {"start": 1444.52, "end": 1448.84, "text": " and you just train the transformer like that."}, {"start": 1448.84, "end": 1451.94, "text": " And they showed that that approach won't do anything better"}, {"start": 1451.94, "end": 1456.46, "text": " than just using a simple 1D positional encoding."}, {"start": 1456.46, "end": 1458.6, "text": " So that was pretty much it."}, {"start": 1458.6, "end": 1461.8, "text": " This was already probably too long."}, {"start": 1461.8, "end": 1464.32, "text": " Hopefully you found this video useful."}, {"start": 1464.32, "end": 1466.76, "text": " This is the first time I'm doing like a deep dive,"}, {"start": 1466.76, "end": 1471.1799999999998, "text": " like a semi deep dive of a research paper on this channel."}, {"start": 1471.18, "end": 1472.74, "text": " If you found it useful,"}, {"start": 1472.74, "end": 1474.4, "text": " please leave a comment down in the description"}, {"start": 1474.4, "end": 1477.5600000000002, "text": " if you'd like me to make more of these."}, {"start": 1477.5600000000002, "end": 1481.28, "text": " And yeah, subscribe if you haven't already"}, {"start": 1481.28, "end": 1483.3600000000001, "text": " and click that bell icon to get notified"}, {"start": 1483.3600000000001, "end": 1485.0, "text": " when I upload a new video."}, {"start": 1485.0, "end": 1488.4, "text": " And until next time, keep learning deep."}, {"start": 1488.4, "end": 1507.96, "text": " Bye."}]