ResNet Strikes Back! | Patches Are All You Need? | Papers Explained
❤️ Become The AI Epiphany 
Patreon ❤️ ► https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

In this video I cover 2 recent papers:
* ResNet strikes back: An improved training procedure in timm
* Patches Are All You Need?

The ResNet paper proposes a novel training procedure and boosts ResNet-50 top-1 accuracy to over 80%!

The "Patches Are All You Need" paper shows that the template (using patch embeddings + isotropic model) may be playing a more important role than the transformer block itself.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ ResNet paper: https://arxiv.org/abs/2110.00476
✅ All You Need paper: https://openreview.net/forum?id=TVHS5Y4dNvM

✅ timm lib: https://github.com/rwightman/pytorch-image-models/
✅ My transformer implementation: https://github.com/gordicaleksa/pytorch-original-transformer
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
00:50 Paper 1: high-level overview
02:50 What caused the accuracy boost?
05:45 Data augmentations explained
07:05 Regularization explained
08:55 Optimizers explained
09:50 Experiments
12:50 Inherent result variance
13:55 Training procedure generalization to other datasets
15:10 Making wrong conclusions is easy
16:30 Paper 2: high-level overview
17:35 Architecture explained
22:25 A note on paper length
23:05 My thoughts
24:05 Single tweet implementation

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Bartłomiej Danek
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR ML PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#resnet #patches #allyouneed
 What's cracking guys? In this video I'm covering two papers. The first one is ResNet Strikes Back and the second one is Patches Are All You Need. And basically the reason I'm doing that is because this paper is super simple and a very neat idea. It's a four pager, it's a very short paper and the second reason is because both of these papers tell us that we don't know what we are doing, basically. So this one tells us that we are oftentimes doing unfair comparisons with some baselines and this one tells us that oftentimes we don't know how to disentangle what is bringing the actual performance, whether it is the actual architecture or the template itself, like something else basically. So I'll dig into the paper a bit later. Let me first start with ResNet Strikes Back and both of these are fairly short. I think 30 minutes will be enough. So anyways, it's a paper by Ross Whitman from the team library, famous image models, PyTorch library, and Hugo Touvron and Hervé Gégaud. I'm butchering their names, sorry for that. Anyways, so let's dig into the paper itself. The influential residual networks designed by Hiahtol remain the gold standard architecture in numerous scientific publications. So the ResNet-50 architecture published back in 2015 is still used as a baseline for various tasks. And they say that in this paper we re-evaluate the performance of the vanilla ResNet-50 when trained with a procedure that integrates such advances, i.e. the advances that appeared from 2015 onwards. And finally, for instance, with our more demanding training setting, a vanilla ResNet-50 achieves 80.4% top-one accuracy at resolution 224 times 224 on image net validation without extra data or distillation. Whereas the, I think, the original number was somewhere around 76 or something, so around five points, which is a decent advancement considering that they are basically freezing the original architecture and just adding novel, like, things, so like optimizers, augmentations, etc. We're going to see that in a second. So they have this, like, symbolic high-level representation formula here, which tells us basically that the accuracy of a certain model is a function of the architecture design, of the training procedure, and of noise and overfitting. And now, basically, it's kind of hard to decouple, I'd say, these two, because you do overfit the training procedure oftentimes of your model to a specific dataset. So I don't know how they decouple these two, but in any case, what they're focusing on in this paper is the training procedure itself. So let's see what they actually have done in this paper, how have they approved the ResNet-50. So they basically have three procedures. You can see them here, so A1, A2, and A3. The first one is the most compute-heavy one, the best one, at least for their setup, and you can see because it has 600 epochs, lots of time to train this one, like, memory intensive, etc., etc., and they achieved the 80.4 number we saw above. So that's achieved with this A1 procedure. Basically, what I've done is across multiple dimensions, such as the loss, such as data augmentations, regularization, and optimization, they kind of introduced the best techniques and approaches we know now in 2021, and they got the results that they reported. So let's see what the thing is about. Okay, so I went ahead and just pasted in the mixup and the cut-mix procedures so that it's easier to understand what I'm talking about here. So basically, they've used, as they say here, they're using mixup and cut-mix, and the thing they modified compared to the original papers is the following. So they say here, in our training, we assume instead that these concepts are all present and treat the classification as a multi-label classification problem, one versus all. So the thing is, as you can see here, what mixup does is both in the image space as well as in the output label space, they mix up various images, various categories. So here you can see that the two techniques do this in a bit different way. So this one applies some blending, and you can see like the cat here and the dog face here as well. So that's the image space, and then what happens in the label space is depending on the blending coefficient, they basically have the output distribution look something like this. Like maybe the cat will be 0.4 probability and the dog will be like 0.6. So this will be your target distribution you're trying to match using cross entropy. Same thing with cut-mix, except that here, as you can see, you're actually like pasting a crop of your second image of the cat here into your original image, and basically depending on the ratio of how much of the original image are you occupying with this crop, you'll be again forming these probabilities and thus forming the target distribution. So what this paper instead like proposes is to simply, instead of reweighting this, just put like one for the cat and one for the dog, and use binary cross entropy instead. So basically for every single class you'll be doing the log thing and like accumulating that to form the loss. So it's a fairly simple like a tweak and it turns out it helps a lot. So that's the first thing they do here. That's the loss component. Then they have the data augmentation component. Let me read this through. So we adopt the following combination of data augmentations. On top of a standard random resized crop, so RRC, so that's your like standard stuff, standard augmentation, you basically take a like a random crop from the image and you resize it into a temp like a 224 times 224. Horizontal flipping, which is obviously just flipping the image. They apply the the theme variance, so that's the library I mentioned, of rand augment and we're going to see what it is in a second. Mix up and cut mix. So what rand augment does is basically has a like a series of augmentations such as shear, as you can see here, such as changing the some photometric transformations such as this auto contrast and they basically have an automatic way to figure out these augmentations. So I don't think it's that important to understand. You can check out the paper yourself, but like all in all they're just using these techniques that people have devised over the last couple of years and they just kind of experimented with a lot of them and found the best configuration for Reson50. They mentioned mix up and cut mix. Let me just see here. So cut mix is this one and mix up is this one. So those two are augmentations as well. So yeah, anyways, regularization is the third thing they change here and they say in addition to adapting the weight decay we use label smooting, repeated augmentation and stochastic depth. So let me quickly go through all of these. So label smooting, what they do is they basically, this was used in the original transformer paper and you can check my implementation of the original transformer where I've actually implemented this label smooting thing, but the idea is super simple. So instead of having like a hard target where you have certain class having like the probability one, you just put something like one minus epsilon and then you take the epsilon, because you want to have the probability sum up to one, right? And so you just basically distribute that, whoops, the epsilon across all of the other classes. So that's going to be epsilon divided by n minus one, where n is the number of classes, so that's going to be a value at any one of these positions. So that's the label smooting. Repeated augmentation is a fairly simple technique as well. So in a batch you take a single image and you apply a couple of augmentations to that image and you include all of those replicas of that original image inside of the same batch and that turns out to kind of improve the results. So as you can see all of these are random heuristics and things just work and yeah. And finally stochastic depth is similar to dropout in a way, so what you do instead is you have a like a resonant module, so you have something like this, you have a block here and this is your regular resonant thing, so you have like a addition here, you're basically having a skip connection and so what this stochastic depth does is with certain probability they're going to short like short circuit these blocks here and that's how the the thing works. So just an additional heuristic like a variation of a dropout for these models. Okay so that's those are the three main components and finally instead of using Atom or SGD they're actually using LAM optimizer. So we therefore focus on LAM with cosine schedule as the default optimizer for training our resonant 50 and the reason is they are using a way larger batch sizes compared to the original resonant 50 training and then like this paper showed and some other transformer papers that LAM is really good and hopefully i'm pronouncing the name correctly but LAM is really good when you have like a lot of images in a batch and that's pretty much it. Like they take resonant 50, they applied lots of these different hacks, they figured out they did a enormous like kudos to the authors because this was i guess enormous amount of work put into this paper to do the extensive experimentations and all to kind of like sweep through the combinatorial space and figure out the best possible combinations. Okay let's see the experiments. So first things first here you can see the thing i just mentioned. So the column here contains all of the variables, the hyperparameters they were tweaking in order to find these optimal optimal trainings procedures. So A1, A2 and A3. As you can see here they're comparing them with PyTorch default training procedure for resonant 50 and some other some other procedures here from like this is the vision transformer etc. The main thing you can notice here is they're using LAM instead of Atom and SGD optimizers. They have different learning rates, they have different schedules, etc. etc. So in some procedures such as in A1 they're using label smooting, whereas in others they are not using label smooting, they're also using stochastic depth etc. So i don't think it's that important to understand all the details. The authors did the all the hard lifting for us. Bottom line is once they found this this like correct configuration they got the 80.4 result and you can see that this result is way better compared to the reported results that all of these other papers reported. So this vision transformer paper reported 78.4 which as you can see is almost 2 percent. The comparison was not fair because they were probably using some advanced like training receipt for for vision transformer, whereas they were not using they didn't try as hard to improve Resin50 which is I mean understandable. You cannot invest that much time into reconstructing baselines and trying to make them as best as possible because it's hard. It takes a lot of computing, it takes a lot of time, so people just take like numbers reported in previous papers. Okay so let's see some tables here. They took the same like training procedures like labeled A1, A2 and A3 and they applied them to various other models in order to see whether this specific training procedure generalizes to other architectures as well as to the Resin50. So the final results are that mostly they do if certain constraints are obeyed. So the model has to be similar in size and depth to Resin50 and the more similar it is the better the results will be. So basically that means they did overfit. And if we take a look at some results here, if we take a look at EfficientNet B4 for example, you can see that A2 is actually better for that model compared to A1, whereas for Resin50 obviously the best results come from A1. Also some deeper models such as this SE Squeeze Excite I guess, Net154, you can see results are a tad better here with A2 compared to A1 which is because it's way deeper compared to Resin50 and I guess you'd have to kind of tweak the procedure to get better results. Okay second next up is, I'll actually just skip this table, you can check it out yourself and let me focus on this result here. So they showed that once they do evaluation on ImageNet V2 versus the original ImageNet, you can kind of see that, so all of these crosses here are like a single model that was trained on ImageNet and then evaluated on these two datasets and you can see that this seems like there is no correlation, at least like just eyeballing this thing, because for example the best top one accuracy on this V2 dataset comes for at the same time the model has, I guess, around mean like top one accuracy on the original ImageNet. So now imagine we had different training procedure and not just different random seed and you can see there is a lot of variance that can happen and so there is a lot of potential confusion that can happen and it's hard to disentangle whether something you added is actually beneficial or not. Yeah anyways, let's continue here. There is a couple more things I want to show you. First thing is this one, so they took the same model and they fine-tuned it on various other datasets and you can see that, like this is a one procedure and you can see that the results on ImageNet are fairly better here and the gap is pretty big, but once you go and compare it with on other datasets you can see that these results are fairly similar. So even here and here we still have some like a gap, I guess this one has a similar distribution to ImageNet, that would be my assumption, but yeah. Anyways, this tells us that even this fancy like training procedure, they took some, like a lot of compute to figure out, is not, will not be like the best one on other datasets and the differences are even smaller when you take a look at A2. So here sometimes they even have worse performance. So you can see here versus this one here. Yeah, I guess all of this is to show that it's, A, it's hard to find a right training procedure and B, even if you find it, it's not gonna generalize to other setups. So yeah, this seems too hacky for me, like following Richard Sutton's like philosophy for this approach, this type of approach to be the way to go in the future. So one more thing, one more table that's gonna strengthen that argument even more is this one here. So they took Resonant 50 and this DATE-S model, which is I think similar in size to Resonant 50, and what they've done is they took the A2 procedure and trained both of them and they got results here. And you can see that Resonant 50 seems to be better, the conclusion is this one is better than VIT. And then they took T2 procedure, which was tailored to this VIT, and you can see that now with this training procedure it seems that this one is better compared to Resonant 50. The point here is when you have a paper and people figure out a certain procedure, that procedure is gonna probably benefit the model that's being published and it's gonna be unfair compared to all of the previous baselines. And then they have, in this second case, once they are evaluating on ImageNet V2 instead of the, here it was ImageNet, you can see that here, like for both procedures, it seems that VIT is better compared to Resonant 50, although this here is not, like the difference is very small, so yeah. In any case, this interplay between where you're evaluating your models and the training procedure you're using, all this makes it easy to to make a wrong conclusion. And yeah, that's pretty much it. That's the Resonant Strikes Again paper. And now let me switch back to this new paper, like just published a couple of days ago, it's still, like the authors are still anonymous, it's a double blind review. And the idea here was the following. So they made this CovMix mixer model and they showed that it's very simple, it's just using convolutions, as we're going to see in a second, and they again have, they reported better results compared to these other baselines such as the VIT, ResNet, and ResMLP. And now the thing is, probably with the procedures like that were advised in the paper I just covered, would make this ResNet way better, maybe even comparable, because we know it went over 80. So yeah, so basically it's very hard to compare like models, especially when you just have a single dimension like parameters here, because there are many other things we may care more about, like throughput, etc. Okay, anyways, let's see what this paper is about. So if you already know something about vision transformers or MLP mixer, this thing will look very familiar. If you haven't watched those videos, if you haven't explored those papers, you can check out my videos, I've covered both of them, and I'll link them somewhere here. Anyways, you can see the common pattern, we have like images split into various patches, then the patches basically projected into like a vector, and then in the case of VIT, what they've done is they just applied a transformer, like a couple of transformer blocks, in the case of MLP mixer, they've done mixing over a spatial extent and over the channel extent, by applying basically MLP, or you can also consider it as a type of a convolution. In any case, what this paper shows is that you do not need to use like self-attention, like in the case of VITs, you can just use simple convolutional layers. So the thing they do here is they apply depth-wise convolution, then they have like a activation function, they have batch norm, and then they do point-wise convolution, activation function, batch norm, and they just repeat this a couple of times, and that's it. It's super simple, we're gonna see it, they can basically fit this whole implementation of the model in a single tweet, they have that in the appendix, so that's a fun fact. And so what depth-wise and point-wise do is the same thing as mixer, they're just doing spatial-wise and channel-wise mixing. A brief recap how those two work, so how depth-wise and point-wise work, is if we have like a activation volume here, so basically, let me just draw it out here, so something like this, so this will be the number of channels, and this will be like, I don't know, this will, this is like width, and this thing here is height. So what depth-wise convolution does is you take a single channel, like maybe this one, and you just apply a kernel across that, that channel, across the spatial extent of the activation volume, and that's the, and you just get like a new feature map. Point-wise, what it does is you basically have the opposite, so you take one, times one, kernel, and that will have basically as many channels as the activation volume has, and basically then, this is how you're going to form novel features by just doing a stride over the spatial extent, and you're going to mix up, as you can see here, channels, whereas here you're mixing up the spatial information. So that's a idea that we've seen already in MLP Mixer, and yeah, in any case, what they argue, because we'll soon see, we saw that their results are better compared to MLP Mixers and VATs in some cases, what they show is that this actual template of having this patch embedding, and then having this, how they call it, isotropic model, basically the thing is you're not reducing neither the number of channels nor the spatial extent, whereas in the CNNs, for example, you're basically usually, you're reducing the spatial extent and you're adding the number of channels as you're going deeper into the network. So these isotropic models basically preserve both the number of channels as well as the spatial extent, and they argue that maybe, just maybe, that template of using patch embeddings plus that isotropic, like, modeling of features is the thing that actually gives us performance, not the self-attention, not the other fancy things we've imported from NLP world. In any case, the model is super simple, it's a very nice baseline, you can see here, this is the whole implementation, as you can see, so they have this initial layer here, is basically just the patch embedding, then they have a couple of these depth, so N of these layers, and finally they add the adaptive average pooling and they flatten out the features and they add a linear classifier on top of it. So it's a fairly simple model and it achieves comparable, if not better, results compared to VAT, compared to ResNet and ResMLP. So now the thing here, the red alert is the throughput is super, super small, they mentioned that it's possible to, in future research, improve this. I mean, this ties back to the thing I mentioned, in this plot here, when you just plot a single dimension, like number of parameters here, you may paint up like a wrong picture. But in any case, this paper is not trying to set SOTA, they're just trying to say that we are maybe focusing on the wrong thing, I guess. So we went all the way from ResNet and convolutional neural networks to transformers and vision transformers to finally get back to convolutional neural networks, again, in their pure form, just changing this, just keeping this isotropic template. Also, I like the end part of this paper, a note on paper length, expecting more text in this paper, wondering if it's a workshop paper, we hastily submitted to iClear. No, this paper presents a simple idea, one where we genuinely believe that a short paper presentation is more effective. Do we really need exactly eight, now nine or ten pages to describe every machine learning architecture and algorithm in existence? We proposed an incredibly simple architecture and made a very simple point that we think is worth more discussion. Patches work well in convolutional architectures. We think that four pages is more than enough space for this. The details of the experiments and architectures are in the appendix for those who want to read through it all. A lot of respect to the authors, to be honest. Aside from this, I really think we need to question this whole template of how we are presenting the research. I think that PDFs, for one, are obsolete. I think that authors need to write like accompanying blogs a lot more. We need something like interactive and types of notebooks, such as Jupyter or Deep Note is also one cool notebook environment. So, I think we need as a community, we need to rethink how we are presenting, because the whole idea of this paper is for other researchers to understand what these guys have done and be able to reconstruct or build upon their ideas. And so, who cares about like 10-pager format? Paper reading in and off itself is hard and I guess any help we can get from the technology advancements we have, such as interactivity, visualizations, animations, will help us as a community. Finally, let me show you the one tweet implementation. So, here it is. An implementation of our model in exactly 280 characters is right here. Twitter is slowly becoming a more important medium, if not already, compared to conferences. You can check out other results in the paper. They have some nice visualizations of the kernels. I'll leave you with that and if you like this video, consider subscribing, share out the video, hit that bell icon to get notified every time I upload a new video, also join the Discord community. We've got a lot of smart people there and until next time, bye-bye!
[{"start": 0.0, "end": 3.68, "text": " What's cracking guys? In this video I'm covering two papers. The first one is"}, {"start": 3.68, "end": 8.24, "text": " ResNet Strikes Back and the second one is Patches Are All You Need."}, {"start": 8.24, "end": 12.24, "text": " And basically the reason I'm doing that is because this paper is super"}, {"start": 12.24, "end": 15.280000000000001, "text": " simple and a very neat idea. It's a four pager,"}, {"start": 15.280000000000001, "end": 19.68, "text": " it's a very short paper and the second reason is because both of these papers"}, {"start": 19.68, "end": 23.6, "text": " tell us that we don't know what we are doing,"}, {"start": 23.6, "end": 27.68, "text": " basically. So this one tells us that we are oftentimes doing unfair"}, {"start": 27.68, "end": 32.08, "text": " comparisons with some baselines and this one tells us that oftentimes"}, {"start": 32.08, "end": 35.36, "text": " we don't know how to disentangle what is"}, {"start": 35.36, "end": 39.28, "text": " bringing the actual performance, whether it is the actual architecture or"}, {"start": 39.28, "end": 43.519999999999996, "text": " the template itself, like something else basically. So I'll dig into the paper a"}, {"start": 43.519999999999996, "end": 48.08, "text": " bit later. Let me first start with ResNet Strikes Back and both of"}, {"start": 48.08, "end": 51.04, "text": " these are fairly short. I think 30 minutes will be"}, {"start": 51.04, "end": 55.04, "text": " enough. So anyways, it's a paper by Ross Whitman"}, {"start": 55.04, "end": 59.839999999999996, "text": " from the team library, famous image models, PyTorch library,"}, {"start": 59.839999999999996, "end": 64.8, "text": " and Hugo Touvron and Herv\u00e9 G\u00e9gaud. I'm butchering their names, sorry for that."}, {"start": 64.8, "end": 68.16, "text": " Anyways, so let's dig into the paper itself."}, {"start": 68.16, "end": 71.75999999999999, "text": " The influential residual networks designed by Hiahtol"}, {"start": 71.75999999999999, "end": 75.36, "text": " remain the gold standard architecture in numerous scientific publications."}, {"start": 75.36, "end": 78.56, "text": " So the ResNet-50 architecture published back in 2015"}, {"start": 78.56, "end": 81.84, "text": " is still used as a baseline for various tasks."}, {"start": 81.84, "end": 85.44, "text": " And they say that in this paper we re-evaluate the performance of the"}, {"start": 85.44, "end": 88.72, "text": " vanilla ResNet-50 when trained with a procedure that"}, {"start": 88.72, "end": 93.92, "text": " integrates such advances, i.e. the advances that appeared from"}, {"start": 93.92, "end": 98.96000000000001, "text": " 2015 onwards. And finally, for instance, with our more demanding training"}, {"start": 98.96000000000001, "end": 103.52000000000001, "text": " setting, a vanilla ResNet-50 achieves 80.4%"}, {"start": 103.52000000000001, "end": 107.76, "text": " top-one accuracy at resolution 224 times 224 on image"}, {"start": 107.76, "end": 111.44, "text": " net validation without extra data or distillation."}, {"start": 111.44, "end": 114.48, "text": " Whereas the, I think, the original number was somewhere around"}, {"start": 114.48, "end": 117.75999999999999, "text": " 76 or something, so around five points, which is a"}, {"start": 117.75999999999999, "end": 121.44, "text": " decent advancement considering that they are basically freezing the"}, {"start": 121.44, "end": 126.64, "text": " original architecture and just adding novel, like, things, so like"}, {"start": 126.64, "end": 131.04, "text": " optimizers, augmentations, etc. We're going to see that in a second."}, {"start": 131.04, "end": 134.88, "text": " So they have this, like, symbolic high-level representation"}, {"start": 134.88, "end": 139.04, "text": " formula here, which tells us basically that the accuracy of a certain model"}, {"start": 139.04, "end": 143.35999999999999, "text": " is a function of the architecture design, of the training"}, {"start": 143.35999999999999, "end": 148.72, "text": " procedure, and of noise and overfitting. And now, basically, it's kind of hard to"}, {"start": 148.72, "end": 153.2, "text": " decouple, I'd say, these two, because you do overfit the training procedure"}, {"start": 153.2, "end": 155.92, "text": " oftentimes of your model to a specific dataset."}, {"start": 155.92, "end": 160.07999999999998, "text": " So I don't know how they decouple these two, but in any case,"}, {"start": 160.07999999999998, "end": 164.48, "text": " what they're focusing on in this paper is the training procedure itself."}, {"start": 164.48, "end": 168.07999999999998, "text": " So let's see what they actually have done in this paper, how"}, {"start": 168.08, "end": 174.24, "text": " have they approved the ResNet-50. So they basically have three procedures."}, {"start": 174.24, "end": 180.24, "text": " You can see them here, so A1, A2, and A3. The first one is the most compute-heavy"}, {"start": 180.24, "end": 184.32000000000002, "text": " one, the best one, at least for their setup, and you"}, {"start": 184.32000000000002, "end": 188.16000000000003, "text": " can see because it has 600 epochs,"}, {"start": 188.16000000000003, "end": 192.88000000000002, "text": " lots of time to train this one, like, memory intensive, etc., etc., and they"}, {"start": 192.88000000000002, "end": 197.44, "text": " achieved the 80.4 number we saw above. So that's achieved with this"}, {"start": 197.44, "end": 202.88, "text": " A1 procedure. Basically, what I've done is across multiple"}, {"start": 202.88, "end": 209.35999999999999, "text": " dimensions, such as the loss, such as data augmentations, regularization,"}, {"start": 209.35999999999999, "end": 213.04, "text": " and optimization, they kind of introduced the best"}, {"start": 213.04, "end": 219.76, "text": " techniques and approaches we know now in 2021, and they got the"}, {"start": 219.76, "end": 224.4, "text": " results that they reported. So let's see what the thing is about."}, {"start": 224.4, "end": 228.48000000000002, "text": " Okay, so I went ahead and just pasted in the mixup and the cut-mix"}, {"start": 228.48000000000002, "end": 231.92000000000002, "text": " procedures so that it's easier to understand what I'm talking about here."}, {"start": 231.92000000000002, "end": 237.04000000000002, "text": " So basically, they've used, as they say here, they're using mixup and cut-mix,"}, {"start": 237.04000000000002, "end": 240.56, "text": " and the thing they modified compared to the original papers is the following."}, {"start": 240.56, "end": 244.08, "text": " So they say here, in our training, we assume instead that these concepts are"}, {"start": 244.08, "end": 246.48000000000002, "text": " all present and treat the classification as a"}, {"start": 246.48000000000002, "end": 250.72, "text": " multi-label classification problem, one versus all. So the thing is, as you can"}, {"start": 250.72, "end": 254.72, "text": " see here, what mixup does is both in the image"}, {"start": 254.72, "end": 258.96, "text": " space as well as in the output label space, they mix up"}, {"start": 258.96, "end": 263.52, "text": " various images, various categories. So here you can see"}, {"start": 263.52, "end": 267.28, "text": " that the two techniques do this in a bit different way."}, {"start": 267.28, "end": 270.48, "text": " So this one applies some blending, and you can see"}, {"start": 270.48, "end": 274.56, "text": " like the cat here and the dog face here as well. So that's the image space, and"}, {"start": 274.56, "end": 277.84, "text": " then what happens in the label space is depending on the blending coefficient,"}, {"start": 277.84, "end": 280.64, "text": " they basically have the output distribution look something like this."}, {"start": 280.64, "end": 285.44, "text": " Like maybe the cat will be 0.4 probability and the"}, {"start": 285.44, "end": 289.28, "text": " dog will be like 0.6. So this will be your target distribution you're trying"}, {"start": 289.28, "end": 293.91999999999996, "text": " to match using cross entropy. Same thing with cut-mix, except that here,"}, {"start": 293.91999999999996, "end": 298.71999999999997, "text": " as you can see, you're actually like pasting a crop"}, {"start": 298.71999999999997, "end": 304.0, "text": " of your second image of the cat here into your original image, and"}, {"start": 304.0, "end": 307.52, "text": " basically depending on the ratio of how much of the"}, {"start": 307.52, "end": 311.28, "text": " original image are you occupying with this crop, you'll be again"}, {"start": 311.28, "end": 315.12, "text": " forming these probabilities and thus forming the"}, {"start": 315.12, "end": 319.68, "text": " target distribution. So what this paper instead"}, {"start": 319.68, "end": 325.52, "text": " like proposes is to simply, instead of reweighting this, just put like one for"}, {"start": 325.52, "end": 330.15999999999997, "text": " the cat and one for the dog, and use binary cross entropy instead."}, {"start": 330.15999999999997, "end": 333.44, "text": " So basically for every single class you'll be doing"}, {"start": 333.44, "end": 337.28, "text": " the log thing and like accumulating that to form the loss."}, {"start": 337.28, "end": 342.32, "text": " So it's a fairly simple like a tweak and it turns out it helps a lot."}, {"start": 342.32, "end": 345.91999999999996, "text": " So that's the first thing they do here. That's the loss component. Then they have"}, {"start": 345.91999999999996, "end": 348.71999999999997, "text": " the data augmentation component. Let me read this through."}, {"start": 348.71999999999997, "end": 352.15999999999997, "text": " So we adopt the following combination of data augmentations. On top of"}, {"start": 352.15999999999997, "end": 356.0, "text": " a standard random resized crop, so RRC, so that's your"}, {"start": 356.0, "end": 359.2, "text": " like standard stuff, standard augmentation, you basically take a"}, {"start": 359.2, "end": 363.52, "text": " like a random crop from the image and you resize it into a temp like a 224"}, {"start": 363.52, "end": 368.32, "text": " times 224. Horizontal flipping, which is obviously just flipping the image."}, {"start": 368.32, "end": 373.35999999999996, "text": " They apply the the theme variance, so that's the library I mentioned,"}, {"start": 373.35999999999996, "end": 377.03999999999996, "text": " of rand augment and we're going to see what it is in a second."}, {"start": 377.03999999999996, "end": 383.2, "text": " Mix up and cut mix. So what rand augment does is basically has a like a"}, {"start": 383.2, "end": 387.44, "text": " series of augmentations such as shear, as you can see here, such as changing the"}, {"start": 387.44, "end": 390.47999999999996, "text": " some photometric transformations such as this auto"}, {"start": 390.48, "end": 394.64000000000004, "text": " contrast and they basically have an automatic way to figure out these"}, {"start": 394.64000000000004, "end": 397.76, "text": " augmentations. So I don't think it's that important to"}, {"start": 397.76, "end": 399.76, "text": " understand. You can check out the paper yourself,"}, {"start": 399.76, "end": 403.36, "text": " but like all in all they're just using these techniques that people have"}, {"start": 403.36, "end": 406.96000000000004, "text": " devised over the last couple of years and they just kind of"}, {"start": 406.96000000000004, "end": 411.76, "text": " experimented with a lot of them and found the best configuration for Reson50."}, {"start": 411.76, "end": 415.28000000000003, "text": " They mentioned mix up and cut mix. Let me just see here."}, {"start": 415.28000000000003, "end": 419.6, "text": " So cut mix is this one and mix up is this one. So those two are augmentations as"}, {"start": 419.6, "end": 422.8, "text": " well. So yeah, anyways,"}, {"start": 422.8, "end": 427.28000000000003, "text": " regularization is the third thing they change here and they say in"}, {"start": 427.28000000000003, "end": 431.20000000000005, "text": " addition to adapting the weight decay we use label smooting,"}, {"start": 431.20000000000005, "end": 435.84000000000003, "text": " repeated augmentation and stochastic depth. So let me quickly go through"}, {"start": 435.84000000000003, "end": 439.28000000000003, "text": " all of these. So label smooting, what they do is"}, {"start": 439.28000000000003, "end": 442.72, "text": " they basically, this was used in the original transformer paper"}, {"start": 442.72, "end": 445.76000000000005, "text": " and you can check my implementation of the original transformer where I've"}, {"start": 445.76000000000005, "end": 447.84000000000003, "text": " actually implemented this label smooting thing,"}, {"start": 447.84, "end": 451.76, "text": " but the idea is super simple. So instead of having like a hard target where you"}, {"start": 451.76, "end": 454.0, "text": " have certain class having like the probability"}, {"start": 454.0, "end": 457.35999999999996, "text": " one, you just put something like one minus"}, {"start": 457.35999999999996, "end": 461.91999999999996, "text": " epsilon and then you take the epsilon, because you want to have the"}, {"start": 461.91999999999996, "end": 465.44, "text": " probability sum up to one, right? And so you just basically distribute that,"}, {"start": 465.44, "end": 468.0, "text": " whoops, the epsilon across all of the other"}, {"start": 468.0, "end": 473.91999999999996, "text": " classes. So that's going to be epsilon divided by n minus one, where n is"}, {"start": 473.91999999999996, "end": 477.2, "text": " the number of classes, so that's going to be a value at any one of these"}, {"start": 477.2, "end": 480.24, "text": " positions. So that's the label smooting. Repeated"}, {"start": 480.24, "end": 483.03999999999996, "text": " augmentation is a fairly simple technique as well."}, {"start": 483.03999999999996, "end": 487.59999999999997, "text": " So in a batch you take a single image and you apply a couple of augmentations to"}, {"start": 487.59999999999997, "end": 491.68, "text": " that image and you include all of those replicas of that original image inside"}, {"start": 491.68, "end": 494.32, "text": " of the same batch and that turns out to kind of improve the"}, {"start": 494.32, "end": 496.8, "text": " results. So as you can see all of these are"}, {"start": 496.8, "end": 500.96, "text": " random heuristics and things just work and yeah."}, {"start": 500.96, "end": 504.56, "text": " And finally stochastic depth is similar to dropout in a way,"}, {"start": 504.56, "end": 508.8, "text": " so what you do instead is you have a like a resonant module,"}, {"start": 508.8, "end": 512.08, "text": " so you have something like this, you have a block here and this is your"}, {"start": 512.08, "end": 516.16, "text": " regular resonant thing, so you have like a addition here, you're basically"}, {"start": 516.16, "end": 520.48, "text": " having a skip connection and so what this stochastic depth does is with"}, {"start": 520.48, "end": 525.36, "text": " certain probability they're going to short like short circuit these blocks"}, {"start": 525.36, "end": 528.48, "text": " here and that's how the the thing works. So"}, {"start": 528.48, "end": 532.0, "text": " just an additional heuristic like a variation of a dropout"}, {"start": 532.0, "end": 537.04, "text": " for these models. Okay so that's those are the three main components"}, {"start": 537.04, "end": 540.4, "text": " and finally instead of using Atom or SGD they're"}, {"start": 540.4, "end": 544.0, "text": " actually using LAM optimizer. So we therefore focus on LAM with"}, {"start": 544.0, "end": 547.92, "text": " cosine schedule as the default optimizer for training our resonant 50"}, {"start": 547.92, "end": 551.44, "text": " and the reason is they are using a way larger batch sizes"}, {"start": 551.44, "end": 555.44, "text": " compared to the original resonant 50 training and then"}, {"start": 555.44, "end": 559.04, "text": " like this paper showed and some other transformer papers that LAM is really"}, {"start": 559.04, "end": 560.48, "text": " good and hopefully i'm pronouncing the name"}, {"start": 560.48, "end": 563.52, "text": " correctly but LAM is really good when you have like a"}, {"start": 563.52, "end": 567.44, "text": " lot of images in a batch and that's pretty much it. Like they"}, {"start": 567.44, "end": 570.5600000000001, "text": " take resonant 50, they applied lots of these different"}, {"start": 570.5600000000001, "end": 574.5600000000001, "text": " hacks, they figured out they did a enormous like"}, {"start": 574.5600000000001, "end": 577.36, "text": " kudos to the authors because this was i guess enormous"}, {"start": 577.36, "end": 582.16, "text": " amount of work put into this paper to do the extensive experimentations and all"}, {"start": 582.16, "end": 586.0, "text": " to kind of like sweep through the combinatorial space and figure out the"}, {"start": 586.0, "end": 588.8000000000001, "text": " best possible combinations. Okay let's see"}, {"start": 588.8, "end": 592.64, "text": " the experiments. So first things first here you can see the"}, {"start": 592.64, "end": 596.9599999999999, "text": " thing i just mentioned. So the column here contains all of the"}, {"start": 596.9599999999999, "end": 601.12, "text": " variables, the hyperparameters they were tweaking in order to find these"}, {"start": 601.12, "end": 606.9599999999999, "text": " optimal optimal trainings procedures. So A1, A2 and A3. As you can see here"}, {"start": 606.9599999999999, "end": 610.24, "text": " they're comparing them with PyTorch default training procedure"}, {"start": 610.24, "end": 614.24, "text": " for resonant 50 and some other some other procedures here from like"}, {"start": 614.24, "end": 617.1999999999999, "text": " this is the vision transformer etc. The main thing you can notice here is"}, {"start": 617.2, "end": 621.84, "text": " they're using LAM instead of Atom and SGD optimizers. They have"}, {"start": 621.84, "end": 625.84, "text": " different learning rates, they have different schedules, etc. etc."}, {"start": 625.84, "end": 630.88, "text": " So in some procedures such as in A1 they're using label smooting,"}, {"start": 630.88, "end": 634.0, "text": " whereas in others they are not using label smooting, they're also using"}, {"start": 634.0, "end": 638.1600000000001, "text": " stochastic depth etc. So i don't think it's that important"}, {"start": 638.1600000000001, "end": 642.24, "text": " to understand all the details. The authors did the all the hard"}, {"start": 642.24, "end": 646.08, "text": " lifting for us. Bottom line is once they found this"}, {"start": 646.08, "end": 652.32, "text": " this like correct configuration they got the 80.4 result and you can see that"}, {"start": 652.32, "end": 656.08, "text": " this result is way better compared to the reported results that all of these"}, {"start": 656.08, "end": 659.12, "text": " other papers reported. So this vision"}, {"start": 659.12, "end": 663.2800000000001, "text": " transformer paper reported 78.4 which as you can see is"}, {"start": 663.2800000000001, "end": 667.44, "text": " almost 2 percent. The comparison was not fair"}, {"start": 667.44, "end": 670.96, "text": " because they were probably using some advanced like training receipt for"}, {"start": 670.96, "end": 673.76, "text": " for vision transformer, whereas they were not using"}, {"start": 673.76, "end": 677.68, "text": " they didn't try as hard to improve Resin50 which is"}, {"start": 677.68, "end": 682.16, "text": " I mean understandable. You cannot invest that much time into"}, {"start": 682.16, "end": 686.56, "text": " reconstructing baselines and trying to make them as best as possible"}, {"start": 686.56, "end": 689.6, "text": " because it's hard. It takes a lot of computing, it takes a lot of time,"}, {"start": 689.6, "end": 693.12, "text": " so people just take like numbers reported in previous papers."}, {"start": 693.12, "end": 699.6, "text": " Okay so let's see some tables here. They took the same like training"}, {"start": 699.6, "end": 703.84, "text": " procedures like labeled A1, A2 and A3 and they"}, {"start": 703.84, "end": 708.32, "text": " applied them to various other models in order to see whether this"}, {"start": 708.32, "end": 711.44, "text": " specific training procedure generalizes to other"}, {"start": 711.44, "end": 716.16, "text": " architectures as well as to the Resin50. So the"}, {"start": 716.16, "end": 721.44, "text": " final results are that mostly they do if certain constraints"}, {"start": 721.44, "end": 727.84, "text": " are obeyed. So the model has to be similar in size and depth to Resin50"}, {"start": 727.84, "end": 730.88, "text": " and the more similar it is the better the results will be. So basically that"}, {"start": 730.88, "end": 733.84, "text": " means they did overfit. And if we take a look at some"}, {"start": 733.84, "end": 737.76, "text": " results here, if we take a look at EfficientNet B4 for"}, {"start": 737.76, "end": 742.32, "text": " example, you can see that A2 is actually better for that model"}, {"start": 742.32, "end": 747.0400000000001, "text": " compared to A1, whereas for Resin50 obviously the best results"}, {"start": 747.0400000000001, "end": 750.48, "text": " come from A1. Also some deeper models such as this"}, {"start": 750.48, "end": 757.2, "text": " SE Squeeze Excite I guess, Net154, you can see results are a tad better"}, {"start": 757.2, "end": 761.76, "text": " here with A2 compared to A1 which is because"}, {"start": 761.76, "end": 766.48, "text": " it's way deeper compared to Resin50 and I guess"}, {"start": 766.48, "end": 770.4000000000001, "text": " you'd have to kind of tweak the procedure to"}, {"start": 770.4000000000001, "end": 776.0, "text": " get better results. Okay second next up is, I'll actually just skip this"}, {"start": 776.0, "end": 780.48, "text": " table, you can check it out yourself and let me focus on this result here. So"}, {"start": 780.48, "end": 786.0, "text": " they showed that once they do evaluation on ImageNet V2"}, {"start": 786.0, "end": 789.36, "text": " versus the original ImageNet, you can kind of"}, {"start": 789.36, "end": 793.52, "text": " see that, so all of these crosses here are like a single model that was"}, {"start": 793.52, "end": 797.44, "text": " trained on ImageNet and then evaluated on these two datasets and you"}, {"start": 797.44, "end": 800.0, "text": " can see that this seems like there is no correlation,"}, {"start": 800.0, "end": 803.92, "text": " at least like just eyeballing this thing, because for example"}, {"start": 803.92, "end": 807.44, "text": " the best top one accuracy on this V2 dataset"}, {"start": 807.44, "end": 813.28, "text": " comes for at the same time the model has, I guess, around mean like top one"}, {"start": 813.28, "end": 817.04, "text": " accuracy on the original ImageNet. So now imagine we had different"}, {"start": 817.04, "end": 819.36, "text": " training procedure and not just different random seed and you can see"}, {"start": 819.36, "end": 821.52, "text": " there is a lot of variance that can happen"}, {"start": 821.52, "end": 825.12, "text": " and so there is a lot of potential confusion that can"}, {"start": 825.12, "end": 828.4, "text": " happen and it's hard to disentangle whether something you added"}, {"start": 828.4, "end": 834.24, "text": " is actually beneficial or not. Yeah anyways, let's continue here."}, {"start": 834.24, "end": 838.8, "text": " There is a couple more things I want to show you. First thing is this one, so"}, {"start": 838.8, "end": 842.0799999999999, "text": " they took the same model and they fine-tuned"}, {"start": 842.08, "end": 846.1600000000001, "text": " it on various other datasets and you can see that,"}, {"start": 846.1600000000001, "end": 850.1600000000001, "text": " like this is a one procedure and you can see that the results on ImageNet"}, {"start": 850.1600000000001, "end": 853.44, "text": " are fairly better here and the gap is pretty big, but once you"}, {"start": 853.44, "end": 857.2, "text": " go and compare it with on other datasets you can see that these results are"}, {"start": 857.2, "end": 861.2800000000001, "text": " fairly similar. So even here and here we still"}, {"start": 861.2800000000001, "end": 865.5200000000001, "text": " have some like a gap, I guess this one has a similar"}, {"start": 865.5200000000001, "end": 868.4000000000001, "text": " distribution to ImageNet, that would be my"}, {"start": 868.4, "end": 872.9599999999999, "text": " assumption, but yeah. Anyways, this tells us that even this"}, {"start": 872.9599999999999, "end": 876.8, "text": " fancy like training procedure, they took some,"}, {"start": 876.8, "end": 880.0799999999999, "text": " like a lot of compute to figure out, is not,"}, {"start": 880.0799999999999, "end": 884.64, "text": " will not be like the best one on other datasets"}, {"start": 884.64, "end": 889.04, "text": " and the differences are even smaller when you take a look at A2."}, {"start": 889.04, "end": 892.24, "text": " So here sometimes they even have worse performance."}, {"start": 892.24, "end": 896.0, "text": " So you can see here versus this one here. Yeah, I guess all of this is to show that"}, {"start": 896.0, "end": 898.32, "text": " it's, A, it's hard to find a right training"}, {"start": 898.32, "end": 902.32, "text": " procedure and B, even if you find it, it's not gonna generalize to other setups."}, {"start": 902.32, "end": 906.72, "text": " So yeah, this seems too hacky for me, like following Richard Sutton's"}, {"start": 906.72, "end": 911.12, "text": " like philosophy for this approach, this type of approach to be the way to go"}, {"start": 911.12, "end": 915.6, "text": " in the future. So one more thing, one more table that's gonna"}, {"start": 915.6, "end": 918.64, "text": " strengthen that argument even more is this one here."}, {"start": 918.64, "end": 923.44, "text": " So they took Resonant 50 and this DATE-S model, which is I think similar in size"}, {"start": 923.44, "end": 926.4000000000001, "text": " to Resonant 50, and what they've done is they took the"}, {"start": 926.4000000000001, "end": 931.0400000000001, "text": " A2 procedure and trained both of them and they got results here."}, {"start": 931.0400000000001, "end": 934.72, "text": " And you can see that Resonant 50 seems to be better,"}, {"start": 934.72, "end": 938.08, "text": " the conclusion is this one is better than VIT."}, {"start": 938.08, "end": 942.0, "text": " And then they took T2 procedure, which was tailored to this VIT,"}, {"start": 942.0, "end": 944.96, "text": " and you can see that now with this training procedure it seems that this"}, {"start": 944.96, "end": 948.1600000000001, "text": " one is better compared to Resonant 50. The point here is"}, {"start": 948.1600000000001, "end": 950.48, "text": " when you have a paper and people figure out a certain"}, {"start": 950.48, "end": 955.36, "text": " procedure, that procedure is gonna probably benefit the model that's"}, {"start": 955.36, "end": 957.36, "text": " being published and it's gonna be unfair"}, {"start": 957.36, "end": 959.04, "text": " compared to all of the previous baselines."}, {"start": 959.04, "end": 962.72, "text": " And then they have, in this second case, once they are"}, {"start": 962.72, "end": 967.12, "text": " evaluating on ImageNet V2 instead of the, here it was ImageNet,"}, {"start": 967.12, "end": 971.12, "text": " you can see that here, like for both procedures,"}, {"start": 971.12, "end": 975.6, "text": " it seems that VIT is better compared to Resonant 50, although this here is not,"}, {"start": 975.6, "end": 979.6, "text": " like the difference is very small, so yeah. In any case, this interplay between"}, {"start": 979.6, "end": 984.4, "text": " where you're evaluating your models and the training procedure you're using,"}, {"start": 984.4, "end": 988.0, "text": " all this makes it easy to to make a wrong conclusion."}, {"start": 988.0, "end": 992.64, "text": " And yeah, that's pretty much it. That's the Resonant Strikes Again paper."}, {"start": 992.64, "end": 996.5600000000001, "text": " And now let me switch back to this new paper,"}, {"start": 996.5600000000001, "end": 1000.24, "text": " like just published a couple of days ago, it's still, like the authors are"}, {"start": 1000.24, "end": 1005.0400000000001, "text": " still anonymous, it's a double blind review. And the idea here was"}, {"start": 1005.04, "end": 1010.4, "text": " the following. So they made this CovMix mixer model"}, {"start": 1010.4, "end": 1014.48, "text": " and they showed that it's very simple, it's just using convolutions, as we're"}, {"start": 1014.48, "end": 1018.4, "text": " going to see in a second, and they again have, they reported"}, {"start": 1018.4, "end": 1022.24, "text": " better results compared to these other baselines such as the VIT,"}, {"start": 1022.24, "end": 1027.36, "text": " ResNet, and ResMLP. And now the thing is,"}, {"start": 1027.36, "end": 1032.8, "text": " probably with the procedures like that were advised in the paper I just"}, {"start": 1032.8, "end": 1036.56, "text": " covered, would make this ResNet way better,"}, {"start": 1036.56, "end": 1040.3999999999999, "text": " maybe even comparable, because we know it went over 80."}, {"start": 1040.3999999999999, "end": 1043.52, "text": " So yeah, so basically it's very hard to compare"}, {"start": 1043.52, "end": 1047.6, "text": " like models, especially when you just have a single dimension like parameters"}, {"start": 1047.6, "end": 1049.36, "text": " here, because there are many other things we"}, {"start": 1049.36, "end": 1053.84, "text": " may care more about, like throughput, etc. Okay, anyways, let's"}, {"start": 1053.84, "end": 1055.9199999999998, "text": " see what this paper is about. So if you already"}, {"start": 1055.9199999999998, "end": 1059.36, "text": " know something about vision transformers or MLP mixer,"}, {"start": 1059.36, "end": 1063.1999999999998, "text": " this thing will look very familiar. If you haven't watched those videos,"}, {"start": 1063.1999999999998, "end": 1066.08, "text": " if you haven't explored those papers, you can check out my videos, I've covered"}, {"start": 1066.08, "end": 1068.8799999999999, "text": " both of them, and I'll link them somewhere here."}, {"start": 1068.8799999999999, "end": 1071.6799999999998, "text": " Anyways, you can see the common pattern, we have"}, {"start": 1071.6799999999998, "end": 1075.6799999999998, "text": " like images split into various patches, then the patches"}, {"start": 1075.6799999999998, "end": 1081.12, "text": " basically projected into like a vector, and then in the case of VIT,"}, {"start": 1081.12, "end": 1084.08, "text": " what they've done is they just applied a transformer,"}, {"start": 1084.08, "end": 1087.4399999999998, "text": " like a couple of transformer blocks, in the case of MLP mixer, they've done"}, {"start": 1087.44, "end": 1091.52, "text": " mixing over a spatial extent and over the channel extent,"}, {"start": 1091.52, "end": 1096.3200000000002, "text": " by applying basically MLP, or you can also consider it"}, {"start": 1096.3200000000002, "end": 1102.0800000000002, "text": " as a type of a convolution. In any case, what this paper shows is that you"}, {"start": 1102.0800000000002, "end": 1105.44, "text": " do not need to use like self-attention, like in the case of"}, {"start": 1105.44, "end": 1109.28, "text": " VITs, you can just use simple convolutional layers. So the"}, {"start": 1109.28, "end": 1112.88, "text": " thing they do here is they apply depth-wise convolution, then they have"}, {"start": 1112.88, "end": 1116.3200000000002, "text": " like a activation function, they have batch norm,"}, {"start": 1116.32, "end": 1120.1599999999999, "text": " and then they do point-wise convolution, activation function, batch norm, and they"}, {"start": 1120.1599999999999, "end": 1123.12, "text": " just repeat this a couple of times, and that's it. It's super simple,"}, {"start": 1123.12, "end": 1126.8799999999999, "text": " we're gonna see it, they can basically fit this whole implementation of the"}, {"start": 1126.8799999999999, "end": 1131.2, "text": " model in a single tweet, they have that in the appendix, so that's a fun fact."}, {"start": 1131.2, "end": 1136.1599999999999, "text": " And so what depth-wise and point-wise do is the same thing as mixer, they're"}, {"start": 1136.1599999999999, "end": 1138.56, "text": " just doing spatial-wise and channel-wise mixing."}, {"start": 1138.56, "end": 1142.72, "text": " A brief recap how those two work, so how depth-wise and point-wise work,"}, {"start": 1142.72, "end": 1146.24, "text": " is if we have like a activation volume here,"}, {"start": 1146.24, "end": 1149.84, "text": " so basically, let me just draw it out here,"}, {"start": 1149.84, "end": 1154.56, "text": " so something like this, so this will be the number of channels,"}, {"start": 1154.56, "end": 1158.64, "text": " and this will be like, I don't know, this will, this is like width,"}, {"start": 1158.64, "end": 1164.72, "text": " and this thing here is height. So what depth-wise convolution does is you"}, {"start": 1164.72, "end": 1167.2, "text": " take a single channel, like maybe this one,"}, {"start": 1167.2, "end": 1170.88, "text": " and you just apply a kernel across that, that channel,"}, {"start": 1170.88, "end": 1174.48, "text": " across the spatial extent of the activation volume,"}, {"start": 1174.48, "end": 1177.7600000000002, "text": " and that's the, and you just get like a new feature map."}, {"start": 1177.7600000000002, "end": 1182.48, "text": " Point-wise, what it does is you basically have the opposite, so you take one,"}, {"start": 1182.48, "end": 1188.72, "text": " times one, kernel, and that will have basically as many channels as"}, {"start": 1188.72, "end": 1194.0800000000002, "text": " the activation volume has, and basically then, this is how you're"}, {"start": 1194.0800000000002, "end": 1197.5200000000002, "text": " going to form novel features by just doing a"}, {"start": 1197.52, "end": 1201.12, "text": " stride over the spatial extent, and you're going to mix up, as you can see"}, {"start": 1201.12, "end": 1204.24, "text": " here, channels, whereas here you're mixing up the spatial"}, {"start": 1204.24, "end": 1208.08, "text": " information. So that's a idea that we've seen already in MLP"}, {"start": 1208.08, "end": 1212.8, "text": " Mixer, and yeah, in any case, what they argue, because"}, {"start": 1212.8, "end": 1217.36, "text": " we'll soon see, we saw that their results are better compared to MLP Mixers and"}, {"start": 1217.36, "end": 1220.6399999999999, "text": " VATs in some cases, what they show is that this actual"}, {"start": 1220.6399999999999, "end": 1224.8, "text": " template of having this patch embedding, and then having this, how they call it,"}, {"start": 1224.8, "end": 1229.04, "text": " isotropic model, basically the thing is you're not reducing"}, {"start": 1229.04, "end": 1233.9199999999998, "text": " neither the number of channels nor the spatial extent, whereas in the"}, {"start": 1233.9199999999998, "end": 1237.68, "text": " CNNs, for example, you're basically usually, you're"}, {"start": 1237.68, "end": 1240.8, "text": " reducing the spatial extent and you're adding the number of channels as you're"}, {"start": 1240.8, "end": 1245.36, "text": " going deeper into the network. So these isotropic models basically"}, {"start": 1245.36, "end": 1248.56, "text": " preserve both the number of channels as well as the spatial extent,"}, {"start": 1248.56, "end": 1253.68, "text": " and they argue that maybe, just maybe, that template of using patch embeddings"}, {"start": 1253.68, "end": 1259.3600000000001, "text": " plus that isotropic, like, modeling of features is the thing that"}, {"start": 1259.3600000000001, "end": 1263.04, "text": " actually gives us performance, not the self-attention, not the"}, {"start": 1263.04, "end": 1267.28, "text": " other fancy things we've imported from NLP world. In any case, the model is super"}, {"start": 1267.28, "end": 1269.3600000000001, "text": " simple, it's a very nice baseline, you can see"}, {"start": 1269.3600000000001, "end": 1274.24, "text": " here, this is the whole implementation, as you can see, so they have this"}, {"start": 1274.24, "end": 1278.64, "text": " initial layer here, is basically just the"}, {"start": 1278.64, "end": 1282.0800000000002, "text": " patch embedding, then they have a couple of these"}, {"start": 1282.08, "end": 1287.28, "text": " depth, so N of these layers, and finally they add the adaptive"}, {"start": 1287.28, "end": 1291.6799999999998, "text": " average pooling and they flatten out the features and they add a"}, {"start": 1291.6799999999998, "end": 1295.84, "text": " linear classifier on top of it. So it's a fairly simple model and it achieves"}, {"start": 1295.84, "end": 1299.9199999999998, "text": " comparable, if not better, results compared to VAT, compared to ResNet"}, {"start": 1299.9199999999998, "end": 1304.6399999999999, "text": " and ResMLP. So now the thing here, the red alert is the throughput is super, super"}, {"start": 1304.6399999999999, "end": 1307.52, "text": " small, they mentioned that it's possible to,"}, {"start": 1307.52, "end": 1311.84, "text": " in future research, improve this. I mean, this ties back to the thing I mentioned,"}, {"start": 1311.84, "end": 1316.48, "text": " in this plot here, when you just plot a single dimension,"}, {"start": 1316.48, "end": 1320.08, "text": " like number of parameters here, you may paint up like a wrong picture."}, {"start": 1320.08, "end": 1323.6, "text": " But in any case, this paper is not trying to set SOTA,"}, {"start": 1323.6, "end": 1327.6799999999998, "text": " they're just trying to say that we are maybe focusing on the wrong thing, I"}, {"start": 1327.6799999999998, "end": 1330.56, "text": " guess. So we went all the way from ResNet and"}, {"start": 1330.56, "end": 1333.04, "text": " convolutional neural networks to transformers"}, {"start": 1333.04, "end": 1336.48, "text": " and vision transformers to finally get back to convolutional"}, {"start": 1336.48, "end": 1340.8799999999999, "text": " neural networks, again, in their pure form, just changing this, just keeping this"}, {"start": 1340.88, "end": 1346.0, "text": " isotropic template. Also, I like the end part of this"}, {"start": 1346.0, "end": 1349.68, "text": " paper, a note on paper length, expecting more text in this paper,"}, {"start": 1349.68, "end": 1354.48, "text": " wondering if it's a workshop paper, we hastily submitted to iClear. No, this"}, {"start": 1354.48, "end": 1358.0800000000002, "text": " paper presents a simple idea, one where we genuinely believe that a short paper"}, {"start": 1358.0800000000002, "end": 1362.64, "text": " presentation is more effective. Do we really need exactly eight, now nine"}, {"start": 1362.64, "end": 1366.88, "text": " or ten pages to describe every machine learning architecture and algorithm in"}, {"start": 1366.88, "end": 1369.44, "text": " existence? We proposed an incredibly simple"}, {"start": 1369.44, "end": 1373.3600000000001, "text": " architecture and made a very simple point that we think is worth more"}, {"start": 1373.3600000000001, "end": 1376.24, "text": " discussion. Patches work well in"}, {"start": 1376.24, "end": 1379.68, "text": " convolutional architectures. We think that four pages is more than enough"}, {"start": 1379.68, "end": 1382.72, "text": " space for this. The details of the experiments and"}, {"start": 1382.72, "end": 1386.48, "text": " architectures are in the appendix for those who want to read through it all."}, {"start": 1386.48, "end": 1389.76, "text": " A lot of respect to the authors, to be honest."}, {"start": 1389.76, "end": 1392.88, "text": " Aside from this, I really think we need to question this"}, {"start": 1392.88, "end": 1396.88, "text": " whole template of how we are presenting the research."}, {"start": 1396.88, "end": 1401.2, "text": " I think that PDFs, for one, are obsolete. I think that authors need to write"}, {"start": 1401.2, "end": 1405.1200000000001, "text": " like accompanying blogs a lot more. We need something like interactive"}, {"start": 1405.1200000000001, "end": 1410.72, "text": " and types of notebooks, such as Jupyter or Deep Note is also one cool notebook"}, {"start": 1410.72, "end": 1414.48, "text": " environment. So, I think we need as a community, we"}, {"start": 1414.48, "end": 1418.24, "text": " need to rethink how we are presenting, because the whole idea of this paper"}, {"start": 1418.24, "end": 1421.7600000000002, "text": " is for other researchers to understand what these guys have done"}, {"start": 1421.7600000000002, "end": 1425.2800000000002, "text": " and be able to reconstruct or build upon their ideas."}, {"start": 1425.28, "end": 1429.68, "text": " And so, who cares about like 10-pager format? Paper reading in and off itself"}, {"start": 1429.68, "end": 1433.76, "text": " is hard and I guess any help we can get from"}, {"start": 1433.76, "end": 1436.32, "text": " the technology advancements we have, such as"}, {"start": 1436.32, "end": 1440.6399999999999, "text": " interactivity, visualizations, animations, will help us"}, {"start": 1440.6399999999999, "end": 1443.36, "text": " as a community. Finally, let me show you the"}, {"start": 1443.36, "end": 1448.72, "text": " one tweet implementation. So, here it is."}, {"start": 1448.72, "end": 1453.52, "text": " An implementation of our model in exactly 280 characters"}, {"start": 1453.52, "end": 1457.52, "text": " is right here. Twitter is slowly becoming a more important medium,"}, {"start": 1457.52, "end": 1461.92, "text": " if not already, compared to conferences."}, {"start": 1461.92, "end": 1465.6, "text": " You can check out other results in the paper. They have some nice"}, {"start": 1465.6, "end": 1470.0, "text": " visualizations of the kernels."}, {"start": 1470.0, "end": 1473.84, "text": " I'll leave you with that and if you like this video, consider subscribing,"}, {"start": 1473.84, "end": 1478.16, "text": " share out the video, hit that bell icon to get notified every time I upload a new"}, {"start": 1478.16, "end": 1480.72, "text": " video, also join the Discord community. We've"}, {"start": 1480.72, "end": 1486.32, "text": " got a lot of smart people there and until next time, bye-bye!"}]