When Vision Transformers Outperform ResNets without Pretraining | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentation paper explained.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2106.01548
✅ LinkedIn post: https://www.linkedin.com/posts/aleksagordic_vision-transformers-mlp-activity-6807372257187442688-7jzF
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Key points of the paper
01:37 Key conclusions
03:00 Inductive biases and biases in a CNN
07:00 SAM explained
11:30 Possibility of heavy pruning, overfitting, sparsity, etc.
14:20 Neural tangent kernel and steepness of curvature
17:30 Results, empirical correlation between SAM and biases
19:00 Deeper look into the Hessians
20:50 Attention visualized, low data regime plots

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Petar Veličković
Zvonimir Sabljic
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#visiontransformer #nopretraining #resnets
 What's up? In this video I'm covering this new paper called When Vision Transformers Outperform Resonance Without Pre-Training or Strong Data Augmentations by the Google research team Chen, Xie and Gong. And the main idea of this paper, the main innovation was actually something that was developed I think half a year ago called Sharpness Aware Minimization Objective and what it does it smoothens out the lost landscape. Let me show you the picture here. So basically we have their testing vision transformers, their testing recently published MLP mixer and you can see that using the same here we get a much smoother like a point of convergence in that lot landscape and what that brings us is higher generalization capabilities that was previously shown that basically converging to flat areas of the lost landscape guarantees better generalization properties and also it shows that we don't need a bunch of pre-training data as before so basically remember and do check out my videos on vision transformer and mixer MLPs if you haven't so if you haven't heard of it so far. Basically they used to train to pre-train VATs on this huge data set called JFT 300m which is a proprietary data set containing 300 million images from Google and now they showed that without that huge pre-training procedure and without extreme data augmentation techniques which were previously advised they achieve on pair or even outperform resonant baselines which is really cool so that's the main bullet point now let me go slowly take you through the walk you through the paper so they say here vision transformers and MLPs signal further efforts on replacing hand-wired features or inductive biases with general-purpose neural architecture so we've seen that trend over the last year basically transformers going from an NLP coming to the computer vision field and basically being having less biases than covnets as well as recently published mixer MLPs which basically just use multilayer perceptions to do token-wise and channel-wise mixing. This paper investigates VATs and NLP mixers from the lens of loss geometry as I already explained that and intending to improve the models data efficiency as training and generalization at inference okay by promoting smoothness with a recently proposed sharpness aware optimizer so the same thing we substantially improve the accuracy and robustness of VATs and MLP mixers on various tasks okay and we show that the improved smoothness attributes to sparser active neurons in the first few layers and the result in VATs outperform resonance of similar size and throughput when trained from scratch on image add without large pre training or strong data augmentations okay next thing that's interesting is this so despite the appealing potential of moving toward general-purpose neural architectures the lack of convolution like inductive bias also challenges the training of vision transformers and MLPs when trained on image net with a conventional inception style data processing which is basically just random crops and random flips super simple I think that was used back in 2014 with the inception when the inception architecture first came out hence inception style just want to yield modest modest accuracies of a few percentage points below resonance of comparable size so what they say here is that if you don't use Sam and you just train your VATs and MLP mixers with the same procedure as resonance we have like even lower performance stand than those resonant baselines so the reason is comments and let me just recap shortly here comments have really useful priors built into them and there is this whole thing going on currently in the deep learning field where we are pretty much going forward with this blank slate paradigm and if you take a look at humans we have a bunch of evolutionary built-in priors and so like my deep belief is that like we will need to to finally once we find really good set of priors we'll need to start using them because it's super expensive to you to be rediscovering them all the time but like I still think this research direction is is very useful and we are just kind of trying to see like what can we learn with as least priors as possible so yeah I still think it's a cool idea but eventually I do believe priors are going to be really important and let me show you so basically for a CNN if you have an image here and what CNN do and exploit is the the fact that the the local pixel neighborhood is highly correlated in natural images so if you have a pixel here it's highly likely that all of these pixels in the neighborhood are going to be a bit less like they'll have a bit less intensity or a bit higher intensity but they won't be we won't have dramatic changes so that that's what I mean when I say core like the neighborhood the pixel intensity is correlate in this neighborhood so what CNN's have is so you hopefully know what how CNN's work you have a kernel and they have at least three biases I can think of are the first one is locality so basically you're assuming that this filter here should be attending only to the local neighborhood of the pixel the second one is weight sharing which basically means yeah you the things you learn here the filter you learn here will be useful not only here but also here and here and all around the image and this leads to something called translational active variance which is super useful property that CNN's have and the third bias I can think of is the hierarchy so basically you know that CNN's if when you go in like into deeper layers the spatial extent diminishes and the volumes get more and more channels and what that practically means is that if you have an image here and I'm just looking from it sideways here if you have a neuron here and a neuron here so this one is only going to attend depending on the kernel size maybe three pixels vertically here whereas the deeper one will pretty much attend over the whole image and so that means that we progressively keep expanding the neighborhood of the pixels which is basically something called a relational inductive bias so yeah that's that CNN's and MLPs just have much less priors and that's why as you can see here they are harder to train prior to this same thing they were like under performing or as in the baselines okay so let's continue here and let me explain what Sam is so the first order optimizers like SGD stochastic gradient and Adam only seek the model parameters that minimize the training error they dismiss the higher order information such as flatness that correlates with the generalization so first order because we are basically just estimating the gradient which is the like basically the first order derivative of the loss with respect to the loss and of the of the model parameters there are also second-order methods like LBFGS is one of the famous ones where we also take into account the second order partial derivatives so Hessians etc but here what they actually advise is something a bit different and we'll see that in a moment so Sam strives to find a solution whose entire neighborhood has low losses rather than focus on any singleton point so let me go to the drawing here again and try to explain what this means so what I've did here in order to visualize the loss function because the models have millions of parameters they've kind of projected the the whole like the the weights into 2d space and now they can plot the loss and so what what what the whole point of the same thing is is if you take a point here in the parameter space and you take a disk around that point so in the 2d space it's going to be a disk in a 3d space it's going to be a sphere in an n dimensional space is going to be an n dimensional hyperbola and what they want to achieve is the following so maybe this is a better example here what they want to achieve is the following so they want to make sure that inside of this disk or hyperbola when we're in the original parameter space we want to make sure that the max loss so if I if I take a point on the disk here and they project it upwards here we'll see that the loss here is really really high even though the loss here is really low the max loss inside the volume of interest disk in this particular case is high and they want to make sure that that max loss is actually minimized and as you can see this is what we get so going from from this thing we get something much smoother and you can see here that if I take a disk of the same size the max loss is still really low lower than this one here so that's the whole point that's the geometric explanation now let me just kind of explain you the formula which tells the same thing and here it is but briefly before that so from the original paper so motivated by the connection between sharpness of the lost landscape and generalization so that's the the key point here somebody the prior work showed that if we converge to a flat part of the lost landscape we're gonna have much better generalization capabilities so we propose a different approach rather than seeking our parameter values W that simply have low training loss value we seek out parameter values whose entire neighborhoods have uniformly low training loss value equivalently neighborhoods having both low loss and low curvature and or from the original paper again here so here is the where they converge with resonance in the original formulation of the optimizer like using Adam and here is where they get using Sam and as you can see again it's much flatter much smoother and that's the whole point okay and here's the formula so it's really easy basically what I see here is for these epsilons so this is called a perturbation vector let's go that way so we want to make sure that the L2 norm is inside this threshold so that just just a mathematical formulation of what they already explained with discs and hyperboles so you want to make sure that W when you when you add this vector over the whole value around this W point we want to make sure that the max loss so max loss is minimized with respect to W so that's the whole point so you're basically you have your W let's let's assume we are in a 2d space here so let's assume we are in a 2d space even though we're going to be like a multi-dimensional space so this is W2 this is W1 let's say we've converged to this point here so what it says here is that so epsilon is just some vector and we are going to trace out so so this is a set of all vectors which are whose L2 norm is inside of row so that's gonna be into the case just a disc so we want to make sure that inside that region so the the max loss the max loss inside that region needs to be minimal and that's what we are minimizing so hopefully yeah that was quite in-depth hopefully you understood it let me get back to the beginning here a couple of things I want to mention so a side observation is that unlike resonance and MLP mixers VATs have extremely sparse active neurons less than 5% for most layers revealing the redundancy of input image patches and the capacity for network pruning and this looks like an interesting follow-up work from this paper basically it seems that we can reduce the memory footprint of VATs heavily and still hopefully contain like keep the performance we achieved here without using the extensive pre-training etc okay and I just kind of highlight the perspicuous here because I really am a strong believer that we should be using like we should make paper papers as clear as possible and not use fancy terms fancy equations if there is no need to do that I literally had to you to to Google this word and it says clearly expressed and easily understood well this sentence is not perspicuous that's what I know cool I'll rant over okay let me walk you through a couple of bullet points here and again they are focusing only on vision transformers and MLP mixers so do watch those videos if you haven't already here are a couple of points they make it's been extensively studied that the convergence to a flat region whose curvature is small benefits generalization of neural networks and I repeated it as multiple times I think this is a really important key point to keep in mind although mixer has fewer parameters than VATs it has smaller training error but much worse test accuracy so that basically means mixers tend to overfit much more and that makes sense because they have less priors in built into them so here are some nice curves that explain that so taking a look at maybe mixer B 16 here you can see the training curve is really low but the test accuracy is not that high on the other hand if we take something like this VAT B 16 the training curves are higher as you can see here so that this this dim orange curve here is higher but also the the test accuracy is higher so as you can see they overfit much much less than than MLP mixers here they just compare how it looks like without Sam and with Sam and again the training curve with Sam is higher and the test accuracy is also higher so that's cool that means we overfit much less to the training data and we generalize much better to the test distribution okay and finally here just the the sparsity constraint they they they noticed that basically in the lower layers using Sam the the number of activated neurons that's the y-axis gets much much lower for for for mixers than them without Sam here a couple of more bullet points here so Xiao et al. showed that the trainability of a neural network can be characterized by the condition number of the associated neural tangent kernel and I won't get into the details of the of the of the kernel but basically it's a it's a simple proxy for trainability so case pretty stable for Resnets echoing previous results the Resnets enjoy superior trainability regardless of the depth however so if you remember like back in 2015 when the paper came out Resnets from Microsoft research they showed for the first time that we can train models from 18 all the way to 151 layers and it just works because of the because of the skip connections or residual connections however you want to call them however we observed that the condition number diverges when it comes to VAT and MLP mixer confirming that the training of VATs desires extra care so they kind of quantify this not kind of they quantify this in this table and you can see it here so here is the NTK here we have Resnets and VATs and mixers so you can see that the NTK for Resnets is pretty much the same so I'm not sure where those numbers whether there is some maybe bug here but like I'd assume that a Resonant 152 should have a bit higher NTK although I'm not sure about exact details but okay basically what I want to show you here is that mixer has much higher NTK which means it's much harder to train the second thing they plot here and I'll go into a bit more detail a bit later but basically Hessians are again a proxy for the curvature of your loss landscape at the point of convergence so what they calculate here as you can see so you can see that the lower is better the lower means it's more flat and you can see that VATs have much higher than than Resonants and mixers have even higher Hessian so this is the just the this is just the max eigenvalue of the eigenvector associated with this Hessian so I don't want to confuse you here but it's just a proxy for the curvature and you can see that after applying SAM it just goes like it falls drops down all the way to 20 something so that's even lower than for Resonant so that's really cool aside from that they they show that the performance is really really great so after applying SAM on ImageNet the accuracy just kind of increases and as well as on ImageNet C which tests the robustness of the model how is that well because if you take a look at the ImageNet C data set you can see that it just has a bunch of different augmentations like Gaussian noise different kinds of noises here in impulse noise blurring like they have motion blurring some special effects and some photometric augmentations here like brightness contrast etc so basically you just want to make sure that you're generalizing to these small shifts in the distribution of your data set and yeah they showed that actually VATs perform even better than comparably sized Resonants okay let's see what else is interesting in this paper and I think I've covered everything pretty much a bunch of results here and summarized in a couple of sentences here so on the ImageNet validation set SAM boosts the top one accuracy of VATs from something to something so basically increase of 5% here and for mixture as well and empirically so this is interesting empirically the degree of improvement negatively correlates with the level of inductive biases built into the architecture so what you're seeing here is the following so let's plot a 2d chart here and on the x-axis let's say we have a bias and on the so the bias ingrained into the architecture itself and on the y-axis we have some like improvement okay and what I say here is we have a negative correlation like something like this some point cloud here and basically CNNs are here so here is a CNN MLP mixtures are probably here and VATs are somewhere here I don't know I'm just qualitatively drawing this and what they say is that the more priors we have in the architecture the less improvement we get from using SAM for that particular architecture so that's something they they empirically found so there is no like theoretical explanation for why that is but it is okay again it's more robust as well not just more accurate but more robust like looking at the image net see that is said nothing interesting there I already mentioned that and this is an interesting table basically they just kind of decompose the Hessians already mentioned which are a proxy for the curvature and you can see looking at the layers like let's focus on VAT or even better like on mixer looking at the lower layer like in the embedding layer itself we have a huge Hessian so the huge eigenvalue of the Hessian and going to deep players block one block six block twelve you can see that the Hessians go down so basically these lower layers contribute to the huge like to the steep curvature of the lost landscape and that's what I've fixed here as you can see using SAM it drops significantly and basically that correlates with the the fact that we have much sparser activations now in those lower layers as we saw on the plot up there so that's this plot here basically you can see that we have much sparser activations so this is the x-axis is the depth of the network so we now see that we have much sparse activations in the shallower layers of the network that correlates with this finding here one thing they notice as well is that the L2 norm of the weight vector increases which may indicate that they've used the weight decay regularization and it seems it's not helping so they need to further investigate that the reason I highlight the recursively here is that basically you can see H of K depends on the H of K plus 1 so then that's the reason why we have higher eigenvalues so this Hessians in the lower layer because it just multiplies with all of the previous deeper layers and it just kind of accumulates and blows up in the shallower layers okay additionally what they found is that the attention maps found by VATs and MLP mixers after the SAM procedure has much better discriminative features you can see that the attention maps do focus on something that's salient in these images much better than before using SAM so yeah just a fun fact and I mean visualizations are super important so kudos to them for doing this a couple of fun results here basically and it's pretty obvious what they show here is that even with using when you're using SAM when you go to when you start reducing the number of data points in your data set so what they did here is they randomly sampled a half of the pictures from the ImageNet 1k and here 1 fourth of the images are randomly sampled and we can see that VAT focusing on the orange curve you can see that the VATs and mixers degrade much more severely when we go into the low data regime here whereas thanks to the biases already mentioned CNN's ResNets here in particular managed to keep up that performance even in the low data regime so again priors are super important and nonetheless I do think that this kind of research where we are just doing this blank slate paradigm is going to be very interesting and informative over the long long run so yeah they also tried some contrastive learning and it kind of improves a bit upon the SAM as well they tried adversarial training as well I won't be focusing on that and they like using this PGD 10 attack basically PGD and I think they just average over 10 attacks and they showed that they get a nice like nice performance boost there as well yeah hopefully that's it hopefully that was informative and useful if you found useful consider subscribing sharing this video and see you next time bye bye
[{"start": 0.0, "end": 3.56, "text": " What's up? In this video I'm covering this new paper called When Vision"}, {"start": 3.56, "end": 7.32, "text": " Transformers Outperform Resonance Without Pre-Training or Strong Data"}, {"start": 7.32, "end": 14.84, "text": " Augmentations by the Google research team Chen, Xie and Gong. And the main idea of"}, {"start": 14.84, "end": 18.36, "text": " this paper, the main innovation was actually something that was developed I"}, {"start": 18.36, "end": 23.68, "text": " think half a year ago called Sharpness Aware Minimization Objective and what it"}, {"start": 23.68, "end": 27.68, "text": " does it smoothens out the lost landscape. Let me show you the picture here. So"}, {"start": 27.68, "end": 31.92, "text": " basically we have their testing vision transformers, their testing recently"}, {"start": 31.92, "end": 39.04, "text": " published MLP mixer and you can see that using the same here we get a much"}, {"start": 39.04, "end": 45.2, "text": " smoother like a point of convergence in that lot landscape and what that brings"}, {"start": 45.2, "end": 50.32, "text": " us is higher generalization capabilities that was previously shown that basically"}, {"start": 50.32, "end": 55.480000000000004, "text": " converging to flat areas of the lost landscape guarantees better"}, {"start": 55.48, "end": 60.519999999999996, "text": " generalization properties and also it shows that we don't need a bunch of"}, {"start": 60.519999999999996, "end": 65.16, "text": " pre-training data as before so basically remember and do check out my videos on"}, {"start": 65.16, "end": 69.88, "text": " vision transformer and mixer MLPs if you haven't so if you haven't heard of it so"}, {"start": 69.88, "end": 76.19999999999999, "text": " far. Basically they used to train to pre-train VATs on this huge data set"}, {"start": 76.19999999999999, "end": 81.52, "text": " called JFT 300m which is a proprietary data set containing 300 million images"}, {"start": 81.52, "end": 87.52, "text": " from Google and now they showed that without that huge pre-training procedure"}, {"start": 87.52, "end": 91.28, "text": " and without extreme data augmentation techniques which were previously advised"}, {"start": 91.28, "end": 96.6, "text": " they achieve on pair or even outperform resonant baselines which is really cool"}, {"start": 96.6, "end": 100.96, "text": " so that's the main bullet point now let me go slowly take you through the walk"}, {"start": 100.96, "end": 106.12, "text": " you through the paper so they say here vision transformers and MLPs signal"}, {"start": 106.12, "end": 111.0, "text": " further efforts on replacing hand-wired features or inductive biases with"}, {"start": 111.0, "end": 115.0, "text": " general-purpose neural architecture so we've seen that trend over the last year"}, {"start": 115.0, "end": 120.12, "text": " basically transformers going from an NLP coming to the computer vision field and"}, {"start": 120.12, "end": 126.08, "text": " basically being having less biases than covnets as well as recently published"}, {"start": 126.08, "end": 132.68, "text": " mixer MLPs which basically just use multilayer perceptions to do token-wise"}, {"start": 132.68, "end": 137.72, "text": " and channel-wise mixing. This paper investigates VATs and NLP mixers from"}, {"start": 137.72, "end": 141.96, "text": " the lens of loss geometry as I already explained that and intending to improve"}, {"start": 141.96, "end": 147.72, "text": " the models data efficiency as training and generalization at inference okay by"}, {"start": 147.72, "end": 152.24, "text": " promoting smoothness with a recently proposed sharpness aware optimizer so the"}, {"start": 152.24, "end": 158.96, "text": " same thing we substantially improve the accuracy and robustness of VATs and"}, {"start": 158.96, "end": 163.44, "text": " MLP mixers on various tasks okay and we show that the improved smoothness"}, {"start": 163.44, "end": 168.72, "text": " attributes to sparser active neurons in the first few layers and the result in"}, {"start": 168.72, "end": 172.2, "text": " VATs outperform resonance of similar size and throughput when trained from"}, {"start": 172.2, "end": 177.32, "text": " scratch on image add without large pre training or strong data augmentations"}, {"start": 177.32, "end": 183.0, "text": " okay next thing that's interesting is this so despite the appealing potential"}, {"start": 183.0, "end": 186.4, "text": " of moving toward general-purpose neural architectures the lack of convolution"}, {"start": 186.4, "end": 190.2, "text": " like inductive bias also challenges the training of vision transformers and"}, {"start": 190.2, "end": 195.28, "text": " MLPs when trained on image net with a conventional inception style data"}, {"start": 195.28, "end": 200.44, "text": " processing which is basically just random crops and random flips super"}, {"start": 200.44, "end": 203.64, "text": " simple I think that was used back in 2014 with the inception when the"}, {"start": 203.64, "end": 208.28, "text": " inception architecture first came out hence inception style just want to"}, {"start": 208.28, "end": 213.23999999999998, "text": " yield modest modest accuracies of a few percentage points below resonance of"}, {"start": 213.23999999999998, "end": 219.12, "text": " comparable size so what they say here is that if you don't use Sam and you just"}, {"start": 219.12, "end": 227.04, "text": " train your VATs and MLP mixers with the same procedure as resonance we have like"}, {"start": 227.04, "end": 232.04, "text": " even lower performance stand than those resonant baselines so the reason is"}, {"start": 232.04, "end": 238.0, "text": " comments and let me just recap shortly here comments have really useful priors"}, {"start": 238.0, "end": 241.84, "text": " built into them and there is this whole thing going on currently in the deep"}, {"start": 241.84, "end": 246.88, "text": " learning field where we are pretty much going forward with this blank slate"}, {"start": 246.88, "end": 251.96, "text": " paradigm and if you take a look at humans we have a bunch of evolutionary"}, {"start": 251.96, "end": 258.44, "text": " built-in priors and so like my deep belief is that like we will need to to"}, {"start": 258.44, "end": 262.84, "text": " finally once we find really good set of priors we'll need to start using them"}, {"start": 262.84, "end": 266.96, "text": " because it's super expensive to you to be rediscovering them all the time but"}, {"start": 266.96, "end": 272.0, "text": " like I still think this research direction is is very useful and we are"}, {"start": 272.0, "end": 277.04, "text": " just kind of trying to see like what can we learn with as least priors as"}, {"start": 277.04, "end": 281.28, "text": " possible so yeah I still think it's a cool idea but eventually I do believe"}, {"start": 281.28, "end": 285.48, "text": " priors are going to be really important and let me show you so basically for a"}, {"start": 285.48, "end": 291.64, "text": " CNN if you have an image here and what CNN do and exploit is the the fact that"}, {"start": 291.64, "end": 296.2, "text": " the the local pixel neighborhood is highly correlated in natural images so"}, {"start": 296.2, "end": 300.6, "text": " if you have a pixel here it's highly likely that all of these pixels in the"}, {"start": 300.6, "end": 306.36, "text": " neighborhood are going to be a bit less like they'll have a bit less intensity"}, {"start": 306.36, "end": 310.44, "text": " or a bit higher intensity but they won't be we won't have dramatic changes so"}, {"start": 310.44, "end": 314.68, "text": " that that's what I mean when I say core like the neighborhood the pixel"}, {"start": 314.68, "end": 319.64000000000004, "text": " intensity is correlate in this neighborhood so what CNN's have is so"}, {"start": 319.64000000000004, "end": 323.76000000000005, "text": " you hopefully know what how CNN's work you have a kernel and they have at"}, {"start": 323.76000000000005, "end": 328.64000000000004, "text": " least three biases I can think of are the first one is locality so basically"}, {"start": 328.64, "end": 334.0, "text": " you're assuming that this filter here should be attending only to the local"}, {"start": 334.0, "end": 337.84, "text": " neighborhood of the pixel the second one is weight sharing which basically"}, {"start": 337.84, "end": 342.32, "text": " means yeah you the things you learn here the filter you learn here will be useful"}, {"start": 342.32, "end": 348.03999999999996, "text": " not only here but also here and here and all around the image and this leads to"}, {"start": 348.03999999999996, "end": 351.8, "text": " something called translational active variance which is super useful property"}, {"start": 351.8, "end": 356.96, "text": " that CNN's have and the third bias I can think of is the hierarchy so basically"}, {"start": 356.96, "end": 363.4, "text": " you know that CNN's if when you go in like into deeper layers the spatial"}, {"start": 363.4, "end": 368.23999999999995, "text": " extent diminishes and the volumes get more and more channels and what that"}, {"start": 368.23999999999995, "end": 371.76, "text": " practically means is that if you have an image here and I'm just looking from it"}, {"start": 371.76, "end": 377.76, "text": " sideways here if you have a neuron here and a neuron here so this one is only"}, {"start": 377.76, "end": 383.24, "text": " going to attend depending on the kernel size maybe three pixels vertically here"}, {"start": 383.24, "end": 389.68, "text": " whereas the deeper one will pretty much attend over the whole image and so that"}, {"start": 389.68, "end": 394.8, "text": " means that we progressively keep expanding the neighborhood of the pixels"}, {"start": 394.8, "end": 400.32, "text": " which is basically something called a relational inductive bias so yeah that's"}, {"start": 400.32, "end": 406.52, "text": " that CNN's and MLPs just have much less priors and that's why as you can see"}, {"start": 406.52, "end": 413.04, "text": " here they are harder to train prior to this same thing they were like under"}, {"start": 413.04, "end": 418.68, "text": " performing or as in the baselines okay so let's continue here and let me explain"}, {"start": 418.68, "end": 422.88, "text": " what Sam is so the first order optimizers like SGD stochastic gradient"}, {"start": 422.88, "end": 428.04, "text": " and Adam only seek the model parameters that minimize the training error they"}, {"start": 428.04, "end": 432.0, "text": " dismiss the higher order information such as flatness that correlates with"}, {"start": 432.0, "end": 437.28000000000003, "text": " the generalization so first order because we are basically just estimating"}, {"start": 437.28000000000003, "end": 441.32000000000005, "text": " the gradient which is the like basically the first order derivative of the loss"}, {"start": 441.32, "end": 445.59999999999997, "text": " with respect to the loss and of the of the model parameters there are also"}, {"start": 445.59999999999997, "end": 450.2, "text": " second-order methods like LBFGS is one of the famous ones where we also take"}, {"start": 450.2, "end": 456.0, "text": " into account the second order partial derivatives so Hessians etc but here"}, {"start": 456.0, "end": 459.48, "text": " what they actually advise is something a bit different and we'll see that in a"}, {"start": 459.48, "end": 464.08, "text": " moment so Sam strives to find a solution whose entire neighborhood has low losses"}, {"start": 464.08, "end": 470.52, "text": " rather than focus on any singleton point so let me go to the drawing here again"}, {"start": 470.52, "end": 476.44, "text": " and try to explain what this means so what I've did here in order to"}, {"start": 476.44, "end": 479.91999999999996, "text": " visualize the loss function because the models have millions of parameters"}, {"start": 479.91999999999996, "end": 485.15999999999997, "text": " they've kind of projected the the whole like the the weights into 2d space and"}, {"start": 485.15999999999997, "end": 490.44, "text": " now they can plot the loss and so what what what the whole point of the same"}, {"start": 490.44, "end": 495.59999999999997, "text": " thing is is if you take a point here in the parameter space and you take a disk"}, {"start": 495.59999999999997, "end": 499.79999999999995, "text": " around that point so in the 2d space it's going to be a disk in a 3d space"}, {"start": 499.8, "end": 503.28000000000003, "text": " it's going to be a sphere in an n dimensional space is going to be an n"}, {"start": 503.28000000000003, "end": 508.64, "text": " dimensional hyperbola and what they want to achieve is the following so maybe"}, {"start": 508.64, "end": 512.2, "text": " this is a better example here what they want to achieve is the following so they"}, {"start": 512.2, "end": 516.12, "text": " want to make sure that inside of this disk or hyperbola when we're in the"}, {"start": 516.12, "end": 521.12, "text": " original parameter space we want to make sure that the max loss so if I if I take"}, {"start": 521.12, "end": 525.2, "text": " a point on the disk here and they project it upwards here we'll see that"}, {"start": 525.2, "end": 529.12, "text": " the loss here is really really high even though the loss here is really low the"}, {"start": 529.12, "end": 533.92, "text": " max loss inside the volume of interest disk in this particular case is high"}, {"start": 533.92, "end": 539.76, "text": " and they want to make sure that that max loss is actually minimized and as you"}, {"start": 539.76, "end": 544.08, "text": " can see this is what we get so going from from this thing we get something"}, {"start": 544.08, "end": 548.64, "text": " much smoother and you can see here that if I take a disk of the same size the"}, {"start": 548.64, "end": 553.6, "text": " max loss is still really low lower than this one here so that's the whole point"}, {"start": 553.6, "end": 557.8, "text": " that's the geometric explanation now let me just kind of explain you the formula"}, {"start": 557.8, "end": 564.16, "text": " which tells the same thing and here it is but briefly before that so from the"}, {"start": 564.16, "end": 567.4, "text": " original paper so motivated by the connection between sharpness of the lost"}, {"start": 567.4, "end": 571.74, "text": " landscape and generalization so that's the the key point here somebody the"}, {"start": 571.74, "end": 576.5999999999999, "text": " prior work showed that if we converge to a flat part of the lost landscape we're"}, {"start": 576.5999999999999, "end": 580.64, "text": " gonna have much better generalization capabilities so we propose a different"}, {"start": 580.64, "end": 585.12, "text": " approach rather than seeking our parameter values W that simply have low"}, {"start": 585.12, "end": 588.92, "text": " training loss value we seek out parameter values whose entire"}, {"start": 588.92, "end": 593.84, "text": " neighborhoods have uniformly low training loss value equivalently"}, {"start": 593.84, "end": 598.52, "text": " neighborhoods having both low loss and low curvature and or from the original"}, {"start": 598.52, "end": 603.36, "text": " paper again here so here is the where they converge with resonance in the"}, {"start": 603.36, "end": 607.88, "text": " original formulation of the optimizer like using Adam and here is where they"}, {"start": 607.88, "end": 612.36, "text": " get using Sam and as you can see again it's much flatter much smoother and"}, {"start": 612.36, "end": 616.16, "text": " that's the whole point okay and here's the formula so it's really easy"}, {"start": 616.16, "end": 622.6800000000001, "text": " basically what I see here is for these epsilons so this is called a perturbation"}, {"start": 622.6800000000001, "end": 628.48, "text": " vector let's go that way so we want to make sure that the L2 norm is inside"}, {"start": 628.48, "end": 633.16, "text": " this threshold so that just just a mathematical formulation of what they"}, {"start": 633.16, "end": 636.32, "text": " already explained with discs and hyperboles so you want to make sure that"}, {"start": 636.32, "end": 643.48, "text": " W when you when you add this vector over the whole value around this W point we"}, {"start": 643.48, "end": 649.2, "text": " want to make sure that the max loss so max loss is minimized with respect to W"}, {"start": 649.2, "end": 653.08, "text": " so that's the whole point so you're basically you have your W let's let's"}, {"start": 653.08, "end": 658.12, "text": " assume we are in a 2d space here so let's assume we are in a 2d space even"}, {"start": 658.12, "end": 661.5200000000001, "text": " though we're going to be like a multi-dimensional space so this is W2"}, {"start": 661.52, "end": 667.04, "text": " this is W1 let's say we've converged to this point here so what it says here is"}, {"start": 667.04, "end": 672.64, "text": " that so epsilon is just some vector and we are going to trace out so so this is"}, {"start": 672.64, "end": 678.56, "text": " a set of all vectors which are whose L2 norm is inside of row so that's gonna be"}, {"start": 678.56, "end": 683.6, "text": " into the case just a disc so we want to make sure that inside that region so the"}, {"start": 683.6, "end": 688.36, "text": " the max loss the max loss inside that region needs to be minimal and that's"}, {"start": 688.36, "end": 692.84, "text": " what we are minimizing so hopefully yeah that was quite in-depth hopefully you"}, {"start": 692.84, "end": 698.24, "text": " understood it let me get back to the beginning here a couple of things I want"}, {"start": 698.24, "end": 702.0, "text": " to mention so a side observation is that unlike resonance and MLP mixers"}, {"start": 702.0, "end": 707.16, "text": " VATs have extremely sparse active neurons less than 5% for most layers"}, {"start": 707.16, "end": 711.64, "text": " revealing the redundancy of input image patches and the capacity for network"}, {"start": 711.64, "end": 715.2, "text": " pruning and this looks like an interesting follow-up work from this"}, {"start": 715.2, "end": 719.6400000000001, "text": " paper basically it seems that we can reduce the memory footprint of VATs"}, {"start": 719.6400000000001, "end": 725.9200000000001, "text": " heavily and still hopefully contain like keep the performance we achieved here"}, {"start": 725.9200000000001, "end": 731.88, "text": " without using the extensive pre-training etc okay and I just kind of highlight the"}, {"start": 731.88, "end": 736.36, "text": " perspicuous here because I really am a strong believer that we should be using"}, {"start": 736.36, "end": 741.72, "text": " like we should make paper papers as clear as possible and not use fancy"}, {"start": 741.72, "end": 745.4, "text": " terms fancy equations if there is no need to do that I literally had to you"}, {"start": 745.4, "end": 749.28, "text": " to to Google this word and it says clearly expressed and easily understood"}, {"start": 749.28, "end": 755.0, "text": " well this sentence is not perspicuous that's what I know cool I'll rant over"}, {"start": 755.0, "end": 759.72, "text": " okay let me walk you through a couple of bullet points here and again they are"}, {"start": 759.72, "end": 763.1600000000001, "text": " focusing only on vision transformers and MLP mixers so do watch those videos if"}, {"start": 763.1600000000001, "end": 768.88, "text": " you haven't already here are a couple of points they make it's been extensively"}, {"start": 768.88, "end": 772.08, "text": " studied that the convergence to a flat region whose curvature is small"}, {"start": 772.08, "end": 775.96, "text": " benefits generalization of neural networks and I repeated it as multiple"}, {"start": 775.96, "end": 779.52, "text": " times I think this is a really important key point to keep in mind"}, {"start": 779.52, "end": 786.0, "text": " although mixer has fewer parameters than VATs it has smaller training error but"}, {"start": 786.0, "end": 790.24, "text": " much worse test accuracy so that basically means mixers tend to overfit"}, {"start": 790.24, "end": 794.28, "text": " much more and that makes sense because they have less priors in built into them"}, {"start": 794.28, "end": 799.52, "text": " so here are some nice curves that explain that so taking a look at maybe"}, {"start": 799.52, "end": 805.9599999999999, "text": " mixer B 16 here you can see the training curve is really low but the test"}, {"start": 805.9599999999999, "end": 811.68, "text": " accuracy is not that high on the other hand if we take something like this VAT"}, {"start": 811.68, "end": 817.64, "text": " B 16 the training curves are higher as you can see here so that this this dim"}, {"start": 817.64, "end": 824.16, "text": " orange curve here is higher but also the the test accuracy is higher so as you"}, {"start": 824.16, "end": 829.7199999999999, "text": " can see they overfit much much less than than MLP mixers here they just compare"}, {"start": 829.7199999999999, "end": 834.52, "text": " how it looks like without Sam and with Sam and again the training curve with"}, {"start": 834.52, "end": 839.88, "text": " Sam is higher and the test accuracy is also higher so that's cool that means we"}, {"start": 839.88, "end": 843.7199999999999, "text": " overfit much less to the training data and we generalize much better to the"}, {"start": 843.7199999999999, "end": 848.36, "text": " test distribution okay and finally here just the the sparsity constraint they"}, {"start": 848.36, "end": 855.96, "text": " they they noticed that basically in the lower layers using Sam the the number of"}, {"start": 855.96, "end": 861.24, "text": " activated neurons that's the y-axis gets much much lower for for for mixers than"}, {"start": 861.24, "end": 866.5600000000001, "text": " them without Sam here a couple of more bullet points here so Xiao et al."}, {"start": 866.5600000000001, "end": 871.5600000000001, "text": " showed that the trainability of a neural network can be characterized by the"}, {"start": 871.5600000000001, "end": 877.08, "text": " condition number of the associated neural tangent kernel and I won't get"}, {"start": 877.08, "end": 880.2, "text": " into the details of the of the of the kernel but basically it's a it's a"}, {"start": 880.2, "end": 885.08, "text": " simple proxy for trainability so case pretty stable for Resnets echoing"}, {"start": 885.08, "end": 889.0200000000001, "text": " previous results the Resnets enjoy superior trainability regardless of the"}, {"start": 889.0200000000001, "end": 894.12, "text": " depth however so if you remember like back in 2015 when the paper came out"}, {"start": 894.12, "end": 898.6, "text": " Resnets from Microsoft research they showed for the first time that we can"}, {"start": 898.6, "end": 904.2800000000001, "text": " train models from 18 all the way to 151 layers and it just works because of the"}, {"start": 904.28, "end": 908.72, "text": " because of the skip connections or residual connections however you want to"}, {"start": 908.72, "end": 912.88, "text": " call them however we observed that the condition number diverges when it comes"}, {"start": 912.88, "end": 919.24, "text": " to VAT and MLP mixer confirming that the training of VATs desires extra care so"}, {"start": 919.24, "end": 925.16, "text": " they kind of quantify this not kind of they quantify this in this table and you"}, {"start": 925.16, "end": 931.68, "text": " can see it here so here is the NTK here we have Resnets and VATs and mixers so"}, {"start": 931.68, "end": 935.64, "text": " you can see that the NTK for Resnets is pretty much the same so I'm not sure"}, {"start": 935.64, "end": 939.8399999999999, "text": " where those numbers whether there is some maybe bug here but like I'd assume"}, {"start": 939.8399999999999, "end": 944.9599999999999, "text": " that a Resonant 152 should have a bit higher NTK although I'm not sure about"}, {"start": 944.9599999999999, "end": 949.4799999999999, "text": " exact details but okay basically what I want to show you here is that mixer has"}, {"start": 949.4799999999999, "end": 954.66, "text": " much higher NTK which means it's much harder to train the second thing they"}, {"start": 954.66, "end": 959.0, "text": " plot here and I'll go into a bit more detail a bit later but basically Hessians"}, {"start": 959.0, "end": 966.64, "text": " are again a proxy for the curvature of your loss landscape at the point of"}, {"start": 966.64, "end": 971.0, "text": " convergence so what they calculate here as you can see so you can see that the"}, {"start": 971.0, "end": 975.52, "text": " lower is better the lower means it's more flat and you can see that VATs have"}, {"start": 975.52, "end": 980.54, "text": " much higher than than Resonants and mixers have even higher Hessian so this"}, {"start": 980.54, "end": 985.72, "text": " is the just the this is just the max eigenvalue of the eigenvector associated"}, {"start": 985.72, "end": 989.4, "text": " with this Hessian so I don't want to confuse you here but it's just a proxy"}, {"start": 989.4, "end": 995.72, "text": " for the curvature and you can see that after applying SAM it just goes like it"}, {"start": 995.72, "end": 999.52, "text": " falls drops down all the way to 20 something so that's even lower than for"}, {"start": 999.52, "end": 1004.12, "text": " Resonant so that's really cool aside from that they they show that the"}, {"start": 1004.12, "end": 1009.0, "text": " performance is really really great so after applying SAM on ImageNet the"}, {"start": 1009.0, "end": 1014.2, "text": " accuracy just kind of increases and as well as on ImageNet C which tests the"}, {"start": 1014.2, "end": 1019.8000000000001, "text": " robustness of the model how is that well because if you take a look at the"}, {"start": 1019.8000000000001, "end": 1024.04, "text": " ImageNet C data set you can see that it just has a bunch of different"}, {"start": 1024.04, "end": 1027.68, "text": " augmentations like Gaussian noise different kinds of noises here in impulse"}, {"start": 1027.68, "end": 1034.88, "text": " noise blurring like they have motion blurring some special effects and some"}, {"start": 1034.88, "end": 1039.8400000000001, "text": " photometric augmentations here like brightness contrast etc so basically you"}, {"start": 1039.8400000000001, "end": 1043.04, "text": " just want to make sure that you're generalizing to these small shifts in"}, {"start": 1043.04, "end": 1048.92, "text": " the distribution of your data set and yeah they showed that actually VATs"}, {"start": 1048.92, "end": 1053.48, "text": " perform even better than comparably sized Resonants okay let's see what else"}, {"start": 1053.48, "end": 1057.68, "text": " is interesting in this paper and I think I've covered everything pretty much a"}, {"start": 1057.68, "end": 1062.56, "text": " bunch of results here and summarized in a couple of sentences here so on the"}, {"start": 1062.56, "end": 1066.6399999999999, "text": " ImageNet validation set SAM boosts the top one accuracy of VATs from"}, {"start": 1066.6399999999999, "end": 1071.12, "text": " something to something so basically increase of 5% here and for mixture as"}, {"start": 1071.12, "end": 1074.4799999999998, "text": " well and empirically so this is interesting empirically the degree of"}, {"start": 1074.4799999999998, "end": 1079.1999999999998, "text": " improvement negatively correlates with the level of inductive biases built into"}, {"start": 1079.1999999999998, "end": 1083.4399999999998, "text": " the architecture so what you're seeing here is the following so let's plot a"}, {"start": 1083.4399999999998, "end": 1089.6, "text": " 2d chart here and on the x-axis let's say we have a bias and on the so the"}, {"start": 1089.6, "end": 1093.4399999999998, "text": " bias ingrained into the architecture itself and on the y-axis we have some"}, {"start": 1093.4399999999998, "end": 1097.76, "text": " like improvement okay and what I say here is we have a negative correlation"}, {"start": 1097.76, "end": 1103.8, "text": " like something like this some point cloud here and basically CNNs are here"}, {"start": 1103.8, "end": 1110.28, "text": " so here is a CNN MLP mixtures are probably here and VATs are somewhere"}, {"start": 1110.28, "end": 1116.68, "text": " here I don't know I'm just qualitatively drawing this and what they say is that"}, {"start": 1116.68, "end": 1121.08, "text": " the more priors we have in the architecture the less improvement we get"}, {"start": 1121.08, "end": 1125.16, "text": " from using SAM for that particular architecture so that's something they"}, {"start": 1125.16, "end": 1130.4, "text": " they empirically found so there is no like theoretical explanation for why that"}, {"start": 1130.4, "end": 1137.48, "text": " is but it is okay again it's more robust as well not just more accurate but more"}, {"start": 1137.48, "end": 1141.52, "text": " robust like looking at the image net see that is said nothing interesting there"}, {"start": 1141.52, "end": 1148.16, "text": " I already mentioned that and this is an interesting table basically they just"}, {"start": 1148.16, "end": 1152.16, "text": " kind of decompose the Hessians already mentioned which are a proxy for the"}, {"start": 1152.16, "end": 1158.28, "text": " curvature and you can see looking at the layers like let's focus on VAT or even"}, {"start": 1158.28, "end": 1163.28, "text": " better like on mixer looking at the lower layer like in the embedding layer"}, {"start": 1163.28, "end": 1170.0400000000002, "text": " itself we have a huge Hessian so the huge eigenvalue of the Hessian and going"}, {"start": 1170.0400000000002, "end": 1176.1200000000001, "text": " to deep players block one block six block twelve you can see that the Hessians go"}, {"start": 1176.1200000000001, "end": 1181.88, "text": " down so basically these lower layers contribute to the huge like to the steep"}, {"start": 1181.88, "end": 1185.88, "text": " curvature of the lost landscape and that's what I've fixed here as you can"}, {"start": 1185.88, "end": 1191.96, "text": " see using SAM it drops significantly and basically that correlates with the the"}, {"start": 1191.96, "end": 1195.3200000000002, "text": " fact that we have much sparser activations now in those lower layers as"}, {"start": 1195.3200000000002, "end": 1200.5200000000002, "text": " we saw on the plot up there so that's this plot here basically you can see"}, {"start": 1200.5200000000002, "end": 1205.5200000000002, "text": " that we have much sparser activations so this is the x-axis is the depth of the"}, {"start": 1205.5200000000002, "end": 1209.4, "text": " network so we now see that we have much sparse activations in the shallower"}, {"start": 1209.4, "end": 1214.72, "text": " layers of the network that correlates with this finding here one thing they"}, {"start": 1214.72, "end": 1220.3200000000002, "text": " notice as well is that the L2 norm of the weight vector increases which may"}, {"start": 1220.3200000000002, "end": 1226.0400000000002, "text": " indicate that they've used the weight decay regularization and it seems it's"}, {"start": 1226.0400000000002, "end": 1230.94, "text": " not helping so they need to further investigate that the reason I highlight"}, {"start": 1230.94, "end": 1236.1200000000001, "text": " the recursively here is that basically you can see H of K depends on the H of"}, {"start": 1236.12, "end": 1242.6, "text": " K plus 1 so then that's the reason why we have higher eigenvalues so this"}, {"start": 1242.6, "end": 1247.6, "text": " Hessians in the lower layer because it just multiplies with all of the"}, {"start": 1247.6, "end": 1252.0, "text": " previous deeper layers and it just kind of accumulates and blows up in the"}, {"start": 1252.0, "end": 1259.28, "text": " shallower layers okay additionally what they found is that the attention maps"}, {"start": 1259.28, "end": 1265.1999999999998, "text": " found by VATs and MLP mixers after the SAM procedure has much better"}, {"start": 1265.2, "end": 1270.3600000000001, "text": " discriminative features you can see that the attention maps do focus on"}, {"start": 1270.3600000000001, "end": 1276.1200000000001, "text": " something that's salient in these images much better than before using SAM so"}, {"start": 1276.1200000000001, "end": 1281.32, "text": " yeah just a fun fact and I mean visualizations are super important so"}, {"start": 1281.32, "end": 1290.28, "text": " kudos to them for doing this a couple of fun results here basically and it's"}, {"start": 1290.28, "end": 1295.72, "text": " pretty obvious what they show here is that even with using when you're using"}, {"start": 1295.72, "end": 1300.84, "text": " SAM when you go to when you start reducing the number of data points in"}, {"start": 1300.84, "end": 1304.8, "text": " your data set so what they did here is they randomly sampled a half of the"}, {"start": 1304.8, "end": 1309.56, "text": " pictures from the ImageNet 1k and here 1 fourth of the images are randomly"}, {"start": 1309.56, "end": 1314.26, "text": " sampled and we can see that VAT focusing on the orange curve you can see that the"}, {"start": 1314.26, "end": 1318.96, "text": " VATs and mixers degrade much more severely when we go into the low data"}, {"start": 1318.96, "end": 1325.56, "text": " regime here whereas thanks to the biases already mentioned CNN's ResNets here in"}, {"start": 1325.56, "end": 1330.72, "text": " particular managed to keep up that performance even in the low data regime"}, {"start": 1330.72, "end": 1336.1200000000001, "text": " so again priors are super important and nonetheless I do think that this kind of"}, {"start": 1336.1200000000001, "end": 1340.44, "text": " research where we are just doing this blank slate paradigm is going to be very"}, {"start": 1340.44, "end": 1347.0, "text": " interesting and informative over the long long run so yeah they"}, {"start": 1347.0, "end": 1352.52, "text": " also tried some contrastive learning and it kind of improves a bit upon the SAM"}, {"start": 1352.52, "end": 1357.56, "text": " as well they tried adversarial training as well I won't be focusing on that and"}, {"start": 1357.56, "end": 1362.84, "text": " they like using this PGD 10 attack basically PGD and I think they just"}, {"start": 1362.84, "end": 1369.92, "text": " average over 10 attacks and they showed that they get a nice like nice"}, {"start": 1369.92, "end": 1375.2, "text": " performance boost there as well yeah hopefully that's it hopefully that was"}, {"start": 1375.2, "end": 1380.6000000000001, "text": " informative and useful if you found useful consider subscribing sharing"}, {"start": 1380.6, "end": 1406.0, "text": " this video and see you next time bye bye"}]