Do Vision Transformers See Like Convolutional Neural Networks? | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I cover the "Do Vision Transformers See Like Convolutional Neural Networks?" paper. They dissect ViTs and ResNets and show the differences in the features learned as well as what contributes to those differences (like the amount of data used, skip connections, etc.).

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2108.08810
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
00:45 Contrasting features in ViTs vs CNNs 
06:45 Global vs Local receptive fields
13:55 Data matters, mr. obvious
17:40 Contrasting receptive fields
20:30 Data flow through CLS vs spatial tokens
23:30 Skip connections matter a lot in ViTs
24:20 Spatial information is preserved in ViTs
30:10 Features evolution with the amount of data
32:20 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR ML PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#vision #transformers #cnns
 What's cracking guys in this video I'm covering do vision transformers see like CNN's by the Google brain team and Before I dig into that I want to remind you that I just created a brand new discord community Feel free to join it. So there are many things that this discord community will bring so whether you're looking for some machine learning a related job or you just want to indulge into some meme content or You can also engage with the members Maybe ask you even ask your questions and somebody will be able to to kind of help you out So do join it and also subscribe to my monthly AI newsletter It's coming the first issue will be coming end of this month And I encourage you to subscribe because I strongly believe that it's gonna bring you a lot of value Okay, so getting back to the paper do vision transformers see like CNN's What basically the authors of this paper try to answer is? Whether there is a fundamental difference between their presentations learned by VAT versus CNN's How the skip connections are affecting those representations how the data size is affecting those representations, etc So there are multiple Tweaks and experiments they've done here. There is not a single one idea. It's more of trying to distill The knowledge they they they they found they accumulated during the experiment duration about what's what's the difference between these two? Model families and so let me just start digging into into diagrams They have a bunch of diagrams like this one. So in this first diagram what I've done is they try to visualize How different representations across different layers of VAT and CNN's relate to each other? So they've used something called this centered kernel alignment metric Which is this fancy formula here, but basically it's fairly simple. So what they have is they have so they have X which is NY so X is basically we have M examples and P1 is just the number of neurons in that particular layer So X mean X is just like M examples and we collect the activation So P1 activations and Y is some different layer potentially different layer It can be also the same one with M examples again and P2 neurons this time And so what I do they form these gram matrices. So K and L So if you watch my neural style transfer playlist, you'll have heard about a gram matrices they just kind of tell you like the covariance information of your data and Going further they apply this H C I's HS IC metric which is Like formed like this Just a second. So here and so they first did this centering by this H matrix and they get K prime and L prime And then they do a simple dot product Normalized with them where M is the number of examples So for the sake of just having a clear mental picture of what this means Because there's many of these in between steps you can you can you can basically treat this as a simple dot product Although it's not working on on vectors. It's working on these gram matrices, which were centralized centered etc But yeah, I think the dot product knowledge you will work Fine enough. So basically you're trying to given up even a vector you're trying to see So the closer a vector in a vector space is to that vector the bigger the dot product will be So if you have in the worst case if you have orthogonal vector the dot product will be zero Obviously if it's told in the opposite direction will have like a negative dot product So having said that let's see what's going on here. So they analyze VAT with L 16 and H H 14 models, which are basically the only differences if you remember and do watch my VAT video That's just currently that's a building block in in air research by now You should have a clear understanding of what VAT is and what transformer is before watching this video, but like a just a simple Explanation here is if you have an image What VATs do is they kind of form bunch of patches across that image and all of the patches will be so we have single patch here And so what what this number means 16 or 14 is just the dimension of the patch So 16 by 16, so that means the hardest number is the less patches will have in an image And that means that the transformer model the VAT will be smaller because the sequences will be shorter Obviously because we have bigger patches, right? So I think the information is pretty much redundant So I'm gonna go just focus on on single VAT model The whole idea here is as you can see so we have on x-axis. We have a layer index on y-axis We also have a layer index and obviously we have this line across this diagonal because Basically, if you do a dot product between a vector and itself, it's gonna give you a high output and That's why we have this this strong line here now because the matrix is symmetric by the nature of things We're just gonna focus on a single on this lower part of the of the of the of the diagram And you can see it's fairly uniform especially compared to to to ResNet And what that means is that our representations tend to stay similar as they kind of propagate through the VAT On the other hand, what happens with ResNet is there is some qualitative difference that happens As the property as the as the information as the as the features are propagating through the ResNet So what you can see here, let me help you parse this this diagram is so layers 80 to 120 Which are deeper layers of ResNet If you take a look at the similarity between the features in those layers and the features in the like Shallower in the first couple of layers of ResNet You can see that like focusing on this part of the diagram It's we have a low low activation here low similarity, which means the features have changed and On the other hand, if we if we take a look at these parts this part of the diagram here where they are comparing How the higher level features are being similar to each other We can see like a high output here as well as here and if we contrast that to to VAT We can see here it's much more uniform So that's the first conclusion they they had here is that for some reason there is this uniformity happening in VATs And it's not present in ResNet And so let's let's continue and see what other diagrams they have So the second thing that they've done here is now we have on the x-axis we have ResNet 50 On the y-axis we have VAT And what this diagram shows us is that ResNet basically takes 60 layers To calculate the features which are calculated in the first 30 something layers of VAT You can see that by looking at this bright blob that basically VAT gets to compute the features in the first 30 something layers Which take ResNet around 60 something layers And then we have the second blob it's a bit more vague here but like still So all of the other layers of ResNet are computing features which basically That's how I'm parsing this diagram at least What this means is that the VAT is managing to compute those similar features to those to the ones that ResNet is computing Until layer 75 or something So that means that VAT basically has this whole advantage here of computing refining the features even further on compared to ResNet And we've seen in the past research papers that VATs have comparable or even outperform ResNet So this kind of visualizes why this is And it all makes sense if you understand how VATs work We have a global attention span compared to CNNs which have by design have local attention only So having said that the third diagram is fairly informative And I guess it's super similar to the diagrams we saw in the original VAT paper So again I encourage you to go and check out that video if you haven't already But what it tells us is the following So we have this encoder block 012223 And what it basically means is again let me just kind of do a block diagram of VAT So this will be a VAT we have an image on the input something like this And so VAT is basically a stack of transformer blocks right And so let me denote this one as 0, this one is 1, and then we have blah blah blah And there is block 22, whatever So I guess 22 not 22 And so what they do is as you can see obviously there is a bunch of layers here in between They focus on these shallow layers as well as on deeper layers And there is a funny trend you can notice here And that's that basically if we focus on the deeper like blocks so these ones You can see that the mean distance and I'm going to explain what it is in a minute Is much higher and it's high across all of the tension heads So remember each block, each transformer layer has multiple tension heads And they just kind of plot all of the heads and all of the mean distances across all of the heads And let me before I start explaining the other curves let me kind of try and explain what this means It's fairly simple so imagine you have a simple image And we have 4x4 patches something like this 4x4 so 1, 2, 3, 4 Ok so what they do is the following So imagine you have this patch a token corresponding to this patch used as a query Again assuming you know how VATs work this query token is going to attend Basically we are going to do a dot product, a scaled dot product with the keys that correspond to other tokens And those are going to give us certain coefficients And so basically that means that this query will maybe be attending It will be attending this key here with maybe value 0.8 And it's going to be attending this key with maybe some lower weights maybe 0.01 And so basically how the mean distance is calculated is You have basically 0.8 times whatever the distance is And so we have let's say we have 3 pixels here and 3 pixels here So basically we have square root of 3 raised to the power of 2 plus 3 raised to the power of 2 So that's square root of 18 if I'm not wrong And so the basic idea is you are waiting the distance of all of the tokens from the query token And that means if we are attending if we are strongly attending further away tokens So these tokens that means that the mean distance will increase And analogously and similarly if we are attending the closer tokens and not the tokens further away Which we can see by those attention coefficients That means that the mean distance will drop down So that's a rough intuition of how this thing works Hopefully you got it And so now having that in mind let's kind of try and interpret this diagram again If we focus on the shallower block so block 0 or block 1 We can see that after we sort the attention heads So maybe this block has whatever like 16 heads I guess And all of the heads will have different mean distance which is natural And so you can see that certain heads here have much like a smaller mean distance So what that basically means is that if we have an image here and we have a certain We focus on a specific token here that means that it's going to attend the neighboring tokens IE pixels that are grouped into those tokens And I guess we saw the same or similar diagram in the original VAT paper What it tells us is that initially the shallower, the first blocks of the VAT Are going to learn both the global attention as you can see here So certain heads are still learning these global attention spans Whereas some of the other heads have something like CNN So that means some heads will be attending like this one And other heads may be attending to further away tokens like these ones here And that's the whole point, we have a mix of all of those different like I guess effective receptive fields And I guess similar trend for the VAT H14 but that's the whole idea Basically the additional flexibility that VATs, transformers in general have Is that they can choose and learn however they want their receptive fields Whereas CNNs are kind of bound to only have local receptive fields So that was already too long, let me continue, hopefully you got that one So this fourth diagram what they show is that basically if you train these models, VATs With not enough data, so all of those were pre-trained on JFT which is a huge data set I think it has 300 million data points If you train the VAT in contrast on ImageNet you can see what happens here So now if we focus on block 0 and block 1 we can see that even all of the heads have kind of global attention span And not local, and they also say here that these models also perform much worse When only trained on ImageNet suggesting that incorporating local features Which is hardcoded into CNNs may be important for strong performance And yeah I guess this is kind of a nice demonstration of that fact Okay, continuing on here, let's see the fifth diagram So what they show here is the following Now we have on X axis we have the mean distance so the same metric I just explained a couple of minutes ago Here we have the similarity metric and so what they do is they try to compare the features in shallower layers And compare them to ResNet-50 and see what's going on there So what they show is that for all of those basically features that have all of those heads That have lower mean distance so that would be in this part of the diagram Those features tend to have higher similarity with ResNet features And as the distance as the mean distance increases going right on the X axis We can see that the similarity decreases which means that VIT is qualitatively learning different representations compared to ResNet Okay similar thing goes for VIT H14 and L16 compared to ResNet-152 So I won't be focusing on the other diagrams but you get the point Basically the very fact that transformers have this extra flexibility is kind of Like being used by transformers to learn some new features which are useful But the thing I find interesting here is that we didn't see that much of an improvement that VIT has brought compared to CNNs So it kind of surprises me that where is all of this additional expressivity being used up when doing for example classification tasks So that's an open question at least in my mind So even though it has so much more flexibility and it's obviously learning some different as we saw here as well So we saw that in the first 75 layers or something we already learned similar features as ResNet So the question is what happens with all of that additional expressivity and why don't we see higher accuracy scores compared to CNNs Why are they still on pair And one thing I forgot to mention here for the data part So I'm not sure why they didn't mention anywhere in this paper the sharpness aware minimization objectives So there's a recent paper you can check out my video I'll link it somewhere here Where they showed that you don't need to pre-train VATs on JFT-300M which is super expensive You can just use the SAM objective and they have not been mentioning that paper anywhere in this paper So that kind of surprised me if somebody knows the answer feel free to write it down in the comments But yeah I guess it would be interesting to see the same kind of experiments now this time with VAT plus SAM And training only on ImageNet instead of training on JFT I guess that would be something I'd like to get an answer to Ok again as I said a bunch of diagrams not necessarily connected so stick with me Hopefully we'll have a bigger picture at the end of this video The second thing they do here is they try to visualize the receptive field of CNNs versus VATs And you're obviously hopefully familiar with how CNNs work and you can see that we have linear increase of basically of the receptive field in CNNs And the reason is let me just kind of quickly backtrack here So let me draw a simple 4x4 pixel image here So we have 4x4 pixels so something like this And so how CNNs work is basically you have a kernel let's assume we have a 2x2 window And we're gonna like do multiplication here then we're gonna slide the window to the right if we assume that the stride is equal to 1 We'll have something like this and all of these 4 pixels we're gonna be mapped into a single feature The second position will be also mapped to like a second feature here The third position will be mapped to yet another feature etc So what happens is that after we do this for the whole image What happens is that the next window that works in this layer so this was layer 0 let's say this is layer 1 So now when we have a sliding window here because remember this blue dot came from the 4 pixels and this blue dot came from 4 pixels here So that means that this window here will actually be attending to 3x3 to this span of the image And you can imagine extrapolating that into deeper layers This happens in every single layer and that's why we have like a linear increase of the receptive field in CNNs On the other hand VATs do not work like CNNs obviously and they can immediately attend whatever pixel in the image in whatever layer of the VAT So and that's kind of reflected here you can see there is a sudden jump here where all of a sudden every token can attend every token And it's kind of I don't like these diagrams to be honest it's kind of hard to understand them in O of 1 what this exactly means But like they basically calculated some absolute values of gradients of the central token of the feature map with respect to the input tokens And that's how they kind of and then they do the averaging but basically you can see these like bright shadows here Basically mean that they are attending every single input token Okay so the receptive field is basically the whole image whereas here it takes some takes a while and takes a deeper layer only there will have like a neuron attending the whole input image Okay let's continue and see some other interesting results so I like this result a lot and let me try and explain what this actually depicts So the colors on this diagram are the following things so let me just recap briefly how transformer layer looks like So this will be let's call this F this will be just like a transformation block just stick with me here so we'll have a skip connection here Those two will get added up and then we have next layer so F is either MLP or self-attention layer so that's your transformer block right So we have each transformer block consists out of MLP actually first goes the self-attention block and then we have sub block sub module and then we have the MLP sub module So that's F and so that will be the input features and obviously we'll have F of Z here after being after we transform the features we have F of Z So what they depict here is a norm simply a norm between like Z we take L2 norm of Z and we take a ratio between that and the F of Z okay So something like this we have a ratio of norms and now having that in mind let's now interpret this diagram here So you can see that yellow means high value and that means that if we focus on zero thought token which is CLS token so the classification token in VAT We can see there is a like a spike here in the lower layers which means if we focus on what this plot is actually plotting That means that the norm of Z vector is much higher than this one which means much more signal is being passed by the skip connection So actually the skip connection carries a lot of the information on this CLS token route okay And then we can see that after like we get to these deep layers from maybe fifth layer onwards we can see that the brightness goes down That means that this kind of changes and now the information is being passed over this main branch that's how they are calling it in the paper So this will be the main branch the long branch and this is the skip connection or the short branch And we can see that the reverse thing happens for all of the other tokens in the VAT model So initially you can see that the more information is carried over the long branch and then later on we have increase here That means that Z increases that means the skip connections are carrying more information To be honest I don't know what to make out of this except that it's really interesting like symmetry going on here and the transition the change of phase here So there is this complementarity between the CLS token and all of the other spatial tokens let's call them that way But yeah to be honest I'm not sure why this is but it's an interesting dynamic in any case Okay so next thing what I show is how that basically we saw that features are fairly uniform going through the whole VAT model And they show that skip connections are fairly important to keep this uniformity of representations So they show that after they remove certain like skip connection the performance drops by 4% and we get these diagrams Which tells us that now deeper layers and these shallower layers have like are way different compared to for example here Which means that there is some qualitative change that happens if we remove a skip connection and performance obviously suffers a lot So skip connections are definitely a vital part of VATs so it seems Okay let's continue on so the next thing they show here is that basically the tokens that are not CLS token are preserving the spatial information in VATs And that's fairly why they do this is because we eventually will want to if they prove to be better than CNNs We'll want to use VATs not just for the classification task but also for some tasks that require more like global spatial awareness Like object detection or semantic segmentation etc. So that's why they do this experiment and they show that basically Again similarity between tokens in deeper layers and tokens in the input image is basically localized Let me try and kind of break that down so basically this is your input image you break it into patches after you kind of So let me denote this as X we take X here we pass it through the transformer so we just pass it through the VAT here And we take a certain layer like we focus on maybe this layer maybe layer whatever 13 or something and what they do is they take a specific patch So now we take a specific patch like maybe this patch here we take this patch here and what we do is we just do similarity with the input image And we can see that the response is highest exactly where this token is which means that the representations are kind of preserved The spatial information is preserved even in the deeper layers that's the whole point so the only difference is when they take a token that's on the edge of the image For some reason it's attending the whole edge like it's attending the all of the edges of the input image and I have not to be honest I haven't seen the answer in this paper for why that is So my first assumption would be even though I know it's incorrect would be that when you do for example CNN when you do the sliding of the windows You basically have to add some padding across the image and then you can see some artifacts going on on the edges of the image But here we have VAT which works completely different so it's kind of funny to see this thing emerging keeping in mind that when you kind of flatten this out in a rest order Which is the way that VAT is usually like flatten out the input image token so if we flatten them out you can see that we'll have a bunch of green pixels And then there will be a single pixel that has like high output and then we'll have a bunch of pixels that have low output and then high so there's this pattern going on Which is kind of funny to be honest yeah if you have any hypothesis why this may be feel free to write it down in the comments I guess it's an open question The whole point is basically if we compare that to CNNs we can see that CNNs are way less localized compared to VATs and they quickly show that the reason for that is the usage of global pooling in the CNN So basically after you have a couple of CNN layers at the end you just do global pooling across the features and that kind of destroys your spatial information And that's why CNNs are poor at least those types of model of CNNs with global average global pooling are poor at spatial localization And they showed that VATs if you do the same thing instead of training it with a CLS token if you instead add like a global averaging global pooling And then you stick a classification hat on top of it you'll also have poor spatial awareness of that VAT So the next thing they did is and these diagrams are fairly interesting so again I'm going to plot VAT here for the sake of argument So VIT and so we'll have output token that corresponds to the CLS tokens let's call it CLS and we'll have the other tokens here So there will be as many of these as there are tokens in the input image so what they do here is they just kind of so all these will be vectors obviously So every token is a vector and they'll just take a linear like a linear layer on top of those vectors and then train a classifier And what they show here is that if you have a VAT trained with CLS you'll see that the test accuracy will be super low And the reason is basically because all of these so all of these tokens are fairly dumb they don't have in the sense that they don't have a They haven't been trained to understand the global context of the image and so if you stick a linear layer on your probe on top of them And train a classifier that way you basically it's going to be a poor classifier and we can see that by the test accuracy So even though the CLS probe will actually work really nicely when you take that and you average because they basically average I think they yeah yeah after you do the averaging across all of those tokens and the CLS token the test accuracy is going to drop even though CLS will have high accuracy by itself and then they show that if they train VAT using the gap instead of CLS logic So that means instead of having the CLS logic what they'll do is they'll just do averaging here and then they'll train the VAT by by sticking classification had here And so all of the tokens by doing this will learn some global context of the input image and when you do that you can see that the test accuracy Increases heavily because all of these tokens now have global information Whereas when you don't do that you obviously have more spatial information as we saw in some of the previous diagrams and finally They did some experiments on how the data influences the performance of the of the model and we can see so not the performance but basically the similarity of How the features evolve as we kind of start diminishing the amount of data that the model is being fed during the training time So on one hand we have a like a like VAT that was trained on on GFT 300m and then what they do as you can see here they kind of slowly start decreasing the amount of data So 30% of the GFT 10% and 3% and you can see that so this is a block index on the on the x-axis which means basically like the block ID of the VAT model And if we focus on the shallower on the first blocks of the VAT we can see that the similarity is fairly high of those first blocks So the features are fairly similar to the model that was trained on the full data set and as we go into deeper layers those layers suffer a lot more Compared to the shallower layers and the situation gets even worse as we decrease the amount of data from 3% you can see these are still kind of similar to the fully trained model Whereas these ones these features are completely different so it kind of deteriorates really quickly with when we decrease the data And again I'd be super happy to see the same results with a SAM objective and how the SAM influences this behavior On the other hand they have a much smaller model here so remember 32 means it's a 32 by 32 patch which means we have a smaller less tokens and thus a smaller transformer the VAT transformer And we can see that similarity is much more robust with the decrease of data here compared to this transformer And yeah I guess it has to do with the overfitting for a bigger model you simply need more data to kind of fine tune those weights to train those weights Finally not much new information here basically the more data you have the better the test accuracy will be in this linear probe setting Yeah that was pretty much it so obviously this paper is a bit different there is not a single core idea they're trying to validate What they did here instead was dissecting the VAT every single component I really like the thoroughness of this paper the experiments all of the experiments they did I guess we knew many of these information from the maybe from some of the previous papers including the original VAT paper So the shallower layers are making use of the fact that they can attend the whole image fairly successfully so that we have a mix of local and global attention in the shallower layers And we saw that in the original paper as well so we can see I guess this is the interesting part the features are kind of uniform across the whole model to just recap the whole thing VAT learns features much earlier compared to ResNets which means there is this additional expressivity that is left unthemed Again this is the same thing as in the original transformer as I said we have both global and local attention in the earlier layers And that kind of deteriorates we have less data if you're not using SAM objective if you're just training VAT the classical way without SAM So by cross entropy in the case of classification we can see that we don't have those like CNN like behavior emerging in these lower layers we only have global attention here So yeah some like experiments in receptive field and yeah this phase information is interesting the fact that we need skip connections The fact that tokens learn the spatial information and preserve it throughout the whole network except when they're using the global pooling So but like when you're training with the VAT with CLS token then the tokens kind of learn to preserve the spatial information Yeah I guess that's pretty much it for this video so if you found it useful consider subscribing share it with a friend Also I encourage you to join the discord community I strongly believe it's going to become a really vibrant community over the next couple of months Also subscribe to the AI monthly newsletter and until next time bye bye
[{"start": 0.0, "end": 4.5600000000000005, "text": " What's cracking guys in this video I'm covering do vision transformers see like CNN's"}, {"start": 5.36, "end": 7.68, "text": " by the Google brain team and"}, {"start": 8.24, "end": 13.1, "text": " Before I dig into that I want to remind you that I just created a brand new discord community"}, {"start": 13.1, "end": 16.44, "text": " Feel free to join it. So there are many things that this discord community will bring"}, {"start": 16.44, "end": 20.400000000000002, "text": " so whether you're looking for some machine learning a related job or"}, {"start": 20.8, "end": 24.0, "text": " you just want to indulge into some meme content or"}, {"start": 24.84, "end": 26.94, "text": " You can also engage with the members"}, {"start": 26.94, "end": 31.86, "text": " Maybe ask you even ask your questions and somebody will be able to to kind of help you out"}, {"start": 31.86, "end": 36.18, "text": " So do join it and also subscribe to my monthly AI newsletter"}, {"start": 36.18, "end": 39.3, "text": " It's coming the first issue will be coming end of this month"}, {"start": 39.3, "end": 43.3, "text": " And I encourage you to subscribe because I strongly believe that it's gonna bring you a lot of value"}, {"start": 43.46, "end": 46.94, "text": " Okay, so getting back to the paper do vision transformers see"}, {"start": 47.5, "end": 49.1, "text": " like CNN's"}, {"start": 49.1, "end": 52.18000000000001, "text": " What basically the authors of this paper try to answer is?"}, {"start": 52.18, "end": 58.78, "text": " Whether there is a fundamental difference between their presentations learned by VAT versus CNN's"}, {"start": 59.22, "end": 66.78, "text": " How the skip connections are affecting those representations how the data size is affecting those representations, etc"}, {"start": 66.98, "end": 68.78, "text": " So there are multiple"}, {"start": 68.78, "end": 73.7, "text": " Tweaks and experiments they've done here. There is not a single one idea. It's more of"}, {"start": 74.42, "end": 76.3, "text": " trying to distill"}, {"start": 76.3, "end": 84.62, "text": " The knowledge they they they they found they accumulated during the experiment duration about what's what's the difference between these two?"}, {"start": 85.22, "end": 90.02, "text": " Model families and so let me just start digging into into diagrams"}, {"start": 90.02, "end": 95.96, "text": " They have a bunch of diagrams like this one. So in this first diagram what I've done is they try to visualize"}, {"start": 96.7, "end": 104.7, "text": " How different representations across different layers of VAT and CNN's relate to each other? So they've used something called this"}, {"start": 104.7, "end": 106.7, "text": " centered kernel alignment metric"}, {"start": 106.7, "end": 113.7, "text": " Which is this fancy formula here, but basically it's fairly simple. So what they have is they have so they have"}, {"start": 114.7, "end": 123.7, "text": " X which is NY so X is basically we have M examples and P1 is just the number of neurons in that particular layer"}, {"start": 123.7, "end": 129.7, "text": " So X mean X is just like M examples and we collect the activation"}, {"start": 129.7, "end": 134.7, "text": " So P1 activations and Y is some different layer potentially different layer"}, {"start": 134.7, "end": 139.7, "text": " It can be also the same one with M examples again and P2 neurons this time"}, {"start": 139.7, "end": 143.7, "text": " And so what I do they form these gram matrices. So K and L"}, {"start": 143.7, "end": 148.7, "text": " So if you watch my neural style transfer playlist, you'll have heard about a gram matrices"}, {"start": 148.7, "end": 153.7, "text": " they just kind of tell you like the covariance information of your data and"}, {"start": 153.7, "end": 160.7, "text": " Going further they apply this H C I's HS IC metric which is"}, {"start": 160.7, "end": 162.7, "text": " Like formed like this"}, {"start": 162.7, "end": 170.7, "text": " Just a second. So here and so they first did this centering by this H matrix and they get K prime and L prime"}, {"start": 170.7, "end": 173.7, "text": " And then they do a simple dot product"}, {"start": 173.7, "end": 176.7, "text": " Normalized with them where M is the number of examples"}, {"start": 176.7, "end": 181.7, "text": " So for the sake of just having a clear mental picture of what this means"}, {"start": 181.7, "end": 187.7, "text": " Because there's many of these in between steps you can you can you can basically treat this as a simple dot product"}, {"start": 187.7, "end": 194.7, "text": " Although it's not working on on vectors. It's working on these gram matrices, which were centralized centered etc"}, {"start": 194.7, "end": 197.7, "text": " But yeah, I think the dot product knowledge you will work"}, {"start": 197.7, "end": 203.7, "text": " Fine enough. So basically you're trying to given up even a vector you're trying to see"}, {"start": 203.7, "end": 209.7, "text": " So the closer a vector in a vector space is to that vector the bigger the dot product will be"}, {"start": 209.7, "end": 214.7, "text": " So if you have in the worst case if you have orthogonal vector the dot product will be zero"}, {"start": 214.7, "end": 219.7, "text": " Obviously if it's told in the opposite direction will have like a negative dot product"}, {"start": 219.7, "end": 228.7, "text": " So having said that let's see what's going on here. So they analyze VAT with L 16 and H"}, {"start": 228.7, "end": 234.7, "text": " H 14 models, which are basically the only differences if you remember and do watch my VAT video"}, {"start": 234.7, "end": 239.7, "text": " That's just currently that's a building block in in air research by now"}, {"start": 239.7, "end": 245.7, "text": " You should have a clear understanding of what VAT is and what transformer is before watching this video, but like a just a simple"}, {"start": 245.7, "end": 248.7, "text": " Explanation here is if you have an image"}, {"start": 248.7, "end": 256.7, "text": " What VATs do is they kind of form bunch of patches across that image and all of the patches will be so we have single patch here"}, {"start": 256.7, "end": 262.7, "text": " And so what what this number means 16 or 14 is just the dimension of the patch"}, {"start": 262.7, "end": 269.7, "text": " So 16 by 16, so that means the hardest number is the less patches will have in an image"}, {"start": 269.7, "end": 274.7, "text": " And that means that the transformer model the VAT will be smaller because the sequences will be shorter"}, {"start": 274.7, "end": 281.7, "text": " Obviously because we have bigger patches, right? So I think the information is pretty much redundant"}, {"start": 281.7, "end": 285.7, "text": " So I'm gonna go just focus on on single VAT model"}, {"start": 285.7, "end": 290.7, "text": " The whole idea here is as you can see so we have on x-axis. We have a layer index on y-axis"}, {"start": 290.7, "end": 297.7, "text": " We also have a layer index and obviously we have this line across this diagonal because"}, {"start": 297.7, "end": 304.7, "text": " Basically, if you do a dot product between a vector and itself, it's gonna give you a high output and"}, {"start": 304.7, "end": 311.7, "text": " That's why we have this this strong line here now because the matrix is symmetric by the nature of things"}, {"start": 311.7, "end": 317.7, "text": " We're just gonna focus on a single on this lower part of the of the of the of the diagram"}, {"start": 317.7, "end": 323.7, "text": " And you can see it's fairly uniform especially compared to to to ResNet"}, {"start": 323.7, "end": 331.7, "text": " And what that means is that our representations tend to stay similar as they kind of propagate through the VAT"}, {"start": 331.7, "end": 335.7, "text": " On the other hand, what happens with ResNet is there is some qualitative difference that happens"}, {"start": 335.7, "end": 341.7, "text": " As the property as the as the information as the as the features are propagating through the ResNet"}, {"start": 341.7, "end": 348.7, "text": " So what you can see here, let me help you parse this this diagram is so layers 80 to 120"}, {"start": 348.7, "end": 351.7, "text": " Which are deeper layers of ResNet"}, {"start": 351.7, "end": 357.7, "text": " If you take a look at the similarity between the features in those layers and the features in the like"}, {"start": 357.7, "end": 361.7, "text": " Shallower in the first couple of layers of ResNet"}, {"start": 361.7, "end": 365.7, "text": " You can see that like focusing on this part of the diagram"}, {"start": 365.7, "end": 373.7, "text": " It's we have a low low activation here low similarity, which means the features have changed and"}, {"start": 373.7, "end": 381.7, "text": " On the other hand, if we if we take a look at these parts this part of the diagram here where they are comparing"}, {"start": 381.7, "end": 385.7, "text": " How the higher level features are being similar to each other"}, {"start": 385.7, "end": 390.7, "text": " We can see like a high output here as well as here and if we contrast that to to VAT"}, {"start": 390.7, "end": 392.7, "text": " We can see here it's much more uniform"}, {"start": 392.7, "end": 399.7, "text": " So that's the first conclusion they they had here is that for some reason there is this uniformity happening in VATs"}, {"start": 399.7, "end": 401.7, "text": " And it's not present in ResNet"}, {"start": 401.7, "end": 406.7, "text": " And so let's let's continue and see what other diagrams they have"}, {"start": 406.7, "end": 413.7, "text": " So the second thing that they've done here is now we have on the x-axis we have ResNet 50"}, {"start": 413.7, "end": 415.7, "text": " On the y-axis we have VAT"}, {"start": 415.7, "end": 421.7, "text": " And what this diagram shows us is that ResNet basically takes 60 layers"}, {"start": 421.7, "end": 427.7, "text": " To calculate the features which are calculated in the first 30 something layers of VAT"}, {"start": 427.7, "end": 439.7, "text": " You can see that by looking at this bright blob that basically VAT gets to compute the features in the first 30 something layers"}, {"start": 439.7, "end": 442.7, "text": " Which take ResNet around 60 something layers"}, {"start": 442.7, "end": 447.7, "text": " And then we have the second blob it's a bit more vague here but like still"}, {"start": 447.7, "end": 454.7, "text": " So all of the other layers of ResNet are computing features which basically"}, {"start": 454.7, "end": 456.7, "text": " That's how I'm parsing this diagram at least"}, {"start": 456.7, "end": 466.7, "text": " What this means is that the VAT is managing to compute those similar features to those to the ones that ResNet is computing"}, {"start": 466.7, "end": 468.7, "text": " Until layer 75 or something"}, {"start": 468.7, "end": 479.7, "text": " So that means that VAT basically has this whole advantage here of computing refining the features even further on compared to ResNet"}, {"start": 479.7, "end": 486.7, "text": " And we've seen in the past research papers that VATs have comparable or even outperform ResNet"}, {"start": 486.7, "end": 490.7, "text": " So this kind of visualizes why this is"}, {"start": 490.7, "end": 494.7, "text": " And it all makes sense if you understand how VATs work"}, {"start": 494.7, "end": 502.7, "text": " We have a global attention span compared to CNNs which have by design have local attention only"}, {"start": 502.7, "end": 509.7, "text": " So having said that the third diagram is fairly informative"}, {"start": 509.7, "end": 515.7, "text": " And I guess it's super similar to the diagrams we saw in the original VAT paper"}, {"start": 515.7, "end": 520.7, "text": " So again I encourage you to go and check out that video if you haven't already"}, {"start": 520.7, "end": 522.7, "text": " But what it tells us is the following"}, {"start": 522.7, "end": 527.7, "text": " So we have this encoder block 012223"}, {"start": 527.7, "end": 533.7, "text": " And what it basically means is again let me just kind of do a block diagram of VAT"}, {"start": 533.7, "end": 538.7, "text": " So this will be a VAT we have an image on the input something like this"}, {"start": 538.7, "end": 542.7, "text": " And so VAT is basically a stack of transformer blocks right"}, {"start": 542.7, "end": 548.7, "text": " And so let me denote this one as 0, this one is 1, and then we have blah blah blah"}, {"start": 548.7, "end": 551.7, "text": " And there is block 22, whatever"}, {"start": 551.7, "end": 554.7, "text": " So I guess 22 not 22"}, {"start": 554.7, "end": 560.7, "text": " And so what they do is as you can see obviously there is a bunch of layers here in between"}, {"start": 560.7, "end": 564.7, "text": " They focus on these shallow layers as well as on deeper layers"}, {"start": 564.7, "end": 567.7, "text": " And there is a funny trend you can notice here"}, {"start": 567.7, "end": 575.7, "text": " And that's that basically if we focus on the deeper like blocks so these ones"}, {"start": 575.7, "end": 579.7, "text": " You can see that the mean distance and I'm going to explain what it is in a minute"}, {"start": 579.7, "end": 583.7, "text": " Is much higher and it's high across all of the tension heads"}, {"start": 583.7, "end": 588.7, "text": " So remember each block, each transformer layer has multiple tension heads"}, {"start": 588.7, "end": 593.7, "text": " And they just kind of plot all of the heads and all of the mean distances across all of the heads"}, {"start": 593.7, "end": 599.7, "text": " And let me before I start explaining the other curves let me kind of try and explain what this means"}, {"start": 599.7, "end": 604.7, "text": " It's fairly simple so imagine you have a simple image"}, {"start": 604.7, "end": 607.7, "text": " And we have 4x4 patches something like this"}, {"start": 607.7, "end": 611.7, "text": " 4x4 so 1, 2, 3, 4"}, {"start": 611.7, "end": 614.7, "text": " Ok so what they do is the following"}, {"start": 614.7, "end": 620.7, "text": " So imagine you have this patch a token corresponding to this patch used as a query"}, {"start": 620.7, "end": 626.7, "text": " Again assuming you know how VATs work this query token is going to attend"}, {"start": 626.7, "end": 632.7, "text": " Basically we are going to do a dot product, a scaled dot product with the keys that correspond to other tokens"}, {"start": 632.7, "end": 635.7, "text": " And those are going to give us certain coefficients"}, {"start": 635.7, "end": 641.7, "text": " And so basically that means that this query will maybe be attending"}, {"start": 641.7, "end": 647.7, "text": " It will be attending this key here with maybe value 0.8"}, {"start": 647.7, "end": 654.7, "text": " And it's going to be attending this key with maybe some lower weights maybe 0.01"}, {"start": 654.7, "end": 660.7, "text": " And so basically how the mean distance is calculated is"}, {"start": 660.7, "end": 665.7, "text": " You have basically 0.8 times whatever the distance is"}, {"start": 665.7, "end": 668.7, "text": " And so we have let's say we have 3 pixels here and 3 pixels here"}, {"start": 668.7, "end": 676.7, "text": " So basically we have square root of 3 raised to the power of 2 plus 3 raised to the power of 2"}, {"start": 676.7, "end": 679.7, "text": " So that's square root of 18 if I'm not wrong"}, {"start": 679.7, "end": 687.7, "text": " And so the basic idea is you are waiting the distance of all of the tokens from the query token"}, {"start": 687.7, "end": 692.7, "text": " And that means if we are attending if we are strongly attending further away tokens"}, {"start": 692.7, "end": 695.7, "text": " So these tokens that means that the mean distance will increase"}, {"start": 695.7, "end": 706.7, "text": " And analogously and similarly if we are attending the closer tokens and not the tokens further away"}, {"start": 706.7, "end": 710.7, "text": " Which we can see by those attention coefficients"}, {"start": 710.7, "end": 713.7, "text": " That means that the mean distance will drop down"}, {"start": 713.7, "end": 715.7, "text": " So that's a rough intuition of how this thing works"}, {"start": 715.7, "end": 717.7, "text": " Hopefully you got it"}, {"start": 717.7, "end": 723.7, "text": " And so now having that in mind let's kind of try and interpret this diagram again"}, {"start": 723.7, "end": 728.7, "text": " If we focus on the shallower block so block 0 or block 1"}, {"start": 728.7, "end": 732.7, "text": " We can see that after we sort the attention heads"}, {"start": 732.7, "end": 736.7, "text": " So maybe this block has whatever like 16 heads I guess"}, {"start": 736.7, "end": 741.7, "text": " And all of the heads will have different mean distance which is natural"}, {"start": 741.7, "end": 748.7, "text": " And so you can see that certain heads here have much like a smaller mean distance"}, {"start": 748.7, "end": 752.7, "text": " So what that basically means is that if we have an image here and we have a certain"}, {"start": 752.7, "end": 758.7, "text": " We focus on a specific token here that means that it's going to attend the neighboring tokens"}, {"start": 758.7, "end": 762.7, "text": " IE pixels that are grouped into those tokens"}, {"start": 762.7, "end": 768.7, "text": " And I guess we saw the same or similar diagram in the original VAT paper"}, {"start": 768.7, "end": 775.7, "text": " What it tells us is that initially the shallower, the first blocks of the VAT"}, {"start": 775.7, "end": 778.7, "text": " Are going to learn both the global attention as you can see here"}, {"start": 778.7, "end": 782.7, "text": " So certain heads are still learning these global attention spans"}, {"start": 782.7, "end": 786.7, "text": " Whereas some of the other heads have something like CNN"}, {"start": 786.7, "end": 789.7, "text": " So that means some heads will be attending like this one"}, {"start": 789.7, "end": 794.7, "text": " And other heads may be attending to further away tokens like these ones here"}, {"start": 794.7, "end": 803.7, "text": " And that's the whole point, we have a mix of all of those different like I guess effective receptive fields"}, {"start": 803.7, "end": 810.7, "text": " And I guess similar trend for the VAT H14 but that's the whole idea"}, {"start": 810.7, "end": 816.7, "text": " Basically the additional flexibility that VATs, transformers in general have"}, {"start": 816.7, "end": 821.7, "text": " Is that they can choose and learn however they want their receptive fields"}, {"start": 821.7, "end": 826.7, "text": " Whereas CNNs are kind of bound to only have local receptive fields"}, {"start": 826.7, "end": 831.7, "text": " So that was already too long, let me continue, hopefully you got that one"}, {"start": 831.7, "end": 838.7, "text": " So this fourth diagram what they show is that basically if you train these models, VATs"}, {"start": 838.7, "end": 844.7, "text": " With not enough data, so all of those were pre-trained on JFT which is a huge data set"}, {"start": 844.7, "end": 847.7, "text": " I think it has 300 million data points"}, {"start": 847.7, "end": 853.7, "text": " If you train the VAT in contrast on ImageNet you can see what happens here"}, {"start": 853.7, "end": 863.7, "text": " So now if we focus on block 0 and block 1 we can see that even all of the heads have kind of global attention span"}, {"start": 863.7, "end": 869.7, "text": " And not local, and they also say here that these models also perform much worse"}, {"start": 869.7, "end": 873.7, "text": " When only trained on ImageNet suggesting that incorporating local features"}, {"start": 873.7, "end": 877.7, "text": " Which is hardcoded into CNNs may be important for strong performance"}, {"start": 877.7, "end": 882.7, "text": " And yeah I guess this is kind of a nice demonstration of that fact"}, {"start": 882.7, "end": 887.7, "text": " Okay, continuing on here, let's see the fifth diagram"}, {"start": 887.7, "end": 890.7, "text": " So what they show here is the following"}, {"start": 890.7, "end": 895.7, "text": " Now we have on X axis we have the mean distance so the same metric I just explained a couple of minutes ago"}, {"start": 895.7, "end": 902.7, "text": " Here we have the similarity metric and so what they do is they try to compare the features in shallower layers"}, {"start": 902.7, "end": 906.7, "text": " And compare them to ResNet-50 and see what's going on there"}, {"start": 906.7, "end": 913.7, "text": " So what they show is that for all of those basically features that have all of those heads"}, {"start": 913.7, "end": 917.7, "text": " That have lower mean distance so that would be in this part of the diagram"}, {"start": 917.7, "end": 922.7, "text": " Those features tend to have higher similarity with ResNet features"}, {"start": 922.7, "end": 928.7, "text": " And as the distance as the mean distance increases going right on the X axis"}, {"start": 928.7, "end": 936.7, "text": " We can see that the similarity decreases which means that VIT is qualitatively learning different representations compared to ResNet"}, {"start": 936.7, "end": 945.7, "text": " Okay similar thing goes for VIT H14 and L16 compared to ResNet-152"}, {"start": 945.7, "end": 948.7, "text": " So I won't be focusing on the other diagrams but you get the point"}, {"start": 948.7, "end": 954.7, "text": " Basically the very fact that transformers have this extra flexibility is kind of"}, {"start": 954.7, "end": 959.7, "text": " Like being used by transformers to learn some new features which are useful"}, {"start": 959.7, "end": 967.7, "text": " But the thing I find interesting here is that we didn't see that much of an improvement that VIT has brought compared to CNNs"}, {"start": 967.7, "end": 975.7, "text": " So it kind of surprises me that where is all of this additional expressivity being used up when doing for example classification tasks"}, {"start": 975.7, "end": 979.7, "text": " So that's an open question at least in my mind"}, {"start": 979.7, "end": 986.7, "text": " So even though it has so much more flexibility and it's obviously learning some different as we saw here as well"}, {"start": 986.7, "end": 994.7, "text": " So we saw that in the first 75 layers or something we already learned similar features as ResNet"}, {"start": 994.7, "end": 1002.7, "text": " So the question is what happens with all of that additional expressivity and why don't we see higher accuracy scores compared to CNNs"}, {"start": 1002.7, "end": 1005.7, "text": " Why are they still on pair"}, {"start": 1005.7, "end": 1010.7, "text": " And one thing I forgot to mention here for the data part"}, {"start": 1010.7, "end": 1017.7, "text": " So I'm not sure why they didn't mention anywhere in this paper the sharpness aware minimization objectives"}, {"start": 1017.7, "end": 1020.7, "text": " So there's a recent paper you can check out my video I'll link it somewhere here"}, {"start": 1020.7, "end": 1028.7, "text": " Where they showed that you don't need to pre-train VATs on JFT-300M which is super expensive"}, {"start": 1028.7, "end": 1036.7, "text": " You can just use the SAM objective and they have not been mentioning that paper anywhere in this paper"}, {"start": 1036.7, "end": 1041.7, "text": " So that kind of surprised me if somebody knows the answer feel free to write it down in the comments"}, {"start": 1041.7, "end": 1050.7, "text": " But yeah I guess it would be interesting to see the same kind of experiments now this time with VAT plus SAM"}, {"start": 1050.7, "end": 1053.7, "text": " And training only on ImageNet instead of training on JFT"}, {"start": 1053.7, "end": 1056.7, "text": " I guess that would be something I'd like to get an answer to"}, {"start": 1056.7, "end": 1063.7, "text": " Ok again as I said a bunch of diagrams not necessarily connected so stick with me"}, {"start": 1063.7, "end": 1067.7, "text": " Hopefully we'll have a bigger picture at the end of this video"}, {"start": 1067.7, "end": 1073.7, "text": " The second thing they do here is they try to visualize the receptive field of CNNs versus VATs"}, {"start": 1073.7, "end": 1084.7, "text": " And you're obviously hopefully familiar with how CNNs work and you can see that we have linear increase of basically of the receptive field in CNNs"}, {"start": 1084.7, "end": 1087.7, "text": " And the reason is let me just kind of quickly backtrack here"}, {"start": 1087.7, "end": 1092.7, "text": " So let me draw a simple 4x4 pixel image here"}, {"start": 1092.7, "end": 1096.7, "text": " So we have 4x4 pixels so something like this"}, {"start": 1096.7, "end": 1101.7, "text": " And so how CNNs work is basically you have a kernel let's assume we have a 2x2 window"}, {"start": 1101.7, "end": 1109.7, "text": " And we're gonna like do multiplication here then we're gonna slide the window to the right if we assume that the stride is equal to 1"}, {"start": 1109.7, "end": 1115.7, "text": " We'll have something like this and all of these 4 pixels we're gonna be mapped into a single feature"}, {"start": 1115.7, "end": 1119.7, "text": " The second position will be also mapped to like a second feature here"}, {"start": 1119.7, "end": 1124.7, "text": " The third position will be mapped to yet another feature etc"}, {"start": 1124.7, "end": 1129.7, "text": " So what happens is that after we do this for the whole image"}, {"start": 1129.7, "end": 1135.7, "text": " What happens is that the next window that works in this layer so this was layer 0 let's say this is layer 1"}, {"start": 1135.7, "end": 1145.7, "text": " So now when we have a sliding window here because remember this blue dot came from the 4 pixels and this blue dot came from 4 pixels here"}, {"start": 1145.7, "end": 1154.7, "text": " So that means that this window here will actually be attending to 3x3 to this span of the image"}, {"start": 1154.7, "end": 1157.7, "text": " And you can imagine extrapolating that into deeper layers"}, {"start": 1157.7, "end": 1164.7, "text": " This happens in every single layer and that's why we have like a linear increase of the receptive field in CNNs"}, {"start": 1164.7, "end": 1176.7, "text": " On the other hand VATs do not work like CNNs obviously and they can immediately attend whatever pixel in the image in whatever layer of the VAT"}, {"start": 1176.7, "end": 1186.7, "text": " So and that's kind of reflected here you can see there is a sudden jump here where all of a sudden every token can attend every token"}, {"start": 1186.7, "end": 1193.7, "text": " And it's kind of I don't like these diagrams to be honest it's kind of hard to understand them in O of 1 what this exactly means"}, {"start": 1193.7, "end": 1203.7, "text": " But like they basically calculated some absolute values of gradients of the central token of the feature map with respect to the input tokens"}, {"start": 1203.7, "end": 1210.7, "text": " And that's how they kind of and then they do the averaging but basically you can see these like bright shadows here"}, {"start": 1210.7, "end": 1214.7, "text": " Basically mean that they are attending every single input token"}, {"start": 1214.7, "end": 1226.7, "text": " Okay so the receptive field is basically the whole image whereas here it takes some takes a while and takes a deeper layer only there will have like a neuron attending the whole input image"}, {"start": 1226.7, "end": 1236.7, "text": " Okay let's continue and see some other interesting results so I like this result a lot and let me try and explain what this actually depicts"}, {"start": 1236.7, "end": 1245.7, "text": " So the colors on this diagram are the following things so let me just recap briefly how transformer layer looks like"}, {"start": 1245.7, "end": 1257.7, "text": " So this will be let's call this F this will be just like a transformation block just stick with me here so we'll have a skip connection here"}, {"start": 1257.7, "end": 1267.7, "text": " Those two will get added up and then we have next layer so F is either MLP or self-attention layer so that's your transformer block right"}, {"start": 1267.7, "end": 1277.7, "text": " So we have each transformer block consists out of MLP actually first goes the self-attention block and then we have sub block sub module and then we have the MLP sub module"}, {"start": 1277.7, "end": 1290.7, "text": " So that's F and so that will be the input features and obviously we'll have F of Z here after being after we transform the features we have F of Z"}, {"start": 1290.7, "end": 1302.7, "text": " So what they depict here is a norm simply a norm between like Z we take L2 norm of Z and we take a ratio between that and the F of Z okay"}, {"start": 1302.7, "end": 1310.7, "text": " So something like this we have a ratio of norms and now having that in mind let's now interpret this diagram here"}, {"start": 1310.7, "end": 1320.7, "text": " So you can see that yellow means high value and that means that if we focus on zero thought token which is CLS token so the classification token in VAT"}, {"start": 1320.7, "end": 1328.7, "text": " We can see there is a like a spike here in the lower layers which means if we focus on what this plot is actually plotting"}, {"start": 1328.7, "end": 1336.7, "text": " That means that the norm of Z vector is much higher than this one which means much more signal is being passed by the skip connection"}, {"start": 1336.7, "end": 1344.7, "text": " So actually the skip connection carries a lot of the information on this CLS token route okay"}, {"start": 1344.7, "end": 1352.7, "text": " And then we can see that after like we get to these deep layers from maybe fifth layer onwards we can see that the brightness goes down"}, {"start": 1352.7, "end": 1360.7, "text": " That means that this kind of changes and now the information is being passed over this main branch that's how they are calling it in the paper"}, {"start": 1360.7, "end": 1364.7, "text": " So this will be the main branch the long branch and this is the skip connection or the short branch"}, {"start": 1364.7, "end": 1372.7, "text": " And we can see that the reverse thing happens for all of the other tokens in the VAT model"}, {"start": 1372.7, "end": 1379.7, "text": " So initially you can see that the more information is carried over the long branch and then later on we have increase here"}, {"start": 1379.7, "end": 1383.7, "text": " That means that Z increases that means the skip connections are carrying more information"}, {"start": 1383.7, "end": 1393.7, "text": " To be honest I don't know what to make out of this except that it's really interesting like symmetry going on here and the transition the change of phase here"}, {"start": 1393.7, "end": 1401.7, "text": " So there is this complementarity between the CLS token and all of the other spatial tokens let's call them that way"}, {"start": 1401.7, "end": 1410.7, "text": " But yeah to be honest I'm not sure why this is but it's an interesting dynamic in any case"}, {"start": 1410.7, "end": 1422.7, "text": " Okay so next thing what I show is how that basically we saw that features are fairly uniform going through the whole VAT model"}, {"start": 1422.7, "end": 1427.7, "text": " And they show that skip connections are fairly important to keep this uniformity of representations"}, {"start": 1427.7, "end": 1436.7, "text": " So they show that after they remove certain like skip connection the performance drops by 4% and we get these diagrams"}, {"start": 1436.7, "end": 1446.7, "text": " Which tells us that now deeper layers and these shallower layers have like are way different compared to for example here"}, {"start": 1446.7, "end": 1454.7, "text": " Which means that there is some qualitative change that happens if we remove a skip connection and performance obviously suffers a lot"}, {"start": 1454.7, "end": 1459.7, "text": " So skip connections are definitely a vital part of VATs so it seems"}, {"start": 1459.7, "end": 1472.7, "text": " Okay let's continue on so the next thing they show here is that basically the tokens that are not CLS token are preserving the spatial information in VATs"}, {"start": 1472.7, "end": 1481.7, "text": " And that's fairly why they do this is because we eventually will want to if they prove to be better than CNNs"}, {"start": 1481.7, "end": 1488.7, "text": " We'll want to use VATs not just for the classification task but also for some tasks that require more like global spatial awareness"}, {"start": 1488.7, "end": 1496.7, "text": " Like object detection or semantic segmentation etc. So that's why they do this experiment and they show that basically"}, {"start": 1496.7, "end": 1504.7, "text": " Again similarity between tokens in deeper layers and tokens in the input image is basically localized"}, {"start": 1504.7, "end": 1512.7, "text": " Let me try and kind of break that down so basically this is your input image you break it into patches after you kind of"}, {"start": 1512.7, "end": 1522.7, "text": " So let me denote this as X we take X here we pass it through the transformer so we just pass it through the VAT here"}, {"start": 1522.7, "end": 1531.7, "text": " And we take a certain layer like we focus on maybe this layer maybe layer whatever 13 or something and what they do is they take a specific patch"}, {"start": 1531.7, "end": 1541.7, "text": " So now we take a specific patch like maybe this patch here we take this patch here and what we do is we just do similarity with the input image"}, {"start": 1541.7, "end": 1550.7, "text": " And we can see that the response is highest exactly where this token is which means that the representations are kind of preserved"}, {"start": 1550.7, "end": 1559.7, "text": " The spatial information is preserved even in the deeper layers that's the whole point so the only difference is when they take a token that's on the edge of the image"}, {"start": 1559.7, "end": 1572.7, "text": " For some reason it's attending the whole edge like it's attending the all of the edges of the input image and I have not to be honest I haven't seen the answer in this paper for why that is"}, {"start": 1572.7, "end": 1580.7, "text": " So my first assumption would be even though I know it's incorrect would be that when you do for example CNN when you do the sliding of the windows"}, {"start": 1580.7, "end": 1587.7, "text": " You basically have to add some padding across the image and then you can see some artifacts going on on the edges of the image"}, {"start": 1587.7, "end": 1598.7, "text": " But here we have VAT which works completely different so it's kind of funny to see this thing emerging keeping in mind that when you kind of flatten this out in a rest order"}, {"start": 1598.7, "end": 1609.7, "text": " Which is the way that VAT is usually like flatten out the input image token so if we flatten them out you can see that we'll have a bunch of green pixels"}, {"start": 1609.7, "end": 1618.7, "text": " And then there will be a single pixel that has like high output and then we'll have a bunch of pixels that have low output and then high so there's this pattern going on"}, {"start": 1618.7, "end": 1627.7, "text": " Which is kind of funny to be honest yeah if you have any hypothesis why this may be feel free to write it down in the comments I guess it's an open question"}, {"start": 1627.7, "end": 1645.7, "text": " The whole point is basically if we compare that to CNNs we can see that CNNs are way less localized compared to VATs and they quickly show that the reason for that is the usage of global pooling in the CNN"}, {"start": 1645.7, "end": 1654.7, "text": " So basically after you have a couple of CNN layers at the end you just do global pooling across the features and that kind of destroys your spatial information"}, {"start": 1654.7, "end": 1664.7, "text": " And that's why CNNs are poor at least those types of model of CNNs with global average global pooling are poor at spatial localization"}, {"start": 1664.7, "end": 1674.7, "text": " And they showed that VATs if you do the same thing instead of training it with a CLS token if you instead add like a global averaging global pooling"}, {"start": 1674.7, "end": 1683.7, "text": " And then you stick a classification hat on top of it you'll also have poor spatial awareness of that VAT"}, {"start": 1683.7, "end": 1691.7, "text": " So the next thing they did is and these diagrams are fairly interesting so again I'm going to plot VAT here for the sake of argument"}, {"start": 1691.7, "end": 1699.7, "text": " So VIT and so we'll have output token that corresponds to the CLS tokens let's call it CLS and we'll have the other tokens here"}, {"start": 1699.7, "end": 1709.7, "text": " So there will be as many of these as there are tokens in the input image so what they do here is they just kind of so all these will be vectors obviously"}, {"start": 1709.7, "end": 1718.7, "text": " So every token is a vector and they'll just take a linear like a linear layer on top of those vectors and then train a classifier"}, {"start": 1718.7, "end": 1726.7, "text": " And what they show here is that if you have a VAT trained with CLS you'll see that the test accuracy will be super low"}, {"start": 1726.7, "end": 1735.7, "text": " And the reason is basically because all of these so all of these tokens are fairly dumb they don't have in the sense that they don't have a"}, {"start": 1735.7, "end": 1743.7, "text": " They haven't been trained to understand the global context of the image and so if you stick a linear layer on your probe on top of them"}, {"start": 1743.7, "end": 1749.7, "text": " And train a classifier that way you basically it's going to be a poor classifier and we can see that by the test accuracy"}, {"start": 1749.7, "end": 1757.7, "text": " So even though the CLS probe will actually work really nicely when you take that and you average because they basically average"}, {"start": 1757.7, "end": 1765.7, "text": " I think they yeah yeah after you do the averaging across all of those tokens and the CLS token the test accuracy is going to drop even though"}, {"start": 1765.7, "end": 1774.7, "text": " CLS will have high accuracy by itself and then they show that if they train VAT using the gap instead of CLS logic"}, {"start": 1774.7, "end": 1785.7, "text": " So that means instead of having the CLS logic what they'll do is they'll just do averaging here and then they'll train the VAT by by sticking classification had here"}, {"start": 1785.7, "end": 1793.7, "text": " And so all of the tokens by doing this will learn some global context of the input image and when you do that you can see that the test accuracy"}, {"start": 1793.7, "end": 1798.7, "text": " Increases heavily because all of these tokens now have global information"}, {"start": 1798.7, "end": 1807.7, "text": " Whereas when you don't do that you obviously have more spatial information as we saw in some of the previous diagrams and finally"}, {"start": 1807.7, "end": 1817.7, "text": " They did some experiments on how the data influences the performance of the of the model and we can see so not the performance but basically the similarity of"}, {"start": 1817.7, "end": 1825.7, "text": " How the features evolve as we kind of start diminishing the amount of data that the model is being fed during the training time"}, {"start": 1825.7, "end": 1836.7, "text": " So on one hand we have a like a like VAT that was trained on on GFT 300m and then what they do as you can see here they kind of slowly start decreasing the amount of data"}, {"start": 1836.7, "end": 1848.7, "text": " So 30% of the GFT 10% and 3% and you can see that so this is a block index on the on the x-axis which means basically like the block ID of the VAT model"}, {"start": 1848.7, "end": 1857.7, "text": " And if we focus on the shallower on the first blocks of the VAT we can see that the similarity is fairly high of those first blocks"}, {"start": 1857.7, "end": 1865.7, "text": " So the features are fairly similar to the model that was trained on the full data set and as we go into deeper layers those layers suffer a lot more"}, {"start": 1865.7, "end": 1878.7, "text": " Compared to the shallower layers and the situation gets even worse as we decrease the amount of data from 3% you can see these are still kind of similar to the fully trained model"}, {"start": 1878.7, "end": 1888.7, "text": " Whereas these ones these features are completely different so it kind of deteriorates really quickly with when we decrease the data"}, {"start": 1888.7, "end": 1898.7, "text": " And again I'd be super happy to see the same results with a SAM objective and how the SAM influences this behavior"}, {"start": 1898.7, "end": 1910.7, "text": " On the other hand they have a much smaller model here so remember 32 means it's a 32 by 32 patch which means we have a smaller less tokens and thus a smaller transformer the VAT transformer"}, {"start": 1910.7, "end": 1919.7, "text": " And we can see that similarity is much more robust with the decrease of data here compared to this transformer"}, {"start": 1919.7, "end": 1929.7, "text": " And yeah I guess it has to do with the overfitting for a bigger model you simply need more data to kind of fine tune those weights to train those weights"}, {"start": 1929.7, "end": 1937.7, "text": " Finally not much new information here basically the more data you have the better the test accuracy will be in this linear probe setting"}, {"start": 1937.7, "end": 1944.7, "text": " Yeah that was pretty much it so obviously this paper is a bit different there is not a single core idea they're trying to validate"}, {"start": 1944.7, "end": 1954.7, "text": " What they did here instead was dissecting the VAT every single component I really like the thoroughness of this paper the experiments all of the experiments they did"}, {"start": 1954.7, "end": 1962.7, "text": " I guess we knew many of these information from the maybe from some of the previous papers including the original VAT paper"}, {"start": 1962.7, "end": 1972.7, "text": " So the shallower layers are making use of the fact that they can attend the whole image fairly successfully so that we have a mix of local and global attention in the shallower layers"}, {"start": 1972.7, "end": 1982.7, "text": " And we saw that in the original paper as well so we can see I guess this is the interesting part the features are kind of uniform across the whole model to just recap the whole thing"}, {"start": 1982.7, "end": 1993.7, "text": " VAT learns features much earlier compared to ResNets which means there is this additional expressivity that is left unthemed"}, {"start": 1993.7, "end": 2002.7, "text": " Again this is the same thing as in the original transformer as I said we have both global and local attention in the earlier layers"}, {"start": 2002.7, "end": 2013.7, "text": " And that kind of deteriorates we have less data if you're not using SAM objective if you're just training VAT the classical way without SAM"}, {"start": 2013.7, "end": 2026.7, "text": " So by cross entropy in the case of classification we can see that we don't have those like CNN like behavior emerging in these lower layers we only have global attention here"}, {"start": 2026.7, "end": 2039.7, "text": " So yeah some like experiments in receptive field and yeah this phase information is interesting the fact that we need skip connections"}, {"start": 2039.7, "end": 2049.7, "text": " The fact that tokens learn the spatial information and preserve it throughout the whole network except when they're using the global pooling"}, {"start": 2049.7, "end": 2057.7, "text": " So but like when you're training with the VAT with CLS token then the tokens kind of learn to preserve the spatial information"}, {"start": 2057.7, "end": 2065.7, "text": " Yeah I guess that's pretty much it for this video so if you found it useful consider subscribing share it with a friend"}, {"start": 2065.7, "end": 2073.7, "text": " Also I encourage you to join the discord community I strongly believe it's going to become a really vibrant community over the next couple of months"}, {"start": 2073.7, "end": 2079.7, "text": " Also subscribe to the AI monthly newsletter and until next time bye bye"}]