DINO: Emerging Properties in Self-Supervised Vision Transformers | Paper Explained!
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I cover DINO (self DIstillation with NO labels) introduced in the "Emerging Properties in Self-Supervised Vision Transformers" paper by Facebook AI.

The idea is to see whether using supervised learning was preventing transformers from showing the same kind of results in CV as they demonstrated in the NLP world (where we use self-supervised learning objectives such as (masked) language modeling).

It turns out some nice properties emerge such as:
* DINO-ViT learns to predict segmentation masks
* features are especially of high quality for the k-NN classification

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2104.14294
✅ Code: https://github.com/facebookresearch/dino
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 

00:00 DINO main ideas, attention maps explained
05:05 DINO explained in depth
10:30 Pseudocode walk-through
13:55 Multi-crop and local-to-global correspondence
15:15 More details on the teacher network
19:00 Results
25:00 Ablations
27:40 Collapse analysis
30:40 Features visualized and outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️


Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#dino #facebook #selfsupervised
 What's cracking guys? In this video I'm covering emerging properties in self-supervised vision transformers paper introducing Dino model from Facebook AI research, INRIA and Sorbonne University. The main motivation behind the paper was investigating why haven't transformers brought such an like a competitive advantage to computer vision as they did in NLP world and the main hypothesis is that using supervised learning as we've have so far with vision transformers have been kind of damping down their performance and so they propose to use self-supervised learning which provides us with a richer training signal and so they mentioned it here nicely. The resulting VAT are competitive with comnets but they have not yet delivered clear benefits over them. They're computationally more demanding, require more training data and their features do not exhibit unique properties. So on a small tangent this part of requiring more training data is not correct anymore so I recorded a video where I explained this novel technique where they use sharpness aware minimization objective to kind of smoothen out the lost landscape of VATs and thus it requires much less data now so yeah that part is not true anymore. But anyways, returning back to the paper, so the reason they want to use self-supervised learning is first because labeling is expensive and secondly because it's a richer training signal because suppose you have a sentence and if you want to predict just a single label for that sentence that's like a poor training signal compared to just trying to predict every single token which is a language modeling objective and so yeah that's the idea. Let's kind of try and use that self-supervised learning and see whether we get some like new properties popping up for transformers like for vision transformers and they say here that in this paper we question if self-supervised learning provides new properties to VATs that stand out compared to CNNs and as we'll soon see that is the case i.e. they do have some properties which even when they use the same training procedure so dyno procedure for comnets they do not achieve the same they do not gain the same properties as VAT. So first of those properties are these segmentation masks. As you see these are the attention maps that we get from the last layer of VAT after we are training with dyno and you can see that the objects are segmented which are salient for us humans. So we see a bird here, we see a boat here, we see a bicycle, toothbrush etc and that's not something we explicitly told a VAT to predict and so it's just an emergent property. And if you're, by the way, if you're not under if you don't understand how these are calculated let me just kind of briefly walk you through that part. They see here we look at the self-attention of the CLS token on the heads of the last layer. So again I recorded a video on vision transformer so if you want to know more details you can check it out. I'm going to briefly kind of explain here. So basically it's a transformer so you have a transformer like blob here and what you do in order to kind of parse images is so first you have your CLS token which is a learnable token and then what you do is you kind of segment your image so you kind of add these patches you kind of create all of these patches and then you for example take it to take these patches in a raster order you flatten them out and you kind of put them here okay and so now what happens is how they actually calculate this attention is the how they calculate how they made this attention map is the following so you kind of process all of these tokens using transformer and then out comes a representation of this CLS token and what you do is and also obviously for every single token here you'll have a corresponding output representation and they just take a this this token here they create a query vector so this will be query and they take these tokens and they just kind of map them to key vectors so we'll have key one here we'll have key two here etc so same here and just basically doing dot product between this query so that corresponds to the CLS token and these keys you'll get scores and then you'll just take these scores so you'll have a you'll get a score for this one you'll get a score for this one and you just take that flattened out structure and you wrap it back into the matrix structure and that's how you get these like these these images so basically these are just scores of these dot products between queries and keys pretty much so that's as simple as that now that was the that was a one of the emergent properties the second one that is super important is that the features that come out of the vision transformer trained by Dino have they perform really well using this k nearest neighbor classification and we'll see that a bit later but those are the two properties that emerge and now let me slowly start explaining the dino architecture itself on a high level what you do you have an input image x and you kind of do different augmentations so you do geometric augmentations such as crop you do photometric augmentations such as color jitter and you apply one set of those augmentations and you get x1 and you apply a distinct set of augmentations and you get x2 okay so now you pass those augmentations through these networks which are so this one will be just vision transformer so that's the whole point of this paper right we have vision transformer we want to kind of test it how it works with self-supervised learning and you form you form the teacher weights by kind of using this ema procedure so exponentially moving average so you'll be taking uh past snapshots of of student uh weights and you'll be using them to form the teacher weights and so now just keeping it on high level what happens here we will we'll output some distribution here which will be a categorical distribution i'm just going to draw it as continuous one because it's easier but it's softmax it's categorical and the whole point will be to kind of match these two distributions so we'll have one coming from the teacher as well and the whole goal will be to kind of make sure these two are the same and why is that so the intuition behind this procedure is so no matter how you kind of deform how you augment your image uh you want to make sure that the student and teacher learn how to extract such representations so that they are invariant to those augmentations so again even though these two uh images uh are completely different because they are different crops different augmentations we'll end up with the same representations because we'll have same distributions and so that means we're extracting relevant information from the image that's the whole point and those features turn out to be super valuable on downstream tasks now going in a bit more detail first how we enforce the uh the these two distributions to be the same is via cross entropy loss so that's this thing here secondly uh important details here is we're using a temperature coefficient here in softmax if you're not familiar with the temperature does is basically let's say you have a distribution something like this and basically so this will be your mode of the distribution if you apply now the temperature and if we let the temperature go down to zero we'll basically end up with something like this so we'll have a probability of one for this mode and everything else will drop down to zero that's also called sharpening if you see that expression now you know what it is basically as you drop down the temperature you'll be sharpening distribution so that only the mode peaks and everything else drops down so that's the idea there uh and on the other hand we have this this centering which is vital for this method otherwise we'd got we would get a like a collapse of the representations and i'll explain those in a bit but basically um we need uh we need these the centering to to avoid collapse and um so briefly like what collapses is um the the easiest thing the easiest thing this this this network could do the system could do in order to make sure that they're the output distributions are the same is to just kind of output the uniform distribution for every single input so they'll just kind of learn to output something like this so this is your distribution and they'll just learn how to output the uniform one and this one will also learn how to output the uniform one and then trivially the loss goes to zero but that's obviously not what we want because that's that's a collapse a second type of collapse would be uh if you instead have just uh like a peak on a specific uh position on specific dimension so here for example you having a peak and on the same position from the teacher network as well so these are two types of collapses that happen and in order to avoid them what they do is they apply this centering and what centering is is you um just basically um so uh let me start like this we have image here and it's usually a batch of images so we'll have a like a b number of these images and so the vector here the tensor here will be b comma k okay where k is just the dimensionality of the representation and we have b of those so what to do is you'll you'll be calculating this c vector so the centering vector by just kind of taking a mean from all of these representations so just take a mean over over over the b dimension over the zero dimension and so you'll end up with a k one vector and you'll additionally be using exponential moving average to calculate this c vector so that means that also the previous batches will be influencing the value of this c vector and then what you do is you just subtract c from all of these uh b representations and then you pass them through softmax that also has temperature coefficient uh second important detail here is this stop gradient operator so that basically means we'll be propagating uh the loss uh like uh the errors through the student network only because that makes sense teacher is only is formed by using this exponentially moving average uh that's that's it hopefully uh that made that made sense um now i'm gonna walk you through the pseudocode that will maybe help you further understand this uh and yeah so here it is we have the teacher network we have the student network we iterate through the data set so x is just a batch of images you apply some augmentations here you apply a different set of augmentations here you end up with x1 and x2 okay so those are two batches with different augmentations applied to them uh then you do a feed forward through the student network and you do a feed forward through the teacher network and you end up with these logits here okay and finally uh what you do here you can see it's kind of symmetric so we're using t1 which is a teacher logit from the this x1 batch of augmented images and we use this uh s2 which is us which which came from this second group of augmented images uh being feed through the the student network so it's basically just symmetry stuff um so let's see let's see how how h is implemented basically here we can see t gets detached so the logits of the teacher gets detached and that's just an implementation of the stop gradient in pytorch uh then we apply softmax and as you can see here so tps is just a temperature parameter for the student um that that's that's what makes it sharper and uh then we just apply softmax and that's the the output distribution we had here so that that's this one and on the other hand we had here we we have the c vector so we'll be subtracting it as i said so that's the centering part and then we have then we apply the sharpening part so that's the this part here and we get t which is the output distribution so that the categorical distribution that came out of from teacher and finally we just apply as you can see here cross entropy and we do a mean because it's a batch of different uh images okay um that's pretty much it hopefully this helped a bit backward we'll calculate the actual gradients for all of the weights in the network update just updates the student network as you see so we only update student network and then here are the two exponentially moving average updates one updates the teacher network as you can see here l times the teacher network's parameters and y minus l times the student network parameters uh on the other hand we have here the the centering vector so t1 and t2 remember those are just logits so these are just logits so we have the these are bk tensors and they'll just concatenate those so that means they'll form basically if you have uh let's let's make it like this so we have b here and k is here okay so that's t1 and then we have t2 which they'll just concatenate here and then they do like a simple mean across this dimension okay and that's how they get the the the the the c vector plus they have as you see the exponentially moving part so they'll be using c's from other batches from the previous batches that's how they form the c vector and that's it in a nutshell that's those are all of the nitty gritty details um but again the main picture is very very simple the idea is to form such representations so that we have the same or similar distributions at the output and uh like it doesn't matter what kind of augmentations we applied those are still the same okay um a couple more interesting points to make here is uh they mentioned here that all crops are passed through the student while only the global views are passed through the teacher therefore encouraging local to global correspondences so that means the following so let's say we have an image here and we have let me draw something inside like maybe a chat programmer okay like a guy with glasses he's buffed okay like this and what you'll do is some of the crops will be uh maybe uh consisting out of more than 50 of the image something like this some of those crops may be smaller maybe like taking only one part one smaller part of the image and you'll be passing the big ones through the that's why they say here you'll be passing the big ones through the teaching network and you'll be passing both types of crops through the student network and again the idea is if you can kind of infer from this small crop if you can infer uh like relevant information that contains the same information as this bigger crop uh that means you you got yourself a robust representation okay and that's the whole idea that this this uh uh like multi-crop strategy enforces uh robust representations which turn out to to to do quite well on those k and evaluations as we'll soon see now let's see some more details about the teacher network here uh they mentioned that unlike knowledge distillation we do not have a teacher uh given a priority and hence we build it from past iterations of the student network so as we saw we were using that uh exponential moving average to form the teacher network so usually in computer vision you have a big uh like cnn maybe something like this so that's a cnn and you train it on a data set maybe something like image net and the whole idea is now having trained that one you freeze the weights and then you have a smaller cnn something like this and so what to do is the following so you take an image from that data set you feed it both here into the big one as well as to your smaller yet to be trained cnn and now instead of using the one hot uh encoding of the label as the target you'll actually be using whatever the the output distribution from this one is so whatever the output distribution here is we'll want to mimic in the smaller one so here we'll want to mimic that one uh by just applying uh k l divergence or or whatnot uh so that's the usual knowledge distillation and here we are not using exactly that because as you as you saw the teacher network is not frozen we're just updating it uh exponentially as the time goes by okay that's that's the first detail uh secondly they they mentioned here that indeed we observed that this teacher performs a form of model in sampling similar to polyac rupert averaging with an exponential decay uh that leads us to this thing and that's we observed that this teacher has better performance than the student throughout the training and hence guides the training of the student by providing target features of higher quality uh so yeah i mean just doing and sampling is a is a is a known method where whereby you can improve the the performance of your model by just taking a bunch of models and then averaging their outputs but here we are doing the averaging in the in the parameter space and that also uh helps obviously uh increase the performance we'll see some curves later on where they quantitatively show that this does happen indeed so that the teacher is always better than the student and so students can can actually learn something from the teacher because if yeah yeah i guess that makes sense um okay so a couple more details worth mentioning is that this whole like system this dino system will be uh free of bet normalization which is a trend we've seen recently with nf nets if you if you if you saw that paper uh it came from deep mind the idea was to kind of ditch the bet normalization because it has some nasty properties um and yeah but it ended up um complicating the whole procedure so i'm not sure it uh it got so popular but yeah still um just if you're able to kind of ditch bet normalization that that's cool and here they managed it because uh they say here that therefore when applying dino to vat we do not use any bet normalization in the projection hats uh vat does not have bn by default and they just made sure not to add bn layers uh on top of those linear layers in the projection head okay i already mentioned that these centering and sharpening operations are vital uh and they say here so as shown experimentally centering prevents one dimension to dominate but encourages collapse to the uniform distribution so that's this first type of uh like collapse i mentioned the uniform one so this one and then they mention um while the sharpening has the opposite effect okay so it kind of makes those uh one dimension explode so applying both operations balances their effects which is sufficient to avoid collapse in presence of a momentum teacher okay that's that's it those are those are all of the details of the architecture and how the system works now let's see for the results um first what they do is they take uh resnet as the as the as the baseline and they train it using dino so this is this method here and they show that uh compared to all of the previous uh like self-supervised methods such as i don't know like bootstrap your own latence bjoll or or moco or or simclear etc so they show that they are they perform better using both types of evaluations so the first one is linear one and the second one is k and n evaluation and just uh briefly if you don't understand how how the k and n works it's fairly simple so what you do is you train your your vat so you train your vat here and then you take uh you basically take uh your training set so this is your training set you freeze these weights so these are frozen so let me just you know it like f you froze these and now you just input an image outcomes are representation and because it's from the training set you'll have a label associated with it and you'll you'll just repeat this for a subset of your training set and you'll store these representation label tuples okay and next thing how you actually evaluate now this system is the following so now you have now you have uh like a validation data set and you pass in an image from the validation set outcomes representation but this time you don't have a label so what you do instead is you search through these representations which you've collected from your training set and you just find the k nearest ones and they were using like 20 uh experimentally they found that number works well so you'll just find 20 closest representations and you'll see what their labels are and by majority voting for example you'll just figure out what the represent what the label should be for this one okay that's the that's the whole idea so you just find the closest ones where closeness is defined by i guess they used uh l2 metric and so yeah that that's it that's how you find a label for for this one you just find the closest ones okay and now the interesting part actually here is that uh once you have vit you can see that the knn performance is like the gap between these two is severely like uh decreased whereas here we had a bigger gap that's the the important thing to notice is that using transformers and ssl has something that's that's um qualitatively different comparing to using cnn's that's the first thing worth mentioning and the yeah in general they compare with other baselines so here so here they compensate for the architecture here they use whatever architecture just the best methods out there and they still show that uh using linear evaluation as well as the knn they they achieve the best results compared to previous like bjoll simclear etc previous ssl baselines uh those are the initial results now let's see some other stuff here so these two results just support their claim that the features are really of high quality and since they perform really well for k nearest neighbor classification they also should perform well for image retrieval and copy detection and they just kind of uh yeah show that that's the case uh interesting thing here is that uh cool thing about ssl methods is you can use them on whatever data set you have you don't have to have labels and so they just pre-trained dino on this google landmarks uh data set which turns out to to boost performance severely compared to just pre-training dino on on image net and yeah they get results which are comparable to even a supervised baseline and that's cool okay let's continue here and that was the that was the first part so those were the the the that was the emergent property of nice k features that are good for knn focusing on these results here we can see that um different colors just you know different hats of the transformer heads attention heads from the vat uh final layer and we can see that the red head focuses on horses on this zebra's head we can see that the yellow head is focusing on on on the neck as well as on the ears and the blue hat is focusing on on this bonds here okay and um what's interesting is they show here that um even some some occluded objects such as these bushes here are detected and segmented uh i mean it's hard even for me to kind of detect these and the network learns how to recognize the bush and segment that information okay that's super cool okay continuing further they showed that if you take supervised baseline and plot the segmentation maps from from that one and from dino on on different data sets on pascal so they pre-trained this on image net so both dino and the supervised method and then they test it on pascal data set and you can see there's bunch of random like sparse dots here so the segmentation masks are not of the same quality as as as in dino and they also show quantitatively that you can see that this is i think uh how they called it uh jacquard similarity or iou just a fancy name for intersection of a union they show it's much higher compared to supervised baseline so again this testifies that um uh the the dino the vat with dino learns how to extract these segmentation masks uh high quality segmentation masks okay uh in this table they show a general applicability of these features to various downstream benchmarks such as cipher 10 cipher 100 etc so just comparing dino with the supervised baseline we can see it pretty much achieves outperforms supervised baseline on all of these downstream tasks which further testifies that the features learned using dino and vat are of super high quality uh some ablations here uh maybe the most important row i want you to focus on is when we uh expel when we don't use the momentum encoder so that's the teacher network being formed from exponential like by the exponential moving average strategy we can see that it completely fails to train so the performance just drops down to almost zero and it does not work so we need we so this method so dino needs momentum encoder and yeah um other ablations like uh multi crop not using multi crop reduces result reduces the performance not using cross entropy reduces performance etc in this plot they show that not that using smaller patches is more important than using a bigger model so if you focus on on this thing here so if you focus on on the eight by eight uh patches and vat small uh as you can see it has better performance than using um vat big so the bigger model but with 16 by 16 patches so the trade-off here obviously exists and uh it's the following basically when you're using eight by eight you need you need a bigger memory footprint and so uh you also kind of uh reduce the throughput of number of images you can feed through the model in a second um but yeah you get some additional performance but all in all you can see these two curves overlap so basically it boils down to how much performance you want and what's the throughput your desired throughput and that's how you can pick the best trade-off between the size of the model and the patch size okay nice um here is the quantitative results i mentioned that the student is always so that the teacher is always better than student you can see as the epochs progress during the training the validation accuracy of the teacher is always higher compared to the student and they just kind of try different strategies of how they update the teacher and it turns out that using momentum is the best strategy using previous epoch so that's something similar to dqn if you watch my video on dqn or you know what dqn is it also uses it also freezes the target network and updates it every couple of epochs or whatever every every end steps in general and so that method also works but it yields somewhat worse performance and using updating the teacher too often so that will be previous iteration or student copy you can see that the method just fails all completely so yeah there are ways to kind of build up this teacher network but the best thing seems to be so far this momentum update these results are pretty interesting let me just kind of explain them again there are two forms of collapse regardless of the input the model output is uniform along all dimensions or dominated by one dimension if we take the cross entropy we saw in the loss so we take the cross entropy loss and kind of split it down into the entropy component and the KL divergence component and plot it here they show the following so if you're using sharpening only and you just ignore you you kind of ditch the centering part in the teacher network this is what happens basically the entropy drops down to zero which means what which means that the network learns to output like something like this you basically have a peak at one dimension and the probabilities one and everything else is zero and so the entropy of this thing is zero so that means that using only sharpening leads to this mode of collapse and using only centering also collapses but as you can see the collapse has a non-zero value so the the the the entropy has non-zero value which means we have something like this which means we have a uniform distribution and this has a non-zero entropy and so that's what we end up with here and it did mention somewhere here let me just check the entropy converges to different values so zero with no centering and minus log one of over k with no sharpening so that's that's the value that we'll have here so here this value will be minus log of one over k the reason being is the support has k values so this is distribution support is has k dimensions so that means this will be one over k so that the integral amounts to one and this means if you just calculate the entropy you'll end up with with this this value here so yeah using both as you can see here the entropy converges to some value but it's neither zero neither this value of minus log one over k and that's it that's pretty much it taking a look at the KL divergence the KL becomes zero for these two cases because both the student and the teacher learn to output either uniform or this kind of distribution and when we're using both then KL divergence does not collapse to zero those are just two different perspectives on the same thing and that's the fact that they collapse to either of these modes depending on what what do we omit either sharpening or centering okay that's that's pretty much it i think i explained everything i wanted finally they mentioned it here in the future we plan to explore if pre-training a large vat model with dino on random uncurated images could push the limits of visual features and i can just imagine something along the gpt3 dimensions being trained and this time with vision transformer so that will be exciting i guess final thing i want to show you are the features that like the some semantics of the features of vat trained with dino as you can see here just extracting those representations and then using t-sneed to kind of plot them in 2d you can see that dino vat with dino learns how to cluster similar objects here we can see like model t that's a car then car wheel ambulance minibus so we have some kind of vehicles cluster here and then we have some like gutter snake king snake ring neck snakes we have some snakes here we have like orangutan some monkeys here so all in all all of these emerge by training dino by training vat using dino so we get this nice property even though we train vat only using ssl objective and we also got those segmentation masks and we also got the features which are of high quality as shown by knn performance etc so hopefully you like this video if you did consider subscribing and sharing and until next time bye bye
[{"start": 0.0, "end": 4.24, "text": " What's cracking guys? In this video I'm covering emerging properties in self-supervised vision"}, {"start": 4.24, "end": 12.16, "text": " transformers paper introducing Dino model from Facebook AI research, INRIA and Sorbonne University."}, {"start": 12.16, "end": 19.28, "text": " The main motivation behind the paper was investigating why haven't transformers brought"}, {"start": 19.28, "end": 25.76, "text": " such an like a competitive advantage to computer vision as they did in NLP world and the main"}, {"start": 25.76, "end": 31.28, "text": " hypothesis is that using supervised learning as we've have so far with vision transformers have"}, {"start": 31.28, "end": 38.72, "text": " been kind of damping down their performance and so they propose to use self-supervised learning"}, {"start": 39.44, "end": 45.52, "text": " which provides us with a richer training signal and so they mentioned it here nicely. The resulting"}, {"start": 45.52, "end": 51.760000000000005, "text": " VAT are competitive with comnets but they have not yet delivered clear benefits over them."}, {"start": 51.76, "end": 57.44, "text": " They're computationally more demanding, require more training data and their features do not"}, {"start": 57.44, "end": 62.48, "text": " exhibit unique properties. So on a small tangent this part of requiring more training data is not"}, {"start": 62.48, "end": 68.08, "text": " correct anymore so I recorded a video where I explained this novel technique where they use"}, {"start": 68.08, "end": 74.88, "text": " sharpness aware minimization objective to kind of smoothen out the lost landscape of VATs and thus"}, {"start": 75.6, "end": 81.44, "text": " it requires much less data now so yeah that part is not true anymore. But anyways, returning back"}, {"start": 81.44, "end": 85.52, "text": " to the paper, so the reason they want to use self-supervised learning is first because"}, {"start": 85.52, "end": 90.64, "text": " labeling is expensive and secondly because it's a richer training signal because suppose you have"}, {"start": 90.64, "end": 95.92, "text": " a sentence and if you want to predict just a single label for that sentence that's like a"}, {"start": 95.92, "end": 100.72, "text": " poor training signal compared to just trying to predict every single token which is a language"}, {"start": 100.72, "end": 107.03999999999999, "text": " modeling objective and so yeah that's the idea. Let's kind of try and use that"}, {"start": 107.04, "end": 111.68, "text": " self-supervised learning and see whether we get some like new properties popping up for"}, {"start": 111.68, "end": 116.96000000000001, "text": " transformers like for vision transformers and they say here that in this paper we question if"}, {"start": 116.96000000000001, "end": 121.84, "text": " self-supervised learning provides new properties to VATs that stand out compared to CNNs and as"}, {"start": 121.84, "end": 128.88, "text": " we'll soon see that is the case i.e. they do have some properties which even when they use the same"}, {"start": 128.88, "end": 135.44, "text": " training procedure so dyno procedure for comnets they do not achieve the same they do not gain the"}, {"start": 135.44, "end": 142.64, "text": " same properties as VAT. So first of those properties are these segmentation masks. As you see these are"}, {"start": 142.64, "end": 148.24, "text": " the attention maps that we get from the last layer of VAT after we are training with dyno and you can"}, {"start": 148.24, "end": 153.28, "text": " see that the objects are segmented which are salient for us humans. So we see a bird here,"}, {"start": 153.28, "end": 159.76, "text": " we see a boat here, we see a bicycle, toothbrush etc and that's not something we explicitly"}, {"start": 159.76, "end": 168.39999999999998, "text": " told a VAT to predict and so it's just an emergent property. And if you're, by the way, if you're not"}, {"start": 168.39999999999998, "end": 173.04, "text": " under if you don't understand how these are calculated let me just kind of briefly walk you"}, {"start": 173.04, "end": 179.28, "text": " through that part. They see here we look at the self-attention of the CLS token on the heads of"}, {"start": 179.28, "end": 184.64, "text": " the last layer. So again I recorded a video on vision transformer so if you want to know more"}, {"start": 184.64, "end": 189.12, "text": " details you can check it out. I'm going to briefly kind of explain here. So basically it's a transformer"}, {"start": 189.12, "end": 196.72, "text": " so you have a transformer like blob here and what you do in order to kind of parse images is so first"}, {"start": 196.72, "end": 203.68, "text": " you have your CLS token which is a learnable token and then what you do is you kind of segment your"}, {"start": 203.68, "end": 209.68, "text": " image so you kind of add these patches you kind of create all of these patches and then you for"}, {"start": 209.68, "end": 214.4, "text": " example take it to take these patches in a raster order you flatten them out and you kind of put"}, {"start": 214.4, "end": 221.20000000000002, "text": " them here okay and so now what happens is how they actually calculate this attention is the how they"}, {"start": 221.20000000000002, "end": 226.24, "text": " calculate how they made this attention map is the following so you kind of process all of these"}, {"start": 226.24, "end": 232.24, "text": " tokens using transformer and then out comes a representation of this CLS token and what you do"}, {"start": 232.24, "end": 237.92000000000002, "text": " is and also obviously for every single token here you'll have a corresponding output representation"}, {"start": 237.92, "end": 244.72, "text": " and they just take a this this token here they create a query vector so this will be query and"}, {"start": 244.72, "end": 250.48, "text": " they take these tokens and they just kind of map them to key vectors so we'll have key one here"}, {"start": 250.48, "end": 258.56, "text": " we'll have key two here etc so same here and just basically doing dot product between this query"}, {"start": 258.56, "end": 265.03999999999996, "text": " so that corresponds to the CLS token and these keys you'll get scores and then you'll just take"}, {"start": 265.04, "end": 269.76000000000005, "text": " these scores so you'll have a you'll get a score for this one you'll get a score for this one and"}, {"start": 269.76000000000005, "end": 274.56, "text": " you just take that flattened out structure and you wrap it back into the matrix structure and"}, {"start": 274.56, "end": 280.32000000000005, "text": " that's how you get these like these these images so basically these are just scores of these dot"}, {"start": 280.32000000000005, "end": 286.64000000000004, "text": " products between queries and keys pretty much so that's as simple as that now that was the that"}, {"start": 286.64000000000004, "end": 291.68, "text": " was a one of the emergent properties the second one that is super important is that the features"}, {"start": 291.68, "end": 297.6, "text": " that come out of the vision transformer trained by Dino have they perform really well using this"}, {"start": 297.6, "end": 302.64, "text": " k nearest neighbor classification and we'll see that a bit later but those are the two properties"}, {"start": 302.64, "end": 311.28000000000003, "text": " that emerge and now let me slowly start explaining the dino architecture itself on a high level what"}, {"start": 311.28000000000003, "end": 317.68, "text": " you do you have an input image x and you kind of do different augmentations so you do geometric"}, {"start": 317.68, "end": 324.24, "text": " augmentations such as crop you do photometric augmentations such as color jitter and you apply"}, {"start": 324.24, "end": 329.36, "text": " one set of those augmentations and you get x1 and you apply a distinct set of augmentations and you"}, {"start": 329.36, "end": 335.2, "text": " get x2 okay so now you pass those augmentations through these networks which are so this one will"}, {"start": 335.2, "end": 339.84000000000003, "text": " be just vision transformer so that's the whole point of this paper right we have vision transformer"}, {"start": 339.84000000000003, "end": 345.76, "text": " we want to kind of test it how it works with self-supervised learning and you form you form"}, {"start": 345.76, "end": 351.12, "text": " the teacher weights by kind of using this ema procedure so exponentially moving average so"}, {"start": 351.12, "end": 357.59999999999997, "text": " you'll be taking uh past snapshots of of student uh weights and you'll be using them to form the"}, {"start": 357.59999999999997, "end": 363.92, "text": " teacher weights and so now just keeping it on high level what happens here we will we'll output some"}, {"start": 363.92, "end": 367.76, "text": " distribution here which will be a categorical distribution i'm just going to draw it as"}, {"start": 367.76, "end": 374.08, "text": " continuous one because it's easier but it's softmax it's categorical and the whole point will be to"}, {"start": 374.08, "end": 379.35999999999996, "text": " kind of match these two distributions so we'll have one coming from the teacher as well and the"}, {"start": 379.35999999999996, "end": 384.8, "text": " whole goal will be to kind of make sure these two are the same and why is that so the intuition"}, {"start": 384.8, "end": 390.96, "text": " behind this procedure is so no matter how you kind of deform how you augment your image uh you want"}, {"start": 390.96, "end": 398.56, "text": " to make sure that the student and teacher learn how to extract such representations so that they"}, {"start": 398.56, "end": 404.88, "text": " are invariant to those augmentations so again even though these two uh images uh are completely"}, {"start": 404.88, "end": 409.04, "text": " different because they are different crops different augmentations we'll end up with the same"}, {"start": 409.04, "end": 414.56, "text": " representations because we'll have same distributions and so that means we're extracting"}, {"start": 414.56, "end": 418.64, "text": " relevant information from the image that's the whole point and those features turn out to be"}, {"start": 418.64, "end": 424.88, "text": " super valuable on downstream tasks now going in a bit more detail first how we enforce the uh the"}, {"start": 424.88, "end": 428.71999999999997, "text": " these two distributions to be the same is via cross entropy loss so that's this thing here"}, {"start": 429.6, "end": 435.6, "text": " secondly uh important details here is we're using a temperature coefficient here in softmax if you're"}, {"start": 435.6, "end": 440.24, "text": " not familiar with the temperature does is basically let's say you have a distribution something like"}, {"start": 440.24, "end": 446.71999999999997, "text": " this and basically so this will be your mode of the distribution if you apply now the temperature"}, {"start": 447.68, "end": 452.08, "text": " and if we let the temperature go down to zero we'll basically end up with something like this so"}, {"start": 452.08, "end": 458.47999999999996, "text": " we'll have a probability of one for this mode and everything else will drop down to zero that's also"}, {"start": 458.47999999999996, "end": 462.96, "text": " called sharpening if you see that expression now you know what it is basically as you drop down the"}, {"start": 462.96, "end": 467.59999999999997, "text": " temperature you'll be sharpening distribution so that only the mode peaks and everything else drops"}, {"start": 467.59999999999997, "end": 474.64, "text": " down so that's the idea there uh and on the other hand we have this this centering which is vital"}, {"start": 474.64, "end": 480.88, "text": " for this method otherwise we'd got we would get a like a collapse of the representations and i'll"}, {"start": 480.88, "end": 488.88, "text": " explain those in a bit but basically um we need uh we need these the centering to to avoid collapse"}, {"start": 488.88, "end": 496.24, "text": " and um so briefly like what collapses is um the the easiest thing the easiest thing this this"}, {"start": 496.24, "end": 500.96, "text": " this network could do the system could do in order to make sure that they're the output distributions"}, {"start": 500.96, "end": 505.52, "text": " are the same is to just kind of output the uniform distribution for every single input so they'll"}, {"start": 505.52, "end": 510.0, "text": " just kind of learn to output something like this so this is your distribution and they'll just learn"}, {"start": 510.0, "end": 515.84, "text": " how to output the uniform one and this one will also learn how to output the uniform one and then"}, {"start": 515.84, "end": 521.28, "text": " trivially the loss goes to zero but that's obviously not what we want because that's that's a collapse"}, {"start": 521.28, "end": 528.24, "text": " a second type of collapse would be uh if you instead have just uh like a peak on a specific"}, {"start": 528.24, "end": 535.52, "text": " uh position on specific dimension so here for example you having a peak and on the same position"}, {"start": 535.52, "end": 541.28, "text": " from the teacher network as well so these are two types of collapses that happen and in order to"}, {"start": 541.28, "end": 547.52, "text": " avoid them what they do is they apply this centering and what centering is is you um just"}, {"start": 547.52, "end": 554.4, "text": " basically um so uh let me start like this we have image here and it's usually a batch of images so"}, {"start": 554.4, "end": 559.76, "text": " we'll have a like a b number of these images and so the vector here the tensor here will be b"}, {"start": 559.76, "end": 567.12, "text": " comma k okay where k is just the dimensionality of the representation and we have b of those so what"}, {"start": 567.12, "end": 572.8, "text": " to do is you'll you'll be calculating this c vector so the centering vector by just kind of taking a"}, {"start": 572.8, "end": 578.72, "text": " mean from all of these representations so just take a mean over over over the b dimension over"}, {"start": 578.72, "end": 586.64, "text": " the zero dimension and so you'll end up with a k one vector and you'll additionally be using"}, {"start": 586.64, "end": 592.24, "text": " exponential moving average to calculate this c vector so that means that also the previous batches"}, {"start": 592.24, "end": 597.68, "text": " will be influencing the value of this c vector and then what you do is you just subtract c from all"}, {"start": 597.68, "end": 603.12, "text": " of these uh b representations and then you pass them through softmax that also has temperature"}, {"start": 603.12, "end": 610.16, "text": " coefficient uh second important detail here is this stop gradient operator so that basically"}, {"start": 610.16, "end": 617.12, "text": " means we'll be propagating uh the loss uh like uh the errors through the student network only"}, {"start": 617.12, "end": 621.68, "text": " because that makes sense teacher is only is formed by using this exponentially moving average"}, {"start": 622.7199999999999, "end": 628.8, "text": " uh that's that's it hopefully uh that made that made sense um now i'm gonna walk you through the"}, {"start": 628.8, "end": 633.6, "text": " pseudocode that will maybe help you further understand this uh and yeah so here it is"}, {"start": 634.48, "end": 638.48, "text": " we have the teacher network we have the student network we iterate through the data"}, {"start": 638.48, "end": 642.96, "text": " set so x is just a batch of images you apply some augmentations here you apply a different set of"}, {"start": 642.96, "end": 648.16, "text": " augmentations here you end up with x1 and x2 okay so those are two batches with different"}, {"start": 648.16, "end": 653.12, "text": " augmentations applied to them uh then you do a feed forward through the student network and you"}, {"start": 653.12, "end": 656.8000000000001, "text": " do a feed forward through the teacher network and you end up with these logits here okay"}, {"start": 657.52, "end": 663.28, "text": " and finally uh what you do here you can see it's kind of symmetric so we're using t1 which is a"}, {"start": 663.28, "end": 671.8399999999999, "text": " teacher logit from the this x1 batch of augmented images and we use this uh s2 which is us which"}, {"start": 671.8399999999999, "end": 677.52, "text": " which came from this second group of augmented images uh being feed through the the student"}, {"start": 677.52, "end": 683.92, "text": " network so it's basically just symmetry stuff um so let's see let's see how how h is implemented"}, {"start": 683.92, "end": 689.68, "text": " basically here we can see t gets detached so the logits of the teacher gets detached and that's"}, {"start": 689.68, "end": 695.04, "text": " just an implementation of the stop gradient in pytorch uh then we apply softmax and as you can"}, {"start": 695.04, "end": 702.16, "text": " see here so tps is just a temperature parameter for the student um that that's that's what makes"}, {"start": 702.16, "end": 708.16, "text": " it sharper and uh then we just apply softmax and that's the the output distribution we had here so"}, {"start": 708.16, "end": 714.8, "text": " that that's this one and on the other hand we had here we we have the c vector so we'll be"}, {"start": 714.8, "end": 719.4399999999999, "text": " subtracting it as i said so that's the centering part and then we have then we apply the sharpening"}, {"start": 719.4399999999999, "end": 724.16, "text": " part so that's the this part here and we get t which is the output distribution so that the"}, {"start": 724.16, "end": 728.88, "text": " categorical distribution that came out of from teacher and finally we just apply as you can see"}, {"start": 728.88, "end": 735.76, "text": " here cross entropy and we do a mean because it's a batch of different uh images okay um that's"}, {"start": 735.76, "end": 740.7199999999999, "text": " pretty much it hopefully this helped a bit backward we'll calculate the actual gradients for all of"}, {"start": 740.72, "end": 746.0, "text": " the weights in the network update just updates the student network as you see so we only update"}, {"start": 746.0, "end": 750.88, "text": " student network and then here are the two exponentially moving average updates one updates"}, {"start": 750.88, "end": 757.12, "text": " the teacher network as you can see here l times the teacher network's parameters and y minus l"}, {"start": 757.12, "end": 763.0400000000001, "text": " times the student network parameters uh on the other hand we have here the the centering vector"}, {"start": 763.0400000000001, "end": 770.1600000000001, "text": " so t1 and t2 remember those are just logits so these are just logits so we have the"}, {"start": 770.16, "end": 777.6, "text": " these are bk tensors and they'll just concatenate those so that means they'll form basically if you"}, {"start": 777.6, "end": 786.0799999999999, "text": " have uh let's let's make it like this so we have b here and k is here okay so that's t1 and then"}, {"start": 786.0799999999999, "end": 795.68, "text": " we have t2 which they'll just concatenate here and then they do like a simple mean across this"}, {"start": 795.68, "end": 802.9599999999999, "text": " dimension okay and that's how they get the the the the the c vector plus they have as you see the"}, {"start": 802.9599999999999, "end": 808.88, "text": " exponentially moving part so they'll be using c's from other batches from the previous batches"}, {"start": 808.88, "end": 813.8399999999999, "text": " that's how they form the c vector and that's it in a nutshell that's those are all of the nitty"}, {"start": 813.8399999999999, "end": 820.88, "text": " gritty details um but again the main picture is very very simple the idea is to form such"}, {"start": 820.88, "end": 828.08, "text": " representations so that we have the same or similar distributions at the output and uh like it doesn't"}, {"start": 828.08, "end": 835.6, "text": " matter what kind of augmentations we applied those are still the same okay um a couple more"}, {"start": 835.6, "end": 840.64, "text": " interesting points to make here is uh they mentioned here that all crops are passed through the"}, {"start": 840.64, "end": 845.44, "text": " student while only the global views are passed through the teacher therefore encouraging local"}, {"start": 845.44, "end": 851.12, "text": " to global correspondences so that means the following so let's say we have an image here and"}, {"start": 851.12, "end": 857.0400000000001, "text": " we have let me draw something inside like maybe a chat programmer okay like a guy with glasses"}, {"start": 857.6800000000001, "end": 866.48, "text": " he's buffed okay like this and what you'll do is some of the crops will be uh maybe uh consisting"}, {"start": 866.48, "end": 872.8000000000001, "text": " out of more than 50 of the image something like this some of those crops may be smaller maybe like"}, {"start": 872.8, "end": 878.16, "text": " taking only one part one smaller part of the image and you'll be passing the big ones through the"}, {"start": 878.16, "end": 881.68, "text": " that's why they say here you'll be passing the big ones through the teaching network and you'll be"}, {"start": 881.68, "end": 886.9599999999999, "text": " passing both types of crops through the student network and again the idea is if you can kind of"}, {"start": 886.9599999999999, "end": 894.7199999999999, "text": " infer from this small crop if you can infer uh like relevant information that contains the same"}, {"start": 894.7199999999999, "end": 901.68, "text": " information as this bigger crop uh that means you you got yourself a robust representation okay"}, {"start": 901.68, "end": 908.3199999999999, "text": " and that's the whole idea that this this uh uh like multi-crop strategy enforces uh robust"}, {"start": 908.3199999999999, "end": 913.52, "text": " representations which turn out to to to do quite well on those k and evaluations as we'll soon see"}, {"start": 913.52, "end": 918.0, "text": " now let's see some more details about the teacher network here uh they mentioned that unlike"}, {"start": 918.0, "end": 924.16, "text": " knowledge distillation we do not have a teacher uh given a priority and hence we build it from"}, {"start": 924.16, "end": 928.88, "text": " past iterations of the student network so as we saw we were using that uh exponential moving average"}, {"start": 928.88, "end": 935.6, "text": " to form the teacher network so usually in computer vision you have a big uh like cnn maybe something"}, {"start": 935.6, "end": 940.8, "text": " like this so that's a cnn and you train it on a data set maybe something like image net"}, {"start": 941.76, "end": 948.4, "text": " and the whole idea is now having trained that one you freeze the weights and then you have a smaller"}, {"start": 948.4, "end": 955.36, "text": " cnn something like this and so what to do is the following so you take an image from that data set"}, {"start": 955.36, "end": 962.8000000000001, "text": " you feed it both here into the big one as well as to your smaller yet to be trained cnn and now"}, {"start": 962.8000000000001, "end": 970.8000000000001, "text": " instead of using the one hot uh encoding of the label as the target you'll actually be using"}, {"start": 970.8000000000001, "end": 974.88, "text": " whatever the the output distribution from this one is so whatever the output distribution here is"}, {"start": 975.6800000000001, "end": 981.76, "text": " we'll want to mimic in the smaller one so here we'll want to mimic that one uh by just applying"}, {"start": 981.76, "end": 988.96, "text": " uh k l divergence or or whatnot uh so that's the usual knowledge distillation and here we are not"}, {"start": 988.96, "end": 994.3199999999999, "text": " using exactly that because as you as you saw the teacher network is not frozen we're just updating"}, {"start": 994.3199999999999, "end": 1000.4, "text": " it uh exponentially as the time goes by okay that's that's the first detail uh secondly they they"}, {"start": 1000.4, "end": 1004.88, "text": " mentioned here that indeed we observed that this teacher performs a form of model in sampling"}, {"start": 1005.4399999999999, "end": 1010.56, "text": " similar to polyac rupert averaging with an exponential decay uh that leads us to this"}, {"start": 1010.56, "end": 1014.7199999999999, "text": " thing and that's we observed that this teacher has better performance than the student throughout"}, {"start": 1014.7199999999999, "end": 1019.04, "text": " the training and hence guides the training of the student by providing target features of higher"}, {"start": 1019.04, "end": 1026.0, "text": " quality uh so yeah i mean just doing and sampling is a is a is a known method where whereby you can"}, {"start": 1026.0, "end": 1029.9199999999998, "text": " improve the the performance of your model by just taking a bunch of models and then averaging"}, {"start": 1030.48, "end": 1036.32, "text": " their outputs but here we are doing the averaging in the in the parameter space and that also uh"}, {"start": 1036.32, "end": 1040.96, "text": " helps obviously uh increase the performance we'll see some curves later on where they"}, {"start": 1040.96, "end": 1046.0, "text": " quantitatively show that this does happen indeed so that the teacher is always better than the"}, {"start": 1046.0, "end": 1051.9199999999998, "text": " student and so students can can actually learn something from the teacher because if yeah yeah"}, {"start": 1051.9199999999998, "end": 1058.24, "text": " i guess that makes sense um okay so a couple more details worth mentioning is that this whole"}, {"start": 1059.2, "end": 1065.04, "text": " like system this dino system will be uh free of bet normalization which is a trend we've seen"}, {"start": 1065.04, "end": 1070.56, "text": " recently with nf nets if you if you if you saw that paper uh it came from deep mind the idea was"}, {"start": 1070.56, "end": 1077.52, "text": " to kind of ditch the bet normalization because it has some nasty properties um and yeah but it ended"}, {"start": 1077.52, "end": 1084.72, "text": " up um complicating the whole procedure so i'm not sure it uh it got so popular but yeah still um"}, {"start": 1085.68, "end": 1089.92, "text": " just if you're able to kind of ditch bet normalization that that's cool and here they"}, {"start": 1089.92, "end": 1095.6000000000001, "text": " managed it because uh they say here that therefore when applying dino to vat we do not use any bet"}, {"start": 1095.6000000000001, "end": 1101.8400000000001, "text": " normalization in the projection hats uh vat does not have bn by default and they just made sure"}, {"start": 1101.8400000000001, "end": 1109.44, "text": " not to add bn layers uh on top of those linear layers in the projection head okay i already"}, {"start": 1109.44, "end": 1116.16, "text": " mentioned that these centering and sharpening operations are vital uh and they say here so as"}, {"start": 1116.16, "end": 1120.8000000000002, "text": " shown experimentally centering prevents one dimension to dominate but encourages collapse"}, {"start": 1120.8000000000002, "end": 1126.24, "text": " to the uniform distribution so that's this first type of uh like collapse i mentioned the uniform"}, {"start": 1126.24, "end": 1134.0800000000002, "text": " one so this one and then they mention um while the sharpening has the opposite effect okay so it"}, {"start": 1134.0800000000002, "end": 1139.76, "text": " kind of makes those uh one dimension explode so applying both operations balances their effects"}, {"start": 1139.76, "end": 1145.52, "text": " which is sufficient to avoid collapse in presence of a momentum teacher okay that's that's it those"}, {"start": 1145.52, "end": 1150.24, "text": " are those are all of the details of the architecture and how the system works now let's see for the"}, {"start": 1150.24, "end": 1156.8799999999999, "text": " results um first what they do is they take uh resnet as the as the as the baseline and they"}, {"start": 1156.8799999999999, "end": 1162.8799999999999, "text": " train it using dino so this is this method here and they show that uh compared to all of the"}, {"start": 1162.8799999999999, "end": 1168.96, "text": " previous uh like self-supervised methods such as i don't know like bootstrap your own latence bjoll"}, {"start": 1168.96, "end": 1176.48, "text": " or or moco or or simclear etc so they show that they are they perform better using both types of"}, {"start": 1176.48, "end": 1182.72, "text": " evaluations so the first one is linear one and the second one is k and n evaluation and just uh"}, {"start": 1182.72, "end": 1188.32, "text": " briefly if you don't understand how how the k and n works it's fairly simple so what you do is you"}, {"start": 1188.32, "end": 1195.3600000000001, "text": " train your your vat so you train your vat here and then you take uh you basically take uh your"}, {"start": 1195.36, "end": 1202.0, "text": " training set so this is your training set you freeze these weights so these are frozen so let"}, {"start": 1202.0, "end": 1208.6399999999999, "text": " me just you know it like f you froze these and now you just input an image outcomes are representation"}, {"start": 1210.0, "end": 1213.52, "text": " and because it's from the training set you'll have a label associated with it"}, {"start": 1214.1599999999999, "end": 1219.1999999999998, "text": " and you'll you'll just repeat this for a subset of your training set and you'll store these"}, {"start": 1219.2, "end": 1226.48, "text": " representation label tuples okay and next thing how you actually evaluate now this system is the"}, {"start": 1226.48, "end": 1233.68, "text": " following so now you have now you have uh like a validation data set and you pass in an image from"}, {"start": 1233.68, "end": 1239.44, "text": " the validation set outcomes representation but this time you don't have a label so what you do"}, {"start": 1239.44, "end": 1245.2, "text": " instead is you search through these representations which you've collected from your training set and"}, {"start": 1245.2, "end": 1250.0800000000002, "text": " you just find the k nearest ones and they were using like 20 uh experimentally they found that"}, {"start": 1250.0800000000002, "end": 1256.24, "text": " number works well so you'll just find 20 closest representations and you'll see what their labels"}, {"start": 1256.24, "end": 1261.3600000000001, "text": " are and by majority voting for example you'll just figure out what the represent what the label"}, {"start": 1261.3600000000001, "end": 1265.92, "text": " should be for this one okay that's the that's the whole idea so you just find the closest ones where"}, {"start": 1265.92, "end": 1271.52, "text": " closeness is defined by i guess they used uh l2 metric and so yeah that that's it that's how you"}, {"start": 1271.52, "end": 1276.48, "text": " find a label for for this one you just find the closest ones okay and now the interesting part"}, {"start": 1276.48, "end": 1284.48, "text": " actually here is that uh once you have vit you can see that the knn performance is like the gap"}, {"start": 1284.48, "end": 1291.44, "text": " between these two is severely like uh decreased whereas here we had a bigger gap that's the the"}, {"start": 1291.44, "end": 1297.36, "text": " important thing to notice is that using transformers and ssl has something that's that's um qualitatively"}, {"start": 1297.36, "end": 1302.32, "text": " different comparing to using cnn's that's the first thing worth mentioning and the yeah in general"}, {"start": 1302.32, "end": 1307.4399999999998, "text": " they compare with other baselines so here so here they compensate for the architecture here they use"}, {"start": 1307.4399999999998, "end": 1312.0, "text": " whatever architecture just the best methods out there and they still show that uh using linear"}, {"start": 1312.0, "end": 1317.6799999999998, "text": " evaluation as well as the knn they they achieve the best results compared to previous like bjoll"}, {"start": 1317.6799999999998, "end": 1323.6, "text": " simclear etc previous ssl baselines uh those are the initial results now let's see some other stuff"}, {"start": 1323.6, "end": 1331.4399999999998, "text": " here so these two results just support their claim that the features are really of high quality and"}, {"start": 1332.1599999999999, "end": 1337.04, "text": " since they perform really well for k nearest neighbor classification they also should perform"}, {"start": 1337.04, "end": 1342.3999999999999, "text": " well for image retrieval and copy detection and they just kind of uh yeah show that that's the"}, {"start": 1342.3999999999999, "end": 1348.8, "text": " case uh interesting thing here is that uh cool thing about ssl methods is you can use them on"}, {"start": 1348.8, "end": 1353.9199999999998, "text": " whatever data set you have you don't have to have labels and so they just pre-trained dino on this"}, {"start": 1353.9199999999998, "end": 1360.56, "text": " google landmarks uh data set which turns out to to boost performance severely compared to just"}, {"start": 1360.56, "end": 1366.32, "text": " pre-training dino on on image net and yeah they get results which are comparable to even a supervised"}, {"start": 1366.32, "end": 1372.96, "text": " baseline and that's cool okay let's continue here and that was the that was the first part so those"}, {"start": 1372.96, "end": 1380.24, "text": " were the the the that was the emergent property of nice k features that are good for knn focusing"}, {"start": 1380.24, "end": 1385.92, "text": " on these results here we can see that um different colors just you know different hats of the"}, {"start": 1385.92, "end": 1391.52, "text": " transformer heads attention heads from the vat uh final layer and we can see that the red head"}, {"start": 1391.52, "end": 1397.52, "text": " focuses on horses on this zebra's head we can see that the yellow head is focusing on on on the neck"}, {"start": 1397.52, "end": 1403.68, "text": " as well as on the ears and the blue hat is focusing on on this bonds here okay and um what's"}, {"start": 1403.68, "end": 1409.92, "text": " interesting is they show here that um even some some occluded objects such as these bushes here"}, {"start": 1409.92, "end": 1415.92, "text": " are detected and segmented uh i mean it's hard even for me to kind of detect these and the network"}, {"start": 1415.92, "end": 1422.24, "text": " learns how to recognize the bush and segment that information okay that's super cool okay continuing"}, {"start": 1422.24, "end": 1431.04, "text": " further they showed that if you take supervised baseline and plot the segmentation maps from from"}, {"start": 1431.04, "end": 1438.24, "text": " that one and from dino on on different data sets on pascal so they pre-trained this on image net so"}, {"start": 1438.24, "end": 1443.6, "text": " both dino and the supervised method and then they test it on pascal data set and you can see there's"}, {"start": 1443.6, "end": 1449.2, "text": " bunch of random like sparse dots here so the segmentation masks are not of the same quality"}, {"start": 1449.2, "end": 1457.04, "text": " as as as in dino and they also show quantitatively that you can see that this is i think uh how they"}, {"start": 1457.04, "end": 1462.88, "text": " called it uh jacquard similarity or iou just a fancy name for intersection of a union they show"}, {"start": 1462.88, "end": 1469.3600000000001, "text": " it's much higher compared to supervised baseline so again this testifies that um uh the the dino"}, {"start": 1469.3600000000001, "end": 1474.8, "text": " the vat with dino learns how to extract these segmentation masks uh high quality segmentation"}, {"start": 1474.8, "end": 1480.3999999999999, "text": " masks okay uh in this table they show a general applicability of these features to various"}, {"start": 1480.3999999999999, "end": 1487.28, "text": " downstream benchmarks such as cipher 10 cipher 100 etc so just comparing dino with the supervised"}, {"start": 1487.28, "end": 1492.56, "text": " baseline we can see it pretty much achieves outperforms supervised baseline on all of these"}, {"start": 1492.56, "end": 1498.24, "text": " downstream tasks which further testifies that the features learned using dino and vat are of super"}, {"start": 1498.24, "end": 1505.52, "text": " high quality uh some ablations here uh maybe the most important row i want you to focus on is when"}, {"start": 1505.52, "end": 1511.6, "text": " we uh expel when we don't use the momentum encoder so that's the teacher network being formed from"}, {"start": 1511.6, "end": 1516.48, "text": " exponential like by the exponential moving average strategy we can see that it completely"}, {"start": 1516.48, "end": 1522.16, "text": " fails to train so the performance just drops down to almost zero and it does not work so we need we"}, {"start": 1522.16, "end": 1530.16, "text": " so this method so dino needs momentum encoder and yeah um other ablations like uh multi crop"}, {"start": 1530.16, "end": 1535.1200000000001, "text": " not using multi crop reduces result reduces the performance not using cross entropy reduces"}, {"start": 1535.1200000000001, "end": 1541.1200000000001, "text": " performance etc in this plot they show that not that using smaller patches is more important than"}, {"start": 1541.1200000000001, "end": 1546.5600000000002, "text": " using a bigger model so if you focus on on this thing here so if you focus on on the eight by eight"}, {"start": 1546.56, "end": 1554.8799999999999, "text": " uh patches and vat small uh as you can see it has better performance than using um vat big so the"}, {"start": 1554.8799999999999, "end": 1561.6, "text": " bigger model but with 16 by 16 patches so the trade-off here obviously exists and uh it's the"}, {"start": 1561.6, "end": 1566.6399999999999, "text": " following basically when you're using eight by eight you need you need a bigger memory footprint"}, {"start": 1566.6399999999999, "end": 1572.3999999999999, "text": " and so uh you also kind of uh reduce the throughput of number of images you can feed through the model"}, {"start": 1572.4, "end": 1577.76, "text": " in a second um but yeah you get some additional performance but all in all you can see these two"}, {"start": 1577.76, "end": 1583.2, "text": " curves overlap so basically it boils down to how much performance you want and what's the throughput"}, {"start": 1583.2, "end": 1587.68, "text": " your desired throughput and that's how you can pick the best trade-off between the size of the"}, {"start": 1587.68, "end": 1595.8400000000001, "text": " model and the patch size okay nice um here is the quantitative results i mentioned that the student"}, {"start": 1595.8400000000001, "end": 1601.1200000000001, "text": " is always so that the teacher is always better than student you can see as the epochs progress"}, {"start": 1601.12, "end": 1606.1599999999999, "text": " during the training the validation accuracy of the teacher is always higher compared to the student"}, {"start": 1607.28, "end": 1613.04, "text": " and they just kind of try different strategies of how they update the teacher and it turns out that"}, {"start": 1613.04, "end": 1619.1999999999998, "text": " using momentum is the best strategy using previous epoch so that's something similar to dqn if you"}, {"start": 1619.1999999999998, "end": 1626.8, "text": " watch my video on dqn or you know what dqn is it also uses it also freezes the target network and"}, {"start": 1626.8, "end": 1635.84, "text": " updates it every couple of epochs or whatever every every end steps in general and so that method"}, {"start": 1635.84, "end": 1643.76, "text": " also works but it yields somewhat worse performance and using updating the teacher too often so that"}, {"start": 1643.76, "end": 1650.6399999999999, "text": " will be previous iteration or student copy you can see that the method just fails all completely so"}, {"start": 1650.64, "end": 1656.96, "text": " yeah there are ways to kind of build up this teacher network but the best thing seems to be so far"}, {"start": 1656.96, "end": 1662.16, "text": " this momentum update these results are pretty interesting let me just kind of explain them"}, {"start": 1662.16, "end": 1666.5600000000002, "text": " again there are two forms of collapse regardless of the input the model output is uniform"}, {"start": 1666.5600000000002, "end": 1674.0800000000002, "text": " along all dimensions or dominated by one dimension if we take the cross entropy we saw in the loss"}, {"start": 1674.72, "end": 1679.5200000000002, "text": " so we take the cross entropy loss and kind of split it down into the entropy component and the"}, {"start": 1679.52, "end": 1685.68, "text": " KL divergence component and plot it here they show the following so if you're using sharpening"}, {"start": 1685.68, "end": 1690.8, "text": " only and you just ignore you you kind of ditch the centering part in the teacher network this"}, {"start": 1690.8, "end": 1695.2, "text": " is what happens basically the entropy drops down to zero which means what which means that the"}, {"start": 1695.2, "end": 1702.72, "text": " network learns to output like something like this you basically have a peak at one dimension and the"}, {"start": 1702.72, "end": 1707.84, "text": " probabilities one and everything else is zero and so the entropy of this thing is zero so that means"}, {"start": 1707.84, "end": 1714.56, "text": " that using only sharpening leads to this mode of collapse and using only centering also collapses"}, {"start": 1714.56, "end": 1720.32, "text": " but as you can see the collapse has a non-zero value so the the the the entropy has non-zero"}, {"start": 1720.32, "end": 1724.48, "text": " value which means we have something like this which means we have a uniform distribution"}, {"start": 1725.6799999999998, "end": 1733.1999999999998, "text": " and this has a non-zero entropy and so that's what we end up with here and it did mention"}, {"start": 1733.2, "end": 1738.48, "text": " somewhere here let me just check the entropy converges to different values so zero with no"}, {"start": 1738.48, "end": 1745.3600000000001, "text": " centering and minus log one of over k with no sharpening so that's that's the value that we'll"}, {"start": 1745.3600000000001, "end": 1754.32, "text": " have here so here this value will be minus log of one over k the reason being is the support has"}, {"start": 1754.32, "end": 1761.8400000000001, "text": " k values so this is distribution support is has k dimensions so that means this will be one over k"}, {"start": 1761.84, "end": 1768.8799999999999, "text": " so that the integral amounts to one and this means if you just calculate the entropy you'll end up"}, {"start": 1768.8799999999999, "end": 1776.56, "text": " with with this this value here so yeah using both as you can see here the entropy converges to some"}, {"start": 1776.56, "end": 1782.8799999999999, "text": " value but it's neither zero neither this value of minus log one over k and that's it that's"}, {"start": 1782.88, "end": 1792.48, "text": " pretty much it taking a look at the KL divergence the KL becomes zero for these two cases because"}, {"start": 1792.48, "end": 1799.2, "text": " both the student and the teacher learn to output either uniform or this kind of distribution and"}, {"start": 1799.2, "end": 1804.0800000000002, "text": " when we're using both then KL divergence does not collapse to zero those are just two different"}, {"start": 1804.0800000000002, "end": 1809.7600000000002, "text": " perspectives on the same thing and that's the fact that they collapse to either of these modes"}, {"start": 1809.76, "end": 1816.16, "text": " depending on what what do we omit either sharpening or centering okay that's that's"}, {"start": 1816.16, "end": 1821.52, "text": " pretty much it i think i explained everything i wanted finally they mentioned it here in the"}, {"start": 1821.52, "end": 1826.96, "text": " future we plan to explore if pre-training a large vat model with dino on random uncurated images"}, {"start": 1826.96, "end": 1833.84, "text": " could push the limits of visual features and i can just imagine something along the gpt3 dimensions"}, {"start": 1833.84, "end": 1840.32, "text": " being trained and this time with vision transformer so that will be exciting i guess final thing i"}, {"start": 1840.32, "end": 1847.52, "text": " want to show you are the features that like the some semantics of the features of vat trained with"}, {"start": 1847.52, "end": 1854.08, "text": " dino as you can see here just extracting those representations and then using t-sneed to kind"}, {"start": 1854.08, "end": 1861.9199999999998, "text": " of plot them in 2d you can see that dino vat with dino learns how to cluster similar objects here"}, {"start": 1861.92, "end": 1868.24, "text": " we can see like model t that's a car then car wheel ambulance minibus so we have some kind of"}, {"start": 1868.24, "end": 1873.76, "text": " vehicles cluster here and then we have some like gutter snake king snake ring neck snakes we have"}, {"start": 1873.76, "end": 1880.64, "text": " some snakes here we have like orangutan some monkeys here so all in all all of these emerge"}, {"start": 1880.64, "end": 1887.6000000000001, "text": " by training dino by training vat using dino so we get this nice property even though we train vat"}, {"start": 1887.6, "end": 1894.24, "text": " only using ssl objective and we also got those segmentation masks and we also got the features"}, {"start": 1894.24, "end": 1900.24, "text": " which are of high quality as shown by knn performance etc so hopefully you like this"}, {"start": 1900.24, "end": 1917.76, "text": " video if you did consider subscribing and sharing and until next time bye bye"}]