Facebook AI's DINO | PyTorch Code Explained
👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

With this video, I kick off a brand new series of coding videos in PyTorch where I'll be explaining some of the most impactful AI research through its code analysis!

In this video I cover DINO from the "Emerging Properties in Self-Supervised Vision Transformers" paper.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2104.14294
✅ Code: https://github.com/facebookresearch/dino
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 DINO paper - short overview
02:38 Code analysis starts - training arguments (argparse)
07:30 Training main function
08:25 DINO augmentations
11:30 Main function resumed
16:05 DINO head (MLP)
18:05 Main function resumed
20:40 DINO loss overview
21:45 Main function resumed
24:05 Augmentations visualized (matplotlib)
27:00 Main function resumed
28:00 DINO Core part!!! (in-depth shape analysis)
33:45 DINO loss in depth
38:00 Main function resumed
39:50 Visualizing attention script explained
48:15 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR ML PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#dino #pytorch #coding
 What's cracking guys in this video we're gonna do something very different So I'm gonna walk you through the code of the dyno model which I've previously covered in one of my videos From Facebook AI research so we're gonna step through the code and understand how exactly the training works line by line in PyTorch So if you find that useful let me know and also Leave a feedback of what you think about this video so that I can improve it over the for the next for the next time So this is super experimental before I dig into the code I'll do like a super short overview of the paper just one minute overview for those of you who haven't watched the video go Ahead and watch it. I'm gonna link it somewhere here And so let me do a quick recap of what happens so that you have some context before we dig into the code Okay, so basically in that video what it what I've explained was I explained how exactly this attention mechanism works But I explained it on a very high level using these diagrams And so from from this image we get this this this attention map And we're gonna see how exactly that works by the way if you if you want me to share these squiggles of the paper With you let me know down in the comments. I'm gonna leave a comment and if I get enough thumbs up I'm gonna leave I'm gonna think about how I can share these with you Okay, so next up we saw this high level diagram of high down a dyno works so basically we have the student branch We have the teacher branch the teacher branch is updated using this Exponentially moving average from the student like weights, and we also have a stop gradient here Which means we won't be updating the teacher Finally after applying the softmax we get our district output distributions here And the whole idea is to make these two distributions be the same And we do that by across entropy as you can see here, so the p2 times log of p1 So the idea is to extract the essence of the image we input images which had different augmentations as the input to student and teacher and Hopefully we want to make sure that the distributions are the same because that's the same image And that's how we extract the kind of the essence of the image, so there will be a high level explanation So we saw the pseudocode now. We're gonna see the actual code. We saw that we are using So I mentioned augmentations. We're using these global crops, and we're using these local crops, and we are again trying to Make the model output the same distributions for both of these Finally let me see what else I had so explain how K nearest neighbor algorithm works and Probably in the next video I'm gonna show you how exactly this looks in code because I think the things I just explained will take like at least an Hour to kind of cover, so let's get back to the code and see how this looks like okay So you've watched the video you have some context from this short recap now. Let's start digging into actual code Okay, so let's put some breakpoint here as I'm going to step through this code and show you exactly how it works So let's start with this get args parser function Okay, so I'm gonna just quickly scheme over these arguments so that you have some context and idea of which variables are gonna be Circulating around the code so first things first. They have the architecture. We're gonna be using the vision transformer small They have many different options, but we're gonna focus on that one batch size I had to increase it to 32 because it was 16 by default and it's for some reason crashing my machine I still haven't been debugging the amount of memory. It's eating up on my GPU So yeah, so output dimension is just 65 65,000 and that's the actual output distributions We just saw in the paper overview which we are going to compare and make sure they are the same for different crops, okay? Then we have just normalized last layer. I'm gonna skip this momentum teacher so 0.996 That's the at the actual coefficient that's being used in the exponentially moving average when we are forming the teacher weights from the student weights Again better realization. This is not that important warm-up temperature teacher temperature Number of epochs all of these are not that important. There's just a schedule of how the temperature of the teacher is gonna change throughout the training Using mix precision training is gonna be on so I obviously won't be focusing on the optimization stuff in this video so weight decay Okay, that's again. We have some schedule so they have the end and initial value Clipping gradients so that's also basically when you're updating the weights They are going to use this to kind of clip the norm of the of all of the weights before doing an update We're gonna see that in a couple of minutes So number of epochs hundred by default where or not to freeze the last layer This is just some trick that makes the training a bit easier learning rate So they basically have a linear warm-up first and then they have some kind of I think cosine schedule Until the end of the training so warm-up epochs just tells you across how many epochs you're you're you're taking to do that linear warm-up thing minimum learning rate that's the pretty much learning rate at the end of the of the training Optimizer they're using Adam Wide Adam so that's because they have distributed training this dino model is trained on 8 or 16 GPUs originally But I have only a single GPU here So that's that's it drop path rate is just an argument that is going to modify the the vision transformer architecture Then we have the global crop scales. It's gonna be from 0 4 to 1 In contrast that to so I'm gonna quickly get back to that one, but we have eight local crops and we have And we have so local crop scale is gonna be from 0 0 5 to 0.4 And that's the same number as here. So that means so that means that This this for the small crops We're gonna take 5% to 40% of the image size and then we're gonna resize that back But like for the big crops we're going to take from 40 to 100% of the image and then we're gonna do the resizing to 24 as for the small crops they're going to be resized to 96 96 pixels Again some details, but like yeah, this video is gonna be about a lot of details We're gonna use image net the thing I did is I actually downloaded a small subset of image net because this thing is huge more than like 150 gigabytes, I think and So from fast AI basically it's called image net Well, I don't know how to pronounce it with two T's and aside from me having to download image net that small subset of image net I also had to do a couple of tricks to make this work on Windows because I'm running on Windows machine So if you want to know about those nitty-gritty details and small bugs and problems I had to solve do join the discord server actually had a thread where as I was in analyzing this code over the past two days I was kind of writing down the steps I took and the problems I encountered Okay, so finally as I said image net where I'm going to save the checkpoints and logs seed seed basically serves to Make this code reproducible because otherwise there is a lot of randomness and every time you run this code is gonna have different output and different results And the images are gonna be loaded in different order It makes it easy for me to have some consistency between different runs and it's easier to debug the code So a number of workers not that important all of this is just some distributed settings and That's not that vital. Okay We parse the arguments we create the output directory where we are going to dump the logs the log files and the checkpoints And finally, let's go to the main function and that's trained, you know So again, this is the in it distributed mode is just a function that's setting up certain settings When you're doing distributed training on 8 or 16 GPUs again, I'm using Only one GPU I had to do some modification because of Windows But yeah, do check out this Discord server for that So we are fixing the random seeds. They're just printing I'm gonna Rise up the console here So they're basically just printing the the Shock code the the hash of this commit with of the code. I'm currently running Then they're just printing some arguments Cudian and benchmark set to true is just a certain optimization again That makes the code run faster under certain assumptions again. Not that important. Okay This is fairly important data augmentation dino. I'm gonna quickly walk you through this code and then I'm gonna step over it in the execution loop But basically as you can see here, they have flip and color jitter Transform so that's just a compose. That means we are composing a couple of different transforms and These are organized so that they do the flipping and the color Like photometric augmentations. So we have as you can see here random horizontal flip with 50 percent With 50 chance, we're gonna flip it Either flip it or not flip it then we have the we're gonna randomly apply with certain probability like 80 percent This color jitter which is going to change the brightness the contrast the saturation the hue So basically our photometric augmentations and finally random grayscale is with 20 chance. We're gonna kind of Apply the the grayscale transformation The then we have normalization. This is your standard stuff. We are converting the the images to pi torch tensor We are normalizing them and don't get confused by these magic numbers Even though it'd be really nice if the author has actually extracted this into a constant Because they've been using this all around the place. So this is just your image image net Statistics so the mean and the standard deviation of the image net images Okay So for the fun part this random resize crop is the thing that does the magic of dino So global crop scale is again the thing I just mentioned So from we're gonna take 40 to 100 percent of the image We're gonna cut it there and then we're gonna resize it back to 224 by 224 Using bicubic interpolation 224 by 224 using bicubic interpolation method Then they're gonna apply some photometric augmentations and the flipping the gaussian blur and we're gonna normalize the images We have a second global Crop because if you remember the details from the paper, we have two global crops and we have eight local crops Although obviously that's you can you can experiment with that So again, a random resize crop everything is the same. They just apply solarization here Uh, if you're not familiar with solarization, i'm just gonna put the image on the screen so you can see a visual example I guess the image is going to explain that much faster than I will Finally, we have the the local crops. So that means we are applying the local crop scale So that means we are now using that 1.5 percent to 40 percent of the image and then we resize that back into 96 by 96 images So these crops are gonna these images that came from the smaller crop of the image are gonna be of smaller resolution And 96 by 96 again flipping and color rendering a gaussian blur normalization Standard stuff so that's it So basically what's gonna happen is this data augmentation dino uh is gonna be passed to the data set in pytorch and Then every time you try to fetch an image from the data set it's gonna apply It's gonna append one crop here one global crop the second global crop And then we're gonna have iterating a for loop. So eight times and do these small local crops And then we're gonna return like a list of 10 crops in total This is just on code. I wrote i'm gonna experiment with that a bit later But let me get back to the actual line where we stopped. Okay, so that was the that was the uh data augmentation Of dino. I think that was pretty important detail by the way, i'll be adding time stamps to all of these logical sections So feel free to skip if you understand some part you can just skip to the part you you want to understand better Okay, continuing on we are creating this data set. Uh, so it's gonna link to the as you can see here image net train Uh subdirectory and we're gonna pass the transform I already mentioned Which means every time you fetch an image from this data set is gonna apply those 10 transformations and we're gonna get those 10 crops back Uh distributed sampler. So just ignore the distributed part This is just going to shuffle the images once we input that into the data loader So as we can see we have a data set here. We have the sampler We have I think 64 images per gpu and since I have a single gpu that's going to be 64 images in total a number of workers just optimization stuff peel memory as well. That just kind of helps you Basically, uh transferred images from the cpu to the gpu faster because there's this uh dedicated slot of the memory So i'm currently going through every single detail, uh, because i'm experimenting but like, uh, i'm later on i'm gonna abstract many details So let me know whether you actually want everything explained or you just want me to cover the main idea Uh, this is experimental video. So, uh, please give me that feedback. And yeah, uh droplast means because we have a data set of I I think uh, like I have around 9 000 something images Because every batch has 64 images as we saw here Uh, that means that probably the last batch won't have 64 because unless the the number of images in our training data set is divisible But 64 we're gonna have the last batch being less than 64 and this droplast just means we're gonna drop The last batch because that can cause some issues probably because of the shape being different So that's the reason you sometimes see this droplast set to true. Okay data loaded there are uh, as you can see here 9469 images, okay um This is just some bureaucracy stuff. Uh, they did because they used to to name these models Date now, uh, they're just kind of replacing them with vat, but the architecture is already vat small. So nothing's gonna happen there um, so basically This says that so vat is just uh, like a shorthand notation for the all of the vision transformer models That exist in this vision transformer pi file So we're gonna because we are vision transformer we're gonna enter this if statement and we're going to as you can see here We're gonna form so this statement here may confuse you. So this is just a list of all of the different like uh functions from this vision transformer file and this is just going to instantiate the model and we're going to instantiate it with the 32 by 32 so the patch will be 32 by 32 pixels and we're gonna use this Uh drop path rate of 0.1. So the quickest way you can understand how this thing What this thing does is just put a print statement in there do this, uh, like just print the dictionary Uh set a breakpoint there stop the code Re-execute again and we're gonna see what happens there. So i'm gonna Just skip these skip these i'm gonna get here. I'm gonna open the console And after I print this you're gonna see Uh, whoops bunch of stuff and as you can see here It's actually printing this whole vision transformer pi file and the thing is uh, it's some of the some of the functions are organized Into dictionaries. So this is just your python stuff. So that means we're gonna have somewhere in here We're gonna have uh, like these functions. Let me let me jump to that file. So we're gonna have somewhere in there We're gonna have this v80 small Function and then we're gonna pass the patch size and uh additional we are passing that drop path Uh coefficient so that's the the thing that happens there. So basically we're instantiating the v80 small model here Uh, then we're doing the same thing for the teacher network. So that was a student network We have the teacher network and since we're using uh Vision transformer small the embedding dimension is going to be 384. Uh, that's just the hyper parameter they they kind of Decided to use okay So they have in these other branches they're using excite and some other stuff we don't care about so Now this is another important part. Basically, uh, what they do is they they wrap the student network So this is the vision transformer and they put this dino head on top of it. So dino head is just a simple mlp Uh with some additional tweaks and i'm gonna jump into those details right now So the the main point is they are forming uh, this this mlp And since number of layers is three by default, that means the mlp is gonna have three linear layers that means we're gonna uh Execute the else branch and we're gonna form the mlp that just consists out of as you can see here We have some potentially some batch norms potentially some uh, like gelu, uh, like uh Activation functions, uh, but most of all most importantly we have this linear function Uh, so basically it's an mlp And then we just initialize the way it's using this self apply that's an inbuilt Function that's a part of this nn module class and then we're gonna form the last layer, uh by creating yet another Linear layer and the linear layer is gonna have so the output dimension here is like 65 000 something So that's the actual output of the whole uh pipeline and then we're gonna compare those outputs by cross entropy later on So that's the last layer. Uh, the interesting part here is this weight norm is just a hook function Uh what it does is once you put this weight g to all of ones That means they are going to normalize the actual linear layer So it's just going to make sure that the l2 norm of the weights of that linear layer are equal to one That's basically it and if they decide to do this normalization of the last layer if this is set to true Then as you can see here that those gradients are going to be set to false which means the following during the training Those uh, those weights will not will not be updated. Which means that the l2 norm is going to be kept at one Uh, which means we're normalizing the last layer hence the norm last layer. I think it's pretty self-explanatory So again simple mlp on top of it. We have the last layer which has the 65 000, uh something Uh number of neurons as the output and that's it. Optionally we have these uh, these uh, like normalization stuff Okay, getting back here. So that's the dino head We stick it on top of the student and one additional detail which we need to go through This is this multi-crop wrapper again. I have timestamps just jump if you don't if you're not interested into this section Uh, so the multi-crop wrapper What it does is the following Um, you know, i'm actually just gonna put a break point here and then after we we get to this part We'll understand what multi-crop wrapper does but basically on a high level for now it's basically just gonna deal with those crops passed them through the the pipeline and then Outcomes the result from the student the final the final output and the final output from the teacher We'll see that in a in a once we we hit that part of the code. Okay, uh getting back here So that was how we formed the student part So again, it's a multi-crop wrapper around student which is vision transformer and dino head which is mlp Same thing goes for for the teacher. We have teacher we have dino head and that's it Some minor differences like whether we are not we are normalizing the last layer or not But that's it and we are not using the bet normalization So this is uh, there is no better normalization in this whole pipeline, which is something we we strive for okay Uh, we're just pushing the the student and teacher to the gpu. That's this dot kuda that that's what it means and finally Because as I said, we don't have any any batch norms That means we're gonna enter the else branch and that means that the teacher network will not be using this distributed data parallel Which is just a wrapper that makes it easy to train these models in a distributed fashion Uh, okay continuing on we wrap the student network, uh into this distributed data parallel But but they only have a single gpu here So the index zero means the default gpu you have on your system. I have a rtx 2080. So that's it Then we are loading the as you can see here. We are loading the state dictionary directly from the student Uh module so and they say here nicely teacher and student start with the same weights So initially we're gonna start with the same weights later on we're gonna use this exponentially moving average to update the teacher weights Uh, finally as I said, there is a stop gradient Uh, if you saw the the diagram there is a stop gradient and this is how you implement the stop gradient in pytorch you basically just iterate through all of the Like parameters and you said there are gradients to false. So that means you won't be trainable I'm gonna skip all of this just f9 and that's it So we we have an output here, uh, the student and teacher are built. They're both vision transformer small networks Okay, this part is fairly important and i'm gonna probably uh step through it a bit later But let me just give you a glimpse. So i'm gonna put uh, like a like a breakpoint here in the forward function And as you can see here, it's just accumulating the parameters that we passed in the init function Uh, it's just registering this center vector. So if register buffer what it does it makes sure that the this this this vector here Uh is a part of the state dictionary even though it's not trainable. So this center vector will not be trainable We're not going to update it using gradients. We're going to update it using the exponentially moving average again So that's the reason they're using register buffer. So it's not trainable So it's not a model parameter in the pytorch, uh, like uh terminology And but we still need it in the actual state dictionary of the model Um, okay. Finally, they have some teacher temperature schedule. So with the current parameters, this is actually going to return 0.04 Throughout all of this all of the epochs, but what it's it's kind of envisioned to have a linear ramp up here Although that's not that important. Let's get back to the dyno loss Okay, i'm gonna step over this i'm gonna later once we are actually using the dyno loss i'm gonna step through the forward function um This just not important Whether we are regularizing or not regularizing certain parameters of the student network. We're gonna create the atom wide optimizer And finally because we're using the mixed precision training We're just going to wrap this create this fp16. So that means we're using 16 bits training these models Okay here they're just creating uh schedulers for the learning rate. So it's going to be a cosine scheduler Um, then we have a weight decay schedule against cosine scheduler and finally momentum scheduler Basically, uh, that's the thing that is used to update the teacher Uh weights using the student weights. So that's that parameter Uh, and again, it's just a cosine scheduler. Okay, so that's it. I think we're we're good to go We're getting closer to the actual training loop Uh, this already took a lot of time. So uh, we're just going to this this restart from checkpoint I think we're gonna skip it because we don't have any checkpoints for now. So just put a breakpoint there. We're gonna enter it Okay, let me step over step over over Okay, let me step over step over. Okay, and we're gonna just return from that function since we have don't have any checkpoints Okay, so that's just uh dummy for now. Um Start epoch is gonna start with zero. We're just uh logging the starting time just some bureaucracy stuff And finally here is the main loop. Okay, we are here so we're starting from the epoch zero we're gonna run for 100 epochs and Uh ignore this line. This is just something that has to do with this distributed sampler. Otherwise, you wouldn't have this line Um, so this is the actual meat of the whole whole script So it's training one epoch is gonna train the the dino for one epoch and then later on we're just gonna have some some Logging nothing nothing very fancy. So i'm gonna step into this f7 And let's see how this function looks like. Okay, so metric logger not an important again Okay, this is where the fun starts. So we have the data loader and we're going to fetch Like this images so this images is going to be a list of 10 elements and each element is going to be a batch of images So each element is going to be 64 images So that means so because 10 elements because as you remember we have two global crops We have eight local crops. So that means if we take like the zeroth element of this images list That's those are the global crops and we have 64 images and if we take uh, like the second index That means that's the first local crop Like set of images a batch of local crops. And so as you can see here I just pre-wrote some code here to make this a bit faster I'm gonna stop the code execution and i'm gonna rerun it again I'll I'll need to make another tweak to make this work So that's that means i'm gonna have to add additional crop So what this does is we are just going to preserve the image as it is without doing any any crops So that's just a reference image and that was that's going to be prepended as the first crop So now we have 11 crops and hopefully now this is going to work. Let me let me let me run this Uh, let me go back to the code where we were. Okay, here we are So what i'm doing here is as you remember I prepended this original image So now we have 11 elements instead of 10 and this zero will just be indexing the original images And i'm just gonna fetch like one image out of those 64 images I'm gonna convert those to non-pi do some move axes because the channel is actually channel first and now we want to have channel last For for the plot function of matplotlib So second line what it does is it will extract the global crop So remember now the the the index one and zero the index one and two are actually referencing The global crops and again, i'm gonna take just one single image Finally, we have the local crop. So index three through ten are all going to contain local crops So i'm just gonna take one arbitrary local crop and finally, let's let's plot those results. So Um, here you can see the the original image, uh, just some dude playing golf So i'm gonna now exit that remember that image and now we're gonna see a global crop as you can see it's uh, There is a crop but there is also severe solarization and other photometric effects So you can't really see it perfectly, but let's see the the local crop So the local crop is as you can see now, it's totally clear We're just taking a single small crop and we're resizing it into this image of 96 by 96 And we also apply some augmentations here. So that's it That's the now you have an understanding of how images look like and I encourage you when you're debugging and understanding the code Do make these visualizations as much as possible because that helps you understand what's going on in your code That plus print statement, I guess that's pretty much how I debug the code. Although there are sometimes more elegant ways to do it. Okay so Um, let's let's continue here. So okay This line is just forming a global step depending on the epoch we are in Depending on the number of images in the data set depending on the current iteration inside of this Inside of this loop. Okay, so that's just global step and Then what they're going to do here is depending on the schedule current schedule. We're gonna update the way decay and learning rate Uh, depending on these param groups, it's not that important. I'm just gonna skip that part So we're just adapting the way decay and learning rates depending on the current step Okay, so skipping that we're putting the images on the gpu. So that's this step Let me just whoops. Let me just skip over that Uh, and finally this part, uh is just a context context manager because we have this mixed precision training And it's not none. So we're gonna enter this but now it's not going to work because I prepended all of this I'm just gonna quickly comment this out Uh and comment this out and i'm gonna restart the training and get back to you. Okay. Here we are Uh, so this is as I said very important step We're taking the global crops and we are passing only the global crops through the teacher to get the teacher output and let's see So analyzing the shape of the output is super important. I can't stress that enough So let's now see whoops. I have uh, okay, so we have a multi crop wrapper Okay, this is the part we we want to understand a little bit better um so x Is just a list of two elements again, uh, each contains a batch of crops These are global crops. So because it's a list we're gonna skip this part So this is just a fancy way of separating the global from the local crops and it doesn't make much sense for the teacher network Because it's only using global crops. So we're just gonna have number two here. So as you can see here Uh index crops just has number two. Uh, this is going to make more sense for the for the student network so now what we do is We iterate here and we're gonna take as you can see from zero to two So that means we're gonna concatenate all of those, uh global crops We have 64 images in one in one in one element. We have 64 images in the second one We're just going to concatenate them and now we have 128 images in a single tensor and then we're gonna pass that through the backbone which is basically the vision transformer Uh of the teach network. So that's going to form what so the the let's try and figure out the shape so 128 and because we are outputting Only the cls token. So that will be the output from the vision transformer. So it has 384 Uh, like uh neurons, right? So we're gonna have hopefully, um What so 128 times 384, okay, let's let's see whether the prediction was right So it's kind of good to have these predictions and then uh tweak your brain depending on whether you made made a mistake or not I guess that's a similar way how these neural networks work. So the out Shape here as you can see the shape is 128 128, um, comma 384 so that's it i'm gonna skip this part and now we're gonna gonna this is going to accumulate the The output and this will again make more sense for the student network. Now is just gonna Uh, we're gonna just copy paste pretty much this output from the vision transformer here and that's it We're gonna exit the loop and now this is the actual dino head So this is the mlp and now we're after passing it here. We're gonna end up with those 65 000 something So we're gonna have 128 times 65,000 something. Let's let's see whether that that's true. Okay Uh, whoops, let me get out of here Teacher output. Let's see whether the teacher output is as I predicted so teacher output. Let me kind of So 128 65 000, that's it. So the predictions were right. So now This is the interesting part. We have a student network and we are passing both the global crops as well as the local crops So let's enter the loop again again. It's a list. So i'm gonna skip this and now it's a bit more interesting. So it's gonna Make uh return the ids where the resolution changes So we have 20 remember the global crops are 224 by 224 and the local crops are 96 by 96 So what this line of code actually does is it finds a separation and returns the indices? Of the place where the resolution changes so it's going to return as you will soon see 2 and 10 I think or something like that. Yeah 2 and 10 so that's because after 2 we drop the small resolution And then we keep the small resolution until the end and that's 10 crops So now we are iterating again as you can see here. We're first going to pass 0 through 2 So that's global crops And we're going to pass that to the the network here. So that's going to be again 128 So that was that is going to be as you can see here 128 384 Uh, and we're just going to concatenate that with this output, which is just a placeholder variable You can see here is torch empty and we're just concatenating the results to it and that's it Now we're gonna put this the start index is now going to be 2 and the end index is going to be 10 So now we're passing 2 Through 10 which means we are fetching the local crops. We're concatenating them So we're gonna form what like let me do it like calculator Here we have 8 crops all of these have what? 64 64 images so that's 512. So let's see whether that's true So we have 512 and we're going to expect the output 512 384 so let's see whether that's true. So we have out Output variable 512 384. That's it. So everything works as we expect So let's continue here again. We're just concatenating these so now we're gonna concatenate these with the outputs from the from the global crops and that means we're gonna have what like 512 for the global crops plus 100 128 so 512 was for the for the local crops 128 is for the Global crops so that's 640. So if we do that, we're gonna see that the output is Let me see here So the output is 640 384. Okay, that's it. Let's continue. We're gonna exit the loop. We're gonna pass all of these through the MLP and that means we're gonna end up with Instead of 384 we're gonna have those 65 000 something Activations, so that's it. That's the whole trick. Let me again exit this thing Okay, exit. Come on. Come on. Come on This sucks ass, okay, so this is the last important detail Let's just analyze the dyno loss and that's pretty much it. You know, you should be able to understand the training So here what happened what happens we have we take the student outputs and remember that's again 600 something times 65 000 something so 640 times 65 000 and we're just going to scale it with this temperature coefficient So remember that's remember from the diagram from the paper. That's what they are doing Now they're gonna chunk this into 10 pieces because remember We have 10 different crops so this is just a way to take these 640 outputs and kind of cluster them into Semantically meaningful categories so that means global crops and local crops So we're gonna see that the student out let's find that variable. So that student out here Let me expand it. We have 640 and now we're gonna have a list of 10 elements and every element will have 64 images as it should be so as you can see here 10 elements in the list and every single one contains 64 images That's it. We do a similar thing for the teacher network. We have some temperature stuff We have the centering of the output and then we divide by the temperature across the feature dimension so that means across the That vector that has 65 000 activations where you're going to do the soft max across it and then we do the detaching because again, we're not training teacher network and We chunk it into two because remember we have two global crops being passed through the teacher network So teacher out let me just expand that variable. So teacher out has 128 it's gonna be split into two elements each will have 64 images So let's step over it and as you can see here two elements 64 images each. So that's it So now we have those different views and now we're gonna do the actual like loss, so What is happening here is we are taking a single so there's gonna be a single like a Global crop and it's gonna contain a batch of images for that global crop and then so this just makes sure we are not operating On the same view. So that's what we're gonna skip in the first iteration because both are zero So that means both are indexing the same global crop and now in the second iteration, this is gonna be one We're gonna skip that and what happens now is we're gonna take a batch of images So 64 images from the from the from the second global crop and we're gonna contrast it with Q Which is the the first global crop and that means and we're gonna do simple cross entropy here So that's Q and at times the log of the student view Softmax is here because we already we already done softmax for the teacher network So we have to do softmax and then we have to apply log and then we have just cross entropy and that's it So focusing on dimensions here on the shape again So if we take expand the Q variable, you can see it's 64 times 65,000 The same thing is gonna be for this student view So we have 64 images here and here and we have their outputs and we're just gonna make sure that the Distributions are the same for all of these 64 images. And then what we're gonna do is Basically sum it up So we're gonna sum up across the that dimension that not the dimension number one And then this mean will do the averaging across the second dimension and that's gonna end up with a scalar So after we do this and this we end up with a scalar and that's it. That's pretty much it now We iterate through every single combination of views. So we have we trade through different global crops here We trade through both the global and local crops here. We take those views We do the cross entropy and that's how we train this whole thing Okay, and the final last detail is we are updating this this center vector which is used as you remember here So the self center is gonna be updated using the teacher outputs in this function here But you're gonna skip it. You can analyze it yourself. It's fairly simple it's basically following the formula from the paper and we're doing the Exponentially moving average update of the of the center vector. That's it Okay, let's exit this thing. Let's exit this thing and that's it We now have just the bureaucracy stuff. So we clear the gradients of the of the model We are going to now do Backward which is gonna calculate the gradients throughout the whole network of all of the parameters that are obviously a trainable We're gonna do some Clipping of gradients basically here. This line is just going to freeze And not train the last layer if you remember there was this parameter called What's the name of it freeze? So since this freeze last layer is only one and current epoch is zero That means we're actually going to freeze the last layer of the model But that's again a small optimization trick to make this thing more stable and trainable Finally we do the step. That means we're gonna actually update the weights and that's it Just some details around mixed precision training nothing that spectacular Finally we take the the the moment value depending on the on the Epoch we are in on the global step we're in and this is the part where we are updating the teacher networks using the Student weights, so we are basically doing Exponential moving average here. So as you can see the teacher weights are gonna be multiplied by M Which is the this momentum parameter and we're gonna add up 1 minus M times the weights from the student And you can see it on the formula on the screen and that's that's pretty much it. Okay, that was it That was pretty much the the core parts of the training everything else is just logging Dumping those log files and checkpoints. This was the the very meat of the dyno training and how the training looks like again Any feedback is appreciated the more feedback I received that the better I'll know for the next time how to make these videos. Okay. I want to end this video by showing you this visualize attention function Let's quickly run through how this thing works and I think it's fairly fairly much simpler compared to the training So let's kind of dig into it. I'm gonna put a breakpoint here. I'm gonna start debugging and Let's see what happens Okay, again, we're using We at small pet size 8 No pre train weights. This is not that important. We have some image path. I already Inputted the path to this image. So we're gonna be using this image here this this bird image and So that's the image path image size is gonna be 480 not that important again, which is gonna dump in the current directory Threshold is also not that important for now. Let me see what I Am using currently so my current threshold is 0.1. I'm actually gonna delete this so I'm gonna just input the image path here as you can see and Now let's apply this and Restart the code again jumping to here Okay, I'm gonna jump here Let's see here. So Device if I have if you have GPU, it's gonna be using GPU. So it's CUDA here for me. I have a GPU So we're gonna form we're gonna create a model. So this is the same line as in dyno training code So we're gonna basically instantiate the vision transformer small Because of this string with pet size 8 and 0 classes means it won't have the output head is just going to have the the final layer is Gonna be just bunch of tokens the CLS tokens and all of the other tokens, but no no actual linear Classifier had okay. Let's do this. So these lines were just gonna freeze the the model Parameters that means we are not we're gonna set them to not trainable So that means this model is just going to be used as is no training involved Eval evaluation mode that basically sets up depending what you have dropout or batch norm This thing can change the the those layers And that's why it's important when you're using a model in an inference mode to call this eval function We are gonna push the device to the GPU to make things a bit faster And finally because I don't have any pre-trained weights, which is gonna Fetch from a certain URL as you can see here. We form an URL we Fetch those from the torch hub. We're gonna just concatenate the URL to this repository And we're gonna fetch the state dict and we're gonna load the state dictionary So the download actually already happened on my local machine So it's just fetching it from the local storage instead of having to go to the cloud Okay, my image the image name was like incorrect So I had to tweak that in the settings and now I rerun the code and here we are again So because we have an image path, we're gonna open it load the image We're gonna load the image. We're gonna do something that's gonna load the image We're gonna load the image. We're gonna do some transformations. So we're gonna resize the image into 480 by 480 We're gonna convert it into tensor. And again, here is here these Numbers we already saw the much the magic numbers. They just basically represent the mean and standard deviation of the image net data set So, okay, so after that we're gonna transform the image So now we have a tensor that's normalized and resized as you can see here Finally, we make sure that it's divisible by the pet size. So the pet size is currently what eight so we're gonna Make sure that the width and height are are actually Divisible by eight and now we're gonna crop the image here if necessary and on squeeze zero means we're just gonna add Another dimension in the front. So that's just some pie torch like stuff And if we take a look at the image dimension here, so let's take a look at the shape So the shape is just three four hundred eighty four hundred eighty and after we execute this line is gonna be one three four hundred Eighty four hundred eighty. So that's the unsqueezed part then we're gonna divide the resolution by the pet size So this will tell you how many tokens you have across the width dimension and we have sixty and here again We have sixty because remember it's four hundred eighty times four hundred eighty image finally What this does it's gonna affect from the vision transformer. It's gonna get the the self-attention of the image So we're gonna take the image we're gonna Like send it to the GPU because the model is also in GPU. So you don't want to have a discrepancy there We're gonna feed the image into the model and it's going to take from the last layer It's gonna return not the tokens is gonna return the attention coefficients. So that means the following so remember we have 60 times 60 Tokens as we just saw up there. So we have 60 here and 60 here So that means we have three hundred three thousand six hundred tokens in in total plus one for the CLS token So that's three hundred three thousand six hundred one. So the attention dimension we should expect should be basically three thousand six hundred one times three hundred six thousand three Three thousand six hundred and one and we also have six Attention head so we're gonna have something like six times this number here So that's the the the numbers I'm the shape I'm expecting so if I step over and After we now let's find the attentions variable. So the shape is as you can see here six Three thousand six four six hundred blah blah the number we expected basically plus the this one as the first dimension again Those are just some idiosyncrasies of pytorch quick recap How those were formed is basically we have so we have as I said Three thousand six hundred one tokens you take a single token and you basically form its query and then you do a dot product a scale dot product with all of the other keys and that's why you have 3601 attention coefficients for that single token, but since we have 3601 tokens we're gonna have this this this shape here. So hopefully that's clear So as you can see here, we extract the number of heads the number of heads is six And now what we do here is we take we just extract this is just because of one We take this zero here and then we take all of the heads, but we take only the zeros token Which means CLS token and we take one through the end Which means we find the attention coefficients that the CLS has across all of the other let's call them spatial tokens So that's the whole idea there. So that's why we have these numbers here So, let's see what what is the shape we're expecting. We're expecting something like let me write down here Maybe it's easier to see so expecting like six And then we expect because we're taking just the CLS token. So expect six three hundred Three thousand six hundred. So that's the shape where we're actually expecting after after this This line of code. So let's see where that's true So attentions as you can see here, that's that's the actual number six Three thousand six hundred. Okay. I'm just gonna ignore this threshold part and we're just going to reshape the those attentions into Into different shape. So instead of having it flattened out, we're gonna return it back to 60 60 And finally, this is just nearest neighbor Interpolation which means you're gonna take that token Whatever the coefficient is we're just going to replicate the coefficient into the 8 times 8 pixels because that's was the original image shape so that we can visualize the actual attention So that's why we need the interpolation part finally just some making some directories Saving the original image and finally here in this in this section here. We're going to as you can see here We're going to take The attentions for every single head and save them in an image So let me just kind of do f9 there and we're gonna end up having Six images which I'm going to show you in a second and finally I'm gonna ignore this threshold in part again And that's that's pretty much it and here are the attention Maps we got so here you can see the bird image Every single head has different attention pattern and that's it and these are the images you you saw in the actual paper Now you know how they are calculated and visualize awesome. This was a long video hopefully liked it if you did share it out with a friend subscribe hit the bell icon and Join the district community until next time bye bye
[{"start": 0.0, "end": 3.6, "text": " What's cracking guys in this video we're gonna do something very different"}, {"start": 3.6, "end": 9.120000000000001, "text": " So I'm gonna walk you through the code of the dyno model which I've previously covered in one of my videos"}, {"start": 9.46, "end": 17.46, "text": " From Facebook AI research so we're gonna step through the code and understand how exactly the training works line by line in PyTorch"}, {"start": 17.7, "end": 20.6, "text": " So if you find that useful let me know and also"}, {"start": 21.1, "end": 27.04, "text": " Leave a feedback of what you think about this video so that I can improve it over the for the next for the next time"}, {"start": 27.04, "end": 30.24, "text": " So this is super experimental before I dig into the code"}, {"start": 30.4, "end": 37.12, "text": " I'll do like a super short overview of the paper just one minute overview for those of you who haven't watched the video go"}, {"start": 37.12, "end": 38.92, "text": " Ahead and watch it. I'm gonna link it somewhere here"}, {"start": 38.92, "end": 43.620000000000005, "text": " And so let me do a quick recap of what happens so that you have some context before we dig into the code"}, {"start": 43.980000000000004, "end": 51.36, "text": " Okay, so basically in that video what it what I've explained was I explained how exactly this attention mechanism works"}, {"start": 51.379999999999995, "end": 54.32, "text": " But I explained it on a very high level using these diagrams"}, {"start": 54.32, "end": 59.1, "text": " And so from from this image we get this this this attention map"}, {"start": 59.1, "end": 64.22, "text": " And we're gonna see how exactly that works by the way if you if you want me to share these squiggles of the paper"}, {"start": 64.32, "end": 68.36, "text": " With you let me know down in the comments. I'm gonna leave a comment and if I get enough thumbs up"}, {"start": 68.36, "end": 70.96000000000001, "text": " I'm gonna leave I'm gonna think about how I can share these with you"}, {"start": 71.44, "end": 78.58, "text": " Okay, so next up we saw this high level diagram of high down a dyno works so basically we have the student branch"}, {"start": 78.58, "end": 81.8, "text": " We have the teacher branch the teacher branch is updated using this"}, {"start": 81.8, "end": 87.94, "text": " Exponentially moving average from the student like weights, and we also have a stop gradient here"}, {"start": 87.94, "end": 89.94, "text": " Which means we won't be updating the teacher"}, {"start": 90.52, "end": 94.47999999999999, "text": " Finally after applying the softmax we get our district output distributions here"}, {"start": 94.47999999999999, "end": 98.28, "text": " And the whole idea is to make these two distributions be the same"}, {"start": 98.75999999999999, "end": 104.12, "text": " And we do that by across entropy as you can see here, so the p2 times log of p1"}, {"start": 104.92, "end": 110.12, "text": " So the idea is to extract the essence of the image we input images which had different"}, {"start": 110.12, "end": 112.80000000000001, "text": " augmentations as the input to student and teacher and"}, {"start": 113.12, "end": 117.32000000000001, "text": " Hopefully we want to make sure that the distributions are the same because that's the same image"}, {"start": 117.32000000000001, "end": 121.64, "text": " And that's how we extract the kind of the essence of the image, so there will be a high level explanation"}, {"start": 122.12, "end": 127.24000000000001, "text": " So we saw the pseudocode now. We're gonna see the actual code. We saw that we are using"}, {"start": 127.64, "end": 134.68, "text": " So I mentioned augmentations. We're using these global crops, and we're using these local crops, and we are again trying to"}, {"start": 136.16, "end": 138.84, "text": " Make the model output the same distributions for both of these"}, {"start": 138.84, "end": 145.96, "text": " Finally let me see what else I had so explain how K nearest neighbor algorithm works and"}, {"start": 146.52, "end": 147.64000000000001, "text": " Probably in the next video"}, {"start": 147.64000000000001, "end": 154.0, "text": " I'm gonna show you how exactly this looks in code because I think the things I just explained will take like at least an"}, {"start": 154.0, "end": 158.84, "text": " Hour to kind of cover, so let's get back to the code and see how this looks like okay"}, {"start": 158.84, "end": 164.46, "text": " So you've watched the video you have some context from this short recap now. Let's start digging into actual code"}, {"start": 164.46, "end": 170.44, "text": " Okay, so let's put some breakpoint here as I'm going to step through this code and show you exactly how it works"}, {"start": 170.70000000000002, "end": 173.42000000000002, "text": " So let's start with this get args parser function"}, {"start": 173.42000000000002, "end": 180.4, "text": " Okay, so I'm gonna just quickly scheme over these arguments so that you have some context and idea of which variables are gonna be"}, {"start": 180.44, "end": 186.08, "text": " Circulating around the code so first things first. They have the architecture. We're gonna be using the vision transformer small"}, {"start": 186.36, "end": 190.12, "text": " They have many different options, but we're gonna focus on that one batch size"}, {"start": 190.12, "end": 195.52, "text": " I had to increase it to 32 because it was 16 by default and it's for some reason crashing my machine"}, {"start": 195.52, "end": 199.68, "text": " I still haven't been debugging the amount of memory. It's eating up on my GPU"}, {"start": 199.68, "end": 203.6, "text": " So yeah, so output dimension is just 65"}, {"start": 204.36, "end": 207.16, "text": " 65,000 and that's the actual output distributions"}, {"start": 207.16, "end": 212.36, "text": " We just saw in the paper overview which we are going to compare and make sure they are the same for different crops, okay?"}, {"start": 213.20000000000002, "end": 218.92000000000002, "text": " Then we have just normalized last layer. I'm gonna skip this momentum teacher so 0.996"}, {"start": 218.92, "end": 226.51999999999998, "text": " That's the at the actual coefficient that's being used in the exponentially moving average when we are forming the teacher weights from the student weights"}, {"start": 227.32, "end": 232.72, "text": " Again better realization. This is not that important warm-up temperature teacher temperature"}, {"start": 233.79999999999998, "end": 239.6, "text": " Number of epochs all of these are not that important. There's just a schedule of how the temperature of the teacher is gonna change"}, {"start": 239.83999999999997, "end": 241.83999999999997, "text": " throughout the training"}, {"start": 242.11999999999998, "end": 247.79999999999998, "text": " Using mix precision training is gonna be on so I obviously won't be focusing on the optimization"}, {"start": 247.8, "end": 249.8, "text": " stuff in this video"}, {"start": 249.92000000000002, "end": 250.84, "text": " so"}, {"start": 250.84, "end": 252.32000000000002, "text": " weight decay"}, {"start": 252.32000000000002, "end": 256.92, "text": " Okay, that's again. We have some schedule so they have the end and initial value"}, {"start": 257.48, "end": 263.06, "text": " Clipping gradients so that's also basically when you're updating the weights"}, {"start": 263.24, "end": 268.88, "text": " They are going to use this to kind of clip the norm of the of all of the weights before doing an update"}, {"start": 268.96000000000004, "end": 270.96000000000004, "text": " We're gonna see that in a couple of minutes"}, {"start": 271.08000000000004, "end": 275.6, "text": " So number of epochs hundred by default where or not to freeze the last layer"}, {"start": 275.6, "end": 278.96000000000004, "text": " This is just some trick that makes the training a bit easier"}, {"start": 279.72, "end": 281.12, "text": " learning rate"}, {"start": 281.12, "end": 286.04, "text": " So they basically have a linear warm-up first and then they have some kind of I think cosine schedule"}, {"start": 286.44, "end": 293.72, "text": " Until the end of the training so warm-up epochs just tells you across how many epochs you're you're you're taking to do that linear"}, {"start": 293.72, "end": 295.20000000000005, "text": " warm-up thing"}, {"start": 295.20000000000005, "end": 301.18, "text": " minimum learning rate that's the pretty much learning rate at the end of the of the training"}, {"start": 301.64000000000004, "end": 303.64000000000004, "text": " Optimizer they're using Adam"}, {"start": 303.64, "end": 310.47999999999996, "text": " Wide Adam so that's because they have distributed training this dino model is trained on 8 or 16 GPUs originally"}, {"start": 310.47999999999996, "end": 312.32, "text": " But I have only a single GPU here"}, {"start": 312.32, "end": 319.62, "text": " So that's that's it drop path rate is just an argument that is going to modify the the vision transformer architecture"}, {"start": 320.2, "end": 324.44, "text": " Then we have the global crop scales. It's gonna be from 0 4 to 1"}, {"start": 325.08, "end": 330.8, "text": " In contrast that to so I'm gonna quickly get back to that one, but we have eight local crops and we have"}, {"start": 330.8, "end": 337.0, "text": " And we have so local crop scale is gonna be from 0 0 5 to 0.4"}, {"start": 337.0, "end": 339.36, "text": " And that's the same number as here. So that means"}, {"start": 340.0, "end": 341.92, "text": " so that means that"}, {"start": 341.92, "end": 344.24, "text": " This this for the small crops"}, {"start": 344.24, "end": 349.74, "text": " We're gonna take 5% to 40% of the image size and then we're gonna resize that back"}, {"start": 349.76, "end": 356.0, "text": " But like for the big crops we're going to take from 40 to 100% of the image and then we're gonna do the resizing to"}, {"start": 356.0, "end": 361.2, "text": " 24 as for the small crops they're going to be resized to 96 96 pixels"}, {"start": 361.76, "end": 365.86, "text": " Again some details, but like yeah, this video is gonna be about a lot of details"}, {"start": 366.44, "end": 373.3, "text": " We're gonna use image net the thing I did is I actually downloaded a small subset of image net because this thing is huge"}, {"start": 373.3, "end": 375.3, "text": " more than like 150 gigabytes, I think"}, {"start": 376.04, "end": 377.96, "text": " and"}, {"start": 377.96, "end": 380.76, "text": " So from fast AI basically it's called image net"}, {"start": 380.76, "end": 387.36, "text": " Well, I don't know how to pronounce it with two T's and aside from me having to download image net that small subset of image net"}, {"start": 387.36, "end": 391.64, "text": " I also had to do a couple of tricks to make this work on Windows because I'm running on Windows machine"}, {"start": 391.64, "end": 396.02, "text": " So if you want to know about those nitty-gritty details and small bugs and problems"}, {"start": 396.02, "end": 402.96, "text": " I had to solve do join the discord server actually had a thread where as I was in analyzing this code over the past two days"}, {"start": 402.96, "end": 408.12, "text": " I was kind of writing down the steps I took and the problems I encountered"}, {"start": 408.12, "end": 414.04, "text": " Okay, so finally as I said image net where I'm going to save the checkpoints and logs"}, {"start": 414.84000000000003, "end": 416.84000000000003, "text": " seed seed basically serves to"}, {"start": 417.16, "end": 424.36, "text": " Make this code reproducible because otherwise there is a lot of randomness and every time you run this code is gonna have different output and different results"}, {"start": 424.36, "end": 426.36, "text": " And the images are gonna be loaded in different order"}, {"start": 426.36, "end": 431.88, "text": " It makes it easy for me to have some consistency between different runs and it's easier to debug the code"}, {"start": 431.88, "end": 436.68, "text": " So a number of workers not that important all of this is just some distributed settings and"}, {"start": 436.68, "end": 438.68, "text": " That's not that vital. Okay"}, {"start": 438.68, "end": 445.72, "text": " We parse the arguments we create the output directory where we are going to dump the logs the log files and the checkpoints"}, {"start": 445.72, "end": 448.76, "text": " And finally, let's go to the main function and that's trained, you know"}, {"start": 449.8, "end": 455.96000000000004, "text": " So again, this is the in it distributed mode is just a function that's setting up certain settings"}, {"start": 455.96000000000004, "end": 460.76, "text": " When you're doing distributed training on 8 or 16 GPUs again, I'm using"}, {"start": 461.4, "end": 464.84000000000003, "text": " Only one GPU I had to do some modification because of Windows"}, {"start": 464.84, "end": 468.59999999999997, "text": " But yeah, do check out this Discord server for that"}, {"start": 469.32, "end": 473.08, "text": " So we are fixing the random seeds. They're just printing I'm gonna"}, {"start": 473.64, "end": 475.64, "text": " Rise up the console here"}, {"start": 475.64, "end": 477.64, "text": " So they're basically just printing the the"}, {"start": 478.67999999999995, "end": 483.96, "text": " Shock code the the hash of this commit with of the code. I'm currently running"}, {"start": 484.67999999999995, "end": 486.67999999999995, "text": " Then they're just printing some arguments"}, {"start": 487.23999999999995, "end": 491.23999999999995, "text": " Cudian and benchmark set to true is just a certain optimization again"}, {"start": 491.24, "end": 497.32, "text": " That makes the code run faster under certain assumptions again. Not that important. Okay"}, {"start": 498.04, "end": 504.84000000000003, "text": " This is fairly important data augmentation dino. I'm gonna quickly walk you through this code and then I'm gonna step over it"}, {"start": 505.56, "end": 507.56, "text": " in the execution loop"}, {"start": 507.88, "end": 512.28, "text": " But basically as you can see here, they have flip and color jitter"}, {"start": 513.0, "end": 518.04, "text": " Transform so that's just a compose. That means we are composing a couple of different transforms and"}, {"start": 518.04, "end": 521.24, "text": " These are organized so that they do the flipping and the color"}, {"start": 521.88, "end": 527.64, "text": " Like photometric augmentations. So we have as you can see here random horizontal flip with 50 percent"}, {"start": 528.1999999999999, "end": 530.1999999999999, "text": " With 50 chance, we're gonna flip it"}, {"start": 530.68, "end": 537.88, "text": " Either flip it or not flip it then we have the we're gonna randomly apply with certain probability like 80 percent"}, {"start": 538.52, "end": 543.56, "text": " This color jitter which is going to change the brightness the contrast the saturation the hue"}, {"start": 543.56, "end": 549.7199999999999, "text": " So basically our photometric augmentations and finally random grayscale is with 20 chance. We're gonna"}, {"start": 550.52, "end": 551.8, "text": " kind of"}, {"start": 551.8, "end": 553.8, "text": " Apply the the grayscale transformation"}, {"start": 554.76, "end": 560.92, "text": " The then we have normalization. This is your standard stuff. We are converting the the images to pi torch tensor"}, {"start": 561.4799999999999, "end": 565.7199999999999, "text": " We are normalizing them and don't get confused by these magic numbers"}, {"start": 566.04, "end": 570.04, "text": " Even though it'd be really nice if the author has actually extracted this into a constant"}, {"start": 570.04, "end": 574.4399999999999, "text": " Because they've been using this all around the place. So this is just your image image net"}, {"start": 575.0, "end": 578.52, "text": " Statistics so the mean and the standard deviation of the image net images"}, {"start": 579.0, "end": 579.4, "text": " Okay"}, {"start": 579.4, "end": 583.56, "text": " So for the fun part this random resize crop is the thing that does the magic of dino"}, {"start": 583.9599999999999, "end": 586.68, "text": " So global crop scale is again the thing I just mentioned"}, {"start": 586.68, "end": 589.56, "text": " So from we're gonna take 40 to 100 percent of the image"}, {"start": 589.56, "end": 594.12, "text": " We're gonna cut it there and then we're gonna resize it back to 224 by 224"}, {"start": 594.76, "end": 596.76, "text": " Using bicubic interpolation"}, {"start": 596.76, "end": 600.4399999999999, "text": " 224 by 224 using bicubic interpolation method"}, {"start": 600.92, "end": 607.0, "text": " Then they're gonna apply some photometric augmentations and the flipping the gaussian blur and we're gonna normalize the images"}, {"start": 607.88, "end": 609.88, "text": " We have a second global"}, {"start": 610.2, "end": 616.12, "text": " Crop because if you remember the details from the paper, we have two global crops and we have eight local crops"}, {"start": 616.12, "end": 619.08, "text": " Although obviously that's you can you can experiment with that"}, {"start": 619.8, "end": 624.52, "text": " So again, a random resize crop everything is the same. They just apply solarization here"}, {"start": 624.52, "end": 629.16, "text": " Uh, if you're not familiar with solarization, i'm just gonna put the image on the screen so you can see a visual example"}, {"start": 629.24, "end": 631.56, "text": " I guess the image is going to explain that much faster than I will"}, {"start": 632.28, "end": 637.72, "text": " Finally, we have the the local crops. So that means we are applying the local crop scale"}, {"start": 637.72, "end": 644.84, "text": " So that means we are now using that 1.5 percent to 40 percent of the image and then we resize that back into 96 by 96 images"}, {"start": 644.84, "end": 651.64, "text": " So these crops are gonna these images that came from the smaller crop of the image are gonna be of smaller resolution"}, {"start": 651.64, "end": 656.84, "text": " And 96 by 96 again flipping and color rendering a gaussian blur normalization"}, {"start": 657.24, "end": 659.24, "text": " Standard stuff so that's it"}, {"start": 659.64, "end": 664.04, "text": " So basically what's gonna happen is this data augmentation dino"}, {"start": 664.4399999999999, "end": 668.4399999999999, "text": " uh is gonna be passed to the data set in pytorch and"}, {"start": 668.68, "end": 672.92, "text": " Then every time you try to fetch an image from the data set it's gonna apply"}, {"start": 673.08, "end": 677.0, "text": " It's gonna append one crop here one global crop the second global crop"}, {"start": 677.0, "end": 683.08, "text": " And then we're gonna have iterating a for loop. So eight times and do these small local crops"}, {"start": 683.56, "end": 686.84, "text": " And then we're gonna return like a list of 10 crops in total"}, {"start": 687.4, "end": 690.28, "text": " This is just on code. I wrote i'm gonna experiment with that a bit later"}, {"start": 690.36, "end": 696.06, "text": " But let me get back to the actual line where we stopped. Okay, so that was the that was the uh data augmentation"}, {"start": 696.76, "end": 702.12, "text": " Of dino. I think that was pretty important detail by the way, i'll be adding time stamps to all of these logical sections"}, {"start": 702.12, "end": 708.76, "text": " So feel free to skip if you understand some part you can just skip to the part you you want to understand better"}, {"start": 709.5600000000001, "end": 716.04, "text": " Okay, continuing on we are creating this data set. Uh, so it's gonna link to the as you can see here image net train"}, {"start": 716.58, "end": 719.96, "text": " Uh subdirectory and we're gonna pass the transform I already mentioned"}, {"start": 720.28, "end": 726.84, "text": " Which means every time you fetch an image from this data set is gonna apply those 10 transformations and we're gonna get those 10 crops back"}, {"start": 727.32, "end": 730.52, "text": " Uh distributed sampler. So just ignore the distributed part"}, {"start": 730.52, "end": 734.76, "text": " This is just going to shuffle the images once we input that into the data loader"}, {"start": 734.76, "end": 737.88, "text": " So as we can see we have a data set here. We have the sampler"}, {"start": 738.12, "end": 743.16, "text": " We have I think 64 images per gpu and since I have a single gpu that's going to be"}, {"start": 743.88, "end": 749.64, "text": " 64 images in total a number of workers just optimization stuff peel memory as well. That just kind of helps you"}, {"start": 751.3199999999999, "end": 757.96, "text": " Basically, uh transferred images from the cpu to the gpu faster because there's this uh dedicated slot of the memory"}, {"start": 757.96, "end": 765.08, "text": " So i'm currently going through every single detail, uh, because i'm experimenting but like, uh, i'm later on i'm gonna abstract many details"}, {"start": 765.1600000000001, "end": 770.36, "text": " So let me know whether you actually want everything explained or you just want me to cover the main idea"}, {"start": 770.6800000000001, "end": 775.0, "text": " Uh, this is experimental video. So, uh, please give me that feedback. And yeah, uh"}, {"start": 775.5400000000001, "end": 781.88, "text": " droplast means because we have a data set of I I think uh, like I have around 9 000 something images"}, {"start": 782.12, "end": 784.9200000000001, "text": " Because every batch has 64 images as we saw here"}, {"start": 784.92, "end": 792.28, "text": " Uh, that means that probably the last batch won't have 64 because unless the the number of images in our training data set is divisible"}, {"start": 792.28, "end": 798.1999999999999, "text": " But 64 we're gonna have the last batch being less than 64 and this droplast just means we're gonna drop"}, {"start": 798.4399999999999, "end": 802.5999999999999, "text": " The last batch because that can cause some issues probably because of the shape being different"}, {"start": 803.3199999999999, "end": 809.24, "text": " So that's the reason you sometimes see this droplast set to true. Okay data loaded there are uh, as you can see here"}, {"start": 810.5, "end": 813.64, "text": " 9469 images, okay"}, {"start": 813.64, "end": 815.24, "text": " um"}, {"start": 815.24, "end": 820.6, "text": " This is just some bureaucracy stuff. Uh, they did because they used to to name these models"}, {"start": 821.16, "end": 828.04, "text": " Date now, uh, they're just kind of replacing them with vat, but the architecture is already vat small. So nothing's gonna happen there"}, {"start": 828.52, "end": 829.8, "text": " um, so"}, {"start": 829.8, "end": 831.48, "text": " basically"}, {"start": 831.48, "end": 837.8, "text": " This says that so vat is just uh, like a shorthand notation for the all of the vision transformer models"}, {"start": 838.2, "end": 840.68, "text": " That exist in this vision transformer pi file"}, {"start": 840.68, "end": 846.8399999999999, "text": " So we're gonna because we are vision transformer we're gonna enter this if statement and we're going to as you can see here"}, {"start": 846.8399999999999, "end": 852.4399999999999, "text": " We're gonna form so this statement here may confuse you. So this is just a list of all of the different"}, {"start": 853.16, "end": 860.3599999999999, "text": " like uh functions from this vision transformer file and this is just going to instantiate the model and we're going to instantiate it with the"}, {"start": 860.76, "end": 866.04, "text": " 32 by 32 so the patch will be 32 by 32 pixels and we're gonna use this"}, {"start": 866.04, "end": 871.24, "text": " Uh drop path rate of 0.1. So the quickest way you can understand how this thing"}, {"start": 871.64, "end": 877.42, "text": " What this thing does is just put a print statement in there do this, uh, like just print the dictionary"}, {"start": 878.1999999999999, "end": 880.5999999999999, "text": " Uh set a breakpoint there stop the code"}, {"start": 881.54, "end": 884.4399999999999, "text": " Re-execute again and we're gonna see what happens there. So i'm gonna"}, {"start": 885.0799999999999, "end": 889.9599999999999, "text": " Just skip these skip these i'm gonna get here. I'm gonna open the console"}, {"start": 890.4399999999999, "end": 892.52, "text": " And after I print this you're gonna see"}, {"start": 892.52, "end": 896.04, "text": " Uh, whoops bunch of stuff and as you can see here"}, {"start": 896.12, "end": 903.56, "text": " It's actually printing this whole vision transformer pi file and the thing is uh, it's some of the some of the functions are organized"}, {"start": 903.64, "end": 908.36, "text": " Into dictionaries. So this is just your python stuff. So that means we're gonna have somewhere in here"}, {"start": 908.6, "end": 913.64, "text": " We're gonna have uh, like these functions. Let me let me jump to that file. So we're gonna have somewhere in there"}, {"start": 913.64, "end": 915.64, "text": " We're gonna have this v80 small"}, {"start": 915.96, "end": 921.16, "text": " Function and then we're gonna pass the patch size and uh additional we are passing that drop path"}, {"start": 921.16, "end": 926.76, "text": " Uh coefficient so that's the the thing that happens there. So basically we're instantiating the v80 small model here"}, {"start": 927.3199999999999, "end": 931.16, "text": " Uh, then we're doing the same thing for the teacher network. So that was a student network"}, {"start": 931.16, "end": 933.9599999999999, "text": " We have the teacher network and since we're using uh"}, {"start": 934.8399999999999, "end": 941.8, "text": " Vision transformer small the embedding dimension is going to be 384. Uh, that's just the hyper parameter they they kind of"}, {"start": 943.24, "end": 945.48, "text": " Decided to use okay"}, {"start": 945.48, "end": 949.9599999999999, "text": " So they have in these other branches they're using excite and some other stuff we don't care about"}, {"start": 949.96, "end": 951.4000000000001, "text": " so"}, {"start": 951.4000000000001, "end": 957.72, "text": " Now this is another important part. Basically, uh, what they do is they they wrap the student network"}, {"start": 957.72, "end": 963.88, "text": " So this is the vision transformer and they put this dino head on top of it. So dino head is just a simple mlp"}, {"start": 964.44, "end": 969.08, "text": " Uh with some additional tweaks and i'm gonna jump into those details right now"}, {"start": 969.4000000000001, "end": 973.5600000000001, "text": " So the the main point is they are forming uh, this this mlp"}, {"start": 974.2800000000001, "end": 979.0, "text": " And since number of layers is three by default, that means the mlp is gonna have three linear layers"}, {"start": 979.0, "end": 981.0, "text": " that means we're gonna uh"}, {"start": 981.0, "end": 986.44, "text": " Execute the else branch and we're gonna form the mlp that just consists out of as you can see here"}, {"start": 986.6, "end": 992.12, "text": " We have some potentially some batch norms potentially some uh, like gelu, uh, like uh"}, {"start": 992.92, "end": 997.64, "text": " Activation functions, uh, but most of all most importantly we have this linear function"}, {"start": 998.12, "end": 1000.12, "text": " Uh, so basically it's an mlp"}, {"start": 1000.68, "end": 1005.08, "text": " And then we just initialize the way it's using this self apply that's an inbuilt"}, {"start": 1005.08, "end": 1008.44, "text": " Function that's a part of this nn module"}, {"start": 1009.4000000000001, "end": 1014.6, "text": " class and then we're gonna form the last layer, uh by creating yet another"}, {"start": 1015.0, "end": 1020.76, "text": " Linear layer and the linear layer is gonna have so the output dimension here is like 65 000 something"}, {"start": 1021.0, "end": 1027.8, "text": " So that's the actual output of the whole uh pipeline and then we're gonna compare those outputs by cross entropy later on"}, {"start": 1028.04, "end": 1032.92, "text": " So that's the last layer. Uh, the interesting part here is this weight norm is just a hook function"}, {"start": 1032.92, "end": 1038.04, "text": " Uh what it does is once you put this weight g to all of ones"}, {"start": 1038.3600000000001, "end": 1041.72, "text": " That means they are going to normalize the actual linear layer"}, {"start": 1041.88, "end": 1047.48, "text": " So it's just going to make sure that the l2 norm of the weights of that linear layer are equal to one"}, {"start": 1047.72, "end": 1052.52, "text": " That's basically it and if they decide to do this normalization of the last layer if this is set to true"}, {"start": 1052.92, "end": 1059.4, "text": " Then as you can see here that those gradients are going to be set to false which means the following during the training"}, {"start": 1059.4, "end": 1065.8000000000002, "text": " Those uh, those weights will not will not be updated. Which means that the l2 norm is going to be kept at one"}, {"start": 1066.3600000000001, "end": 1071.0, "text": " Uh, which means we're normalizing the last layer hence the norm last layer. I think it's pretty self-explanatory"}, {"start": 1071.5600000000002, "end": 1077.5600000000002, "text": " So again simple mlp on top of it. We have the last layer which has the 65 000, uh something"}, {"start": 1078.0400000000002, "end": 1084.3600000000001, "text": " Uh number of neurons as the output and that's it. Optionally we have these uh, these uh, like normalization stuff"}, {"start": 1084.76, "end": 1087.3200000000002, "text": " Okay, getting back here. So that's the dino head"}, {"start": 1087.32, "end": 1092.12, "text": " We stick it on top of the student and one additional detail which we need to go through"}, {"start": 1092.12, "end": 1098.04, "text": " This is this multi-crop wrapper again. I have timestamps just jump if you don't if you're not interested into this section"}, {"start": 1098.6, "end": 1100.6, "text": " Uh, so the multi-crop wrapper"}, {"start": 1101.3999999999999, "end": 1103.0, "text": " What it does is the following"}, {"start": 1103.0, "end": 1107.72, "text": " Um, you know, i'm actually just gonna put a break point here and then after we we get to this part"}, {"start": 1107.72, "end": 1111.8, "text": " We'll understand what multi-crop wrapper does but basically on a high level for now"}, {"start": 1111.8, "end": 1116.36, "text": " it's basically just gonna deal with those crops passed them through the the pipeline and then"}, {"start": 1116.36, "end": 1121.08, "text": " Outcomes the result from the student the final the final output and the final output from the teacher"}, {"start": 1121.3999999999999, "end": 1126.52, "text": " We'll see that in a in a once we we hit that part of the code. Okay, uh getting back here"}, {"start": 1126.6, "end": 1129.0, "text": " So that was how we formed the student part"}, {"start": 1129.3999999999999, "end": 1134.6799999999998, "text": " So again, it's a multi-crop wrapper around student which is vision transformer and dino head which is mlp"}, {"start": 1135.08, "end": 1140.28, "text": " Same thing goes for for the teacher. We have teacher we have dino head and that's it"}, {"start": 1140.6, "end": 1144.84, "text": " Some minor differences like whether we are not we are normalizing the last layer or not"}, {"start": 1144.84, "end": 1147.6399999999999, "text": " But that's it and we are not using the bet normalization"}, {"start": 1147.6399999999999, "end": 1154.04, "text": " So this is uh, there is no better normalization in this whole pipeline, which is something we we strive for"}, {"start": 1155.08, "end": 1156.12, "text": " okay"}, {"start": 1156.12, "end": 1162.84, "text": " Uh, we're just pushing the the student and teacher to the gpu. That's this dot kuda that that's what it means and finally"}, {"start": 1163.3999999999999, "end": 1166.6, "text": " Because as I said, we don't have any any batch norms"}, {"start": 1167.24, "end": 1173.8, "text": " That means we're gonna enter the else branch and that means that the teacher network will not be using this distributed data parallel"}, {"start": 1173.8, "end": 1178.52, "text": " Which is just a wrapper that makes it easy to train these models in a distributed fashion"}, {"start": 1179.0, "end": 1184.68, "text": " Uh, okay continuing on we wrap the student network, uh into this distributed data parallel"}, {"start": 1184.68, "end": 1187.08, "text": " But but they only have a single gpu here"}, {"start": 1187.08, "end": 1193.32, "text": " So the index zero means the default gpu you have on your system. I have a rtx 2080. So that's it"}, {"start": 1194.12, "end": 1199.1599999999999, "text": " Then we are loading the as you can see here. We are loading the state dictionary directly from the student"}, {"start": 1199.16, "end": 1204.1200000000001, "text": " Uh module so and they say here nicely teacher and student start with the same weights"}, {"start": 1204.52, "end": 1210.68, "text": " So initially we're gonna start with the same weights later on we're gonna use this exponentially moving average to update the teacher weights"}, {"start": 1211.16, "end": 1213.96, "text": " Uh, finally as I said, there is a stop gradient"}, {"start": 1214.1200000000001, "end": 1219.3200000000002, "text": " Uh, if you saw the the diagram there is a stop gradient and this is how you implement the stop gradient in pytorch"}, {"start": 1219.48, "end": 1221.48, "text": " you basically just iterate through all of the"}, {"start": 1222.28, "end": 1226.8400000000001, "text": " Like parameters and you said there are gradients to false. So that means you won't be trainable"}, {"start": 1226.84, "end": 1231.1599999999999, "text": " I'm gonna skip all of this just f9 and that's it"}, {"start": 1231.32, "end": 1237.6399999999999, "text": " So we we have an output here, uh, the student and teacher are built. They're both vision transformer small networks"}, {"start": 1237.72, "end": 1243.0, "text": " Okay, this part is fairly important and i'm gonna probably uh step through it a bit later"}, {"start": 1243.32, "end": 1248.12, "text": " But let me just give you a glimpse. So i'm gonna put uh, like a like a breakpoint here in the forward function"}, {"start": 1248.6799999999998, "end": 1254.36, "text": " And as you can see here, it's just accumulating the parameters that we passed in the init function"}, {"start": 1254.36, "end": 1263.1599999999999, "text": " Uh, it's just registering this center vector. So if register buffer what it does it makes sure that the this this this vector here"}, {"start": 1263.7199999999998, "end": 1270.9199999999998, "text": " Uh is a part of the state dictionary even though it's not trainable. So this center vector will not be trainable"}, {"start": 1271.1599999999999, "end": 1277.56, "text": " We're not going to update it using gradients. We're going to update it using the exponentially moving average again"}, {"start": 1277.7199999999998, "end": 1280.4399999999998, "text": " So that's the reason they're using register buffer. So it's not trainable"}, {"start": 1280.44, "end": 1284.52, "text": " So it's not a model parameter in the pytorch, uh, like uh terminology"}, {"start": 1285.0, "end": 1288.76, "text": " And but we still need it in the actual state dictionary of the model"}, {"start": 1289.24, "end": 1295.72, "text": " Um, okay. Finally, they have some teacher temperature schedule. So with the current parameters, this is actually going to return 0.04"}, {"start": 1296.68, "end": 1302.6000000000001, "text": " Throughout all of this all of the epochs, but what it's it's kind of envisioned to have a linear ramp up here"}, {"start": 1302.8400000000001, "end": 1305.3200000000002, "text": " Although that's not that important. Let's get back to the dyno loss"}, {"start": 1305.32, "end": 1311.6399999999999, "text": " Okay, i'm gonna step over this i'm gonna later once we are actually using the dyno loss i'm gonna step through the forward function"}, {"start": 1312.52, "end": 1314.04, "text": " um"}, {"start": 1314.04, "end": 1316.04, "text": " This just not important"}, {"start": 1317.1599999999999, "end": 1323.24, "text": " Whether we are regularizing or not regularizing certain parameters of the student network. We're gonna create the atom"}, {"start": 1323.8799999999999, "end": 1325.8799999999999, "text": " wide optimizer"}, {"start": 1325.8799999999999, "end": 1329.24, "text": " And finally because we're using the mixed precision training"}, {"start": 1329.24, "end": 1336.6, "text": " We're just going to wrap this create this fp16. So that means we're using 16 bits training these models"}, {"start": 1337.16, "end": 1342.04, "text": " Okay here they're just creating uh schedulers for the learning rate. So it's going to be a cosine scheduler"}, {"start": 1342.84, "end": 1348.76, "text": " Um, then we have a weight decay schedule against cosine scheduler and finally momentum scheduler"}, {"start": 1349.56, "end": 1354.84, "text": " Basically, uh, that's the thing that is used to update the teacher"}, {"start": 1354.84, "end": 1358.76, "text": " Uh weights using the student weights. So that's that parameter"}, {"start": 1359.1599999999999, "end": 1364.36, "text": " Uh, and again, it's just a cosine scheduler. Okay, so that's it. I think we're we're good to go"}, {"start": 1364.6799999999998, "end": 1366.6799999999998, "text": " We're getting closer to the actual training loop"}, {"start": 1367.1599999999999, "end": 1373.0, "text": " Uh, this already took a lot of time. So uh, we're just going to this this restart from checkpoint"}, {"start": 1373.0, "end": 1379.32, "text": " I think we're gonna skip it because we don't have any checkpoints for now. So just put a breakpoint there. We're gonna enter it"}, {"start": 1380.4399999999998, "end": 1382.76, "text": " Okay, let me step over step over over"}, {"start": 1382.76, "end": 1389.32, "text": " Okay, let me step over step over. Okay, and we're gonna just return from that function since we have don't have any checkpoints"}, {"start": 1389.56, "end": 1392.28, "text": " Okay, so that's just uh dummy for now. Um"}, {"start": 1392.92, "end": 1398.28, "text": " Start epoch is gonna start with zero. We're just uh logging the starting time just some bureaucracy stuff"}, {"start": 1398.44, "end": 1401.8799999999999, "text": " And finally here is the main loop. Okay, we are here"}, {"start": 1402.36, "end": 1406.76, "text": " so we're starting from the epoch zero we're gonna run for 100 epochs and"}, {"start": 1406.76, "end": 1412.68, "text": " Uh ignore this line. This is just something that has to do with this distributed sampler. Otherwise, you wouldn't have this line"}, {"start": 1413.24, "end": 1417.08, "text": " Um, so this is the actual meat of the whole whole script"}, {"start": 1417.32, "end": 1423.64, "text": " So it's training one epoch is gonna train the the dino for one epoch and then later on we're just gonna have some some"}, {"start": 1424.12, "end": 1429.72, "text": " Logging nothing nothing very fancy. So i'm gonna step into this f7"}, {"start": 1429.72, "end": 1436.1200000000001, "text": " And let's see how this function looks like. Okay, so metric logger not an important again"}, {"start": 1437.0, "end": 1442.52, "text": " Okay, this is where the fun starts. So we have the data loader and we're going to fetch"}, {"start": 1443.16, "end": 1449.72, "text": " Like this images so this images is going to be a list of 10 elements and each element is going to be a batch of images"}, {"start": 1450.04, "end": 1452.44, "text": " So each element is going to be 64 images"}, {"start": 1452.68, "end": 1457.4, "text": " So that means so because 10 elements because as you remember we have two global crops"}, {"start": 1457.4, "end": 1462.0400000000002, "text": " We have eight local crops. So that means if we take like the zeroth element of this images list"}, {"start": 1462.3600000000001, "end": 1468.2800000000002, "text": " That's those are the global crops and we have 64 images and if we take uh, like the second index"}, {"start": 1468.44, "end": 1470.44, "text": " That means that's the first local crop"}, {"start": 1471.0, "end": 1474.92, "text": " Like set of images a batch of local crops. And so as you can see here"}, {"start": 1474.92, "end": 1477.88, "text": " I just pre-wrote some code here to make this a bit faster"}, {"start": 1478.2800000000002, "end": 1481.8000000000002, "text": " I'm gonna stop the code execution and i'm gonna rerun it again"}, {"start": 1482.0400000000002, "end": 1484.44, "text": " I'll I'll need to make another tweak to make this work"}, {"start": 1484.44, "end": 1487.8, "text": " So that's that means i'm gonna have to add additional crop"}, {"start": 1488.3600000000001, "end": 1493.8, "text": " So what this does is we are just going to preserve the image as it is without doing any any crops"}, {"start": 1494.1200000000001, "end": 1499.3200000000002, "text": " So that's just a reference image and that was that's going to be prepended as the first crop"}, {"start": 1499.3200000000002, "end": 1504.92, "text": " So now we have 11 crops and hopefully now this is going to work. Let me let me let me run this"}, {"start": 1505.96, "end": 1509.16, "text": " Uh, let me go back to the code where we were. Okay, here we are"}, {"start": 1509.16, "end": 1514.1200000000001, "text": " So what i'm doing here is as you remember I prepended this original image"}, {"start": 1514.76, "end": 1520.76, "text": " So now we have 11 elements instead of 10 and this zero will just be indexing the original images"}, {"start": 1521.0, "end": 1524.3600000000001, "text": " And i'm just gonna fetch like one image out of those 64 images"}, {"start": 1524.76, "end": 1531.48, "text": " I'm gonna convert those to non-pi do some move axes because the channel is actually channel first and now we want to have channel last"}, {"start": 1531.8000000000002, "end": 1533.96, "text": " For for the plot function of matplotlib"}, {"start": 1534.52, "end": 1538.2, "text": " So second line what it does is it will extract the global crop"}, {"start": 1538.2, "end": 1544.76, "text": " So remember now the the the index one and zero the index one and two are actually referencing"}, {"start": 1545.64, "end": 1549.0800000000002, "text": " The global crops and again, i'm gonna take just one single image"}, {"start": 1549.64, "end": 1555.56, "text": " Finally, we have the local crop. So index three through ten are all going to contain local crops"}, {"start": 1555.56, "end": 1560.76, "text": " So i'm just gonna take one arbitrary local crop and finally, let's let's plot those results. So"}, {"start": 1560.76, "end": 1568.6, "text": " Um, here you can see the the original image, uh, just some dude playing golf"}, {"start": 1569.08, "end": 1575.48, "text": " So i'm gonna now exit that remember that image and now we're gonna see a global crop as you can see it's uh,"}, {"start": 1575.8, "end": 1580.2, "text": " There is a crop but there is also severe solarization and other photometric effects"}, {"start": 1580.28, "end": 1583.4, "text": " So you can't really see it perfectly, but let's see the the local crop"}, {"start": 1583.8799999999999, "end": 1587.24, "text": " So the local crop is as you can see now, it's totally clear"}, {"start": 1587.24, "end": 1592.68, "text": " We're just taking a single small crop and we're resizing it into this image of 96 by 96"}, {"start": 1592.92, "end": 1595.64, "text": " And we also apply some augmentations here. So that's it"}, {"start": 1595.72, "end": 1601.88, "text": " That's the now you have an understanding of how images look like and I encourage you when you're debugging and understanding the code"}, {"start": 1602.36, "end": 1608.2, "text": " Do make these visualizations as much as possible because that helps you understand what's going on in your code"}, {"start": 1608.84, "end": 1614.92, "text": " That plus print statement, I guess that's pretty much how I debug the code. Although there are sometimes more elegant ways to do it. Okay"}, {"start": 1614.92, "end": 1616.76, "text": " so"}, {"start": 1616.76, "end": 1619.3200000000002, "text": " Um, let's let's continue here. So okay"}, {"start": 1619.4, "end": 1622.92, "text": " This line is just forming a global step depending on the epoch we are in"}, {"start": 1623.4, "end": 1627.88, "text": " Depending on the number of images in the data set depending on the current iteration inside of this"}, {"start": 1628.76, "end": 1632.1200000000001, "text": " Inside of this loop. Okay, so that's just global step and"}, {"start": 1633.0, "end": 1639.5600000000002, "text": " Then what they're going to do here is depending on the schedule current schedule. We're gonna update the way decay and learning rate"}, {"start": 1639.96, "end": 1643.5600000000002, "text": " Uh, depending on these param groups, it's not that important. I'm just gonna skip that part"}, {"start": 1643.56, "end": 1647.72, "text": " So we're just adapting the way decay and learning rates depending on the current step"}, {"start": 1647.72, "end": 1652.84, "text": " Okay, so skipping that we're putting the images on the gpu. So that's this step"}, {"start": 1653.8, "end": 1656.04, "text": " Let me just whoops. Let me just skip over that"}, {"start": 1657.3999999999999, "end": 1665.08, "text": " Uh, and finally this part, uh is just a context context manager because we have this mixed precision training"}, {"start": 1665.08, "end": 1670.52, "text": " And it's not none. So we're gonna enter this but now it's not going to work because I prepended all of this"}, {"start": 1670.52, "end": 1672.52, "text": " I'm just gonna quickly comment this out"}, {"start": 1672.52, "end": 1678.28, "text": " Uh and comment this out and i'm gonna restart the training and get back to you. Okay. Here we are"}, {"start": 1678.36, "end": 1680.84, "text": " Uh, so this is as I said very important step"}, {"start": 1681.0, "end": 1687.56, "text": " We're taking the global crops and we are passing only the global crops through the teacher to get the teacher output and let's see"}, {"start": 1687.8799999999999, "end": 1691.96, "text": " So analyzing the shape of the output is super important. I can't stress that enough"}, {"start": 1692.36, "end": 1697.4, "text": " So let's now see whoops. I have uh, okay, so we have a multi crop wrapper"}, {"start": 1697.56, "end": 1700.76, "text": " Okay, this is the part we we want to understand a little bit better"}, {"start": 1700.76, "end": 1703.0, "text": " um"}, {"start": 1703.0, "end": 1704.28, "text": " so x"}, {"start": 1704.28, "end": 1710.04, "text": " Is just a list of two elements again, uh, each contains a batch of crops"}, {"start": 1710.52, "end": 1713.96, "text": " These are global crops. So because it's a list we're gonna skip this part"}, {"start": 1714.04, "end": 1720.04, "text": " So this is just a fancy way of separating the global from the local crops and it doesn't make much sense for the teacher network"}, {"start": 1720.52, "end": 1725.0, "text": " Because it's only using global crops. So we're just gonna have number two here. So as you can see here"}, {"start": 1725.0, "end": 1730.36, "text": " Uh index crops just has number two. Uh, this is going to make more sense for the for the student network"}, {"start": 1730.68, "end": 1732.68, "text": " so now what we do is"}, {"start": 1733.48, "end": 1738.76, "text": " We iterate here and we're gonna take as you can see from zero to two"}, {"start": 1739.08, "end": 1742.68, "text": " So that means we're gonna concatenate all of those, uh global crops"}, {"start": 1742.68, "end": 1747.8, "text": " We have 64 images in one in one in one element. We have 64 images in the second one"}, {"start": 1747.96, "end": 1750.92, "text": " We're just going to concatenate them and now we have 128"}, {"start": 1750.92, "end": 1758.28, "text": " images in a single tensor and then we're gonna pass that through the backbone which is basically the vision transformer"}, {"start": 1758.8400000000001, "end": 1765.0800000000002, "text": " Uh of the teach network. So that's going to form what so the the let's try and figure out the shape so"}, {"start": 1765.94, "end": 1768.28, "text": " 128 and because we are outputting"}, {"start": 1769.16, "end": 1774.68, "text": " Only the cls token. So that will be the output from the vision transformer. So it has 384"}, {"start": 1774.68, "end": 1780.8400000000001, "text": " Uh, like uh neurons, right? So we're gonna have hopefully, um"}, {"start": 1781.48, "end": 1786.92, "text": " What so 128 times 384, okay, let's let's see whether the prediction was right"}, {"start": 1787.0800000000002, "end": 1793.0, "text": " So it's kind of good to have these predictions and then uh tweak your brain depending on whether you made made a mistake or not"}, {"start": 1793.24, "end": 1797.0, "text": " I guess that's a similar way how these neural networks work. So the out"}, {"start": 1797.5600000000002, "end": 1800.2, "text": " Shape here as you can see the shape is 128"}, {"start": 1800.2, "end": 1807.24, "text": " 128, um, comma 384 so that's it i'm gonna skip this part and now we're gonna gonna this is going to accumulate the"}, {"start": 1807.88, "end": 1812.1200000000001, "text": " The output and this will again make more sense for the student network. Now is just gonna"}, {"start": 1812.68, "end": 1817.72, "text": " Uh, we're gonna just copy paste pretty much this output from the vision transformer here and that's it"}, {"start": 1817.8, "end": 1820.8400000000001, "text": " We're gonna exit the loop and now this is the actual dino head"}, {"start": 1820.8400000000001, "end": 1826.76, "text": " So this is the mlp and now we're after passing it here. We're gonna end up with those 65 000 something"}, {"start": 1826.76, "end": 1832.92, "text": " So we're gonna have 128 times 65,000 something. Let's let's see whether that that's true. Okay"}, {"start": 1833.32, "end": 1836.04, "text": " Uh, whoops, let me get out of here"}, {"start": 1836.6, "end": 1842.44, "text": " Teacher output. Let's see whether the teacher output is as I predicted so teacher output. Let me kind of"}, {"start": 1844.92, "end": 1846.76, "text": " So 128"}, {"start": 1846.76, "end": 1849.8, "text": " 65 000, that's it. So the predictions were right. So now"}, {"start": 1850.36, "end": 1855.48, "text": " This is the interesting part. We have a student network and we are passing both the global crops as well as the local crops"}, {"start": 1855.48, "end": 1861.4, "text": " So let's enter the loop again again. It's a list. So i'm gonna skip this and now it's a bit more interesting. So it's gonna"}, {"start": 1862.3600000000001, "end": 1865.88, "text": " Make uh return the ids where the resolution changes"}, {"start": 1865.88, "end": 1872.2, "text": " So we have 20 remember the global crops are 224 by 224 and the local crops are 96 by 96"}, {"start": 1872.44, "end": 1876.84, "text": " So what this line of code actually does is it finds a separation and returns the indices?"}, {"start": 1877.56, "end": 1882.44, "text": " Of the place where the resolution changes so it's going to return as you will soon see 2 and 10"}, {"start": 1882.44, "end": 1887.8, "text": " I think or something like that. Yeah 2 and 10 so that's because after 2 we drop the small resolution"}, {"start": 1887.8, "end": 1891.56, "text": " And then we keep the small resolution until the end and that's 10 crops"}, {"start": 1891.96, "end": 1897.3200000000002, "text": " So now we are iterating again as you can see here. We're first going to pass 0 through 2"}, {"start": 1897.72, "end": 1899.72, "text": " So that's global crops"}, {"start": 1900.04, "end": 1905.5800000000002, "text": " And we're going to pass that to the the network here. So that's going to be again 128"}, {"start": 1906.6000000000001, "end": 1911.1000000000001, "text": " So that was that is going to be as you can see here 128 384"}, {"start": 1911.1, "end": 1917.5, "text": " Uh, and we're just going to concatenate that with this output, which is just a placeholder variable"}, {"start": 1917.58, "end": 1922.62, "text": " You can see here is torch empty and we're just concatenating the results to it and that's it"}, {"start": 1922.86, "end": 1927.8999999999999, "text": " Now we're gonna put this the start index is now going to be 2 and the end index is going to be 10"}, {"start": 1928.3, "end": 1930.1399999999999, "text": " So now we're passing 2"}, {"start": 1930.1399999999999, "end": 1934.86, "text": " Through 10 which means we are fetching the local crops. We're concatenating them"}, {"start": 1935.1, "end": 1938.3, "text": " So we're gonna form what like let me do it like calculator"}, {"start": 1938.3, "end": 1941.6599999999999, "text": " Here we have 8 crops all of these have what?"}, {"start": 1942.44, "end": 1943.74, "text": " 64"}, {"start": 1943.74, "end": 1948.06, "text": " 64 images so that's 512. So let's see whether that's true"}, {"start": 1948.06, "end": 1951.82, "text": " So we have 512 and we're going to expect the output 512"}, {"start": 1952.86, "end": 1956.78, "text": " 384 so let's see whether that's true. So we have out"}, {"start": 1957.8999999999999, "end": 1963.58, "text": " Output variable 512 384. That's it. So everything works as we expect"}, {"start": 1963.58, "end": 1971.26, "text": " So let's continue here again. We're just concatenating these so now we're gonna concatenate these with the outputs from the from the"}, {"start": 1971.58, "end": 1974.54, "text": " global crops and that means we're gonna have what like"}, {"start": 1975.6399999999999, "end": 1979.82, "text": " 512 for the global crops plus 100"}, {"start": 1980.76, "end": 1985.1799999999998, "text": " 128 so 512 was for the for the local crops 128 is for the"}, {"start": 1985.82, "end": 1991.26, "text": " Global crops so that's 640. So if we do that, we're gonna see that the output is"}, {"start": 1991.26, "end": 1993.26, "text": " Let me see here"}, {"start": 1993.66, "end": 1995.66, "text": " So the output is 640"}, {"start": 1996.36, "end": 2002.62, "text": " 384. Okay, that's it. Let's continue. We're gonna exit the loop. We're gonna pass all of these"}, {"start": 2003.5, "end": 2004.62, "text": " through the"}, {"start": 2004.62, "end": 2007.26, "text": " MLP and that means we're gonna end up with"}, {"start": 2008.94, "end": 2012.14, "text": " Instead of 384 we're gonna have those 65 000 something"}, {"start": 2013.0, "end": 2017.58, "text": " Activations, so that's it. That's the whole trick. Let me again exit this thing"}, {"start": 2017.58, "end": 2021.02, "text": " Okay, exit. Come on. Come on. Come on"}, {"start": 2023.1799999999998, "end": 2026.3799999999999, "text": " This sucks ass, okay, so this is the last important detail"}, {"start": 2026.46, "end": 2031.1, "text": " Let's just analyze the dyno loss and that's pretty much it. You know, you should be able to understand the training"}, {"start": 2032.06, "end": 2037.34, "text": " So here what happened what happens we have we take the student outputs and remember that's again"}, {"start": 2038.6999999999998, "end": 2043.1799999999998, "text": " 600 something times 65 000 something so 640 times"}, {"start": 2043.18, "end": 2047.44, "text": " 65 000 and we're just going to scale it with this temperature coefficient"}, {"start": 2047.44, "end": 2051.52, "text": " So remember that's remember from the diagram from the paper. That's what they are doing"}, {"start": 2052.2400000000002, "end": 2056.56, "text": " Now they're gonna chunk this into 10 pieces because remember"}, {"start": 2058.96, "end": 2063.12, "text": " We have 10 different crops so this is just a way to take these 640"}, {"start": 2063.84, "end": 2066.2400000000002, "text": " outputs and kind of cluster them into"}, {"start": 2067.2000000000003, "end": 2070.64, "text": " Semantically meaningful categories so that means global crops and local crops"}, {"start": 2070.64, "end": 2075.2, "text": " So we're gonna see that the student out let's find that variable. So that student out here"}, {"start": 2075.8399999999997, "end": 2082.48, "text": " Let me expand it. We have 640 and now we're gonna have a list of 10 elements and every element will have"}, {"start": 2082.7, "end": 2090.3199999999997, "text": " 64 images as it should be so as you can see here 10 elements in the list and every single one contains 64 images"}, {"start": 2090.4, "end": 2094.7999999999997, "text": " That's it. We do a similar thing for the teacher network. We have some temperature stuff"}, {"start": 2094.8799999999997, "end": 2099.12, "text": " We have the centering of the output and then we divide by the temperature"}, {"start": 2099.12, "end": 2101.7599999999998, "text": " across the feature dimension so that means across the"}, {"start": 2102.4, "end": 2108.16, "text": " That vector that has 65 000 activations where you're going to do the soft max across it and"}, {"start": 2109.0, "end": 2113.8399999999997, "text": " then we do the detaching because again, we're not training teacher network and"}, {"start": 2114.2799999999997, "end": 2120.2799999999997, "text": " We chunk it into two because remember we have two global crops being passed through the teacher network"}, {"start": 2120.2799999999997, "end": 2124.3199999999997, "text": " So teacher out let me just expand that variable. So teacher out has"}, {"start": 2124.32, "end": 2129.54, "text": " 128 it's gonna be split into two elements each will have 64 images"}, {"start": 2129.54, "end": 2135.78, "text": " So let's step over it and as you can see here two elements 64 images each. So that's it"}, {"start": 2135.78, "end": 2139.82, "text": " So now we have those different views and now we're gonna do the actual"}, {"start": 2140.5800000000004, "end": 2142.5800000000004, "text": " like loss, so"}, {"start": 2142.7400000000002, "end": 2149.2400000000002, "text": " What is happening here is we are taking a single so there's gonna be a single like a"}, {"start": 2149.24, "end": 2156.8799999999997, "text": " Global crop and it's gonna contain a batch of images for that global crop and then so this just makes sure we are not operating"}, {"start": 2156.8799999999997, "end": 2161.3599999999997, "text": " On the same view. So that's what we're gonna skip in the first iteration because both are zero"}, {"start": 2161.4799999999996, "end": 2166.4799999999996, "text": " So that means both are indexing the same global crop and now in the second iteration, this is gonna be one"}, {"start": 2166.4799999999996, "end": 2171.2, "text": " We're gonna skip that and what happens now is we're gonna take a batch of images"}, {"start": 2171.2, "end": 2177.58, "text": " So 64 images from the from the from the second global crop and we're gonna contrast it with Q"}, {"start": 2177.58, "end": 2183.16, "text": " Which is the the first global crop and that means and we're gonna do simple cross entropy here"}, {"start": 2183.16, "end": 2186.46, "text": " So that's Q and at times the log of the student view"}, {"start": 2186.92, "end": 2191.52, "text": " Softmax is here because we already we already done softmax for the teacher network"}, {"start": 2191.52, "end": 2197.04, "text": " So we have to do softmax and then we have to apply log and then we have just cross entropy and that's it"}, {"start": 2197.04, "end": 2199.58, "text": " So focusing on dimensions here on the shape again"}, {"start": 2199.58, "end": 2204.92, "text": " So if we take expand the Q variable, you can see it's 64 times 65,000"}, {"start": 2204.92, "end": 2207.48, "text": " The same thing is gonna be for this student view"}, {"start": 2207.48, "end": 2212.98, "text": " So we have 64 images here and here and we have their outputs and we're just gonna make sure that the"}, {"start": 2213.12, "end": 2217.8, "text": " Distributions are the same for all of these 64 images. And then what we're gonna do is"}, {"start": 2218.6800000000003, "end": 2220.44, "text": " Basically sum it up"}, {"start": 2220.44, "end": 2224.66, "text": " So we're gonna sum up across the that dimension that not the dimension number one"}, {"start": 2224.66, "end": 2231.4, "text": " And then this mean will do the averaging across the second dimension and that's gonna end up with a scalar"}, {"start": 2231.4, "end": 2236.94, "text": " So after we do this and this we end up with a scalar and that's it. That's pretty much it now"}, {"start": 2236.94, "end": 2243.42, "text": " We iterate through every single combination of views. So we have we trade through different global crops here"}, {"start": 2243.52, "end": 2247.6800000000003, "text": " We trade through both the global and local crops here. We take those views"}, {"start": 2247.6800000000003, "end": 2250.8, "text": " We do the cross entropy and that's how we train this whole thing"}, {"start": 2250.88, "end": 2257.54, "text": " Okay, and the final last detail is we are updating this this center vector which is used as you remember here"}, {"start": 2257.54, "end": 2262.94, "text": " So the self center is gonna be updated using the teacher outputs in this function here"}, {"start": 2262.94, "end": 2266.12, "text": " But you're gonna skip it. You can analyze it yourself. It's fairly simple"}, {"start": 2266.8, "end": 2271.22, "text": " it's basically following the formula from the paper and we're doing the"}, {"start": 2272.44, "end": 2275.7599999999998, "text": " Exponentially moving average update of the of the center vector. That's it"}, {"start": 2276.44, "end": 2281.62, "text": " Okay, let's exit this thing. Let's exit this thing and that's it"}, {"start": 2281.62, "end": 2287.66, "text": " We now have just the bureaucracy stuff. So we clear the gradients of the of the model"}, {"start": 2288.2, "end": 2290.2799999999997, "text": " We are going to now do"}, {"start": 2291.16, "end": 2297.8399999999997, "text": " Backward which is gonna calculate the gradients throughout the whole network of all of the parameters that are obviously a trainable"}, {"start": 2298.56, "end": 2300.56, "text": " We're gonna do some"}, {"start": 2301.2799999999997, "end": 2305.2, "text": " Clipping of gradients basically here. This line is just going to freeze"}, {"start": 2305.72, "end": 2309.72, "text": " And not train the last layer if you remember there was this parameter called"}, {"start": 2309.72, "end": 2311.72, "text": " What's the name of it freeze?"}, {"start": 2312.16, "end": 2317.04, "text": " So since this freeze last layer is only one and current epoch is zero"}, {"start": 2317.04, "end": 2320.3599999999997, "text": " That means we're actually going to freeze the last layer of the model"}, {"start": 2320.3599999999997, "end": 2325.72, "text": " But that's again a small optimization trick to make this thing more stable and trainable"}, {"start": 2326.4399999999996, "end": 2331.8799999999997, "text": " Finally we do the step. That means we're gonna actually update the weights and that's it"}, {"start": 2331.8799999999997, "end": 2335.48, "text": " Just some details around mixed precision training nothing that spectacular"}, {"start": 2335.48, "end": 2338.84, "text": " Finally we take the the the"}, {"start": 2339.56, "end": 2341.96, "text": " moment value depending on the on the"}, {"start": 2341.96, "end": 2348.64, "text": " Epoch we are in on the global step we're in and this is the part where we are updating the teacher networks"}, {"start": 2349.12, "end": 2350.72, "text": " using the"}, {"start": 2350.72, "end": 2353.44, "text": " Student weights, so we are basically doing"}, {"start": 2353.96, "end": 2360.2400000000002, "text": " Exponential moving average here. So as you can see the teacher weights are gonna be multiplied by M"}, {"start": 2360.24, "end": 2367.24, "text": " Which is the this momentum parameter and we're gonna add up 1 minus M times the weights from the student"}, {"start": 2367.8399999999997, "end": 2372.4399999999996, "text": " And you can see it on the formula on the screen and that's that's pretty much it. Okay, that was it"}, {"start": 2372.4399999999996, "end": 2376.68, "text": " That was pretty much the the core parts of the training everything else is just logging"}, {"start": 2377.0, "end": 2383.16, "text": " Dumping those log files and checkpoints. This was the the very meat of the dyno training and how the training looks like again"}, {"start": 2383.16, "end": 2386.7999999999997, "text": " Any feedback is appreciated the more feedback I received that the better"}, {"start": 2386.8, "end": 2393.1600000000003, "text": " I'll know for the next time how to make these videos. Okay. I want to end this video by showing you this visualize attention function"}, {"start": 2393.1600000000003, "end": 2401.4, "text": " Let's quickly run through how this thing works and I think it's fairly fairly much simpler compared to the training"}, {"start": 2401.4, "end": 2408.0, "text": " So let's kind of dig into it. I'm gonna put a breakpoint here. I'm gonna start debugging and"}, {"start": 2408.48, "end": 2410.48, "text": " Let's see what happens"}, {"start": 2411.48, "end": 2413.48, "text": " Okay, again, we're using"}, {"start": 2413.48, "end": 2415.92, "text": " We at small pet size 8"}, {"start": 2416.76, "end": 2422.64, "text": " No pre train weights. This is not that important. We have some image path. I already"}, {"start": 2423.56, "end": 2428.44, "text": " Inputted the path to this image. So we're gonna be using this image here this this bird image and"}, {"start": 2429.12, "end": 2435.48, "text": " So that's the image path image size is gonna be 480 not that important again, which is gonna dump in the current directory"}, {"start": 2436.32, "end": 2439.6, "text": " Threshold is also not that important for now. Let me see what I"}, {"start": 2439.6, "end": 2444.52, "text": " Am using currently so my current threshold is 0.1. I'm actually gonna delete this"}, {"start": 2445.04, "end": 2448.3199999999997, "text": " so I'm gonna just input the image path here as you can see and"}, {"start": 2448.96, "end": 2450.96, "text": " Now let's apply this and"}, {"start": 2451.72, "end": 2453.72, "text": " Restart the code again"}, {"start": 2454.24, "end": 2456.24, "text": " jumping to here"}, {"start": 2456.24, "end": 2458.24, "text": " Okay, I'm gonna jump here"}, {"start": 2458.24, "end": 2460.24, "text": " Let's see here. So"}, {"start": 2460.24, "end": 2465.44, "text": " Device if I have if you have GPU, it's gonna be using GPU. So it's CUDA here for me. I have a GPU"}, {"start": 2465.44, "end": 2471.2400000000002, "text": " So we're gonna form we're gonna create a model. So this is the same line as in dyno training code"}, {"start": 2471.2400000000002, "end": 2474.76, "text": " So we're gonna basically instantiate the vision transformer small"}, {"start": 2475.4, "end": 2482.56, "text": " Because of this string with pet size 8 and 0 classes means it won't have the output"}, {"start": 2483.08, "end": 2485.92, "text": " head is just going to have the the final layer is"}, {"start": 2486.4, "end": 2491.7200000000003, "text": " Gonna be just bunch of tokens the CLS tokens and all of the other tokens, but no no actual linear"}, {"start": 2491.72, "end": 2496.52, "text": " Classifier had okay. Let's do this. So these lines were just gonna"}, {"start": 2497.24, "end": 2499.24, "text": " freeze the the model"}, {"start": 2499.48, "end": 2503.2, "text": " Parameters that means we are not we're gonna set them to not trainable"}, {"start": 2503.2, "end": 2508.04, "text": " So that means this model is just going to be used as is no training involved"}, {"start": 2508.68, "end": 2514.2, "text": " Eval evaluation mode that basically sets up depending what you have dropout or batch norm"}, {"start": 2514.2, "end": 2517.7999999999997, "text": " This thing can change the the those layers"}, {"start": 2517.8, "end": 2523.76, "text": " And that's why it's important when you're using a model in an inference mode to call this eval function"}, {"start": 2524.8, "end": 2528.44, "text": " We are gonna push the device to the GPU to make things a bit faster"}, {"start": 2528.44, "end": 2531.8, "text": " And finally because I don't have any pre-trained weights, which is gonna"}, {"start": 2532.44, "end": 2536.48, "text": " Fetch from a certain URL as you can see here. We form an URL we"}, {"start": 2537.2400000000002, "end": 2543.1600000000003, "text": " Fetch those from the torch hub. We're gonna just concatenate the URL to this repository"}, {"start": 2543.16, "end": 2547.3599999999997, "text": " And we're gonna fetch the state dict and we're gonna load the state dictionary"}, {"start": 2547.3599999999997, "end": 2549.96, "text": " So the download actually already happened on my local machine"}, {"start": 2549.96, "end": 2554.0, "text": " So it's just fetching it from the local storage instead of having to go to the cloud"}, {"start": 2554.0, "end": 2557.2799999999997, "text": " Okay, my image the image name was like incorrect"}, {"start": 2557.2799999999997, "end": 2561.48, "text": " So I had to tweak that in the settings and now I rerun the code and here we are again"}, {"start": 2561.48, "end": 2565.56, "text": " So because we have an image path, we're gonna open it load the image"}, {"start": 2566.56, "end": 2570.16, "text": " We're gonna load the image. We're gonna do something that's gonna load the image"}, {"start": 2570.16, "end": 2577.12, "text": " We're gonna load the image. We're gonna do some transformations. So we're gonna resize the image into 480 by 480"}, {"start": 2577.2799999999997, "end": 2580.8799999999997, "text": " We're gonna convert it into tensor. And again, here is here these"}, {"start": 2581.56, "end": 2588.2799999999997, "text": " Numbers we already saw the much the magic numbers. They just basically represent the mean and standard deviation of the image net data set"}, {"start": 2589.8799999999997, "end": 2592.8399999999997, "text": " So, okay, so after that we're gonna transform the image"}, {"start": 2592.8399999999997, "end": 2597.12, "text": " So now we have a tensor that's normalized and resized as you can see here"}, {"start": 2597.12, "end": 2604.16, "text": " Finally, we make sure that it's divisible by the pet size. So the pet size is currently what eight so we're gonna"}, {"start": 2604.7999999999997, "end": 2608.1, "text": " Make sure that the width and height are are actually"}, {"start": 2609.2799999999997, "end": 2616.64, "text": " Divisible by eight and now we're gonna crop the image here if necessary and on squeeze zero means we're just gonna add"}, {"start": 2617.04, "end": 2622.2799999999997, "text": " Another dimension in the front. So that's just some pie torch like stuff"}, {"start": 2622.28, "end": 2627.34, "text": " And if we take a look at the image dimension here, so let's take a look at the shape"}, {"start": 2627.4, "end": 2634.6800000000003, "text": " So the shape is just three four hundred eighty four hundred eighty and after we execute this line is gonna be one three four hundred"}, {"start": 2634.6800000000003, "end": 2640.76, "text": " Eighty four hundred eighty. So that's the unsqueezed part then we're gonna divide the resolution by the pet size"}, {"start": 2640.8, "end": 2648.0, "text": " So this will tell you how many tokens you have across the width dimension and we have sixty and here again"}, {"start": 2648.0, "end": 2652.14, "text": " We have sixty because remember it's four hundred eighty times four hundred eighty image"}, {"start": 2653.28, "end": 2654.44, "text": " finally"}, {"start": 2654.44, "end": 2661.52, "text": " What this does it's gonna affect from the vision transformer. It's gonna get the the self-attention of the image"}, {"start": 2661.52, "end": 2663.52, "text": " So we're gonna take the image we're gonna"}, {"start": 2664.08, "end": 2669.04, "text": " Like send it to the GPU because the model is also in GPU. So you don't want to have a discrepancy there"}, {"start": 2669.2, "end": 2674.48, "text": " We're gonna feed the image into the model and it's going to take from the last layer"}, {"start": 2674.48, "end": 2681.48, "text": " It's gonna return not the tokens is gonna return the attention coefficients. So that means the following so remember we have"}, {"start": 2682.0, "end": 2684.0, "text": " 60 times 60"}, {"start": 2684.16, "end": 2688.96, "text": " Tokens as we just saw up there. So we have 60 here and 60 here"}, {"start": 2688.96, "end": 2695.56, "text": " So that means we have three hundred three thousand six hundred tokens in in total plus one for the CLS token"}, {"start": 2695.56, "end": 2702.2, "text": " So that's three hundred three thousand six hundred one. So the attention dimension we should expect should be"}, {"start": 2702.2, "end": 2707.96, "text": " basically three thousand six hundred one times three hundred six thousand three"}, {"start": 2709.12, "end": 2712.8999999999996, "text": " Three thousand six hundred and one and we also have six"}, {"start": 2713.7599999999998, "end": 2718.16, "text": " Attention head so we're gonna have something like six times this number here"}, {"start": 2718.2, "end": 2721.7599999999998, "text": " So that's the the the numbers I'm the shape I'm expecting"}, {"start": 2722.68, "end": 2724.9199999999996, "text": " so if I step over and"}, {"start": 2724.92, "end": 2732.6800000000003, "text": " After we now let's find the attentions variable. So the shape is as you can see here six"}, {"start": 2733.2400000000002, "end": 2740.66, "text": " Three thousand six four six hundred blah blah the number we expected basically plus the this one as the first dimension again"}, {"start": 2740.66, "end": 2743.84, "text": " Those are just some idiosyncrasies of pytorch quick recap"}, {"start": 2743.84, "end": 2747.46, "text": " How those were formed is basically we have so we have as I said"}, {"start": 2747.46, "end": 2755.2200000000003, "text": " Three thousand six hundred one tokens you take a single token and you basically form its query and then you do a dot product"}, {"start": 2755.2200000000003, "end": 2758.4, "text": " a scale dot product with all of the other keys and that's why you have"}, {"start": 2759.7, "end": 2763.7, "text": " 3601 attention coefficients for that single token, but since we have"}, {"start": 2764.56, "end": 2768.86, "text": " 3601 tokens we're gonna have this this this shape here. So hopefully that's clear"}, {"start": 2768.86, "end": 2772.58, "text": " So as you can see here, we extract the number of heads the number of heads is six"}, {"start": 2772.58, "end": 2779.1, "text": " And now what we do here is we take we just extract this is just because of one"}, {"start": 2779.1, "end": 2785.62, "text": " We take this zero here and then we take all of the heads, but we take only the zeros token"}, {"start": 2787.2999999999997, "end": 2791.42, "text": " Which means CLS token and we take one through the end"}, {"start": 2791.46, "end": 2798.72, "text": " Which means we find the attention coefficients that the CLS has across all of the other let's call them spatial tokens"}, {"start": 2798.72, "end": 2801.7, "text": " So that's the whole idea there. So that's why we have these numbers here"}, {"start": 2801.7, "end": 2806.9399999999996, "text": " So, let's see what what is the shape we're expecting. We're expecting something like let me write down here"}, {"start": 2806.9399999999996, "end": 2809.18, "text": " Maybe it's easier to see so expecting like six"}, {"start": 2809.8599999999997, "end": 2815.7, "text": " And then we expect because we're taking just the CLS token. So expect six three hundred"}, {"start": 2816.2999999999997, "end": 2820.16, "text": " Three thousand six hundred. So that's the shape where we're actually expecting after after this"}, {"start": 2820.72, "end": 2823.4399999999996, "text": " This line of code. So let's see where that's true"}, {"start": 2823.7, "end": 2828.16, "text": " So attentions as you can see here, that's that's the actual number six"}, {"start": 2828.16, "end": 2835.2, "text": " Three thousand six hundred. Okay. I'm just gonna ignore this threshold part and we're just going to reshape"}, {"start": 2835.7599999999998, "end": 2837.7599999999998, "text": " the those attentions into"}, {"start": 2838.48, "end": 2843.68, "text": " Into different shape. So instead of having it flattened out, we're gonna return it back to 60 60"}, {"start": 2844.08, "end": 2846.44, "text": " And finally, this is just nearest neighbor"}, {"start": 2847.24, "end": 2850.8399999999997, "text": " Interpolation which means you're gonna take that token"}, {"start": 2851.48, "end": 2856.3999999999996, "text": " Whatever the coefficient is we're just going to replicate the coefficient into the 8 times 8"}, {"start": 2856.4, "end": 2861.5, "text": " pixels because that's was the original image shape so that we can visualize the actual attention"}, {"start": 2861.6, "end": 2863.8, "text": " So that's why we need the interpolation part"}, {"start": 2864.48, "end": 2867.1, "text": " finally just some making some directories"}, {"start": 2867.76, "end": 2874.2000000000003, "text": " Saving the original image and finally here in this in this section here. We're going to as you can see here"}, {"start": 2874.32, "end": 2876.32, "text": " We're going to take"}, {"start": 2877.6, "end": 2881.46, "text": " The attentions for every single head and save them in an image"}, {"start": 2881.46, "end": 2886.4, "text": " So let me just kind of do f9 there and we're gonna end up having"}, {"start": 2887.94, "end": 2893.82, "text": " Six images which I'm going to show you in a second and finally I'm gonna ignore this threshold in part again"}, {"start": 2893.82, "end": 2896.66, "text": " And that's that's pretty much it and here are the attention"}, {"start": 2897.26, "end": 2900.3, "text": " Maps we got so here you can see the bird image"}, {"start": 2900.66, "end": 2907.9, "text": " Every single head has different attention pattern and that's it and these are the images you you saw in the actual paper"}, {"start": 2907.9, "end": 2912.6, "text": " Now you know how they are calculated and visualize awesome. This was a long video"}, {"start": 2912.78, "end": 2917.7400000000002, "text": " hopefully liked it if you did share it out with a friend subscribe hit the bell icon and"}, {"start": 2917.74, "end": 2937.4199999999996, "text": " Join the district community until next time bye bye"}]