Facebook DETR | ML Coding Series | End to end object detection with transformers
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

The second video in the ML coding series! The longest video I ever made! 

In this video I do a code walkthrough of Facebook's DETR model from the "End-to-End Object Detection with Transformers" paper.

Let me know what you'd like me to cover next!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ DETR GitHub: https://github.com/facebookresearch/detr
✅ DETR Paper: https://arxiv.org/abs/2005.12872
✅ Mini COCO: https://github.com/giddyyupp/coco-minitrain
I edited the download/sampling scripts and ended up downloading a valid dataset that has only 100 images.
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00:00 DETR model recap
00:07:30 DETR demo notebook
00:18:55 Visualizing attention notebook
00:29:50 Visualizing encoder attention
00:36:45 Going through the training script
00:38:45 Backbone construction
00:42:05 DETR construction
00:54:00 Data loading and nested tensors
01:02:30 Forward pass through ResNet backbone
01:09:00 Forward pass through the transformer
01:22:00 Hungarian matching algorithm
01:32:50 Loss calculation
01:40:50 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#detr #transformers #coding
 What's cracking guys? This is the second video in the machine learning coding series and I'll be covering Facebook Deter, so that's a short for Detection Transformer and some of you might recall that I've already covered the paper in great depth I think a year ago, so if you're not familiar with what Deter is maybe consider checking out that video first, but I think this one will be like self-sufficient as well. So as I said it's a model for object detection and just for those of you who are new to the channel, object detection is this task where you have some salient objects in the image, as you can see we have some cats and remote controllers here and you can see that we have a bounding box around this cat here and we also have a label associated with the bounding box similarly for the remote controller here, for the second one here and for the other cat this blue box and finally we can see a purple box around this whole image and basically the class being couch which is the correct class in this particular case. So what I've done is I went ahead and basically cloned the official repo and as I said I'm gonna walk you through the code base in a lot of details. Before we go there I want to quickly walk you through the pipeline on the high level so that you have a mental model going into this video. The model itself is fairly simple and we'll be focusing on three components in this particular video. So first is understanding how the actual architecture works. So how do we basically, let me take a pen here, so how do we do this transformation here? How do we go from the image all the way to this set of box predictions? So understanding the architecture of the actual model. The second part is given the predictions, given the the bounding boxes predictions and the associated classes, how do we match those with the set of ground truth bounding boxes here? Okay so basically what this model outputs as we'll soon see is, and this is a hyper parameter but the model they've used 100 as the as the default, so that means we are always outputting 100 bounding boxes and so we'll have two vectors as the output from our model. One is going to be a 104 so that's going to be our bounding boxes. Obviously the four points are enough to describe the actual bounding box. The format is I think first the center point coordinates XY and then the height and the width but that's less important. The second vector that the model will output is just like 100 basically containing the classes of those associated bounding boxes here. Now as I said the hard part will be how do we match these outputs with like correct ground truth bounding boxes? So imagine we have this red bounding box represented as this red square and consider we have, I'll just pick another color like green because I don't have yellow at disposal. So imagine this yellow is this green and so the idea is how do we decide how do we know that we need to connect this one with this particular ground truth bounding box and how do we know that we need to connect this red one with this red one here? And also we need to take all of the other predictions and map them into this no object into this special class that no object class and okay so that's the step number two. And the way they do that is using something called Hungarian matching algorithm. We'll see how it works in a very like a very in a lot of details. Finally once we have the matching we now need to calculate some sort of a loss and then do the back prop. So they'll have three types of losses. The first one is gonna be the label loss so that means for this for the match boxes we want to make sure that the ground truth that the probability of the ground truth label is as high as possible. That's first loss. The second loss is gonna be L1 loss. So basically how far off are our predictions from from the associated ground truth boxes. So imagine we have a prediction here and imagine we have a different prediction basically maybe sorry the ground truth prediction being here. We calculate the L1 loss and I'll get into the details of how that's calculated but basically well you just treat the bounding boxes as a four-dimensional vector and you just find the L1 between the difference of those two vectors. That's how we calculate L1 loss. And the final one is the generalized intersection over union which is basically a fancy version of your theme of the of the classic IOU metric. IOU is just a way to so so what it does is it finds the area of this intersection here and it basically divides that with the union so hence the intersection of a union and you can imagine that as this tends to one that means that these two boxes are completely overlapping and you can see as the less the overlap the smaller the intersection area and the and when you divide that by the actual union that goes towards zero as the boxes don't so so yeah it will be ultimately equals zero once we don't have any overlap. So that's the third type of a loss again don't worry if you didn't understand everything we'll go into a lot of details in this particular video. Okay let me just show you a bit more detailed view of the architecture here so the idea will be to encode so we'll pass the first we pass the image to a CNN we get some set of image features we are going to then basically tokenize those image features so we'll just treat them as tokens for the NLP transformer so that part is here we'll have some so so basically I said we're gonna pass them here and then encode them using the encoder stack of the transformer we'll add some spatial positional encodings and on the other hand side we have this very interesting thing going on so we have these let me just change the color so we basically have these object fairies which are going to be learnable like hundred learnable embedding vectors and it's hundred because as I said that's just a hyper parameter and because we have hundred we'll also have output hundred like bounding boxes and you can see here what happens is that in this middle lab layer of the transformer layer we have cross attention going on so we have that these object queries are basically each of them are attending to this encoded version of the image features and each of them are looking for something in the in the image so that's and that's how on the high level obviously very hand baby how this pipeline works once we get the output set of hundred tokens here we just map them individually using MLPs or just a simple linear layer and we map so we take this token we map it into basically a class so we'll have associated class and we also map it into a four dimensional vector here and that's gonna be that's going to represent the bounding box that this particular token encodes and that's it on a very high level with this prerequisite knowledge we can now jump into the actual true piter notebook so let's see what's going on here let's start from the beginning of the notebook I'm just going to run these imports here I'm going to import the necessary libraries we're gonna be working with pytorch because this is Facebook after all or meta as they are now known to as here is a small minimal implementation of debtor it's not as complicated and it's not the same as the one they actually have in the official code base but it's good enough and gives an understanding of what's going on so I'm gonna run the cell and then explain what's going on again quickly we have a backbone resident 50 we have this a comb layer which is going to take the 2048 channels of the output embedding volume from the from the CNN and just reduce the number of dimensions to 256 and that's going to be the ultimate dimension of the tokens that we pass into the transformer so let me quickly just kind of explain what that exactly means so as I said we have an image here so this is an image we have a CNN block here I'm just gonna denote it as a box here we pass the image and out comes some some volume basically obviously the spatial so here the spatial extent was we had three channels here so this was this was three here and after encoding it through a CNN we're gonna end up with something that has like a smaller spatial extent so let me try and denote that like the following so basically we have a smaller spatial extent but then we have much more channels okay so something like this I'm really bad at drawing today so something like this basically this number of channels here is gonna be 248 for resonance 50 and this is just like your your down sample spatial extent so this is like small H small W whereas this was like big H here and big W so the the convolutional layer I just mentioned has the kernel of size one times one which means each is just going to reduce the number of channels of this of this volume here from 2048 into I think 256 and then the idea is to just like treat these as a set of tokens so we can in a raster order we can just do the following we can just do this and we take this vector this is gonna be the first embedding vector which we pass into the transformer then we take the second one here so this is gonna be the second vector this one here is the second one and so yeah we basically do that in the raster order let me can zoom in here a little bit and let's show you what I mean exactly so this is vector number one vector number two vector number three vector number four five six etc etc and that's basically how this this part of the pipeline looks like okay let me go back to the code so that's this this what this conversion layer does then we have a simple transformer again encoder and decoder stacks and finally we have the two linear layers linear class and linear b-box I showed you what these do these take the output embeddings from the decoder and just map them into either the distribution over classes for the cocoa data set or this second one just gives us the bounding boxes for each of the output tokens okay output embedding vectors here the query at the query vectors the learnable query vectors we have hundreds of them and that's it and finally we have a row embed and call embed these are just going to be the the positional encodings okay and once we do so here is how the forward path looks like we basically pass the inputs so that's a set of images we pass that through the comp one then we apply some batch normalization ReLU max pooling whatnot and then we apply the four layers of the of the resonance 50 and finally we apply this small this this convolutional layer where we we basically reduce number of dimensions as I previously mentioned and what happens here is we construct the positional encodings then we're going to take those positional encodings just sum them up with these tokens here so that's this part here don't worry about the actual details we'll get to those a bit later so what is important is we have some tokens from the image we add the positional encodings we pass that into the encoder stack and then through the decoder we pass the query the query embedding vectors that gives us as the output a set of embedding vectors here which we then map using the linear class and linear b-box linear layers we additionally apply sigmoid because we want to have normalized bounding box coordinates so from 0 to 1 okay that's it now let's see what happens let's just run this this cell is going to load a pre-trained data model so I'm going to just run this and here we're just going to run the this cell that has that defines the classes from the cocoa data set and some associated colors which are going to help us visualize stuff so again the number of classes in cocoa is let's see what we have here so if I add a code cell let's bring the number of classes so shape well shape will not work because this is what this is like a list so we're gonna do length is that so length and this should be I think 91 or 91 yeah 91 and then we'll add the additional no object class we'll see that a bit later okay here's some helper functions this one helps us just do some type of processing to the input images we do some resize operator then we convert the images into pie torch tensors and then we do image net normalization so these are statistics calculated from the actual image net data set this is going to be like the mean and this is going to be the standard deviation statistic of over the calculated over the image net data set these helper functions just map as you can see from the format CX CY WH which is as you specify the center coordinate of the bounding box and width and height and instead of that it's going to convert it into a format where we specify the top left point and the bottom right point and those are obviously equally informative representations that take equal amount of space it's just that this representation is a bit more suitable to some of the API's they're actually using and we'll see that later not that important rescale boxes so as I said we do predict the bounding boxes which are normalized which means from 0 to 1 this is just going to return a map those predictions into the original input image space like a range okay did we run the cell and just run the cell and finally we have the detection function here what we do is we pass an image we apply the transforms we just saw we unsqueeze so that we add like the batch dimension so this adds a dummy batch dimension one because that's what by torch models expect just some error checking we pass the images through the model we get the outputs and finally what we do here is we take the prediction logits we apply the softmax because remember this is just going to apply linear layer so these are still unnormalized logits so we apply softmax after we've normalized the logits using softmax we just extract all of the classes except for the no object class here that's what we have up to minus one and then what we do is we take those probabilities we find the max across all of the hundred outputs and we find those predictions where the peak of the distribution across classes is at least 0.7 in that way we kind of take out only those very confident bounding box predictions okay that's the idea there and then we use that vector the key vector to basically extract only those bounding boxes and then we just rescale those and return back the bounding boxes we additionally return the associated probabilities here so if I run this and if we run this here we're going to get as the output the scores and the bounding boxes so let's kind of quickly examine what's going on here let's see what the shapes are here so if I do print score shape that's going to give us 591 so it's five because only five of the hundred bounding boxes were like confident enough and if I was to print the boxes this should be what this should be five four that's the expectation so let's run it's five four indeed so five boxes that are good enough so I'm gonna quickly just modify this and return back like this probability vector so this should contain if I'm not wrong so let me just actually return back this original output so like this and I'm just gonna call that logits so I'm gonna run that cell I'm gonna add a logits whoops variable here so just add logits logits run that and see what the shape of that thing is so that should be a hundred ninety one I assume right so if I were to run this we have 192 so it's 192 because as I said here we extract the no object box so if we were to take this probus whatever like it's a weird variable name if we were to take that variable and return it instead that's gonna be that's going to give us 91 instead of 92 so let me show you here and as you can see here that's that's pretty much it okay now we can finally visualize what's the output of this of this function detect function and what we do is we iterate over this over the probabilities over the bounding boxes and we just plot rectangles using the coordinates from those boxes and we plot the basically the most probable class from the distribution there and if I were to run this I mean we already had the image so let me let me show you this is what we get so this is the image with which we started this video I showed you this one in the beginning okay so far so good let's now jump to the second notebook this was just like a quick like a high level overview of how minimal implementation of a debtor could look like and it turns out it works already fairly fairly nicely let's now go to this one where you're gonna see some attention visualization so I'm gonna again run the necessary imports I'm gonna again run the cocoa classes here we have the same type of helper functions that transform functions we have the these functions that change the box format they do the rescaling so I'm just gonna run those again we have the same plot results function as from the previous two Python notebook the details are not important it's just like your math plotlib stuff I'm gonna focus on the decor and the gist and that's the actual data model and how those three components that I showed you in the beginning of the video so that the architecture the Hungarian matching and the loss calculation look like that's gonna be the focus of the video again let's load some of the pre-trained models here we set the model into eval mode so eval just for those of you who are not familiar with pytorch sets so that's very important because some of the layers such as batch norm behave differently whether they are in the depending on whether they are in the train or eval setting okay and we did the same thing we just load load an image here and then we have a similar type of a forward a prop as in the previous detect function from the previous notebook so we just pass the images we transform them and squeeze them pass through the through the model we get the outputs we do the same type of transform here so we get the probabilities and this time the threshold is a bit higher we use 0.9 here although I think I am the one who modified this not sure let's let's assume this is 0.9 and then we do the rescaling so let's see what we get so yeah for the most part this part was the same as in the previous notebook so we get the output predictions now that we have all of that calculated let's go and add some hooks to our model so what the hook does is as the image is passing through the actual data model we're going to log certain activations and the ones we are going to be interested in is going to be the activations from the actual backbone model so that's going to be the calm features let me let me show you what I mean by that so it's going to be we're going to log these features here better yet so we're gonna log actually these features here and we're going to log some of the basically attention coefficients I think from the let's see I think from the multi hat attention like the cross attention component here so let's just make sure that's the case so we get we take from from the encoder from the last layer of the encoder we're gonna grab the self attention module so that means we're going to grab from the last layer so as you can see we have and here I think in practice they use six so from the sixth one we're gonna grab the activations from this module here okay multi had self attention in particular we're gonna take the matrix like when you multiply query vectors with with key vectors you get a matrix of scores which after you which after you apply the softmax you get basically the attention scores so that's the one we're gonna visualize and the second one if I'm not wrong I think it's this one here but let's double check if I go back to the code here we can see we again we take the decoder the last layer of decoder multi had attention and I'm fairly sure that's the second the cross attention one so let me quickly go to the transformer file here let me just make sure that's the case so transformer decoder layer we are going to have let's see so self attention and multi head attention the multi head yeah the multi head attention is the second is the cross attention layer okay so we are basically taking the activations from this particular multi head attention head okay not yet but the module sorry okay let's go back that's enough rambling let's go back to the notebook after we attach these hooks and this is just your pie tour syntax I will not get into details there and we run the inference all of those are gonna be locked in the associated lists then we just remove the hooks and then we extract the zeroth element from the list because here as you can see we just append it so we by doing this we just extract the collected data so I'm gonna run this cell and let's see what happens okay so we end up with those shapes I'm going to quickly analyze what's going on so con features let's print what we get here so what I expect is I assume something like the following shape like maybe one to 48 and then some spatial extent which is gonna be I think something like age is gonna be 19 and double is gonna be 29 that's less important but let's kind of print this and see what we get okay so we get that this uh-huh oh yeah or I remember so there is this particular detail of the implementation so we have to actually index we have to index by this key and only then do we actually get the okay and then we have to do additional tensors because additional implementation detail because this is gonna give us nested tensor and we just want the tensor info so finally okay this is what we get as I said I kind of missed the small H and small W but that was not the point that the same overall format is as we expected let's copy paste this and let me do the following here I'm not sure whether this is gonna be nested vectors let me just try and do shapes like this I'm gonna comment this out comment this out see whether this works and it does so 850 850 that's the I said the scores but what you get by multiplying the queries and key vectors similarly here if I were to do this and we're gonna visualize the detection attention weights let's see the shape here so we're gonna have I guess the same exact shape oh no actually it's hundred because here we are in the decoder stack so because of that we have a hundred eight hundred fifty because we have hundred very the learnable vectors and we have eight hundred fifty keys from the encoder stack that's why we get this this different shape it all makes sense nice we've seen the shapes now let's visualize the attention so we're gonna in particular focus on visualizing the decoder attention weights and as you can see here we they just grab the height and width from the com features that's gonna be the 2534 here and the each way through the bounding boxes we previously predicted so the five bounding boxes we previously predicted and visualized above here so these are the two remote controllers two cats and a couch and they are then going to as you can see here grab a particular index so remember this is gonna be of the shape hundred because we have hundred query embeddings and 850 because we have 850 of the basically keys coming from the image features of the 3d encoder stack and so this is going to grab 850 coefficients that basically correspond to the scores that this particular query vector has when it attends to the keys okay just to quickly clarify what I mean by that is it means the following basically so again remember we are we have encoder output here so we have basically 850 outputs here and then we have the decoder stack here and as I said we grab the cross attention layer and that cross attention layer has hundred hundred query vectors we have in total hundred query vectors there and we take a particular one that's associated with the bounding box we care about so we take for example we take this one here and we see how that query vector basically we do the dot product between that query vector and all of these key vectors and those are the 850 numbers that we're now going to visualize okay that's that's the basic idea so let's see what we get so after we do that and run this cell we have this so for this particular remote controller you can see that this is the pattern that that we attend to in the these are the image features that we attend to and you can see there is obviously a correlation between these two similarly for this image here similarly for the this is I guess the couch and so we are basically attending the ends of the image that's how it figures out that this is the couch that we are seeing and then you can see kind of a faint figure of the cat here and finally similarly here we can see a faint figure of the cat here okay so hopefully this gave you some understanding of what's what's going on and that I hope that this this image here helped you understand again what's going on so let me just recap one part so the shape here is 2534 if you multiply those two numbers you get 850 that's why we have 850 here so 2534 so remember that we started here from the image features that were precisely 2534 and then we took those in a rest order and we kind of flattened those out and that's how we got these 850 tokens so that means by attending to these 850 tokens you're actually attending to the original image but just a down sampled version that's why we are able to get these interesting attention maps here so that's how you connect these 850 with the images we get here so with these images here okay so yeah that's the best I can do for now but bear with me we'll get into we'll understand all of the details a bit later so what I do here is just they just plot the they just print these shapes and we saw all of this already so I'm just gonna skip through all of that because we already saw it what they additionally do is they take the now the encoder attention weights instead of decoder ones and they just reshape those such that we end up with 2534 2534 instead of with 850 850 so this is just going to be a bit easier to understand what's going on when we index into this array into this tensor okay let's now visualize the encoder attention patterns so they grab four points from the original image and then using those points they're going to visualize the encoder attention so let's see what exactly happens here so you can see original image they picked four points denoted here by these red circles and then we visualize the where these points attend to in the original input image because we have the self-attention in the encoder layer which are the image tokens and that's why we get these image patterns and you can see the interesting fact about data is that the encoder already learns to do some sort of instance segmentation which is super useful if you later want to create a bounding boxes around those objects and this was not explicitly trained for so this is just something that emerges from the from the way that data was trained which is super nice so let's kind of dig through this code and understand what's going on it's a bit messy here so I'm gonna just keep it on a high level basically they create a bunch of subplots that's not that interesting and then the important part is let's see so the important part is this so we have image show of the attention pattern so we take the index so indices are again the points we picked right and so we grab those points those points are initially in the image space and so because of that we want to down sample because that's what the calm layer does the backbone is going to don't sample by 32 that's why we divide by fact and this fact is going to be equal 32 that's how we get the indices for the down sampled image features and then we just basically grab the points and index into this attention tensor okay and by doing that we basically figure out the following I'm gonna again use my one note because this is very hard to I realize this can be hard to understand but just me speaking about it so what what we've done is the following so again remember we have this volume here and so what we are going to do with it is the following so this is our volume these are our tokens basically so this is the volume that came out from the CNN so we take a particular point so let me kind of create a grid structure here so make it easier to visualize we take a particular point so this one here so that's as you can recall that's the tens that's the token that is fed into the encoder transformer so somewhere in the transformer we're gonna map this into a query vector and then we're gonna take that very vector and basically do a dot product with the key vectors of all of these other tokens right all of these other tokens and by doing that we basically understand how this image token is attending all of the other image tokens in this in this volume here so that's the idea by the way obviously this is flattened in the actual transformer I'm just visualizing this as in this in this spatial format just because it's easier to map what's happening here and between and why do we have the images that we get here so hopefully that was somewhat understandable and now this final cell just makes this basically interactive so you can pick your own image and do the same thing in an interactive fashion so if I were to run this they created this nice widget where you basically have let me show you what happens so you can see here you can kind of move this red point here and you can see how the attention patterns change so if I were to move this a bit more directly on to the cat well we get some weird patterns so apparently when we are closer to the boundary of the cat that's when we we get the nicest patterns of attention here so you can also play and just use some other image from the internet and this is gonna work so that's the idea with this with this Jupyter hopefully you got some understanding of how of the things that this data model is learning so you can see that we have interesting patterns going on in the encoder so instance segmentation types of of masks here and we also saw nice patterns in the actual decoder where we saw where the query tokens from the decoder stack are attending to in the original down sampled image features again quickly this is decoder so the query embedding vectors attending the embedding vectors from the encoder stack so that's this part here so again query vector attending the embedding vectors from the top of the encoding stack and by visualizing that you basically visualize the where this query token is attending to the query vector and the second image is about grabbing the image token and visualizing where that image token is attending to over the over the neighborhood so that's that's this particular image here so should be fairly simple and easy to understand cool now let's start digging through the actual code base and for that I'll first have to check out the main branch so I'm gonna have to check out the main branch because the colabs are so the the Jupyter notebooks are checked into this colab branch so if I do get check out main now I'll be able to find the code that I need so let's go back here and let's start debugging the main file so I'm gonna skip a lot of unnecessary details like because the code was developed to be trained on a you know like a multi node multi GPU setup we don't care about that we just care about understanding how data works also there is a lot of details about pen optic segmentation because data can easily be adjusted so that it works for the pen optic segmentation desk we're also going to ignore everything there and finally the actual cocoa implementation details are not a part of this code base but they are using an existing API so we'll kind of skim through that as well so the main focus is let's understand the three components of data the architecture the Hungarian matching algorithm and finally how do we calculate those three types of losses we saw in the beginning of the video so we offered our do let's start here so we are going to just ignore these parts this just prints the basically the shot of the of the current commit but that's again not vital I'm gonna skip all of this we just grab the device so I have a GPU so it's gonna be GPU is gonna be used so for reproducibility reasons they they set the seeds for various libraries pytorch non pythons random etc so that not that important now this is where the actual important part starts let's see how the model is built so we first have basically we picked a number of classes because we'll have 91 here because we are dealing with cocoa so as you can see here we have 91 and then as I said we're gonna ignore the pen optic part so now let's build the backbone so the belt bone is the resonance 50 but let's see the a bit more details there so first of all we're gonna build the position in coatings and the interesting part of this positioning coatings is going to be in the actual for a function of this of this module so we can just kind of skip through that because the learning rate is bigger than zero we are gonna make backbone trainable nothing fancy there so now we form the backbone and this join our object built which will see what it is in a second so it's basically just gonna join the backbone which is gonna be the resonant 50 and the positioning beddings so let's step into the constructor of the backbone here you can see what happens is that name is going to be the resident 50 so we get the we fetch the class the resident 50 class from the torch vision models and you can see here that sedulation is set to false and because of that we'll just fetch your regular resident 50 not the one with dilated convolution and if you're wondering this is what dilated convolution is all about basically here is a nice visualization of how it works so it has a similar effect when it comes to down sampling the spatial extent so it's instead of using strides to down sample the output map you just do this type of you create this type of a kernel where you only where you basically well it's basically dilated and you're attending only certain pixels of the image here and that's the way you you you manage to down sample the output map okay but that's not that important just like a fun fact okay finally they're using these frozen batch norms again just in other detail not that crucial it's just a common thing in these detection pipelines okay so number of channel is gonna be 2048 and finally we construct the backbone base some more details here let's kind of scheme through this they're gonna freeze some of the parameters basically the only parameters that are going to be trainable are these layer 2 layer 3 and layer 4 blocks let me kind of scheme over that click f5 to jump over all of that and finally we are gonna care we're just going to extract the features from this layer 4 and we are going to take those features and basically put them as a value with the associated key that's gonna be this zero key okay so that's the reason we during the the past through the Jupiter we had to have that weird zero key and then the tensors will see that in a second but yeah this is the detail why that had to happen okay so finally we just take the resonance 50 and we call this intermediate layer gatherer which is just gonna make sure that we only fetch layer 4 representations from this resonance 50 backbone that's that's that's it nothing fancy there joiner nothing fancy there as well we'll see the interesting part actually lies not in the neat function of the joiner but in the for function again most of the things are gonna happen in the actual for a prop but this is just giving us some idea of how the model is implemented so let's continue here next up we build the transformer your standard transformer not nothing interesting in the in the neat function so we're just gonna focus on the forward prop a bit later now this is interesting that the better part is gonna be probably worth kind of skimming through the neat function a number of queries is gonna be hundred as previously mentioned we just stored a transformer here we store the model dimension which is 256 the internal transformer dimension we create these class in bed and d-box in bed if you recall those are gonna map the output decoder output embedding vectors they're gonna map those into the distribution of classes and the four in the bounding boxes okay so you can see here we map from 256 which is the side dimensionality of the output embedding vector we map that into 92 because we have 91 classes in cocoa plus one for the no object for the special class okay for the b-box so for the body box we'll have MLP instead instead of a simple linear layer so this is kind of the API here is kind of convoluted but what this says is hey I want the output of this MLP to be four I want to have three layers this is these are the hidden dimensions of the layer and this is the input dimension so basically this is some type of an MLP that outputs number four dimensional vectors at the end okay we form the query embeddings just embedding table so hundred of these and dimensionality is again 256 finally this is the input projection so that's the one that maps from 2048 into 256 and we just stored the backbone and this auxiliary losses is set to true we'll see what it is basically what it does is the following let me let me go through the notebook again it's going to make sure that we fetch the outputs of every single out so so not just for the last layer of the decoder but for all of the six decoder transformer layers so that's they showed in the paper that that helps improve the performance by having intermediate losses and not just the loss at the end of the decoder stack that's the idea there okay so let's continue here again ignoring this part because this is just relevant for the panoptic segmentation pathway we don't care about that one and this is gonna construct the Hungarian matter so that's the second important part of the of the of the pipeline but again the whole magic happens in the forward pass so we're just gonna ignore all of this for now weighted dictionary is just gonna obviously define the weights for the associated loss so this loss is the label loss this is gonna be the bounding box loss etc etc so this one is gonna be five this is one again these are just some hyper parameters that they figured out during the training okay we skip again this part because that's the panoptic segmentation and because auxiliary loss is set to true we are also gonna have to define weights for all of the intermediate losses as well so that's what this is gonna do I'm just gonna skip all over this and let me let me print you what our weighted deck is gonna look like at the end so here it is if I kind of pass it here you can see that we have various different weights for a bunch of different losses that we'll have so let's see how many losses we have so like 18 losses that's that's a lot like that's three times six because we have six layers in the decoder and we have three light layers in each of those three losses in each of those okay so again we defined here the types of losses we care about this cardinality is not actually a loss we'll see what it is a bit later and yeah let's keep over this and finally this is the the final important layer during the for prop of this object here we're gonna do the matching plus the loss calculation so that's the third component of our data model that we are gonna care about okay let's continue here so here is how the in the function looks like we store the Hungarian metric here we stored a bunch of different objects nothing super vital there so one interesting detail is that we have this empty weight matrix vector and it has so as you can see the dimensionality of it it's gonna be I think it should be 92 so if I were to print this 92 so that's the number of classes plus the no object class and the reason we set the last so so all of the weights are gonna be one except that for the no object class so you can see here that's why we specify it so for the 90 so for the last element of this array we're gonna set the coefficient to 0.1 and that's going to help us put our smaller weight on this class during the label loss and the reason we have to do that is because for the for the most part we'll have only a couple of box predictions and all of the other ones are gonna be the no object prediction and remember we have hundreds of predictions so that's why we want to kind of reduce the weight for these no object predictions and that's just a common thing people do for to to kind of counter this class imbalance basically okay we registered the buffer meaning we this is not a trainable vector it's a constant but it's still a part of the model so we want to store it in the model so that's why we do register buffer again pie torch syntax continuing on we are almost done with the data model so we are almost done building the the whole thing so we just placed that this this object onto GPU and then we store some post-processing we're gonna see what those are in the forward pass okay finally we return back the model so that's the better criterion that contains the Hungarian matter and the loss calculation and we turn back this these post processors which are gonna be more relevant during the evaluation than during training but yeah bear with me for now okay so again we move the model weights on to the GPU and here yeah just some variable like renaming we don't do distributed so I can skip this here we calculate number of parameters basically we trade through all the parameters and if they require grad then we just find the number of elements in that particular layer and we sum up all of those numbers and we get that number is equal to forty something million I think so let's find where the output is so we can see here we have 41 million trainable parameters for the data model okay what this piece of code here does is separates the trainable weights into two groups one group is the backbone group the other one is the non backbone a group of parameters so we'll have a different learning rate for for the backbone and a different learning rate for the other parameters so again this is what this logic does let's not lose too much time there and they're using Adam W which additionally adds the weight decay logic into into the optimization procedure finally this is just a special learning rate scheduler that's going to drop but I think 10x after 200 epochs so after 200 epochs it's going to drop the learning rate by 10x that's the idea okay now we need to build the cocoa datasets so let me quickly go through what's going on here so we build the cocoa we we form we are going to transform each of our images using various transforms such as normalization using the image net statistics and then you can see here that we'll have some random horizontal flipping and then either random resizing or this combination of resizing cropping etc etc so just bunch of details not that interesting okay so we just store those transforms as well as some other processing here again we'll see all of that a bit later during the forward pass okay this is how the data set was formed by the way I kind of skipped through this part but let me let me quickly show you so image folder is let me print you the actual paths so we can see here the actual path towards the image directory and then we have the annotation file and so what I've done here is the following thing so I kind of took I found this repo I'm gonna link it down in the video description and they they've down sampled the training set of the of the cocoa data set into so that it only contains 25,000 images instead of 117,000 or something and I've additionally literally just extracted hundred images and then created a valid like a valid annotation file that contains only hundred annotations so those are these 391 kilobytes these small annotation files which are gonna be valid annotation files for those hundred images so by doing that I have a minimal cocoa data set that helps me be able to run and debug this code without having to download like 20 gigabytes of data or something so do let me know if you want me to share the small modifications I've done so that I have this mini data set ready for this video okay so that's just a minor detail so again this data set now loaded all of our hundred images and the associated annotations that's the idea let's continue here we do the same thing for validation I'm just gonna skip to here I'm just gonna skip all of this let's keep all of this and we are back to the main function so we have the data sets we have the optimizers we have the model here we have everything now we need to form the data loaders so for that they just create some random sampling of the training data and they're gonna have sequential sampling of the validation data nothing too interesting then they create this batch sampler so we we basically batch two training images during during training nothing fancy there just standard pie-trot stuff finally we create the data loaded loaders here using the training data set using this batch sampler and we'll have this collate function which is going to be interesting and we'll see how it works a bit later again we have data loaders ready and that's it okay finally some cocoa specific idiosyncrasies so I'll just skip this as well not that vital we don't have any frozen weights that's again relevant only for the panoptic segmentation let's continue here we don't want to resume because that's when we have a checkpoint which we don't we don't want to do eval we just want to focus on the training okay here we start so we are gonna run this for 300 epochs obviously I'm not gonna show you that in this video I'm just gonna do a single iteration a single like a loop through the training and let's see how that things work so this is this part here is the the most important part of the whole codebase let's step into the training for one epoch and here we are so we set the model to trainable as well as the criterion again as a reminder it contains the Hungarian matter plus the loss calculation logic okay I'm gonna skip all of these logging functionalities nothing super vital there and finally here we start loading the images okay so this is where the fun starts basically let's step through this and see what happens okay so we are in the cocoa detection so this is the data set class and we are grabbing a single image so let's see what happens here so we're gonna end up with I think this image is gonna be a pillow yeah type so image pillow exactly and target is just gonna be the set addiction like a list containing all of the associated annotations and you can see there is much fun tations like segmentation is crowd bonding box blah blah blah we only care about the actual bounty box so that's it let's continue here less important details like image ID is gonna be 42 415 I think that's gonna be the file name so if we if you take a look at the actual training images here I think it just grabbed yeah four two four one five it just grabbed one of these images from the training data set took its ID and that's what it's using as this image ID here nothing too interesting there let's continue so prepare just kind of processes the target this this set of annotations it's a bit cleaner nothing nothing fancy there let's continue here transform is gonna apply the set of horizontal flipping etc etc to the image so that's it so again we just fetch the image we fetch the notations we process the image we process the annotations and now let's continue let's see what happens now so we do that again the reason is remember we have batch size set to two so that's why we're gonna call get item of the data set twice and then we're gonna collate those to form the final batch okay so I'm gonna skip through all of this because we saw how it works already and now we have the call a function going on okay so let's see what's going on here so batch is going to be basically a list of couples containing as you can see image annotations image annotations and instead of doing that we want to regroup it so that we have a couple of images and a couple of annotations so that's what this align does here we unpack the batch we do the zipping we do the listing and then we end up with batch that's gonna be as you can see here image image and then annotations annotations that's that's all this is an interesting part and an important part because images in the case of object detection are gonna be oftentimes of different shapes and we don't want to resize them into the same dimension we want to operate on the original images so let's quickly step into this nasty tensor from tensor list function and see what it does so here we index with zero which means we're gonna fetch the top all that contains the tensors of images so let's do f10 here and we end up in that function okay so again the images are obviously gonna be of dimensionality they're gonna have three dimensions if I'm going to let me just plot this let me just print this oh my god so we have three seven sixty eight eleven fifty one so that's the number of channels and this is a spatial extent of the image okay let's continue here and basically here what we do is we iterate through all of the images in the batch and we find the maximum basically well let's call it the like a convex hull or a bounding box so we find the biggest box that encompasses all of the spatial resolutions of all of the images in our data set if that makes sense so let me show you what I mean by that so tensor list we saw we just saw how it looks how it looks like for the zeroth image let's see the shape for the second one and you can see that this dimension is bigger than this one and this dimension is bigger than this one and that's what why our result in resultant max size is gonna be the same shape as this tensor here so if I am to step over this and if I were to print the max size we'll see we get as I said the biggest possible bounding box that encompasses all of these spatial extents so it has to be bigger than we basically do a max across all of the images in the batch across this dimension and across this dimension and that's how we get this final max size okay so finally we we form the the the batch shape into which we're going to now copy paste all of these individual images so we end up with a batch shape of two and then the same shape as this one here then we just extract individual so the batch size the channel size the height and the width so let's continue here we just grab the types and the devices nothing fancy there and we create that this is gonna be the like the container tensor and we just initialize it with zeros so it's going to have this patch shape as we just saw here next up we're gonna form a mask that's gonna tell us later on which tokens we need to ignore when we pass them through the transformer because some of these you saw that the second image is smaller one and so some of the the tokens will have to be ignored from that smaller image that's why we have a mask and we initialize it initially with all ones let's see now what happens so we iterate through the tens or lists so that's a list containing our images and we trade through the basically this this place this this container tensor and we're just gonna like we're just gonna copy our image into the into that container tensor and then we just take the mask and we fill in the false such that the parts that remain true are the part are the parts that need to be padded for that image if that makes sense so for this first one I assume that everything is gonna be false because the bigger image has the same shape as the as the container so that's why if I were to print the mask you'll see that all of these are false and that makes sense because again the bigger image is of the same shape as the container object so let's now step through it again so now this time mask is gonna be a bit different so you can see here that we now have true here because we do need padding for the second image because it is smaller than the container tensor that's the idea and that's it like after this you can see that we've created successfully created like this container tensor we've stored our image data inside and we have the mask that also knows which tokens we need to ignore later on during during the the forward pass okay let's see what this nested tensor is basically we just store it's a simple simple object that source that tensors and the masks nothing fancy there okay so that was this part that's that was the what this collate function does and now we can return back to the main training loop okay so we can just skip all of this let me just go back to the file here so we were here we just need to skip all the way through here so f5 nice so samples just contains the nested tensors we just saw so we just created those so let me just show you that that's indeed the case if I were to print type of samples we're gonna have a nested tensor and if I were to reach for the tensors here and then do the shape we'll see that that contains our images and if I were to do instead of tensors I think a mask yeah mask and then shape here we can see that we have a mask stored so mask obviously doesn't need to have the number of channels because we just carry care about masking out certain tokens from the from the image space that's it okay so these two lines are just gonna push the labels and the images to the GPU in case you have one and now comes the fun part so this is literally the most important part of the code base so inside of the train one epoch these two lines are the main that the gist of what's going on so we have we do the the forward pass here so we get the bounding box predictions and then the criterion is gonna do the Hungarian matching followed by loss calculation so let's see how that's gonna work okay so we are in the data for pass here as you can see here so because samples is already a nested list we're just gonna skip this and the first step is to take the images so so the samples and pass them through the backbone so let's see how that's gonna work okay so here is the backbone first things first thing that this backbone is gonna do again remember there is this joiner object it's nothing fancy so this self zero let me show you what it is so self zero is simply your backbone so that's your resonant 50 so if I were to print the type of this you can see it's a backbone you later see this self one this self one is just gonna do the positionally encoding so if we were to if I were to replace this zero by one we can see that this is just position embeddings sign so the sinusoid from the like the same type of positionally embeddings as from the original transformer paper okay so that means that by doing this we are passing the nested I think this is the nested tensor let me kind of double check this again so let's just take the tensor list yeah it says here the notation says it already it's a nested tensor we passed it through the backbone so let's see the forward pass of the backbone okay here it is the backbone base we basically pass the so we extract the tensions which are the images and we just pass that through the body which is gonna fetch us the as you recall here it's just gonna fetch us the features from the layer 4 of the resonant 50 so let's do that and we end up with XS XS is just gonna have I think one key and that's this key zero and and we we've specified it here exactly so that this tells us hey if you want access layer four features access me via key zero and that's what we are going to do so if I if I were to take excess of zero we're gonna access the features and the features are gonna be as you can see of the following dimension so we have 2048 channels and we can see that the number of the spatial extent kind of reduced here so it started with we started with this shape let me show you what we had here so we started with two three and the original image resolution and we got two more channels and smaller spatial resolution which is expected from a convolutional neural network such as resonant 50 okay next up what we do is the following we grab the features from access and we take the mask and now see what we do so we basically now down sample the mask such that this mask is no longer of the same spatial resolution as the original image but it's of the same spatial resolution as the as the image features that came out of resonant 50 because remember we want to know which of those image features we want to mask out because we are gonna later on pass them through the transformer so that's why we do this step so basically you can see that mask currently has the following shape whoops let me see why what it so it's called them M shape so this is the original shape after we do the down sampling we end up with mask shape so to 1929 which is of the same resolution as the image features that came out of resonant 50 so let me now print mask and you can see again we have all false for the first image and we have some of these being true for the second image and that's it okay finally we store the image features and the associated mask into a nested vector and we return that back basically in this out dictionary that's it that's the first step of the of the backbone so now we just want to calculate the positional embeddings let's see how we do that again we grab the image features and the down sample mask here we just store that into this out dictionary so again let me remind you that X is just if I were to access tensors and bring the shape that contains the image features and if I were to ask excess mask that contains the down sample masks that's the X object here and we store it into this out list here we do just a simple positional encoding calculation nothing too fancy basically we give it the X and it's gonna do its own magic so a bunch of code but that's your just your like signs and cosines calculation so that we ultimately end up with this positional embedding like set of vectors of the shape as you can see the following so he has two because we have remember we have batch dimension of two we have the same resolution here as the as the as the image features that came out of the resonant and we have 256 because that's going to be the hidden dimension that the transformers are gonna be using that's why we have all of this okay so I encourage you to check out this this code if you're interested but I'm just gonna treat it as a black box for now it's nothing to fancy there okay we store that in this positional list and we return that back so that's the first step of the data for pass so data model again for pass and we return back the image features the positional vectors and the mask next up we just decompose the nested tensor into the source and masks the source again image features and mask the down sample mask nothing fancy there now this is the interesting part so we take the source which is again recall this is going to have two thousand twenty four sorry wait 2048 and then the down sampled image resolution so again source shape here it is and after we apply the input projection that's going to reduce this number of channels as I previously explained to 256 so that's going to happen there and then we just passed the mask so this is what we pass to the encoder and then we also pass the query vectors to the decoder and we pass the positional encodings so let's step through this piece of code let's do f10 here and we are hitting the transformer so this is a transformer layer this is the for pass and let me now show you the source that it indeed is now has 256 when it comes to a number of channels so you can see here to 256 so that's the consequence of applying that convolutional layer before passing it into the transformer so nothing fancy there now we're going to do some flattening and permutation so that we have a shape that's suitable for passing it into the transformer so that's something I mentioned here previously basically you take this and in the rest order you're gonna flatten out this this volume into a sequence of tokens that's that's the basic idea and so again let me show you the shape the shape is going to be 551 to 256 so this is the batch size this is the hidden dimension and this is the number of tokens and it's kind of weird permutation but that's what the this original pie torch implementation of the transformer expects okay instead of having batch as the first dimensionality and the reason is some optimization tricks we do the same thing for the positional embeddings so if I were to show you the positional embedding shape here we're gonna end up with literally the same shape as as the source because we want to add them up later on and finally we do a similar thing with query embedding basically they are initially just going to be 100 256 I think so hundred two fifty six and here we just add dimension like at the location one and then we repeat it whatever the number of patches is that's how many times we just copy paste these very actors and then we end up with basically the same so hundred to 256 okay we do the flattening of the mask that we can so we can pass it into interest former and finally we form this target so targets is going to be so we have a source we have the targets targets are going to be the tokens that we pass into the decoder part of the transformer layer whereas we're gonna just be using these query vectors as the positionally coatings that we're gonna sum up with the target tokens we're gonna see that in a second okay so finally we now have encoding part and decoding part let's jump and see how those are gonna work let me just see whether I have a breakpoint here I'm gonna add a breakpoint to forward post post because we're gonna have layer norm applied after the residual connection not before that's why we have this for a priest we're gonna just be using this version of the function here let's see let's see whether I should add in honor I should probably add breakpoint here as well so now let's do f10 just step over so as you can see here this is your encoder layer we do for like six times because let me show you the the length we have like six of these as you can see here so we're gonna six times just apply the transformer layer and we pass basically the source here so those are the image tokens just flattened out and we passed the mask so that's the mask that we got as the odd but if you recall from the back bone and finally we we add the positional encodings let me now go into the actual layer so let's see what's going on here so we sum up the positional encoding so those are the sinusoid we computed we sum them up with the image tokens and we get the queries and the keys and then we're just going to do a self-attention as you can see here so the only difference is that the value vectors will not contain the positional encodings that's it and then we have this the part where we mask out some parts so let me show you how this looks like so it's mostly false but then there are some parts that are true which means we're going to ignore those tokens because if you recall the that part of the image is non-existent let me quickly explain what I mean by that exactly so imagine so remember that we passed two images into our like debtor so we had image one and we had I'm gonna just draw like something like this that's of the same shape as this one here but instead so the second image was maybe like this okay so that means that we have to pad all of these tokens out we have to ignore all of these tokens and that information is now stored in this mask because it's gonna be set to true for all of these image tokens so that's kind of just a visualization that I have in my mind when I think about it so let's go back here let's step through this and that's just gonna be a single step of self-attention of image tokens across each other then this is just your like regular transform logic we have this inverse bottleneck MLP so so nothing nothing fancy there I'm not gonna lose any time doing that so let's step back here and let's continue with the decoder portion so I'm gonna click f5 and we're gonna get to the decoder okay so we end up with memory the memory is gonna be of the same dimensionality as the input source token so source tokens again those are the image tokens are of this shape and you can see that the memory is gonna be of the same shape here okay and again memory is just let me show you the the image here so that's gonna be the tokens we got we get here so we start with image features here and we end up with with the tensor that has the same dimensionality as the input image features so that's where we are at at the moment so now we need to pass through the decoder part of the transform so let me show you how that's gonna look like we passed the target which is gonna be initially all zeros we passed the memory so that's the output of the decoder we then pass what we passed the mask so that we know which tokens we should not attend to when doing the cross-attention positional embeddings query vectors bunch of details there basically everything that we have in this image here is gonna be passed into the decoder so object varies positional encodings memory targets tokens which are not displayed here but all zeros initially so all that needs to be passed into the decoder layer so now let's go back here and let me step over the let me just kind of add like a breakpoint somewhere here so here okay let's now click f10 and we're gonna hit decoder part I'm gonna ignore this so let's step through that code so here we are in a particular decoder layer so here it probably works we have the query position positional embedding so those are these positional embedding vectors here so we are going to add those to the target tokens which are gonna be all zeros initially so we end up with query and keys being equal to those query embedding vectors again we have hundred of those let me print a shape here so we have hundred of those and dimensionality is 256 okay so then we apply as you can see here we just apply a self-attention so those query vectors are gonna basically self attend to each other and nothing nothing super fancy there the interesting part in this decoder our layer happens actually at the cross attention module here so here as you can see again we are adding the query position to the target tokens and we use as keys memory so that's the output of the encoder and we add the positional encodings on top of the memory tokens and we use as the value vectors the memory vectors so we're gonna somehow combine the outputs of the encoder to form our novel representations after this so target two okay and again we'll be passing the padding information so the masking information because we do not want to attend all of the encoder outputs because some of them are not non valid because some of them stem from the parts of the image where we just added padding so it's like gonna be black if you look at the image in the image space okay so that's that part and then we just have regular transformer logic the MLP part and that's pretty much it okay guys so we're just gonna repeat this six times and as you can see we are storing the intermediate representations for the auxiliary loss I mentioned but yeah let me now step over and let's get to this part here see if I were to click f5 let me remove the breakpoint here we are now finished decoder we got decoder output so let me show you the shape of the output here we have 600 to 256 so why does this make sense so 100 is because we have 100 query embedding vectors to become because we have batch dimension of 2 to 56 is the dimensionality of the embedding vectors throughout the decoder stack and throughout the encoder stack and finally we have six because we have six decoder layers and we're storing or all of the intermediate representations because we are gonna apply this auxiliary auxiliary loss a bit later so that's it so this is the thing we're gonna be using this HS after we do this transpose operation okay so finally we applied the class embedding so that's gonna map our representation HS from this shape into we're gonna have instead of 256 92 because that's the number of classes plus the no object class so let me show you the the shape of the output class so this is gonna be as I told you 92 here then we do b-box embeds that's the MLP that's gonna give us the outputs like the bounding box coordinates so let me do f10 there and let me show you the shape so here is the shape we have four here is the output and that's it and finally we grab only the last so this is gonna grab only the output from the last decoder layer and then we're gonna store all of the intermediate representations inside of coupled with this this particular key here auxiliary outputs okay so we store the classes the logits we store the bounding boxes and then we store all of the other ones excluding the last one inside of inside of inside of this basically dictionary so let me kind of step over that and if I were to show you the keys here these are the keys we get but then inside of this key we have this list with all of the other outputs yeah hopefully yeah I know this is a bunch of details but do let me know if you if you if you get anything out of this video and even if you if you watched all the way to here like congrats like this was like very I can assume very like intensive and so I'd be super happy to hear the feedback of all of you guys who who stuck all the way to here so let's continue that was the one of the most important parts of this whole code base and now there is a second part which is much easier we're gonna soon finish up this video so let's see the the Hungarian matching algorithm and the loss calculation so let's dig into it so here we fetch just the outputs from the last layer of the decoder we ignore this auxiliary outputs and finally comes the matcher so this is the Hungarian matching algorithm let's see how this thing works fairly simple logic to be honest so let me kind of explain what's going on okay so first we are gonna grab the batch size and the number of various it's gonna be two and hundred here then we're gonna flatten out the prediction logic so again prediction logic so let me show you the shape here and this is how it looks like 292 we're gonna flat nod the first two and we're gonna apply the softmax instead of using unnormal as logits so let me do f10 here and so we end up with a shape 292 as you can see here we do the same thing for the bounding boxes so I do it here let's not do this it's gonna be 204 as you can see here so we just flatten across all of the batches and across all of the like tokens and now we'll be able to compare that to ground truth so here's the ground truth we trade through this targets dictionary that contains the annotations the ground truth annotations we grab the labels and we concatenate those so we'll have we'll have two of these for the two of images and we end up with I think 16 I already went through this code so it's 16 because we have like I think two labels for the first image and 14 labels for the second image so that means two bounding boxes for the first image and 14 for the second one we do the same thing here we just grab the bounding boxes the ground truth money boxes that we end up with 16 for let me show you the shape here so 64 and now for the fun part so this is the fun part of this whole logic so what this thing is going to do is so we take these 16 ground truth indices so this tells us as you can see on the screen here that the first bounding blocks class is 7 and then the second is 1 and then 53 53 53 etc so for all of the 200 boxes that we predicted for the batch size of 2 we're going to take those particular 16 lodges for the 16 ground truth classes and extract them into this cost class matrix okay so we're gonna end up with what 216 instead of 292 so again we start with out prop being 292 and we end up with having a subset of those with 216 okay so we'll see why that makes sense then we do the following we calculate the distance between all of our predicted boxes and the target bounding boxes this is just going to do L1 distance nothing nothing fancy there and again we're going to end up with a matrix that's 216 and it contains basically the distance between each of our 200 predictions and each of the 16 ground truth labels so we'll know basically yeah how far away each of them are like a fully connected graph okay if I click F10 let me show you and convince you that we have the shape 216 so quickly what this distance function does is let me show you so it's a L1 distance between two bounding boxes so imagine we have one bounding box that's like maybe here is one here's one so we have a bounding box here and let's imagine we have a second one here okay so okay that's it and now if you were to represent this bounding box as a vector let's use the XY XY representation so that's gonna we're gonna end up with having so that what's the coordinate of this X coordinate is 0 y coordinate is 1 then we have X coordinate is 1 y coordinate is 0 so this is going to be a representation for this bounding box and then we have this bounding box here which is going to be what X coordinate is 2 y coordinate is 1 we're taking a look at this point here and then we have 3 and we have 0 okay so now as you can imagine the how L1 is calculated is you basically subtract these two vectors and so you'll end up with what you'll end up with like and you take the absolute value so you'll have 2 0 you'll have 2 and you'll have 0 and then I think you just add this up and so you end up with like 4 so the distance between these two is gonna be 4 and so we do this for all of the predicted boxes and the ground truth boxes that's it that's the logic we apply and then we have this and I'm gonna ignore the implementation but we just find the same type of so the intersectional reunion logic I just I think I mentioned an hour ago or something okay so we end up with this cost matrix that's also 216 so after that we sum up all of those matrices and we have associated weights here 5 and we have 1 and we have 2 so what's the semantics what's the idea why are we doing this well the idea is the following we care about we're trying to match the predicted box with the ground truth box there is multiple angles at which we could look at this so we can only focus maybe on what's the closest one but then if they are close but the class is completely off is that a good match so if on the other hand we have so let me show you what I mean by that so imagine this is imagine this is a ground truth box here imagine we have a ground truth box here and imagine we have one box that's predicted like this it has some significant overlap but the class of this one is maybe cat and the class of this one is maybe dog and now let's assume we have a different bounding box so that bounding box is a little bit less overlap but it has a correct class it's cat so which one is a better one which better is a which one of these is better match so there is no correct answer and that's why we have to kind of combine both the intersectional reunion both the distance both the class to form the final matrix and then solve this basically graph flow problem to find the optimal assignment so here so we finally have the cost matrix and then we just kind of change the change the shape of that matrix so that we have let me show you here so we have two hundred and sixteen now we collect sizes because we want to separately match the prediction bounding boxes from the first batch with the ground truth boxes from the first batch from the first element of the batch and similarly for the second one so we want to have those are completely orthogonal so that's why we have to do this separation so size is just going to be 2 and 14 so 2 and 14 because we have two bounding boxes for the first image and 14 for the second one okay and now we just do this split over sizes which is basically going to split this into 102 and hundred fourteen so the two separate problems as I mentioned before so let me see whether this is gonna work if I were to print this and allocate it to a and B let's print shape of a and let's print shape of B as you can see here this is what we got okay and after splitting you can see that we have this indexing by I it's gonna be zero in the first iteration and then one in the in the second one so that means we'll be grabbing only the zero with batch here and the first one here and then basically doing this assignment problem on 102 cost matrix and hundred fourteen and it's literally our minimal flow type of an algorithm so this is what this linear sum assignment is gonna find it's gonna find such a choice of edges in the bipartite graph so as to minimize the cost and so what I mean by that let's for example take the hundred two hundred fourteen one so what is gonna happen is we have a matrix that's like let's say hundred fourteen okay and it tells you how so this row here tells you the cost between this particular predicted box and all of the fourteen ground truth boxes and so you're gonna have like what's the cost between each of those and so we can visualize this this matrix is the following way as well so you imagine you have like literally hundred nodes here and you only have 14 nodes here and so we have a fully connected graph okay something like this something like this and then we have it all of the edges are connected here we have a by part of a graph and so the the idea is to find two such edges that will minimize the the cost of this of this bipartite graph so hopefully that makes sense plus there is a constraint that you cannot pick edges that go to the same notes you have to cover all of the output nodes so you have to cover both this node and all of the other nodes and this one by doing that you literally found for each of the node you found some ideal match so maybe you found for this one maybe this one is ideal and for this one some of these ones here is ideal one so that's the idea that's the Hungarian match algorithm if you just think about it a little bit this should make sense it's not that hard okay let's get back to the code after we run that we'll end up with a set of indices and you will see what we get here basically you can see that the bounding box 17 is matched with the bonding ground truth bonding box one the bonding box 66 is matched with the ground box zero and all of this is for the first image in the batch and then for the second image we have four and the 11 are matched seven and one twenty six and ten etc etc so we got our matching that's it that's the Hungarian matching that's the second component hopefully this was clear enough let's continue on now we just need to calculate the losses and we are pretty much done that's that's everything there is to data so we just calculate the number of boxes so we can use that for normalization nothing fancy there is gonna be 16 for our particular batch so that's 16 and now we do the loss computation so let's dig into this so we have three losses we're calculating one is labels one is boxes we'll see that boxes actually encompasses two losses and then cardinality is not a loss per se but that's just an implementation detail let's let's kind of dig into this okay so loss labels is the first loss what do we do here we grab the prediction the predicted logits so that's gonna be the output of our decoder stack and the shape is gonna be this one okay so we have hundred tokens nine two logits and we have two batches we grab this we call this function which is going to basically give us how we should index this this output here so as to get those boxes we just found using the Hungarian matching algorithm so those predict the boxes that are matched to the ground truth boxes that's what this IDX index is gonna contain so you can see here box 1766 all the way through 99 those are the ones that were matched with with our ground truth boxes next up here we are just going to take the indices we got from our Hungarian matching algorithm we're just gonna focus on the ground truth indices ground truth bounding boxes indices let me remind you so this is the ground truth bounding box indices we're just gonna focus on this one and on this one and why we do that is this is just gonna be a permutation for the labels so we want to now permute the ground truth labels so that we have exact match with the predicted boxes that match those ground truth boxes so let me go through that so we end up with basically 16 classes there that are permuted in the exactly the right manner next up we create this tensor of target classes that's going to be of the following shape so 200 and it's basically going to contain 91 across all so it's gonna just contain 91s everywhere and that basically means that currently these are the classes that are output output bounding boxes should should should have but then we just copy paste the target the actual target classes so now let me print you that the target classes here you can see that some of them are now correctly whoops what have I done man let me just do that again and not click anything so let me show you what we have here so now you can see that we basically for the match boxes so let's find the first one so 53 this is the first predicted box it's gonna be 0 1 2 3 4 so the fourth predicted box should be of class 53 because that predicted box is matched with a certain ground truth box okay that's that's the basic idea again I'm just gonna kind of skim through this this is a lot of details already but hopefully this gives you some understanding and now we just do the cross entropy loss which is going to force such that all of these classes have a probability that goes up to 1 and we also have this waiting for the all of these different classes such that the no object class which is this 91 thing has lesser weight in the cross entropy which makes sense because you can see how many of these are there okay let me quickly show you in the one note what this exactly means so just to make it a bit clear imagine we have our output here let me just change the color here let me first erase this thing so this is our output we have hundred outputs here I'm gonna just picture that a bit differently so imagine we have hundred outputs here we have I'm gonna just write down maybe four okay and imagine we have two bonding boxes so we have one blue one and we have one that's a green one so now imagine that we matched this one with this one and imagine that we've matched maybe the second one with this one so if the class for this one is 23 and is the class for this one is like 15 let me change the color so if the class for this one is 15 we'll want to make sure that the logits here so that for the whatever the 15 indexes we will have like 92 here so this is gonna be 92 outputs we'll want to make sure that we push the distribution such that we have a peak around 15 because that's the ground truth class so that's what this loss is going to enforce it's going to enforce that the logic distribution for this particular output token right because remember we have output tokens which then we map into both the boxes as well as this distribution so this is gonna push that distribution so that we have the correct class similarly for here so we'll make sure that for this particular class for this particular token that at 23 we're gonna have a peak so that's what this this loss is about hopefully this kind of helps it a little bit okay let's continue on and see the second loss the second loss is gonna be the boxes loss so I'm just gonna kind of quickly go through this we get the predicted boxes here we get the target boxes we just calculate the loss between the sources boxes and the target boxes by doing the L1 I already explained that in one note and then we just do some normalization with the number of boxes we do a similar thing here so just generalized box IOU we hit that's the second component of the loss and we just do some realization and those are the two additional losses that we we have that's it and the final one is this cardinality that is just going to tell us how many of the of the how many errors do we have when it comes to computing the no object boxes okay so basically it tells us for each of the hundred predictions how many of those were different from the no object class so that's what this this thing here does so let me show you let me print this here you can see that because the model is still not trained pretty much all of these IE none of these predicts the no object bounding box because currently is just kind of random so basically if I were to print this thing let me see what we get so as you can see none of these is predicting 91 which is the no object class and that's where we get hundred hundred here and now we just do l1 loss between that and the target lengths so we expect 16 boxes 2 and 14 and so that's it like just calculating that type of discrepancy between between the ground truth and the predictions nothing fancy there that's it guys we just repeat the same procedure for all of the other five layers of the decoder I'm just gonna skip all of that okay here we are those losses are not computed and finally we just use that a weighted dictionary to sum up so for all of the losses as you can see here we go through the radio dictionary we go through the last dictionary and if if we have a corresponding weight in our weighted dictionary we just multiply those and we sum them up and we get losses so you can imagine that this is like probably the most complex loss I've seen in my life like let me let me see what's the length of this thing number of keys here 18 so we have like 18 losses in total here being computed that's it and after this this is just some boilerplate to be honest we just grab the loss final loss we do the zero grads so clean the gradients of the associated parameters in the in the torch computational graph we do the backward which is gonna compute the gradients for the last we just computed we just do some gradient clipping with 0.1 max norm and we do a step of the optimizer guys that's it this is like the longest video I ever made I'm experimenting I am super curious to know how many of you followed through the whole video this is probably the last video I'll make that this is that's this long unless I get a very good feedback on this one because it takes a lot of time to prepare these and it takes a lot of time to film and to edit super huge videos such as this one in any case I really hope you learn something new I would super appreciate your feedback do let me know if you like this video please share it out that's the best way you can help out this channel and finally subscribe to this channel if you haven't in any case until next time bye bye
[{"start": 0.0, "end": 4.12, "text": " What's cracking guys? This is the second video in the machine learning coding"}, {"start": 4.12, "end": 9.200000000000001, "text": " series and I'll be covering Facebook Deter, so that's a short for Detection"}, {"start": 9.200000000000001, "end": 15.4, "text": " Transformer and some of you might recall that I've already covered the paper in"}, {"start": 15.4, "end": 20.68, "text": " great depth I think a year ago, so if you're not familiar with what Deter is"}, {"start": 20.68, "end": 26.28, "text": " maybe consider checking out that video first, but I think this one will be like"}, {"start": 26.28, "end": 31.96, "text": " self-sufficient as well. So as I said it's a model for object detection and"}, {"start": 31.96, "end": 36.56, "text": " just for those of you who are new to the channel, object detection is this"}, {"start": 36.56, "end": 41.04, "text": " task where you have some salient objects in the image, as you can see we have some"}, {"start": 41.04, "end": 45.0, "text": " cats and remote controllers here and you can see that we have a bounding box"}, {"start": 45.0, "end": 49.6, "text": " around this cat here and we also have a label associated with the bounding box"}, {"start": 49.6, "end": 53.96, "text": " similarly for the remote controller here, for the second one here and for the"}, {"start": 53.96, "end": 58.96, "text": " other cat this blue box and finally we can see a purple box around this"}, {"start": 58.96, "end": 63.92, "text": " whole image and basically the class being couch which is the correct class"}, {"start": 63.92, "end": 69.2, "text": " in this particular case. So what I've done is I went ahead and basically"}, {"start": 69.2, "end": 74.24000000000001, "text": " cloned the official repo and as I said I'm gonna walk you through the code base"}, {"start": 74.24000000000001, "end": 81.72, "text": " in a lot of details. Before we go there I want to quickly walk you through the"}, {"start": 81.72, "end": 85.52, "text": " pipeline on the high level so that you have a mental model going into this"}, {"start": 85.52, "end": 90.92, "text": " video. The model itself is fairly simple and we'll be focusing on three"}, {"start": 90.92, "end": 94.84, "text": " components in this particular video. So first is understanding how the actual"}, {"start": 94.84, "end": 100.92, "text": " architecture works. So how do we basically, let me take a pen here, so how"}, {"start": 100.92, "end": 105.64, "text": " do we do this transformation here? How do we go from the image all the way to this"}, {"start": 105.64, "end": 113.16, "text": " set of box predictions? So understanding the architecture of the actual model. The"}, {"start": 113.16, "end": 118.16, "text": " second part is given the predictions, given the the bounding boxes predictions"}, {"start": 118.16, "end": 123.72, "text": " and the associated classes, how do we match those with the set of ground truth"}, {"start": 123.72, "end": 128.68, "text": " bounding boxes here? Okay so basically what this model outputs as we'll soon"}, {"start": 128.68, "end": 135.2, "text": " see is, and this is a hyper parameter but the model they've used 100 as the"}, {"start": 135.2, "end": 139.04, "text": " as the default, so that means we are always outputting 100 bounding boxes and"}, {"start": 139.04, "end": 144.39999999999998, "text": " so we'll have two vectors as the output from our model. One is going to be a"}, {"start": 144.39999999999998, "end": 149.23999999999998, "text": " 104 so that's going to be our bounding boxes. Obviously the four points are"}, {"start": 149.23999999999998, "end": 154.0, "text": " enough to describe the actual bounding box. The format is I think"}, {"start": 154.0, "end": 160.2, "text": " first the center point coordinates XY and then the height and the width but"}, {"start": 160.2, "end": 164.79999999999998, "text": " that's less important. The second vector that the model will output is just like"}, {"start": 164.8, "end": 171.68, "text": " 100 basically containing the classes of those associated bounding boxes here. Now"}, {"start": 171.68, "end": 176.4, "text": " as I said the hard part will be how do we match these outputs"}, {"start": 176.4, "end": 182.36, "text": " with like correct ground truth bounding boxes? So imagine we have this"}, {"start": 182.36, "end": 188.48000000000002, "text": " red bounding box represented as this red square and consider we have, I'll"}, {"start": 188.48000000000002, "end": 193.04000000000002, "text": " just pick another color like green because I don't have yellow at disposal."}, {"start": 193.04, "end": 197.32, "text": " So imagine this yellow is this green and so the idea is how do we decide how do"}, {"start": 197.32, "end": 201.76, "text": " we know that we need to connect this one with this particular ground truth"}, {"start": 201.76, "end": 207.56, "text": " bounding box and how do we know that we need to connect this red one with this"}, {"start": 207.56, "end": 212.88, "text": " red one here? And also we need to take all of the other predictions and map"}, {"start": 212.88, "end": 217.95999999999998, "text": " them into this no object into this special class that no object class and"}, {"start": 217.96, "end": 223.08, "text": " okay so that's the step number two. And the way they do that is using something"}, {"start": 223.08, "end": 228.24, "text": " called Hungarian matching algorithm. We'll see how it works in a very like a"}, {"start": 228.24, "end": 234.0, "text": " very in a lot of details. Finally once we have the matching we now need to"}, {"start": 234.0, "end": 238.8, "text": " calculate some sort of a loss and then do the back prop. So they'll have three"}, {"start": 238.8, "end": 243.48000000000002, "text": " types of losses. The first one is gonna be the label loss so that means for this"}, {"start": 243.48000000000002, "end": 247.64000000000001, "text": " for the match boxes we want to make sure that the ground truth that the"}, {"start": 247.64, "end": 252.48, "text": " probability of the ground truth label is as high as possible. That's first loss."}, {"start": 252.48, "end": 258.28, "text": " The second loss is gonna be L1 loss. So basically how far off are our"}, {"start": 258.28, "end": 262.68, "text": " predictions from from the associated ground truth boxes. So imagine we have a"}, {"start": 262.68, "end": 269.12, "text": " prediction here and imagine we have a different prediction basically maybe"}, {"start": 269.12, "end": 274.12, "text": " sorry the ground truth prediction being here. We calculate the L1 loss and I'll"}, {"start": 274.12, "end": 279.72, "text": " get into the details of how that's calculated but basically well you just"}, {"start": 279.72, "end": 283.08, "text": " treat the bounding boxes as a four-dimensional vector and you just"}, {"start": 283.08, "end": 286.4, "text": " find the L1 between the difference of those two vectors. That's how we"}, {"start": 286.4, "end": 291.0, "text": " calculate L1 loss. And the final one is the generalized intersection over union"}, {"start": 291.0, "end": 296.72, "text": " which is basically a fancy version of your theme of the of the classic IOU"}, {"start": 296.72, "end": 303.68, "text": " metric. IOU is just a way to so so what it does is it finds the area of this"}, {"start": 303.68, "end": 309.04, "text": " intersection here and it basically divides that with the union so hence the"}, {"start": 309.04, "end": 313.56, "text": " intersection of a union and you can imagine that as this tends to one that"}, {"start": 313.56, "end": 316.88, "text": " means that these two boxes are completely overlapping and you can see"}, {"start": 316.88, "end": 322.72, "text": " as the less the overlap the smaller the intersection area and the and when you"}, {"start": 322.72, "end": 327.32, "text": " divide that by the actual union that goes towards zero as the boxes don't so"}, {"start": 327.32, "end": 331.84000000000003, "text": " so yeah it will be ultimately equals zero once we don't have any overlap. So"}, {"start": 331.84, "end": 335.35999999999996, "text": " that's the third type of a loss again don't worry if you didn't understand"}, {"start": 335.35999999999996, "end": 340.2, "text": " everything we'll go into a lot of details in this particular video. Okay"}, {"start": 340.2, "end": 345.08, "text": " let me just show you a bit more detailed view of the architecture here so the"}, {"start": 345.08, "end": 350.55999999999995, "text": " idea will be to encode so we'll pass the first we pass the image to a CNN we get"}, {"start": 350.55999999999995, "end": 356.52, "text": " some set of image features we are going to then basically tokenize those"}, {"start": 356.52, "end": 361.82, "text": " image features so we'll just treat them as tokens for the NLP transformer"}, {"start": 361.82, "end": 367.0, "text": " so that part is here we'll have some so so basically I said we're gonna pass"}, {"start": 367.0, "end": 370.84, "text": " them here and then encode them using the encoder stack of the transformer we'll"}, {"start": 370.84, "end": 375.52, "text": " add some spatial positional encodings and on the other hand side we have this"}, {"start": 375.52, "end": 379.2, "text": " very interesting thing going on so we have these let me just change the color"}, {"start": 379.2, "end": 385.12, "text": " so we basically have these object fairies which are going to be learnable"}, {"start": 385.12, "end": 389.4, "text": " like hundred learnable embedding vectors and it's hundred because as I said"}, {"start": 389.4, "end": 393.0, "text": " that's just a hyper parameter and because we have hundred we'll also have"}, {"start": 393.0, "end": 399.0, "text": " output hundred like bounding boxes and you can see here what happens is that in"}, {"start": 399.0, "end": 403.88, "text": " this middle lab layer of the transformer layer we have cross attention going on"}, {"start": 403.88, "end": 409.15999999999997, "text": " so we have that these object queries are basically each of them are attending to"}, {"start": 409.15999999999997, "end": 413.76, "text": " this encoded version of the image features and each of them are looking"}, {"start": 413.76, "end": 417.91999999999996, "text": " for something in the in the image so that's and that's how on the high level"}, {"start": 417.92, "end": 422.8, "text": " obviously very hand baby how this pipeline works once we get the output"}, {"start": 422.8, "end": 428.76, "text": " set of hundred tokens here we just map them individually using MLPs or just a"}, {"start": 428.76, "end": 434.12, "text": " simple linear layer and we map so we take this token we map it into basically"}, {"start": 434.12, "end": 437.40000000000003, "text": " a class so we'll have associated class and we also map it into a four"}, {"start": 437.40000000000003, "end": 443.16, "text": " dimensional vector here and that's gonna be that's going to represent the"}, {"start": 443.16, "end": 448.72, "text": " bounding box that this particular token encodes and that's it on a very high"}, {"start": 448.72, "end": 453.74, "text": " level with this prerequisite knowledge we can now jump into the actual true"}, {"start": 453.74, "end": 459.40000000000003, "text": " piter notebook so let's see what's going on here let's start from the beginning"}, {"start": 459.40000000000003, "end": 464.68, "text": " of the notebook I'm just going to run these imports here I'm going to import"}, {"start": 464.68, "end": 468.28000000000003, "text": " the necessary libraries we're gonna be working with pytorch because this is"}, {"start": 468.28, "end": 474.96, "text": " Facebook after all or meta as they are now known to as here is a small minimal"}, {"start": 474.96, "end": 480.76, "text": " implementation of debtor it's not as complicated and it's not the same as"}, {"start": 480.76, "end": 483.52, "text": " the one they actually have in the official code base but it's good enough"}, {"start": 483.52, "end": 486.52, "text": " and gives an understanding of what's going on so I'm gonna run the cell and"}, {"start": 486.52, "end": 490.15999999999997, "text": " then explain what's going on again quickly we have a backbone"}, {"start": 490.16, "end": 498.16, "text": " resident 50 we have this a comb layer which is going to take the 2048 channels"}, {"start": 498.16, "end": 503.16, "text": " of the output embedding volume from the from the CNN and just reduce the number"}, {"start": 503.16, "end": 507.40000000000003, "text": " of dimensions to 256 and that's going to be the ultimate dimension of the tokens"}, {"start": 507.40000000000003, "end": 511.28000000000003, "text": " that we pass into the transformer so let me quickly just kind of explain what"}, {"start": 511.28000000000003, "end": 518.9200000000001, "text": " that exactly means so as I said we have an image here so this is an image we"}, {"start": 518.92, "end": 523.4799999999999, "text": " have a CNN block here I'm just gonna denote it as a box here we pass the"}, {"start": 523.4799999999999, "end": 529.7199999999999, "text": " image and out comes some some volume basically obviously the spatial so here"}, {"start": 529.7199999999999, "end": 536.12, "text": " the spatial extent was we had three channels here so this was this was three"}, {"start": 536.12, "end": 541.36, "text": " here and after encoding it through a CNN we're gonna end up with something that"}, {"start": 541.36, "end": 546.48, "text": " has like a smaller spatial extent so let me try and denote that like the"}, {"start": 546.48, "end": 551.0, "text": " following so basically we have a smaller spatial extent but then we have much"}, {"start": 551.0, "end": 556.96, "text": " more channels okay so something like this I'm really bad at drawing today so"}, {"start": 556.96, "end": 563.28, "text": " something like this basically this number of channels here is gonna be 248"}, {"start": 563.28, "end": 568.5600000000001, "text": " for resonance 50 and this is just like your your down sample spatial extent so"}, {"start": 568.5600000000001, "end": 575.96, "text": " this is like small H small W whereas this was like big H here and big W so the"}, {"start": 575.96, "end": 581.0, "text": " the convolutional layer I just mentioned has the kernel of size one times one"}, {"start": 581.0, "end": 585.5600000000001, "text": " which means each is just going to reduce the number of channels of this of this"}, {"start": 585.5600000000001, "end": 594.6800000000001, "text": " volume here from 2048 into I think 256 and then the idea is to just like treat"}, {"start": 594.6800000000001, "end": 599.08, "text": " these as a set of tokens so we can in a raster order we can just do the"}, {"start": 599.08, "end": 603.5600000000001, "text": " following we can just do this and we take this vector this is gonna be the"}, {"start": 603.56, "end": 608.8, "text": " first embedding vector which we pass into the transformer then we take the"}, {"start": 608.8, "end": 615.3599999999999, "text": " second one here so this is gonna be the second vector this one here is the"}, {"start": 615.3599999999999, "end": 619.8, "text": " second one and so yeah we basically do that in the raster order let me can zoom"}, {"start": 619.8, "end": 624.26, "text": " in here a little bit and let's show you what I mean exactly so this is vector"}, {"start": 624.26, "end": 629.28, "text": " number one vector number two vector number three vector number four five six"}, {"start": 629.28, "end": 634.56, "text": " etc etc and that's basically how this this part of the pipeline looks like"}, {"start": 634.56, "end": 639.24, "text": " okay let me go back to the code so that's this this what this conversion"}, {"start": 639.24, "end": 644.56, "text": " layer does then we have a simple transformer again encoder and decoder"}, {"start": 644.56, "end": 650.6, "text": " stacks and finally we have the two linear layers linear class and linear"}, {"start": 650.6, "end": 654.76, "text": " b-box I showed you what these do these take the output embeddings from the"}, {"start": 654.76, "end": 660.24, "text": " decoder and just map them into either the distribution over classes for the"}, {"start": 660.24, "end": 666.4399999999999, "text": " cocoa data set or this second one just gives us the bounding boxes for each of"}, {"start": 666.4399999999999, "end": 672.64, "text": " the output tokens okay output embedding vectors here the query at the query"}, {"start": 672.64, "end": 677.56, "text": " vectors the learnable query vectors we have hundreds of them and that's it and"}, {"start": 677.56, "end": 682.52, "text": " finally we have a row embed and call embed these are just going to be the"}, {"start": 682.52, "end": 686.8, "text": " the positional encodings okay and once we do so here is how the forward path"}, {"start": 686.8, "end": 691.4, "text": " looks like we basically pass the inputs so that's a set of images we pass that"}, {"start": 691.4, "end": 696.14, "text": " through the comp one then we apply some batch normalization ReLU max pooling"}, {"start": 696.14, "end": 701.0, "text": " whatnot and then we apply the four layers of the of the resonance 50 and"}, {"start": 701.0, "end": 708.12, "text": " finally we apply this small this this convolutional layer where we we basically"}, {"start": 708.12, "end": 713.32, "text": " reduce number of dimensions as I previously mentioned and what happens"}, {"start": 713.32, "end": 718.8, "text": " here is we construct the positional encodings then we're going to take those"}, {"start": 718.8, "end": 723.36, "text": " positional encodings just sum them up with these tokens here so that's this"}, {"start": 723.36, "end": 728.36, "text": " part here don't worry about the actual details we'll get to those a bit later"}, {"start": 728.36, "end": 732.8, "text": " so what is important is we have some tokens from the image we add the"}, {"start": 732.8, "end": 737.04, "text": " positional encodings we pass that into the encoder stack and then through the"}, {"start": 737.04, "end": 741.68, "text": " decoder we pass the query the query embedding vectors that gives us as the"}, {"start": 741.68, "end": 746.04, "text": " output a set of embedding vectors here which we then map using the linear class"}, {"start": 746.04, "end": 751.4599999999999, "text": " and linear b-box linear layers we additionally apply sigmoid because we"}, {"start": 751.4599999999999, "end": 758.28, "text": " want to have normalized bounding box coordinates so from 0 to 1 okay that's"}, {"start": 758.28, "end": 766.52, "text": " it now let's see what happens let's just run this this cell is going to load a"}, {"start": 766.52, "end": 773.68, "text": " pre-trained data model so I'm going to just run this and here we're just going"}, {"start": 773.68, "end": 778.6, "text": " to run the this cell that has that defines the classes from the cocoa data"}, {"start": 778.6, "end": 783.24, "text": " set and some associated colors which are going to help us visualize stuff so"}, {"start": 783.24, "end": 789.64, "text": " again the number of classes in cocoa is let's see what we have here so if I add"}, {"start": 789.64, "end": 798.08, "text": " a code cell let's bring the number of classes so shape well shape will not"}, {"start": 798.08, "end": 803.64, "text": " work because this is what this is like a list so we're gonna do length is that so"}, {"start": 803.64, "end": 809.68, "text": " length and this should be I think 91 or 91 yeah 91 and then we'll add the"}, {"start": 809.68, "end": 816.48, "text": " additional no object class we'll see that a bit later okay here's some helper"}, {"start": 816.48, "end": 823.48, "text": " functions this one helps us just do some type of processing to the input images"}, {"start": 823.48, "end": 829.8000000000001, "text": " we do some resize operator then we convert the images into pie torch"}, {"start": 829.8000000000001, "end": 834.76, "text": " tensors and then we do image net normalization so these are statistics"}, {"start": 834.76, "end": 838.24, "text": " calculated from the actual image net data set this is going to be like the"}, {"start": 838.24, "end": 844.24, "text": " mean and this is going to be the standard deviation statistic of over the"}, {"start": 844.24, "end": 849.08, "text": " calculated over the image net data set these helper functions just map as you"}, {"start": 849.08, "end": 855.72, "text": " can see from the format CX CY WH which is as you specify the center coordinate"}, {"start": 855.72, "end": 859.6, "text": " of the bounding box and width and height and instead of that it's going to"}, {"start": 859.6, "end": 864.52, "text": " convert it into a format where we specify the top left point and the bottom"}, {"start": 864.52, "end": 869.04, "text": " right point and those are obviously equally informative representations"}, {"start": 869.04, "end": 873.52, "text": " that take equal amount of space it's just that this representation is a bit"}, {"start": 873.52, "end": 877.72, "text": " more suitable to some of the API's they're actually using and we'll see that"}, {"start": 877.72, "end": 882.8, "text": " later not that important rescale boxes so as I said we do predict the bounding"}, {"start": 882.8, "end": 886.88, "text": " boxes which are normalized which means from 0 to 1 this is just going to return"}, {"start": 886.88, "end": 893.92, "text": " a map those predictions into the original input image space like a range"}, {"start": 893.92, "end": 899.84, "text": " okay did we run the cell and just run the cell and finally we have the"}, {"start": 899.84, "end": 905.36, "text": " detection function here what we do is we pass an image we apply the transforms"}, {"start": 905.36, "end": 909.84, "text": " we just saw we unsqueeze so that we add like the batch dimension so this adds a"}, {"start": 909.84, "end": 915.5600000000001, "text": " dummy batch dimension one because that's what by torch models expect just some"}, {"start": 915.5600000000001, "end": 920.72, "text": " error checking we pass the images through the model we get the outputs and"}, {"start": 920.72, "end": 926.36, "text": " finally what we do here is we take the prediction logits we apply the softmax"}, {"start": 926.36, "end": 931.2, "text": " because remember this is just going to apply linear layer so these are still"}, {"start": 931.2, "end": 935.92, "text": " unnormalized logits so we apply softmax after we've normalized the logits using"}, {"start": 935.92, "end": 941.8000000000001, "text": " softmax we just extract all of the classes except for the no object class"}, {"start": 941.8000000000001, "end": 947.4, "text": " here that's what we have up to minus one and then what we do is we take those"}, {"start": 947.4, "end": 955.52, "text": " probabilities we find the max across all of the hundred outputs and we find those"}, {"start": 955.52, "end": 961.16, "text": " predictions where the peak of the distribution across classes is at least"}, {"start": 961.16, "end": 967.48, "text": " 0.7 in that way we kind of take out only those very confident bounding box"}, {"start": 967.48, "end": 972.4399999999999, "text": " predictions okay that's the idea there and then we use that vector the key"}, {"start": 972.4399999999999, "end": 978.0, "text": " vector to basically extract only those bounding boxes and then we just rescale"}, {"start": 978.0, "end": 983.3199999999999, "text": " those and return back the bounding boxes we additionally return the associated"}, {"start": 983.32, "end": 989.0400000000001, "text": " probabilities here so if I run this and if we run this here we're going to get"}, {"start": 989.0400000000001, "end": 992.72, "text": " as the output the scores and the bounding boxes so let's kind of quickly"}, {"start": 992.72, "end": 997.24, "text": " examine what's going on here let's see what the shapes are here so if I do"}, {"start": 997.24, "end": 1004.5200000000001, "text": " print score shape that's going to give us 591 so it's five because only five of"}, {"start": 1004.5200000000001, "end": 1012.5600000000001, "text": " the hundred bounding boxes were like confident enough and if I was to"}, {"start": 1012.56, "end": 1018.4799999999999, "text": " print the boxes this should be what this should be five four that's the"}, {"start": 1018.4799999999999, "end": 1024.24, "text": " expectation so let's run it's five four indeed so five boxes that are good"}, {"start": 1024.24, "end": 1033.6399999999999, "text": " enough so I'm gonna quickly just modify this and return back like this"}, {"start": 1033.6399999999999, "end": 1039.96, "text": " probability vector so this should contain if I'm not wrong so let me just"}, {"start": 1039.96, "end": 1046.16, "text": " actually return back this original output so like this and I'm just gonna"}, {"start": 1046.16, "end": 1052.56, "text": " call that logits so I'm gonna run that cell I'm gonna add a logits whoops"}, {"start": 1052.56, "end": 1061.44, "text": " variable here so just add logits logits run that and see what the shape of that"}, {"start": 1061.44, "end": 1067.8400000000001, "text": " thing is so that should be a hundred ninety one I assume right so if I were"}, {"start": 1067.84, "end": 1074.04, "text": " to run this we have 192 so it's 192 because as I said here we extract the"}, {"start": 1074.04, "end": 1081.12, "text": " no object box so if we were to take this probus whatever like it's a weird"}, {"start": 1081.12, "end": 1087.24, "text": " variable name if we were to take that variable and return it instead that's"}, {"start": 1087.24, "end": 1092.9199999999998, "text": " gonna be that's going to give us 91 instead of 92 so let me show you here"}, {"start": 1092.9199999999998, "end": 1096.8, "text": " and as you can see here that's that's pretty much it okay now we can finally"}, {"start": 1096.8, "end": 1103.36, "text": " visualize what's the output of this of this function detect function and what"}, {"start": 1103.36, "end": 1109.9199999999998, "text": " we do is we iterate over this over the probabilities over the bounding boxes"}, {"start": 1109.9199999999998, "end": 1116.0, "text": " and we just plot rectangles using the coordinates from those boxes and we"}, {"start": 1116.0, "end": 1123.6399999999999, "text": " plot the basically the most probable class from the distribution there and"}, {"start": 1123.64, "end": 1128.5600000000002, "text": " if I were to run this I mean we already had the image so let me let me show you"}, {"start": 1128.5600000000002, "end": 1132.8400000000001, "text": " this is what we get so this is the image with which we started this video I"}, {"start": 1132.8400000000001, "end": 1137.16, "text": " showed you this one in the beginning okay so far so good let's now jump to"}, {"start": 1137.16, "end": 1143.2800000000002, "text": " the second notebook this was just like a quick like a high level overview of how"}, {"start": 1143.2800000000002, "end": 1149.8000000000002, "text": " minimal implementation of a debtor could look like and it turns out it works"}, {"start": 1149.8, "end": 1154.68, "text": " already fairly fairly nicely let's now go to this one where you're gonna see"}, {"start": 1154.68, "end": 1158.76, "text": " some attention visualization so I'm gonna again run the necessary imports I'm"}, {"start": 1158.76, "end": 1165.72, "text": " gonna again run the cocoa classes here we have the same type of helper"}, {"start": 1165.72, "end": 1170.68, "text": " functions that transform functions we have the these functions that change the"}, {"start": 1170.68, "end": 1176.24, "text": " box format they do the rescaling so I'm just gonna run those again we have the"}, {"start": 1176.24, "end": 1180.04, "text": " same plot results function as from the previous two Python notebook the details"}, {"start": 1180.04, "end": 1184.1200000000001, "text": " are not important it's just like your math plotlib stuff I'm gonna focus on"}, {"start": 1184.1200000000001, "end": 1188.56, "text": " the decor and the gist and that's the actual data model and how those three"}, {"start": 1188.56, "end": 1191.24, "text": " components that I showed you in the beginning of the video so that the"}, {"start": 1191.24, "end": 1195.92, "text": " architecture the Hungarian matching and the loss calculation look like that's"}, {"start": 1195.92, "end": 1201.04, "text": " gonna be the focus of the video again let's load some of the pre-trained"}, {"start": 1201.04, "end": 1207.36, "text": " models here we set the model into eval mode so eval just for those of you who"}, {"start": 1207.36, "end": 1211.48, "text": " are not familiar with pytorch sets so that's very important because some of"}, {"start": 1211.48, "end": 1214.8799999999999, "text": " the layers such as batch norm behave differently whether they are in the"}, {"start": 1214.8799999999999, "end": 1221.72, "text": " depending on whether they are in the train or eval setting okay and we did"}, {"start": 1221.72, "end": 1226.52, "text": " the same thing we just load load an image here and then we have a similar"}, {"start": 1226.52, "end": 1230.96, "text": " type of a forward a prop as in the previous detect function from the"}, {"start": 1230.96, "end": 1234.48, "text": " previous notebook so we just pass the images we transform them and squeeze"}, {"start": 1234.48, "end": 1239.52, "text": " them pass through the through the model we get the outputs we do the same type"}, {"start": 1239.52, "end": 1243.56, "text": " of transform here so we get the probabilities and this time the"}, {"start": 1243.56, "end": 1249.04, "text": " threshold is a bit higher we use 0.9 here although I think I am the one who"}, {"start": 1249.04, "end": 1254.4, "text": " modified this not sure let's let's assume this is 0.9 and then we do the"}, {"start": 1254.4, "end": 1259.48, "text": " rescaling so let's see what we get so yeah for the most part this part was the"}, {"start": 1259.48, "end": 1264.16, "text": " same as in the previous notebook so we get the output predictions now that we"}, {"start": 1264.16, "end": 1271.0400000000002, "text": " have all of that calculated let's go and add some hooks to our model so what the"}, {"start": 1271.0400000000002, "end": 1276.8400000000001, "text": " hook does is as the image is passing through the actual data model we're"}, {"start": 1276.8400000000001, "end": 1283.3600000000001, "text": " going to log certain activations and the ones we are going to be interested in is"}, {"start": 1283.36, "end": 1287.84, "text": " going to be the activations from the actual backbone model so that's going to"}, {"start": 1287.84, "end": 1290.84, "text": " be the calm features let me let me show you what I mean by that so it's going to"}, {"start": 1290.84, "end": 1297.04, "text": " be we're going to log these features here better yet so we're gonna log"}, {"start": 1297.04, "end": 1303.6399999999999, "text": " actually these features here and we're going to log some of the basically"}, {"start": 1303.6399999999999, "end": 1307.1999999999998, "text": " attention coefficients I think from the let's see I think from the multi hat"}, {"start": 1307.2, "end": 1313.76, "text": " attention like the cross attention component here so let's just make sure"}, {"start": 1313.76, "end": 1318.0800000000002, "text": " that's the case so we get we take from from the encoder from the last layer of"}, {"start": 1318.0800000000002, "end": 1322.6000000000001, "text": " the encoder we're gonna grab the self attention module so that means we're"}, {"start": 1322.6000000000001, "end": 1327.4, "text": " going to grab from the last layer so as you can see we have and here I think in"}, {"start": 1327.4, "end": 1332.76, "text": " practice they use six so from the sixth one we're gonna grab the activations"}, {"start": 1332.76, "end": 1338.0, "text": " from this module here okay multi had self attention in particular we're gonna"}, {"start": 1338.0, "end": 1342.52, "text": " take the matrix like when you multiply query vectors with with key vectors you"}, {"start": 1342.52, "end": 1348.2, "text": " get a matrix of scores which after you which after you apply the softmax you"}, {"start": 1348.2, "end": 1351.56, "text": " get basically the attention scores so that's the one we're gonna visualize and"}, {"start": 1351.56, "end": 1356.0, "text": " the second one if I'm not wrong I think it's this one here but let's double"}, {"start": 1356.0, "end": 1362.56, "text": " check if I go back to the code here we can see we again we take the decoder the"}, {"start": 1362.56, "end": 1367.84, "text": " last layer of decoder multi had attention and I'm fairly sure that's the"}, {"start": 1367.84, "end": 1373.8799999999999, "text": " second the cross attention one so let me quickly go to the transformer file here"}, {"start": 1373.8799999999999, "end": 1379.6799999999998, "text": " let me just make sure that's the case so transformer decoder layer we are going"}, {"start": 1379.6799999999998, "end": 1386.24, "text": " to have let's see so self attention and multi head attention the multi head yeah"}, {"start": 1386.24, "end": 1391.0, "text": " the multi head attention is the second is the cross attention layer okay so we"}, {"start": 1391.0, "end": 1396.56, "text": " are basically taking the activations from this particular multi head attention"}, {"start": 1396.56, "end": 1401.88, "text": " head okay not yet but the module sorry okay let's go back that's enough"}, {"start": 1401.88, "end": 1408.96, "text": " rambling let's go back to the notebook after we attach these hooks and this is"}, {"start": 1408.96, "end": 1413.66, "text": " just your pie tour syntax I will not get into details there and we run the"}, {"start": 1413.66, "end": 1419.88, "text": " inference all of those are gonna be locked in the associated lists then we"}, {"start": 1419.88, "end": 1424.6000000000001, "text": " just remove the hooks and then we extract the zeroth element from the list"}, {"start": 1424.6000000000001, "end": 1427.92, "text": " because here as you can see we just append it so we by doing this we just"}, {"start": 1427.92, "end": 1432.8000000000002, "text": " extract the collected data so I'm gonna run this cell and let's see what happens"}, {"start": 1432.8000000000002, "end": 1438.9, "text": " okay so we end up with those shapes I'm going to quickly analyze what's going on"}, {"start": 1438.9, "end": 1446.2, "text": " so con features let's print what we get here so what I expect is I assume"}, {"start": 1446.2, "end": 1451.68, "text": " something like the following shape like maybe one to 48 and then some spatial"}, {"start": 1451.68, "end": 1455.92, "text": " extent which is gonna be I think something like age is gonna be 19 and"}, {"start": 1455.92, "end": 1459.0, "text": " double is gonna be 29 that's less important but let's kind of print this"}, {"start": 1459.0, "end": 1465.3600000000001, "text": " and see what we get okay so we get that this uh-huh oh yeah or I remember so"}, {"start": 1465.3600000000001, "end": 1469.76, "text": " there is this particular detail of the implementation so we have to actually"}, {"start": 1469.76, "end": 1477.8, "text": " index we have to index by this key and only then do we actually get the okay"}, {"start": 1477.8, "end": 1481.32, "text": " and then we have to do additional tensors because additional implementation"}, {"start": 1481.32, "end": 1485.0, "text": " detail because this is gonna give us nested tensor and we just want the tensor"}, {"start": 1485.0, "end": 1490.72, "text": " info so finally okay this is what we get as I said I kind of missed the small H"}, {"start": 1490.72, "end": 1496.2, "text": " and small W but that was not the point that the same overall format is as we"}, {"start": 1496.2, "end": 1504.04, "text": " expected let's copy paste this and let me do the following here I'm not sure"}, {"start": 1504.04, "end": 1507.64, "text": " whether this is gonna be nested vectors let me just try and do shapes like this"}, {"start": 1507.64, "end": 1512.4, "text": " I'm gonna comment this out comment this out see whether this works and it does"}, {"start": 1512.4, "end": 1520.04, "text": " so 850 850 that's the I said the scores but what you get by multiplying the"}, {"start": 1520.04, "end": 1526.8799999999999, "text": " queries and key vectors similarly here if I were to do this and we're gonna"}, {"start": 1526.8799999999999, "end": 1531.72, "text": " visualize the detection attention weights let's see the shape here so we're"}, {"start": 1531.72, "end": 1536.8, "text": " gonna have I guess the same exact shape oh no actually it's hundred because here"}, {"start": 1536.8, "end": 1542.24, "text": " we are in the decoder stack so because of that we have a hundred eight hundred"}, {"start": 1542.24, "end": 1548.12, "text": " fifty because we have hundred very the learnable vectors and we have eight"}, {"start": 1548.12, "end": 1553.36, "text": " hundred fifty keys from the encoder stack that's why we get this this"}, {"start": 1553.36, "end": 1558.6, "text": " different shape it all makes sense nice we've seen the shapes now let's"}, {"start": 1558.6, "end": 1562.8799999999999, "text": " visualize the attention so we're gonna in particular focus on visualizing the"}, {"start": 1562.8799999999999, "end": 1569.1599999999999, "text": " decoder attention weights and as you can see here we they just grab the height"}, {"start": 1569.1599999999999, "end": 1577.4399999999998, "text": " and width from the com features that's gonna be the 2534 here and the each way"}, {"start": 1577.44, "end": 1580.92, "text": " through the bounding boxes we previously predicted so the five bounding boxes we"}, {"start": 1580.92, "end": 1585.96, "text": " previously predicted and visualized above here so these are the two remote"}, {"start": 1585.96, "end": 1592.44, "text": " controllers two cats and a couch and they are then going to as you can see"}, {"start": 1592.44, "end": 1598.0800000000002, "text": " here grab a particular index so remember this is gonna be of the shape hundred"}, {"start": 1598.0800000000002, "end": 1605.04, "text": " because we have hundred query embeddings and 850 because we have 850 of the"}, {"start": 1605.04, "end": 1611.24, "text": " basically keys coming from the image features of the 3d encoder stack and so"}, {"start": 1611.24, "end": 1621.24, "text": " this is going to grab 850 coefficients that basically correspond to the scores"}, {"start": 1621.24, "end": 1627.36, "text": " that this particular query vector has when it attends to the keys okay just to"}, {"start": 1627.36, "end": 1633.0, "text": " quickly clarify what I mean by that is it means the following basically so"}, {"start": 1633.0, "end": 1640.68, "text": " again remember we are we have encoder output here so we have basically 850"}, {"start": 1640.68, "end": 1646.72, "text": " outputs here and then we have the decoder stack here and as I said we"}, {"start": 1646.72, "end": 1653.88, "text": " grab the cross attention layer and that cross attention layer has hundred"}, {"start": 1653.88, "end": 1660.32, "text": " hundred query vectors we have in total hundred query vectors there and we take"}, {"start": 1660.32, "end": 1663.56, "text": " a particular one that's associated with the bounding box we care about so we"}, {"start": 1663.56, "end": 1670.08, "text": " take for example we take this one here and we see how that query vector"}, {"start": 1670.08, "end": 1674.46, "text": " basically we do the dot product between that query vector and all of these key"}, {"start": 1674.46, "end": 1679.28, "text": " vectors and those are the 850 numbers that we're now going to visualize okay"}, {"start": 1679.28, "end": 1685.72, "text": " that's that's the basic idea so let's see what we get so after we do that and"}, {"start": 1685.72, "end": 1691.28, "text": " run this cell we have this so for this particular remote controller you can see"}, {"start": 1691.28, "end": 1699.52, "text": " that this is the pattern that that we attend to in the these are the image"}, {"start": 1699.52, "end": 1703.56, "text": " features that we attend to and you can see there is obviously a correlation"}, {"start": 1703.56, "end": 1709.06, "text": " between these two similarly for this image here similarly for the this is I"}, {"start": 1709.06, "end": 1713.6000000000001, "text": " guess the couch and so we are basically attending the ends of the image that's"}, {"start": 1713.6, "end": 1719.1999999999998, "text": " how it figures out that this is the couch that we are seeing and then you"}, {"start": 1719.1999999999998, "end": 1725.0, "text": " can see kind of a faint figure of the cat here and finally similarly here we"}, {"start": 1725.0, "end": 1729.0, "text": " can see a faint figure of the cat here okay so hopefully this gave you some"}, {"start": 1729.0, "end": 1733.9399999999998, "text": " understanding of what's what's going on and that I hope that this this image"}, {"start": 1733.9399999999998, "end": 1739.34, "text": " here helped you understand again what's going on so let me just recap one part"}, {"start": 1739.34, "end": 1745.76, "text": " so the shape here is 2534 if you multiply those two numbers you get 850"}, {"start": 1745.76, "end": 1752.6399999999999, "text": " that's why we have 850 here so 2534 so remember that we started here from the"}, {"start": 1752.6399999999999, "end": 1759.86, "text": " image features that were precisely 2534 and then we took those in a"}, {"start": 1759.86, "end": 1766.12, "text": " rest order and we kind of flattened those out and that's how we got these 850"}, {"start": 1766.12, "end": 1770.04, "text": " tokens so that means by attending to these 850 tokens you're actually"}, {"start": 1770.04, "end": 1775.08, "text": " attending to the original image but just a down sampled version that's why we are"}, {"start": 1775.08, "end": 1780.6, "text": " able to get these interesting attention maps here so that's how you connect"}, {"start": 1780.6, "end": 1789.9599999999998, "text": " these 850 with the images we get here so with these images here okay so yeah"}, {"start": 1789.9599999999998, "end": 1795.4799999999998, "text": " that's the best I can do for now but bear with me we'll get into we'll"}, {"start": 1795.48, "end": 1800.88, "text": " understand all of the details a bit later so what I do here is just they"}, {"start": 1800.88, "end": 1807.24, "text": " just plot the they just print these shapes and we saw all of this already so"}, {"start": 1807.24, "end": 1812.4, "text": " I'm just gonna skip through all of that because we already saw it what they"}, {"start": 1812.4, "end": 1816.68, "text": " additionally do is they take the now the encoder attention weights instead of"}, {"start": 1816.68, "end": 1823.8, "text": " decoder ones and they just reshape those such that we end up with 2534 2534"}, {"start": 1823.8, "end": 1830.36, "text": " instead of with 850 850 so this is just going to be a bit easier to understand"}, {"start": 1830.36, "end": 1837.96, "text": " what's going on when we index into this array into this tensor okay let's now"}, {"start": 1837.96, "end": 1844.54, "text": " visualize the encoder attention patterns so they grab four points from the"}, {"start": 1844.54, "end": 1851.12, "text": " original image and then using those points they're going to visualize the"}, {"start": 1851.12, "end": 1855.8799999999999, "text": " encoder attention so let's see what exactly happens here so you can see"}, {"start": 1855.8799999999999, "end": 1861.12, "text": " original image they picked four points denoted here by these red circles and"}, {"start": 1861.12, "end": 1869.8799999999999, "text": " then we visualize the where these points attend to in the original input image"}, {"start": 1869.8799999999999, "end": 1875.28, "text": " because we have the self-attention in the encoder layer which are the image"}, {"start": 1875.28, "end": 1878.84, "text": " tokens and that's why we get these image patterns and you can see the"}, {"start": 1878.84, "end": 1883.9199999999998, "text": " interesting fact about data is that the encoder already learns to do some sort"}, {"start": 1883.9199999999998, "end": 1888.9199999999998, "text": " of instance segmentation which is super useful if you later want to create a"}, {"start": 1888.9199999999998, "end": 1894.1999999999998, "text": " bounding boxes around those objects and this was not explicitly trained for so"}, {"start": 1894.1999999999998, "end": 1897.9599999999998, "text": " this is just something that emerges from the from the way that data was trained"}, {"start": 1897.9599999999998, "end": 1902.56, "text": " which is super nice so let's kind of dig through this code and understand what's"}, {"start": 1902.56, "end": 1908.32, "text": " going on it's a bit messy here so I'm gonna just keep it on a high level"}, {"start": 1908.32, "end": 1916.56, "text": " basically they create a bunch of subplots that's not that interesting and"}, {"start": 1916.56, "end": 1925.12, "text": " then the important part is let's see so the important part is this so we have"}, {"start": 1925.12, "end": 1932.2, "text": " image show of the attention pattern so we take the index so indices are again"}, {"start": 1932.2, "end": 1937.24, "text": " the points we picked right and so we grab those points those points are"}, {"start": 1937.24, "end": 1943.6, "text": " initially in the image space and so because of that we want to down sample"}, {"start": 1943.6, "end": 1948.76, "text": " because that's what the calm layer does the backbone is going to don't sample"}, {"start": 1948.76, "end": 1953.92, "text": " by 32 that's why we divide by fact and this fact is going to be equal 32 that's"}, {"start": 1953.92, "end": 1959.84, "text": " how we get the indices for the down sampled image features and then we just"}, {"start": 1959.84, "end": 1967.16, "text": " basically grab the points and index into this attention tensor okay and by"}, {"start": 1967.16, "end": 1971.48, "text": " doing that we basically figure out the following I'm gonna again use my one"}, {"start": 1971.48, "end": 1974.8400000000001, "text": " note because this is very hard to I realize this can be hard to understand"}, {"start": 1974.8400000000001, "end": 1980.3600000000001, "text": " but just me speaking about it so what what we've done is the following so"}, {"start": 1980.3600000000001, "end": 1987.76, "text": " again remember we have this volume here and"}, {"start": 1987.76, "end": 1996.68, "text": " so what we are going to do with it is the following so this is our volume"}, {"start": 1996.68, "end": 2003.16, "text": " these are our tokens basically so this is the volume that came out from the"}, {"start": 2003.16, "end": 2009.16, "text": " CNN so we take a particular point so let me kind of create a grid structure here"}, {"start": 2009.16, "end": 2016.12, "text": " so make it easier to visualize we take a particular point so this one here so"}, {"start": 2016.12, "end": 2023.9799999999998, "text": " that's as you can recall that's the tens that's the token that is fed into the"}, {"start": 2023.9799999999998, "end": 2029.08, "text": " encoder transformer so somewhere in the transformer we're gonna map this into a"}, {"start": 2029.08, "end": 2034.08, "text": " query vector and then we're gonna take that very vector and basically do a"}, {"start": 2034.08, "end": 2040.0, "text": " dot product with the key vectors of all of these other tokens right all of these"}, {"start": 2040.0, "end": 2044.7399999999998, "text": " other tokens and by doing that we basically understand how this image"}, {"start": 2044.74, "end": 2051.6, "text": " token is attending all of the other image tokens in this in this volume here"}, {"start": 2051.6, "end": 2056.24, "text": " so that's the idea by the way obviously this is flattened in the actual"}, {"start": 2056.24, "end": 2062.44, "text": " transformer I'm just visualizing this as in this in this spatial format just"}, {"start": 2062.44, "end": 2067.56, "text": " because it's easier to map what's happening here and between and why do we"}, {"start": 2067.56, "end": 2072.56, "text": " have the images that we get here so hopefully that was somewhat"}, {"start": 2072.56, "end": 2077.72, "text": " understandable and now this final cell just makes this basically interactive"}, {"start": 2077.72, "end": 2082.48, "text": " so you can pick your own image and do the same thing in an interactive fashion"}, {"start": 2082.48, "end": 2087.88, "text": " so if I were to run this they created this nice widget where you basically"}, {"start": 2087.88, "end": 2092.56, "text": " have let me show you what happens so you can see here you can kind of move this"}, {"start": 2092.56, "end": 2099.4, "text": " red point here and you can see how the attention patterns change so if I were"}, {"start": 2099.4, "end": 2104.1600000000003, "text": " to move this a bit more directly on to the cat well we get some weird patterns"}, {"start": 2104.1600000000003, "end": 2111.08, "text": " so apparently when we are closer to the boundary of the cat that's when we we"}, {"start": 2111.08, "end": 2118.2400000000002, "text": " get the nicest patterns of attention here so you can also play and just use"}, {"start": 2118.2400000000002, "end": 2122.2400000000002, "text": " some other image from the internet and this is gonna work so that's the idea"}, {"start": 2122.2400000000002, "end": 2129.04, "text": " with this with this Jupyter hopefully you got some understanding of how of the"}, {"start": 2129.04, "end": 2132.7599999999998, "text": " things that this data model is learning so you can see that we have interesting"}, {"start": 2132.7599999999998, "end": 2137.96, "text": " patterns going on in the encoder so instance segmentation types of of masks"}, {"start": 2137.96, "end": 2144.92, "text": " here and we also saw nice patterns in the actual decoder where we saw where"}, {"start": 2144.92, "end": 2150.0, "text": " the query tokens from the decoder stack are attending to in the original"}, {"start": 2150.0, "end": 2156.32, "text": " down sampled image features again quickly this is decoder so the query"}, {"start": 2156.32, "end": 2163.6000000000004, "text": " embedding vectors attending the embedding vectors from the encoder stack"}, {"start": 2163.6000000000004, "end": 2170.44, "text": " so that's this part here so again query vector attending the embedding vectors"}, {"start": 2170.44, "end": 2175.1200000000003, "text": " from the top of the encoding stack and by visualizing that you basically"}, {"start": 2175.1200000000003, "end": 2182.52, "text": " visualize the where this query token is attending to the query vector and the"}, {"start": 2182.52, "end": 2191.44, "text": " second image is about grabbing the image token and visualizing where that image"}, {"start": 2191.44, "end": 2197.68, "text": " token is attending to over the over the neighborhood so that's that's this"}, {"start": 2197.68, "end": 2202.8, "text": " particular image here so should be fairly simple and easy to understand"}, {"start": 2202.8, "end": 2206.74, "text": " cool now let's start digging through the actual code base and for that I'll"}, {"start": 2206.74, "end": 2211.2, "text": " first have to check out the main branch so I'm gonna have to check out the main"}, {"start": 2211.2, "end": 2216.8399999999997, "text": " branch because the colabs are so the the Jupyter notebooks are checked into this"}, {"start": 2216.8399999999997, "end": 2222.64, "text": " colab branch so if I do get check out main now I'll be able to find the code"}, {"start": 2222.64, "end": 2228.68, "text": " that I need so let's go back here and let's start debugging the main file so"}, {"start": 2228.68, "end": 2232.2, "text": " I'm gonna skip a lot of unnecessary details like because the code was"}, {"start": 2232.2, "end": 2239.12, "text": " developed to be trained on a you know like a multi node multi GPU setup we"}, {"start": 2239.12, "end": 2242.56, "text": " don't care about that we just care about understanding how data works also there"}, {"start": 2242.56, "end": 2246.92, "text": " is a lot of details about pen optic segmentation because data can easily be"}, {"start": 2246.92, "end": 2250.56, "text": " adjusted so that it works for the pen optic segmentation desk we're also"}, {"start": 2250.56, "end": 2255.8399999999997, "text": " going to ignore everything there and finally the actual cocoa implementation"}, {"start": 2255.8399999999997, "end": 2259.8399999999997, "text": " details are not a part of this code base but they are using an existing API so"}, {"start": 2259.8399999999997, "end": 2264.24, "text": " we'll kind of skim through that as well so the main focus is let's understand"}, {"start": 2264.24, "end": 2268.52, "text": " the three components of data the architecture the Hungarian matching"}, {"start": 2268.52, "end": 2272.32, "text": " algorithm and finally how do we calculate those three types of losses we"}, {"start": 2272.32, "end": 2277.44, "text": " saw in the beginning of the video so we offered our do let's start here so we"}, {"start": 2277.44, "end": 2282.92, "text": " are going to just ignore these parts this just prints the basically the shot"}, {"start": 2282.92, "end": 2288.72, "text": " of the of the current commit but that's again not vital I'm gonna skip all of"}, {"start": 2288.72, "end": 2293.96, "text": " this we just grab the device so I have a GPU so it's gonna be GPU is gonna be"}, {"start": 2293.96, "end": 2299.6, "text": " used so for reproducibility reasons they they set the seeds for various libraries"}, {"start": 2299.6, "end": 2306.04, "text": " pytorch non pythons random etc so that not that important now this is where"}, {"start": 2306.04, "end": 2310.84, "text": " the actual important part starts let's see how the model is built so we first"}, {"start": 2310.84, "end": 2317.6, "text": " have basically we picked a number of classes because we'll have 91 here"}, {"start": 2317.6, "end": 2324.16, "text": " because we are dealing with cocoa so as you can see here we have 91 and then as"}, {"start": 2324.16, "end": 2328.36, "text": " I said we're gonna ignore the pen optic part so now let's build the backbone so"}, {"start": 2328.36, "end": 2333.48, "text": " the belt bone is the resonance 50 but let's see the a bit more details there"}, {"start": 2333.48, "end": 2339.16, "text": " so first of all we're gonna build the position in coatings and the interesting"}, {"start": 2339.16, "end": 2342.56, "text": " part of this positioning coatings is going to be in the actual for a function"}, {"start": 2342.56, "end": 2349.08, "text": " of this of this module so we can just kind of skip through that because the"}, {"start": 2349.08, "end": 2355.32, "text": " learning rate is bigger than zero we are gonna make backbone trainable nothing"}, {"start": 2355.32, "end": 2360.04, "text": " fancy there so now we form the backbone and this join our object built which"}, {"start": 2360.04, "end": 2364.64, "text": " will see what it is in a second so it's basically just gonna join the backbone"}, {"start": 2364.64, "end": 2368.64, "text": " which is gonna be the resonant 50 and the positioning beddings so let's step"}, {"start": 2368.64, "end": 2372.68, "text": " into the constructor of the backbone here you can see what happens is that"}, {"start": 2372.68, "end": 2378.16, "text": " name is going to be the resident 50 so we get the we fetch the class the"}, {"start": 2378.16, "end": 2382.68, "text": " resident 50 class from the torch vision models and you can see here that"}, {"start": 2382.68, "end": 2389.08, "text": " sedulation is set to false and because of that we'll just fetch your regular"}, {"start": 2389.08, "end": 2393.2, "text": " resident 50 not the one with dilated convolution and if you're wondering this"}, {"start": 2393.2, "end": 2398.3599999999997, "text": " is what dilated convolution is all about basically here is a nice visualization"}, {"start": 2398.36, "end": 2403.2000000000003, "text": " of how it works so it has a similar effect when it comes to down sampling"}, {"start": 2403.2000000000003, "end": 2408.4, "text": " the spatial extent so it's instead of using strides to down sample the output"}, {"start": 2408.4, "end": 2414.84, "text": " map you just do this type of you create this type of a kernel where you only"}, {"start": 2414.84, "end": 2419.04, "text": " where you basically well it's basically dilated and you're attending only"}, {"start": 2419.04, "end": 2424.56, "text": " certain pixels of the image here and that's the way you you you manage to"}, {"start": 2424.56, "end": 2428.64, "text": " down sample the output map okay but that's not that important just like a"}, {"start": 2428.64, "end": 2434.68, "text": " fun fact okay finally they're using these frozen batch norms again just in"}, {"start": 2434.68, "end": 2439.4, "text": " other detail not that crucial it's just a common thing in these detection"}, {"start": 2439.4, "end": 2447.7999999999997, "text": " pipelines okay so number of channel is gonna be 2048 and finally we construct"}, {"start": 2447.7999999999997, "end": 2453.72, "text": " the backbone base some more details here let's kind of scheme through this they're"}, {"start": 2453.72, "end": 2458.16, "text": " gonna freeze some of the parameters basically the only parameters that are"}, {"start": 2458.16, "end": 2463.7599999999998, "text": " going to be trainable are these layer 2 layer 3 and layer 4 blocks let me kind"}, {"start": 2463.7599999999998, "end": 2469.2, "text": " of scheme over that click f5 to jump over all of that and finally we are"}, {"start": 2469.2, "end": 2474.12, "text": " gonna care we're just going to extract the features from this layer 4 and we"}, {"start": 2474.12, "end": 2478.9199999999996, "text": " are going to take those features and basically put them as a value with the"}, {"start": 2478.9199999999996, "end": 2483.2, "text": " associated key that's gonna be this zero key okay so that's the reason we"}, {"start": 2483.2, "end": 2488.6, "text": " during the the past through the Jupiter we had to have that weird zero key and"}, {"start": 2488.6, "end": 2492.8799999999997, "text": " then the tensors will see that in a second but yeah this is the detail why"}, {"start": 2492.8799999999997, "end": 2496.96, "text": " that had to happen okay so finally we just take the"}, {"start": 2496.96, "end": 2502.2799999999997, "text": " resonance 50 and we call this intermediate layer gatherer which is"}, {"start": 2502.2799999999997, "end": 2507.72, "text": " just gonna make sure that we only fetch layer 4 representations from this"}, {"start": 2507.72, "end": 2513.3599999999997, "text": " resonance 50 backbone that's that's that's it nothing fancy there joiner"}, {"start": 2513.3599999999997, "end": 2517.6, "text": " nothing fancy there as well we'll see the interesting part actually lies not"}, {"start": 2517.6, "end": 2521.68, "text": " in the neat function of the joiner but in the for function again most of the"}, {"start": 2521.68, "end": 2525.08, "text": " things are gonna happen in the actual for a prop but this is just giving us"}, {"start": 2525.08, "end": 2529.72, "text": " some idea of how the model is implemented so let's continue here next"}, {"start": 2529.72, "end": 2534.12, "text": " up we build the transformer your standard transformer not nothing"}, {"start": 2534.12, "end": 2537.2, "text": " interesting in the in the neat function so we're just gonna focus on the"}, {"start": 2537.2, "end": 2542.0, "text": " forward prop a bit later now this is interesting that the better part is"}, {"start": 2542.0, "end": 2548.48, "text": " gonna be probably worth kind of skimming through the neat function a number of"}, {"start": 2548.48, "end": 2551.64, "text": " queries is gonna be hundred as previously mentioned we just stored a"}, {"start": 2551.64, "end": 2557.12, "text": " transformer here we store the model dimension which is 256 the internal"}, {"start": 2557.12, "end": 2562.7999999999997, "text": " transformer dimension we create these class in bed and d-box in bed if you"}, {"start": 2562.8, "end": 2568.92, "text": " recall those are gonna map the output decoder output embedding vectors they're"}, {"start": 2568.92, "end": 2573.5600000000004, "text": " gonna map those into the distribution of classes and the four in the bounding"}, {"start": 2573.5600000000004, "end": 2578.2000000000003, "text": " boxes okay so you can see here we map from 256 which is the side"}, {"start": 2578.2000000000003, "end": 2584.04, "text": " dimensionality of the output embedding vector we map that into 92 because we"}, {"start": 2584.04, "end": 2589.0, "text": " have 91 classes in cocoa plus one for the no object for the special class okay"}, {"start": 2589.0, "end": 2594.08, "text": " for the b-box so for the body box we'll have MLP instead instead of a simple"}, {"start": 2594.08, "end": 2600.2, "text": " linear layer so this is kind of the API here is kind of convoluted but what this"}, {"start": 2600.2, "end": 2604.8, "text": " says is hey I want the output of this MLP to be four I want to have three"}, {"start": 2604.8, "end": 2608.78, "text": " layers this is these are the hidden dimensions of the layer and this is the"}, {"start": 2608.78, "end": 2613.08, "text": " input dimension so basically this is some type of an MLP that outputs number"}, {"start": 2613.08, "end": 2618.48, "text": " four dimensional vectors at the end okay we form the query embeddings just"}, {"start": 2618.48, "end": 2624.72, "text": " embedding table so hundred of these and dimensionality is again 256 finally this"}, {"start": 2624.72, "end": 2632.18, "text": " is the input projection so that's the one that maps from 2048 into 256 and we"}, {"start": 2632.18, "end": 2636.16, "text": " just stored the backbone and this auxiliary losses is set to true we'll"}, {"start": 2636.16, "end": 2640.16, "text": " see what it is basically what it does is the following let me let me go through"}, {"start": 2640.16, "end": 2647.98, "text": " the notebook again it's going to make sure that we fetch the outputs of every"}, {"start": 2647.98, "end": 2653.8, "text": " single out so so not just for the last layer of the decoder but for all of the"}, {"start": 2653.8, "end": 2659.6, "text": " six decoder transformer layers so that's they showed in the paper that that"}, {"start": 2659.6, "end": 2663.58, "text": " helps improve the performance by having intermediate losses and not just the"}, {"start": 2663.58, "end": 2668.72, "text": " loss at the end of the decoder stack that's the idea there okay so let's"}, {"start": 2668.72, "end": 2673.4, "text": " continue here again ignoring this part because this is just relevant for the"}, {"start": 2673.4, "end": 2677.28, "text": " panoptic segmentation pathway we don't care about that one and this is gonna"}, {"start": 2677.28, "end": 2681.52, "text": " construct the Hungarian matter so that's the second important part of the of the"}, {"start": 2681.52, "end": 2684.92, "text": " of the pipeline but again the whole magic happens in the forward pass so"}, {"start": 2684.92, "end": 2690.48, "text": " we're just gonna ignore all of this for now weighted dictionary is just gonna"}, {"start": 2690.48, "end": 2695.76, "text": " obviously define the weights for the associated loss so this loss is the"}, {"start": 2695.76, "end": 2699.36, "text": " label loss this is gonna be the bounding box loss etc etc so this one is"}, {"start": 2699.36, "end": 2704.36, "text": " gonna be five this is one again these are just some hyper parameters that they"}, {"start": 2704.36, "end": 2709.6600000000003, "text": " figured out during the training okay we skip again this part because that's the"}, {"start": 2709.6600000000003, "end": 2715.6400000000003, "text": " panoptic segmentation and because auxiliary loss is set to true we are"}, {"start": 2715.6400000000003, "end": 2720.1600000000003, "text": " also gonna have to define weights for all of the intermediate losses as well"}, {"start": 2720.1600000000003, "end": 2724.48, "text": " so that's what this is gonna do I'm just gonna skip all over this and let me let"}, {"start": 2724.48, "end": 2729.4, "text": " me print you what our weighted deck is gonna look like at the end so here it is"}, {"start": 2729.4, "end": 2734.56, "text": " if I kind of pass it here you can see that we have various different weights"}, {"start": 2734.56, "end": 2738.7200000000003, "text": " for a bunch of different losses that we'll have so let's see how many losses"}, {"start": 2738.7200000000003, "end": 2743.32, "text": " we have so like 18 losses that's that's a lot like that's three times six"}, {"start": 2743.32, "end": 2746.84, "text": " because we have six layers in the decoder and we have three light layers"}, {"start": 2746.84, "end": 2752.92, "text": " in each of those three losses in each of those okay so again we defined here the"}, {"start": 2752.92, "end": 2756.7000000000003, "text": " types of losses we care about this cardinality is not actually a loss we'll"}, {"start": 2756.7, "end": 2764.52, "text": " see what it is a bit later and yeah let's keep over this and finally this is"}, {"start": 2764.52, "end": 2770.96, "text": " the the final important layer during the for prop of this object here we're gonna"}, {"start": 2770.96, "end": 2774.12, "text": " do the matching plus the loss calculation so that's the third"}, {"start": 2774.12, "end": 2778.8799999999997, "text": " component of our data model that we are gonna care about okay let's continue"}, {"start": 2778.8799999999997, "end": 2786.54, "text": " here so here is how the in the function looks like we store the Hungarian"}, {"start": 2786.54, "end": 2792.12, "text": " metric here we stored a bunch of different objects nothing super vital"}, {"start": 2792.12, "end": 2797.96, "text": " there so one interesting detail is that we have this empty weight matrix vector"}, {"start": 2797.96, "end": 2802.36, "text": " and it has so as you can see the dimensionality of it it's gonna be I"}, {"start": 2802.36, "end": 2808.24, "text": " think it should be 92 so if I were to print this 92 so that's the number of"}, {"start": 2808.24, "end": 2814.2, "text": " classes plus the no object class and the reason we set the last so so all of the"}, {"start": 2814.2, "end": 2818.3599999999997, "text": " weights are gonna be one except that for the no object class so you can see here"}, {"start": 2818.3599999999997, "end": 2825.2, "text": " that's why we specify it so for the 90 so for the last element of this array"}, {"start": 2825.2, "end": 2830.16, "text": " we're gonna set the coefficient to 0.1 and that's going to help us put our"}, {"start": 2830.16, "end": 2836.3199999999997, "text": " smaller weight on this class during the label loss and the reason we have to do"}, {"start": 2836.3199999999997, "end": 2841.2, "text": " that is because for the for the most part we'll have only a couple of box"}, {"start": 2841.2, "end": 2846.3999999999996, "text": " predictions and all of the other ones are gonna be the no object prediction"}, {"start": 2846.3999999999996, "end": 2849.6, "text": " and remember we have hundreds of predictions so that's why we want to"}, {"start": 2849.6, "end": 2855.16, "text": " kind of reduce the weight for these no object predictions and that's just a"}, {"start": 2855.16, "end": 2863.3999999999996, "text": " common thing people do for to to kind of counter this class imbalance basically"}, {"start": 2863.3999999999996, "end": 2869.8799999999997, "text": " okay we registered the buffer meaning we this is not a trainable vector it's a"}, {"start": 2869.88, "end": 2874.58, "text": " constant but it's still a part of the model so we want to store it in the"}, {"start": 2874.58, "end": 2879.08, "text": " model so that's why we do register buffer again pie torch syntax"}, {"start": 2879.08, "end": 2883.8, "text": " continuing on we are almost done with the data model so we are almost done"}, {"start": 2883.8, "end": 2892.2400000000002, "text": " building the the whole thing so we just placed that this this object onto GPU and"}, {"start": 2892.2400000000002, "end": 2896.2400000000002, "text": " then we store some post-processing we're gonna see what those are in the forward"}, {"start": 2896.24, "end": 2900.6, "text": " pass okay finally we return back the model so that's the better criterion"}, {"start": 2900.6, "end": 2905.3199999999997, "text": " that contains the Hungarian matter and the loss calculation and we turn back"}, {"start": 2905.3199999999997, "end": 2908.9199999999996, "text": " this these post processors which are gonna be more relevant during the"}, {"start": 2908.9199999999996, "end": 2913.9199999999996, "text": " evaluation than during training but yeah bear with me for now okay so again we"}, {"start": 2913.9199999999996, "end": 2920.9199999999996, "text": " move the model weights on to the GPU and here yeah just some variable like"}, {"start": 2920.9199999999996, "end": 2925.8399999999997, "text": " renaming we don't do distributed so I can skip this here we calculate number"}, {"start": 2925.84, "end": 2930.32, "text": " of parameters basically we trade through all the parameters and if they require"}, {"start": 2930.32, "end": 2936.0, "text": " grad then we just find the number of elements in that particular layer and we"}, {"start": 2936.0, "end": 2940.4, "text": " sum up all of those numbers and we get that number is equal to forty something"}, {"start": 2940.4, "end": 2943.92, "text": " million I think so let's find where the output is so we can see here we have"}, {"start": 2943.92, "end": 2952.2400000000002, "text": " 41 million trainable parameters for the data model okay what this piece of code"}, {"start": 2952.24, "end": 2958.56, "text": " here does is separates the trainable weights into two groups one group is the"}, {"start": 2958.56, "end": 2963.2799999999997, "text": " backbone group the other one is the non backbone a group of parameters so we'll"}, {"start": 2963.2799999999997, "end": 2966.64, "text": " have a different learning rate for for the backbone and a different learning"}, {"start": 2966.64, "end": 2971.08, "text": " rate for the other parameters so again this is what this logic does let's not"}, {"start": 2971.08, "end": 2977.7599999999998, "text": " lose too much time there and they're using Adam W which additionally adds the"}, {"start": 2977.76, "end": 2982.44, "text": " weight decay logic into into the optimization procedure finally this is"}, {"start": 2982.44, "end": 2987.6400000000003, "text": " just a special learning rate scheduler that's going to drop but I think 10x"}, {"start": 2987.6400000000003, "end": 2992.84, "text": " after 200 epochs so after 200 epochs it's going to drop the learning rate by"}, {"start": 2992.84, "end": 2999.0400000000004, "text": " 10x that's the idea okay now we need to build the cocoa datasets so let me"}, {"start": 2999.0400000000004, "end": 3005.44, "text": " quickly go through what's going on here so we build the cocoa we we form we are"}, {"start": 3005.44, "end": 3009.84, "text": " going to transform each of our images using various transforms such as"}, {"start": 3009.84, "end": 3014.36, "text": " normalization using the image net statistics and then you can see here"}, {"start": 3014.36, "end": 3019.52, "text": " that we'll have some random horizontal flipping and then either random"}, {"start": 3019.52, "end": 3024.68, "text": " resizing or this combination of resizing cropping etc etc so just bunch of details"}, {"start": 3024.68, "end": 3031.12, "text": " not that interesting okay so we just store those transforms as well as some"}, {"start": 3031.12, "end": 3036.24, "text": " other processing here again we'll see all of that a bit later during the"}, {"start": 3036.24, "end": 3041.2799999999997, "text": " forward pass okay this is how the data set was formed by the way I kind of"}, {"start": 3041.2799999999997, "end": 3045.7999999999997, "text": " skipped through this part but let me let me quickly show you so image folder is"}, {"start": 3045.7999999999997, "end": 3052.3199999999997, "text": " let me print you the actual paths so we can see here the actual path towards the"}, {"start": 3052.3199999999997, "end": 3057.16, "text": " image directory and then we have the annotation file and so what I've done"}, {"start": 3057.16, "end": 3063.44, "text": " here is the following thing so I kind of took I found this repo I'm gonna link it"}, {"start": 3063.44, "end": 3069.12, "text": " down in the video description and they they've down sampled the training set"}, {"start": 3069.12, "end": 3075.08, "text": " of the of the cocoa data set into so that it only contains 25,000 images"}, {"start": 3075.08, "end": 3081.72, "text": " instead of 117,000 or something and I've additionally literally just extracted"}, {"start": 3081.72, "end": 3087.52, "text": " hundred images and then created a valid like a valid annotation file that"}, {"start": 3087.52, "end": 3093.04, "text": " contains only hundred annotations so those are these 391 kilobytes these"}, {"start": 3093.04, "end": 3096.64, "text": " small annotation files which are gonna be valid annotation files for those"}, {"start": 3096.64, "end": 3102.7799999999997, "text": " hundred images so by doing that I have a minimal cocoa data set that helps me be"}, {"start": 3102.7799999999997, "end": 3106.52, "text": " able to run and debug this code without having to download like 20 gigabytes of"}, {"start": 3106.52, "end": 3111.92, "text": " data or something so do let me know if you want me to share the small"}, {"start": 3111.92, "end": 3117.36, "text": " modifications I've done so that I have this mini data set ready for this video"}, {"start": 3117.36, "end": 3122.4, "text": " okay so that's just a minor detail so again this data set now loaded all of"}, {"start": 3122.4, "end": 3126.88, "text": " our hundred images and the associated annotations that's the idea let's"}, {"start": 3126.88, "end": 3132.04, "text": " continue here we do the same thing for validation I'm just gonna skip to here"}, {"start": 3132.04, "end": 3136.96, "text": " I'm just gonna skip all of this let's keep all of this and we are back to the"}, {"start": 3136.96, "end": 3141.56, "text": " main function so we have the data sets we have the optimizers we have the model"}, {"start": 3141.56, "end": 3150.48, "text": " here we have everything now we need to form the data loaders so for that they"}, {"start": 3150.48, "end": 3153.52, "text": " just create some random sampling of the training data and they're gonna have"}, {"start": 3153.52, "end": 3158.36, "text": " sequential sampling of the validation data nothing too interesting then they"}, {"start": 3158.36, "end": 3162.92, "text": " create this batch sampler so we we basically batch two training images"}, {"start": 3162.92, "end": 3168.2000000000003, "text": " during during training nothing fancy there just standard pie-trot stuff"}, {"start": 3168.2000000000003, "end": 3173.7200000000003, "text": " finally we create the data loaded loaders here using the training data set"}, {"start": 3173.7200000000003, "end": 3178.52, "text": " using this batch sampler and we'll have this collate function which is going to"}, {"start": 3178.52, "end": 3183.32, "text": " be interesting and we'll see how it works a bit later again we have data"}, {"start": 3183.32, "end": 3188.08, "text": " loaders ready and that's it okay finally some cocoa specific idiosyncrasies so"}, {"start": 3188.08, "end": 3191.92, "text": " I'll just skip this as well not that vital we don't have any frozen weights"}, {"start": 3191.92, "end": 3197.04, "text": " that's again relevant only for the panoptic segmentation let's continue"}, {"start": 3197.04, "end": 3201.0, "text": " here we don't want to resume because that's when we have a checkpoint which"}, {"start": 3201.0, "end": 3205.24, "text": " we don't we don't want to do eval we just want to focus on the training okay"}, {"start": 3205.24, "end": 3210.64, "text": " here we start so we are gonna run this for 300 epochs obviously I'm not gonna"}, {"start": 3210.64, "end": 3214.4, "text": " show you that in this video I'm just gonna do a single iteration a single"}, {"start": 3214.4, "end": 3218.7200000000003, "text": " like a loop through the training and let's see how that things work so this"}, {"start": 3218.7200000000003, "end": 3223.64, "text": " is this part here is the the most important part of the whole codebase"}, {"start": 3223.64, "end": 3229.88, "text": " let's step into the training for one epoch and here we are so we set the"}, {"start": 3229.88, "end": 3235.2000000000003, "text": " model to trainable as well as the criterion again as a reminder it"}, {"start": 3235.2000000000003, "end": 3241.2000000000003, "text": " contains the Hungarian matter plus the loss calculation logic okay I'm gonna"}, {"start": 3241.2, "end": 3245.7999999999997, "text": " skip all of these logging functionalities nothing super vital there"}, {"start": 3245.7999999999997, "end": 3250.0, "text": " and finally here we start loading the images okay so this is where the fun"}, {"start": 3250.0, "end": 3258.24, "text": " starts basically let's step through this and see what happens okay so we are in"}, {"start": 3258.24, "end": 3261.96, "text": " the cocoa detection so this is the data set class and we are grabbing a single"}, {"start": 3261.96, "end": 3266.7, "text": " image so let's see what happens here so we're gonna end up with I think this"}, {"start": 3266.7, "end": 3272.68, "text": " image is gonna be a pillow yeah type so image pillow exactly and target is just"}, {"start": 3272.68, "end": 3277.3599999999997, "text": " gonna be the set addiction like a list containing all of the associated"}, {"start": 3277.3599999999997, "end": 3281.12, "text": " annotations and you can see there is much fun tations like segmentation is"}, {"start": 3281.12, "end": 3286.5, "text": " crowd bonding box blah blah blah we only care about the actual bounty box so"}, {"start": 3286.5, "end": 3292.2, "text": " that's it let's continue here less important details like image ID is gonna"}, {"start": 3292.2, "end": 3298.4399999999996, "text": " be 42 415 I think that's gonna be the file name so if we if you take a look at"}, {"start": 3298.4399999999996, "end": 3304.7599999999998, "text": " the actual training images here I think it just grabbed yeah four two four one"}, {"start": 3304.7599999999998, "end": 3308.7599999999998, "text": " five it just grabbed one of these images from the training data set took its ID"}, {"start": 3308.7599999999998, "end": 3314.52, "text": " and that's what it's using as this image ID here nothing too interesting there"}, {"start": 3314.52, "end": 3322.0, "text": " let's continue so prepare just kind of processes the target this this set of"}, {"start": 3322.0, "end": 3330.04, "text": " annotations it's a bit cleaner nothing nothing fancy there let's continue here"}, {"start": 3330.68, "end": 3335.68, "text": " transform is gonna apply the set of horizontal flipping etc etc to the image"}, {"start": 3335.68, "end": 3341.04, "text": " so that's it so again we just fetch the image we fetch the notations we process"}, {"start": 3341.04, "end": 3345.56, "text": " the image we process the annotations and now let's continue let's see what"}, {"start": 3345.56, "end": 3350.16, "text": " happens now so we do that again the reason is remember we have batch size"}, {"start": 3350.16, "end": 3355.56, "text": " set to two so that's why we're gonna call get item of the data set twice and"}, {"start": 3355.56, "end": 3361.44, "text": " then we're gonna collate those to form the final batch okay so I'm gonna skip"}, {"start": 3361.44, "end": 3364.92, "text": " through all of this because we saw how it works already and now we have the"}, {"start": 3364.92, "end": 3370.2, "text": " call a function going on okay so let's see what's going on here so batch is"}, {"start": 3370.2, "end": 3375.92, "text": " going to be basically a list of couples containing as you can see image"}, {"start": 3375.92, "end": 3382.28, "text": " annotations image annotations and instead of doing that we want to regroup"}, {"start": 3382.28, "end": 3386.0, "text": " it so that we have a couple of images and a couple of annotations so that's"}, {"start": 3386.0, "end": 3390.2400000000002, "text": " what this align does here we unpack the batch we do the zipping we do the"}, {"start": 3390.2400000000002, "end": 3394.84, "text": " listing and then we end up with batch that's gonna be as you can see here"}, {"start": 3394.84, "end": 3399.7400000000002, "text": " image image and then annotations annotations that's that's all this is an"}, {"start": 3399.7400000000002, "end": 3405.0, "text": " interesting part and an important part because images in the case of object"}, {"start": 3405.0, "end": 3409.32, "text": " detection are gonna be oftentimes of different shapes and we don't want to"}, {"start": 3409.32, "end": 3414.64, "text": " resize them into the same dimension we want to operate on the original images"}, {"start": 3414.64, "end": 3418.72, "text": " so let's quickly step into this nasty tensor from tensor list function and see"}, {"start": 3418.72, "end": 3423.48, "text": " what it does so here we index with zero which means we're gonna fetch the top"}, {"start": 3423.48, "end": 3429.4, "text": " all that contains the tensors of images so let's do f10 here and we end up in"}, {"start": 3429.4, "end": 3436.36, "text": " that function okay so again the images are obviously gonna be of dimensionality"}, {"start": 3436.36, "end": 3441.28, "text": " they're gonna have three dimensions if I'm going to let me just plot this let"}, {"start": 3441.28, "end": 3447.2000000000003, "text": " me just print this oh my god so we have three seven sixty eight eleven fifty one"}, {"start": 3447.2000000000003, "end": 3450.4, "text": " so that's the number of channels and this is a spatial extent of the image"}, {"start": 3450.4, "end": 3458.44, "text": " okay let's continue here and basically here what we do is we iterate through"}, {"start": 3458.44, "end": 3464.08, "text": " all of the images in the batch and we find the maximum basically well let's"}, {"start": 3464.08, "end": 3471.64, "text": " call it the like a convex hull or a bounding box so we find the biggest box"}, {"start": 3471.64, "end": 3475.78, "text": " that encompasses all of the spatial resolutions of all of the images in our"}, {"start": 3475.78, "end": 3479.52, "text": " data set if that makes sense so let me show you what I mean by that so tensor"}, {"start": 3479.52, "end": 3484.9, "text": " list we saw we just saw how it looks how it looks like for the zeroth image"}, {"start": 3484.9, "end": 3489.08, "text": " let's see the shape for the second one and you can see that this dimension is"}, {"start": 3489.08, "end": 3492.48, "text": " bigger than this one and this dimension is bigger than this one and that's what"}, {"start": 3492.48, "end": 3498.48, "text": " why our result in resultant max size is gonna be the same shape as this tensor"}, {"start": 3498.48, "end": 3504.84, "text": " here so if I am to step over this and if I were to print the max size we'll see"}, {"start": 3504.84, "end": 3511.36, "text": " we get as I said the biggest possible bounding box that encompasses all of"}, {"start": 3511.36, "end": 3515.36, "text": " these spatial extents so it has to be bigger than we basically do a max"}, {"start": 3515.36, "end": 3518.88, "text": " across all of the images in the batch across this dimension and across this"}, {"start": 3518.88, "end": 3524.04, "text": " dimension and that's how we get this final max size okay so finally we we"}, {"start": 3524.04, "end": 3529.28, "text": " form the the the batch shape into which we're going to now copy paste all of"}, {"start": 3529.28, "end": 3536.44, "text": " these individual images so we end up with a batch shape of two and then the"}, {"start": 3536.44, "end": 3542.64, "text": " same shape as this one here then we just extract individual so the batch size the"}, {"start": 3542.64, "end": 3546.68, "text": " channel size the height and the width so let's continue here we just grab the"}, {"start": 3546.68, "end": 3550.68, "text": " types and the devices nothing fancy there and we create that this is gonna"}, {"start": 3550.68, "end": 3557.56, "text": " be the like the container tensor and we just initialize it with zeros so it's"}, {"start": 3557.56, "end": 3561.92, "text": " going to have this patch shape as we just saw here next up we're gonna form a"}, {"start": 3561.92, "end": 3568.46, "text": " mask that's gonna tell us later on which tokens we need to ignore when we pass"}, {"start": 3568.46, "end": 3574.2000000000003, "text": " them through the transformer because some of these you saw that the second"}, {"start": 3574.2000000000003, "end": 3578.6, "text": " image is smaller one and so some of the the tokens will have to be ignored from"}, {"start": 3578.6, "end": 3582.48, "text": " that smaller image that's why we have a mask and we initialize it initially with"}, {"start": 3582.48, "end": 3587.2400000000002, "text": " all ones let's see now what happens so we iterate through the tens or lists so"}, {"start": 3587.24, "end": 3592.08, "text": " that's a list containing our images and we trade through the basically this this"}, {"start": 3592.08, "end": 3596.4799999999996, "text": " place this this container tensor and we're just gonna like we're just gonna"}, {"start": 3596.4799999999996, "end": 3602.68, "text": " copy our image into the into that container tensor and then we just take"}, {"start": 3602.68, "end": 3610.68, "text": " the mask and we fill in the false such that the parts that remain true are the"}, {"start": 3610.68, "end": 3614.3599999999997, "text": " part are the parts that need to be padded for that image if that makes sense"}, {"start": 3614.36, "end": 3620.44, "text": " so for this first one I assume that everything is gonna be false because the"}, {"start": 3620.44, "end": 3625.48, "text": " bigger image has the same shape as the as the container so that's why if I were"}, {"start": 3625.48, "end": 3629.96, "text": " to print the mask you'll see that all of these are false and that makes sense"}, {"start": 3629.96, "end": 3635.7200000000003, "text": " because again the bigger image is of the same shape as the container object so"}, {"start": 3635.7200000000003, "end": 3641.08, "text": " let's now step through it again so now this time mask is gonna be a bit"}, {"start": 3641.08, "end": 3645.48, "text": " different so you can see here that we now have true here because we do need"}, {"start": 3645.48, "end": 3649.2799999999997, "text": " padding for the second image because it is smaller than the container tensor"}, {"start": 3649.2799999999997, "end": 3655.7599999999998, "text": " that's the idea and that's it like after this you can see that we've created"}, {"start": 3655.7599999999998, "end": 3662.24, "text": " successfully created like this container tensor we've stored our image data"}, {"start": 3662.24, "end": 3666.84, "text": " inside and we have the mask that also knows which tokens we need to ignore"}, {"start": 3666.84, "end": 3672.92, "text": " later on during during the the forward pass okay let's see what this nested"}, {"start": 3672.92, "end": 3677.96, "text": " tensor is basically we just store it's a simple simple object that source that"}, {"start": 3677.96, "end": 3683.88, "text": " tensors and the masks nothing fancy there okay so that was this part that's"}, {"start": 3683.88, "end": 3688.6400000000003, "text": " that was the what this collate function does and now we can return back to the"}, {"start": 3688.6400000000003, "end": 3695.96, "text": " main training loop okay so we can just skip all of this let me just go back to"}, {"start": 3695.96, "end": 3703.04, "text": " the file here so we were here we just need to skip all the way through here so"}, {"start": 3703.04, "end": 3711.12, "text": " f5 nice so samples just contains the nested tensors we just saw so we just"}, {"start": 3711.12, "end": 3715.96, "text": " created those so let me just show you that that's indeed the case if I were"}, {"start": 3715.96, "end": 3721.76, "text": " to print type of samples we're gonna have a nested tensor and if I were to"}, {"start": 3721.76, "end": 3727.44, "text": " reach for the tensors here and then do the shape we'll see that that contains"}, {"start": 3727.44, "end": 3733.92, "text": " our images and if I were to do instead of tensors I think a mask yeah mask and"}, {"start": 3733.92, "end": 3739.0800000000004, "text": " then shape here we can see that we have a mask stored so mask obviously doesn't"}, {"start": 3739.0800000000004, "end": 3743.44, "text": " need to have the number of channels because we just carry care about masking"}, {"start": 3743.44, "end": 3749.88, "text": " out certain tokens from the from the image space that's it okay so these two"}, {"start": 3749.88, "end": 3755.96, "text": " lines are just gonna push the labels and the images to the GPU in case you have"}, {"start": 3755.96, "end": 3762.2400000000002, "text": " one and now comes the fun part so this is literally the most important part of"}, {"start": 3762.2400000000002, "end": 3767.76, "text": " the code base so inside of the train one epoch these two lines are the main that"}, {"start": 3767.76, "end": 3772.6, "text": " the gist of what's going on so we have we do the the forward pass here so we"}, {"start": 3772.6, "end": 3776.36, "text": " get the bounding box predictions and then the criterion is gonna do the"}, {"start": 3776.36, "end": 3780.48, "text": " Hungarian matching followed by loss calculation so let's see how that's"}, {"start": 3780.48, "end": 3787.08, "text": " gonna work okay so we are in the data for pass here as you can see here so"}, {"start": 3787.08, "end": 3793.48, "text": " because samples is already a nested list we're just gonna skip this and the first"}, {"start": 3793.48, "end": 3798.8, "text": " step is to take the images so so the samples and pass them through the"}, {"start": 3798.8, "end": 3803.6, "text": " backbone so let's see how that's gonna work okay so here is the backbone first"}, {"start": 3803.6, "end": 3807.56, "text": " things first thing that this backbone is gonna do again remember there is this"}, {"start": 3807.56, "end": 3812.92, "text": " joiner object it's nothing fancy so this self zero let me show you what it is so"}, {"start": 3812.92, "end": 3817.46, "text": " self zero is simply your backbone so that's your resonant 50 so if I were to"}, {"start": 3817.46, "end": 3821.4, "text": " print the type of this you can see it's a backbone you later see this self one"}, {"start": 3821.4, "end": 3825.88, "text": " this self one is just gonna do the positionally encoding so if we were to"}, {"start": 3825.88, "end": 3830.36, "text": " if I were to replace this zero by one we can see that this is just position"}, {"start": 3830.36, "end": 3835.48, "text": " embeddings sign so the sinusoid from the like the same type of positionally"}, {"start": 3835.48, "end": 3839.76, "text": " embeddings as from the original transformer paper okay so that means"}, {"start": 3839.76, "end": 3845.6400000000003, "text": " that by doing this we are passing the nested I think this is the nested tensor"}, {"start": 3845.6400000000003, "end": 3851.6, "text": " let me kind of double check this again so let's just take the tensor list yeah"}, {"start": 3851.6, "end": 3855.84, "text": " it says here the notation says it already it's a nested tensor we passed"}, {"start": 3855.84, "end": 3859.48, "text": " it through the backbone so let's see the forward pass of the backbone okay here"}, {"start": 3859.48, "end": 3864.92, "text": " it is the backbone base we basically pass the so we extract the tensions"}, {"start": 3864.92, "end": 3868.84, "text": " which are the images and we just pass that through the body which is gonna"}, {"start": 3868.84, "end": 3873.16, "text": " fetch us the as you recall here it's just gonna fetch us the features from"}, {"start": 3873.16, "end": 3880.2, "text": " the layer 4 of the resonant 50 so let's do that and we end up with XS XS is"}, {"start": 3880.2, "end": 3885.04, "text": " just gonna have I think one key and that's this key zero and and we we've"}, {"start": 3885.04, "end": 3889.56, "text": " specified it here exactly so that this tells us hey if you want access layer"}, {"start": 3889.56, "end": 3894.32, "text": " four features access me via key zero and that's what we are going to do so if I"}, {"start": 3894.32, "end": 3899.12, "text": " if I were to take excess of zero we're gonna access the features and the"}, {"start": 3899.12, "end": 3902.92, "text": " features are gonna be as you can see of the following dimension so we have 2048"}, {"start": 3902.92, "end": 3906.64, "text": " channels and we can see that the number of the spatial extent kind of reduced"}, {"start": 3906.64, "end": 3911.6, "text": " here so it started with we started with this shape let me show you what we had"}, {"start": 3911.6, "end": 3919.04, "text": " here so we started with two three and the original image resolution and we"}, {"start": 3919.04, "end": 3925.48, "text": " got two more channels and smaller spatial resolution which is expected from"}, {"start": 3925.48, "end": 3931.24, "text": " a convolutional neural network such as resonant 50 okay next up what we do is"}, {"start": 3931.24, "end": 3938.36, "text": " the following we grab the features from access and we take the mask and now see"}, {"start": 3938.36, "end": 3944.44, "text": " what we do so we basically now down sample the mask such that this mask is"}, {"start": 3944.44, "end": 3950.76, "text": " no longer of the same spatial resolution as the original image but it's of the"}, {"start": 3950.76, "end": 3955.04, "text": " same spatial resolution as the as the image features that came out of resonant"}, {"start": 3955.04, "end": 3959.1200000000003, "text": " 50 because remember we want to know which of those image features we want to"}, {"start": 3959.1200000000003, "end": 3963.56, "text": " mask out because we are gonna later on pass them through the transformer so"}, {"start": 3963.56, "end": 3968.64, "text": " that's why we do this step so basically you can see that mask currently has the"}, {"start": 3968.64, "end": 3976.7999999999997, "text": " following shape whoops let me see why what it so it's called them M shape so"}, {"start": 3976.7999999999997, "end": 3982.08, "text": " this is the original shape after we do the down sampling we end up with mask"}, {"start": 3982.08, "end": 3990.24, "text": " shape so to 1929 which is of the same resolution as the image features that"}, {"start": 3990.24, "end": 3995.68, "text": " came out of resonant 50 so let me now print mask and you can see again we have"}, {"start": 3995.68, "end": 4000.7599999999998, "text": " all false for the first image and we have some of these being true for the"}, {"start": 4000.7599999999998, "end": 4006.8399999999997, "text": " second image and that's it okay finally we store the image features and the"}, {"start": 4006.8399999999997, "end": 4013.2, "text": " associated mask into a nested vector and we return that back basically in this"}, {"start": 4013.2, "end": 4020.04, "text": " out dictionary that's it that's the first step of the of the backbone so now"}, {"start": 4020.04, "end": 4025.8399999999997, "text": " we just want to calculate the positional embeddings let's see how we do that"}, {"start": 4025.8399999999997, "end": 4031.72, "text": " again we grab the image features and the down sample mask here we just store that"}, {"start": 4031.72, "end": 4037.6, "text": " into this out dictionary so again let me remind you that X is just if I were to"}, {"start": 4037.6, "end": 4042.64, "text": " access tensors and bring the shape that contains the image features and if I"}, {"start": 4042.64, "end": 4047.04, "text": " were to ask excess mask that contains the down sample masks that's the X"}, {"start": 4047.04, "end": 4055.52, "text": " object here and we store it into this out list here we do just a simple"}, {"start": 4055.52, "end": 4061.7599999999998, "text": " positional encoding calculation nothing too fancy basically we give it the X and"}, {"start": 4061.7599999999998, "end": 4069.4, "text": " it's gonna do its own magic so a bunch of code but that's your just your like"}, {"start": 4069.4, "end": 4075.28, "text": " signs and cosines calculation so that we ultimately end up with this positional"}, {"start": 4075.28, "end": 4082.2400000000002, "text": " embedding like set of vectors of the shape as you can see the following so he"}, {"start": 4082.2400000000002, "end": 4087.52, "text": " has two because we have remember we have batch dimension of two we have the same"}, {"start": 4087.52, "end": 4091.96, "text": " resolution here as the as the as the image features that came out of the"}, {"start": 4091.96, "end": 4097.4800000000005, "text": " resonant and we have 256 because that's going to be the hidden dimension that"}, {"start": 4097.48, "end": 4101.919999999999, "text": " the transformers are gonna be using that's why we have all of this okay so I"}, {"start": 4101.919999999999, "end": 4105.44, "text": " encourage you to check out this this code if you're interested but I'm just"}, {"start": 4105.44, "end": 4110.719999999999, "text": " gonna treat it as a black box for now it's nothing to fancy there okay we"}, {"start": 4110.719999999999, "end": 4117.08, "text": " store that in this positional list and we return that back so that's the first"}, {"start": 4117.08, "end": 4124.759999999999, "text": " step of the data for pass so data model again for pass and we return back the"}, {"start": 4124.76, "end": 4131.360000000001, "text": " image features the positional vectors and the mask next up we just decompose"}, {"start": 4131.360000000001, "end": 4136.72, "text": " the nested tensor into the source and masks the source again image features and"}, {"start": 4136.72, "end": 4141.2, "text": " mask the down sample mask nothing fancy there now this is the interesting part"}, {"start": 4141.2, "end": 4148.320000000001, "text": " so we take the source which is again recall this is going to have two thousand"}, {"start": 4148.32, "end": 4157.04, "text": " twenty four sorry wait 2048 and then the down sampled image resolution so again"}, {"start": 4157.04, "end": 4162.48, "text": " source shape here it is and after we apply the input projection that's going"}, {"start": 4162.48, "end": 4168.4, "text": " to reduce this number of channels as I previously explained to 256 so that's"}, {"start": 4168.4, "end": 4173.0, "text": " going to happen there and then we just passed the mask so this is what we pass"}, {"start": 4173.0, "end": 4179.36, "text": " to the encoder and then we also pass the query vectors to the decoder and we pass"}, {"start": 4179.36, "end": 4185.56, "text": " the positional encodings so let's step through this piece of code let's do f10"}, {"start": 4185.56, "end": 4191.0, "text": " here and we are hitting the transformer so this is a transformer layer this is"}, {"start": 4191.0, "end": 4197.7, "text": " the for pass and let me now show you the source that it indeed is now has 256 when"}, {"start": 4197.7, "end": 4201.64, "text": " it comes to a number of channels so you can see here to 256 so that's the"}, {"start": 4201.64, "end": 4207.04, "text": " consequence of applying that convolutional layer before passing it"}, {"start": 4207.04, "end": 4213.52, "text": " into the transformer so nothing fancy there now we're going to do some"}, {"start": 4213.52, "end": 4220.68, "text": " flattening and permutation so that we have a shape that's suitable for passing"}, {"start": 4220.68, "end": 4224.08, "text": " it into the transformer so that's something I mentioned here previously"}, {"start": 4224.08, "end": 4230.08, "text": " basically you take this and in the rest order you're gonna flatten out this this"}, {"start": 4230.08, "end": 4237.44, "text": " volume into a sequence of tokens that's that's the basic idea and so again let"}, {"start": 4237.44, "end": 4243.48, "text": " me show you the shape the shape is going to be 551 to 256 so this is the batch"}, {"start": 4243.48, "end": 4248.32, "text": " size this is the hidden dimension and this is the number of tokens and it's"}, {"start": 4248.32, "end": 4255.2, "text": " kind of weird permutation but that's what the this original pie torch"}, {"start": 4255.2, "end": 4259.5599999999995, "text": " implementation of the transformer expects okay instead of having batch as"}, {"start": 4259.56, "end": 4265.240000000001, "text": " the first dimensionality and the reason is some optimization tricks we do the"}, {"start": 4265.240000000001, "end": 4269.64, "text": " same thing for the positional embeddings so if I were to show you the positional"}, {"start": 4269.64, "end": 4275.52, "text": " embedding shape here we're gonna end up with literally the same shape as as the"}, {"start": 4275.52, "end": 4280.4800000000005, "text": " source because we want to add them up later on and finally we do a similar"}, {"start": 4280.4800000000005, "end": 4286.04, "text": " thing with query embedding basically they are initially just going to be 100"}, {"start": 4286.04, "end": 4293.0, "text": " 256 I think so hundred two fifty six and here we just add dimension like at the"}, {"start": 4293.0, "end": 4297.84, "text": " location one and then we repeat it whatever the number of patches is that's"}, {"start": 4297.84, "end": 4302.68, "text": " how many times we just copy paste these very actors and then we end up with"}, {"start": 4302.68, "end": 4310.04, "text": " basically the same so hundred to 256 okay we do the flattening of the mask"}, {"start": 4310.04, "end": 4314.6, "text": " that we can so we can pass it into interest former and finally we form this"}, {"start": 4314.6, "end": 4318.280000000001, "text": " target so targets is going to be so we have a source we have the targets"}, {"start": 4318.280000000001, "end": 4322.68, "text": " targets are going to be the tokens that we pass into the decoder part of the"}, {"start": 4322.68, "end": 4327.200000000001, "text": " transformer layer whereas we're gonna just be using these query vectors as the"}, {"start": 4327.200000000001, "end": 4331.400000000001, "text": " positionally coatings that we're gonna sum up with the target tokens we're"}, {"start": 4331.400000000001, "end": 4337.4400000000005, "text": " gonna see that in a second okay so finally we now have encoding part and"}, {"start": 4337.4400000000005, "end": 4344.400000000001, "text": " decoding part let's jump and see how those are gonna work let me just see"}, {"start": 4344.4, "end": 4348.96, "text": " whether I have a breakpoint here I'm gonna add a breakpoint to forward post"}, {"start": 4348.96, "end": 4354.04, "text": " post because we're gonna have layer norm applied after the residual connection"}, {"start": 4354.04, "end": 4357.719999999999, "text": " not before that's why we have this for a priest we're gonna just be using this"}, {"start": 4357.719999999999, "end": 4363.96, "text": " version of the function here let's see let's see whether I should add in honor"}, {"start": 4363.96, "end": 4370.4, "text": " I should probably add breakpoint here as well so now let's do f10 just step over"}, {"start": 4370.4, "end": 4375.92, "text": " so as you can see here this is your encoder layer we do for like six times"}, {"start": 4375.92, "end": 4381.5199999999995, "text": " because let me show you the the length we have like six of these as you can see"}, {"start": 4381.5199999999995, "end": 4385.759999999999, "text": " here so we're gonna six times just apply the transformer layer and we pass"}, {"start": 4385.759999999999, "end": 4392.5199999999995, "text": " basically the source here so those are the image tokens just flattened out and"}, {"start": 4392.5199999999995, "end": 4397.5599999999995, "text": " we passed the mask so that's the mask that we got as the odd but if you recall"}, {"start": 4397.56, "end": 4405.200000000001, "text": " from the back bone and finally we we add the positional encodings let me now go"}, {"start": 4405.200000000001, "end": 4409.160000000001, "text": " into the actual layer so let's see what's going on here so we sum up the"}, {"start": 4409.160000000001, "end": 4413.0, "text": " positional encoding so those are the sinusoid we computed we sum them up"}, {"start": 4413.0, "end": 4418.320000000001, "text": " with the image tokens and we get the queries and the keys and then we're just"}, {"start": 4418.320000000001, "end": 4423.240000000001, "text": " going to do a self-attention as you can see here so the only difference is that"}, {"start": 4423.24, "end": 4428.32, "text": " the value vectors will not contain the positional encodings that's it and then"}, {"start": 4428.32, "end": 4433.28, "text": " we have this the part where we mask out some parts so let me show you how this"}, {"start": 4433.28, "end": 4437.0599999999995, "text": " looks like so it's mostly false but then there are some parts that are true which"}, {"start": 4437.0599999999995, "end": 4442.76, "text": " means we're going to ignore those tokens because if you recall the that part of"}, {"start": 4442.76, "end": 4446.719999999999, "text": " the image is non-existent let me quickly explain what I mean by that exactly so"}, {"start": 4446.72, "end": 4453.8, "text": " imagine so remember that we passed two images into our like debtor so we had"}, {"start": 4453.8, "end": 4462.38, "text": " image one and we had I'm gonna just draw like something like this that's of the"}, {"start": 4462.38, "end": 4467.6, "text": " same shape as this one here but instead so the second image was maybe like this"}, {"start": 4467.6, "end": 4475.04, "text": " okay so that means that we have to pad all of these tokens out we have to"}, {"start": 4475.04, "end": 4482.4, "text": " ignore all of these tokens and that information is now stored in this mask"}, {"start": 4482.4, "end": 4488.2, "text": " because it's gonna be set to true for all of these image tokens so that's kind"}, {"start": 4488.2, "end": 4491.84, "text": " of just a visualization that I have in my mind when I think about it so let's"}, {"start": 4491.84, "end": 4499.96, "text": " go back here let's step through this and that's just gonna be a single step of"}, {"start": 4499.96, "end": 4507.16, "text": " self-attention of image tokens across each other then this is just your like"}, {"start": 4507.16, "end": 4513.24, "text": " regular transform logic we have this inverse bottleneck MLP so so nothing"}, {"start": 4513.24, "end": 4518.88, "text": " nothing fancy there I'm not gonna lose any time doing that so let's step back"}, {"start": 4518.88, "end": 4526.2, "text": " here and let's continue with the decoder portion so I'm gonna click f5 and we're"}, {"start": 4526.2, "end": 4530.28, "text": " gonna get to the decoder okay so we end up with memory the memory is gonna be of"}, {"start": 4530.28, "end": 4534.24, "text": " the same dimensionality as the input source token so source tokens again"}, {"start": 4534.24, "end": 4540.36, "text": " those are the image tokens are of this shape and you can see that the memory is"}, {"start": 4540.36, "end": 4545.92, "text": " gonna be of the same shape here okay and again memory is just let me show you the"}, {"start": 4545.92, "end": 4551.679999999999, "text": " the image here so that's gonna be the tokens we got we get here so we start"}, {"start": 4551.68, "end": 4556.72, "text": " with image features here and we end up with with the tensor that has the same"}, {"start": 4556.72, "end": 4561.52, "text": " dimensionality as the input image features so that's where we are at at"}, {"start": 4561.52, "end": 4564.84, "text": " the moment so now we need to pass through the decoder part of the"}, {"start": 4564.84, "end": 4569.280000000001, "text": " transform so let me show you how that's gonna look like we passed the target"}, {"start": 4569.280000000001, "end": 4573.56, "text": " which is gonna be initially all zeros we passed the memory so that's the output"}, {"start": 4573.56, "end": 4581.280000000001, "text": " of the decoder we then pass what we passed the mask so that we know which"}, {"start": 4581.280000000001, "end": 4586.200000000001, "text": " tokens we should not attend to when doing the cross-attention positional"}, {"start": 4586.200000000001, "end": 4591.160000000001, "text": " embeddings query vectors bunch of details there basically everything that"}, {"start": 4591.160000000001, "end": 4595.52, "text": " we have in this image here is gonna be passed into the decoder so object"}, {"start": 4595.52, "end": 4602.76, "text": " varies positional encodings memory targets tokens which are not displayed"}, {"start": 4602.76, "end": 4607.2, "text": " here but all zeros initially so all that needs to be passed into the decoder"}, {"start": 4607.2, "end": 4613.8, "text": " layer so now let's go back here and let me step over the let me just kind of add"}, {"start": 4613.8, "end": 4621.6, "text": " like a breakpoint somewhere here so here okay let's now click f10 and we're gonna"}, {"start": 4621.6, "end": 4631.0, "text": " hit decoder part I'm gonna ignore this so let's step through that code so here"}, {"start": 4631.0, "end": 4636.0, "text": " we are in a particular decoder layer so here it probably works we have the query"}, {"start": 4636.0, "end": 4643.7, "text": " position positional embedding so those are these positional embedding vectors"}, {"start": 4643.7, "end": 4651.16, "text": " here so we are going to add those to the target tokens which are gonna be all"}, {"start": 4651.16, "end": 4657.16, "text": " zeros initially so we end up with query and keys being equal to those query"}, {"start": 4657.16, "end": 4661.68, "text": " embedding vectors again we have hundred of those let me print a shape here so we"}, {"start": 4661.68, "end": 4667.72, "text": " have hundred of those and dimensionality is 256 okay so then we apply as you can"}, {"start": 4667.72, "end": 4672.36, "text": " see here we just apply a self-attention so those query vectors are gonna"}, {"start": 4672.36, "end": 4678.84, "text": " basically self attend to each other and nothing nothing super fancy there the"}, {"start": 4678.84, "end": 4683.08, "text": " interesting part in this decoder our layer happens actually at the cross"}, {"start": 4683.08, "end": 4688.88, "text": " attention module here so here as you can see again we are adding the query"}, {"start": 4688.88, "end": 4696.36, "text": " position to the target tokens and we use as keys memory so that's the output of"}, {"start": 4696.36, "end": 4701.48, "text": " the encoder and we add the positional encodings on top of the memory tokens"}, {"start": 4701.48, "end": 4705.24, "text": " and we use as the value vectors the memory vectors so we're gonna somehow"}, {"start": 4705.24, "end": 4711.16, "text": " combine the outputs of the encoder to form our novel representations after"}, {"start": 4711.16, "end": 4719.4, "text": " this so target two okay and again we'll be passing the padding information so"}, {"start": 4719.4, "end": 4723.4, "text": " the masking information because we do not want to attend all of the encoder"}, {"start": 4723.4, "end": 4728.92, "text": " outputs because some of them are not non valid because some of them stem from the"}, {"start": 4728.92, "end": 4733.24, "text": " parts of the image where we just added padding so it's like gonna be black if"}, {"start": 4733.24, "end": 4739.16, "text": " you look at the image in the image space okay so that's that part and then we"}, {"start": 4739.16, "end": 4746.16, "text": " just have regular transformer logic the MLP part and that's pretty much it okay"}, {"start": 4746.16, "end": 4751.96, "text": " guys so we're just gonna repeat this six times and as you can see we are storing"}, {"start": 4751.96, "end": 4758.12, "text": " the intermediate representations for the auxiliary loss I mentioned but yeah let"}, {"start": 4758.12, "end": 4765.8, "text": " me now step over and let's get to this part here see if I were to click f5 let"}, {"start": 4765.8, "end": 4771.4400000000005, "text": " me remove the breakpoint here we are now finished decoder we got decoder output"}, {"start": 4771.4400000000005, "end": 4778.68, "text": " so let me show you the shape of the output here we have 600 to 256 so why"}, {"start": 4778.68, "end": 4783.0, "text": " does this make sense so 100 is because we have 100 query embedding vectors to"}, {"start": 4783.0, "end": 4789.56, "text": " become because we have batch dimension of 2 to 56 is the dimensionality of the"}, {"start": 4789.56, "end": 4793.360000000001, "text": " embedding vectors throughout the decoder stack and throughout the encoder stack"}, {"start": 4793.36, "end": 4797.24, "text": " and finally we have six because we have six decoder layers and we're storing or"}, {"start": 4797.24, "end": 4800.5599999999995, "text": " all of the intermediate representations because we are gonna apply this"}, {"start": 4800.5599999999995, "end": 4805.4, "text": " auxiliary auxiliary loss a bit later so that's it so this is the thing we're"}, {"start": 4805.4, "end": 4813.339999999999, "text": " gonna be using this HS after we do this transpose operation okay so finally we"}, {"start": 4813.339999999999, "end": 4819.28, "text": " applied the class embedding so that's gonna map our representation HS from"}, {"start": 4819.28, "end": 4823.96, "text": " this shape into we're gonna have instead of 256 92 because that's the number of"}, {"start": 4823.96, "end": 4829.28, "text": " classes plus the no object class so let me show you the the shape of the output"}, {"start": 4829.28, "end": 4835.5599999999995, "text": " class so this is gonna be as I told you 92 here then we do b-box embeds that's"}, {"start": 4835.5599999999995, "end": 4840.599999999999, "text": " the MLP that's gonna give us the outputs like the bounding box coordinates so"}, {"start": 4840.599999999999, "end": 4845.639999999999, "text": " let me do f10 there and let me show you the shape so here is the shape we have"}, {"start": 4845.64, "end": 4850.6, "text": " four here is the output and that's it and finally we grab only the last so"}, {"start": 4850.6, "end": 4856.76, "text": " this is gonna grab only the output from the last decoder layer and then we're"}, {"start": 4856.76, "end": 4862.0, "text": " gonna store all of the intermediate representations inside of coupled with"}, {"start": 4862.0, "end": 4866.700000000001, "text": " this this particular key here auxiliary outputs okay so we store the classes the"}, {"start": 4866.700000000001, "end": 4871.280000000001, "text": " logits we store the bounding boxes and then we store all of the other ones"}, {"start": 4871.28, "end": 4878.4, "text": " excluding the last one inside of inside of inside of this basically dictionary"}, {"start": 4878.4, "end": 4887.639999999999, "text": " so let me kind of step over that and if I were to show you the keys here these"}, {"start": 4887.639999999999, "end": 4894.44, "text": " are the keys we get but then inside of this key we have this list with all of"}, {"start": 4894.44, "end": 4899.32, "text": " the other outputs yeah hopefully yeah I know this is a bunch of details but do"}, {"start": 4899.32, "end": 4902.88, "text": " let me know if you if you if you get anything out of this video and even if"}, {"start": 4902.88, "end": 4907.759999999999, "text": " you if you watched all the way to here like congrats like this was like very I"}, {"start": 4907.759999999999, "end": 4913.5599999999995, "text": " can assume very like intensive and so I'd be super happy to hear the feedback"}, {"start": 4913.5599999999995, "end": 4919.92, "text": " of all of you guys who who stuck all the way to here so let's continue that was"}, {"start": 4919.92, "end": 4924.96, "text": " the one of the most important parts of this whole code base and now there is a"}, {"start": 4924.96, "end": 4930.0, "text": " second part which is much easier we're gonna soon finish up this video so let's"}, {"start": 4930.0, "end": 4934.68, "text": " see the the Hungarian matching algorithm and the loss calculation so let's dig"}, {"start": 4934.68, "end": 4940.6, "text": " into it so here we fetch just the outputs from the last layer of the"}, {"start": 4940.6, "end": 4947.4800000000005, "text": " decoder we ignore this auxiliary outputs and finally comes the matcher so this is"}, {"start": 4947.4800000000005, "end": 4951.78, "text": " the Hungarian matching algorithm let's see how this thing works fairly simple"}, {"start": 4951.78, "end": 4957.92, "text": " logic to be honest so let me kind of explain what's going on okay so first we"}, {"start": 4957.92, "end": 4962.32, "text": " are gonna grab the batch size and the number of various it's gonna be two and"}, {"start": 4962.32, "end": 4967.48, "text": " hundred here then we're gonna flatten out the prediction logic so again"}, {"start": 4967.48, "end": 4971.12, "text": " prediction logic so let me show you the shape here and this is how it looks like"}, {"start": 4971.12, "end": 4977.4, "text": " 292 we're gonna flat nod the first two and we're gonna apply the softmax"}, {"start": 4977.4, "end": 4983.2, "text": " instead of using unnormal as logits so let me do f10 here and so we end up with"}, {"start": 4983.2, "end": 4989.879999999999, "text": " a shape 292 as you can see here we do the same thing for the bounding boxes so"}, {"start": 4989.879999999999, "end": 4996.0, "text": " I do it here let's not do this it's gonna be 204 as you can see here so we"}, {"start": 4996.0, "end": 5000.5199999999995, "text": " just flatten across all of the batches and across all of the like tokens and"}, {"start": 5000.5199999999995, "end": 5004.799999999999, "text": " now we'll be able to compare that to ground truth so here's the ground truth"}, {"start": 5004.8, "end": 5009.16, "text": " we trade through this targets dictionary that contains the annotations the ground"}, {"start": 5009.16, "end": 5015.18, "text": " truth annotations we grab the labels and we concatenate those so we'll have we'll"}, {"start": 5015.18, "end": 5022.2, "text": " have two of these for the two of images and we end up with I think 16 I already"}, {"start": 5022.2, "end": 5026.92, "text": " went through this code so it's 16 because we have like I think two labels"}, {"start": 5026.92, "end": 5031.28, "text": " for the first image and 14 labels for the second image so that means two"}, {"start": 5031.28, "end": 5035.599999999999, "text": " bounding boxes for the first image and 14 for the second one we do the same"}, {"start": 5035.599999999999, "end": 5038.759999999999, "text": " thing here we just grab the bounding boxes the ground truth money boxes that"}, {"start": 5038.759999999999, "end": 5044.24, "text": " we end up with 16 for let me show you the shape here so 64 and now for the"}, {"start": 5044.24, "end": 5049.88, "text": " fun part so this is the fun part of this whole logic so what this thing is going"}, {"start": 5049.88, "end": 5056.2, "text": " to do is so we take these 16 ground truth indices so this tells us as you"}, {"start": 5056.2, "end": 5060.12, "text": " can see on the screen here that the first bounding blocks class is 7 and"}, {"start": 5060.12, "end": 5066.76, "text": " then the second is 1 and then 53 53 53 etc so for all of the 200 boxes that we"}, {"start": 5066.76, "end": 5072.36, "text": " predicted for the batch size of 2 we're going to take those particular 16 lodges"}, {"start": 5072.36, "end": 5077.04, "text": " for the 16 ground truth classes and extract them into this cost class"}, {"start": 5077.04, "end": 5083.8, "text": " matrix okay so we're gonna end up with what 216 instead of 292 so again we"}, {"start": 5083.8, "end": 5091.4800000000005, "text": " start with out prop being 292 and we end up with having a subset of those with"}, {"start": 5091.4800000000005, "end": 5099.28, "text": " 216 okay so we'll see why that makes sense then we do the following we"}, {"start": 5099.28, "end": 5104.400000000001, "text": " calculate the distance between all of our predicted boxes and the target"}, {"start": 5104.400000000001, "end": 5109.8, "text": " bounding boxes this is just going to do L1 distance nothing nothing fancy there"}, {"start": 5109.8, "end": 5115.64, "text": " and again we're going to end up with a matrix that's 216 and it contains"}, {"start": 5115.64, "end": 5122.04, "text": " basically the distance between each of our 200 predictions and each of the 16"}, {"start": 5122.04, "end": 5128.320000000001, "text": " ground truth labels so we'll know basically yeah how far away each of them"}, {"start": 5128.320000000001, "end": 5133.56, "text": " are like a fully connected graph okay if I click F10 let me show you and convince"}, {"start": 5133.56, "end": 5139.0, "text": " you that we have the shape 216 so quickly what this distance function does"}, {"start": 5139.0, "end": 5145.56, "text": " is let me show you so it's a L1 distance between two bounding boxes so imagine we"}, {"start": 5145.56, "end": 5150.84, "text": " have one bounding box that's like maybe here is one here's one so we have a"}, {"start": 5150.84, "end": 5157.08, "text": " bounding box here and let's imagine we have a second one here okay so okay"}, {"start": 5157.08, "end": 5161.96, "text": " that's it and now if you were to represent this bounding box as a vector"}, {"start": 5161.96, "end": 5166.84, "text": " let's use the XY XY representation so that's gonna we're gonna end up with"}, {"start": 5166.84, "end": 5171.2, "text": " having so that what's the coordinate of this X coordinate is 0 y coordinate is"}, {"start": 5171.2, "end": 5176.2, "text": " 1 then we have X coordinate is 1 y coordinate is 0 so this is going to be"}, {"start": 5176.2, "end": 5180.6, "text": " a representation for this bounding box and then we have this bounding box here"}, {"start": 5180.6, "end": 5186.32, "text": " which is going to be what X coordinate is 2 y coordinate is 1 we're taking a"}, {"start": 5186.32, "end": 5193.84, "text": " look at this point here and then we have 3 and we have 0 okay so now as you can"}, {"start": 5193.84, "end": 5200.32, "text": " imagine the how L1 is calculated is you basically subtract these two vectors and"}, {"start": 5200.32, "end": 5204.12, "text": " so you'll end up with what you'll end up with like and you take the absolute"}, {"start": 5204.12, "end": 5212.22, "text": " value so you'll have 2 0 you'll have 2 and you'll have 0 and then I think you"}, {"start": 5212.22, "end": 5216.72, "text": " just add this up and so you end up with like 4 so the distance between these two"}, {"start": 5216.72, "end": 5222.16, "text": " is gonna be 4 and so we do this for all of the predicted boxes and the ground"}, {"start": 5222.16, "end": 5227.639999999999, "text": " truth boxes that's it that's the logic we apply and then we have this and I'm"}, {"start": 5227.639999999999, "end": 5231.84, "text": " gonna ignore the implementation but we just find the same type of so the"}, {"start": 5231.84, "end": 5237.4, "text": " intersectional reunion logic I just I think I mentioned an hour ago or"}, {"start": 5237.4, "end": 5245.76, "text": " something okay so we end up with this cost matrix that's also 216 so after"}, {"start": 5245.76, "end": 5253.08, "text": " that we sum up all of those matrices and we have associated weights here 5 and we"}, {"start": 5253.08, "end": 5257.72, "text": " have 1 and we have 2 so what's the semantics what's the idea why are we"}, {"start": 5257.72, "end": 5262.16, "text": " doing this well the idea is the following we care about we're trying to"}, {"start": 5262.16, "end": 5267.76, "text": " match the predicted box with the ground truth box there is multiple angles at"}, {"start": 5267.76, "end": 5271.6, "text": " which we could look at this so we can only focus maybe on what's the closest"}, {"start": 5271.6, "end": 5276.92, "text": " one but then if they are close but the class is completely off is that a good"}, {"start": 5276.92, "end": 5280.88, "text": " match so if on the other hand we have so let me show you what I mean by that so"}, {"start": 5280.88, "end": 5286.56, "text": " imagine this is imagine this is a ground truth box here imagine we have a ground"}, {"start": 5286.56, "end": 5293.14, "text": " truth box here and imagine we have one box that's predicted like this it has"}, {"start": 5293.14, "end": 5299.200000000001, "text": " some significant overlap but the class of this one is maybe cat and the class"}, {"start": 5299.2, "end": 5303.66, "text": " of this one is maybe dog and now let's assume we have a different bounding box"}, {"start": 5303.66, "end": 5310.28, "text": " so that bounding box is a little bit less overlap but it has a correct class"}, {"start": 5310.28, "end": 5314.639999999999, "text": " it's cat so which one is a better one which better is a which one of these is"}, {"start": 5314.639999999999, "end": 5318.44, "text": " better match so there is no correct answer and that's why we have to kind of"}, {"start": 5318.44, "end": 5324.36, "text": " combine both the intersectional reunion both the distance both the class to form"}, {"start": 5324.36, "end": 5329.599999999999, "text": " the final matrix and then solve this basically graph flow problem to find the"}, {"start": 5329.599999999999, "end": 5338.32, "text": " optimal assignment so here so we finally have the cost matrix and then we just"}, {"start": 5338.32, "end": 5343.92, "text": " kind of change the change the shape of that matrix so that we have let me show"}, {"start": 5343.92, "end": 5348.88, "text": " you here so we have two hundred and sixteen now we collect sizes because we"}, {"start": 5348.88, "end": 5354.4800000000005, "text": " want to separately match the prediction bounding boxes from the first batch with"}, {"start": 5354.4800000000005, "end": 5357.68, "text": " the ground truth boxes from the first batch from the first element of the"}, {"start": 5357.68, "end": 5361.32, "text": " batch and similarly for the second one so we want to have those are completely"}, {"start": 5361.32, "end": 5365.400000000001, "text": " orthogonal so that's why we have to do this separation so size is just going to"}, {"start": 5365.400000000001, "end": 5370.0, "text": " be 2 and 14 so 2 and 14 because we have two bounding boxes for the first image"}, {"start": 5370.0, "end": 5376.36, "text": " and 14 for the second one okay and now we just do this split over sizes which"}, {"start": 5376.36, "end": 5381.44, "text": " is basically going to split this into 102 and hundred fourteen so the two"}, {"start": 5381.44, "end": 5384.759999999999, "text": " separate problems as I mentioned before so let me see whether this is gonna"}, {"start": 5384.759999999999, "end": 5393.0, "text": " work if I were to print this and allocate it to a and B let's print"}, {"start": 5393.0, "end": 5401.4, "text": " shape of a and let's print shape of B as you can see here this is what we got"}, {"start": 5401.4, "end": 5406.96, "text": " okay and after splitting you can see that we have this indexing by I it's"}, {"start": 5406.96, "end": 5410.96, "text": " gonna be zero in the first iteration and then one in the in the second one so"}, {"start": 5410.96, "end": 5418.4, "text": " that means we'll be grabbing only the zero with batch here and the first one"}, {"start": 5418.4, "end": 5425.879999999999, "text": " here and then basically doing this assignment problem on 102 cost matrix"}, {"start": 5425.88, "end": 5432.96, "text": " and hundred fourteen and it's literally our minimal flow type of an algorithm so"}, {"start": 5432.96, "end": 5437.08, "text": " this is what this linear sum assignment is gonna find it's gonna find such a"}, {"start": 5437.08, "end": 5441.56, "text": " choice of edges in the bipartite graph so as to minimize the cost and so what I"}, {"start": 5441.56, "end": 5446.04, "text": " mean by that let's for example take the hundred two hundred fourteen one so what"}, {"start": 5446.04, "end": 5452.8, "text": " is gonna happen is we have a matrix that's like let's say hundred fourteen"}, {"start": 5452.8, "end": 5459.68, "text": " okay and it tells you how so this row here tells you the cost between this"}, {"start": 5459.68, "end": 5465.84, "text": " particular predicted box and all of the fourteen ground truth boxes and so you're"}, {"start": 5465.84, "end": 5470.92, "text": " gonna have like what's the cost between each of those and so we can visualize"}, {"start": 5470.92, "end": 5474.56, "text": " this this matrix is the following way as well so you imagine you have like"}, {"start": 5474.56, "end": 5481.04, "text": " literally hundred nodes here and you only have 14 nodes here and so we have"}, {"start": 5481.04, "end": 5486.92, "text": " a fully connected graph okay something like this something like this and then"}, {"start": 5486.92, "end": 5491.16, "text": " we have it all of the edges are connected here we have a by part of a"}, {"start": 5491.16, "end": 5497.96, "text": " graph and so the the idea is to find two such edges that will minimize the the"}, {"start": 5497.96, "end": 5503.64, "text": " cost of this of this bipartite graph so hopefully that makes sense plus there is"}, {"start": 5503.64, "end": 5507.96, "text": " a constraint that you cannot pick edges that go to the same notes you have to"}, {"start": 5507.96, "end": 5512.16, "text": " cover all of the output nodes so you have to cover both this node and all of"}, {"start": 5512.16, "end": 5516.44, "text": " the other nodes and this one by doing that you literally found for each of"}, {"start": 5516.44, "end": 5521.56, "text": " the node you found some ideal match so maybe you found for this one maybe this"}, {"start": 5521.56, "end": 5526.08, "text": " one is ideal and for this one some of these ones here is ideal one so that's"}, {"start": 5526.08, "end": 5530.36, "text": " the idea that's the Hungarian match algorithm if you just think about it a"}, {"start": 5530.36, "end": 5534.8, "text": " little bit this should make sense it's not that hard okay let's get back to the"}, {"start": 5534.8, "end": 5541.8, "text": " code after we run that we'll end up with a set of indices and you will see what"}, {"start": 5541.8, "end": 5549.68, "text": " we get here basically you can see that the bounding box 17 is matched with the"}, {"start": 5549.68, "end": 5554.64, "text": " bonding ground truth bonding box one the bonding box 66 is matched with the"}, {"start": 5554.64, "end": 5558.92, "text": " ground box zero and all of this is for the first image in the batch and then"}, {"start": 5558.92, "end": 5564.6, "text": " for the second image we have four and the 11 are matched seven and one twenty six"}, {"start": 5564.6, "end": 5569.32, "text": " and ten etc etc so we got our matching that's it that's the Hungarian matching"}, {"start": 5569.32, "end": 5572.88, "text": " that's the second component hopefully this was clear enough let's continue on"}, {"start": 5572.88, "end": 5577.2, "text": " now we just need to calculate the losses and we are pretty much done that's"}, {"start": 5577.2, "end": 5580.88, "text": " that's everything there is to data so we just calculate the number of boxes so we"}, {"start": 5580.88, "end": 5585.32, "text": " can use that for normalization nothing fancy there is gonna be 16 for our"}, {"start": 5585.32, "end": 5591.719999999999, "text": " particular batch so that's 16 and now we do the loss computation so let's dig"}, {"start": 5591.719999999999, "end": 5596.12, "text": " into this so we have three losses we're calculating one is labels one is boxes"}, {"start": 5596.12, "end": 5599.759999999999, "text": " we'll see that boxes actually encompasses two losses and then"}, {"start": 5599.759999999999, "end": 5604.32, "text": " cardinality is not a loss per se but that's just an implementation detail"}, {"start": 5604.32, "end": 5610.28, "text": " let's let's kind of dig into this okay so loss labels is the first loss what do"}, {"start": 5610.28, "end": 5615.5199999999995, "text": " we do here we grab the prediction the predicted logits so that's gonna be the"}, {"start": 5615.5199999999995, "end": 5620.16, "text": " output of our decoder stack and the shape is gonna be this one okay so we"}, {"start": 5620.16, "end": 5627.04, "text": " have hundred tokens nine two logits and we have two batches we grab this we call"}, {"start": 5627.04, "end": 5633.04, "text": " this function which is going to basically give us how we should index"}, {"start": 5633.04, "end": 5639.5, "text": " this this output here so as to get those boxes we just found using the Hungarian"}, {"start": 5639.5, "end": 5643.72, "text": " matching algorithm so those predict the boxes that are matched to the ground"}, {"start": 5643.72, "end": 5648.66, "text": " truth boxes that's what this IDX index is gonna contain so you can see here"}, {"start": 5648.66, "end": 5655.16, "text": " box 1766 all the way through 99 those are the ones that were matched with with"}, {"start": 5655.16, "end": 5659.68, "text": " our ground truth boxes next up here we are just going to take the indices we"}, {"start": 5659.68, "end": 5663.92, "text": " got from our Hungarian matching algorithm we're just gonna focus on the"}, {"start": 5663.92, "end": 5669.96, "text": " ground truth indices ground truth bounding boxes indices let me remind you"}, {"start": 5669.96, "end": 5674.04, "text": " so this is the ground truth bounding box indices we're just gonna focus on this"}, {"start": 5674.04, "end": 5679.28, "text": " one and on this one and why we do that is this is just gonna be a permutation"}, {"start": 5679.28, "end": 5686.36, "text": " for the labels so we want to now permute the ground truth labels so that we have"}, {"start": 5686.36, "end": 5692.68, "text": " exact match with the predicted boxes that match those ground truth boxes so"}, {"start": 5692.68, "end": 5698.0, "text": " let me go through that so we end up with basically 16 classes there that are"}, {"start": 5698.0, "end": 5704.0, "text": " permuted in the exactly the right manner next up we create this tensor of target"}, {"start": 5704.0, "end": 5709.72, "text": " classes that's going to be of the following shape so 200 and it's"}, {"start": 5709.72, "end": 5716.320000000001, "text": " basically going to contain 91 across all so it's gonna just contain 91s"}, {"start": 5716.320000000001, "end": 5722.4400000000005, "text": " everywhere and that basically means that currently these are the classes that are"}, {"start": 5722.44, "end": 5726.759999999999, "text": " output output bounding boxes should should should have but then we just"}, {"start": 5726.759999999999, "end": 5731.4, "text": " copy paste the target the actual target classes so now let me print you that the"}, {"start": 5731.4, "end": 5736.32, "text": " target classes here you can see that some of them are now correctly whoops"}, {"start": 5736.32, "end": 5743.04, "text": " what have I done man let me just do that again and not click anything so let me"}, {"start": 5743.04, "end": 5748.0, "text": " show you what we have here so now you can see that we basically for the match"}, {"start": 5748.0, "end": 5753.44, "text": " boxes so let's find the first one so 53 this is the first predicted box it's"}, {"start": 5753.44, "end": 5760.8, "text": " gonna be 0 1 2 3 4 so the fourth predicted box should be of class 53"}, {"start": 5760.8, "end": 5765.7, "text": " because that predicted box is matched with a certain ground truth box okay"}, {"start": 5765.7, "end": 5770.08, "text": " that's that's the basic idea again I'm just gonna kind of skim through this"}, {"start": 5770.08, "end": 5773.2, "text": " this is a lot of details already but hopefully this gives you some"}, {"start": 5773.2, "end": 5777.24, "text": " understanding and now we just do the cross entropy loss which is going to"}, {"start": 5777.24, "end": 5782.76, "text": " force such that all of these classes have a probability that goes up to 1 and"}, {"start": 5782.76, "end": 5789.32, "text": " we also have this waiting for the all of these different classes such that the"}, {"start": 5789.32, "end": 5795.639999999999, "text": " no object class which is this 91 thing has lesser weight in the cross entropy"}, {"start": 5795.639999999999, "end": 5799.96, "text": " which makes sense because you can see how many of these are there okay let me"}, {"start": 5799.96, "end": 5805.48, "text": " quickly show you in the one note what this exactly means so just to make it a"}, {"start": 5805.48, "end": 5811.2, "text": " bit clear imagine we have our output here let me just change the color here"}, {"start": 5811.2, "end": 5819.799999999999, "text": " let me first erase this thing so this is our output we have hundred outputs here"}, {"start": 5819.799999999999, "end": 5825.04, "text": " I'm gonna just picture that a bit differently so imagine we have hundred"}, {"start": 5825.04, "end": 5832.799999999999, "text": " outputs here we have I'm gonna just write down maybe four okay and imagine"}, {"start": 5832.8, "end": 5839.24, "text": " we have two bonding boxes so we have one blue one and we have one that's a green"}, {"start": 5839.24, "end": 5848.96, "text": " one so now imagine that we matched this one with this one and imagine that we've"}, {"start": 5848.96, "end": 5857.16, "text": " matched maybe the second one with this one so if the class for this one is 23"}, {"start": 5857.16, "end": 5863.72, "text": " and is the class for this one is like 15 let me change the color so if the class"}, {"start": 5863.72, "end": 5873.68, "text": " for this one is 15 we'll want to make sure that the logits here so that for"}, {"start": 5873.68, "end": 5880.48, "text": " the whatever the 15 indexes we will have like 92 here so this is gonna be 92"}, {"start": 5880.48, "end": 5886.0, "text": " outputs we'll want to make sure that we push the distribution such that we have"}, {"start": 5886.0, "end": 5889.76, "text": " a peak around 15 because that's the ground truth class so that's what this"}, {"start": 5889.76, "end": 5894.28, "text": " loss is going to enforce it's going to enforce that the logic distribution for"}, {"start": 5894.28, "end": 5901.32, "text": " this particular output token right because remember we have output tokens"}, {"start": 5901.32, "end": 5906.96, "text": " which then we map into both the boxes as well as this distribution so this is"}, {"start": 5906.96, "end": 5911.36, "text": " gonna push that distribution so that we have the correct class similarly for"}, {"start": 5911.36, "end": 5916.16, "text": " here so we'll make sure that for this particular class for this particular"}, {"start": 5916.16, "end": 5926.36, "text": " token that at 23 we're gonna have a peak so that's what this this loss is about"}, {"start": 5926.36, "end": 5934.839999999999, "text": " hopefully this kind of helps it a little bit okay let's continue on and see the"}, {"start": 5934.839999999999, "end": 5938.92, "text": " second loss the second loss is gonna be the boxes loss so I'm just gonna kind of"}, {"start": 5938.92, "end": 5944.28, "text": " quickly go through this we get the predicted boxes here we get the target"}, {"start": 5944.28, "end": 5948.52, "text": " boxes we just calculate the loss between the sources boxes and the target boxes"}, {"start": 5948.52, "end": 5953.56, "text": " by doing the L1 I already explained that in one note and then we just do some"}, {"start": 5953.56, "end": 5957.52, "text": " normalization with the number of boxes we do a similar thing here so just"}, {"start": 5957.52, "end": 5962.4800000000005, "text": " generalized box IOU we hit that's the second component of the loss and we just"}, {"start": 5962.4800000000005, "end": 5966.36, "text": " do some realization and those are the two additional losses that we we have"}, {"start": 5966.36, "end": 5971.04, "text": " that's it and the final one is this cardinality that is just going to tell"}, {"start": 5971.04, "end": 5976.28, "text": " us how many of the of the how many errors do we have when it comes to"}, {"start": 5976.28, "end": 5985.2, "text": " computing the no object boxes okay so basically it tells us for each of the"}, {"start": 5985.2, "end": 5993.36, "text": " hundred predictions how many of those were different from the no object class"}, {"start": 5993.36, "end": 5997.679999999999, "text": " so that's what this this thing here does so let me show you let me print this"}, {"start": 5997.679999999999, "end": 6003.16, "text": " here you can see that because the model is still not trained pretty much all of"}, {"start": 6003.16, "end": 6007.799999999999, "text": " these IE none of these predicts the no object bounding box because currently"}, {"start": 6007.799999999999, "end": 6016.5599999999995, "text": " is just kind of random so basically if I were to print this thing let me see what"}, {"start": 6016.5599999999995, "end": 6022.08, "text": " we get so as you can see none of these is predicting 91 which is the no object"}, {"start": 6022.08, "end": 6027.48, "text": " class and that's where we get hundred hundred here and now we just do l1 loss"}, {"start": 6027.48, "end": 6034.76, "text": " between that and the target lengths so we expect 16 boxes 2 and 14 and so that's"}, {"start": 6034.76, "end": 6039.6, "text": " it like just calculating that type of discrepancy between between the ground"}, {"start": 6039.6, "end": 6044.36, "text": " truth and the predictions nothing fancy there that's it guys we just repeat the"}, {"start": 6044.36, "end": 6048.32, "text": " same procedure for all of the other five layers of the decoder I'm just gonna"}, {"start": 6048.32, "end": 6055.04, "text": " skip all of that okay here we are those losses are not computed and finally we"}, {"start": 6055.04, "end": 6060.24, "text": " just use that a weighted dictionary to sum up so for all of the losses as you"}, {"start": 6060.24, "end": 6063.679999999999, "text": " can see here we go through the radio dictionary we go through the last"}, {"start": 6063.679999999999, "end": 6067.48, "text": " dictionary and if if we have a corresponding weight in our weighted"}, {"start": 6067.48, "end": 6072.12, "text": " dictionary we just multiply those and we sum them up and we get losses so you"}, {"start": 6072.12, "end": 6075.799999999999, "text": " can imagine that this is like probably the most complex loss I've seen in my"}, {"start": 6075.8, "end": 6080.84, "text": " life like let me let me see what's the length of this thing number of keys here"}, {"start": 6080.84, "end": 6087.400000000001, "text": " 18 so we have like 18 losses in total here being computed that's it and after"}, {"start": 6087.400000000001, "end": 6094.2, "text": " this this is just some boilerplate to be honest we just grab the loss final loss"}, {"start": 6094.2, "end": 6098.72, "text": " we do the zero grads so clean the gradients of the associated parameters"}, {"start": 6098.72, "end": 6102.2, "text": " in the in the torch computational graph we do the backward which is gonna"}, {"start": 6102.2, "end": 6107.32, "text": " compute the gradients for the last we just computed we just do some gradient"}, {"start": 6107.32, "end": 6115.96, "text": " clipping with 0.1 max norm and we do a step of the optimizer guys that's it"}, {"start": 6115.96, "end": 6122.8, "text": " this is like the longest video I ever made I'm experimenting I am super"}, {"start": 6122.8, "end": 6127.44, "text": " curious to know how many of you followed through the whole video this is probably"}, {"start": 6127.44, "end": 6132.32, "text": " the last video I'll make that this is that's this long unless I get a very"}, {"start": 6132.32, "end": 6135.919999999999, "text": " good feedback on this one because it takes a lot of time to prepare these and"}, {"start": 6135.919999999999, "end": 6141.16, "text": " it takes a lot of time to film and to edit super huge videos such as this one"}, {"start": 6141.16, "end": 6146.879999999999, "text": " in any case I really hope you learn something new I would super appreciate"}, {"start": 6146.879999999999, "end": 6151.12, "text": " your feedback do let me know if you like this video please share it out that's"}, {"start": 6151.12, "end": 6155.08, "text": " the best way you can help out this channel and finally subscribe to this"}, {"start": 6155.08, "end": 6161.16, "text": " channel if you haven't in any case until next time bye bye"}]