DETR: End-to-End Object Detection with Transformers | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I cover DETR, an end-to-end object detection pipeline with transformers.

The main 2 ideas are:
* Using transformers instead of specialized vision architectures
* Using Hungarian matching and loss to train the system e2e

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2005.12872
✅ Code: https://github.com/facebookresearch/detr
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 

00:00 Intro: DETR main ideas
00:45 Non-max suppression
03:20 High-level pipeline overview
07:50 Architecture in more detail
12:10 Matching loss
18:35 Hungarian loss
21:00 Results
24:05 Visualization
27:35 Ablations
30:00 Outro: Segmentation results

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️


Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#detr #objectdetection #transformers
 What's cracking guys? In this video I'm covering end-to-end object detection with transformers or DETER for short by Nicolas Carrion, Francisco Massa and others from Facebook AI Research. So obviously it's about object detection and they are using transformers. So they mentioned here that the main ingredients of the new framework called DETER are set-based global loss that forces unique predictions via bipartite matching. It's a mouthful, we'll break that down in a minute. And a transformer encoder decoder architecture. So the basic premise here is they are training this detection pipeline using end-to-end learning and they're using as the differentiable blob they're using transformers. So those are the two main ideas and comparing that to previous object detection pipelines, like if you're familiar with that, basically most of them consist out of multiple stages. So as you can see here on the left side, basically we have, so the stage one just outputs bunch of these different region proposals. So all of these boxes are kind of claiming that they see a car inside of that box. And the idea is to just kind of filter out and leave just the best one. And in this case this non-max suppression algorithm works perfectly. So basically after stage one proposes these regions we have this heuristic called non-max suppression algorithm and it just basically filters out the best box. And the way it works is super simple. Basically you find the box with the highest confidence that this is a car and then what you do, you just set some threshold and you find the IOU or the intersection of reunion with other boxes and you just kind of filter out those boxes that have that whose IOU crosses that threshold. Okay so I just went ahead and pasted this IOU definition so that I can better explain how this NMS algorithm, so non-max suppression, works. So as I said we have two stages. Usually the first stage proposes a bunch of these different regions and all of those boxes kind of claim that they see a car inside of it. But we want to just leave a single one that has the highest confidence and the best bounding box. And the way we do that is the following. So out of all of these boxes we'll just find the one with the highest confidence score and then we're just gonna set the threshold for example I don't know like 0.7 and we're going to remove all of those boxes whose IOU is bigger than 0.7. That means in human language those boxes that overlap too much with the box with the highest confidence those will be removed. And IOU is just just defined like this. So you find the intersection between the two boxes you find the union and you just divide the two. As you can see as the the overlap becomes bigger this IOU will will tend to one and as the overlap becomes smaller you'll basically this IOU will drop to zero. Okay that's the basic idea. So we want to the idea the whole idea of the DETER pipeline is to remove this this heuristic and kind of train this whole system detection pipeline end to end without these multiple stages without these heuristics such as the NMS. And this is not the only method out there but that's the one that I took as an as an example and yeah that's it. So with the motivation out of the way let's see how DETER works. So jumping to this high level explanation of the pipeline we have how the thing like flows is we have an image at the input we pass it through a CNN which will down sample the spatial extent and increase the number of channels so we'll end up with multiple feature maps here. So something like hw times c where c is the number of channels and initially we maybe have something like h0 we have w0 and we have three in the case of RGB images. So after we have these feature maps what we do is we're going to flatten them out and pass them as tokens into the transformer and later I'll just we'll just unpack how this works in the next diagram but here just stick with me. So basically we just kind of process all of these tokens similar to division transformer you can you can treat it that way and what what what happens is at the output of the decoder we have a bunch of these different region proposals. So we'll have a fixed number of these and they were using 100 so we'll have 100 proposals and each of these boxes has two pieces of information so we can treat this box it will just be a two tuple so we'll have two tuples so we'll have one tuple will tell us about the actual box so what's the the center coordinate and what's the height and width of the box so that will be four numbers so we'll have like cx that's the center coordinate on the x-axis then we have the center coordinate on the y-axis we have the height of the box and we have the width of the box. So in the case of this red one so obviously this would be the center then we had the height and we've had the width so that's the output. Now the second piece of information that's vital is which kind of object is that and that's where the second tuple comes in and this tuple will just contain c number of classes plus one where this is just the cardinal number of the of the classes set so that means for example if we have I don't know 10 classes like dog cat whatever we'll have 10 here and plus one is because of the this no object class so sometimes a box like this green one just is just a spurious box and it doesn't contain any objects so that's why we include plus one so that will be the output and all of these hundred outputs will contain exactly these two tuples so that's the output information now the interesting part is how do we actually match these so that we can compare them and how do we back prop these through the encoder and the cnn and train the data pipeline so the idea is fairly simple the first thing we do is this bipartite matching and i'll explain that in a bit more detail but the rough idea is to find the permutation so basically if we if we focus on the ground truth the ground truth here will be something like this so we'll have let me just draw it like this so we'll have a bunch of outputs and the first one will be this this let's say this red box is the first output in the ground truth and then the second output will be this maybe this yellow one i'll just denote like this so we'll have so this will be the two outputs and then we have a bunch of these no object like proposals and so as you can see the the goal will be to kind of learn how to find a permutation how to associate this one to this one and how to associate this box with this box so we'll basically want to find a way how to sort this so that we have this red box standing in the first place and then the yellow box being in the second place and all of the other no object proposals should be here and once we have that then we can just kind of compare them because they're matched right because if we didn't do that it just wouldn't make any sense so again on the high level we first do the sorting using this hungarian matching algorithm and the second thing is we just compare them so obviously you can think of it we'll basically want to compare the boxes so the the red box here and the red box here will somehow include the iou in the loss will somehow include all of those four coordinates in the loss as well as the class information so for example this here so the the red class for this proposal will be equal to one so the probability is 100 here maybe our prediction is not that good so it's predicting maybe 0.7 for this red class and we want to push that 0.7 towards one okay that was it in a nutshell now let me start digging into more details so that this becomes way more clear than it currently is okay let's find and i'll explain all of these equations in a second i just want to focus first on on the on the expanded diagram of that of that same pipeline so again we have cnn we have these we have these feature maps and we'll have positionally coding so that's standard stuff for transformers except that for for a small detail they are actually adding this positionally codings in every single layer of the encoder and of the decoder but that's a minor detail again if you're not familiar how transformers work please check out my video on transformer i'll link it somewhere here but like in a nutshell i'll try to explain it quickly here basically every of these tokens will be a map to a query key and value and what you'll do so you'll have queries so let me denote queries with with this like like an arrow so that's a vector and query will be in a red so we'll have like queries and then we'll have let's focus on on a specific query so on this one and basically this query will be uh we'll do adopt products so that's a that's a way to find the similarity between that query and all of the keys so we'll have keys for every single uh like token and as you can see here this red uh this red uh arrow and this blue arrow are super similar they're pointing in the same direction so that means the the score will be really high between these two so so basically the score between these two between this blue key and this red query will be really high so that means that the value vector and i'll denote value vectors with with green uh that means that this value vector will have a high coefficient when we sum up all of the all of the value vectors so the basic idea is every token attends every other token so we have this global attention and we'll just basically figure out these scores by doing a query and a key dot product and we'll use the scores to kind of do a weighted average of these value vectors and that's how we form a novel representation so that's a super high level overview that's how it works you can think of it as information routing and you do a bunch of those you just propagate that through the encoder you get these representations and now the second part is interesting so they have these object queries and they're basically learned so these are just randomized vectors initially and you can think of it as this one is maybe so this vector here this red one is looking for a particular thing in an image so it's going to attend all of this information and it's going to focus on a particular things so they have a nice visualization a bit later in the paper let me just find it for a second as you can as you can see here basically one of these so they have as i said they have 100 output proposals here they just depicted 20 of those and they just so each of these dots is just the center of the of the box that that proposal it's outputting across the validation set so that's maybe a mouthful but in a nutshell what that means is this one particular proposal so maybe the first one from the output of the decoder so the leftmost proposal in in the of the decoder is basically attending on all of the small so the green dots just mean a small bounding box so it's attending all of the small bounding boxes in the in the bottom left part of the image and as you can see those different output tokens will be attending different parts of the image so this one as you can see is attending to the center most part of the image and also it's attending to those small boxes etc so basically all of these let me go back here so all of these will be focusing on particular like semantics of the image let's call it that way the only difference compared to the original transformer is that here we'll actually be outputting this in parallel we won't be doing this all regressively so we're just gonna output all of the outputs at once in parallel and the final step is once we have these output tokens we just pass them through these fully like feed forward networks and we output the class and box information so that's the as i already mentioned this is for tuple this is the just the c c plus one number of classes and that's that's it in a nutshell so that's how the pipeline is structured now i'm gonna go and explain the most important part and that's how we do how do we match the the proposals and how do we actually do how the loss looks like so let's focus on that now okay so okay this is the this is the main part basically uh the whole idea is to find a permutation let me go here so again i just copy pasted the the diagram from the above and so the whole idea will be the following so you have your ground truth so in in this case in the case of this image we have one black one red box as you can see here we have one yellow box and all of the other boxes will just be no object uh like uh proposals okay and so the idea is to find a permutation of this output here so that the loss is minimized and uh that means we want to push this this this red one should be in the first place this yellow one should be in the second place and all of the other ones should be here so that's the permutation that will minimize the loss and as you can see here so let's focus on this so y i is just a tuple of the class information and the box information and this is just the sigma of i is just the uh current uh permutation okay so we want to minimize this match loss between those two so n just stands for the number of proposals as i said they are using a hundred of these and that's hard coded so basically none of the images in the data set will ever have more than 100 objects so they just took a like a upper threshold and set it to 100 so we sum up all of the match losses are over all of the proposals and we find the permutation that minimizes this matching loss and that will be the permutation that we end up using okay so that may be abstract still so i'll try and break it down here is the the main the main this is how the matching loss looks like broken down into two components we have this left term with the like class probability and we have the right term with this box loss and let me try and break this down so first things first is if we focus on on on this on this proposal okay so this is one of the outputs as i said it's going to have the the four tuple that kind of tells you where the box is and it's going to have this distribution over classes and because it's red it's probably going to have like this red component being the strongest one maybe 0.6 and in the current permutation just treat this as the current permutation of the algorithm so we'll be comparing red with the yellow one and because the the yellow class has a probability of 0.2 that means here we'll have minus 0.2 and that's not good enough because we want to minimize this as you as you remember so this matching loss needs to go down so we we need to minimize it as much as possible and the minimal value would be if we had one for the for the yellow class so instead of having two here we have one so again let me just dig into this into this loss and kind of break it down one just means an index function so that means that when the when the ground truth that's the ci class is different from the no object class so that means when we have some class so here we have this let's call it red class we here we have yellow class so in this case for the for the i equals one we see here the indices so indices go from one to six so c of i c of one is red so that means that because it's different from the no object we'll have one here so this index function will be one okay so c of two is yellow so that means it's still different from no object so that means we're gonna have one here so it's minus one times this and now for the fun part the probability of the of the yellow class for the second index here as we can see is 0.2 and that means we'll end up with your minus 0.2 here that's minus 0.2 and this is going to give us some value whatever that may be let's just denote it as x as you can see the goal will be to find the permutation that minimizes all of these so if you imagine the permutation where this red is here and the yellow is the second one in that case we'd have a much much lower matching loss so for that permutation because this is the red class we can see it has 0.6 we'll have minus 0.6 here and that means we we are going to minimize this loss even more so in a nutshell that that's the way how this Hungarian algorithm works like it just iterates through different permutations it just calculates this these expressions and finds the permutation that has the minimal matching loss and it turns out that that corresponds to having red as the first proposal then the yellow one and then all of the four ones it doesn't matter the order because basically as you can see here these indexing this index function will just be zero because those proposals have ground truth to no object so all of these have ground truth of no object class okay hopefully that was clear now let's break down the loss the box loss and here it is here is how it's defined is defined as a weighted average of this iou loss and of l1 loss as you can see bi as i mentioned that's the four tuple so that's the center coordinates plus the height and width of the box and what we do is we just take the so this is the the ground truth proposal this is the current permutation proposal and we just find the iou as i said iou is distinct the second term just has this l1 between these bounding box vectors so there's one thing that's kind of confusing me here because this goes with a plus sign let me just see here so this goes with a plus sign the the box loss that means we want to minimize it and that means that obviously when the when the bounding box vectors co-align when they're the same this will go down to zero that's cool but the more they intersect so that means it's a better aligned this thing is going up so i suspect that we need a minus sign here or something to to kind of this for this to make sense or basically maybe the a specific choice of these hyperparameters can help it work out but like no i'm i'm kind of confused by by this part i think there should be a minus here or i'm misinterpreting this a bit okay basically that's it that's the l box and as i said after we find the correct matching after we find the correct permutation we do the following thing and this is the loss component this is how the detrim model is trained we basically go over all of the end proposals so that's as i said hundred in their case and what they do is they just find the whatever the ground truth class is we find what's the current probability of that class and we obviously want to push that towards one and that's when log becomes zero so this is minimized and the second part is whenever we have a class that's not a no object so basically a class that has something meaningful inside will want to include that box loss and that's the same box loss as the one as i just explained for the matching procedure so that's it that's that's that's how everything works again i'm a bit confused by the l box loss and let me just kind of step out and try and explain this once more on the high level so again we are looking for a permutation so for this sigma that minimizes a sum over all of these hundred proposals of this l match so this matching loss and that will lead us to find the correct like let's go to sort the output so that we have perfect matching with the ground truth once we have the alignment between the output and the ground truth we just apply this this simple loss here so we both want to penalize for the class is not being as confident enough and we want to also penalize if the l box if the boxes are two different so we want to make sure they that both the class is correct as well as the bounding box that's all the information you need to know about how the model works now let's focus on the results there is one more detail though here and that's we add prediction feed for networks and hungarian loss after each decoder layer so that's just a minor detail but it will be kind of important later on so that means that we are not only doing these ffns at the output at the final layer of the decoder we're also doing it here and here and here and although it's conceptually not that like necessary to do this they they figure out that improves the performance and so they included it and they'll have later some ablations on this so that's why i wanted to mention this so again we'll have the same procedure i just mentioned so the matching plus the the loss the hungarian loss across all of the layers of the output of the decoder okay that's it now let's see for the experiments they compare with this faster rcnn object detection baseline and they show some really nice results so again these architectures are super handcrafted there was many years of iterations to improve these and they managed to using this end-to-end learning pipeline using transformers they managed to outperform the faster rcnn baseline as you can see here focusing on all of the so first thing i really like here is they they kind of mentioned the flops the fps and the number of parameters because only when you kind of even those out then you can then you can kind of claim that you have better performance or whatever or own pair and in this case as you can see on this apl so that's just the the precision for the larger objects we can see that we are the detr outperforms faster rcnn by quite a lot as well as on the medium sized objects but here on the smaller sized objects it's kind of under performs but all in all looking at the this ap metric we are like the detr model is better than than the faster rcnn and they did mention it somewhere here in the paper let me find it um okay so here so that demonstrates significantly better performance on large objects or is not likely enabled by the non-local computations of the transformer it obtains however lower performances on small objects so the the main idea here is because certain objects span all like a huge portion of the image and the transformer having its global attention on all of the layers has the advantage compared to cnns which need to go into couple you need to have a couple of layers before you have the receptive field that can kind of perceive that huge object in the scene in the image and so that's the reason why they they hypothesize the transformer is performing better for the larger objects whereas it has some problems with the smaller objects and i can suppose that the reason being is the the the spacial reduction we have in the cnn so if we were to not reduce the spatial extent but then we'd kind of have to increase the computation and the memory footprint probably like this this can be further improved with some of the novel like efficient transformer architectures yeah i guess that's left for for the future work okay let me get back to the results we saw we saw the the comparison with a faster rcnn now let's see some ablations so they showed that and this is pretty straightforward i mean they showed that that the encoder a part of the transformer is necessary so the more layers you add the better it performs but the the biggest gap goes from not using just like encoder to using it you can see that the ap like increases significantly as well as obviously the the number of parameters as well as the computation and fps but we have a huge improvement here and then we kind of start saturating adding more and more layers they ended up using six i think in their encoder and that's pretty much it okay so let's continue here here is a nice visualization of the output of the encoder so what they do is they take a query token from the output of the encoder and they then attend using that query vector they just attend all of the keys all the key vectors of this image and by doing that they they as you can see here this query attends to this cow here so it does some sort of an instant segmentation as you can see here so focusing on this query vector here so this one basically it attends as you can see the small cow so that's again instant segmentation similarly for all of the other query tokens continuing and seeing other results this one is super interesting basically what i do is they form this synthetic image where they have six times four so 24 giraffes and they mentioned that in the training set they don't have more than 13 giraffes in a single image and despite that this data model as you can see pretty successfully manages to detect all of the 24 giraffes with a high confidence scores you can see 100 100 most of them are above 90 90 as i can see here so that's a nice out of distribution like generalization and that's cool for this second chart here what they do is i mentioned that on the every layer of the decoder they have this hungarian laws being applied and what they show is that when they are using one of the intermediate layers instead of the final layer as the as the output they get you can see that this progression so obviously taking the first layer taking the output from the first layer is much worse than if we take it from the sixth layer so that's the final layer and additionally what they show here so i'm just focusing on this ap metric ignoring this ap 50 metric what they additionally show is that this non max suppression heuristic which i mentioned in the beginning of the video basically helps if we are taking the outputs from the initial couple of layers but then later on it does not help and that's and that's by design because the hungarian laws and the matching was meant to make this uh like the the proposals unique and so we don't have to kind of filter out similar proposals because that's handled by the loss implicitly okay uh that's that part here they have a interesting visualization of the this time decoder attention mechanisms so we take one of the proposals one of the tokens from the output of the decoder we use that and we attend to the outputs of the encoder network and that's what we got and basically as you can see focusing on one of those output tokens so whatever like the the whatever the the index of the orange token is you can see that it attends to this to the extremities of the of the elephant this kind of explains why why the encoder is necessary so encoder kind of does this uh this initial segmentation of the instance and then what the decoder does it learns how to kind of do contours contours of the object which is necessary for it to find the convex hull or the the bounding box in this case uh of the of that object they just kind of work in symbiosis and help each other uh and again for the blue token you can see that it again attends the extremities and that's how it figures out so so these components here are really important for for the detector to figure out that this is an elephant and similarly in this image for zebras you can see a green token attending here to this zebra the blue one to this zebra and the orange one to the zebra okay that's it um they did a couple more ablations i mentioned that the spatial these these uh positional encodings they have um it's pretty intricate i can kind of go to appendix and show you this part so as you can see here the spatial encoding is added to the keys to values in every single layer and it's also added to the keys of the of the decoder as well and you can see these object queries which are also positional encodings but they are learned not fixed in this in this particular case it's again added here and there so it's it's pretty convoluted but like uh you can also do it the same way as the as the original transformer which was only at the input but they showed that this increased the performance in the ablation table i was just showing you so let me get back here so yeah basically here they showed that adding these uh positional encodings on every layer at the attention module just increases the the ap metric uh compared to all of the other combinations so i don't think we we can extract something like something super intuitive here it's just that this experimentation showed that it's just better to kind of include them in all of the layers okay they also try and uh like the ablation of the L box component which i showed you and they try using only L1 or try using only IOU and it turns out that um if you if you ignore the IOU you have a significant decrease in this AP metric so you have to include the IOU L1 is just extra and using both of those obviously just increases across just boosts up all of the metrics uh okay that's it yeah i already mentioned this one as as i said so this just shows how different uh output those they call it they call those prediction slots i just call them tokens of the decoder basically you can see that each of them attends different different regions of the image and different types of objects of of boxes so you can you can notice a trend that this red dots are uh equally present across all of these 20 prediction slots out of the 100 they have and the reason being those are i think huge they say some of here are huge horizontal boxes or something so red to large horizontal boxes and it seems that the MSCoco data set just has a bunch of those objects which span the whole image horizontally so that's why all of the all of these uh prediction slots or tokens i'll learn how to kind of focus and attend to those okay that's that's pretty much it um let me see i'm going to briefly uh just tell you about this uh they also showed that just by simply modifying the data pipeline uh by adding this this this additional like segmentation head they can also do really well on this panoptic segmentation task and the idea is they just do regular object detection training of the data pipeline so that remains the same and then they just append the head and they trend that they freeze the data and they just train the the head itself and they showed some nice results you can see here like segmentation of this giraffe image and boss etc yeah they just showed that this works really well here in this table comparing it to other baselines like this panoptic fpn etc and they just report nice results there as well so that's pretty much it it's exciting uh so the main two components again are using transformers and learning end-to-end using this this match plus hungarian loss logic and that's it if you if you like this video share it out subscribe to this channel and until next time bye bye
[{"start": 0.0, "end": 4.08, "text": " What's cracking guys? In this video I'm covering end-to-end object detection with"}, {"start": 4.08, "end": 11.6, "text": " transformers or DETER for short by Nicolas Carrion, Francisco Massa and others from Facebook AI"}, {"start": 11.6, "end": 18.0, "text": " Research. So obviously it's about object detection and they are using transformers. So they mentioned"}, {"start": 18.0, "end": 23.92, "text": " here that the main ingredients of the new framework called DETER are set-based global loss that forces"}, {"start": 23.92, "end": 29.2, "text": " unique predictions via bipartite matching. It's a mouthful, we'll break that down in a minute."}, {"start": 29.2, "end": 35.28, "text": " And a transformer encoder decoder architecture. So the basic premise here is they are training"}, {"start": 35.28, "end": 41.68, "text": " this detection pipeline using end-to-end learning and they're using as the"}, {"start": 41.68, "end": 47.28, "text": " differentiable blob they're using transformers. So those are the two main ideas and comparing that"}, {"start": 47.28, "end": 54.4, "text": " to previous object detection pipelines, like if you're familiar with that, basically most of them"}, {"start": 54.4, "end": 59.839999999999996, "text": " consist out of multiple stages. So as you can see here on the left side, basically we have, so the"}, {"start": 59.839999999999996, "end": 65.52, "text": " stage one just outputs bunch of these different region proposals. So all of these boxes are kind of"}, {"start": 67.75999999999999, "end": 73.68, "text": " claiming that they see a car inside of that box. And the idea is to just kind of filter out and"}, {"start": 73.68, "end": 79.28, "text": " leave just the best one. And in this case this non-max suppression algorithm works perfectly."}, {"start": 79.28, "end": 84.8, "text": " So basically after stage one proposes these regions we have this heuristic called non-max"}, {"start": 84.8, "end": 90.56, "text": " suppression algorithm and it just basically filters out the best box. And the way it works"}, {"start": 90.56, "end": 97.04, "text": " is super simple. Basically you find the box with the highest confidence that this is a car and then"}, {"start": 97.04, "end": 102.64, "text": " what you do, you just set some threshold and you find the IOU or the intersection of reunion"}, {"start": 102.64, "end": 109.52, "text": " with other boxes and you just kind of filter out those boxes that have that whose IOU crosses that"}, {"start": 109.52, "end": 114.24, "text": " threshold. Okay so I just went ahead and pasted this IOU definition so that I can better explain"}, {"start": 114.24, "end": 120.64, "text": " how this NMS algorithm, so non-max suppression, works. So as I said we have two stages. Usually"}, {"start": 120.64, "end": 126.32, "text": " the first stage proposes a bunch of these different regions and all of those boxes kind of claim that"}, {"start": 126.32, "end": 131.52, "text": " they see a car inside of it. But we want to just leave a single one that has the highest"}, {"start": 131.52, "end": 137.04000000000002, "text": " confidence and the best bounding box. And the way we do that is the following. So out of all of these"}, {"start": 137.04000000000002, "end": 141.84, "text": " boxes we'll just find the one with the highest confidence score and then we're just gonna set"}, {"start": 141.84, "end": 148.32000000000002, "text": " the threshold for example I don't know like 0.7 and we're going to remove all of those boxes"}, {"start": 148.32000000000002, "end": 155.44, "text": " whose IOU is bigger than 0.7. That means in human language those boxes that overlap too much with"}, {"start": 155.44, "end": 160.48000000000002, "text": " the box with the highest confidence those will be removed. And IOU is just just defined like this. So"}, {"start": 160.48, "end": 165.04, "text": " you find the intersection between the two boxes you find the union and you just divide the two."}, {"start": 165.04, "end": 171.6, "text": " As you can see as the the overlap becomes bigger this IOU will will tend to one and as the overlap"}, {"start": 171.6, "end": 177.67999999999998, "text": " becomes smaller you'll basically this IOU will drop to zero. Okay that's the basic idea. So we"}, {"start": 177.67999999999998, "end": 183.12, "text": " want to the idea the whole idea of the DETER pipeline is to remove this this heuristic and"}, {"start": 183.12, "end": 189.28, "text": " kind of train this whole system detection pipeline end to end without these multiple stages without"}, {"start": 189.28, "end": 194.88, "text": " these heuristics such as the NMS. And this is not the only method out there but that's the one that"}, {"start": 194.88, "end": 201.2, "text": " I took as an as an example and yeah that's it. So with the motivation out of the way let's see how"}, {"start": 201.2, "end": 208.32, "text": " DETER works. So jumping to this high level explanation of the pipeline we have how the thing"}, {"start": 208.88, "end": 214.64, "text": " like flows is we have an image at the input we pass it through a CNN which will down sample the"}, {"start": 214.64, "end": 219.51999999999998, "text": " spatial extent and increase the number of channels so we'll end up with multiple feature maps here."}, {"start": 219.51999999999998, "end": 228.07999999999998, "text": " So something like hw times c where c is the number of channels and initially we maybe have something"}, {"start": 228.07999999999998, "end": 237.92, "text": " like h0 we have w0 and we have three in the case of RGB images. So after we have these feature maps"}, {"start": 237.92, "end": 243.44, "text": " what we do is we're going to flatten them out and pass them as tokens into the transformer and later"}, {"start": 243.44, "end": 249.92, "text": " I'll just we'll just unpack how this works in the next diagram but here just stick with me. So"}, {"start": 249.92, "end": 254.72, "text": " basically we just kind of process all of these tokens similar to division transformer you can"}, {"start": 254.72, "end": 260.64, "text": " you can treat it that way and what what what happens is at the output of the decoder we have"}, {"start": 260.64, "end": 265.76, "text": " a bunch of these different region proposals. So we'll have a fixed number of these and they were"}, {"start": 265.76, "end": 272.48, "text": " using 100 so we'll have 100 proposals and each of these boxes has two pieces of information so we can"}, {"start": 272.48, "end": 277.92, "text": " treat this box it will just be a two tuple so we'll have two tuples so we'll have one tuple will"}, {"start": 277.92, "end": 283.20000000000005, "text": " tell us about the actual box so what's the the center coordinate and what's the height and width"}, {"start": 283.20000000000005, "end": 288.88, "text": " of the box so that will be four numbers so we'll have like cx that's the center coordinate on the"}, {"start": 288.88, "end": 294.64000000000004, "text": " x-axis then we have the center coordinate on the y-axis we have the height of the box and we have"}, {"start": 294.64000000000004, "end": 301.20000000000005, "text": " the width of the box. So in the case of this red one so obviously this would be the center then we"}, {"start": 301.2, "end": 305.92, "text": " had the height and we've had the width so that's the output. Now the second piece of information"}, {"start": 305.92, "end": 312.71999999999997, "text": " that's vital is which kind of object is that and that's where the second tuple comes in and this"}, {"start": 312.71999999999997, "end": 321.03999999999996, "text": " tuple will just contain c number of classes plus one where this is just the cardinal number of the"}, {"start": 321.03999999999996, "end": 325.91999999999996, "text": " of the classes set so that means for example if we have I don't know 10 classes like dog cat whatever"}, {"start": 325.92, "end": 332.64000000000004, "text": " we'll have 10 here and plus one is because of the this no object class so sometimes a box like this"}, {"start": 332.64000000000004, "end": 338.48, "text": " green one just is just a spurious box and it doesn't contain any objects so that's why we include"}, {"start": 338.48, "end": 343.52000000000004, "text": " plus one so that will be the output and all of these hundred outputs will contain exactly"}, {"start": 343.52000000000004, "end": 348.08000000000004, "text": " these two tuples so that's the output information now the interesting part is how do we actually"}, {"start": 348.08000000000004, "end": 352.64, "text": " match these so that we can compare them and how do we back prop these through the encoder and the"}, {"start": 352.64, "end": 359.52, "text": " cnn and train the data pipeline so the idea is fairly simple the first thing we do is this"}, {"start": 359.52, "end": 364.96, "text": " bipartite matching and i'll explain that in a bit more detail but the rough idea is to find the"}, {"start": 364.96, "end": 369.36, "text": " permutation so basically if we if we focus on the ground truth the ground truth here will be"}, {"start": 369.36, "end": 375.12, "text": " something like this so we'll have let me just draw it like this so we'll have a bunch of outputs and"}, {"start": 375.12, "end": 380.56, "text": " the first one will be this this let's say this red box is the first output in the ground truth and"}, {"start": 380.56, "end": 386.64, "text": " then the second output will be this maybe this yellow one i'll just denote like this so we'll"}, {"start": 386.64, "end": 393.84000000000003, "text": " have so this will be the two outputs and then we have a bunch of these no object like proposals"}, {"start": 393.84000000000003, "end": 400.08, "text": " and so as you can see the the goal will be to kind of learn how to find a permutation how to"}, {"start": 400.08, "end": 407.52, "text": " associate this one to this one and how to associate this box with this box so we'll basically want to"}, {"start": 407.52, "end": 413.03999999999996, "text": " find a way how to sort this so that we have this red box standing in the first place and then the"}, {"start": 413.03999999999996, "end": 418.88, "text": " yellow box being in the second place and all of the other no object proposals should be here and"}, {"start": 418.88, "end": 423.68, "text": " once we have that then we can just kind of compare them because they're matched right because if we"}, {"start": 423.68, "end": 428.79999999999995, "text": " didn't do that it just wouldn't make any sense so again on the high level we first do the sorting"}, {"start": 428.79999999999995, "end": 435.35999999999996, "text": " using this hungarian matching algorithm and the second thing is we just compare them so obviously"}, {"start": 435.36, "end": 441.04, "text": " you can think of it we'll basically want to compare the boxes so the the red box here and the red box"}, {"start": 441.04, "end": 447.92, "text": " here will somehow include the iou in the loss will somehow include all of those four coordinates in"}, {"start": 447.92, "end": 454.96000000000004, "text": " the loss as well as the class information so for example this here so the the red class for this"}, {"start": 454.96000000000004, "end": 460.72, "text": " proposal will be equal to one so the probability is 100 here maybe our prediction is not that good"}, {"start": 460.72, "end": 467.44000000000005, "text": " so it's predicting maybe 0.7 for this red class and we want to push that 0.7 towards one okay that"}, {"start": 467.44000000000005, "end": 472.88000000000005, "text": " was it in a nutshell now let me start digging into more details so that this becomes way more clear"}, {"start": 472.88000000000005, "end": 478.24, "text": " than it currently is okay let's find and i'll explain all of these equations in a second i"}, {"start": 478.24, "end": 485.36, "text": " just want to focus first on on the on the expanded diagram of that of that same pipeline so again we"}, {"start": 485.36, "end": 489.92, "text": " have cnn we have these we have these feature maps and we'll have positionally coding so that's"}, {"start": 489.92, "end": 494.48, "text": " standard stuff for transformers except that for for a small detail they are actually adding this"}, {"start": 494.48, "end": 500.16, "text": " positionally codings in every single layer of the encoder and of the decoder but that's a minor"}, {"start": 500.16, "end": 506.32, "text": " detail again if you're not familiar how transformers work please check out my video on transformer"}, {"start": 506.32, "end": 511.92, "text": " i'll link it somewhere here but like in a nutshell i'll try to explain it quickly here basically"}, {"start": 511.92, "end": 517.84, "text": " every of these tokens will be a map to a query key and value and what you'll do so you'll have"}, {"start": 517.84, "end": 524.0, "text": " queries so let me denote queries with with this like like an arrow so that's a vector and query"}, {"start": 524.0, "end": 529.6, "text": " will be in a red so we'll have like queries and then we'll have let's focus on on a specific query"}, {"start": 529.6, "end": 536.32, "text": " so on this one and basically this query will be uh we'll do adopt products so that's a that's a way"}, {"start": 536.32, "end": 541.6800000000001, "text": " to find the similarity between that query and all of the keys so we'll have keys for every single"}, {"start": 541.68, "end": 549.28, "text": " uh like token and as you can see here this red uh this red uh arrow and this blue arrow are super"}, {"start": 549.28, "end": 554.2399999999999, "text": " similar they're pointing in the same direction so that means the the score will be really high"}, {"start": 554.2399999999999, "end": 561.76, "text": " between these two so so basically the score between these two between this blue key and this red query"}, {"start": 561.76, "end": 567.1999999999999, "text": " will be really high so that means that the value vector and i'll denote value vectors with with"}, {"start": 567.2, "end": 573.9200000000001, "text": " green uh that means that this value vector will have a high coefficient when we sum up all of the"}, {"start": 573.9200000000001, "end": 579.6, "text": " all of the value vectors so the basic idea is every token attends every other token so we have this"}, {"start": 579.6, "end": 587.2800000000001, "text": " global attention and we'll just basically figure out these scores by doing a query and a key dot"}, {"start": 587.2800000000001, "end": 592.96, "text": " product and we'll use the scores to kind of do a weighted average of these value vectors and that's"}, {"start": 592.96, "end": 597.6, "text": " how we form a novel representation so that's a super high level overview that's how it works"}, {"start": 597.6, "end": 603.0400000000001, "text": " you can think of it as information routing and you do a bunch of those you just propagate that"}, {"start": 603.0400000000001, "end": 607.2, "text": " through the encoder you get these representations and now the second part is interesting so they"}, {"start": 607.2, "end": 612.5600000000001, "text": " have these object queries and they're basically learned so these are just randomized vectors"}, {"start": 612.5600000000001, "end": 618.32, "text": " initially and you can think of it as this one is maybe so this vector here this red one is looking"}, {"start": 618.32, "end": 623.12, "text": " for a particular thing in an image so it's going to attend all of this information and it's going"}, {"start": 623.12, "end": 628.1600000000001, "text": " to focus on a particular things so they have a nice visualization a bit later in the paper"}, {"start": 628.88, "end": 635.36, "text": " let me just find it for a second as you can as you can see here basically one of these so they have"}, {"start": 635.36, "end": 641.6, "text": " as i said they have 100 output proposals here they just depicted 20 of those and they just so each of"}, {"start": 641.6, "end": 648.5600000000001, "text": " these dots is just the center of the of the box that that proposal it's outputting across the validation"}, {"start": 648.5600000000001, "end": 653.44, "text": " set so that's maybe a mouthful but in a nutshell what that means is this one particular proposal"}, {"start": 653.44, "end": 658.16, "text": " so maybe the first one from the output of the decoder so the leftmost proposal in in the of the"}, {"start": 658.16, "end": 663.9200000000001, "text": " decoder is basically attending on all of the small so the green dots just mean a small bounding box"}, {"start": 663.9200000000001, "end": 670.88, "text": " so it's attending all of the small bounding boxes in the in the bottom left part of the image and as"}, {"start": 670.88, "end": 675.4399999999999, "text": " you can see those different output tokens will be attending different parts of the image so this one"}, {"start": 675.4399999999999, "end": 680.48, "text": " as you can see is attending to the center most part of the image and also it's attending to those"}, {"start": 680.48, "end": 689.36, "text": " small boxes etc so basically all of these let me go back here so all of these will be focusing on"}, {"start": 689.36, "end": 694.0, "text": " particular like semantics of the image let's call it that way the only difference compared to the"}, {"start": 694.0, "end": 699.28, "text": " original transformer is that here we'll actually be outputting this in parallel we won't be doing"}, {"start": 699.28, "end": 705.12, "text": " this all regressively so we're just gonna output all of the outputs at once in parallel and the"}, {"start": 705.12, "end": 710.72, "text": " final step is once we have these output tokens we just pass them through these fully like feed"}, {"start": 710.72, "end": 715.52, "text": " forward networks and we output the class and box information so that's the as i already mentioned"}, {"start": 715.52, "end": 724.16, "text": " this is for tuple this is the just the c c plus one number of classes and that's that's it in a"}, {"start": 724.16, "end": 729.68, "text": " nutshell so that's how the pipeline is structured now i'm gonna go and explain the most important"}, {"start": 729.68, "end": 735.76, "text": " part and that's how we do how do we match the the proposals and how do we actually do how the loss"}, {"start": 735.76, "end": 744.0, "text": " looks like so let's focus on that now okay so okay this is the this is the main part basically uh the"}, {"start": 744.0, "end": 750.72, "text": " whole idea is to find a permutation let me go here so again i just copy pasted the the diagram from"}, {"start": 750.72, "end": 755.76, "text": " the above and so the whole idea will be the following so you have your ground truth so in"}, {"start": 755.76, "end": 760.5600000000001, "text": " in this case in the case of this image we have one black one red box as you can see here we have one"}, {"start": 760.5600000000001, "end": 768.08, "text": " yellow box and all of the other boxes will just be no object uh like uh proposals okay and so the idea"}, {"start": 768.08, "end": 774.24, "text": " is to find a permutation of this output here so that the loss is minimized and uh that means we"}, {"start": 774.24, "end": 779.36, "text": " want to push this this this red one should be in the first place this yellow one should be in the"}, {"start": 779.36, "end": 783.2, "text": " second place and all of the other ones should be here so that's the permutation that will minimize"}, {"start": 783.2, "end": 790.24, "text": " the loss and as you can see here so let's focus on this so y i is just a tuple of the class"}, {"start": 790.24, "end": 796.8000000000001, "text": " information and the box information and this is just the sigma of i is just the uh current uh"}, {"start": 797.44, "end": 804.0, "text": " permutation okay so we want to minimize this match loss between those two so n just stands for the"}, {"start": 804.0, "end": 809.84, "text": " number of proposals as i said they are using a hundred of these and that's hard coded so basically"}, {"start": 809.84, "end": 814.96, "text": " none of the images in the data set will ever have more than 100 objects so they just took a like a"}, {"start": 814.96, "end": 821.36, "text": " upper threshold and set it to 100 so we sum up all of the match losses are over all of the proposals"}, {"start": 821.36, "end": 826.4, "text": " and we find the permutation that minimizes this matching loss and that will be the permutation"}, {"start": 826.4, "end": 833.52, "text": " that we end up using okay so that may be abstract still so i'll try and break it down here is the"}, {"start": 833.52, "end": 839.04, "text": " the main the main this is how the matching loss looks like broken down into two components we"}, {"start": 839.04, "end": 846.24, "text": " have this left term with the like class probability and we have the right term with this box loss and"}, {"start": 846.24, "end": 852.56, "text": " let me try and break this down so first things first is if we focus on on on this on this"}, {"start": 852.56, "end": 857.92, "text": " proposal okay so this is one of the outputs as i said it's going to have the the four tuple that"}, {"start": 857.92, "end": 863.04, "text": " kind of tells you where the box is and it's going to have this distribution over classes and because"}, {"start": 863.04, "end": 868.4, "text": " it's red it's probably going to have like this red component being the strongest one maybe 0.6"}, {"start": 868.4, "end": 873.76, "text": " and in the current permutation just treat this as the current permutation of the algorithm so we'll"}, {"start": 873.76, "end": 880.64, "text": " be comparing red with the yellow one and because the the yellow class has a probability of 0.2"}, {"start": 880.64, "end": 886.9599999999999, "text": " that means here we'll have minus 0.2 and that's not good enough because we want to minimize this"}, {"start": 886.9599999999999, "end": 892.7199999999999, "text": " as you as you remember so this matching loss needs to go down so we we need to minimize it as much"}, {"start": 892.72, "end": 899.12, "text": " as possible and the minimal value would be if we had one for the for the yellow class so instead"}, {"start": 899.12, "end": 904.88, "text": " of having two here we have one so again let me just dig into this into this loss and kind of"}, {"start": 904.88, "end": 911.44, "text": " break it down one just means an index function so that means that when the when the ground truth"}, {"start": 911.44, "end": 917.6, "text": " that's the ci class is different from the no object class so that means when we have some class so here"}, {"start": 917.6, "end": 923.52, "text": " we have this let's call it red class we here we have yellow class so in this case for the for the"}, {"start": 923.52, "end": 930.72, "text": " i equals one we see here the indices so indices go from one to six so c of i c of one is red so"}, {"start": 931.6800000000001, "end": 936.08, "text": " that means that because it's different from the no object we'll have one here so this"}, {"start": 936.08, "end": 942.48, "text": " index function will be one okay so c of two is yellow so that means it's still different from no"}, {"start": 942.48, "end": 948.08, "text": " object so that means we're gonna have one here so it's minus one times this and now for the fun part"}, {"start": 948.08, "end": 956.24, "text": " the probability of the of the yellow class for the second index here as we can see is 0.2 and that"}, {"start": 956.24, "end": 962.88, "text": " means we'll end up with your minus 0.2 here that's minus 0.2 and this is going to give us some value"}, {"start": 962.88, "end": 969.04, "text": " whatever that may be let's just denote it as x as you can see the goal will be to find the permutation"}, {"start": 969.04, "end": 975.36, "text": " that minimizes all of these so if you imagine the permutation where this red is here and the yellow"}, {"start": 975.36, "end": 984.0, "text": " is the second one in that case we'd have a much much lower matching loss so for that permutation"}, {"start": 984.0, "end": 992.4, "text": " because this is the red class we can see it has 0.6 we'll have minus 0.6 here and that means we"}, {"start": 992.4, "end": 997.12, "text": " we are going to minimize this loss even more so in a nutshell that that's the way how this Hungarian"}, {"start": 997.12, "end": 1002.32, "text": " algorithm works like it just iterates through different permutations it just calculates this"}, {"start": 1002.32, "end": 1008.08, "text": " these expressions and finds the permutation that has the minimal matching loss and it turns out"}, {"start": 1008.08, "end": 1013.68, "text": " that that corresponds to having red as the first proposal then the yellow one and then all of the"}, {"start": 1013.68, "end": 1019.52, "text": " four ones it doesn't matter the order because basically as you can see here these indexing"}, {"start": 1020.48, "end": 1026.24, "text": " this index function will just be zero because those proposals have ground truth to no object"}, {"start": 1026.24, "end": 1033.76, "text": " so all of these have ground truth of no object class okay hopefully that was clear now let's"}, {"start": 1033.76, "end": 1039.84, "text": " break down the loss the box loss and here it is here is how it's defined is defined as a weighted"}, {"start": 1039.84, "end": 1046.56, "text": " average of this iou loss and of l1 loss as you can see bi as i mentioned that's the four tuple"}, {"start": 1046.56, "end": 1052.8, "text": " so that's the center coordinates plus the height and width of the box and what we do is we just"}, {"start": 1052.8, "end": 1059.36, "text": " take the so this is the the ground truth proposal this is the current permutation proposal and we"}, {"start": 1059.36, "end": 1065.68, "text": " just find the iou as i said iou is distinct the second term just has this l1 between these"}, {"start": 1065.68, "end": 1070.8799999999999, "text": " bounding box vectors so there's one thing that's kind of confusing me here because this goes with"}, {"start": 1070.8799999999999, "end": 1076.1599999999999, "text": " a plus sign let me just see here so this goes with a plus sign the the box loss that means we want"}, {"start": 1076.16, "end": 1083.52, "text": " to minimize it and that means that obviously when the when the bounding box vectors co-align when"}, {"start": 1083.52, "end": 1090.24, "text": " they're the same this will go down to zero that's cool but the more they intersect so that means it's"}, {"start": 1090.24, "end": 1096.88, "text": " a better aligned this thing is going up so i suspect that we need a minus sign here or something to"}, {"start": 1096.88, "end": 1103.44, "text": " to kind of this for this to make sense or basically maybe the a specific choice of these"}, {"start": 1103.44, "end": 1109.8400000000001, "text": " hyperparameters can help it work out but like no i'm i'm kind of confused by by this part i think"}, {"start": 1109.8400000000001, "end": 1115.76, "text": " there should be a minus here or i'm misinterpreting this a bit okay basically that's it that's the l"}, {"start": 1115.76, "end": 1122.24, "text": " box and as i said after we find the correct matching after we find the correct permutation"}, {"start": 1122.24, "end": 1126.96, "text": " we do the following thing and this is the loss component this is how the detrim model is trained"}, {"start": 1126.96, "end": 1132.88, "text": " we basically go over all of the end proposals so that's as i said hundred in their case and"}, {"start": 1132.88, "end": 1138.8000000000002, "text": " what they do is they just find the whatever the ground truth class is we find what's the current"}, {"start": 1138.8000000000002, "end": 1144.0800000000002, "text": " probability of that class and we obviously want to push that towards one and that's when log becomes"}, {"start": 1144.0800000000002, "end": 1152.48, "text": " zero so this is minimized and the second part is whenever we have a class that's not a no object"}, {"start": 1153.44, "end": 1158.16, "text": " so basically a class that has something meaningful inside will want to include that box loss and"}, {"start": 1158.16, "end": 1163.2, "text": " that's the same box loss as the one as i just explained for the matching procedure so that's it"}, {"start": 1164.0800000000002, "end": 1171.0400000000002, "text": " that's that's that's how everything works again i'm a bit confused by the l box loss and let me just"}, {"start": 1171.0400000000002, "end": 1177.76, "text": " kind of step out and try and explain this once more on the high level so again we are looking"}, {"start": 1177.76, "end": 1183.8400000000001, "text": " for a permutation so for this sigma that minimizes a sum over all of these hundred proposals of this"}, {"start": 1183.84, "end": 1190.48, "text": " l match so this matching loss and that will lead us to find the correct like let's go to sort the"}, {"start": 1190.48, "end": 1196.0, "text": " output so that we have perfect matching with the ground truth once we have the alignment between"}, {"start": 1196.0, "end": 1201.6799999999998, "text": " the output and the ground truth we just apply this this simple loss here so we both want to penalize"}, {"start": 1201.6799999999998, "end": 1209.12, "text": " for the class is not being as confident enough and we want to also penalize if the l box if the boxes"}, {"start": 1209.12, "end": 1214.08, "text": " are two different so we want to make sure they that both the class is correct as well as the bounding box"}, {"start": 1214.08, "end": 1218.08, "text": " that's all the information you need to know about how the model works now let's focus on the results"}, {"start": 1219.36, "end": 1225.28, "text": " there is one more detail though here and that's we add prediction feed for networks and hungarian"}, {"start": 1225.28, "end": 1230.08, "text": " loss after each decoder layer so that's just a minor detail but it will be kind of important later"}, {"start": 1230.08, "end": 1236.3999999999999, "text": " on so that means that we are not only doing these ffns at the output at the final layer of the"}, {"start": 1236.4, "end": 1242.96, "text": " decoder we're also doing it here and here and here and although it's conceptually not that like"}, {"start": 1242.96, "end": 1248.3200000000002, "text": " necessary to do this they they figure out that improves the performance and so they included it"}, {"start": 1248.3200000000002, "end": 1252.5600000000002, "text": " and they'll have later some ablations on this so that's why i wanted to mention this so again"}, {"start": 1252.5600000000002, "end": 1258.5600000000002, "text": " we'll have the same procedure i just mentioned so the matching plus the the loss the hungarian loss"}, {"start": 1258.5600000000002, "end": 1263.8400000000001, "text": " across all of the layers of the output of the decoder okay that's it now let's see for the"}, {"start": 1263.84, "end": 1270.56, "text": " experiments they compare with this faster rcnn object detection baseline and they show some really"}, {"start": 1270.56, "end": 1277.6, "text": " nice results so again these architectures are super handcrafted there was many years of iterations to"}, {"start": 1277.6, "end": 1283.76, "text": " improve these and they managed to using this end-to-end learning pipeline using transformers they"}, {"start": 1283.76, "end": 1290.9599999999998, "text": " managed to outperform the faster rcnn baseline as you can see here focusing on all of the so first"}, {"start": 1290.96, "end": 1295.8400000000001, "text": " thing i really like here is they they kind of mentioned the flops the fps and the number of"}, {"start": 1295.8400000000001, "end": 1300.8, "text": " parameters because only when you kind of even those out then you can then you can kind of"}, {"start": 1301.52, "end": 1307.28, "text": " claim that you have better performance or whatever or own pair and in this case as you can see on"}, {"start": 1307.28, "end": 1313.28, "text": " this apl so that's just the the precision for the larger objects we can see that we are the"}, {"start": 1313.28, "end": 1320.48, "text": " detr outperforms faster rcnn by quite a lot as well as on the medium sized objects but here on"}, {"start": 1320.48, "end": 1326.24, "text": " the smaller sized objects it's kind of under performs but all in all looking at the this ap"}, {"start": 1326.96, "end": 1332.96, "text": " metric we are like the detr model is better than than the faster rcnn and they did mention it"}, {"start": 1332.96, "end": 1338.8, "text": " somewhere here in the paper let me find it um okay so here so that demonstrates significantly"}, {"start": 1338.8, "end": 1343.6, "text": " better performance on large objects or is not likely enabled by the non-local computations of"}, {"start": 1343.6, "end": 1349.92, "text": " the transformer it obtains however lower performances on small objects so the the main idea here is"}, {"start": 1349.92, "end": 1355.1200000000001, "text": " because certain objects span all like a huge portion of the image and the transformer having"}, {"start": 1355.1200000000001, "end": 1361.04, "text": " its global attention on all of the layers has the advantage compared to cnns which need to go into"}, {"start": 1361.04, "end": 1365.44, "text": " couple you need to have a couple of layers before you have the receptive field that can kind of"}, {"start": 1365.44, "end": 1370.72, "text": " perceive that huge object in the scene in the image and so that's the reason why they they"}, {"start": 1370.72, "end": 1376.72, "text": " hypothesize the transformer is performing better for the larger objects whereas it has some problems"}, {"start": 1376.72, "end": 1382.72, "text": " with the smaller objects and i can suppose that the reason being is the the the spacial reduction"}, {"start": 1382.72, "end": 1388.96, "text": " we have in the cnn so if we were to not reduce the spatial extent but then we'd kind of have to"}, {"start": 1388.96, "end": 1394.48, "text": " increase the computation and the memory footprint probably like this this can be further improved"}, {"start": 1394.48, "end": 1401.2, "text": " with some of the novel like efficient transformer architectures yeah i guess that's left for for the"}, {"start": 1401.2, "end": 1407.76, "text": " future work okay let me get back to the results we saw we saw the the comparison with a faster rcnn"}, {"start": 1407.76, "end": 1412.72, "text": " now let's see some ablations so they showed that and this is pretty straightforward i mean"}, {"start": 1413.52, "end": 1419.76, "text": " they showed that that the encoder a part of the transformer is necessary so the more layers you"}, {"start": 1419.76, "end": 1426.16, "text": " add the better it performs but the the biggest gap goes from not using just like encoder to using it"}, {"start": 1426.16, "end": 1431.92, "text": " you can see that the ap like increases significantly as well as obviously the the"}, {"start": 1431.92, "end": 1436.5600000000002, "text": " number of parameters as well as the computation and fps but we have a huge improvement here and"}, {"start": 1436.5600000000002, "end": 1441.76, "text": " then we kind of start saturating adding more and more layers they ended up using six i think in"}, {"start": 1441.76, "end": 1447.44, "text": " their encoder and that's pretty much it okay so let's continue here here is a nice visualization"}, {"start": 1447.44, "end": 1453.6000000000001, "text": " of the output of the encoder so what they do is they take a query token from the output of the"}, {"start": 1453.6, "end": 1459.28, "text": " encoder and they then attend using that query vector they just attend all of the keys all the"}, {"start": 1459.28, "end": 1466.32, "text": " key vectors of this image and by doing that they they as you can see here this query attends to"}, {"start": 1466.32, "end": 1471.9199999999998, "text": " this cow here so it does some sort of an instant segmentation as you can see here so focusing on"}, {"start": 1471.9199999999998, "end": 1477.04, "text": " this query vector here so this one basically it attends as you can see the small cow so that's"}, {"start": 1477.04, "end": 1483.84, "text": " again instant segmentation similarly for all of the other query tokens continuing and seeing other"}, {"start": 1483.84, "end": 1488.72, "text": " results this one is super interesting basically what i do is they form this synthetic image"}, {"start": 1488.72, "end": 1494.24, "text": " where they have six times four so 24 giraffes and they mentioned that in the training set they don't"}, {"start": 1494.24, "end": 1500.72, "text": " have more than 13 giraffes in a single image and despite that this data model as you can see pretty"}, {"start": 1500.72, "end": 1505.44, "text": " successfully manages to detect all of the 24 giraffes with a high confidence scores you can"}, {"start": 1505.44, "end": 1512.16, "text": " see 100 100 most of them are above 90 90 as i can see here so that's a nice out of distribution"}, {"start": 1512.72, "end": 1519.28, "text": " like generalization and that's cool for this second chart here what they do is i mentioned that on the"}, {"start": 1519.28, "end": 1525.28, "text": " every layer of the decoder they have this hungarian laws being applied and what they show is that when"}, {"start": 1525.28, "end": 1531.3600000000001, "text": " they are using one of the intermediate layers instead of the final layer as the as the output"}, {"start": 1531.36, "end": 1537.84, "text": " they get you can see that this progression so obviously taking the first layer taking the output"}, {"start": 1537.84, "end": 1542.8799999999999, "text": " from the first layer is much worse than if we take it from the sixth layer so that's the final layer"}, {"start": 1542.8799999999999, "end": 1548.6399999999999, "text": " and additionally what they show here so i'm just focusing on this ap metric ignoring this ap 50"}, {"start": 1548.6399999999999, "end": 1553.9199999999998, "text": " metric what they additionally show is that this non max suppression heuristic which i mentioned in"}, {"start": 1553.9199999999998, "end": 1559.6, "text": " the beginning of the video basically helps if we are taking the outputs from the initial couple of"}, {"start": 1559.6, "end": 1564.24, "text": " layers but then later on it does not help and that's and that's by design because the hungarian"}, {"start": 1564.24, "end": 1569.9199999999998, "text": " laws and the matching was meant to make this uh like the the proposals unique and so we don't have"}, {"start": 1569.9199999999998, "end": 1577.04, "text": " to kind of filter out similar proposals because that's handled by the loss implicitly okay uh"}, {"start": 1577.52, "end": 1582.9599999999998, "text": " that's that part here they have a interesting visualization of the this time decoder attention"}, {"start": 1582.9599999999998, "end": 1589.1999999999998, "text": " mechanisms so we take one of the proposals one of the tokens from the output of the decoder"}, {"start": 1589.2, "end": 1595.44, "text": " we use that and we attend to the outputs of the encoder network and that's what we got and basically"}, {"start": 1595.44, "end": 1603.2, "text": " as you can see focusing on one of those output tokens so whatever like the the whatever the"}, {"start": 1603.2, "end": 1609.28, "text": " the index of the orange token is you can see that it attends to this to the extremities of the of"}, {"start": 1609.28, "end": 1613.92, "text": " the elephant this kind of explains why why the encoder is necessary so encoder kind of does this"}, {"start": 1613.92, "end": 1620.0800000000002, "text": " uh this initial segmentation of the instance and then what the decoder does it learns how to kind"}, {"start": 1620.0800000000002, "end": 1625.1200000000001, "text": " of do contours contours of the object which is necessary for it to find the convex hull or the"}, {"start": 1625.1200000000001, "end": 1629.92, "text": " the bounding box in this case uh of the of that object they just kind of work in symbiosis and"}, {"start": 1629.92, "end": 1635.3600000000001, "text": " help each other uh and again for the blue token you can see that it again attends the extremities"}, {"start": 1635.3600000000001, "end": 1641.04, "text": " and that's how it figures out so so these components here are really important for for"}, {"start": 1641.04, "end": 1647.2, "text": " the detector to figure out that this is an elephant and similarly in this image for zebras you can see"}, {"start": 1647.76, "end": 1653.36, "text": " a green token attending here to this zebra the blue one to this zebra and the orange one to the"}, {"start": 1653.36, "end": 1660.32, "text": " zebra okay that's it um they did a couple more ablations i mentioned that the spatial these"}, {"start": 1660.32, "end": 1665.76, "text": " these uh positional encodings they have um it's pretty intricate i can kind of go to appendix and"}, {"start": 1665.76, "end": 1671.44, "text": " show you this part so as you can see here the spatial encoding is added to the keys to values"}, {"start": 1671.44, "end": 1677.76, "text": " in every single layer and it's also added to the keys of the of the decoder as well and you can see"}, {"start": 1677.76, "end": 1682.64, "text": " these object queries which are also positional encodings but they are learned not fixed in this"}, {"start": 1682.64, "end": 1688.0, "text": " in this particular case it's again added here and there so it's it's pretty convoluted but like uh"}, {"start": 1688.0, "end": 1693.28, "text": " you can also do it the same way as the as the original transformer which was only at the input"}, {"start": 1693.28, "end": 1698.56, "text": " but they showed that this increased the performance in the ablation table i was just showing you"}, {"start": 1698.56, "end": 1704.16, "text": " so let me get back here so yeah basically here they showed that adding these uh positional"}, {"start": 1704.16, "end": 1712.0, "text": " encodings on every layer at the attention module just increases the the ap metric uh compared to"}, {"start": 1712.0, "end": 1718.32, "text": " all of the other combinations so i don't think we we can extract something like something super"}, {"start": 1718.32, "end": 1722.8, "text": " intuitive here it's just that this experimentation showed that it's just better to kind of include"}, {"start": 1722.8, "end": 1729.6, "text": " them in all of the layers okay they also try and uh like the ablation of the L box component"}, {"start": 1729.6, "end": 1736.6399999999999, "text": " which i showed you and they try using only L1 or try using only IOU and it turns out that um if you"}, {"start": 1736.6399999999999, "end": 1742.08, "text": " if you ignore the IOU you have a significant decrease in this AP metric so you have to include"}, {"start": 1742.08, "end": 1749.12, "text": " the IOU L1 is just extra and using both of those obviously just increases across just boosts up all"}, {"start": 1749.12, "end": 1754.9599999999998, "text": " of the metrics uh okay that's it yeah i already mentioned this one as as i said so this just shows"}, {"start": 1754.9599999999998, "end": 1759.76, "text": " how different uh output those they call it they call those prediction slots i just call them"}, {"start": 1759.76, "end": 1765.1999999999998, "text": " tokens of the decoder basically you can see that each of them attends different different regions"}, {"start": 1765.1999999999998, "end": 1770.8799999999999, "text": " of the image and different types of objects of of boxes so you can you can notice a trend that this"}, {"start": 1770.8799999999999, "end": 1777.36, "text": " red dots are uh equally present across all of these 20 prediction slots out of the 100 they have"}, {"start": 1777.36, "end": 1782.6399999999999, "text": " and the reason being those are i think huge they say some of here are huge horizontal boxes or"}, {"start": 1782.6399999999999, "end": 1788.6399999999999, "text": " something so red to large horizontal boxes and it seems that the MSCoco data set just has a bunch of"}, {"start": 1788.6399999999999, "end": 1793.6799999999998, "text": " those objects which span the whole image horizontally so that's why all of the all of these"}, {"start": 1793.6799999999998, "end": 1799.84, "text": " uh prediction slots or tokens i'll learn how to kind of focus and attend to those okay that's"}, {"start": 1799.84, "end": 1805.52, "text": " that's pretty much it um let me see i'm going to briefly uh just tell you about this uh they also"}, {"start": 1805.52, "end": 1813.6, "text": " showed that just by simply modifying the data pipeline uh by adding this this this additional"}, {"start": 1813.6, "end": 1819.36, "text": " like segmentation head they can also do really well on this panoptic segmentation task and the"}, {"start": 1819.36, "end": 1824.56, "text": " idea is they just do regular object detection training of the data pipeline so that remains"}, {"start": 1824.56, "end": 1830.32, "text": " the same and then they just append the head and they trend that they freeze the data and they just"}, {"start": 1830.32, "end": 1834.16, "text": " train the the head itself and they showed some nice results you can see here"}, {"start": 1834.16, "end": 1840.4, "text": " like segmentation of this giraffe image and boss etc yeah they just showed that this works really"}, {"start": 1840.4, "end": 1846.5600000000002, "text": " well here in this table comparing it to other baselines like this panoptic fpn etc and they just"}, {"start": 1846.5600000000002, "end": 1852.96, "text": " report nice results there as well so that's pretty much it it's exciting uh so the main two components"}, {"start": 1852.96, "end": 1861.68, "text": " again are using transformers and learning end-to-end using this this match plus hungarian loss logic and"}, {"start": 1861.68, "end": 1869.6000000000001, "text": " that's it if you if you like this video share it out subscribe to this channel and until next time"}, {"start": 1869.6, "end": 1892.7199999999998, "text": " bye bye"}]