DeepMind DetCon: Efficient Visual Pretraining with Contrastive Detection | Paper Explained
📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video, I cover DetCon: Efficient Visual Pretraining with Contrastive Detection - a novel self-supervised method that achieves SOTA results on various transfer learning tasks. 

The main idea is to add the semantic segmentation information into the contrastive objective (they used various heuristics to obtain semantic segmentation information in an unsupervised fashion).

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2103.10957
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 News: Discord and AI newsletter!
01:24 Self-supervised learning, BYOL, and SimCLR
03:50 DetCon method overview
11:35 Semantic segmentation heuristics
12:45 Overview of BYOL and SimCLR
20:15 Results
24:10 Impact of segmentation heuristics
27:20 Outro
28:10 Turn on the notification bell, much love!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

TODO:
Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR ML PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#detcon #selfsupervised #contrastive
 What's up guys, I just came back from vacation and in this video I'm covering efficient visual pre-training with contrastive detection or DATCON for short by the DeepMind team, but before I even dig into the paper I want to announce a couple of news I have for you So the first news is I just created a brand new Discord server so that means we'll be able to drive the engagement in this community so much more and you'll be able to Get answers to your questions without relying on me answering every single question on YouTube and everything and all the goodies the discord brings with it Also, you'll be able to suggest a repost I should do the code walkthroughs off because I want to be more bullish going forward doing those like hands-on coding videos and Hopefully you'll find that useful Anyways sign up for the discord. I'll just link it down in the description. So check it out So the second news is I'm creating a new monthly AI newsletter Where I'm gonna cover like the hottest the latest and greatest things that happened in the field of AI On a monthly basis and I'm gonna keep it high level. So that means it will be suitable for engineers Researchers and entrepreneurs as well. So compared to these videos where I'm in a zoom-in state in a particular paper there I'm gonna give you a brief understanding of what happened over the last month if you want to just quickly catch up with the things that Happened That's what this newsletter is gonna be all about and again just sign up for the newsletter I'm gonna link it down in the description. So having said that let me jump back into the paper and See what's happening here. So that con is another self supervised learning paper and it's just a Basically piggybacks off of dual and seem clear research with initial additional idea of including the segmentation mask information in order to create those representations representations, okay, so so let's start from the main motivation Self-supervised pre training has been shown to yield powerful representation for transfer learning these performance gains come at a large computational cost However with state-of-the-art methods requiring an order of magnitude more computation than supervised pre training So the trade-off here is fairly clear. So on one hand you have Like you don't need like expensive labeling annotation teams So you can kind of save up on the cost like there But on the other hand you have to invest in more like computational So your infrastructure your compute has will rise with some of the older methods so that's the trade-off you you kind of have to accept and You're still not going to approach the the representations that you get from from having super from from applying a supervised Paradigm, but like SSL is definitely getting closer and this paper is right like just in the right direction. I'd say Okay, so as you can see here on the x-axis We have like a number of epochs on image net So we're training these different methods on image and and they are then fine tuning them on cocoa and kind of reporting the Detection accuracy and you can see that using sim clear which is one of the older self supervised learning methods We can see that the performance is nowhere near as good as with a supervised method and then we can see that this novel method So that con actually surpasses Even the supervised method in 5x less computations You need like around 200 epochs instead of thousand epochs to achieve higher Accuracy on the cocoa detection desk. So I already mentioned that or I think I mentioned that Basically this paper piggybacks off of sim clear and Buell and I'm going to dig a bit deeper into those two But like first of all, let me kind of show you how this whole method works So here it is and if you read any of those older papers like moco sim clear Buell Whatever, you'll be fairly familiar with this similar pipeline So what we do is we have an image And then we do like two different augmentations So we'll have something like T Augmentation T and augmentation T prime and as you can see here So here we had some crop the elephant is now cropped and zoomed in and here we had some other types of augmentations So what we do next is We just apply encoder which is your regular CNN They've used ResNet across this paper to gain to get these Convolutional features so this here is just a spatial extent of the features and there is a third dimension here not shown Which can be maybe like 1k dimensions there okay, so now the important thing to notice here is that they are using these segmentation masks and That's that's so that's the difference compared to Buell. That's a difference compared to other SSL methods that we've seen so far and We're gonna see how they actually calculate the masks but for now just let's let's treat that as a black box and Now that the whole idea of the paper is the following so you have you have an elephant here as you can see it here And you can see these red dots represent the feature vectors that correspond to that spatial extent of the image here We have more red dots because the elephant occupies nearly the whole image and so now in order to form representations what they do is they take those masks so they take the information from the mask so with that like Segmentation mask information they can pull exactly those feature vectors which correspond to that red mask or to the elephant mask And so that means effectively you'll use you'll take these like as I said, these are maybe 1k Dimensional feature vectors you'll just take them and you'll do average over all of those 5 vectors in this case here or whatever the number of array vectors here is and that's how you get the final Representation here. Okay, you do that the same procedure for all of the other masks So because this image had different augmentations compared to this one here We still have this yellow object whatever this is it like a human so that means we have that vector here Whereas we don't have it here and now the whole idea is the following We just want to make sure that the red vectors get have a representation such that they are closer together in the vector space Whereas we want to make sure to push away the red from the blue vectors here And again, we want to pull the the the blue vectors and we want to because we only have yellow here But we don't have yellow here. We want to just push away the blue vector and the red vector So that's the main idea using these segmentation masks They have a richer learning signal and they are able to do this fine-grained push and pull Standard contrastive learning scheme. Okay, so let's dig into some formulas here To make this a bit clearer. So here is just a formulaic representation of the thing I just explained so basically M is your segmentation mask and it's gonna be a big binary mask So basically what M will do is the following so M will be something like this So for a red mask, it's gonna have ones here. So these will be ones ones ones ones Okay, and all of the other ones will be zeros So these will be zeros, okay and Now as you can see here, so H is just a feature vector that came from the CNN So that means so these here are H vectors. Okay, so and as you can see for every Particular slot so for this one The mask this binary mask will decide whether we want to use that feature vector behind it So whether we want to use the feature vector or not, okay, and That's simply described here as this So when when M equals 1 that means that the mask is active that means we're gonna use that feature vector and kind of sum them up and then divide by the number of Like active units. So let me kind of clarify that so here we have a 5 times 1 So there's gonna be 5 so that means we're gonna take these five vectors add them together and divide by 5 Which is basically what I already explained and that's average pooling across those across those vectors. So that's how we form the representations So that's this part now. The second part is as I said, they are piggybacking piggybacking off of Be all and simpler and so in order to like having formed those HM representations We now need to form these VM representations before we actually apply this contrastive detection objective Which I'll explain in a second. Okay, so Again, these here are only HM s and now we have to form VMS before we actually applied the objective And now I'm gonna briefly explain so as I said This here these formulas correspond to seem clear and these ones here correspond to be all and then For now, let's just treat it as a black box. We somehow like a calculate these VMS and What we do next up is as I said, we do this push and pull This this pooling and pushing away of these feature vectors So once we have those VMS we can then do the thing I just explained pictorially here So this this pushing away of different of vectors that correspond to different objects and pooling together The vectors that correspond to the same object in the image. So this is just a formulaically described here basically as you can see so VM and this VM prime just correspond to the same mask so M is for example red mask and Prime just means different augmentation. So this means basically the following So we will let's say this is just for the sake of argument. This is now VM just ignore It's it's actually HM, but let's say we pre process it using this fuel or sim clear and now we have VM here So what we want to do is so this will be the VM and we want to make sure that this one is As close as possible to this one. So those are two vectors. We want to make sure that the dot product between those two is Super high that means they are similar on the other hand. We want to make sure that this so this red vector is Pushed away from this blue vector and same for the other vectors So that's the whole that's the whole idea and that's kind of captured in this formula here so we want to make sure that the Vectors corresponding to the same mask So for example the the red mask the elephant mask are similar where we want to make sure that these negative ones are Pushed away. So that means this will get negative that means Exponential function raised to like a big negative number goes to zero. This will roughly go to zero Let's say and that means that that's how this loss will get minimized because minus log looks something like this So this is your minus log function which means you want to push this value here to one and that's how the loss is minimized and By doing that but that so that's a numerical intuition But what happens in that process is that as I said the red vectors or the the vectors corresponding to the same mask Get clustered together. Whereas all of the other vectors get kind of Like pushed away from each other and that's the whole idea Nothing, nothing fancy there Now briefly before I start like giving you a some overview of Buell and SimClear Let me show you the segmentation masks and what I've used. So Obviously, you don't know it wouldn't make sense to have a self-surprise learning method if you Presuppose that you have semantic segmentation masks because those require obviously labeling from humans So what I've done is they tried a couple of heuristics so some of these heuristics for calculating the segmentation masks are super simple and Like primitive like this one's the spatial heuristic where you basically cluster and assume that the neighboring pixels Belong to a specific object and so they tried this one. They also tried this FH algorithm heuristic this MCG and those are not that important Basically, they're just some more advanced heuristics compared to the spatial heuristic obviously and finally we they have they used human annotated segmentation masks And they later showed some intuitive results at having better segmentation masks like this one Actually improved the methods by by quite a lot. Okay, so those are the segmentation masks now Let me slowly dig into these Buell and SimClear methods Again, the ideas of many of these self supervised learning Papers are fairly similar So what I do here again is you have an image and you do you apply two different distinct Augmentations and that's how we get view t and view t prime of the same image So that would be something like this, right? So like you had an elephant this picture here and you get two different images. So that's that's the idea Okay, so once you have those what Buell does is it does not use negative samples So that's a that's the that was the novel thing that Buell did and it managed to actually create Like high quality representations without them collapsing and I'm gonna slowly digest what that means but stick with me for now so As I said, you have a view here you apply an encoder which is again usually your CNN you get some representation Then they have this G theta which is basically like a shallow MLP network Which is a like a projection head you get this projection representation and then they apply additionally this this additional MLP Which actually was one of the important ingredients to avoid the collapse So this a symmetry between the upper branch and the lower branch was important fact in in the important piece of information in Buell Okay, so On the other hand the lower branch is just constructed basically from the upper branch by using something called Exponentially moving average so that means you just take snapshots of the upper network in time and you use the exponential Logic to kind of average those out and that's how you form the these f-x i or whatever the discrete letter is pronounced like and Then you you kind of propagate and you get this final representation and the whole goal will be to After you do some L2 normalization. So whatever this vector is, let's call it V What they do is they first normalize it using L2 So they just find the L2 of that vector denormalize it by dividing it by the L2 norm and then they want to make sure that these two representations Basically by doing this MSC loss mean square error They want to ensure that these vectors are the same and so intuitively what that computational thing does is it forces the network? To learn how to abstract the important parts of the image So no matter the augmentations you want to extract something that's inherently present in those images and by learning that you learn valuable Representations which are agnostic to different augmentations. You can still capture and understand understand what's in the image so that's the whole point of y'all and They showed empirically that by doing this they avoid the collapse. So why would the collapse happen? Well, it's simple like the easiest most trivial way to kind of make sure you are minimizing this this loss function is To always output a constant representation. So no matter the views and give me an image whatever the images Whatever the augmentations are after you pass them through F theta. You'll get some constant representation Let's call it like V C like constant vector and you get V C here as well. And that means basically Not here, but like here at the output you get constant representations and that means that this is basically Approaching zero without you learning any interesting representations. So you'll avoid the disc collapse and back then when the paper came out like there they didn't have like Super strong theoretical understanding of why that is but some recent paper actually I kind of explained this a bit better I forgot the name of the paper. But yeah now sim clear is Similar but different it does so I'd say the main one of the main differences is that it uses negative sampling So what's in clear does is the following again? We have t and t prime so we apply different augmentations to the same image and this time they apply the same encoder In both branches and then they apply the same projection head in both branches and now the whole idea here is to make sure that the Augmentations belonging to the same image are similar. So those are printed representations are close together in the vector space Whereas like whereas representations from different images are further away from each other So let me try and kind of visualize that a little bit So this is your image and you have a batch of images to apply You apply the first augmentation and you get image t1 and you apply the second augmentation to the same image you get t1 Prime now you have a second image and you again apply augmentation and you get Image t2 and you get t2 prime and you do this Obviously n times because you have n images in your batch and now the whole point will be the following You take the representation you get from this t1 image after passing it through f and g And you want to make sure that that representation is close to this one because they belong to the same image And then you want to make sure that it's far away from these representations from this one from this one as well You want to pull it? You want to push it away from those representations and the same for every single image in the batch you're gonna Push them away and the only thing that's getting pulled together are those two images here So that's the whole idea of sim clear. So that's just kind of formulaically Showed here you basically have as I said, so similarity usually dot product between those representations There's some temperature coefficient But a whole point at the end is to kind of do what what I showed you here is just symbolic representation of the thing I just explained you here. Okay having understand that let me now go back to the Method and here we are. So again, here is the here is the sim clear method so you can now see that HM is the representation that comes from the F so from the encoder and then they just applied the projection head and That's it. That's that's everything you need to form representation for sim clear and then you apply this this this novel this novel object objective function to get representations for Buell they additionally just add this this head which I Mentioned which gives you this as symmetry in the in the pipeline and obviously they also use this a network So that's the network form from the exponential moving average. Okay, so now you can kind of hopefully ground these these Formulas to the dual and sim clear papers and again once you have the representations those VMs and VM primes You can do the logic I explained a couple minutes ago where you're forcing the vectors for Where you're forcing the vectors that correspond to the same masks to be closer together Whereas you're pushing away the vectors belonging to different masks And whereas masks correspond hopefully to different objects and that depends on the quality of your segmentation masks and they did some ablations on How those different so how these different algorithms? Affect the performance of the method and we'll see that in a couple minutes. Okay Let's see the results finally of this stat-con method and as you can see here So having pre training on the image net and then fine-tuning different tasks like cocoa instance segmentation Pascal semantic segmentation cityscapes Semantic segmentation and NYU depth estimation you can see in all of these four tasks, which are fairly different Like the that come method performs way better compared to the supervised and sim clear baselines So I think let's let's focus for now on one chart and then I'll tell you some interesting things I see here. So first things first is You can see here that sim clear after certain amount of given enough computation like Surpasses the performance of supervised like paradigm and that's something we saw in the beginning of this paper So given enough compute that's a trade-off you're gonna eventually get there, but you need bunch of compute, right? Whereas here you can see focusing on that con that like with After only 200 epochs, you're already better compared to these two paradigms and you have better performance Okay, so the the the same thing applies pretty much for all of the other tasks I won't be digging into every single chart here. But the interesting thing here to notice is that supervised like paradigm Completely fails for these two tasks like depth estimation and semantic segmentation Which means that the pre training supervised pre training on image net is not informative enough so the features are not informative enough for these particular tasks where it's in clear and that can seem to generalize better to different tasks, which is Another plus and bonus point for applying SSL because you get more generalization out of the method. Okay I'll skip this table. It just has like a numeric representation of the thing We just saw here So basically the the main point they're trying to drive here is that with 10x less computation They can achieve performance that's much better than all the other baselines So here you can see with only 100 approximately 100 epochs It's already better than seem clear which is way better that supervised learning on this city scape semantic segmentation desk So that's the whole point. Okay going forward. Let's see. What else is there Okay here they compare like That con again against various different like baselines not only Buell and seem clear both so moco Supervised suave and other methods and they just showed that it consistently outperforms all of those baselines So one thing they've worried about is that maybe scaling the number of parameters will kind of give us something qualitatively different behavior with that con compared to the previous baselines and this basically just confirms that this method does scale with with scale and Like the performance does not change and it's still all performed So that's con the blue one still outperforms all of the other baselines like we all seem clear and supervised Baseline on these cocoa detection and cocoa instance segmentation tasks. Okay Moving further they even showed that this recently published methods here Which has billion which used billion images for for pre training and has way more parameters It's still like the the dead con outperforms those methods Which is a nice result and they kind of mentioned that it's not super comparable because Sierra uses The data that's here uses is way more Noisy, it's kind of not a fair comparison, but still it's an impressive result Okay, let's see. What else interesting is here They showed how using different Segmentation mass so different heuristics which I briefly described before Improves and changes the performance. So as you can see here using those spatial like heuristics, so that's this one Let me just go briefly up there So this is a spatial heuristic and here we have so this upper here is one by one here. We have two by two Grid and here we have four by four. Okay, so that will hopefully help you understand this chart a bit better What these numbers mean so here is one by one? Here is using two by two ten by ten and five by five seems to be working the best on this particular Like desk and I guess this is not that surprising. So the better the Semantic segmentation you do initially so FH obviously has better as you can see on the x-axis this means that the overlap between the The masks that this method found is way way higher To the GT which is obviously one because by default GT overlaps completely with itself. That's why we have one here and these methods have even smaller overlap Which means they are of lower quality So that basically means the higher the quality of your segmentation masks the higher the detection accuracy and that's kind of I guess intuitive But it's nice. You have a chart like proving exactly that. Okay They did a couple more oblations of pre-training this time on cocoa instead of image net and they still have they still Outperform the supervised and the same clear like baselines. That's that that's the point there. Finally, they showed that using GT labels which kind of makes sense Helps helps them even in increasing the the pre-training image resolution and always testing in on this resolution Consistently improves the performance when you're using GT masks Whereas when you're using FH the performance does not improve with resolution which I guess makes sense They mentioned that somewhere here in the in the discussion portion of the paper. So, okay this part How can this be so one interpretation is that other images give us clean negative examples? Because images in cocoa depict different scenes However, it appears that negatives from the same image provide a stronger learning signal blah blah blah I were not pushing features from the same object apart Positives from the same image are also at least as good as those across segmentations if again, they are clean We are not pulling together features from different objects So basically if you have if you don't have perfect cementation mass that means you'll have overlap between classes Which means sometimes you're gonna be pushing away Two feature vectors even though there is an overlap where those two vectors are formed from the same object So that's something not something you want to do. You want to pull push away different objects You want to pull together the same object and that's all like a hand wavy explanation of why you need better better cement cementing better cementing segmentation masks Okay wrapping up this paper The main idea again is usage of these segmentation masks in order to form a richer training signal So whereas Sinclair will in this example just we won't have any cementation masks We won't have any semantic segmentation mask, which means Sinclair will just be pulling together these two because they belong to the same image Whereas here we have a much richer signal because of the presence of semantic segmentation So that's pretty much it for this video if you have any suggestion For which repo I should do a code walkthrough off. Let me know down in the comments So I was thinking maybe dyno or some of those newer papers that came out So if you have any suggestion Please let me know down in the comments and I definitely read those comments and you can influence the next video other than that If you found this video useful consider subscribing and hit that notification Button so that you can get notified the moment I upload a new video I've noticed that only 10% of you have the notification bell on and only 5 or 6% of you Are actually receiving those notifications because your YouTube settings are set in such manner that you are not receiving them instantaneously So yeah, definitely toggle on that notification button and until next time. Bye. Bye. I
[{"start": 0.0, "end": 6.6000000000000005, "text": " What's up guys, I just came back from vacation and in this video I'm covering efficient visual pre-training with contrastive detection or"}, {"start": 7.38, "end": 12.16, "text": " DATCON for short by the DeepMind team, but before I even dig into the paper"}, {"start": 12.16, "end": 14.74, "text": " I want to announce a couple of news I have for you"}, {"start": 14.74, "end": 18.34, "text": " So the first news is I just created a brand new Discord server"}, {"start": 18.54, "end": 24.38, "text": " so that means we'll be able to drive the engagement in this community so much more and you'll be able to"}, {"start": 24.38, "end": 31.88, "text": " Get answers to your questions without relying on me answering every single question on YouTube and everything and all the goodies the discord brings with it"}, {"start": 32.14, "end": 34.42, "text": " Also, you'll be able to suggest a repost"}, {"start": 34.42, "end": 41.5, "text": " I should do the code walkthroughs off because I want to be more bullish going forward doing those like hands-on"}, {"start": 42.08, "end": 43.78, "text": " coding videos and"}, {"start": 43.78, "end": 45.5, "text": " Hopefully you'll find that useful"}, {"start": 45.5, "end": 49.54, "text": " Anyways sign up for the discord. I'll just link it down in the description. So check it out"}, {"start": 49.54, "end": 53.3, "text": " So the second news is I'm creating a new monthly AI newsletter"}, {"start": 53.3, "end": 58.98, "text": " Where I'm gonna cover like the hottest the latest and greatest things that happened in the field of AI"}, {"start": 59.339999999999996, "end": 64.3, "text": " On a monthly basis and I'm gonna keep it high level. So that means it will be suitable for engineers"}, {"start": 64.78, "end": 71.24, "text": " Researchers and entrepreneurs as well. So compared to these videos where I'm in a zoom-in state in a particular paper there"}, {"start": 71.24, "end": 76.97999999999999, "text": " I'm gonna give you a brief understanding of what happened over the last month if you want to just quickly catch up with the things that"}, {"start": 76.97999999999999, "end": 77.97999999999999, "text": " Happened"}, {"start": 77.97999999999999, "end": 82.14, "text": " That's what this newsletter is gonna be all about and again just sign up for the newsletter"}, {"start": 82.14, "end": 87.46000000000001, "text": " I'm gonna link it down in the description. So having said that let me jump back into the paper and"}, {"start": 88.34, "end": 90.5, "text": " See what's happening here. So"}, {"start": 91.22, "end": 95.82, "text": " that con is another self supervised learning paper and"}, {"start": 96.5, "end": 98.5, "text": " it's just a"}, {"start": 98.58, "end": 106.54, "text": " Basically piggybacks off of dual and seem clear research with initial additional idea of including the segmentation mask"}, {"start": 106.82, "end": 108.94, "text": " information in order to create those representations"}, {"start": 108.94, "end": 113.14, "text": " representations, okay, so so let's start from the main motivation"}, {"start": 113.94, "end": 121.1, "text": " Self-supervised pre training has been shown to yield powerful representation for transfer learning these performance gains come at a large computational cost"}, {"start": 121.62, "end": 128.06, "text": " However with state-of-the-art methods requiring an order of magnitude more computation than supervised pre training"}, {"start": 128.06, "end": 132.06, "text": " So the trade-off here is fairly clear. So on one hand you have"}, {"start": 132.66, "end": 136.94, "text": " Like you don't need like expensive labeling annotation teams"}, {"start": 136.94, "end": 140.78, "text": " So you can kind of save up on the cost like there"}, {"start": 140.78, "end": 144.82, "text": " But on the other hand you have to invest in more like computational"}, {"start": 144.82, "end": 149.98, "text": " So your infrastructure your compute has will rise with some of the older methods"}, {"start": 149.98, "end": 153.14, "text": " so that's the trade-off you you kind of have to accept and"}, {"start": 153.9, "end": 161.18, "text": " You're still not going to approach the the representations that you get from from having super from from applying a supervised"}, {"start": 161.18, "end": 168.94, "text": " Paradigm, but like SSL is definitely getting closer and this paper is right like just in the right direction. I'd say"}, {"start": 169.62, "end": 174.14000000000001, "text": " Okay, so as you can see here on the x-axis"}, {"start": 174.14000000000001, "end": 176.94, "text": " We have like a number of epochs on image net"}, {"start": 176.94, "end": 183.74, "text": " So we're training these different methods on image and and they are then fine tuning them on cocoa and kind of reporting the"}, {"start": 184.06, "end": 189.14000000000001, "text": " Detection accuracy and you can see that using sim clear which is one of the older"}, {"start": 189.14, "end": 191.38, "text": " self supervised learning methods"}, {"start": 191.38, "end": 199.77999999999997, "text": " We can see that the performance is nowhere near as good as with a supervised method and then we can see that this novel method"}, {"start": 199.77999999999997, "end": 201.77999999999997, "text": " So that con actually"}, {"start": 202.1, "end": 203.22, "text": " surpasses"}, {"start": 203.22, "end": 207.7, "text": " Even the supervised method in 5x less computations"}, {"start": 207.7, "end": 212.42, "text": " You need like around 200 epochs instead of thousand epochs to achieve higher"}, {"start": 212.42, "end": 219.33999999999997, "text": " Accuracy on the cocoa detection desk. So I already mentioned that or I think I mentioned that"}, {"start": 219.89999999999998, "end": 226.66, "text": " Basically this paper piggybacks off of sim clear and Buell and I'm going to dig a bit deeper into those two"}, {"start": 226.73999999999998, "end": 230.82, "text": " But like first of all, let me kind of show you how this whole method works"}, {"start": 231.01999999999998, "end": 237.06, "text": " So here it is and if you read any of those older papers like moco sim clear Buell"}, {"start": 237.06, "end": 241.48, "text": " Whatever, you'll be fairly familiar with this similar pipeline"}, {"start": 241.48, "end": 243.79999999999998, "text": " So what we do is we have an image"}, {"start": 244.35999999999999, "end": 247.26, "text": " And then we do like two different augmentations"}, {"start": 247.26, "end": 249.16, "text": " So we'll have something like T"}, {"start": 249.16, "end": 252.89999999999998, "text": " Augmentation T and augmentation T prime and as you can see here"}, {"start": 252.89999999999998, "end": 259.84, "text": " So here we had some crop the elephant is now cropped and zoomed in and here we had some other types of augmentations"}, {"start": 260.03999999999996, "end": 262.03999999999996, "text": " So what we do next is"}, {"start": 262.03999999999996, "end": 265.15999999999997, "text": " We just apply encoder which is your regular CNN"}, {"start": 265.15999999999997, "end": 269.28, "text": " They've used ResNet across this paper to gain to get these"}, {"start": 269.28, "end": 276.44, "text": " Convolutional features so this here is just a spatial extent of the features and there is a third dimension here not shown"}, {"start": 276.55999999999995, "end": 279.23999999999995, "text": " Which can be maybe like 1k dimensions there"}, {"start": 279.23999999999995, "end": 285.4, "text": " okay, so now the important thing to notice here is that they are using these segmentation masks and"}, {"start": 286.08, "end": 293.11999999999995, "text": " That's that's so that's the difference compared to Buell. That's a difference compared to other SSL methods that we've seen so far and"}, {"start": 293.96, "end": 296.52, "text": " We're gonna see how they actually calculate the masks"}, {"start": 296.52, "end": 299.47999999999996, "text": " but for now just let's let's treat that as a black box and"}, {"start": 300.12, "end": 305.32, "text": " Now that the whole idea of the paper is the following so you have you have an elephant here as you can see it here"}, {"start": 305.79999999999995, "end": 313.4, "text": " And you can see these red dots represent the feature vectors that correspond to that spatial extent of the image here"}, {"start": 313.4, "end": 315.56, "text": " We have more red dots because the elephant"}, {"start": 316.47999999999996, "end": 321.64, "text": " occupies nearly the whole image and so now in order to form"}, {"start": 321.64, "end": 329.32, "text": " representations what they do is they take those masks so they take the information from the mask so with that like"}, {"start": 329.68, "end": 337.56, "text": " Segmentation mask information they can pull exactly those feature vectors which correspond to that red mask or to the elephant mask"}, {"start": 337.56, "end": 343.44, "text": " And so that means effectively you'll use you'll take these like as I said, these are maybe 1k"}, {"start": 344.08, "end": 348.2, "text": " Dimensional feature vectors you'll just take them and you'll do average over all of those"}, {"start": 348.2, "end": 355.44, "text": " 5 vectors in this case here or whatever the number of array vectors here is and that's how you get the final"}, {"start": 356.03999999999996, "end": 360.52, "text": " Representation here. Okay, you do that the same procedure for all of the other masks"}, {"start": 360.52, "end": 364.44, "text": " So because this image had different augmentations compared to this one here"}, {"start": 364.44, "end": 370.56, "text": " We still have this yellow object whatever this is it like a human so that means we have that vector here"}, {"start": 370.56, "end": 374.59999999999997, "text": " Whereas we don't have it here and now the whole idea is the following"}, {"start": 374.6, "end": 382.68, "text": " We just want to make sure that the red vectors get have a representation such that they are closer together in the vector space"}, {"start": 383.16, "end": 388.36, "text": " Whereas we want to make sure to push away the red from the blue vectors here"}, {"start": 388.36, "end": 395.3, "text": " And again, we want to pull the the the blue vectors and we want to because we only have yellow here"}, {"start": 395.3, "end": 400.46000000000004, "text": " But we don't have yellow here. We want to just push away the blue vector and the red vector"}, {"start": 400.46000000000004, "end": 403.16, "text": " So that's the main idea using these segmentation masks"}, {"start": 403.16, "end": 408.56, "text": " They have a richer learning signal and they are able to do this fine-grained push and pull"}, {"start": 409.48, "end": 414.94, "text": " Standard contrastive learning scheme. Okay, so let's dig into some formulas here"}, {"start": 415.68, "end": 421.64000000000004, "text": " To make this a bit clearer. So here is just a formulaic representation of the thing"}, {"start": 421.64000000000004, "end": 427.42, "text": " I just explained so basically M is your segmentation mask and it's gonna be a big binary mask"}, {"start": 427.42, "end": 431.20000000000005, "text": " So basically what M will do is the following so M will be something like this"}, {"start": 431.2, "end": 436.82, "text": " So for a red mask, it's gonna have ones here. So these will be ones ones ones ones"}, {"start": 436.82, "end": 439.47999999999996, "text": " Okay, and all of the other ones will be zeros"}, {"start": 441.2, "end": 443.2, "text": " So these will be zeros, okay and"}, {"start": 443.52, "end": 449.08, "text": " Now as you can see here, so H is just a feature vector that came from the CNN"}, {"start": 449.08, "end": 456.24, "text": " So that means so these here are H vectors. Okay, so and as you can see for every"}, {"start": 457.08, "end": 459.08, "text": " Particular slot so for this one"}, {"start": 459.08, "end": 465.52, "text": " The mask this binary mask will decide whether we want to use that feature vector behind it"}, {"start": 465.68, "end": 469.24, "text": " So whether we want to use the feature vector or not, okay, and"}, {"start": 469.88, "end": 471.59999999999997, "text": " That's simply described here as this"}, {"start": 471.59999999999997, "end": 475.68, "text": " So when when M equals 1 that means that the mask is active"}, {"start": 475.68, "end": 481.32, "text": " that means we're gonna use that feature vector and kind of sum them up and then divide by the number of"}, {"start": 482.12, "end": 487.62, "text": " Like active units. So let me kind of clarify that so here we have a 5 times 1"}, {"start": 487.62, "end": 492.96, "text": " So there's gonna be 5 so that means we're gonna take these five vectors add them together and divide by 5"}, {"start": 492.96, "end": 500.58, "text": " Which is basically what I already explained and that's average pooling across those across those vectors. So that's how we form the representations"}, {"start": 500.58, "end": 506.38, "text": " So that's this part now. The second part is as I said, they are piggybacking piggybacking off of"}, {"start": 507.52, "end": 514.0600000000001, "text": " Be all and simpler and so in order to like having formed those HM representations"}, {"start": 514.06, "end": 520.8199999999999, "text": " We now need to form these VM representations before we actually apply this contrastive detection objective"}, {"start": 520.8199999999999, "end": 523.0, "text": " Which I'll explain in a second. Okay, so"}, {"start": 524.0799999999999, "end": 531.64, "text": " Again, these here are only HM s and now we have to form VMS before we actually applied the objective"}, {"start": 531.64, "end": 534.78, "text": " And now I'm gonna briefly explain so as I said"}, {"start": 535.4599999999999, "end": 541.38, "text": " This here these formulas correspond to seem clear and these ones here correspond to be all and"}, {"start": 541.38, "end": 543.18, "text": " then"}, {"start": 543.18, "end": 549.14, "text": " For now, let's just treat it as a black box. We somehow like a calculate these VMS and"}, {"start": 549.82, "end": 554.18, "text": " What we do next up is as I said, we do this push and pull"}, {"start": 554.58, "end": 558.72, "text": " This this pooling and pushing away of these feature vectors"}, {"start": 558.72, "end": 564.1, "text": " So once we have those VMS we can then do the thing I just explained pictorially here"}, {"start": 564.1, "end": 571.4200000000001, "text": " So this this pushing away of different of vectors that correspond to different objects and pooling together"}, {"start": 571.84, "end": 578.38, "text": " The vectors that correspond to the same object in the image. So this is just a formulaically described here"}, {"start": 578.74, "end": 581.98, "text": " basically as you can see so VM and"}, {"start": 582.5, "end": 584.5, "text": " this VM prime"}, {"start": 584.94, "end": 589.9, "text": " just correspond to the same mask so M is for example red mask and"}, {"start": 589.9, "end": 594.3, "text": " Prime just means different augmentation. So this means basically the following"}, {"start": 594.3, "end": 599.26, "text": " So we will let's say this is just for the sake of argument. This is now VM just ignore"}, {"start": 599.26, "end": 605.66, "text": " It's it's actually HM, but let's say we pre process it using this fuel or sim clear and now we have VM here"}, {"start": 605.74, "end": 610.86, "text": " So what we want to do is so this will be the VM and we want to make sure that this one is"}, {"start": 611.02, "end": 617.5, "text": " As close as possible to this one. So those are two vectors. We want to make sure that the dot product between those two is"}, {"start": 617.5, "end": 622.86, "text": " Super high that means they are similar on the other hand. We want to make sure that this"}, {"start": 623.62, "end": 625.54, "text": " so this red vector is"}, {"start": 625.54, "end": 629.54, "text": " Pushed away from this blue vector and same for the other vectors"}, {"start": 629.54, "end": 634.42, "text": " So that's the whole that's the whole idea and that's kind of captured in this formula here"}, {"start": 634.42, "end": 636.88, "text": " so we want to make sure that the"}, {"start": 639.22, "end": 640.86, "text": " Vectors corresponding to the same mask"}, {"start": 640.86, "end": 646.54, "text": " So for example the the red mask the elephant mask are similar where we want to make sure that these negative ones are"}, {"start": 646.54, "end": 651.36, "text": " Pushed away. So that means this will get negative that means"}, {"start": 652.3, "end": 657.4599999999999, "text": " Exponential function raised to like a big negative number goes to zero. This will roughly go to zero"}, {"start": 657.4599999999999, "end": 663.98, "text": " Let's say and that means that that's how this loss will get minimized because minus log looks something like this"}, {"start": 664.66, "end": 666.66, "text": " So this is your minus log function"}, {"start": 666.8199999999999, "end": 673.14, "text": " which means you want to push this value here to one and that's how the loss is minimized and"}, {"start": 673.14, "end": 676.66, "text": " By doing that but that so that's a numerical intuition"}, {"start": 676.66, "end": 683.3, "text": " But what happens in that process is that as I said the red vectors or the the vectors corresponding to the same mask"}, {"start": 683.34, "end": 686.74, "text": " Get clustered together. Whereas all of the other vectors get kind of"}, {"start": 687.38, "end": 691.9, "text": " Like pushed away from each other and that's the whole idea"}, {"start": 693.1, "end": 695.1, "text": " Nothing, nothing fancy there"}, {"start": 695.54, "end": 700.7, "text": " Now briefly before I start like giving you a some overview of Buell and SimClear"}, {"start": 700.7, "end": 704.5400000000001, "text": " Let me show you the segmentation masks and what I've used. So"}, {"start": 705.26, "end": 710.7800000000001, "text": " Obviously, you don't know it wouldn't make sense to have a self-surprise learning method if you"}, {"start": 711.1800000000001, "end": 716.58, "text": " Presuppose that you have semantic segmentation masks because those require obviously labeling from humans"}, {"start": 716.74, "end": 719.86, "text": " So what I've done is they tried a couple of heuristics"}, {"start": 720.0200000000001, "end": 724.9000000000001, "text": " so some of these heuristics for calculating the segmentation masks are super simple and"}, {"start": 724.9, "end": 732.62, "text": " Like primitive like this one's the spatial heuristic where you basically cluster and assume that the neighboring pixels"}, {"start": 732.8199999999999, "end": 737.74, "text": " Belong to a specific object and so they tried this one. They also tried this"}, {"start": 738.42, "end": 743.62, "text": " FH algorithm heuristic this MCG and those are not that important"}, {"start": 744.62, "end": 752.4, "text": " Basically, they're just some more advanced heuristics compared to the spatial heuristic obviously and finally we they have they used human annotated segmentation masks"}, {"start": 752.4, "end": 757.1999999999999, "text": " And they later showed some intuitive results at having better segmentation masks like this one"}, {"start": 758.12, "end": 763.9599999999999, "text": " Actually improved the methods by by quite a lot. Okay, so those are the segmentation masks now"}, {"start": 763.9599999999999, "end": 767.42, "text": " Let me slowly dig into these Buell and SimClear methods"}, {"start": 768.24, "end": 772.54, "text": " Again, the ideas of many of these self supervised learning"}, {"start": 773.1999999999999, "end": 775.1999999999999, "text": " Papers are fairly similar"}, {"start": 775.2, "end": 782.24, "text": " So what I do here again is you have an image and you do you apply two different distinct"}, {"start": 782.6800000000001, "end": 787.6, "text": " Augmentations and that's how we get view t and view t prime of the same image"}, {"start": 788.08, "end": 790.08, "text": " So that would be something like this, right?"}, {"start": 790.08, "end": 795.08, "text": " So like you had an elephant this picture here and you get two different images. So that's that's the idea"}, {"start": 795.24, "end": 800.72, "text": " Okay, so once you have those what Buell does is it does not use negative samples"}, {"start": 800.72, "end": 807.1600000000001, "text": " So that's a that's the that was the novel thing that Buell did and it managed to actually create"}, {"start": 807.4, "end": 813.74, "text": " Like high quality representations without them collapsing and I'm gonna slowly digest what that means but stick with me for now"}, {"start": 814.12, "end": 815.8000000000001, "text": " so"}, {"start": 815.8000000000001, "end": 823.12, "text": " As I said, you have a view here you apply an encoder which is again usually your CNN you get some representation"}, {"start": 823.12, "end": 828.9, "text": " Then they have this G theta which is basically like a shallow MLP network"}, {"start": 828.9, "end": 835.68, "text": " Which is a like a projection head you get this projection representation and then they apply additionally this this additional MLP"}, {"start": 836.0799999999999, "end": 841.04, "text": " Which actually was one of the important ingredients to avoid the collapse"}, {"start": 841.04, "end": 848.72, "text": " So this a symmetry between the upper branch and the lower branch was important fact in in the important piece of information in Buell"}, {"start": 848.88, "end": 850.88, "text": " Okay, so"}, {"start": 850.9599999999999, "end": 857.0, "text": " On the other hand the lower branch is just constructed basically from the upper branch by using something called"}, {"start": 857.0, "end": 865.0, "text": " Exponentially moving average so that means you just take snapshots of the upper network in time and you use the exponential"}, {"start": 865.2, "end": 869.04, "text": " Logic to kind of average those out and that's how you form the these"}, {"start": 869.84, "end": 873.64, "text": " f-x i or whatever the discrete letter is pronounced like and"}, {"start": 874.6, "end": 879.96, "text": " Then you you kind of propagate and you get this final representation and the whole goal will be to"}, {"start": 880.52, "end": 885.04, "text": " After you do some L2 normalization. So whatever this vector is, let's call it V"}, {"start": 885.04, "end": 888.3199999999999, "text": " What they do is they first normalize it using L2"}, {"start": 888.3199999999999, "end": 895.48, "text": " So they just find the L2 of that vector denormalize it by dividing it by the L2 norm and then they want to make sure that"}, {"start": 895.48, "end": 897.48, "text": " these two representations"}, {"start": 898.1999999999999, "end": 900.9599999999999, "text": " Basically by doing this MSC loss mean square error"}, {"start": 901.8, "end": 910.0, "text": " They want to ensure that these vectors are the same and so intuitively what that computational thing does is it forces the network?"}, {"start": 910.0, "end": 913.88, "text": " To learn how to abstract the important parts of the image"}, {"start": 913.88, "end": 922.16, "text": " So no matter the augmentations you want to extract something that's inherently present in those images and by learning that you learn valuable"}, {"start": 922.76, "end": 930.2, "text": " Representations which are agnostic to different augmentations. You can still capture and understand understand what's in the image"}, {"start": 930.24, "end": 932.32, "text": " so that's the whole point of y'all and"}, {"start": 932.84, "end": 939.04, "text": " They showed empirically that by doing this they avoid the collapse. So why would the collapse happen?"}, {"start": 939.04, "end": 946.0799999999999, "text": " Well, it's simple like the easiest most trivial way to kind of make sure you are minimizing this this loss function is"}, {"start": 946.36, "end": 952.52, "text": " To always output a constant representation. So no matter the views and give me an image whatever the images"}, {"start": 952.64, "end": 958.4, "text": " Whatever the augmentations are after you pass them through F theta. You'll get some constant representation"}, {"start": 958.4, "end": 965.3199999999999, "text": " Let's call it like V C like constant vector and you get V C here as well. And that means basically"}, {"start": 965.32, "end": 972.1600000000001, "text": " Not here, but like here at the output you get constant representations and that means that this is basically"}, {"start": 973.1600000000001, "end": 980.84, "text": " Approaching zero without you learning any interesting representations. So you'll avoid the disc collapse and"}, {"start": 981.2800000000001, "end": 985.96, "text": " back then when the paper came out like there they didn't have like"}, {"start": 986.96, "end": 992.6, "text": " Super strong theoretical understanding of why that is but some recent paper actually I kind of explained this a bit better"}, {"start": 992.6, "end": 995.08, "text": " I forgot the name of the paper. But yeah"}, {"start": 996.0, "end": 998.0, "text": " now sim clear is"}, {"start": 998.32, "end": 1004.6, "text": " Similar but different it does so I'd say the main one of the main differences is that it uses negative sampling"}, {"start": 1004.6, "end": 1006.9200000000001, "text": " So what's in clear does is the following again?"}, {"start": 1006.9200000000001, "end": 1013.16, "text": " We have t and t prime so we apply different augmentations to the same image and this time they apply the same encoder"}, {"start": 1013.4, "end": 1018.6600000000001, "text": " In both branches and then they apply the same projection head in both branches"}, {"start": 1018.66, "end": 1024.12, "text": " and now the whole idea here is to make sure that the"}, {"start": 1024.6, "end": 1030.68, "text": " Augmentations belonging to the same image are similar. So those are printed representations are close together in the vector space"}, {"start": 1031.08, "end": 1035.84, "text": " Whereas like whereas representations from different images are further away from each other"}, {"start": 1035.92, "end": 1039.76, "text": " So let me try and kind of visualize that a little bit"}, {"start": 1039.76, "end": 1042.8799999999999, "text": " So this is your image and you have a batch of images to apply"}, {"start": 1042.88, "end": 1049.96, "text": " You apply the first augmentation and you get image t1 and you apply the second augmentation to the same image you get t1"}, {"start": 1050.2800000000002, "end": 1054.44, "text": " Prime now you have a second image and you again apply"}, {"start": 1055.1200000000001, "end": 1056.64, "text": " augmentation and you get"}, {"start": 1056.64, "end": 1060.72, "text": " Image t2 and you get t2 prime and you do this"}, {"start": 1061.3200000000002, "end": 1067.24, "text": " Obviously n times because you have n images in your batch and now the whole point will be the following"}, {"start": 1067.24, "end": 1073.16, "text": " You take the representation you get from this t1 image after passing it through f and g"}, {"start": 1073.16, "end": 1080.16, "text": " And you want to make sure that that representation is close to this one because they belong to the same image"}, {"start": 1080.16, "end": 1087.1200000000001, "text": " And then you want to make sure that it's far away from these representations from this one from this one as well"}, {"start": 1087.1200000000001, "end": 1088.76, "text": " You want to pull it?"}, {"start": 1088.76, "end": 1095.28, "text": " You want to push it away from those representations and the same for every single image in the batch you're gonna"}, {"start": 1095.28, "end": 1102.0, "text": " Push them away and the only thing that's getting pulled together are those two images here"}, {"start": 1102.08, "end": 1106.56, "text": " So that's the whole idea of sim clear. So that's just kind of"}, {"start": 1107.48, "end": 1109.3999999999999, "text": " formulaically"}, {"start": 1109.3999999999999, "end": 1115.92, "text": " Showed here you basically have as I said, so similarity usually dot product between those representations"}, {"start": 1115.96, "end": 1117.52, "text": " There's some temperature coefficient"}, {"start": 1117.52, "end": 1123.96, "text": " But a whole point at the end is to kind of do what what I showed you here is just symbolic representation of the thing"}, {"start": 1123.96, "end": 1128.96, "text": " I just explained you here. Okay having understand that let me now go back to the"}, {"start": 1130.28, "end": 1136.3, "text": " Method and here we are. So again, here is the here is the sim clear method"}, {"start": 1136.4, "end": 1143.92, "text": " so you can now see that HM is the representation that comes from the F so from the encoder and then they just applied the"}, {"start": 1144.28, "end": 1145.76, "text": " projection head and"}, {"start": 1145.76, "end": 1153.0, "text": " That's it. That's that's everything you need to form representation for sim clear and then you apply this this this novel this novel"}, {"start": 1153.0, "end": 1155.0, "text": " object objective function"}, {"start": 1155.12, "end": 1161.6, "text": " to get representations for Buell they additionally just add this this head which I"}, {"start": 1161.92, "end": 1168.44, "text": " Mentioned which gives you this as symmetry in the in the pipeline and obviously they also use this a network"}, {"start": 1168.44, "end": 1173.48, "text": " So that's the network form from the exponential moving average. Okay, so now you can kind of"}, {"start": 1174.2, "end": 1176.66, "text": " hopefully ground these these"}, {"start": 1176.66, "end": 1183.5400000000002, "text": " Formulas to the dual and sim clear papers and again once you have the representations those VMs and VM primes"}, {"start": 1183.5800000000002, "end": 1188.92, "text": " You can do the logic I explained a couple minutes ago where you're forcing the"}, {"start": 1189.74, "end": 1191.74, "text": " vectors for"}, {"start": 1192.02, "end": 1196.46, "text": " Where you're forcing the vectors that correspond to the same masks to be closer together"}, {"start": 1196.94, "end": 1200.5, "text": " Whereas you're pushing away the vectors belonging to different masks"}, {"start": 1200.5, "end": 1208.46, "text": " And whereas masks correspond hopefully to different objects and that depends on the quality of your segmentation masks and they did some ablations on"}, {"start": 1208.74, "end": 1211.86, "text": " How those different so how these different algorithms?"}, {"start": 1212.42, "end": 1216.4, "text": " Affect the performance of the method and we'll see that in a couple minutes. Okay"}, {"start": 1216.94, "end": 1221.94, "text": " Let's see the results finally of this stat-con method and as you can see here"}, {"start": 1222.74, "end": 1229.74, "text": " So having pre training on the image net and then fine-tuning different tasks like cocoa instance segmentation"}, {"start": 1229.74, "end": 1231.74, "text": " Pascal semantic segmentation"}, {"start": 1232.02, "end": 1233.46, "text": " cityscapes"}, {"start": 1233.46, "end": 1240.34, "text": " Semantic segmentation and NYU depth estimation you can see in all of these four tasks, which are fairly different"}, {"start": 1241.14, "end": 1246.78, "text": " Like the that come method performs way better compared to the supervised and sim clear baselines"}, {"start": 1247.18, "end": 1254.82, "text": " So I think let's let's focus for now on one chart and then I'll tell you some interesting things"}, {"start": 1254.82, "end": 1257.26, "text": " I see here. So first things first is"}, {"start": 1257.26, "end": 1260.02, "text": " You can see here that sim clear"}, {"start": 1260.74, "end": 1262.74, "text": " after certain amount of"}, {"start": 1262.82, "end": 1264.82, "text": " given enough computation"}, {"start": 1264.94, "end": 1266.46, "text": " like"}, {"start": 1266.46, "end": 1272.62, "text": " Surpasses the performance of supervised like paradigm and that's something we saw in the beginning of this paper"}, {"start": 1272.62, "end": 1278.86, "text": " So given enough compute that's a trade-off you're gonna eventually get there, but you need bunch of compute, right?"}, {"start": 1279.26, "end": 1282.3, "text": " Whereas here you can see focusing on that con that"}, {"start": 1282.86, "end": 1284.34, "text": " like with"}, {"start": 1284.34, "end": 1290.54, "text": " After only 200 epochs, you're already better compared to these two paradigms and you have better performance"}, {"start": 1290.54, "end": 1295.6599999999999, "text": " Okay, so the the the same thing applies pretty much for all of the other tasks"}, {"start": 1295.6599999999999, "end": 1300.86, "text": " I won't be digging into every single chart here. But the interesting thing here to notice is that supervised"}, {"start": 1301.78, "end": 1303.26, "text": " like paradigm"}, {"start": 1303.26, "end": 1307.86, "text": " Completely fails for these two tasks like depth estimation and semantic segmentation"}, {"start": 1307.86, "end": 1313.34, "text": " Which means that the pre training supervised pre training on image net is not informative enough"}, {"start": 1313.34, "end": 1318.06, "text": " so the features are not informative enough for these particular tasks where it's in clear and"}, {"start": 1318.58, "end": 1321.78, "text": " that can seem to generalize better to different tasks, which is"}, {"start": 1322.4599999999998, "end": 1329.9199999999998, "text": " Another plus and bonus point for applying SSL because you get more generalization out of the method. Okay"}, {"start": 1330.6999999999998, "end": 1336.12, "text": " I'll skip this table. It just has like a numeric representation of the thing"}, {"start": 1336.12, "end": 1336.9399999999998, "text": " We just saw here"}, {"start": 1336.9399999999998, "end": 1342.1, "text": " So basically the the main point they're trying to drive here is that with 10x less computation"}, {"start": 1342.1, "end": 1347.02, "text": " They can achieve performance that's much better than all the other baselines"}, {"start": 1347.02, "end": 1350.4599999999998, "text": " So here you can see with only 100 approximately 100 epochs"}, {"start": 1350.5, "end": 1356.82, "text": " It's already better than seem clear which is way better that supervised learning on this city scape semantic segmentation desk"}, {"start": 1356.82, "end": 1360.26, "text": " So that's the whole point. Okay going forward. Let's see. What else is"}, {"start": 1361.02, "end": 1362.9399999999998, "text": " there"}, {"start": 1362.9399999999998, "end": 1364.3, "text": " Okay"}, {"start": 1364.3, "end": 1366.3, "text": " here they compare like"}, {"start": 1366.3, "end": 1373.6399999999999, "text": " That con again against various different like baselines not only Buell and seem clear both so moco"}, {"start": 1374.26, "end": 1381.06, "text": " Supervised suave and other methods and they just showed that it consistently outperforms all of those baselines"}, {"start": 1381.94, "end": 1389.12, "text": " So one thing they've worried about is that maybe scaling the number of parameters will kind of give us something qualitatively different"}, {"start": 1389.5, "end": 1392.86, "text": " behavior with that con compared to the previous baselines and"}, {"start": 1392.86, "end": 1398.4599999999998, "text": " this basically just confirms that this method does scale with with scale and"}, {"start": 1399.2199999999998, "end": 1402.58, "text": " Like the performance does not change and it's still all performed"}, {"start": 1402.58, "end": 1408.3799999999999, "text": " So that's con the blue one still outperforms all of the other baselines like we all seem clear and supervised"}, {"start": 1408.9799999999998, "end": 1413.3, "text": " Baseline on these cocoa detection and cocoa instance segmentation tasks. Okay"}, {"start": 1414.82, "end": 1419.4599999999998, "text": " Moving further they even showed that this recently published methods here"}, {"start": 1419.46, "end": 1425.42, "text": " Which has billion which used billion images for for pre training and has way more parameters"}, {"start": 1425.82, "end": 1430.26, "text": " It's still like the the dead con outperforms those methods"}, {"start": 1430.26, "end": 1435.66, "text": " Which is a nice result and they kind of mentioned that it's not super comparable because"}, {"start": 1436.38, "end": 1438.38, "text": " Sierra uses"}, {"start": 1438.38, "end": 1440.5, "text": " The data that's here uses is way more"}, {"start": 1441.22, "end": 1444.8600000000001, "text": " Noisy, it's kind of not a fair comparison, but still it's an impressive result"}, {"start": 1445.66, "end": 1448.54, "text": " Okay, let's see. What else interesting is here"}, {"start": 1448.54, "end": 1452.26, "text": " They showed how using different"}, {"start": 1453.1399999999999, "end": 1457.04, "text": " Segmentation mass so different heuristics which I briefly described before"}, {"start": 1458.1399999999999, "end": 1464.62, "text": " Improves and changes the performance. So as you can see here using those spatial like heuristics, so that's this one"}, {"start": 1464.62, "end": 1466.62, "text": " Let me just go briefly up there"}, {"start": 1466.8999999999999, "end": 1472.94, "text": " So this is a spatial heuristic and here we have so this upper here is one by one here. We have two by two"}, {"start": 1472.94, "end": 1480.78, "text": " Grid and here we have four by four. Okay, so that will hopefully help you understand this chart a bit better"}, {"start": 1481.5800000000002, "end": 1484.22, "text": " What these numbers mean so here is one by one?"}, {"start": 1484.94, "end": 1490.48, "text": " Here is using two by two ten by ten and five by five seems to be working the best on this particular"}, {"start": 1491.5, "end": 1495.9, "text": " Like desk and I guess this is not that surprising. So the better the"}, {"start": 1495.9, "end": 1502.46, "text": " Semantic segmentation you do initially so FH obviously has better as you can see on the x-axis"}, {"start": 1502.46, "end": 1504.46, "text": " this means that the overlap between the"}, {"start": 1504.9, "end": 1509.7, "text": " The masks that this method found is way way higher"}, {"start": 1510.5, "end": 1514.1000000000001, "text": " To the GT which is obviously one because by default"}, {"start": 1514.66, "end": 1520.74, "text": " GT overlaps completely with itself. That's why we have one here and these methods have even smaller overlap"}, {"start": 1520.74, "end": 1522.3000000000002, "text": " Which means they are of lower quality"}, {"start": 1522.3, "end": 1529.78, "text": " So that basically means the higher the quality of your segmentation masks the higher the detection accuracy and that's kind of I guess intuitive"}, {"start": 1529.82, "end": 1533.78, "text": " But it's nice. You have a chart like proving exactly that. Okay"}, {"start": 1535.3799999999999, "end": 1537.86, "text": " They did a couple more oblations of"}, {"start": 1538.82, "end": 1543.3, "text": " pre-training this time on cocoa instead of image net and they still have they still"}, {"start": 1544.02, "end": 1551.06, "text": " Outperform the supervised and the same clear like baselines. That's that that's the point there. Finally, they showed that"}, {"start": 1551.06, "end": 1552.34, "text": " using"}, {"start": 1552.34, "end": 1554.78, "text": " GT labels which kind of makes sense"}, {"start": 1555.4199999999998, "end": 1558.7, "text": " Helps helps them even in increasing the the pre-training"}, {"start": 1559.3799999999999, "end": 1562.54, "text": " image resolution and always testing in on this resolution"}, {"start": 1563.5, "end": 1566.7, "text": " Consistently improves the performance when you're using GT masks"}, {"start": 1566.7, "end": 1572.5, "text": " Whereas when you're using FH the performance does not improve with resolution which I guess makes sense"}, {"start": 1573.22, "end": 1578.86, "text": " They mentioned that somewhere here in the in the discussion portion of the paper. So, okay this part"}, {"start": 1578.86, "end": 1583.8999999999999, "text": " How can this be so one interpretation is that other images give us clean negative examples?"}, {"start": 1583.8999999999999, "end": 1586.4599999999998, "text": " Because images in cocoa depict different scenes"}, {"start": 1586.4599999999998, "end": 1591.82, "text": " However, it appears that negatives from the same image provide a stronger learning signal blah blah blah"}, {"start": 1591.82, "end": 1595.6999999999998, "text": " I were not pushing features from the same object apart"}, {"start": 1595.6999999999998, "end": 1601.5, "text": " Positives from the same image are also at least as good as those across segmentations if again, they are clean"}, {"start": 1601.5, "end": 1605.1399999999999, "text": " We are not pulling together features from different objects"}, {"start": 1605.14, "end": 1610.74, "text": " So basically if you have if you don't have perfect cementation mass that means you'll have overlap between classes"}, {"start": 1610.74, "end": 1613.8600000000001, "text": " Which means sometimes you're gonna be pushing away"}, {"start": 1614.5800000000002, "end": 1620.3400000000001, "text": " Two feature vectors even though there is an overlap where those two vectors are formed from the same object"}, {"start": 1620.3400000000001, "end": 1624.5800000000002, "text": " So that's something not something you want to do. You want to pull push away different objects"}, {"start": 1624.5800000000002, "end": 1631.0600000000002, "text": " You want to pull together the same object and that's all like a hand wavy explanation of why you need better"}, {"start": 1631.6200000000001, "end": 1633.46, "text": " better cement cementing"}, {"start": 1633.46, "end": 1636.14, "text": " better cementing segmentation masks"}, {"start": 1636.74, "end": 1637.8600000000001, "text": " Okay"}, {"start": 1637.8600000000001, "end": 1639.8600000000001, "text": " wrapping up this paper"}, {"start": 1640.54, "end": 1647.82, "text": " The main idea again is usage of these segmentation masks in order to form a richer training signal"}, {"start": 1647.82, "end": 1653.14, "text": " So whereas Sinclair will in this example just we won't have any cementation masks"}, {"start": 1653.3400000000001, "end": 1661.02, "text": " We won't have any semantic segmentation mask, which means Sinclair will just be pulling together these two because they belong to the same image"}, {"start": 1661.02, "end": 1665.7, "text": " Whereas here we have a much richer signal because of the presence of semantic segmentation"}, {"start": 1665.86, "end": 1669.42, "text": " So that's pretty much it for this video if you have any suggestion"}, {"start": 1669.94, "end": 1675.02, "text": " For which repo I should do a code walkthrough off. Let me know down in the comments"}, {"start": 1675.02, "end": 1679.62, "text": " So I was thinking maybe dyno or some of those newer papers that came out"}, {"start": 1679.78, "end": 1681.58, "text": " So if you have any suggestion"}, {"start": 1681.58, "end": 1687.62, "text": " Please let me know down in the comments and I definitely read those comments and you can influence the next video other than that"}, {"start": 1687.62, "end": 1691.6999999999998, "text": " If you found this video useful consider subscribing and hit that notification"}, {"start": 1692.4199999999998, "end": 1696.62, "text": " Button so that you can get notified the moment I upload a new video"}, {"start": 1696.7399999999998, "end": 1704.02, "text": " I've noticed that only 10% of you have the notification bell on and only 5 or 6% of you"}, {"start": 1704.7399999999998, "end": 1712.3799999999999, "text": " Are actually receiving those notifications because your YouTube settings are set in such manner that you are not receiving them instantaneously"}, {"start": 1712.38, "end": 1717.8200000000002, "text": " So yeah, definitely toggle on that notification button and until next time. Bye. Bye. I"}]