VOS: Learning What You Don't Know By Virtual Outlier Synthesis | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this paper I cover the "VOS: Learning What You Don't Know By Virtual Outlier Synthesis" paper - where they introduce a clever way of sampling OOD (out-of-distribution) data in the feature space in order to produce a more robust ID (in-distribution) image classification/object detection that's OOD aware.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2202.01197
✅ Code: https://github.com/deeplearning-wisc/vos
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro to the OOD problem
04:40 High-level VOS explanation
09:38 Alternative synthesis approach (GANs)
11:40 Diving deeper into the method
17:40 Uncertainty loss component
22:45 Inference-time OOD detection
24:05 Method step-by-step overview
26:35 Results
27:45 Computational cost
29:00 Ablations, visualization
30:23 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#generalization #OOD #virtualoutliersynthesis
 What's up guys, in this video I'm covering this paper called VOS, Learning What You Don't Know by Virtual Outlier Synthesis, by this nice group of people here whose names I don't want to butcher with my poor pronunciation. And what the paper is all about is handling this out of distribution data in the context of image classification and in the context of object detection. So here we can see on this left hand side image, basically we see a moose being detected and then labeled as a pedestrian with high confidence. And this image may not even be the best one to present here because pedestrian is semantically very close to what this is. I mean the moose is crossing the street so it is a pedestrian in a way. But you get a point, like here it could have been mislabeled as like a truck or whatnot. And that would be very bad, especially if the model is very confident about that. So why this happens is because these models are trained on certain ID data, so the in-distribution data, data sets. And then once you show them certain objects which were not a part of their training set, so the OOD data, they usually are very confident and very incorrect at the same time. So this paper is trying to address exactly that. So why does this type of behavior happen in the first place? And the reason is due to the training procedure. You're basically training these object detection models in such a way that you're just feeding them the ID data. So as soon as you give them some object that was not seen in the training data set, then you have no idea and no guarantees of how the model is going to behave with that new data point because the model's decision boundaries may be completely unknown outside of your domain, of your initial training data set. So to understand that better, let's shift our focus to the two diagrams on the right-hand side here. So we have two different models here. And let me first help you understand how to parse these diagrams. So we have our data set, which is a contrived data set of two deep points sampled from a Gaussian mixture model here. So we have these three clusters here, here, here, and here. And the model is trained to predict whether the data is in distribution or out of distribution. And you can see that this color bar on the right-hand side, so if it's very red, that means model is fairly certain that data point belongs, is in the ID data indeed. And if the color is like very deep blue, then basically it thinks that it's OOD data, so out of distribution data. And here you can see that this landscape, that these decision boundaries are quite bad, because if we were to take, and let me change the color to something that's not blue or red here, let's say green. So if we were to take a data point here, this model, however it was trained, would say that this data point is most certainly ID, so that means in distribution, which obviously doesn't make any sense. As you can see here, as a human looking at this, it's very far off from all of these three clusters and doesn't make any sense. On the other hand, the model arbitrarily decided to, that these data points here are OOD. So that's a behavior that's not desired. And then on the right hand side here, we can see the VOS, so that's the model that was proposed in this paper. We can see a much nicer behavior, because we have that model is predicting that the nearby data points here are actually very likely to be ID. And then that slowly dropping as we're radially going outwards from the clusters, you can see that it goes towards the white color, which means it's becoming less and less certain. It's becoming more and more certain that the data point is OOD data point. And so we can see that these data points here are clearly marked as OOD. And that's a behavior, that's the type of behavior we want to enforce. So now this is, we can all agree that this is the behavior we want to see. This is the type of decision boundaries we want to see in the models, but how do we enforce such decision boundaries? And that's what I'm going to explain you next up. I'm going to focus a lot on the method in this paper and just briefly show you the results a bit later. So let's focus on this high level diagram here. So I mentioned that this paper works in the context of image detection, object detection, and image classification. For the sake of argument, and for the sake of this method, it doesn't even matter. We can focus on image classification because it's even simpler or we can just do object detection. It doesn't matter both ways. So you can see here the pipeline looks the following. You feed an image into some backbone network, which is usually, I guess, either like a CNN or some type of a transformer, like vision transformer or some hybrid or whatnot. So that's less important for the sake of this paper. And we have this proposal generator, we generate some bounding boxes. And the main point here is we somehow take our input images, our input data points, which they denote as x. So let me just change the color. So they denote those as x. And they feed also the bounding box here. And they somehow form these embeddings that they denote as h. And if you take images from various different classes, so let's say we have, let me zoom in a bit here. So let's say we have some class like a that's blue circle class, whatever that means. And then we take some images that are like yellow square. And we basically feed all of those. So we take a bunch of images that belong to the blue circle class, we feed them through this like basically backbone networks. And we can see that they tend to cluster together here. Now that will obviously not happen initially as the network is highly random, like random. But like they actually start doing this process a bit later throughout the training. Initially, they just use these losses for the classification and for the basically location or the bounding box loss here. And only later do they include the most important component and the most important contribution of this paper. And that's this uncertainty loss here. But I'll get to those details in a bit later. Now let's just keep it high level. So for the time being, let's assume that the embedding vectors are indeed clustered. So basically for a particular class, the embedding vectors are going to be clustered roughly together in the same part of the space. The main idea is to form these class conditional Gaussians. So these here, using the embedding vectors here. So you can imagine calculating a mean vector for this cluster, and basically a covariance matrix. And then using those two matrices to model like a Gaussian. And what this distribution tells you is whether a certain embedding is likely to belong to that class. So if you fit it, as you can see here on the image like this, then this model is fairly certain then that if we were to take this point here, it's highly unlikely that this data point here belongs to this class here. So they form the same types of class conditional Gaussians for all of the classes. So you can see here for the yellow squares, they form a Gaussian. For the blue circles, they form a Gaussian as well. And now the cool thing is you can, in this feature space, you can start sampling the outliers. Because by definition, we know that the ID data for this class lies somewhere in this radius here. So then that means that we can sample these less likely samples and use them as the outliers for that particular class. And that's exactly what I do. And now the final idea is to just use both those outliers as well as the ID data, form these energy scores. So energy terms, which they also call uncertainty scores. And make sure to push the energy for the ID data towards minus infinity and to push the scores or the energy term for the outliers towards plus infinity. So you just want to make that distinction. And they kind of depict that here as this separate, you can see like a fairly nice separation between the ID data and the OOD data as a consequence of having this uncertainty loss here. So yeah, we'll get to the details in a couple of seconds. But like that's the high level overview here. So this is the main component. This uncertainty loss component is the main one. And finally, we can see that the model this time correctly labels moose as OOD. So whatever type of object we had here, if it wasn't in the training data set, the model is going to correctly label all of those as OOD. Ideally, of course, that's what we're trying to get at. And this model seems to have a fairly nice results. So now let's start digging a bit deeper. They say here, while a straightforward idea is to train generative models such as GANs, synthesizing images in the high dimensional pixel space can be difficult to optimize. Instead, our key idea is to synthesize virtual outliers in the future space, which is more tractable given lower dimensionality. So we've seen that the system is actually dealing with embedding vectors in this so-called feature space. And one like alternative way how you could approach this problem, which is a fairly intuitive one, is to use GANs. So that means you would train a GAN architecture here additionally, that would generate OOD data. So you'd have a GAN here. And you would basically be generating images which are OOD and then feeding those into your pipeline. And then basically want to push this uncertainty score to different parts of the spectrum for the ID data versus the OOD data. But it's hard to train GANs. And also it's a question how you would, how would you cover the whole space of OOD data? Like would you just train GAN on some different data set or multiple different data sets that have the OOD classes? Or would you just try to push the image generation towards something that's not ID? So I guess those two approaches pop to my head. Although I'm not sure whether those images generated in such a manner would even be looking like natural images and whether that would be helpful. This method here definitely seems to have its own appeal in the computational sense because it's dealing with embedding vectors which are inherently lower dimensional as well as just the semantics. Just you're basically sampling data right near to your decision boundary. I mean this is a two-dimensional data set so that's the case. But in the case of multi-dimensional vectors it's going to be a bit different. But still this definitely seems to be a better approach than generating in the image space. Okay, nice. Let's get back to the formula here. So this is just a like a formula explaining what I just explained visually. Basically this probability distribution here for a condition on a specific class k, it's modeling the probability of these embedding vectors as and it's modeling those as a Gaussian. And now the question is how do you form the mu case so that's the means of this of this Gaussian distribution and as well as the covariance. And the trick is to, as I said, to just use the clusters that belong to a specific class. So you can see here if we want to generate the mean for a particular class we take only those embedding vectors whose class is k, which makes a lot of sense I guess. You just average them out and that's how you get the mu k and then you just form the covariance matrix actually using all of the classes. So here I'm a bit confused why not just use the specific class here. So my gut feeling would be to first try to form these covariance matrices using only class specific information and not all of the classes. You can see this sum over k. It's kind of confusing. I didn't see any ablations on this. If anybody has an idea why this might be better feel free to comment down below. So you can see one potential problem with this thing here because after every single weight update of your system you'll have to recompute these and that's quite expensive. So you'll have to go through all of to pass your all of your data points again because this time after the weight update you'll have different embedding vectors h and so you'll have to keep recomputing this. And so how they make this computationally tractable is the following. They say that we use online estimation for efficient training where we maintain a class conditional queue with this many object instances from each class. And I think in practice they use something around thousand of these object instances and when they say object instances what I mean is they basically feed these embedding vectors h in separate queues. So every class will have a dedicated queue basically. So they mention here that in each iteration we enqueue the embeddings of objects to their corresponding class conditional queues and the queue the same number of object embeddings. Cool that's how we form the class conditional gaussians now in the feature space. So now let's see how we sample the virtual outliers and it's fairly simple. So you can see here that the set of virtual outliers for a class k denoted like this. So it's a set of vks such that basically the probability of those vks has to be smaller than some epsilon. And this whole ugly expression is just your multi-dimensional multivariate Gaussian distribution and this one is parameterized using the mean and the covariance for this particular class k. So what this tells you basically is to sample these outliers from low probability regions of the class conditional Gaussian. So that may be a mouthful but it's fairly simple. Basically you can imagine in this simple case of a 2D data set here that the peak of this Gaussian is somewhere here and then as you're radially moving outwards here, so if you're radially moving outwards the probability is dropping down. So if I were to do a cross cut through this distribution here and basically let's take this as the origin point then how this would look like is the following. So you have something like this, you have the peak at zero, so this is zero and you'd be sampling your outliers starting at certain cutoff probabilities. So they denote that one as epsilon, so that means let's say somewhere here this is your epsilon and that means you'd be sampling only these data points here. Okay that explanation was more for those of you who are new to this channel. Let's now continue and understand how do they leverage these outliers to form the uncertainty loss component I just mentioned above on the high level diagram. And the main idea is to form this log partition function that looks something like this. So we have a minus log sum over the number of classes where big K denotes the number of classes in your image classification or object detection task and you sum up these terms here. So I guess the main question is what is FK exactly and what FK is it's just a logit so it's basically you just take the embedding vector so HK you take that one and you linearly project that to the logit so you you basically have a linear layer here and as the output you have the logits so this here the length of this vector is big K which is the number of classes so this is just your dense or feedforward layer and that's how you form the FKs and now they're going to make sure that E for ID data goes to minus infinity and that E for OOD data goes to plus infinity and thus that's going to force these logits to have a different distribution depending whether your ID or OOD data point. And that's a high level intuitive idea why this works. So let's see this formula here. So here is the uncertainty component I was mentioning. So you can see how it's constructed here. So we have these in index this is known as an index function so basically when this expression here is true so when this is true then this equals one otherwise it equals zero so that means you want to make sure that this is like false almost all of the time and you also want to make sure that this is false almost all of the time because that way you're going to minimize the uncertainty loss components. Your goal is to minimize this and in order to minimize this you want to make sure that the energy component for the outliers goes towards minus infinity and you want to make sure that the energy component for the ID data goes to plus infinity which is completely the opposite as what I've just mentioned a couple of seconds ago because I think there is a missed error in this formula. I think the signs here should be switched so this should be like smaller than and this should be greater or equal than. I don't want to confuse you here the whole point is to make sure that the energy terms go to opposite parts of the spectrum whether the the ID data goes to minus infinity or to plus infinity does not matter as long as the OOD data goes to the opposite part of the spectrum basically but I'm fairly sure this is a mistake here somebody may correct me if I'm wrong here. Okay so there was an intuitive explanation you obviously want to include the expectations here so you have expectation over the data distribution here you have an expectation over the virtual outlier distribution here and you're trying to minimize this whole component this whole sum here. Now we don't know how to optimize index function and so what I've done is they did a smooth approximation of this of this index function they also call it 0 0 slash 1 loss using the binary sigmoid loss so here is the exact same formula just a realistic implementation and you can notice a couple of things here so first off if if you were to just replace this with x so this term here with x this is your familiar sigmoid function so let's just do a simple limit analysis here so if if e was to go to minus infinity then both of these terms would go to plus infinity and so this term the numerator over denominator is going to converge in the limit is going to go to one that means that log of one is going to go to zero and thus we are minimizing this so that means that this current this formula here is forcing the energy of the id data to go towards minus infinity so as e of x and i'm just going to ignore the this part the theta part so it's obviously parameterized by by theta so if e goes to minus infinity then that implies that the the loss goes to zero at least for for this term here and vice versa here you can see that when e for the virtual outliers so when e for the virtual outliers goes to plus infinity then this term goes to zero because e raised to the power of minus infinity goes to zero and then you have log of one which is zero so this term goes down to zero as as e of as the energy of the virtual outliers is approaching plus infinity and that's it it's fairly simple obviously the sign does not matter it's just important that you separate the energy terms for id and ood data that's it okay let's continue here basically this is just the same thing just for the object detection task so obviously here we were ignoring the bounding boxes because we were dealing with image classification so they're switching between image classification and object detection but for the purpose of understanding their fundamental idea and those are the virtual outliers and how to combine them to do separation between id and ood data you don't need to know the specifics i'm just going to skip over it this is the final loss component the final loss so it's basically a sum of your classical so this is a classification loss this is your location so bounding box loss in case you have in case you're dealing with object detection you'll have this one in case you're just dealing with image classification you can just ditch this one so you have those classic losses plus a weighted uncertainty loss here which is the main part of this paper and you're trying to minimize this the expected value across your data set of this loss here by tweaking thetas and that's that's it now once you train the model in such a way you can use that energy term to basically have a probabilistic output of whether your data is id or ood so in case of this formula here you can see that for if the model was properly trained for id data e goes to minus infinity which means this whole thing goes to one and that means this basically signals if it's going towards one so it's going to be on a spectrum between zero and one so zero let me just do it like this so zero and one so this indicates that it's id and this indicates that it's ood finally you can set a certain threshold and when you're above certain threshold gamma you say hey this is id otherwise you say hey this is ood data point and that's the final thing we need in order to do the type of classifications we saw in the example above so let me show you what what i mean by that basically this thing here so this moose was obviously the the output of that g model was towards zero so lower than gamma and that's why the model was labeled as ood object and that's it okay let me go back here let me show you the algorithm finally the whole picture is fairly simple so given the id data here so the data set that consists out of data points images corresponding bounding boxes again in the case of object detection task and corresponding labels so we have n of those data points i images we have randomly initialized detector with parameter theta so that's our neural network we have q size for gaussian density estimation so all of those are parameters for the algorithm we have the weight for uncertainty regularization beta and we have the epsilon which is if you remember the threshold that we use to determine how to sample the virtual outliers so the probability under the class conditional gaussian has to be lower than this parameter epsilon if we were to sample those virtual outliers okay so let's see what the algorithm does so when we were training we update the idq with the training object so this is kind of weird statement you basically take the you embed a certain amount of data points for each of the classes you take the embedding vectors and you update those corresponding cues then you estimate the multivariate distributions based on the id on these embedding vectors using using equations one and two and equations one and two are these ones so let me go back up here basically you form the the mean and you form the the covariance matrix for your given embedding vectors so that's the next step of the algorithm and then they say sample virtual outliers v using equation three so that means we take the equation three here so having formed the class conditional gaussians using the parameters above so the the mean here and the covariance we can basically sample a set of these outliers from those distributions so that's the next step here and then they say calculate the regularization loss using equation five and update the parameters using equation seven so regularization number five basically that's how we calculate the loss the uncertainty component and then finally the whole loss consists out of the classification loss in case we have object detection we have this localization loss i guess and uncertainty component okay and finally once you train the model like that you can use the equations eight and nine to determine whether the object at hand is od or id and that's pretty much it there was i'm aware there was a lot of nitty-gritty details now let me show you the results so first thing to keep in mind is that they are only comparing against methods that are not using external outlier data sets if you do have external outlier data set those models those yeah those methods tend to perform better and they did have some results comparing against those methods and they are a bit worse now the problem with those data sets is they are very usually very expensive depends on your domain of application they're going to be more or less expensive to acquire so having methods such as this one where you don't need any additional data set you just need your id and then you have this clever way of sampling the the virtual outliers in the space in the feature space and this just works very nicely compared to other baselines here you can see that across all of these metrics it outperforms the previous methods but again keep in mind they are not comparing i think they even say here yeah they say it here all baseline methods are based on a model train on id data only that's the important part here okay cool now one thing worth thinking about is the following for each in distribution class we use thousand samples to estimate the class conditional gossians so that means they have to take thousand images embed them and then calculate the the mean and the and the and the covariance matrix and then they say since the threshold epsilon can be infinitesimally small we instead choose epsilon based on the teeth smallest likelihood in a pool of 10 000 samples per class generated from the class conditional gossian distribution so once you form the the gossian distribution using these thousand samples then what they do is as far as i understand here they take another 10 000 samples and then whatever out of those 10 000 samples what whatever whatever was the the lowest probability of a sample that's what they use as the epsilon and then they use that epsilon to sample the virtual outliers now the problem here is that this can be very expensive and the thing is in none of these tables did they compare the computational budget used to train these methods so that's something to definitely keep in mind if you're if you're on budget anyways let's continue here and see what other results they have so here they compare against some other types of synthesis methods i did mention the gan approach where you're trying to in the image space generate the the the outliers so the ood data and then use those to to train the system they also compare against these negative proposal methods etc etc and they again show up like superior results here okay let's continue yeah yeah they do all these ablations like having k plus one classes instead of k classes so basically this plus one is where the training happens so they use that plus one class as a detector of whether the data whether the um embedding vector is belongs to id data or to od data that's the trick there okay let me just show you the visualizations and we are pretty much done on the top here you can see faster rcnn and on the bottom you can see vos and you can see that eos successfully detects that the bet here and some other objects are od whereas the faster rcnn fails completely and is and it's confident in the prediction so it's confident that the the the bet here is car with 90 97 percent certainty that's that's weird um cool let me just show you one single detail here in the appendix if i can find it um and that's this part here so they say that uh importantly we show that uncertainty regularization should be added in the middle of the training if it is added too early the feature space is not sufficiently discriminative for gossian distribution estimation and that's something i mentioned in the beginning of the video basically initially you're just training for the classif you're just minimizing the classification loss the object detection loss if you're doing object detection and only later during the training when when basically the feature space becomes sufficiently discriminative i.e. the the the embedding vectors from the same class tend to cluster together only then can you start using this method and then it works in any case hopefully like this video if you did consider subscribing uh leave a comment down below if you have anything to ask or or any feedback uh in any case until next time bye bye
[{"start": 0.0, "end": 4.16, "text": " What's up guys, in this video I'm covering this paper called VOS,"}, {"start": 4.16, "end": 7.28, "text": " Learning What You Don't Know by Virtual Outlier Synthesis,"}, {"start": 8.08, "end": 14.16, "text": " by this nice group of people here whose names I don't want to butcher with my poor pronunciation."}, {"start": 14.88, "end": 21.76, "text": " And what the paper is all about is handling this out of distribution data in the context of"}, {"start": 21.76, "end": 26.96, "text": " image classification and in the context of object detection."}, {"start": 26.96, "end": 35.92, "text": " So here we can see on this left hand side image, basically we see a moose being detected and then"}, {"start": 35.92, "end": 41.92, "text": " labeled as a pedestrian with high confidence. And this image may not even be the best one to"}, {"start": 41.92, "end": 48.88, "text": " present here because pedestrian is semantically very close to what this is. I mean the moose is"}, {"start": 48.88, "end": 54.32, "text": " crossing the street so it is a pedestrian in a way. But you get a point, like here it could have"}, {"start": 54.32, "end": 60.0, "text": " been mislabeled as like a truck or whatnot. And that would be very bad, especially if the model"}, {"start": 60.0, "end": 65.84, "text": " is very confident about that. So why this happens is because these models are trained on certain"}, {"start": 66.4, "end": 74.48, "text": " ID data, so the in-distribution data, data sets. And then once you show them certain objects which"}, {"start": 74.48, "end": 79.92, "text": " were not a part of their training set, so the OOD data, they usually are very confident and"}, {"start": 79.92, "end": 84.96000000000001, "text": " very incorrect at the same time. So this paper is trying to address exactly that. So why does this"}, {"start": 84.96000000000001, "end": 89.92, "text": " type of behavior happen in the first place? And the reason is due to the training procedure."}, {"start": 89.92, "end": 94.72, "text": " You're basically training these object detection models in such a way that you're just feeding"}, {"start": 94.72, "end": 102.08, "text": " them the ID data. So as soon as you give them some object that was not seen in the training data set,"}, {"start": 102.08, "end": 108.4, "text": " then you have no idea and no guarantees of how the model is going to behave with that new data"}, {"start": 108.4, "end": 116.4, "text": " point because the model's decision boundaries may be completely unknown outside of your domain,"}, {"start": 116.4, "end": 123.84, "text": " of your initial training data set. So to understand that better, let's shift our focus to the two"}, {"start": 123.84, "end": 131.04000000000002, "text": " diagrams on the right-hand side here. So we have two different models here. And let me first help"}, {"start": 131.04000000000002, "end": 137.36, "text": " you understand how to parse these diagrams. So we have our data set, which is a contrived data set"}, {"start": 137.36, "end": 143.92000000000002, "text": " of two deep points sampled from a Gaussian mixture model here. So we have these three clusters here,"}, {"start": 144.48000000000002, "end": 151.28, "text": " here, here, and here. And the model is trained to predict whether the data is in distribution or"}, {"start": 151.28, "end": 156.96, "text": " out of distribution. And you can see that this color bar on the right-hand side, so if it's very"}, {"start": 156.96, "end": 164.4, "text": " red, that means model is fairly certain that data point belongs, is in the ID data indeed. And if the"}, {"start": 164.4, "end": 172.8, "text": " color is like very deep blue, then basically it thinks that it's OOD data, so out of distribution"}, {"start": 172.8, "end": 177.68, "text": " data. And here you can see that this landscape, that these decision boundaries are quite bad,"}, {"start": 177.68, "end": 183.36, "text": " because if we were to take, and let me change the color to something that's not blue or red here,"}, {"start": 183.36, "end": 189.20000000000002, "text": " let's say green. So if we were to take a data point here, this model, however it was trained,"}, {"start": 189.2, "end": 196.48, "text": " would say that this data point is most certainly ID, so that means in distribution, which obviously"}, {"start": 196.48, "end": 201.76, "text": " doesn't make any sense. As you can see here, as a human looking at this, it's very far off from"}, {"start": 201.76, "end": 207.51999999999998, "text": " all of these three clusters and doesn't make any sense. On the other hand, the model arbitrarily"}, {"start": 207.51999999999998, "end": 216.72, "text": " decided to, that these data points here are OOD. So that's a behavior that's not desired. And then"}, {"start": 216.72, "end": 221.52, "text": " on the right hand side here, we can see the VOS, so that's the model that was proposed in this paper."}, {"start": 221.52, "end": 227.28, "text": " We can see a much nicer behavior, because we have that model is predicting that the nearby data"}, {"start": 227.28, "end": 233.76, "text": " points here are actually very likely to be ID. And then that slowly dropping as we're radially"}, {"start": 233.76, "end": 240.56, "text": " going outwards from the clusters, you can see that it goes towards the white color, which means"}, {"start": 240.56, "end": 245.84, "text": " it's becoming less and less certain. It's becoming more and more certain that the data point is OOD"}, {"start": 245.84, "end": 255.36, "text": " data point. And so we can see that these data points here are clearly marked as OOD. And that's"}, {"start": 255.36, "end": 260.88, "text": " a behavior, that's the type of behavior we want to enforce. So now this is, we can all agree that"}, {"start": 260.88, "end": 267.44, "text": " this is the behavior we want to see. This is the type of decision boundaries we want to see in the"}, {"start": 267.44, "end": 272.56, "text": " models, but how do we enforce such decision boundaries? And that's what I'm going to explain"}, {"start": 272.56, "end": 279.2, "text": " you next up. I'm going to focus a lot on the method in this paper and just briefly show you the"}, {"start": 279.2, "end": 286.8, "text": " results a bit later. So let's focus on this high level diagram here. So I mentioned that this paper"}, {"start": 286.8, "end": 292.88, "text": " works in the context of image detection, object detection, and image classification. For the sake"}, {"start": 292.88, "end": 297.44, "text": " of argument, and for the sake of this method, it doesn't even matter. We can focus on image"}, {"start": 297.44, "end": 301.84000000000003, "text": " classification because it's even simpler or we can just do object detection. It doesn't matter"}, {"start": 301.84, "end": 306.88, "text": " both ways. So you can see here the pipeline looks the following. You feed an image into some backbone"}, {"start": 306.88, "end": 313.28, "text": " network, which is usually, I guess, either like a CNN or some type of a transformer, like vision"}, {"start": 313.28, "end": 318.71999999999997, "text": " transformer or some hybrid or whatnot. So that's less important for the sake of this paper."}, {"start": 319.59999999999997, "end": 327.2, "text": " And we have this proposal generator, we generate some bounding boxes. And the main point here is"}, {"start": 327.2, "end": 335.28, "text": " we somehow take our input images, our input data points, which they denote as x. So let me just"}, {"start": 335.28, "end": 341.2, "text": " change the color. So they denote those as x. And they feed also the bounding box here. And they"}, {"start": 341.2, "end": 352.48, "text": " somehow form these embeddings that they denote as h. And if you take images from various different"}, {"start": 352.48, "end": 357.84000000000003, "text": " classes, so let's say we have, let me zoom in a bit here. So let's say we have some class like a"}, {"start": 357.84000000000003, "end": 363.52000000000004, "text": " that's blue circle class, whatever that means. And then we take some images that are like yellow"}, {"start": 363.52000000000004, "end": 369.12, "text": " square. And we basically feed all of those. So we take a bunch of images that belong to the blue"}, {"start": 369.68, "end": 376.24, "text": " circle class, we feed them through this like basically backbone networks. And we can see that"}, {"start": 376.24, "end": 381.76, "text": " they tend to cluster together here. Now that will obviously not happen initially as the network is"}, {"start": 381.76, "end": 388.56, "text": " highly random, like random. But like they actually start doing this process a bit later throughout"}, {"start": 388.56, "end": 393.84, "text": " the training. Initially, they just use these losses for the classification and for the"}, {"start": 394.64, "end": 400.64, "text": " basically location or the bounding box loss here. And only later do they include the most important"}, {"start": 400.64, "end": 405.03999999999996, "text": " component and the most important contribution of this paper. And that's this uncertainty loss here."}, {"start": 405.68, "end": 411.36, "text": " But I'll get to those details in a bit later. Now let's just keep it high level. So for the time"}, {"start": 411.36, "end": 416.32, "text": " being, let's assume that the embedding vectors are indeed clustered. So basically for a particular"}, {"start": 416.32, "end": 420.40000000000003, "text": " class, the embedding vectors are going to be clustered roughly together in the same part of"}, {"start": 420.40000000000003, "end": 430.16, "text": " the space. The main idea is to form these class conditional Gaussians. So these here, using the"}, {"start": 430.16, "end": 436.48, "text": " embedding vectors here. So you can imagine calculating a mean vector for this cluster,"}, {"start": 436.48, "end": 447.84000000000003, "text": " and basically a covariance matrix. And then using those two matrices to model like a Gaussian. And"}, {"start": 447.84000000000003, "end": 453.76, "text": " what this distribution tells you is whether a certain embedding is likely to belong to that"}, {"start": 453.76, "end": 460.40000000000003, "text": " class. So if you fit it, as you can see here on the image like this, then this model is fairly"}, {"start": 460.40000000000003, "end": 466.32, "text": " certain then that if we were to take this point here, it's highly unlikely that this data"}, {"start": 466.32, "end": 474.15999999999997, "text": " point here belongs to this class here. So they form the same types of class conditional Gaussians for"}, {"start": 474.15999999999997, "end": 480.24, "text": " all of the classes. So you can see here for the yellow squares, they form a Gaussian. For the"}, {"start": 480.24, "end": 485.28, "text": " blue circles, they form a Gaussian as well. And now the cool thing is you can, in this feature"}, {"start": 485.28, "end": 490.8, "text": " space, you can start sampling the outliers. Because by definition, we know that the"}, {"start": 490.8, "end": 499.2, "text": " ID data for this class lies somewhere in this radius here. So then that means that we can sample"}, {"start": 499.2, "end": 505.68, "text": " these less likely samples and use them as the outliers for that particular class. And that's"}, {"start": 505.68, "end": 513.6, "text": " exactly what I do. And now the final idea is to just use both those outliers as well as the ID data,"}, {"start": 513.6, "end": 520.96, "text": " form these energy scores. So energy terms, which they also call uncertainty scores. And make sure"}, {"start": 520.96, "end": 529.9200000000001, "text": " to push the energy for the ID data towards minus infinity and to push the scores or the energy term"}, {"start": 529.9200000000001, "end": 536.08, "text": " for the outliers towards plus infinity. So you just want to make that distinction. And they kind of"}, {"start": 536.08, "end": 541.6800000000001, "text": " depict that here as this separate, you can see like a fairly nice separation between the ID data"}, {"start": 541.68, "end": 546.7199999999999, "text": " and the OOD data as a consequence of having this uncertainty loss here. So yeah, we'll get to the"}, {"start": 546.7199999999999, "end": 552.0799999999999, "text": " details in a couple of seconds. But like that's the high level overview here. So this is the main"}, {"start": 552.0799999999999, "end": 558.64, "text": " component. This uncertainty loss component is the main one. And finally, we can see that the model"}, {"start": 558.64, "end": 567.12, "text": " this time correctly labels moose as OOD. So whatever type of object we had here, if it wasn't in the"}, {"start": 567.12, "end": 572.5600000000001, "text": " training data set, the model is going to correctly label all of those as OOD. Ideally, of course,"}, {"start": 572.5600000000001, "end": 579.2, "text": " that's what we're trying to get at. And this model seems to have a fairly nice results. So now let's"}, {"start": 579.2, "end": 584.96, "text": " start digging a bit deeper. They say here, while a straightforward idea is to train generative"}, {"start": 584.96, "end": 590.24, "text": " models such as GANs, synthesizing images in the high dimensional pixel space can be difficult to"}, {"start": 590.24, "end": 595.84, "text": " optimize. Instead, our key idea is to synthesize virtual outliers in the future space, which is"}, {"start": 595.84, "end": 600.72, "text": " more tractable given lower dimensionality. So we've seen that the system is actually"}, {"start": 600.72, "end": 608.08, "text": " dealing with embedding vectors in this so-called feature space. And one like alternative way how"}, {"start": 608.08, "end": 613.44, "text": " you could approach this problem, which is a fairly intuitive one, is to use GANs. So that means you"}, {"start": 613.44, "end": 621.12, "text": " would train a GAN architecture here additionally, that would generate OOD data. So you'd have a GAN"}, {"start": 621.12, "end": 631.12, "text": " here. And you would basically be generating images which are OOD and then feeding those into your"}, {"start": 631.12, "end": 636.48, "text": " pipeline. And then basically want to push this uncertainty score to different parts of the"}, {"start": 636.48, "end": 644.88, "text": " spectrum for the ID data versus the OOD data. But it's hard to train GANs. And also it's a question"}, {"start": 644.88, "end": 652.0, "text": " how you would, how would you cover the whole space of OOD data? Like would you just train GAN on some"}, {"start": 652.0, "end": 657.84, "text": " different data set or multiple different data sets that have the OOD classes? Or would you just try"}, {"start": 657.84, "end": 663.92, "text": " to push the image generation towards something that's not ID? So I guess those two approaches"}, {"start": 663.92, "end": 669.2, "text": " pop to my head. Although I'm not sure whether those images generated in such a manner would even be"}, {"start": 669.2, "end": 674.72, "text": " looking like natural images and whether that would be helpful. This method here definitely seems to"}, {"start": 674.72, "end": 680.24, "text": " have its own appeal in the computational sense because it's dealing with embedding vectors which"}, {"start": 680.24, "end": 687.6, "text": " are inherently lower dimensional as well as just the semantics. Just you're basically sampling data"}, {"start": 687.6, "end": 693.84, "text": " right near to your decision boundary. I mean this is a two-dimensional data set so that's the case."}, {"start": 693.84, "end": 697.5200000000001, "text": " But in the case of multi-dimensional vectors it's going to be a bit different. But still"}, {"start": 697.52, "end": 703.28, "text": " this definitely seems to be a better approach than generating in the image space. Okay, nice."}, {"start": 703.28, "end": 710.8, "text": " Let's get back to the formula here. So this is just a like a formula explaining what I just explained"}, {"start": 710.8, "end": 716.88, "text": " visually. Basically this probability distribution here for a condition on a specific class k,"}, {"start": 718.3199999999999, "end": 726.24, "text": " it's modeling the probability of these embedding vectors as and it's modeling those as a Gaussian."}, {"start": 726.24, "end": 731.84, "text": " And now the question is how do you form the mu case so that's the means of this of this"}, {"start": 731.84, "end": 737.76, "text": " Gaussian distribution and as well as the covariance. And the trick is to, as I said, to just use the"}, {"start": 739.52, "end": 746.0, "text": " clusters that belong to a specific class. So you can see here if we want to generate the mean"}, {"start": 746.0, "end": 752.96, "text": " for a particular class we take only those embedding vectors whose class is k, which makes a lot of"}, {"start": 752.96, "end": 758.64, "text": " sense I guess. You just average them out and that's how you get the mu k and then you just"}, {"start": 758.64, "end": 764.32, "text": " form the covariance matrix actually using all of the classes. So here I'm a bit confused why not"}, {"start": 764.32, "end": 773.36, "text": " just use the specific class here. So my gut feeling would be to first try to form these"}, {"start": 773.36, "end": 778.72, "text": " covariance matrices using only class specific information and not all of the classes. You can"}, {"start": 778.72, "end": 784.0, "text": " see this sum over k. It's kind of confusing. I didn't see any ablations on this. If anybody"}, {"start": 784.0, "end": 791.52, "text": " has an idea why this might be better feel free to comment down below. So you can see one potential"}, {"start": 791.52, "end": 797.2, "text": " problem with this thing here because after every single weight update of your system"}, {"start": 798.0, "end": 802.64, "text": " you'll have to recompute these and that's quite expensive. So you'll have to go through"}, {"start": 802.64, "end": 809.28, "text": " all of to pass your all of your data points again because this time after the weight update you'll"}, {"start": 809.28, "end": 815.1999999999999, "text": " have different embedding vectors h and so you'll have to keep recomputing this. And so how they"}, {"start": 815.1999999999999, "end": 819.76, "text": " make this computationally tractable is the following. They say that we use online estimation"}, {"start": 819.76, "end": 826.24, "text": " for efficient training where we maintain a class conditional queue with this many object instances"}, {"start": 826.24, "end": 832.96, "text": " from each class. And I think in practice they use something around thousand of these object instances"}, {"start": 833.84, "end": 839.28, "text": " and when they say object instances what I mean is they basically feed these embedding vectors h"}, {"start": 840.0, "end": 846.08, "text": " in separate queues. So every class will have a dedicated queue basically. So they mention here"}, {"start": 846.08, "end": 851.2, "text": " that in each iteration we enqueue the embeddings of objects to their corresponding class conditional"}, {"start": 851.2, "end": 856.96, "text": " queues and the queue the same number of object embeddings. Cool that's how we form the class"}, {"start": 856.96, "end": 864.88, "text": " conditional gaussians now in the feature space. So now let's see how we sample the virtual outliers"}, {"start": 864.88, "end": 871.9200000000001, "text": " and it's fairly simple. So you can see here that the set of virtual outliers for a class k denoted"}, {"start": 871.92, "end": 884.24, "text": " like this. So it's a set of vks such that basically the probability of those vks has to be smaller"}, {"start": 884.24, "end": 890.88, "text": " than some epsilon. And this whole ugly expression is just your multi-dimensional multivariate"}, {"start": 891.5999999999999, "end": 899.1999999999999, "text": " Gaussian distribution and this one is parameterized using the mean and the covariance for this"}, {"start": 899.2, "end": 907.2, "text": " particular class k. So what this tells you basically is to sample these outliers from"}, {"start": 908.5600000000001, "end": 913.44, "text": " low probability regions of the class conditional Gaussian. So that may be a mouthful but it's fairly"}, {"start": 913.44, "end": 919.2800000000001, "text": " simple. Basically you can imagine in this simple case of a 2D data set here that the peak of this"}, {"start": 919.2800000000001, "end": 924.8000000000001, "text": " Gaussian is somewhere here and then as you're radially moving outwards here, so if you're"}, {"start": 924.8, "end": 929.8399999999999, "text": " radially moving outwards the probability is dropping down. So if I were to do a cross cut"}, {"start": 930.4799999999999, "end": 937.04, "text": " through this distribution here and basically let's take this as the origin point then how this would"}, {"start": 937.04, "end": 944.0, "text": " look like is the following. So you have something like this, you have the peak at zero, so this is zero"}, {"start": 946.4, "end": 954.0, "text": " and you'd be sampling your outliers starting at certain cutoff probabilities. So they denote"}, {"start": 954.0, "end": 961.04, "text": " that one as epsilon, so that means let's say somewhere here this is your epsilon and that"}, {"start": 961.04, "end": 968.64, "text": " means you'd be sampling only these data points here. Okay that explanation was more for those"}, {"start": 968.64, "end": 975.36, "text": " of you who are new to this channel. Let's now continue and understand how do they leverage these"}, {"start": 975.36, "end": 983.44, "text": " outliers to form the uncertainty loss component I just mentioned above on the high level diagram."}, {"start": 984.32, "end": 988.64, "text": " And the main idea is to form this log partition function that looks something like this. So we"}, {"start": 988.64, "end": 995.04, "text": " have a minus log sum over the number of classes where big K denotes the number of classes in your"}, {"start": 995.04, "end": 1000.96, "text": " image classification or object detection task and you sum up these terms here. So I guess the main"}, {"start": 1000.96, "end": 1008.32, "text": " question is what is FK exactly and what FK is it's just a logit so it's basically you just take the"}, {"start": 1009.52, "end": 1019.44, "text": " embedding vector so HK you take that one and you linearly project that to the logit so you"}, {"start": 1019.44, "end": 1027.3600000000001, "text": " you basically have a linear layer here and as the output you have the logits so this here the length"}, {"start": 1027.36, "end": 1032.8799999999999, "text": " of this vector is big K which is the number of classes so this is just your dense or feedforward"}, {"start": 1032.8799999999999, "end": 1040.24, "text": " layer and that's how you form the FKs and now they're going to make sure that E for ID data"}, {"start": 1040.24, "end": 1047.28, "text": " goes to minus infinity and that E for OOD data goes to plus infinity and thus that's going to"}, {"start": 1047.84, "end": 1054.4799999999998, "text": " force these logits to have a different distribution depending whether your ID or OOD data point."}, {"start": 1054.48, "end": 1062.4, "text": " And that's a high level intuitive idea why this works. So let's see this formula here."}, {"start": 1062.4, "end": 1067.84, "text": " So here is the uncertainty component I was mentioning. So you can see how it's constructed"}, {"start": 1067.84, "end": 1073.28, "text": " here. So we have these in index this is known as an index function so basically when this expression"}, {"start": 1073.28, "end": 1082.48, "text": " here is true so when this is true then this equals one otherwise it equals zero so that means you"}, {"start": 1082.48, "end": 1088.4, "text": " want to make sure that this is like false almost all of the time and you also want to make sure"}, {"start": 1088.4, "end": 1095.1200000000001, "text": " that this is false almost all of the time because that way you're going to minimize the"}, {"start": 1095.1200000000001, "end": 1099.68, "text": " uncertainty loss components. Your goal is to minimize this and in order to minimize this you"}, {"start": 1099.68, "end": 1106.48, "text": " want to make sure that the energy component for the outliers goes towards minus infinity and you"}, {"start": 1106.48, "end": 1112.32, "text": " want to make sure that the energy component for the ID data goes to plus infinity which is completely"}, {"start": 1112.32, "end": 1119.9199999999998, "text": " the opposite as what I've just mentioned a couple of seconds ago because I think there is a missed"}, {"start": 1119.9199999999998, "end": 1123.9199999999998, "text": " error in this formula. I think the signs here should be switched so this should be like"}, {"start": 1124.72, "end": 1130.48, "text": " smaller than and this should be greater or equal than. I don't want to confuse you here the whole"}, {"start": 1130.48, "end": 1138.32, "text": " point is to make sure that the energy terms go to opposite parts of the spectrum whether the the ID"}, {"start": 1138.32, "end": 1144.32, "text": " data goes to minus infinity or to plus infinity does not matter as long as the OOD data goes to"}, {"start": 1144.32, "end": 1149.2, "text": " the opposite part of the spectrum basically but I'm fairly sure this is a mistake here somebody"}, {"start": 1149.2, "end": 1154.0, "text": " may correct me if I'm wrong here. Okay so there was an intuitive explanation you obviously want"}, {"start": 1154.0, "end": 1158.32, "text": " to include the expectations here so you have expectation over the data distribution here you"}, {"start": 1158.32, "end": 1164.1599999999999, "text": " have an expectation over the virtual outlier distribution here and you're trying to minimize"}, {"start": 1164.16, "end": 1171.6000000000001, "text": " this whole component this whole sum here. Now we don't know how to optimize index function"}, {"start": 1171.6000000000001, "end": 1177.6000000000001, "text": " and so what I've done is they did a smooth approximation of this of this index function"}, {"start": 1178.16, "end": 1185.2, "text": " they also call it 0 0 slash 1 loss using the binary sigmoid loss so here is the exact same"}, {"start": 1185.2, "end": 1191.6000000000001, "text": " formula just a realistic implementation and you can notice a couple of things here so first off"}, {"start": 1191.6, "end": 1198.1599999999999, "text": " if if you were to just replace this with x so this term here with x this is your familiar"}, {"start": 1198.1599999999999, "end": 1205.52, "text": " sigmoid function so let's just do a simple limit analysis here so if if e was to go to minus"}, {"start": 1205.52, "end": 1212.9599999999998, "text": " infinity then both of these terms would go to plus infinity and so this term the numerator over"}, {"start": 1212.9599999999998, "end": 1219.28, "text": " denominator is going to converge in the limit is going to go to one that means that log of one is"}, {"start": 1219.28, "end": 1225.44, "text": " going to go to zero and thus we are minimizing this so that means that this current this formula"}, {"start": 1225.44, "end": 1235.36, "text": " here is forcing the energy of the id data to go towards minus infinity so as e of x and i'm just"}, {"start": 1235.36, "end": 1242.3999999999999, "text": " going to ignore the this part the theta part so it's obviously parameterized by by theta so if"}, {"start": 1242.4, "end": 1250.64, "text": " e goes to minus infinity then that implies that the the loss goes to zero at least for for this"}, {"start": 1250.64, "end": 1261.0400000000002, "text": " term here and vice versa here you can see that when e for the virtual outliers so when e for the"}, {"start": 1261.0400000000002, "end": 1269.68, "text": " virtual outliers goes to plus infinity then this term goes to zero because e raised to the power of"}, {"start": 1269.68, "end": 1276.72, "text": " minus infinity goes to zero and then you have log of one which is zero so this term goes down to zero"}, {"start": 1276.72, "end": 1283.28, "text": " as as e of as the energy of the virtual outliers is approaching plus infinity and that's it it's"}, {"start": 1283.28, "end": 1290.0, "text": " fairly simple obviously the sign does not matter it's just important that you separate the energy"}, {"start": 1290.0, "end": 1300.48, "text": " terms for id and ood data that's it okay let's continue here basically this is just the same"}, {"start": 1300.48, "end": 1304.56, "text": " thing just for the object detection task so obviously here we were ignoring the bounding"}, {"start": 1304.56, "end": 1307.68, "text": " boxes because we were dealing with image classification so they're switching between"}, {"start": 1307.68, "end": 1312.48, "text": " image classification and object detection but for the purpose of understanding their fundamental idea"}, {"start": 1313.12, "end": 1319.12, "text": " and those are the virtual outliers and how to combine them to do separation between id and ood"}, {"start": 1319.12, "end": 1324.2399999999998, "text": " data you don't need to know the specifics i'm just going to skip over it this is the final loss"}, {"start": 1324.2399999999998, "end": 1332.08, "text": " component the final loss so it's basically a sum of your classical so this is a classification loss"}, {"start": 1332.08, "end": 1337.6799999999998, "text": " this is your location so bounding box loss in case you have in case you're dealing with object"}, {"start": 1337.6799999999998, "end": 1342.0, "text": " detection you'll have this one in case you're just dealing with image classification you can just"}, {"start": 1342.0, "end": 1352.24, "text": " ditch this one so you have those classic losses plus a weighted uncertainty loss here which is the"}, {"start": 1352.24, "end": 1358.24, "text": " main part of this paper and you're trying to minimize this the expected value across your"}, {"start": 1358.24, "end": 1367.84, "text": " data set of this loss here by tweaking thetas and that's that's it now once you train the model"}, {"start": 1367.84, "end": 1375.28, "text": " in such a way you can use that energy term to basically have a probabilistic output of whether"}, {"start": 1375.28, "end": 1384.1599999999999, "text": " your data is id or ood so in case of this formula here you can see that for if the model was properly"}, {"start": 1384.1599999999999, "end": 1393.52, "text": " trained for id data e goes to minus infinity which means this whole thing goes to one and that means"}, {"start": 1393.52, "end": 1398.16, "text": " this basically signals if it's going towards one so it's going to be on a spectrum between zero and"}, {"start": 1398.16, "end": 1405.12, "text": " one so zero let me just do it like this so zero and one so this indicates that it's id and this"}, {"start": 1405.12, "end": 1411.68, "text": " indicates that it's ood finally you can set a certain threshold and when you're above certain"}, {"start": 1411.68, "end": 1418.6399999999999, "text": " threshold gamma you say hey this is id otherwise you say hey this is ood data point and that's the"}, {"start": 1418.64, "end": 1425.2, "text": " final thing we need in order to do the type of classifications we saw in the example above so"}, {"start": 1425.2, "end": 1433.44, "text": " let me show you what what i mean by that basically this thing here so this moose was obviously the"}, {"start": 1434.0800000000002, "end": 1439.6000000000001, "text": " the output of that g model was towards zero so lower than gamma and that's why the model was"}, {"start": 1439.6000000000001, "end": 1447.6000000000001, "text": " labeled as ood object and that's it okay let me go back here let me show you the algorithm finally"}, {"start": 1447.6, "end": 1454.1599999999999, "text": " the whole picture is fairly simple so given the id data here so the data set that consists out of"}, {"start": 1454.1599999999999, "end": 1459.84, "text": " data points images corresponding bounding boxes again in the case of object detection task and"}, {"start": 1459.84, "end": 1466.6399999999999, "text": " corresponding labels so we have n of those data points i images we have randomly initialized"}, {"start": 1466.6399999999999, "end": 1472.08, "text": " detector with parameter theta so that's our neural network we have q size for gaussian density"}, {"start": 1472.08, "end": 1476.3999999999999, "text": " estimation so all of those are parameters for the algorithm we have the weight for uncertainty"}, {"start": 1476.4, "end": 1482.5600000000002, "text": " regularization beta and we have the epsilon which is if you remember the threshold that we use"}, {"start": 1482.5600000000002, "end": 1489.3600000000001, "text": " to determine how to sample the virtual outliers so the probability under the class conditional"}, {"start": 1489.3600000000001, "end": 1495.1200000000001, "text": " gaussian has to be lower than this parameter epsilon if we were to sample those virtual"}, {"start": 1495.1200000000001, "end": 1501.68, "text": " outliers okay so let's see what the algorithm does so when we were training we update the idq"}, {"start": 1501.68, "end": 1508.48, "text": " with the training object so this is kind of weird statement you basically take the you embed a"}, {"start": 1508.48, "end": 1512.8, "text": " certain amount of data points for each of the classes you take the embedding vectors and you"}, {"start": 1512.8, "end": 1518.48, "text": " update those corresponding cues then you estimate the multivariate distributions based on the id"}, {"start": 1519.1200000000001, "end": 1525.1200000000001, "text": " on these embedding vectors using using equations one and two and equations one and two are these"}, {"start": 1525.12, "end": 1532.4799999999998, "text": " ones so let me go back up here basically you form the the mean and you form the the covariance matrix"}, {"start": 1533.52, "end": 1538.7199999999998, "text": " for your given embedding vectors so that's the next step of the algorithm and then they say"}, {"start": 1538.7199999999998, "end": 1545.4399999999998, "text": " sample virtual outliers v using equation three so that means we take the equation three here"}, {"start": 1545.4399999999998, "end": 1551.4399999999998, "text": " so having formed the class conditional gaussians using the parameters above so the the mean here"}, {"start": 1551.44, "end": 1560.16, "text": " and the covariance we can basically sample a set of these outliers from those distributions so that's"}, {"start": 1560.16, "end": 1565.3600000000001, "text": " the next step here and then they say calculate the regularization loss using equation five"}, {"start": 1566.0, "end": 1573.6000000000001, "text": " and update the parameters using equation seven so regularization number five basically that's how"}, {"start": 1573.6000000000001, "end": 1578.8, "text": " we calculate the loss the uncertainty component and then finally the whole loss consists out of"}, {"start": 1578.8, "end": 1585.04, "text": " the classification loss in case we have object detection we have this localization loss i guess"}, {"start": 1585.04, "end": 1589.28, "text": " and uncertainty component okay and finally once you train the model like that you can use"}, {"start": 1590.0, "end": 1597.2, "text": " the equations eight and nine to determine whether the object at hand is od or id and that's pretty"}, {"start": 1597.2, "end": 1601.52, "text": " much it there was i'm aware there was a lot of nitty-gritty details now let me show you the"}, {"start": 1601.52, "end": 1607.04, "text": " results so first thing to keep in mind is that they are only comparing against methods that are"}, {"start": 1607.04, "end": 1615.76, "text": " not using external outlier data sets if you do have external outlier data set those models those"}, {"start": 1616.3999999999999, "end": 1621.92, "text": " yeah those methods tend to perform better and they did have some results comparing against those"}, {"start": 1621.92, "end": 1627.68, "text": " methods and they are a bit worse now the problem with those data sets is they are very usually very"}, {"start": 1627.68, "end": 1632.8799999999999, "text": " expensive depends on your domain of application they're going to be more or less expensive to"}, {"start": 1632.88, "end": 1639.0400000000002, "text": " acquire so having methods such as this one where you don't need any additional data set you just"}, {"start": 1639.0400000000002, "end": 1643.8400000000001, "text": " need your id and then you have this clever way of sampling the the virtual outliers in the space in"}, {"start": 1643.8400000000001, "end": 1650.16, "text": " the feature space and this just works very nicely compared to other baselines here you can see that"}, {"start": 1650.8000000000002, "end": 1655.68, "text": " across all of these metrics it outperforms the previous methods but again keep in mind"}, {"start": 1655.68, "end": 1660.64, "text": " they are not comparing i think they even say here yeah they say it here all baseline methods are"}, {"start": 1660.64, "end": 1668.16, "text": " based on a model train on id data only that's the important part here okay cool now one thing worth"}, {"start": 1669.0400000000002, "end": 1674.3200000000002, "text": " thinking about is the following for each in distribution class we use thousand samples"}, {"start": 1674.3200000000002, "end": 1678.8000000000002, "text": " to estimate the class conditional gossians so that means they have to take thousand images"}, {"start": 1679.6000000000001, "end": 1685.2800000000002, "text": " embed them and then calculate the the mean and the and the and the covariance matrix and then they"}, {"start": 1685.28, "end": 1692.08, "text": " say since the threshold epsilon can be infinitesimally small we instead choose epsilon based on the teeth"}, {"start": 1692.6399999999999, "end": 1698.6399999999999, "text": " smallest likelihood in a pool of 10 000 samples per class generated from the class conditional"}, {"start": 1698.6399999999999, "end": 1704.32, "text": " gossian distribution so once you form the the gossian distribution using these thousand samples"}, {"start": 1704.32, "end": 1711.04, "text": " then what they do is as far as i understand here they take another 10 000 samples and then whatever"}, {"start": 1711.04, "end": 1717.28, "text": " out of those 10 000 samples what whatever whatever was the the lowest probability of a sample"}, {"start": 1717.28, "end": 1723.76, "text": " that's what they use as the epsilon and then they use that epsilon to sample the virtual outliers"}, {"start": 1723.76, "end": 1729.92, "text": " now the problem here is that this can be very expensive and the thing is in none of these"}, {"start": 1729.92, "end": 1736.72, "text": " tables did they compare the computational budget used to train these methods so that's something"}, {"start": 1736.72, "end": 1743.1200000000001, "text": " to definitely keep in mind if you're if you're on budget anyways let's continue here and see"}, {"start": 1743.1200000000001, "end": 1747.92, "text": " what other results they have so here they compare against some other types of synthesis methods"}, {"start": 1747.92, "end": 1754.64, "text": " i did mention the gan approach where you're trying to in the image space generate the the the outliers"}, {"start": 1754.64, "end": 1762.96, "text": " so the ood data and then use those to to train the system they also compare against these negative"}, {"start": 1762.96, "end": 1769.68, "text": " proposal methods etc etc and they again show up like superior results here okay let's continue"}, {"start": 1770.32, "end": 1777.04, "text": " yeah yeah they do all these ablations like having k plus one classes instead of k classes so basically"}, {"start": 1777.04, "end": 1783.44, "text": " this plus one is where the training happens so they use that plus one class as a detector of"}, {"start": 1783.44, "end": 1791.92, "text": " whether the data whether the um embedding vector is belongs to id data or to od data that's the"}, {"start": 1791.92, "end": 1795.92, "text": " trick there okay let me just show you the visualizations and we are pretty much done"}, {"start": 1797.1200000000001, "end": 1804.0800000000002, "text": " on the top here you can see faster rcnn and on the bottom you can see vos and you can see that"}, {"start": 1804.0800000000002, "end": 1812.48, "text": " eos successfully detects that the bet here and some other objects are od whereas the faster rcnn"}, {"start": 1812.48, "end": 1818.24, "text": " fails completely and is and it's confident in the prediction so it's confident that the the the bet"}, {"start": 1818.24, "end": 1827.04, "text": " here is car with 90 97 percent certainty that's that's weird um cool let me just show you one"}, {"start": 1827.04, "end": 1835.28, "text": " single detail here in the appendix if i can find it um and that's this part here so they say that"}, {"start": 1835.28, "end": 1840.32, "text": " uh importantly we show that uncertainty regularization should be added in the middle of"}, {"start": 1840.32, "end": 1845.2, "text": " the training if it is added too early the feature space is not sufficiently discriminative for"}, {"start": 1845.2, "end": 1849.3600000000001, "text": " gossian distribution estimation and that's something i mentioned in the beginning of the video"}, {"start": 1850.0, "end": 1854.24, "text": " basically initially you're just training for the classif you're just minimizing the classification"}, {"start": 1854.24, "end": 1859.1200000000001, "text": " loss the object detection loss if you're doing object detection and only later during the training"}, {"start": 1859.1200000000001, "end": 1865.2, "text": " when when basically the feature space becomes sufficiently discriminative i.e. the the the"}, {"start": 1865.2, "end": 1870.48, "text": " embedding vectors from the same class tend to cluster together only then can you start using"}, {"start": 1870.48, "end": 1876.72, "text": " this method and then it works in any case hopefully like this video if you did consider subscribing"}, {"start": 1876.72, "end": 1880.64, "text": " uh leave a comment down below if you have anything to ask or or any feedback"}, {"start": 1880.64, "end": 1900.4, "text": " uh in any case until next time bye bye"}]