OpenAI CLIP - Connecting Text and Images | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video, I cover the CLIP paper - Learning Transferable Visual Models from Natural Language Supervision.

You'll learn about:
✔️ How the contrastive learning behind CLIP works
✔️ All the nitty-gritty details behind the paper

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Bitter lessons by Sutton: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
✅ CLIP paper: https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language.pdf
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 

00:00 OpenAI's CLIP
02:10 Detailed explanation of the method
06:00 Comparision with SimCLR
12:55 How does the zero-shot part work
20:45 WIT dataset
21:30 Why this method, hint efficiency
28:35 Zero-shot - generalizing to new tasks
31:30 Prompt programming and ensembling
34:00 Zero-shot perf
36:20 Few-shot comparison with best baselines
38:20 How good the zero-shot classifier is?
40:45 Compute error correlation
41:20 Quality of CLIP's embedding space
43:05 Robustness to distribution shift
49:10 Limitations (MNIST failure)
50:30 A short recap

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#clip #openai #nlpsupervision
 Hi there. So a couple of days ago OpenAI published two really exciting research works and among them Dali took pretty much all of the media attention and was all the hype but I found this work clip or learning transferable visual models from natural language supervision really exciting and I want to cover it in this video. So basically the method itself is not new so they it's just a modified version of convert paper but the thing there they introduced here is a concept of using zero shot in computer vision. So they're probably not the first ones but they are really bullish on it. They're just pretty much a continuation of their exploration of the GPT and all of the zero-shot capabilities it had and they're now trying to take that NLP paradigm and bring it back into the computer vision. So yeah, so this paper is mostly about how this model can, how can we create models which are, which mimic the human brain and learn the way we do. So with that being said let me jump into the method and I'll first explain the method and then we'll go through the different experiments they did. Okay, so they say here we demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn soda image representations from scratch on a data set of 400 million image sex pairs collected from the internet and so they they also introduced this new data set because all of so because they really so that's open AI they really are bullish on computation and all the existing data sets weren't leveraging the scale as much as they could and especially since most of the computer vision tasks are benchmarks data sets like like image net are only associated with a simple like structured label and they just want to associate images and text so they had to create their own data set. So without further ado let's let's jump into the method itself and I'll explain it in depth. I'll start doing some high-level explanation and we'll slowly dig deeper into the method. Okay, so this is this is how it looks like. Basically what they do is the following. So they have as I said associated with they have associated image text pairs and the way the thing they do is they first do some data augmentations here for the for the image part and then they have image encoder and they have text encoder and here they just used Resnets and the newly introduced vision transformer whereas for the text encoder they just use the original Vaslani transformer with certain modifications they did for the GPT family of models as well and once you once you have that setup what you do is the following. So you take the text sequence and you just encode it and how do you do that? Well because this is transformer so you basically just embed the text sequence here and you just take the start of sentence token and you take the end of sentence tokens and you place them you just bracket the text sequence using those and now if you know how the transformer works and I hope you do so you just have the forward pass through the transformer and whatever comes out from the last layer of the transformer above the EOS end of sentence token that's the representation they're gonna use. So they just do some simple layer normalization here and they project this using some linear projection layer into the embedding space and we are here. So one through N is just N is just the size of the batch so these are the encodings we get for the text sentences sequences. On the other hand image encoder is pretty straightforward you basically feed through the the image and you get certain representation at the output. So if this was resonant you'd probably just take the output at the after the average pooling part whereas they did some modifications with resonant I won't get into those details right now but yeah you can imagine you just take out some certain representation at after some after that pooling part and you use that as your image representation. You also do that linear projection and you're finally inside this contrastive embedding space. So once we're here we basically on the high level we just want to make sure that this image image one has the highest cosine similarity with this text embedding vector T1 and you want to push all of these down to zero and you want to do the same thing along this column. So you want to make sure because we have asymmetric setting here we have on one side we have textual information and on the other side we have visual information in our embedding space so this is a multimodal embedding space and so they do it both ways so they want to also make sure this one goes up to to to one and all of these are pushed down to zero and you just repeat all of that across the batch and that's it. Now let's let's get one level deeper and see how this exactly works. So this paper was heavily inspired by this convert paper that was working on medical images and they actually achieved some really awesome results so the visual features they trained using their method proved to be really good at some downstream medical tasks. So you can see the the how the pipeline looks like and it's the same thing as I just explained. So we have sequence of we have a sequence of images and sequence of associated textual sequences and basically what they do is they do data augmentations which is really really popular thing to do in these contrastive learning methods and they did some textual augmentations so just extracting single sentences from their text sequences. So this paper on the other hand ditched this thing here because most of the images they had had only one single sentence associated with image so they can't just remove the sentence otherwise they'll they'll lose the supervision signal. So they just ditch this away and they simplified the this augmentation part where the only thing they do is basically they resize the image and they take a square crop. That's everything. No color jitter, no blurring, nothing. So that's that's everything. Now the second thing they did is so they modified is basically this was MLP in the previous work and also so by the way this work this convert paper was inspired by SimClear framework which you probably heard of which achieved really nice results on self-supervision visual representation learning and it introduced this concept of using MLP between the embedding space and between the contrastive embedding space. So in this paper they just ditch this away and they just use a linear projection layer and that's it. So no activation functions you just have one matrix here that projects from the embedding space into the contrasting embedding space. So that's one of the other modification they did to the pipeline. Okay so once we have that let's let's see like some formulas for the things I already intuitively explain how how it functions. So so what we do is the cosine similarity so this thing with these like angle brackets just symbolize the cosine similarity. So once again you have two vectors let's say this is VI and this thing here is UI and because this is a cosine similarity that means these two vectors are normalized they are basically unit norm so the norm is equal to to 1 and so basically when you do the the dot product you end up having cosine of t. So cosine of t where of theta where theta is the angle between those two. So again if the theta is 0 we'll have cosine similarity 1 if the theta is 0 that means they're orthogonal etc. So when we say we want to make these two be equal to 1 that means those vectors are pretty much aligned and that's that's the idea. So t here is the temperature coefficient just it just modifies the softmax distribution to make it a bit more steep and it's actually trainable parameter in this paper and not a hyperparam and this is a familiar softmax thing so basically what you're doing your your summing so you take the cosine this cosine similarity and you just average so in the denominator denominator you just sum up all of these and you get the probability distribution and then you just have a simple cross entropy loss over that softmax distribution which is a pretty usual thing to do. So you can treat this as a classification problem basically you're trying to classify which out of these combinations is the true combination so what's the true class and because as I already mentioned this problem is a symmetric you're basically doing the same thing just this time you put UI here and your your denominator part goes sums over the column and the final loss vector is just a weighted combination of those two and you just go over the whole batch you sum over the batch and you average out so that means you take these and then you also add this one to the loss and you add the row part and you just do that for the whole batch so yeah that hopefully that was detailed enough for you to understand the exact way this method works so so once you have that loss you just back prop the information the gradients through your encoders and you train the image and the text encoders and that's it your model is ready for this zero-shot learning and I'll get to that in a second I just want to mention one more thing about data augmentations and the reason they just took they just did this simple geometric augmentation whereby they just cropped from from the resized image and not use color data so seem clear this framework I already mentioned found this this nice finding so one composition of augmentation stands out random cropping and random color jitter we conjecture that one serious issue when using only random cropping as data augmentation is that most patches from an image share a similar color distribution so that means that the the model can learn to cheat by just looking at the histogram distribution and figure out that those are the same instances now the difference between this framework and and clip model is that they had they just compared images to images in contrast to loss whereas we have now text an image so the reason I think they just ditched away with with color jitter is the efficiency and we'll see that in a couple of minutes because yeah that's open AI there they're just trying to scale things and and see how they how they perform with that scale so what you can see here is the following so this thing here is basically you have an image and if you do know if you if you if you just ditch the color jitter and you just take a crop and you take another patch and you take another patch and you take another patch so all of these patches will have as you can see here for four different patches they'll have a similar color distribution and so once you do the contrastive learning between those augmented patches they it will be a really easy to even they they are visually different the histograms are the same so it will be a really easy problem to just place them in the same part of the embedding space where here once you do the color jitter you can see that the model can't explain cannot exploit this this this this this trick and that's the reason they use color jitter but as I said probably because of the efficiency reasons so it was a trade-off they just used the random cropping okay that was it now let's focus on this part and this is probably arguably the most important part of the of this paper and that's how did they use this thing now so after they trained it how did they use it in a zero shot setting and this is how so basically let's say you have you now want to just test this one in a zero shot a setting to a new you know on the nude like downstream computer vision tasks like we have cipher 10 and that's a pretty famous and simple benchmark in computer vision and you basically have ten classes right and so maybe plane car dog bird etc so what you do is the following you basically embed you embed those classes in sentences of this construction maybe a photo of and then you just fill in the blank you put a plane your car dog whatever so and once you do that you just encode those sentences in the same way so you have end of sentence token blah blah this is transformer but this time this one is trained so we have it initialized with certain weights after the training after this contrastive learning training and so we just prepare all of those embeddings so we just pre compute them once and that's it we are we are finished with text encoder we just have this dispatch of embeddings here and now how we do the zero shot classification is the following we just take the image that we want to get classified we find its embedding vector and we just do a simple cosine similarity between this embedding vector and between these embedding textual embedding vectors and whatever the highest similarity is that's our class and you can see here a photo of a dog and because t3 corresponds to that particular sentence and had the highest similarity so maybe this one has 0.7 and all of the others had maybe I know 0.1 this one had even smaller similarity etc so that that's how the method works in a nutshell and yeah hopefully this was clear enough and they also mentioned here at test time the learned text encoder synthesizes our zero shot linear classifier so synthesizes that's an interesting word because you can basically treat this as a as a hyper network so you can basically treat this whole system with text encoder and with these sentences as an adaptable as a hyper network so basically depending on the on these on your data set you construct certain corpus of sentences and those will adapt this text encoder so that it's like a configurable linear classifier in a way let's call it that way so if you're still having problems understanding why this is a linear configurable classifier in a way let's look at it from a different perspective so as I mentioned we have those linear projection layers from the embedding space of the encoders into the contrast of embedding space so that means after the embedding space here we'll have a linear projection which will project this vector into this vector so that we'll end up having same dimensions across these image vectors and these text vectors so that means this one is d dimensions and all of these have d dimensions so let me take a look let me just extract this box and represent it as a vector which it is and these are also d vectors so when we do when we do cosine similarity when we do dot product we're basically doing in a way where we are creating a fully connected layer so these weights here these weights correspond to maybe t1 and we'll end up with a value here and we do the same thing with t2 t2 is also fully connected layer over this vector here and we end up with a second value and we do that n times so we'll have n values here so that's like n heads and you can see that all of these weights here are basically configured depending on the text corpus because text encoder is pre-trained and it's fixed and we'll just we just input these this text corpus that's created out of the labels for that particular data set and we end up with particular weights here which again are basically specifying the weights in this classifier and that's why it's a configurable classifier and so that's really interesting so as your shot linear classifier by embedding the names or descriptions of target data set classes and that's we already saw that so we just embedded so one detail I'll just I'll omit it here is that basically what I do here is in sampling and prompt programming so if you're familiar with the GPT family of models you know that prop programming is a thing that's just it's called like a software 3.0 and the idea is depending on how you construct these these these the skeleton around the class you'll get different performance out of your model so what I ended up doing is they play with different constructions and they also assembled bunch of these constructions and they just average out these the features here before they they project them into the contrasting embedding space so what that means maybe you maybe they'll have so for car they maybe have a photo of a car but they'll have another sentence like a nice photo of a car for example and they'll just take both of those embeddings and they'll average it out before they project it into contrast space and they they found that that small trick actually improves performance significantly but we'll get to that in a bit more detail a bit later okay so that was that was the explanation and now let's let's dug even a bit deeper and then we'll jump to experiments so let's see as I said we find the clip similar to the GPT family learns to perform a wide set of tasks during pre training including OCR geolocalization action recognition and many others so again if you're familiar with GPT family of models you know that there is this emerging let's call it phenomena happening so where you're basically training your model on on some tasks like GPT-3 was just trained to predict the next token and once you do that they figured out that it learned how to do many other tasks that it wasn't explicitly trained to do like maybe you input a sentence into GPT and you just append too long didn't read words here and then after you sample out of it you'll notice that it actually learned to summarize the this input sentence even though it wasn't trained to do that and the same thing happened here with clip they trained it to as we saw to just associate the the most probable text sequence with the image and along the way it learned to do all of these things so it's that's interesting okay I'll get to this pick in a second let's just go over a couple more things so learning from natural language also has an important advantage over most unsupervised blah blah so basically this says that the way how we train it enabled us to do this flexible zero-shot transfer as you saw with those with the hyper network story I just explained you and yeah I mentioned that they created this WIT data set which has 400 million pairs of images in text so why they did it is because they want to leverage the scale and none of the existing data sets had that so that's why they did it and it's interesting how they created it so they had like a half a million queries which they created like by collecting words that appear at least hundred times in English version of Wikipedia and some other heuristics and they tried to balance it out so that every one of those queries have has a like a similar number of images associated with them that's how they created the data set so but more importantly let's jump into this part about efficiency so let's see why they ended up having the exact same task that I just explained so they say here that Mahajan and his collaborators folks from Facebook required 19 GPU years to train this Resnext model she and others are required 33 TPU v3 core years to train the noisy student so that's a lot of compute and the only thing they are trying to do is predict hundred image net classes and compare that like comparing that to the to learning the open set of visual concepts from natural language seems daunting so if this was so hard to do then just imagine what how much harder it is to train the the the model of predicting like just natural language labels that they're open for they did try first exploring with this discriminating method that has a predictive objective instead of the contrastive objective I just showed you and what vertex paper did is they basically also had image encoder and they had a transformer above it they were trying to predict the exact same words from the caption of that image so you have an image you have a caption associated with it and you're trying to predict the exact same word so that's not the new approach like we've been doing these captioning predictions for a long time and people notice that the visual representations you get out of this tasks so when you for example do some linear probing on these representations that they are really useful for many downstream tasks classification etc so the thing is they tried this one and it was too expensive and we'll see the one of the charts above which will nicely explain this and the second thing they also consider these generative methods like image GPT and what it does it learns to autoregressively just regress the image the input image and it also learns really nice visual representations but the thing is it's not efficient and like efficiency is the key point here this is open AI they want to make the most efficient method and then leverage the computer have both in the Harvard sense and the huge data set they collected so like the bitter lessons by sudden are a nice read if you're not familiar with this but basically the more computer you can throw at the problem usually that that's that's a good idea I'm not I'm not I'm not saying that's the only way we will go forward with the research but that's definitely an option okay and having said that let's let's just jump to this chart and what it did is the following so this transformer language model that's basically that vertex approach I just mentioned so you just basically stick the transformer up on top of the image encoder and you can see that by trying to predict the exact same birds that know from the caption it's the the zero shot image net accuracy is really low and you need a lot of data to accomplish it compare that to just a simple baseline where instead of trying to predict the exact same birds from the transformer you're just trying to predict which words appear in your caption so that's a much simpler problem so you have a like a vocab here like maybe 30k of words and you're just trying to predict which words are present in your current caption and that's this orange line you can see and we already noticed that we have a 3x efficiency gain and finally once we ditch the predictive objective altogether so trying to predict the words and just trying to associate the the textual as sequence with the image we get a boost in efficiency a further boost and now we can truly leverage the computation okay so that was that was it that was it about the efficiency part and now I'll go into some more details about the method and then we'll we'll just jump to exploring the easier shot results they got so they say here that they we train clip from scratch without initializing the image encoder with image net weights or text encoder with pre-trained weights and why they do that is because work like a vision transformer already showed that once you have enough compute and enough data you really don't need to bias your model in any way so you just so what vision transformer showed is that this particular model is better than CNN's given enough data like they use JFT 300 data set and that's Google's proprietary data set and they show that it outperforms CNN's and it learns something similar to CNN's just a bit better obviously so that's why they don't encode it why they don't initialize it with pre-trained weights so I explained most of these so I'll just skip them yeah this is an interesting detail they are not using so they for a resnet they just swap this average pulling layer with the attention pulling layer that's implemented in a transformer style and yeah they're using as I said was funny for the text encoder and I also mentioned this one about EOS so the interesting part about how they scaled their approach is that they used efficient nets ideas so they used ideas from this paper to scale I don't know how to write today okay to scale their their technique and I just highlighted these as a just to show you how much engineering goes into all of their all of their research work and like you can see all of this mixed precision gradient checkpointing is all about making use as efficient as possible of the memory available in their on their hardware so half precision half precision and yeah it still takes a lot of time to train their biggest models so it took 18 days to train on almost 600 we hunted GPUs and 12 days on 256 for the vision transformer which is a vision transformer is more efficient than CNN's and that's one of the points that that paper made and that's why it takes a lot less compute and finally the the best model they had is the vision transformer L that was trained for one epoch on on a higher resolution of 336 pixels and that's that's a final clip model that they that they are using throughout this paper okay finally experiments so as I mentioned the target the goal of the paper is to show that clip has a really nice zero shot transfer performance and what they mean by zero shot transfer performance is a bit more general than what you usually mean by saying that in computer vision so usually and dates here they say here most of these tasks are are testing for distribution shift and domain generalization so what that means is the previous models in computer vision basically you want you want to test if it's if it's if it's having a really good performance even when you change the this underlying distribution so basically here you have a banana like a real-world banana like a sketch and those are called different domains and the other thing that we're testing for is that for example if the model was testing test that was trained on real world objects like banana I don't know dogs cats whatever and you want to see how how easy it is to just do some fine-tuning or do some linear probe on top of the feature space to maybe learn also about a new object like maybe horse and what they mean by zero shot transfer learning in this and this paper is not generalizing to those to those so distribution shifts and domain generalizations but to new tasks so for example that they say here well it is reasonable to say that the SVH and data set which is this one measures the task of street number transcription on the distribution of Google Street View photos is unclear what real task the cipher 10 data set measures and this is cipher 10 so basically if you train the model maybe image net and then you you want to test it on on on cipher 10 like this you're you're basically doing what I already said you're doing basically just a different different distribute just has a different distribution and maybe some different objects whereas if you're testing on SVH n you're basically learning how to do OCR in the wild if your model learns how to extract numbers from these images so that's it so we we want to generalize not only to distribution shifts and generalizations to new objects we also want to generalize to new tasks like OCR and not only two different classification datasets okay let's let's start exploring the paper first this table is not so important they just show so visual engrams are our method from 2016 which had a similar similar notion of zero-shot transfer and that's why they compare with them although it's not comparable because many things like transformers did not exist back then and so yeah they just have much better clip just as much better results on these datasets like image net so before seeing the zero shop and few shot performance of clip let's first go through prompt engineering and assembling details so the first thing they had to cope with is the fact that there is a polysemy problem in some of the downstream computer vision tasks and that means that for example in some of the datasets like image net you basically have two words which mean two different things so different semantics and maybe they give example here construction cranes and cranes the fly so the first thing they had to do is for a given data set and labels to disambiguate between those labels which have the polysemy problems the second thing and I already mentioned this one is they had to construct the the second the sequences like this so instead of just using labels so for example if you have a on a downstream class task you have labeled dog instead of using dog to in your zero shot classifier to configure it you use something like this a photo of a label and by doing that you get a 1.3 percent increase in accuracy and that's like a low low-hanging fruit the second thing they did is they they again they play with prompt programming so instead of just using a photo of a label they also experiment with adding maybe a type of pet if that made sense for the so we found several fine-grained image classification datasets that it helped to specify the category so for example in Oxford pets using that one would even help them even boost the performance even higher so on OCR just putting the quotes around the numbers would also gained gave them some additional boost they also played with and sampling as I mentioned so here is a concrete example so you by using a photo of a big for example dog and a photo of a small dog and combining those in the in the embedding feature space they get they get an additional boosting performance so they did not average in the probability space they just average before they do the the the soft max and everything so considering those two things together they get a boost of almost 5% and that's on image net so that's that's a really long low-hanging fruit and they made use of it so let's see that on the chart here basically for a given compute by just using those additional in sampling and pro programming programming methods they just get a huge improvements in the average score so that's that's that's the first thing I wanted to mention now let's jump to the zero shot performance of clip so what I did here is they took off-the-shelf models such as resonant hundred I think no resonance 50 here and they just fully pre fine-tuned it on every single data set you see here so it they pre trained it on STL 10 Stanford cars etc and they took on the other side it took clip in a zero shot setting and we can see that clip is much better than fully supervised resident resident 50 on many of the data sets and now if we just analyze the extremes here this part and this part that will probably give us some more information what's happening here so first things first the data set zero shot clip improves by the most is STL 10 a data set designed to encourage efficient learning by providing only a limited number of labeled examples so what that means is that for this particular data set because resonant was you basically take the resident features that were that we got from image net and we freeze those and then we take the classifier and training that one with this many with this like this small amount of examples just didn't didn't give us a good results so you can see that clip was much better than resonant and on the other hand we have this other part of spectrum where if we analyze these data sets you can see that these data sets are complex and specialized so they say it here looking where it underperforms it works really weak on on these data sets like satellite image classification lymph node tumor counting objects blah blah blah and these results highlight the poor capability of zero shot clip or more complex complex tasks so that's where it underperforms resonant but it's still quite fascinating that a fully supervised model using labeled information is underperforming clip on on many of the data sets so it's worth mentioning of course resonant 50 is by no means state-of-the-art currently so let's now examine what happens when we compare clip with state-of-the-art but in a few shots few shot setting this time so these models are not trained like with all of the labels they just take a couple of labels so let's analyze the chart and they took three models first is resonant 50 sim clear the self supervised framework I mentioned and bit big transfer so this one was trained on JFT 300 so a lot of data and we can see that with 16 so we just so what we do here is we take those models like bit and we freeze the weights we add the classifier and we we pre-trip we we fine-tune on these different data sets and we see that for 16 examples we are approaching the zero shot clip which is a nice nice nice result because these are the best methods available currently and what's interesting here is that there is this drop between zero shot and one shot and basically that discrepancy shouldn't be there in in ideal world because humans for example when you when you give us one example we won't be underperforming like yeah we'll we'll we'll be better so that's something they're trying to to to to work on and the potential solution they propose is just to initialize so once you take the vision encoder of clip and instead of training a classifier from random they just initialize it with the zero classifier zero shot classifiers weights for that particular data set and use that but that still didn't help them so they just left it out for for the future for the future exploration let's continue and see how the how good the zero shot classifier actually is and here we see what we see in this chart is for example how many labeled examples do we need to achieve the same accuracy as zero shot classifier so for fair for fair 2013 if we just take that division encoder of clip and use a random initial classifier we need hundred eighty four examples to achieve the same performance as zero shot clip so that's that's a good result we we want to make zero shot classifier as good as possible so that even with a lot of label data we can improve upon it so that would be an ideal result if every single data set required a lot of labeled examples to improve upon that randomly initialized classifier that would be a good result here on the other chart we can see the difference between a linear probe clip performance versus a zero shot performance so this time you take that classifier and you basically train it with all the available data so it's fully supervised and we use all of the data available in a particular data set and we see that for a given data set like maybe this one the the performance is maybe 45% once we use all once we leverage all of the data but by using zero shot we have only 35 so obviously zero shot a classifier is suboptimal for this data set and we can see the general trend so in a perfect world zero shot would be as good as this classifier that used the whole corpus of label data and but it's you can see the trend it's suboptimal and it's down by maybe 20 or 15 points and only a couple of data says like STL 10 and if you remember that one has a little like a small amount of a label data available so here the the zero shot performance is similar to this fully fully trained classifier so that shows us that there is a lot more like space to to improve upon this zero shot classifier let's continue analyze other charts so this one what it tells us is that basically increasing the compute will decrease the error which is something we could expect but the interesting thing is it's really noisy so if we take a particular data set we can see that with compute sometimes the error increase increases and then it decreases so it's really noisy but the overall trend is still strong in favor of more computation is better here what we see is that basically the future space of clip is high-quality because once we do once we compare it with different models like you can see here efficient nets and be all and moco sim cleared like self supervised methods and bid which is fully supervised we can see that the feature space is higher quality for clip and so what we see on this diagram is the amount of compute on x-axis and the average score across a suit of different like 27 data sets and every single model was had a had a linear probe fully fully trained on the frozen feature space and a clip again shows much better much better average score which is something that we ideally want we want the clip to to be the features we train for clip are are really good for many downstream data sets and it showed that here if we take one specific model like maybe efficient net noisy student and we analyze we break down those instead of using the average score we just see the performance across every single data set we can see where the the results for clip are much better and where it's a bit worse than efficient net and because efficient net was basically pre trained on image net so it does make sense that it has better performance than clip that the feature space he developed is better for image net but only for image and I was as we can see on most of the other data sets the future space developed by clip is way more high quality than than efficient nets feature space okay let's continue and see some other results I'll skip this one not that interesting and I'll jump to this one basically if we take what we see here is the following we want to see we want to see how how the clip accuracy behaves when we change the underlying distribution so we see here basically same class same object same classes but like different distribution so here we have sketches here we have real-world objects some like again some drawings etc and if we take for example a model that had 80% accuracy on image net top 1 accuracy we see that for all of those standard image net models that they severely have like have much lower accuracy on those other related data sets were just with different distribution whereas on the other hand if we take the results for for clip it has much higher much higher accuracy than then then then image at once and ideally again if we have 80% on image net we want to have 80% on all of those similar data sets just with different distributions if we take a look at the tabular data here what we see is when we take resonant 101 and so it was just pre trained on image and we don't fine-tune it on any of these new datasets with different distributions we just use it as it is and we see that the performance that the accuracy just severely drops and for these adversarial examples like you can see here some chopped banana something that the model never saw during training it has really really bad accuracy on the other hand clips got much much higher accuracy and it improves like even 75% in this case on this data set so that shows that clip is much more robust to distributional shift than these other other methods let's see some other results they showed and this one is interesting so this line is the same as the one above so we just use clip in a zero shot setting and if we actually fine-tune clip for image net we can see that even though it improves the image net accuracy the the actual robustness to to distributional shift so again we have those data sets like those seven data sets we see that the curves actually go go down and it's it's less robust to to all of those data sets from another perspective if we look at this chart it's it's had some improvements on image net whereas on many other data sets it just has less less good performance so was this adapt to class shift so I think I noted somewhere here basically because some of these data sets like YouTube BB and image net vid have different classes than image net which has thousand classes so it has so for the person class in YouTube BB we have what we have to do for those image net models is at collect the scores over baseball player bridegroom and scuba diver and that's how we give the prediction for person on YouTube BB which is suboptimal but since clip is configurable we can just use this new class and create the construction a type of a person or whatever to to have much better results than than these fixed image net models and when we do that we we have we can see that the trend improves and the robustness increases further which is really nice and particularly on those couple of data sets which has which have that superset of classes this small trick helps whereas the other methods as I already explained have to do the following so for example this this data set and has maybe class person so when we're testing when we're testing efficient net we have to pull over three classes out of thousand that he has to give an estimate of what person would be since he doesn't know of the concept of a person and on the other hand we can just use the flexibility of a zero shot classifier and get the direct results instead I'll just I'll just skip some of these because this is similar experiments to these ones is just that instead of doing showing us the fully supervised curve so this model is clip this that's fine-tuned to image net fully fine-tuned and instead of that they just show a trend once we start from few shots so two shots four shot as we start increasing the number of labels from image and we can see that we are slowly increasing the the image net top one accuracy but the we're still underneath zero shot clip on this on these seven natural distribution data sets okay there's there's a lot of information here the main idea is basically that clip clips feature space is way better suited to to just extend to many other downstream tasks and it's much more robust as we saw two different natural shifts so in a way it's it's a better way to and additionally it's not using those costly supervised data sets is just using those pairings and so it's easier to to to just get a get a get a grab of those of that data that's the main idea of the paper this shows some contamination analysis and finally I just want to show that there are some examples where it's not working really good and a nice nice example of a failure is on amnest where it achieves only 88 percent accuracy on amnest just for comparison if you take a simplest baseline so you just take amnest image which is 28 by 28 you just flatten out the pixels and you use a simple feed for net this one will achieve higher higher accuracy than clip which was trained on a lot of data and lots of compute so it really failed on amnest so as they say here this suggests clip does little to address the underlying problem of brittle generalization of deep learning models instead clips tries to circumvent the problem and hopes that by training in such a large and varied data set at all data will effectively be in distribution and this is a naive assumption that as amnest demonstrates is easy easy to violate so obviously this is not the the final the final way to go to towards towards really generalized model to really general models but it's a it's a it's a step forward and I'm excited about this work so yeah that was that was pretty much it a short recap because there was a lot of details and hopefully this will make it kind of consolidate in your head and so they care about zero-shot performance generalization to new tasks and not only a new distributions and a domain generalization they show that prompt programming and assembling does help boost the performance of clip models so that's important they show that even against a fully supervised baseline such as the resonance 50 they are comparable and those are exciting results they show that even against the best baselines in a few shot setting they are comparable in a zero shot setting and they are better when they're doing the same thing when they're also fine-tuning their model I linear probing actually here next they show that zero shot classifier can be further improved it's suboptimal to the classifiers which can which are imaginable by just fine-tuning them on specific data sets so there is a lot of there is more work that can be done on improving that again compute doesn't does improve a clip features are better so feature space that clip provides is better than other baselines in compared to a specific baseline efficient net here we can see that it's much better on many data sets so again feature space is richer than than the other baselines here it shows a impressive generalization to a robustness to distribution shift and same thing here and they show that if you want if you if you kind of add a classifier and linear probe the clip onto image net you basically do improve image and performance but you lose the robustness to other data sets so you don't want to overfit to imagine basically and finally they showed some limitations among them that it's failing on on on simple data sets such as MNIST because it never saw those images during training so it just fails miserably even compared to the simplest of baselines okay those are there was a much longer video than I expected but hopefully you learned something from this video if you did consider subscribing and sharing this video and also hit the bell icon to get notified when I upload a new video until next time keep learning deep
[{"start": 0.0, "end": 7.5600000000000005, "text": " Hi there. So a couple of days ago OpenAI published two really exciting research"}, {"start": 7.5600000000000005, "end": 14.24, "text": " works and among them Dali took pretty much all of the media attention and was"}, {"start": 14.24, "end": 20.84, "text": " all the hype but I found this work clip or learning transferable"}, {"start": 20.84, "end": 26.48, "text": " visual models from natural language supervision really exciting and I want"}, {"start": 26.48, "end": 32.24, "text": " to cover it in this video. So basically the method itself is not new so they"}, {"start": 32.24, "end": 38.86, "text": " it's just a modified version of convert paper but the thing there they"}, {"start": 38.86, "end": 44.32, "text": " introduced here is a concept of using zero shot in computer vision. So they're"}, {"start": 44.32, "end": 47.92, "text": " probably not the first ones but they are really bullish on it. They're just"}, {"start": 47.92, "end": 52.84, "text": " pretty much a continuation of their exploration of the GPT and all of the"}, {"start": 52.84, "end": 57.28, "text": " zero-shot capabilities it had and they're now trying to take that NLP"}, {"start": 57.28, "end": 63.800000000000004, "text": " paradigm and bring it back into the computer vision. So yeah, so this paper is"}, {"start": 63.800000000000004, "end": 68.76, "text": " mostly about how this model can, how can we create models which are, which mimic"}, {"start": 68.76, "end": 73.76, "text": " the human brain and learn the way we do. So with that being said let me jump into"}, {"start": 73.76, "end": 78.08000000000001, "text": " the method and I'll first explain the method and then we'll go through the"}, {"start": 78.08, "end": 84.44, "text": " different experiments they did. Okay, so they say here we demonstrate that the"}, {"start": 84.44, "end": 88.72, "text": " simple pre-training task of predicting which caption goes with which image is"}, {"start": 88.72, "end": 93.44, "text": " an efficient and scalable way to learn soda image representations from scratch"}, {"start": 93.44, "end": 98.52, "text": " on a data set of 400 million image sex pairs collected from the internet and so"}, {"start": 98.52, "end": 104.2, "text": " they they also introduced this new data set because all of so because they"}, {"start": 104.2, "end": 109.56, "text": " really so that's open AI they really are bullish on computation and all the"}, {"start": 109.56, "end": 115.2, "text": " existing data sets weren't leveraging the scale as much as they could and"}, {"start": 115.2, "end": 121.2, "text": " especially since most of the computer vision tasks are benchmarks data sets"}, {"start": 121.2, "end": 127.64, "text": " like like image net are only associated with a simple like structured label and"}, {"start": 127.64, "end": 132.24, "text": " they just want to associate images and text so they had to create their own"}, {"start": 132.24, "end": 136.32000000000002, "text": " data set. So without further ado let's let's jump into the method itself and"}, {"start": 136.32000000000002, "end": 141.36, "text": " I'll explain it in depth. I'll start doing some high-level explanation and"}, {"start": 141.36, "end": 147.48000000000002, "text": " we'll slowly dig deeper into the method. Okay, so this is this is how it looks"}, {"start": 147.48000000000002, "end": 151.20000000000002, "text": " like. Basically what they do is the following. So they have as I said"}, {"start": 151.20000000000002, "end": 157.68, "text": " associated with they have associated image text pairs and the way the thing"}, {"start": 157.68, "end": 162.4, "text": " they do is they first do some data augmentations here for the for the image"}, {"start": 162.4, "end": 168.92000000000002, "text": " part and then they have image encoder and they have text encoder and here they"}, {"start": 168.92000000000002, "end": 174.32, "text": " just used Resnets and the newly introduced vision transformer whereas"}, {"start": 174.32, "end": 179.60000000000002, "text": " for the text encoder they just use the original Vaslani transformer with"}, {"start": 179.60000000000002, "end": 184.36, "text": " certain modifications they did for the GPT family of models as well and once"}, {"start": 184.36, "end": 189.72000000000003, "text": " you once you have that setup what you do is the following. So you take the text"}, {"start": 189.72000000000003, "end": 194.0, "text": " sequence and you just encode it and how do you do that? Well because this is"}, {"start": 194.0, "end": 198.44000000000003, "text": " transformer so you basically just embed the text sequence here and you just"}, {"start": 198.44000000000003, "end": 206.44000000000003, "text": " take the start of sentence token and you take the end of sentence tokens and you"}, {"start": 206.44000000000003, "end": 213.64000000000001, "text": " place them you just bracket the text sequence using those and now if you know"}, {"start": 213.64, "end": 217.07999999999998, "text": " how the transformer works and I hope you do so you just have the forward pass"}, {"start": 217.07999999999998, "end": 221.39999999999998, "text": " through the transformer and whatever comes out from the last layer of the"}, {"start": 221.39999999999998, "end": 226.79999999999998, "text": " transformer above the EOS end of sentence token that's the representation"}, {"start": 226.79999999999998, "end": 231.32, "text": " they're gonna use. So they just do some simple layer normalization here and they"}, {"start": 231.32, "end": 234.76, "text": " project this using some linear projection layer into the embedding"}, {"start": 234.76, "end": 239.56, "text": " space and we are here. So one through N is just N is just the size of the batch"}, {"start": 239.56, "end": 246.8, "text": " so these are the encodings we get for the text sentences sequences. On the"}, {"start": 246.8, "end": 252.36, "text": " other hand image encoder is pretty straightforward you basically feed"}, {"start": 252.36, "end": 257.2, "text": " through the the image and you get certain representation at the output. So"}, {"start": 257.2, "end": 262.22, "text": " if this was resonant you'd probably just take the output at the after the"}, {"start": 262.22, "end": 266.2, "text": " average pooling part whereas they did some modifications with resonant I"}, {"start": 266.2, "end": 269.8, "text": " won't get into those details right now but yeah you can imagine you just take"}, {"start": 269.8, "end": 275.92, "text": " out some certain representation at after some after that pooling part and you use"}, {"start": 275.92, "end": 280.44, "text": " that as your image representation. You also do that linear projection and"}, {"start": 280.44, "end": 287.32, "text": " you're finally inside this contrastive embedding space. So once we're here we"}, {"start": 287.32, "end": 292.32, "text": " basically on the high level we just want to make sure that this image image one"}, {"start": 292.32, "end": 301.12, "text": " has the highest cosine similarity with this text embedding vector T1 and you"}, {"start": 301.12, "end": 305.44, "text": " want to push all of these down to zero and you want to do the same thing along"}, {"start": 305.44, "end": 309.56, "text": " this column. So you want to make sure because we have asymmetric setting here"}, {"start": 309.56, "end": 313.88, "text": " we have on one side we have textual information and on the other side we"}, {"start": 313.88, "end": 318.12, "text": " have visual information in our embedding space so this is a multimodal embedding"}, {"start": 318.12, "end": 322.44, "text": " space and so they do it both ways so they want to also make sure this one goes"}, {"start": 322.44, "end": 327.68, "text": " up to to to one and all of these are pushed down to zero and you just repeat"}, {"start": 327.68, "end": 334.08, "text": " all of that across the batch and that's it. Now let's let's get one level deeper"}, {"start": 334.08, "end": 340.76, "text": " and see how this exactly works. So this paper was heavily inspired by this"}, {"start": 340.76, "end": 345.42, "text": " convert paper that was working on medical images and they actually"}, {"start": 345.42, "end": 351.04, "text": " achieved some really awesome results so the visual features they trained using"}, {"start": 351.04, "end": 357.48, "text": " their method proved to be really good at some downstream medical tasks. So you can"}, {"start": 357.48, "end": 361.28000000000003, "text": " see the the how the pipeline looks like and it's the same thing as I just"}, {"start": 361.28000000000003, "end": 365.92, "text": " explained. So we have sequence of we have a sequence of images and sequence of"}, {"start": 365.92, "end": 372.08000000000004, "text": " associated textual sequences and basically what they do is they do data"}, {"start": 372.08, "end": 375.32, "text": " augmentations which is really really popular thing to do in these"}, {"start": 375.32, "end": 381.32, "text": " contrastive learning methods and they did some textual augmentations so just"}, {"start": 381.32, "end": 386.4, "text": " extracting single sentences from their text sequences. So this paper on the"}, {"start": 386.4, "end": 392.03999999999996, "text": " other hand ditched this thing here because most of the images they had had"}, {"start": 392.03999999999996, "end": 396.34, "text": " only one single sentence associated with image so they can't just remove the"}, {"start": 396.34, "end": 399.84, "text": " sentence otherwise they'll they'll lose the supervision signal. So they just"}, {"start": 399.84, "end": 405.08, "text": " ditch this away and they simplified the this augmentation part where the only"}, {"start": 405.08, "end": 410.76, "text": " thing they do is basically they resize the image and they take a square crop."}, {"start": 410.76, "end": 416.55999999999995, "text": " That's everything. No color jitter, no blurring, nothing. So that's that's"}, {"start": 416.55999999999995, "end": 422.12, "text": " everything. Now the second thing they did is so they modified is basically this was"}, {"start": 422.12, "end": 427.71999999999997, "text": " MLP in the previous work and also so by the way this work this convert paper"}, {"start": 427.72, "end": 434.28000000000003, "text": " was inspired by SimClear framework which you probably heard of which achieved"}, {"start": 434.28000000000003, "end": 440.16, "text": " really nice results on self-supervision visual representation learning and it"}, {"start": 440.16, "end": 444.24, "text": " introduced this concept of using MLP between the embedding space and between"}, {"start": 444.24, "end": 448.88000000000005, "text": " the contrastive embedding space. So in this paper they just ditch this away and"}, {"start": 448.88000000000005, "end": 453.68, "text": " they just use a linear projection layer and that's it. So no activation functions"}, {"start": 453.68, "end": 458.16, "text": " you just have one matrix here that projects from the embedding space into"}, {"start": 458.16, "end": 463.16, "text": " the contrasting embedding space. So that's one of the other modification"}, {"start": 463.16, "end": 468.82, "text": " they did to the pipeline. Okay so once we have that let's let's see like some"}, {"start": 468.82, "end": 472.64, "text": " formulas for the things I already intuitively explain how how it functions."}, {"start": 472.64, "end": 478.56, "text": " So so what we do is the cosine similarity so this thing with these like"}, {"start": 478.56, "end": 484.64, "text": " angle brackets just symbolize the cosine similarity. So once again you have two"}, {"start": 484.64, "end": 491.96, "text": " vectors let's say this is VI and this thing here is UI and because this is a"}, {"start": 491.96, "end": 495.76, "text": " cosine similarity that means these two vectors are normalized they are"}, {"start": 495.76, "end": 502.6, "text": " basically unit norm so the norm is equal to to 1 and so basically when you do the"}, {"start": 502.6, "end": 510.48, "text": " the dot product you end up having cosine of t. So cosine of t where of theta where"}, {"start": 510.48, "end": 515.78, "text": " theta is the angle between those two. So again if the theta is 0 we'll have"}, {"start": 515.78, "end": 522.4, "text": " cosine similarity 1 if the theta is 0 that means they're orthogonal etc. So when"}, {"start": 522.4, "end": 529.2, "text": " we say we want to make these two be equal to 1 that means those vectors are"}, {"start": 529.2, "end": 534.96, "text": " pretty much aligned and that's that's the idea. So t here is the temperature"}, {"start": 534.96, "end": 539.6, "text": " coefficient just it just modifies the softmax distribution to make it a bit"}, {"start": 539.6, "end": 544.12, "text": " more steep and it's actually trainable parameter in this paper and not a"}, {"start": 544.12, "end": 549.6800000000001, "text": " hyperparam and this is a familiar softmax thing so basically what you're"}, {"start": 549.6800000000001, "end": 554.8000000000001, "text": " doing your your summing so you take the cosine this cosine similarity and you"}, {"start": 554.8, "end": 561.04, "text": " just average so in the denominator denominator you just sum up all of these"}, {"start": 561.04, "end": 566.0799999999999, "text": " and you get the probability distribution and then you just have a simple cross"}, {"start": 566.0799999999999, "end": 570.68, "text": " entropy loss over that softmax distribution which is a pretty usual"}, {"start": 570.68, "end": 574.16, "text": " thing to do. So you can treat this as a classification problem basically you're"}, {"start": 574.16, "end": 579.52, "text": " trying to classify which out of these combinations is the true combination so"}, {"start": 579.52, "end": 584.24, "text": " what's the true class and because as I already mentioned this problem is a"}, {"start": 584.24, "end": 589.6800000000001, "text": " symmetric you're basically doing the same thing just this time you put UI"}, {"start": 589.6800000000001, "end": 597.52, "text": " here and your your denominator part goes sums over the column and the final loss"}, {"start": 597.52, "end": 603.12, "text": " vector is just a weighted combination of those two and you just go over the whole"}, {"start": 603.12, "end": 607.88, "text": " batch you sum over the batch and you average out so that means you take"}, {"start": 607.88, "end": 614.96, "text": " these and then you also add this one to the loss and you add the row part and"}, {"start": 614.96, "end": 621.12, "text": " you just do that for the whole batch so yeah that hopefully that was detailed"}, {"start": 621.12, "end": 627.6, "text": " enough for you to understand the exact way this method works so so once you"}, {"start": 627.6, "end": 631.4, "text": " have that loss you just back prop the information the gradients through your"}, {"start": 631.4, "end": 637.6, "text": " encoders and you train the image and the text encoders and that's it your model"}, {"start": 637.6, "end": 643.9, "text": " is ready for this zero-shot learning and I'll get to that in a second I just want"}, {"start": 643.9, "end": 647.6, "text": " to mention one more thing about data augmentations and the reason they just"}, {"start": 647.6, "end": 653.48, "text": " took they just did this simple geometric augmentation whereby they just cropped"}, {"start": 653.48, "end": 659.6800000000001, "text": " from from the resized image and not use color data so seem clear this framework"}, {"start": 659.6800000000001, "end": 665.36, "text": " I already mentioned found this this nice finding so one composition of"}, {"start": 665.36, "end": 670.6800000000001, "text": " augmentation stands out random cropping and random color jitter we conjecture"}, {"start": 670.6800000000001, "end": 675.08, "text": " that one serious issue when using only random cropping as data augmentation is"}, {"start": 675.08, "end": 680.2, "text": " that most patches from an image share a similar color distribution so that means"}, {"start": 680.2, "end": 685.4, "text": " that the the model can learn to cheat by just looking at the histogram"}, {"start": 685.4, "end": 689.36, "text": " distribution and figure out that those are the same instances now the difference"}, {"start": 689.36, "end": 695.6800000000001, "text": " between this framework and and clip model is that they had they just"}, {"start": 695.6800000000001, "end": 699.6800000000001, "text": " compared images to images in contrast to loss whereas we have now text an image"}, {"start": 699.6800000000001, "end": 704.92, "text": " so the reason I think they just ditched away with with color jitter is the"}, {"start": 704.92, "end": 709.48, "text": " efficiency and we'll see that in a couple of minutes because yeah that's"}, {"start": 709.48, "end": 713.8000000000001, "text": " open AI there they're just trying to scale things and and see how they how"}, {"start": 713.8, "end": 719.92, "text": " they perform with that scale so what you can see here is the following so this"}, {"start": 719.92, "end": 725.56, "text": " thing here is basically you have an image and if you do know if you if you"}, {"start": 725.56, "end": 730.5999999999999, "text": " if you just ditch the color jitter and you just take a crop and you take"}, {"start": 730.5999999999999, "end": 734.4, "text": " another patch and you take another patch and you take another patch so all of"}, {"start": 734.4, "end": 738.5999999999999, "text": " these patches will have as you can see here for four different patches they'll"}, {"start": 738.5999999999999, "end": 743.56, "text": " have a similar color distribution and so once you do the contrastive learning"}, {"start": 743.56, "end": 748.68, "text": " between those augmented patches they it will be a really easy to even they they"}, {"start": 748.68, "end": 752.2399999999999, "text": " are visually different the histograms are the same so it will be a really easy"}, {"start": 752.2399999999999, "end": 757.8399999999999, "text": " problem to just place them in the same part of the embedding space where here"}, {"start": 757.8399999999999, "end": 762.5999999999999, "text": " once you do the color jitter you can see that the model can't explain cannot"}, {"start": 762.5999999999999, "end": 767.1199999999999, "text": " exploit this this this this this trick and that's the reason they use color"}, {"start": 767.1199999999999, "end": 771.4, "text": " jitter but as I said probably because of the efficiency reasons so it was a"}, {"start": 771.4, "end": 777.3199999999999, "text": " trade-off they just used the random cropping okay that was it now let's"}, {"start": 777.3199999999999, "end": 781.36, "text": " focus on this part and this is probably arguably the most important part of the"}, {"start": 781.36, "end": 786.9599999999999, "text": " of this paper and that's how did they use this thing now so after they trained"}, {"start": 786.9599999999999, "end": 791.04, "text": " it how did they use it in a zero shot setting and this is how so basically"}, {"start": 791.04, "end": 796.16, "text": " let's say you have you now want to just test this one in a zero shot a setting"}, {"start": 796.16, "end": 801.52, "text": " to a new you know on the nude like downstream computer vision tasks like"}, {"start": 801.52, "end": 807.64, "text": " we have cipher 10 and that's a pretty famous and simple benchmark in computer"}, {"start": 807.64, "end": 812.7199999999999, "text": " vision and you basically have ten classes right and so maybe plane car dog"}, {"start": 812.7199999999999, "end": 818.56, "text": " bird etc so what you do is the following you basically embed you embed those"}, {"start": 818.56, "end": 824.98, "text": " classes in sentences of this construction maybe a photo of and then"}, {"start": 824.98, "end": 831.96, "text": " you just fill in the blank you put a plane your car dog whatever so and once"}, {"start": 831.96, "end": 835.6800000000001, "text": " you do that you just encode those sentences in the same way so you have"}, {"start": 835.6800000000001, "end": 841.0600000000001, "text": " end of sentence token blah blah this is transformer but this time this one is"}, {"start": 841.0600000000001, "end": 846.04, "text": " trained so we have it initialized with certain weights after the training after"}, {"start": 846.04, "end": 850.84, "text": " this contrastive learning training and so we just prepare all of those"}, {"start": 850.84, "end": 856.52, "text": " embeddings so we just pre compute them once and that's it we are we are finished"}, {"start": 856.52, "end": 863.2800000000001, "text": " with text encoder we just have this dispatch of embeddings here and now how"}, {"start": 863.2800000000001, "end": 868.4, "text": " we do the zero shot classification is the following we just take the image"}, {"start": 868.4, "end": 873.88, "text": " that we want to get classified we find its embedding vector and we just do a"}, {"start": 873.88, "end": 878.64, "text": " simple cosine similarity between this embedding vector and between these"}, {"start": 878.64, "end": 883.48, "text": " embedding textual embedding vectors and whatever the highest similarity is"}, {"start": 883.48, "end": 891.84, "text": " that's our class and you can see here a photo of a dog and because t3 corresponds"}, {"start": 891.84, "end": 896.28, "text": " to that particular sentence and had the highest similarity so maybe this one has"}, {"start": 896.28, "end": 902.92, "text": " 0.7 and all of the others had maybe I know 0.1 this one had even smaller"}, {"start": 902.92, "end": 908.56, "text": " similarity etc so that that's how the method works in a nutshell and yeah"}, {"start": 908.56, "end": 914.0799999999999, "text": " hopefully this was clear enough and they also mentioned here at test time the"}, {"start": 914.0799999999999, "end": 919.52, "text": " learned text encoder synthesizes our zero shot linear classifier so"}, {"start": 919.52, "end": 923.56, "text": " synthesizes that's an interesting word because you can basically treat this as"}, {"start": 923.56, "end": 928.1199999999999, "text": " a as a hyper network so you can basically treat this whole system with"}, {"start": 928.1199999999999, "end": 932.8399999999999, "text": " text encoder and with these sentences as an adaptable as a hyper network so"}, {"start": 932.8399999999999, "end": 938.16, "text": " basically depending on the on these on your data set you construct certain"}, {"start": 938.16, "end": 945.0, "text": " corpus of sentences and those will adapt this text encoder so that it's like a"}, {"start": 945.0, "end": 949.4, "text": " configurable linear classifier in a way let's call it that way so if you're"}, {"start": 949.4, "end": 954.0, "text": " still having problems understanding why this is a linear configurable classifier"}, {"start": 954.0, "end": 958.9599999999999, "text": " in a way let's look at it from a different perspective so as I mentioned"}, {"start": 958.9599999999999, "end": 964.36, "text": " we have those linear projection layers from the embedding space of the encoders"}, {"start": 964.36, "end": 969.16, "text": " into the contrast of embedding space so that means after the embedding space here"}, {"start": 969.16, "end": 973.6800000000001, "text": " we'll have a linear projection which will project this vector into this"}, {"start": 973.6800000000001, "end": 979.4, "text": " vector so that we'll end up having same dimensions across these image vectors"}, {"start": 979.4, "end": 984.92, "text": " and these text vectors so that means this one is d dimensions and all of"}, {"start": 984.92, "end": 990.08, "text": " these have d dimensions so let me take a look let me just extract this box and"}, {"start": 990.08, "end": 996.08, "text": " represent it as a vector which it is and these are also d vectors so when we do"}, {"start": 996.08, "end": 1000.9200000000001, "text": " when we do cosine similarity when we do dot product we're basically doing in a"}, {"start": 1000.9200000000001, "end": 1005.36, "text": " way where we are creating a fully connected layer so these weights here"}, {"start": 1005.36, "end": 1011.5600000000001, "text": " these weights correspond to maybe t1 and we'll end up with a value here and we"}, {"start": 1011.5600000000001, "end": 1017.72, "text": " do the same thing with t2 t2 is also fully connected layer over this vector"}, {"start": 1017.72, "end": 1022.6, "text": " here and we end up with a second value and we do that n times so we'll have n"}, {"start": 1022.6, "end": 1029.76, "text": " values here so that's like n heads and you can see that all of these weights"}, {"start": 1029.76, "end": 1036.0, "text": " here are basically configured depending on the text corpus because text encoder"}, {"start": 1036.0, "end": 1042.4, "text": " is pre-trained and it's fixed and we'll just we just input these this text"}, {"start": 1042.4, "end": 1046.8, "text": " corpus that's created out of the labels for that particular data set and we end"}, {"start": 1046.8, "end": 1053.04, "text": " up with particular weights here which again are basically specifying the"}, {"start": 1053.04, "end": 1056.6399999999999, "text": " weights in this classifier and that's why it's a configurable classifier and"}, {"start": 1056.6399999999999, "end": 1060.84, "text": " so that's really interesting so as your shot linear classifier by embedding the"}, {"start": 1060.84, "end": 1065.6399999999999, "text": " names or descriptions of target data set classes and that's we already saw that"}, {"start": 1065.6399999999999, "end": 1072.1599999999999, "text": " so we just embedded so one detail I'll just I'll omit it here is that basically"}, {"start": 1072.16, "end": 1078.16, "text": " what I do here is in sampling and prompt programming so if you're familiar with"}, {"start": 1078.16, "end": 1082.64, "text": " the GPT family of models you know that prop programming is a thing that's just"}, {"start": 1082.64, "end": 1088.64, "text": " it's called like a software 3.0 and the idea is depending on how you construct"}, {"start": 1088.64, "end": 1095.16, "text": " these these these the skeleton around the class you'll get different"}, {"start": 1095.16, "end": 1099.72, "text": " performance out of your model so what I ended up doing is they play with"}, {"start": 1099.72, "end": 1103.88, "text": " different constructions and they also assembled bunch of these constructions"}, {"start": 1103.88, "end": 1110.72, "text": " and they just average out these the features here before they they project"}, {"start": 1110.72, "end": 1114.2, "text": " them into the contrasting embedding space so what that means maybe you maybe"}, {"start": 1114.2, "end": 1119.32, "text": " they'll have so for car they maybe have a photo of a car but they'll have"}, {"start": 1119.32, "end": 1125.68, "text": " another sentence like a nice photo of a car for example and they'll just take"}, {"start": 1125.68, "end": 1130.4, "text": " both of those embeddings and they'll average it out before they project it"}, {"start": 1130.4, "end": 1136.2, "text": " into contrast space and they they found that that small trick actually improves"}, {"start": 1136.2, "end": 1140.8, "text": " performance significantly but we'll get to that in a bit more detail a bit later"}, {"start": 1140.8, "end": 1149.28, "text": " okay so that was that was the explanation and now let's let's dug even"}, {"start": 1149.28, "end": 1158.04, "text": " a bit deeper and then we'll jump to experiments so let's see as I said we"}, {"start": 1158.04, "end": 1161.8799999999999, "text": " find the clip similar to the GPT family learns to perform a wide set of tasks"}, {"start": 1161.8799999999999, "end": 1166.76, "text": " during pre training including OCR geolocalization action recognition and"}, {"start": 1166.76, "end": 1171.44, "text": " many others so again if you're familiar with GPT family of models you know that"}, {"start": 1171.44, "end": 1177.24, "text": " there is this emerging let's call it phenomena happening so where you're"}, {"start": 1177.24, "end": 1181.6, "text": " basically training your model on on some tasks like GPT-3 was just trained to"}, {"start": 1181.6, "end": 1187.08, "text": " predict the next token and once you do that they figured out that it learned"}, {"start": 1187.08, "end": 1191.4, "text": " how to do many other tasks that it wasn't explicitly trained to do like"}, {"start": 1191.4, "end": 1199.76, "text": " maybe you input a sentence into GPT and you just append too long didn't read"}, {"start": 1199.76, "end": 1205.16, "text": " words here and then after you sample out of it you'll notice that it actually"}, {"start": 1205.16, "end": 1209.96, "text": " learned to summarize the this input sentence even though it wasn't trained"}, {"start": 1209.96, "end": 1214.4, "text": " to do that and the same thing happened here with clip they trained it to as we"}, {"start": 1214.4, "end": 1219.0600000000002, "text": " saw to just associate the the most probable text sequence with the image"}, {"start": 1219.0600000000002, "end": 1225.2, "text": " and along the way it learned to do all of these things so it's that's"}, {"start": 1225.2, "end": 1231.0800000000002, "text": " interesting okay I'll get to this pick in a second let's just go over a couple"}, {"start": 1231.0800000000002, "end": 1234.52, "text": " more things so learning from natural language also has an important"}, {"start": 1234.52, "end": 1240.8799999999999, "text": " advantage over most unsupervised blah blah so basically this says that the way"}, {"start": 1240.8799999999999, "end": 1245.32, "text": " how we train it enabled us to do this flexible zero-shot transfer as you saw"}, {"start": 1245.32, "end": 1252.96, "text": " with those with the hyper network story I just explained you and yeah I mentioned"}, {"start": 1252.96, "end": 1259.4, "text": " that they created this WIT data set which has 400 million pairs of images"}, {"start": 1259.4, "end": 1264.08, "text": " in text so why they did it is because they want to leverage the scale and none"}, {"start": 1264.08, "end": 1268.52, "text": " of the existing data sets had that so that's why they did it and it's"}, {"start": 1268.52, "end": 1271.96, "text": " interesting how they created it so they had like a half a million queries which"}, {"start": 1271.96, "end": 1276.1599999999999, "text": " they created like by collecting words that appear at least hundred times in"}, {"start": 1276.1599999999999, "end": 1280.52, "text": " English version of Wikipedia and some other heuristics and they tried to"}, {"start": 1280.52, "end": 1285.76, "text": " balance it out so that every one of those queries have has a like a similar"}, {"start": 1285.76, "end": 1290.3999999999999, "text": " number of images associated with them that's how they created the data set so"}, {"start": 1290.4, "end": 1294.5600000000002, "text": " but more importantly let's jump into this part about efficiency so let's see"}, {"start": 1294.5600000000002, "end": 1300.24, "text": " why they ended up having the exact same task that I just explained so they say"}, {"start": 1300.24, "end": 1307.6000000000001, "text": " here that Mahajan and his collaborators folks from Facebook required 19 GPU years"}, {"start": 1307.6000000000001, "end": 1315.3600000000001, "text": " to train this Resnext model she and others are required 33 TPU v3 core years"}, {"start": 1315.3600000000001, "end": 1320.16, "text": " to train the noisy student so that's a lot of compute and the only thing they"}, {"start": 1320.16, "end": 1325.3200000000002, "text": " are trying to do is predict hundred image net classes and compare that like"}, {"start": 1325.3200000000002, "end": 1329.48, "text": " comparing that to the to learning the open set of visual concepts from"}, {"start": 1329.48, "end": 1334.8000000000002, "text": " natural language seems daunting so if this was so hard to do then just imagine"}, {"start": 1334.8000000000002, "end": 1342.4, "text": " what how much harder it is to train the the the model of predicting like just"}, {"start": 1342.4, "end": 1350.16, "text": " natural language labels that they're open for they did try first exploring"}, {"start": 1350.16, "end": 1354.16, "text": " with this discriminating method that has a predictive objective instead of the"}, {"start": 1354.16, "end": 1360.8400000000001, "text": " contrastive objective I just showed you and what vertex paper did is they"}, {"start": 1360.8400000000001, "end": 1365.8400000000001, "text": " basically also had image encoder and they had a transformer above it they"}, {"start": 1365.8400000000001, "end": 1370.72, "text": " were trying to predict the exact same words from the caption of that image so"}, {"start": 1370.72, "end": 1374.1200000000001, "text": " you have an image you have a caption associated with it and you're trying to"}, {"start": 1374.1200000000001, "end": 1378.24, "text": " predict the exact same word so that's not the new approach like we've been"}, {"start": 1378.24, "end": 1385.24, "text": " doing these captioning predictions for a long time and people notice that"}, {"start": 1385.24, "end": 1390.76, "text": " the visual representations you get out of this tasks so when you for example do"}, {"start": 1390.76, "end": 1393.44, "text": " some linear probing on these representations that they are really"}, {"start": 1393.44, "end": 1399.44, "text": " useful for many downstream tasks classification etc so the thing is they"}, {"start": 1399.44, "end": 1404.3200000000002, "text": " tried this one and it was too expensive and we'll see the one of the charts"}, {"start": 1404.3200000000002, "end": 1410.64, "text": " above which will nicely explain this and the second thing they also consider"}, {"start": 1410.64, "end": 1417.76, "text": " these generative methods like image GPT and what it does it learns to"}, {"start": 1417.76, "end": 1426.4, "text": " autoregressively just regress the image the input image and it also learns"}, {"start": 1426.4, "end": 1430.6000000000001, "text": " really nice visual representations but the thing is it's not efficient and like"}, {"start": 1430.6000000000001, "end": 1436.5600000000002, "text": " efficiency is the key point here this is open AI they want to make the most"}, {"start": 1436.5600000000002, "end": 1440.48, "text": " efficient method and then leverage the computer have both in the Harvard sense"}, {"start": 1440.48, "end": 1445.3200000000002, "text": " and the huge data set they collected so like the bitter lessons by sudden are a"}, {"start": 1445.3200000000002, "end": 1449.0800000000002, "text": " nice read if you're not familiar with this but basically the more computer you"}, {"start": 1449.0800000000002, "end": 1454.1200000000001, "text": " can throw at the problem usually that that's that's a good idea I'm not I'm"}, {"start": 1454.12, "end": 1458.1599999999999, "text": " not I'm not saying that's the only way we will go forward with the research"}, {"start": 1458.1599999999999, "end": 1463.6, "text": " but that's definitely an option okay and having said that let's let's just jump"}, {"start": 1463.6, "end": 1469.52, "text": " to this chart and what it did is the following so this transformer language"}, {"start": 1469.52, "end": 1474.76, "text": " model that's basically that vertex approach I just mentioned so you just"}, {"start": 1474.76, "end": 1481.62, "text": " basically stick the transformer up on top of the image encoder and you can see"}, {"start": 1481.62, "end": 1486.0, "text": " that by trying to predict the exact same birds that know from the caption it's"}, {"start": 1486.0, "end": 1491.52, "text": " the the zero shot image net accuracy is really low and you need a lot of data"}, {"start": 1491.52, "end": 1496.8, "text": " to accomplish it compare that to just a simple baseline where instead of trying"}, {"start": 1496.8, "end": 1501.3999999999999, "text": " to predict the exact same birds from the transformer you're just trying to"}, {"start": 1501.3999999999999, "end": 1505.32, "text": " predict which words appear in your caption so that's a much simpler"}, {"start": 1505.32, "end": 1510.8799999999999, "text": " problem so you have a like a vocab here like maybe 30k of words and you're just"}, {"start": 1510.88, "end": 1515.68, "text": " trying to predict which words are present in your current caption and"}, {"start": 1515.68, "end": 1521.92, "text": " that's this orange line you can see and we already noticed that we have a 3x"}, {"start": 1521.92, "end": 1527.6000000000001, "text": " efficiency gain and finally once we ditch the predictive objective altogether"}, {"start": 1527.6000000000001, "end": 1532.92, "text": " so trying to predict the words and just trying to associate the the textual as"}, {"start": 1532.92, "end": 1539.5600000000002, "text": " sequence with the image we get a boost in efficiency a further boost and now we"}, {"start": 1539.56, "end": 1546.44, "text": " can truly leverage the computation okay so that was that was it that was it"}, {"start": 1546.44, "end": 1552.8, "text": " about the efficiency part and now I'll go into some more details about the"}, {"start": 1552.8, "end": 1559.72, "text": " method and then we'll we'll just jump to exploring the easier shot results they"}, {"start": 1559.72, "end": 1566.08, "text": " got so they say here that they we train clip from scratch without initializing"}, {"start": 1566.08, "end": 1570.96, "text": " the image encoder with image net weights or text encoder with pre-trained weights"}, {"start": 1570.96, "end": 1577.6, "text": " and why they do that is because work like a vision transformer already showed"}, {"start": 1577.6, "end": 1583.6399999999999, "text": " that once you have enough compute and enough data you really don't need to"}, {"start": 1583.6399999999999, "end": 1590.36, "text": " bias your model in any way so you just so what vision transformer showed is that"}, {"start": 1590.36, "end": 1596.32, "text": " this particular model is better than CNN's given enough data like they use"}, {"start": 1596.32, "end": 1602.52, "text": " JFT 300 data set and that's Google's proprietary data set and they show that"}, {"start": 1602.52, "end": 1607.4399999999998, "text": " it outperforms CNN's and it learns something similar to CNN's just a bit"}, {"start": 1607.4399999999998, "end": 1612.8799999999999, "text": " better obviously so that's why they don't encode it why they don't initialize it"}, {"start": 1612.8799999999999, "end": 1618.4399999999998, "text": " with pre-trained weights so I explained most of these so I'll just skip them"}, {"start": 1618.44, "end": 1623.48, "text": " yeah this is an interesting detail they are not using so they for a resnet they"}, {"start": 1623.48, "end": 1627.48, "text": " just swap this average pulling layer with the attention pulling layer that's"}, {"start": 1627.48, "end": 1634.28, "text": " implemented in a transformer style and yeah they're using as I said was funny"}, {"start": 1634.28, "end": 1639.92, "text": " for the text encoder and I also mentioned this one about EOS so the"}, {"start": 1639.92, "end": 1643.3600000000001, "text": " interesting part about how they scaled their approach is that they used"}, {"start": 1643.36, "end": 1649.9199999999998, "text": " efficient nets ideas so they used ideas from this paper to scale I don't know"}, {"start": 1649.9199999999998, "end": 1655.28, "text": " how to write today okay to scale their their technique and I just highlighted"}, {"start": 1655.28, "end": 1662.32, "text": " these as a just to show you how much engineering goes into all of their all"}, {"start": 1662.32, "end": 1668.08, "text": " of their research work and like you can see all of this mixed precision gradient"}, {"start": 1668.08, "end": 1673.8, "text": " checkpointing is all about making use as efficient as possible of the memory"}, {"start": 1673.8, "end": 1679.72, "text": " available in their on their hardware so half precision half precision and yeah"}, {"start": 1679.72, "end": 1684.1999999999998, "text": " it still takes a lot of time to train their biggest models so it took 18 days"}, {"start": 1684.1999999999998, "end": 1691.12, "text": " to train on almost 600 we hunted GPUs and 12 days on 256 for the vision"}, {"start": 1691.12, "end": 1694.6399999999999, "text": " transformer which is a vision transformer is more efficient than CNN's"}, {"start": 1694.64, "end": 1699.64, "text": " and that's one of the points that that paper made and that's why it takes a lot"}, {"start": 1699.64, "end": 1704.92, "text": " less compute and finally the the best model they had is the vision transformer"}, {"start": 1704.92, "end": 1713.3600000000001, "text": " L that was trained for one epoch on on a higher resolution of 336 pixels and"}, {"start": 1713.3600000000001, "end": 1718.92, "text": " that's that's a final clip model that they that they are using throughout this"}, {"start": 1718.92, "end": 1724.5200000000002, "text": " paper okay finally experiments so as I mentioned the target the goal of the"}, {"start": 1724.52, "end": 1729.44, "text": " paper is to show that clip has a really nice zero shot transfer performance and"}, {"start": 1729.44, "end": 1733.6, "text": " what they mean by zero shot transfer performance is a bit more general than"}, {"start": 1733.6, "end": 1739.2, "text": " what you usually mean by saying that in computer vision so usually and dates"}, {"start": 1739.2, "end": 1744.52, "text": " here they say here most of these tasks are are testing for distribution shift"}, {"start": 1744.52, "end": 1748.36, "text": " and domain generalization so what that means is the previous models in computer"}, {"start": 1748.36, "end": 1754.08, "text": " vision basically you want you want to test if it's if it's if it's having a"}, {"start": 1754.08, "end": 1759.1599999999999, "text": " really good performance even when you change the this underlying distribution"}, {"start": 1759.1599999999999, "end": 1763.9199999999998, "text": " so basically here you have a banana like a real-world banana like a sketch and"}, {"start": 1763.9199999999998, "end": 1768.32, "text": " those are called different domains and the other thing that we're testing for"}, {"start": 1768.32, "end": 1773.0, "text": " is that for example if the model was testing test that was trained on real"}, {"start": 1773.0, "end": 1777.1799999999998, "text": " world objects like banana I don't know dogs cats whatever and you want to see"}, {"start": 1777.1799999999998, "end": 1782.6, "text": " how how easy it is to just do some fine-tuning or do some linear probe on"}, {"start": 1782.6, "end": 1788.08, "text": " top of the feature space to maybe learn also about a new object like maybe horse"}, {"start": 1788.08, "end": 1794.4399999999998, "text": " and what they mean by zero shot transfer learning in this and this paper is not"}, {"start": 1794.4399999999998, "end": 1797.54, "text": " generalizing to those to those so distribution shifts and domain"}, {"start": 1797.54, "end": 1802.12, "text": " generalizations but to new tasks so for example that they say here well it is"}, {"start": 1802.12, "end": 1807.8, "text": " reasonable to say that the SVH and data set which is this one measures the task"}, {"start": 1807.8, "end": 1811.6799999999998, "text": " of street number transcription on the distribution of Google Street View"}, {"start": 1811.68, "end": 1816.8400000000001, "text": " photos is unclear what real task the cipher 10 data set measures and this is"}, {"start": 1816.8400000000001, "end": 1822.68, "text": " cipher 10 so basically if you train the model maybe image net and then you you"}, {"start": 1822.68, "end": 1827.2, "text": " want to test it on on on cipher 10 like this you're you're basically doing what"}, {"start": 1827.2, "end": 1832.2, "text": " I already said you're doing basically just a different different distribute"}, {"start": 1832.2, "end": 1837.3600000000001, "text": " just has a different distribution and maybe some different objects whereas if"}, {"start": 1837.36, "end": 1842.56, "text": " you're testing on SVH n you're basically learning how to do OCR in the wild if"}, {"start": 1842.56, "end": 1847.28, "text": " your model learns how to extract numbers from these images so that's it so we we"}, {"start": 1847.28, "end": 1850.8, "text": " want to generalize not only to distribution shifts and generalizations"}, {"start": 1850.8, "end": 1855.4199999999998, "text": " to new objects we also want to generalize to new tasks like OCR and not"}, {"start": 1855.4199999999998, "end": 1859.6399999999999, "text": " only two different classification datasets okay let's let's start"}, {"start": 1859.6399999999999, "end": 1864.6399999999999, "text": " exploring the paper first this table is not so important they just show so"}, {"start": 1864.64, "end": 1869.3200000000002, "text": " visual engrams are our method from 2016 which had a similar similar notion of"}, {"start": 1869.3200000000002, "end": 1872.5600000000002, "text": " zero-shot transfer and that's why they compare with them although it's not"}, {"start": 1872.5600000000002, "end": 1876.8000000000002, "text": " comparable because many things like transformers did not exist back then and"}, {"start": 1876.8000000000002, "end": 1882.1200000000001, "text": " so yeah they just have much better clip just as much better results on these"}, {"start": 1882.1200000000001, "end": 1887.72, "text": " datasets like image net so before seeing the zero shop and few shot performance"}, {"start": 1887.72, "end": 1892.0800000000002, "text": " of clip let's first go through prompt engineering and assembling details so"}, {"start": 1892.08, "end": 1895.6, "text": " the first thing they had to cope with is the fact that there is a polysemy"}, {"start": 1895.6, "end": 1898.96, "text": " problem in some of the downstream computer vision tasks and that means"}, {"start": 1898.96, "end": 1903.04, "text": " that for example in some of the datasets like image net you basically"}, {"start": 1903.04, "end": 1908.56, "text": " have two words which mean two different things so different semantics and maybe"}, {"start": 1908.56, "end": 1913.4399999999998, "text": " they give example here construction cranes and cranes the fly so the first"}, {"start": 1913.4399999999998, "end": 1917.48, "text": " thing they had to do is for a given data set and labels to disambiguate"}, {"start": 1917.48, "end": 1921.8799999999999, "text": " between those labels which have the polysemy problems the second thing and"}, {"start": 1921.88, "end": 1927.0800000000002, "text": " I already mentioned this one is they had to construct the the second the"}, {"start": 1927.0800000000002, "end": 1930.64, "text": " sequences like this so instead of just using labels so for example if you have"}, {"start": 1930.64, "end": 1937.0400000000002, "text": " a on a downstream class task you have labeled dog instead of using dog to in"}, {"start": 1937.0400000000002, "end": 1941.5600000000002, "text": " your zero shot classifier to configure it you use something like this a photo"}, {"start": 1941.5600000000002, "end": 1947.92, "text": " of a label and by doing that you get a 1.3 percent increase in accuracy and"}, {"start": 1947.92, "end": 1955.0800000000002, "text": " that's like a low low-hanging fruit the second thing they did is they they again"}, {"start": 1955.0800000000002, "end": 1959.24, "text": " they play with prompt programming so instead of just using a photo of a label"}, {"start": 1959.24, "end": 1963.96, "text": " they also experiment with adding maybe a type of pet if that made sense for the"}, {"start": 1963.96, "end": 1968.24, "text": " so we found several fine-grained image classification datasets that it helped"}, {"start": 1968.24, "end": 1974.2, "text": " to specify the category so for example in Oxford pets using that one would even"}, {"start": 1974.2, "end": 1979.04, "text": " help them even boost the performance even higher so on OCR just putting the"}, {"start": 1979.04, "end": 1985.04, "text": " quotes around the numbers would also gained gave them some additional boost"}, {"start": 1985.04, "end": 1989.44, "text": " they also played with and sampling as I mentioned so here is a concrete example"}, {"start": 1989.44, "end": 1994.32, "text": " so you by using a photo of a big for example dog and a photo of a small dog"}, {"start": 1994.32, "end": 1999.16, "text": " and combining those in the in the embedding feature space they get they get"}, {"start": 1999.16, "end": 2003.52, "text": " an additional boosting performance so they did not average in the probability"}, {"start": 2003.52, "end": 2010.6, "text": " space they just average before they do the the the soft max and everything so"}, {"start": 2010.6, "end": 2015.24, "text": " considering those two things together they get a boost of almost 5% and that's"}, {"start": 2015.24, "end": 2021.28, "text": " on image net so that's that's a really long low-hanging fruit and they made use"}, {"start": 2021.28, "end": 2028.76, "text": " of it so let's see that on the chart here basically for a given compute by"}, {"start": 2028.76, "end": 2034.0, "text": " just using those additional in sampling and pro programming programming methods"}, {"start": 2034.0, "end": 2040.56, "text": " they just get a huge improvements in the average score so that's that's that's"}, {"start": 2040.56, "end": 2045.2, "text": " the first thing I wanted to mention now let's jump to the zero shot performance"}, {"start": 2045.2, "end": 2050.96, "text": " of clip so what I did here is they took off-the-shelf models such as resonant"}, {"start": 2050.96, "end": 2057.6, "text": " hundred I think no resonance 50 here and they just fully pre fine-tuned it on"}, {"start": 2057.6, "end": 2062.92, "text": " every single data set you see here so it they pre trained it on STL 10"}, {"start": 2062.92, "end": 2068.52, "text": " Stanford cars etc and they took on the other side it took clip in a zero shot"}, {"start": 2068.52, "end": 2075.7999999999997, "text": " setting and we can see that clip is much better than fully supervised"}, {"start": 2075.7999999999997, "end": 2082.3199999999997, "text": " resident resident 50 on many of the data sets and now if we just analyze the"}, {"start": 2082.3199999999997, "end": 2086.4, "text": " extremes here this part and this part that will probably give us some more"}, {"start": 2086.4, "end": 2091.36, "text": " information what's happening here so first things first the data set zero"}, {"start": 2091.36, "end": 2096.48, "text": " shot clip improves by the most is STL 10 a data set designed to encourage"}, {"start": 2096.48, "end": 2100.96, "text": " efficient learning by providing only a limited number of labeled examples so"}, {"start": 2100.96, "end": 2105.96, "text": " what that means is that for this particular data set because resonant"}, {"start": 2105.96, "end": 2110.48, "text": " was you basically take the resident features that were that we got from"}, {"start": 2110.48, "end": 2116.2400000000002, "text": " image net and we freeze those and then we take the classifier and training"}, {"start": 2116.24, "end": 2121.8399999999997, "text": " that one with this many with this like this small amount of examples just"}, {"start": 2121.8399999999997, "end": 2127.72, "text": " didn't didn't give us a good results so you can see that clip was much better"}, {"start": 2127.72, "end": 2135.64, "text": " than resonant and on the other hand we have this other part of spectrum where"}, {"start": 2135.64, "end": 2140.9199999999996, "text": " if we analyze these data sets you can see that these data sets are complex and"}, {"start": 2140.92, "end": 2146.6, "text": " specialized so they say it here looking where it underperforms it works really"}, {"start": 2146.6, "end": 2151.4, "text": " weak on on these data sets like satellite image classification lymph node"}, {"start": 2151.4, "end": 2155.92, "text": " tumor counting objects blah blah blah and these results highlight the poor"}, {"start": 2155.92, "end": 2160.44, "text": " capability of zero shot clip or more complex complex tasks so that's where it"}, {"start": 2160.44, "end": 2165.04, "text": " underperforms resonant but it's still quite fascinating that a fully"}, {"start": 2165.04, "end": 2170.48, "text": " supervised model using labeled information is underperforming clip on"}, {"start": 2170.48, "end": 2176.12, "text": " on many of the data sets so it's worth mentioning of course resonant 50 is by"}, {"start": 2176.12, "end": 2182.28, "text": " no means state-of-the-art currently so let's now examine what happens when we"}, {"start": 2182.28, "end": 2186.2, "text": " compare clip with state-of-the-art but in a few shots few shot setting this time"}, {"start": 2186.2, "end": 2191.52, "text": " so these models are not trained like with all of the labels they just take a"}, {"start": 2191.52, "end": 2199.32, "text": " couple of labels so let's analyze the chart and they took three models first"}, {"start": 2199.32, "end": 2204.1600000000003, "text": " is resonant 50 sim clear the self supervised framework I mentioned and"}, {"start": 2204.1600000000003, "end": 2210.6000000000004, "text": " bit big transfer so this one was trained on JFT 300 so a lot of data and we can"}, {"start": 2210.6000000000004, "end": 2217.0800000000004, "text": " see that with 16 so we just so what we do here is we take those models like bit"}, {"start": 2217.0800000000004, "end": 2223.38, "text": " and we freeze the weights we add the classifier and we we pre-trip we we"}, {"start": 2223.38, "end": 2232.12, "text": " fine-tune on these different data sets and we see that for 16 examples we are"}, {"start": 2232.12, "end": 2239.6400000000003, "text": " approaching the zero shot clip which is a nice nice nice result because these are"}, {"start": 2239.6400000000003, "end": 2245.2400000000002, "text": " the best methods available currently and what's interesting here is that there is"}, {"start": 2245.24, "end": 2254.08, "text": " this drop between zero shot and one shot and basically that discrepancy"}, {"start": 2254.08, "end": 2259.6, "text": " shouldn't be there in in ideal world because humans for example when you when"}, {"start": 2259.6, "end": 2265.68, "text": " you give us one example we won't be underperforming like yeah we'll we'll"}, {"start": 2265.68, "end": 2270.8399999999997, "text": " we'll be better so that's something they're trying to to to to work on and"}, {"start": 2270.84, "end": 2276.88, "text": " the potential solution they propose is just to initialize so once you take the"}, {"start": 2276.88, "end": 2284.4, "text": " vision encoder of clip and instead of training a classifier from random they"}, {"start": 2284.4, "end": 2289.2400000000002, "text": " just initialize it with the zero classifier zero shot classifiers weights"}, {"start": 2289.2400000000002, "end": 2294.36, "text": " for that particular data set and use that but that still didn't help them so"}, {"start": 2294.36, "end": 2301.88, "text": " they just left it out for for the future for the future exploration let's continue"}, {"start": 2301.88, "end": 2311.56, "text": " and see how the how good the zero shot classifier actually is and here we see"}, {"start": 2311.56, "end": 2317.88, "text": " what we see in this chart is for example how many labeled examples do we need to"}, {"start": 2317.88, "end": 2326.2000000000003, "text": " achieve the same accuracy as zero shot classifier so for fair for fair 2013 if"}, {"start": 2326.2000000000003, "end": 2333.4, "text": " we just take that division encoder of clip and use a random initial classifier"}, {"start": 2333.4, "end": 2338.96, "text": " we need hundred eighty four examples to achieve the same performance as zero"}, {"start": 2338.96, "end": 2344.92, "text": " shot clip so that's that's a good result we we want to make zero shot classifier"}, {"start": 2344.92, "end": 2350.52, "text": " as good as possible so that even with a lot of label data we can improve upon"}, {"start": 2350.52, "end": 2355.32, "text": " it so that would be an ideal result if every single data set required a lot of"}, {"start": 2355.32, "end": 2360.96, "text": " labeled examples to improve upon that randomly initialized classifier that"}, {"start": 2360.96, "end": 2365.56, "text": " would be a good result here on the other chart we can see the difference between"}, {"start": 2365.56, "end": 2373.06, "text": " a linear probe clip performance versus a zero shot performance so this time you"}, {"start": 2373.06, "end": 2380.88, "text": " take that classifier and you basically train it with all the available data so"}, {"start": 2380.88, "end": 2385.96, "text": " it's fully supervised and we use all of the data available in a particular data"}, {"start": 2385.96, "end": 2392.48, "text": " set and we see that for a given data set like maybe this one the the performance"}, {"start": 2392.48, "end": 2398.0, "text": " is maybe 45% once we use all once we leverage all of the data but by using"}, {"start": 2398.0, "end": 2403.92, "text": " zero shot we have only 35 so obviously zero shot a classifier is suboptimal"}, {"start": 2403.92, "end": 2408.8, "text": " for this data set and we can see the general trend so in a perfect world zero"}, {"start": 2408.8, "end": 2414.48, "text": " shot would be as good as this classifier that used the whole corpus of label data"}, {"start": 2414.48, "end": 2420.44, "text": " and but it's you can see the trend it's suboptimal and it's down by maybe 20 or"}, {"start": 2420.44, "end": 2427.3, "text": " 15 points and only a couple of data says like STL 10 and if you remember that one"}, {"start": 2427.3, "end": 2432.0800000000004, "text": " has a little like a small amount of a label data available so here the the"}, {"start": 2432.0800000000004, "end": 2440.7200000000003, "text": " zero shot performance is similar to this fully fully trained classifier so that"}, {"start": 2440.7200000000003, "end": 2446.32, "text": " shows us that there is a lot more like space to to improve upon this zero shot"}, {"start": 2446.32, "end": 2452.6400000000003, "text": " classifier let's continue analyze other charts so this one what it tells us is"}, {"start": 2452.64, "end": 2459.6, "text": " that basically increasing the compute will decrease the error which is"}, {"start": 2459.6, "end": 2463.44, "text": " something we could expect but the interesting thing is it's really noisy"}, {"start": 2463.44, "end": 2467.3599999999997, "text": " so if we take a particular data set we can see that with compute sometimes the"}, {"start": 2467.3599999999997, "end": 2473.2799999999997, "text": " error increase increases and then it decreases so it's really noisy but the"}, {"start": 2473.2799999999997, "end": 2480.64, "text": " overall trend is still strong in favor of more computation is better here what"}, {"start": 2480.64, "end": 2487.24, "text": " we see is that basically the future space of clip is high-quality because"}, {"start": 2487.24, "end": 2491.68, "text": " once we do once we compare it with different models like you can see here"}, {"start": 2491.68, "end": 2498.72, "text": " efficient nets and be all and moco sim cleared like self supervised methods"}, {"start": 2498.72, "end": 2503.52, "text": " and bid which is fully supervised we can see that the feature space is higher"}, {"start": 2503.52, "end": 2508.8799999999997, "text": " quality for clip and so what we see on this diagram is the amount of compute on"}, {"start": 2508.88, "end": 2514.52, "text": " x-axis and the average score across a suit of different like 27 data sets and"}, {"start": 2514.52, "end": 2522.8, "text": " every single model was had a had a linear probe fully fully trained on the"}, {"start": 2522.8, "end": 2529.08, "text": " frozen feature space and a clip again shows much better much better average"}, {"start": 2529.08, "end": 2534.8, "text": " score which is something that we ideally want we want the clip to to be the"}, {"start": 2534.8, "end": 2540.0800000000004, "text": " features we train for clip are are really good for many downstream data sets"}, {"start": 2540.0800000000004, "end": 2545.48, "text": " and it showed that here if we take one specific model like maybe efficient net"}, {"start": 2545.48, "end": 2551.04, "text": " noisy student and we analyze we break down those instead of using the average"}, {"start": 2551.04, "end": 2555.52, "text": " score we just see the performance across every single data set we can see where"}, {"start": 2555.52, "end": 2560.2400000000002, "text": " the the results for clip are much better and where it's a bit worse than"}, {"start": 2560.2400000000002, "end": 2564.32, "text": " efficient net and because efficient net was basically pre trained on image net"}, {"start": 2564.32, "end": 2569.0, "text": " so it does make sense that it has better performance than clip that the feature"}, {"start": 2569.0, "end": 2572.96, "text": " space he developed is better for image net but only for image and I was as we"}, {"start": 2572.96, "end": 2576.48, "text": " can see on most of the other data sets the future space developed by clip is"}, {"start": 2576.48, "end": 2582.6000000000004, "text": " way more high quality than than efficient nets feature space okay let's"}, {"start": 2582.6000000000004, "end": 2589.44, "text": " continue and see some other results I'll skip this one not that interesting and"}, {"start": 2589.44, "end": 2596.28, "text": " I'll jump to this one basically if we take what we see here is the following"}, {"start": 2596.28, "end": 2604.4, "text": " we want to see we want to see how how the clip accuracy behaves when we change"}, {"start": 2604.4, "end": 2610.2000000000003, "text": " the underlying distribution so we see here basically same class same object"}, {"start": 2610.2000000000003, "end": 2613.56, "text": " same classes but like different distribution so here we have sketches"}, {"start": 2613.56, "end": 2621.36, "text": " here we have real-world objects some like again some drawings etc and if we"}, {"start": 2621.36, "end": 2628.08, "text": " take for example a model that had 80% accuracy on image net top 1 accuracy we"}, {"start": 2628.08, "end": 2636.88, "text": " see that for all of those standard image net models that they severely have like"}, {"start": 2636.88, "end": 2643.7200000000003, "text": " have much lower accuracy on those other related data sets were just with"}, {"start": 2643.7200000000003, "end": 2649.1600000000003, "text": " different distribution whereas on the other hand if we take the results for"}, {"start": 2649.1600000000003, "end": 2656.08, "text": " for clip it has much higher much higher accuracy than then then then image at"}, {"start": 2656.08, "end": 2664.28, "text": " once and ideally again if we have 80% on image net we want to have 80% on all of"}, {"start": 2664.28, "end": 2669.6800000000003, "text": " those similar data sets just with different distributions if we take a"}, {"start": 2669.6800000000003, "end": 2676.28, "text": " look at the tabular data here what we see is when we take resonant 101 and so"}, {"start": 2676.28, "end": 2681.2000000000003, "text": " it was just pre trained on image and we don't fine-tune it on any of these new"}, {"start": 2681.2000000000003, "end": 2685.96, "text": " datasets with different distributions we just use it as it is and we see that the"}, {"start": 2685.96, "end": 2690.7200000000003, "text": " performance that the accuracy just severely drops and for these adversarial"}, {"start": 2690.72, "end": 2694.7599999999998, "text": " examples like you can see here some chopped banana something that the model"}, {"start": 2694.7599999999998, "end": 2700.3999999999996, "text": " never saw during training it has really really bad accuracy on the other hand"}, {"start": 2700.3999999999996, "end": 2707.7599999999998, "text": " clips got much much higher accuracy and it improves like even 75% in this case"}, {"start": 2707.7599999999998, "end": 2713.64, "text": " on this data set so that shows that clip is much more robust to"}, {"start": 2713.64, "end": 2720.92, "text": " distributional shift than these other other methods let's see some other results"}, {"start": 2720.92, "end": 2726.24, "text": " they showed and this one is interesting so this line is the same as the one above"}, {"start": 2726.24, "end": 2732.0, "text": " so we just use clip in a zero shot setting and if we actually fine-tune"}, {"start": 2732.0, "end": 2737.6, "text": " clip for image net we can see that even though it improves the image net"}, {"start": 2737.6, "end": 2743.72, "text": " accuracy the the actual robustness to to distributional shift so again we have"}, {"start": 2743.72, "end": 2749.16, "text": " those data sets like those seven data sets we see that the curves actually go"}, {"start": 2749.16, "end": 2756.44, "text": " go down and it's it's less robust to to all of those data sets from another"}, {"start": 2756.44, "end": 2761.56, "text": " perspective if we look at this chart it's it's had some improvements on image"}, {"start": 2761.56, "end": 2769.32, "text": " net whereas on many other data sets it just has less less good performance so"}, {"start": 2769.32, "end": 2774.6, "text": " was this adapt to class shift so I think I noted somewhere here basically"}, {"start": 2774.6, "end": 2778.84, "text": " because some of these data sets like YouTube BB and image net vid have"}, {"start": 2778.84, "end": 2783.6, "text": " different classes than image net which has thousand classes so it has so for"}, {"start": 2783.6, "end": 2789.2, "text": " the person class in YouTube BB we have what we have to do for those image net"}, {"start": 2789.2, "end": 2796.4399999999996, "text": " models is at collect the scores over baseball player bridegroom and scuba"}, {"start": 2796.4399999999996, "end": 2801.3999999999996, "text": " diver and that's how we give the prediction for person on YouTube BB"}, {"start": 2801.3999999999996, "end": 2808.2799999999997, "text": " which is suboptimal but since clip is configurable we can just use this new"}, {"start": 2808.2799999999997, "end": 2814.12, "text": " class and create the construction a type of a person or whatever to to have much"}, {"start": 2814.12, "end": 2821.2, "text": " better results than than these fixed image net models and when we do that we"}, {"start": 2821.2, "end": 2825.6, "text": " we have we can see that the trend improves and the robustness increases"}, {"start": 2825.6, "end": 2830.08, "text": " further which is really nice and particularly on those couple of data"}, {"start": 2830.08, "end": 2835.72, "text": " sets which has which have that superset of classes this small trick helps"}, {"start": 2835.72, "end": 2840.3599999999997, "text": " whereas the other methods as I already explained have to do the following so"}, {"start": 2840.36, "end": 2847.6, "text": " for example this this data set and has maybe class person so when we're testing"}, {"start": 2847.6, "end": 2853.52, "text": " when we're testing efficient net we have to pull over three classes out of"}, {"start": 2853.52, "end": 2857.96, "text": " thousand that he has to give an estimate of what person would be since he doesn't"}, {"start": 2857.96, "end": 2862.1600000000003, "text": " know of the concept of a person and on the other hand we can just use the"}, {"start": 2862.1600000000003, "end": 2869.2400000000002, "text": " flexibility of a zero shot classifier and get the direct results instead I'll"}, {"start": 2869.24, "end": 2874.6, "text": " just I'll just skip some of these because this is similar experiments to"}, {"start": 2874.6, "end": 2881.0, "text": " these ones is just that instead of doing showing us the fully supervised curve so"}, {"start": 2881.0, "end": 2887.0, "text": " this model is clip this that's fine-tuned to image net fully fine-tuned"}, {"start": 2887.0, "end": 2892.3199999999997, "text": " and instead of that they just show a trend once we start from few shots so"}, {"start": 2892.3199999999997, "end": 2896.3599999999997, "text": " two shots four shot as we start increasing the number of labels from"}, {"start": 2896.36, "end": 2901.6400000000003, "text": " image and we can see that we are slowly increasing the the image net top one"}, {"start": 2901.6400000000003, "end": 2909.2000000000003, "text": " accuracy but the we're still underneath zero shot clip on this on these seven"}, {"start": 2909.2000000000003, "end": 2913.84, "text": " natural distribution data sets okay there's there's a lot of information"}, {"start": 2913.84, "end": 2921.1600000000003, "text": " here the main idea is basically that clip clips feature space is way better"}, {"start": 2921.16, "end": 2927.56, "text": " suited to to just extend to many other downstream tasks and it's much more"}, {"start": 2927.56, "end": 2933.3999999999996, "text": " robust as we saw two different natural shifts so in a way it's it's a better"}, {"start": 2933.3999999999996, "end": 2937.96, "text": " way to and additionally it's not using those costly supervised data sets is"}, {"start": 2937.96, "end": 2944.64, "text": " just using those pairings and so it's easier to to to just get a get a get a"}, {"start": 2944.64, "end": 2949.8399999999997, "text": " grab of those of that data that's the main idea of the paper this shows some"}, {"start": 2949.84, "end": 2952.88, "text": " contamination analysis and finally I just want to show that there are some"}, {"start": 2952.88, "end": 2959.6800000000003, "text": " examples where it's not working really good and a nice nice example of a"}, {"start": 2959.6800000000003, "end": 2966.36, "text": " failure is on amnest where it achieves only 88 percent accuracy on amnest just"}, {"start": 2966.36, "end": 2971.36, "text": " for comparison if you take a simplest baseline so you just take amnest image"}, {"start": 2971.36, "end": 2976.96, "text": " which is 28 by 28 you just flatten out the pixels and you use a simple feed"}, {"start": 2976.96, "end": 2982.18, "text": " for net this one will achieve higher higher accuracy than clip which was"}, {"start": 2982.18, "end": 2988.0, "text": " trained on a lot of data and lots of compute so it really failed on amnest so"}, {"start": 2988.0, "end": 2993.7200000000003, "text": " as they say here this suggests clip does little to address the underlying problem"}, {"start": 2993.7200000000003, "end": 2997.64, "text": " of brittle generalization of deep learning models instead clips tries to"}, {"start": 2997.64, "end": 3001.68, "text": " circumvent the problem and hopes that by training in such a large and varied"}, {"start": 3001.68, "end": 3006.4, "text": " data set at all data will effectively be in distribution and this is a naive"}, {"start": 3006.4, "end": 3012.6, "text": " assumption that as amnest demonstrates is easy easy to violate so obviously"}, {"start": 3012.6, "end": 3019.2400000000002, "text": " this is not the the final the final way to go to towards towards really"}, {"start": 3019.2400000000002, "end": 3024.12, "text": " generalized model to really general models but it's a it's a it's a step"}, {"start": 3024.12, "end": 3030.28, "text": " forward and I'm excited about this work so yeah that was that was pretty much it"}, {"start": 3030.28, "end": 3033.96, "text": " a short recap because there was a lot of details and hopefully this will make it"}, {"start": 3033.96, "end": 3039.2400000000002, "text": " kind of consolidate in your head and so they care about zero-shot performance"}, {"start": 3039.2400000000002, "end": 3044.28, "text": " generalization to new tasks and not only a new distributions and a domain"}, {"start": 3044.28, "end": 3049.04, "text": " generalization they show that prompt programming and assembling does help"}, {"start": 3049.04, "end": 3053.92, "text": " boost the performance of clip models so that's important they show that even"}, {"start": 3053.92, "end": 3059.16, "text": " against a fully supervised baseline such as the resonance 50 they are comparable"}, {"start": 3059.16, "end": 3064.2799999999997, "text": " and those are exciting results they show that even against the best baselines in"}, {"start": 3064.2799999999997, "end": 3068.7599999999998, "text": " a few shot setting they are comparable in a zero shot setting and they are"}, {"start": 3068.7599999999998, "end": 3073.6, "text": " better when they're doing the same thing when they're also fine-tuning their"}, {"start": 3073.6, "end": 3081.52, "text": " model I linear probing actually here next they show that zero shot classifier"}, {"start": 3081.52, "end": 3086.96, "text": " can be further improved it's suboptimal to the classifiers which can which are"}, {"start": 3086.96, "end": 3092.28, "text": " imaginable by just fine-tuning them on specific data sets so there is a lot of"}, {"start": 3092.28, "end": 3097.92, "text": " there is more work that can be done on improving that again compute doesn't"}, {"start": 3097.92, "end": 3103.96, "text": " does improve a clip features are better so feature space that clip provides is"}, {"start": 3103.96, "end": 3108.88, "text": " better than other baselines in compared to a specific baseline efficient net"}, {"start": 3108.88, "end": 3114.16, "text": " here we can see that it's much better on many data sets so again feature space is"}, {"start": 3114.16, "end": 3122.68, "text": " richer than than the other baselines here it shows a impressive"}, {"start": 3122.68, "end": 3129.16, "text": " generalization to a robustness to distribution shift and same thing here"}, {"start": 3129.16, "end": 3134.7999999999997, "text": " and they show that if you want if you if you kind of add a classifier and linear"}, {"start": 3134.7999999999997, "end": 3140.08, "text": " probe the clip onto image net you basically do improve image and performance"}, {"start": 3140.08, "end": 3144.56, "text": " but you lose the robustness to other data sets so you don't want to overfit"}, {"start": 3144.56, "end": 3151.3199999999997, "text": " to imagine basically and finally they showed some limitations among them that"}, {"start": 3151.3199999999997, "end": 3157.6, "text": " it's failing on on on simple data sets such as MNIST because it never saw those"}, {"start": 3157.6, "end": 3162.7599999999998, "text": " images during training so it just fails miserably even compared to the simplest"}, {"start": 3162.7599999999998, "end": 3166.4, "text": " of baselines okay those are there was a much longer video than I expected but"}, {"start": 3166.4, "end": 3170.2400000000002, "text": " hopefully you learned something from this video if you did consider"}, {"start": 3170.2400000000002, "end": 3174.64, "text": " subscribing and sharing this video and also hit the bell icon to get notified"}, {"start": 3174.64, "end": 3197.4, "text": " when I upload a new video until next time keep learning deep"}]