Multimodal Few-Shot Learning with Frozen Language Models | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I cover "Multimodal Few-Shot Learning with Frozen Language Models" from DeepMind. They introduce Frozen - which is able to handle both visual and textual inputs and shows good generalization capabilities to novel visual question answering datasets combined with fast binding mechanisms even though it was only trained on image captioning.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2106.13884
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable:

00:00 Intro
02:20 GPT-3 and emerging few-shot properties
04:20 Training procedure for Frozen
07:45 Inference
10:15 Strong generalization?
11:55 Prompting mechanisms and the hardest task
13:25 Quantitative results
19:50 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️


Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#frozen #deepmind #multimodal
 What's up? In this video I'm covering this novel paper called Multimodal Few Shot Learning with Frozen Language Models by Maria Zimpucalli, Jacob Manik, Serkan Kabi, Ali Aslami, Oriol Vinyals and Felix Hill of DeepMind. So what this paper shows in a nutshell is that they have this model called Frozen where basically they take a huge pre-trained language model such as GPT-2, they freeze it, hence frozen, and they basically train this vision encoder so that they can parse images as well. So it will just convert images into like tokens which are compatible with the language model and then they show that they can have this model do few-shot learning across different tasks which involve both the visual cues as well as the linguistic cues. To make it a bit less abstract, let me show you this example to show you what I mean. So this is an input, an example of an input that goes into this frozen model. So we have an image, it will be tokenized, then we have this person is like, and then we have this smiley because this girl is happy, this person is like sad, and finally we prompt the model with this image and we prompt it with this person is like, and as we can see the model generates this like terrified smiley, a dot, and end of a sentence token. Okay, which is what we've expected, which is really cool. And okay, thing to keep in mind during this whole video is that these examples are curated, they mentioned that multiple times, so again the whole point is that they can have this few-shot learning capability rather than they have like a highly performant model on all of these tasks. Okay, nonetheless let me show you this one. So this was invented by Zacharias Janssen, Thomas Edison, blah blah blah. So the point here is that just looking at this image you can't deduce what's the answer, whereas here you could have. And so as you can see the output here is the right brothers dot end of sentence token. So the whole point here as we'll soon see is that the language model can now pull the factual knowledge in and we can answer questions like this because of the language model, not because of the vision portion of the model. Okay, let's dig deeper into the paper now. So that was a high level like a glimpse of this paper. Basically they can handle both modalities and they can like learn the more examples you give the model the better it becomes. That's really cool. Okay, let's now dive deeper. When trained at a sufficient scale autoregressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples. And this should be like familiar to you. If you haven't watched my video on GPT-3 or the original transformer paper, do check them out. I'll link them somewhere here. But basically what those models showed is the following, especially the GPT family of models from OpenAI. They showed that even though the model was trained as a language model, so basically you have an unsupervised task of next token prediction and that's everything. And they show that you can actually do machine translation for example. And here is an example of how it looks like. So we have a one-shot setting here where we prompt the model with translate English to French and then we give it one example, C author to Lutre Des Morts, I don't speak French. And then you prompt with cheese and you have this symbol and the model will actually learn how to translate this. So to perform machine translation even though it's never seen such a task during this training. And here we just have additionally a few-shot example set up where we have multiple examples and then we prompt the model to translate. And they show that with these multiple examples the performance just gets better and better. Obviously it saturates after a certain point, but like the trend is clear. Okay, so that's the first thing. Now here we present a simple yet effective approach for transferring this few-shot learning ability to a multi-modal setting. And in particular they focus on vision as we'll soon see. Okay, finally here's a motivation behind all this. So despite these impressive capabilities, such large-scale language models are blind to modalities other than text, preventing us from communicating visual tasks, questions or concepts to them. So that's the reason they kind of integrated this vision component. Okay, let me now explain to you how this whole system looks like and how it works. So the system as a whole is pretty simple. So we have a language model as you can see here and it's frozen. So the parameters are all frozen and we have this vision encoder. So just to be a bit more specific they were using GPT-2 for the language model. They were using NF ResNet for the vision encoder, but that's not that crucial. You can pick some other language model, you can pick some different vision encoder, maybe something like vision image transformer, but that's not that important. Like the fact that these components exist and they are wired the way they are. Okay, so what I do is, so they first need to kind of adapt the image into the input that the transformer model is expecting and that's these tokens. So what I do is they have this vision encoder and it will output after some pulling layer a vector. And what they do is they take a linear layer, they just project this into this novel space. Let me draw it like this and the dimension will be n times d, where basically they found this n to be like the best value was two, but that's not important. As you can see here they have only two tokens, but it can be an arbitrary number in general. So d is, that's important. D is a dimension that's the same as these tokens that go into language model. So obviously that's the prerequisite so that we can feed these tokens into the language model that come from the image. And so that's the image part. Now how do they train this is fairly simple. So basically I think it would be wiser to write this down as, so we have start of sentence token here and all of these other words will be kind of translated here. So a will get here, this small will be here, etc. So we'll have small here and so now what you're trying to do is to predict the target sequence. So as you can see here, so let me just focus on predicting the word small. So the word small will have as a context these image tokens that came here as well as this start of sentence token as well as a because we have causal masking obviously you don't want to have if you saw small in the input then you're kind of cheating and it's easy to predict the output small. You can just kind of copy the values and the transformer will learn how just to copy copy paste the values and that's not what we want. We want to predict tokens. So that's how the test would look like. Basically here you'd have you'd output the distribution the usual way so you're just trying to maximize the likelihood. So here maybe we have some distribution and we find the token that corresponds to the word small maybe this one and we'll just want to maximize this to one and push all of the other probabilities down to zero and we do that by just simple like cross entropy so it will be a minus log of p so when p goes to one the loss goes to zero and so in a nutshell that's how the system works that's how it's trained the gradients are back propped through these weights which are frozen and these weights are then tweaked so that basically what happens is that this vision encoder learns such representations so that they're useful in order to do this captioning task okay and that's the whole system it's fairly trivial. Let me now show you how they use this thing and here we have in the first example so we have a vision encoder so we have an image we encode it into these two tokens and then we prompt it with like this text so question what color is the car and the model generates blue and then end of sentence token so by the way just a short remark here they actually even though you can see here a word and a single token what actually in practice what I do is they use this tokenizer called sentence piece so this boat or this word small maybe like separated into sub words maybe like maybe this sma part will be like one we'll have one token associated with it and then ll will have second so that's just an example but in general you'll have more tokens than you have words in your sentence okay just a minor detail and so that's this is one of the tasks that they're gonna evaluate this model on the second example here will require the model to have some knowledge base and additionally I forgot to mention that the captioning so during this captioning training all of the like named entities are masked so if you have a name like alexa or something you'll mask it with a person so you'll have you'll just like put a person instead of a name there so that means that this vision encoder cannot learn those named entities obviously because it never saw like named entities and so the examples you see here like like this model generated steve jobs that did not certainly didn't come from the vision encoder and that's an important thing I want you to notice here so the knowledge of that steve jobs was the guy who invented iphone actually came from the language model itself and that's the important from the language model itself so that's important and the third task they evaluate this model on is this fast binding task where you present the model with the image then you say you kind of have this made up word called dex and you just want the model to associate this visual category of an apple with the novel word dex and you do the same thing here you have an orange blanket and finally you prompt it with an image and you say question what is this answer and the model generates this is a dex so we saw dex is apple so it properly generated this is a dex sentence okay so having said that let me now kind of see let me show you the quantitative results they got because that's that's that's interesting before that just a short remark here so they say here in contrast our work enables strong generalization to new multi-modal tasks blah blah blah what i want to say here is i don't like this part because basically we don't even have a strong definition of what strong generalization is and the closest thing i could think of was fransois chollet's paper on the measure of intelligence where he described this terminology where he basically says that all of the current models pretty much that we have in deep learning field and machine learning are only like able to do this local generalization whereas this broad generalization and extreme generalization is something that only humans can currently basically do and so yeah i pretty much agree with him on this on this one because this language model that they use in frozen has seen like a lot of data so that means it has a lot of experience which we need to count in when we calculate these generalization capabilities so arguably because of all of that like immense amount of experience you cannot actually claim that it's generalizing like strongly it does generalize but like just the level yeah it's kind of yeah rent over pretty much okay these are just some details i already explained they have a huge seven billion pre-trained language model they freeze it and that's how they train the model i also explained this one how how do we map the images into into tokens which language model can then parse okay i'll just skip all of those small detail there is in positionally coding that helped them and let me focus on this so important thing i want you to notice is that they have a bunch of different ways to prompt this model so that it can generate the answers much better and here's one example so they have a pretty intricate uh like terminology here so two ways zero repeats two inner shots so let's see what i mean so first they have this task induction which they quantitatively show it helps a lot so answer with dex or blicket kind of prompts kind of tunes the model into answering with these two words in a sense and then they have so two shots basically this is the first part and uh the reason it's two ways because we have two made up words like blanket and ducks here and because the reason it's two inner shots is because they have two independent examples here as you can see and finally they prompt the model and blicket would be the correct answer because as you can see blicket is lion okay and the the last task they'll they'll they'll evaluate is this one where again you have to associate a visual category with a novel word and finally you actually not only have to output like here it's basically recognition you recognize it's a lion and you output blicket but here you have to reason because it says what is the the question says what is the dex made of so you first need to understand that dex is a table and then you need to understand what it is made of and the answer is wood so this is the final task they evaluate this frozen model on and now let's see the quantitative results okay here are the results for the visual question answering and there are these baselines which i'll shortly explain now especially this blind baseline so the strength of the pre-trained language model is a double edge sword it powers the generalization abilities of frozen but also enables the model to perform surprisingly well without considering the visual input at all so you can learn to ignore the visual input and still answer the question so to guard against this possibility we also train blind baselines in which the image presented to the visual encoder is blacked out but the common weights are still trained this amounts to prefix tuning so just as a short reminder how it works basically here instead of an image for these baselines what they'll do instead is they'll just black out or blue out in my case these images and so what happens is basically the model will have to learn some representation which is constant which won't which won't change because it will only be presented with black images and so that constant kind of needs to help in this captioning task and so that's why they call it prefix tuning because it finds some representation that helps the overall system do better job at captioning okay so that's one of the baselines the blind baseline and having explained that one let's now focus on the quantitative results okay here we are um so frozen here is the the the version we were talking about here is the version trained from scratch so that means you just stitch all of the weights in the pre-trained transformer and try and train it from scratch um here the the setups they have so this is zero shot one shot four shot etc and we can see that as we add more examples we have this trend that the the accuracy is improving which is desirable okay we see that the from scratch model totally fails we see that the fine-tune model is worse so fine-tuning so basically here they do not they do not freeze the language model but they initialize the model with a pre-trained weight the blind baseline i've just explained it as well so um again oscar is some dedicated baseline you can see it's much better than frozen but the whole point here is that we have this improve improving ability uh that also was pertinent to gpt3 models and language models in general so that's cool um here you can just see when you additionally fine-tune on this very data set visual visual question answering data set the performance obviously ramps up and that's that's expected i guess now the second thing here is the test on this okay vqa data set so this data set contains those examples where you need some additional knowledge base in order to answer the questions and again they so they show that um the the model improves with additional examples as a new baseline they use this 400 million parameter model and by the way this one has seven billion and again obviously the the bigger the model the the better the performance is something that's not that surprising i guess uh like in 2021 and um yeah again the this baseline is much better they just want to stress out that uh we have this improvement uh like being transferred to this multi-multi-multi-modal uh setup and that's cool okay so that was the the first experiments they did um here let me just uh kind of reiterate this because it's really important uh so this conceptual captions data set is hypernamed meaning that for example proper names are replaced with a general person a word-like person okay so this enables us to rigorously study the transfer of factual knowledge because all knowledge of named entities comes from language model pre-training consequently when we show the model an image of an airplane and ask who invented this the visual encoder has determined that the image contains an airplane and the language model has used this to retrieve the factual knowledge that airplanes were invented by the Wright brothers and finally jumping to fast concept binding these are the last tasks they tested this frozen model on um let me just kind of connect this table with the actual tasks so this is the task at hand we have this uh visual binding uh task so this is a two-way binding because we have again two novel made up words text and blick it they'll show like five way binding task where obviously you have now five made upwards and we'll soon see that it fails on that one but it succeeds on the two-way binding task so on the two-way binding you can see again they have a bunch of different ways to to prompt the model so that they can elicit better outputs and again here you can see it's improving with more examples second thing they tried is they used real names instead of those made upwards they do this so that they can quantify how harder it is for the model to learn the binding uh and how hard the task itself is and you can see so this kind of bugs me i'm not sure whether this is a typo but like you can it has better performance and then you kind of saturate saturates after three examples um again this annual baseline is better as i mentioned uh five way um binding fails so here it's literally like uh same as random chance 20 and it kind of improves here then goes back so yeah it's inconclusive here they mentioned here somewhere here just a sec so in table four we show that the observed effects on open-ended mini image net do not transfer to the five-way setting where frozen is not significantly above chance this shows that learning to bind five new names to five visual categories in a single forward pass is beyond the current capabilities of frozen okay so it kind of fails there and they leave it up as a future research the final task is the one i showed you where you need to aside from fast binding you need to reason not much new conclusions can be made here so again it's improving with more examples the interesting part maybe is so if we focus on this blind baseline we can see that even repeating uh so again remember the image is blacked out so we just just repeat the text a couple of times and those linguistic cues help boost the performance of this blind model which means that these improvements above basically are a combination of both linguistic cues as well as the visual cues okay um i think that's pretty much it i like this paper a lot i like this uh inclusion of visual uh information into this whole pipeline it slowly starts resembling the way we humans operate so we have these like we have vision obviously and we kind of uh somehow represent that information and then we have this like computation engine inside our head by the way one of the previous papers called uh like pre-trained transformers or universal computation engine or something they showed that like if you take a huge pre-trained language model and you just tweak some layer norm parameters and some embedding weights you can basically fine tune it very fast onto novel tasks and that's cool i guess this work is a follow-up in a way and yeah having said that i hope you like this video if you did consider subscribing sharing and until next time bye bye
[{"start": 0.0, "end": 2.88, "text": " What's up? In this video I'm covering this novel paper called"}, {"start": 2.88, "end": 6.32, "text": " Multimodal Few Shot Learning with Frozen Language Models"}, {"start": 6.32, "end": 13.92, "text": " by Maria Zimpucalli, Jacob Manik, Serkan Kabi, Ali Aslami, Oriol Vinyals and Felix Hill of DeepMind."}, {"start": 13.92, "end": 18.32, "text": " So what this paper shows in a nutshell is that they have this model called Frozen"}, {"start": 18.32, "end": 22.88, "text": " where basically they take a huge pre-trained language model such as GPT-2,"}, {"start": 22.88, "end": 27.36, "text": " they freeze it, hence frozen, and they basically train this vision encoder"}, {"start": 27.36, "end": 31.36, "text": " so that they can parse images as well. So it will just convert images into"}, {"start": 31.36, "end": 36.4, "text": " like tokens which are compatible with the language model and then they show that they can have"}, {"start": 36.96, "end": 42.72, "text": " this model do few-shot learning across different tasks which involve both the"}, {"start": 43.6, "end": 47.120000000000005, "text": " visual cues as well as the linguistic cues. To make it a bit less abstract,"}, {"start": 47.120000000000005, "end": 51.760000000000005, "text": " let me show you this example to show you what I mean. So this is an input,"}, {"start": 51.760000000000005, "end": 55.6, "text": " an example of an input that goes into this frozen model. So we have an image,"}, {"start": 55.6, "end": 60.72, "text": " it will be tokenized, then we have this person is like, and then we have this smiley because"}, {"start": 60.72, "end": 67.12, "text": " this girl is happy, this person is like sad, and finally we prompt the model with this image"}, {"start": 67.12, "end": 72.16, "text": " and we prompt it with this person is like, and as we can see the model generates this"}, {"start": 72.16, "end": 77.84, "text": " like terrified smiley, a dot, and end of a sentence token. Okay, which is what we've expected,"}, {"start": 77.84, "end": 84.24000000000001, "text": " which is really cool. And okay, thing to keep in mind during this whole video is that these"}, {"start": 84.24, "end": 88.96, "text": " examples are curated, they mentioned that multiple times, so again the whole point is that they can"}, {"start": 89.52, "end": 94.88, "text": " have this few-shot learning capability rather than they have like a highly performant model"}, {"start": 94.88, "end": 100.39999999999999, "text": " on all of these tasks. Okay, nonetheless let me show you this one. So this was invented by Zacharias"}, {"start": 100.39999999999999, "end": 106.08, "text": " Janssen, Thomas Edison, blah blah blah. So the point here is that just looking at this image you"}, {"start": 106.08, "end": 112.8, "text": " can't deduce what's the answer, whereas here you could have. And so as you can see the output here"}, {"start": 112.8, "end": 119.2, "text": " is the right brothers dot end of sentence token. So the whole point here as we'll soon see is that"}, {"start": 119.2, "end": 124.56, "text": " the language model can now pull the factual knowledge in and we can answer questions like"}, {"start": 124.56, "end": 129.44, "text": " this because of the language model, not because of the vision portion of the model. Okay, let's"}, {"start": 129.44, "end": 135.12, "text": " dig deeper into the paper now. So that was a high level like a glimpse of this paper. Basically"}, {"start": 135.12, "end": 141.28, "text": " they can handle both modalities and they can like learn the more examples you give the model the"}, {"start": 141.28, "end": 147.12, "text": " better it becomes. That's really cool. Okay, let's now dive deeper. When trained at a sufficient scale"}, {"start": 147.12, "end": 152.0, "text": " autoregressive language models exhibit the notable ability to learn a new language task"}, {"start": 152.0, "end": 158.8, "text": " after being prompted with just a few examples. And this should be like familiar to you. If you"}, {"start": 158.8, "end": 164.56, "text": " haven't watched my video on GPT-3 or the original transformer paper, do check them out. I'll link"}, {"start": 164.56, "end": 169.04, "text": " them somewhere here. But basically what those models showed is the following, especially the"}, {"start": 169.04, "end": 176.16, "text": " GPT family of models from OpenAI. They showed that even though the model was trained as a language"}, {"start": 176.16, "end": 181.12, "text": " model, so basically you have an unsupervised task of next token prediction and that's everything."}, {"start": 181.12, "end": 185.04, "text": " And they show that you can actually do machine translation for example. And here is an example"}, {"start": 185.04, "end": 190.07999999999998, "text": " of how it looks like. So we have a one-shot setting here where we prompt the model with translate"}, {"start": 190.07999999999998, "end": 195.92, "text": " English to French and then we give it one example, C author to Lutre Des Morts, I don't speak French."}, {"start": 195.92, "end": 200.55999999999997, "text": " And then you prompt with cheese and you have this symbol and the model will actually learn how to"}, {"start": 200.55999999999997, "end": 206.48, "text": " translate this. So to perform machine translation even though it's never seen such a task during"}, {"start": 206.48, "end": 214.39999999999998, "text": " this training. And here we just have additionally a few-shot example set up where we have multiple"}, {"start": 215.27999999999997, "end": 220.56, "text": " examples and then we prompt the model to translate. And they show that with these multiple examples"}, {"start": 220.56, "end": 224.79999999999998, "text": " the performance just gets better and better. Obviously it saturates after a certain point,"}, {"start": 224.8, "end": 230.88000000000002, "text": " but like the trend is clear. Okay, so that's the first thing. Now here we present a simple"}, {"start": 230.88000000000002, "end": 236.16000000000003, "text": " yet effective approach for transferring this few-shot learning ability to a multi-modal setting."}, {"start": 236.16000000000003, "end": 243.36, "text": " And in particular they focus on vision as we'll soon see. Okay, finally here's a motivation behind"}, {"start": 243.36, "end": 248.4, "text": " all this. So despite these impressive capabilities, such large-scale language models are blind to"}, {"start": 248.4, "end": 253.60000000000002, "text": " modalities other than text, preventing us from communicating visual tasks, questions or concepts"}, {"start": 253.6, "end": 258.64, "text": " to them. So that's the reason they kind of integrated this vision component. Okay, let me"}, {"start": 258.64, "end": 263.2, "text": " now explain to you how this whole system looks like and how it works. So the system as a whole"}, {"start": 263.2, "end": 268.0, "text": " is pretty simple. So we have a language model as you can see here and it's frozen. So the parameters"}, {"start": 268.0, "end": 273.92, "text": " are all frozen and we have this vision encoder. So just to be a bit more specific they were using"}, {"start": 273.92, "end": 279.28, "text": " GPT-2 for the language model. They were using NF ResNet for the vision encoder, but that's not"}, {"start": 279.28, "end": 283.59999999999997, "text": " that crucial. You can pick some other language model, you can pick some different vision encoder,"}, {"start": 283.59999999999997, "end": 288.47999999999996, "text": " maybe something like vision image transformer, but that's not that important. Like the fact that"}, {"start": 288.47999999999996, "end": 296.0, "text": " these components exist and they are wired the way they are. Okay, so what I do is, so they first need"}, {"start": 296.0, "end": 301.11999999999995, "text": " to kind of adapt the image into the input that the transformer model is expecting and that's these"}, {"start": 301.11999999999995, "end": 306.23999999999995, "text": " tokens. So what I do is they have this vision encoder and it will output after some pulling"}, {"start": 306.24, "end": 311.28000000000003, "text": " layer a vector. And what they do is they take a linear layer, they just project this into"}, {"start": 312.64, "end": 321.04, "text": " this novel space. Let me draw it like this and the dimension will be n times d, where basically"}, {"start": 321.68, "end": 326.16, "text": " they found this n to be like the best value was two, but that's not important. As you can see here"}, {"start": 326.16, "end": 333.44, "text": " they have only two tokens, but it can be an arbitrary number in general. So d is, that's"}, {"start": 333.44, "end": 339.2, "text": " important. D is a dimension that's the same as these tokens that go into language model. So obviously"}, {"start": 339.2, "end": 344.08, "text": " that's the prerequisite so that we can feed these tokens into the language model that come from the"}, {"start": 344.08, "end": 353.12, "text": " image. And so that's the image part. Now how do they train this is fairly simple. So basically"}, {"start": 353.12, "end": 358.64, "text": " I think it would be wiser to write this down as, so we have start of sentence token here"}, {"start": 358.64, "end": 363.2, "text": " and all of these other words will be kind of translated here. So a will get here,"}, {"start": 363.76, "end": 371.2, "text": " this small will be here, etc. So we'll have small here and so now what you're trying to do is to"}, {"start": 371.2, "end": 378.15999999999997, "text": " predict the target sequence. So as you can see here, so let me just focus on predicting the word"}, {"start": 378.15999999999997, "end": 386.32, "text": " small. So the word small will have as a context these image tokens that came here as well as"}, {"start": 386.32, "end": 392.24, "text": " this start of sentence token as well as a because we have causal masking obviously you don't want to"}, {"start": 392.24, "end": 398.08, "text": " have if you saw small in the input then you're kind of cheating and it's easy to predict the"}, {"start": 398.08, "end": 402.56, "text": " output small. You can just kind of copy the values and the transformer will learn how just to copy"}, {"start": 402.56, "end": 407.28, "text": " copy paste the values and that's not what we want. We want to predict tokens. So that's how the"}, {"start": 407.28, "end": 412.32, "text": " test would look like. Basically here you'd have you'd output the distribution the usual way so"}, {"start": 412.32, "end": 418.15999999999997, "text": " you're just trying to maximize the likelihood. So here maybe we have some distribution and we find"}, {"start": 418.15999999999997, "end": 423.36, "text": " the token that corresponds to the word small maybe this one and we'll just want to maximize this to"}, {"start": 423.36, "end": 429.6, "text": " one and push all of the other probabilities down to zero and we do that by just simple like cross"}, {"start": 429.6, "end": 435.68, "text": " entropy so it will be a minus log of p so when p goes to one the loss goes to zero and so in a"}, {"start": 435.68, "end": 440.56, "text": " nutshell that's how the system works that's how it's trained the gradients are back propped"}, {"start": 440.56, "end": 447.52, "text": " through these weights which are frozen and these weights are then tweaked so that basically"}, {"start": 447.52, "end": 453.68, "text": " what happens is that this vision encoder learns such representations so that they're useful"}, {"start": 453.68, "end": 459.12, "text": " in order to do this captioning task okay and that's the whole system it's fairly"}, {"start": 459.12, "end": 465.6, "text": " trivial. Let me now show you how they use this thing and here we have in the first example"}, {"start": 465.6, "end": 471.44, "text": " so we have a vision encoder so we have an image we encode it into these two tokens and then we"}, {"start": 471.44, "end": 477.44, "text": " prompt it with like this text so question what color is the car and the model generates blue"}, {"start": 477.92, "end": 484.72, "text": " and then end of sentence token so by the way just a short remark here they actually even though you"}, {"start": 484.72, "end": 489.44, "text": " can see here a word and a single token what actually in practice what I do is they use this"}, {"start": 489.44, "end": 496.32, "text": " tokenizer called sentence piece so this boat or this word small maybe like separated into"}, {"start": 496.32, "end": 502.96, "text": " sub words maybe like maybe this sma part will be like one we'll have one token associated with it"}, {"start": 502.96, "end": 508.15999999999997, "text": " and then ll will have second so that's just an example but in general you'll have more tokens"}, {"start": 508.15999999999997, "end": 514.24, "text": " than you have words in your sentence okay just a minor detail and so that's this is one of the"}, {"start": 514.24, "end": 521.52, "text": " tasks that they're gonna evaluate this model on the second example here will require the model"}, {"start": 521.52, "end": 526.96, "text": " to have some knowledge base and additionally I forgot to mention that the captioning so during"}, {"start": 526.96, "end": 534.08, "text": " this captioning training all of the like named entities are masked so if you have a name like"}, {"start": 534.08, "end": 540.16, "text": " alexa or something you'll mask it with a person so you'll have you'll just like put a person"}, {"start": 540.16, "end": 545.8399999999999, "text": " instead of a name there so that means that this vision encoder cannot learn those named entities"}, {"start": 545.8399999999999, "end": 552.0799999999999, "text": " obviously because it never saw like named entities and so the examples you see here like"}, {"start": 552.0799999999999, "end": 558.0799999999999, "text": " like this model generated steve jobs that did not certainly didn't come from the vision encoder and"}, {"start": 558.0799999999999, "end": 562.88, "text": " that's an important thing I want you to notice here so the knowledge of that steve jobs was the"}, {"start": 562.88, "end": 568.8, "text": " guy who invented iphone actually came from the language model itself and that's the important"}, {"start": 568.8, "end": 574.16, "text": " from the language model itself so that's important and the third task they evaluate this model on is"}, {"start": 574.16, "end": 581.76, "text": " this fast binding task where you present the model with the image then you say you kind of have this"}, {"start": 581.76, "end": 588.24, "text": " made up word called dex and you just want the model to associate this visual category of an apple with"}, {"start": 588.24, "end": 594.0, "text": " the novel word dex and you do the same thing here you have an orange blanket and finally you prompt"}, {"start": 594.0, "end": 601.12, "text": " it with an image and you say question what is this answer and the model generates this is a dex so"}, {"start": 601.12, "end": 608.96, "text": " we saw dex is apple so it properly generated this is a dex sentence okay so having said that let me"}, {"start": 608.96, "end": 614.8, "text": " now kind of see let me show you the quantitative results they got because that's that's that's"}, {"start": 614.8, "end": 620.96, "text": " interesting before that just a short remark here so they say here in contrast our work enables"}, {"start": 620.96, "end": 627.6800000000001, "text": " strong generalization to new multi-modal tasks blah blah blah what i want to say here is i don't"}, {"start": 627.6800000000001, "end": 634.64, "text": " like this part because basically we don't even have a strong definition of what strong generalization"}, {"start": 634.64, "end": 641.2, "text": " is and the closest thing i could think of was fransois chollet's paper on the measure of"}, {"start": 641.2, "end": 647.6, "text": " intelligence where he described this terminology where he basically says that all of the current"}, {"start": 647.6, "end": 653.52, "text": " models pretty much that we have in deep learning field and machine learning are only like able to"}, {"start": 653.52, "end": 658.8000000000001, "text": " do this local generalization whereas this broad generalization and extreme generalization is"}, {"start": 658.8000000000001, "end": 665.76, "text": " something that only humans can currently basically do and so yeah i pretty much agree with him on"}, {"start": 665.76, "end": 672.08, "text": " this on this one because this language model that they use in frozen has seen like a lot of data so"}, {"start": 672.08, "end": 677.84, "text": " that means it has a lot of experience which we need to count in when we calculate these generalization"}, {"start": 677.84, "end": 684.32, "text": " capabilities so arguably because of all of that like immense amount of experience you cannot"}, {"start": 684.32, "end": 690.24, "text": " actually claim that it's generalizing like strongly it does generalize but like just the level yeah"}, {"start": 690.24, "end": 694.8000000000001, "text": " it's kind of yeah rent over pretty much okay these are just some details i already explained"}, {"start": 695.6800000000001, "end": 700.96, "text": " they have a huge seven billion pre-trained language model they freeze it and that's how they train the"}, {"start": 700.96, "end": 707.76, "text": " model i also explained this one how how do we map the images into into tokens which language model"}, {"start": 707.76, "end": 713.12, "text": " can then parse okay i'll just skip all of those small detail there is in positionally coding that"}, {"start": 713.12, "end": 720.4000000000001, "text": " helped them and let me focus on this so important thing i want you to notice is that they have a"}, {"start": 720.4000000000001, "end": 726.32, "text": " bunch of different ways to prompt this model so that it can generate the answers much better and"}, {"start": 726.32, "end": 732.24, "text": " here's one example so they have a pretty intricate uh like terminology here so two ways zero repeats"}, {"start": 732.24, "end": 737.0400000000001, "text": " two inner shots so let's see what i mean so first they have this task induction which they"}, {"start": 737.0400000000001, "end": 744.48, "text": " quantitatively show it helps a lot so answer with dex or blicket kind of prompts kind of tunes the"}, {"start": 744.48, "end": 749.84, "text": " model into answering with these two words in a sense and then they have so two shots basically"}, {"start": 749.84, "end": 755.36, "text": " this is the first part and uh the reason it's two ways because we have two made up words like"}, {"start": 755.36, "end": 760.08, "text": " blanket and ducks here and because the reason it's two inner shots is because they have two"}, {"start": 760.08, "end": 765.04, "text": " independent examples here as you can see and finally they prompt the model and blicket would"}, {"start": 765.04, "end": 770.4, "text": " be the correct answer because as you can see blicket is lion okay and the the last task they'll"}, {"start": 770.4, "end": 776.72, "text": " they'll they'll evaluate is this one where again you have to associate a visual category with a"}, {"start": 776.72, "end": 782.72, "text": " novel word and finally you actually not only have to output like here it's basically recognition"}, {"start": 782.72, "end": 788.32, "text": " you recognize it's a lion and you output blicket but here you have to reason because it says what"}, {"start": 788.32, "end": 795.0400000000001, "text": " is the the question says what is the dex made of so you first need to understand that dex is a table"}, {"start": 795.0400000000001, "end": 801.28, "text": " and then you need to understand what it is made of and the answer is wood so this is the final"}, {"start": 801.28, "end": 805.9200000000001, "text": " task they evaluate this frozen model on and now let's see the quantitative results okay here are"}, {"start": 805.9200000000001, "end": 811.6, "text": " the results for the visual question answering and there are these baselines which i'll shortly explain"}, {"start": 811.6, "end": 817.9200000000001, "text": " now especially this blind baseline so the strength of the pre-trained language model is a double edge"}, {"start": 817.9200000000001, "end": 823.12, "text": " sword it powers the generalization abilities of frozen but also enables the model to perform"}, {"start": 823.12, "end": 828.8000000000001, "text": " surprisingly well without considering the visual input at all so you can learn to ignore the visual"}, {"start": 828.8000000000001, "end": 833.84, "text": " input and still answer the question so to guard against this possibility we also train blind"}, {"start": 833.84, "end": 839.0400000000001, "text": " baselines in which the image presented to the visual encoder is blacked out but the common"}, {"start": 839.04, "end": 845.76, "text": " weights are still trained this amounts to prefix tuning so just as a short reminder how it works"}, {"start": 845.76, "end": 852.0, "text": " basically here instead of an image for these baselines what they'll do instead is they'll"}, {"start": 852.0, "end": 859.12, "text": " just black out or blue out in my case these images and so what happens is basically the model will"}, {"start": 859.12, "end": 864.56, "text": " have to learn some representation which is constant which won't which won't change because it will only"}, {"start": 864.56, "end": 871.4399999999999, "text": " be presented with black images and so that constant kind of needs to help in this captioning task"}, {"start": 871.4399999999999, "end": 877.1999999999999, "text": " and so that's why they call it prefix tuning because it finds some representation that helps"}, {"start": 877.1999999999999, "end": 882.16, "text": " the overall system do better job at captioning okay so that's one of the baselines the blind"}, {"start": 882.16, "end": 887.52, "text": " baseline and having explained that one let's now focus on the quantitative results okay here we are"}, {"start": 887.52, "end": 894.56, "text": " um so frozen here is the the the version we were talking about here is the version trained from"}, {"start": 894.56, "end": 899.52, "text": " scratch so that means you just stitch all of the weights in the pre-trained transformer and try"}, {"start": 899.52, "end": 906.0799999999999, "text": " and train it from scratch um here the the setups they have so this is zero shot one shot four shot"}, {"start": 906.0799999999999, "end": 913.28, "text": " etc and we can see that as we add more examples we have this trend that the the accuracy is"}, {"start": 913.28, "end": 919.04, "text": " improving which is desirable okay we see that the from scratch model totally fails we see that the"}, {"start": 919.04, "end": 924.72, "text": " fine-tune model is worse so fine-tuning so basically here they do not they do not freeze"}, {"start": 924.72, "end": 929.1999999999999, "text": " the language model but they initialize the model with a pre-trained weight the blind baseline"}, {"start": 929.1999999999999, "end": 935.68, "text": " i've just explained it as well so um again oscar is some dedicated baseline you can see it's much"}, {"start": 935.68, "end": 942.48, "text": " better than frozen but the whole point here is that we have this improve improving ability uh that"}, {"start": 942.48, "end": 949.44, "text": " also was pertinent to gpt3 models and language models in general so that's cool um here you can"}, {"start": 949.44, "end": 955.04, "text": " just see when you additionally fine-tune on this very data set visual visual question answering"}, {"start": 955.04, "end": 961.04, "text": " data set the performance obviously ramps up and that's that's expected i guess now the second"}, {"start": 961.04, "end": 967.12, "text": " thing here is the test on this okay vqa data set so this data set contains those examples where"}, {"start": 967.12, "end": 973.92, "text": " you need some additional knowledge base in order to answer the questions and again they so they"}, {"start": 973.92, "end": 981.28, "text": " show that um the the model improves with additional examples as a new baseline they use this 400"}, {"start": 981.28, "end": 987.2, "text": " million parameter model and by the way this one has seven billion and again obviously the the"}, {"start": 987.2, "end": 991.2, "text": " bigger the model the the better the performance is something that's not that surprising i guess"}, {"start": 991.2, "end": 999.0400000000001, "text": " uh like in 2021 and um yeah again the this baseline is much better they just want to stress out that"}, {"start": 999.0400000000001, "end": 1004.72, "text": " uh we have this improvement uh like being transferred to this multi-multi-multi-modal"}, {"start": 1004.72, "end": 1012.48, "text": " uh setup and that's cool okay so that was the the first experiments they did um here let me just"}, {"start": 1012.48, "end": 1017.12, "text": " uh kind of reiterate this because it's really important uh so this conceptual captions data"}, {"start": 1017.12, "end": 1022.24, "text": " set is hypernamed meaning that for example proper names are replaced with a general person"}, {"start": 1022.24, "end": 1027.76, "text": " a word-like person okay so this enables us to rigorously study the transfer of factual knowledge"}, {"start": 1027.76, "end": 1033.28, "text": " because all knowledge of named entities comes from language model pre-training consequently when we"}, {"start": 1033.28, "end": 1039.04, "text": " show the model an image of an airplane and ask who invented this the visual encoder has determined"}, {"start": 1039.04, "end": 1043.52, "text": " that the image contains an airplane and the language model has used this to retrieve the"}, {"start": 1043.52, "end": 1048.08, "text": " factual knowledge that airplanes were invented by the Wright brothers and finally jumping to"}, {"start": 1048.08, "end": 1055.44, "text": " fast concept binding these are the last tasks they tested this frozen model on um let me just"}, {"start": 1055.44, "end": 1061.52, "text": " kind of connect this table with the actual tasks so this is the task at hand we have this uh visual"}, {"start": 1061.52, "end": 1067.36, "text": " binding uh task so this is a two-way binding because we have again two novel made up words"}, {"start": 1067.36, "end": 1074.24, "text": " text and blick it they'll show like five way binding task where obviously you have now five"}, {"start": 1074.24, "end": 1080.1599999999999, "text": " made upwards and we'll soon see that it fails on that one but it succeeds on the two-way binding"}, {"start": 1080.1599999999999, "end": 1086.0, "text": " task so on the two-way binding you can see again they have a bunch of different ways to to prompt"}, {"start": 1086.0, "end": 1093.12, "text": " the model so that they can elicit better outputs and again here you can see it's improving with"}, {"start": 1093.12, "end": 1098.6399999999999, "text": " more examples second thing they tried is they used real names instead of those made upwards"}, {"start": 1098.6399999999999, "end": 1104.4799999999998, "text": " they do this so that they can quantify how harder it is for the model to learn the binding uh and"}, {"start": 1104.4799999999998, "end": 1109.4399999999998, "text": " how hard the task itself is and you can see so this kind of bugs me i'm not sure whether this"}, {"start": 1109.4399999999998, "end": 1115.36, "text": " is a typo but like you can it has better performance and then you kind of saturate saturates after three"}, {"start": 1115.36, "end": 1126.08, "text": " examples um again this annual baseline is better as i mentioned uh five way um binding fails so here"}, {"start": 1126.08, "end": 1134.1599999999999, "text": " it's literally like uh same as random chance 20 and it kind of improves here then goes back so"}, {"start": 1134.1599999999999, "end": 1138.6399999999999, "text": " yeah it's inconclusive here they mentioned here somewhere here just a sec so in table four we show"}, {"start": 1138.6399999999999, "end": 1144.08, "text": " that the observed effects on open-ended mini image net do not transfer to the five-way setting"}, {"start": 1144.08, "end": 1149.6799999999998, "text": " where frozen is not significantly above chance this shows that learning to bind five new names"}, {"start": 1149.6799999999998, "end": 1154.8799999999999, "text": " to five visual categories in a single forward pass is beyond the current capabilities of frozen okay"}, {"start": 1155.6, "end": 1161.12, "text": " so it kind of fails there and they leave it up as a future research the final task is the one i"}, {"start": 1161.12, "end": 1166.72, "text": " showed you where you need to aside from fast binding you need to reason not much new conclusions can"}, {"start": 1166.72, "end": 1172.6399999999999, "text": " be made here so again it's improving with more examples the interesting part maybe is so if we"}, {"start": 1172.64, "end": 1179.2800000000002, "text": " focus on this blind baseline we can see that even repeating uh so again remember the image is blacked"}, {"start": 1179.2800000000002, "end": 1185.8400000000001, "text": " out so we just just repeat the text a couple of times and those linguistic cues help boost the"}, {"start": 1185.8400000000001, "end": 1192.8000000000002, "text": " performance of this blind model which means that these improvements above basically are a combination"}, {"start": 1192.8000000000002, "end": 1200.4, "text": " of both linguistic cues as well as the visual cues okay um i think that's pretty much it i like this"}, {"start": 1200.4, "end": 1208.88, "text": " paper a lot i like this uh inclusion of visual uh information into this whole pipeline it slowly"}, {"start": 1208.88, "end": 1214.5600000000002, "text": " starts resembling the way we humans operate so we have these like we have vision obviously and we"}, {"start": 1214.5600000000002, "end": 1221.3600000000001, "text": " kind of uh somehow represent that information and then we have this like computation engine inside"}, {"start": 1221.3600000000001, "end": 1226.16, "text": " our head by the way one of the previous papers called uh like pre-trained transformers or"}, {"start": 1226.16, "end": 1232.4, "text": " universal computation engine or something they showed that like if you take a huge pre-trained"}, {"start": 1232.4, "end": 1237.2, "text": " language model and you just tweak some layer norm parameters and some embedding weights you can"}, {"start": 1237.2, "end": 1243.1200000000001, "text": " basically fine tune it very fast onto novel tasks and that's cool i guess this work is a follow-up"}, {"start": 1243.1200000000001, "end": 1248.64, "text": " in a way and yeah having said that i hope you like this video if you did consider subscribing"}, {"start": 1248.64, "end": 1260.64, "text": " sharing and until next time bye bye"}]