T0: Multitask Prompted Training Enables Zero-Shot Task Generalization | Paper Explained
❤️ Become The AI Epiphany 
Patreon ❤️ ► https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

In this video I cover the "Multitask Prompted Training Enables Zero-Shot Task Generalization" paper that introduced the T0 transformer.

T0 basically = Google's T5 + LM pretraining + additional training on prompted datasets.

The paper came out of the BigScience workshop.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2110.08207
✅ BigScience: https://bigscience.huggingface.co/
✅ Models on HF hub: https://huggingface.co/bigscience
✅ Prompt tool: https://github.com/bigscience-workshop/promptsource
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 BigScience workshop, HuggingFace hub
03:00 A high-level overview
04:40 Are we really doing implicit training?
06:30 T0 training and prompt templates
10:55 Choosing the val dataset
13:20 How is T0 trained?
14:55 Results (vs GPT3)
18:25 Ablations (varying prompts and data)
21:30 Discrepancy in GPT3 API vs reported results
22:30 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Bartłomiej Danek
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR ML PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#t0 #bigscience #huggingface
 What's cracking guys? In this video I'm covering multitask prompted training enables zero-shot task generalization by this huge group of people here and all of them came from this thing called Big Science Workshop. Let me show you here. So a one-year long research workshop on large multilingual models and datasets. So they took inspiration from scientific creation schemes such as CERN and the Large Hadron Collider and it's basically a collaboration between various companies and the whole effort is put straight up by Hugging Face, GenC and this Idris company and they even have a supercomputer on their disposal. So like 28 petaflops supercomputer located near Paris, France. So yeah if you're interested, if you fancy training huge language models do check out this Big Science Workshop and yeah I mean outside of Hugging Face and these companies there are many other like supporting institutions like Mila, Stanford, even Microsoft here. So it's a huge thing. Other than that the model, the T0 model I'm about to cover is already published on Hugging Face Hub. So you can find all of the versions of this T0, T0+, T0++ models are already here and additionally the model card is already available as well as this small widget and if you're not familiar with Hugging Face what basically happens here is that once somebody trains a model they can publish it to this hub and they are responsible to create this model card because that will like enable people to use that model so that's in their best interest as well as in the best interest of the audience, of the people who are going to use this, of customers. And finally they have this small like applet in this widget style applet where you can see, you can run the model in the background so we can just click compute here and the T0 is going to do classification on this sentence. So we have last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app. Select the category for the above sentence from mobile, website, billing and account access. So you can see that the prompt here is in a natural language form, you don't have just a sentence and then the output you have the prompt and then you have the categories here and the model finally figures out the correct category. So if I were to change website to iOS because that's a word that appears here I guess it's gonna now output iOS. If I press compute here it's gonna output iOS. Bottom line you can play with T0 already or you can like here in this small widget or even better you can you can download it locally and then start the whole thing. Other than that aside from models they also like open sourced the this tooling they were using to collect the prompts. I'm gonna see what those are in a second but like yeah all in all huge kudos to the whole organization for making all this public and usable for others. Okay getting back to the paper let's dig into it. So large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks. So that's the GPT-3 paper. It has been hypothesized that this is a consequence of implicit multitask learning in language model training. Can zero-shot generalization instead be directly induced by explicit multitask training? So basically the hypothesis is the following. When you're optimizing the language modeling objective so you're trying to predict the next word in the sentence the model basically figures out that it's way easier to learn how to solve certain NLP tasks such as summarization such as question answering which are relevant to us humans in order to to have an easier time to predict the next word. So all of those are implicitly learned as a consequence of trying to to minimize the cross entropy loss on the language modeling objective. The concern that the author has expressed here is that maybe just maybe those models are actually training explicitly and we just don't know about it so we call it implicit multitask training. We want to see that in a couple of minutes. Anyways they say here the model attains strong zero-shot performance on several standard data sets often outperforming models up to 16x its size. So they were comparing as we'll soon see this model t0 with GPT-3 family of models and oftentimes this model all performs GPT-3 which is super amazing. Okay let's continue here. I mentioned the problem with implicit multitask learning and they raise the concern here. So yet it is an open question how implicit this multitask learning really is. For example there are many websites that simply contain lists of trivia questions and answers and this data is precisely supervised training data for the task of closed book question answering. So let me open up that that website. Okay so here is like example of a website where we have those answer and question types of of of text. So you can see here what geometric shape is generally used for stop signs answer octagon. What is synophobia answer fear of dogs. So you can imagine that while the model was trying to to predict the next word it was actually learning how to answer this this question because once you prompt it with this thing and then you you force it to answer like this you're basically teaching it arguably how to do the question answering in this closed book setting. And you can imagine similar document somewhere in the huge data set that GPT-3 and T0 T5 models were using for training and that the implicit multitask may actually be explicit just that we are not aware of it. So therefore they insist on training this model explicitly and see whether that increases the zero shot generalization whether we can like get off with smaller models and whether we can have more robustness to those natural language prompts. And so they say here our goal is to induce a model to better generalize to unseen tasks without requiring massive scale as well as being more robust to the wording choices of the prompts. And that's very important because we see more and more lately like the the prompt engineering type of job where the whole point of some someone's job is to find the correct prompt in order to squeeze out the additional performance from from the language model. Anyways okay let's see how T0 is actually trained. So there are multiple tasks they they used we're going to see those in a bit detail a bit later but here here a couple of examples. So we have summarization task and you can see like the template here is the picture appeared on the wall of a pound land store on Weimark Avenue blah blah blah. So that's the question or the text and then you have how would you rephrase that in a few words. So that's the natural language prompt and the thing is this prompt here so they made so this whole group that wrote this paper they made an open call and they asked for contributors who would create novel creative prompts for these types of examples. So summarization problem summarization task does not have only this single type of prompt they have multiple prompts I think that's 8.1 on average or something. Anyways the model is expected to regress this type of a sentence graffiti artist banksy blah blah blah blah. Okay so other than that we have paraphrasing identification tasks so you can see here how is your traffic controlled that's one question then we have how do you become an air traffic controller that's the second question and then we have the prompt pick one these questions are duplicates or not duplicates and the model is supposed to say not duplicates in this specific case. So you can see that again this is a particular template for this particular task and there can be many more prompts as we'll soon see and yeah basically once you train the t0 model on various tasks you're gonna evaluate it on a held out set of tasks like natural language inference. Okay so let's go further here let me show you something so to make this a bit more clear because this prompting part is arguably the most important like novelty let's let's call it that way in this paper so here is a task of paraphrasing we can see again how is air traffic controlled that's question one we have we have question two and we have a label and here the label in the original data set is like numeric one so we have zero and zero corresponds to i think like not duplicates or one or whatever and in any case here is one particular prompt that people contributed so you can see here uh question one question two that's the template and then pick one these questions are duplicates or not duplicates and then you have choices uh label so that's gonna map zero to not duplicates or duplicates so basically you can see that this thing here is a template exact template for the actual instantiation we saw up there so that's this thing here so we have question question pick one blah blah blah so that's one particular template but other than that as i already mentioned we have other types of prompts like i received the questions question one and question two are they duplicates and again so you can see that semantics hasn't changed here but the like it's syntax wise it's completely different and that makes it makes it tougher for the for the model to to learn to solve that task uh similarly for every other like task you can see here the summary different prompts let me show you in even more detail how these guys made this uh like happen they basically created this uh like uh let's call it template language which helps automate the mapping from the original data set examples to the input output pair of examples using the natural language prompting so here we can see the example of how this template language works so we have a variable text and it's going to be eventually replaced by some concrete like text so we have here mark told pete many lies about himself which pete etc so that's one example but you can imagine any other example in the data set is going to pass through this scheme and out comes the actual input so here we'll have this replaced and then in the previous sentence does the pronoun and then they have span to text dot lower which is a python syntax and we have span to text is he so they ask does the pronoun he refer to spend one text that's going to be mark so does it refer to mark yes or no second example is here we have text and then here by he they mean uh mark yes or no so that's a second example and then you can have a bunch of these so text in other words blah blah true or false so basically all of these uh templates are gonna map the input examples into desired output format and that's how uh they created this novel uh prompted data sets let's call them that way anyways let me go back to the paper uh and let me show you a couple more things so i mentioned tasks and um it's a heuristic they they they they mentioned um the authors are well aware that this is not ideal because sometimes you can learn certain skills here and then even though you're testing the model on a held out data set you already learned the skill somewhere here so it's hard to decouple uh skills and they they opted to to do this way of separating it by tasks so we have um like close book q and a sentiment classification summarization etc etc and every single task has multiple data sets you can see here even like we have five data sets in total for the summarization task and then they have four uh like held out tasks as well as this big bench uh like set of data sets and they mentioned here that noting the grouping by task is an imperfect heuristic we are on the side of organizing our task taxonomy based on the task format as opposed to required skill largely based on conventions in the literature so as you can see this is pretty much still an open problem how do you separate nicely uh the the skills and the data sets from the held out ones so that you can um evaluate the model in a fair fashion and since they are comparing uh t0 with gpt3 they say that additionally we do not train our main model on any data sets that gpt3 use for evaluation so that our main results will be a fair zero shot comparison we verify that data for those tasks is not leaked through the pre-training corpus and i mean um this is a very tough problem uh even i if i if i recall correctly the original gpt3 paper actually had problems with leaking and subsequently they've done a post-hoc uh analysis and determined that the fact that the data leaked uh did not cause the gpt3 model to be much better on those on those tasks but yeah still uh like once you're dealing with such a huge internet scale types of data sets like like the ones used for gpt3 or t0 or t5 model upon which t0 is based on uh you have these problems of like leaking and as i said you you cannot decouple whether you learn a certain skill on the training data set and then try and evaluate the model in a totally different separate set of skills so that's still an open problem in the whole nlp field anyways i thought that was worth mentioning so let me continue here okay we saw this one and i mentioned that t0 is based on t5 and actually so the the the the fact is so the thing is t5 was originally trained on this mass language modeling objective and because uh they say here that it's quite different from the conditional text generation format used in our prompted data sets so because of that they actually took the t5 model that came from google and they additionally trained it as a like on a language modeling task and they refer to that model as the t5 plus lm and that's going to be used as a baseline throughout this paper in what is called t0 okay so let's see how the t0 is finally trained they mentioned here at a high level we assemble our multitask training mixture simply by combining all of the examples from all training data sets and shuffling the result this is equivalent to sampling from each data set in proportion to the number of examples in the data set so yeah basically you you take all of those question answering summarization tasks you you take their examples you map them using that template language and then you just kind of shuffle all of those of those examples and train this model by just patching those there are certain details here probably worth mentioning so some data sets are like even to order of magnitudes bigger than other ones so if you just if you were to just sample using this naive scheme the model would be overwhelmed by those a couple of those super huge data sets and so they have certain way to threshold the amount of examples coming from those bigger data sets in a nutshell okay so let's see the results finally we can see a comparison here between gpt family of models so we have gpt3 models here with different sizes we have 6.7 billion 13 billion and 175 so that's the actual gpt3 model the biggest one this is the t5 plus lm baseline and this is t0 model and we can see that most of the time on all of these tasks the t0 is way better compared to gpt3 so even the biggest one except on this data set and except on helloswag data set which are amazing results considering that gpt3 is way bigger compared to t0 like it's an order of magnitude bigger than t0 they argumented that on this task they perform worse because gpt3 use certain specific techniques that were like developed for that specific task and as for this one i haven't seen any any argument for why there is this huge gap in performance i guess that's an interesting open question for further research okay so they mentioned that note that brown at also the gpt3 paper authors reports performance on a single prompt whereas we report the median and interquartile range of performance across all prompts so they basically say that the the gpt3 paper they they cherry picked the best prompt and reported those results and we can see that on the chart here we have error bars for t0 and t5 plus lm baseline but we don't have any error bars here so this is the best case scenario pretty much the best prompt was cherry picked for this for this gpt3 model so that's something worth keeping in mind whereas here the model was evaluated on multiple prompts and all of the results are are plotted here like using these error bars okay let's continue we have other results on this big bench benchmark and again the trend is very similar only on this strategy qa is the t0 worse compared to these baselines whereas the baselines now are not gpt3 so keep that in mind they are just some models that the like people from google created those baselines for this big band a big bench benchmark uh okay so what's interesting here for me is the non-monotonic nature of of these models so a quick mention here we have t0 we know what it is we have t0 plus and we have t0 plus plus so t0 plus is just trained on additional data compared to t0 and t0 plus plus on even more data so let me just briefly go and show you what exactly that is so t0 plus is the same model but trained on a mixture that adds gpt3's evaluation data sets for t0 plus plus we add gpt3's and superglue data sets okay so that's the difference we have more data for these models so what's interesting going back here is that this non-monotonic behavior so on most tasks we have that basically adding more data is is a good thing but then we have certain tasks like this logical deduction where we have the inverse here happening so the the the model with the least data performs the best and then as we are adding data we have worse and worse performance and i guess investigating what happened here what kind of whether we had some overfitting or whatnot uh would be worth a try i guess okay finally let's see uh certain ablations they've done um the first one is uh training with various amounts of prompts per data set if that makes sense so they they take uh the t0 model and they uh they basically p equals zero means there are zero prompts that means they're basically using t5 plus lm baseline p equals one means for every data set they use a single prompt that was randomly chosen from from from the set of prompts and p equals all is basically all of the prompts for each data set which is on average i think around 8.1 or something and we can see the results here on various different tasks and uh we have this this this this candlestick diagram and you can see that the mean usually increases as we are adding so the blue one is the p equals zero the red one is the p equals one and the green one is the p equals all uh case so we can see that the median is constantly improving pretty much so the thing that worries me here is that the the spread is way bigger when we when when we set p equals to one but as we uh progress p to to to be equal to all we basically have uh like small spread and we have uh like higher median so that's cool so now question i have here is whether this is a like consequence of uh actually adding the prompts or is it just the fact that we have much more computation that went into training this this p equals all model compared to these two models so it'd be nice to to to uh um overlay a comparison here of models that were trained with less prompts but but had more compute time uh compared to these models here if that makes sense okay so they did aside from this uh in other uh in other like uh ablation and here uh they hold the number of prompts constant so they use all prompts uh but they vary so they have they vary the number of data sets that were used to train the t0 models so those were the t0 t0 plus and t0 plus plus models so what is interesting here is that uh it's not quite a clear cut uh which model is better here because first of all it seems like that t0 plus is overall worse compared to t like t0 like it has bigger spread and it most of the times even the median is lower compared to the uh t0 baseline so i mean yeah and then we have a trend where once we add this super glue data set we have a bump in performance here but again the spread is usually bigger way bigger compared to t0 here as well compare this to this and compare this to this this is definitely worth a further investigation i mean it is it is obviously not so this model here on this particular nly task is definitely not robust that means depending on how you pick your prompt you're gonna have a huge like difference in your performance and that's that's a bad thing that means this model is not robust uh when we increase the amount of data that will be like oh fun conclusion i have here okay uh let me wrap up this paper uh mentioning this part here so note that one of our templates is identical to brown at all again gpt3 paper authors reporter prompt this prompt scores 58.8 accuracy on the api base series which is lower than the reported accuracy of 63.5 from the paper itself so we can see a huge gap in performance between what the paper uh like reported and between and between what the actual api is offering so that's a like a point of concern for me there and they mentioned here that all other nine prompts however yield roughly random guessing performance with median accuracy roughly equal to 50 and interquartile range 1.28 these results suggest that t0 is more robust to prompt formulation than gpt3 so yeah i think this is a nice supporting evidence that training the models like this uh will make the models more robust so this offloads some of the work from those pro prompt programmer people maybe not so far in the future prompt programming will cease to exist and that's that's so sad um anyways um i'm gonna wrap it up with mentioning this um we release all models training this paper in addition to the collection of prompts we created and our prompt annotation tool so huge kudos to the authors uh for for doing this um this whole big science workshop sounds very exciting uh they have a supercomputer on disposal uh they are doing uh very nice interesting experiments and they are open sourcing all of that along the way so again do check it out and okay guys that's it for this video if you found it useful consider sharing it out with your friends as well as subscribing to this youtube channel and finally do join the discord community we have an ever-increasing number of smart people there so definitely worth checking out uh until next time bye bye
[{"start": 0.0, "end": 5.2, "text": " What's cracking guys? In this video I'm covering multitask prompted training enables zero-shot"}, {"start": 5.2, "end": 12.0, "text": " task generalization by this huge group of people here and all of them came from this thing called"}, {"start": 12.0, "end": 17.12, "text": " Big Science Workshop. Let me show you here. So a one-year long research workshop on large"}, {"start": 17.12, "end": 23.52, "text": " multilingual models and datasets. So they took inspiration from scientific creation schemes such"}, {"start": 23.52, "end": 29.04, "text": " as CERN and the Large Hadron Collider and it's basically a collaboration between various companies"}, {"start": 29.04, "end": 34.96, "text": " and the whole effort is put straight up by Hugging Face, GenC and this Idris company and they even"}, {"start": 34.96, "end": 42.08, "text": " have a supercomputer on their disposal. So like 28 petaflops supercomputer located near Paris,"}, {"start": 42.08, "end": 49.36, "text": " France. So yeah if you're interested, if you fancy training huge language models do check out this"}, {"start": 49.36, "end": 55.519999999999996, "text": " Big Science Workshop and yeah I mean outside of Hugging Face and these companies there are many"}, {"start": 55.52, "end": 63.36, "text": " other like supporting institutions like Mila, Stanford, even Microsoft here. So it's a huge"}, {"start": 63.36, "end": 70.0, "text": " thing. Other than that the model, the T0 model I'm about to cover is already published on Hugging"}, {"start": 70.0, "end": 76.24000000000001, "text": " Face Hub. So you can find all of the versions of this T0, T0+, T0++ models are already here"}, {"start": 76.24000000000001, "end": 82.24000000000001, "text": " and additionally the model card is already available as well as this small widget and if"}, {"start": 82.24, "end": 86.47999999999999, "text": " you're not familiar with Hugging Face what basically happens here is that once somebody"}, {"start": 86.47999999999999, "end": 91.44, "text": " trains a model they can publish it to this hub and they are responsible to create this model card"}, {"start": 91.44, "end": 96.96, "text": " because that will like enable people to use that model so that's in their best interest as well as"}, {"start": 96.96, "end": 101.44, "text": " in the best interest of the audience, of the people who are going to use this, of customers."}, {"start": 101.44, "end": 108.56, "text": " And finally they have this small like applet in this widget style applet where you can see,"}, {"start": 108.56, "end": 113.84, "text": " you can run the model in the background so we can just click compute here and the T0 is going to"}, {"start": 113.84, "end": 119.36, "text": " do classification on this sentence. So we have last week I upgraded my iOS version and ever since then"}, {"start": 119.36, "end": 124.48, "text": " my phone has been overheating whenever I use your app. Select the category for the above sentence"}, {"start": 124.48, "end": 130.72, "text": " from mobile, website, billing and account access. So you can see that the prompt here is in a natural"}, {"start": 130.72, "end": 135.12, "text": " language form, you don't have just a sentence and then the output you have the prompt and then you"}, {"start": 135.12, "end": 141.04, "text": " have the categories here and the model finally figures out the correct category. So if I were"}, {"start": 141.04, "end": 148.24, "text": " to change website to iOS because that's a word that appears here I guess it's gonna now output"}, {"start": 148.24, "end": 154.0, "text": " iOS. If I press compute here it's gonna output iOS. Bottom line you can play with T0 already or you"}, {"start": 154.0, "end": 158.56, "text": " can like here in this small widget or even better you can you can download it locally and then"}, {"start": 158.56, "end": 165.84, "text": " start the whole thing. Other than that aside from models they also like open sourced the"}, {"start": 166.64000000000001, "end": 170.96, "text": " this tooling they were using to collect the prompts. I'm gonna see what those are in a second"}, {"start": 170.96, "end": 178.64000000000001, "text": " but like yeah all in all huge kudos to the whole organization for making all this public and"}, {"start": 178.64000000000001, "end": 185.52, "text": " usable for others. Okay getting back to the paper let's dig into it. So large language models have"}, {"start": 185.52, "end": 191.68, "text": " recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks. So that's"}, {"start": 191.68, "end": 198.64000000000001, "text": " the GPT-3 paper. It has been hypothesized that this is a consequence of implicit multitask learning"}, {"start": 198.64000000000001, "end": 204.24, "text": " in language model training. Can zero-shot generalization instead be directly induced by"}, {"start": 204.24, "end": 210.08, "text": " explicit multitask training? So basically the hypothesis is the following. When you're optimizing"}, {"start": 210.08, "end": 215.12, "text": " the language modeling objective so you're trying to predict the next word in the sentence the model"}, {"start": 215.12, "end": 221.6, "text": " basically figures out that it's way easier to learn how to solve certain NLP tasks such as"}, {"start": 221.6, "end": 226.72, "text": " summarization such as question answering which are relevant to us humans in order to to"}, {"start": 228.8, "end": 234.88, "text": " have an easier time to predict the next word. So all of those are implicitly learned as a consequence"}, {"start": 234.88, "end": 240.32, "text": " of trying to to minimize the cross entropy loss on the language modeling objective. The concern"}, {"start": 240.32, "end": 246.23999999999998, "text": " that the author has expressed here is that maybe just maybe those models are actually training"}, {"start": 246.23999999999998, "end": 251.44, "text": " explicitly and we just don't know about it so we call it implicit multitask training. We want to"}, {"start": 251.44, "end": 257.12, "text": " see that in a couple of minutes. Anyways they say here the model attains strong zero-shot performance"}, {"start": 257.12, "end": 263.68, "text": " on several standard data sets often outperforming models up to 16x its size. So they were comparing"}, {"start": 263.68, "end": 270.96, "text": " as we'll soon see this model t0 with GPT-3 family of models and oftentimes this model all performs"}, {"start": 270.96, "end": 278.64, "text": " GPT-3 which is super amazing. Okay let's continue here. I mentioned the problem with"}, {"start": 278.64, "end": 284.72, "text": " implicit multitask learning and they raise the concern here. So yet it is an open question how"}, {"start": 284.72, "end": 290.64, "text": " implicit this multitask learning really is. For example there are many websites that simply contain"}, {"start": 290.64, "end": 295.76, "text": " lists of trivia questions and answers and this data is precisely supervised training data for"}, {"start": 295.76, "end": 302.15999999999997, "text": " the task of closed book question answering. So let me open up that that website. Okay so here is"}, {"start": 302.15999999999997, "end": 308.56, "text": " like example of a website where we have those answer and question types of of of text. So you"}, {"start": 308.56, "end": 314.4, "text": " can see here what geometric shape is generally used for stop signs answer octagon. What is"}, {"start": 314.4, "end": 320.64, "text": " synophobia answer fear of dogs. So you can imagine that while the model was trying to to predict the"}, {"start": 320.64, "end": 325.03999999999996, "text": " next word it was actually learning how to answer this this question because once you prompt it with"}, {"start": 325.03999999999996, "end": 331.03999999999996, "text": " this thing and then you you force it to answer like this you're basically teaching it arguably"}, {"start": 331.03999999999996, "end": 337.91999999999996, "text": " how to do the question answering in this closed book setting. And you can imagine similar document"}, {"start": 337.92, "end": 345.92, "text": " somewhere in the huge data set that GPT-3 and T0 T5 models were using for training and that the"}, {"start": 345.92, "end": 351.12, "text": " implicit multitask may actually be explicit just that we are not aware of it. So therefore they"}, {"start": 351.12, "end": 356.40000000000003, "text": " insist on training this model explicitly and see whether that increases the zero shot generalization"}, {"start": 356.40000000000003, "end": 361.84000000000003, "text": " whether we can like get off with smaller models and whether we can have more robustness to those"}, {"start": 361.84000000000003, "end": 367.36, "text": " natural language prompts. And so they say here our goal is to induce a model to better generalize to"}, {"start": 367.36, "end": 372.64, "text": " unseen tasks without requiring massive scale as well as being more robust to the wording choices"}, {"start": 372.64, "end": 377.52000000000004, "text": " of the prompts. And that's very important because we see more and more lately like the the prompt"}, {"start": 377.52000000000004, "end": 383.36, "text": " engineering type of job where the whole point of some someone's job is to find the correct prompt"}, {"start": 383.36, "end": 389.12, "text": " in order to squeeze out the additional performance from from the language model. Anyways okay let's"}, {"start": 389.12, "end": 394.56, "text": " see how T0 is actually trained. So there are multiple tasks they they used we're going to see"}, {"start": 394.56, "end": 400.08, "text": " those in a bit detail a bit later but here here a couple of examples. So we have summarization task"}, {"start": 400.08, "end": 406.16, "text": " and you can see like the template here is the picture appeared on the wall of a pound land store"}, {"start": 406.16, "end": 412.08, "text": " on Weimark Avenue blah blah blah. So that's the question or the text and then you have how would"}, {"start": 412.08, "end": 416.4, "text": " you rephrase that in a few words. So that's the natural language prompt and the thing is this"}, {"start": 416.4, "end": 423.12, "text": " prompt here so they made so this whole group that wrote this paper they made an open call and they"}, {"start": 423.12, "end": 429.84000000000003, "text": " asked for contributors who would create novel creative prompts for these types of examples. So"}, {"start": 429.84000000000003, "end": 434.64, "text": " summarization problem summarization task does not have only this single type of prompt they have"}, {"start": 434.64, "end": 440.96, "text": " multiple prompts I think that's 8.1 on average or something. Anyways the model is expected to regress"}, {"start": 440.96, "end": 446.16, "text": " this type of a sentence graffiti artist banksy blah blah blah blah. Okay so other than that we"}, {"start": 446.16, "end": 451.6, "text": " have paraphrasing identification tasks so you can see here how is your traffic controlled that's one"}, {"start": 451.6, "end": 456.88, "text": " question then we have how do you become an air traffic controller that's the second question"}, {"start": 456.88, "end": 461.04, "text": " and then we have the prompt pick one these questions are duplicates or not duplicates and"}, {"start": 461.04, "end": 466.88, "text": " the model is supposed to say not duplicates in this specific case. So you can see that again"}, {"start": 466.88, "end": 473.04, "text": " this is a particular template for this particular task and there can be many more prompts as we'll"}, {"start": 473.04, "end": 479.76000000000005, "text": " soon see and yeah basically once you train the t0 model on various tasks you're gonna evaluate it"}, {"start": 479.76, "end": 487.12, "text": " on a held out set of tasks like natural language inference. Okay so let's go further here let me"}, {"start": 487.12, "end": 493.03999999999996, "text": " show you something so to make this a bit more clear because this prompting part is arguably the"}, {"start": 493.03999999999996, "end": 500.56, "text": " most important like novelty let's let's call it that way in this paper so here is a task of"}, {"start": 500.56, "end": 505.68, "text": " paraphrasing we can see again how is air traffic controlled that's question one we have we have"}, {"start": 505.68, "end": 511.04, "text": " question two and we have a label and here the label in the original data set is like numeric"}, {"start": 511.04, "end": 516.08, "text": " one so we have zero and zero corresponds to i think like not duplicates or one or whatever"}, {"start": 516.08, "end": 520.64, "text": " and in any case here is one particular prompt that people contributed so you can see here"}, {"start": 520.64, "end": 525.36, "text": " uh question one question two that's the template and then pick one these questions are duplicates"}, {"start": 525.36, "end": 531.2, "text": " or not duplicates and then you have choices uh label so that's gonna map zero to not duplicates"}, {"start": 531.2, "end": 536.88, "text": " or duplicates so basically you can see that this thing here is a template exact template for the"}, {"start": 536.88, "end": 542.6400000000001, "text": " actual instantiation we saw up there so that's this thing here so we have question question pick"}, {"start": 542.6400000000001, "end": 547.36, "text": " one blah blah blah so that's one particular template but other than that as i already"}, {"start": 547.36, "end": 551.9200000000001, "text": " mentioned we have other types of prompts like i received the questions question one and question"}, {"start": 551.9200000000001, "end": 557.2, "text": " two are they duplicates and again so you can see that semantics hasn't changed here but the like"}, {"start": 557.2, "end": 562.6400000000001, "text": " it's syntax wise it's completely different and that makes it makes it tougher for the for the"}, {"start": 562.6400000000001, "end": 569.36, "text": " model to to learn to solve that task uh similarly for every other like task you can see here the"}, {"start": 569.36, "end": 576.5600000000001, "text": " summary different prompts let me show you in even more detail how these guys made this uh like happen"}, {"start": 576.5600000000001, "end": 582.88, "text": " they basically created this uh like uh let's call it template language which helps automate the"}, {"start": 582.88, "end": 589.52, "text": " mapping from the original data set examples to the input output pair of examples using the natural"}, {"start": 589.52, "end": 593.92, "text": " language prompting so here we can see the example of how this template language works so we have a"}, {"start": 593.92, "end": 598.88, "text": " variable text and it's going to be eventually replaced by some concrete like text so we have"}, {"start": 598.88, "end": 604.08, "text": " here mark told pete many lies about himself which pete etc so that's one example but you can imagine"}, {"start": 604.08, "end": 608.96, "text": " any other example in the data set is going to pass through this scheme and out comes the actual"}, {"start": 608.96, "end": 614.08, "text": " input so here we'll have this replaced and then in the previous sentence does the pronoun and then"}, {"start": 614.08, "end": 621.44, "text": " they have span to text dot lower which is a python syntax and we have span to text is he so they ask"}, {"start": 621.44, "end": 628.0, "text": " does the pronoun he refer to spend one text that's going to be mark so does it refer to mark yes or"}, {"start": 628.0, "end": 635.44, "text": " no second example is here we have text and then here by he they mean uh mark yes or no so that's"}, {"start": 635.44, "end": 639.9200000000001, "text": " a second example and then you can have a bunch of these so text in other words blah blah true or"}, {"start": 639.9200000000001, "end": 646.6400000000001, "text": " false so basically all of these uh templates are gonna map the input examples into desired output"}, {"start": 646.6400000000001, "end": 652.1600000000001, "text": " format and that's how uh they created this novel uh prompted data sets let's call them that way"}, {"start": 652.72, "end": 658.8000000000001, "text": " anyways let me go back to the paper uh and let me show you a couple more things so i mentioned"}, {"start": 658.8, "end": 666.24, "text": " tasks and um it's a heuristic they they they they mentioned um the authors are well aware that this"}, {"start": 666.24, "end": 671.3599999999999, "text": " is not ideal because sometimes you can learn certain skills here and then even though you're"}, {"start": 671.3599999999999, "end": 677.28, "text": " testing the model on a held out data set you already learned the skill somewhere here so it's"}, {"start": 677.28, "end": 684.64, "text": " hard to decouple uh skills and they they opted to to do this way of separating it by tasks so we have"}, {"start": 684.64, "end": 691.12, "text": " um like close book q and a sentiment classification summarization etc etc and every single task has"}, {"start": 691.12, "end": 696.48, "text": " multiple data sets you can see here even like we have five data sets in total for the summarization"}, {"start": 696.48, "end": 703.36, "text": " task and then they have four uh like held out tasks as well as this big bench uh like set of"}, {"start": 703.36, "end": 709.4399999999999, "text": " data sets and they mentioned here that noting the grouping by task is an imperfect heuristic we are"}, {"start": 709.44, "end": 714.6400000000001, "text": " on the side of organizing our task taxonomy based on the task format as opposed to required skill"}, {"start": 714.6400000000001, "end": 718.8800000000001, "text": " largely based on conventions in the literature so as you can see this is pretty much still an open"}, {"start": 718.8800000000001, "end": 724.6400000000001, "text": " problem how do you separate nicely uh the the skills and the data sets from the held out ones"}, {"start": 724.6400000000001, "end": 730.48, "text": " so that you can um evaluate the model in a fair fashion and since they are comparing uh t0 with"}, {"start": 730.48, "end": 736.8000000000001, "text": " gpt3 they say that additionally we do not train our main model on any data sets that gpt3 use for"}, {"start": 736.8, "end": 742.9599999999999, "text": " evaluation so that our main results will be a fair zero shot comparison we verify that data for those"}, {"start": 742.9599999999999, "end": 750.4, "text": " tasks is not leaked through the pre-training corpus and i mean um this is a very tough problem"}, {"start": 750.4, "end": 756.8, "text": " uh even i if i if i recall correctly the original gpt3 paper actually had problems with leaking"}, {"start": 756.8, "end": 762.4, "text": " and subsequently they've done a post-hoc uh analysis and determined that the fact that the"}, {"start": 762.4, "end": 768.3199999999999, "text": " data leaked uh did not cause the gpt3 model to be much better on those on those tasks but yeah"}, {"start": 768.3199999999999, "end": 773.92, "text": " still uh like once you're dealing with such a huge internet scale types of data sets like like the"}, {"start": 773.92, "end": 781.68, "text": " ones used for gpt3 or t0 or t5 model upon which t0 is based on uh you have these problems of"}, {"start": 781.68, "end": 788.24, "text": " like leaking and as i said you you cannot decouple whether you learn a certain skill on the training"}, {"start": 788.24, "end": 794.5600000000001, "text": " data set and then try and evaluate the model in a totally different separate set of skills so that's"}, {"start": 794.5600000000001, "end": 798.8, "text": " still an open problem in the whole nlp field anyways i thought that was worth mentioning so"}, {"start": 798.8, "end": 806.5600000000001, "text": " let me continue here okay we saw this one and i mentioned that t0 is based on t5 and actually"}, {"start": 806.5600000000001, "end": 813.52, "text": " so the the the the fact is so the thing is t5 was originally trained on this mass language modeling"}, {"start": 813.52, "end": 818.64, "text": " objective and because uh they say here that it's quite different from the conditional text generation"}, {"start": 818.64, "end": 824.16, "text": " format used in our prompted data sets so because of that they actually took the t5 model that came"}, {"start": 824.16, "end": 830.16, "text": " from google and they additionally trained it as a like on a language modeling task and they refer"}, {"start": 830.16, "end": 835.52, "text": " to that model as the t5 plus lm and that's going to be used as a baseline throughout this paper"}, {"start": 836.3199999999999, "end": 842.88, "text": " in what is called t0 okay so let's see how the t0 is finally trained they mentioned here at a high"}, {"start": 842.88, "end": 848.96, "text": " level we assemble our multitask training mixture simply by combining all of the examples from all"}, {"start": 848.96, "end": 853.84, "text": " training data sets and shuffling the result this is equivalent to sampling from each data set in"}, {"start": 853.84, "end": 859.52, "text": " proportion to the number of examples in the data set so yeah basically you you take all of those"}, {"start": 859.52, "end": 865.04, "text": " question answering summarization tasks you you take their examples you map them using that template"}, {"start": 865.04, "end": 870.08, "text": " language and then you just kind of shuffle all of those of those examples and train this model by"}, {"start": 870.08, "end": 875.6800000000001, "text": " just patching those there are certain details here probably worth mentioning so some data sets are"}, {"start": 875.6800000000001, "end": 882.08, "text": " like even to order of magnitudes bigger than other ones so if you just if you were to just sample"}, {"start": 882.08, "end": 886.88, "text": " using this naive scheme the model would be overwhelmed by those a couple of those super"}, {"start": 886.88, "end": 891.84, "text": " huge data sets and so they have certain way to threshold the amount of examples coming from"}, {"start": 891.84, "end": 898.0, "text": " those bigger data sets in a nutshell okay so let's see the results finally we can see a"}, {"start": 898.0, "end": 904.4, "text": " comparison here between gpt family of models so we have gpt3 models here with different sizes we"}, {"start": 904.4, "end": 911.2, "text": " have 6.7 billion 13 billion and 175 so that's the actual gpt3 model the biggest one this is the t5"}, {"start": 911.2, "end": 916.88, "text": " plus lm baseline and this is t0 model and we can see that most of the time on all of these tasks"}, {"start": 917.52, "end": 925.52, "text": " the t0 is way better compared to gpt3 so even the biggest one except on this data set and except on"}, {"start": 925.52, "end": 931.04, "text": " helloswag data set which are amazing results considering that gpt3 is way bigger compared to"}, {"start": 931.04, "end": 939.4399999999999, "text": " t0 like it's an order of magnitude bigger than t0 they argumented that on this task they perform"}, {"start": 939.4399999999999, "end": 946.4, "text": " worse because gpt3 use certain specific techniques that were like developed for that specific task"}, {"start": 947.04, "end": 952.96, "text": " and as for this one i haven't seen any any argument for why there is this huge gap in performance i"}, {"start": 952.96, "end": 959.12, "text": " guess that's an interesting open question for further research okay so they mentioned that"}, {"start": 960.1600000000001, "end": 967.36, "text": " note that brown at also the gpt3 paper authors reports performance on a single prompt whereas"}, {"start": 967.36, "end": 974.0, "text": " we report the median and interquartile range of performance across all prompts so they basically"}, {"start": 974.0, "end": 982.1600000000001, "text": " say that the the gpt3 paper they they cherry picked the best prompt and reported those results"}, {"start": 982.16, "end": 989.68, "text": " and we can see that on the chart here we have error bars for t0 and t5 plus lm baseline but we"}, {"start": 989.68, "end": 994.8, "text": " don't have any error bars here so this is the best case scenario pretty much the best prompt was"}, {"start": 994.8, "end": 999.28, "text": " cherry picked for this for this gpt3 model so that's something worth keeping in mind"}, {"start": 999.28, "end": 1006.3199999999999, "text": " whereas here the model was evaluated on multiple prompts and all of the results are are plotted"}, {"start": 1006.32, "end": 1015.0400000000001, "text": " here like using these error bars okay let's continue we have other results on this big bench"}, {"start": 1015.0400000000001, "end": 1022.48, "text": " benchmark and again the trend is very similar only on this strategy qa is the t0 worse compared to"}, {"start": 1022.48, "end": 1027.44, "text": " these baselines whereas the baselines now are not gpt3 so keep that in mind they are just some models"}, {"start": 1027.44, "end": 1033.52, "text": " that the like people from google created those baselines for this big band a big bench benchmark"}, {"start": 1033.52, "end": 1040.8, "text": " uh okay so what's interesting here for me is the non-monotonic nature of of these models so"}, {"start": 1041.76, "end": 1048.72, "text": " a quick mention here we have t0 we know what it is we have t0 plus and we have t0 plus plus"}, {"start": 1048.72, "end": 1054.96, "text": " so t0 plus is just trained on additional data compared to t0 and t0 plus plus on even more data"}, {"start": 1054.96, "end": 1060.24, "text": " so let me just briefly go and show you what exactly that is so t0 plus is the same model"}, {"start": 1060.24, "end": 1066.24, "text": " but trained on a mixture that adds gpt3's evaluation data sets for t0 plus plus we add"}, {"start": 1066.24, "end": 1072.8, "text": " gpt3's and superglue data sets okay so that's the difference we have more data for these models so"}, {"start": 1072.8, "end": 1078.64, "text": " what's interesting going back here is that this non-monotonic behavior so on most tasks we have"}, {"start": 1078.64, "end": 1085.36, "text": " that basically adding more data is is a good thing but then we have certain tasks like this logical"}, {"start": 1085.36, "end": 1091.6799999999998, "text": " deduction where we have the inverse here happening so the the the model with the least data performs"}, {"start": 1091.6799999999998, "end": 1097.4399999999998, "text": " the best and then as we are adding data we have worse and worse performance and i guess investigating"}, {"start": 1097.4399999999998, "end": 1102.4799999999998, "text": " what happened here what kind of whether we had some overfitting or whatnot uh would be worth a"}, {"start": 1102.4799999999998, "end": 1108.3999999999999, "text": " try i guess okay finally let's see uh certain ablations they've done um the first one is uh"}, {"start": 1108.3999999999999, "end": 1114.0, "text": " training with various amounts of prompts per data set if that makes sense so they they take"}, {"start": 1114.0, "end": 1119.36, "text": " uh the t0 model and they uh they basically p equals zero means there are zero prompts that"}, {"start": 1119.36, "end": 1125.76, "text": " means they're basically using t5 plus lm baseline p equals one means for every data set they use a"}, {"start": 1125.76, "end": 1131.36, "text": " single prompt that was randomly chosen from from from the set of prompts and p equals all is"}, {"start": 1131.36, "end": 1136.08, "text": " basically all of the prompts for each data set which is on average i think around 8.1 or something"}, {"start": 1136.8, "end": 1141.76, "text": " and we can see the results here on various different tasks and uh we have this this this"}, {"start": 1141.76, "end": 1148.4, "text": " this candlestick diagram and you can see that the mean usually increases as we are adding so the"}, {"start": 1148.4, "end": 1154.8, "text": " blue one is the p equals zero the red one is the p equals one and the green one is the p equals all"}, {"start": 1155.28, "end": 1160.8, "text": " uh case so we can see that the median is constantly improving pretty much so the thing that worries me"}, {"start": 1160.8, "end": 1167.76, "text": " here is that the the spread is way bigger when we when when we set p equals to one but as we uh"}, {"start": 1167.76, "end": 1174.16, "text": " progress p to to to be equal to all we basically have uh like small spread and we have uh like"}, {"start": 1174.16, "end": 1181.52, "text": " higher median so that's cool so now question i have here is whether this is a like consequence of uh"}, {"start": 1181.52, "end": 1187.12, "text": " actually adding the prompts or is it just the fact that we have much more computation that went into"}, {"start": 1187.12, "end": 1193.12, "text": " training this this p equals all model compared to these two models so it'd be nice to to to uh"}, {"start": 1193.12, "end": 1199.6, "text": " um overlay a comparison here of models that were trained with less prompts but but had more compute"}, {"start": 1199.6, "end": 1207.12, "text": " time uh compared to these models here if that makes sense okay so they did aside from this uh"}, {"start": 1207.12, "end": 1214.32, "text": " in other uh in other like uh ablation and here uh they hold the number of prompts constant so they"}, {"start": 1214.32, "end": 1219.28, "text": " use all prompts uh but they vary so they have they vary the number of data sets that were used to"}, {"start": 1219.28, "end": 1224.24, "text": " train the t0 models so those were the t0 t0 plus and t0 plus plus models so what is interesting"}, {"start": 1224.24, "end": 1231.28, "text": " here is that uh it's not quite a clear cut uh which model is better here because first of all"}, {"start": 1231.28, "end": 1238.3999999999999, "text": " it seems like that t0 plus is overall worse compared to t like t0 like it has bigger spread"}, {"start": 1238.3999999999999, "end": 1246.24, "text": " and it most of the times even the median is lower compared to the uh t0 baseline so i mean yeah and"}, {"start": 1246.24, "end": 1253.68, "text": " then we have a trend where once we add this super glue data set we have a bump in performance here"}, {"start": 1253.68, "end": 1260.8, "text": " but again the spread is usually bigger way bigger compared to t0 here as well compare this to this"}, {"start": 1260.8, "end": 1266.64, "text": " and compare this to this this is definitely worth a further investigation i mean it is it is"}, {"start": 1266.64, "end": 1272.88, "text": " obviously not so this model here on this particular nly task is definitely not robust that means"}, {"start": 1272.88, "end": 1279.5200000000002, "text": " depending on how you pick your prompt you're gonna have a huge like difference in your performance"}, {"start": 1279.5200000000002, "end": 1284.8000000000002, "text": " and that's that's a bad thing that means this model is not robust uh when we increase the amount"}, {"start": 1284.8000000000002, "end": 1291.8400000000001, "text": " of data that will be like oh fun conclusion i have here okay uh let me wrap up this paper uh"}, {"start": 1291.8400000000001, "end": 1298.48, "text": " mentioning this part here so note that one of our templates is identical to brown at all again gpt3"}, {"start": 1298.48, "end": 1306.4, "text": " paper authors reporter prompt this prompt scores 58.8 accuracy on the api base series which is"}, {"start": 1306.4, "end": 1313.44, "text": " lower than the reported accuracy of 63.5 from the paper itself so we can see a huge gap in performance"}, {"start": 1313.44, "end": 1319.92, "text": " between what the paper uh like reported and between and between what the actual api is offering"}, {"start": 1320.48, "end": 1326.56, "text": " so that's a like a point of concern for me there and they mentioned here that all other nine prompts"}, {"start": 1326.56, "end": 1333.2, "text": " however yield roughly random guessing performance with median accuracy roughly equal to 50 and"}, {"start": 1333.2, "end": 1340.24, "text": " interquartile range 1.28 these results suggest that t0 is more robust to prompt formulation than gpt3"}, {"start": 1340.24, "end": 1344.3999999999999, "text": " so yeah i think this is a nice supporting evidence that training the models like this"}, {"start": 1344.3999999999999, "end": 1348.8, "text": " uh will make the models more robust so this offloads some of the work from those pro"}, {"start": 1348.8, "end": 1353.6799999999998, "text": " prompt programmer people maybe not so far in the future prompt programming will cease to exist and"}, {"start": 1353.68, "end": 1360.88, "text": " that's that's so sad um anyways um i'm gonna wrap it up with mentioning this um we release all models"}, {"start": 1360.88, "end": 1364.96, "text": " training this paper in addition to the collection of prompts we created and our prompt annotation"}, {"start": 1364.96, "end": 1371.92, "text": " tool so huge kudos to the authors uh for for doing this um this whole big science workshop"}, {"start": 1371.92, "end": 1378.0, "text": " sounds very exciting uh they have a supercomputer on disposal uh they are doing uh very nice"}, {"start": 1378.0, "end": 1383.04, "text": " interesting experiments and they are open sourcing all of that along the way so again do check it out"}, {"start": 1383.04, "end": 1388.1599999999999, "text": " and okay guys that's it for this video if you found it useful consider sharing it out with your"}, {"start": 1388.1599999999999, "end": 1393.52, "text": " friends as well as subscribing to this youtube channel and finally do join the discord community"}, {"start": 1393.52, "end": 1399.12, "text": " we have an ever-increasing number of smart people there so definitely worth checking out uh until"}, {"start": 1399.12, "end": 1414.08, "text": " next time bye bye"}]