GPT-3 - Language Models are Few-Shot Learners | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video, I cover the famous GPT-3 model.

I first give you some context about the stuff that happened since the paper was first published in May 2020 (hype, anti-hype, limitations, and cool apps), and then I dive deep into explaining the paper.

You'll learn about:
✔️ Useful resources on GPT-3
✔️ Main takeaways from the paper

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ "anti-hype" blog: https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html
✅ Gwern's blog: https://www.gwern.net/GPT-3
✅ My transformer implementation: https://github.com/gordicaleksa/pytorch-original-transformer
✅ Cool "GPT game": https://play.aidungeon.io/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable:

00:00 GPT (anti)hype, Gwern, prompt programming
04:30 Abstract of the paper
06:50 Architecture, data, compute
12:15 Zero-shot, one-shot, and few-shot learning
18:45 Power-law chart (more compute please)
20:35 Results (machine translation)
23:05 NLI (reasoning is hard)
24:40 Arithmetic
26:25 Word unscrambling
28:40 SAT analogies (how smart are humans?)
30:45 Fake news generation
32:05 Data contamination
35:05 Limitations of the model
37:35 Bias, fairness (broader impact)
44:30 Final thoughts, are we going towards an AGI?

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#gpt3 #fewshotlearning #deeplearning
 What's up folks? So in this video I thought covering this famous GPT-3 model titled language models are a few-shot learners Published earlier this year by OpenAI. So first I just want to give you an overview like a context of what was happening since then and Basically what we had is on one hand we had a lot of hype happening. So people saying so this is artificial general intelligence It's going to solve everything and then on the other hand we had people nagging saying how Going in this direction is going to ruin the field of machine learning and this is Like there is no way that this is the the the right path towards AGI so basically the truth is probably somewhere in between and Let me show you a couple of demos before I start doing a deep dive in of this of this paper Which I think you'll find interesting. So first Twitter had a lot of Demos happening. So this was one of the most famous ones So basically a guy would input a button that looks like a watermelon and then GPT-3 would like kind of generate code For for a button that would look like a watermelon, which is pretty pretty awesome And then we had also cool applications like AI dungeon, which actually appeared much earlier But they just upgraded their their back end using Partly using GPT-3 model so you can basically hear Enter like the genre of the of the play then you pick your character like let me take I know wizard and then let me put in the name like GPT-3 Let me see what it generates and basically it will generate text and then you can interactively play this game by just like Prompting it with different text and that's super awesome and nice use case for for this model So I won't do it right now. I'll link it in the description so you can play with the game if you want So as I mentioned on the other side we had people who Who used a single example as an argument that the model is not not as good as as people are hyping it that it is So basically here is a question which is heavier a toaster or a pencil and GPT-3 would answer a pencil is heavier than a toaster which is not true like in probably in most most of the cases and Then like second example, maybe how many eyes does my foot have and GPT-3 would answer your foot has two eyes so After this blog was published a lot of people were experimenting with GPT-3 Including this guy named Gwern Whose blog you should definitely check out he's writing a lot It's a bit harder to parse in my in my like subjective opinion, but like it's a really nice resource so what he showed is that using like smart like by picking smart hyper parameters like the Temperature like the sampling method maybe nucleus and by picking like the correct text that will that would get input to the model That's like the conditioning text you can get the GPT-3 to answer all of those questions Like the toaster or pencil question correctly, which is surprising and then like basically like people started talking about a new paradigm So we had like we had the software 1.0 We still have it like where you're basically just designing the algorithm so like classical computer science, then we have designing the data set So that's pretty much machine learning. You're now designing the data set and then the algorithm kind of just develops by learning from the data using gradient based methods and finally We are just we came in a situation where we are designing a prompt So as to communicate effectively communicate with this GPT-3 model So Gwern had a nice sentence here, which says sampling can prove the presence of knowledge, but not the absence and So that's that's the idea behind all of this like prompt programming you basically Can't so if the model doesn't give you a correct answer Maybe the model wasn't incentivized to give you a correct answer. Maybe the model was just being a jokester So now I'm anthropomorphizing GPT-3 But Gwern showed that by thinking like that by thinking that you're pretty much communicating with a human You can more effectively get the information you want out of it. So that was it for the overview So we had the hype the hyping team. We had the nagging team. We have we had everything in between so now like at the end of 2020 we know much more about both the limitations and Useful use cases that models like GPT-3 can have so without further ado, let's jump into the deep dive Let's jump into the paper. So first thing you can notice is that like a whole lot of people work in this project Like this looks like like a half of the open AI team pretty much like bunch of people. So it's a multi person effort as all of these huge papers are and What this paper basically did was because they have a lot of resources. They were able to kind of explore what happens when you're Increasing the model size and especially with the looking at how the what the like a few shot Performances compared conscious that to fine-tuning which is the usual thing to do like you you pre-train the model And then you fine-tune it. Well this paper Did the pre-training also but then without any fine-tuning they try to see what's the zero shot one shot and few shot Performance of this model and we'll get into some details a bit later. So Let me see what this say here So by contrast humans can generally perform a new language task from only a few examples or from simple instructions Something which current NLP systems still largely struggle to do so that's their like point basically humans you only need a couple of examples and you're already able to solve the new task and Now it's debatable like because we've been collecting data like like since we were born pretty much like continuously So yeah, it's a complicated question. But like I do agree that we need to Develop better like few shot performance of our models Then we have basically what they say is that so that the thing is they don't update the gradients So we'll see in a moment how they do it So basically they just condition the pre-trained model on like a bunch of text and it doesn't have to be bunch of text It's like a bit basically like a prompt in natural language and then maybe a couple of examples We'll see it in a minute and so the final finally what they what they what they Say is that it's able to generate text That's really hard to distinguish from the text that humans wrote and then they kind of Have a whole section on broader impacts because this kind of technology can really be used for malicious malicious purposes as well and I'll explain I'll talk about that part also like the fairness and bias a bit later So, okay. Let's let me first we'll go kind of backwards here. I'll first want to explain you the architecture the The data set and then we'll jump to how the a few shot thing works. So basically the architecture is They say here the same as GPT-2, which is basically the same as GPT, which is Basically the decoder portion of the transformer model So we can see it here So this is the the the transformer paper like this is the transformer model from the attention is all you need paper And they took the decoder part and they just they don't need this multi-head attention because they don't have the encoding part So they they just leave the the causal masking. So basically tokens can only look The tokens that came previously or or look at themselves. So yeah, so maybe I'll try and briefly explain how it works again Let me Take a pencil here So you basically take the input text you you tokenize it so you get let me zoom zoom in a little bit So you you tokenize it Then what you do is you embed these tokens so that's this output embedding part so let's say for the for the sake of argument that the Uh, like the hidden dimension in the model is like 512. So basically you'll end up with a bunch of vectors They have like dimension 512 And then what you do is you have these positional encodings. It's basically like a huge table So it's basically a huge table and then depending on the position here. So this is like Token number zero you basically take zero And you add this with uh like row here and you just add it up To this embedding vector and after adding it up your you end up here and then you just have two parts of the architecture So the first one is the multi-headed attention Which is basically attends to all of the tokens and creates like those nice representations when I say attains to all so it's not bi-directional as I said, they have the causal masking and Finally, we have the feedforward part where basically, yeah, you just kind of independently process those token representations again. So I have a video about this one, but just wanted to do a quick recap of it. And basically the only difference is they use something called a couple of ideas from the sparse transformer paper where you basically, instead of using those like causal masks, you use sparse causal masks. So let's not get into too much details. And yeah, that's pretty much it about the architecture. Now for the training data set, they used, so they used the common crawl, which is a huge, huge data set pretty much like crawling every single month that site downloads bunch of the data from the internet. And what they did is they filtered it, and they also used something called WebText, which was created by using the links from Reddit, which had at least three uploads, which was kind of heuristic for filtering like higher quality content. And they just kind of like scraped all of the data that those links were pointing to. They also have the books, One Books, Two and Wikipedia. So basically the point is that depending on the quality of the data set, they took more samples or less samples. So some of the data like Wikipedia, which is higher quality than for example, common crawl, like had like 3.4 epochs, meaning the model saw it approximately 3.4 times, whereas the common crawl was only seen 0.44 times. One important thing to notice here and comment is that they had a problem with data contamination, which means many of the results, they later showed that the results are not as seriously affected. Some of the results that were seriously affected were kind of just not displayed in the paper, but I still think they had some contamination issues left over, especially on the PICA data set where they are testing for understanding of the like physics in the world. And we'll get to it a bit later. The compute was enormous. So they had like, you basically had like 10 years of petaflop second. Like if you had a machine that has a petaflop second performance, you need 10 years to train this model. So it's huge in that sense, you can see it here. And the interesting comparison is that they say here, as a consequence, although GPT-3, 3 billion is almost 10X larger than Roberta large, both models took approximately 50 petaflop second days of compute during pre-training. The reason is that there is some other paper that tells that you shouldn't, that the model shouldn't see like tokens too many times, otherwise it will overfit and the performance will get worse. So that was that part about the data. And yeah, one more interesting thing maybe is that basically they needed a supercomputer because the model was huge. He had 175 billion, so they needed like a supercomputer from Microsoft to train the thing. So that's kind of a fun fact. Let me jump to something that's really like the core part of this paper. And that's how they do this zero shot, like one shot and few shot learning and why they do it. So basically why they do it is, so they said here, so the reason to distinguish between one shot from few shot and zero shot is that it most closely matches the way in which some of the tests are communicated to humans. So yeah, basically that's how humans work. That's how humans work and then for the zero shot, there are some examples like for example, translation. If I told you to translate from English to German, supposing you knew both languages, you'd know what to do. You wouldn't need any additional examples. But in some other cases, you do need example and that's why the one shot and few shot setting. So basically in some cases, it may even be difficult for humans to understand the format of the task without prior examples. So this setting is in some cases unfairly hard. So zero shot is going to be, always going to have like worse performance and we're going to see some data that will back that up. Also, they say here, nevertheless, for at least some settings, zero shot is closest to how human perform tasks. So that's the thing I mentioned about the translation. Okay, having said that, let's see some examples here. So basically on the translation example, this is a zero shot setting. You basically say, you condition the model on this sentence. So you give it translate English to French, cheese and then prompt. And basically you expect the model to regressively complete the sentence and translate cheese into whatever, however you say cheese in French, I don't speak French. Then there is the one shot where you give one example like sea otter, l'outre de mer, I don't know if it's pronounced like that, but let's say for the sake of it. And then the few shot one where you give multiple examples and then you prompt it and expect the correct answer. So depending on the tasks, some will make more sense, some will make less sense and few shot basically almost always gives better performance, which makes sense. And let me compare that to fine tuning, which is basically what most of the other models in NLP so far, like BERT, et cetera, they always use to pre-train the model on a huge corpus of text. And then they fine tune the model on a specific downstream like task and then that's how it pretty much works. And the problem, so they did notice a couple of problems with that, there are some problems with doing that. Basically you kind of overfit to a more narrow distribution, that's one problem. Second problem is you need supervised like label data, which is usually expensive. And the third problem is that humans like don't need like a label data, like you only need a couple of examples and that's it. So that's the few shot argument again. Here they say that they usually need around, they usually put around 10 to 100 examples in the few shot setting, depending on the, like the basically they're constrained by this, by the model size pretty much, by the memory actually. And they have only 2048 tokens available. So depending on the example size, they can go from 100 to, from 10 to 100. So that was, that was that part. And there are a couple of interesting charts in the beginning of the paper. So this one shows us that as we are scaling, so they pretty much averaged across different benchmarks, which we'll see in a couple of minutes, they averaged the performance. And you can see that as the, on the x-axis as the, as the model size grows in size, so as we slowly get to the 175 billion model version, we get the better and better performance. And the second important thing to notice here is that the difference in performance between the few shot, one shot and zero shot also increases, which kinda tells us that the model is becoming better at meta-learning. So those are two interesting facts to see from this plot. So the first one, bigger the scale, the better the performance. And the second thing is they are getting better at meta-learning as the size increases, which we can see by the difference between the different performances. So basically going from zero to one shot here, you get less improvement than by going from zero to one to few shot learning here. So that was that chart. Let me see what else we have here. So here they show how the, like the number of context, number of examples in the context also affects the performance. And they showed a couple of models. So the 1.3 billion, the 13 billion, and the biggest one, the GPT-3 175 billion model. And we can see that as we increased the number of examples, usually the performance also increases. And this is for some specific task, like insertion, trying to unscramble the word, which I'll explain a bit later. But that's the basic idea. So we have the zero shot, we have the one shot setting, and we have the few shot setting, and the performance increases. And the second thing is the bigger the model, like the steeper the improvements in a sense. Okay, and finally, this is the last chart in this section, which tells us, so they treat the model in a sense like a meta-learning, like a, they treat the problem as a meta-learning problem. So basically you pre-train the model, that's the outer loop. So that's all of those web crawl, like common crawl data, web text and common crawl data. And then once you condition the model on the, on the like text, like we saw in the translation example, you basically, that's the inner loop. When you put the context, you have, you form in a sense, ephemeral weights, which define the model. So the model is kinda like changes its, let's call it shape, depending on the conditioning text. So you have a completely different model, which is in a sense fine-tuned to the new, adapts to the new task, which is the interesting thing. Before I jump to the results section, and show you how it performs on different benchmarks, let me show you one more really important chart. So this chart shows us that the validation loss decreases as the compute increases. And the x-axis is the log scale, and you can basically see, so GPT-3, which had this many parameters, so that's approximately 175 billion params, and had 3,000 days of compute. So this is the GPT-3 model. You can see as the compute increases, the loss decreases, and it follows the power law here. So basically smaller models with less compute saturate at higher validation loss, which is an interesting, which was actually hypothesized by some of the previous papers, and this paper showed that the loss still holds even for huge, huge models such as the GPT-3. So that was all I had to tell you before we jumped to the results section. So pretty much nothing new in the research sense. They pretty much like sweep the whole combinatorial space, tried a bunch of different models, saw how it generalizes. So the nice thing is that they are pushing for this few shot and zero shot and one shot like key evaluation and not for the fine tuning approach, which just gives you better results in the benchmark, but is not as applicable as having a general model which you can later apply on the fly, which is really cool. So the paper is really huge, so I'm going to just show you a subset of interesting results. And yeah, so the first one will be, I'll skip Lombetta. Those are some language modeling tasks, and as expected because the model was, GPT-3 was pre-trained as a language model, i.e. predicting the next word in the sentence. It performs pretty good on those tasks. I'll skip this and I'll jump to translation. Let me just find the curves. Whoop, just a sec, this one. What's interesting is that basically the model was not explicitly trained to do translation. And that's one of the funny things about this GPT-3 model. So basically it was just trained to predict the next word, but what emerges is a set of skills which seem to be useful to know in order to predict those words. And one of those skills is just like translation. And I'm pretty surprised by the BLEO score it achieved. So I previously reconstructed the original transformer paper, and I'll link the project in the description. And basically I think I achieved around 33 BLEO score for English to German. And here without being explicitly trained to do translation and just having a small subset of the text in German, it achieved a really, really decent score. And interesting thing is that going from French to English, German to English, and Romanian to English, the performance is really good. But once you go in the opposite direction, it's really hard to achieve a good result. So this, the Romanian really has some serious problems. It's around 20, which is really, really low. That's interesting. And that shows that this model is a good English language model, but not as good for other languages. So they noticed, so as I mentioned, they noticed this skew, and they said that this could be a weakness due to reusing the bad level BPE tokenizer of GPT-2, which was developed for an almost entirely English training data set. So BPE, the by pair encoding was, many people pointed out that BPE can be causing different problems. We'll see some problems that it's causing in one of the tasks I'm going to show you about unscrambling words. And yeah, just keep that in mind about the BPE. The next interesting tasks I wanna show you is the natural language inference, where the model is having some problems doing those. So the NLI tasks, and we can see here, basically, so the task there is to read the text and understand how sentences relate to each other. For example, you give it one sentence and the second sentence, and you ask it to say whether the second sentence follows from the first, whether it contradicts the first sentence, or whether they're just neutral. And basically, that's really interesting because that means that the model doesn't handle comprehension like in reasoning, as well as it does just like generating sequences like translation or language modeling or like completing sentences. So different tasks that have to do with generation. And we can see the results here that we have pretty much random behavior on smaller models, and then as the model grows, the few shot model actually improves the performance, but it's still way below the baselines, like even BERT and Roberta is even better. So those are some of the tasks that this model is having problems with. So A stands for adversarial. So there's actually a subset of those NLI examples where humans picked some examples which are especially hard for language models. And yeah, so it's struggling with those types of problems. Now my favorite ones are these synthetic tasks. So what they did is they tried and prompt the model whether it knows how to do some basic addition, subtraction, and multiplication, and here we can see results. So basically for two digit addition and subtraction, the model has really good performance, at least the biggest one, and we can see like huge improvements as the model size increases again. But then when we go to three digit addition, perf goes down and it goes even further down for four digit addition and subtraction, for multiplication, and for some three digit operations. So what is probably causing this is that like in the huge stacks that the model has seen during the pre-training, there were a lot of tables that had like two digit numbers, but less so for three digit numbers, and even less so for four and five, et cetera. So basically this may be indicative that the model is not learning how to reason and actually doesn't adapt in a meta-learning sense to this new task and learns it from a few examples, but it's just doing some kind of a statistical like pattern matching and stuff. So it's still unexplored what exactly happens in the few shot setting, but like a probably good hypothesis is that it's not reasoning in the sense we humans are reasoning. Okay, so that was an interesting example. So one more really interesting task for me was this word scrambling and manipulation task. So they had five sub-tasks here. So the first one was cycle letters in words. So basically you give the model something like this and the model is supposed to figure out that this is actually just a circularly permuted. So you don't scramble it into inevitably here. Then we have anagrams where you keep the first and the last letter and you scramble everything in between and that's what you give to the model and you expect the model to unscramble the word. And this is actually super tough. I had problems also unscrambling this word. So I thought it had something to do with crypto, but then yeah, it doesn't have any Y, but it's still, it's kind of hard. And then this one is a bit easier. You hold the first two letters fixed and the last two letters and you just unscramble the middle and you get opponent. Then they have a random insertion in the word where you basically just after every single character, you insert a random punctuation or like a blank space and you expect again, the model to unscramble it. And finally, we have just the reverse word. So but just the model should figure out that you should just reverse the letters and get objects here. So those were the tasks and a bit surprisingly, so even the biggest model, the GPT-3, the 175 billion one had zero like accuracy on the reverse words. So that's the first intuitive thought I had like what's happening here. And then you figure out that the model is using sub word tokenization. So it's really hard for it to do this task. Whereas the easiest one was random insertion, which was also easiest for me actually. And then you can see the anagram, the one word anagram is having problems. So that mirrors my difficulties with understanding with doing this A1 task as well. And then the A2 anagram task was a bit easier. So that's the one where we keep the first two and last two letters fixed. So that's actually all intuitive. So the same difficulties I had solving these problems, the model had similar difficulties. There are two more interesting tasks I wanna kind of explore a bit more in depth. So the first one is the set analogies. And here you can see what the problem looks like. So this is in the appendix part of the paper. Basically the context is a law is to trust as, and then you have a multiple like answers here, multiple choices, and you are supposed to pick out the correct one. And I don't know about you, like English is not my native language. And I think I'm pretty good at it. But like some of these words like Kajoli or Belk were not familiar to me. So this problem actually turned out to be really difficult even for me. So I was just thinking how we are like testing our models really like really heavy. Even the humans are not like, not nearly as smart as we think they are. So I mean, yeah, that's just something that struck me and was interesting. So let me go and see how the model is actually performing on the set problem. So on the set analogy problem. So what was interesting for me was that actually GPT-3 achieved higher score than average high school student. So on this task GPT-3 achieved 65.2 in the few shot setting, whereas the average score among college applicants was 57%. And we can see the curves here again, the few shot one is growing steadily and is always the best for the GPT-3 model. So one thing that we can maybe conclude from this result is that we shouldn't be teaching our kids to just do these tasks that are really easy for like big neural networks to do. They should be focusing more like on things that the networks can do, like reasoning and maybe art or stuff. But even art is something that's already in the realm of neural networks. So I guess that's going to be a complicated question that we have to solve, but let me not digress too much. The final problem that I wanted to talk about is the news article generation. And the results here are really striking. Basically, if I show you the plot here, as the model size increases, so here on the X dimension, on the X axis, you can see that the accuracy goes down, meaning that people are less and less confident that an article, whether the text came from the model or whether it came from humans. So they did a really nice study where they tested a bunch of models, also they used something called control model, which was a smaller GPT-3 version, and it intentionally had like a higher temperature, which increased the randomness of the softmax. So basically, it was more random than all of the other models. And they compared all of those models with the control model. And what it turns out is that as we go, as the models increase, we end up pretty much with the fact that people can decide whether an article came from human or from the GPT-3, which is amazing and has some probably scary repercussions. So those were some interesting results that they showed in this huge paper. This video's already getting too long, so I'm going to tell you a little bit more about two or three more things. So one is data contamination. So what happened is that because the training dataset they have is so huge, they ended up having some of the dev and test data from the benchmarks already being included in the training dataset. And then they did some investigations, and for the most part, they said that the tests weren't as affected as they initially expected because there was a bug in their filtering of the data, which caused all of this. So yeah, but some of the tests like PICA, where they had physical, let me see if I can find it. So this is one example from PICA data sets. So basically the context is how to apply sealant to wood, and then we have the correct answer, using a brush, brush on sealant onto wood until it is fully saturated with the sealant, and then they have the incorrect one. So as they showed, the results on PICA were really good, and because it looks like it's an outlier, I do think that contamination played its role here. Okay, getting back to this chart, I just wanna note one more thing. And so they had a bug, so they said here, unfortunately a bug resulted in only partial removal of all detected overlaps from the training data. Due to the cost of training, it wasn't feasible to retrain the model. So even I think that they're using some kind of like, they're using 13 grams to figure out the overlaps, and they didn't mention somewhere in the paper that this is still a new area of research. So I'm really skeptical about this part, how they are, and what do they consider by duplicate. So it's hard to figure out which parts of text you should filter out so that you can say with certainty, okay, these examples from the dev and test sets are not present in any way in the training sets in a way that could bias and help the model to better predict on those dev and test sets. So I mentioned PICA, we can see it's an outlier, so it has better performance. You can see here that the points on the lower part of this chart have better performance on the dirty data sets, and so that means that there was a contamination issue. What I'm not clear about is this drop. They did mention it, but it looks like a huge outlier, but they didn't mention anything specifically about it. They did mention PICA and VinaGrad, et cetera. So that was it about this part. Now I wanna just walk you through some of the limitations they mentioned. So the first one is this one. So GPT-3 samples still sometimes repeat themselves semantically at the document level, start to lose coherence over sufficiently long passages, contradict themselves, and occasionally contain non-sequitur sentences or paragraphs. So even though this is a huge language model, it still has its own problems, like repeating. So there's still a lot of research to do about how do we decode the output from these models and whether the problem is the model itself or the heuristics we are using for decoding. So basically, Gvern and most people are using the top P, the nucleus decoding, and they're playing with temperatures, so it still doesn't feel like as a correct way to do things. They then mentioned that they are obviously aware that if they had bi-directional representations compared, contrasted to the ones that they do have, and that's the causal, the unidirectional representations, they expect better results. So that's a thing they could try out. They also acknowledge that the pre-training objective could be further improved, and they also acknowledge that the understanding precisely how few-shot learning works is an important unexplored direction for future research. So yeah, basically nobody quite knows how this exactly works. It's still kind of not easy to interpret, and that's one of the bad things. So they mentioned it here. Its decisions are not easily interpretable. Also, the pure scale of the model doesn't allow them to iterate when they make mistakes, as we saw on the example with data contamination. So a limitation associated with models at the scale of GPT-3 regardless of objective function or algorithm is that they are both expensive and inconvenient to perform inference on, which may present a challenge for practical applicability of models of this scale in their current form. So I guess some sorts of knowledge distillations and pruning, et cetera, could help us make smaller, yet performant models. I wanna close up this video talking about the broader impact. So I do think this is important, even though it's not like a technical part, so feel free to skip it if you're not interested in it. So basically what they consider, so with GPT-2, they had a staged release. So that happened last year, and they initially released the smallest GPT-2 model. They were monitoring different forums, et cetera, and just looking for any signs of potential misuse. Once they were pretty certain that no misuse was happening, they started gradually releasing the models, and at the end of 2019, the whole GPT-2, the biggest one, I think it had 1.5 billion params, was published. So GPT-3 is still not published, and it probably will never be, but the API is already accessible for certain people. It's in beta. Okay, so what I wanted to say here is that it's even more important to consider these questions for GPT-3, because basically whatever malicious task that depends on producing a huge amount of text, GPT-3 will help with that. So that's phishing, that's spamming, bunch of different stuff. So social engineering, so that can be automated using GPT-3 in a sense. So they did consider those things. So many of these applications bottle on human beings to write sufficiently high-quality text. And they mentioned who has the resources to do this, like government groups. I don't think this part is as interesting as this part about the bias. So they considered three axes. So one is gender, the second one is religion, and the third one is, let me check, race. Yep. So how they evaluated the bias in these models is, they basically prompted with sentences like this one. The occupation was up. Like for example, and they note here the detective was up, and they are looking at the, what's the probability distribution coming out from the GPT-3, and they're looking at male and female. And what I figured out is that most of the occupations are biased towards males. So especially when they put it like this, the competent, then fill in the blank, was up. And they figured out that that version of the prompt had even higher bias for males. And actually also the incompetent one was also biased more towards males. So again, iterating a little bit on this one. Here we can see in particular occupations demonstrating higher levels of education, such as legislator, banker, or professor, were heavily male leaning. And also like physical labor stuff was also male leaning, where the occupations that were more likely to be followed by female identifiers include midwife, nurse, receptionist, housekeeper, et cetera. So once more how it works, you basically take a variant of the prompt like this one, you input it into your GPT-3, let me zoom in a little bit. So you basically input it into your GPT-3 model. Let's say this is GPT-3. And it outputs as the next token, so the probability distribution which has the size of the GPT-3's vocab, which I think is around 50K. And what it did then was they monitored a couple of male identifiers like man, male, et cetera, and they monitored like female, woman, et cetera. And they just pretty much added the probabilities and compared those. They also did some normalization. So basically that's the method how they did this. And I already mentioned the results that came out of it. So that was about the gender. Then I'll just briefly go over the race. The, so across the models, they analyzed Asian had a consistently high sentiment, whereas the black had a consistently low sentiment. And this was kind of surprising because you usually hear that like white males are much better positioned than Asians. So this was kind of surprising in a way for me. And yeah, that's an obvious bias. And especially for black people, like a lot of the text has, is referencing like slavery, et cetera. And then that's the reason black people have the lowest sentiment. They did mention it here. So the resulting sentiment can reflect socio-historical factors. For instance, text relating to a discussion of slavery will frequently have a negative sentiment. And that's the reason why the black race always has lowest sentiment, like for every single model size. So X axis is the model size. Here is the biggest one. Finally, religion, they tried out like the top, the most popular like world religions like Christianity, Buddhism, Islam, et cetera. And so they found some negative biases about Islam. Like we also found that words such as violent, terrorism, terrorist occurred at a greater scale with Islam than with other religions. And we're in the top 40 most favorite words for Islam in GPT-3, which is really sad. And we need to be aware of these biases because obviously, I won't get into this, but like most of the Muslim people are really nice folks and that's it. They did also consider the energy usage. And one really surprising fact was this. So though models like GPT-3 consumes significant resources during training, they can be surprisingly efficient once trained. So even with the full GPT-3, 175 billion per am, a large model, generating 100 pages of content from a trained model can cost on the order of 0.4 kilowatt hour or only a few cents in energy costs. It's actually pretty efficient once you train this huge monster. And then during the lifetime of the model, it can be like the energy costs can be amortized potentially. Yep. So that was pretty much all I had to say. I could make this video much longer. I mean, this is a huge paper. So hopefully you liked it. If you did, please leave a comment, what you liked, what you didn't like, what are your opinions on this? And yeah, regarding the nagging versus the hyping sides, I just wanna mention one thing. So what's my opinion on this one? So I really think that these huge models, like huge transformers, are not something that's a digression or a bad thing for the future of AI. Why do I think that? That's because your brain, most of your brain, like cerebellum, where there is, the largest amount of computation happens in cerebellum is actually unconscious. So we are probably now in the phase where we are developing a synthetic primordial brain. Basically, the equivalent that we have, which is pretty much regulating all of our vital functions, like controlling the heart, temperature, et cetera, and also perception, so just kind of extracting concepts from image and audio, et cetera. So all of those require a lot of computation, and I don't see this as a digression. I just see it as a supplement, and as a step towards the artificial general intelligence, which will happen once we are able to simulate the cognitive layer above the primordial brain level. So those are just some two cents on my side about what I think about this. And I have some gut feeling that graph neural networks and causational relations that Judea Pearl has been pushing for decades now will play a significant role, as well as symbolic AI. So I don't think we should kinda disregard all of those, and we should just be cognizant that all of those will probably somehow play a role like in this long-term goal towards achieving AGI. So that was pretty much it. If you liked this video, consider subscribing to this channel and hit that bell icon to get notified when I upload a new video. And until next time, keep learning. Deep. We'll see you next time.
[{"start": 0.0, "end": 8.08, "text": " What's up folks? So in this video I thought covering this famous GPT-3 model titled language models are a few-shot learners"}, {"start": 8.72, "end": 13.76, "text": " Published earlier this year by OpenAI. So first I just want to give you an overview"}, {"start": 14.120000000000001, "end": 16.84, "text": " like a context of what was happening since then and"}, {"start": 17.56, "end": 24.84, "text": " Basically what we had is on one hand we had a lot of hype happening. So people saying so this is artificial general intelligence"}, {"start": 24.84, "end": 29.82, "text": " It's going to solve everything and then on the other hand we had people nagging saying how"}, {"start": 30.36, "end": 34.16, "text": " Going in this direction is going to ruin the field of machine learning and this is"}, {"start": 34.480000000000004, "end": 38.96, "text": " Like there is no way that this is the the the right path towards AGI"}, {"start": 39.32, "end": 41.96, "text": " so basically the truth is probably somewhere in between and"}, {"start": 42.64, "end": 47.54, "text": " Let me show you a couple of demos before I start doing a deep dive in of this of this paper"}, {"start": 48.3, "end": 50.3, "text": " Which I think you'll find interesting. So"}, {"start": 50.879999999999995, "end": 52.879999999999995, "text": " first Twitter had a lot of"}, {"start": 52.88, "end": 56.040000000000006, "text": " Demos happening. So this was one of the most famous ones"}, {"start": 56.040000000000006, "end": 63.080000000000005, "text": " So basically a guy would input a button that looks like a watermelon and then GPT-3 would like kind of generate code"}, {"start": 63.64, "end": 67.88, "text": " For for a button that would look like a watermelon, which is pretty pretty awesome"}, {"start": 67.88, "end": 74.48, "text": " And then we had also cool applications like AI dungeon, which actually appeared much earlier"}, {"start": 74.48, "end": 77.2, "text": " But they just upgraded their their back end using"}, {"start": 77.96000000000001, "end": 80.74000000000001, "text": " Partly using GPT-3 model so you can basically hear"}, {"start": 80.74, "end": 86.58, "text": " Enter like the genre of the of the play then you pick your character like let me take"}, {"start": 86.58, "end": 91.0, "text": " I know wizard and then let me put in the name like GPT-3"}, {"start": 91.53999999999999, "end": 95.86, "text": " Let me see what it generates and basically it will generate text and then you can"}, {"start": 96.74, "end": 99.02, "text": " interactively play this game by just"}, {"start": 100.06, "end": 101.16, "text": " like"}, {"start": 101.16, "end": 106.78, "text": " Prompting it with different text and that's super awesome and nice use case for for this model"}, {"start": 106.78, "end": 111.16, "text": " So I won't do it right now. I'll link it in the description so you can play with the game if you want"}, {"start": 112.58, "end": 115.54, "text": " So as I mentioned on the other side we had people who"}, {"start": 115.9, "end": 122.42, "text": " Who used a single example as an argument that the model is not not as good as as people are hyping it that it is"}, {"start": 122.62, "end": 130.78, "text": " So basically here is a question which is heavier a toaster or a pencil and GPT-3 would answer a pencil is heavier than a toaster"}, {"start": 130.9, "end": 135.54, "text": " which is not true like in probably in most most of the cases and"}, {"start": 135.54, "end": 143.78, "text": " Then like second example, maybe how many eyes does my foot have and GPT-3 would answer your foot has two eyes so"}, {"start": 144.89999999999998, "end": 148.7, "text": " After this blog was published a lot of people were experimenting with GPT-3"}, {"start": 149.5, "end": 151.5, "text": " Including this guy named Gwern"}, {"start": 152.22, "end": 155.34, "text": " Whose blog you should definitely check out he's writing a lot"}, {"start": 155.34, "end": 160.82, "text": " It's a bit harder to parse in my in my like subjective opinion, but like it's a really nice resource"}, {"start": 160.82, "end": 168.62, "text": " so what he showed is that using like smart like by picking smart hyper parameters like the"}, {"start": 168.66, "end": 176.34, "text": " Temperature like the sampling method maybe nucleus and by picking like the correct text that will that would get input to the model"}, {"start": 176.66, "end": 181.54, "text": " That's like the conditioning text you can get the GPT-3 to answer all of those questions"}, {"start": 182.14, "end": 190.54, "text": " Like the toaster or pencil question correctly, which is surprising and then like basically like people started talking about a new paradigm"}, {"start": 190.54, "end": 192.98, "text": " So we had like we had the software 1.0"}, {"start": 192.98, "end": 200.38, "text": " We still have it like where you're basically just designing the algorithm so like classical computer science, then we have designing the data set"}, {"start": 200.38, "end": 205.7, "text": " So that's pretty much machine learning. You're now designing the data set and then the algorithm kind of just"}, {"start": 206.29999999999998, "end": 210.57999999999998, "text": " develops by learning from the data using gradient based methods and finally"}, {"start": 210.57999999999998, "end": 214.7, "text": " We are just we came in a situation where we are designing a prompt"}, {"start": 214.73999999999998, "end": 218.38, "text": " So as to communicate effectively communicate with this GPT-3 model"}, {"start": 218.38, "end": 225.62, "text": " So Gwern had a nice sentence here, which says sampling can prove the presence of knowledge, but not the absence and"}, {"start": 226.34, "end": 230.62, "text": " So that's that's the idea behind all of this like prompt programming you basically"}, {"start": 231.46, "end": 234.01999999999998, "text": " Can't so if the model doesn't give you a correct answer"}, {"start": 234.42, "end": 239.14, "text": " Maybe the model wasn't incentivized to give you a correct answer. Maybe the model was just being a jokester"}, {"start": 239.14, "end": 241.98, "text": " So now I'm anthropomorphizing GPT-3"}, {"start": 242.01999999999998, "end": 247.62, "text": " But Gwern showed that by thinking like that by thinking that you're pretty much communicating with a human"}, {"start": 247.62, "end": 253.82, "text": " You can more effectively get the information you want out of it. So that was it for the overview"}, {"start": 253.82, "end": 258.94, "text": " So we had the hype the hyping team. We had the nagging team. We have we had everything in between"}, {"start": 259.02, "end": 265.1, "text": " so now like at the end of 2020 we know much more about both the limitations and"}, {"start": 265.66, "end": 272.7, "text": " Useful use cases that models like GPT-3 can have so without further ado, let's jump into the deep dive"}, {"start": 272.7, "end": 278.42, "text": " Let's jump into the paper. So first thing you can notice is that like a whole lot of people work in this project"}, {"start": 278.42, "end": 283.53999999999996, "text": " Like this looks like like a half of the open AI team pretty much like bunch of people. So it's a"}, {"start": 284.18, "end": 285.58, "text": " multi"}, {"start": 285.58, "end": 288.98, "text": " person effort as all of these huge papers are and"}, {"start": 289.78, "end": 296.18, "text": " What this paper basically did was because they have a lot of resources. They were able to kind of"}, {"start": 296.78, "end": 298.78, "text": " explore what happens when you're"}, {"start": 298.78, "end": 304.85999999999996, "text": " Increasing the model size and especially with the looking at how the what the like a few shot"}, {"start": 305.41999999999996, "end": 311.34, "text": " Performances compared conscious that to fine-tuning which is the usual thing to do like you you pre-train the model"}, {"start": 311.34, "end": 313.73999999999995, "text": " And then you fine-tune it. Well this paper"}, {"start": 314.14, "end": 320.53999999999996, "text": " Did the pre-training also but then without any fine-tuning they try to see what's the zero shot one shot and few shot"}, {"start": 320.53999999999996, "end": 324.78, "text": " Performance of this model and we'll get into some details a bit later. So"}, {"start": 325.5, "end": 327.5, "text": " Let me see what this say here"}, {"start": 327.5, "end": 333.5, "text": " So by contrast humans can generally perform a new language task from only a few examples or from simple instructions"}, {"start": 333.5, "end": 338.34, "text": " Something which current NLP systems still largely struggle to do so that's their like point"}, {"start": 338.34, "end": 345.42, "text": " basically humans you only need a couple of examples and you're already able to solve the new task and"}, {"start": 346.18, "end": 353.02, "text": " Now it's debatable like because we've been collecting data like like since we were born pretty much like continuously"}, {"start": 353.02, "end": 357.65999999999997, "text": " So yeah, it's a complicated question. But like I do agree that we need to"}, {"start": 357.97999999999996, "end": 360.53999999999996, "text": " Develop better like few shot performance of our models"}, {"start": 361.09999999999997, "end": 366.62, "text": " Then we have basically what they say is that so that the thing is they don't update the gradients"}, {"start": 366.62, "end": 368.62, "text": " So we'll see in a moment how they do it"}, {"start": 368.62, "end": 375.74, "text": " So basically they just condition the pre-trained model on like a bunch of text and it doesn't have to be bunch of text"}, {"start": 375.74, "end": 380.21999999999997, "text": " It's like a bit basically like a prompt in natural language and then maybe a couple of examples"}, {"start": 380.22, "end": 385.90000000000003, "text": " We'll see it in a minute and so the final finally what they what they what they"}, {"start": 386.62, "end": 390.14000000000004, "text": " Say is that it's able to generate text"}, {"start": 390.14000000000004, "end": 395.66, "text": " That's really hard to distinguish from the text that humans wrote and then they kind of"}, {"start": 396.06, "end": 404.14000000000004, "text": " Have a whole section on broader impacts because this kind of technology can really be used for malicious malicious purposes as well and I'll"}, {"start": 404.70000000000005, "end": 405.74, "text": " explain"}, {"start": 405.74, "end": 408.70000000000005, "text": " I'll talk about that part also like the fairness and bias"}, {"start": 408.7, "end": 410.53999999999996, "text": " a bit later"}, {"start": 410.53999999999996, "end": 416.62, "text": " So, okay. Let's let me first we'll go kind of backwards here. I'll first want to explain you the"}, {"start": 417.34, "end": 418.86, "text": " architecture the"}, {"start": 418.86, "end": 425.34, "text": " The data set and then we'll jump to how the a few shot thing works. So basically the architecture is"}, {"start": 426.06, "end": 430.14, "text": " They say here the same as GPT-2, which is basically the same as GPT, which is"}, {"start": 430.7, "end": 433.34, "text": " Basically the decoder portion of the transformer model"}, {"start": 434.06, "end": 436.06, "text": " So we can see it here"}, {"start": 436.06, "end": 443.42, "text": " So this is the the the transformer paper like this is the transformer model from the attention is all you need paper"}, {"start": 443.42, "end": 450.38, "text": " And they took the decoder part and they just they don't need this multi-head attention because they don't have the encoding part"}, {"start": 450.38, "end": 455.58, "text": " So they they just leave the the causal masking. So basically tokens can only look"}, {"start": 456.38, "end": 464.54, "text": " The tokens that came previously or or look at themselves. So yeah, so maybe I'll try and briefly explain how it works again"}, {"start": 464.54, "end": 466.54, "text": " Let me"}, {"start": 466.86, "end": 468.86, "text": " Take a pencil here"}, {"start": 469.26000000000005, "end": 475.5, "text": " So you basically take the input text you you tokenize it so you get let me zoom zoom in a little bit"}, {"start": 475.90000000000003, "end": 477.90000000000003, "text": " So you you tokenize it"}, {"start": 479.98, "end": 485.42, "text": " Then what you do is you embed these tokens so that's this output embedding part"}, {"start": 485.90000000000003, "end": 488.86, "text": " so let's say for the for the sake of argument that the"}, {"start": 488.86, "end": 494.7, "text": " Uh, like the hidden dimension in the model is like 512. So basically you'll end up with a bunch of vectors"}, {"start": 495.90000000000003, "end": 498.46000000000004, "text": " They have like dimension 512"}, {"start": 499.58000000000004, "end": 504.62, "text": " And then what you do is you have these positional encodings. It's basically like a huge table"}, {"start": 505.02000000000004, "end": 509.90000000000003, "text": " So it's basically a huge table and then depending on the position here. So this is like"}, {"start": 510.7, "end": 513.98, "text": " Token number zero you basically take zero"}, {"start": 513.98, "end": 518.62, "text": " And you add this with uh like row here and you just add it up"}, {"start": 519.34, "end": 525.98, "text": " To this embedding vector and after adding it up your you end up here and then you just have two parts of the architecture"}, {"start": 526.14, "end": 528.86, "text": " So the first one is the multi-headed attention"}, {"start": 529.5, "end": 536.86, "text": " Which is basically attends to all of the tokens and creates like those nice representations when I say attains to all so it's not"}, {"start": 536.86, "end": 539.9, "text": " bi-directional as I said, they have the causal masking and"}, {"start": 539.9, "end": 544.9, "text": " Finally, we have the feedforward part where basically,"}, {"start": 546.3, "end": 549.1, "text": " yeah, you just kind of independently process"}, {"start": 549.1, "end": 551.6999999999999, "text": " those token representations again."}, {"start": 551.6999999999999, "end": 554.54, "text": " So I have a video about this one,"}, {"start": 554.54, "end": 558.3, "text": " but just wanted to do a quick recap of it."}, {"start": 558.3, "end": 560.74, "text": " And basically the only difference is they use something"}, {"start": 560.74, "end": 564.02, "text": " called a couple of ideas from the sparse transformer paper"}, {"start": 564.02, "end": 567.6999999999999, "text": " where you basically, instead of using those like causal masks,"}, {"start": 567.6999999999999, "end": 569.4399999999999, "text": " you use sparse causal masks."}, {"start": 569.44, "end": 572.1400000000001, "text": " So let's not get into too much details."}, {"start": 572.1400000000001, "end": 575.6600000000001, "text": " And yeah, that's pretty much it about the architecture."}, {"start": 576.62, "end": 580.4200000000001, "text": " Now for the training data set, they used,"}, {"start": 580.4200000000001, "end": 584.1, "text": " so they used the common crawl, which is a huge, huge data set"}, {"start": 584.1, "end": 587.9200000000001, "text": " pretty much like crawling every single month"}, {"start": 587.9200000000001, "end": 591.34, "text": " that site downloads bunch of the data from the internet."}, {"start": 591.34, "end": 593.5600000000001, "text": " And what they did is they filtered it,"}, {"start": 593.5600000000001, "end": 595.7, "text": " and they also used something called WebText,"}, {"start": 595.7, "end": 600.7, "text": " which was created by using the links from Reddit,"}, {"start": 601.74, "end": 604.26, "text": " which had at least three uploads,"}, {"start": 604.26, "end": 607.82, "text": " which was kind of heuristic for filtering"}, {"start": 607.82, "end": 609.82, "text": " like higher quality content."}, {"start": 609.82, "end": 612.82, "text": " And they just kind of like scraped all of the data"}, {"start": 612.82, "end": 616.1800000000001, "text": " that those links were pointing to."}, {"start": 616.1800000000001, "end": 619.26, "text": " They also have the books, One Books, Two and Wikipedia."}, {"start": 619.26, "end": 622.1800000000001, "text": " So basically the point is that depending on the quality"}, {"start": 622.18, "end": 626.06, "text": " of the data set, they took more samples or less samples."}, {"start": 626.06, "end": 627.66, "text": " So some of the data like Wikipedia,"}, {"start": 627.66, "end": 631.02, "text": " which is higher quality than for example, common crawl,"}, {"start": 631.02, "end": 635.38, "text": " like had like 3.4 epochs, meaning the model"}, {"start": 635.38, "end": 637.5799999999999, "text": " saw it approximately 3.4 times,"}, {"start": 637.5799999999999, "end": 642.14, "text": " whereas the common crawl was only seen 0.44 times."}, {"start": 642.14, "end": 646.06, "text": " One important thing to notice here and comment"}, {"start": 646.06, "end": 649.5, "text": " is that they had a problem with data contamination,"}, {"start": 649.5, "end": 651.8199999999999, "text": " which means many of the results,"}, {"start": 651.82, "end": 654.1, "text": " they later showed that the results"}, {"start": 654.1, "end": 655.82, "text": " are not as seriously affected."}, {"start": 655.82, "end": 658.38, "text": " Some of the results that were seriously affected"}, {"start": 658.38, "end": 662.1, "text": " were kind of just not displayed in the paper,"}, {"start": 662.1, "end": 666.3000000000001, "text": " but I still think they had some contamination issues"}, {"start": 666.3000000000001, "end": 668.6600000000001, "text": " left over, especially on the PICA data set"}, {"start": 668.6600000000001, "end": 671.1400000000001, "text": " where they are testing for understanding"}, {"start": 671.1400000000001, "end": 675.0600000000001, "text": " of the like physics in the world."}, {"start": 675.0600000000001, "end": 676.6600000000001, "text": " And we'll get to it a bit later."}, {"start": 677.8000000000001, "end": 679.46, "text": " The compute was enormous."}, {"start": 679.46, "end": 682.7, "text": " So they had like, you basically had like 10 years"}, {"start": 682.7, "end": 684.1, "text": " of petaflop second."}, {"start": 684.1, "end": 685.5, "text": " Like if you had a machine that has"}, {"start": 685.5, "end": 687.14, "text": " a petaflop second performance,"}, {"start": 687.14, "end": 688.7, "text": " you need 10 years to train this model."}, {"start": 688.7, "end": 691.7, "text": " So it's huge in that sense, you can see it here."}, {"start": 691.7, "end": 694.5400000000001, "text": " And the interesting comparison is that they say here,"}, {"start": 694.5400000000001, "end": 696.9000000000001, "text": " as a consequence, although GPT-3, 3 billion"}, {"start": 696.9000000000001, "end": 699.98, "text": " is almost 10X larger than Roberta large,"}, {"start": 699.98, "end": 703.3000000000001, "text": " both models took approximately 50 petaflop second days"}, {"start": 703.3000000000001, "end": 705.5, "text": " of compute during pre-training."}, {"start": 705.5, "end": 709.02, "text": " The reason is that there is some other paper"}, {"start": 709.02, "end": 710.54, "text": " that tells that you shouldn't,"}, {"start": 711.54, "end": 714.34, "text": " that the model shouldn't see like tokens too many times,"}, {"start": 714.34, "end": 717.38, "text": " otherwise it will overfit and the performance will get worse."}, {"start": 718.5799999999999, "end": 720.8199999999999, "text": " So that was that part about the data."}, {"start": 720.8199999999999, "end": 722.78, "text": " And yeah, one more interesting thing maybe"}, {"start": 722.78, "end": 725.78, "text": " is that basically they needed a supercomputer"}, {"start": 725.78, "end": 726.9399999999999, "text": " because the model was huge."}, {"start": 726.9399999999999, "end": 730.8199999999999, "text": " He had 175 billion, so they needed like a supercomputer"}, {"start": 730.8199999999999, "end": 733.38, "text": " from Microsoft to train the thing."}, {"start": 733.38, "end": 735.18, "text": " So that's kind of a fun fact."}, {"start": 735.18, "end": 737.5, "text": " Let me jump to something that's really like"}, {"start": 737.5, "end": 738.76, "text": " the core part of this paper."}, {"start": 738.76, "end": 742.78, "text": " And that's how they do this zero shot,"}, {"start": 742.78, "end": 746.02, "text": " like one shot and few shot learning and why they do it."}, {"start": 746.02, "end": 749.38, "text": " So basically why they do it is, so they said here,"}, {"start": 749.38, "end": 751.36, "text": " so the reason to distinguish between one shot"}, {"start": 751.36, "end": 756.04, "text": " from few shot and zero shot is that it most closely matches"}, {"start": 756.04, "end": 758.42, "text": " the way in which some of the tests"}, {"start": 758.42, "end": 760.38, "text": " are communicated to humans."}, {"start": 760.38, "end": 765.38, "text": " So yeah, basically that's how humans work."}, {"start": 765.38, "end": 768.8, "text": " That's how humans work and then for the zero shot,"}, {"start": 770.22, "end": 772.64, "text": " there are some examples like for example, translation."}, {"start": 772.64, "end": 774.88, "text": " If I told you to translate from English to German,"}, {"start": 774.88, "end": 778.14, "text": " supposing you knew both languages, you'd know what to do."}, {"start": 778.14, "end": 780.36, "text": " You wouldn't need any additional examples."}, {"start": 780.36, "end": 782.62, "text": " But in some other cases, you do need example"}, {"start": 782.62, "end": 785.26, "text": " and that's why the one shot and few shot setting."}, {"start": 785.26, "end": 789.22, "text": " So basically in some cases, it may even be difficult"}, {"start": 789.22, "end": 791.42, "text": " for humans to understand the format of the task"}, {"start": 791.42, "end": 792.5, "text": " without prior examples."}, {"start": 792.5, "end": 795.78, "text": " So this setting is in some cases unfairly hard."}, {"start": 795.78, "end": 800.02, "text": " So zero shot is going to be, always going to have"}, {"start": 800.02, "end": 803.3, "text": " like worse performance and we're going to see some data"}, {"start": 803.3, "end": 804.36, "text": " that will back that up."}, {"start": 805.86, "end": 808.14, "text": " Also, they say here, nevertheless,"}, {"start": 808.14, "end": 810.58, "text": " for at least some settings, zero shot is closest"}, {"start": 810.58, "end": 813.14, "text": " to how human perform tasks."}, {"start": 813.14, "end": 816.04, "text": " So that's the thing I mentioned about the translation."}, {"start": 816.04, "end": 820.2, "text": " Okay, having said that, let's see some examples here."}, {"start": 820.2, "end": 823.26, "text": " So basically on the translation example,"}, {"start": 823.26, "end": 825.46, "text": " this is a zero shot setting."}, {"start": 825.46, "end": 829.7800000000001, "text": " You basically say, you condition the model on this sentence."}, {"start": 829.7800000000001, "end": 833.1, "text": " So you give it translate English to French,"}, {"start": 833.1, "end": 835.4200000000001, "text": " cheese and then prompt."}, {"start": 835.4200000000001, "end": 839.1800000000001, "text": " And basically you expect the model to regressively"}, {"start": 839.1800000000001, "end": 842.4200000000001, "text": " complete the sentence and translate cheese into whatever,"}, {"start": 842.4200000000001, "end": 845.7, "text": " however you say cheese in French, I don't speak French."}, {"start": 845.7, "end": 849.38, "text": " Then there is the one shot where you give one example"}, {"start": 849.38, "end": 852.22, "text": " like sea otter, l'outre de mer, I don't know if it's"}, {"start": 852.22, "end": 854.98, "text": " pronounced like that, but let's say for the sake of it."}, {"start": 854.98, "end": 858.46, "text": " And then the few shot one where you give multiple examples"}, {"start": 858.46, "end": 861.42, "text": " and then you prompt it and expect the correct answer."}, {"start": 861.42, "end": 864.66, "text": " So depending on the tasks, some will make more sense,"}, {"start": 864.66, "end": 867.58, "text": " some will make less sense and few shot basically"}, {"start": 867.58, "end": 870.54, "text": " almost always gives better performance, which makes sense."}, {"start": 870.54, "end": 874.38, "text": " And let me compare that to fine tuning,"}, {"start": 874.38, "end": 877.86, "text": " which is basically what most of the other models"}, {"start": 877.86, "end": 881.14, "text": " in NLP so far, like BERT, et cetera,"}, {"start": 881.14, "end": 883.22, "text": " they always use to pre-train the model"}, {"start": 883.22, "end": 885.36, "text": " on a huge corpus of text."}, {"start": 885.36, "end": 888.92, "text": " And then they fine tune the model on a specific downstream"}, {"start": 888.92, "end": 893.02, "text": " like task and then that's how it pretty much works."}, {"start": 893.02, "end": 896.38, "text": " And the problem, so they did notice a couple of problems"}, {"start": 896.38, "end": 898.6800000000001, "text": " with that, there are some problems with doing that."}, {"start": 898.6800000000001, "end": 902.6, "text": " Basically you kind of overfit to a more narrow distribution,"}, {"start": 902.6, "end": 903.78, "text": " that's one problem."}, {"start": 903.78, "end": 907.82, "text": " Second problem is you need supervised like label data,"}, {"start": 907.82, "end": 909.74, "text": " which is usually expensive."}, {"start": 909.74, "end": 913.98, "text": " And the third problem is that humans like don't need"}, {"start": 913.98, "end": 917.82, "text": " like a label data, like you only need a couple of examples"}, {"start": 917.82, "end": 918.6600000000001, "text": " and that's it."}, {"start": 918.6600000000001, "end": 920.6, "text": " So that's the few shot argument again."}, {"start": 920.6, "end": 923.82, "text": " Here they say that they usually need around,"}, {"start": 923.82, "end": 927.46, "text": " they usually put around 10 to 100 examples"}, {"start": 927.46, "end": 929.82, "text": " in the few shot setting, depending on the,"}, {"start": 929.82, "end": 934.5, "text": " like the basically they're constrained by this,"}, {"start": 934.5, "end": 937.5400000000001, "text": " by the model size pretty much, by the memory actually."}, {"start": 937.54, "end": 941.5, "text": " And they have only 2048 tokens available."}, {"start": 941.5, "end": 943.5999999999999, "text": " So depending on the example size,"}, {"start": 943.5999999999999, "end": 946.8, "text": " they can go from 100 to, from 10 to 100."}, {"start": 946.8, "end": 949.4599999999999, "text": " So that was, that was that part."}, {"start": 949.4599999999999, "end": 952.18, "text": " And there are a couple of interesting charts"}, {"start": 952.18, "end": 953.04, "text": " in the beginning of the paper."}, {"start": 953.04, "end": 956.8199999999999, "text": " So this one shows us that as we are scaling,"}, {"start": 956.8199999999999, "end": 959.8399999999999, "text": " so they pretty much averaged across different benchmarks,"}, {"start": 959.8399999999999, "end": 961.8399999999999, "text": " which we'll see in a couple of minutes,"}, {"start": 961.8399999999999, "end": 963.4399999999999, "text": " they averaged the performance."}, {"start": 963.44, "end": 967.62, "text": " And you can see that as the, on the x-axis as the,"}, {"start": 967.62, "end": 969.5400000000001, "text": " as the model size grows in size,"}, {"start": 969.5400000000001, "end": 974.5400000000001, "text": " so as we slowly get to the 175 billion model version,"}, {"start": 975.2600000000001, "end": 977.82, "text": " we get the better and better performance."}, {"start": 977.82, "end": 980.7, "text": " And the second important thing to notice here"}, {"start": 980.7, "end": 982.7, "text": " is that the difference in performance"}, {"start": 982.7, "end": 986.44, "text": " between the few shot, one shot and zero shot also increases,"}, {"start": 986.44, "end": 990.1400000000001, "text": " which kinda tells us that the model is becoming better"}, {"start": 990.1400000000001, "end": 991.44, "text": " at meta-learning."}, {"start": 991.44, "end": 994.5400000000001, "text": " So those are two interesting facts to see from this plot."}, {"start": 994.5400000000001, "end": 996.7800000000001, "text": " So the first one, bigger the scale,"}, {"start": 996.7800000000001, "end": 998.2600000000001, "text": " the better the performance."}, {"start": 998.2600000000001, "end": 1001.0200000000001, "text": " And the second thing is they are getting better"}, {"start": 1001.0200000000001, "end": 1003.86, "text": " at meta-learning as the size increases,"}, {"start": 1003.86, "end": 1005.24, "text": " which we can see by the difference"}, {"start": 1005.24, "end": 1006.74, "text": " between the different performances."}, {"start": 1006.74, "end": 1011.3000000000001, "text": " So basically going from zero to one shot here,"}, {"start": 1011.3000000000001, "end": 1014.7800000000001, "text": " you get less improvement than by going from zero to one"}, {"start": 1014.7800000000001, "end": 1018.48, "text": " to few shot learning here."}, {"start": 1018.48, "end": 1019.82, "text": " So that was that chart."}, {"start": 1019.82, "end": 1021.74, "text": " Let me see what else we have here."}, {"start": 1021.74, "end": 1026.38, "text": " So here they show how the, like the number of context,"}, {"start": 1026.38, "end": 1028.3, "text": " number of examples in the context"}, {"start": 1028.3, "end": 1029.78, "text": " also affects the performance."}, {"start": 1029.78, "end": 1032.1000000000001, "text": " And they showed a couple of models."}, {"start": 1032.1000000000001, "end": 1034.78, "text": " So the 1.3 billion, the 13 billion,"}, {"start": 1034.78, "end": 1038.48, "text": " and the biggest one, the GPT-3 175 billion model."}, {"start": 1038.48, "end": 1042.78, "text": " And we can see that as we increased the number of examples,"}, {"start": 1042.78, "end": 1045.92, "text": " usually the performance also increases."}, {"start": 1045.92, "end": 1048.18, "text": " And this is for some specific task,"}, {"start": 1048.18, "end": 1052.18, "text": " like insertion, trying to unscramble the word,"}, {"start": 1052.18, "end": 1054.42, "text": " which I'll explain a bit later."}, {"start": 1054.42, "end": 1055.44, "text": " But that's the basic idea."}, {"start": 1055.44, "end": 1058.3400000000001, "text": " So we have the zero shot, we have the one shot setting,"}, {"start": 1058.3400000000001, "end": 1060.1000000000001, "text": " and we have the few shot setting,"}, {"start": 1060.1000000000001, "end": 1061.68, "text": " and the performance increases."}, {"start": 1061.68, "end": 1064.5800000000002, "text": " And the second thing is the bigger the model,"}, {"start": 1065.78, "end": 1068.8600000000001, "text": " like the steeper the improvements in a sense."}, {"start": 1070.18, "end": 1075.18, "text": " Okay, and finally, this is the last chart in this section,"}, {"start": 1075.18, "end": 1079.5800000000002, "text": " which tells us, so they treat the model in a sense"}, {"start": 1079.5800000000002, "end": 1082.26, "text": " like a meta-learning, like a,"}, {"start": 1082.26, "end": 1083.98, "text": " they treat the problem as a meta-learning problem."}, {"start": 1083.98, "end": 1087.66, "text": " So basically you pre-train the model, that's the outer loop."}, {"start": 1087.66, "end": 1091.3, "text": " So that's all of those web crawl, like common crawl data,"}, {"start": 1091.3, "end": 1093.0600000000002, "text": " web text and common crawl data."}, {"start": 1093.0600000000002, "end": 1097.88, "text": " And then once you condition the model on the,"}, {"start": 1097.88, "end": 1102.5800000000002, "text": " on the like text, like we saw in the translation example,"}, {"start": 1102.58, "end": 1106.06, "text": " you basically, that's the inner loop."}, {"start": 1106.06, "end": 1108.1399999999999, "text": " When you put the context, you have,"}, {"start": 1108.1399999999999, "end": 1111.22, "text": " you form in a sense, ephemeral weights,"}, {"start": 1111.22, "end": 1112.58, "text": " which define the model."}, {"start": 1112.58, "end": 1116.86, "text": " So the model is kinda like changes its,"}, {"start": 1116.86, "end": 1120.06, "text": " let's call it shape, depending on the conditioning text."}, {"start": 1120.06, "end": 1122.04, "text": " So you have a completely different model,"}, {"start": 1122.04, "end": 1125.06, "text": " which is in a sense fine-tuned to the new,"}, {"start": 1125.06, "end": 1127.58, "text": " adapts to the new task, which is the interesting thing."}, {"start": 1127.58, "end": 1130.58, "text": " Before I jump to the results section,"}, {"start": 1130.58, "end": 1133.6599999999999, "text": " and show you how it performs on different benchmarks,"}, {"start": 1133.6599999999999, "end": 1136.1, "text": " let me show you one more really important chart."}, {"start": 1136.1, "end": 1140.22, "text": " So this chart shows us that the validation loss"}, {"start": 1141.6399999999999, "end": 1145.98, "text": " decreases as the compute increases."}, {"start": 1145.98, "end": 1149.1, "text": " And the x-axis is the log scale,"}, {"start": 1149.1, "end": 1151.62, "text": " and you can basically see, so GPT-3,"}, {"start": 1151.62, "end": 1154.54, "text": " which had this many parameters,"}, {"start": 1154.54, "end": 1158.62, "text": " so that's approximately 175 billion params,"}, {"start": 1158.62, "end": 1161.08, "text": " and had 3,000 days of compute."}, {"start": 1161.08, "end": 1163.06, "text": " So this is the GPT-3 model."}, {"start": 1163.06, "end": 1166.1399999999999, "text": " You can see as the compute increases,"}, {"start": 1166.1399999999999, "end": 1171.1399999999999, "text": " the loss decreases, and it follows the power law here."}, {"start": 1172.7399999999998, "end": 1175.34, "text": " So basically smaller models with less compute"}, {"start": 1176.34, "end": 1179.9399999999998, "text": " saturate at higher validation loss,"}, {"start": 1179.9399999999998, "end": 1182.54, "text": " which is an interesting, which was actually"}, {"start": 1183.4599999999998, "end": 1185.86, "text": " hypothesized by some of the previous papers,"}, {"start": 1185.86, "end": 1189.4199999999998, "text": " and this paper showed that the loss still holds"}, {"start": 1189.4199999999998, "end": 1193.4199999999998, "text": " even for huge, huge models such as the GPT-3."}, {"start": 1193.4199999999998, "end": 1197.6799999999998, "text": " So that was all I had to tell you"}, {"start": 1197.6799999999998, "end": 1200.1, "text": " before we jumped to the results section."}, {"start": 1200.1, "end": 1205.1, "text": " So pretty much nothing new in the research sense."}, {"start": 1205.5, "end": 1209.6, "text": " They pretty much like sweep the whole combinatorial space,"}, {"start": 1209.6, "end": 1211.24, "text": " tried a bunch of different models,"}, {"start": 1211.24, "end": 1212.6, "text": " saw how it generalizes."}, {"start": 1212.6, "end": 1214.6599999999999, "text": " So the nice thing is that they are pushing"}, {"start": 1214.66, "end": 1219.1000000000001, "text": " for this few shot and zero shot and one shot"}, {"start": 1219.1000000000001, "end": 1222.74, "text": " like key evaluation and not for the fine tuning approach,"}, {"start": 1222.74, "end": 1225.78, "text": " which just gives you better results in the benchmark,"}, {"start": 1225.78, "end": 1230.3400000000001, "text": " but is not as applicable as having a general model"}, {"start": 1230.3400000000001, "end": 1234.8200000000002, "text": " which you can later apply on the fly, which is really cool."}, {"start": 1234.8200000000002, "end": 1235.98, "text": " So the paper is really huge,"}, {"start": 1235.98, "end": 1237.26, "text": " so I'm going to just show you"}, {"start": 1237.26, "end": 1239.68, "text": " a subset of interesting results."}, {"start": 1239.68, "end": 1242.0, "text": " And yeah, so the first one will be,"}, {"start": 1242.98, "end": 1244.38, "text": " I'll skip Lombetta."}, {"start": 1244.38, "end": 1246.0200000000002, "text": " Those are some language modeling tasks,"}, {"start": 1246.0200000000002, "end": 1247.9, "text": " and as expected because the model was,"}, {"start": 1247.9, "end": 1250.3000000000002, "text": " GPT-3 was pre-trained as a language model,"}, {"start": 1250.3000000000002, "end": 1253.1000000000001, "text": " i.e. predicting the next word in the sentence."}, {"start": 1253.1000000000001, "end": 1256.3400000000001, "text": " It performs pretty good on those tasks."}, {"start": 1256.3400000000001, "end": 1258.7800000000002, "text": " I'll skip this and I'll jump to translation."}, {"start": 1258.7800000000002, "end": 1260.3400000000001, "text": " Let me just find the curves."}, {"start": 1260.3400000000001, "end": 1262.0600000000002, "text": " Whoop, just a sec, this one."}, {"start": 1263.3400000000001, "end": 1267.14, "text": " What's interesting is that basically"}, {"start": 1268.46, "end": 1271.3200000000002, "text": " the model was not explicitly trained to do translation."}, {"start": 1271.3200000000002, "end": 1274.2600000000002, "text": " And that's one of the funny things about this GPT-3 model."}, {"start": 1274.26, "end": 1276.98, "text": " So basically it was just trained to predict the next word,"}, {"start": 1276.98, "end": 1279.66, "text": " but what emerges is a set of skills"}, {"start": 1279.66, "end": 1282.9, "text": " which seem to be useful to know"}, {"start": 1282.9, "end": 1284.44, "text": " in order to predict those words."}, {"start": 1284.44, "end": 1287.06, "text": " And one of those skills is just like translation."}, {"start": 1287.06, "end": 1292.06, "text": " And I'm pretty surprised by the BLEO score it achieved."}, {"start": 1292.26, "end": 1294.94, "text": " So I previously reconstructed"}, {"start": 1294.94, "end": 1296.42, "text": " the original transformer paper,"}, {"start": 1296.42, "end": 1300.66, "text": " and I'll link the project in the description."}, {"start": 1300.66, "end": 1304.18, "text": " And basically I think I achieved around 33 BLEO score"}, {"start": 1304.18, "end": 1307.02, "text": " for English to German."}, {"start": 1307.02, "end": 1311.3400000000001, "text": " And here without being explicitly trained to do translation"}, {"start": 1311.3400000000001, "end": 1316.3400000000001, "text": " and just having a small subset of the text in German,"}, {"start": 1316.9, "end": 1319.98, "text": " it achieved a really, really decent score."}, {"start": 1319.98, "end": 1324.6200000000001, "text": " And interesting thing is that going from French to English,"}, {"start": 1324.6200000000001, "end": 1326.92, "text": " German to English, and Romanian to English,"}, {"start": 1326.92, "end": 1328.5800000000002, "text": " the performance is really good."}, {"start": 1328.5800000000002, "end": 1331.0800000000002, "text": " But once you go in the opposite direction,"}, {"start": 1331.08, "end": 1336.08, "text": " it's really hard to achieve a good result."}, {"start": 1336.3, "end": 1340.54, "text": " So this, the Romanian really has some serious problems."}, {"start": 1340.54, "end": 1343.5, "text": " It's around 20, which is really, really low."}, {"start": 1343.5, "end": 1344.34, "text": " That's interesting."}, {"start": 1344.34, "end": 1345.1799999999998, "text": " And that shows that this model"}, {"start": 1345.1799999999998, "end": 1347.4199999999998, "text": " is a good English language model,"}, {"start": 1347.4199999999998, "end": 1350.78, "text": " but not as good for other languages."}, {"start": 1352.1399999999999, "end": 1357.1399999999999, "text": " So they noticed, so as I mentioned,"}, {"start": 1357.5, "end": 1358.82, "text": " they noticed this skew,"}, {"start": 1358.82, "end": 1360.6599999999999, "text": " and they said that this could be a weakness"}, {"start": 1360.66, "end": 1364.5600000000002, "text": " due to reusing the bad level BPE tokenizer of GPT-2,"}, {"start": 1364.5600000000002, "end": 1366.66, "text": " which was developed for an almost entirely"}, {"start": 1366.66, "end": 1368.1000000000001, "text": " English training data set."}, {"start": 1368.1000000000001, "end": 1372.9, "text": " So BPE, the by pair encoding was,"}, {"start": 1372.9, "end": 1374.5400000000002, "text": " many people pointed out that BPE"}, {"start": 1374.5400000000002, "end": 1376.24, "text": " can be causing different problems."}, {"start": 1376.24, "end": 1378.6200000000001, "text": " We'll see some problems that it's causing"}, {"start": 1378.6200000000001, "end": 1381.78, "text": " in one of the tasks I'm going to show you"}, {"start": 1381.78, "end": 1383.38, "text": " about unscrambling words."}, {"start": 1383.38, "end": 1387.0400000000002, "text": " And yeah, just keep that in mind about the BPE."}, {"start": 1387.94, "end": 1390.3200000000002, "text": " The next interesting tasks I wanna show you"}, {"start": 1390.32, "end": 1392.46, "text": " is the natural language inference,"}, {"start": 1392.46, "end": 1395.78, "text": " where the model is having some problems doing those."}, {"start": 1395.78, "end": 1400.78, "text": " So the NLI tasks, and we can see here, basically,"}, {"start": 1401.06, "end": 1403.9399999999998, "text": " so the task there is to read the text"}, {"start": 1403.9399999999998, "end": 1408.5, "text": " and understand how sentences relate to each other."}, {"start": 1408.5, "end": 1410.34, "text": " For example, you give it one sentence"}, {"start": 1410.34, "end": 1411.5, "text": " and the second sentence,"}, {"start": 1411.5, "end": 1414.5, "text": " and you ask it to say whether the second sentence"}, {"start": 1414.5, "end": 1415.8999999999999, "text": " follows from the first,"}, {"start": 1415.8999999999999, "end": 1417.86, "text": " whether it contradicts the first sentence,"}, {"start": 1417.86, "end": 1419.6399999999999, "text": " or whether they're just neutral."}, {"start": 1419.64, "end": 1423.7, "text": " And basically, that's really interesting"}, {"start": 1423.7, "end": 1425.26, "text": " because that means that the model"}, {"start": 1425.26, "end": 1428.9, "text": " doesn't handle comprehension like in reasoning,"}, {"start": 1428.9, "end": 1433.44, "text": " as well as it does just like generating sequences"}, {"start": 1433.44, "end": 1436.26, "text": " like translation or language modeling"}, {"start": 1436.26, "end": 1438.18, "text": " or like completing sentences."}, {"start": 1438.18, "end": 1441.14, "text": " So different tasks that have to do with generation."}, {"start": 1443.26, "end": 1445.5600000000002, "text": " And we can see the results here"}, {"start": 1445.5600000000002, "end": 1448.3000000000002, "text": " that we have pretty much random behavior"}, {"start": 1448.3, "end": 1451.58, "text": " on smaller models, and then as the model grows,"}, {"start": 1451.58, "end": 1456.22, "text": " the few shot model actually improves the performance,"}, {"start": 1456.22, "end": 1459.06, "text": " but it's still way below the baselines,"}, {"start": 1459.06, "end": 1461.8999999999999, "text": " like even BERT and Roberta is even better."}, {"start": 1462.86, "end": 1464.3, "text": " So those are some of the tasks"}, {"start": 1464.3, "end": 1467.3, "text": " that this model is having problems with."}, {"start": 1467.3, "end": 1470.1, "text": " So A stands for adversarial."}, {"start": 1470.1, "end": 1473.68, "text": " So there's actually a subset of those NLI examples"}, {"start": 1473.68, "end": 1476.52, "text": " where humans picked some examples"}, {"start": 1476.52, "end": 1479.46, "text": " which are especially hard for language models."}, {"start": 1479.46, "end": 1484.1, "text": " And yeah, so it's struggling with those types of problems."}, {"start": 1484.1, "end": 1488.58, "text": " Now my favorite ones are these synthetic tasks."}, {"start": 1488.58, "end": 1493.58, "text": " So what they did is they tried and prompt the model"}, {"start": 1493.98, "end": 1497.74, "text": " whether it knows how to do some basic addition,"}, {"start": 1497.74, "end": 1500.02, "text": " subtraction, and multiplication,"}, {"start": 1500.02, "end": 1502.34, "text": " and here we can see results."}, {"start": 1502.34, "end": 1507.34, "text": " So basically for two digit addition and subtraction,"}, {"start": 1507.54, "end": 1509.54, "text": " the model has really good performance,"}, {"start": 1509.54, "end": 1511.06, "text": " at least the biggest one,"}, {"start": 1511.06, "end": 1512.9399999999998, "text": " and we can see like huge improvements"}, {"start": 1512.9399999999998, "end": 1517.4199999999998, "text": " as the model size increases again."}, {"start": 1517.4199999999998, "end": 1521.1799999999998, "text": " But then when we go to three digit addition,"}, {"start": 1521.1799999999998, "end": 1525.06, "text": " perf goes down and it goes even further down"}, {"start": 1525.06, "end": 1526.8999999999999, "text": " for four digit addition and subtraction,"}, {"start": 1526.9, "end": 1531.9, "text": " for multiplication, and for some three digit operations."}, {"start": 1534.02, "end": 1538.1000000000001, "text": " So what is probably causing this is that"}, {"start": 1538.1000000000001, "end": 1542.14, "text": " like in the huge stacks that the model has seen"}, {"start": 1542.14, "end": 1543.98, "text": " during the pre-training,"}, {"start": 1543.98, "end": 1547.6200000000001, "text": " there were a lot of tables that had like two digit numbers,"}, {"start": 1547.6200000000001, "end": 1549.5600000000002, "text": " but less so for three digit numbers,"}, {"start": 1549.5600000000002, "end": 1551.98, "text": " and even less so for four and five, et cetera."}, {"start": 1551.98, "end": 1555.8200000000002, "text": " So basically this may be indicative"}, {"start": 1555.82, "end": 1558.6599999999999, "text": " that the model is not learning how to reason"}, {"start": 1558.6599999999999, "end": 1562.8, "text": " and actually doesn't adapt in a meta-learning sense"}, {"start": 1562.8, "end": 1565.26, "text": " to this new task and learns it from a few examples,"}, {"start": 1565.26, "end": 1569.78, "text": " but it's just doing some kind of a statistical"}, {"start": 1569.78, "end": 1572.58, "text": " like pattern matching and stuff."}, {"start": 1572.58, "end": 1576.3, "text": " So it's still unexplored what exactly happens"}, {"start": 1576.3, "end": 1577.36, "text": " in the few shot setting,"}, {"start": 1577.36, "end": 1580.5, "text": " but like a probably good hypothesis is that"}, {"start": 1580.5, "end": 1583.1399999999999, "text": " it's not reasoning in the sense we humans are reasoning."}, {"start": 1583.14, "end": 1586.6200000000001, "text": " Okay, so that was an interesting example."}, {"start": 1586.6200000000001, "end": 1589.8200000000002, "text": " So one more really interesting task for me was this"}, {"start": 1589.8200000000002, "end": 1591.5400000000002, "text": " word scrambling and manipulation task."}, {"start": 1591.5400000000002, "end": 1593.9, "text": " So they had five sub-tasks here."}, {"start": 1593.9, "end": 1595.9, "text": " So the first one was cycle letters in words."}, {"start": 1595.9, "end": 1598.7800000000002, "text": " So basically you give the model something like this"}, {"start": 1598.7800000000002, "end": 1600.8600000000001, "text": " and the model is supposed to figure out"}, {"start": 1600.8600000000001, "end": 1605.0600000000002, "text": " that this is actually just a circularly permuted."}, {"start": 1605.0600000000002, "end": 1608.3600000000001, "text": " So you don't scramble it into inevitably here."}, {"start": 1608.3600000000001, "end": 1611.0600000000002, "text": " Then we have anagrams where you keep the first"}, {"start": 1611.06, "end": 1614.3, "text": " and the last letter and you scramble everything in between"}, {"start": 1614.3, "end": 1615.62, "text": " and that's what you give to the model"}, {"start": 1615.62, "end": 1618.34, "text": " and you expect the model to unscramble the word."}, {"start": 1618.34, "end": 1619.78, "text": " And this is actually super tough."}, {"start": 1619.78, "end": 1623.26, "text": " I had problems also unscrambling this word."}, {"start": 1623.26, "end": 1625.5, "text": " So I thought it had something to do with crypto,"}, {"start": 1625.5, "end": 1627.06, "text": " but then yeah, it doesn't have any Y,"}, {"start": 1627.06, "end": 1628.7, "text": " but it's still, it's kind of hard."}, {"start": 1628.7, "end": 1630.6399999999999, "text": " And then this one is a bit easier."}, {"start": 1630.6399999999999, "end": 1632.98, "text": " You hold the first two letters fixed"}, {"start": 1632.98, "end": 1635.86, "text": " and the last two letters and you just unscramble the middle"}, {"start": 1635.86, "end": 1636.96, "text": " and you get opponent."}, {"start": 1636.96, "end": 1639.1, "text": " Then they have a random insertion in the word"}, {"start": 1639.1, "end": 1641.54, "text": " where you basically just after every single character,"}, {"start": 1641.54, "end": 1644.98, "text": " you insert a random punctuation or like a blank space"}, {"start": 1644.98, "end": 1648.1799999999998, "text": " and you expect again, the model to unscramble it."}, {"start": 1648.1799999999998, "end": 1651.02, "text": " And finally, we have just the reverse word."}, {"start": 1651.02, "end": 1653.3, "text": " So but just the model should figure out"}, {"start": 1653.3, "end": 1655.86, "text": " that you should just reverse the letters"}, {"start": 1655.86, "end": 1658.62, "text": " and get objects here."}, {"start": 1658.62, "end": 1662.82, "text": " So those were the tasks and a bit surprisingly,"}, {"start": 1662.82, "end": 1667.82, "text": " so even the biggest model, the GPT-3, the 175 billion one"}, {"start": 1672.1, "end": 1676.26, "text": " had zero like accuracy on the reverse words."}, {"start": 1676.26, "end": 1678.5, "text": " So that's the first intuitive thought I had"}, {"start": 1678.5, "end": 1680.1799999999998, "text": " like what's happening here."}, {"start": 1680.1799999999998, "end": 1681.9399999999998, "text": " And then you figure out that the model"}, {"start": 1681.9399999999998, "end": 1684.58, "text": " is using sub word tokenization."}, {"start": 1684.58, "end": 1688.22, "text": " So it's really hard for it to do this task."}, {"start": 1688.22, "end": 1691.0, "text": " Whereas the easiest one was random insertion,"}, {"start": 1691.0, "end": 1693.42, "text": " which was also easiest for me actually."}, {"start": 1693.42, "end": 1696.22, "text": " And then you can see the anagram,"}, {"start": 1696.22, "end": 1699.9, "text": " the one word anagram is having problems."}, {"start": 1699.9, "end": 1703.46, "text": " So that mirrors my difficulties with understanding"}, {"start": 1703.46, "end": 1706.66, "text": " with doing this A1 task as well."}, {"start": 1706.66, "end": 1710.92, "text": " And then the A2 anagram task was a bit easier."}, {"start": 1710.92, "end": 1713.34, "text": " So that's the one where we keep the first two"}, {"start": 1713.34, "end": 1715.22, "text": " and last two letters fixed."}, {"start": 1715.22, "end": 1717.94, "text": " So that's actually all intuitive."}, {"start": 1717.94, "end": 1722.02, "text": " So the same difficulties I had solving these problems,"}, {"start": 1722.02, "end": 1724.8600000000001, "text": " the model had similar difficulties."}, {"start": 1724.8600000000001, "end": 1726.3600000000001, "text": " There are two more interesting tasks"}, {"start": 1726.3600000000001, "end": 1730.02, "text": " I wanna kind of explore a bit more in depth."}, {"start": 1730.02, "end": 1732.3400000000001, "text": " So the first one is the set analogies."}, {"start": 1732.3400000000001, "end": 1737.14, "text": " And here you can see what the problem looks like."}, {"start": 1737.14, "end": 1739.6200000000001, "text": " So this is in the appendix part of the paper."}, {"start": 1739.6200000000001, "end": 1743.6200000000001, "text": " Basically the context is a law is to trust as,"}, {"start": 1743.6200000000001, "end": 1747.16, "text": " and then you have a multiple like answers here,"}, {"start": 1747.16, "end": 1749.98, "text": " multiple choices, and you are supposed to pick out"}, {"start": 1749.98, "end": 1750.8600000000001, "text": " the correct one."}, {"start": 1750.8600000000001, "end": 1752.22, "text": " And I don't know about you,"}, {"start": 1752.22, "end": 1754.02, "text": " like English is not my native language."}, {"start": 1754.02, "end": 1756.0600000000002, "text": " And I think I'm pretty good at it."}, {"start": 1756.0600000000002, "end": 1759.88, "text": " But like some of these words like Kajoli or Belk"}, {"start": 1759.88, "end": 1761.1000000000001, "text": " were not familiar to me."}, {"start": 1761.1000000000001, "end": 1762.7, "text": " So this problem actually turned out"}, {"start": 1762.7, "end": 1764.5, "text": " to be really difficult even for me."}, {"start": 1764.5, "end": 1768.5800000000002, "text": " So I was just thinking how we are like testing our models"}, {"start": 1768.5800000000002, "end": 1770.26, "text": " really like really heavy."}, {"start": 1770.26, "end": 1772.4, "text": " Even the humans are not like,"}, {"start": 1773.5, "end": 1775.98, "text": " not nearly as smart as we think they are."}, {"start": 1775.98, "end": 1778.6, "text": " So I mean, yeah, that's just something that struck me"}, {"start": 1778.6, "end": 1779.8, "text": " and was interesting."}, {"start": 1779.8, "end": 1784.8, "text": " So let me go and see how the model is actually performing"}, {"start": 1784.9, "end": 1787.06, "text": " on the set problem."}, {"start": 1787.06, "end": 1789.34, "text": " So on the set analogy problem."}, {"start": 1790.94, "end": 1794.58, "text": " So what was interesting for me was that actually"}, {"start": 1794.58, "end": 1796.46, "text": " GPT-3 achieved higher score"}, {"start": 1796.46, "end": 1799.54, "text": " than average high school student."}, {"start": 1799.54, "end": 1804.54, "text": " So on this task GPT-3 achieved 65.2 in the few shot setting,"}, {"start": 1804.54, "end": 1809.54, "text": " whereas the average score among college applicants was 57%."}, {"start": 1810.1399999999999, "end": 1814.58, "text": " And we can see the curves here again,"}, {"start": 1814.58, "end": 1817.22, "text": " the few shot one is growing steadily"}, {"start": 1817.22, "end": 1822.22, "text": " and is always the best for the GPT-3 model."}, {"start": 1822.5, "end": 1826.1399999999999, "text": " So one thing that we can maybe conclude from this result"}, {"start": 1826.1399999999999, "end": 1827.98, "text": " is that we shouldn't be teaching our kids"}, {"start": 1827.98, "end": 1831.26, "text": " to just do these tasks that are really easy"}, {"start": 1831.26, "end": 1834.3, "text": " for like big neural networks to do."}, {"start": 1834.3, "end": 1836.6599999999999, "text": " They should be focusing more like on things"}, {"start": 1836.6599999999999, "end": 1838.82, "text": " that the networks can do,"}, {"start": 1838.82, "end": 1842.26, "text": " like reasoning and maybe art or stuff."}, {"start": 1842.26, "end": 1843.94, "text": " But even art is something that's already"}, {"start": 1843.94, "end": 1845.6599999999999, "text": " in the realm of neural networks."}, {"start": 1845.6599999999999, "end": 1848.58, "text": " So I guess that's going to be a complicated question"}, {"start": 1848.58, "end": 1851.4199999999998, "text": " that we have to solve, but let me not digress too much."}, {"start": 1852.58, "end": 1854.56, "text": " The final problem that I wanted to talk about"}, {"start": 1854.56, "end": 1856.82, "text": " is the news article generation."}, {"start": 1856.82, "end": 1860.54, "text": " And the results here are really striking."}, {"start": 1860.54, "end": 1864.46, "text": " Basically, if I show you the plot here,"}, {"start": 1864.46, "end": 1866.98, "text": " as the model size increases,"}, {"start": 1866.98, "end": 1871.1, "text": " so here on the X dimension, on the X axis,"}, {"start": 1871.1, "end": 1873.46, "text": " you can see that the accuracy goes down,"}, {"start": 1873.46, "end": 1876.6599999999999, "text": " meaning that people are less and less confident"}, {"start": 1876.6599999999999, "end": 1881.6599999999999, "text": " that an article, whether the text came from the model"}, {"start": 1881.6599999999999, "end": 1884.1599999999999, "text": " or whether it came from humans."}, {"start": 1884.1599999999999, "end": 1887.56, "text": " So they did a really nice study"}, {"start": 1887.56, "end": 1890.46, "text": " where they tested a bunch of models,"}, {"start": 1890.46, "end": 1892.66, "text": " also they used something called control model,"}, {"start": 1892.66, "end": 1895.3400000000001, "text": " which was a smaller GPT-3 version,"}, {"start": 1895.3400000000001, "end": 1899.64, "text": " and it intentionally had like a higher temperature,"}, {"start": 1899.64, "end": 1902.38, "text": " which increased the randomness of the softmax."}, {"start": 1902.38, "end": 1904.74, "text": " So basically, it was more random"}, {"start": 1904.74, "end": 1906.58, "text": " than all of the other models."}, {"start": 1906.58, "end": 1908.38, "text": " And they compared all of those models"}, {"start": 1908.38, "end": 1909.5, "text": " with the control model."}, {"start": 1909.5, "end": 1912.46, "text": " And what it turns out is that as we go,"}, {"start": 1912.46, "end": 1917.46, "text": " as the models increase, we end up pretty much"}, {"start": 1917.46, "end": 1920.52, "text": " with the fact that people can decide"}, {"start": 1920.52, "end": 1924.8, "text": " whether an article came from human or from the GPT-3,"}, {"start": 1924.8, "end": 1929.56, "text": " which is amazing and has some probably scary repercussions."}, {"start": 1929.56, "end": 1931.68, "text": " So those were some interesting results"}, {"start": 1931.68, "end": 1934.8, "text": " that they showed in this huge paper."}, {"start": 1934.8, "end": 1936.3600000000001, "text": " This video's already getting too long,"}, {"start": 1936.3600000000001, "end": 1939.24, "text": " so I'm going to tell you a little bit more"}, {"start": 1939.24, "end": 1942.92, "text": " about two or three more things."}, {"start": 1942.92, "end": 1945.16, "text": " So one is data contamination."}, {"start": 1945.16, "end": 1947.88, "text": " So what happened is that because the training dataset"}, {"start": 1947.88, "end": 1951.6000000000001, "text": " they have is so huge, they ended up having"}, {"start": 1951.6000000000001, "end": 1955.0800000000002, "text": " some of the dev and test data from the benchmarks"}, {"start": 1955.0800000000002, "end": 1957.6000000000001, "text": " already being included in the training dataset."}, {"start": 1957.6000000000001, "end": 1961.88, "text": " And then they did some investigations,"}, {"start": 1961.88, "end": 1966.76, "text": " and for the most part, they said that the tests"}, {"start": 1966.76, "end": 1970.64, "text": " weren't as affected as they initially expected"}, {"start": 1970.64, "end": 1973.64, "text": " because there was a bug in their filtering of the data,"}, {"start": 1973.64, "end": 1975.0800000000002, "text": " which caused all of this."}, {"start": 1975.0800000000002, "end": 1978.48, "text": " So yeah, but some of the tests like PICA,"}, {"start": 1978.48, "end": 1983.48, "text": " where they had physical, let me see if I can find it."}, {"start": 1984.0, "end": 1986.92, "text": " So this is one example from PICA data sets."}, {"start": 1986.92, "end": 1990.3200000000002, "text": " So basically the context is how to apply sealant to wood,"}, {"start": 1990.3200000000002, "end": 1992.24, "text": " and then we have the correct answer,"}, {"start": 1992.24, "end": 1994.3600000000001, "text": " using a brush, brush on sealant onto wood"}, {"start": 1994.3600000000001, "end": 1996.72, "text": " until it is fully saturated with the sealant,"}, {"start": 1996.72, "end": 1998.68, "text": " and then they have the incorrect one."}, {"start": 1998.68, "end": 2003.68, "text": " So as they showed, the results on PICA were really good,"}, {"start": 2004.3200000000002, "end": 2007.44, "text": " and because it looks like it's an outlier,"}, {"start": 2007.44, "end": 2010.9, "text": " I do think that contamination played its role here."}, {"start": 2010.9, "end": 2013.3200000000002, "text": " Okay, getting back to this chart,"}, {"start": 2013.3200000000002, "end": 2015.3600000000001, "text": " I just wanna note one more thing."}, {"start": 2015.3600000000001, "end": 2018.52, "text": " And so they had a bug, so they said here,"}, {"start": 2018.52, "end": 2022.0800000000002, "text": " unfortunately a bug resulted in only partial removal"}, {"start": 2022.0800000000002, "end": 2024.94, "text": " of all detected overlaps from the training data."}, {"start": 2024.94, "end": 2026.2, "text": " Due to the cost of training,"}, {"start": 2026.2, "end": 2028.76, "text": " it wasn't feasible to retrain the model."}, {"start": 2028.76, "end": 2033.2, "text": " So even I think that they're using some kind of like,"}, {"start": 2033.2, "end": 2037.16, "text": " they're using 13 grams to figure out the overlaps,"}, {"start": 2037.16, "end": 2039.26, "text": " and they didn't mention somewhere in the paper"}, {"start": 2039.26, "end": 2041.68, "text": " that this is still a new area of research."}, {"start": 2041.68, "end": 2044.3400000000001, "text": " So I'm really skeptical about this part,"}, {"start": 2044.3400000000001, "end": 2049.34, "text": " how they are, and what do they consider by duplicate."}, {"start": 2049.92, "end": 2052.8, "text": " So it's hard to figure out which parts of text"}, {"start": 2052.8, "end": 2055.7200000000003, "text": " you should filter out so that you can say with certainty,"}, {"start": 2055.72, "end": 2059.3599999999997, "text": " okay, these examples from the dev and test sets"}, {"start": 2059.3599999999997, "end": 2063.3599999999997, "text": " are not present in any way in the training sets"}, {"start": 2063.3599999999997, "end": 2065.68, "text": " in a way that could bias and help the model"}, {"start": 2065.68, "end": 2069.2799999999997, "text": " to better predict on those dev and test sets."}, {"start": 2069.2799999999997, "end": 2073.9599999999996, "text": " So I mentioned PICA, we can see it's an outlier,"}, {"start": 2073.9599999999996, "end": 2076.12, "text": " so it has better performance."}, {"start": 2076.12, "end": 2079.64, "text": " You can see here that the points"}, {"start": 2079.64, "end": 2083.8399999999997, "text": " on the lower part of this chart have better performance"}, {"start": 2083.84, "end": 2087.8, "text": " on the dirty data sets,"}, {"start": 2087.8, "end": 2092.8, "text": " and so that means that there was a contamination issue."}, {"start": 2093.1200000000003, "end": 2095.36, "text": " What I'm not clear about is this drop."}, {"start": 2095.36, "end": 2099.32, "text": " They did mention it, but it looks like a huge outlier,"}, {"start": 2101.1600000000003, "end": 2104.0, "text": " but they didn't mention anything specifically about it."}, {"start": 2104.0, "end": 2107.44, "text": " They did mention PICA and VinaGrad, et cetera."}, {"start": 2107.44, "end": 2109.92, "text": " So that was it about this part."}, {"start": 2109.92, "end": 2111.76, "text": " Now I wanna just walk you through"}, {"start": 2111.76, "end": 2113.6000000000004, "text": " some of the limitations they mentioned."}, {"start": 2113.6, "end": 2115.12, "text": " So the first one is this one."}, {"start": 2115.12, "end": 2118.44, "text": " So GPT-3 samples still sometimes repeat themselves"}, {"start": 2118.44, "end": 2120.36, "text": " semantically at the document level,"}, {"start": 2120.36, "end": 2123.7599999999998, "text": " start to lose coherence over sufficiently long passages,"}, {"start": 2123.7599999999998, "end": 2126.48, "text": " contradict themselves, and occasionally contain"}, {"start": 2126.48, "end": 2129.48, "text": " non-sequitur sentences or paragraphs."}, {"start": 2129.48, "end": 2132.56, "text": " So even though this is a huge language model,"}, {"start": 2132.56, "end": 2137.56, "text": " it still has its own problems, like repeating."}, {"start": 2137.6, "end": 2142.44, "text": " So there's still a lot of research to do"}, {"start": 2142.44, "end": 2145.6, "text": " about how do we decode the output from these models"}, {"start": 2145.6, "end": 2147.76, "text": " and whether the problem is the model itself"}, {"start": 2147.76, "end": 2150.58, "text": " or the heuristics we are using for decoding."}, {"start": 2150.58, "end": 2155.58, "text": " So basically, Gvern and most people are using the top P,"}, {"start": 2155.88, "end": 2159.36, "text": " the nucleus decoding, and they're playing with temperatures,"}, {"start": 2159.36, "end": 2162.6, "text": " so it still doesn't feel like"}, {"start": 2162.6, "end": 2164.12, "text": " as a correct way to do things."}, {"start": 2165.88, "end": 2169.2000000000003, "text": " They then mentioned that they are obviously aware"}, {"start": 2169.2000000000003, "end": 2172.2000000000003, "text": " that if they had bi-directional representations"}, {"start": 2172.2, "end": 2175.68, "text": " compared, contrasted to the ones that they do have,"}, {"start": 2175.68, "end": 2179.8399999999997, "text": " and that's the causal, the unidirectional representations,"}, {"start": 2179.8399999999997, "end": 2181.7999999999997, "text": " they expect better results."}, {"start": 2181.7999999999997, "end": 2184.7999999999997, "text": " So that's a thing they could try out."}, {"start": 2184.7999999999997, "end": 2187.72, "text": " They also acknowledge that the pre-training objective"}, {"start": 2187.72, "end": 2190.8799999999997, "text": " could be further improved, and they also acknowledge"}, {"start": 2190.8799999999997, "end": 2194.3599999999997, "text": " that the understanding precisely how few-shot learning works"}, {"start": 2194.3599999999997, "end": 2197.72, "text": " is an important unexplored direction for future research."}, {"start": 2197.72, "end": 2201.0, "text": " So yeah, basically nobody quite knows"}, {"start": 2201.0, "end": 2202.48, "text": " how this exactly works."}, {"start": 2202.48, "end": 2205.3, "text": " It's still kind of not easy to interpret,"}, {"start": 2205.3, "end": 2206.8, "text": " and that's one of the bad things."}, {"start": 2206.8, "end": 2208.12, "text": " So they mentioned it here."}, {"start": 2208.12, "end": 2211.8, "text": " Its decisions are not easily interpretable."}, {"start": 2211.8, "end": 2215.4, "text": " Also, the pure scale of the model"}, {"start": 2217.12, "end": 2220.2, "text": " doesn't allow them to iterate when they make mistakes,"}, {"start": 2220.2, "end": 2223.32, "text": " as we saw on the example with data contamination."}, {"start": 2223.32, "end": 2228.12, "text": " So a limitation associated with models at the scale of GPT-3"}, {"start": 2228.12, "end": 2230.68, "text": " regardless of objective function or algorithm"}, {"start": 2230.68, "end": 2233.68, "text": " is that they are both expensive and inconvenient"}, {"start": 2233.68, "end": 2236.7599999999998, "text": " to perform inference on, which may present a challenge"}, {"start": 2236.7599999999998, "end": 2239.2, "text": " for practical applicability of models of this scale"}, {"start": 2239.2, "end": 2240.3599999999997, "text": " in their current form."}, {"start": 2242.3399999999997, "end": 2246.2, "text": " So I guess some sorts of knowledge distillations"}, {"start": 2246.2, "end": 2251.2, "text": " and pruning, et cetera, could help us make smaller,"}, {"start": 2253.12, "end": 2255.7599999999998, "text": " yet performant models."}, {"start": 2255.7599999999998, "end": 2257.7599999999998, "text": " I wanna close up this video talking about"}, {"start": 2257.7599999999998, "end": 2258.68, "text": " the broader impact."}, {"start": 2258.68, "end": 2260.2, "text": " So I do think this is important,"}, {"start": 2260.2, "end": 2262.74, "text": " even though it's not like a technical part,"}, {"start": 2262.74, "end": 2265.3199999999997, "text": " so feel free to skip it if you're not interested in it."}, {"start": 2265.3199999999997, "end": 2268.16, "text": " So basically what they consider,"}, {"start": 2268.16, "end": 2270.7599999999998, "text": " so with GPT-2, they had a staged release."}, {"start": 2270.7599999999998, "end": 2272.24, "text": " So that happened last year,"}, {"start": 2272.24, "end": 2274.98, "text": " and they initially released the smallest GPT-2 model."}, {"start": 2274.98, "end": 2279.04, "text": " They were monitoring different forums, et cetera,"}, {"start": 2279.04, "end": 2282.96, "text": " and just looking for any signs of potential misuse."}, {"start": 2282.96, "end": 2284.52, "text": " Once they were pretty certain"}, {"start": 2284.52, "end": 2287.7599999999998, "text": " that no misuse was happening,"}, {"start": 2287.7599999999998, "end": 2289.96, "text": " they started gradually releasing the models,"}, {"start": 2289.96, "end": 2294.04, "text": " and at the end of 2019, the whole GPT-2,"}, {"start": 2294.04, "end": 2297.6, "text": " the biggest one, I think it had 1.5 billion params,"}, {"start": 2297.6, "end": 2298.64, "text": " was published."}, {"start": 2298.64, "end": 2300.68, "text": " So GPT-3 is still not published,"}, {"start": 2300.68, "end": 2302.28, "text": " and it probably will never be,"}, {"start": 2302.28, "end": 2307.28, "text": " but the API is already accessible for certain people."}, {"start": 2309.32, "end": 2310.26, "text": " It's in beta."}, {"start": 2310.26, "end": 2314.32, "text": " Okay, so what I wanted to say here is that"}, {"start": 2316.08, "end": 2318.52, "text": " it's even more important to consider these questions"}, {"start": 2318.52, "end": 2323.52, "text": " for GPT-3, because basically whatever malicious task"}, {"start": 2325.24, "end": 2328.86, "text": " that depends on producing a huge amount of text,"}, {"start": 2328.86, "end": 2330.14, "text": " GPT-3 will help with that."}, {"start": 2330.14, "end": 2332.24, "text": " So that's phishing, that's spamming,"}, {"start": 2333.48, "end": 2334.38, "text": " bunch of different stuff."}, {"start": 2334.38, "end": 2335.68, "text": " So social engineering,"}, {"start": 2335.68, "end": 2340.22, "text": " so that can be automated using GPT-3 in a sense."}, {"start": 2340.22, "end": 2343.04, "text": " So they did consider those things."}, {"start": 2343.04, "end": 2345.92, "text": " So many of these applications bottle on human beings"}, {"start": 2345.92, "end": 2349.3, "text": " to write sufficiently high-quality text."}, {"start": 2350.66, "end": 2355.1, "text": " And they mentioned who has the resources to do this,"}, {"start": 2355.1, "end": 2356.1, "text": " like government groups."}, {"start": 2356.1, "end": 2358.16, "text": " I don't think this part is as interesting"}, {"start": 2358.16, "end": 2360.1800000000003, "text": " as this part about the bias."}, {"start": 2360.1800000000003, "end": 2362.1, "text": " So they considered three axes."}, {"start": 2362.1, "end": 2365.3, "text": " So one is gender, the second one is religion,"}, {"start": 2365.3, "end": 2368.58, "text": " and the third one is, let me check, race."}, {"start": 2368.58, "end": 2369.42, "text": " Yep."}, {"start": 2369.42, "end": 2374.42, "text": " So how they evaluated the bias in these models is,"}, {"start": 2376.1, "end": 2380.06, "text": " they basically prompted with sentences like this one."}, {"start": 2380.06, "end": 2381.52, "text": " The occupation was up."}, {"start": 2381.52, "end": 2385.62, "text": " Like for example, and they note here the detective was up,"}, {"start": 2385.62, "end": 2388.26, "text": " and they are looking at the,"}, {"start": 2389.54, "end": 2392.14, "text": " what's the probability distribution coming out"}, {"start": 2392.14, "end": 2395.14, "text": " from the GPT-3, and they're looking at male and female."}, {"start": 2395.14, "end": 2398.26, "text": " And what I figured out is that most of the occupations"}, {"start": 2398.26, "end": 2401.2000000000003, "text": " are biased towards males."}, {"start": 2401.2000000000003, "end": 2404.0600000000004, "text": " So especially when they put it like this,"}, {"start": 2404.0600000000004, "end": 2409.0600000000004, "text": " the competent, then fill in the blank, was up."}, {"start": 2409.82, "end": 2414.1000000000004, "text": " And they figured out that that version of the prompt"}, {"start": 2414.1000000000004, "end": 2416.78, "text": " had even higher bias for males."}, {"start": 2416.78, "end": 2419.44, "text": " And actually also the incompetent one"}, {"start": 2419.44, "end": 2421.5800000000004, "text": " was also biased more towards males."}, {"start": 2421.5800000000004, "end": 2424.46, "text": " So again, iterating a little bit on this one."}, {"start": 2424.46, "end": 2427.46, "text": " Here we can see in particular occupations"}, {"start": 2427.46, "end": 2429.58, "text": " demonstrating higher levels of education,"}, {"start": 2429.58, "end": 2432.66, "text": " such as legislator, banker, or professor,"}, {"start": 2433.66, "end": 2435.7200000000003, "text": " were heavily male leaning."}, {"start": 2436.86, "end": 2440.52, "text": " And also like physical labor stuff was also male leaning,"}, {"start": 2440.52, "end": 2443.58, "text": " where the occupations that were more likely to be followed"}, {"start": 2443.58, "end": 2446.42, "text": " by female identifiers include midwife, nurse,"}, {"start": 2446.42, "end": 2448.7, "text": " receptionist, housekeeper, et cetera."}, {"start": 2448.7, "end": 2453.1, "text": " So once more how it works, you basically take a variant"}, {"start": 2453.1, "end": 2456.82, "text": " of the prompt like this one, you input it into your GPT-3,"}, {"start": 2456.82, "end": 2458.2200000000003, "text": " let me zoom in a little bit."}, {"start": 2458.2200000000003, "end": 2461.38, "text": " So you basically input it into your GPT-3 model."}, {"start": 2462.46, "end": 2464.34, "text": " Let's say this is GPT-3."}, {"start": 2464.34, "end": 2467.26, "text": " And it outputs as the next token,"}, {"start": 2467.26, "end": 2470.6600000000003, "text": " so the probability distribution which has the size"}, {"start": 2470.6600000000003, "end": 2475.54, "text": " of the GPT-3's vocab, which I think is around 50K."}, {"start": 2477.52, "end": 2482.52, "text": " And what it did then was they monitored a couple"}, {"start": 2482.52, "end": 2486.92, "text": " of male identifiers like man, male, et cetera,"}, {"start": 2486.92, "end": 2491.2, "text": " and they monitored like female, woman, et cetera."}, {"start": 2491.2, "end": 2495.2, "text": " And they just pretty much added the probabilities"}, {"start": 2495.2, "end": 2497.32, "text": " and compared those."}, {"start": 2497.32, "end": 2498.88, "text": " They also did some normalization."}, {"start": 2498.88, "end": 2501.44, "text": " So basically that's the method how they did this."}, {"start": 2501.44, "end": 2505.44, "text": " And I already mentioned the results that came out of it."}, {"start": 2507.12, "end": 2508.28, "text": " So that was about the gender."}, {"start": 2508.28, "end": 2511.92, "text": " Then I'll just briefly go over the race."}, {"start": 2511.92, "end": 2516.92, "text": " The, so across the models, they analyzed Asian"}, {"start": 2517.36, "end": 2519.36, "text": " had a consistently high sentiment,"}, {"start": 2519.36, "end": 2522.7200000000003, "text": " whereas the black had a consistently low sentiment."}, {"start": 2522.7200000000003, "end": 2527.34, "text": " And this was kind of surprising because you usually hear"}, {"start": 2527.34, "end": 2532.26, "text": " that like white males are much better positioned"}, {"start": 2532.26, "end": 2533.5, "text": " than Asians."}, {"start": 2533.5, "end": 2537.08, "text": " So this was kind of surprising in a way for me."}, {"start": 2537.08, "end": 2539.6800000000003, "text": " And yeah, that's an obvious bias."}, {"start": 2539.68, "end": 2544.68, "text": " And especially for black people, like a lot of the text"}, {"start": 2545.3599999999997, "end": 2550.3599999999997, "text": " has, is referencing like slavery, et cetera."}, {"start": 2550.48, "end": 2553.2, "text": " And then that's the reason black people"}, {"start": 2553.2, "end": 2555.64, "text": " have the lowest sentiment."}, {"start": 2555.64, "end": 2556.9199999999996, "text": " They did mention it here."}, {"start": 2556.9199999999996, "end": 2558.7999999999997, "text": " So the resulting sentiment can reflect"}, {"start": 2558.7999999999997, "end": 2560.44, "text": " socio-historical factors."}, {"start": 2560.44, "end": 2563.44, "text": " For instance, text relating to a discussion of slavery"}, {"start": 2563.44, "end": 2565.8799999999997, "text": " will frequently have a negative sentiment."}, {"start": 2565.88, "end": 2570.88, "text": " And that's the reason why the black race always has"}, {"start": 2573.48, "end": 2578.4, "text": " lowest sentiment, like for every single model size."}, {"start": 2578.4, "end": 2580.32, "text": " So X axis is the model size."}, {"start": 2580.32, "end": 2582.08, "text": " Here is the biggest one."}, {"start": 2582.08, "end": 2586.1600000000003, "text": " Finally, religion, they tried out like the top,"}, {"start": 2586.1600000000003, "end": 2591.08, "text": " the most popular like world religions like Christianity,"}, {"start": 2591.08, "end": 2594.88, "text": " Buddhism, Islam, et cetera."}, {"start": 2594.88, "end": 2599.88, "text": " And so they found some negative biases about Islam."}, {"start": 2600.04, "end": 2603.1600000000003, "text": " Like we also found that words such as violent, terrorism,"}, {"start": 2603.1600000000003, "end": 2605.96, "text": " terrorist occurred at a greater scale with Islam"}, {"start": 2605.96, "end": 2607.2000000000003, "text": " than with other religions."}, {"start": 2607.2000000000003, "end": 2611.04, "text": " And we're in the top 40 most favorite words for Islam"}, {"start": 2611.04, "end": 2612.86, "text": " in GPT-3, which is really sad."}, {"start": 2612.86, "end": 2616.3, "text": " And we need to be aware of these biases"}, {"start": 2617.26, "end": 2619.6400000000003, "text": " because obviously, I won't get into this,"}, {"start": 2619.6400000000003, "end": 2621.32, "text": " but like most of the Muslim people"}, {"start": 2621.32, "end": 2625.06, "text": " are really nice folks and that's it."}, {"start": 2625.06, "end": 2628.8, "text": " They did also consider the energy usage."}, {"start": 2628.8, "end": 2632.76, "text": " And one really surprising fact was this."}, {"start": 2632.76, "end": 2636.4, "text": " So though models like GPT-3 consumes significant resources"}, {"start": 2636.4, "end": 2639.28, "text": " during training, they can be surprisingly efficient"}, {"start": 2639.28, "end": 2640.1600000000003, "text": " once trained."}, {"start": 2640.1600000000003, "end": 2644.32, "text": " So even with the full GPT-3, 175 billion per am,"}, {"start": 2644.32, "end": 2647.2400000000002, "text": " a large model, generating 100 pages of content"}, {"start": 2647.24, "end": 2652.24, "text": " from a trained model can cost on the order of 0.4 kilowatt hour"}, {"start": 2653.4799999999996, "end": 2655.52, "text": " or only a few cents in energy costs."}, {"start": 2655.52, "end": 2656.7999999999997, "text": " It's actually pretty efficient"}, {"start": 2656.7999999999997, "end": 2659.3799999999997, "text": " once you train this huge monster."}, {"start": 2659.3799999999997, "end": 2664.3799999999997, "text": " And then during the lifetime of the model,"}, {"start": 2664.72, "end": 2669.72, "text": " it can be like the energy costs can be amortized potentially."}, {"start": 2670.2, "end": 2671.64, "text": " Yep."}, {"start": 2671.64, "end": 2674.52, "text": " So that was pretty much all I had to say."}, {"start": 2674.52, "end": 2677.56, "text": " I could make this video much longer."}, {"start": 2677.56, "end": 2679.56, "text": " I mean, this is a huge paper."}, {"start": 2679.56, "end": 2681.88, "text": " So hopefully you liked it."}, {"start": 2681.88, "end": 2683.4, "text": " If you did, please leave a comment,"}, {"start": 2683.4, "end": 2685.08, "text": " what you liked, what you didn't like,"}, {"start": 2685.08, "end": 2687.16, "text": " what are your opinions on this?"}, {"start": 2687.16, "end": 2691.72, "text": " And yeah, regarding the nagging versus the hyping sides,"}, {"start": 2691.72, "end": 2692.88, "text": " I just wanna mention one thing."}, {"start": 2692.88, "end": 2694.44, "text": " So what's my opinion on this one?"}, {"start": 2694.44, "end": 2696.56, "text": " So I really think that these huge models,"}, {"start": 2696.56, "end": 2699.94, "text": " like huge transformers, are not something"}, {"start": 2699.94, "end": 2703.68, "text": " that's a digression or a bad thing for the future of AI."}, {"start": 2703.68, "end": 2704.6, "text": " Why do I think that?"}, {"start": 2704.6, "end": 2708.52, "text": " That's because your brain, most of your brain,"}, {"start": 2708.52, "end": 2710.96, "text": " like cerebellum, where there is,"}, {"start": 2710.96, "end": 2714.7599999999998, "text": " the largest amount of computation happens in cerebellum"}, {"start": 2714.7599999999998, "end": 2716.16, "text": " is actually unconscious."}, {"start": 2716.16, "end": 2720.4199999999996, "text": " So we are probably now in the phase"}, {"start": 2720.4199999999996, "end": 2725.04, "text": " where we are developing a synthetic primordial brain."}, {"start": 2725.04, "end": 2727.52, "text": " Basically, the equivalent that we have,"}, {"start": 2727.52, "end": 2730.44, "text": " which is pretty much regulating all of our"}, {"start": 2730.44, "end": 2734.8, "text": " vital functions, like controlling the heart,"}, {"start": 2734.8, "end": 2737.6, "text": " temperature, et cetera, and also perception,"}, {"start": 2737.6, "end": 2739.68, "text": " so just kind of extracting concepts"}, {"start": 2739.68, "end": 2741.5, "text": " from image and audio, et cetera."}, {"start": 2741.5, "end": 2743.84, "text": " So all of those require a lot of computation,"}, {"start": 2743.84, "end": 2745.6, "text": " and I don't see this as a digression."}, {"start": 2745.6, "end": 2747.48, "text": " I just see it as a supplement,"}, {"start": 2747.48, "end": 2750.96, "text": " and as a step towards the artificial general intelligence,"}, {"start": 2750.96, "end": 2754.8, "text": " which will happen once we are able to simulate"}, {"start": 2754.8, "end": 2757.64, "text": " the cognitive layer above the primordial brain level."}, {"start": 2757.64, "end": 2761.24, "text": " So those are just some two cents on my side"}, {"start": 2761.24, "end": 2762.68, "text": " about what I think about this."}, {"start": 2762.68, "end": 2765.8399999999997, "text": " And I have some gut feeling that graph neural networks"}, {"start": 2765.8399999999997, "end": 2768.6, "text": " and causational relations that Judea Pearl"}, {"start": 2770.24, "end": 2772.6, "text": " has been pushing for decades now"}, {"start": 2772.6, "end": 2776.2, "text": " will play a significant role, as well as symbolic AI."}, {"start": 2776.2, "end": 2779.2, "text": " So I don't think we should kinda disregard all of those,"}, {"start": 2779.2, "end": 2782.7599999999998, "text": " and we should just be cognizant that all of those"}, {"start": 2782.7599999999998, "end": 2785.16, "text": " will probably somehow play a role"}, {"start": 2785.16, "end": 2789.3199999999997, "text": " like in this long-term goal towards achieving AGI."}, {"start": 2789.3199999999997, "end": 2790.8399999999997, "text": " So that was pretty much it."}, {"start": 2790.8399999999997, "end": 2793.0, "text": " If you liked this video, consider subscribing"}, {"start": 2793.0, "end": 2794.8399999999997, "text": " to this channel and hit that bell icon"}, {"start": 2794.8399999999997, "end": 2797.02, "text": " to get notified when I upload a new video."}, {"start": 2797.02, "end": 2799.7, "text": " And until next time, keep learning."}, {"start": 2799.7, "end": 2800.54, "text": " Deep."}, {"start": 2800.54, "end": 2815.54, "text": " We'll see you next time."}]