GPT-NeoX-20B | BigScience BLOOM | OPT-175B | Training Large Language Models | Papers Explained
🚀 Find out how to get started using Weights & Biases 🚀
http://wandb.me/ai-epiphany

❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

In this video I cover 3 publicly shared LLM 🚀 projects/papers and the pain they experience training them (🍿🍿🍿):

1. "What Language Model to Train if You Have One Million GPU Hours?" introducing BLOOM 176 billion parameter model by BigScience!

2. "OPT: Open Pre-trained Transformer Language Models" introducing 175 billion parameter model OPT-175B!

3. "GPT-NeoX-20B: An Open-Source Autoregressive Language Model" introducing, well, GPT-NeoX-20B, a 20 billion parameter LLM!

All 3 projects shared their weights, code, and papers, so it's a great way to dig into large language models and understand them better. 

I walk you through the papers and the chronicles/logs they've shared sharing the pain they experienced training at these scales! :))

Cluster deletion, 2 million consecutive backslash symbols in the dataset, and much more - it's fun!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ BLOOM paper: https://openreview.net/forum?id=rI7BL3fHIZq
✅ OPT paper: https://arxiv.org/abs/2205.01068
✅ GPT-NeoX-20B paper: https://arxiv.org/abs/2204.06745
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
00:33 (sponsored) Weights & Biases
02:43 BLOOM paper
19:37 BLOOM chronicles
25:08 OPT paper
31:20 OPT chronicles
33:46 GPT-NeoX-20B paper
45:30 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#gptneox #bigscience #opt
 What's up guys in this video I'll be covering three very famous large language model papers And all of them have in common that they've publicly shared both the code as well as the weights So the idea will be to kind of show you Some of the connections the commonalities between them will not be digging into the actual technical details that much that much because it's basically just A transformer architecture and I already assume most of you know how hot transformers work If you don't know you can check out my video. I've covered a long time ago on how transformers work But before we go there this video is sponsored by weights and biases weights and biases is an amazing machine learning platform that helps you Track your experiments do data set versioning and model management You can see how easy it is to get started with weights and biases They are framework agnostics So that means you can you can either pick pytorch keras or whatever you want and it's only a couple of lines of code so you import the weights and biases after installing it by doing pip install and then if After initialize the project here It will start automatically logging both the configs as well as various system metrics such as GPU utilization Etc etc and if you instruct it to do so you can obviously log curves images videos and much more Here is an example for run that I want to show you quickly. That's very representative So here are some of the system metrics we are logging such as GPU temperature memory location Etc if you instruct it to do so you can log obviously losses validation loss is accuracy whatever you want and finally in this example here you can see a hyper parameter sweep optimization experiment Where we can for example select those runs? They have very low loss and then we can draw some conclusions such as we can see here that higher epochs tend to perform best and Then lower learning rate tends to help etc etc You can additionally log videos not only images and even more structured Information and data such as tables and here you can see this is not just like a your dummy UI You can apply various filters such as for example show me those rows where the where the ground truth equals the actual Guess so that's our prediction and after you click apply here. You'll literally see filtered out exactly those data points So very cool. Finally, one of the features are really love and there are many more features I won't show you here but like creating super rich media reports such as this one you can see how interactive it can be and how beautiful and you can make it as complex as you want as you can see here with many cells in the grid and animations, etc, etc weights and biases are free for personal use and academics So check out the link down below to support this channel and now let's get back into the video So, let's see which papers I'm covering. So the first one is What language model to train if you have 1 million GPU hours coming from the big science event and they managed to train a 176 billion parameter large language model, which is huge. That's the biggest currently open sourced LLM then we have the opt model for meta so open pre trained transformer large language models, which was 175 billion parameters and here I'll mostly focus on some of the struggles they were experiencing will so that we'll see those a bit later And finally, the third people will be GPT neo X 20 B and open source or regressive language model from the illu theory community, okay So before we go there, let me get kind of quickly show you the the glimpse of what we'll see so I'll be walking you through some of the chronicles that opt folks as well as the bloom folks shared So here is one very interesting diagram where you can see on the wire Because you see the perplexity of the language model So basically the loss and you can see the the part I want to I want you to focus on is on the gaps here And so you can see that like for in this duration of like a month or something. They had these huge like Gaps where the model could not train because of our idea hardware hardware failure or software A box or whatnot. You can also see a bunch of spikes You can see how these are kind of almost almost don't don't don't like for my like a smooth curve And that's because they were oftentimes they had to obviously start from from a previous checkpoint And so there was a lot of struggle in there. Just looking at this graph. It's kind of clear And one interesting detail here as well is like the cluster deletion So that you managed so their cloud provider managed to delete their cluster in the middle of the run And so yeah, those are some of the interesting nuggets. We'll see a bit later Okay, having said that let's get back to the papers I'm gonna quickly go through all of those the reason I'm covering these papers is because over the next videos I'll probably do a code walkthroughs of some of these projects. So stay tuned for that So here I'm gonna give you some context and theoretical background Okay. So first of all, we can see here That the bloom folks come up with the same types of scaling laws here So we have the loss on the y-axis and we have the compute in the petaflop days on the x-axis And we can basically see the very similar curves to what Kaplan at all from the scaling loss paper from open AI Kind of already showed us petaflop days by the way stands for so one petaflop day is sustaining one petaflop of compute for a whole day So that that many operations, okay, so Let me kind of go through here. This is gonna be Basically a recurring theme Everyone all of these three papers literally took inspiration from the GPT-3 model So you can see here they say we follow the architecture and hyper parameters of GPT-3 model That's going to be like a commonality between all of these three papers So let's continue here They say we base this model size on Kaplan at all. So that's the the scaling loss paper However, after this paper was accepted for publication, but before it came out Huffman at all provided an alternative approach for selecting the model size. So that's the chinchilla paper from DeepMind I'm going to quickly just show you the the graphs for those of you who are not familiar with that with the scaling loss paper I strongly recommend you at least skim it. It's a very interesting finding and it's also very empirical and very It's over fitted to a particular type of a transformer. So here you can see what the the basic like postulate behind the scaling laws is is if you you can see on the x-axis if we increase compute so here on the on the on this left diagram here we have compute and then we have data set size here and finally we have Number of parameters. So if you increase either three of those You can see that the loss on the y-axis here the test loss is decreasing and we don't see any sign of situation here So that's the that's that was the paper that kind of showed us that we just need to scale the models And we'll get more and more performance now. It's a completely different topic and problematic whether like having a smaller loss Like perplexity actually correlates with the metrics we actually care about but that turns usually to be the case but yeah, okay, so one thing that the this paper the scaling laws paper showed is This one so they say that the the performance penalty depends predictably on the ratio You can see this one here meaning that every time we increase the model size Atex we need we only need to increase the data but roughly five times to avoid a penalty So basically to avoid the over fitting and so this is in starting point Contrast with the chinchilla paper. So what I say here is that you basically need to Increase the number of tokens of your training data set much slower According to this relationship you can see on the screen According to this formula here compared to the number of parameters and then the chinchilla paper actually showed a completely different Finding and that's that so they say specifically given a 10x increase in computational budget They suggest that the size of the model should increase 5.5 By day they refer to the kaplan paper while the number of training tokens should only increase 1.8x Instead we find that the model size so this is the important finding of this paper the model size and the number of training Tokens should be scaled in equal proportions, okay So the problem is there is um, no no strong theoretical underpinning Behind these and and like who knows how how these uh laws will and whether they will be valid for for different Configurations of transformers if you change something if you start using I don't know maybe a libby instead of relative positional encodings Maybe then the laws change and we are not sure how sensitive they are Um, I might be wrong. Maybe the chinchilla paper has like some stronger theoretical insights, but that's i'm recalling from my memory Okay, so let's get back to the paper here So the first thing they show is that obviously it's very important to have high quality data set and by high quality They mean like so so having these diverse cross-domain high quality data Such as the pile data set so that's the data set that we have here And we'll see that their paper a bit later And you can see that using the the pile compared to these other data sets such as the oscar or c4 Gives them basically better better accuracy on on these various nlp tasks. You can see that the blue model here Is on pair and even a bit better compared to the eluthorized paper The gpt neo paper is the same as the The gpt neo x20b and you can also see that basically the the open eyes models to outperform them by by some amount So that's that's um worth noticing Okay, guys, so let's continue here Let's see. What else is interesting in this paper? Um, so as I said diverse cross-domain pre-training data combining web calls with high With curated data and with high quality data So as I said diverse cross-domain pre-training data combining web calls with high with curated high quality sources significantly proves zero shot Generalization over pre-trained data sets constructed from common crawl only Okay, that's one finding the second finding they had was that although learned positional embeddings help perform rotary embeddings Alibi yields significant better results than all the other alternatives now This might be as well very very specific to their particular set of high parameters in architecture I wouldn't like take anything fundamental out of this statement But um, yeah, i'm just kind of walking you through some of the findings behind behind the papers, by the way I did cover the alibi paper I think like eight months ago or something when it came out So do check it out if you're curious to know more and we'll see what the rotary embeddings if you're not familiar Um, what those are? Um a bit later in the in the gpt neo x paper Okay, some of the other findings they had was they tried a bunch of different activation functions And then they say here we present our results in table three bubble us swigloo produces slightly better results than jellu However, this comes at a cost of reducing the throughput by approximately a third So I think they ultimately decided not to use it because of the throughput Okay So that's that's one more finding and then they they they mention how the and we already know this basically, it very it matters a lot where you put the Uh layer normalization and how many layer normalization layers do you include in your transformer blocks and and they say here adding layer Normalization after the embedding layer incurs a significant penalty on zero shot generalization But the thing is when you scale the model to very big sizes Uh, then this thing helps stabilize the training Uh, and when you're on the when you're dealing with lower scales Uh in that case, you probably don't want to include it because you'll be sacrificing as you can see here that the zero shot generalization Okay, guys, let's continue here One thing that I kind of noted to myself as well. I was kind of This is a curious finding. Uh, so they say instead of manually writing So this is about the multilingual, uh, basically evaluation So trying to evaluate on multiple different human languages And they say instead of manually writing prompts for each language, which would obviously be very bothersome We followed the strategy proposed by linetal using english prompts for non-english examples This can be viewed as cross-lingual zero shot generalization Uh, they validated this strategy by demonstrating its ability to achieve zero shot performance on pair With and sometimes even better than human written language specific prompts. This is kind of cool. So you literally take a uh, um, you you have data sets with various different languages and you always take the english prompt and then you concatenate the the Uh, the answer in that different language you repeat it a couple of times and then you evaluate and it's and it's apparently the model Like learn to to understand, uh this which kind of makes sense because you're training the disjoint Uh, like a space of multi disjoint multilingual space Uh when you're training transformer on on on diverse data sets And we always always know that the machine translation kind of emerges as a consequence of this Next token prediction, uh, if you give it enough big data sets with with some presence of different languages So thing worth mentioning and this is something we're probably all aware of at this point And that's that the multilingual pre-training Very significantly diminishes english zero shot generalization. So the thing with this bloom model is that they trained Uh, this is uh, like a multilingual model and because of that it's not as strong Uh on on on english as as uh, basically some of the other baselines Okay, let's continue here. Um A couple things worth mentioning, this is how they've chosen the final configuration of the bloom 176 billion parameter model So they they reference here the the the the scaling loss paper It is notable that this pareto optimal frontier describes very large models trained on few tokens We call this training to optimality So that's basically what I said before basically you want to the the scaling loss paper show that you want to increase the parameters more Than the number of tokens of your data set and then they say this is in stark contrast with the common practice of training much smaller Models on many more tokens to convergence. Okay And now because they're training They have a couple of arguments here But one of them is that they're training on a multilingual data set and because of that They they don't want to strictly follow the the scaling loss. And so this is what they do So we choose to use the optimality front as an upper bound on model size So that's like the biggest model size There will be determined by the scaling loss paper and the lower lower bound on number of training tokens So they'll probably have smaller model and more tokens Okay compared to what the scaling loss paper suggests the second very influential paper that had impact on on the Like modeling decisions in this paper is this levin et al paper So they proposed a theoretically motivated and empirically backed law describing the optimal compromise compromise between width and height Within depth, sorry. So for a gpt3 size model with 175 billion parameters They predict an ideal depth of 80 layers. Okay so now it's just basically Like combining lego bricks you have some constraints from the scaling loss paper You have some constraints from this paper and then given the compute they have and that's Roughly 1 million gp hours on a 100 machines They decide on the on the on the on the hyper parameters of the model. So let's let's see what those are So they they lose this many tokens as you can see here 70 to 80 layers because of this paper and Et cetera, et cetera. So finally they end up with three best configurations And the one they pick at the end is this one here. So they they pick this one with 176 billion parameters 70 layers this many hidden dimensions attention hats Dimensionality of attention hats. This is the memory footprint and the performance. Okay, so that's pretty much it That's the first paper. Let me get kind of show you these numbers side by side in the actual code base And then we'll switch to the second paper. Okay guys, here it is And as I said, i'll i'll do a walkthrough of of the code base. So basically don't worry I'll i'll show you this code in much more depth later on Uh, I just want to kind of briefly show you so this is the big science workshop Repo and you can see the configuration for the 176 billion model And you can see here the numbers are exactly what you see here in the table. So we have 176 layers so you can see that somewhere here. Let me try Nope, sorry, that's the number of uh, that's the number of parameters. Uh, so we have the 70 layers So here is the the variable for 70 layers. We have 14336 you can see that number of hidden is is exactly that we have number of hats 112 etc, etc. So basically yeah, this paper shows you how how they've Decided to to pick the uh, all of the hyper parameters of the model Some of that was inspired by the gpt3 paper. Some of that was inspired by the scaling loss paper. And finally, uh, the The levin et al paper had a had an impact as well on the on the width and depth now I'm gonna quickly walk you through the chronicles they created So basically, uh, i'm gonna start here So first of all one more thing that's kind of common across all of the three papers is that all of them are using the nvdm Megatron library that helps you do model parallelism do check out my scaling Uh, like ultimate guide to scaling video where I cover a couple of these uh influential scaling papers including megatron The second thing they they they all have in common is the deep speed library Again, I did cover that one in my previous video So do check it out and the bloom folks because they had 384 gpus at their disposal basically decided to to Arrange those gpus in this 3d Fashion so the so-called 3d parallelism where one axis is basically pipeline parallelism Where you break the model such that you take a couple of first layers you put them on one gpu Then you take the next uh end layers you put them on the second one, etc, etc Then you have the model parallelism where you break the actual layers uh And then kind of take the weights of that layer and put one part on one gpu the second part on the second one Etc, etc And the third dimension is the the data parallel dimension here where you basically replicate Your models across different gpus and then you feed them different portions of your of your input patch Okay, next up. I want to walk you through some of the chronicles. Uh, they've shared So there is this document called lessons learned. I link the repo down in the description So you can check it out You can go I strongly encourage you to go through uh, at least skim through all of these documents I think it's a valuable learning experience. So a couple of things so they have this title how training divergences were Overcome so how they overcame the the divergences one of the the methods was to just have a different initialization method And it literally means so much like I think they mentioned it somewhere So it has made a huge difference to the training stability and not only stability but the the final Uh actual loss you you converge to and I remember back from my days when I trained the transformer model So I had the original transformer implementation right here and uh Initialization blah blah. I remember this this diagram here. So here's what I said here So initialization matters a lot for transformers. I initially thought that the other implementation using savir Uh initialization is again one of those arbitrary heuristics and the pytorch default in it will do I was wrong You can see here three curves that these two are using pytorch default and this one is using the xavier initialization So that was the moment for me when I realized how sensitive the models are to initialization and how vital that is Okay getting back to the lessons learned document here adding embedding layer norms, so Hopefully we're all aware of this basically the position whether it's before there is a jollo connection or after How many of them do you include etc? etc All of that matters a lot for the final stability and finally patience So in some cases in so they say here in some cases in the case of a huge spike It was taking 2k iterations for training to return to the same Loss it spiked from and then it continued training as if nothing happened So sometimes in order to overcome the spike you just have to wait Uh, but more often than not the training won't recover from a spike Yet in another situation as the training diverged slowly without any spikes. Okay, so let me show you the spikes so here is the Uh, this is probably the most interesting chronicle i've seen so the the the way they've trained the 104 billion parameter model They had the most problems with this model and then actually with the 176b model It was the training was much more stable and there are many reasons why that was first of all They I guess they learned a lot during during this this whole trial and then the second thing is I think like they use b Float so the brain float format instead of the the the the usual float 16 and all of that kind of brought the additional stability So I just want to kind of quickly skim through Some of the diagrams here you can see that the problems they had with spikes So during the training, uh, the spike would appear and then the training would diverge And they had a lot of experiments. They tried a bunch of things to try to to kind of cope with those spikes But they kept getting the spikes. They kept getting the spikes like throughout many many like experiments and many many weeks they constantly they couldn't get the model to Uh to converge it was constantly diverging and then I think only at the end Did they start getting some better results after multiple months? Um, i'll show you the the the 176b model a bit later. So let me show you one more thing. See here This is interesting. So the they had two million backslash only samples in the data set um, so Hugh Negan, I don't know how to pronounce this surname Discovered that there are huge records with just backslashes in oscar English version so we set out to look at their occurrence And so what happens when when your model basically encounters this sequence? Uh, it can produce the the the spiking behavior the divergence So all of these details are this is the this is the messy reality of training of training large language models And it's very nice that they've shared this Okay, I did mention that the 176b was much more stable They literally say here what makes the the model so so stable and they compare so here is the 104 Uh billion parameter model you can see Uh bunch of curves, uh superposition here and you can see the the the 176b So it's literally like a smooth curve converging and the loss is lowering as expected So, uh, here is the kind of the overlapped image and then they say it's a combination of probably a few of these improvements It's kind of hard to pinpoint a particular thing, but I I have a strong hunch that the bfloat Uh format is what had a very very significant impact here. So very clean data. They have better data set for the for the big model They had the bfloat, uh mixed precision regime The fp32 accumulation for the pipeline blah blah blah very low initialization Word embedding layer norm But it was also using okay, so then I guess that doesn't doesn't explain it Uh, okay, that's guys. That's pretty much it They also have this document where they show, uh how they decided to To pick the parameters of the 3d parallelism. So basically they decide how they decided on on picking the tensor parallelism Parameter the pipeline parallelism the data parallelism all of those details so you can kind of walk through this document at your own pace It's too long for me to kind of extract any any particular insight here But that's pretty much it. Okay guys, so now let's go to the second paper I'm gonna open up the um The opt paper and let's start from the beginning here. So open pretrained transformer language models So these guys had a lot of problems training their models like reading through the the the logs was was such a such a joy So basically let me kind of show you a couple of snippets from a huge document They have like there is like 140 pages or something of their logs Internal logs, that's very cool. They shared those so here here are a couple of those once we identify the bad node We need to document these lists of things to complain to csp and get our money back. Give us our money back you bastards Okay, it looks like 26 tried to immediately upload the checkpoint and failed its cp commands Then it took another step lowered its scalar and tried uploading again and again the humanity We are already at loss scale 0.25 and so on and so on like oh hallelujah Things looked good, but nope just kind of complete confusion So I again encourage you to just kind of skim through the the log book. It's a very interesting read Okay, let's start with this one First of all our models and hyper parameters largely follow brown at all So that's the gpt3 paper again with variations in batch size mostly to obtain increased computational efficiency So this is something I did mention in the previous paper as well Oops, so this is kind of a common thread for all of these three papers They all follow the gpt3 hyper parameters and and uh advices Next up they say we found that the pile was particularly full of duplicate documents and advise future researchers using the pile to perform additional deduplication processing this is in particular relevant to the next paper i'll cover from eluthor ai Because they did use pile and they did not duplicate it because of the lack of resources And they also which is very interesting. They've done more than a single epoch, which is the common practice training lms But we'll get there in a couple of minutes a couple of interesting curves again, um, you can see Iterations on the x-axis learning rate on the y-axis. You can see that this was fairly ad hoc like basically the model would crash They would retrieve the the older checkpoint and then lower the low the the learning rate such that in order to avoid Try to avoid the instabilities and as a consequence of that you get this piecewise weird graph here Okay, you can also see the the validation perplexity Has a lot of these bumps and uh, it looks very funny Uh, so here is the part, uh, that's uh joy to read Basically a lot of a lot of problems during during the training So the the thing to mention in metals defense is that they literally had only a couple of months to do this It was a very tight deadline and so yeah, they didn't have the time to to do this properly But here is a couple of insights from from the paper. So in total hardware failures contributed to at least 35 manual restarts and the cycling of over 100 hosts over the course of two months Given the difference between the number of hosts cycled out and the number of manual restarts We estimate 70 plus automatic restarts due to hardware failures. I guess everything that will that can break will break the marfais law thing Okay, we noticed a correlation between loss divergence our dynamic loss scalar crashing to zero and the l2 norm of the activations Of the final layer spiking these observations led us to pick restart points for which our dynamic loss scalar was still in a healthy state So bigger or equal than one and after which our activation norms would trend downward instead of growing Unboundedly, so this is just some detail about how they were handling the um, how What was the heuristic they were using to to pick the best checkpoint in order to avoid the the future spikes? Okay, guys, uh, let's continue here. So the results here as you can see on 14 nlp tasks averaged for the zero shot Evaluation their own pair with gpt. That's very cool on the other side for the multi-shot Evaluation where it does like one or 32 shots. You can see that the uh, the dashed line the gpt model did outperform Them on on on those tasks. So that means It's kind of better few shot learner than the op model and um, yeah They say that the trend follows the gpt3. That's what what I said So let's continue here a couple of more interesting details. First of all, um, hate speech detection. They showed that um That the 175b model the op model considerably outperforms da vinci in all settings So that's very cool And then you think about it and this is all with the Double edged sword. So if it's better at detection, that means it's probably more susceptible to generating toxic And hateful speech so and that's indeed the case. So let me show you that here And they mention it. So overall we see that op 175b has a higher toxicity rate than either palm or da vinci da vinci being the the gpt3 model That's the code name. Okay, so let's continue here Da vinci being the the gpt3 model. That's the code name. Okay, you can see here Prompt toxicity probability toxicity probability of continuation. You can see that the blue curve the op model has the highest toxicity Uh for for for across across the the skip across the whole axis here Okay, guys, um I think that's pretty much it They they kind of summarize saying that we still believe this technology is premature for commercial deployment. And yeah, I mean, uh understanding how these models work, uh, Just interpreting what's going on and why the model is up but in certain results all of that We are still in the early days We are much better at at getting raw numbers on benchmarks and then knowing how to handle and align These models with human preferences, but it's a topic for for a different video. I think that's pretty much it. That's the second paper As I said, i'm just quickly giving you Uh the the the overview here and then in the next videos i'll be doing the coding Series, okay. Um, yeah before this let's let me show you the the opt logbook So here's the the big logbook I mentioned this is the this contains a bunch of notes That they were kind of accumulating throughout this period of a couple months I encourage you to check it out at your own pace. You can skim it. It literally took me maybe 20 30 minutes to just skim through this and kind of extract some of the insights Okay, so I did show you this diagram in the beginning of the video So again, you can see the torque or the number of gaps here and how many failures they were experiencing So we saw the number 70 plus or something restarts. I don't want to count but yeah, this might be At least in that range. Okay. Finally, let's go to their final update Uh, they have a couple of things like some some insights here Um, let me pick a couple of them. So scaling to 1024 a 100 to handle a real Workload of this size is highly non-trivial. We will discuss infrastructure pain points below Okay, ensuring training converges at this scale is also highly non-trivial without sufficient ablations at medium scale Results obtained from training at small scale also do not necessarily hold when scaled up We will cover these learnings in a note to be released in the upcoming weeks. Okay So let me quickly read you the cluster deletion story. That's kind of funny So given the holidays we requested a pool of 12 buffer nodes to guard against hardware failures Two machines go down every day on average in the process of replenishing this pool The cloud provider support team accidentally deleted our entire cluster on december 21st First while the cluster was restored fairly quickly it unfortunately came back with 16 machines that did not pass our infrastructure checks Etc, etc. So that's one of the interesting parts Second interesting diagram here is this one. They showed that they didn't quite match the the loss Of the gpt3 model, but we did see that on the on various nlp tasks the zero shot performance of the model did reach gpt3 so that's again the the the discrepancy between Um, like the loss value the final loss value and the actual performance might not always be that clear Although we did see that we did saw that the gpt3 handles much better the few shot prompts And that might be because of lower of lower loss here again just a speculation, but that's usually the case That's it guys again encourage you to go through it at your own pace I'm going to continue on and explore the gpt neo x20b paper So an open source autoregressive language model And let's see what the insights here are Okay, so we trained on a data set that contains duplicated data for more than one epoch But see no evidence of performance loss While these authors claims that the few shot prompting doesn't improve performance on their task We find that this is actually a phenomenon unique to gpt3 And does not apply to either gpt neo x20b or fair sec models. Okay, so that's just a note that uh, it's dangerous to just take a single model and Draw out extrapolate from from there Because other models might have different behavior so we'll see later in this paper that uh, The probable explanation is basically data set the underlying data set for that they use to train gpt neo x20b i.e the pile Is of different quality and distribution compared to the data sets that were used for gpt3 We don't even know what data was used. So yeah Okay So again, they mentioned here that they largely followed the gpt3 So so blah blah blah the coder model whose architecture largely follows that of gpt3 With a few notable deviations described below. So that's that's again the common thread that I mentioned multiple times But this paper did have like, um, many more modifications compared to the previous ones we saw. Okay Before this paper was published and the ways were were released They had a couple of iterations Behind them. One of them was gpt neo and the second one Was gptj which was basically supported by the elutrii community but executed by ban wang and komatsuzaki So that's kind of worth mentioning. So the architecture of this paper closely follows the gptj paper So let's see what's the diff between those two? So the gptj and this one compared to gpt3 So first of all, they use rotary embeddings I will not get into a lot of details how these work but like here is the here's your uh, classical formulation of the of this uh attention so you can see the formula here we have the uh Not token here. We map it into the query. So this is going to be the query We map this one into the key then we do the dot product blah blah blah And then the rest of the notation is kind of weird because it's not summation you want to have You want to have these? scores In separate spatial locations, you don't want to sum them up. So this is kind of weird notation Um, I mean looking at it. It's not even correct. Like you cannot just sum up these scores. It's kind of weird Uh, yeah, let me just make a diff here. So The the the difference with the rotary medics is you you have this this matrix r here Which basically rotates your intermediate representation. So here's the here's kind of the idea So if you have a key or a query So let's say we have a text here in hands transformer with rotary position embeddings and let's say that each of these words Has a corresponding uh vector here. Let's say this is um, either Uh a key or or a query vector, uh, and basically what I do is they kind of chunk Uh this vector into d over two Groups, uh, and that basically means that each of the groups is two-dimensional and then so once you have this chunk here What you then do is you literally rotate it depending on this position and depending on the on the on the group index So what I mean by that is so this is group one group two Uh group d over two so because this is from group one You basically apply the theta one angle and because it's position one you apply m and that's how you rotate And then you get the output representation here and that's the final representation here. So that's the modification that happens that happens Precisely here. Okay guys Um, so that's one of the modifications they've used the second one is this one We compute the attention and few forward layers in parallel and sum the results rather than running them in series Okay uh, that's a fairly, uh like a known technique for for uh increasing the throughput And uh basically does not sacrifice any of the of the of the performance So they say here this led to a 15% Throughput increase while having comparable loss curves with running them in series during early training So I guess there is a minor um penalty you're paying for for for getting the additional throughput. Okay Then they mentioned due to an oversight in our uh code We unintentionally apply two independent layer norms instead of using a tight layer norm the way that um These authors here do so instead of computing this thing you can see that they they actually decouple l and one and l and two So just just the bug they had in their code. Okay. Um um Okay, they use here's here's the uh infrastructure they are using And there is a nice diagram explaining how how the setup looks like the actual uh, like node setup looks like this So we train gpt new x20b on 12 Super micro servers each with eight nvidia a100 gpus and configured with two cpus. Okay So here you can see that the the complex diagram Um, even though it's a high level diagram and still very complex You can see how uh each of the four gpus connects to a single cpu here And then you have the different group here is connected with this cpu And all of these here is what is commonly referred to as a node. So this thing here is called a node You'll see this terminology a lot, okay guys So again a common thread That we opted to use the values from the gpt3 paper to guide our choice of hyper parameters To achieve a higher training throughput. We opt to use the same batch size as open ai's Uh 175b model approximately 3.15 million tokens We extend addmw with zero optimizer. So that's um, I kind of mentioned that because I did cover zero in one in in my ultimate scaling guide video I'll link that video somewhere in the in the descriptions if you if you're curious to know more about zero Check it out. But the the tldr is instead of replicating the the model states and the residual states So the gradients the the the optimizer states the weight etc Instead of that you just partition them and that's how you achieve the the zero redundancy. So that's that's what zero stands for um Finally a note before we see the results Uh, they say when comparing results in this work to gpt3 the training data is almost certainly the biggest known unknown factor Okay, and yeah data is becoming the name of the game pretty much A couple more differences, uh, they have compared to gpt3 is the tokenizer So we use a bp based tokenizer similar to that used in gpt2 gpt2 With the same total vocab size of 50 000 with three major changes to the tokenizer So first we train a new bpe tokenizer based on the pile which makes sense you want to because they're training on pile they want to optimize the tokenizer for that particular data Second in contrast to the gpt2 Tokenizer which took which treats tokenization as a start of a string as a non-space delimited token The gpt neo x20b tokenizer applies consistent space limitation Uh regardless and third our tokenizer contains tokens for repeated space tokens All positive integer amounts of repeated spaces up to and including 24. Okay, this is kind of a mouthful um, and just some idiosyncrasies of tokenizers Um, but like let me show you like a graphical representation of what's going on So here is the gpt2 and how gpt2 gpt2 tokenizer would tokenize this piece of text this snippet of code and here is how gpt neo x20b would tokenize that same text so this is what I mentioned about the uh Basically the spaces so you can see that basically all of these spaces here are encoded as a single token similarly here Etc etc up to 24. So literally until you get to 24. It's a single token That's that's how I understood this part. Whereas on the other side, um, you can see that gpt2 gpt2 literally encodes each of these spaces as a separate token and that kind of bloats the number of tokens to 55 here compared to 39 uh in this paper Okay, guys, so I mentioned this a couple of times we see no drop In test validation loss after crossing the one epoch boundary Here are some of the diagrams they have and it's a common practice to always train only for a single epoch I guess partially because you have so much data That going for multiple epochs would be super compute intensive. So people almost never go to more than than a single epoch And there are some papers showing that the model starts memorizing very quickly if you if you start Going multiple epochs. So people usually just do single epoch in this paper They literally continued training and they didn't see any any any signs of uh Of validation loss starting to diverge or anything. So they just kind of continued and uh, yeah, I mean um, I guess Yeah, I guess, um the common wisdom should be Questioned because the wisdom is not based on any solid theoretical understanding of why this works and like llm Large language model training is pretty much like dark magic and alchemy as many as many refer to it And so I think it's completely legit to do something like this if it's giving you better performance Why not? Okay, so Let's continue and let's wrap up this paper. Um, let me show you the results Uh, they get so this is the zero shot performance of gpt new x compared to gptj Fairsec and open eyes models. You can see it's pretty much on pair with them But then it really starts shining once you start using it on a day mathematical data sets and like these arithmetic tasks, which is I guess partially because a huge part of I mean not a huge part, but like a significant portions of uh of pile Are are like these scientific websites such as archive papers, etc, etc So because of the data set, uh, it seems that this model came much better compared to to the other baselines You can see the the orange dashed lines here. It's kind of Uh over outperforming all of the other baselines if i'm not wrong I think there are few shot performance not only in these data sets, but in general is also better compared to GPT models, but I don't see the diagrams here. So So yeah, okay. This is actually the one i'm referring to so this is the five shot evaluation You can see that again gpt new x is much better when you give it multiple shots So it's kind of a better few shot learner apparently compared to the gpt model and fairsec, etc And final load here We opted to choose hyper parameters based on on a mixture of experiments at smaller scales and by interpolating parameters Appropriate for our model size based on previously published work But because I think they so however several aspects of both our model architecture and training setup including the data and the tokenizer Diverged significantly from gpt3 as such It is almost certainly the case that the hyper params used for our model are no longer optimal and potentially never were In the appendix they go on to say And argument why it's a good idea to release the model weights because back at the at the time when they released this model This was the biggest model out there now. We have bloom now We have opt but like this was a very important contribution to the community and just a very valuable thing for like researchers and people who are not as as well-financed as some other research labs And that's it guys So those are the three papers I wanted to cover so again gpt neo x20b opt and bloom paper So we saw a couple of commonalities between them. They were all following gpt3 Hyper parameters and recommendations we saw that they were all using pretty much except for the opt paper Both the deep speed and Megatron Whereas opt was using only the Megatron library to scale up these models and train them Which is completely non-trivial thing to do most importantly that the value of these projects Is that they've shared both the code as well as the process as well as the weights which is a huge contribution to the community So kudos for that Guys if you like this video Subscribe to this channel share it out with your friends and Expect new coding videos coming up where I'll be walking you through some of the code bases behind these papers So stay tuned for that and until next time bye bye
[{"start": 0.0, "end": 5.7, "text": " What's up guys in this video I'll be covering three very famous large language model papers"}, {"start": 6.18, "end": 11.92, "text": " And all of them have in common that they've publicly shared both the code as well as the weights"}, {"start": 12.0, "end": 14.5, "text": " So the idea will be to kind of show you"}, {"start": 15.08, "end": 22.56, "text": " Some of the connections the commonalities between them will not be digging into the actual technical details that much that much because it's basically just"}, {"start": 22.56, "end": 27.52, "text": " A transformer architecture and I already assume most of you know how hot transformers work"}, {"start": 27.52, "end": 33.32, "text": " If you don't know you can check out my video. I've covered a long time ago on how transformers work"}, {"start": 33.32, "end": 40.64, "text": " But before we go there this video is sponsored by weights and biases weights and biases is an amazing machine learning platform that helps you"}, {"start": 40.64, "end": 44.16, "text": " Track your experiments do data set versioning and model management"}, {"start": 44.16, "end": 47.04, "text": " You can see how easy it is to get started with weights and biases"}, {"start": 47.239999999999995, "end": 48.82, "text": " They are framework agnostics"}, {"start": 48.82, "end": 54.32, "text": " So that means you can you can either pick pytorch keras or whatever you want and it's only a couple of lines of code"}, {"start": 54.32, "end": 59.28, "text": " so you import the weights and biases after installing it by doing pip install and then if"}, {"start": 59.6, "end": 61.480000000000004, "text": " After initialize the project here"}, {"start": 61.480000000000004, "end": 66.32, "text": " It will start automatically logging both the configs as well as various system metrics such as GPU utilization"}, {"start": 66.44, "end": 72.48, "text": " Etc etc and if you instruct it to do so you can obviously log curves images videos and much more"}, {"start": 72.8, "end": 77.48, "text": " Here is an example for run that I want to show you quickly. That's very representative"}, {"start": 77.48, "end": 81.56, "text": " So here are some of the system metrics we are logging such as GPU temperature memory location"}, {"start": 81.56, "end": 85.2, "text": " Etc if you instruct it to do so you can log obviously losses"}, {"start": 85.84, "end": 92.72, "text": " validation loss is accuracy whatever you want and finally in this example here you can see a hyper parameter sweep optimization experiment"}, {"start": 93.2, "end": 96.32000000000001, "text": " Where we can for example select those runs?"}, {"start": 96.4, "end": 103.52000000000001, "text": " They have very low loss and then we can draw some conclusions such as we can see here that higher epochs tend to perform best and"}, {"start": 103.52000000000001, "end": 106.64, "text": " Then lower learning rate tends to help etc etc"}, {"start": 106.64, "end": 112.36, "text": " You can additionally log videos not only images and even more structured"}, {"start": 113.04, "end": 117.92, "text": " Information and data such as tables and here you can see this is not just like a your dummy UI"}, {"start": 118.08, "end": 123.6, "text": " You can apply various filters such as for example show me those rows where the where the ground truth"}, {"start": 124.44, "end": 126.44, "text": " equals the actual"}, {"start": 127.16, "end": 134.92000000000002, "text": " Guess so that's our prediction and after you click apply here. You'll literally see filtered out exactly those data points"}, {"start": 134.92, "end": 138.64, "text": " So very cool. Finally, one of the features are really love and there are many more features"}, {"start": 138.64, "end": 143.27999999999997, "text": " I won't show you here but like creating super rich media"}, {"start": 144.11999999999998, "end": 150.6, "text": " reports such as this one you can see how interactive it can be and how beautiful and you can make it as complex as you want as"}, {"start": 150.6, "end": 153.0, "text": " you can see here with many cells in the grid and"}, {"start": 153.88, "end": 158.16, "text": " animations, etc, etc weights and biases are free for personal use and academics"}, {"start": 158.16, "end": 162.92, "text": " So check out the link down below to support this channel and now let's get back into the video"}, {"start": 162.92, "end": 165.28, "text": " So, let's see which papers I'm covering. So the first one is"}, {"start": 165.83999999999997, "end": 173.0, "text": " What language model to train if you have 1 million GPU hours coming from the big science event and they managed to train a"}, {"start": 174.23999999999998, "end": 180.95999999999998, "text": " 176 billion parameter large language model, which is huge. That's the biggest currently open sourced"}, {"start": 181.39999999999998, "end": 184.32, "text": " LLM then we have the opt model for meta"}, {"start": 185.0, "end": 189.35999999999999, "text": " so open pre trained transformer large language models, which was"}, {"start": 189.36, "end": 197.76000000000002, "text": " 175 billion parameters and here I'll mostly focus on some of the struggles they were experiencing will so that we'll see those a bit later"}, {"start": 198.48000000000002, "end": 206.0, "text": " And finally, the third people will be GPT neo X 20 B and open source or regressive language model from the illu theory"}, {"start": 206.72000000000003, "end": 208.56, "text": " community, okay"}, {"start": 208.56, "end": 214.0, "text": " So before we go there, let me get kind of quickly show you the the glimpse of what we'll see so"}, {"start": 214.0, "end": 219.32, "text": " I'll be walking you through some of the chronicles that opt folks as well as the bloom folks shared"}, {"start": 219.32, "end": 223.68, "text": " So here is one very interesting diagram where you can see on the wire"}, {"start": 223.68, "end": 226.52, "text": " Because you see the perplexity of the language model"}, {"start": 226.52, "end": 233.28, "text": " So basically the loss and you can see the the part I want to I want you to focus on is on the gaps here"}, {"start": 233.28, "end": 240.12, "text": " And so you can see that like for in this duration of like a month or something. They had these huge like"}, {"start": 240.12, "end": 245.4, "text": " Gaps where the model could not train because of our idea hardware hardware failure or software"}, {"start": 245.72, "end": 248.52, "text": " A box or whatnot. You can also see a bunch of spikes"}, {"start": 248.52, "end": 255.16, "text": " You can see how these are kind of almost almost don't don't don't like for my like a smooth curve"}, {"start": 255.16, "end": 261.88, "text": " And that's because they were oftentimes they had to obviously start from from a previous checkpoint"}, {"start": 262.68, "end": 266.52, "text": " And so there was a lot of struggle in there. Just looking at this graph. It's kind of clear"}, {"start": 266.52, "end": 270.28, "text": " And one interesting detail here as well is like the cluster deletion"}, {"start": 270.28, "end": 275.24, "text": " So that you managed so their cloud provider managed to delete their cluster in the middle of the run"}, {"start": 275.24, "end": 278.91999999999996, "text": " And so yeah, those are some of the interesting nuggets. We'll see a bit later"}, {"start": 278.91999999999996, "end": 281.32, "text": " Okay, having said that let's get back to the papers"}, {"start": 281.32, "end": 288.28, "text": " I'm gonna quickly go through all of those the reason I'm covering these papers is because over the next videos"}, {"start": 288.28, "end": 294.2, "text": " I'll probably do a code walkthroughs of some of these projects. So stay tuned for that"}, {"start": 294.2, "end": 298.2, "text": " So here I'm gonna give you some context and theoretical background"}, {"start": 299.0, "end": 301.47999999999996, "text": " Okay. So first of all, we can see here"}, {"start": 302.28, "end": 307.96, "text": " That the bloom folks come up with the same types of scaling laws here"}, {"start": 307.96, "end": 313.56, "text": " So we have the loss on the y-axis and we have the compute in the petaflop days on the x-axis"}, {"start": 313.56, "end": 321.48, "text": " And we can basically see the very similar curves to what Kaplan at all from the scaling loss paper from open AI"}, {"start": 321.48, "end": 332.36, "text": " Kind of already showed us petaflop days by the way stands for so one petaflop day is sustaining one petaflop of compute for a whole day"}, {"start": 332.36, "end": 335.72, "text": " So that that many operations, okay, so"}, {"start": 336.36, "end": 339.40000000000003, "text": " Let me kind of go through here. This is gonna be"}, {"start": 339.96000000000004, "end": 341.96000000000004, "text": " Basically a recurring theme"}, {"start": 342.52000000000004, "end": 347.48, "text": " Everyone all of these three papers literally took inspiration from the GPT-3 model"}, {"start": 347.48, "end": 354.44, "text": " So you can see here they say we follow the architecture and hyper parameters of GPT-3 model"}, {"start": 354.44, "end": 358.12, "text": " That's going to be like a commonality between all of these three papers"}, {"start": 358.92, "end": 360.92, "text": " So let's continue here"}, {"start": 361.48, "end": 366.92, "text": " They say we base this model size on Kaplan at all. So that's the the scaling loss paper"}, {"start": 366.92, "end": 370.84000000000003, "text": " However, after this paper was accepted for publication, but before it came out"}, {"start": 370.84, "end": 377.79999999999995, "text": " Huffman at all provided an alternative approach for selecting the model size. So that's the chinchilla paper from DeepMind"}, {"start": 377.79999999999995, "end": 383.15999999999997, "text": " I'm going to quickly just show you the the graphs for those of you who are not familiar with that with the scaling loss paper"}, {"start": 383.15999999999997, "end": 388.84, "text": " I strongly recommend you at least skim it. It's a very interesting finding and it's also very"}, {"start": 389.64, "end": 391.64, "text": " empirical and very"}, {"start": 392.2, "end": 397.55999999999995, "text": " It's over fitted to a particular type of a transformer. So here you can see what the the basic"}, {"start": 397.56, "end": 403.88, "text": " like postulate behind the scaling laws is is if you you can see on the x-axis if we increase compute"}, {"start": 403.88, "end": 410.2, "text": " so here on the on the on this left diagram here we have compute and then we have"}, {"start": 411.48, "end": 414.84000000000003, "text": " data set size here and finally we have"}, {"start": 415.88, "end": 419.64, "text": " Number of parameters. So if you increase either three of those"}, {"start": 420.2, "end": 424.04, "text": " You can see that the loss on the y-axis here the test loss is"}, {"start": 424.04, "end": 428.92, "text": " decreasing and we don't see any sign of situation here"}, {"start": 428.92, "end": 434.28000000000003, "text": " So that's the that's that was the paper that kind of showed us that we just need to scale the models"}, {"start": 434.92, "end": 440.28000000000003, "text": " And we'll get more and more performance now. It's a completely different topic and problematic whether"}, {"start": 441.64000000000004, "end": 443.64000000000004, "text": " like having a smaller loss"}, {"start": 444.36, "end": 450.44, "text": " Like perplexity actually correlates with the metrics we actually care about but that turns usually to be the case"}, {"start": 450.44, "end": 455.8, "text": " but yeah, okay, so one thing that the this paper the scaling laws paper showed is"}, {"start": 457.08, "end": 462.52, "text": " This one so they say that the the performance penalty depends predictably on the ratio"}, {"start": 463.08, "end": 467.08, "text": " You can see this one here meaning that every time we increase the model size"}, {"start": 467.72, "end": 472.84, "text": " Atex we need we only need to increase the data but roughly five times to avoid a penalty"}, {"start": 472.84, "end": 477.48, "text": " So basically to avoid the over fitting and so this is in starting point"}, {"start": 477.48, "end": 482.76, "text": " Contrast with the chinchilla paper. So what I say here is that you basically need to"}, {"start": 484.28000000000003, "end": 488.84000000000003, "text": " Increase the number of tokens of your training data set much slower"}, {"start": 489.40000000000003, "end": 492.12, "text": " According to this relationship you can see on the screen"}, {"start": 493.08000000000004, "end": 499.72, "text": " According to this formula here compared to the number of parameters and then the chinchilla paper actually showed a completely different"}, {"start": 499.72, "end": 504.68, "text": " Finding and that's that so they say specifically given a 10x increase in computational budget"}, {"start": 504.68, "end": 507.56, "text": " They suggest that the size of the model should increase"}, {"start": 508.28000000000003, "end": 509.40000000000003, "text": " 5.5"}, {"start": 509.40000000000003, "end": 514.84, "text": " By day they refer to the kaplan paper while the number of training tokens should only increase"}, {"start": 515.4, "end": 517.0, "text": " 1.8x"}, {"start": 517.0, "end": 521.8, "text": " Instead we find that the model size so this is the important finding of this paper the model size"}, {"start": 522.28, "end": 524.28, "text": " and the number of training"}, {"start": 524.84, "end": 527.32, "text": " Tokens should be scaled in equal"}, {"start": 528.36, "end": 530.2, "text": " proportions, okay"}, {"start": 530.2, "end": 535.8000000000001, "text": " So the problem is there is um, no no strong theoretical underpinning"}, {"start": 536.5200000000001, "end": 543.72, "text": " Behind these and and like who knows how how these uh laws will and whether they will be valid for for different"}, {"start": 544.2800000000001, "end": 551.1600000000001, "text": " Configurations of transformers if you change something if you start using I don't know maybe a libby instead of relative positional encodings"}, {"start": 551.48, "end": 555.48, "text": " Maybe then the laws change and we are not sure how sensitive they are"}, {"start": 555.48, "end": 563.0, "text": " Um, I might be wrong. Maybe the chinchilla paper has like some stronger theoretical insights, but that's i'm recalling from my memory"}, {"start": 563.8000000000001, "end": 566.44, "text": " Okay, so let's get back to the paper here"}, {"start": 566.9200000000001, "end": 573.0, "text": " So the first thing they show is that obviously it's very important to have high quality data set and by high quality"}, {"start": 573.8000000000001, "end": 578.84, "text": " They mean like so so having these diverse cross-domain high quality data"}, {"start": 579.4, "end": 581.88, "text": " Such as the pile data set so that's the data set that we have here"}, {"start": 581.88, "end": 585.08, "text": " And we'll see that their paper a bit later"}, {"start": 585.72, "end": 591.16, "text": " And you can see that using the the pile compared to these other data sets such as the oscar or c4"}, {"start": 591.96, "end": 598.68, "text": " Gives them basically better better accuracy on on these various nlp tasks. You can see that the blue model here"}, {"start": 599.24, "end": 603.64, "text": " Is on pair and even a bit better compared to the eluthorized paper"}, {"start": 604.2, "end": 606.6, "text": " The gpt neo paper is the same as the"}, {"start": 606.6, "end": 615.32, "text": " The gpt neo x20b and you can also see that basically the the open eyes models to outperform them by by some amount"}, {"start": 615.88, "end": 618.36, "text": " So that's that's um worth noticing"}, {"start": 619.5600000000001, "end": 621.5600000000001, "text": " Okay, guys, so let's continue here"}, {"start": 622.28, "end": 628.84, "text": " Let's see. What else is interesting in this paper? Um, so as I said diverse cross-domain pre-training data combining web calls with high"}, {"start": 629.4, "end": 632.12, "text": " With curated data and with high quality data"}, {"start": 632.12, "end": 639.4, "text": " So as I said diverse cross-domain pre-training data combining web calls with high with curated high quality sources significantly proves zero shot"}, {"start": 639.86, "end": 643.5600000000001, "text": " Generalization over pre-trained data sets constructed from common crawl only"}, {"start": 644.52, "end": 650.52, "text": " Okay, that's one finding the second finding they had was that although learned positional embeddings help perform rotary embeddings"}, {"start": 650.76, "end": 655.0, "text": " Alibi yields significant better results than all the other alternatives now"}, {"start": 655.0, "end": 659.96, "text": " This might be as well very very specific to their particular set of high parameters in architecture"}, {"start": 659.96, "end": 663.08, "text": " I wouldn't like take anything fundamental out of this statement"}, {"start": 663.08, "end": 667.32, "text": " But um, yeah, i'm just kind of walking you through some of the findings behind behind the papers, by the way"}, {"start": 667.5600000000001, "end": 669.5600000000001, "text": " I did cover the alibi paper"}, {"start": 670.2, "end": 672.6, "text": " I think like eight months ago or something when it came out"}, {"start": 673.0, "end": 677.72, "text": " So do check it out if you're curious to know more and we'll see what the rotary embeddings if you're not familiar"}, {"start": 677.8000000000001, "end": 682.0400000000001, "text": " Um, what those are? Um a bit later in the in the gpt neo x paper"}, {"start": 683.1600000000001, "end": 687.88, "text": " Okay, some of the other findings they had was they tried a bunch of different activation functions"}, {"start": 687.88, "end": 694.52, "text": " And then they say here we present our results in table three bubble us swigloo produces slightly better results than jellu"}, {"start": 694.84, "end": 699.4, "text": " However, this comes at a cost of reducing the throughput by approximately a third"}, {"start": 699.56, "end": 703.56, "text": " So I think they ultimately decided not to use it because of the throughput"}, {"start": 704.2, "end": 705.08, "text": " Okay"}, {"start": 705.08, "end": 710.68, "text": " So that's that's one more finding and then they they they mention how the and we already know this"}, {"start": 711.24, "end": 713.88, "text": " basically, it very it matters a lot where you put the"}, {"start": 713.88, "end": 721.16, "text": " Uh layer normalization and how many layer normalization layers do you include in your transformer blocks and and they say here adding layer"}, {"start": 721.48, "end": 726.52, "text": " Normalization after the embedding layer incurs a significant penalty on zero shot generalization"}, {"start": 726.84, "end": 730.12, "text": " But the thing is when you scale the model to very big sizes"}, {"start": 730.6, "end": 733.64, "text": " Uh, then this thing helps stabilize the training"}, {"start": 733.96, "end": 737.24, "text": " Uh, and when you're on the when you're dealing with lower scales"}, {"start": 737.24, "end": 743.64, "text": " Uh in that case, you probably don't want to include it because you'll be sacrificing as you can see here that the zero shot generalization"}, {"start": 744.6, "end": 746.6, "text": " Okay, guys, let's continue here"}, {"start": 747.08, "end": 751.24, "text": " One thing that I kind of noted to myself as well. I was kind of"}, {"start": 752.12, "end": 756.04, "text": " This is a curious finding. Uh, so they say instead of manually writing"}, {"start": 756.2, "end": 759.48, "text": " So this is about the multilingual, uh, basically evaluation"}, {"start": 759.64, "end": 763.0, "text": " So trying to evaluate on multiple different human languages"}, {"start": 763.0, "end": 767.8, "text": " And they say instead of manually writing prompts for each language, which would obviously be very bothersome"}, {"start": 768.2, "end": 773.48, "text": " We followed the strategy proposed by linetal using english prompts for non-english examples"}, {"start": 773.88, "end": 777.48, "text": " This can be viewed as cross-lingual zero shot generalization"}, {"start": 778.04, "end": 783.64, "text": " Uh, they validated this strategy by demonstrating its ability to achieve zero shot performance on pair"}, {"start": 784.12, "end": 790.04, "text": " With and sometimes even better than human written language specific prompts. This is kind of cool. So you literally take a"}, {"start": 790.04, "end": 796.68, "text": " uh, um, you you have data sets with various different languages and you always take the english prompt and"}, {"start": 796.92, "end": 798.92, "text": " then you concatenate the the"}, {"start": 799.0799999999999, "end": 806.4399999999999, "text": " Uh, the answer in that different language you repeat it a couple of times and then you evaluate and it's and it's apparently the model"}, {"start": 806.5999999999999, "end": 812.5999999999999, "text": " Like learn to to understand, uh this which kind of makes sense because you're training the disjoint"}, {"start": 812.92, "end": 817.24, "text": " Uh, like a space of multi disjoint multilingual space"}, {"start": 817.24, "end": 820.6, "text": " Uh when you're training transformer on on on diverse data sets"}, {"start": 821.4, "end": 826.6, "text": " And we always always know that the machine translation kind of emerges as a consequence of this"}, {"start": 827.16, "end": 834.36, "text": " Next token prediction, uh, if you give it enough big data sets with with some presence of different languages"}, {"start": 835.5600000000001, "end": 840.12, "text": " So thing worth mentioning and this is something we're probably all aware of at this point"}, {"start": 840.2, "end": 842.2, "text": " And that's that the multilingual pre-training"}, {"start": 842.2, "end": 849.4000000000001, "text": " Very significantly diminishes english zero shot generalization. So the thing with this bloom model is that they trained"}, {"start": 849.48, "end": 853.8000000000001, "text": " Uh, this is uh, like a multilingual model and because of that it's not as strong"}, {"start": 854.2800000000001, "end": 858.2, "text": " Uh on on on english as as uh, basically some of the other"}, {"start": 859.0, "end": 860.5200000000001, "text": " baselines"}, {"start": 860.5200000000001, "end": 862.9200000000001, "text": " Okay, let's continue here. Um"}, {"start": 863.88, "end": 869.4000000000001, "text": " A couple things worth mentioning, this is how they've chosen the final configuration of the bloom"}, {"start": 869.4, "end": 872.9399999999999, "text": " 176 billion parameter model"}, {"start": 873.5799999999999, "end": 878.14, "text": " So they they reference here the the the the scaling loss paper"}, {"start": 878.22, "end": 883.74, "text": " It is notable that this pareto optimal frontier describes very large models trained on few tokens"}, {"start": 883.98, "end": 885.98, "text": " We call this training to optimality"}, {"start": 885.9, "end": 891.8199999999999, "text": " So that's basically what I said before basically you want to the the scaling loss paper show that you want to increase the parameters more"}, {"start": 892.14, "end": 898.54, "text": " Than the number of tokens of your data set and then they say this is in stark contrast with the common practice of training much smaller"}, {"start": 898.54, "end": 901.18, "text": " Models on many more tokens to convergence. Okay"}, {"start": 901.98, "end": 903.98, "text": " And now because they're training"}, {"start": 905.02, "end": 906.38, "text": " They have a couple of arguments here"}, {"start": 906.38, "end": 910.2199999999999, "text": " But one of them is that they're training on a multilingual data set and because of that"}, {"start": 910.3, "end": 914.4599999999999, "text": " They they don't want to strictly follow the the scaling loss. And so this is what they do"}, {"start": 914.54, "end": 918.4599999999999, "text": " So we choose to use the optimality front as an upper bound on model size"}, {"start": 918.6999999999999, "end": 920.62, "text": " So that's like the biggest model size"}, {"start": 921.26, "end": 927.66, "text": " There will be determined by the scaling loss paper and the lower lower bound on number of training tokens"}, {"start": 927.66, "end": 930.4599999999999, "text": " So they'll probably have smaller model and more tokens"}, {"start": 930.78, "end": 937.9, "text": " Okay compared to what the scaling loss paper suggests the second very influential paper that had impact on on the"}, {"start": 938.62, "end": 943.98, "text": " Like modeling decisions in this paper is this levin et al paper"}, {"start": 944.06, "end": 951.74, "text": " So they proposed a theoretically motivated and empirically backed law describing the optimal compromise compromise between width and height"}, {"start": 951.74, "end": 958.38, "text": " Within depth, sorry. So for a gpt3 size model with 175 billion parameters"}, {"start": 958.46, "end": 961.58, "text": " They predict an ideal depth of 80 layers. Okay"}, {"start": 962.38, "end": 964.38, "text": " so now it's just basically"}, {"start": 964.54, "end": 969.66, "text": " Like combining lego bricks you have some constraints from the scaling loss paper"}, {"start": 969.66, "end": 974.38, "text": " You have some constraints from this paper and then given the compute they have and that's"}, {"start": 974.94, "end": 978.3, "text": " Roughly 1 million gp hours on a 100 machines"}, {"start": 978.3, "end": 983.66, "text": " They decide on the on the on the on the hyper parameters of the model. So let's let's see what those are"}, {"start": 983.66, "end": 987.02, "text": " So they they lose this many tokens as you can see here"}, {"start": 987.66, "end": 989.66, "text": " 70 to 80 layers"}, {"start": 989.66, "end": 991.66, "text": " because of this paper and"}, {"start": 992.62, "end": 996.2199999999999, "text": " Et cetera, et cetera. So finally they end up with three"}, {"start": 996.8599999999999, "end": 998.78, "text": " best configurations"}, {"start": 998.78, "end": 1005.0999999999999, "text": " And the one they pick at the end is this one here. So they they pick this one with 176 billion"}, {"start": 1005.66, "end": 1007.66, "text": " parameters 70 layers"}, {"start": 1007.66, "end": 1008.9399999999999, "text": " this many"}, {"start": 1008.9399999999999, "end": 1010.4599999999999, "text": " hidden dimensions"}, {"start": 1010.4599999999999, "end": 1011.98, "text": " attention hats"}, {"start": 1011.98, "end": 1018.14, "text": " Dimensionality of attention hats. This is the memory footprint and the performance. Okay, so that's pretty much it"}, {"start": 1018.2199999999999, "end": 1023.8199999999999, "text": " That's the first paper. Let me get kind of show you these numbers side by side in the actual code base"}, {"start": 1024.22, "end": 1026.94, "text": " And then we'll switch to the second paper. Okay guys, here it is"}, {"start": 1027.26, "end": 1031.8999999999999, "text": " And as I said, i'll i'll do a walkthrough of of the code base. So basically don't worry"}, {"start": 1031.98, "end": 1035.26, "text": " I'll i'll show you this code in much more depth later on"}, {"start": 1035.26, "end": 1039.42, "text": " Uh, I just want to kind of briefly show you so this is the big science workshop"}, {"start": 1039.98, "end": 1044.22, "text": " Repo and you can see the configuration for the 176 billion model"}, {"start": 1044.94, "end": 1048.54, "text": " And you can see here the numbers are exactly what you see here in the table. So we have"}, {"start": 1049.48, "end": 1053.58, "text": " 176 layers so you can see that somewhere here. Let me try"}, {"start": 1054.3, "end": 1059.26, "text": " Nope, sorry, that's the number of uh, that's the number of parameters. Uh, so we have the 70 layers"}, {"start": 1059.42, "end": 1062.14, "text": " So here is the the variable for 70 layers. We have"}, {"start": 1062.14, "end": 1068.72, "text": " 14336 you can see that number of hidden is is exactly that we have number of hats"}, {"start": 1069.4, "end": 1074.24, "text": " 112 etc, etc. So basically yeah, this paper shows you how how they've"}, {"start": 1075.0400000000002, "end": 1079.1200000000001, "text": " Decided to to pick the uh, all of the hyper parameters of the model"}, {"start": 1080.0, "end": 1086.0800000000002, "text": " Some of that was inspired by the gpt3 paper. Some of that was inspired by the scaling loss paper. And finally, uh, the"}, {"start": 1086.08, "end": 1091.76, "text": " The levin et al paper had a had an impact as well on the on the width and depth now"}, {"start": 1091.76, "end": 1094.6399999999999, "text": " I'm gonna quickly walk you through the chronicles they created"}, {"start": 1095.6799999999998, "end": 1097.76, "text": " So basically, uh, i'm gonna start here"}, {"start": 1097.84, "end": 1104.8799999999999, "text": " So first of all one more thing that's kind of common across all of the three papers is that all of them are using the nvdm"}, {"start": 1104.8799999999999, "end": 1110.1599999999999, "text": " Megatron library that helps you do model parallelism do check out my scaling"}, {"start": 1110.32, "end": 1114.0, "text": " Uh, like ultimate guide to scaling video where I cover a couple of these"}, {"start": 1114.0, "end": 1116.0, "text": " uh influential scaling papers"}, {"start": 1116.86, "end": 1118.38, "text": " including megatron"}, {"start": 1118.38, "end": 1123.22, "text": " The second thing they they they all have in common is the deep speed library"}, {"start": 1123.58, "end": 1125.82, "text": " Again, I did cover that one in my previous video"}, {"start": 1125.82, "end": 1131.42, "text": " So do check it out and the bloom folks because they had 384 gpus at their disposal"}, {"start": 1132.78, "end": 1134.78, "text": " basically decided to to"}, {"start": 1136.22, "end": 1139.02, "text": " Arrange those gpus in this 3d"}, {"start": 1139.02, "end": 1145.28, "text": " Fashion so the so-called 3d parallelism where one axis is basically pipeline parallelism"}, {"start": 1145.5, "end": 1150.06, "text": " Where you break the model such that you take a couple of first layers you put them on one gpu"}, {"start": 1150.1399999999999, "end": 1153.98, "text": " Then you take the next uh end layers you put them on the second one, etc, etc"}, {"start": 1154.3799999999999, "end": 1157.42, "text": " Then you have the model parallelism where you break the actual layers"}, {"start": 1157.9, "end": 1158.7, "text": " uh"}, {"start": 1158.7, "end": 1164.06, "text": " And then kind of take the weights of that layer and put one part on one gpu the second part on the second one"}, {"start": 1164.06, "end": 1165.02, "text": " Etc, etc"}, {"start": 1165.02, "end": 1170.06, "text": " And the third dimension is the the data parallel dimension here where you basically replicate"}, {"start": 1170.46, "end": 1175.9, "text": " Your models across different gpus and then you feed them different portions of your of your input patch"}, {"start": 1176.54, "end": 1181.18, "text": " Okay, next up. I want to walk you through some of the chronicles. Uh, they've shared"}, {"start": 1181.42, "end": 1186.3799999999999, "text": " So there is this document called lessons learned. I link the repo down in the description"}, {"start": 1186.62, "end": 1187.5, "text": " So you can check it out"}, {"start": 1187.5, "end": 1192.46, "text": " You can go I strongly encourage you to go through uh, at least skim through all of these documents"}, {"start": 1192.46, "end": 1198.7, "text": " I think it's a valuable learning experience. So a couple of things so they have this title how training divergences were"}, {"start": 1199.42, "end": 1206.6200000000001, "text": " Overcome so how they overcame the the divergences one of the the methods was to just have a different initialization method"}, {"start": 1207.02, "end": 1211.18, "text": " And it literally means so much like I think they mentioned it somewhere"}, {"start": 1211.26, "end": 1216.3, "text": " So it has made a huge difference to the training stability and not only stability but the the final"}, {"start": 1216.3, "end": 1222.3799999999999, "text": " Uh actual loss you you converge to and I remember back from my days when I trained the transformer model"}, {"start": 1222.54, "end": 1224.54, "text": " So I had the original"}, {"start": 1224.7, "end": 1226.7, "text": " transformer implementation right here"}, {"start": 1227.4199999999998, "end": 1228.76, "text": " and uh"}, {"start": 1228.76, "end": 1233.8999999999999, "text": " Initialization blah blah. I remember this this diagram here. So here's what I said here"}, {"start": 1233.8999999999999, "end": 1238.7, "text": " So initialization matters a lot for transformers. I initially thought that the other implementation using savir"}, {"start": 1239.18, "end": 1245.18, "text": " Uh initialization is again one of those arbitrary heuristics and the pytorch default in it will do I was wrong"}, {"start": 1245.18, "end": 1251.5800000000002, "text": " You can see here three curves that these two are using pytorch default and this one is using the xavier initialization"}, {"start": 1251.66, "end": 1258.6200000000001, "text": " So that was the moment for me when I realized how sensitive the models are to initialization and how vital that is"}, {"start": 1259.02, "end": 1261.02, "text": " Okay getting back to the"}, {"start": 1261.66, "end": 1263.66, "text": " lessons learned document here"}, {"start": 1264.14, "end": 1266.38, "text": " adding embedding layer norms, so"}, {"start": 1267.1000000000001, "end": 1273.18, "text": " Hopefully we're all aware of this basically the position whether it's before there is a jollo connection or after"}, {"start": 1273.18, "end": 1275.18, "text": " How many of them do you include etc?"}, {"start": 1275.18, "end": 1275.5800000000002, "text": " etc"}, {"start": 1275.5800000000002, "end": 1279.98, "text": " All of that matters a lot for the final stability and finally patience"}, {"start": 1280.22, "end": 1284.7, "text": " So in some cases in so they say here in some cases in the case of a huge spike"}, {"start": 1285.1000000000001, "end": 1288.46, "text": " It was taking 2k iterations for training to return to the same"}, {"start": 1289.1000000000001, "end": 1293.3400000000001, "text": " Loss it spiked from and then it continued training as if nothing happened"}, {"start": 1293.66, "end": 1296.46, "text": " So sometimes in order to overcome the spike you just have to wait"}, {"start": 1296.94, "end": 1300.38, "text": " Uh, but more often than not the training won't recover from a spike"}, {"start": 1300.38, "end": 1306.8600000000001, "text": " Yet in another situation as the training diverged slowly without any spikes. Okay, so let me show you the spikes"}, {"start": 1306.8600000000001, "end": 1308.3000000000002, "text": " so here is the"}, {"start": 1308.3000000000002, "end": 1315.2600000000002, "text": " Uh, this is probably the most interesting chronicle i've seen so the the the way they've trained the 104 billion parameter model"}, {"start": 1315.66, "end": 1321.66, "text": " They had the most problems with this model and then actually with the 176b model"}, {"start": 1322.0600000000002, "end": 1326.94, "text": " It was the training was much more stable and there are many reasons why that was first of all"}, {"start": 1326.94, "end": 1333.26, "text": " They I guess they learned a lot during during this this whole trial and then the second thing is I think like they use b"}, {"start": 1333.26, "end": 1341.8200000000002, "text": " Float so the brain float format instead of the the the the usual float 16 and all of that kind of brought the additional stability"}, {"start": 1342.38, "end": 1344.38, "text": " So I just want to kind of quickly skim through"}, {"start": 1345.02, "end": 1347.98, "text": " Some of the diagrams here you can see that the problems they had with spikes"}, {"start": 1348.22, "end": 1352.3, "text": " So during the training, uh, the spike would appear and then the training would diverge"}, {"start": 1352.3, "end": 1358.86, "text": " And they had a lot of experiments. They tried a bunch of things to try to to kind of cope with those spikes"}, {"start": 1358.86, "end": 1363.8999999999999, "text": " But they kept getting the spikes. They kept getting the spikes like throughout many many"}, {"start": 1364.46, "end": 1369.18, "text": " like experiments and many many weeks they constantly they couldn't get the model to"}, {"start": 1369.82, "end": 1375.1, "text": " Uh to converge it was constantly diverging and then I think only at the end"}, {"start": 1375.5, "end": 1378.78, "text": " Did they start getting some better results after multiple months?"}, {"start": 1378.78, "end": 1384.78, "text": " Um, i'll show you the the the 176b model a bit later. So let me show you one more thing. See here"}, {"start": 1385.02, "end": 1390.06, "text": " This is interesting. So the they had two million backslash only samples in the data set"}, {"start": 1390.54, "end": 1391.98, "text": " um, so"}, {"start": 1391.98, "end": 1392.86, "text": " Hugh"}, {"start": 1392.86, "end": 1394.94, "text": " Negan, I don't know how to pronounce this surname"}, {"start": 1395.58, "end": 1400.22, "text": " Discovered that there are huge records with just backslashes in oscar"}, {"start": 1400.7, "end": 1403.82, "text": " English version so we set out to look at their occurrence"}, {"start": 1403.82, "end": 1408.3799999999999, "text": " And so what happens when when your model basically encounters this sequence?"}, {"start": 1408.78, "end": 1412.78, "text": " Uh, it can produce the the the spiking behavior the divergence"}, {"start": 1412.86, "end": 1419.5, "text": " So all of these details are this is the this is the messy reality of training of training large language models"}, {"start": 1419.5, "end": 1421.5, "text": " And it's very nice that they've shared this"}, {"start": 1421.82, "end": 1425.02, "text": " Okay, I did mention that the 176b was much more stable"}, {"start": 1425.26, "end": 1430.78, "text": " They literally say here what makes the the model so so stable and they compare so here is the 104"}, {"start": 1431.1, "end": 1433.1, "text": " Uh billion parameter model you can see"}, {"start": 1433.1, "end": 1438.86, "text": " Uh bunch of curves, uh superposition here and you can see the the the 176b"}, {"start": 1438.9399999999998, "end": 1443.74, "text": " So it's literally like a smooth curve converging and the loss is lowering as expected"}, {"start": 1444.3, "end": 1450.6999999999998, "text": " So, uh, here is the kind of the overlapped image and then they say it's a combination of probably a few of these improvements"}, {"start": 1450.6999999999998, "end": 1456.3799999999999, "text": " It's kind of hard to pinpoint a particular thing, but I I have a strong hunch that the bfloat"}, {"start": 1456.38, "end": 1464.38, "text": " Uh format is what had a very very significant impact here. So very clean data. They have better data set for the for the big model"}, {"start": 1464.7, "end": 1467.3400000000001, "text": " They had the bfloat, uh mixed precision regime"}, {"start": 1468.5400000000002, "end": 1473.68, "text": " The fp32 accumulation for the pipeline blah blah blah very low initialization"}, {"start": 1475.18, "end": 1477.18, "text": " Word embedding layer norm"}, {"start": 1477.9, "end": 1481.5800000000002, "text": " But it was also using okay, so then I guess that doesn't doesn't explain it"}, {"start": 1482.14, "end": 1483.98, "text": " Uh, okay, that's guys. That's pretty much it"}, {"start": 1483.98, "end": 1487.74, "text": " They also have this document where they show, uh how they decided to"}, {"start": 1488.38, "end": 1494.94, "text": " To pick the parameters of the 3d parallelism. So basically they decide how they decided on on picking the tensor parallelism"}, {"start": 1495.48, "end": 1502.14, "text": " Parameter the pipeline parallelism the data parallelism all of those details so you can kind of walk through this document at your own pace"}, {"start": 1502.22, "end": 1506.22, "text": " It's too long for me to kind of extract any any particular insight here"}, {"start": 1507.02, "end": 1511.02, "text": " But that's pretty much it. Okay guys, so now let's go to the second paper"}, {"start": 1511.02, "end": 1513.66, "text": " I'm gonna open up the um"}, {"start": 1514.94, "end": 1521.42, "text": " The opt paper and let's start from the beginning here. So open pretrained transformer language models"}, {"start": 1521.5, "end": 1528.3, "text": " So these guys had a lot of problems training their models like reading through the the the logs was was such a such a joy"}, {"start": 1529.5, "end": 1533.26, "text": " So basically let me kind of show you a couple of snippets from a huge document"}, {"start": 1533.26, "end": 1536.7, "text": " They have like there is like 140 pages or something of their logs"}, {"start": 1536.7, "end": 1543.66, "text": " Internal logs, that's very cool. They shared those so here here are a couple of those once we identify the bad node"}, {"start": 1543.82, "end": 1551.66, "text": " We need to document these lists of things to complain to csp and get our money back. Give us our money back you bastards"}, {"start": 1552.46, "end": 1557.74, "text": " Okay, it looks like 26 tried to immediately upload the checkpoint and failed its cp commands"}, {"start": 1557.98, "end": 1562.94, "text": " Then it took another step lowered its scalar and tried uploading again and again the humanity"}, {"start": 1562.94, "end": 1567.76, "text": " We are already at loss scale 0.25 and so on and so on like oh hallelujah"}, {"start": 1568.46, "end": 1571.8200000000002, "text": " Things looked good, but nope just kind of complete confusion"}, {"start": 1572.46, "end": 1578.6200000000001, "text": " So I again encourage you to just kind of skim through the the log book. It's a very interesting read"}, {"start": 1579.5, "end": 1581.5, "text": " Okay, let's start with this one"}, {"start": 1582.8600000000001, "end": 1586.22, "text": " First of all our models and hyper parameters largely follow brown at all"}, {"start": 1586.22, "end": 1593.18, "text": " So that's the gpt3 paper again with variations in batch size mostly to obtain increased computational efficiency"}, {"start": 1593.18, "end": 1595.58, "text": " So this is something I did mention in the previous paper as well"}, {"start": 1595.74, "end": 1600.8600000000001, "text": " Oops, so this is kind of a common thread for all of these three papers"}, {"start": 1601.18, "end": 1605.84, "text": " They all follow the gpt3 hyper parameters and and uh advices"}, {"start": 1607.58, "end": 1615.02, "text": " Next up they say we found that the pile was particularly full of duplicate documents and advise future researchers using the pile to perform"}, {"start": 1615.02, "end": 1622.24, "text": " additional deduplication processing this is in particular relevant to the next paper i'll cover from eluthor ai"}, {"start": 1622.48, "end": 1627.36, "text": " Because they did use pile and they did not duplicate it because of the lack of resources"}, {"start": 1627.68, "end": 1634.24, "text": " And they also which is very interesting. They've done more than a single epoch, which is the common practice training lms"}, {"start": 1635.12, "end": 1637.52, "text": " But we'll get there in a couple of minutes"}, {"start": 1638.32, "end": 1641.2, "text": " a couple of interesting curves again, um, you can see"}, {"start": 1641.2, "end": 1648.72, "text": " Iterations on the x-axis learning rate on the y-axis. You can see that this was fairly ad hoc like basically the model would crash"}, {"start": 1648.72, "end": 1655.52, "text": " They would retrieve the the older checkpoint and then lower the low the the learning rate such that in order to avoid"}, {"start": 1655.52, "end": 1661.68, "text": " Try to avoid the instabilities and as a consequence of that you get this piecewise weird graph here"}, {"start": 1662.48, "end": 1665.28, "text": " Okay, you can also see the the validation perplexity"}, {"start": 1665.92, "end": 1669.8400000000001, "text": " Has a lot of these bumps and uh, it looks very funny"}, {"start": 1669.84, "end": 1673.36, "text": " Uh, so here is the part, uh, that's uh joy to read"}, {"start": 1674.08, "end": 1677.6, "text": " Basically a lot of a lot of problems during during the training"}, {"start": 1677.84, "end": 1682.3999999999999, "text": " So the the thing to mention in metals defense is that they literally had only a couple of months to do this"}, {"start": 1682.72, "end": 1688.48, "text": " It was a very tight deadline and so yeah, they didn't have the time to to do this properly"}, {"start": 1689.04, "end": 1695.28, "text": " But here is a couple of insights from from the paper. So in total hardware failures contributed to at least"}, {"start": 1695.28, "end": 1700.8, "text": " 35 manual restarts and the cycling of over 100 hosts over the course of two months"}, {"start": 1701.6, "end": 1705.68, "text": " Given the difference between the number of hosts cycled out and the number of manual restarts"}, {"start": 1705.68, "end": 1713.2, "text": " We estimate 70 plus automatic restarts due to hardware failures. I guess everything that will that can break will break the marfais law thing"}, {"start": 1714.08, "end": 1721.76, "text": " Okay, we noticed a correlation between loss divergence our dynamic loss scalar crashing to zero and the l2 norm of the activations"}, {"start": 1721.76, "end": 1729.28, "text": " Of the final layer spiking these observations led us to pick restart points for which our dynamic loss scalar was still in a healthy state"}, {"start": 1729.28, "end": 1736.0, "text": " So bigger or equal than one and after which our activation norms would trend downward instead of growing"}, {"start": 1736.4, "end": 1741.04, "text": " Unboundedly, so this is just some detail about how they were handling the um, how"}, {"start": 1741.76, "end": 1747.36, "text": " What was the heuristic they were using to to pick the best checkpoint in order to avoid the the future spikes?"}, {"start": 1747.36, "end": 1755.28, "text": " Okay, guys, uh, let's continue here. So the results here as you can see on 14 nlp tasks averaged for the zero shot"}, {"start": 1755.84, "end": 1760.8799999999999, "text": " Evaluation their own pair with gpt. That's very cool on the other side for the multi-shot"}, {"start": 1761.4399999999998, "end": 1769.52, "text": " Evaluation where it does like one or 32 shots. You can see that the uh, the dashed line the gpt model did outperform"}, {"start": 1770.32, "end": 1772.7199999999998, "text": " Them on on on those tasks. So that means"}, {"start": 1772.72, "end": 1777.68, "text": " It's kind of better few shot learner than the op model and um, yeah"}, {"start": 1778.0, "end": 1780.96, "text": " They say that the trend follows the gpt3. That's what what I said"}, {"start": 1781.44, "end": 1787.6000000000001, "text": " So let's continue here a couple of more interesting details. First of all, um, hate speech detection. They showed that um"}, {"start": 1788.32, "end": 1794.48, "text": " That the 175b model the op model considerably outperforms da vinci in all settings"}, {"start": 1794.48, "end": 1795.6000000000001, "text": " So that's very cool"}, {"start": 1795.6000000000001, "end": 1798.32, "text": " And then you think about it and this is all with the"}, {"start": 1798.32, "end": 1804.56, "text": " Double edged sword. So if it's better at detection, that means it's probably more susceptible to generating toxic"}, {"start": 1804.8799999999999, "end": 1809.2, "text": " And hateful speech so and that's indeed the case. So let me show you that here"}, {"start": 1810.0, "end": 1813.04, "text": " And they mention it. So overall we see that op"}, {"start": 1813.6, "end": 1820.56, "text": " 175b has a higher toxicity rate than either palm or da vinci da vinci being the the gpt3 model"}, {"start": 1821.12, "end": 1823.9199999999998, "text": " That's the code name. Okay, so let's continue here"}, {"start": 1823.92, "end": 1828.88, "text": " Da vinci being the the gpt3 model. That's the code name. Okay, you can see here"}, {"start": 1829.92, "end": 1836.88, "text": " Prompt toxicity probability toxicity probability of continuation. You can see that the blue curve the op model has the highest toxicity"}, {"start": 1837.44, "end": 1841.76, "text": " Uh for for for across across the the skip across the whole axis here"}, {"start": 1842.5600000000002, "end": 1844.3200000000002, "text": " Okay, guys, um"}, {"start": 1844.3200000000002, "end": 1845.68, "text": " I think that's pretty much it"}, {"start": 1845.68, "end": 1854.0, "text": " They they kind of summarize saying that we still believe this technology is premature for commercial deployment. And yeah, I mean, uh understanding how these models work, uh,"}, {"start": 1854.0800000000002, "end": 1858.4, "text": " Just interpreting what's going on and why the model is up but in certain results all of that"}, {"start": 1858.4, "end": 1859.8400000000001, "text": " We are still in the early days"}, {"start": 1859.8400000000001, "end": 1865.3600000000001, "text": " We are much better at at getting raw numbers on benchmarks and then knowing how to handle and align"}, {"start": 1865.68, "end": 1871.52, "text": " These models with human preferences, but it's a topic for for a different video. I think that's pretty much it. That's the second paper"}, {"start": 1872.4, "end": 1874.4, "text": " As I said, i'm just quickly giving you"}, {"start": 1874.4, "end": 1879.8400000000001, "text": " Uh the the the overview here and then in the next videos i'll be doing the coding"}, {"start": 1880.4, "end": 1885.44, "text": " Series, okay. Um, yeah before this let's let me show you the the opt logbook"}, {"start": 1885.44, "end": 1890.64, "text": " So here's the the big logbook I mentioned this is the this contains a bunch of notes"}, {"start": 1891.52, "end": 1894.5600000000002, "text": " That they were kind of accumulating throughout this period of a couple months"}, {"start": 1894.72, "end": 1899.2, "text": " I encourage you to check it out at your own pace. You can skim it. It literally took me maybe"}, {"start": 1899.2, "end": 1904.0800000000002, "text": " 20 30 minutes to just skim through this and kind of extract some of the insights"}, {"start": 1905.1200000000001, "end": 1908.72, "text": " Okay, so I did show you this diagram in the beginning of the video"}, {"start": 1908.8, "end": 1914.96, "text": " So again, you can see the torque or the number of gaps here and how many failures they were experiencing"}, {"start": 1914.96, "end": 1920.8, "text": " So we saw the number 70 plus or something restarts. I don't want to count but yeah, this might be"}, {"start": 1921.76, "end": 1925.8400000000001, "text": " At least in that range. Okay. Finally, let's go to their final update"}, {"start": 1925.84, "end": 1929.6, "text": " Uh, they have a couple of things like some some insights here"}, {"start": 1930.0, "end": 1932.6399999999999, "text": " Um, let me pick a couple of them. So scaling to"}, {"start": 1933.26, "end": 1935.76, "text": " 1024 a 100 to handle a real"}, {"start": 1936.24, "end": 1941.1999999999998, "text": " Workload of this size is highly non-trivial. We will discuss infrastructure pain points below"}, {"start": 1941.4399999999998, "end": 1947.28, "text": " Okay, ensuring training converges at this scale is also highly non-trivial without sufficient ablations at medium scale"}, {"start": 1947.84, "end": 1952.9599999999998, "text": " Results obtained from training at small scale also do not necessarily hold when scaled up"}, {"start": 1952.96, "end": 1957.2, "text": " We will cover these learnings in a note to be released in the upcoming weeks. Okay"}, {"start": 1957.92, "end": 1961.68, "text": " So let me quickly read you the cluster deletion story. That's kind of funny"}, {"start": 1961.76, "end": 1967.1200000000001, "text": " So given the holidays we requested a pool of 12 buffer nodes to guard against hardware failures"}, {"start": 1967.8400000000001, "end": 1973.04, "text": " Two machines go down every day on average in the process of replenishing this pool"}, {"start": 1973.28, "end": 1978.8, "text": " The cloud provider support team accidentally deleted our entire cluster on december 21st"}, {"start": 1978.8, "end": 1985.84, "text": " First while the cluster was restored fairly quickly it unfortunately came back with 16 machines that did not pass our infrastructure checks"}, {"start": 1986.08, "end": 1988.08, "text": " Etc, etc. So that's one of the interesting parts"}, {"start": 1988.8, "end": 1994.08, "text": " Second interesting diagram here is this one. They showed that they didn't quite match the the loss"}, {"start": 1994.6399999999999, "end": 1999.36, "text": " Of the gpt3 model, but we did see that on the on various nlp tasks"}, {"start": 1999.6, "end": 2003.68, "text": " the zero shot performance of the model did reach gpt3 so that's again the"}, {"start": 2004.32, "end": 2006.32, "text": " the the discrepancy between"}, {"start": 2006.32, "end": 2012.72, "text": " Um, like the loss value the final loss value and the actual performance might not always be that clear"}, {"start": 2012.96, "end": 2018.24, "text": " Although we did see that we did saw that the gpt3 handles much better the few shot prompts"}, {"start": 2018.96, "end": 2025.84, "text": " And that might be because of lower of lower loss here again just a speculation, but that's usually the case"}, {"start": 2026.08, "end": 2029.6, "text": " That's it guys again encourage you to go through it at your own pace"}, {"start": 2029.84, "end": 2033.6799999999998, "text": " I'm going to continue on and explore the gpt neo x20b paper"}, {"start": 2033.68, "end": 2036.3200000000002, "text": " So an open source autoregressive language model"}, {"start": 2036.88, "end": 2040.16, "text": " And let's see what the insights here are"}, {"start": 2040.5600000000002, "end": 2045.68, "text": " Okay, so we trained on a data set that contains duplicated data for more than one epoch"}, {"start": 2045.76, "end": 2048.4, "text": " But see no evidence of performance loss"}, {"start": 2048.88, "end": 2053.76, "text": " While these authors claims that the few shot prompting doesn't improve performance on their task"}, {"start": 2054.0, "end": 2057.3, "text": " We find that this is actually a phenomenon unique to gpt3"}, {"start": 2057.3, "end": 2063.1400000000003, "text": " And does not apply to either gpt neo x20b or fair sec models. Okay, so"}, {"start": 2063.7000000000003, "end": 2067.7000000000003, "text": " that's just a note that uh, it's dangerous to just take a single model and"}, {"start": 2068.42, "end": 2070.7400000000002, "text": " Draw out extrapolate from from there"}, {"start": 2071.38, "end": 2073.94, "text": " Because other models might have different behavior"}, {"start": 2074.42, "end": 2076.5800000000004, "text": " so we'll see later in this paper that uh,"}, {"start": 2076.82, "end": 2083.6200000000003, "text": " The probable explanation is basically data set the underlying data set for that they use to train gpt neo x20b"}, {"start": 2084.34, "end": 2086.02, "text": " i.e the pile"}, {"start": 2086.02, "end": 2091.72, "text": " Is of different quality and distribution compared to the data sets that were used for gpt3"}, {"start": 2091.94, "end": 2093.94, "text": " We don't even know what data was used. So yeah"}, {"start": 2094.66, "end": 2095.94, "text": " Okay"}, {"start": 2095.94, "end": 2100.02, "text": " So again, they mentioned here that they largely followed the gpt3"}, {"start": 2100.02, "end": 2104.66, "text": " So so blah blah blah the coder model whose architecture largely follows that of gpt3"}, {"start": 2105.14, "end": 2112.02, "text": " With a few notable deviations described below. So that's that's again the common thread that I mentioned"}, {"start": 2112.58, "end": 2113.94, "text": " multiple times"}, {"start": 2113.94, "end": 2120.02, "text": " But this paper did have like, um, many more modifications compared to the previous ones we saw. Okay"}, {"start": 2120.7400000000002, "end": 2125.46, "text": " Before this paper was published and the ways were were released"}, {"start": 2126.1, "end": 2128.02, "text": " They had a couple of iterations"}, {"start": 2128.02, "end": 2131.94, "text": " Behind them. One of them was gpt neo and the second one"}, {"start": 2132.58, "end": 2139.48, "text": " Was gptj which was basically supported by the elutrii community but executed by ban wang and komatsuzaki"}, {"start": 2139.48, "end": 2145.96, "text": " So that's kind of worth mentioning. So the architecture of this paper closely follows the gptj paper"}, {"start": 2146.6, "end": 2152.12, "text": " So let's see what's the diff between those two? So the gptj and this one compared to gpt3"}, {"start": 2152.12, "end": 2154.62, "text": " So first of all, they use rotary embeddings"}, {"start": 2156.28, "end": 2161.16, "text": " I will not get into a lot of details how these work but like here is the here's your uh,"}, {"start": 2161.46, "end": 2168.12, "text": " classical formulation of the of this uh attention so you can see the formula here we have the uh"}, {"start": 2168.12, "end": 2172.68, "text": " Not token here. We map it into the query. So this is going to be the query"}, {"start": 2172.92, "end": 2177.08, "text": " We map this one into the key then we do the dot product blah blah blah"}, {"start": 2177.3199999999997, "end": 2181.3199999999997, "text": " And then the rest of the notation is kind of weird because it's not summation you want to have"}, {"start": 2182.04, "end": 2184.04, "text": " You want to have these?"}, {"start": 2184.3399999999997, "end": 2185.64, "text": " scores"}, {"start": 2185.64, "end": 2190.12, "text": " In separate spatial locations, you don't want to sum them up. So this is kind of weird notation"}, {"start": 2190.44, "end": 2196.12, "text": " Um, I mean looking at it. It's not even correct. Like you cannot just sum up these scores. It's kind of weird"}, {"start": 2196.12, "end": 2199.0, "text": " Uh, yeah, let me just make a diff here. So"}, {"start": 2199.72, "end": 2204.3599999999997, "text": " The the the difference with the rotary medics is you you have this this matrix r here"}, {"start": 2205.48, "end": 2210.6, "text": " Which basically rotates your intermediate representation. So here's the here's kind of the idea"}, {"start": 2210.6, "end": 2212.3599999999997, "text": " So if you have a key or a query"}, {"start": 2212.3599999999997, "end": 2217.96, "text": " So let's say we have a text here in hands transformer with rotary position embeddings and let's say that each of these words"}, {"start": 2218.2799999999997, "end": 2222.2799999999997, "text": " Has a corresponding uh vector here. Let's say this is um, either"}, {"start": 2222.28, "end": 2228.1200000000003, "text": " Uh a key or or a query vector, uh, and basically what I do is they kind of chunk"}, {"start": 2228.44, "end": 2231.0800000000004, "text": " Uh this vector into d over two"}, {"start": 2231.8, "end": 2238.6800000000003, "text": " Groups, uh, and that basically means that each of the groups is two-dimensional and then so once you have this chunk here"}, {"start": 2239.5600000000004, "end": 2246.28, "text": " What you then do is you literally rotate it depending on this position and depending on the on the on the group index"}, {"start": 2246.52, "end": 2249.2400000000002, "text": " So what I mean by that is so this is group one group two"}, {"start": 2249.24, "end": 2253.24, "text": " Uh group d over two so because this is from group one"}, {"start": 2253.64, "end": 2259.9599999999996, "text": " You basically apply the theta one angle and because it's position one you apply m and that's how you rotate"}, {"start": 2260.12, "end": 2267.3199999999997, "text": " And then you get the output representation here and that's the final representation here. So that's the modification that happens that happens"}, {"start": 2267.7999999999997, "end": 2269.7999999999997, "text": " Precisely here. Okay guys"}, {"start": 2269.9599999999996, "end": 2273.8799999999997, "text": " Um, so that's one of the modifications they've used the second one is this one"}, {"start": 2273.88, "end": 2280.76, "text": " We compute the attention and few forward layers in parallel and sum the results rather than running them in series"}, {"start": 2281.32, "end": 2282.2000000000003, "text": " Okay"}, {"start": 2282.2000000000003, "end": 2283.96, "text": " uh, that's a"}, {"start": 2283.96, "end": 2285.48, "text": " fairly, uh"}, {"start": 2285.48, "end": 2287.32, "text": " like a known technique"}, {"start": 2287.32, "end": 2289.4, "text": " for for uh increasing the throughput"}, {"start": 2290.2000000000003, "end": 2294.52, "text": " And uh basically does not sacrifice any of the of the of the performance"}, {"start": 2294.52, "end": 2296.94, "text": " So they say here this led to a 15%"}, {"start": 2297.56, "end": 2302.52, "text": " Throughput increase while having comparable loss curves with running them in series during early training"}, {"start": 2302.52, "end": 2308.6, "text": " So I guess there is a minor um penalty you're paying for for for getting the additional throughput. Okay"}, {"start": 2309.64, "end": 2312.36, "text": " Then they mentioned due to an oversight in our uh code"}, {"start": 2312.6, "end": 2318.68, "text": " We unintentionally apply two independent layer norms instead of using a tight layer norm the way that um"}, {"start": 2319.24, "end": 2326.36, "text": " These authors here do so instead of computing this thing you can see that they they actually decouple l and one and l and two"}, {"start": 2326.68, "end": 2329.96, "text": " So just just the bug they had in their code. Okay. Um"}, {"start": 2329.96, "end": 2331.4, "text": " um"}, {"start": 2331.4, "end": 2334.84, "text": " Okay, they use here's here's the uh infrastructure they are using"}, {"start": 2335.7200000000003, "end": 2341.96, "text": " And there is a nice diagram explaining how how the setup looks like the actual uh, like node setup looks like this"}, {"start": 2342.28, "end": 2345.0, "text": " So we train gpt new x20b on 12"}, {"start": 2345.7200000000003, "end": 2353.2400000000002, "text": " Super micro servers each with eight nvidia a100 gpus and configured with two cpus. Okay"}, {"start": 2354.44, "end": 2358.04, "text": " So here you can see that the the complex diagram"}, {"start": 2358.04, "end": 2360.84, "text": " Um, even though it's a high level diagram and still very complex"}, {"start": 2360.84, "end": 2364.68, "text": " You can see how uh each of the four gpus connects to a single cpu here"}, {"start": 2365.48, "end": 2368.92, "text": " And then you have the different group here is connected with this cpu"}, {"start": 2369.64, "end": 2376.04, "text": " And all of these here is what is commonly referred to as a node. So this thing here is called a node"}, {"start": 2377.08, "end": 2379.08, "text": " You'll see this terminology"}, {"start": 2379.48, "end": 2381.48, "text": " a lot, okay guys"}, {"start": 2381.88, "end": 2383.88, "text": " So again a common thread"}, {"start": 2383.88, "end": 2389.96, "text": " That we opted to use the values from the gpt3 paper to guide our choice of hyper parameters"}, {"start": 2390.6800000000003, "end": 2395.1600000000003, "text": " To achieve a higher training throughput. We opt to use the same batch size as open ai's"}, {"start": 2395.6400000000003, "end": 2400.04, "text": " Uh 175b model approximately 3.15 million tokens"}, {"start": 2401.2400000000002, "end": 2408.76, "text": " We extend addmw with zero optimizer. So that's um, I kind of mentioned that because I did cover zero in one in in my"}, {"start": 2409.08, "end": 2411.2400000000002, "text": " ultimate scaling guide video"}, {"start": 2411.24, "end": 2416.12, "text": " I'll link that video somewhere in the in the descriptions if you if you're curious to know more about zero"}, {"start": 2416.7599999999998, "end": 2422.2, "text": " Check it out. But the the tldr is instead of replicating the the model states and the residual states"}, {"start": 2422.3599999999997, "end": 2426.2799999999997, "text": " So the gradients the the the optimizer states the weight etc"}, {"start": 2426.6, "end": 2432.6, "text": " Instead of that you just partition them and that's how you achieve the the zero redundancy. So that's that's what zero stands for"}, {"start": 2433.08, "end": 2434.04, "text": " um"}, {"start": 2434.04, "end": 2436.04, "text": " Finally a note before we see the results"}, {"start": 2436.04, "end": 2442.12, "text": " Uh, they say when comparing results in this work to gpt3 the training data is almost certainly the biggest known unknown factor"}, {"start": 2442.44, "end": 2445.48, "text": " Okay, and yeah data is becoming the name of the game pretty much"}, {"start": 2448.44, "end": 2453.02, "text": " A couple more differences, uh, they have compared to gpt3 is the tokenizer"}, {"start": 2453.48, "end": 2456.92, "text": " So we use a bp based tokenizer similar to that used in gpt2"}, {"start": 2457.32, "end": 2458.52, "text": " gpt2"}, {"start": 2458.52, "end": 2463.4, "text": " With the same total vocab size of 50 000 with three major changes to the tokenizer"}, {"start": 2463.4, "end": 2470.04, "text": " So first we train a new bpe tokenizer based on the pile which makes sense you want to because they're training on pile they want to"}, {"start": 2470.92, "end": 2474.52, "text": " optimize the tokenizer for that particular data"}, {"start": 2475.32, "end": 2477.2400000000002, "text": " Second in contrast to the gpt2"}, {"start": 2478.02, "end": 2483.88, "text": " Tokenizer which took which treats tokenization as a start of a string as a non-space delimited token"}, {"start": 2484.2000000000003, "end": 2488.7000000000003, "text": " The gpt neo x20b tokenizer applies consistent space limitation"}, {"start": 2488.7, "end": 2494.8799999999997, "text": " Uh regardless and third our tokenizer contains tokens for repeated space tokens"}, {"start": 2495.2, "end": 2501.04, "text": " All positive integer amounts of repeated spaces up to and including 24. Okay, this is kind of a mouthful"}, {"start": 2501.4399999999996, "end": 2504.56, "text": " um, and just some idiosyncrasies of tokenizers"}, {"start": 2505.2, "end": 2509.6, "text": " Um, but like let me show you like a graphical representation of what's going on"}, {"start": 2509.68, "end": 2512.56, "text": " So here is the gpt2 and how gpt2"}, {"start": 2512.56, "end": 2518.64, "text": " gpt2 tokenizer would tokenize this piece of text this snippet of code and here is how gpt neo x20b would"}, {"start": 2518.88, "end": 2522.16, "text": " tokenize that same text so this is what I mentioned about the uh"}, {"start": 2522.72, "end": 2528.56, "text": " Basically the spaces so you can see that basically all of these spaces here are encoded as a single token"}, {"start": 2529.12, "end": 2530.96, "text": " similarly here"}, {"start": 2530.96, "end": 2536.08, "text": " Etc etc up to 24. So literally until you get to 24. It's a single token"}, {"start": 2536.16, "end": 2541.04, "text": " That's that's how I understood this part. Whereas on the other side, um, you can see that gpt2"}, {"start": 2541.04, "end": 2549.04, "text": " gpt2 literally encodes each of these spaces as a separate token and that kind of bloats the number of tokens to 55 here compared to 39"}, {"start": 2549.6, "end": 2551.6, "text": " uh in this paper"}, {"start": 2552.16, "end": 2556.08, "text": " Okay, guys, so I mentioned this a couple of times we see no drop"}, {"start": 2556.8, "end": 2560.0, "text": " In test validation loss after crossing the one epoch boundary"}, {"start": 2560.4, "end": 2566.4, "text": " Here are some of the diagrams they have and it's a common practice to always train only for a single epoch"}, {"start": 2566.88, "end": 2569.04, "text": " I guess partially because you have so much data"}, {"start": 2569.04, "end": 2574.72, "text": " That going for multiple epochs would be super compute intensive. So people almost never go to"}, {"start": 2575.36, "end": 2577.36, "text": " more than than a single epoch"}, {"start": 2577.7599999999998, "end": 2582.96, "text": " And there are some papers showing that the model starts memorizing very quickly if you if you start"}, {"start": 2583.52, "end": 2587.68, "text": " Going multiple epochs. So people usually just do single epoch in this paper"}, {"start": 2588.56, "end": 2592.96, "text": " They literally continued training and they didn't see any any any signs of uh"}, {"start": 2592.96, "end": 2598.7400000000002, "text": " Of validation loss starting to diverge or anything. So they just kind of continued and uh, yeah, I mean"}, {"start": 2599.46, "end": 2601.46, "text": " um, I guess"}, {"start": 2601.94, "end": 2603.7, "text": " Yeah, I guess, um"}, {"start": 2603.7, "end": 2605.7, "text": " the common wisdom should be"}, {"start": 2605.7, "end": 2612.58, "text": " Questioned because the wisdom is not based on any solid theoretical understanding of why this works and like llm"}, {"start": 2613.3, "end": 2618.7400000000002, "text": " Large language model training is pretty much like dark magic and alchemy as many as many refer to it"}, {"start": 2618.74, "end": 2623.4799999999996, "text": " And so I think it's completely legit to do something like this if it's giving you better performance"}, {"start": 2624.18, "end": 2625.7799999999997, "text": " Why not?"}, {"start": 2625.7799999999997, "end": 2627.22, "text": " Okay, so"}, {"start": 2627.22, "end": 2630.9799999999996, "text": " Let's continue and let's wrap up this paper. Um, let me show you the results"}, {"start": 2631.7, "end": 2636.4399999999996, "text": " Uh, they get so this is the zero shot performance of gpt new x compared to gptj"}, {"start": 2637.06, "end": 2641.4599999999996, "text": " Fairsec and open eyes models. You can see it's pretty much on pair with them"}, {"start": 2641.7799999999997, "end": 2646.58, "text": " But then it really starts shining once you start using it on a"}, {"start": 2646.58, "end": 2653.7999999999997, "text": " day mathematical data sets and like these arithmetic tasks, which is I guess partially because a huge part of"}, {"start": 2654.2, "end": 2658.44, "text": " I mean not a huge part, but like a significant portions of uh of pile"}, {"start": 2659.08, "end": 2664.04, "text": " Are are like these scientific websites such as archive papers, etc, etc"}, {"start": 2664.2799999999997, "end": 2669.56, "text": " So because of the data set, uh, it seems that this model came much better compared to to the other baselines"}, {"start": 2669.56, "end": 2672.7599999999998, "text": " You can see the the orange dashed lines here. It's kind of"}, {"start": 2672.76, "end": 2676.6000000000004, "text": " Uh over outperforming all of the other baselines if i'm not wrong"}, {"start": 2676.6000000000004, "end": 2682.2000000000003, "text": " I think there are few shot performance not only in these data sets, but in general is also better compared to"}, {"start": 2682.8, "end": 2685.5600000000004, "text": " GPT models, but I don't see the diagrams here. So"}, {"start": 2686.6400000000003, "end": 2691.6400000000003, "text": " So yeah, okay. This is actually the one i'm referring to so this is the five shot evaluation"}, {"start": 2691.6400000000003, "end": 2696.0800000000004, "text": " You can see that again gpt new x is much better when you give it multiple shots"}, {"start": 2696.1600000000003, "end": 2701.88, "text": " So it's kind of a better few shot learner apparently compared to the gpt model and fairsec, etc"}, {"start": 2701.88, "end": 2703.08, "text": " And final load here"}, {"start": 2703.08, "end": 2709.6800000000003, "text": " We opted to choose hyper parameters based on on a mixture of experiments at smaller scales and by interpolating parameters"}, {"start": 2709.88, "end": 2713.08, "text": " Appropriate for our model size based on previously published work"}, {"start": 2713.2400000000002, "end": 2720.48, "text": " But because I think they so however several aspects of both our model architecture and training setup including the data and the tokenizer"}, {"start": 2721.12, "end": 2723.88, "text": " Diverged significantly from gpt3 as such"}, {"start": 2723.88, "end": 2731.04, "text": " It is almost certainly the case that the hyper params used for our model are no longer optimal and potentially never were"}, {"start": 2731.04, "end": 2733.6, "text": " In the appendix they go on to say"}, {"start": 2734.0, "end": 2741.4, "text": " And argument why it's a good idea to release the model weights because back at the at the time when they released this model"}, {"start": 2741.4, "end": 2744.98, "text": " This was the biggest model out there now. We have bloom now"}, {"start": 2744.98, "end": 2752.84, "text": " We have opt but like this was a very important contribution to the community and just a very valuable thing for"}, {"start": 2753.6, "end": 2756.68, "text": " like researchers and people who are not as as"}, {"start": 2756.68, "end": 2760.56, "text": " well-financed as some other research labs"}, {"start": 2761.24, "end": 2763.08, "text": " And that's it guys"}, {"start": 2763.08, "end": 2770.7999999999997, "text": " So those are the three papers I wanted to cover so again gpt neo x20b opt and bloom paper"}, {"start": 2771.44, "end": 2776.22, "text": " So we saw a couple of commonalities between them. They were all following gpt3"}, {"start": 2777.08, "end": 2784.08, "text": " Hyper parameters and recommendations we saw that they were all using pretty much except for the opt paper"}, {"start": 2784.08, "end": 2786.7999999999997, "text": " Both the deep speed and Megatron"}, {"start": 2786.88, "end": 2791.68, "text": " Whereas opt was using only the Megatron library to scale up these models and train them"}, {"start": 2791.68, "end": 2796.64, "text": " Which is completely non-trivial thing to do most importantly that the value of these"}, {"start": 2797.88, "end": 2798.84, "text": " projects"}, {"start": 2798.84, "end": 2806.2799999999997, "text": " Is that they've shared both the code as well as the process as well as the weights which is a huge contribution to the community"}, {"start": 2806.2799999999997, "end": 2807.96, "text": " So kudos for that"}, {"start": 2807.96, "end": 2809.96, "text": " Guys if you like this video"}, {"start": 2809.96, "end": 2813.64, "text": " Subscribe to this channel share it out with your friends and"}, {"start": 2814.52, "end": 2820.46, "text": " Expect new coding videos coming up where I'll be walking you through some of the code bases behind these papers"}, {"start": 2820.46, "end": 2840.14, "text": " So stay tuned for that and until next time bye bye"}]