OPT-175B: Open Pretrained Transformer | ML Coding Series
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I do a deep dive into the metaseq (Open Pretrained Transformer, OPT-175B) codebase.

I first show you how to set up the code on your machine, and then I walk you through the codebase behind Meta's large language model (LLM).

Along the way I cover key concepts behind mixed precision training such as loss scaling and unscaling, and much more.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ My fork of metaseq: https://github.com/gordicaleksa/metaseq
✅ Instructions for Apex: https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/main/start_fast.md
✅ Ultimate guide to scaling vid: https://www.youtube.com/watch?v=hc0u4avAkuM
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00:00 Intro - open pretrained transformer
00:01:25 Setup (creating the cond env)
00:06:00 Setup (patch the code)
00:11:40 Collecting train script arguments
00:17:49 Training script walk-through
00:25:25 Constructing a dummy task
00:28:30 Building the transformer model
00:35:22 CUDA kernels (C++ code)
00:46:30 Preparing a dummy dataset
00:50:00 Training loop
00:54:05 Zero grad (loss scaling)
00:56:15 Forward pass through a transformer
01:02:20 (IMPORTANT) loss, scaling, mixed precision, error handling
01:13:25 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#opt #meta #largelanguagemodels #transformers
 What's cracking guys? Alex here. In this video, I'll be walking you through the code base behind the recent Opt 175B Transformer from Meta and if you're curious to learn about the actual theory about the paper I covered this paper as well as GPT Neo X20B and as well as Big Science blue models So the corresponding papers in the previous video, so I'm gonna link the card somewhere up there in this video I'll be walking you through as I said through the code base behind the opt model basically, that's the meta sec code base and My idea will be to first show you how to set up this on your local machine Because the system was designed the code base was designed to be run on the Linux distributed system So that means multi node multi GPU setting whereas I have windows single GPU machine So that kind of made it a bit harder to get started So that's the first thing the second thing I want to show you some of the concepts I introduced in my Ultimate guide to scaling video. I'm gonna link the card as well somewhere here Basically, I'll focus on the mixed precision training and I'll show you some of the concepts such as loss scaling Etc and also you'll just get to see how this product well not a production level But like a code base behind one of these large language models looks like so hopefully that's gonna be valuable Okay, let's get started. The first thing we need to do is open up the setup guide So there is the setup instructions here, which we're going to follow. It's fairly straightforward There's a couple of caveats, but that's pretty much it. So I'm gonna open up my anaconda prompt here I'm going to first create a conda environment. So conda conda create Name I'm gonna use opt to because I already have Opt environment setup so I have to change the name there and I'm gonna use the Python 3.9 so I'm gonna hit enter there and This will be quickly over and once we have conda Installed will start installing the necessary packages directly inside of it. Okay, this is gonna be quite fast. So activate the environment Clear just to clear the screen a bit and now I'm gonna just copy paste this command here So we do this pip install of basically torch and torch vision etc. So just hit this command and That should execute in a couple of minutes. Okay guys, so that's that's installed now. Let's continue So the next step would be to install the apex library from from Nvidia because I'm on windows This does not work windows is not supported so you can follow these steps on your own if you're on windows Just skip this part. There is a bit better instruction for how to install apex in case you have some mismatch of the CUDA versions, etc, etc and Like some some basically tips on how to overcome those potential problems in this big science Basically repo I'll link that link down in the video description so you can check it out. So I'm skipping the apex Let's now continue on installing the Megatron. So let's just copy paste this part here Well before that, sorry, let's just first clone the the actual repo. So I'm gonna do the following and I'm just create a Directory here. I'm gonna name it opt I'm going to just do a clone operation inside of the directory if I manage to open up it the folder So here's the the git bash Just go and basically copy the link the URL do git clone here and that's gonna download the actual meta sec repo in the in this target directory So once we are there, let me just navigate using my my environment. So I'm gonna open up this thing here I'm gonna navigate here in the in the console to that to the root of my code base and Now I'm gonna follow the rest of the instructions. So let's now do the git clone So let's do the Megatron installation You can just copy paste this directly in the terminal everything is gonna execute automatically except for the last line So hit the pip install command here That's gonna install the Megatron. Hopefully fairly quickly. Yeah. Okay, that's cool. Let's continue on the second step now the important thing you need to do here is just change go go back to the root to to meta sec and Don't install don't obviously don't you don't want to install inside of the Megatron the fair scale library So I'm just gonna copy paste this again here. We're installing the fair scale and I hit pip install For the last line and we'll install the fair scale as well. It's kind of smooth so far No, no problems here. Okay, finally Because we've already installed we already cloned the meta sec. We just need to go to the root so change directory to the root and now we do pip install but before that there is a Additional tweak I have to do and that's in the setup file here I have to basically open up this this file and find the Aim library and this one also doesn't work on Windows for whatever reason I just didn't want to hustle around it so I just kind of comment it out and that's it after this you can hit the the pip install command and Everything should work as expected. Let me paste this likely enter and that's gonna be installed now Okay, there is one more command we have to execute and that's not in this manual So that's this pip install setup tools this version 59 point five point. Oh There is some weird version error going on if I don't do this So I had to I kind of found this on the on some random website as a solution and finally This is not necessary unless you want to to like basically contribute to the to the repo this pre-commit install But let's just do it either way. Okay, cool. I'm gonna clear the console here. Everything is ready now now Let's open up the actual code. So I'm gonna use VS code as usually VS code here and I'm gonna just do the whoops open folder Find this meta sec directory and just select folder That should be pretty much it So now what we want to do is select the correct interpreter and that's the opt-to so let's just open something Let me open up some of the files will need a bit later. So this is gonna be the training file Let me hit the nope. I don't want to dead So I just want to hit the ctrl shift P select interpreter and let's select the opt-to Interpreter here and that's now it the final thing we have to do is a couple of patches Obviously I had to sort it out before before the video. So let me just show you what those are. I Have a working code here. So just to get this tool to show you some of the differences Some of the patches I had to apply so first thing was and we've already done This is to comment out the aim this might not be necessary for for your case But just like that's something that I had to do. Okay, then there is this line in trains I'm just gonna copy paste this one. You obviously won't have to do this because I'll basically create a clone Directly and and push it to my to my github, but let me let me return here So this is something we have to add Basically here. I'm gonna just save it there Then let's see what else so let me exit here We have a couple more steps Metasec tasks Metasec tasks base task we have to add this weird line to convert it to long Otherwise, he's gonna complain. So that's the base task inside of meta task tasks So let me open up that file again You will have you can skip this all of this because you can just start from my from my code So I'm gonna do this so meta sec find the tasks find the best base task and find this line so this is basically Let me find this thing so filter oops filter examples blah blah blah, so it's here So I just paste that line here. It can be cleaner. Obviously I'm just doing it like this like you should import it in on top of the file But this is just the whole point is for me to enable this code So that I can easily run it and and do a stepping through of the code base Okay, so I think that's one step then we have the sweep We can actually skip this part. We have we don't have to do this Then there is this thing so we have to change from 2 to 1 because I only have one GPU Otherwise this will fail because as you can see here, this is the model parallel argument So I'm just gonna do that. So that's meta sec launcher up job constants. I'm gonna just patch that quickly here so let's go to the Metasec launcher so meta sec here is the launcher here the constants and We just put ones here. These are the small smallest models You can see the 8 million the 125 million and the 350 million the tiny small and medium ones. Okay And the final thing we have to do is in the up baselines We have to comment out this if arcs number of GPUs smaller than 2 then raise an error Because obviously I have just single ones I'll have to do that and there is this one here with the environment thingy So it was obviously a lot of trial and error to to to get this uh, like sorted out but yeah Okay, let me find the uh up baselines so arcs Num of GPUs Here it is. So I have to comment out this line And the final patch that I have to apply is this one here. I'm just gonna do Copy paste of this on top of on the bottom of the file and that's it Okay guys, that's pretty much it Okay, having done all of this, let's go back to the repo and now we want to train this thing So let's see what the instructions say so they have the this training link here You open it up and the only thing you have is this so from here I have to had to work my way around and figure out what's going on. So Let's let's see what i'm gonna do. So i'm gonna instead just copy paste the uh The launch arguments I already have in the in in my code Which I've already previously Been running so i'm just gonna copy paste all of these arguments here and you'll see what's the difference basically Let's just give me a second here So i'm gonna open up this one Of baselines i'm gonna create Uh, basically a config file here for the for the for running the code and let's just paste those arguments So let's see what's the difference between this thing and what they show here So first things first, uh, you can notice I have the benchmark in the local The local makes it so that I can run this on the local machine obviously and the benchmark Was necessary to basically So that you can skip downloading bigdp So that you can skip downloading big data sets and this will just load some use some dummy data loaders and Everything else will stay the same. So that's kind of cool. Okay, so number of nodes was obviously I had to change it to one It was two here And number of gpus I had to set it to one instead of eight here And other than that everything else remains the same Okay, so now let's run this code. I'm gonna zoom in a little bit here And i'm going to set a breakpoint here and let's just run this code. Okay, let me run this Hit run And the whole point of this script will be to collect the arguments That then we are going to use for the train script because they don't have any instructions for how to start the actual train script So this is a workaround. I kind of figured out um, let's see what's going on here, so we have to Go here And then sweep main so this is maybe helpful for you to understand my my process and then basically In case you need to figure out something on your own, uh, you can you can follow a similar workflow. So we end up here Then we enter this uh back end main i'm gonna put a breakpoint here Let's just continue the whole point was to find so let's continue here The whole point was to find the actual command line that's being used to call the train script So we'll see how that looks like in a second. Okay, let me do this Let me just find uh the so train commands. We don't care about this code We just want to find the the the final code that's going to be run. So i'm gonna put uh Breakpoint here hit f5 Uh, let's continue here. Let me just find yeah local run. So this is where we want to go um f5 Enter the local run and here it is. So here is the actual piece of code that will call the train script So we have this train command. So it's python blah blah blah. It's calling as you can see here, hopefully um Let me just see whether I can shift this. Okay. So what i'm gonna do is this i'm gonna open up the debug console here and i'm simply going to Capture all of these arguments which we will then use to to to run the the the train script So we don't care about this n thingy. Uh, we only care about about the uh actual commands that were passed to the train script Okay, you can see here python blah blah. It's calling the train script. So this is the thing you can do. So we open um, basically command Uh take take stay uh, and then write mode Uh as f and then we basically want to write to that file This train command and this is going to obviously save This command and capture it and so i'm gonna do that and let's see what's going on. Okay, so we've saved that somewhere here Hopefully, yeah commands And then you can just copy paste all of these and start playing from there So that was my first part of the process So it took some time to figure out how to how to call the train script and this was my workaround To to get the actual commands and then came the tweaking part. So i'm gonna just again Kind of speed this up by by copy pasting the actual Arguments you can see it's super duper verbose Um, i'm gonna copy paste all of this and i'm gonna well i'm actually gonna copy paste everything Including these comments I made here so I can explain what i've done. Uh, so let's go to the code here again You will not have to do this because because um, you'll have this already Uh in my in on my github, uh repo, okay, let's go here. Um Let's rename this as as opt baselines And i'll have to create another config here For the train run, I will just add configuration Python python file and then i'll just add the arguments here. Okay That's it. Hopefully if I haven't made any mistakes here, we don't need this And that's pretty much it. I'm going to rename this file to This this config to train script Uh, and let me show you what's going on here. So a lot of arguments obviously, uh, this is um What you get when you're working with large language models and in general with with libraries such as such as metasec which is I think cloned and then um Modified version of the fairsec library. Okay so this is pretty much the same thing as um as what we get when we run the Previous script with some modifications So some of those modifications I had to me was like we we I can't use fully sharding because I have a single gpu I can't use the parallel version of cross entropy So I had to kind of change it to cross entropy instead of cross entropy parallel or something I was just using transformer lm instead of transformer lm megatron or something and finally, um Windows was complaining I guess any any operating system will complain about super long paths So there was some super long path that we get dumped from the previous script So I had to kind of shorten them into this one So test v0 and finally I had to remove the checkpoint activations and distribute checkpoint activations This is not well, it's actually necessary Otherwise you you will face some bugs but like I didn't bother to try and figure it out So if you want to kind of play with the checkpointing, uh, which is the the technique where you basically trade off the memory versus the the the the speed of your backward pass Uh, then you can kind of enable this and try and figure it out yourself But like i'm going to focus on the mixed precision a part of the of the code. Okay Made a couple more tweaks. Uh, you will be able to just copy paste this Okay, I think a couple of things I had to change was like the batch size because otherwise you I was getting oom's Out of memory exceptions and I had to change to like have 128 tokens per per batch Uh per sentence instead of like 2,048 by default or something like that, but that's it again You'll copy paste this thing, uh, and you can start from here Okay, guys, let's finally get to the transcript. Let me set a breakpoint here Uh, let me use the transcript config And actually i'm gonna set another breakpoint down below Here and now we are ready to to to run Okay, guys, let's start going through the code here i'm gonna focus on on on the Well, i'm gonna show you details that that are different compared to your uh toy Uh repos so to speak because this is after all using this fairly powerful Library, uh that was developed by meta So yeah, there is a lot of code that that makes this harder to read Uh, but like more robust to various errors. Okay, so let's continue here So first things first i'm gonna enter this parser function and I want to focus on a couple of details Like first of all, I think it's quite kind of interesting how they are picking up, uh various arguments uh, it might not be the most readable way to do it, but it's uh Interesting in the sense that let me let me show what I mean So here there is this common config so common config will be just a data class that contains a bunch of these fields that uh basically define uh various, uh variables which are which belong to the common group of variables of arguments such as uh logging interval tensorboard logdir Uh various uh thingies like this aim repo. So that's the this visualization tool there Then there is the weights and biases project here Azure ml logging bunch of details. So yeah, I won't focus on those but I just kind of want to stress that that's neat so once we Basically execute this line. We'll pick up all of the arguments and default values from this config Okay Next up they have this these registries. So this is probably less interesting Let me just see here. Uh, so yeah, they're picking up They're picking up values automatically. I mean i'm gonna skip this part because otherwise this will take too long and it's not that vital so Bottom line is now the same thing will happen in all of these other functions. So this is if you enter here You'll see that there is this uh generate parser from data class and then there is the data set config so here is the where they define some of the default values for the uh, data set Group and then uh, yeah, so we can basically continue. Let me just go back here So yeah, by the way, if you don't know how to use vs code, there is this nice call stack like a window here You just hit the uh, whatever is on top of the stack and you get back to wherever your your debugger has stopped That's kind of neat to know. Um, yeah So again bunch of different groups will add different arguments and finally we end up with a parser And that's it um now some conversion to configs it's going to be a dictionary or something and Now because we're not profiling. We're not doing this. We won't enter this branch. We're going here So let's enter the the actual main Function we can skip all of this and we enter the single gpu main branch. So the only difference is um Well, there are some differences but ultimately all of the code pets whether you're running this in distributed setting or on a single gpu We'll call the same main function So that means we'll be actually stepping through the actual code that was used to train the the huge Opt 175 billion parameter model, which is awesome. So let's let me just show you quickly this thingy here Again, there are many details Here that are relevant if you're running this in a distributed setting That might be interesting interesting for you to check it out. But here i'm going to focus on on on on this code path here So let's enter the main and let's keep on stepping through this. So let me see what's interesting. So here this is the patch I applied So finally we save the config. So this is a config that we got from all those parser steps So ultimately we just save it here to config yaml and we'll see how how the config looks like Okay. Let me just now show you how this thing's going to look like. So this is hopefully the right. Yeah, this is the right directory So let's open up the um test v0. Let's open up the config. I'm just gonna um Basically open up another visual studio instance here And let me open that file. So let's open. Oops Let's open the file here This one and here is the config yaml So i'm going to open it up here side by side. Okay. So as you can see here So as you can see here, there is a lot of a lot of arguments like I think over 400 or something like a huge amount of arguments um, this could use some some cleanup and refactoring but um I guess it works Kind of they did struggle a lot. So I don't know Uh, cool. So I won't be going through the arguments here. Obviously it beats the purpose. There is too many things going on So i'm just gonna Explain you those that we are hitting as we are executing along this code path. Um, okay So there is this nice, um Is is master function? I think it just checks so because the distributed rank is sql zero. So rank means basically if you have multiple gpus in a in a node, uh rank zero would be GPU zero in that node the rank one would be gpu one It's just like basically naming for your gpus in the node because here we are zero because we are running a single gpu Uh, we are going to be the master because we are the only gpu. So we are the master automatically. Okay Let's continue here. Um I'm going to skip all of that just some asserts Ignoring all of that seeding the the code so that we can have reproducibility Uh, and then some additional seeding that kind of probably in the background. I think it just seeds the torch thingies Yeah torch manual seed blah blah blah I'm gonna skip those details Um verify checkpoint directory. I think this is kind of neat. So again this video I'm just showing you some details that that go into the engineering behind these llms is not only like the the actual Ml logic. That's what i'm trying to get at is super simple like you have a transformer you do everything You can literally write all of this logic conceptually. It may be Hundred lines of code or something. I might be exaggerating but uh, the point is there is so many things That you have to add to make this robust and to make this uh work on on such a large scale So this is one of the of the of the details. So let me enter this I'm gonna put a breakpoint here hit f5 Uh, okay. So let's see what they're doing. So they are literally testing with this if this test v0 directory exists And we can actually we actually have the access the the right rights to write to that directory So if it exists, uh, if it doesn't exist we create it because it does exist. We don't create it We fetch the global rank. It's going to be zero. I assume because yeah, as I said, uh, we create the temporary file So we just create some random File here and we try to open it up in a write mode that shows us that we have write access So as you can see here, we're not hitting the exception. So everything's fine there. And finally we try to remove the file So you can see here. So if I were to Go here, there is the dummy file. And now if I go Uh f10 so step over you can see how this thing is deleted So that's how they do the check that you can actually write to that directory and that's robust Okay now some logging how much memory I have etc. Etc Logging the the whole config you can see there is a bunch of data here Everything we saw in the actual yaml file in the vs code And yeah, okay. I'm gonna skip this There is the setup task. I'm gonna enter this part of the code as well. So let's see what's going on there um blah blah blah So we have the task name that's called dummylm And this is a consequence of me using the benchmark flag if you recall And um, yeah, so that's gonna be like a super simple task. We're gonna see what it does in a couple of seconds so here We just pick some uh, basically um Associated config with that task. I will not get into how they populate these Uh, there is a neat method they use in the background. Well, i'm not sure whether it's neat. It's kind of hard to Read and understand but again It's understandable Let's enter this one Let's enter the actual constructor of the dummylm task and here we are. Okay, so Let me see whether it's interesting to see the base task. I think nothing there is fan Yeah, okay So i'm gonna just skip the init function of the base task and let's see how this is gonna work So we create a dictionary and this dictionary is just gonna be adding a bunch of symbols Uh to this to this task, which is not important. It's gonna add like 50 000 or something Of yeah, 51 000 196 Symbols, uh, these will not be important for for for the training as you'll see in a second And then what they do is they pad to multiple fate. So Why do they do this is because sometimes um On the hardware on the underlying hardware if things are divisible by eight things go much faster So that's a constraint that you kind of have to know of the of the underlying gpus you're using So I guess nvidia a 100s or h100s new ones. So yeah again Details that matter when you train on such a large scale Okay, and finally what we do here is we create as you can see here 128 um basically numbers we have plus one because here We'll create one will be the the source sequence and the second one will be the dummy, uh, the the the target sequence So that's basically your transformer logic. You have a sequence and then you shift it You try to predict the next token and that's why you have here You grab the first 128 elements and then you grab the you move by one and then you grab the 128 there And that's what you're trying to predict. Hopefully that makes sense um There is additionally some some some logic here, which is not necessary for this dummy task But um in general, uh, what it does is makes that you're not predicting the padding token So that's why you have this but it doesn't matter for now. That's that's enough. Okay, so that's the setup task As I said super simple. We just have these two, uh dummy sequences That will be later passing through the transformer during the forward pass Okay, because we're not charted i'm gonna skip all of this But you see that if we were running this on a on a on an actual distributed system would be hitting this branch Instead and you can see there is this fsdp Wrap that help helps with the model Training on a distributed system basically, okay build model i'm going to enter this branch as well Uh, let me see what's going on here. So build model We are using the transformer lm uh as again, I cannot run the megatron version because it Assumes I have a at least two gpus and because of that i'm i'm getting Uh, basically exceptions if I do this so Let's just uh pick up the model the actual class is so this is what this registry thing does It takes the string transformer lm and picks up the actual class that we'll be using to instantiate the model. Okay? And then here it just picks up the associated config So transform role language model config It doesn't matter for for for us at this point of time. Let's enter the build model, uh branch Uh, and let's see what's going on here. So We first build the embedding table. Okay, so that's the first part So so that's how transformers work, right? We have a piece of text a sentence We tokenize it into a bunch of integers which are usually of a long data type And then we use those integers to index into this embedding table to extract a particular Uh embedding vector and then we're gonna feed those vectors through the transformer. That's what this thing does okay, and being a bit more detailed than than in general because Yeah, I think it makes sense to do it now Um, let's see how this is gonna work. They call some embedding thingy here And once you enter there, you see that ultimately it boils down to torch pi torch embedding So we fetch the device here it's gonna be uh kuda Zero because we have that's the index of my single gpu And because this is set to true we'll be using half precision So everything is going to be in half precision So you can see here what we do is number of embeddings is 51,200 because that's the i3 we call that's the dictionary size. So that's the uh, Even though we're using a dummy task, um, then we have embedding the dimension which is 512 And we are running this on a on a gpu in half precision. So that's float 16 So let's create this empty tensor And then we just basically initialize it normally and then i'm not sure why we're doing this Basically, what I do is they set the padding Vector to be all zeros. So currently if I were to open up this one Let me show you this. So if I use the weight of padding index And if I do this, uh, you can see there is a bunch of values after we step over here Now if I execute it we'll have all zeros. So i'm not sure why they're doing this because ultimately embeddings are learnable. Um, So so so yeah, um, i'm not completely sure what's going on Embeddings Constructed we get back from the call and we are here The next thing we do is we construct the actual decoder. So this is going to be a decoder only transformer So let's enter into this thingy and And let's execute. Okay Um, let me see whether there's something interesting here incremental decoder. I think it's just going to store the dictionary So we don't have to enter the Constructors of the parent classes. I'm going to skip over all of that Uh, let's focus on the important parts. Um, nothing fancy here. We're just storing some variables Blah blah blah Okay So the first thing that we're gonna do is we if the embedding dimension is different from the input embedding dimension, which they are Uh, then we'll create this linear layer That's gonna take our embedding vectors and just uh increase the dimensionality from 512 to to seven some 700 something So let's see what it is. So we have a linear layer So from 512 is gonna map them into 768 dimensions. Okay We're not using a lib in this particular setup so Um, we can ignore all of this we can ignore all of this Um, and let's see where we are now. So now we are generating the positional embeddings Let me enter inside of there. So I just hit f12 to enter. I set a breakpoint here I hit that five and we are in okay. This is learnable. So we enter this branch here um number of embeddings is gonna be Uh, whatever the number of readings here is plus padding plus one Because I assume it's for the beginning of sentence token or something That's why they have to create 130 of these instead of 128. So remember we only need as many uh positional embeddings as as the longest sequence will be feeding to the transformer because we're using this dummy task We're only feeding 128 tokens. So that's why we have this many uh positional embeddings So then we call this learn positional embedding thingy and that's just going to Do what let me see Uh, well, it's just embedding. It's literally just a torch embedding again. So we can just skip over that He's just gonna create embedding table and then we initialize it again and um Again padding index is set to zero and we return back the embedding. So that's it. Okay Similarly to what we've done for the input tokens embedding table okay so So now we're gonna do this precision conversion. So we'll be uh grabbing the Embed positions and we'll convert them to fp 16 basically, so let me enter this function here i'm gonna hit hit the f uh, like a breakpoint here hit f5 and uh, let's see what's gonna happen so we assert that it's not the B-float 16. So it's the brain float format Uh check out my um again ultimate guide to scaling to understand how these formats look like how they work And why they're important in a nutshell it just increases the range, but I don't have bf 16 on my machine So we're just going to use the half which is the fp 16 Uh format that's a bit has a smaller range and that means it's uh, the the the the weights during the training and the gradients and everything Is more susceptible to both underflow as well as the overflow Okay, so we basically transform them to have precision and we get back here. Okay, let's continue Next up we generate 12 of these decoder layers By calling a build decoder layer a function here and this one is going to be fairly interesting. So i'm gonna Put a breakpoint here hit f5 enter here Let's first build the base decoder layer. So i'm gonna hit f11 to enter f11 again to enter Here and let's see where we are. So we are in the decoder layer Okay, so we're trying to construct the first one now. This part is fairly interesting So this is something you don't see in a regular code base. That's not super optimized Here we'll be using megatron fused kernels. Well, actually for this particular setup I have I will not be using these kernels but still we'll see how they how they are loading them and how it looks like in practice Okay, so let's go through all of this. That's not important And now this part is interesting. So let's enter the the load the fuse kernels load function F11 and here we are okay So first we pick up some some some metadata about the kuda. That's not that important um, let's just Step over all of these and then this one is fairly interesting. So this one is going to load certain Uh kernels written in c++. That's gonna be super efficient compared to running in python So let me just go here So fused softmax, we just defined the function and this is set to true. So we enter this we just Create some flags and you can see here We are now creating a path towards certain cpp file and this cu file kuda kernel file. So So by the way, i'm not super familiar. I've never had to write any code into any kuda kernels myself. So yeah, um, just fyi um, so here are the sources And now we're gonna load those files using the cpp extension load helper. So this function here This is gonna load from those from those files. Let me show you those those files here So here it is Uh, we have to enter the megatron. We have to enter the megatron again and then diffuse kernels And here are those files. So we are scaled upper triang Mast softmax So here it is. I'm gonna open it up I think that's the one and then we have to open up the kuda one, right? so That one let me just open it up with vs code And then there is this one. So let me open it up with vs code as well. And that's the only two files I'm gonna open up just for two to show you how how it roughly looks like So here is the cpp one. So you can see here they define forward and backward for this particular Softmax layer and they make it super efficient So in this particular part in this file, they just create a binding to python so that we can execute a c++ code directly from python Uh, so here is the actual code behind that file So here is the blah blah blah the backward kuda file Uh, and everything boils down to this dispatch half and and and bfloat function Let me see where I can do f12 and see how that function looks like um Yeah, I won't go there man. That's gonna be wicked in any case This was just more of an fyi than than anything else I don't want to dig too much into the c++ plus code because that's gonna get like messy very quickly Okay, let's go back to the um, let's find my my prompt Let's find the debugger and let's continue execution execution here. So this might take a while Uh, just the loading of this i'm gonna i'm gonna skip this loading part and i'm gonna just hit f5 to To get to the mask softmax here and I will get back to you as soon as yeah here it is because i've actually pre-built it before But the first time we run this it will take probably five minutes, uh in case Uh, you you it's hanging there is one trick you have to do and that's to just delete this build directory again Don't ask me why I know this Uh, yeah, the tears on my keyboard know it Okay, let's continue here Let's load these scaled masks softmax once and as I said, we will not actually be using them But it's kind of funny to to see uh that uh, they are doing that Okay, guys, I actually realized that the actual implementation is in the h file So if we open up this one, let me just quickly show you so you can see the actual code You can find it here. So this is how that these functions are defined in the background just a cpp function Uh c++ function and the optimized version of the of those of that functionality. So yeah, let me go back now here Uh, let me show you one more thing. I'm gonna copy this Use the search functionality here in vs code to find where this one is used And you can see that um, so we have in this file here where it's actually being used is somewhere here So let's just find it Uh, here it is. Let me try and see whether yeah forward f line. Okay, so there is this fused layer norm f1 function Uh, that's inside of the megatron library actually And I think it relies on the apex potentially So so yeah, that's i'm just kind of showing you to so that you know your way around this code But yeah, um, so if we were to actually use this this this class somewhere in our code path Uh in the background this part of the code would actually be c++ code. So that's kind of interesting Okay, let's get back here and let's exit out of this function let's load this one and then let me show you the The creation of this of this Decoder layer. So again, we are in the transformer decoder layer Let's continue We're building the self-attention. I'm going to skip that part because that's that's common common knowledge, I guess um Let's continue here layer norms I initialize blah blah blah again. We're converting the layers the the attention layers to to uh float i'm gonna Basically remove this one Uh, so that we don't have to enter anymore. So everything has been converted to fp16 Uh, so that's the mixed precision thingy Uh, and um, yeah, let's continue here. We just fetch the activation function, which is going to be a rally I think Yeah, and then let me just see what this one is So yeah, you can see we're just because activation was set to rally We'll just return the f value the functional from the functional module. We fetch the rally function. Okay Let's go back here Um now we create the fully connected layers So we have obviously two fully connected layers, uh in the transformer block. The first one is after the attention mechanism We have the fc1 and then we have the fc2. No, actually this is the sorry this is the I guess the mlp after after the attention thing is is is is gone but like the actual attention logic also has the Mapping the linear mapping once you do the Attention thingy with value vectors and everything if that makes sense So the reason they split up into fc1 and fc2 is because they use megatron And so I think that's some limitation that they that the megatron imposes on the users You have to kind of split these such that once you do the model parallelism You can split these fully connected layers onto multiple gpus. That's why they have to separately Uh create the function. So if that's confusing you That's the answer Okay, so it's just gonna build in the background as you can see here. It's just gonna build a linear layer But the reason they're doing it like this is again model parallelism And those are all of the details you kind of have to deal with when you're when you're um Working on on large scale there. There are details that you kind of have to to to learn and uh, yeah It's kind of different compared to small scale again conversion to fp16. Let's exit here And that's the base layer um Let me skip over all of this And now additionally if checkpoint was set to true with basically wrap this layer inside of the checkpoint wrapper And again, that's what it does in the back in the background is during the backward pass You didn't store all the activations and instead you'll be recomputing the activations and by doing that you'll be obviously Increasing the time it takes to do the backward. I think it's going to be 2x slower But as a as a consequence you you save up a lot of memory because you don't have to save every activation Along every layer. That's the checkpoint in logic, but we're going to skip it now Uh, let's continue here. This is going to be no op. Nothing's going to happen in this one And let's continue. Okay, so that's going to be repeated 12 times I'm gonna do disable all breakpoints put f put the breakpoint here hit f5 Okay, and let's continue from here. Now. I'm gonna re-enable all of the breakpoints and that's it. Okay guys And we just packed those layers into module list Nothing fancy there Nothing fancy here. We have the project again projection the output dimension projection from I guess 712 to 768 to 512 uh and That's it Okay here because share input output embed it's set to true. This is one of the things you again kind of um kept to note and that's that usually people tie The input embedding table with the output and so here instead of having a separate set of weights for this output projection instead We'll just reuse the same weights that we use to embed our our initial tokens And you can see we're literally using this embed tokens, which is our embedding table as you can see 51 200 and 512 that's the initial embedding table we created and we're going to use those weights to populate this linear layer So then you can imagine what this does is you take the output token Which is 512 and once you multiply with this output projection you end up with this 51 000 something which is basically the vocab size Of your of your output. Okay. Hopefully that makes sense Let me exit this. We are not using a lib so we can exit this we have our decoder We have we have the embedding table. That's it model constructed Okay, now we have the um Criterion we're using cross entropy. I don't think this is anything super interesting Let me see what's going on here You can skip all of this. We can skip all of this Let's enter the builder. I think we can skip all of this as well Well Nothing interesting Here, so here's just the cross entropy criterion I'm gonna skip all of this. This is not interesting Okay. So now there is some logging going on if you open up the terminal here. You'll just see what those logs are Uh, it's just logging various stuff. So we're using dummy element task. We're using transformer language model. We're using cross entropy as the loss Let's continue. We are counting the number of parameters here. We have 112 Um million Uh, blah blah blah. Let's continue here. Now we'll load the data set. So let's enter the Data loading part. So i'm gonna hit f11 Here we are. So let me continue here Okay, so what's this batch size is one as I told you i'm using one because otherwise I would be hitting Outer memory exceptions and now we just construct a dummy data set So here is how it looks like this this one we basically just stored the batch. It's a super simple Well, literally dummy data set the name says it itself. So what we do is we just create this Batch you can see here And what we do is we have source tokens. We have uh, we just have the the array of of 128 elements We created if you recall before so we just use that as a source tokens and then um Um, we set as the target tokens those dummy targets and you can see see they're shifted by one So here we have three four five and there we had for the source one. We had two Three four, so they're exactly shifted by one So this is going to be our simple, um dummy data set. Uh, we just have number of sentences is one We only have 128 tokens blah blah blah So we literally have batch size one and only 128 tokens as simple as it gets as simple as it gets So let me just construct this thingy And that's it. So that's the construction of the data set. It's very easy and simple So now the trainer is just a abstraction they use to to sort everything inside So the task and the data set and everything so here they extract the shared parameters because I told you they're binding the input embedding with the output embedding output projection layer Again, that's now less interesting. I'm going to skip over all of these They are storing the criteria and the model Uh blah blah blah Uh, they are basically again casting to uh half precision float fp 16 both the model and the criterion uh, and uh Now they're pushing it to the gpu and That's it Nothing, nothing interesting there. I'm going to skip all of this I'm going to skip all of this. I'm just going to put f5 somewhere here Sorry, put the breakpoint and hit f5. My brain is glitching nice Okay. So again, it's not fetching the environment for some reason so you can see here I have rtx 2080 8 gigabytes, uh 7.5 are the major and minor And they just store that for whatever reason. I'm not sure why they're using it probably just to For the logging purposes so they they know exactly which devices in case you have heterogeneous distributed system Where you're using multiple gpus some are maybe p 100 some are maybe m 40 some are maybe v 100 Some are a 100 so when I have that additional, uh, as you can see here, they're just Logging everything basically. Okay. We're back to the train. This is the the main function In case you're confused where we are. So, um Let's now i'm gonna skip this because this is just gonna Uh give us back this iterator and there is like literally four layers of wrapping they do here They have some counting wrapper and then this wrapper and this wrapper gets very very unwieldy very quickly So now skip that part and just focus on on the train loop now um So here we are here's the train loop. Let's enter the train loop. Okay, so we're actually hitting this Aggregate function i'm going to show you what it is So i'm gonna hit i'm gonna enter i'm gonna hit f12 to train i'm gonna put the breakpoint here You can see there is this um wrapper. They're using this decorator Aggregate and it's that's just going to make sure that when they're logging something inside of that scope That all of those logging details are being stored into this particular train Dictionary and then they'll have multiple of these, uh aggregate Uh, um decorators and that's just gonna gonna like nicely organize where each of the logs is going to so they know exactly From which part of the code which logs came from that's the basic idea Okay, so we're gonna just um hit f5 and exit this and hit the initial part of the train function Let's just grab the iterator. I can skip over all of this Um, basically iterator is just Uh as the name suggests counting iterators, so it's just gonna feed us those 128 Tokens in in on each call That's it. We grab some progress bar again. I'm gonna skip all of that again. I told you this is this is what it How the code looks like when you're training? Uh bigger models like there is a lot of things that you don't care about because they're conceptually not important But then again, they're super important if you want to make sure if you want to be able to debug Uh things when they go wrong, so i'm gonna skip over all of this. Let's get to here Here is the actual train function. So there is train instead of train instead of training again A lot of layers of of kind of complexity. Uh, let's skip over all of this So here is the definition of the train we hit f5 we get here and this is where the actual magic starts happening So we we're not Gonna iterate and fetch the sample. We will not enter this particular branch because this new profiler is set to false Um, and so we'll we'll just hit this branch here And now i'll hit the uh f11 and enter this train function and instead of this train function There is yet another train function. So let me hit f5 and we end up here so we end up here train step so again, uh the The outer train is the epoch train. This one is the batch trick This one is going to be like the batch train. So we pass just the samples and we do a single forward pass So let's see what the samples are samples Samples is a dictionary as you can see we have all those tokens from the dummy data set loader Two interesting things to notice again. They have this aggregate thingy going on It's kind of convenient. It might be confusing if you're not used to it, but uh, it It does the job and secondly they have this profiler record function So what this thing does is first of all, they enable the profiler. I'm going to show you that in a second Secondly, this record function basically names this particular part of the code with train underscore step dash Some number of the iteration and that's useful because later on the when you analyze the profiler output you'll you'll exactly know How long or like you have the details the profiling details about this particular piece of code? That's why they do it. That's why they have the record function here So again, the profiler should have been set up somewhere here Uh, let me just see. Yeah So when we were here there was the profiler call here and you can see profile memory true with stack true record shapes true So that means they'll be profiling the memory. What was the memory footprint like? In case they're hitting oms this helps you debug it Recording the shapes of the variables and also stack. Um, if I recall correctly Uh, um, yeah, I forgot what the command is. You can always enter the profile here and see what the um Stack means so let me just find that variable Okay, record source information file and line number for the operation. So additional metadata for for the operations Let's go back here and let's enter the train step Enter the train step again. I'll have to it keeps on entering these aggregate Functions that kind of sucks. So I have to enter the train step using f12 set the break point and then hit f5 So now we are here. So let's see what's going on So we set the seed we set the models to train criterion to train We call zero grad and i'll actually enter zero grad and there is a particular reason i'm doing that Let me let me show you what why is that why that is? Okay, let's enter this one Let's enter this one Let's enter This one. Oh my god And here it is So the reason is we have this lost scaling and that's because we're doing mixed precision And so lost scale is set to four and if you recall from my video on again the ultimate guide to scaling Why they do this is because you want to shift all of the values such that you're you're you're less likely to hit the underflow in the fp16 precision So this kind of shifts all of the values and that's why they uh do One over Well, they're doing one over four because the multiply factor is going to do the unscaling So we'll see the this is going to do the scaling and the multiply factor is going to do the unscaling Okay, we'll see that a bit later, but that's a detail. I wanted to show you so that's the zero grad. Um, Let's continue here We grab the sample we do prepare sample which is gonna do two things One is to push the cuda and the second one is to convert to fp16 because of that. I'm just gonna skip Um, let's enter The the final train step and it's happening instead of a task, which is kind of unintuitive Why would you why would the train step be associated with a task? like this Like this way of structuring the the the the programming model is kind of weird and does not resonate with my with my mind Okay, guys, let me enter that one. I'm gonna hit f11 And here we are. This is the train step again. Take a look at the stack here Like the stack is horrible. We have train train train step instead of a trainer train step inside of a task But it is what it is It's kind of ambitious to try to explain this code on the youtube. So yeah I'm giving my best so again record function forward. So we'll know exactly the details of this part of code We'll know how much memory is spent. We'll know the shapes. We'll know the lines everything so that kind of makes it easier to debug Let's enter this f11 Here we are. We do a forward pass through the model. We passed on that input. That's gonna pass us the the actual tokens So here we are. We hit the coder. We pass the tokens to the decoder And now we call this extract features function the extract features function is gonna do a forward pass through the Decoder transformer. I'm not sure why they called it like that I mean, it's not incorrect to to to name it like that but kind of weird extract features. Okay um Let's continue here Let's enter the extract features and instead of extract features There is the extract features scriptable and this the reason why they do this is because of torch script There was again some some constraint that if you want to have optimization you kind of have to do this and And it makes the code horrible. I think they know they kind of mentioned that somewhere here Blah blah. I don't see it now, but I yeah Um a scriptable subclass of this class has an extract features method and calls super extract features But super is not supported in torch script a copy of this function is made to be used in the subclass instead Okay Let's continue here Let's continue here. Okay. This part is not interesting. So the first thing we do is forward embedding so that's gonna take our our Basically tokens so from two to whatever we have 128 of those And we are going to now convert them. We're gonna map them into Multi-dimensional space. So let's let me enter this part So let me enter this forward part and let's see what's going on So basically we just call All yeah, this is the uh, the torch embedding. So it's just gonna embed basically everything out of the box basically Okay, so this was actually the position. Okay, we first embedded the positions. Let me see. Where is the actual embedding going on um Okay, so here's where where we embedded tokens so we embed them here. So now we have token embeddings So let me show you this so we have token embeddings the shape of this thing is 128 512 let's see. What's the shape of the position? Positions it should be also 128 but we have 768 here. So that means we we would have a mismatch So because of that we'll have to first do the projection um Okay, embedding scale is set to one. So this is kind of no up we do the projection From 512 to 768 and now we can add positions on top of x So that's the initial part of the transformer logic We do the embedding we add on top of it the positionally encodings and now we start doing the forward pass Okay, let's continue here and let's exit we're returning back The embeddings the positions and the combined embeddings and positions after we do the additional projection. Okay, let's exit this part We are again in the forward pass of the transformer decoder You can always take a look at here to to figure out where we are transformer decoder We're doing the forward pass. We've done the embedding part here. We create the mask the Attention mask. So let me show you how it's going to look like i'm gonna kind of quickly walk you through this part It's going to create this Tri u is going to create a triangular matrix It's going to be filled with bunch of minus int So negative infinity and that's going to be used to create a causal decoder, right? So we'll just add that to the attention scores and that's going to after applying the softmax You'll end up with tokens that can only attend to itself and the previous tokens So again here we create 128 times 128 Matrix of zeros then we fill it with negative infinity and then we just call this This function that will basically create a triangular map mask out of this. So let me step over this and let me show you how this looks like Here is how it looks like again Zeros and then on this upper the upper triangle is minus infinities Okay Let's continue here. We're not using a lib so we can step over all of this and we just return back that that mask We just created. Okay, so that's the attention mask We do some transpose here operation. So because we are batch number of tokens number of channels now we are Uh number of tokens and then batch and the number of channels again optimization detail the lot of transformer implementations do Okay, we just store some interstates blah blah. Now we do A bunch of passes through all of the layers. We have 12 layers and we are going to do four pass through all of them i'm going to skip the actual layer logic Uh, that would be too much i'm going to skip it, but it's just classic transformer stuff. So let me exit this Let me exit here uh, let me just Put breakpoint here hit f5 and let's exit the transformer We do some again now we now we have again, let me show you the um So we now passed all those vectors through the transformer 12 layers in total And we end up with the shape 128 768 and so you see it's kind of messed up. So we just do the transpose operation again and now we end up with the correct um the usual shape Okay, that's it. And finally the projection so we have 512 instead of 768 Let me do f10. That's it. We return the interstates. We return the attention blah blah blah And we turn x which are the actual output embeddings Okay, let's continue. That's the that part and now we call the output layer And that's gonna project this into I guess, um vocab size. So yeah, you can see here 51 200 Let's continue Let's continue and that's the forward pass through the model now we compute the loss Um, i'm gonna kind of skip through this a little bit So we first to get normal probabilities. It's gonna this is gonna apply the log softmax on top of our um outputs here then we do some reshaping and Let's see the shapes And those of you who are still watching I'd love to hear your feedback. Like if you have any feedback, it would be super appreciated. So 128 51 1,200 whereas now these are normalized because we applied the softmax. Okay, so We get the targets. This is just going to fetch those sample targets so that's going to fetch those 128 tokens that were shifted by one to the right I'm gonna convince you that that's indeed the case so you can see here three four five That's the target sequence we were using the whole time and now we apply the nl loss. So that's the Um negative log likelihood loss and you can see it's a bit more complex than than your usual implementation So your usual implementation will be just this but when you're dealing with um I guess uh, like big dimensionality and a lot of a big vocab Then you can see here they have this if branching. So if the number of elements is smaller than What is this like? 2 billion then you just execute this standard f and l loss if it's not the case then you have to do some some a bit more intricate logic because otherwise, um, you would encounter some uh, Numerical issues so because we are smaller than this Let's see what the number of elements is for us. We have 51 000 and then Dimensionality is 128. So I guess it's yeah, it's below The threshold so we just do this. Um Negative log likelihood and that's it. That's everything That's how transformers work like you apply just you just apply this nl loss on the output and Compared with the targets that are shifted by one and that's it the whole magic behind transformers Okay. Now here's where the actual mixed precision magic happens and we are almost done with this video I'm gonna quickly walk you through all of this. We're just like logging some outputs. That's not important And i'm gonna skip all of this We're just storing some stuff blah blah blah. We're just storing some stuff. Let's go back here Let's return back the loss. So here is the loss and now let's go for the second part So this is the interesting part So we do backward pass and we again do record function backward to just profile this piece of code And let's do um, let's do f11 here. Let's enter so now because we have scaling. Let's see what happens. So um Scaler is not none. So we enter this branch And what we do is we scale the loss, okay So whatever the loss is and it's thousand four hundred something and it's that big because we were just doing summation in the nl loss As the reduction method So if I enter f11 here, you can see what what what we do is we simply take that number and we multiply it by four And that's what I previously explained you because of that That when you multiply the loss by four you trivially multiply all of the gradients are going to be uh multiplied by four that's how the Chain rule works. So let's hit f10 Let's now do the backward pass. I don't think anything special is going to happen here I'm just gonna yeah Everything is yeah, we are we exited the the um train step and now let's Let's go back. We are not we're now level up in the train step of the trainer not in the train step of the task Let's hit f10. We just append the outputs blah blah blah. That part is not interesting and okay, let's continue so here you can see this is kind of uh wrapped up inside of a uh, runtime error out of memory exception so that the whole training procedure is wrapped up in that uh, and um I think this is this is it. We just have a single batch. That's why we exit the this this for loop through the samples and um Let's continue here sample size was 128 because we only had 128 tokens Um, this is some logic that will be necessary if we had multiple machines, which we don't so that's fine Uh, okay, so we now enter this interesting part And this is pretty much the end of the video. So try accept as you can see here There is so many accepts they're catching so they're catching the floating point error. They're catching the overflow error They're catching the outer memory error So all of these things are necessary when you're training at these big scales You want to catch the errors you want to handle them gracefully so you can see here If there is a floating point error, I think they do have this nan detector. Yeah, we'll get to that a bit later Let me now first walk you through the uh reduced gradients So here actually nothing is going to happen because we have a single gpu. Otherwise, uh, you would take Uh gradients from across the devices and then you would do the all reduce So that means you literally take all of the gradients you push them onto a single device You divide them by the number of devices and then you do that for all all the devices and all of the devices will have The same state now If that makes sense, but less important for now, let's focus on the loss scaling Multiply grads. Let's see what's going on here. Okay. So what we do is We divide so we have this multiply grads and we divide this is one divided by 128 the reason we divide by 128 is because nll loss had as a reduction method had a sum And because we had 128 elements, we now want to divide by 128 if that makes sense So let's enter here Let's see what's going to happen here uh, so Multiply grads. So we just take that multiply factor, which is now one over four because we want to do the unscaling of gradients And we additionally multiply by this one one over 128. This is the c constant That's that number and then that's kind of all aggregated inside of this variable and we're going to later apply the variable to actual gradients We'll see that a bit later. Okay, let's exit here Now gradient clipping is going to happen again, I will have to skip this part, but um, Okay. No, actually let me quickly enter it. Let me see what's going on there. I think it's kind of fairly intricate There's a lot of as you can see a lot of steps, but I think there might be a part that's kind of interesting for us Oh, yeah, there is so here is a multiple factor. Here's where it comes into the picture. So we we basically grab the gradients of our we grab the norm of the gradients and then we multiply that by the The multiply factor and that's kind of optimization detail because you didn't have to multiply the actual gradients You just took the norm and then Multiply by this factor and you get the equivalent result. That's kind of neat Okay Let's see what's going on here. Let's continue here max norm grad norm And again, we have another multiplication for the multiply factor. Why because The gradients were the norm was 16 but we want to clip it to to 1 and that means we have to we have to take this number and Multiply with that number of gradients and then they will obey this property of having gradient norm of 1 if that makes sense That's why we have another multiplication. So now the multiply factor has three things going on We have the loss on scaling we have we've divided with a sample size and finally we've divided by this ratio of the norms Let's continue we check some overflow again a bunch of details And let's continue we check the grade norms. Let me see what this is I Think this is just going to be skipped. So yeah, it's just skipped. Okay. Now we make sure that we don't have none of the values in the grads are infinite so if we had nans then this would be This exception would be raised and it would be caught in this particular context here where they have the nan detector That kind of let me show you let me show you what it does. So nan detector Blah blah blah. It basically is going to capture Some of the values so that we can debug it later on detects the first man or in in forward and or backward pass and logs Together with the module name. Okay, so it's just gonna help her to to debug Where the man occurred? And then once you have once they do that they do another for pass here Under that context and it's gonna catch the first man So they say here rerun the forward and backward pass with hooks attached to print out where it fails Okay, so that's kind of how they cope with the errors Let's finally enter the optimizer step. That's where we're gonna finally apply the multiplier the famous multiplier So let me show you here step step step Let's continue here. Okay unscaled grads. Let's see what it is. I think that's going to be our our logic So here we are we enter this part multiply grads with the multiply factor Okay, that's finally the moment where we apply the multiply factor and then we reset it back to one And then the whole procedure will will continue So here you can see here we just go through the parameters and we multiply the so parameter the gradient associated with that parameter with that weight Data and then multiply by c and c is the the the multiplier we were accumulating over time So we had three factors again, if you recall the sample size the gradient on scaling and finally the ratio of the norms I'm gonna Step out of this. So hit this step out Error and we are done. We are done here. We've done the stepping We update the scalar. So the scalar is going to do what? Okay, nothing interesting here We can exit all of this and we are back to our train step in the trainer We just debug some information we didn't have any nonce and that's pretty much it So I think I can skip all of this. I can skip all of this a lot of details Yeah, go at your own pace explore this code if you're curious to learn more But uh, I think we saw most of the interesting parts. So let me just do this Let me see why we are okay. So i'll just have to put f breakpoint there hit f5 here. We are So as you can see, we're finally back in the in the in the in the train.py file And we are inside of the train function here Let me see whether there is something interesting And the feedback, uh, we're gonna just skip this part. Nothing's gonna happen. It's gonna be a no-wisdom Um, and uh If we should stop And for now we are not stopping so that means we are again gonna fetch the next samples and uh Continue this whole Loop and that's it. So we're just gonna keep on doing this training Guys, that's it. Uh, you hopefully saw what it takes to Kind of saw what it takes to build these llms. Um, we did Exclude the whole distributed part of the equation, which is probably the hardest part So like doing the model of parallelism the data Parallelism the the pipeline parallelism all of that Goodness, and then there is a zero optimizer where you do the partitioning of the activation. So there is so many details Again, we also skipped the checkpointing. So like Yeah tragic cool. I'm gonna stop here Let me know please let me know what you think Um, depending on that i'm gonna probably modify my my my uh, the videos i'm gonna be creating because By doing this I learned a lot and so I thought just kind of walking you through Uh the learning so it didn't feel like like a big of an effort But like going forward I'll probably won't be doing this unless people find it useful So do let me know down in the comments whether this was actually interesting and useful for you Having said that if you like this video, please share it out. That's the best way to do it This channel also subscribe to this channel if you haven't already and see you next time
[{"start": 0.0, "end": 7.0, "text": " What's cracking guys? Alex here. In this video, I'll be walking you through the code base behind the recent Opt"}, {"start": 7.84, "end": 9.44, "text": " 175B"}, {"start": 9.44, "end": 14.58, "text": " Transformer from Meta and if you're curious to learn about the actual theory about the paper"}, {"start": 14.58, "end": 21.14, "text": " I covered this paper as well as GPT Neo X20B and as well as Big Science blue models"}, {"start": 21.44, "end": 28.080000000000002, "text": " So the corresponding papers in the previous video, so I'm gonna link the card somewhere up there in this video"}, {"start": 28.08, "end": 31.56, "text": " I'll be walking you through as I said through the code base behind the opt model"}, {"start": 32.04, "end": 34.72, "text": " basically, that's the meta sec code base and"}, {"start": 35.239999999999995, "end": 39.9, "text": " My idea will be to first show you how to set up this on your local machine"}, {"start": 40.12, "end": 45.78, "text": " Because the system was designed the code base was designed to be run on the Linux distributed system"}, {"start": 45.78, "end": 52.3, "text": " So that means multi node multi GPU setting whereas I have windows single GPU machine"}, {"start": 52.3, "end": 54.980000000000004, "text": " So that kind of made it a bit harder to get started"}, {"start": 54.98, "end": 60.31999999999999, "text": " So that's the first thing the second thing I want to show you some of the concepts I introduced in my"}, {"start": 60.739999999999995, "end": 64.66, "text": " Ultimate guide to scaling video. I'm gonna link the card as well somewhere here"}, {"start": 65.53999999999999, "end": 73.52, "text": " Basically, I'll focus on the mixed precision training and I'll show you some of the concepts such as loss scaling"}, {"start": 74.14, "end": 78.69999999999999, "text": " Etc and also you'll just get to see how this product well not a production level"}, {"start": 78.69999999999999, "end": 84.8, "text": " But like a code base behind one of these large language models looks like so hopefully that's gonna be valuable"}, {"start": 84.8, "end": 90.53999999999999, "text": " Okay, let's get started. The first thing we need to do is open up the setup guide"}, {"start": 90.7, "end": 96.17999999999999, "text": " So there is the setup instructions here, which we're going to follow. It's fairly straightforward"}, {"start": 96.17999999999999, "end": 101.69999999999999, "text": " There's a couple of caveats, but that's pretty much it. So I'm gonna open up my anaconda prompt here"}, {"start": 102.17999999999999, "end": 106.78, "text": " I'm going to first create a conda environment. So conda conda create"}, {"start": 107.42, "end": 110.52, "text": " Name I'm gonna use opt to because I already have"}, {"start": 110.52, "end": 116.36, "text": " Opt environment setup so I have to change the name there and I'm gonna use the Python 3.9"}, {"start": 116.36, "end": 118.88, "text": " so I'm gonna hit enter there and"}, {"start": 119.39999999999999, "end": 123.39999999999999, "text": " This will be quickly over and once we have conda"}, {"start": 123.88, "end": 130.44, "text": " Installed will start installing the necessary packages directly inside of it. Okay, this is gonna be quite fast. So activate the"}, {"start": 131.2, "end": 132.16, "text": " environment"}, {"start": 132.16, "end": 136.76, "text": " Clear just to clear the screen a bit and now I'm gonna just copy paste this command here"}, {"start": 136.76, "end": 144.28, "text": " So we do this pip install of basically torch and torch vision etc. So just hit this command and"}, {"start": 144.6, "end": 151.12, "text": " That should execute in a couple of minutes. Okay guys, so that's that's installed now. Let's continue"}, {"start": 151.12, "end": 157.32, "text": " So the next step would be to install the apex library from from Nvidia because I'm on windows"}, {"start": 157.32, "end": 161.92, "text": " This does not work windows is not supported so you can follow these steps on your own if you're on windows"}, {"start": 161.92, "end": 164.64, "text": " Just skip this part. There is a bit better"}, {"start": 164.64, "end": 170.83999999999997, "text": " instruction for how to install apex in case you have some mismatch of the CUDA versions, etc, etc and"}, {"start": 171.27999999999997, "end": 178.39999999999998, "text": " Like some some basically tips on how to overcome those potential problems in this big science"}, {"start": 179.51999999999998, "end": 185.64, "text": " Basically repo I'll link that link down in the video description so you can check it out. So I'm skipping the apex"}, {"start": 185.64, "end": 191.61999999999998, "text": " Let's now continue on installing the Megatron. So let's just copy paste this part here"}, {"start": 191.62, "end": 198.54, "text": " Well before that, sorry, let's just first clone the the actual repo. So I'm gonna do the following and I'm just create a"}, {"start": 199.26, "end": 201.22, "text": " Directory here. I'm gonna name it opt"}, {"start": 201.22, "end": 207.86, "text": " I'm going to just do a clone operation inside of the directory if I manage to open up it the folder"}, {"start": 207.86, "end": 209.86, "text": " So here's the the git bash"}, {"start": 210.14000000000001, "end": 218.86, "text": " Just go and basically copy the link the URL do git clone here and that's gonna download the actual meta sec"}, {"start": 218.86, "end": 222.42000000000002, "text": " repo in the in this target directory"}, {"start": 223.06, "end": 228.38000000000002, "text": " So once we are there, let me just navigate using my my environment. So I'm gonna open up this thing here"}, {"start": 228.38000000000002, "end": 235.42000000000002, "text": " I'm gonna navigate here in the in the console to that to the root of my code base and"}, {"start": 235.96, "end": 240.94000000000003, "text": " Now I'm gonna follow the rest of the instructions. So let's now do the git clone"}, {"start": 241.42000000000002, "end": 243.9, "text": " So let's do the Megatron installation"}, {"start": 243.9, "end": 249.38, "text": " You can just copy paste this directly in the terminal everything is gonna execute automatically except for the last line"}, {"start": 249.38, "end": 251.38, "text": " So hit the pip install"}, {"start": 251.58, "end": 253.18, "text": " command here"}, {"start": 253.18, "end": 260.18, "text": " That's gonna install the Megatron. Hopefully fairly quickly. Yeah. Okay, that's cool. Let's continue on the second step"}, {"start": 261.1, "end": 268.22, "text": " now the important thing you need to do here is just change go go back to the root to to meta sec and"}, {"start": 268.22, "end": 273.34000000000003, "text": " Don't install don't obviously don't you don't want to install inside of the Megatron the fair scale library"}, {"start": 273.34, "end": 279.4, "text": " So I'm just gonna copy paste this again here. We're installing the fair scale and I hit pip install"}, {"start": 280.0, "end": 285.29999999999995, "text": " For the last line and we'll install the fair scale as well. It's kind of smooth so far"}, {"start": 285.65999999999997, "end": 288.29999999999995, "text": " No, no problems here. Okay, finally"}, {"start": 288.97999999999996, "end": 293.47999999999996, "text": " Because we've already installed we already cloned the meta sec. We just need to go"}, {"start": 294.09999999999997, "end": 299.58, "text": " to the root so change directory to the root and now we do pip install but before that there is a"}, {"start": 299.58, "end": 304.3, "text": " Additional tweak I have to do and that's in the setup file here"}, {"start": 304.3, "end": 309.74, "text": " I have to basically open up this this file and find the"}, {"start": 309.97999999999996, "end": 313.94, "text": " Aim library and this one also doesn't work on Windows for whatever reason"}, {"start": 313.94, "end": 316.41999999999996, "text": " I just didn't want to hustle around it"}, {"start": 316.41999999999996, "end": 323.74, "text": " so I just kind of comment it out and that's it after this you can hit the the pip install command and"}, {"start": 323.74, "end": 330.26, "text": " Everything should work as expected. Let me paste this likely enter and that's gonna be installed now"}, {"start": 330.34000000000003, "end": 334.90000000000003, "text": " Okay, there is one more command we have to execute and that's not in this manual"}, {"start": 334.90000000000003, "end": 338.98, "text": " So that's this pip install setup tools this version 59 point five point. Oh"}, {"start": 339.86, "end": 343.1, "text": " There is some weird version error going on if I don't do this"}, {"start": 343.1, "end": 349.14, "text": " So I had to I kind of found this on the on some random website as a solution and finally"}, {"start": 349.14, "end": 355.26, "text": " This is not necessary unless you want to to like basically contribute to the to the repo this pre-commit install"}, {"start": 355.26, "end": 361.46, "text": " But let's just do it either way. Okay, cool. I'm gonna clear the console here. Everything is ready now now"}, {"start": 361.46, "end": 367.58, "text": " Let's open up the actual code. So I'm gonna use VS code as usually VS code here and"}, {"start": 368.62, "end": 371.97999999999996, "text": " I'm gonna just do the whoops open folder"}, {"start": 372.7, "end": 376.82, "text": " Find this meta sec directory and just select folder"}, {"start": 376.82, "end": 378.82, "text": " That should be pretty much it"}, {"start": 378.82, "end": 385.82, "text": " So now what we want to do is select the correct interpreter and that's the opt-to so let's just open something"}, {"start": 386.18, "end": 390.9, "text": " Let me open up some of the files will need a bit later. So this is gonna be the training file"}, {"start": 390.9, "end": 393.34, "text": " Let me hit the nope. I don't want to dead"}, {"start": 393.34, "end": 399.7, "text": " So I just want to hit the ctrl shift P select interpreter and let's select the opt-to"}, {"start": 400.18, "end": 404.98, "text": " Interpreter here and that's now it the final thing we have to do is a couple of patches"}, {"start": 404.98, "end": 409.74, "text": " Obviously I had to sort it out before before the video. So let me just show you what those are. I"}, {"start": 410.06, "end": 416.3, "text": " Have a working code here. So just to get this tool to show you some of the differences"}, {"start": 417.3, "end": 421.86, "text": " Some of the patches I had to apply so first thing was and we've already done"}, {"start": 421.86, "end": 426.26, "text": " This is to comment out the aim this might not be necessary for for your case"}, {"start": 426.26, "end": 430.74, "text": " But just like that's something that I had to do. Okay, then there is this line in trains"}, {"start": 430.74, "end": 436.58, "text": " I'm just gonna copy paste this one. You obviously won't have to do this because I'll basically create a clone"}, {"start": 437.18, "end": 442.46000000000004, "text": " Directly and and push it to my to my github, but let me let me return here"}, {"start": 443.26, "end": 445.26, "text": " So this is something we have to add"}, {"start": 445.86, "end": 449.06, "text": " Basically here. I'm gonna just save it there"}, {"start": 450.3, "end": 454.22, "text": " Then let's see what else so let me exit here"}, {"start": 454.90000000000003, "end": 456.90000000000003, "text": " We have a couple more steps"}, {"start": 456.90000000000003, "end": 458.90000000000003, "text": " Metasec tasks"}, {"start": 458.9, "end": 466.09999999999997, "text": " Metasec tasks base task we have to add this weird line to convert it to long"}, {"start": 466.46, "end": 471.53999999999996, "text": " Otherwise, he's gonna complain. So that's the base task inside of meta task tasks"}, {"start": 471.62, "end": 473.62, "text": " So let me open up that file again"}, {"start": 473.62, "end": 478.15999999999997, "text": " You will have you can skip this all of this because you can just start from my from my code"}, {"start": 478.21999999999997, "end": 485.78, "text": " So I'm gonna do this so meta sec find the tasks find the best base task and find this line"}, {"start": 485.85999999999996, "end": 487.85999999999996, "text": " so this is basically"}, {"start": 487.86, "end": 490.78000000000003, "text": " Let me find this thing so filter oops"}, {"start": 491.42, "end": 493.42, "text": " filter examples"}, {"start": 493.98, "end": 495.98, "text": " blah blah blah, so it's here"}, {"start": 495.98, "end": 499.14, "text": " So I just paste that line here. It can be cleaner. Obviously"}, {"start": 499.14, "end": 502.78000000000003, "text": " I'm just doing it like this like you should import it in on top of the file"}, {"start": 502.78000000000003, "end": 506.42, "text": " But this is just the whole point is for me to enable this code"}, {"start": 506.42, "end": 511.3, "text": " So that I can easily run it and and do a stepping through of the code base"}, {"start": 511.3, "end": 515.3000000000001, "text": " Okay, so I think that's one step then we have the sweep"}, {"start": 515.3, "end": 518.66, "text": " We can actually skip this part. We have we don't have to do this"}, {"start": 519.3, "end": 524.26, "text": " Then there is this thing so we have to change from 2 to 1 because I only have one GPU"}, {"start": 524.26, "end": 528.42, "text": " Otherwise this will fail because as you can see here, this is the model parallel argument"}, {"start": 528.42, "end": 534.5799999999999, "text": " So I'm just gonna do that. So that's meta sec launcher up job constants. I'm gonna just patch that quickly here"}, {"start": 535.38, "end": 537.38, "text": " so let's go to the"}, {"start": 537.62, "end": 543.62, "text": " Metasec launcher so meta sec here is the launcher here the constants and"}, {"start": 543.62, "end": 546.98, "text": " We just put ones here. These are the small smallest models"}, {"start": 546.98, "end": 554.02, "text": " You can see the 8 million the 125 million and the 350 million the tiny small and medium ones. Okay"}, {"start": 554.5, "end": 558.34, "text": " And the final thing we have to do is in the up baselines"}, {"start": 558.82, "end": 564.42, "text": " We have to comment out this if arcs number of GPUs smaller than 2 then raise an error"}, {"start": 564.74, "end": 567.0600000000001, "text": " Because obviously I have just single ones"}, {"start": 567.0600000000001, "end": 571.94, "text": " I'll have to do that and there is this one here with the environment thingy"}, {"start": 571.94, "end": 578.2600000000001, "text": " So it was obviously a lot of trial and error to to to get this uh, like sorted out"}, {"start": 578.82, "end": 580.2600000000001, "text": " but yeah"}, {"start": 580.2600000000001, "end": 583.46, "text": " Okay, let me find the uh up baselines"}, {"start": 584.2600000000001, "end": 586.2600000000001, "text": " so arcs"}, {"start": 586.4200000000001, "end": 588.4200000000001, "text": " Num of GPUs"}, {"start": 588.74, "end": 591.22, "text": " Here it is. So I have to comment out this line"}, {"start": 591.94, "end": 597.3000000000001, "text": " And the final patch that I have to apply is this one here. I'm just gonna do"}, {"start": 597.3, "end": 602.02, "text": " Copy paste of this on top of on the bottom of the file and that's it"}, {"start": 603.3, "end": 605.3, "text": " Okay guys, that's pretty much it"}, {"start": 605.62, "end": 611.9399999999999, "text": " Okay, having done all of this, let's go back to the repo and now we want to train this thing"}, {"start": 611.9399999999999, "end": 616.3399999999999, "text": " So let's see what the instructions say so they have the this training link here"}, {"start": 616.3399999999999, "end": 620.74, "text": " You open it up and the only thing you have is this so from here"}, {"start": 620.74, "end": 624.3399999999999, "text": " I have to had to work my way around and figure out what's going on. So"}, {"start": 624.34, "end": 628.1800000000001, "text": " Let's let's see what i'm gonna do. So i'm gonna instead just copy paste the uh"}, {"start": 628.9, "end": 632.1, "text": " The launch arguments I already have in the in in my code"}, {"start": 632.9, "end": 634.9, "text": " Which I've already previously"}, {"start": 635.5400000000001, "end": 641.14, "text": " Been running so i'm just gonna copy paste all of these arguments here and you'll see what's the difference basically"}, {"start": 641.14, "end": 643.14, "text": " Let's just give me a second here"}, {"start": 643.7, "end": 645.7, "text": " So i'm gonna open up this one"}, {"start": 646.1, "end": 648.26, "text": " Of baselines i'm gonna create"}, {"start": 648.26, "end": 654.66, "text": " Uh, basically a config file here for the for the for running the code and let's just paste those arguments"}, {"start": 654.9, "end": 659.22, "text": " So let's see what's the difference between this thing and what they show here"}, {"start": 659.7, "end": 663.86, "text": " So first things first, uh, you can notice I have the benchmark in the local"}, {"start": 664.26, "end": 668.74, "text": " The local makes it so that I can run this on the local machine obviously and the benchmark"}, {"start": 669.22, "end": 671.22, "text": " Was necessary to basically"}, {"start": 671.9399999999999, "end": 674.66, "text": " So that you can skip downloading bigdp"}, {"start": 674.66, "end": 682.42, "text": " So that you can skip downloading big data sets and this will just load some use some dummy data loaders and"}, {"start": 682.98, "end": 689.3, "text": " Everything else will stay the same. So that's kind of cool. Okay, so number of nodes was obviously I had to change it to one"}, {"start": 689.62, "end": 691.62, "text": " It was two here"}, {"start": 691.6999999999999, "end": 694.98, "text": " And number of gpus I had to set it to one instead of eight here"}, {"start": 695.9399999999999, "end": 698.98, "text": " And other than that everything else remains the same"}, {"start": 699.62, "end": 703.6999999999999, "text": " Okay, so now let's run this code. I'm gonna zoom in a little bit here"}, {"start": 703.7, "end": 709.3000000000001, "text": " And i'm going to set a breakpoint here and let's just run this code. Okay, let me run this"}, {"start": 710.9000000000001, "end": 712.9000000000001, "text": " Hit run"}, {"start": 713.0600000000001, "end": 716.26, "text": " And the whole point of this script will be to collect the arguments"}, {"start": 716.9000000000001, "end": 722.9000000000001, "text": " That then we are going to use for the train script because they don't have any instructions for how to start the actual train script"}, {"start": 723.22, "end": 725.22, "text": " So this is a workaround. I kind of figured out"}, {"start": 725.7, "end": 728.6600000000001, "text": " um, let's see what's going on here, so we have to"}, {"start": 729.46, "end": 731.0600000000001, "text": " Go here"}, {"start": 731.06, "end": 737.78, "text": " And then sweep main so this is maybe helpful for you to understand my my process and then basically"}, {"start": 738.66, "end": 745.78, "text": " In case you need to figure out something on your own, uh, you can you can follow a similar workflow. So we end up here"}, {"start": 746.7399999999999, "end": 751.38, "text": " Then we enter this uh back end main i'm gonna put a breakpoint here"}, {"start": 751.6999999999999, "end": 755.14, "text": " Let's just continue the whole point was to find so let's continue here"}, {"start": 755.4599999999999, "end": 760.5799999999999, "text": " The whole point was to find the actual command line that's being used to call the train script"}, {"start": 760.58, "end": 764.5, "text": " So we'll see how that looks like in a second. Okay, let me do this"}, {"start": 764.82, "end": 769.14, "text": " Let me just find uh the so train commands. We don't care about this code"}, {"start": 769.7, "end": 775.7, "text": " We just want to find the the the final code that's going to be run. So i'm gonna put uh"}, {"start": 776.34, "end": 778.34, "text": " Breakpoint here hit f5"}, {"start": 778.9000000000001, "end": 783.86, "text": " Uh, let's continue here. Let me just find yeah local run. So this is where we want to go"}, {"start": 784.82, "end": 786.74, "text": " um f5"}, {"start": 786.74, "end": 793.78, "text": " Enter the local run and here it is. So here is the actual piece of code that will call the train script"}, {"start": 793.94, "end": 800.02, "text": " So we have this train command. So it's python blah blah blah. It's calling as you can see here, hopefully"}, {"start": 800.58, "end": 801.7, "text": " um"}, {"start": 801.7, "end": 808.5, "text": " Let me just see whether I can shift this. Okay. So what i'm gonna do is this i'm gonna open up the debug console here"}, {"start": 809.62, "end": 811.62, "text": " and i'm simply going to"}, {"start": 811.62, "end": 816.98, "text": " Capture all of these arguments which we will then use to to to run the the the train script"}, {"start": 817.3, "end": 824.18, "text": " So we don't care about this n thingy. Uh, we only care about about the uh actual commands that were passed to the train script"}, {"start": 824.5, "end": 831.62, "text": " Okay, you can see here python blah blah. It's calling the train script. So this is the thing you can do. So we open"}, {"start": 832.18, "end": 834.18, "text": " um, basically command"}, {"start": 835.22, "end": 837.22, "text": " Uh take take stay"}, {"start": 837.62, "end": 839.62, "text": " uh, and then write mode"}, {"start": 839.62, "end": 844.18, "text": " Uh as f and then we basically want to write to that file"}, {"start": 844.66, "end": 848.18, "text": " This train command and this is going to obviously save"}, {"start": 848.82, "end": 856.18, "text": " This command and capture it and so i'm gonna do that and let's see what's going on. Okay, so we've saved that somewhere here"}, {"start": 857.54, "end": 859.54, "text": " Hopefully, yeah commands"}, {"start": 859.54, "end": 863.46, "text": " And then you can just copy paste all of these and start playing from there"}, {"start": 863.62, "end": 865.62, "text": " So that was my first part of the process"}, {"start": 865.62, "end": 870.34, "text": " So it took some time to figure out how to how to call the train script and this was my workaround"}, {"start": 870.9, "end": 875.22, "text": " To to get the actual commands and then came the tweaking part. So i'm gonna just again"}, {"start": 876.1, "end": 879.54, "text": " Kind of speed this up by by copy pasting the actual"}, {"start": 880.26, "end": 883.86, "text": " Arguments you can see it's super duper verbose"}, {"start": 884.9, "end": 891.38, "text": " Um, i'm gonna copy paste all of this and i'm gonna well i'm actually gonna copy paste everything"}, {"start": 891.38, "end": 898.26, "text": " Including these comments I made here so I can explain what i've done. Uh, so let's go to the code here again"}, {"start": 898.26, "end": 900.26, "text": " You will not have to do this"}, {"start": 900.34, "end": 903.78, "text": " because because um, you'll have this already"}, {"start": 904.26, "end": 908.98, "text": " Uh in my in on my github, uh repo, okay, let's go here. Um"}, {"start": 909.54, "end": 912.36, "text": " Let's rename this as as opt baselines"}, {"start": 913.14, "end": 915.46, "text": " And i'll have to create another config here"}, {"start": 916.34, "end": 918.34, "text": " For the train run, I will just"}, {"start": 918.9, "end": 920.58, "text": " add configuration"}, {"start": 920.58, "end": 925.62, "text": " Python python file and then i'll just add the arguments here. Okay"}, {"start": 926.58, "end": 931.0600000000001, "text": " That's it. Hopefully if I haven't made any mistakes here, we don't need this"}, {"start": 931.7800000000001, "end": 934.74, "text": " And that's pretty much it. I'm going to rename this file to"}, {"start": 935.5400000000001, "end": 937.5400000000001, "text": " This this config to train script"}, {"start": 938.6600000000001, "end": 944.1, "text": " Uh, and let me show you what's going on here. So a lot of arguments obviously, uh, this is um"}, {"start": 944.74, "end": 950.1800000000001, "text": " What you get when you're working with large language models and in general with with libraries such as such as metasec"}, {"start": 950.18, "end": 951.78, "text": " which is I think"}, {"start": 951.78, "end": 953.78, "text": " cloned and then um"}, {"start": 953.78, "end": 955.8599999999999, "text": " Modified version of the fairsec library. Okay"}, {"start": 955.8599999999999, "end": 960.66, "text": " so this is pretty much the same thing as um as what we get when we run the"}, {"start": 961.2199999999999, "end": 963.3, "text": " Previous script with some modifications"}, {"start": 963.3, "end": 969.2199999999999, "text": " So some of those modifications I had to me was like we we I can't use fully sharding because I have a single gpu"}, {"start": 969.62, "end": 972.0999999999999, "text": " I can't use the parallel version of cross entropy"}, {"start": 972.0999999999999, "end": 976.42, "text": " So I had to kind of change it to cross entropy instead of cross entropy parallel or something"}, {"start": 976.42, "end": 981.54, "text": " I was just using transformer lm instead of transformer lm megatron or something and finally, um"}, {"start": 982.42, "end": 986.9, "text": " Windows was complaining I guess any any operating system will complain about super long paths"}, {"start": 987.14, "end": 990.74, "text": " So there was some super long path that we get dumped from the previous script"}, {"start": 990.74, "end": 993.06, "text": " So I had to kind of shorten them into this one"}, {"start": 993.38, "end": 999.4, "text": " So test v0 and finally I had to remove the checkpoint activations and distribute checkpoint activations"}, {"start": 1000.02, "end": 1002.0999999999999, "text": " This is not well, it's actually necessary"}, {"start": 1002.1, "end": 1007.22, "text": " Otherwise you you will face some bugs but like I didn't bother to try and figure it out"}, {"start": 1007.7, "end": 1012.66, "text": " So if you want to kind of play with the checkpointing, uh, which is the the technique where you basically"}, {"start": 1013.3000000000001, "end": 1018.5, "text": " trade off the memory versus the the the the speed of your backward pass"}, {"start": 1018.98, "end": 1021.94, "text": " Uh, then you can kind of enable this and try and figure it out yourself"}, {"start": 1021.94, "end": 1025.6200000000001, "text": " But like i'm going to focus on the mixed precision a part of the of the code. Okay"}, {"start": 1026.34, "end": 1030.18, "text": " Made a couple more tweaks. Uh, you will be able to just copy paste this"}, {"start": 1030.18, "end": 1036.42, "text": " Okay, I think a couple of things I had to change was like the batch size because otherwise you I was getting oom's"}, {"start": 1036.66, "end": 1043.3, "text": " Out of memory exceptions and I had to change to like have 128 tokens per per batch"}, {"start": 1043.7, "end": 1048.8200000000002, "text": " Uh per sentence instead of like 2,048 by default or something like that, but that's it again"}, {"start": 1048.8200000000002, "end": 1051.94, "text": " You'll copy paste this thing, uh, and you can start from here"}, {"start": 1052.5800000000002, "end": 1057.54, "text": " Okay, guys, let's finally get to the transcript. Let me set a breakpoint here"}, {"start": 1057.54, "end": 1060.82, "text": " Uh, let me use the transcript config"}, {"start": 1061.54, "end": 1064.6599999999999, "text": " And actually i'm gonna set another breakpoint down below"}, {"start": 1065.46, "end": 1069.3, "text": " Here and now we are ready to to to run"}, {"start": 1070.02, "end": 1073.54, "text": " Okay, guys, let's start going through the code here"}, {"start": 1074.26, "end": 1076.34, "text": " i'm gonna focus on on on the"}, {"start": 1077.22, "end": 1083.22, "text": " Well, i'm gonna show you details that that are different compared to your uh toy"}, {"start": 1083.22, "end": 1088.44, "text": " Uh repos so to speak because this is after all using this fairly powerful"}, {"start": 1089.08, "end": 1091.08, "text": " Library, uh that was developed by meta"}, {"start": 1091.72, "end": 1095.4, "text": " So yeah, there is a lot of code that that makes this harder to read"}, {"start": 1095.8, "end": 1100.52, "text": " Uh, but like more robust to various errors. Okay, so let's continue here"}, {"start": 1101.48, "end": 1106.44, "text": " So first things first i'm gonna enter this parser function and I want to focus on a couple of details"}, {"start": 1106.52, "end": 1111.64, "text": " Like first of all, I think it's quite kind of interesting how they are picking up, uh various arguments"}, {"start": 1111.64, "end": 1116.0400000000002, "text": " uh, it might not be the most readable way to do it, but it's uh"}, {"start": 1116.44, "end": 1118.44, "text": " Interesting in the sense that let me let me show what I mean"}, {"start": 1118.8400000000001, "end": 1125.16, "text": " So here there is this common config so common config will be just a data class that contains a bunch of these fields"}, {"start": 1125.64, "end": 1133.5600000000002, "text": " that uh basically define uh various, uh variables which are which belong to the common group of variables of arguments"}, {"start": 1133.8000000000002, "end": 1137.0, "text": " such as uh logging interval tensorboard logdir"}, {"start": 1137.0, "end": 1142.6, "text": " Uh various uh thingies like this aim repo. So that's the this visualization tool there"}, {"start": 1142.68, "end": 1144.76, "text": " Then there is the weights and biases project here"}, {"start": 1145.32, "end": 1151.88, "text": " Azure ml logging bunch of details. So yeah, I won't focus on those but I just kind of want to stress that that's neat"}, {"start": 1152.52, "end": 1154.36, "text": " so once we"}, {"start": 1154.36, "end": 1159.24, "text": " Basically execute this line. We'll pick up all of the arguments and default values from this config"}, {"start": 1160.2, "end": 1161.32, "text": " Okay"}, {"start": 1161.32, "end": 1166.12, "text": " Next up they have this these registries. So this is probably less interesting"}, {"start": 1166.12, "end": 1168.6799999999998, "text": " Let me just see here. Uh, so yeah, they're"}, {"start": 1169.56, "end": 1171.3999999999999, "text": " picking up"}, {"start": 1171.3999999999999, "end": 1178.04, "text": " They're picking up values automatically. I mean i'm gonna skip this part because otherwise this will take too long and it's not that vital so"}, {"start": 1178.84, "end": 1184.36, "text": " Bottom line is now the same thing will happen in all of these other functions. So this is if you enter here"}, {"start": 1184.36, "end": 1189.4799999999998, "text": " You'll see that there is this uh generate parser from data class and then there is the data set config"}, {"start": 1189.4799999999998, "end": 1194.28, "text": " so here is the where they define some of the default values for the uh, data set"}, {"start": 1194.28, "end": 1199.3999999999999, "text": " Group and then uh, yeah, so we can basically continue. Let me just go back here"}, {"start": 1199.3999999999999, "end": 1204.84, "text": " So yeah, by the way, if you don't know how to use vs code, there is this nice call stack like a window here"}, {"start": 1204.84, "end": 1210.52, "text": " You just hit the uh, whatever is on top of the stack and you get back to wherever your your debugger has stopped"}, {"start": 1210.52, "end": 1213.0, "text": " That's kind of neat to know. Um, yeah"}, {"start": 1213.0, "end": 1218.6, "text": " So again bunch of different groups will add different arguments and finally we end up with a parser"}, {"start": 1219.3999999999999, "end": 1221.3999999999999, "text": " And that's it"}, {"start": 1221.4, "end": 1226.68, "text": " um now some conversion to configs it's going to be a dictionary or something and"}, {"start": 1227.24, "end": 1232.2, "text": " Now because we're not profiling. We're not doing this. We won't enter this branch. We're going here"}, {"start": 1232.68, "end": 1234.68, "text": " So let's enter the the actual main"}, {"start": 1236.1200000000001, "end": 1242.92, "text": " Function we can skip all of this and we enter the single gpu main branch. So the only difference is"}, {"start": 1243.48, "end": 1244.2800000000002, "text": " um"}, {"start": 1244.28, "end": 1251.48, "text": " Well, there are some differences but ultimately all of the code pets whether you're running this in distributed setting or on a single gpu"}, {"start": 1251.72, "end": 1253.16, "text": " We'll call the same main function"}, {"start": 1253.16, "end": 1258.52, "text": " So that means we'll be actually stepping through the actual code that was used to train the the huge"}, {"start": 1259.0, "end": 1265.6399999999999, "text": " Opt 175 billion parameter model, which is awesome. So let's let me just show you quickly this thingy here"}, {"start": 1266.44, "end": 1268.44, "text": " Again, there are many details"}, {"start": 1269.08, "end": 1272.6, "text": " Here that are relevant if you're running this in a distributed setting"}, {"start": 1272.6, "end": 1278.6799999999998, "text": " That might be interesting interesting for you to check it out. But here i'm going to focus on on on on this code path here"}, {"start": 1278.6799999999998, "end": 1285.8799999999999, "text": " So let's enter the main and let's keep on stepping through this. So let me see what's interesting. So here this is the patch I applied"}, {"start": 1286.84, "end": 1291.8799999999999, "text": " So finally we save the config. So this is a config that we got from all those parser steps"}, {"start": 1292.28, "end": 1298.12, "text": " So ultimately we just save it here to config yaml and we'll see how how the config looks like"}, {"start": 1298.12, "end": 1303.6399999999999, "text": " Okay. Let me just now show you how this thing's going to look like. So this is hopefully the right. Yeah, this is the right directory"}, {"start": 1303.6399999999999, "end": 1309.08, "text": " So let's open up the um test v0. Let's open up the config. I'm just gonna um"}, {"start": 1309.7199999999998, "end": 1312.6, "text": " Basically open up another visual studio instance here"}, {"start": 1313.6399999999999, "end": 1316.9199999999998, "text": " And let me open that file. So let's open. Oops"}, {"start": 1317.4799999999998, "end": 1319.4799999999998, "text": " Let's open the file here"}, {"start": 1320.04, "end": 1322.76, "text": " This one and here is the config yaml"}, {"start": 1323.56, "end": 1326.9199999999998, "text": " So i'm going to open it up here side by side. Okay. So as you can see here"}, {"start": 1326.92, "end": 1334.76, "text": " So as you can see here, there is a lot of a lot of arguments like I think over 400 or something like a huge amount of arguments"}, {"start": 1335.3200000000002, "end": 1339.24, "text": " um, this could use some some cleanup and refactoring but um"}, {"start": 1340.28, "end": 1341.88, "text": " I guess it works"}, {"start": 1341.88, "end": 1343.88, "text": " Kind of they did struggle a lot. So I don't know"}, {"start": 1345.0800000000002, "end": 1350.6000000000001, "text": " Uh, cool. So I won't be going through the arguments here. Obviously it beats the purpose. There is too many things going on"}, {"start": 1350.68, "end": 1352.2, "text": " So i'm just gonna"}, {"start": 1352.2, "end": 1357.96, "text": " Explain you those that we are hitting as we are executing along this code path. Um, okay"}, {"start": 1358.3600000000001, "end": 1360.3600000000001, "text": " So there is this nice, um"}, {"start": 1360.76, "end": 1367.96, "text": " Is is master function? I think it just checks so because the distributed rank is sql zero. So rank means"}, {"start": 1368.76, "end": 1372.8400000000001, "text": " basically if you have multiple gpus in a in a node, uh rank zero would be"}, {"start": 1374.28, "end": 1377.24, "text": " GPU zero in that node the rank one would be gpu one"}, {"start": 1377.24, "end": 1384.04, "text": " It's just like basically naming for your gpus in the node because here we are zero because we are running a single gpu"}, {"start": 1384.44, "end": 1389.4, "text": " Uh, we are going to be the master because we are the only gpu. So we are the master automatically. Okay"}, {"start": 1390.6, "end": 1392.6, "text": " Let's continue here. Um"}, {"start": 1393.8, "end": 1395.8, "text": " I'm going to skip all of that"}, {"start": 1396.28, "end": 1398.28, "text": " just some asserts"}, {"start": 1398.28, "end": 1402.7, "text": " Ignoring all of that seeding the the code so that we can have reproducibility"}, {"start": 1402.7, "end": 1410.22, "text": " Uh, and then some additional seeding that kind of probably in the background. I think it just seeds the torch thingies"}, {"start": 1410.38, "end": 1412.38, "text": " Yeah torch manual seed blah blah blah"}, {"start": 1412.8600000000001, "end": 1414.8600000000001, "text": " I'm gonna skip those details"}, {"start": 1415.18, "end": 1420.06, "text": " Um verify checkpoint directory. I think this is kind of neat. So again this video"}, {"start": 1420.06, "end": 1427.5, "text": " I'm just showing you some details that that go into the engineering behind these llms is not only like the the actual"}, {"start": 1427.5, "end": 1432.94, "text": " Ml logic. That's what i'm trying to get at is super simple like you have a transformer you do everything"}, {"start": 1433.1, "end": 1436.22, "text": " You can literally write all of this logic conceptually. It may be"}, {"start": 1436.7, "end": 1441.82, "text": " Hundred lines of code or something. I might be exaggerating but uh, the point is there is so many things"}, {"start": 1442.06, "end": 1447.42, "text": " That you have to add to make this robust and to make this uh work on on such a large scale"}, {"start": 1447.66, "end": 1450.7, "text": " So this is one of the of the of the details. So let me enter this"}, {"start": 1451.66, "end": 1453.82, "text": " I'm gonna put a breakpoint here hit f5"}, {"start": 1453.82, "end": 1460.3, "text": " Uh, okay. So let's see what they're doing. So they are literally testing with this if this test v0 directory exists"}, {"start": 1460.46, "end": 1465.58, "text": " And we can actually we actually have the access the the right rights to write to that directory"}, {"start": 1465.58, "end": 1470.9399999999998, "text": " So if it exists, uh, if it doesn't exist we create it because it does exist. We don't create it"}, {"start": 1471.26, "end": 1477.34, "text": " We fetch the global rank. It's going to be zero. I assume because yeah, as I said, uh, we create the temporary file"}, {"start": 1477.5, "end": 1479.5, "text": " So we just create some random"}, {"start": 1479.5, "end": 1485.58, "text": " File here and we try to open it up in a write mode that shows us that we have write access"}, {"start": 1485.98, "end": 1491.66, "text": " So as you can see here, we're not hitting the exception. So everything's fine there. And finally we try to remove the file"}, {"start": 1491.74, "end": 1493.9, "text": " So you can see here. So if I were to"}, {"start": 1494.94, "end": 1498.22, "text": " Go here, there is the dummy file. And now if I go"}, {"start": 1498.86, "end": 1502.54, "text": " Uh f10 so step over you can see how this thing is deleted"}, {"start": 1502.7, "end": 1507.1, "text": " So that's how they do the check that you can actually write to that directory and that's robust"}, {"start": 1507.1, "end": 1511.02, "text": " Okay now some logging how much memory I have etc. Etc"}, {"start": 1511.74, "end": 1515.82, "text": " Logging the the whole config you can see there is a bunch of data here"}, {"start": 1516.3, "end": 1519.98, "text": " Everything we saw in the actual yaml file in the vs code"}, {"start": 1520.9399999999998, "end": 1522.9399999999998, "text": " And yeah, okay. I'm gonna skip this"}, {"start": 1523.4199999999998, "end": 1528.4599999999998, "text": " There is the setup task. I'm gonna enter this part of the code as well. So let's see what's going on there"}, {"start": 1528.9399999999998, "end": 1530.9399999999998, "text": " um blah blah blah"}, {"start": 1531.02, "end": 1533.6599999999999, "text": " So we have the task name that's called dummylm"}, {"start": 1533.66, "end": 1537.3400000000001, "text": " And this is a consequence of me using the benchmark flag if you recall"}, {"start": 1537.74, "end": 1542.78, "text": " And um, yeah, so that's gonna be like a super simple task. We're gonna see what it does in a couple of seconds"}, {"start": 1543.26, "end": 1544.78, "text": " so here"}, {"start": 1544.78, "end": 1546.78, "text": " We just pick some uh, basically"}, {"start": 1547.3200000000002, "end": 1548.6200000000001, "text": " um"}, {"start": 1548.6200000000001, "end": 1553.5800000000002, "text": " Associated config with that task. I will not get into how they populate these"}, {"start": 1553.9, "end": 1557.98, "text": " Uh, there is a neat method they use in the background. Well, i'm not sure whether it's neat. It's kind of"}, {"start": 1559.02, "end": 1560.38, "text": " hard to"}, {"start": 1560.38, "end": 1562.38, "text": " Read and understand but again"}, {"start": 1562.38, "end": 1564.38, "text": " It's understandable"}, {"start": 1565.0200000000002, "end": 1567.0200000000002, "text": " Let's enter this one"}, {"start": 1567.5800000000002, "end": 1572.46, "text": " Let's enter the actual constructor of the dummylm task and here we are. Okay, so"}, {"start": 1573.3400000000001, "end": 1577.66, "text": " Let me see whether it's interesting to see the base task. I think nothing there is fan"}, {"start": 1577.74, "end": 1578.22, "text": " Yeah, okay"}, {"start": 1578.22, "end": 1582.8600000000001, "text": " So i'm gonna just skip the init function of the base task and let's see how this is gonna work"}, {"start": 1583.0200000000002, "end": 1586.7800000000002, "text": " So we create a dictionary and this dictionary is just gonna be adding"}, {"start": 1587.18, "end": 1589.18, "text": " a bunch of symbols"}, {"start": 1589.18, "end": 1594.54, "text": " Uh to this to this task, which is not important. It's gonna add like 50 000 or something"}, {"start": 1595.18, "end": 1597.18, "text": " Of yeah, 51 000 196"}, {"start": 1598.14, "end": 1603.42, "text": " Symbols, uh, these will not be important for for for the training as you'll see in a second"}, {"start": 1603.98, "end": 1606.94, "text": " And then what they do is they pad to multiple fate. So"}, {"start": 1607.5800000000002, "end": 1610.7, "text": " Why do they do this is because sometimes um"}, {"start": 1611.66, "end": 1616.94, "text": " On the hardware on the underlying hardware if things are divisible by eight things go much faster"}, {"start": 1616.94, "end": 1621.5800000000002, "text": " So that's a constraint that you kind of have to know of the of the underlying gpus you're using"}, {"start": 1621.5800000000002, "end": 1626.14, "text": " So I guess nvidia a 100s or h100s new ones. So yeah"}, {"start": 1626.8600000000001, "end": 1627.8200000000002, "text": " again"}, {"start": 1627.8200000000002, "end": 1631.42, "text": " Details that matter when you train on such a large scale"}, {"start": 1632.46, "end": 1637.02, "text": " Okay, and finally what we do here is we create as you can see here 128"}, {"start": 1637.5800000000002, "end": 1638.6200000000001, "text": " um"}, {"start": 1638.6200000000001, "end": 1642.46, "text": " basically numbers we have plus one because here"}, {"start": 1642.46, "end": 1648.8600000000001, "text": " We'll create one will be the the source sequence and the second one will be the dummy, uh, the the the target sequence"}, {"start": 1649.02, "end": 1652.78, "text": " So that's basically your transformer logic. You have a sequence and then you shift it"}, {"start": 1652.78, "end": 1655.5, "text": " You try to predict the next token and that's why you have here"}, {"start": 1656.46, "end": 1662.22, "text": " You grab the first 128 elements and then you grab the you move by one and then you grab the 128 there"}, {"start": 1662.38, "end": 1665.1000000000001, "text": " And that's what you're trying to predict. Hopefully that makes sense"}, {"start": 1665.58, "end": 1666.3, "text": " um"}, {"start": 1666.3, "end": 1671.26, "text": " There is additionally some some some logic here, which is not necessary for this dummy task"}, {"start": 1671.26, "end": 1676.46, "text": " But um in general, uh, what it does is makes that you're not predicting the padding token"}, {"start": 1676.46, "end": 1681.9, "text": " So that's why you have this but it doesn't matter for now. That's that's enough. Okay, so that's the setup task"}, {"start": 1682.54, "end": 1686.46, "text": " As I said super simple. We just have these two, uh dummy sequences"}, {"start": 1687.5, "end": 1690.94, "text": " That will be later passing through the transformer during the forward pass"}, {"start": 1692.06, "end": 1694.78, "text": " Okay, because we're not charted i'm gonna skip all of this"}, {"start": 1694.78, "end": 1701.02, "text": " But you see that if we were running this on a on a on an actual distributed system would be hitting this branch"}, {"start": 1701.02, "end": 1703.6, "text": " Instead and you can see there is this fsdp"}, {"start": 1704.86, "end": 1706.94, "text": " Wrap that help helps with the model"}, {"start": 1708.86, "end": 1713.1, "text": " Training on a distributed system basically, okay build model i'm going to enter this branch as well"}, {"start": 1714.1399999999999, "end": 1717.02, "text": " Uh, let me see what's going on here. So build model"}, {"start": 1717.98, "end": 1719.98, "text": " We are using the transformer lm"}, {"start": 1720.22, "end": 1723.5, "text": " uh as again, I cannot run the megatron version because it"}, {"start": 1724.1399999999999, "end": 1728.3, "text": " Assumes I have a at least two gpus and because of that i'm i'm getting"}, {"start": 1728.3, "end": 1730.94, "text": " Uh, basically exceptions if I do this"}, {"start": 1731.82, "end": 1733.02, "text": " so"}, {"start": 1733.02, "end": 1738.86, "text": " Let's just uh pick up the model the actual class is so this is what this registry thing does"}, {"start": 1739.18, "end": 1746.62, "text": " It takes the string transformer lm and picks up the actual class that we'll be using to instantiate the model. Okay?"}, {"start": 1748.06, "end": 1751.18, "text": " And then here it just picks up the associated config"}, {"start": 1751.8999999999999, "end": 1754.22, "text": " So transform role language model config"}, {"start": 1754.22, "end": 1759.58, "text": " It doesn't matter for for for us at this point of time. Let's enter the build model, uh branch"}, {"start": 1759.98, "end": 1762.94, "text": " Uh, and let's see what's going on here. So"}, {"start": 1763.58, "end": 1766.7, "text": " We first build the embedding table. Okay, so that's the first part"}, {"start": 1766.7, "end": 1770.3, "text": " So so that's how transformers work, right? We have a piece of text a sentence"}, {"start": 1770.6200000000001, "end": 1775.42, "text": " We tokenize it into a bunch of integers which are usually of a long data type"}, {"start": 1775.82, "end": 1781.18, "text": " And then we use those integers to index into this embedding table to extract a particular"}, {"start": 1781.18, "end": 1786.8600000000001, "text": " Uh embedding vector and then we're gonna feed those vectors through the transformer. That's what this thing does"}, {"start": 1787.1000000000001, "end": 1789.98, "text": " okay, and being a bit more detailed than than in general because"}, {"start": 1791.18, "end": 1793.3400000000001, "text": " Yeah, I think it makes sense to do it now"}, {"start": 1793.98, "end": 1798.22, "text": " Um, let's see how this is gonna work. They call some embedding thingy here"}, {"start": 1799.3400000000001, "end": 1804.8600000000001, "text": " And once you enter there, you see that ultimately it boils down to torch pi torch embedding"}, {"start": 1805.1000000000001, "end": 1807.1000000000001, "text": " So we fetch the device here"}, {"start": 1807.42, "end": 1809.02, "text": " it's gonna be uh"}, {"start": 1809.02, "end": 1810.3, "text": " kuda"}, {"start": 1810.3, "end": 1813.26, "text": " Zero because we have that's the index of my single gpu"}, {"start": 1814.22, "end": 1817.34, "text": " And because this is set to true we'll be using half precision"}, {"start": 1817.74, "end": 1819.5, "text": " So everything is going to be in half precision"}, {"start": 1819.5, "end": 1822.54, "text": " So you can see here what we do is number of embeddings is"}, {"start": 1823.48, "end": 1828.54, "text": " 51,200 because that's the i3 we call that's the dictionary size. So that's the uh,"}, {"start": 1829.26, "end": 1833.82, "text": " Even though we're using a dummy task, um, then we have embedding the dimension which is 512"}, {"start": 1834.3, "end": 1839.58, "text": " And we are running this on a on a gpu in half precision. So that's float 16"}, {"start": 1839.58, "end": 1842.46, "text": " So let's create this empty tensor"}, {"start": 1843.1799999999998, "end": 1848.78, "text": " And then we just basically initialize it normally and then i'm not sure why we're doing this"}, {"start": 1849.26, "end": 1851.26, "text": " Basically, what I do is they set the padding"}, {"start": 1851.74, "end": 1855.58, "text": " Vector to be all zeros. So currently if I were to open up this one"}, {"start": 1856.46, "end": 1859.8999999999999, "text": " Let me show you this. So if I use the weight of padding index"}, {"start": 1860.6999999999998, "end": 1866.22, "text": " And if I do this, uh, you can see there is a bunch of values after we step over here"}, {"start": 1866.22, "end": 1874.54, "text": " Now if I execute it we'll have all zeros. So i'm not sure why they're doing this because ultimately embeddings are learnable. Um,"}, {"start": 1875.26, "end": 1878.38, "text": " So so so yeah, um, i'm not completely sure what's going on"}, {"start": 1879.4, "end": 1880.76, "text": " Embeddings"}, {"start": 1880.76, "end": 1884.22, "text": " Constructed we get back from the call and we are here"}, {"start": 1884.78, "end": 1889.58, "text": " The next thing we do is we construct the actual decoder. So this is going to be a decoder only transformer"}, {"start": 1890.8600000000001, "end": 1892.3, "text": " So let's enter"}, {"start": 1892.3, "end": 1894.3, "text": " into this thingy and"}, {"start": 1894.3, "end": 1896.3, "text": " And let's execute. Okay"}, {"start": 1896.7, "end": 1902.06, "text": " Um, let me see whether there's something interesting here incremental decoder. I think it's just going to store the dictionary"}, {"start": 1902.06, "end": 1904.06, "text": " So we don't have to enter the"}, {"start": 1904.12, "end": 1908.1399999999999, "text": " Constructors of the parent classes. I'm going to skip over all of that"}, {"start": 1908.62, "end": 1914.0, "text": " Uh, let's focus on the important parts. Um, nothing fancy here. We're just storing some variables"}, {"start": 1914.54, "end": 1915.5, "text": " Blah blah blah"}, {"start": 1915.5, "end": 1915.74, "text": " Okay"}, {"start": 1915.74, "end": 1922.06, "text": " So the first thing that we're gonna do is we if the embedding dimension is different from the input embedding dimension, which they are"}, {"start": 1922.06, "end": 1924.62, "text": " Uh, then we'll create this linear layer"}, {"start": 1924.94, "end": 1931.8999999999999, "text": " That's gonna take our embedding vectors and just uh increase the dimensionality from 512 to to seven some 700 something"}, {"start": 1932.1399999999999, "end": 1934.94, "text": " So let's see what it is. So we have a linear layer"}, {"start": 1935.26, "end": 1940.3799999999999, "text": " So from 512 is gonna map them into 768 dimensions. Okay"}, {"start": 1941.5, "end": 1944.1399999999999, "text": " We're not using a lib in this particular setup"}, {"start": 1944.86, "end": 1945.82, "text": " so"}, {"start": 1945.82, "end": 1949.74, "text": " Um, we can ignore all of this we can ignore all of this"}, {"start": 1949.74, "end": 1955.98, "text": " Um, and let's see where we are now. So now we are generating the positional embeddings"}, {"start": 1956.7, "end": 1961.58, "text": " Let me enter inside of there. So I just hit f12 to enter. I set a breakpoint here"}, {"start": 1961.58, "end": 1966.86, "text": " I hit that five and we are in okay. This is learnable. So we enter this branch here"}, {"start": 1968.06, "end": 1969.42, "text": " um"}, {"start": 1969.42, "end": 1971.42, "text": " number of embeddings is gonna be"}, {"start": 1972.14, "end": 1974.54, "text": " Uh, whatever the number of readings here is"}, {"start": 1975.1, "end": 1977.1, "text": " plus padding plus one"}, {"start": 1977.1, "end": 1981.6599999999999, "text": " Because I assume it's for the beginning of sentence token or something"}, {"start": 1982.1399999999999, "end": 1987.98, "text": " That's why they have to create 130 of these instead of 128. So remember we only need as many"}, {"start": 1988.62, "end": 1994.86, "text": " uh positional embeddings as as the longest sequence will be feeding to the transformer because we're using this dummy task"}, {"start": 1994.86, "end": 2000.2199999999998, "text": " We're only feeding 128 tokens. So that's why we have this many uh positional embeddings"}, {"start": 2000.86, "end": 2005.74, "text": " So then we call this learn positional embedding thingy and that's just going to"}, {"start": 2005.74, "end": 2007.74, "text": " Do what let me see"}, {"start": 2007.74, "end": 2013.1, "text": " Uh, well, it's just embedding. It's literally just a torch embedding again. So we can just skip over that"}, {"start": 2013.9, "end": 2020.14, "text": " He's just gonna create embedding table and then we initialize it again and um"}, {"start": 2020.78, "end": 2026.22, "text": " Again padding index is set to zero and we return back the embedding. So that's it. Okay"}, {"start": 2026.78, "end": 2031.66, "text": " Similarly to what we've done for the input tokens embedding table"}, {"start": 2032.3, "end": 2033.42, "text": " okay"}, {"start": 2033.42, "end": 2034.38, "text": " so"}, {"start": 2034.38, "end": 2041.18, "text": " So now we're gonna do this precision conversion. So we'll be uh grabbing the"}, {"start": 2042.22, "end": 2044.7, "text": " Embed positions and we'll convert them to fp"}, {"start": 2045.5, "end": 2048.2200000000003, "text": " 16 basically, so let me enter this function here"}, {"start": 2048.94, "end": 2054.94, "text": " i'm gonna hit hit the f uh, like a breakpoint here hit f5 and uh, let's see what's gonna happen"}, {"start": 2055.02, "end": 2057.02, "text": " so we assert that it's not the"}, {"start": 2057.7400000000002, "end": 2059.98, "text": " B-float 16. So it's the brain float format"}, {"start": 2059.98, "end": 2066.78, "text": " Uh check out my um again ultimate guide to scaling to understand how these formats look like how they work"}, {"start": 2067.26, "end": 2073.1, "text": " And why they're important in a nutshell it just increases the range, but I don't have bf 16 on my machine"}, {"start": 2073.26, "end": 2076.22, "text": " So we're just going to use the half which is the fp 16"}, {"start": 2076.62, "end": 2084.06, "text": " Uh format that's a bit has a smaller range and that means it's uh, the the the the weights during the training and the gradients and everything"}, {"start": 2084.38, "end": 2088.3, "text": " Is more susceptible to both underflow as well as the overflow"}, {"start": 2088.3, "end": 2095.5, "text": " Okay, so we basically transform them to have precision and we get back here. Okay, let's continue"}, {"start": 2096.2200000000003, "end": 2098.94, "text": " Next up we generate 12 of these decoder layers"}, {"start": 2099.9, "end": 2105.82, "text": " By calling a build decoder layer a function here and this one is going to be fairly interesting. So i'm gonna"}, {"start": 2106.46, "end": 2109.02, "text": " Put a breakpoint here hit f5 enter here"}, {"start": 2110.1400000000003, "end": 2116.78, "text": " Let's first build the base decoder layer. So i'm gonna hit f11 to enter f11 again to enter"}, {"start": 2116.78, "end": 2120.38, "text": " Here and let's see where we are. So we are in the decoder layer"}, {"start": 2120.5400000000004, "end": 2125.02, "text": " Okay, so we're trying to construct the first one now. This part is fairly interesting"}, {"start": 2125.1000000000004, "end": 2129.6600000000003, "text": " So this is something you don't see in a regular code base. That's not super optimized"}, {"start": 2130.86, "end": 2135.42, "text": " Here we'll be using megatron fused kernels. Well, actually for this particular setup I have"}, {"start": 2135.82, "end": 2141.98, "text": " I will not be using these kernels but still we'll see how they how they are loading them and how it looks like in practice"}, {"start": 2142.46, "end": 2146.1400000000003, "text": " Okay, so let's go through all of this. That's not important"}, {"start": 2146.14, "end": 2152.2999999999997, "text": " And now this part is interesting. So let's enter the the load the fuse kernels load function"}, {"start": 2152.8599999999997, "end": 2154.8599999999997, "text": " F11 and here we are"}, {"start": 2154.94, "end": 2156.14, "text": " okay"}, {"start": 2156.14, "end": 2161.9, "text": " So first we pick up some some some metadata about the kuda. That's not that important"}, {"start": 2162.54, "end": 2164.54, "text": " um, let's just"}, {"start": 2164.7, "end": 2170.06, "text": " Step over all of these and then this one is fairly interesting. So this one is going to load certain"}, {"start": 2170.06, "end": 2176.7, "text": " Uh kernels written in c++. That's gonna be super efficient compared to running in python"}, {"start": 2177.2599999999998, "end": 2179.34, "text": " So let me just go here"}, {"start": 2180.2999999999997, "end": 2186.2999999999997, "text": " So fused softmax, we just defined the function and this is set to true. So we enter this we just"}, {"start": 2187.02, "end": 2189.5, "text": " Create some flags and you can see here"}, {"start": 2190.14, "end": 2197.98, "text": " We are now creating a path towards certain cpp file and this cu file kuda kernel file. So"}, {"start": 2197.98, "end": 2206.32, "text": " So by the way, i'm not super familiar. I've never had to write any code into any kuda kernels myself. So yeah, um, just fyi"}, {"start": 2206.86, "end": 2208.86, "text": " um, so here are the sources"}, {"start": 2209.26, "end": 2215.1, "text": " And now we're gonna load those files using the cpp extension load helper. So this function here"}, {"start": 2215.5, "end": 2220.7, "text": " This is gonna load from those from those files. Let me show you those those files here"}, {"start": 2221.9, "end": 2223.26, "text": " So here it is"}, {"start": 2223.26, "end": 2228.1400000000003, "text": " Uh, we have to enter the megatron. We have to enter the megatron again and then diffuse kernels"}, {"start": 2228.7000000000003, "end": 2232.2200000000003, "text": " And here are those files. So we are scaled upper"}, {"start": 2233.0, "end": 2235.0, "text": " triang"}, {"start": 2235.0200000000004, "end": 2236.86, "text": " Mast softmax"}, {"start": 2236.86, "end": 2239.42, "text": " So here it is. I'm gonna open it up"}, {"start": 2239.5800000000004, "end": 2242.94, "text": " I think that's the one and then we have to open up the kuda one, right?"}, {"start": 2243.5, "end": 2244.3, "text": " so"}, {"start": 2244.3, "end": 2246.78, "text": " That one let me just open it up with vs code"}, {"start": 2246.78, "end": 2253.26, "text": " And then there is this one. So let me open it up with vs code as well. And that's the only two files"}, {"start": 2253.26, "end": 2257.1800000000003, "text": " I'm gonna open up just for two to show you how how it roughly looks like"}, {"start": 2258.0600000000004, "end": 2263.1800000000003, "text": " So here is the cpp one. So you can see here they define forward and backward for this particular"}, {"start": 2263.7400000000002, "end": 2266.46, "text": " Softmax layer and they make it super efficient"}, {"start": 2266.46, "end": 2274.46, "text": " So in this particular part in this file, they just create a binding to python so that we can execute a c++ code directly from python"}, {"start": 2274.46, "end": 2276.62, "text": " Uh, so here is the actual code"}, {"start": 2277.18, "end": 2279.18, "text": " behind that file"}, {"start": 2279.98, "end": 2283.58, "text": " So here is the blah blah blah the backward kuda file"}, {"start": 2284.2200000000003, "end": 2288.94, "text": " Uh, and everything boils down to this dispatch half and and and bfloat function"}, {"start": 2289.18, "end": 2292.46, "text": " Let me see where I can do f12 and see how that function looks like"}, {"start": 2293.1, "end": 2294.54, "text": " um"}, {"start": 2294.54, "end": 2298.54, "text": " Yeah, I won't go there man. That's gonna be wicked in any case"}, {"start": 2298.54, "end": 2301.82, "text": " This was just more of an fyi than than anything else"}, {"start": 2301.82, "end": 2308.3, "text": " I don't want to dig too much into the c++ plus code because that's gonna get like messy very quickly"}, {"start": 2309.02, "end": 2313.98, "text": " Okay, let's go back to the um, let's find my my prompt"}, {"start": 2314.54, "end": 2320.7000000000003, "text": " Let's find the debugger and let's continue execution execution here. So this might take a while"}, {"start": 2320.86, "end": 2326.38, "text": " Uh, just the loading of this i'm gonna i'm gonna skip this loading part and i'm gonna just hit f5 to"}, {"start": 2326.38, "end": 2332.7000000000003, "text": " To get to the mask softmax here and I will get back to you as soon as yeah here it is because i've actually pre-built it before"}, {"start": 2333.02, "end": 2335.6600000000003, "text": " But the first time we run this it will take probably"}, {"start": 2336.46, "end": 2338.46, "text": " five minutes, uh in case"}, {"start": 2338.86, "end": 2344.94, "text": " Uh, you you it's hanging there is one trick you have to do and that's to just delete this build directory again"}, {"start": 2345.1, "end": 2347.1, "text": " Don't ask me why I know this"}, {"start": 2347.1800000000003, "end": 2349.26, "text": " Uh, yeah, the tears on my keyboard know it"}, {"start": 2351.02, "end": 2353.02, "text": " Okay, let's continue here"}, {"start": 2353.02, "end": 2359.42, "text": " Let's load these scaled masks softmax once and as I said, we will not actually be using them"}, {"start": 2360.3, "end": 2364.22, "text": " But it's kind of funny to to see uh that uh, they are doing that"}, {"start": 2364.7, "end": 2369.2599999999998, "text": " Okay, guys, I actually realized that the actual implementation is in the h file"}, {"start": 2369.5, "end": 2373.82, "text": " So if we open up this one, let me just quickly show you so you can see the actual code"}, {"start": 2373.9, "end": 2378.86, "text": " You can find it here. So this is how that these functions are defined in the background just a cpp function"}, {"start": 2378.86, "end": 2385.6600000000003, "text": " Uh c++ function and the optimized version of the of those of that functionality. So yeah, let me go back now here"}, {"start": 2386.1400000000003, "end": 2388.7000000000003, "text": " Uh, let me show you one more thing. I'm gonna copy this"}, {"start": 2389.34, "end": 2393.34, "text": " Use the search functionality here in vs code to find where this one is used"}, {"start": 2393.82, "end": 2401.6600000000003, "text": " And you can see that um, so we have in this file here where it's actually being used is somewhere here"}, {"start": 2402.86, "end": 2404.38, "text": " So let's just find it"}, {"start": 2404.38, "end": 2411.9, "text": " Uh, here it is. Let me try and see whether yeah forward f line. Okay, so there is this fused layer norm f1 function"}, {"start": 2412.54, "end": 2415.1, "text": " Uh, that's inside of the megatron library actually"}, {"start": 2415.6600000000003, "end": 2418.2200000000003, "text": " And I think it relies on the apex potentially"}, {"start": 2419.26, "end": 2424.46, "text": " So so yeah, that's i'm just kind of showing you to so that you know your way around this code"}, {"start": 2424.94, "end": 2430.54, "text": " But yeah, um, so if we were to actually use this this this class somewhere in our code path"}, {"start": 2430.54, "end": 2436.06, "text": " Uh in the background this part of the code would actually be c++ code. So that's kind of interesting"}, {"start": 2436.7, "end": 2441.1, "text": " Okay, let's get back here and let's exit out of this function"}, {"start": 2442.22, "end": 2444.86, "text": " let's load this one and then let me show you the"}, {"start": 2447.34, "end": 2449.34, "text": " The creation of this of this"}, {"start": 2451.82, "end": 2454.54, "text": " Decoder layer. So again, we are in the transformer decoder layer"}, {"start": 2455.42, "end": 2457.18, "text": " Let's continue"}, {"start": 2457.18, "end": 2461.66, "text": " We're building the self-attention. I'm going to skip that part because that's that's common"}, {"start": 2462.8599999999997, "end": 2464.8599999999997, "text": " common knowledge, I guess um"}, {"start": 2465.66, "end": 2467.66, "text": " Let's continue here layer norms"}, {"start": 2467.8999999999996, "end": 2475.1, "text": " I initialize blah blah blah again. We're converting the layers the the attention layers to to uh float i'm gonna"}, {"start": 2475.8999999999996, "end": 2477.66, "text": " Basically remove this one"}, {"start": 2477.66, "end": 2482.0, "text": " Uh, so that we don't have to enter anymore. So everything has been converted to fp16"}, {"start": 2482.8599999999997, "end": 2484.8599999999997, "text": " Uh, so that's the mixed precision thingy"}, {"start": 2484.86, "end": 2491.26, "text": " Uh, and um, yeah, let's continue here. We just fetch the activation function, which is going to be a rally I think"}, {"start": 2491.9, "end": 2494.54, "text": " Yeah, and then let me just see what this one is"}, {"start": 2495.1800000000003, "end": 2498.86, "text": " So yeah, you can see we're just because activation was set to rally"}, {"start": 2499.1, "end": 2504.46, "text": " We'll just return the f value the functional from the functional module. We fetch the rally function. Okay"}, {"start": 2505.5, "end": 2507.02, "text": " Let's go back here"}, {"start": 2507.02, "end": 2510.54, "text": " Um now we create the fully connected layers"}, {"start": 2510.54, "end": 2516.86, "text": " So we have obviously two fully connected layers, uh in the transformer block. The first one is after the attention mechanism"}, {"start": 2516.94, "end": 2521.34, "text": " We have the fc1 and then we have the fc2. No, actually this is the sorry"}, {"start": 2521.34, "end": 2524.54, "text": " this is the I guess the mlp after after the"}, {"start": 2525.34, "end": 2530.22, "text": " attention thing is is is is gone but like the actual attention logic also has the"}, {"start": 2530.94, "end": 2533.9, "text": " Mapping the linear mapping once you do the"}, {"start": 2534.86, "end": 2537.58, "text": " Attention thingy with value vectors and everything if that makes sense"}, {"start": 2537.58, "end": 2542.14, "text": " So the reason they split up into fc1 and fc2 is because they use megatron"}, {"start": 2542.46, "end": 2548.2999999999997, "text": " And so I think that's some limitation that they that the megatron imposes on the users"}, {"start": 2548.38, "end": 2552.38, "text": " You have to kind of split these such that once you do the model parallelism"}, {"start": 2552.54, "end": 2557.5, "text": " You can split these fully connected layers onto multiple gpus. That's why they have to separately"}, {"start": 2557.98, "end": 2560.7, "text": " Uh create the function. So if that's confusing you"}, {"start": 2561.58, "end": 2563.1, "text": " That's the answer"}, {"start": 2563.1, "end": 2568.2999999999997, "text": " Okay, so it's just gonna build in the background as you can see here. It's just gonna build a linear layer"}, {"start": 2568.54, "end": 2571.52, "text": " But the reason they're doing it like this is again model parallelism"}, {"start": 2572.86, "end": 2576.86, "text": " And those are all of the details you kind of have to deal with when you're when you're um"}, {"start": 2577.74, "end": 2583.58, "text": " Working on on large scale there. There are details that you kind of have to to to learn and uh, yeah"}, {"start": 2583.9, "end": 2589.18, "text": " It's kind of different compared to small scale again conversion to fp16. Let's exit here"}, {"start": 2590.14, "end": 2592.14, "text": " And that's the base layer"}, {"start": 2592.14, "end": 2592.8599999999997, "text": " um"}, {"start": 2592.8599999999997, "end": 2594.8599999999997, "text": " Let me skip over all of this"}, {"start": 2595.66, "end": 2603.42, "text": " And now additionally if checkpoint was set to true with basically wrap this layer inside of the checkpoint wrapper"}, {"start": 2603.98, "end": 2607.8199999999997, "text": " And again, that's what it does in the back in the background is during the backward pass"}, {"start": 2608.14, "end": 2615.2599999999998, "text": " You didn't store all the activations and instead you'll be recomputing the activations and by doing that you'll be"}, {"start": 2616.2799999999997, "end": 2617.5, "text": " obviously"}, {"start": 2617.5, "end": 2622.22, "text": " Increasing the time it takes to do the backward. I think it's going to be 2x slower"}, {"start": 2622.38, "end": 2627.18, "text": " But as a as a consequence you you save up a lot of memory because you don't have to save every activation"}, {"start": 2627.5, "end": 2631.1, "text": " Along every layer. That's the checkpoint in logic, but we're going to skip it now"}, {"start": 2632.14, "end": 2637.1, "text": " Uh, let's continue here. This is going to be no op. Nothing's going to happen in this one"}, {"start": 2638.14, "end": 2642.7, "text": " And let's continue. Okay, so that's going to be repeated 12 times"}, {"start": 2642.7, "end": 2649.02, "text": " I'm gonna do disable all breakpoints put f put the breakpoint here hit f5"}, {"start": 2649.8199999999997, "end": 2656.22, "text": " Okay, and let's continue from here. Now. I'm gonna re-enable all of the breakpoints and that's it. Okay guys"}, {"start": 2656.7, "end": 2659.3399999999997, "text": " And we just packed those layers into module list"}, {"start": 2660.06, "end": 2662.06, "text": " Nothing fancy there"}, {"start": 2662.22, "end": 2668.8599999999997, "text": " Nothing fancy here. We have the project again projection the output dimension projection from I guess 712 to"}, {"start": 2669.8799999999997, "end": 2671.8799999999997, "text": " 768 to 512"}, {"start": 2671.88, "end": 2673.88, "text": " uh and"}, {"start": 2674.6800000000003, "end": 2676.52, "text": " That's it"}, {"start": 2676.52, "end": 2681.96, "text": " Okay here because share input output embed it's set to true. This is one of the things you again"}, {"start": 2682.92, "end": 2686.36, "text": " kind of um kept to note and that's that usually people tie"}, {"start": 2686.92, "end": 2693.8, "text": " The input embedding table with the output and so here instead of having a separate set of weights for this output projection instead"}, {"start": 2693.88, "end": 2699.0, "text": " We'll just reuse the same weights that we use to embed our our initial tokens"}, {"start": 2699.0, "end": 2704.04, "text": " And you can see we're literally using this embed tokens, which is our embedding table as you can see 51"}, {"start": 2704.5, "end": 2710.84, "text": " 200 and 512 that's the initial embedding table we created and we're going to use those weights to populate this linear layer"}, {"start": 2710.92, "end": 2715.24, "text": " So then you can imagine what this does is you take the output token"}, {"start": 2715.24, "end": 2719.24, "text": " Which is 512 and once you multiply with this output projection you end up with this"}, {"start": 2719.86, "end": 2722.6, "text": " 51 000 something which is basically the vocab size"}, {"start": 2723.32, "end": 2727.48, "text": " Of your of your output. Okay. Hopefully that makes sense"}, {"start": 2727.48, "end": 2734.12, "text": " Let me exit this. We are not using a lib so we can exit this we have our decoder"}, {"start": 2734.36, "end": 2737.96, "text": " We have we have the embedding table. That's it model constructed"}, {"start": 2739.4, "end": 2741.4, "text": " Okay, now we have the um"}, {"start": 2741.46, "end": 2746.14, "text": " Criterion we're using cross entropy. I don't think this is anything super interesting"}, {"start": 2746.92, "end": 2748.92, "text": " Let me see what's going on here"}, {"start": 2749.32, "end": 2751.8, "text": " You can skip all of this. We can skip all of this"}, {"start": 2752.44, "end": 2755.8, "text": " Let's enter the builder. I think we can skip all of this as well"}, {"start": 2755.8, "end": 2757.1600000000003, "text": " Well"}, {"start": 2757.1600000000003, "end": 2759.1600000000003, "text": " Nothing interesting"}, {"start": 2759.1600000000003, "end": 2761.5600000000004, "text": " Here, so here's just the cross entropy criterion"}, {"start": 2763.0800000000004, "end": 2765.2400000000002, "text": " I'm gonna skip all of this. This is not interesting"}, {"start": 2765.96, "end": 2771.32, "text": " Okay. So now there is some logging going on if you open up the terminal here. You'll just see what those logs are"}, {"start": 2772.6000000000004, "end": 2779.0800000000004, "text": " Uh, it's just logging various stuff. So we're using dummy element task. We're using transformer language model. We're using cross entropy as the loss"}, {"start": 2779.8, "end": 2783.82, "text": " Let's continue. We are counting the number of parameters here. We have 112"}, {"start": 2783.82, "end": 2785.82, "text": " Um million"}, {"start": 2787.02, "end": 2792.6200000000003, "text": " Uh, blah blah blah. Let's continue here. Now we'll load the data set. So let's enter the"}, {"start": 2793.6600000000003, "end": 2796.0800000000004, "text": " Data loading part. So i'm gonna hit f11"}, {"start": 2797.34, "end": 2799.34, "text": " Here we are. So let me continue here"}, {"start": 2800.78, "end": 2806.46, "text": " Okay, so what's this batch size is one as I told you i'm using one because otherwise I would be hitting"}, {"start": 2806.86, "end": 2809.7400000000002, "text": " Outer memory exceptions and now we just construct a dummy data set"}, {"start": 2809.74, "end": 2814.62, "text": " So here is how it looks like this this one we basically just stored the batch. It's a super simple"}, {"start": 2814.9399999999996, "end": 2820.54, "text": " Well, literally dummy data set the name says it itself. So what we do is we just create this"}, {"start": 2821.18, "end": 2823.18, "text": " Batch you can see here"}, {"start": 2823.18, "end": 2829.58, "text": " And what we do is we have source tokens. We have uh, we just have the the array of of 128 elements"}, {"start": 2829.58, "end": 2835.02, "text": " We created if you recall before so we just use that as a source tokens and then um"}, {"start": 2835.02, "end": 2840.54, "text": " Um, we set as the target tokens those dummy targets and you can see see they're shifted by one"}, {"start": 2840.7, "end": 2845.2599999999998, "text": " So here we have three four five and there we had for the source one. We had two"}, {"start": 2845.82, "end": 2848.14, "text": " Three four, so they're exactly shifted by one"}, {"start": 2848.7, "end": 2854.38, "text": " So this is going to be our simple, um dummy data set. Uh, we just have number of sentences is one"}, {"start": 2854.62, "end": 2857.58, "text": " We only have 128 tokens blah blah blah"}, {"start": 2857.66, "end": 2863.1, "text": " So we literally have batch size one and only 128 tokens as simple as it gets as simple as it gets"}, {"start": 2863.1, "end": 2865.3399999999997, "text": " So let me just construct this thingy"}, {"start": 2866.7, "end": 2871.98, "text": " And that's it. So that's the construction of the data set. It's very easy and simple"}, {"start": 2872.14, "end": 2876.54, "text": " So now the trainer is just a abstraction they use to to sort everything inside"}, {"start": 2876.62, "end": 2884.7, "text": " So the task and the data set and everything so here they extract the shared parameters because I told you they're binding the input embedding"}, {"start": 2884.7, "end": 2887.3399999999997, "text": " with the output embedding output projection layer"}, {"start": 2888.38, "end": 2892.14, "text": " Again, that's now less interesting. I'm going to skip over all of these"}, {"start": 2892.14, "end": 2894.14, "text": " They are storing the criteria and the model"}, {"start": 2894.7, "end": 2896.2999999999997, "text": " Uh blah blah blah"}, {"start": 2896.2999999999997, "end": 2900.8599999999997, "text": " Uh, they are basically again casting to uh half precision float"}, {"start": 2901.42, "end": 2903.98, "text": " fp 16 both the model and the criterion"}, {"start": 2904.46, "end": 2906.46, "text": " uh, and uh"}, {"start": 2906.8599999999997, "end": 2908.8599999999997, "text": " Now they're pushing it to the gpu"}, {"start": 2909.5, "end": 2910.3799999999997, "text": " and"}, {"start": 2910.3799999999997, "end": 2912.14, "text": " That's it"}, {"start": 2912.14, "end": 2914.7799999999997, "text": " Nothing, nothing interesting there. I'm going to skip all of this"}, {"start": 2915.42, "end": 2918.8599999999997, "text": " I'm going to skip all of this. I'm just going to put f5 somewhere here"}, {"start": 2918.86, "end": 2922.54, "text": " Sorry, put the breakpoint and hit f5. My brain is glitching nice"}, {"start": 2923.58, "end": 2929.82, "text": " Okay. So again, it's not fetching the environment for some reason so you can see here I have rtx 2080"}, {"start": 2930.38, "end": 2933.98, "text": " 8 gigabytes, uh 7.5 are the major and minor"}, {"start": 2935.26, "end": 2939.9, "text": " And they just store that for whatever reason. I'm not sure why they're using it probably just to"}, {"start": 2940.86, "end": 2946.6200000000003, "text": " For the logging purposes so they they know exactly which devices in case you have heterogeneous distributed system"}, {"start": 2946.62, "end": 2953.3399999999997, "text": " Where you're using multiple gpus some are maybe p 100 some are maybe m 40 some are maybe v 100"}, {"start": 2953.9, "end": 2957.98, "text": " Some are a 100 so when I have that additional, uh, as you can see here, they're just"}, {"start": 2958.62, "end": 2963.3399999999997, "text": " Logging everything basically. Okay. We're back to the train. This is the the main function"}, {"start": 2963.98, "end": 2966.46, "text": " In case you're confused where we are. So, um"}, {"start": 2967.58, "end": 2970.62, "text": " Let's now i'm gonna skip this because this is just gonna"}, {"start": 2970.62, "end": 2976.22, "text": " Uh give us back this iterator and there is like literally four layers of wrapping they do here"}, {"start": 2976.22, "end": 2982.94, "text": " They have some counting wrapper and then this wrapper and this wrapper gets very very unwieldy very quickly"}, {"start": 2983.3399999999997, "end": 2986.46, "text": " So now skip that part and just focus on on the train loop now"}, {"start": 2986.94, "end": 2988.54, "text": " um"}, {"start": 2988.54, "end": 2993.02, "text": " So here we are here's the train loop. Let's enter the train loop. Okay, so we're actually hitting this"}, {"start": 2994.38, "end": 2996.62, "text": " Aggregate function i'm going to show you what it is"}, {"start": 2996.62, "end": 3002.06, "text": " So i'm gonna hit i'm gonna enter i'm gonna hit f12 to train i'm gonna put the breakpoint here"}, {"start": 3002.38, "end": 3005.74, "text": " You can see there is this um wrapper. They're using this decorator"}, {"start": 3006.2799999999997, "end": 3012.2999999999997, "text": " Aggregate and it's that's just going to make sure that when they're logging something inside of that scope"}, {"start": 3012.94, "end": 3017.1, "text": " That all of those logging details are being stored into this particular train"}, {"start": 3017.64, "end": 3021.3399999999997, "text": " Dictionary and then they'll have multiple of these, uh aggregate"}, {"start": 3021.34, "end": 3029.28, "text": " Uh, um decorators and that's just gonna gonna like nicely organize where each of the logs is going to so they know exactly"}, {"start": 3029.36, "end": 3033.2000000000003, "text": " From which part of the code which logs came from that's the basic idea"}, {"start": 3033.76, "end": 3039.28, "text": " Okay, so we're gonna just um hit f5 and exit this and hit the initial part of the train function"}, {"start": 3040.4, "end": 3043.52, "text": " Let's just grab the iterator. I can skip over all of this"}, {"start": 3044.1600000000003, "end": 3046.1600000000003, "text": " Um, basically iterator is just"}, {"start": 3046.16, "end": 3051.8399999999997, "text": " Uh as the name suggests counting iterators, so it's just gonna feed us those 128"}, {"start": 3053.44, "end": 3055.44, "text": " Tokens in in on each call"}, {"start": 3056.56, "end": 3063.2799999999997, "text": " That's it. We grab some progress bar again. I'm gonna skip all of that again. I told you this is this is what it"}, {"start": 3063.8399999999997, "end": 3065.8399999999997, "text": " How the code looks like when you're training?"}, {"start": 3066.08, "end": 3071.04, "text": " Uh bigger models like there is a lot of things that you don't care about because they're conceptually not important"}, {"start": 3071.12, "end": 3075.2, "text": " But then again, they're super important if you want to make sure if you want to be able to debug"}, {"start": 3075.2, "end": 3080.72, "text": " Uh things when they go wrong, so i'm gonna skip over all of this. Let's get to here"}, {"start": 3082.16, "end": 3086.16, "text": " Here is the actual train function. So there is train instead of train instead of training again"}, {"start": 3086.64, "end": 3091.68, "text": " A lot of layers of of kind of complexity. Uh, let's skip over all of this"}, {"start": 3092.16, "end": 3097.8399999999997, "text": " So here is the definition of the train we hit f5 we get here and this is where the actual magic starts happening"}, {"start": 3097.9199999999996, "end": 3099.68, "text": " So we we're not"}, {"start": 3099.68, "end": 3107.7599999999998, "text": " Gonna iterate and fetch the sample. We will not enter this particular branch because this new profiler is set to false"}, {"start": 3108.56, "end": 3111.3599999999997, "text": " Um, and so we'll we'll just hit this branch here"}, {"start": 3111.9199999999996, "end": 3117.2799999999997, "text": " And now i'll hit the uh f11 and enter this train function and instead of this train function"}, {"start": 3117.52, "end": 3121.12, "text": " There is yet another train function. So let me hit f5 and we end up here"}, {"start": 3121.9199999999996, "end": 3125.12, "text": " so we end up here train step so again, uh the"}, {"start": 3125.12, "end": 3129.7599999999998, "text": " The outer train is the epoch train. This one is the batch trick"}, {"start": 3129.8399999999997, "end": 3134.88, "text": " This one is going to be like the batch train. So we pass just the samples and we do a single forward pass"}, {"start": 3135.2, "end": 3137.44, "text": " So let's see what the samples are samples"}, {"start": 3138.08, "end": 3142.88, "text": " Samples is a dictionary as you can see we have all those tokens from the dummy"}, {"start": 3143.7599999999998, "end": 3145.2799999999997, "text": " data set loader"}, {"start": 3145.2799999999997, "end": 3149.6, "text": " Two interesting things to notice again. They have this aggregate thingy going on"}, {"start": 3150.08, "end": 3154.4, "text": " It's kind of convenient. It might be confusing if you're not used to it, but uh, it"}, {"start": 3154.4, "end": 3159.12, "text": " It does the job and secondly they have this profiler record function"}, {"start": 3159.28, "end": 3163.6800000000003, "text": " So what this thing does is first of all, they enable the profiler. I'm going to show you that in a second"}, {"start": 3164.0, "end": 3171.52, "text": " Secondly, this record function basically names this particular part of the code with train underscore step"}, {"start": 3172.1600000000003, "end": 3173.12, "text": " dash"}, {"start": 3173.12, "end": 3180.7200000000003, "text": " Some number of the iteration and that's useful because later on the when you analyze the profiler output you'll you'll exactly know"}, {"start": 3180.72, "end": 3186.08, "text": " How long or like you have the details the profiling details about this particular piece of code?"}, {"start": 3186.08, "end": 3188.9599999999996, "text": " That's why they do it. That's why they have the record function here"}, {"start": 3189.68, "end": 3192.08, "text": " So again, the profiler should have been set up somewhere here"}, {"start": 3192.64, "end": 3194.3199999999997, "text": " Uh, let me just see. Yeah"}, {"start": 3194.3199999999997, "end": 3201.2, "text": " So when we were here there was the profiler call here and you can see profile memory true with stack true record shapes true"}, {"start": 3201.2, "end": 3205.4399999999996, "text": " So that means they'll be profiling the memory. What was the memory footprint like?"}, {"start": 3205.8399999999997, "end": 3208.64, "text": " In case they're hitting oms this helps you debug it"}, {"start": 3208.64, "end": 3213.92, "text": " Recording the shapes of the variables and also stack. Um, if I recall correctly"}, {"start": 3214.56, "end": 3220.96, "text": " Uh, um, yeah, I forgot what the command is. You can always enter the profile here and see what the um"}, {"start": 3221.44, "end": 3224.16, "text": " Stack means so let me just find that variable"}, {"start": 3224.64, "end": 3230.56, "text": " Okay, record source information file and line number for the operation. So additional metadata for for the operations"}, {"start": 3230.56, "end": 3233.68, "text": " Let's go back here and let's enter the train step"}, {"start": 3233.68, "end": 3238.8799999999997, "text": " Enter the train step again. I'll have to it keeps on entering these aggregate"}, {"start": 3239.6, "end": 3246.24, "text": " Functions that kind of sucks. So I have to enter the train step using f12 set the break point and then hit f5"}, {"start": 3246.3999999999996, "end": 3248.3999999999996, "text": " So now we are here. So let's see what's going on"}, {"start": 3248.56, "end": 3252.24, "text": " So we set the seed we set the models to train criterion to train"}, {"start": 3252.48, "end": 3257.04, "text": " We call zero grad and i'll actually enter zero grad and there is a particular reason i'm doing that"}, {"start": 3257.12, "end": 3260.8799999999997, "text": " Let me let me show you what why is that why that is? Okay, let's enter this one"}, {"start": 3260.88, "end": 3262.88, "text": " Let's enter this one"}, {"start": 3263.6800000000003, "end": 3265.6800000000003, "text": " Let's enter"}, {"start": 3265.6800000000003, "end": 3267.6800000000003, "text": " This one. Oh my god"}, {"start": 3267.6800000000003, "end": 3269.6800000000003, "text": " And here it is"}, {"start": 3269.6800000000003, "end": 3273.6800000000003, "text": " So the reason is we have this lost scaling and that's because we're doing mixed precision"}, {"start": 3273.6800000000003, "end": 3279.6800000000003, "text": " And so lost scale is set to four and if you recall from my video on again the ultimate guide to scaling"}, {"start": 3279.6800000000003, "end": 3289.6800000000003, "text": " Why they do this is because you want to shift all of the values such that you're you're you're less likely to hit the underflow in the fp16 precision"}, {"start": 3289.68, "end": 3293.9199999999996, "text": " So this kind of shifts all of the values and that's why they uh do"}, {"start": 3294.8799999999997, "end": 3296.24, "text": " One over"}, {"start": 3296.24, "end": 3300.08, "text": " Well, they're doing one over four because the multiply factor is going to do the unscaling"}, {"start": 3300.3999999999996, "end": 3304.56, "text": " So we'll see the this is going to do the scaling and the multiply factor is going to do the unscaling"}, {"start": 3305.04, "end": 3309.7599999999998, "text": " Okay, we'll see that a bit later, but that's a detail. I wanted to show you so that's the zero grad. Um,"}, {"start": 3310.3199999999997, "end": 3311.7599999999998, "text": " Let's continue here"}, {"start": 3311.7599999999998, "end": 3316.16, "text": " We grab the sample we do prepare sample which is gonna do two things"}, {"start": 3316.16, "end": 3321.6, "text": " One is to push the cuda and the second one is to convert to fp16 because of that. I'm just gonna skip"}, {"start": 3322.16, "end": 3324.16, "text": " Um, let's enter"}, {"start": 3324.24, "end": 3329.44, "text": " The the final train step and it's happening instead of a task, which is kind of unintuitive"}, {"start": 3329.44, "end": 3333.12, "text": " Why would you why would the train step be associated with a task?"}, {"start": 3333.92, "end": 3335.2799999999997, "text": " like this"}, {"start": 3335.2799999999997, "end": 3342.3199999999997, "text": " Like this way of structuring the the the the programming model is kind of weird and does not resonate with my with my mind"}, {"start": 3342.32, "end": 3346.42, "text": " Okay, guys, let me enter that one. I'm gonna hit f11"}, {"start": 3347.28, "end": 3351.52, "text": " And here we are. This is the train step again. Take a look at the stack here"}, {"start": 3351.84, "end": 3358.48, "text": " Like the stack is horrible. We have train train train step instead of a trainer train step inside of a task"}, {"start": 3358.96, "end": 3360.96, "text": " But it is what it is"}, {"start": 3361.76, "end": 3364.88, "text": " It's kind of ambitious to try to explain this code on the youtube. So yeah"}, {"start": 3365.6800000000003, "end": 3371.44, "text": " I'm giving my best so again record function forward. So we'll know exactly the details of this part of code"}, {"start": 3371.44, "end": 3376.88, "text": " We'll know how much memory is spent. We'll know the shapes. We'll know the lines everything so that kind of makes it easier to debug"}, {"start": 3377.52, "end": 3379.52, "text": " Let's enter this f11"}, {"start": 3379.92, "end": 3386.2400000000002, "text": " Here we are. We do a forward pass through the model. We passed on that input. That's gonna pass us the the actual tokens"}, {"start": 3386.7200000000003, "end": 3390.48, "text": " So here we are. We hit the coder. We pass the tokens to the decoder"}, {"start": 3391.44, "end": 3397.44, "text": " And now we call this extract features function the extract features function is gonna do a forward pass through the"}, {"start": 3398.2400000000002, "end": 3400.7200000000003, "text": " Decoder transformer. I'm not sure why they called it like that"}, {"start": 3400.72, "end": 3407.52, "text": " I mean, it's not incorrect to to to name it like that but kind of weird extract features. Okay"}, {"start": 3408.16, "end": 3409.8399999999997, "text": " um"}, {"start": 3409.8399999999997, "end": 3411.8399999999997, "text": " Let's continue here"}, {"start": 3411.9199999999996, "end": 3415.68, "text": " Let's enter the extract features and instead of extract features"}, {"start": 3415.68, "end": 3420.72, "text": " There is the extract features scriptable and this the reason why they do this is because of torch script"}, {"start": 3420.72, "end": 3426.3999999999996, "text": " There was again some some constraint that if you want to have optimization you kind of have to do this and"}, {"start": 3426.4, "end": 3430.96, "text": " And it makes the code horrible. I think they know they kind of mentioned that somewhere here"}, {"start": 3431.76, "end": 3433.76, "text": " Blah blah. I don't see it now, but I yeah"}, {"start": 3434.32, "end": 3439.2000000000003, "text": " Um a scriptable subclass of this class has an extract features method and calls super extract features"}, {"start": 3439.2000000000003, "end": 3444.56, "text": " But super is not supported in torch script a copy of this function is made to be used in the subclass instead"}, {"start": 3445.2000000000003, "end": 3446.56, "text": " Okay"}, {"start": 3446.56, "end": 3448.56, "text": " Let's continue here"}, {"start": 3449.92, "end": 3455.52, "text": " Let's continue here. Okay. This part is not interesting. So the first thing we do is forward embedding"}, {"start": 3455.52, "end": 3457.52, "text": " so that's gonna take our our"}, {"start": 3458.0, "end": 3463.04, "text": " Basically tokens so from two to whatever we have 128 of those"}, {"start": 3463.84, "end": 3467.84, "text": " And we are going to now convert them. We're gonna map them into"}, {"start": 3468.46, "end": 3471.52, "text": " Multi-dimensional space. So let's let me enter this part"}, {"start": 3472.64, "end": 3476.88, "text": " So let me enter this forward part and let's see what's going on"}, {"start": 3477.52, "end": 3479.52, "text": " So basically we just call"}, {"start": 3479.52, "end": 3486.64, "text": " All yeah, this is the uh, the torch embedding. So it's just gonna embed basically everything out of the box basically"}, {"start": 3487.7599999999998, "end": 3493.44, "text": " Okay, so this was actually the position. Okay, we first embedded the positions. Let me see. Where is the actual"}, {"start": 3494.08, "end": 3495.52, "text": " embedding going on"}, {"start": 3495.52, "end": 3496.56, "text": " um"}, {"start": 3496.56, "end": 3502.0, "text": " Okay, so here's where where we embedded tokens so we embed them here. So now we have token embeddings"}, {"start": 3502.16, "end": 3506.16, "text": " So let me show you this so we have token embeddings the shape of this thing is"}, {"start": 3506.16, "end": 3510.8199999999997, "text": " 128 512 let's see. What's the shape of the position?"}, {"start": 3512.02, "end": 3518.74, "text": " Positions it should be also 128 but we have 768 here. So that means we we would have a mismatch"}, {"start": 3519.14, "end": 3521.7, "text": " So because of that we'll have to first do the projection"}, {"start": 3522.2599999999998, "end": 3523.14, "text": " um"}, {"start": 3523.14, "end": 3526.42, "text": " Okay, embedding scale is set to one. So this is kind of no up"}, {"start": 3527.54, "end": 3529.06, "text": " we do the"}, {"start": 3529.06, "end": 3530.18, "text": " projection"}, {"start": 3530.18, "end": 3535.3799999999997, "text": " From 512 to 768 and now we can add positions on top of x"}, {"start": 3535.38, "end": 3538.26, "text": " So that's the initial part of the transformer logic"}, {"start": 3538.26, "end": 3542.6600000000003, "text": " We do the embedding we add on top of it the positionally encodings and now we start doing the forward pass"}, {"start": 3543.1400000000003, "end": 3546.02, "text": " Okay, let's continue here and let's exit we're returning back"}, {"start": 3546.6600000000003, "end": 3554.98, "text": " The embeddings the positions and the combined embeddings and positions after we do the additional projection. Okay, let's exit this part"}, {"start": 3556.1800000000003, "end": 3559.2200000000003, "text": " We are again in the forward pass of the transformer decoder"}, {"start": 3559.3, "end": 3563.06, "text": " You can always take a look at here to to figure out where we are transformer decoder"}, {"start": 3563.06, "end": 3567.14, "text": " We're doing the forward pass. We've done the embedding part here. We create the"}, {"start": 3568.58, "end": 3570.58, "text": " mask the"}, {"start": 3570.74, "end": 3575.94, "text": " Attention mask. So let me show you how it's going to look like i'm gonna kind of quickly walk you through this part"}, {"start": 3576.42, "end": 3578.42, "text": " It's going to create this"}, {"start": 3578.58, "end": 3581.86, "text": " Tri u is going to create a triangular matrix"}, {"start": 3582.9, "end": 3584.9, "text": " It's going to be filled with bunch of"}, {"start": 3585.94, "end": 3587.46, "text": " minus int"}, {"start": 3587.46, "end": 3592.34, "text": " So negative infinity and that's going to be used to create a causal decoder, right?"}, {"start": 3592.34, "end": 3598.9, "text": " So we'll just add that to the attention scores and that's going to after applying the softmax"}, {"start": 3598.9, "end": 3603.1400000000003, "text": " You'll end up with tokens that can only attend to itself and the previous tokens"}, {"start": 3604.1000000000004, "end": 3607.46, "text": " So again here we create 128 times 128"}, {"start": 3608.02, "end": 3613.46, "text": " Matrix of zeros then we fill it with negative infinity and then we just call this"}, {"start": 3613.46, "end": 3621.2200000000003, "text": " This function that will basically create a triangular map mask out of this. So let me step over this and let me show you how this looks like"}, {"start": 3622.7400000000002, "end": 3624.7400000000002, "text": " Here is how it looks like again"}, {"start": 3625.3, "end": 3630.26, "text": " Zeros and then on this upper the upper triangle is minus infinities"}, {"start": 3630.9, "end": 3632.1, "text": " Okay"}, {"start": 3632.1, "end": 3638.98, "text": " Let's continue here. We're not using a lib so we can step over all of this and we just return back that that mask"}, {"start": 3638.98, "end": 3641.46, "text": " We just created. Okay, so that's the attention mask"}, {"start": 3641.46, "end": 3648.98, "text": " We do some transpose here operation. So because we are batch number of tokens number of channels now we are"}, {"start": 3649.7, "end": 3655.62, "text": " Uh number of tokens and then batch and the number of channels again optimization detail the lot of transformer implementations do"}, {"start": 3656.7400000000002, "end": 3660.02, "text": " Okay, we just store some interstates blah blah. Now we do"}, {"start": 3660.58, "end": 3666.58, "text": " A bunch of passes through all of the layers. We have 12 layers and we are going to do four pass through all of them"}, {"start": 3666.82, "end": 3668.82, "text": " i'm going to skip the"}, {"start": 3668.9, "end": 3670.9, "text": " actual layer logic"}, {"start": 3670.9, "end": 3676.82, "text": " Uh, that would be too much i'm going to skip it, but it's just classic transformer stuff. So let me exit this"}, {"start": 3677.3, "end": 3678.98, "text": " Let me exit here"}, {"start": 3678.98, "end": 3680.58, "text": " uh, let me just"}, {"start": 3680.58, "end": 3685.14, "text": " Put breakpoint here hit f5 and let's exit the transformer"}, {"start": 3686.02, "end": 3689.86, "text": " We do some again now we now we have again, let me show you the um"}, {"start": 3690.02, "end": 3694.02, "text": " So we now passed all those vectors through the transformer 12 layers in total"}, {"start": 3694.9, "end": 3696.9, "text": " And we end up with the shape"}, {"start": 3696.9, "end": 3704.28, "text": " 128 768 and so you see it's kind of messed up. So we just do the transpose operation again and now we end up with the"}, {"start": 3704.84, "end": 3705.88, "text": " correct"}, {"start": 3705.88, "end": 3708.04, "text": " um the usual shape"}, {"start": 3708.84, "end": 3714.12, "text": " Okay, that's it. And finally the projection so we have 512 instead of 768"}, {"start": 3714.52, "end": 3719.56, "text": " Let me do f10. That's it. We return the interstates. We return the attention blah blah blah"}, {"start": 3719.88, "end": 3722.76, "text": " And we turn x which are the actual output embeddings"}, {"start": 3722.76, "end": 3728.6200000000003, "text": " Okay, let's continue. That's the that part and now we call the output layer"}, {"start": 3729.8, "end": 3736.2000000000003, "text": " And that's gonna project this into I guess, um vocab size. So yeah, you can see here 51 200"}, {"start": 3737.6400000000003, "end": 3739.32, "text": " Let's continue"}, {"start": 3739.32, "end": 3742.84, "text": " Let's continue and that's the forward pass through the model now we compute the loss"}, {"start": 3743.32, "end": 3746.36, "text": " Um, i'm gonna kind of skip through this a little bit"}, {"start": 3746.36, "end": 3754.52, "text": " So we first to get normal probabilities. It's gonna this is gonna apply the log softmax on top of our um outputs here"}, {"start": 3755.4, "end": 3757.4, "text": " then we do some reshaping and"}, {"start": 3758.48, "end": 3760.48, "text": " Let's see the shapes"}, {"start": 3760.84, "end": 3762.84, "text": " And those of you who are still watching"}, {"start": 3763.6400000000003, "end": 3769.56, "text": " I'd love to hear your feedback. Like if you have any feedback, it would be super appreciated. So 128 51"}, {"start": 3769.56, "end": 3776.04, "text": " 1,200 whereas now these are normalized because we applied the softmax. Okay, so"}, {"start": 3776.7599999999998, "end": 3779.56, "text": " We get the targets. This is just going to fetch those"}, {"start": 3780.7599999999998, "end": 3782.7599999999998, "text": " sample targets so that's going to fetch those"}, {"start": 3783.54, "end": 3786.04, "text": " 128 tokens that were shifted by one to the right"}, {"start": 3787.0, "end": 3791.72, "text": " I'm gonna convince you that that's indeed the case so you can see here three four five"}, {"start": 3791.7999999999997, "end": 3796.68, "text": " That's the target sequence we were using the whole time and now we apply the nl loss. So that's the"}, {"start": 3796.68, "end": 3802.7599999999998, "text": " Um negative log likelihood loss and you can see it's a bit more complex than than your usual implementation"}, {"start": 3802.7599999999998, "end": 3806.3599999999997, "text": " So your usual implementation will be just this but when you're dealing with um"}, {"start": 3807.3199999999997, "end": 3811.48, "text": " I guess uh, like big dimensionality and a lot of a big vocab"}, {"start": 3812.2799999999997, "end": 3817.0, "text": " Then you can see here they have this if branching. So if the number of elements is smaller than"}, {"start": 3817.64, "end": 3819.64, "text": " What is this like?"}, {"start": 3819.64, "end": 3827.16, "text": " 2 billion then you just execute this standard f and l loss if it's not the case then you have to do some"}, {"start": 3827.56, "end": 3833.4, "text": " some a bit more intricate logic because otherwise, um, you would encounter some uh,"}, {"start": 3834.52, "end": 3837.48, "text": " Numerical issues so because we are smaller than this"}, {"start": 3838.6, "end": 3842.68, "text": " Let's see what the number of elements is for us. We have 51 000 and then"}, {"start": 3843.7, "end": 3847.24, "text": " Dimensionality is 128. So I guess it's yeah, it's below"}, {"start": 3847.24, "end": 3850.04, "text": " The threshold so we just do this. Um"}, {"start": 3851.08, "end": 3853.7999999999997, "text": " Negative log likelihood and that's it. That's everything"}, {"start": 3853.7999999999997, "end": 3859.56, "text": " That's how transformers work like you apply just you just apply this nl loss on the output and"}, {"start": 3859.7999999999997, "end": 3864.52, "text": " Compared with the targets that are shifted by one and that's it the whole magic behind transformers"}, {"start": 3865.7999999999997, "end": 3870.3599999999997, "text": " Okay. Now here's where the actual mixed precision magic happens and we are almost done with this video"}, {"start": 3871.0, "end": 3875.9599999999996, "text": " I'm gonna quickly walk you through all of this. We're just like logging some outputs. That's not important"}, {"start": 3875.96, "end": 3878.12, "text": " And i'm gonna skip all of this"}, {"start": 3879.4, "end": 3884.12, "text": " We're just storing some stuff blah blah blah. We're just storing some stuff. Let's go back here"}, {"start": 3884.44, "end": 3889.56, "text": " Let's return back the loss. So here is the loss and now let's go for the second part"}, {"start": 3889.8, "end": 3891.0, "text": " So this is the interesting part"}, {"start": 3891.0, "end": 3896.6, "text": " So we do backward pass and we again do record function backward to just profile this piece of code"}, {"start": 3897.2400000000002, "end": 3903.7200000000003, "text": " And let's do um, let's do f11 here. Let's enter so now because we have scaling. Let's see what happens. So"}, {"start": 3904.28, "end": 3905.7200000000003, "text": " um"}, {"start": 3905.72, "end": 3907.72, "text": " Scaler is not none. So we enter this branch"}, {"start": 3908.2, "end": 3911.0, "text": " And what we do is we scale the loss, okay"}, {"start": 3911.08, "end": 3917.7999999999997, "text": " So whatever the loss is and it's thousand four hundred something and it's that big because we were just doing summation in the nl loss"}, {"start": 3917.8799999999997, "end": 3919.24, "text": " As the reduction method"}, {"start": 3919.24, "end": 3926.4399999999996, "text": " So if I enter f11 here, you can see what what what we do is we simply take that number and we multiply it by four"}, {"start": 3926.68, "end": 3929.64, "text": " And that's what I previously explained you because of that"}, {"start": 3929.64, "end": 3936.44, "text": " That when you multiply the loss by four you trivially multiply all of the gradients are going to be uh multiplied by four"}, {"start": 3937.0, "end": 3938.92, "text": " that's how the"}, {"start": 3938.92, "end": 3941.3199999999997, "text": " Chain rule works. So let's hit f10"}, {"start": 3942.3599999999997, "end": 3946.52, "text": " Let's now do the backward pass. I don't think anything special is going to happen here"}, {"start": 3947.16, "end": 3948.7599999999998, "text": " I'm just gonna yeah"}, {"start": 3948.7599999999998, "end": 3954.92, "text": " Everything is yeah, we are we exited the the um train step and now let's"}, {"start": 3954.92, "end": 3962.52, "text": " Let's go back. We are not we're now level up in the train step of the trainer not in the train step of the task"}, {"start": 3963.56, "end": 3970.76, "text": " Let's hit f10. We just append the outputs blah blah blah. That part is not interesting and okay, let's continue"}, {"start": 3971.16, "end": 3974.6800000000003, "text": " so here you can see this is kind of uh wrapped up inside of a uh,"}, {"start": 3975.48, "end": 3980.04, "text": " runtime error out of memory exception so that the whole training procedure is wrapped up in that"}, {"start": 3980.84, "end": 3982.52, "text": " uh, and um"}, {"start": 3982.52, "end": 3986.52, "text": " I think this is this is it. We just have a single batch. That's why we exit the"}, {"start": 3987.32, "end": 3989.32, "text": " this this for loop"}, {"start": 3989.4, "end": 3991.4, "text": " through the samples"}, {"start": 3991.48, "end": 3993.08, "text": " and um"}, {"start": 3993.08, "end": 3997.48, "text": " Let's continue here sample size was 128 because we only had 128 tokens"}, {"start": 3998.2, "end": 4003.56, "text": " Um, this is some logic that will be necessary if we had multiple machines, which we don't so that's fine"}, {"start": 4004.52, "end": 4006.92, "text": " Uh, okay, so we now enter this interesting part"}, {"start": 4007.56, "end": 4011.72, "text": " And this is pretty much the end of the video. So try accept as you can see here"}, {"start": 4011.72, "end": 4016.3599999999997, "text": " There is so many accepts they're catching so they're catching the floating point error. They're catching the overflow error"}, {"start": 4016.3599999999997, "end": 4018.2, "text": " They're catching the outer memory error"}, {"start": 4018.2, "end": 4021.8799999999997, "text": " So all of these things are necessary when you're training at these big scales"}, {"start": 4021.8799999999997, "end": 4025.16, "text": " You want to catch the errors you want to handle them gracefully so you can see here"}, {"start": 4025.3999999999996, "end": 4031.0, "text": " If there is a floating point error, I think they do have this nan detector. Yeah, we'll get to that a bit later"}, {"start": 4031.08, "end": 4034.12, "text": " Let me now first walk you through the uh reduced gradients"}, {"start": 4034.6, "end": 4040.52, "text": " So here actually nothing is going to happen because we have a single gpu. Otherwise, uh, you would take"}, {"start": 4040.52, "end": 4045.8, "text": " Uh gradients from across the devices and then you would do the all reduce"}, {"start": 4045.88, "end": 4049.96, "text": " So that means you literally take all of the gradients you push them onto a single device"}, {"start": 4050.12, "end": 4055.96, "text": " You divide them by the number of devices and then you do that for all all the devices and all of the devices will have"}, {"start": 4056.04, "end": 4057.64, "text": " The same state now"}, {"start": 4057.64, "end": 4060.7599999999998, "text": " If that makes sense, but less important for now, let's focus on the loss scaling"}, {"start": 4061.4, "end": 4066.36, "text": " Multiply grads. Let's see what's going on here. Okay. So what we do is"}, {"start": 4066.36, "end": 4071.08, "text": " We divide so we have this multiply grads and we divide this is one"}, {"start": 4071.8, "end": 4079.56, "text": " divided by 128 the reason we divide by 128 is because nll loss had as a reduction method had a sum"}, {"start": 4080.1200000000003, "end": 4085.0, "text": " And because we had 128 elements, we now want to divide by 128 if that makes sense"}, {"start": 4086.1200000000003, "end": 4088.1200000000003, "text": " So let's enter here"}, {"start": 4088.2000000000003, "end": 4090.2000000000003, "text": " Let's see what's going to happen here"}, {"start": 4090.2000000000003, "end": 4092.04, "text": " uh, so"}, {"start": 4092.04, "end": 4099.0, "text": " Multiply grads. So we just take that multiply factor, which is now one over four because we want to do the unscaling of gradients"}, {"start": 4099.24, "end": 4104.28, "text": " And we additionally multiply by this one one over 128. This is the c constant"}, {"start": 4105.64, "end": 4113.56, "text": " That's that number and then that's kind of all aggregated inside of this variable and we're going to later apply the variable to actual gradients"}, {"start": 4113.56, "end": 4116.04, "text": " We'll see that a bit later. Okay, let's exit here"}, {"start": 4116.68, "end": 4118.68, "text": " Now gradient clipping is going to happen"}, {"start": 4118.68, "end": 4122.04, "text": " again, I will have to skip this part, but um,"}, {"start": 4122.4400000000005, "end": 4127.96, "text": " Okay. No, actually let me quickly enter it. Let me see what's going on there. I think it's kind of fairly intricate"}, {"start": 4129.08, "end": 4134.84, "text": " There's a lot of as you can see a lot of steps, but I think there might be a part that's kind of interesting for us"}, {"start": 4134.84, "end": 4139.16, "text": " Oh, yeah, there is so here is a multiple factor. Here's where it comes into the picture. So we"}, {"start": 4140.200000000001, "end": 4147.400000000001, "text": " we basically grab the gradients of our we grab the norm of the gradients and then we multiply that by the"}, {"start": 4147.4, "end": 4153.639999999999, "text": " The multiply factor and that's kind of optimization detail because you didn't have to multiply the actual gradients"}, {"start": 4153.639999999999, "end": 4155.639999999999, "text": " You just took the norm and then"}, {"start": 4156.28, "end": 4160.04, "text": " Multiply by this factor and you get the equivalent result. That's kind of neat"}, {"start": 4161.16, "end": 4162.36, "text": " Okay"}, {"start": 4162.36, "end": 4166.36, "text": " Let's see what's going on here. Let's continue here"}, {"start": 4167.32, "end": 4169.32, "text": " max norm grad norm"}, {"start": 4169.719999999999, "end": 4174.839999999999, "text": " And again, we have another multiplication for the multiply factor. Why because"}, {"start": 4174.84, "end": 4178.04, "text": " The gradients were the norm was 16"}, {"start": 4178.04, "end": 4182.92, "text": " but we want to clip it to to 1 and that means we have to we have to take this number and"}, {"start": 4183.4800000000005, "end": 4191.0, "text": " Multiply with that number of gradients and then they will obey this property of having gradient norm of 1 if that makes sense"}, {"start": 4191.16, "end": 4195.72, "text": " That's why we have another multiplication. So now the multiply factor has three things going on"}, {"start": 4195.96, "end": 4202.04, "text": " We have the loss on scaling we have we've divided with a sample size and finally we've divided by this"}, {"start": 4202.04, "end": 4204.04, "text": " ratio of the norms"}, {"start": 4204.84, "end": 4208.28, "text": " Let's continue we check some overflow again a bunch of details"}, {"start": 4209.0, "end": 4212.92, "text": " And let's continue we check the grade norms. Let me see what this is"}, {"start": 4215.16, "end": 4216.04, "text": " I"}, {"start": 4216.04, "end": 4222.5199999999995, "text": " Think this is just going to be skipped. So yeah, it's just skipped. Okay. Now we make sure that we don't have"}, {"start": 4223.16, "end": 4225.64, "text": " none of the values in the grads are"}, {"start": 4226.5199999999995, "end": 4230.5199999999995, "text": " infinite so if we had nans then this would be"}, {"start": 4230.52, "end": 4237.160000000001, "text": " This exception would be raised and it would be caught in this particular context here where they have the nan detector"}, {"start": 4237.72, "end": 4241.240000000001, "text": " That kind of let me show you let me show you what it does. So nan detector"}, {"start": 4241.8, "end": 4244.84, "text": " Blah blah blah. It basically is going to capture"}, {"start": 4245.240000000001, "end": 4252.040000000001, "text": " Some of the values so that we can debug it later on detects the first man or in in forward and or backward pass and logs"}, {"start": 4252.280000000001, "end": 4256.76, "text": " Together with the module name. Okay, so it's just gonna help her to to debug"}, {"start": 4257.400000000001, "end": 4259.400000000001, "text": " Where the man occurred?"}, {"start": 4259.4, "end": 4265.24, "text": " And then once you have once they do that they do another for pass here"}, {"start": 4266.759999999999, "end": 4269.4, "text": " Under that context and it's gonna catch the first man"}, {"start": 4269.639999999999, "end": 4274.12, "text": " So they say here rerun the forward and backward pass with hooks attached to print out where it fails"}, {"start": 4274.28, "end": 4277.08, "text": " Okay, so that's kind of how they cope with the errors"}, {"start": 4277.719999999999, "end": 4283.48, "text": " Let's finally enter the optimizer step. That's where we're gonna finally apply the multiplier the famous multiplier"}, {"start": 4283.639999999999, "end": 4286.2, "text": " So let me show you here step step step"}, {"start": 4286.2, "end": 4292.84, "text": " Let's continue here. Okay unscaled grads. Let's see what it is. I think that's going to be our our logic"}, {"start": 4293.88, "end": 4298.5199999999995, "text": " So here we are we enter this part multiply grads with the multiply factor"}, {"start": 4298.5199999999995, "end": 4303.72, "text": " Okay, that's finally the moment where we apply the multiply factor and then we reset it back to one"}, {"start": 4304.12, "end": 4306.44, "text": " And then the whole procedure will will continue"}, {"start": 4306.679999999999, "end": 4314.679999999999, "text": " So here you can see here we just go through the parameters and we multiply the so parameter the gradient associated with that parameter with that weight"}, {"start": 4314.68, "end": 4320.52, "text": " Data and then multiply by c and c is the the the multiplier we were accumulating over time"}, {"start": 4320.52, "end": 4328.4400000000005, "text": " So we had three factors again, if you recall the sample size the gradient on scaling and finally the ratio of the norms"}, {"start": 4329.56, "end": 4330.84, "text": " I'm gonna"}, {"start": 4330.84, "end": 4333.320000000001, "text": " Step out of this. So hit this step out"}, {"start": 4334.200000000001, "end": 4338.84, "text": " Error and we are done. We are done here. We've done the stepping"}, {"start": 4339.8, "end": 4343.400000000001, "text": " We update the scalar. So the scalar is going to do what?"}, {"start": 4343.4, "end": 4345.4, "text": " Okay, nothing interesting here"}, {"start": 4346.5199999999995, "end": 4351.4, "text": " We can exit all of this and we are back to our train step in the trainer"}, {"start": 4353.0, "end": 4357.799999999999, "text": " We just debug some information we didn't have any nonce and that's pretty much it"}, {"start": 4357.799999999999, "end": 4362.839999999999, "text": " So I think I can skip all of this. I can skip all of this a lot of details"}, {"start": 4363.639999999999, "end": 4367.879999999999, "text": " Yeah, go at your own pace explore this code if you're curious to learn more"}, {"start": 4367.88, "end": 4373.0, "text": " But uh, I think we saw most of the interesting parts. So let me just do this"}, {"start": 4373.4800000000005, "end": 4379.32, "text": " Let me see why we are okay. So i'll just have to put f breakpoint there hit f5 here. We are"}, {"start": 4380.2, "end": 4384.28, "text": " So as you can see, we're finally back in the in the in the in the train.py file"}, {"start": 4385.0, "end": 4387.08, "text": " And we are inside of the train function here"}, {"start": 4387.72, "end": 4389.74, "text": " Let me see whether there is something interesting"}, {"start": 4390.36, "end": 4395.72, "text": " And the feedback, uh, we're gonna just skip this part. Nothing's gonna happen. It's gonna be a no-wisdom"}, {"start": 4395.72, "end": 4397.72, "text": " Um, and uh"}, {"start": 4398.360000000001, "end": 4400.360000000001, "text": " If we should stop"}, {"start": 4400.6, "end": 4406.52, "text": " And for now we are not stopping so that means we are again gonna fetch the next samples and uh"}, {"start": 4407.0, "end": 4408.4400000000005, "text": " Continue this whole"}, {"start": 4408.4400000000005, "end": 4411.4800000000005, "text": " Loop and that's it. So we're just gonna keep on doing this training"}, {"start": 4412.52, "end": 4415.8, "text": " Guys, that's it. Uh, you hopefully saw what it takes to"}, {"start": 4416.360000000001, "end": 4419.88, "text": " Kind of saw what it takes to build these llms. Um, we did"}, {"start": 4419.88, "end": 4425.72, "text": " Exclude the whole distributed part of the equation, which is probably the hardest part"}, {"start": 4425.72, "end": 4428.36, "text": " So like doing the model of parallelism the data"}, {"start": 4428.92, "end": 4431.8, "text": " Parallelism the the pipeline parallelism all of that"}, {"start": 4432.36, "end": 4437.8, "text": " Goodness, and then there is a zero optimizer where you do the partitioning of the activation. So there is so many details"}, {"start": 4438.36, "end": 4440.92, "text": " Again, we also skipped the checkpointing. So like"}, {"start": 4441.8, "end": 4444.36, "text": " Yeah tragic cool. I'm gonna stop here"}, {"start": 4445.24, "end": 4447.4800000000005, "text": " Let me know please let me know what you think"}, {"start": 4447.48, "end": 4454.04, "text": " Um, depending on that i'm gonna probably modify my my my uh, the videos i'm gonna be creating because"}, {"start": 4454.839999999999, "end": 4458.679999999999, "text": " By doing this I learned a lot and so I thought just kind of walking you through"}, {"start": 4459.24, "end": 4461.959999999999, "text": " Uh the learning so it didn't feel like like a big of an effort"}, {"start": 4462.04, "end": 4465.639999999999, "text": " But like going forward I'll probably won't be doing this unless people find it useful"}, {"start": 4465.639999999999, "end": 4469.5599999999995, "text": " So do let me know down in the comments whether this was actually interesting and useful for you"}, {"start": 4469.959999999999, "end": 4473.5599999999995, "text": " Having said that if you like this video, please share it out. That's the best way to do it"}, {"start": 4473.56, "end": 4479.1, "text": " This channel also subscribe to this channel if you haven't already and see you next time"}]