Ultimate Guide To Scaling ML Models - Megatron-LM | ZeRO | DeepSpeed | Mixed Precision
🚀 Sign up for AssemblyAI's speech API using my link 🚀
https://www.assemblyai.com/?utm_source=youtube&utm_medium=social&utm_campaign=theaiepiphany 

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I show you what it takes to scale ML models up to trillions of parameters!

I cover the fundamental ideas behind all of the recent big ML models you must have heard of like Meta's OPT-175B, BigScience BLOOM 176B, EleutherAI's GPT-NeoX-20B, GPT-J, OpenAI's GPT-3, Google's PaLM, DeepMind's Chinchilla/Gopher models, etc.

I cover the ideas of data parallelism, model/pipeline parallelism (e.g. GPipe, PipeDream, etc.), model/tensor parallelism (Megatron-LM), activation checkpointing, mixed precision training, ZeRO (zero redundancy optimizer) from Microsoft's DeepSpeed library and many more.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
Papers:
✅ Megatron-LM paper: https://arxiv.org/abs/1909.08053
✅ ZeRO (DeepSpeed) paper: https://arxiv.org/abs/1910.02054v3
✅ Mixed precision training paper: https://arxiv.org/abs/1710.03740
✅ Gpipe (pipeline parallelism) paper: https://arxiv.org/abs/1811.06965

Articles:
✅ Collective ops: https://en.wikipedia.org/wiki/Collective_operation
✅ IEEE float16 format: https://en.wikipedia.org/wiki/Half-precision_floating-point_format 
✅ Google Brain's bfloat16 format: https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00:00 Intro to training Large ML models (trillions of params!)
00:02:04 (sponsored) AssemblyAI's speech transcription API
00:03:31 Data parallelism
00:01:52 Pipeline/model parallelism
00:14:52 Megatron-LM paper (tensor/model parallelism)
00:18:22 Splitting the MLP block vertically
00:30:07 Splitting the attention block vertically
00:39:24 Activation checkpointing
00:42:12Combining data + model parallelism
00:45:42 Scaling is all you need and 3D parallelism
00:47:57 Mixed precision training paper
00:49:57 Single vs half vs bfloat number formats
00:51:32 Storing master weights in single precision
00:55:41 Loss scaling
00:58:13 Arithmetic precision matters
01:00:32 ZeRO optimizer paper (DeepSpeed library)
01:06:37 Partitioning is all you need?
01:11:02 Where did all the memory go?
01:21:42 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#scaling #deepspeed #megatron
 What's cracking guys? The idea of this video is going to be extremely ambitious. I'm going to try and give you the most seminal and important ideas for how to scale your models up to scales of hundreds of billions and even trillions of parameters. So I'm gonna cover multiple papers as you can see here so we have Megatron LM from Nvidia here introducing the tensor or model parallelism. I'm gonna walk you through quickly through pipeline parallelism idea from the GPy paper. I'm gonna walk you through the data parallelism idea which is probably something most of you are already familiar with but just to round them up I'm gonna cover that one as well. I'm gonna quickly go through the mixed precision training paper as well and finally I'm gonna go through the zero optimizer so the zero redundancy optimizer from Microsoft that enabled training models up to 1 trillion parameters big. So why should we why should you care about this video? Well the thing is the following so if you ever want to train a model that's bigger than let's say well if you're dealing with say GPT style of a model transformer if you want to go like above 1.5 billion parameters you literally need more than 32 gigabytes of VRAM which most of you do not have. So up until basically 1.5 billion a single GPU with 32 gigabytes of VRAM can handle that training if you don't have any of these optimizations. Other than that it's impossible to train bigger models so combining all of these ideas you'll be able to literally train super big models some of them you must have seen recently published models such as the bloom from the big science organization and also opt 175 B model from from meta so all of those models are like hundreds and hundreds of billions of parameters other examples are like chinchilla from deep mind go for poem so yeah guys before we go there I want to make a quick shout out to assembly AI and thank them for sponsoring this video basically they offer a super powerful speech understanding and transcription API you can see how easy it is to get started in Python so literally a couple of lines of code and voila you get your transcription back as well as in this case sentiment analysis they offer various other understanding features such as entity detection etc etc you can also check out the no code solution although I guess this is not that interesting for most of you but if you just want to quickly understand the features they support you can check out this page as well if you are struggling to get started which I doubt because it's super simple there is even a tutorial very detailed tutorial on how to get started and if you like ideas for projects they have a YouTube channel with cool videos such as this one which is like literally 30 minutes long explaining how to summarize various podcasts with with their API so that's very cool finally they have a very cool set of blocks in general and tutorials on how to get started with various other models such as the Lee so yeah I think that's that's fairly cool so give them a go support the channel I'm linking the free API token down in the video description so you can use it and get started in a couple of seconds literally you get for the toy projects you have no problems with the free account you get three hundred hours of transcription if you want to upgrade you can always do that and having said that let's go back to the video so first let's start actually with data parallelism because that's probably the simplest idea to explain the idea is the following so you have a neural network and what you do is you literally take the weights of your neural network and you just replicate it so you copy paste those weights across multiple devices so you have like let's say and GPUs and you literally copy paste that same network across all of the devices and then what you do is you take a batch of data so you basically take a batch of data and so it's gonna look something like this so literally you have a batch of data here and what do you do is you split it into multiple parts so let's assume here we just have three devices so that means you're gonna split the batch so this is the batch dimension by the way this here I'm gonna simplify it like just gonna represent the input tensor as a 2d object even though it's usually gonna be depending on your data 3d 40 whatnot so yeah so you grab the first chunk the first part and you feed that into the first device here then you grab the second chunk here you grab the second chunk and you pass that second chunk of data into the second GPU GPU 2 and finally you grab the final piece here and you pass that one here then you do a forward pass through your networks you're gonna do a forward pass here for a pass here for a pass here and then you calculate the gradients so now now what happens is that all of these devices have different gradients as a consequence of you feeding different parts of your input batch and now we just need to basically reduce all of those gradients and when I say reduce I literally mean you you sum them up and then divide by the number of devices to get the average gradient so there are multiple ways how you can implement this in this particular image there is this thing called parameter server so you do the basically reduce and scatter operation so you grab all of the the gradients you're gonna send the gradients here so you're gonna send and by the way I'm sorry I'm kind of glitching with this like highlighter here but hopefully you can you can see what I'm doing so we send the gradients here we send the gradients here we send the gradients here we do the averaging of those gradients and then you literally so in this particular implementation the weights are updated directly on the parameter server and then you do the scatter operation you literally send back that new the new weights back onto onto your devices you send back the the weights okay and that's how you achieve training with a much higher throughput because you can imagine here although there is some overhead obviously you literally have 3x higher throughput compared to if you were to train on a single GPU okay and when I say GPU it can be a TPU it can be arbitrary device basically so what's the problem with this idea the problem is is it's the underlines presupposition here is that all of these networks can fit into a single device which is the problem because we know that our models are much bigger and usually cannot fit into a single device so what do we do then well then we do like things such as model parallelism so I'm gonna show you the first thing so that's the data parallelism I'm going to show you the the model parallelism idea so this is a similar paper from Nvidia called Megatron LM training multi-billion parameter language models using model parallelism it's literally the idea behind both bloom both opt 175 B from meta I think some of the illu theory eyes models were also backed up by Megatron at least implicitly because I think they're using deep speed and GPT new X code base and deep speed basically leverages Megatron in the background I could be wrong here I'm fairly sure this is this is correct anyway so let's dig into the paper so they say here this is called intro layer model parallel approach I want to contrast that quickly and I'm going to explain this in a bit more depth a bit later with the pipeline approach which is the inter layer splitting so the difference is the following so imagine you have a transformer and I'm gonna be using transformer as the running example so imagine you have a transformer here the main difference between these two approaches is the following so you have n layers here so you have n blocks of your transformer and the pipeline approach does the following thing you literally grab let's say we have four devices if we have four devices then we're gonna do the following we're gonna take device the layers one through four and then five through what eight and we're gonna send them on to separate devices so that's the pipeline approach so literally take this and you put that onto a separate device here then you take this one and you put it so let me let me let me draw it here and I'm literally not explaining the pipeline idea so I might as well explain it the whole way so literally you take the layers five through eight and then you feed them into the different device here and then you take the layers the next four layers because I don't know how to add I'm just gonna use yeah whatever so literally you feed that into the third block and then the last one the last four layers are gonna be fed into the last block here okay so that's what the what the pipeline parallelism approach does it literally splits the model so called well I think this is yeah this is the horizontal splitting and you can now imagine that each of these devices needs to hold literally four times less memory roughly I mean at least when it comes to weights there is also the up to optimizer states etc etc okay so the idea on the opposite side here with model parallelism from the Megatron LM paper is instead you're splitting you're doing the so-called vertical split so you do the following you instead let me take some color such as green and let's imagine we have just two devices gonna be easier to draw so you literally split your transformer like this so literally vertical split it and then you send this piece here on to one device and you send the other piece the other half of the transformer on to the other device so that's the model parallelism or the tensor parallelism as a different different name for that same like concept okay because I'm already halfway through explaining the pipeline parallelism let me let me quickly go through through that so that's the idea I think well one of the papers that mentioned that idea was the G pipe from Google others were I think like pipe dream or something there are multiple papers implementing similar ideas but the idea is as I mentioned this one and what's the problem here well the problem is if you just naively approach training a model that's and that's split using the pipeline parallelism you have a problem with with these so-called bubbles so let me try and demonstrate what I mean by that so imagine you feed a batch of data through this pipeline okay so you have a batch of data here I'm gonna denote it as well I guess like this just to make a difference between the layers and and the data so the thing is while this while you're doing the feed forward through the first device I through the first let's say four layers the other three devices are idle and you can see that on this diagram like this so you can see here you can literally see here that these devices here are idle so these three are doing exactly nothing okay there is no no computation going on so that's kind of wasteful right and then once you do the feed forward through the first four layers then you end up you basically send the activations to the second device and you start doing the feed forward here and at this point of time the three devices the first and the second the third and the fourth one are idle so you can see here literally these devices are doing nothing so you can you can see here this is gonna be idle these two devices are idle etc etc and so then you end up you propagate through all of the four devices and then you then you start doing the backprop so you want to calculate the gradients and when you start doing that again in the in the backward pass you are you're literally waiting so so you're doing backprop here but these three devices are being idle so this is super wasteful so there is a simple way how you can avoid this and it's called micro batching so split your mini batch into multiple micro batches and then you start feeding them into your pipeline and increase the throughput let me let me show you what I mean by that so you can see the diagram is here but let me explain conceptually what it means so here what you do is you split the simple data into multiple chunks into so-called micro batches and you feed this first part here so you feed it here and while it's running here and then the moment it starts running through the through the second device you immediately feed the second part the second micro batch through this device and now you have two devices not being idle and then as soon as this device passes on the activations to the third device you can feed the third micro batch into this device and so on and so forth you get the point and you can see the diagram looks like like this so initially for the duration of the first micro batch you literally have you literally and let me just change the color here you literally have four devices being idle so you can still here see that the four devices are being idle which is wasteful and then only two devices are being being idle and then only one and finally here you can see that all of the devices are being busy and you have the the the 4x the throughput compared to the naive implementation of the pipeline parallelism so in order to get as many of these columns where all of the four devices are active you want to have a lot of micro batches and what the GPy paper showed basically empirically is that you want to have M greater or equal than 4k where K is the number of partitions or the number of devices a fancy name saying number of devices so that means if you have four devices you want to have 32 plus micro batches and then literally they showed that you almost have linear scaling of the throughput which is awesome okay so that's one way you can take a huge model split it across multiple devices each of the devices will have only a portion a slice of the network weights and thus you can train this whole system faster because you have faster throughput and it's also feasible in the first place because yeah you cannot kind of stick it into a single device okay so that's the second idea now let's go through the actual paper this one is very fun let's get back to the first sentence so we started here intralayer model parallel approach we now know what that means so it's intro we're splitting the actual layer we're not splitting across layers we take a single layer and we split it into two parts or four parts or eight parts or whatnot our approach does not require a new compiler or library changes is orthogonal and complementary to pipeline model parallelism and can be fully implemented with insertion of a few communication operations in native part or so the cool thing is the complementary part is very important so this part is super important because that means you can combine these two approaches to scale to even bigger model sizes okay then they say this we sustain 15.1 petaflops across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline that sustains 39 teraflops which is 30% of peak flops so let me kind of break this down first of all I forgot to mention here they're training a model that has 8.3 billion parameters and they're training on 512 GPUs okay so what I say here is that they achieve a throughput of 15.1 peta so peta is I guess 10 to the power of 15 or something so it's like thousand tera or 1024 depends when you ask flops is the the floating point operations per second so that means you have this many operations per second that's a huge number okay and what they say is that they have a 60 76 percent scaling efficiency so what does that mean in practice that means the following let me open up a calculator here so we have that the single device single GPU has 39 teraflops so if you multiply 39 times the number of GPUs ideally this will be ideal scaling we'd end up with 19 petaflops and you can see here that they only end up with 15.1 and that's where this 60 76 percent scaling efficiency comes into place if I multiply this with 0.76 we end up with exactly 15.1 petaflops okay so basically you can see that the GPU baseline has 30% of the theoretical peak so that the GPU will never have its its its full theory it will never reach its full theoretical peak and then when you additionally scale across multiple devices you'll never have ideal linear scaling but it's close enough so that's cool okay so let's continue here let me see what's interesting we're gonna see how we are gonna split the two particular blocks so we're gonna see how we split the MLP block first and then I'm gonna show you this is probably a bit easier how do we split the the attention block and they also split the output embeddings as well as the input embeddings and finally a detail worth mentioning is that this residual connections additions layer norms are actually replicated across devices so there is some duplication going on for various reasons they decided that's that's the most optimal thing to do but that's the plan let's see how do we how do we vertically break up the MLP how do we vertical split that the attention block that's gonna be the the main gist of this paper okay let's start here here you can see the the diagrams of how how the splitting happens so the upper diagram here shows you the basically oops shows you the MLP block and you can see this is the first linear layer the second near layer and this is how it's kind of split down but let me go into the formulas I'm gonna slowly understand how this works but first on the high level what happens here is that you can see that this weights you take the weights of this linear layer you split them column wise and you keep a 2 on one device and you keep a a 1 on the second device you split the matrix that you use for your feed forward layer and you split it into two parts you do the same thing here but the difference is here you split it row wise so you can see here we don't have column wise splitting we have row wise splitting okay and so you take B1 you put B1 on one device you put B2 on the second device so basically you can see here that all of these operations are happening on a single device okay and then there is this G operation going on which is gonna synchronize the two devices more on that a bit later and the second device obviously here is processing the second part of the MLP okay so it has the a1 and B1 so that's and that's how we achieve the vertical splitting but that's only a high-level diagram let's understand why this works and how is it equivalent to just having everything on the same device it might be like maybe tough to grasp on the on the first fly but it's fairly simple once you think about it a bit okay so here is the implementation here is what the layer does here is what the the feed forward layer does you do the this is your input data X this is your weight matrix a and this is the activation unit jelly in this particular case so there is multiple ways we can split we saw the column wise and row wise let's see the advantages and disadvantages of both if we split the weight matrix basically row wise as you can see here then we have a problem and that problem is we need to synchronize the devices after this pass so let's see why that's the case so if we now multiply this you get X one times a one plus X two times a two and then you need to apply jelly so the problem here is the plus and that means that the two devices will need to communicate so that that's the only way you can you can well add those two separate activations right and then you apply the jelly the problem is the jelly is not like is a nonlinear function so jelly of of adding these two is not the same as jelly of the first term plus the jelly of the second term and so they say this approach will require a synchronization point before the jelly function okay let me try and map this matrix multiplication into the diagram with which you're probably familiar with okay so let's start like this we have the input data X I'm gonna assume our batch size is only one so we have something like this so we have X here and we split X into two parts so we have X one and we have X two here okay next up we need to pass that through the feed forward layer so we have a weight matrix we're going to multiply with it so this is going to be just some box here I'm gonna fill in the details a bit later and we end up with the final representation I'm gonna denote it as a red rectangle here and that's it okay and now I'm gonna draw like a matrix a here and we said that we are going to split the matrix a basically row wise right so you're gonna split the matrix a row wise so let's see what happens now so if we multiply this part here and then we can assume in a little bit here if you multiply this part here with this part here what does that mean so that means that we do the following thing we'll literally do this whoops we'll literally do this so we do this is kind of how it's gonna be how it's gonna look like if you if you write like a neural if you if you draw a neural network okay so that's what you've done by multiplying this part with this part and then you multiply this with the second part okay so now you multiply it with the second piece here and that second piece is gonna give you the second output and you end up with something like this and then you do this here and this here and you end up with four outputs so I'm kind of using only four to to to yeah simplify the drawing so the problem is that all of these outputs here are still incomplete because they are missing the contribution from the x2 portion that's why we have to add the the actual representations so what I do now is let me change the color into let's say red so now you'll have to do the multiplication between this one and this one to end up with the contribution to the first output element okay so this gives us this and then you just repeat repeat repeat and you get the contributions to this one to this one and to this one okay so that's hopefully kind of helps you understand how this maps to the neural network diagrams with which you are used to so as you see we if you if you just do x1 times a1 we only get partial contributions to the output and because of that we also have to add additionally the x2 a2 and only then can we apply the jelly okay now let's go to the second like a way of breaking down the weight matrix and that's the column splitting okay so another option is to split a along its columns like this this partitioning allows the jelly non-linearity to be independently applied to the output of each partitioned gem so you can see what's the what's the benefit here you can literally send X so this time you're sending not X1 is here so please notice this thing this is a very important detail so here we have X1 here we have X so that means we're passing the whole input and as well here and here we had X2 so but a benefit we get here is that we can literally do everything on that device and then apply jelly and that's it we don't have any synchronization going on so that's very cool because synchronization slows us down we have to communicate the weights and sum them up and then return them back okay so this is advantageous as it removes a synchronization point hence we partition the first gem in this column parallel fashion and split the second gem along its rows so it takes the output of the jelly layer directly without requiring any communication as shown in figure 3a so that's the hack we're gonna do so let me now do the following let me first show you how the column splitting is gonna look like so I'm gonna do the same diagram again let me just change the color so I'm gonna do this similar diagram again so we have our X here so this is our X we have something going on here I'm gonna just draw like this like a box here a placeholder and then I'm gonna pick the red color so we denote the output representation with red and let's now multiply the input X so this is again X let's multiply X with our matrix that split column wise this time so that's the main difference okay so now we split it so here you saw the horizontal one here we have a vertical splitting okay so what happens now let's go through this so now when you multiply this with this you literally end up with a full representation so this output here has everything it needs so it's completely has all of the necessary information from the input then we do the same thing and now you can see that basically what is going to happen let me change the color maybe to gray after you keep on multiplying this with this part with this matrix you're gonna end up with a half of the output representation vector but this half is for like is going to be actually contain it's gonna contain all of the information necessary so we don't have to do any summation that's the advantage here okay so we do that on two devices we end up with these representations we apply the gel use and then and then we feed so that so now this is this is the nice part we'll literally we'll literally just take these representations here so we apply jellies again so we apply the jelly and we then feed them as x1 to the this layer where we split the weight matrix horizontally so that's the nice part so now when you concatenate these two you get an MLP and you have literally half of the weights on one device and half of the weights on the second device that's it as simple as that beautiful idea if you ask me very simple but very beautiful and that's drawn here again you can see it here so we start with X we replicate X then we we have because this one is column split we can just do X times a 2 and then we apply jellies and then we just feed that here and then we multiply with b2 and here remember we have horizontal so row wise splitting and then we have the G which is the synchronization operation so that means we're gonna sum them up and finally we end up with the final representation is that here that's it that's how we split the MLP layer okay let me see where there is something interesting here this approach splits both gems in the MLP block across GPUs and requires only a single all reduce operation in the forward pass so the G operator and a single already using the backward pass the F operator so let me show you how that looks like so again we saw that already so in the forward pass the only point where we have to synchronize to do the already use is here the G and when we go backwards once we get the gradients here we somehow have to combine them right to get the representations here so we literally have to well I guess just add them and divide by two whatever we have to have the synchronization primitive there okay now for the attention part and I really hope now you're you're kind of appreciating the whole engineering the infrastructure that goes into these big language models it's not just like you set like a hyper parameter in your code to from from like five layers to like 120 layers of transformer and expect everything is gonna work magically on a single device there is a lot of engineering going on and here trying to explain you some of the methods okay attention is a bit easier I think in my opinion so the thing is you literally take the query matrix and I'm gonna I'm not going to explain the transformers please do check out my transformer video I'm gonna link it somewhere here in case you're not familiar with the details of the transformer model you take the query matrix you take the key matrix you take the value matrix here and basically you split them column wise and each of these columns is gonna literally compute the queries for a single head so we have the multi head attention let's imagine for the sake of argument here that we only have two heads and by the way that brings me let me just quickly mention this so even though we just had two here so we were splitting the so-called two-way model parallelism you can imagine splitting the weight matrices into much more like well parts so you can have like a four-way model of parallelism or eight way and you can kind of convince yourself that you're going to keep the equivalent so if we have eight if we if we divide this into eight rows and if we divide the input data into eight pieces as well you'll just have x1 a1 plus x2 a2 plus x3 a3 plus etc etc until you get to x8 a8 and then you have to synchronize them and then you apply jelly so nothing is preventing us from from splitting this into multiple parts okay so let's get back to the attention so here let's imagine for the sake of explanation we only have two heads and basically what will happen is that this device up here so this device up here is going to compute the first head so the the representation for the first head and the bottom layer the bottom part of this diagram is going to compute the second head and now we have this is the head one representations and these are the head two representations and then we just feed them here and this is the the already familiar hopefully breakdown of the feed forward layer where we do the row by splitting and you can see here that we again have to have the G because we have to synchronize them and then we get the out-propresentations okay let me try and demonstrate this again with a small diagram quickly so we start with the input so let's assume we have that one and sequence size one so we literally have a single token so let's that's gonna be the easiest case but it's gonna make hopefully make the point okay so here is our input data X again it's a single token so B the the batch size is one S which is usually along this dimension is one we just have a single token that has some dimensionality okay so some maybe H the hidden dimension okay so what we do is we basically map it using the key query and values so the first device will literally map this one into let's say like this we have we have the the query we have the key we have the value and you can see that the it's like twice smaller dimensionality compared to the input one and now you're literally gonna do your your magic of the transformer and you end up with the final representation which is again 2x smaller so this is H2 if this is H this is H2 okay and you just do the same thing on the separate device so you'll do the same logic I'm just gonna kind of do it like this and you end up with the same well it's gonna be a different representation but the same structure here so we end up with this H over 2 okay so now we pay pass those tokens we pass them here into the feed forward layer we do the multiplication so now we literally have the the same example with X1 and X2 so we split our data into two parts that's what we've done here and now you literally do the when you multiply would be one you're gonna end up with this so let me let me change the color to like blue so you're gonna end up with this so something like this blah blah blah blah blah so we end up with the full representation again H but if you recall we literally have a partial representation so we'll have to add up on top of this one the the mapping that we get from so the representation that we get from this representation here so now we'll have to add up this thing here and only now do we have the full representation and that's the the G the G operator here so hopefully that the idea is clear until now that was the main idea let me kind of walk you through the text here so partitioning the gems associated with the with key query and value in a column parallel fashion such that the matrix multiply corresponding to each attention head is done locally on one GPU this allows us to split per attention head parameters and workload across the GPUs and doesn't require any immediate communication to complete the self-attention that's very important we don't need any immediate communication okay so this approach for both the MLP and self-attention layer fuses groups of two gems so general matrix multiplies eliminates a synchronization point in between and results in better scaling so what I mean by that is you can see here in both of these diagrams you can see that we have a like a general matrix multiply and then something happens and then general matrix multiply and compilers are smart enough to optimize these to fuse these operations together so that you literally have a single matrix multiply or something I'm not super familiar with those low-level details I've never had to do that but like that's my understanding at the moment so that's what I see here and this results in better scaling this enables us to perform all gems in a simple transformer layer using only two all reduces in the forward path and two in the backward path again that's because we have because the transformer layer consists of one attention we have a single already use here and then we pass the data that comes out of here we pass it here and then we have another already use here the G so that's two in the in the in the single single pass through the transformer layer okay guys that's pretty much it now I did say that they also have to split the output embeddings the input embeddings etc etc I'm just gonna kind of rush you through this the logic is very similar I'm not gonna bother explaining this one we parallelize the input embedding matrix denoted as E and then the subsequent is H with a hidden dimension and V the vocabulary size along the vocab dimension so e1 e2 column wise since since each partition now only contains a portion of the embedding table and already use the G operator is required after the input embedding so again we'll be able to split the e matrix which is usually huge like the vocab can sometimes be like 50,000 etc etc so you can split it into multiple pieces send each of those pieces onto the GPUs then you do the mapping and then you'll just have to synchronize again same idea as what we saw before okay then they also mentioned for the output it's a bit more intricate so blah blah blah if you just do it simply then it's super expensive but then they say so however for this case the all getter will communicate B times s times V elements B is the batch size s is the sequence length and B is the vocab size which is huge due to the vocab size being large to reduce the communication size we fuse the output of the parallel gem with the cross-entropy loss which reduces the dimensions to B times s communicating scalar losses instead of logits is a huge reduction in communication that improves the efficiency of our model parallel approach okay and finally they they mentioned this so those are the almost all the pieces that make the transformer but there is also the residual connections the layer norms all of that so let me just mention that briefly rather than having one GPU compute part of the dropout layer normalization or residual connections and broadcast the results to other GPUs we choose to duplicate the communication across GPUs I encourage you to read at your own pace why this is the case but I think you got the gist of this of this paper already a couple more details before before jumping to the mixed precision training so you can understand and appreciate why we need that method let me just kind of go here so to train our models efficiently we utilize mixed precision training as I said with dynamic loss scaling we're gonna learn what it is and why we need it then you can see a bunch of details like how do we initialize the model how do you scale it you do some norm clipping blah blah blah a lot of engineering going on here lastly to better manage our memory footprint we utilize activation checkpointing after every transformer layer this is a super important idea I'm gonna briefly explain it here because it's used all like all the time all the time so activation checkpointing you'll hear it you'll hear this concept being thrown around all the time so the idea is the following again we have a transformer here okay it's a big model has a lot of layers so now you do a feed forward through that model okay so you do a feed forward through the model and you keep on collecting the activations okay so each of the layers will be storing the activations why well because when you do the back prop in order to calculate the gradients you need to have activations of each of the corresponding layers so you can already probably notice the problem with this approach you're literally storing bunch of data here here here and as you're progressing through the network the memory peak so the memory consumption is rising rising rising so the idea with the activation checkpointing is the following it's very simple and it's a beautiful idea it's covered in one of the other papers I obviously don't have enough time to cover it but if I get enough requests I might cover it in the future merge with some more advanced techniques that came after all these papers I am explaining in this video so did is fairly simple don't store activations for every single layer just checkpoint them so basically let's imagine we have four let's imagine we have four layers okay and instead of storing the activations for all of the feed forward layers inside of the block you just stored the activations at the at the at the end of the block so you literally just store the the ultimate activations okay so like suggest the activations that are at the end of the transformer block so just store these ones okay so you just store these ones and now when you're doing the back prop so now let's see what happens when you do the back prop okay so you start doing the let me change the color into maybe gray so we start doing the back prop here we are good we are good we're good now we hit this part where we don't have the activations and what we do is we just recompute them so now we trigger the recomputation from this checkpoint here okay so we trigger the recomputation from here and then we can continue our back prop and then we hit this part here no activations we hit the we hit the recomputation again and we end up with activations obviously this is gonna happen in a bit more clever way so while they're doing the back prop we are computing the we're triggering the recomputation so that we don't have to wait so all of those optimization details but this is the gist of it this is how the activation checkpointing works okay hopefully you can appreciate that now let me quickly walk you through a couple more details and we are done with this paper so our infrastructure is optimized for multi-node deep learning applications with three hundred gigabytes per second bandwidth between GPUs inside a server so that's important by this NS and VS switch and hundred gigabytes of interconnect bandwidth between servers okay that's important so we have we have 300 gigabytes per second between the GPUs and we have only hundred only hundred a smaller number that's that's the whole point between the servers so why am I mentioning this the reason being is you want when you're doing the metal the model of parallelism because you have all of those already use operations so the synchronization points you want to have those synchronization points happening inside of a server where the bandwidth is super high okay so that's why when you have a cluster where your node only has eight GPUs you don't want to go more than eight way model parallelism you don't want to go 16 because if you have 16 then you have two servers communicating that during the synchronization points and that sucks okay so hopefully that that makes sense now let me show you some some results they achieve you can see here that scaling to from one to two to four to eight GPUs you can see that the the efficiency kind of drops but not significantly so here is the 77% for the eight way model parallelism and if you have if you combine it with the data parallel approach then you have still 74% for 512 GPUs that's huge that's huge I'm going to quickly explain how we combine the model parallelism with the data parallelism it's a very very nice visualization so quickly this is how it's going to look like as I said you're gonna because the servers let's say have eight GPUs basically you're gonna do the following thing you're gonna have something like this and then you're gonna slice this server into eight pieces because all the pieces are a single GPU and they are very densely connected and so you literally do you have one eighth of the of the model weights here one eighth of the model weights here etc etc and now remember the data parallelism you just replicate you copy paste these multiple times so precisely they use the 64 way data parallelism so you literally you're going to replicate this so that you have so that you end up with let me draw it like this so you have eight here whoops so yeah eight here and then you have eight here and you end up with 64 times eight and that's 512 and this is gonna be your setup so you have literally 64 servers each of the servers has eight GPUs with high bandwidth internally and a bit lower bandwidth externally so internode versus intranode and yeah then you basically do the logic I just explained so in this particular example you have a huge batch you're gonna split it into 64 chunks and then you're gonna pass each of the chunks to one of the servers and then you do the logic we just saw from the Megatron LM paper with all of the synchronization points etc etc and then calculate the gradients and then you're gonna do the all reviews and update the whole cluster that's it fairly simple okay at least conceptually one thing I want to the last thing I want to mention from this paper is it came before GPT-3 was published which is probably the most famous language model and the ML model in general and they already noticed the trend everybody knew already that scaling is leading to better and better performance so here by performance I mean lower perplexity for example so you can see that 8.3 billion model has lower perplexity than the 2.5 billion etc etc they mentioned that multiple times as the model size increases the validation perplexity decreases and reaches a validation perplexity of blah blah blah we observed the trend of decreasing model size also leads to lower perplexity blah blah blah recently researchers from Microsoft in collaboration with Nvidia trained a 17 billion parameter GPT-2 called Turing NLG using Megatron and showed that the accuracy is further improved as they scale the model highlighting the value of larger models it was super obvious that somebody's gonna do it it just happened that open AI was the first one to do it and yeah kind of worth mentioning a historical note I guess yeah they also have contribution of this paper is they show that you kind of have to reconfigure where you put your layer norms so they can so that you can train much bigger birds otherwise you have these instabilities you can see how the the model loss explodes if you if you just use this default bird architecture but that's not important for our like well it is you kind of have to be aware that everything matters that even the model architecture is is very sensitive if you change rearrange something all of a sudden you cannot train the bigger models and so on and so forth okay guys now on to the mixed precision training I'm gonna quickly walk you through this one and then we're gonna dig into the zero paper one important quick note before before this is all the three approaches I showed you the pipeline parallelism the data parallelism and the model or the tensor parallelism all of them can be combined together they are complementary and I think it was branded as 3d parallelism so when you combine all of these three you can achieve like huge scales and then when you combine it with mixed precision plus the zero optimizer which you're gonna see in a couple of minutes then you can achieve truly astounding scales okay that's that's it now let's dig into this paper so a couple of contributions from this paper so let's gonna read it firstly we recommend maintaining a single precision copy of weights that accumulates the gradients after each optimizer step this copy is rounded to have precision for the forward and back prop so this is the first part so you're gonna literally have three components that I want to explain and then we're done the second part is we propose loss scaling we don't see what that concept means to preserve gradient values with small small magnitudes thirdly we use half precision arithmetic that accumulates into single precision outputs that's FP 32 versus FP 16 which are converted to have precision before storing to memory okay so today is fairly simple usually when you train neural networks historically people use the FP 32 format that means a single numbers like literally needs 32 bits instead of that people have shown that you didn't you do not need 32 bits you can have lower accuracy for a neural network training so like 16 works and that's it by just having the the amount of amount of data amount of memory that that a single number takes you literally can save a bunch of a bunch of memory and still they show in this paper keep the same accuracy the same performance of the model as the baseline which uses the FP 32 that's very cool okay so why does it matter not only do we store the memory it's also faster on the modern hardware so have precision math throughput in recent GPUs is 2x to 8x higher than single precision in addition to speed improvements reduce precision formats also reduce the amount of memory required for training that's obvious I already said that one okay okay let's see what's going on so specifically we train various neural networks using the IEEE half precision format let me quickly show you I'll link these somewhere below in the video but this is the this is the the format they're they're referring to you so you have the exponent has five bits the fraction has ten bits and then there is a sine bit and this is how you represent numbers like the flow 32 is basically obviously has a bigger exponent and bigger fraction and well while I'm here explaining the formats let me quickly mention that the B float 16 or the brain float because it was originally invented into Google brain AI lab and you can see what's the difference the B float has much bigger exponent and the main insight here is for neural network training you care about the range you don't care about the actual accuracy and this significant or mantissa is what determines the how fine-tuned your numbers are whereas the exponent determines how big can you go so obviously with B float you can have much bigger and much smaller numbers without getting the overflow the infinities or the or the NANDs or whatnot and so basically TPUs natively support this this B float format and like people have shown like when you take a look at the problems that for example Boris Dyma training the Lee mini had compared to training on GPUs TPUs basically are much more robust to these divergences of losses because of the B float format so that's my so that's basically my understanding I might be wrong here but that's I wanted to quickly mention the all of these formats now let's go back to the paper okay so now that you know that let's see what's going on so we have to store the master weights they showed for most models this is kind of necessary so they say here the need for FP32 master weights is not universal but for the most models they trained they did need to do this so here is the idea you store the weights as FP32 but then when you do the forward prop the back prop you do everything in half precision so you literally do this float to half operation so you literally kind of convert the FP32 to FP16 then you do the feed forward you can see here you start with the activations which are also FP16 so you can kind of always make sure that's the case because you control your data so let's say the input the input is your image you can literally feed image as FP16 image and then combine that with FP16 weights do the forward pass you get the activations and then in the backward you do the same thing the weights are FP16 you calculate the activation grads from the activation grads you get the weight grads and all of those are in FP16 and because of that you benefit from like a smaller memory like a blueprint as well as the well higher throughput as we saw that some modern GPUs basically have higher throughput for FP16 compared to FP32 that's that's the bottom line okay so this this is a schematic how everything works so they now mention one explanation why they need the FP32 master weights one explanation is that updates weight gradients multiplied by the learning rate become too small to be represented in FP16 any value whose magnitude is smaller than 2 raised to the power of minus 24 becomes 0 in FP 16 and by the way you might be confused why 24 and not 14 I'm not gonna dig into that but just Google for the sub normalized so sub normal or something like that sub normal values just open up the the the float 16 and float 32 Viki pages I'm gonna link those down below and you'll see what those are so we can see in figure 2b that approximately 5% of weight gradient values have exponents smaller than minus 24 okay let me show you what I mean by that so let's go to figure 2b you can see here this is the training when the model is trained in in FP32 and you can see that most of the gradients here fall below this magic red line the the minus 24 which is the where the representation range of the FP 16 ends and so that means that all of these are gonna be rounded to 0 and you lose a lot of information during training and because of that the accuracy the final accuracy for model may or may not drop okay so that's one kind of idea why why you need FP 32 there is a couple of other explanations here a bit more complex to explain here I'm gonna I'm gonna kind of skip it for now so worth mentioning now that you have to store this additional copy doesn't that blow up the memory and here to explain that's not the case because the activations are what matters so even though maintaining an additional copy of weights increases the memory requirements for the weights by 50% compared with single precision training impact on overall memory usage is much smaller for training memory consumption is dominated by activations due to larger batch sizes and activations of each layer being saved for reuse in the back prop pass since activations are also stored in half precision format the overall memory consumption for training deep neural networks is roughly halved okay they are not obviously taking into account activation checkpointing they just assume that you have to store all of the activations in that case this thing that they've just said here holds but yeah okay now let's go into the law scaling this is a second important concept you need to get the mixed precision training to work know that much of the fp16 representable range was left unused while many values were below the minimum representable range and became zeros scaling up the gradients will shift them to occupy more of the representable range and preserve values that are otherwise lost to zeros this particular network diverges when gradients are not scaled but scaling them by a factor of 8 increasing the exponents by 3 is sufficient to make the accuracy achieved with fp32 training this suggests that activation gradient below mine to raise to the power of minus 27 in magnitude were irrelevant to the training of this model but values in this range were important to preserve okay let me break this down for you okay so here is a histogram of basically activation gradient values you can see what I said it's shifted outside of the range that fp16 basically support so this is the range that fp16 loves okay this is this subnormal or the denormalized range so from minus 24 to minus 14 you can see that most of these values are outside of the range and if you multiply the gradients by 8 you literally shift all of these rightwards or equivalently we move the red line here and now you have all of these being representable in fp16 thus we will not round them to zero we will not lose this information and it turned out they say here blah blah blah that that this particular range here and you can see there is a lot of data here was very important to preserve and so scaling by 8 for this particular model it doesn't even matter what the model is just the concept is important it kind of saved the training okay so one efficient way to shift the gradient values into fp16 representable range is to scale the last value computed in the forward pass prior to starting back propagation by chain rule back propagation ensures that all gradient values are scaled by the same amount so that's a easy way how you can scale up all these gradients you just take the final loss you scale it up by let's say 8 and by doing that once you start doing the back prop because how chain rule works you literally multiply all of the gradients by 8 and thus you'll be shifting this the gradients into the representable range this is the yeah this is the main the main intuition I want you to take out of this paper and here is the here's the third technique they do to maintain model accuracy we found that some networks require that the fp16 vector dot product accumulates the partial products into an fp32 value which is converted to fp16 before writing to memory without this accumulation into fp32 some fp16 models did not match the accuracy of the baseline models okay also for large reductions for example sums across elements of a vector they should all be carried out in fp32 such reductions mostly come up in batch normalization layers when accumulating statistics and softmax layers as well so both of the layer types in our implementation still read and write fp16 tensors from memory performing the arithmetic in fp32 this did not slow down the training process since these layers are memory bandwidth limited and not sensitive to arithmetic speed okay again just a additional detail that needs to be taken care of basically there are three categories so they say here this is this cannot relevant by and by and large neural network arithmetic falls into three categories so the vector dot products the reductions and the point wise operations such as the activation functions and that they just mentioned for these two so they mentioned that for vector dot products and for reductions you sometimes have to convert to fp32 and do the operations there such that you can preserve the accuracy that's it guys that's the idea behind mixed precision training now let me show you the results they show that literally baseline they compare they are almost always even better than the baselines so here you can if you just check out these numbers at your own base you'll see that the numbers are always a bit better so it maybe works as a like a regularizer almost to to to do the training in fp16 for some models as I mentioned the last skill and technique was not required for successful mix precision training of these networks but for some others it was very important let me show you one so here you can see that mixed precision training when the last scale is one diverges when the last scale is 128 it does not diverge and they literally show for every four different classes of models like this again here language models machine translation they always match the baseline when it comes to accuracy so that's awesome and the final idea least last but not least definitely not least because this is such an exciting paper this paper is basically introduces deep speed library you might have heard of it if you've seen if you've went through any of the popular LLMs most of them use deep speed illutri I know they use deep speed bloom use deep speed opt 175 be that the meta folks I'm not sure whether they used it but yeah so it's a literally a paper from Microsoft it's called zero memory optimizations toward training trillion parameter models okay I love this paper let's go through it step by step let's think step by step zero redundancy optimizer that's that's what the acronym stands for we'll see why zero redundancy they literally have well they reduce all of the redundancy when it comes to model states and residual states we're gonna see what those are but that's the idea zero eliminates memory redundancies in data and model parallel training zero has the potential to scale beyond one to them parameters using today's hardware zero can train large models of up to 13 billion parameters for example larger than Megatron GPT 8.3 B so that's a model that was trained in the Megatron LM paper the first paper I covered and the t5 the Google's model that was 11 billion parameter big without even requiring model parallelism so that's a big breakthrough because now with with this set of optimizations you as a practitioner can literally train models as big as 13 B if you have just a single but very good albeit a very good like GPU but that's amazing and obviously when you use the actual optimizations you can you can scale to much bigger sizes they show they have shown indeed that you can scale up to 1 trillion parameters with a small caveat and that's that it becomes very slow I mean very slow you literally for some of the models they've shown you need a year to train it so that's practically not not really practical but yeah they train a model of hundred something billion so that's that's kind of cool okay so let's let's dig into it MP so the model parallelism splits the model vertically partitioning the computation and parameters in each layer across multiple devices requiring significant communication between each layer okay as a result they work well within a single node I did mention this where the inter GPU communication bandwidth is high but the efficiency degree degrades quickly beyond a single node we tested a 40 B parameter model using Megatron LM across 2d GX 2 nodes and observe about 5 tera flops per V 100 GPU less than 5% of hardware peak so you can see it drops dramatically as soon as you start doing the model parallelism across multiple nodes okay then they say this for large models the majority of the memory is occupied by model states this is an important piece of terminology they're gonna be reusing it throughout the paper so let's kind of stress that so the model states which include the optimizer states such as momentum and variances in atom gradients and parameters so they consider those to be the model states the remaining memory is consumed by activation temporary buffers and unusable whoops some glitch unusable fragmented memory which we refer to collectively as residual states so again two pieces of terminology new terminology here we have residual states and we have model states and they're gonna address each one of those separately okay so as your DPU removes the memory state redundancies across data parallel processes by partitioning the model states instead of replicating them so that's the main idea that's literally the main idea of this paper they are partitioning instead of replicating I mean simple conceptually but there is a lot of work that goes into making this to work efficiently that's important so I'm gonna show you the diagram I'm not gonna explain it yet what this means I'm gonna see in a couple of seconds so they have a couple of optimizations optimizer state partitioning so when you partition the optimizer states instead of replicating it then you can do the gradient partitioning and then you can also do the parameter partition and they show that the memory reduction is linear with DP degree so that means the number of devices you use to replicate your models across when you do the data parallelism for example splitting across 64 GPUs will yield a 64 X memory reduction is huge there is a modest 50% increase in communication volume though okay with all three stages enabled zero can train a Trillium parameter model on just 1024 Nvidia GPUs which is crazy okay this is the main diagram let me first walk you through that this piece of text here and then I'm gonna basically yeah jump into explaining the dire for activations stored from forward pass in order to perform regular pass we noticed checkpointing helps I did explain activation checkpointing maybe half an hour ago but not sufficient for large models thus zero are optimizes activation memory by identifying and removing the activation replication in existing empty approaches through activation partition again partitioning partitioning it also offloads activations to CPU when appropriate okay that's another detail that can be done okay zero DP and zero are combined together forms a powerful system of memory optimizations for DL training that we collectively refer to as zero again just another piece of terminology if you ever start playing with deep speed this might be useful to know okay so let's see what's going on here is how the that the opera the baseline is how your classical change the color here this is how your data parallelism works you have three types of basically what you have parameters radiance and optimizer states you can see that optimizer states take that the biggest part of the memory and you can see across when you do the data parallelism I did mention that we literally replicate all of these you copy paste them and that's super inefficient okay you can see that the memory consumed is 2 plus 2 plus K times psi I'm going to explain what these are in a second and for I think they're using some GPT model here whatnot they show that you need 120 gigabytes of memory per device that's huge so psi is the number of parameters I think it's 1.5 billion in this particular example they're using the the GPT model and the GPT-2 2 is because you use two bytes when you use mixed precision training you use two bytes for weights you use two bytes for gradients so that's why you have 2 plus 2 and then K is equal to 12 because these are the optimizer states because we'll see the calculation a bit later but for Adam K equals 12 and so you have 16 times 1.5 B you end up with with I guess this number is that correct just see what I messed something up at 1.5 times 16 wait I'm not sure yeah we'll see why yeah well sorry it's 7.5 B it says here okay that's why we get 100 now the first optimization is instead of having a copy of the optimizer states everywhere you instead partition it so you store one piece of the optimizer states on one GPU you store the second part here you store the the the the nth part here and you can see so sorry we have NGP's here so that's why we are gonna have ultimately when you cannot combine all of them together you'll have still the same the same information as here right and then you can do this same thing for everything else for gradients you can partition them for parameters you can partition them and you can see how the memory is slowly decreasing so here we have 2 psi plus 2 psi because we are not doing the the parameter and gradient optimization but we do save up on the optimizer state and that's why we have K times psi divided by ND where ND is the number of devices in your data parallel setup it's gonna be 64 in in their particular case and you can see how the memory dramatically reduces so you started with 120 gigabytes and you ultimately end up with 1.9 gigabytes which is crazy I mean this is an amazing optimization okay let's continue here this is what I mentioned before the complete set of optimizations in Xero could allow us to run models with trillion parameters on the high-end hardware cluster today for example with 1000 V100 GPUs however the hardware compute capacity is still too limited and training time can be in practically long over a year so always take with a grain of salt when somebody says one trillion you have to kind of read the fine print and understand the details of the method zero power is the largest language model with 17 D parameters and record-breaking accuracy cheering energy important to mention this was literally I think a month before GPT-3 was published which was 10x bigger than this model which is like a huge we share zero as a part of our open source DL training optimization library called deep speed so I did mention this library multiple times you can check it out especially if you want to play with the models by the way I'm making and I'm going through these papers precisely because over the next videos I'll be covering some of the largest ML models out there and do let me know if you if you if you have any any suggestions what you want to cover or not feel free to comment down in the comments also congrats if you if you sticked up through the whole video until here okay so let's do a quick memory analysis to understand why case 12 etc etc so a 1.5 billion parameter GPT-2 model requires 3 gigabytes of memory for its weights if we assume FB 16 or in 16-bit precision yet it cannot be trained on a single GPU with 32 gigabytes of memory using tensorflow or PyTorch at least without any of the optimizations I assume that PyTorch and tensorflow now have a bunch of optimizations included one may wonder where all the memory goes so now let's see where it goes let's take Adam as a concrete example so the mixed precision training of a model with psi parameters using Adam requires enough memory to hold an FB 16 copy of the parameters and gradients with memory requirements of two psi and two psi bytes respectively so we've seen that one in the table before basically yeah so in addition it needs to hold the optimizer states and FB 32 copy of the parameters so we know why this is the case because of the mixed precision training we always need to store the master copy in FB 32 okay and that's kind of gonna bloat up the memory but it's necessary the momentum and the variance so these are Adam idiosyncrasies and then with memory requirements of 4 psi 4 psi and 4 psi bytes so 4 because here we have FB 32 okay and because of that if you sum them up you end up with 12 and that's why K equals 12 so in total this results in 16 psi bytes of memory requirements so yeah that's why you cannot train a GPT 1.5b on the GPU unless you have some of these optimizations okay now they break up the residual memory consumption blah blah blah they mention for some set of parameters like if you're using GPT 2 with a sequence length of 1000 that size of 32 you require 60 gigabytes of memory unless you do activation checkpoint then they say here activation checkpoint here or activation computation or computation is a common approach to reduce the creation memory by approximately the square root of the total activations at the expense of 33% of recomputation overhead this would reduce the activation memory consumption of this model to about 8 gigabytes that's a huge saving okay when the size of the model is large these temporary buffer sizes are non-trivial okay so they go on to say that there are three components to the residual number consumption so the activations the temporary buffers and the memory fragmentation yeah I'm gonna skip this one it's not that vital but yeah you you kind of have to have these temporary buffers when you want to do some reductions etc etc yeah that's too much detail for this video okay let's get some hints why this works zero DP partitions the model states instead of replicating them and uses a dynamic communication schedule that exploits the intrinsically temporal nature of the model states while minimizing the communication volume okay so let that sink in the intrinsically temporal nature so let me quickly go back to the diagram here I think that's gonna be the easiest thing to do so the thing is let's say let's take parameters as an example so the the blue the blue line here literally when you're doing a forward pass obviously the forward pass is always at a certain point at a certain layer in your model and the only parameters you care about are the parameters that are needed to do the immediately next computation right so you can kind of discard everything before and you don't care about everything that's about to come so they literally explore that temporal thingy and they in a smart way communicate the parameters the gradients the optimizer states such that you have basically an illusion that everything is already present there and they kind of mask the the overhead by the well because of certain intrinsic latencies during the model training I'm not gonna go into much more detail there so let's go back here let's go back here we are here I think okay a couple more things and I think we're done here the arithmetic dense intensity ratio of the amount of computation per iteration to amount of activation checkpoints per iteration is very large and increases linearly with hidden dimension making it possible to hide so this is the thing I just said the data movement cost for the activation checkpoints even when the bandwidth is low for very large model zero can even choose to move the activation partitions to the CPU memory and then you literally have like a zero cost of memory older and and in case you manage to mask that transitioning the movement of the data to the CPU then you literally like you get it for free so unfortunately the paper is very textual hopefully you got some insight from the diagrams and the thing I things I explained so far let me try and read you a couple more things I highlighted with which I think are insightful so for a DP degree again DP is standing for a data parallelism of ND we group the optimizer states into ND equal partitions okay such that the I data parallel process only updates the optimizer states corresponding to the I partition thus each data parallel process only needs to store and update one over ND of the total optimizer states and then only update one over ND of the parameters we perform an all gatherer across the data parallel process at the end of each step to get the fully updated parameters across all data parallel process okay so it's kind of hard to explain everything I'm going to show you one thing here so there's this nice Vicky page where you can see these collective operations such as reduce or reduce blah blah blah all gather scatter all to all all these are implemented in libraries such as NCCL from from Nvidia so that stands for Nvidia collective communications library so you can kind of check it out the diagrams here are useful it's gonna help you kind of understand this but ultimately someone needs to create maybe to me something that visualizes this much like nicely such it so they can see understand what's going on but yeah hopefully this gives you some intuition and that's what we are aiming for just like have at least the basic terminology and the basic understanding of all of these optimization methods in place and then you can always read on the documentation of the particular library you're using if you care to learn more okay I'm gonna skip all of these here interesting piece of theory but not that practical they show that because of we saw these formulas already in the table but we ultimately end up with 16 psi over MD formula for the amount of memory which means is Andy in the limit of going to infinity this goes to zero but yeah it becomes impractical body this shows that this has a profound implication zero powers DP to fit model with arbitrary size as long as there are sufficient number of devices to share the model states but yeah it becomes after a certain point you hear the situation point and this becomes very very slow okay this is the last last part let me kind of read and walk you through this it might be insightful so more specifically once the forward propagation for a layer of a model is computed the input activations are partitioned across all the model parallel process until it is needed again during the back propagation okay so what they say here is let me try and cannot draw this down so let's say we have a bunch of devices here something like this let's let's say we have three devices and let's say you just computed you're just you're currently here in this part of the network and you computed some of the activations okay and then you proceed forward so after that you proceed forward you continue your forward prop here okay so instead of having that checkpoint so what's usually done that checkpoint is stored here instead of doing that why not partition it and basically why not store the literally change the color so you instead store only a chunk here so you only store a chunk here you store the second time here and you store the third time here so that's the idea and then once we need these activations again we just communicate back those activations back to this device so that we can compute the gradients etc etc so that's the basic idea I mean now how that happens obviously there is a lot of computer science going on in the background the algorithms data structures all of that hardware electronics but yeah we have to kind of stop at a certain level of abstraction okay so it works in conjunction with activation checkpointing storing partitioned activation checkpoints only instead of replicated copy so that's what I told you here so furthermore in the case of very large models and very limited device memory these partitioned activation checkpoints can also be offloaded to the CPU reducing the activation memory overhead to nearly zero at an additional communication cost I didn't mention this before okay if we check point a single activation for each transformer layer it will require about 33 gigabytes of memory per GPU just to store the activation checkpoints but with this optimization the PA and zero it can be reduced to about 2 gigabytes per GPU because yeah if you have 16 degree of 16 of your model parallelism then you just divide by 16 you get roughly 2 gigabytes furthermore this 2 gigabytes can be offloaded to the CPU reducing the memory footprint for activations to nearly zero okay guys this is this is pretty much it the paper is huge obviously there is a lot of heuristics in the background so they mentioned something here so given model and hardware characteristics will leverage the above analysis to decide if and when to apply PA and PA plus CPU so the activation checkpointing the partitioning of the activation checkpoints optimization and the CPU offloading optimization and the above analysis is something I encourage you to read but the idea is depending on like various latencies they decide whether this is a good idea or not so there is a lot of details there but nothing conceptually fundamental that that yeah it will lead you to an Eureka moment to epiphany I guess yeah okay guys hopefully you like this video let me find let me end up with this nice chart here okay so hopefully you like this video if you did please share it out with with your friends with anyone for whom you think would benefit from understanding these scaling techniques so we covered a lot of ground here we covered the data parallelism approach we covered the tensor parallelism approach we covered the pipeline parallelism approach we covered the mixed precision training and finally we covered the zero optimizer where we saw how we can basically cut down and do the partitioning of various model states and digital states so that we can train much bigger models and even on a single GPU so yeah do let me know whether you found this video interesting leave any feedback down below subscribe to the channel and until next time bye bye
[{"start": 0.0, "end": 4.92, "text": " What's cracking guys? The idea of this video is going to be extremely ambitious."}, {"start": 4.92, "end": 11.36, "text": " I'm going to try and give you the most seminal and important ideas for how to"}, {"start": 11.36, "end": 15.52, "text": " scale your models up to scales of hundreds of billions and even trillions"}, {"start": 15.52, "end": 20.48, "text": " of parameters. So I'm gonna cover multiple papers as you can see here so"}, {"start": 20.48, "end": 26.64, "text": " we have Megatron LM from Nvidia here introducing the tensor or model"}, {"start": 26.64, "end": 30.48, "text": " parallelism. I'm gonna walk you through quickly through pipeline parallelism idea"}, {"start": 30.48, "end": 34.24, "text": " from the GPy paper. I'm gonna walk you through the data parallelism idea which"}, {"start": 34.24, "end": 38.52, "text": " is probably something most of you are already familiar with but just to round"}, {"start": 38.52, "end": 42.78, "text": " them up I'm gonna cover that one as well. I'm gonna quickly go through the mixed"}, {"start": 42.78, "end": 47.24, "text": " precision training paper as well and finally I'm gonna go through the zero"}, {"start": 47.24, "end": 52.56, "text": " optimizer so the zero redundancy optimizer from Microsoft that enabled"}, {"start": 52.56, "end": 58.68, "text": " training models up to 1 trillion parameters big. So why should we why"}, {"start": 58.68, "end": 62.68000000000001, "text": " should you care about this video? Well the thing is the following so if you"}, {"start": 62.68000000000001, "end": 67.92, "text": " ever want to train a model that's bigger than let's say well if you're dealing"}, {"start": 67.92, "end": 73.68, "text": " with say GPT style of a model transformer if you want to go like above"}, {"start": 73.68, "end": 79.52000000000001, "text": " 1.5 billion parameters you literally need more than 32 gigabytes of VRAM which"}, {"start": 79.52, "end": 86.6, "text": " most of you do not have. So up until basically 1.5 billion a single GPU with"}, {"start": 86.6, "end": 91.67999999999999, "text": " 32 gigabytes of VRAM can handle that training if you don't have any of these"}, {"start": 91.67999999999999, "end": 96.24, "text": " optimizations. Other than that it's impossible to train bigger models so"}, {"start": 96.24, "end": 100.84, "text": " combining all of these ideas you'll be able to literally train super big models"}, {"start": 100.84, "end": 106.0, "text": " some of them you must have seen recently published models such as the bloom from"}, {"start": 106.0, "end": 115.0, "text": " the big science organization and also opt 175 B model from from meta so all"}, {"start": 115.0, "end": 119.52, "text": " of those models are like hundreds and hundreds of billions of parameters other"}, {"start": 119.52, "end": 125.4, "text": " examples are like chinchilla from deep mind go for poem so yeah guys before we"}, {"start": 125.4, "end": 128.72, "text": " go there I want to make a quick shout out to assembly AI and thank them for"}, {"start": 128.72, "end": 132.6, "text": " sponsoring this video basically they offer a super powerful speech"}, {"start": 132.6, "end": 137.92, "text": " understanding and transcription API you can see how easy it is to get started in"}, {"start": 137.92, "end": 141.16, "text": " Python so literally a couple of lines of code and voila you get your"}, {"start": 141.16, "end": 145.16, "text": " transcription back as well as in this case sentiment analysis they offer"}, {"start": 145.16, "end": 150.68, "text": " various other understanding features such as entity detection etc etc you can"}, {"start": 150.68, "end": 153.79999999999998, "text": " also check out the no code solution although I guess this is not that"}, {"start": 153.79999999999998, "end": 157.0, "text": " interesting for most of you but if you just want to quickly understand the"}, {"start": 157.0, "end": 161.28, "text": " features they support you can check out this page as well if you are struggling"}, {"start": 161.28, "end": 165.2, "text": " to get started which I doubt because it's super simple there is even a"}, {"start": 165.2, "end": 170.12, "text": " tutorial very detailed tutorial on how to get started and if you like ideas for"}, {"start": 170.12, "end": 174.04, "text": " projects they have a YouTube channel with cool videos such as this one which"}, {"start": 174.04, "end": 178.44, "text": " is like literally 30 minutes long explaining how to summarize various"}, {"start": 178.44, "end": 184.88, "text": " podcasts with with their API so that's very cool finally they have a very cool"}, {"start": 184.88, "end": 189.76, "text": " set of blocks in general and tutorials on how to get started with various other"}, {"start": 189.76, "end": 193.88, "text": " models such as the Lee so yeah I think that's that's fairly cool so give them a"}, {"start": 193.88, "end": 197.64, "text": " go support the channel I'm linking the free API token down in the video"}, {"start": 197.64, "end": 200.72, "text": " description so you can use it and get started in a couple of seconds"}, {"start": 200.72, "end": 204.84, "text": " literally you get for the toy projects you have no problems with the free"}, {"start": 204.84, "end": 208.04, "text": " account you get three hundred hours of transcription if you want to upgrade you"}, {"start": 208.04, "end": 212.12, "text": " can always do that and having said that let's go back to the video so first"}, {"start": 212.12, "end": 215.72, "text": " let's start actually with data parallelism because that's probably the"}, {"start": 215.72, "end": 220.32, "text": " simplest idea to explain the idea is the following so you have a neural network"}, {"start": 220.32, "end": 225.2, "text": " and what you do is you literally take the weights of your neural network and"}, {"start": 225.2, "end": 232.12, "text": " you just replicate it so you copy paste those weights across multiple devices so"}, {"start": 232.12, "end": 237.72, "text": " you have like let's say and GPUs and you literally copy paste that same network"}, {"start": 237.72, "end": 241.76, "text": " across all of the devices and then what you do is you take a batch of data so"}, {"start": 241.76, "end": 245.92, "text": " you basically take a batch of data and so it's gonna look something like this"}, {"start": 245.92, "end": 252.0, "text": " so literally you have a batch of data here and what do you do is you split it"}, {"start": 252.0, "end": 256.08, "text": " into multiple parts so let's assume here we just have three devices so that means"}, {"start": 256.08, "end": 259.96, "text": " you're gonna split the batch so this is the batch dimension by the way this here"}, {"start": 259.96, "end": 264.56, "text": " I'm gonna simplify it like just gonna represent the input tensor as a 2d"}, {"start": 264.56, "end": 270.08, "text": " object even though it's usually gonna be depending on your data 3d 40 whatnot so"}, {"start": 270.08, "end": 275.08, "text": " yeah so you grab the first chunk the first part and you feed that into the"}, {"start": 275.08, "end": 280.96, "text": " first device here then you grab the second chunk here you grab the second"}, {"start": 280.96, "end": 285.36, "text": " chunk and you pass that second chunk of data into the second GPU GPU 2 and"}, {"start": 285.36, "end": 292.24, "text": " finally you grab the final piece here and you pass that one here then you do a"}, {"start": 292.24, "end": 295.56, "text": " forward pass through your networks you're gonna do a forward pass here for"}, {"start": 295.56, "end": 301.56, "text": " a pass here for a pass here and then you calculate the gradients so now now what"}, {"start": 301.56, "end": 305.48, "text": " happens is that all of these devices have different gradients as a consequence"}, {"start": 305.48, "end": 310.32, "text": " of you feeding different parts of your input batch and now we just need to"}, {"start": 310.32, "end": 316.2, "text": " basically reduce all of those gradients and when I say reduce I literally mean"}, {"start": 316.2, "end": 321.08, "text": " you you sum them up and then divide by the number of devices to get the average"}, {"start": 321.08, "end": 326.35999999999996, "text": " gradient so there are multiple ways how you can implement this in this particular"}, {"start": 326.35999999999996, "end": 331.35999999999996, "text": " image there is this thing called parameter server so you do the basically"}, {"start": 331.35999999999996, "end": 335.84, "text": " reduce and scatter operation so you grab all of the the gradients you're gonna"}, {"start": 335.84, "end": 340.64, "text": " send the gradients here so you're gonna send and by the way I'm sorry I'm kind"}, {"start": 340.64, "end": 344.76, "text": " of glitching with this like highlighter here but hopefully you can you can see"}, {"start": 344.76, "end": 348.68, "text": " what I'm doing so we send the gradients here we send the gradients here we send"}, {"start": 348.68, "end": 353.56, "text": " the gradients here we do the averaging of those gradients and then you literally"}, {"start": 353.56, "end": 357.64, "text": " so in this particular implementation the weights are updated directly on the"}, {"start": 357.64, "end": 361.40000000000003, "text": " parameter server and then you do the scatter operation you literally send"}, {"start": 361.40000000000003, "end": 368.36, "text": " back that new the new weights back onto onto your devices you send back the the"}, {"start": 368.36, "end": 374.48, "text": " weights okay and that's how you achieve training with a much higher throughput"}, {"start": 374.48, "end": 379.20000000000005, "text": " because you can imagine here although there is some overhead obviously you"}, {"start": 379.20000000000005, "end": 384.20000000000005, "text": " literally have 3x higher throughput compared to if you were to train on a"}, {"start": 384.20000000000005, "end": 390.88, "text": " single GPU okay and when I say GPU it can be a TPU it can be arbitrary device"}, {"start": 390.88, "end": 398.12, "text": " basically so what's the problem with this idea the problem is is it's the"}, {"start": 398.12, "end": 403.86, "text": " underlines presupposition here is that all of these networks can fit into a"}, {"start": 403.86, "end": 408.2, "text": " single device which is the problem because we know that our models are much"}, {"start": 408.2, "end": 412.64, "text": " bigger and usually cannot fit into a single device so what do we do then well"}, {"start": 412.64, "end": 416.96000000000004, "text": " then we do like things such as model parallelism so I'm gonna show you the"}, {"start": 416.96000000000004, "end": 419.68, "text": " first thing so that's the data parallelism I'm going to show you the"}, {"start": 419.68, "end": 425.12, "text": " the model parallelism idea so this is a similar paper from Nvidia called"}, {"start": 425.12, "end": 429.52000000000004, "text": " Megatron LM training multi-billion parameter language models using model"}, {"start": 429.52, "end": 436.35999999999996, "text": " parallelism it's literally the idea behind both bloom both opt 175 B from"}, {"start": 436.35999999999996, "end": 441.82, "text": " meta I think some of the illu theory eyes models were also backed up by"}, {"start": 441.82, "end": 446.14, "text": " Megatron at least implicitly because I think they're using deep speed and GPT"}, {"start": 446.14, "end": 451.35999999999996, "text": " new X code base and deep speed basically leverages Megatron in the background I"}, {"start": 451.35999999999996, "end": 455.28, "text": " could be wrong here I'm fairly sure this is this is correct anyway so let's dig"}, {"start": 455.28, "end": 461.52, "text": " into the paper so they say here this is called intro layer model parallel"}, {"start": 461.52, "end": 466.03999999999996, "text": " approach I want to contrast that quickly and I'm going to explain this in a bit"}, {"start": 466.03999999999996, "end": 470.35999999999996, "text": " more depth a bit later with the pipeline approach which is the inter layer"}, {"start": 470.35999999999996, "end": 474.84, "text": " splitting so the difference is the following so imagine you have a"}, {"start": 474.84, "end": 479.08, "text": " transformer and I'm gonna be using transformer as the running example so"}, {"start": 479.08, "end": 484.08, "text": " imagine you have a transformer here the main difference between these two"}, {"start": 484.08, "end": 488.08, "text": " approaches is the following so you have n layers here so you have n blocks of"}, {"start": 488.08, "end": 493.28, "text": " your transformer and the pipeline approach does the following thing you"}, {"start": 493.28, "end": 497.4, "text": " literally grab let's say we have four devices if we have four devices then"}, {"start": 497.4, "end": 501.56, "text": " we're gonna do the following we're gonna take device the layers one through four"}, {"start": 501.56, "end": 507.06, "text": " and then five through what eight and we're gonna send them on to separate"}, {"start": 507.06, "end": 511.74, "text": " devices so that's the pipeline approach so literally take this and you put that"}, {"start": 511.74, "end": 518.48, "text": " onto a separate device here then you take this one and you put it so let me"}, {"start": 518.48, "end": 521.96, "text": " let me let me draw it here and I'm literally not explaining the pipeline"}, {"start": 521.96, "end": 525.92, "text": " idea so I might as well explain it the whole way so literally you take the"}, {"start": 525.92, "end": 530.12, "text": " layers five through eight and then you feed them into the different device here"}, {"start": 530.12, "end": 536.76, "text": " and then you take the layers the next four layers because I don't know how to"}, {"start": 536.76, "end": 542.4, "text": " add I'm just gonna use yeah whatever so literally you feed that into the third"}, {"start": 542.4, "end": 548.4399999999999, "text": " block and then the last one the last four layers are gonna be fed into the"}, {"start": 548.4399999999999, "end": 554.28, "text": " last block here okay so that's what the what the pipeline parallelism approach"}, {"start": 554.28, "end": 558.92, "text": " does it literally splits the model so called well I think this is yeah this is"}, {"start": 558.92, "end": 564.6, "text": " the horizontal splitting and you can now imagine that each of these devices needs"}, {"start": 564.6, "end": 571.4, "text": " to hold literally four times less memory roughly I mean at least when it comes to"}, {"start": 571.4, "end": 577.64, "text": " weights there is also the up to optimizer states etc etc okay so the idea on the"}, {"start": 577.64, "end": 581.6, "text": " opposite side here with model parallelism from the Megatron LM paper is"}, {"start": 581.6, "end": 587.0, "text": " instead you're splitting you're doing the so-called vertical split so you do"}, {"start": 587.0, "end": 593.76, "text": " the following you instead let me take some color such as green and let's"}, {"start": 593.76, "end": 597.24, "text": " imagine we have just two devices gonna be easier to draw so you literally split"}, {"start": 597.24, "end": 602.8199999999999, "text": " your transformer like this so literally vertical split it and then you send this"}, {"start": 602.8199999999999, "end": 608.88, "text": " piece here on to one device and you send the other piece the other half of the"}, {"start": 608.88, "end": 612.68, "text": " transformer on to the other device so that's the model parallelism or the"}, {"start": 612.68, "end": 616.56, "text": " tensor parallelism as a different different name for that same like"}, {"start": 616.56, "end": 619.96, "text": " concept okay because I'm already halfway through explaining the pipeline"}, {"start": 619.96, "end": 624.6, "text": " parallelism let me let me quickly go through through that so that's the idea"}, {"start": 624.6, "end": 629.96, "text": " I think well one of the papers that mentioned that idea was the G pipe from"}, {"start": 629.96, "end": 634.44, "text": " Google others were I think like pipe dream or something there are multiple"}, {"start": 634.44, "end": 638.88, "text": " papers implementing similar ideas but the idea is as I mentioned this one and"}, {"start": 638.88, "end": 643.6800000000001, "text": " what's the problem here well the problem is if you just naively approach training"}, {"start": 643.6800000000001, "end": 648.1600000000001, "text": " a model that's and that's split using the pipeline parallelism you have a"}, {"start": 648.16, "end": 652.88, "text": " problem with with these so-called bubbles so let me try and demonstrate"}, {"start": 652.88, "end": 658.12, "text": " what I mean by that so imagine you feed a batch of data through this pipeline"}, {"start": 658.12, "end": 664.3199999999999, "text": " okay so you have a batch of data here I'm gonna denote it as well I guess like"}, {"start": 664.3199999999999, "end": 670.04, "text": " this just to make a difference between the layers and and the data so the thing"}, {"start": 670.04, "end": 674.56, "text": " is while this while you're doing the feed forward through the first device I"}, {"start": 674.56, "end": 680.4399999999999, "text": " through the first let's say four layers the other three devices are idle and you"}, {"start": 680.4399999999999, "end": 684.76, "text": " can see that on this diagram like this so you can see here you can literally"}, {"start": 684.76, "end": 690.64, "text": " see here that these devices here are idle so these three are doing exactly"}, {"start": 690.64, "end": 695.76, "text": " nothing okay there is no no computation going on so that's kind of wasteful"}, {"start": 695.76, "end": 700.0799999999999, "text": " right and then once you do the feed forward through the first four layers"}, {"start": 700.08, "end": 705.84, "text": " then you end up you basically send the activations to the second device and you"}, {"start": 705.84, "end": 710.2800000000001, "text": " start doing the feed forward here and at this point of time the three devices the"}, {"start": 710.2800000000001, "end": 715.0, "text": " first and the second the third and the fourth one are idle so you can see here"}, {"start": 715.0, "end": 719.2, "text": " literally these devices are doing nothing so you can you can see here this"}, {"start": 719.2, "end": 725.84, "text": " is gonna be idle these two devices are idle etc etc and so then you end up you"}, {"start": 725.84, "end": 729.48, "text": " propagate through all of the four devices and then you then you start doing"}, {"start": 729.48, "end": 733.28, "text": " the backprop so you want to calculate the gradients and when you start doing"}, {"start": 733.28, "end": 738.9200000000001, "text": " that again in the in the backward pass you are you're literally waiting so so"}, {"start": 738.9200000000001, "end": 744.12, "text": " you're doing backprop here but these three devices are being idle so this is"}, {"start": 744.12, "end": 748.72, "text": " super wasteful so there is a simple way how you can avoid this and it's called"}, {"start": 748.72, "end": 754.32, "text": " micro batching so split your mini batch into multiple micro batches and then you"}, {"start": 754.32, "end": 758.52, "text": " start feeding them into your pipeline and increase the throughput let me let"}, {"start": 758.52, "end": 764.1999999999999, "text": " me show you what I mean by that so you can see the diagram is here but let me"}, {"start": 764.1999999999999, "end": 769.28, "text": " explain conceptually what it means so here what you do is you split the simple"}, {"start": 769.28, "end": 774.24, "text": " data into multiple chunks into so-called micro batches and you feed this first"}, {"start": 774.24, "end": 780.36, "text": " part here so you feed it here and while it's running here and then the moment"}, {"start": 780.36, "end": 785.48, "text": " it starts running through the through the second device you immediately feed"}, {"start": 785.48, "end": 789.52, "text": " the second part the second micro batch through this device and now you have two"}, {"start": 789.52, "end": 796.0, "text": " devices not being idle and then as soon as this device passes on the activations"}, {"start": 796.0, "end": 800.28, "text": " to the third device you can feed the third micro batch into this device and"}, {"start": 800.28, "end": 804.4, "text": " so on and so forth you get the point and you can see the diagram looks like like"}, {"start": 804.4, "end": 809.9200000000001, "text": " this so initially for the duration of the first micro batch you literally have"}, {"start": 809.9200000000001, "end": 813.72, "text": " you literally and let me just change the color here you literally have four"}, {"start": 813.72, "end": 818.44, "text": " devices being idle so you can still here see that the four devices are being"}, {"start": 818.44, "end": 823.0, "text": " idle which is wasteful and then only two devices are being being idle and then"}, {"start": 823.0, "end": 828.0, "text": " only one and finally here you can see that all of the devices are being busy"}, {"start": 828.0, "end": 832.88, "text": " and you have the the the 4x the throughput compared to the naive"}, {"start": 832.88, "end": 837.24, "text": " implementation of the pipeline parallelism so in order to get as many of"}, {"start": 837.24, "end": 842.48, "text": " these columns where all of the four devices are active you want to have a"}, {"start": 842.48, "end": 847.8000000000001, "text": " lot of micro batches and what the GPy paper showed basically empirically is"}, {"start": 847.8000000000001, "end": 853.64, "text": " that you want to have M greater or equal than 4k where K is the number of"}, {"start": 853.64, "end": 858.0, "text": " partitions or the number of devices a fancy name saying number of devices so"}, {"start": 858.0, "end": 862.64, "text": " that means if you have four devices you want to have 32 plus micro batches and"}, {"start": 862.64, "end": 868.04, "text": " then literally they showed that you almost have linear scaling of the"}, {"start": 868.04, "end": 875.1999999999999, "text": " throughput which is awesome okay so that's one way you can take a huge model"}, {"start": 875.1999999999999, "end": 880.7199999999999, "text": " split it across multiple devices each of the devices will have only a portion a"}, {"start": 880.7199999999999, "end": 888.3199999999999, "text": " slice of the network weights and thus you can train this whole system faster"}, {"start": 888.3199999999999, "end": 891.56, "text": " because you have faster throughput and it's also feasible in the first place"}, {"start": 891.56, "end": 895.8399999999999, "text": " because yeah you cannot kind of stick it into a single device okay so that's the"}, {"start": 895.84, "end": 901.24, "text": " second idea now let's go through the actual paper this one is very fun let's"}, {"start": 901.24, "end": 903.6800000000001, "text": " get back to the first sentence so we started here"}, {"start": 903.6800000000001, "end": 907.12, "text": " intralayer model parallel approach we now know what that means so it's intro"}, {"start": 907.12, "end": 910.44, "text": " we're splitting the actual layer we're not splitting across layers we take a"}, {"start": 910.44, "end": 914.24, "text": " single layer and we split it into two parts or four parts or eight parts or"}, {"start": 914.24, "end": 919.2800000000001, "text": " whatnot our approach does not require a new compiler or library changes is"}, {"start": 919.2800000000001, "end": 923.52, "text": " orthogonal and complementary to pipeline model parallelism and can be fully"}, {"start": 923.52, "end": 927.4399999999999, "text": " implemented with insertion of a few communication operations in native"}, {"start": 927.4399999999999, "end": 932.48, "text": " part or so the cool thing is the complementary part is very important so"}, {"start": 932.48, "end": 935.88, "text": " this part is super important because that means you can combine these two"}, {"start": 935.88, "end": 942.4399999999999, "text": " approaches to scale to even bigger model sizes okay then they say this we"}, {"start": 942.4399999999999, "end": 949.72, "text": " sustain 15.1 petaflops across the entire application with 76% scaling"}, {"start": 949.72, "end": 954.8000000000001, "text": " efficiency when compared to a strong single GPU baseline that sustains 39"}, {"start": 954.8000000000001, "end": 960.4, "text": " teraflops which is 30% of peak flops so let me kind of break this down first of"}, {"start": 960.4, "end": 964.6, "text": " all I forgot to mention here they're training a model that has 8.3 billion"}, {"start": 964.6, "end": 969.6800000000001, "text": " parameters and they're training on 512 GPUs okay so what I say here is that"}, {"start": 969.6800000000001, "end": 976.9200000000001, "text": " they achieve a throughput of 15.1 peta so peta is I guess 10 to the power of 15 or"}, {"start": 976.92, "end": 982.8, "text": " something so it's like thousand tera or 1024 depends when you ask flops is the"}, {"start": 982.8, "end": 986.92, "text": " the floating point operations per second so that means you have this many"}, {"start": 986.92, "end": 991.7199999999999, "text": " operations per second that's a huge number okay and what they say is that"}, {"start": 991.7199999999999, "end": 997.36, "text": " they have a 60 76 percent scaling efficiency so what does that mean in"}, {"start": 997.36, "end": 1001.0, "text": " practice that means the following let me open up a calculator here so we have"}, {"start": 1001.0, "end": 1006.88, "text": " that the single device single GPU has 39 teraflops so if you multiply 39"}, {"start": 1006.88, "end": 1012.04, "text": " times the number of GPUs ideally this will be ideal scaling we'd end up with"}, {"start": 1012.04, "end": 1017.8, "text": " 19 petaflops and you can see here that they only end up with 15.1 and that's"}, {"start": 1017.8, "end": 1022.82, "text": " where this 60 76 percent scaling efficiency comes into place if I"}, {"start": 1022.82, "end": 1032.88, "text": " multiply this with 0.76 we end up with exactly 15.1 petaflops okay so basically"}, {"start": 1032.88, "end": 1037.7600000000002, "text": " you can see that the GPU baseline has 30% of the theoretical peak so that the"}, {"start": 1037.7600000000002, "end": 1041.6000000000001, "text": " GPU will never have its its its full theory it will never reach its full"}, {"start": 1041.6000000000001, "end": 1045.3200000000002, "text": " theoretical peak and then when you additionally scale across multiple"}, {"start": 1045.3200000000002, "end": 1050.8400000000001, "text": " devices you'll never have ideal linear scaling but it's close enough so that's"}, {"start": 1050.8400000000001, "end": 1056.96, "text": " cool okay so let's continue here let me see what's interesting we're gonna see"}, {"start": 1056.96, "end": 1064.08, "text": " how we are gonna split the two particular blocks so we're gonna see how"}, {"start": 1064.08, "end": 1068.96, "text": " we split the MLP block first and then I'm gonna show you this is probably a"}, {"start": 1068.96, "end": 1074.48, "text": " bit easier how do we split the the attention block and they also split the"}, {"start": 1074.48, "end": 1079.8400000000001, "text": " output embeddings as well as the input embeddings and finally a detail worth"}, {"start": 1079.8400000000001, "end": 1083.8, "text": " mentioning is that this residual connections additions layer norms are"}, {"start": 1083.8, "end": 1088.68, "text": " actually replicated across devices so there is some duplication going on for"}, {"start": 1088.68, "end": 1093.8799999999999, "text": " various reasons they decided that's that's the most optimal thing to do but"}, {"start": 1093.8799999999999, "end": 1098.8, "text": " that's the plan let's see how do we how do we vertically break up the MLP how"}, {"start": 1098.8, "end": 1103.84, "text": " do we vertical split that the attention block that's gonna be the the main gist"}, {"start": 1103.84, "end": 1109.72, "text": " of this paper okay let's start here here you can see the the diagrams of how how"}, {"start": 1109.72, "end": 1115.84, "text": " the splitting happens so the upper diagram here shows you the basically"}, {"start": 1115.84, "end": 1123.96, "text": " oops shows you the MLP block and you can see this is the first linear layer the"}, {"start": 1123.96, "end": 1129.2, "text": " second near layer and this is how it's kind of split down but let me go into"}, {"start": 1129.2, "end": 1133.64, "text": " the formulas I'm gonna slowly understand how this works but first on the high"}, {"start": 1133.64, "end": 1141.2, "text": " level what happens here is that you can see that this weights you take the"}, {"start": 1141.2, "end": 1147.1200000000001, "text": " weights of this linear layer you split them column wise and you keep a 2 on one"}, {"start": 1147.1200000000001, "end": 1152.8400000000001, "text": " device and you keep a a 1 on the second device you split the matrix that you use"}, {"start": 1152.8400000000001, "end": 1157.16, "text": " for your feed forward layer and you split it into two parts you do the same"}, {"start": 1157.16, "end": 1162.4, "text": " thing here but the difference is here you split it row wise so you can see"}, {"start": 1162.4, "end": 1166.6000000000001, "text": " here we don't have column wise splitting we have row wise splitting okay and so"}, {"start": 1166.6000000000001, "end": 1170.96, "text": " you take B1 you put B1 on one device you put B2 on the second device so"}, {"start": 1170.96, "end": 1175.68, "text": " basically you can see here that all of these operations are happening on a"}, {"start": 1175.68, "end": 1181.6000000000001, "text": " single device okay and then there is this G operation going on which is gonna"}, {"start": 1181.6000000000001, "end": 1186.88, "text": " synchronize the two devices more on that a bit later and the second device"}, {"start": 1186.88, "end": 1194.4, "text": " obviously here is processing the second part of the MLP okay so it has the a1"}, {"start": 1194.4, "end": 1198.16, "text": " and B1 so that's and that's how we achieve the vertical splitting but"}, {"start": 1198.16, "end": 1202.0800000000002, "text": " that's only a high-level diagram let's understand why this works and how is it"}, {"start": 1202.0800000000002, "end": 1207.8000000000002, "text": " equivalent to just having everything on the same device it might be like maybe"}, {"start": 1207.8000000000002, "end": 1211.88, "text": " tough to grasp on the on the first fly but it's fairly simple once you think"}, {"start": 1211.88, "end": 1216.0, "text": " about it a bit okay so here is the implementation here is what the layer"}, {"start": 1216.0, "end": 1221.76, "text": " does here is what the the feed forward layer does you do the this is your input"}, {"start": 1221.76, "end": 1227.96, "text": " data X this is your weight matrix a and this is the activation unit jelly in"}, {"start": 1227.96, "end": 1233.16, "text": " this particular case so there is multiple ways we can split we saw the"}, {"start": 1233.16, "end": 1237.88, "text": " column wise and row wise let's see the advantages and disadvantages of both if"}, {"start": 1237.88, "end": 1245.88, "text": " we split the weight matrix basically row wise as you can see here then we have a"}, {"start": 1245.88, "end": 1250.5600000000002, "text": " problem and that problem is we need to synchronize the devices after this pass"}, {"start": 1250.5600000000002, "end": 1257.5200000000002, "text": " so let's see why that's the case so if we now multiply this you get X one times"}, {"start": 1257.5200000000002, "end": 1263.5200000000002, "text": " a one plus X two times a two and then you need to apply jelly so the problem"}, {"start": 1263.5200000000002, "end": 1268.7600000000002, "text": " here is the plus and that means that the two devices will need to communicate so"}, {"start": 1268.7600000000002, "end": 1274.0400000000002, "text": " that that's the only way you can you can well add those two separate activations"}, {"start": 1274.04, "end": 1279.2, "text": " right and then you apply the jelly the problem is the jelly is not like is a"}, {"start": 1279.2, "end": 1286.76, "text": " nonlinear function so jelly of of adding these two is not the same as jelly of"}, {"start": 1286.76, "end": 1291.32, "text": " the first term plus the jelly of the second term and so they say this"}, {"start": 1291.32, "end": 1296.28, "text": " approach will require a synchronization point before the jelly function okay let"}, {"start": 1296.28, "end": 1304.48, "text": " me try and map this matrix multiplication into the diagram with"}, {"start": 1304.48, "end": 1309.48, "text": " which you're probably familiar with okay so let's start like this we have the"}, {"start": 1309.48, "end": 1315.36, "text": " input data X I'm gonna assume our batch size is only one so we have something"}, {"start": 1315.36, "end": 1323.36, "text": " like this so we have X here and we split X into two parts so we have X one and we"}, {"start": 1323.36, "end": 1330.84, "text": " have X two here okay next up we need to pass that through the feed forward layer"}, {"start": 1330.84, "end": 1336.6399999999999, "text": " so we have a weight matrix we're going to multiply with it so this is going to"}, {"start": 1336.6399999999999, "end": 1341.6, "text": " be just some box here I'm gonna fill in the details a bit later and we end up"}, {"start": 1341.6, "end": 1349.12, "text": " with the final representation I'm gonna denote it as a red rectangle here and"}, {"start": 1349.12, "end": 1356.9599999999998, "text": " that's it okay and now I'm gonna draw like a matrix a here and we said that we"}, {"start": 1356.9599999999998, "end": 1362.52, "text": " are going to split the matrix a basically row wise right so you're gonna"}, {"start": 1362.52, "end": 1371.3999999999999, "text": " split the matrix a row wise so let's see what happens now so if we multiply this"}, {"start": 1371.3999999999999, "end": 1377.32, "text": " part here and then we can assume in a little bit here if you multiply this"}, {"start": 1377.32, "end": 1384.48, "text": " part here with this part here what does that mean so that means that we do the"}, {"start": 1384.48, "end": 1391.04, "text": " following thing we'll literally do this whoops we'll literally do this so we do"}, {"start": 1391.04, "end": 1396.36, "text": " this is kind of how it's gonna be how it's gonna look like if you if you write"}, {"start": 1396.36, "end": 1400.3999999999999, "text": " like a neural if you if you draw a neural network okay so that's what you've done"}, {"start": 1400.3999999999999, "end": 1405.72, "text": " by multiplying this part with this part and then you multiply this with the"}, {"start": 1405.72, "end": 1409.32, "text": " second part okay so now you multiply it with the second piece here and that"}, {"start": 1409.32, "end": 1414.68, "text": " second piece is gonna give you the second output and you end up with"}, {"start": 1414.68, "end": 1419.6000000000001, "text": " something like this and then you do this here and this here and you end up with"}, {"start": 1419.6000000000001, "end": 1425.92, "text": " four outputs so I'm kind of using only four to to to yeah simplify the drawing"}, {"start": 1425.92, "end": 1431.96, "text": " so the problem is that all of these outputs here are still incomplete"}, {"start": 1431.96, "end": 1436.64, "text": " because they are missing the contribution from the x2 portion that's"}, {"start": 1436.64, "end": 1442.32, "text": " why we have to add the the actual representations so what I do now is let"}, {"start": 1442.32, "end": 1446.44, "text": " me change the color into let's say red so now you'll have to do the"}, {"start": 1446.44, "end": 1452.64, "text": " multiplication between this one and this one to end up with the contribution to"}, {"start": 1452.64, "end": 1457.92, "text": " the first output element okay so this gives us this and then you just repeat"}, {"start": 1457.92, "end": 1461.76, "text": " repeat repeat and you get the contributions to this one to this one and"}, {"start": 1461.76, "end": 1466.68, "text": " to this one okay so that's hopefully kind of helps you understand how this"}, {"start": 1466.68, "end": 1471.92, "text": " maps to the neural network diagrams with which you are used to so as you see we"}, {"start": 1471.92, "end": 1477.5, "text": " if you if you just do x1 times a1 we only get partial contributions to the"}, {"start": 1477.5, "end": 1482.92, "text": " output and because of that we also have to add additionally the x2 a2 and only"}, {"start": 1482.92, "end": 1489.24, "text": " then can we apply the jelly okay now let's go to the second like a way of"}, {"start": 1489.24, "end": 1493.36, "text": " breaking down the weight matrix and that's the column splitting okay so"}, {"start": 1493.36, "end": 1497.2, "text": " another option is to split a along its columns like this this partitioning"}, {"start": 1497.2, "end": 1502.32, "text": " allows the jelly non-linearity to be independently applied to the output of"}, {"start": 1502.32, "end": 1507.48, "text": " each partitioned gem so you can see what's the what's the benefit here you"}, {"start": 1507.48, "end": 1513.04, "text": " can literally send X so this time you're sending not X1 is here so please notice"}, {"start": 1513.04, "end": 1517.04, "text": " this thing this is a very important detail so here we have X1 here we have"}, {"start": 1517.04, "end": 1521.72, "text": " X so that means we're passing the whole input and as well here and here we had"}, {"start": 1521.72, "end": 1527.44, "text": " X2 so but a benefit we get here is that we can literally do everything on that"}, {"start": 1527.44, "end": 1531.48, "text": " device and then apply jelly and that's it we don't have any synchronization"}, {"start": 1531.48, "end": 1534.8799999999999, "text": " going on so that's very cool because synchronization slows us down we have"}, {"start": 1534.8799999999999, "end": 1539.76, "text": " to communicate the weights and sum them up and then return them back okay so"}, {"start": 1539.76, "end": 1543.12, "text": " this is advantageous as it removes a synchronization point hence we"}, {"start": 1543.12, "end": 1548.4399999999998, "text": " partition the first gem in this column parallel fashion and split the second"}, {"start": 1548.4399999999998, "end": 1553.4399999999998, "text": " gem along its rows so it takes the output of the jelly layer directly"}, {"start": 1553.4399999999998, "end": 1558.6, "text": " without requiring any communication as shown in figure 3a so that's the hack"}, {"start": 1558.6, "end": 1563.6399999999999, "text": " we're gonna do so let me now do the following let me first show you how the"}, {"start": 1563.6399999999999, "end": 1569.4799999999998, "text": " column splitting is gonna look like so I'm gonna do the same diagram again let"}, {"start": 1569.48, "end": 1574.52, "text": " me just change the color so I'm gonna do this similar diagram again so we have"}, {"start": 1574.52, "end": 1583.2, "text": " our X here so this is our X we have something going on here I'm gonna just"}, {"start": 1583.2, "end": 1589.84, "text": " draw like this like a box here a placeholder and then I'm gonna pick the"}, {"start": 1589.84, "end": 1596.52, "text": " red color so we denote the output representation with red and let's now"}, {"start": 1596.52, "end": 1604.44, "text": " multiply the input X so this is again X let's multiply X with our matrix that"}, {"start": 1604.44, "end": 1609.0, "text": " split column wise this time so that's the main difference okay so now we"}, {"start": 1609.0, "end": 1613.84, "text": " split it so here you saw the horizontal one here we have a vertical splitting"}, {"start": 1613.84, "end": 1618.68, "text": " okay so what happens now let's go through this so now when you multiply"}, {"start": 1618.68, "end": 1624.8, "text": " this with this you literally end up with a full representation so this output"}, {"start": 1624.8, "end": 1632.9199999999998, "text": " here has everything it needs so it's completely has all of the necessary"}, {"start": 1632.9199999999998, "end": 1636.9199999999998, "text": " information from the input then we do the same thing and now you can see that"}, {"start": 1636.9199999999998, "end": 1642.08, "text": " basically what is going to happen let me change the color maybe to gray after you"}, {"start": 1642.08, "end": 1647.52, "text": " keep on multiplying this with this part with this matrix you're gonna end up"}, {"start": 1647.52, "end": 1654.96, "text": " with a half of the output representation vector but this half is for like is"}, {"start": 1654.96, "end": 1658.56, "text": " going to be actually contain it's gonna contain all of the information necessary"}, {"start": 1658.56, "end": 1663.32, "text": " so we don't have to do any summation that's the advantage here okay so we do"}, {"start": 1663.32, "end": 1668.0, "text": " that on two devices we end up with these representations we apply the gel use and"}, {"start": 1668.0, "end": 1672.16, "text": " then and then we feed so that so now this is this is the nice part we'll"}, {"start": 1672.16, "end": 1676.32, "text": " literally we'll literally just take these representations here so we apply"}, {"start": 1676.32, "end": 1684.4399999999998, "text": " jellies again so we apply the jelly and we then feed them as x1 to the this"}, {"start": 1684.4399999999998, "end": 1690.8, "text": " layer where we split the weight matrix horizontally so that's the nice part so"}, {"start": 1690.8, "end": 1695.24, "text": " now when you concatenate these two you get an MLP and you have literally half"}, {"start": 1695.24, "end": 1699.56, "text": " of the weights on one device and half of the weights on the second device that's"}, {"start": 1699.56, "end": 1704.24, "text": " it as simple as that beautiful idea if you ask me very simple but very"}, {"start": 1704.24, "end": 1709.88, "text": " beautiful and that's drawn here again you can see it here so we start with X"}, {"start": 1709.88, "end": 1715.1200000000001, "text": " we replicate X then we we have because this one is column split we can just do"}, {"start": 1715.1200000000001, "end": 1720.92, "text": " X times a 2 and then we apply jellies and then we just feed that here and then"}, {"start": 1720.92, "end": 1725.9, "text": " we multiply with b2 and here remember we have horizontal so row wise splitting"}, {"start": 1725.9, "end": 1730.36, "text": " and then we have the G which is the synchronization operation so that means"}, {"start": 1730.36, "end": 1734.24, "text": " we're gonna sum them up and finally we end up with the final representation is"}, {"start": 1734.24, "end": 1739.56, "text": " that here that's it that's how we split the MLP layer okay let me see where"}, {"start": 1739.56, "end": 1744.36, "text": " there is something interesting here this approach splits both gems in the MLP"}, {"start": 1744.36, "end": 1748.24, "text": " block across GPUs and requires only a single all reduce operation in the"}, {"start": 1748.24, "end": 1752.34, "text": " forward pass so the G operator and a single already using the backward pass"}, {"start": 1752.34, "end": 1757.56, "text": " the F operator so let me show you how that looks like so again we saw that"}, {"start": 1757.56, "end": 1761.1599999999999, "text": " already so in the forward pass the only point where we have to synchronize to do"}, {"start": 1761.1599999999999, "end": 1765.84, "text": " the already use is here the G and when we go backwards once we get the"}, {"start": 1765.84, "end": 1769.12, "text": " gradients here we somehow have to combine them right to get the"}, {"start": 1769.12, "end": 1773.76, "text": " representations here so we literally have to well I guess just add them and"}, {"start": 1773.76, "end": 1778.6, "text": " divide by two whatever we have to have the synchronization primitive there okay"}, {"start": 1778.6, "end": 1782.9199999999998, "text": " now for the attention part and I really hope now you're you're kind of"}, {"start": 1782.92, "end": 1788.3200000000002, "text": " appreciating the whole engineering the infrastructure that goes into these big"}, {"start": 1788.3200000000002, "end": 1793.52, "text": " language models it's not just like you set like a hyper parameter in your code"}, {"start": 1793.52, "end": 1798.64, "text": " to from from like five layers to like 120 layers of transformer and expect"}, {"start": 1798.64, "end": 1801.52, "text": " everything is gonna work magically on a single device there is a lot of"}, {"start": 1801.52, "end": 1806.4, "text": " engineering going on and here trying to explain you some of the methods okay"}, {"start": 1806.4, "end": 1811.6000000000001, "text": " attention is a bit easier I think in my opinion so the thing is you literally"}, {"start": 1811.6, "end": 1815.04, "text": " take the query matrix and I'm gonna I'm not going to explain the transformers"}, {"start": 1815.04, "end": 1818.12, "text": " please do check out my transformer video I'm gonna link it somewhere here in"}, {"start": 1818.12, "end": 1822.4399999999998, "text": " case you're not familiar with the details of the transformer model you"}, {"start": 1822.4399999999998, "end": 1827.1999999999998, "text": " take the query matrix you take the key matrix you take the value matrix here and"}, {"start": 1827.1999999999998, "end": 1833.36, "text": " basically you split them column wise and each of these columns is gonna"}, {"start": 1833.36, "end": 1838.56, "text": " literally compute the queries for a single head so we have the multi head"}, {"start": 1838.56, "end": 1842.52, "text": " attention let's imagine for the sake of argument here that we only have two heads"}, {"start": 1842.52, "end": 1846.32, "text": " and by the way that brings me let me just quickly mention this so even though"}, {"start": 1846.32, "end": 1850.44, "text": " we just had two here so we were splitting the so-called two-way model"}, {"start": 1850.44, "end": 1854.8799999999999, "text": " parallelism you can imagine splitting the weight matrices into much more like"}, {"start": 1854.8799999999999, "end": 1859.6399999999999, "text": " well parts so you can have like a four-way model of parallelism or eight"}, {"start": 1859.6399999999999, "end": 1864.08, "text": " way and you can kind of convince yourself that you're going to keep the"}, {"start": 1864.08, "end": 1870.3999999999999, "text": " equivalent so if we have eight if we if we divide this into eight rows and if we"}, {"start": 1870.3999999999999, "end": 1876.52, "text": " divide the input data into eight pieces as well you'll just have x1 a1 plus x2"}, {"start": 1876.52, "end": 1883.8, "text": " a2 plus x3 a3 plus etc etc until you get to x8 a8 and then you have to"}, {"start": 1883.8, "end": 1887.8799999999999, "text": " synchronize them and then you apply jelly so nothing is preventing us from"}, {"start": 1887.8799999999999, "end": 1893.28, "text": " from splitting this into multiple parts okay so let's get back to the attention"}, {"start": 1893.28, "end": 1898.92, "text": " so here let's imagine for the sake of explanation we only have two heads and"}, {"start": 1898.92, "end": 1904.48, "text": " basically what will happen is that this device up here so this device up here is"}, {"start": 1904.48, "end": 1912.36, "text": " going to compute the first head so the the representation for the first head"}, {"start": 1912.36, "end": 1918.16, "text": " and the bottom layer the bottom part of this diagram is going to compute the"}, {"start": 1918.16, "end": 1922.32, "text": " second head and now we have this is the head one representations and these are"}, {"start": 1922.32, "end": 1926.76, "text": " the head two representations and then we just feed them here and this is the the"}, {"start": 1926.76, "end": 1932.84, "text": " already familiar hopefully breakdown of the feed forward layer where we do the"}, {"start": 1932.84, "end": 1936.02, "text": " row by splitting and you can see here that we again have to have the G"}, {"start": 1936.02, "end": 1939.96, "text": " because we have to synchronize them and then we get the out-propresentations"}, {"start": 1939.96, "end": 1945.08, "text": " okay let me try and demonstrate this again with a small diagram quickly so we"}, {"start": 1945.08, "end": 1951.04, "text": " start with the input so let's assume we have that one and sequence size one so"}, {"start": 1951.04, "end": 1954.2, "text": " we literally have a single token so let's that's gonna be the easiest case"}, {"start": 1954.2, "end": 1961.84, "text": " but it's gonna make hopefully make the point okay so here is our input data X"}, {"start": 1961.84, "end": 1968.92, "text": " again it's a single token so B the the batch size is one S which is usually"}, {"start": 1968.92, "end": 1972.52, "text": " along this dimension is one we just have a single token that has some"}, {"start": 1972.52, "end": 1979.44, "text": " dimensionality okay so some maybe H the hidden dimension okay so what we do is"}, {"start": 1979.44, "end": 1988.3200000000002, "text": " we basically map it using the key query and values so the first device will"}, {"start": 1988.3200000000002, "end": 1996.76, "text": " literally map this one into let's say like this we have we have the the query"}, {"start": 1996.76, "end": 2005.56, "text": " we have the key we have the value and you can see that the it's like twice"}, {"start": 2005.56, "end": 2011.76, "text": " smaller dimensionality compared to the input one and now you're literally gonna"}, {"start": 2011.76, "end": 2016.6399999999999, "text": " do your your magic of the transformer and you end up with the final"}, {"start": 2016.6399999999999, "end": 2023.6399999999999, "text": " representation which is again 2x smaller so this is H2 if this is H this is H2"}, {"start": 2023.6399999999999, "end": 2028.72, "text": " okay and you just do the same thing on the separate device so you'll do the"}, {"start": 2028.72, "end": 2033.9199999999998, "text": " same logic I'm just gonna kind of do it like this and you end up with the same"}, {"start": 2033.92, "end": 2037.4, "text": " well it's gonna be a different representation but the same structure"}, {"start": 2037.4, "end": 2043.6000000000001, "text": " here so we end up with this H over 2 okay so now we pay pass those tokens we"}, {"start": 2043.6000000000001, "end": 2048.92, "text": " pass them here into the feed forward layer we do the multiplication so now"}, {"start": 2048.92, "end": 2054.92, "text": " we literally have the the same example with X1 and X2 so we split our data into"}, {"start": 2054.92, "end": 2061.2400000000002, "text": " two parts that's what we've done here and now you literally do the when you"}, {"start": 2061.24, "end": 2065.3599999999997, "text": " multiply would be one you're gonna end up with this so let me let me change the"}, {"start": 2065.3599999999997, "end": 2070.4799999999996, "text": " color to like blue so you're gonna end up with this so something like this blah"}, {"start": 2070.4799999999996, "end": 2075.8799999999997, "text": " blah blah blah blah so we end up with the full representation again H but if"}, {"start": 2075.8799999999997, "end": 2080.3599999999997, "text": " you recall we literally have a partial representation so we'll have to add up"}, {"start": 2080.3599999999997, "end": 2085.6, "text": " on top of this one the the mapping that we get from so the representation that"}, {"start": 2085.6, "end": 2091.16, "text": " we get from this representation here so now we'll have to add up this thing here"}, {"start": 2091.16, "end": 2096.64, "text": " and only now do we have the full representation and that's the the G"}, {"start": 2096.64, "end": 2102.3599999999997, "text": " the G operator here so hopefully that the idea is clear until now that was the"}, {"start": 2102.3599999999997, "end": 2105.3599999999997, "text": " main idea let me kind of walk you through the text here so partitioning the"}, {"start": 2105.3599999999997, "end": 2110.2799999999997, "text": " gems associated with the with key query and value in a column parallel fashion"}, {"start": 2110.2799999999997, "end": 2113.8799999999997, "text": " such that the matrix multiply corresponding to each attention head is"}, {"start": 2113.8799999999997, "end": 2120.12, "text": " done locally on one GPU this allows us to split per attention head parameters"}, {"start": 2120.12, "end": 2126.04, "text": " and workload across the GPUs and doesn't require any immediate communication to"}, {"start": 2126.04, "end": 2129.3599999999997, "text": " complete the self-attention that's very important we don't need any immediate"}, {"start": 2129.3599999999997, "end": 2134.44, "text": " communication okay so this approach for both the MLP and self-attention layer"}, {"start": 2134.44, "end": 2140.08, "text": " fuses groups of two gems so general matrix multiplies eliminates a"}, {"start": 2140.08, "end": 2144.96, "text": " synchronization point in between and results in better scaling so what I mean"}, {"start": 2144.96, "end": 2149.2799999999997, "text": " by that is you can see here in both of these diagrams you can see that we have"}, {"start": 2149.28, "end": 2153.48, "text": " a like a general matrix multiply and then something happens and then general"}, {"start": 2153.48, "end": 2158.2000000000003, "text": " matrix multiply and compilers are smart enough to optimize these to fuse these"}, {"start": 2158.2000000000003, "end": 2161.88, "text": " operations together so that you literally have a single matrix multiply"}, {"start": 2161.88, "end": 2165.6800000000003, "text": " or something I'm not super familiar with those low-level details I've never had"}, {"start": 2165.6800000000003, "end": 2170.32, "text": " to do that but like that's my understanding at the moment so that's"}, {"start": 2170.32, "end": 2173.4, "text": " what I see here and this results in better scaling this enables us to"}, {"start": 2173.4, "end": 2178.2000000000003, "text": " perform all gems in a simple transformer layer using only two all reduces in the"}, {"start": 2178.2, "end": 2183.04, "text": " forward path and two in the backward path again that's because we have"}, {"start": 2183.04, "end": 2187.0, "text": " because the transformer layer consists of one attention we have a single"}, {"start": 2187.0, "end": 2191.96, "text": " already use here and then we pass the data that comes out of here we pass it"}, {"start": 2191.96, "end": 2197.0, "text": " here and then we have another already use here the G so that's two in the in"}, {"start": 2197.0, "end": 2201.68, "text": " the in the single single pass through the transformer layer okay guys that's"}, {"start": 2201.68, "end": 2209.2799999999997, "text": " pretty much it now I did say that they also have to split the output embeddings"}, {"start": 2209.2799999999997, "end": 2212.6, "text": " the input embeddings etc etc I'm just gonna kind of rush you through this the"}, {"start": 2212.6, "end": 2216.6, "text": " logic is very similar I'm not gonna bother explaining this one we parallelize"}, {"start": 2216.6, "end": 2222.68, "text": " the input embedding matrix denoted as E and then the subsequent is H with a"}, {"start": 2222.68, "end": 2227.4199999999996, "text": " hidden dimension and V the vocabulary size along the vocab dimension so e1"}, {"start": 2227.42, "end": 2233.04, "text": " e2 column wise since since each partition now only contains a portion of"}, {"start": 2233.04, "end": 2237.76, "text": " the embedding table and already use the G operator is required after the input"}, {"start": 2237.76, "end": 2242.28, "text": " embedding so again we'll be able to split the e matrix which is usually huge"}, {"start": 2242.28, "end": 2248.12, "text": " like the vocab can sometimes be like 50,000 etc etc so you can split it into"}, {"start": 2248.12, "end": 2252.92, "text": " multiple pieces send each of those pieces onto the GPUs then you do the"}, {"start": 2252.92, "end": 2257.52, "text": " mapping and then you'll just have to synchronize again same idea as what we"}, {"start": 2257.52, "end": 2263.52, "text": " saw before okay then they also mentioned for the output it's a bit more intricate"}, {"start": 2263.52, "end": 2268.2400000000002, "text": " so blah blah blah if you just do it simply then it's super expensive but"}, {"start": 2268.2400000000002, "end": 2272.44, "text": " then they say so however for this case the all getter will communicate B times"}, {"start": 2272.44, "end": 2276.86, "text": " s times V elements B is the batch size s is the sequence length and B is the"}, {"start": 2276.86, "end": 2280.88, "text": " vocab size which is huge due to the vocab size being large to reduce the"}, {"start": 2280.88, "end": 2285.56, "text": " communication size we fuse the output of the parallel gem with the cross-entropy"}, {"start": 2285.56, "end": 2289.88, "text": " loss which reduces the dimensions to B times s communicating scalar losses"}, {"start": 2289.88, "end": 2293.7200000000003, "text": " instead of logits is a huge reduction in communication that improves the"}, {"start": 2293.7200000000003, "end": 2299.2000000000003, "text": " efficiency of our model parallel approach okay and finally they they"}, {"start": 2299.2000000000003, "end": 2304.84, "text": " mentioned this so those are the almost all the pieces that make the transformer"}, {"start": 2304.84, "end": 2307.96, "text": " but there is also the residual connections the layer norms all of that"}, {"start": 2307.96, "end": 2312.28, "text": " so let me just mention that briefly rather than having one GPU compute part"}, {"start": 2312.28, "end": 2316.0, "text": " of the dropout layer normalization or residual connections and broadcast the"}, {"start": 2316.0, "end": 2320.2, "text": " results to other GPUs we choose to duplicate the communication across"}, {"start": 2320.2, "end": 2325.6, "text": " GPUs I encourage you to read at your own pace why this is the case but I think"}, {"start": 2325.6, "end": 2330.32, "text": " you got the gist of this of this paper already a couple more details before"}, {"start": 2330.32, "end": 2335.52, "text": " before jumping to the mixed precision training so you can understand and"}, {"start": 2335.52, "end": 2340.88, "text": " appreciate why we need that method let me just kind of go here so to train our"}, {"start": 2340.88, "end": 2345.0, "text": " models efficiently we utilize mixed precision training as I said with"}, {"start": 2345.0, "end": 2348.96, "text": " dynamic loss scaling we're gonna learn what it is and why we need it then you"}, {"start": 2348.96, "end": 2352.52, "text": " can see a bunch of details like how do we initialize the model how do you scale"}, {"start": 2352.52, "end": 2357.24, "text": " it you do some norm clipping blah blah blah a lot of engineering going on here"}, {"start": 2357.24, "end": 2361.44, "text": " lastly to better manage our memory footprint we utilize activation"}, {"start": 2361.44, "end": 2366.28, "text": " checkpointing after every transformer layer this is a super important idea I'm"}, {"start": 2366.28, "end": 2370.84, "text": " gonna briefly explain it here because it's used all like all the time all the"}, {"start": 2370.84, "end": 2376.16, "text": " time so activation checkpointing you'll hear it you'll hear this concept being"}, {"start": 2376.16, "end": 2380.2400000000002, "text": " thrown around all the time so the idea is the following again we have a"}, {"start": 2380.2400000000002, "end": 2387.28, "text": " transformer here okay it's a big model has a lot of layers so now you do a feed"}, {"start": 2387.28, "end": 2392.1600000000003, "text": " forward through that model okay so you do a feed forward through the model and"}, {"start": 2392.1600000000003, "end": 2396.0400000000004, "text": " you keep on collecting the activations okay so each of the layers will be"}, {"start": 2396.0400000000004, "end": 2401.28, "text": " storing the activations why well because when you do the back prop in order to"}, {"start": 2401.28, "end": 2405.1200000000003, "text": " calculate the gradients you need to have activations of each of the"}, {"start": 2405.1200000000003, "end": 2410.8, "text": " corresponding layers so you can already probably notice the problem with this"}, {"start": 2410.8, "end": 2415.6000000000004, "text": " approach you're literally storing bunch of data here here here and as you're"}, {"start": 2415.6, "end": 2420.3199999999997, "text": " progressing through the network the memory peak so the memory consumption is"}, {"start": 2420.3199999999997, "end": 2424.48, "text": " rising rising rising so the idea with the activation checkpointing is the"}, {"start": 2424.48, "end": 2427.72, "text": " following it's very simple and it's a beautiful idea it's covered in one of"}, {"start": 2427.72, "end": 2431.7599999999998, "text": " the other papers I obviously don't have enough time to cover it but if I get"}, {"start": 2431.7599999999998, "end": 2436.04, "text": " enough requests I might cover it in the future merge with some more advanced"}, {"start": 2436.04, "end": 2440.44, "text": " techniques that came after all these papers I am explaining in this video so"}, {"start": 2440.44, "end": 2445.7200000000003, "text": " did is fairly simple don't store activations for every single layer just"}, {"start": 2445.7200000000003, "end": 2450.76, "text": " checkpoint them so basically let's imagine we have four let's imagine we"}, {"start": 2450.76, "end": 2459.44, "text": " have four layers okay and instead of storing the activations for all of the"}, {"start": 2459.44, "end": 2464.94, "text": " feed forward layers inside of the block you just stored the activations at the"}, {"start": 2464.94, "end": 2469.16, "text": " at the at the end of the block so you literally just store the the ultimate"}, {"start": 2469.16, "end": 2475.6, "text": " activations okay so like suggest the activations that are at the end of the"}, {"start": 2475.6, "end": 2479.68, "text": " transformer block so just store these ones okay so you just store these ones"}, {"start": 2479.68, "end": 2483.24, "text": " and now when you're doing the back prop so now let's see what happens when you"}, {"start": 2483.24, "end": 2487.3599999999997, "text": " do the back prop okay so you start doing the let me change the color into maybe"}, {"start": 2487.3599999999997, "end": 2492.0, "text": " gray so we start doing the back prop here we are good we are good we're good"}, {"start": 2492.0, "end": 2496.7999999999997, "text": " now we hit this part where we don't have the activations and what we do is we"}, {"start": 2496.8, "end": 2501.32, "text": " just recompute them so now we trigger the recomputation from this checkpoint"}, {"start": 2501.32, "end": 2505.6000000000004, "text": " here okay so we trigger the recomputation from here and then we can"}, {"start": 2505.6000000000004, "end": 2510.6000000000004, "text": " continue our back prop and then we hit this part here no activations we hit the"}, {"start": 2510.6000000000004, "end": 2515.4, "text": " we hit the recomputation again and we end up with activations obviously this is"}, {"start": 2515.4, "end": 2519.5600000000004, "text": " gonna happen in a bit more clever way so while they're doing the back prop we are"}, {"start": 2519.5600000000004, "end": 2523.1600000000003, "text": " computing the we're triggering the recomputation so that we don't have to"}, {"start": 2523.16, "end": 2527.12, "text": " wait so all of those optimization details but this is the gist of it this"}, {"start": 2527.12, "end": 2530.96, "text": " is how the activation checkpointing works okay hopefully you can appreciate"}, {"start": 2530.96, "end": 2535.72, "text": " that now let me quickly walk you through a couple more details and we are done"}, {"start": 2535.72, "end": 2540.2799999999997, "text": " with this paper so our infrastructure is optimized for multi-node deep learning"}, {"start": 2540.2799999999997, "end": 2544.3199999999997, "text": " applications with three hundred gigabytes per second bandwidth between"}, {"start": 2544.3199999999997, "end": 2550.3999999999996, "text": " GPUs inside a server so that's important by this NS and VS switch and hundred"}, {"start": 2550.4, "end": 2555.0, "text": " gigabytes of interconnect bandwidth between servers okay that's important so"}, {"start": 2555.0, "end": 2561.32, "text": " we have we have 300 gigabytes per second between the GPUs and we have only hundred"}, {"start": 2561.32, "end": 2565.44, "text": " only hundred a smaller number that's that's the whole point between the"}, {"start": 2565.44, "end": 2570.62, "text": " servers so why am I mentioning this the reason being is you want when you're"}, {"start": 2570.62, "end": 2574.88, "text": " doing the metal the model of parallelism because you have all of those already"}, {"start": 2574.88, "end": 2578.36, "text": " use operations so the synchronization points you want to have those"}, {"start": 2578.36, "end": 2583.0, "text": " synchronization points happening inside of a server where the bandwidth is super"}, {"start": 2583.0, "end": 2587.96, "text": " high okay so that's why when you have a cluster where your node only has eight"}, {"start": 2587.96, "end": 2591.92, "text": " GPUs you don't want to go more than eight way model parallelism you don't"}, {"start": 2591.92, "end": 2596.1600000000003, "text": " want to go 16 because if you have 16 then you have two servers communicating"}, {"start": 2596.1600000000003, "end": 2600.88, "text": " that during the synchronization points and that sucks okay so hopefully that"}, {"start": 2600.88, "end": 2606.2400000000002, "text": " that makes sense now let me show you some some results they achieve you can"}, {"start": 2606.24, "end": 2611.24, "text": " see here that scaling to from one to two to four to eight GPUs you can see that"}, {"start": 2611.24, "end": 2617.04, "text": " the the efficiency kind of drops but not significantly so here is the 77% for the"}, {"start": 2617.04, "end": 2621.3599999999997, "text": " eight way model parallelism and if you have if you combine it with the data"}, {"start": 2621.3599999999997, "end": 2630.4399999999996, "text": " parallel approach then you have still 74% for 512 GPUs that's huge that's huge"}, {"start": 2630.4399999999996, "end": 2634.9199999999996, "text": " I'm going to quickly explain how we combine the model parallelism with the"}, {"start": 2634.92, "end": 2639.64, "text": " data parallelism it's a very very nice visualization so quickly this is how it's"}, {"start": 2639.64, "end": 2643.48, "text": " going to look like as I said you're gonna because the servers let's say have"}, {"start": 2643.48, "end": 2647.56, "text": " eight GPUs basically you're gonna do the following thing you're gonna have"}, {"start": 2647.56, "end": 2652.52, "text": " something like this and then you're gonna slice this server into eight pieces"}, {"start": 2652.52, "end": 2657.6800000000003, "text": " because all the pieces are a single GPU and they are very densely connected and"}, {"start": 2657.6800000000003, "end": 2662.84, "text": " so you literally do you have one eighth of the of the model weights here one"}, {"start": 2662.84, "end": 2667.52, "text": " eighth of the model weights here etc etc and now remember the data parallelism"}, {"start": 2667.52, "end": 2673.1600000000003, "text": " you just replicate you copy paste these multiple times so precisely they use the"}, {"start": 2673.1600000000003, "end": 2681.48, "text": " 64 way data parallelism so you literally you're going to replicate this so that"}, {"start": 2681.48, "end": 2688.56, "text": " you have so that you end up with let me draw it like this so you have eight here"}, {"start": 2688.56, "end": 2697.96, "text": " whoops so yeah eight here and then you have eight here and you end up with 64"}, {"start": 2697.96, "end": 2703.44, "text": " times eight and that's 512 and this is gonna be your setup so you have"}, {"start": 2703.44, "end": 2709.36, "text": " literally 64 servers each of the servers has eight GPUs with high bandwidth"}, {"start": 2709.36, "end": 2714.36, "text": " internally and a bit lower bandwidth externally so internode versus"}, {"start": 2714.36, "end": 2720.28, "text": " intranode and yeah then you basically do the logic I just explained so in this"}, {"start": 2720.28, "end": 2725.04, "text": " particular example you have a huge batch you're gonna split it into 64 chunks and"}, {"start": 2725.04, "end": 2728.8, "text": " then you're gonna pass each of the chunks to one of the servers and then"}, {"start": 2728.8, "end": 2733.2000000000003, "text": " you do the logic we just saw from the Megatron LM paper with all of the"}, {"start": 2733.2000000000003, "end": 2736.96, "text": " synchronization points etc etc and then calculate the gradients and then you're"}, {"start": 2736.96, "end": 2741.6800000000003, "text": " gonna do the all reviews and update the whole cluster that's it fairly simple"}, {"start": 2741.68, "end": 2746.3999999999996, "text": " okay at least conceptually one thing I want to the last thing I want to mention"}, {"start": 2746.3999999999996, "end": 2751.96, "text": " from this paper is it came before GPT-3 was published which is probably the most"}, {"start": 2751.96, "end": 2757.08, "text": " famous language model and the ML model in general and they already noticed the"}, {"start": 2757.08, "end": 2762.44, "text": " trend everybody knew already that scaling is leading to better and better"}, {"start": 2762.44, "end": 2767.8799999999997, "text": " performance so here by performance I mean lower perplexity for example so you"}, {"start": 2767.88, "end": 2773.12, "text": " can see that 8.3 billion model has lower perplexity than the 2.5 billion etc etc"}, {"start": 2773.12, "end": 2776.98, "text": " they mentioned that multiple times as the model size increases the validation"}, {"start": 2776.98, "end": 2780.1600000000003, "text": " perplexity decreases and reaches a validation perplexity of blah blah blah"}, {"start": 2780.1600000000003, "end": 2783.96, "text": " we observed the trend of decreasing model size also leads to lower perplexity"}, {"start": 2783.96, "end": 2787.6, "text": " blah blah blah recently researchers from Microsoft in collaboration with Nvidia"}, {"start": 2787.6, "end": 2792.6400000000003, "text": " trained a 17 billion parameter GPT-2 called Turing NLG using Megatron and"}, {"start": 2792.6400000000003, "end": 2795.44, "text": " showed that the accuracy is further improved as they scale the model"}, {"start": 2795.44, "end": 2799.64, "text": " highlighting the value of larger models it was super obvious that somebody's"}, {"start": 2799.64, "end": 2805.4, "text": " gonna do it it just happened that open AI was the first one to do it and yeah"}, {"start": 2805.4, "end": 2810.36, "text": " kind of worth mentioning a historical note I guess yeah they also have"}, {"start": 2810.36, "end": 2814.08, "text": " contribution of this paper is they show that you kind of have to reconfigure"}, {"start": 2814.08, "end": 2817.84, "text": " where you put your layer norms so they can so that you can train much bigger"}, {"start": 2817.84, "end": 2821.48, "text": " birds otherwise you have these instabilities you can see how the the"}, {"start": 2821.48, "end": 2826.16, "text": " model loss explodes if you if you just use this default bird architecture but"}, {"start": 2826.16, "end": 2831.2400000000002, "text": " that's not important for our like well it is you kind of have to be aware that"}, {"start": 2831.2400000000002, "end": 2837.08, "text": " everything matters that even the model architecture is is very sensitive if you"}, {"start": 2837.08, "end": 2840.48, "text": " change rearrange something all of a sudden you cannot train the bigger"}, {"start": 2840.48, "end": 2845.56, "text": " models and so on and so forth okay guys now on to the mixed precision training"}, {"start": 2845.56, "end": 2848.96, "text": " I'm gonna quickly walk you through this one and then we're gonna dig into the"}, {"start": 2848.96, "end": 2853.84, "text": " zero paper one important quick note before before this is all the three"}, {"start": 2853.84, "end": 2858.52, "text": " approaches I showed you the pipeline parallelism the data parallelism and the"}, {"start": 2858.52, "end": 2862.52, "text": " model or the tensor parallelism all of them can be combined together they are"}, {"start": 2862.52, "end": 2868.92, "text": " complementary and I think it was branded as 3d parallelism so when you combine"}, {"start": 2868.92, "end": 2872.88, "text": " all of these three you can achieve like huge scales and then when you combine it"}, {"start": 2872.88, "end": 2876.56, "text": " with mixed precision plus the zero optimizer which you're gonna see in a"}, {"start": 2876.56, "end": 2881.64, "text": " couple of minutes then you can achieve truly astounding scales okay that's"}, {"start": 2881.64, "end": 2886.56, "text": " that's it now let's dig into this paper so a couple of contributions from this"}, {"start": 2886.56, "end": 2891.2599999999998, "text": " paper so let's gonna read it firstly we recommend maintaining a single"}, {"start": 2891.2599999999998, "end": 2894.52, "text": " precision copy of weights that accumulates the gradients after each"}, {"start": 2894.52, "end": 2898.0, "text": " optimizer step this copy is rounded to have precision for the forward and back"}, {"start": 2898.0, "end": 2901.64, "text": " prop so this is the first part so you're gonna literally have three components"}, {"start": 2901.64, "end": 2906.04, "text": " that I want to explain and then we're done the second part is we propose loss"}, {"start": 2906.04, "end": 2910.04, "text": " scaling we don't see what that concept means to preserve gradient values with"}, {"start": 2910.04, "end": 2915.48, "text": " small small magnitudes thirdly we use half precision arithmetic that accumulates"}, {"start": 2915.48, "end": 2921.7599999999998, "text": " into single precision outputs that's FP 32 versus FP 16 which are converted to"}, {"start": 2921.7599999999998, "end": 2926.68, "text": " have precision before storing to memory okay so today is fairly simple usually"}, {"start": 2926.68, "end": 2932.04, "text": " when you train neural networks historically people use the FP 32 format"}, {"start": 2932.04, "end": 2936.6, "text": " that means a single numbers like literally needs 32 bits instead of that"}, {"start": 2936.6, "end": 2941.56, "text": " people have shown that you didn't you do not need 32 bits you can have lower"}, {"start": 2941.56, "end": 2947.4, "text": " accuracy for a neural network training so like 16 works and that's it by just"}, {"start": 2947.4, "end": 2952.68, "text": " having the the amount of amount of data amount of memory that that a single"}, {"start": 2952.68, "end": 2957.2799999999997, "text": " number takes you literally can save a bunch of a bunch of memory and still"}, {"start": 2957.28, "end": 2962.1200000000003, "text": " they show in this paper keep the same accuracy the same performance of the"}, {"start": 2962.1200000000003, "end": 2967.96, "text": " model as the baseline which uses the FP 32 that's very cool okay so why does it"}, {"start": 2967.96, "end": 2972.88, "text": " matter not only do we store the memory it's also faster on the modern hardware"}, {"start": 2972.88, "end": 2979.76, "text": " so have precision math throughput in recent GPUs is 2x to 8x higher than"}, {"start": 2979.76, "end": 2983.6000000000004, "text": " single precision in addition to speed improvements reduce precision formats"}, {"start": 2983.6000000000004, "end": 2986.1600000000003, "text": " also reduce the amount of memory required for training that's obvious I"}, {"start": 2986.16, "end": 2991.3999999999996, "text": " already said that one okay okay let's see what's going on so specifically we"}, {"start": 2991.3999999999996, "end": 2996.3199999999997, "text": " train various neural networks using the IEEE half precision format let me"}, {"start": 2996.3199999999997, "end": 3003.04, "text": " quickly show you I'll link these somewhere below in the video but this is"}, {"start": 3003.04, "end": 3007.12, "text": " the this is the the format they're they're referring to you so you have the"}, {"start": 3007.12, "end": 3010.96, "text": " exponent has five bits the fraction has ten bits and then there is a sine bit"}, {"start": 3010.96, "end": 3017.6, "text": " and this is how you represent numbers like the flow 32 is basically obviously"}, {"start": 3017.6, "end": 3023.2, "text": " has a bigger exponent and bigger fraction and well while I'm here"}, {"start": 3023.2, "end": 3027.7200000000003, "text": " explaining the formats let me quickly mention that the B float 16 or the brain"}, {"start": 3027.7200000000003, "end": 3034.48, "text": " float because it was originally invented into Google brain AI lab and you can see"}, {"start": 3034.48, "end": 3038.48, "text": " what's the difference the B float has much bigger exponent and the main"}, {"start": 3038.48, "end": 3043.48, "text": " insight here is for neural network training you care about the range you"}, {"start": 3043.48, "end": 3047.68, "text": " don't care about the actual accuracy and this significant or mantissa is what"}, {"start": 3047.68, "end": 3052.92, "text": " determines the how fine-tuned your numbers are whereas the exponent"}, {"start": 3052.92, "end": 3058.52, "text": " determines how big can you go so obviously with B float you can have much"}, {"start": 3058.52, "end": 3062.36, "text": " bigger and much smaller numbers without getting the overflow the infinities or"}, {"start": 3062.36, "end": 3068.04, "text": " the or the NANDs or whatnot and so basically TPUs natively support this"}, {"start": 3068.04, "end": 3073.56, "text": " this B float format and like people have shown like when you take a look at the"}, {"start": 3073.56, "end": 3079.2599999999998, "text": " problems that for example Boris Dyma training the Lee mini had compared to"}, {"start": 3079.2599999999998, "end": 3085.4, "text": " training on GPUs TPUs basically are much more robust to these divergences of"}, {"start": 3085.4, "end": 3090.36, "text": " losses because of the B float format so that's my so that's basically my"}, {"start": 3090.36, "end": 3094.46, "text": " understanding I might be wrong here but that's I wanted to quickly mention the"}, {"start": 3094.46, "end": 3099.12, "text": " all of these formats now let's go back to the paper okay so now that you know"}, {"start": 3099.12, "end": 3104.2400000000002, "text": " that let's see what's going on so we have to store the master weights they"}, {"start": 3104.2400000000002, "end": 3109.16, "text": " showed for most models this is kind of necessary so they say here the need for"}, {"start": 3109.16, "end": 3114.44, "text": " FP32 master weights is not universal but for the most models they trained they"}, {"start": 3114.44, "end": 3121.52, "text": " did need to do this so here is the idea you store the weights as FP32 but then"}, {"start": 3121.52, "end": 3125.8, "text": " when you do the forward prop the back prop you do everything in half precision"}, {"start": 3125.8, "end": 3129.24, "text": " so you literally do this float to half operation so you literally kind of"}, {"start": 3129.24, "end": 3134.88, "text": " convert the FP32 to FP16 then you do the feed forward you can see here you start"}, {"start": 3134.88, "end": 3139.0, "text": " with the activations which are also FP16 so you can kind of always make sure"}, {"start": 3139.0, "end": 3143.72, "text": " that's the case because you control your data so let's say the input the input is"}, {"start": 3143.72, "end": 3149.04, "text": " your image you can literally feed image as FP16 image and then combine that with"}, {"start": 3149.04, "end": 3153.64, "text": " FP16 weights do the forward pass you get the activations and then in the"}, {"start": 3153.64, "end": 3157.36, "text": " backward you do the same thing the weights are FP16 you calculate the"}, {"start": 3157.36, "end": 3161.72, "text": " activation grads from the activation grads you get the weight grads and all"}, {"start": 3161.72, "end": 3166.64, "text": " of those are in FP16 and because of that you benefit from like a smaller memory"}, {"start": 3166.64, "end": 3174.6, "text": " like a blueprint as well as the well higher throughput as we saw that some"}, {"start": 3174.6, "end": 3179.2, "text": " modern GPUs basically have higher throughput for FP16 compared to FP32"}, {"start": 3179.2, "end": 3183.6, "text": " that's that's the bottom line okay so this this is a schematic how everything"}, {"start": 3183.6, "end": 3188.48, "text": " works so they now mention one explanation why they need the FP32"}, {"start": 3188.48, "end": 3193.18, "text": " master weights one explanation is that updates weight gradients multiplied by"}, {"start": 3193.18, "end": 3198.7799999999997, "text": " the learning rate become too small to be represented in FP16 any value whose"}, {"start": 3198.7799999999997, "end": 3204.36, "text": " magnitude is smaller than 2 raised to the power of minus 24 becomes 0 in FP"}, {"start": 3204.36, "end": 3210.76, "text": " 16 and by the way you might be confused why 24 and not 14 I'm not gonna dig into"}, {"start": 3210.76, "end": 3218.0, "text": " that but just Google for the sub normalized so sub normal or something"}, {"start": 3218.0, "end": 3227.6, "text": " like that sub normal values just open up the the the float 16 and float 32 Viki"}, {"start": 3227.6, "end": 3232.6800000000003, "text": " pages I'm gonna link those down below and you'll see what those are so we can"}, {"start": 3232.68, "end": 3237.08, "text": " see in figure 2b that approximately 5% of weight gradient values have exponents"}, {"start": 3237.08, "end": 3242.7999999999997, "text": " smaller than minus 24 okay let me show you what I mean by that so let's go to"}, {"start": 3242.7999999999997, "end": 3249.0, "text": " figure 2b you can see here this is the training when the model is trained in in"}, {"start": 3249.0, "end": 3254.2599999999998, "text": " FP32 and you can see that most of the gradients here fall below this"}, {"start": 3254.2599999999998, "end": 3260.72, "text": " magic red line the the minus 24 which is the where the representation range of"}, {"start": 3260.72, "end": 3266.08, "text": " the FP 16 ends and so that means that all of these are gonna be rounded to 0"}, {"start": 3266.08, "end": 3269.9199999999996, "text": " and you lose a lot of information during training and because of that the"}, {"start": 3269.9199999999996, "end": 3275.6, "text": " accuracy the final accuracy for model may or may not drop okay so that's one"}, {"start": 3275.6, "end": 3284.8799999999997, "text": " kind of idea why why you need FP 32 there is a couple of other explanations"}, {"start": 3284.8799999999997, "end": 3289.16, "text": " here a bit more complex to explain here I'm gonna I'm gonna kind of skip it for"}, {"start": 3289.16, "end": 3293.56, "text": " now so worth mentioning now that you have to store this additional copy"}, {"start": 3293.56, "end": 3297.08, "text": " doesn't that blow up the memory and here to explain that's not the case"}, {"start": 3297.08, "end": 3301.0, "text": " because the activations are what matters so even though maintaining an additional"}, {"start": 3301.0, "end": 3304.3199999999997, "text": " copy of weights increases the memory requirements for the weights by 50%"}, {"start": 3304.3199999999997, "end": 3309.2799999999997, "text": " compared with single precision training impact on overall memory usage is much"}, {"start": 3309.2799999999997, "end": 3313.44, "text": " smaller for training memory consumption is dominated by activations due to"}, {"start": 3313.44, "end": 3318.56, "text": " larger batch sizes and activations of each layer being saved for reuse in the"}, {"start": 3318.56, "end": 3323.96, "text": " back prop pass since activations are also stored in half precision format the"}, {"start": 3323.96, "end": 3327.68, "text": " overall memory consumption for training deep neural networks is roughly halved"}, {"start": 3327.68, "end": 3332.12, "text": " okay they are not obviously taking into account activation checkpointing they"}, {"start": 3332.12, "end": 3336.08, "text": " just assume that you have to store all of the activations in that case this"}, {"start": 3336.08, "end": 3343.04, "text": " thing that they've just said here holds but yeah okay now let's go into the law"}, {"start": 3343.04, "end": 3346.7599999999998, "text": " scaling this is a second important concept you need to get the mixed"}, {"start": 3346.76, "end": 3351.44, "text": " precision training to work know that much of the fp16 representable range was left"}, {"start": 3351.44, "end": 3356.36, "text": " unused while many values were below the minimum representable range and became"}, {"start": 3356.36, "end": 3360.0600000000004, "text": " zeros scaling up the gradients will shift them to occupy more of the"}, {"start": 3360.0600000000004, "end": 3364.0400000000004, "text": " representable range and preserve values that are otherwise lost to zeros this"}, {"start": 3364.0400000000004, "end": 3367.2400000000002, "text": " particular network diverges when gradients are not scaled but scaling"}, {"start": 3367.2400000000002, "end": 3371.0, "text": " them by a factor of 8 increasing the exponents by 3 is sufficient to make"}, {"start": 3371.0, "end": 3377.56, "text": " the accuracy achieved with fp32 training this suggests that activation gradient"}, {"start": 3377.56, "end": 3382.88, "text": " below mine to raise to the power of minus 27 in magnitude were irrelevant to"}, {"start": 3382.88, "end": 3387.2, "text": " the training of this model but values in this range were important to preserve"}, {"start": 3387.2, "end": 3392.88, "text": " okay let me break this down for you okay so here is a histogram of basically"}, {"start": 3392.88, "end": 3398.3, "text": " activation gradient values you can see what I said it's shifted outside of the"}, {"start": 3398.3, "end": 3403.5800000000004, "text": " range that fp16 basically support so this is the range that fp16 loves okay"}, {"start": 3403.5800000000004, "end": 3409.04, "text": " this is this subnormal or the denormalized range so from minus 24 to"}, {"start": 3409.04, "end": 3414.2000000000003, "text": " minus 14 you can see that most of these values are outside of the range and if"}, {"start": 3414.2000000000003, "end": 3421.2000000000003, "text": " you multiply the gradients by 8 you literally shift all of these rightwards"}, {"start": 3421.2000000000003, "end": 3427.48, "text": " or equivalently we move the red line here and now you have all of these being"}, {"start": 3427.48, "end": 3432.96, "text": " representable in fp16 thus we will not round them to zero we will not lose this"}, {"start": 3432.96, "end": 3439.36, "text": " information and it turned out they say here blah blah blah that that this"}, {"start": 3439.36, "end": 3442.96, "text": " particular range here and you can see there is a lot of data here was very"}, {"start": 3442.96, "end": 3448.04, "text": " important to preserve and so scaling by 8 for this particular model it doesn't"}, {"start": 3448.04, "end": 3452.08, "text": " even matter what the model is just the concept is important it kind of saved"}, {"start": 3452.08, "end": 3457.08, "text": " the training okay so one efficient way to shift the gradient values into fp16"}, {"start": 3457.08, "end": 3461.96, "text": " representable range is to scale the last value computed in the forward pass prior"}, {"start": 3461.96, "end": 3466.52, "text": " to starting back propagation by chain rule back propagation ensures that all"}, {"start": 3466.52, "end": 3470.4, "text": " gradient values are scaled by the same amount so that's a easy way how you can"}, {"start": 3470.4, "end": 3474.08, "text": " scale up all these gradients you just take the final loss you scale it up by"}, {"start": 3474.08, "end": 3478.44, "text": " let's say 8 and by doing that once you start doing the back prop because how"}, {"start": 3478.44, "end": 3482.56, "text": " chain rule works you literally multiply all of the gradients by 8 and thus you'll"}, {"start": 3482.56, "end": 3487.72, "text": " be shifting this the gradients into the representable range this is the yeah"}, {"start": 3487.72, "end": 3491.7999999999997, "text": " this is the main the main intuition I want you to take out of this paper and"}, {"start": 3491.7999999999997, "end": 3497.44, "text": " here is the here's the third technique they do to maintain model accuracy we"}, {"start": 3497.44, "end": 3501.52, "text": " found that some networks require that the fp16 vector dot product accumulates"}, {"start": 3501.52, "end": 3506.58, "text": " the partial products into an fp32 value which is converted to fp16 before"}, {"start": 3506.58, "end": 3511.36, "text": " writing to memory without this accumulation into fp32 some fp16 models"}, {"start": 3511.36, "end": 3515.88, "text": " did not match the accuracy of the baseline models okay also for large"}, {"start": 3515.88, "end": 3519.92, "text": " reductions for example sums across elements of a vector they should all be"}, {"start": 3519.92, "end": 3524.28, "text": " carried out in fp32 such reductions mostly come up in batch normalization"}, {"start": 3524.28, "end": 3529.1200000000003, "text": " layers when accumulating statistics and softmax layers as well so both of the"}, {"start": 3529.1200000000003, "end": 3532.88, "text": " layer types in our implementation still read and write fp16 tensors from memory"}, {"start": 3532.88, "end": 3537.38, "text": " performing the arithmetic in fp32 this did not slow down the training process"}, {"start": 3537.38, "end": 3540.3, "text": " since these layers are memory bandwidth limited and not sensitive to"}, {"start": 3540.3, "end": 3546.4, "text": " arithmetic speed okay again just a additional detail that needs to be"}, {"start": 3546.4, "end": 3551.6400000000003, "text": " taken care of basically there are three categories so they say here this is this"}, {"start": 3551.6400000000003, "end": 3556.5600000000004, "text": " cannot relevant by and by and large neural network arithmetic falls into"}, {"start": 3556.5600000000004, "end": 3560.84, "text": " three categories so the vector dot products the reductions and the point"}, {"start": 3560.84, "end": 3564.7200000000003, "text": " wise operations such as the activation functions and that they just mentioned"}, {"start": 3564.7200000000003, "end": 3569.6800000000003, "text": " for these two so they mentioned that for vector dot products and for"}, {"start": 3569.68, "end": 3575.2599999999998, "text": " reductions you sometimes have to convert to fp32 and do the operations there such"}, {"start": 3575.2599999999998, "end": 3580.0, "text": " that you can preserve the accuracy that's it guys that's the idea behind"}, {"start": 3580.0, "end": 3584.02, "text": " mixed precision training now let me show you the results they show that"}, {"start": 3584.02, "end": 3588.68, "text": " literally baseline they compare they are almost always even better than the"}, {"start": 3588.68, "end": 3592.56, "text": " baselines so here you can if you just check out these numbers at your own base"}, {"start": 3592.56, "end": 3597.24, "text": " you'll see that the numbers are always a bit better so it maybe works as a like a"}, {"start": 3597.24, "end": 3604.7599999999998, "text": " regularizer almost to to to do the training in fp16 for some models as I"}, {"start": 3604.7599999999998, "end": 3608.08, "text": " mentioned the last skill and technique was not required for successful mix"}, {"start": 3608.08, "end": 3612.4799999999996, "text": " precision training of these networks but for some others it was very important"}, {"start": 3612.4799999999996, "end": 3617.68, "text": " let me show you one so here you can see that mixed precision training when the"}, {"start": 3617.68, "end": 3623.56, "text": " last scale is one diverges when the last scale is 128 it does not diverge and"}, {"start": 3623.56, "end": 3627.08, "text": " they literally show for every four different classes of models like this"}, {"start": 3627.08, "end": 3633.24, "text": " again here language models machine translation they always match the"}, {"start": 3633.24, "end": 3637.68, "text": " baseline when it comes to accuracy so that's awesome and the final idea least"}, {"start": 3637.68, "end": 3642.7999999999997, "text": " last but not least definitely not least because this is such an exciting paper"}, {"start": 3642.7999999999997, "end": 3650.1, "text": " this paper is basically introduces deep speed library you might have heard of it"}, {"start": 3650.1, "end": 3654.96, "text": " if you've seen if you've went through any of the popular LLMs most of them use"}, {"start": 3654.96, "end": 3662.3399999999997, "text": " deep speed illutri I know they use deep speed bloom use deep speed opt 175 be"}, {"start": 3662.3399999999997, "end": 3665.7999999999997, "text": " that the meta folks I'm not sure whether they used it but yeah so it's a"}, {"start": 3665.7999999999997, "end": 3670.3199999999997, "text": " literally a paper from Microsoft it's called zero memory optimizations toward"}, {"start": 3670.3199999999997, "end": 3675.48, "text": " training trillion parameter models okay I love this paper let's go through it"}, {"start": 3675.48, "end": 3681.84, "text": " step by step let's think step by step zero redundancy optimizer that's that's"}, {"start": 3681.84, "end": 3685.8, "text": " what the acronym stands for we'll see why zero redundancy they literally have"}, {"start": 3685.8, "end": 3690.08, "text": " well they reduce all of the redundancy when it comes to model states and"}, {"start": 3690.08, "end": 3694.2, "text": " residual states we're gonna see what those are but that's the idea zero"}, {"start": 3694.2, "end": 3698.52, "text": " eliminates memory redundancies in data and model parallel training zero has the"}, {"start": 3698.52, "end": 3702.96, "text": " potential to scale beyond one to them parameters using today's hardware zero"}, {"start": 3702.96, "end": 3707.88, "text": " can train large models of up to 13 billion parameters for example larger"}, {"start": 3707.88, "end": 3712.92, "text": " than Megatron GPT 8.3 B so that's a model that was trained in the Megatron LM"}, {"start": 3712.92, "end": 3717.6, "text": " paper the first paper I covered and the t5 the Google's model that was 11"}, {"start": 3717.6, "end": 3722.88, "text": " billion parameter big without even requiring model parallelism so that's a"}, {"start": 3722.88, "end": 3727.7200000000003, "text": " big breakthrough because now with with this set of optimizations you as a"}, {"start": 3727.72, "end": 3733.6, "text": " practitioner can literally train models as big as 13 B if you have just a single"}, {"start": 3733.6, "end": 3740.52, "text": " but very good albeit a very good like GPU but that's amazing and obviously"}, {"start": 3740.52, "end": 3746.24, "text": " when you use the actual optimizations you can you can scale to much bigger"}, {"start": 3746.24, "end": 3750.3999999999996, "text": " sizes they show they have shown indeed that you can scale up to 1 trillion"}, {"start": 3750.3999999999996, "end": 3755.16, "text": " parameters with a small caveat and that's that it becomes very slow I mean"}, {"start": 3755.16, "end": 3758.7999999999997, "text": " very slow you literally for some of the models they've shown you need a year to"}, {"start": 3758.7999999999997, "end": 3764.96, "text": " train it so that's practically not not really practical but yeah they train a"}, {"start": 3764.96, "end": 3769.0, "text": " model of hundred something billion so that's that's kind of cool okay so let's"}, {"start": 3769.0, "end": 3772.7999999999997, "text": " let's dig into it MP so the model parallelism splits the"}, {"start": 3772.7999999999997, "end": 3776.2, "text": " model vertically partitioning the computation and parameters in each layer"}, {"start": 3776.2, "end": 3779.8399999999997, "text": " across multiple devices requiring significant communication between each"}, {"start": 3779.8399999999997, "end": 3784.8399999999997, "text": " layer okay as a result they work well within a single node I did mention this"}, {"start": 3784.84, "end": 3789.8, "text": " where the inter GPU communication bandwidth is high but the efficiency"}, {"start": 3789.8, "end": 3795.6400000000003, "text": " degree degrades quickly beyond a single node we tested a 40 B parameter model"}, {"start": 3795.6400000000003, "end": 3803.1200000000003, "text": " using Megatron LM across 2d GX 2 nodes and observe about 5 tera flops per V 100"}, {"start": 3803.1200000000003, "end": 3808.08, "text": " GPU less than 5% of hardware peak so you can see it drops dramatically as soon as"}, {"start": 3808.08, "end": 3815.2, "text": " you start doing the model parallelism across multiple nodes okay then they say"}, {"start": 3815.2, "end": 3819.72, "text": " this for large models the majority of the memory is occupied by model states"}, {"start": 3819.72, "end": 3823.72, "text": " this is an important piece of terminology they're gonna be reusing it"}, {"start": 3823.72, "end": 3828.3199999999997, "text": " throughout the paper so let's kind of stress that so the model states which"}, {"start": 3828.3199999999997, "end": 3833.24, "text": " include the optimizer states such as momentum and variances in atom"}, {"start": 3833.24, "end": 3840.64, "text": " gradients and parameters so they consider those to be the model states the"}, {"start": 3840.64, "end": 3845.7999999999997, "text": " remaining memory is consumed by activation temporary buffers and"}, {"start": 3845.7999999999997, "end": 3851.7599999999998, "text": " unusable whoops some glitch unusable fragmented memory which we refer to"}, {"start": 3851.7599999999998, "end": 3856.8399999999997, "text": " collectively as residual states so again two pieces of terminology new"}, {"start": 3856.8399999999997, "end": 3862.72, "text": " terminology here we have residual states and we have model states and they're"}, {"start": 3862.72, "end": 3869.24, "text": " gonna address each one of those separately okay so as your DPU removes"}, {"start": 3869.24, "end": 3872.9599999999996, "text": " the memory state redundancies across data parallel processes by partitioning"}, {"start": 3872.9599999999996, "end": 3877.4399999999996, "text": " the model states instead of replicating them so that's the main idea that's"}, {"start": 3877.4399999999996, "end": 3881.3199999999997, "text": " literally the main idea of this paper they are partitioning instead of"}, {"start": 3881.3199999999997, "end": 3886.2, "text": " replicating I mean simple conceptually but there is a lot of work that goes"}, {"start": 3886.2, "end": 3893.2799999999997, "text": " into making this to work efficiently that's important so I'm gonna show you"}, {"start": 3893.2799999999997, "end": 3896.52, "text": " the diagram I'm not gonna explain it yet what this means I'm gonna see in a"}, {"start": 3896.52, "end": 3901.52, "text": " couple of seconds so they have a couple of optimizations optimizer state"}, {"start": 3901.52, "end": 3904.68, "text": " partitioning so when you partition the optimizer states instead of replicating"}, {"start": 3904.68, "end": 3908.3199999999997, "text": " it then you can do the gradient partitioning and then you can also do"}, {"start": 3908.3199999999997, "end": 3912.4399999999996, "text": " the parameter partition and they show that the memory reduction is linear with"}, {"start": 3912.44, "end": 3918.04, "text": " DP degree so that means the number of devices you use to replicate your models"}, {"start": 3918.04, "end": 3923.44, "text": " across when you do the data parallelism for example splitting across 64 GPUs"}, {"start": 3923.44, "end": 3929.92, "text": " will yield a 64 X memory reduction is huge there is a modest 50% increase in"}, {"start": 3929.92, "end": 3934.04, "text": " communication volume though okay with all three stages enabled zero can train a"}, {"start": 3934.04, "end": 3942.32, "text": " Trillium parameter model on just 1024 Nvidia GPUs which is crazy okay this is"}, {"start": 3942.32, "end": 3946.48, "text": " the main diagram let me first walk you through that this piece of text here and"}, {"start": 3946.48, "end": 3951.68, "text": " then I'm gonna basically yeah jump into explaining the dire for activations"}, {"start": 3951.68, "end": 3954.84, "text": " stored from forward pass in order to perform regular pass we noticed"}, {"start": 3954.84, "end": 3958.92, "text": " checkpointing helps I did explain activation checkpointing maybe half an"}, {"start": 3958.92, "end": 3964.4, "text": " hour ago but not sufficient for large models thus zero are optimizes"}, {"start": 3964.4, "end": 3968.36, "text": " activation memory by identifying and removing the activation replication in"}, {"start": 3968.36, "end": 3972.64, "text": " existing empty approaches through activation partition again partitioning"}, {"start": 3972.64, "end": 3976.8, "text": " partitioning it also offloads activations to CPU when appropriate okay"}, {"start": 3976.8, "end": 3982.16, "text": " that's another detail that can be done okay zero DP and zero are combined"}, {"start": 3982.16, "end": 3985.92, "text": " together forms a powerful system of memory optimizations for DL training"}, {"start": 3985.92, "end": 3990.96, "text": " that we collectively refer to as zero again just another piece of terminology"}, {"start": 3990.96, "end": 3995.7200000000003, "text": " if you ever start playing with deep speed this might be useful to know okay"}, {"start": 3995.7200000000003, "end": 4002.84, "text": " so let's see what's going on here is how the that the opera the baseline is how"}, {"start": 4002.84, "end": 4011.08, "text": " your classical change the color here this is how your data parallelism works"}, {"start": 4011.08, "end": 4017.84, "text": " you have three types of basically what you have parameters radiance and"}, {"start": 4017.84, "end": 4022.6, "text": " optimizer states you can see that optimizer states take that the biggest"}, {"start": 4022.6, "end": 4028.4, "text": " part of the memory and you can see across when you do the data parallelism I did"}, {"start": 4028.4, "end": 4032.72, "text": " mention that we literally replicate all of these you copy paste them and that's"}, {"start": 4032.72, "end": 4037.88, "text": " super inefficient okay you can see that the memory consumed is 2 plus 2 plus K"}, {"start": 4037.88, "end": 4042.44, "text": " times psi I'm going to explain what these are in a second and for I think"}, {"start": 4042.44, "end": 4047.6400000000003, "text": " they're using some GPT model here whatnot they show that you need 120"}, {"start": 4047.6400000000003, "end": 4054.0, "text": " gigabytes of memory per device that's huge so psi is the number of parameters"}, {"start": 4054.0, "end": 4058.36, "text": " I think it's 1.5 billion in this particular example they're using the"}, {"start": 4058.36, "end": 4065.52, "text": " the GPT model and the GPT-2 2 is because you use two bytes when you use mixed"}, {"start": 4065.52, "end": 4070.8, "text": " precision training you use two bytes for weights you use two bytes for gradients"}, {"start": 4070.8, "end": 4076.6, "text": " so that's why you have 2 plus 2 and then K is equal to 12 because these are the"}, {"start": 4076.6, "end": 4081.9, "text": " optimizer states because we'll see the calculation a bit later but for Adam K"}, {"start": 4081.9, "end": 4087.16, "text": " equals 12 and so you have 16 times 1.5 B you end up with with I guess this number"}, {"start": 4087.16, "end": 4095.48, "text": " is that correct just see what I messed something up at 1.5 times 16 wait I'm"}, {"start": 4095.48, "end": 4102.92, "text": " not sure yeah we'll see why yeah well sorry it's 7.5 B it says here okay that's"}, {"start": 4102.92, "end": 4107.8, "text": " why we get 100 now the first optimization is instead of having a"}, {"start": 4107.8, "end": 4112.32, "text": " copy of the optimizer states everywhere you instead partition it so you store"}, {"start": 4112.32, "end": 4117.48, "text": " one piece of the optimizer states on one GPU you store the second part here you"}, {"start": 4117.48, "end": 4124.36, "text": " store the the the the nth part here and you can see so sorry we have NGP's here"}, {"start": 4124.36, "end": 4128.92, "text": " so that's why we are gonna have ultimately when you cannot combine all"}, {"start": 4128.92, "end": 4134.44, "text": " of them together you'll have still the same the same information as here right"}, {"start": 4134.44, "end": 4139.04, "text": " and then you can do this same thing for everything else for gradients you can"}, {"start": 4139.04, "end": 4142.679999999999, "text": " partition them for parameters you can partition them and you can see how the"}, {"start": 4142.679999999999, "end": 4147.04, "text": " memory is slowly decreasing so here we have 2 psi plus 2 psi because we are"}, {"start": 4147.04, "end": 4153.299999999999, "text": " not doing the the parameter and gradient optimization but we do save up on the"}, {"start": 4153.3, "end": 4158.96, "text": " optimizer state and that's why we have K times psi divided by ND where ND is the"}, {"start": 4158.96, "end": 4163.08, "text": " number of devices in your data parallel setup it's gonna be 64 in in their"}, {"start": 4163.08, "end": 4167.16, "text": " particular case and you can see how the memory dramatically reduces so you"}, {"start": 4167.16, "end": 4172.2, "text": " started with 120 gigabytes and you ultimately end up with 1.9 gigabytes"}, {"start": 4172.2, "end": 4178.2, "text": " which is crazy I mean this is an amazing optimization okay let's continue here"}, {"start": 4178.2, "end": 4183.599999999999, "text": " this is what I mentioned before the complete set of optimizations in Xero"}, {"start": 4183.599999999999, "end": 4187.08, "text": " could allow us to run models with trillion parameters on the high-end"}, {"start": 4187.08, "end": 4193.2, "text": " hardware cluster today for example with 1000 V100 GPUs however the hardware"}, {"start": 4193.2, "end": 4197.84, "text": " compute capacity is still too limited and training time can be in practically"}, {"start": 4197.84, "end": 4201.92, "text": " long over a year so always take with a grain of salt when somebody says one"}, {"start": 4201.92, "end": 4206.5599999999995, "text": " trillion you have to kind of read the fine print and understand the details of"}, {"start": 4206.56, "end": 4213.4800000000005, "text": " the method zero power is the largest language model with 17 D parameters and"}, {"start": 4213.4800000000005, "end": 4218.68, "text": " record-breaking accuracy cheering energy important to mention this was"}, {"start": 4218.68, "end": 4223.4400000000005, "text": " literally I think a month before GPT-3 was published which was 10x bigger than"}, {"start": 4223.4400000000005, "end": 4228.72, "text": " this model which is like a huge we share zero as a part of our open source DL"}, {"start": 4228.72, "end": 4234.64, "text": " training optimization library called deep speed so I did mention this library"}, {"start": 4234.64, "end": 4237.400000000001, "text": " multiple times you can check it out especially if you want to play with the"}, {"start": 4237.400000000001, "end": 4241.8, "text": " models by the way I'm making and I'm going through these papers precisely"}, {"start": 4241.8, "end": 4246.08, "text": " because over the next videos I'll be covering some of the largest ML models"}, {"start": 4246.08, "end": 4252.68, "text": " out there and do let me know if you if you if you have any any suggestions what"}, {"start": 4252.68, "end": 4257.200000000001, "text": " you want to cover or not feel free to comment down in the comments also"}, {"start": 4257.200000000001, "end": 4262.56, "text": " congrats if you if you sticked up through the whole video until here okay"}, {"start": 4262.56, "end": 4270.240000000001, "text": " so let's do a quick memory analysis to understand why case 12 etc etc so a 1.5"}, {"start": 4270.240000000001, "end": 4275.280000000001, "text": " billion parameter GPT-2 model requires 3 gigabytes of memory for its weights if"}, {"start": 4275.280000000001, "end": 4281.160000000001, "text": " we assume FB 16 or in 16-bit precision yet it cannot be trained on a single"}, {"start": 4281.160000000001, "end": 4287.0, "text": " GPU with 32 gigabytes of memory using tensorflow or PyTorch at least without"}, {"start": 4287.0, "end": 4290.64, "text": " any of the optimizations I assume that PyTorch and tensorflow now have a bunch"}, {"start": 4290.64, "end": 4296.200000000001, "text": " of optimizations included one may wonder where all the memory goes so now let's"}, {"start": 4296.200000000001, "end": 4299.92, "text": " see where it goes let's take Adam as a concrete example so the mixed precision"}, {"start": 4299.92, "end": 4304.360000000001, "text": " training of a model with psi parameters using Adam requires enough memory to"}, {"start": 4304.360000000001, "end": 4309.8, "text": " hold an FB 16 copy of the parameters and gradients with memory requirements of"}, {"start": 4309.8, "end": 4313.320000000001, "text": " two psi and two psi bytes respectively so we've seen that one in the table"}, {"start": 4313.320000000001, "end": 4319.4800000000005, "text": " before basically yeah so in addition it needs to hold the optimizer states and"}, {"start": 4319.48, "end": 4324.44, "text": " FB 32 copy of the parameters so we know why this is the case because of the"}, {"start": 4324.44, "end": 4329.44, "text": " mixed precision training we always need to store the master copy in FB 32 okay"}, {"start": 4329.44, "end": 4334.44, "text": " and that's kind of gonna bloat up the memory but it's necessary the momentum"}, {"start": 4334.44, "end": 4340.36, "text": " and the variance so these are Adam idiosyncrasies and then with memory"}, {"start": 4340.36, "end": 4345.879999999999, "text": " requirements of 4 psi 4 psi and 4 psi bytes so 4 because here we have FB 32"}, {"start": 4345.88, "end": 4350.2, "text": " okay and because of that if you sum them up you end up with 12 and that's why"}, {"start": 4350.2, "end": 4356.32, "text": " K equals 12 so in total this results in 16 psi bytes of memory requirements so"}, {"start": 4356.32, "end": 4362.2, "text": " yeah that's why you cannot train a GPT 1.5b on the GPU unless you have some of"}, {"start": 4362.2, "end": 4366.4400000000005, "text": " these optimizations okay now they break up the residual memory consumption blah"}, {"start": 4366.4400000000005, "end": 4369.84, "text": " blah blah they mention for some set of parameters like if you're using GPT 2"}, {"start": 4369.84, "end": 4376.6, "text": " with a sequence length of 1000 that size of 32 you require 60 gigabytes of memory"}, {"start": 4376.6, "end": 4380.72, "text": " unless you do activation checkpoint then they say here activation checkpoint here"}, {"start": 4380.72, "end": 4384.24, "text": " or activation computation or computation is a common approach to reduce the"}, {"start": 4384.24, "end": 4388.52, "text": " creation memory by approximately the square root of the total activations at"}, {"start": 4388.52, "end": 4393.72, "text": " the expense of 33% of recomputation overhead this would reduce the activation"}, {"start": 4393.72, "end": 4399.24, "text": " memory consumption of this model to about 8 gigabytes that's a huge saving"}, {"start": 4399.24, "end": 4404.12, "text": " okay when the size of the model is large these temporary buffer sizes are"}, {"start": 4404.12, "end": 4407.679999999999, "text": " non-trivial okay so they go on to say that there are three components to the"}, {"start": 4407.679999999999, "end": 4411.639999999999, "text": " residual number consumption so the activations the temporary buffers and"}, {"start": 4411.639999999999, "end": 4417.48, "text": " the memory fragmentation yeah I'm gonna skip this one it's not that vital but"}, {"start": 4417.48, "end": 4421.08, "text": " yeah you you kind of have to have these temporary buffers when you want to do"}, {"start": 4421.08, "end": 4428.12, "text": " some reductions etc etc yeah that's too much detail for this video okay let's"}, {"start": 4428.12, "end": 4432.92, "text": " get some hints why this works zero DP partitions the model states instead of"}, {"start": 4432.92, "end": 4438.08, "text": " replicating them and uses a dynamic communication schedule that exploits the"}, {"start": 4438.08, "end": 4442.96, "text": " intrinsically temporal nature of the model states while minimizing the"}, {"start": 4442.96, "end": 4449.32, "text": " communication volume okay so let that sink in the intrinsically temporal"}, {"start": 4449.32, "end": 4454.04, "text": " nature so let me quickly go back to the diagram here I think that's gonna be the"}, {"start": 4454.04, "end": 4459.88, "text": " easiest thing to do so the thing is let's say let's take parameters as an"}, {"start": 4459.88, "end": 4464.64, "text": " example so the the blue the blue line here literally when you're doing a"}, {"start": 4464.64, "end": 4470.04, "text": " forward pass obviously the forward pass is always at a certain point at a"}, {"start": 4470.04, "end": 4474.44, "text": " certain layer in your model and the only parameters you care about are the"}, {"start": 4474.44, "end": 4479.96, "text": " parameters that are needed to do the immediately next computation right so"}, {"start": 4479.96, "end": 4482.88, "text": " you can kind of discard everything before and you don't care about"}, {"start": 4482.88, "end": 4487.96, "text": " everything that's about to come so they literally explore that temporal thingy"}, {"start": 4487.96, "end": 4492.76, "text": " and they in a smart way communicate the parameters the gradients the optimizer"}, {"start": 4492.76, "end": 4500.28, "text": " states such that you have basically an illusion that everything is already"}, {"start": 4500.28, "end": 4506.52, "text": " present there and they kind of mask the the overhead by the well because of"}, {"start": 4506.52, "end": 4512.96, "text": " certain intrinsic latencies during the model training I'm not gonna go into"}, {"start": 4512.96, "end": 4519.360000000001, "text": " much more detail there so let's go back here let's go back here we are here I"}, {"start": 4519.360000000001, "end": 4523.160000000001, "text": " think okay a couple more things and I think we're done here the arithmetic"}, {"start": 4523.160000000001, "end": 4527.0, "text": " dense intensity ratio of the amount of computation per iteration to amount of"}, {"start": 4527.0, "end": 4532.0, "text": " activation checkpoints per iteration is very large and increases linearly with"}, {"start": 4532.0, "end": 4536.400000000001, "text": " hidden dimension making it possible to hide so this is the thing I just said"}, {"start": 4536.4, "end": 4541.96, "text": " the data movement cost for the activation checkpoints even when the"}, {"start": 4541.96, "end": 4546.42, "text": " bandwidth is low for very large model zero can even choose to move the"}, {"start": 4546.42, "end": 4550.96, "text": " activation partitions to the CPU memory and then you literally have like a zero"}, {"start": 4550.96, "end": 4557.92, "text": " cost of memory older and and in case you manage to mask that transitioning the"}, {"start": 4557.92, "end": 4565.08, "text": " movement of the data to the CPU then you literally like you get it for free so"}, {"start": 4565.08, "end": 4569.6, "text": " unfortunately the paper is very textual hopefully you got some insight from the"}, {"start": 4569.6, "end": 4573.48, "text": " diagrams and the thing I things I explained so far let me try and read you"}, {"start": 4573.48, "end": 4577.88, "text": " a couple more things I highlighted with which I think are insightful so for a"}, {"start": 4577.88, "end": 4582.6, "text": " DP degree again DP is standing for a data parallelism of ND we group the"}, {"start": 4582.6, "end": 4588.48, "text": " optimizer states into ND equal partitions okay such that the I data"}, {"start": 4588.48, "end": 4593.16, "text": " parallel process only updates the optimizer states corresponding to the"}, {"start": 4593.16, "end": 4598.92, "text": " I partition thus each data parallel process only needs to store and update"}, {"start": 4598.92, "end": 4605.36, "text": " one over ND of the total optimizer states and then only update one over ND"}, {"start": 4605.36, "end": 4610.76, "text": " of the parameters we perform an all gatherer across the data parallel"}, {"start": 4610.76, "end": 4616.48, "text": " process at the end of each step to get the fully updated parameters across all"}, {"start": 4616.48, "end": 4621.9, "text": " data parallel process okay so it's kind of hard to explain everything I'm going"}, {"start": 4621.9, "end": 4626.48, "text": " to show you one thing here so there's this nice Vicky page where you can see"}, {"start": 4626.48, "end": 4631.719999999999, "text": " these collective operations such as reduce or reduce blah blah blah all"}, {"start": 4631.719999999999, "end": 4638.44, "text": " gather scatter all to all all these are implemented in libraries such as NCCL"}, {"start": 4638.44, "end": 4642.24, "text": " from from Nvidia so that stands for Nvidia collective communications"}, {"start": 4642.24, "end": 4646.0, "text": " library so you can kind of check it out the diagrams here are useful it's gonna"}, {"start": 4646.0, "end": 4651.679999999999, "text": " help you kind of understand this but ultimately someone needs to create maybe"}, {"start": 4651.68, "end": 4657.4400000000005, "text": " to me something that visualizes this much like nicely such it so they can see"}, {"start": 4657.4400000000005, "end": 4660.240000000001, "text": " understand what's going on but yeah hopefully this gives you some intuition"}, {"start": 4660.240000000001, "end": 4664.56, "text": " and that's what we are aiming for just like have at least the basic terminology"}, {"start": 4664.56, "end": 4667.92, "text": " and the basic understanding of all of these optimization methods in place and"}, {"start": 4667.92, "end": 4671.400000000001, "text": " then you can always read on the documentation of the particular library"}, {"start": 4671.400000000001, "end": 4678.64, "text": " you're using if you care to learn more okay I'm gonna skip all of these here"}, {"start": 4678.64, "end": 4684.04, "text": " interesting piece of theory but not that practical they show that because of we"}, {"start": 4684.04, "end": 4688.76, "text": " saw these formulas already in the table but we ultimately end up with 16 psi"}, {"start": 4688.76, "end": 4694.84, "text": " over MD formula for the amount of memory which means is Andy in the limit of"}, {"start": 4694.84, "end": 4699.56, "text": " going to infinity this goes to zero but yeah it becomes impractical body this"}, {"start": 4699.56, "end": 4703.6, "text": " shows that this has a profound implication zero powers DP to fit model"}, {"start": 4703.6, "end": 4707.400000000001, "text": " with arbitrary size as long as there are sufficient number of devices to share"}, {"start": 4707.4, "end": 4710.799999999999, "text": " the model states but yeah it becomes after a certain point you hear the"}, {"start": 4710.799999999999, "end": 4716.04, "text": " situation point and this becomes very very slow okay this is the last last"}, {"start": 4716.04, "end": 4720.44, "text": " part let me kind of read and walk you through this it might be insightful so"}, {"start": 4720.44, "end": 4724.0, "text": " more specifically once the forward propagation for a layer of a model is"}, {"start": 4724.0, "end": 4729.32, "text": " computed the input activations are partitioned across all the model"}, {"start": 4729.32, "end": 4734.94, "text": " parallel process until it is needed again during the back propagation okay"}, {"start": 4734.94, "end": 4741.16, "text": " so what they say here is let me try and cannot draw this down so let's say we"}, {"start": 4741.16, "end": 4747.719999999999, "text": " have a bunch of devices here something like this let's let's say we have three"}, {"start": 4747.719999999999, "end": 4752.4, "text": " devices and let's say you just computed you're just you're currently here in"}, {"start": 4752.4, "end": 4757.32, "text": " this part of the network and you computed some of the activations okay and"}, {"start": 4757.32, "end": 4761.679999999999, "text": " then you proceed forward so after that you proceed forward you continue your"}, {"start": 4761.68, "end": 4767.96, "text": " forward prop here okay so instead of having that checkpoint so what's usually"}, {"start": 4767.96, "end": 4774.240000000001, "text": " done that checkpoint is stored here instead of doing that why not partition"}, {"start": 4774.240000000001, "end": 4779.320000000001, "text": " it and basically why not store the literally change the color so you"}, {"start": 4779.320000000001, "end": 4784.64, "text": " instead store only a chunk here so you only store a chunk here you store the"}, {"start": 4784.64, "end": 4789.4800000000005, "text": " second time here and you store the third time here so that's the idea and then"}, {"start": 4789.48, "end": 4795.48, "text": " once we need these activations again we just communicate back those activations"}, {"start": 4795.48, "end": 4800.08, "text": " back to this device so that we can compute the gradients etc etc so that's"}, {"start": 4800.08, "end": 4803.2, "text": " the basic idea I mean now how that happens obviously there is a lot of"}, {"start": 4803.2, "end": 4805.719999999999, "text": " computer science going on in the background the algorithms data"}, {"start": 4805.719999999999, "end": 4809.44, "text": " structures all of that hardware electronics but yeah we have to kind of"}, {"start": 4809.44, "end": 4814.36, "text": " stop at a certain level of abstraction okay so it works in conjunction with"}, {"start": 4814.36, "end": 4819.639999999999, "text": " activation checkpointing storing partitioned activation checkpoints only"}, {"start": 4819.639999999999, "end": 4823.639999999999, "text": " instead of replicated copy so that's what I told you here so furthermore in"}, {"start": 4823.639999999999, "end": 4828.839999999999, "text": " the case of very large models and very limited device memory these partitioned"}, {"start": 4828.839999999999, "end": 4833.719999999999, "text": " activation checkpoints can also be offloaded to the CPU reducing the"}, {"start": 4833.719999999999, "end": 4838.16, "text": " activation memory overhead to nearly zero at an additional communication cost I"}, {"start": 4838.16, "end": 4842.08, "text": " didn't mention this before okay if we check point a single activation for each"}, {"start": 4842.08, "end": 4847.04, "text": " transformer layer it will require about 33 gigabytes of memory per GPU just to"}, {"start": 4847.04, "end": 4852.0, "text": " store the activation checkpoints but with this optimization the PA and zero"}, {"start": 4852.0, "end": 4857.48, "text": " it can be reduced to about 2 gigabytes per GPU because yeah if you have 16"}, {"start": 4857.48, "end": 4861.08, "text": " degree of 16 of your model parallelism then you just divide by 16 you get"}, {"start": 4861.08, "end": 4865.6, "text": " roughly 2 gigabytes furthermore this 2 gigabytes can be offloaded to the CPU"}, {"start": 4865.6, "end": 4870.32, "text": " reducing the memory footprint for activations to nearly zero okay guys this"}, {"start": 4870.32, "end": 4876.2, "text": " is this is pretty much it the paper is huge obviously there is a lot of"}, {"start": 4876.2, "end": 4880.4, "text": " heuristics in the background so they mentioned something here so given model"}, {"start": 4880.4, "end": 4883.4, "text": " and hardware characteristics will leverage the above analysis to decide if"}, {"start": 4883.4, "end": 4888.5199999999995, "text": " and when to apply PA and PA plus CPU so the activation checkpointing the"}, {"start": 4888.5199999999995, "end": 4892.4, "text": " partitioning of the activation checkpoints optimization and the CPU"}, {"start": 4892.4, "end": 4896.48, "text": " offloading optimization and the above analysis is something I encourage you"}, {"start": 4896.48, "end": 4900.5199999999995, "text": " to read but the idea is depending on like various latencies they decide"}, {"start": 4900.5199999999995, "end": 4904.48, "text": " whether this is a good idea or not so there is a lot of details there but"}, {"start": 4904.48, "end": 4909.28, "text": " nothing conceptually fundamental that that yeah it will lead you to an Eureka"}, {"start": 4909.28, "end": 4916.719999999999, "text": " moment to epiphany I guess yeah okay guys hopefully you like this video let"}, {"start": 4916.719999999999, "end": 4921.24, "text": " me find let me end up with this nice chart here okay so hopefully you like"}, {"start": 4921.24, "end": 4925.5599999999995, "text": " this video if you did please share it out with with your friends with anyone"}, {"start": 4925.56, "end": 4930.92, "text": " for whom you think would benefit from understanding these scaling techniques so"}, {"start": 4930.92, "end": 4934.96, "text": " we covered a lot of ground here we covered the data parallelism approach we"}, {"start": 4934.96, "end": 4938.4800000000005, "text": " covered the tensor parallelism approach we covered the pipeline parallelism"}, {"start": 4938.4800000000005, "end": 4942.360000000001, "text": " approach we covered the mixed precision training and finally we covered the zero"}, {"start": 4942.360000000001, "end": 4947.280000000001, "text": " optimizer where we saw how we can basically cut down and do the"}, {"start": 4947.280000000001, "end": 4952.6, "text": " partitioning of various model states and digital states so that we can train much"}, {"start": 4952.6, "end": 4957.96, "text": " bigger models and even on a single GPU so yeah do let me know whether you found"}, {"start": 4957.96, "end": 4962.0, "text": " this video interesting leave any feedback down below subscribe to the"}, {"start": 4962.0, "end": 4982.64, "text": " channel and until next time bye bye"}]