Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (μTransfer)
👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I cover "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (μTransfer)" paper that makes optimal hyperparameters stable w.r.t. width scaling!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2203.03466
✅ Their previous paper: https://arxiv.org/abs/2011.14522
✅ μTransfer tool: https://github.com/microsoft/mup
✅ DeepNet paper: https://arxiv.org/abs/2203.00555
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 uTransfer introduced
04:30 Previous work (tensor programs IV)
10:00 NTK - neural tangent kernel recap
16:30 abc parametrization
19:25 How does learning happen in NTK?
33:50 Connections to Central Limit Theorem
39:30 Maximal Update Parametrization in Practice
45:50 DeepNet paper connection
48:20 Results (width is all you need?)

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
THE AI EPIPHANY PATREONS WALL ❤️ 

Eli Mahler
Kevin Stone
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#μTransfer #MaximalUpdateParametrization #scalingl
 What's cracking guys? In this video I'm covering Tensor Programs 5 Tuning Large Neural Networks by a Zero-Shot Hyperparameter Transfer by Greg Young, Edward Hu and others from Microsoft and OpenAI. As a quick note, I've been uploading somewhat less regularly over the last month and a half. Basically, I've been moving around London and a lot of other stuff has been going on. But going forward, basically I'm going to be much more consistent and also a very cool series is coming up, so stay tuned for that one. Without further ado, let's jump back to this paper. It's one of the most interesting theoretical and practical papers I've read in a while. There's a bunch of theory going behind it, as you can tell by this Roman numeral 5 here. This is basically a fifth paper in this sequence of papers with Tensor Programs papers. There's a bunch of theory. I'm going to try and give you the necessary background theories, although it's going to be a lot of hand-waving, but hopefully you're going to get the gist of it. Let's see what's the main punchline of the paper. We showed that in the recently discovered maximal update parameterization, so do remember this concept. I'm going to try and explain what exactly this thing is in this video. So many optimal HPs, hyperparameters, remain stable even as model size changes. Okay? So then they go and say, by transferring pre-training HPs from a model of 13 million parameters, we outperform published numbers of birth large 350 million parameters with a total tuning cost equivalent to pre-training birth large ones. So this is quite a big deal. Basically, what they do, they have a nice diagram here. They take a huge model, like let's say birth large or GPT-3, then they scale it down, as you can see on the diagram here. They tune this smaller model, so they do the parameter tuning on the smaller models, which is, as you can tell, much more cost-effective, time-effective, and easier. It's easier to just kind of sweep the search space. And then what they show is that by doing this microparametrization, the thing is those optimal hyperparameters are going to stay optimal even for this much bigger model. So you can see here, they can scale it up to the original size, just apply the hyperparameters they found on the smaller model, and it's going to be optimal for the bigger model as well. So that's the punchline, and that's super promising, because so far, as you may know, they usually just kind of find hyperparameters directly on the bigger model, and it's nowhere near often exhaustive search, so it's suboptimal. Here are some other diagrams where they show that, so on the x-axis, we have learning rate. On the y-axis, we have training loss, and we can see that as we are changing widths of our neural networks, basically for the standard parameterization, you can see that the optimum is shifting here. So that means that depending on the size of your model, your optimal learning rate is going to be different. And here on the right-hand side, you can see that for this novel parameterization, once you find the optimal learning rate, it stays pretty much constant with width for all of these models, and you can see that progressively as we add more and more neurons along the width dimension, we get better performance, i.e., a lower training loss. Here, it's not quite the same situation, because you can see that this model that has 2,000 parameters is actually the optimal model here as the lowest loss, so we are not benefiting from having bigger models. Now, there is a gotcha here. So they actually have a theoretical guarantee that this is going to be the case when you're trying to scale the model along the width dimension. But here, Greg Yang, one of the main authors of the paper, says in his Twitter thread, weight can shrink the model only in width, and he says, bad news, there is not much theoretical guarantee for non-width stuff, but good news is we empirically tested transfer across depth, batch size, sequence length, and timestamp, and all of those work within reasonable ranges on pre-LN transformers. Okay, so let's understand what kind of source you realize behind this paper, and for that purpose, I have a bunch of snippets from their last paper, the fourth paper in this series of papers, and let's see some definitions and understand what's going on and what type of parameterization is this microparameterization thingy. Okay, so they say here, this paper studies a natural class of parameterizations, which we call the ABC parameterization and describe here. Consider an L hidden layer perception. For weight matrices, W1, which has N times D dimensions, whereas D is basically your input dimensionality, and N is the hidden layer dimensionality. Then we have layers W2 through WL, which are all of the N times N dimensionality, which means we keep the same number of neurons across each of the hidden layers. We have a non-linearity phi, just a scalar like mapping from real numbers to real numbers, so that's your ReLU, sigmoid, 10H, whatnot. Such a neural network on input, Xi, which is D dimensional, as I mentioned, is given by this H1, and you just get that by multiplying W1 with the input. So your matrix, you multiply your input with your weights of your first layer. Those give you the preactivations, and then after you apply the activation function here, you get the activations, which they denote with X. L just stands as the index into the layer in this deep MLP. Basically, in order to get the next layer preactivations, you just multiply your activations from your last layer with the next layer's weights here. And they say that the network output, also called logits, is denoted as F of Xi, and that's basically, finally, you just apply on the last activations, you apply this linear mapping, and you get your logits, and that's it. So now, ABC parameterization is specified by a set of numbers, AL, BL tuples. So we have for each of the layers in the MLP, we're going to have one of these tuples. And then we have this number C, which is going to be the same across the whole network, because it's going to modulate the learning rate, as we're going to see in a second. So let's understand why ABC, what does ABC mean. And ABC basically comes from these three statements here. So we have the A statement, B statement, and C statement. The first one says, we parameterize each weight, so WL, as N raised to the power of minus AL WL, where WLs are the actual learnable, trainable parameters, and N is just basically the width of your layer, and this is just a constant. So that means this is going to be for a given network architecture, and for a given set of ALs and these parameters, this is going to be constant, and this is learnable, so W is learnable. So that's the A part of this parameterization. The B part is we initialize each weight, so they just index using alpha and beta. I'm not sure why they're not using something like I and J, which is much more common, but yeah, I guess fancy Greek symbols are cool. So they initialize that as a Gaussian that has zero mean, and it has a variance of N raised to the power of minus 2 BL. So that's where the BL parameters come into play. And finally, the statement C here, the SGD learning rate is this nu times N minus C. Let me delete this so you can see it a bit better. So if I delete this, you can see we have nu times N raised to the power of minus C for some width independent nu. So that's like the master. I think they kind of call that the master learning rate. Now, as you can understand, this ABC parameterization is very general. So depending on how you set these parameters, you can get various different behaviors in this limit. And this whole paper is going to be about this infinite width limit analysis. So keep that in mind. So here they mention a couple of instantiations, like specific instantiations of this ABC parameterization. So one of those is NTK parameterization. So that's neural tangent kernel. And I'm fairly sure most of you don't know this. So I'm going to do a small digression and explain what it is, because it's fairly important to understand NTK, to understand the theory behind this paper. So NTK has you set A1 to 0, you set AL to 1 half for all of the outer layers. So the first layer will have A1 equals to 0, and then all of the outer layers are going to have AL equals to 0.5. BL equals 0 for all L, and then C equals 0. Similarly, setting some other set of As, Bs, and Cs, you get this mean field parameterization, which is not that important for our discussion at the moment. And we have also standard parameterization, whereby standard, they mean just the thing that's currently implemented as a default in PyTorch. And I guess TensorFlow has pretty much the same parameterization. OK, a brief detour into NTK. Bear with me. It's going to be, this whole paper is a bit tougher when it comes to theory, but hopefully it's going to pay off and you'll understand stuff a bit more after this video. So what's NTK all about? NTK is basically this thing where people notice that if you have like a network with increasingly bigger and bigger width, what happens is during the training, as you're trying to learn on a particular dataset, people observe that the weights of the neural network are changing very, very slightly. And extrapolating from there, they understood that we can basically linearize our neural network, which is highly unlinear, by just doing up like a Taylor expansion and just keeping the first term of the Taylor expansion. So you basically end up with something like this. So you have your highly nonlinear neural network F and it's basically you feed the input X and you parameterize it by some weights W. So if you now, if we were to now just do the Taylor expansion, we'll get this. We'll get F of X at, we're going to expand it at this, these weights W0, which are basically our initialization weights. So now we add the first term, which is the first order derivatives, basically. So plus gradient of F at this same point W0. So basically gradients of F with respect to neural network parameters. And we need to multiply this with W minus W0. And we just have to transpose this because this is going to be obviously a vector. And so this is a row vector. This is a column vector and we get our results. So Taylor expansion in and of itself will now have higher order derivatives. So basically you'll now have a Hessian here and then third order derivatives, et cetera, et cetera. But if in certain conditions you can ignore all of the other terms and this will be a good enough approximation. And so you've basically managed to linearize your neural network. The approximation becomes better and better as the width goes to infinity. So we have smaller and smaller approximation error. So where does the kernel come into play? Why a neural tangent kernel? So the trick is the following. So if you know your linear regression and if you notice here, this thing, this whole thing here, since we have a set of W0. So W0 is kind of constant. We initialize an approach somehow. And so this is going to map X into some representation. So this is going to be your F of X, Phi of X, sorry. And basically in linear regression, you usually just have like W's here. We have minus W0, but it doesn't change anything. So we end up with a linear regression problem. Obviously the features are nonlinear, but this is still called a linear regression because basically it's linear in the parameters. W. The kernel comes into play if you do. So kernel is basically if you take Phi of X and you do an outer product with Phi of X transposed. So why would you want to do that is because sometimes it's too expensive to calculate these Phi of X's. And so people just use directly this kernel. And so this is your neural tangent kernel. And I'm going to break it down a little bit more now. So neural tangent kernel because, so kernel because it's a kernel, neural because we are basically using neural network to form these nonlinear transformations of our input X. So that's the neural part. And then we have the tangent part. The tangent part is because we are basically creating this linear model, the tangent model of a neural network. Now the reason I mentioned this neural tangent kernel, the reason I did this regression is because it turns out that during this neural, when we're in this NTK regime, the network F is changing. So we are learning something. But the thing is the weights and the features are not changing. And that may be super counterintuitive because it is. And there is a whole snippet I'm going to explain in a couple of minutes here that is going to kind of give you a rough intuition for why this is happening. And because of that, precisely, we don't want to be in this kernel regime because we usually want to do transfer learning. And in order to do transfer learning, you need to learn some useful weights. You need to modify the weights and then transfer them to other tasks. So that's some hand wavy explanation here. I'm going to dig into a bit more details. But to just kind of drive home the point of how this thing works quickly, if we have a one dimensional function. So let's assume we have a one dimensional function here, something like this. Let me zoom in a little bit more. The reason why this Taylor approximation works is the following. So this is your neural network. Imagine this is your F, just a single dimensional case, but still it's your nonlinear function. And you're trying to to basically describe this function using just these two terms. So what you end up doing is if we have if this is w. So this here is w zero. We're basically doing the following thing. So we are describing the network as. So this is F of w zero F of w zero, and then we just add this term here, which is the derivative of this function at this point. So this is basically this this this tangent here. And then if you want to and you can see that for very close points, so for the points are very close to two w zero. This approximation that I've just done. So this tangent is going to be fairly close to the actual function. So let me just change the color here to use green instead. You can see the green approximates our function very well when we are in the in the in the neighborhood in the Epsilon neighborhood of w w zero. OK, that's enough with NTK. Let's get back to the last subroutine. And that's explaining ABC parameterization. OK, so it turns out and this is a simple simplistic representation of this of this ABC parameterization space. But it turns out roughly that we want to be so we want to find ABC parameterization such that we have certain properties. So first of all, we don't want to be in the part of the space where the network training is unstable in this infinite width limit. And that's this whole space, this gray area here. So what an unstable means in this context is that the logits, so the last basically pre activations of the of the network of the output of the network start diverging. So they start going towards infinity in this in this regime. So that's bad. We want to have stable converging logits. Next up, it's not enough that we just like are the logits are not exploding. We also don't want to be in the trivial regime. Trivial would mean, OK, imagine you have logits are all like set to zero and you're learning your training your network, but nothing is changing. The weights are just kind of frozen to zero. And that's stable. OK, because the logits are not exploding. But obviously it's trivial because we're not learning anything smart. So we don't want to be in that regime either. So we end up with being inside of this of this polygon shape part of the space. And again, this is what I explained. This is the kernel regime where we are actually not learning any any useful features. So that means we can do any useful transfer learning. And so we don't want to be here either. And I want to explain why, as I said, I'm going to give you some intuition why how is this possible that we are learning F but not learning features. It's kind of confusing. I'm still trying to grasp this theory completely myself. I'm not an expert here, but like hopefully going to get a better feeling after this video. So what we end up so we end up wanting to be in this part of the space. So on these brown lines here, as you can see here. So these are the lines where we have this feature learning regime where basically we're stable. So the logits are not exploding. We are doing something non trivial. So we are learning features and we also want to be learning the features as much as possible. But just below like the divergence mode where we were the logical start exploding. So that's this maximal update parameterization point. So that's the apex of this of this polygon here. It turns out that the classical the standard parameterization, as they call it here, is actually unstable in the infinite width limit, which is the reason why we have that the instability of the optimal hyperparameters as we are changing the width dimension. OK, so now brace yourselves. I'm going to try and explain this the intuition behind why why is this possible? How can we be learning function F and not learning the actual features? So the features are not changing, but the F is changing. OK, let's see what's going on there. And this is from the appendix of their fourth paper, which I'm going to link down in the video description. So how does the function change if the limit does not allow features to evolve? Then how does learning occur to answer this question? Note that Delta and I'm going to explain all of these symbols and what they mean in a second. So Delta F T T means we are we are looking at function F after T steps of gradient of SGD of stochastic gradient descent. Delta means that we are basically looking at the difference between F T and F zero. So F zero is your neural network. Sorry, your your logits F represents the logits of your network during the initialization. So zero steps of SGD. That's the initialization. So that's equal to V zero, blah, blah, blah, blah, blah. And before I go there, let me just paint a picture here. What is we what is what is all of this? So basically they're just looking at an MLP. And so we have something like this. So we have an MLP. So this is your input. We have an MLP, a hidden layer, and we have another projection here. And here are the logits. This matrix here is called V. This matrix here is called U. And that's all we care about. So now let's see. Let's get back here. Yeah. One thing I forgot to mention. The activations here are called X. So now we have our terminology, our symbols grounded into the physical network here. OK, so equals V zero. And this this very fancy expression where they they take so easier is the the the weights of that network. We during initialization, then they have XT, which are the the activations here basically because it's delta. That's going to be XT minus X zero. And then we have plus a couple of other terms. So instead of me jabbering here, let me just expand this so that we understand what's going on. We're going to end up with the following. So we zero. XT minus X zero. OK, so that's the first term here. Then we have plus we have this part here. So we have VT minus V zero times X zero. And then we have this this this last term, which is basically VT minus V zero times XT. Minus X zero. And if we were to just kind of cancel out the terms here, what we end up with here very trivially is this. So we have VT XT minus V zero. X zero. And now the semantics behind this expression is the following. So here we take the matrix at initialization, so matrix F. So the weights at initialization, we multiply them with the signal here with the activations at initialization. And we subtract that from the same the same thing just at step T. So after T steps of SGD. And so this is kind of quantifying how are the logits of our network evolving throughout training? OK, so we want this to change. We want to to to if we are learning something, this thing is changing. It's evolving. OK, let's continue here. In short, then the evolution of FT of psi where psi is the the input. In the NTK limit is predominantly due to V zero delta XT and delta VT X zero. So why is that? That's because they show that this delta is not changing by a lot in the NTK regime. So it's a very small quantity. And that means that this when you have two small quantities multiplied with one another, this is going to rapidly go down to zero. And so it doesn't it does not contribute to the evolution of our logits. OK, next up, they give a concrete example. So let's imagine now that T equals one. So we've applied only a single SGD update to our network and we get the following expression. So we get delta F one is going to be this fancy expression. So all of these and raised to the power of minus two AV blah, blah, blah comes from our ABC parameterization. If you recall correctly, basically by just plugging in the ABC parameterization into this expression here, we get the following three terms. So the details here don't don't don't don't worry about the details. Let's just see the big picture here. So in NTP, so that's the neural tangent parameterization. Every AV equals one half. So the term and raised to the power of blah, blah, blah is theta of one. OK, confusing piece of notation here as well. I'm going to I'm going to try and dissect that as well. So basically, that means that this term here is state of one, whatever that means. And it roughly means that the this random vector, if you take the coordinate, the standard deviation is roughly theta of one. So that means in the limit, it's going to be upper bounded and lower bounded by by some constant function. Then they see on the other hand, this expression here and raised to the power of blah, blah, blah. So that's that's this expression here is of O of one over square root of N. OK, so that means that as N grows, this is going to go to zero. It's going to converge to zero. And finally, they say likewise for this last term. So for this term here, they kind of manage to reform it into this different shape here. And they show that this is also theta of one. No, sorry, that's the term C actually. So all in all, this term here is going to be theta one over N. So that's this here. And because we have a sum of N elements that's going to cancel out and this is also theta of one. And so we end up in the grand scheme of things, we end up that this term is also theta of one. So we end up with the following. We end up with this being theta of one, this being theta one and this being one over square root of N, which means this is going to go to zero. So in summary, even though this delta X as N is approaching infinity is converging towards zero, which means that with big enough width of a network, even after a single step, because we have one here, even after a single step of this, we don't have any change whatsoever in the features. So this X stays the same after a single step. On the other hand, we see that the F, so the logits of the neural network are actually evolving. And that's why we are able to learn the task because of this weird nonlinear relationship. Okay, that was the best I could do for now. Let's not continue. Hopefully you got something out of that. So I mentioned this theta expression. Basically, they define it quite formally here. Basically, what that means is that your vector X and N is kind of unfortunate here. It should be subscript because N means Nth element in the sequence and not X raised to the power of N. So we take the norm from the Nth random vector in our sequence, we square it, we divide it by N, and we take a square root of all of that. And basically this sequence here is theta of N raised to the power of minus A, which means we can find by the usual definition of big theta, we can find constants A and B such that we can bound that sequence of elements here. So how I like to think about it is, and it's a great simplification, is imagine all of the coordinates here are the same. Then if you just can expand this, this is going to be X1, so the first coordinate of your random vector X raised to the power of 2 plus second coordinate. And because they're the same, let's just kind of put X1 here, square. And we'll have, so plus plus plus, so we'll have N of these. So we end up with NX1 squared. And so because we have N here, so this thing here cancels out with this one. And so we end up with something like this. So you end up with the result that the standard deviation is roughly equal to, of a single coordinate of this vector here, X is equal to this. So that's how you can parse this result that delta F of t is theta of 1. Okay, let me show you just a couple more results from that paper. And we're going back to the main paper, which is going to be shorter than the explanation, this explanation I just gave you here. So for L, so for layers L from 0 to big L, which is the depth of your MLP, we say that this WL, so that the weights of a particular layer L is updated maximally if this delta WTL XTL-1, so basically you take the activations from your previous layer at step t and you multiply that with the weights of your L layer at step t of SGD, but you just have the delta here. So basically if this thing has theta of 1 coordinates for some training routine, some time that's greater than or equal to 1, and some input Xi. So what this roughly means is that if you think about what this is, you're multiplying with weight matrix, you're multiplying the activations, you're getting the pre-activations, so you're basically saying that the pre-activations are evolving as the training progresses. And then they find this complex expression here that shows that you need, so the A, Bs and Cs need to respect this complicated equation here if you were to get this maximal update parameterization. They finally state that this is, there is a unique stable ABC parameterization that has this property, and that's what we saw on this diagram here as the apex of this ABC space. Okay, I'm well aware that this was all very hand-wavy. It's hard to explain this amount of math in a single video, but hopefully you got some understanding of this previous thing that came before this paper. So yeah, in a nutshell, we have this ABC parameterization. We somehow find, they somehow find the relationship between A, Bs and Cs such that we have this maximal learning of features and being stable and all of these nice properties. Now it's much easier to just understand what you need to do as a practitioner. You just need to set a couple of numbers, and this is all going to work magically for you in the background. Okay, so let's get back to the paper. So they say here algorithm one is the following. So they first parameterize target model in maximal update parameterization. So that means pick A, Bs and Cs in a particular way. Then you just reduce the model, and because you parameterize your model, you'll know how to change your initialization, you'll know how to change your weight, your parameter coefficients, and you'll know how to change your learning rate. Then you tune a smaller version of your model, and finally you copy the tuned hyperparameters to the target model, to the big model. Because of all of this, you can now have all of your compute budget used to search for the optimal, for the most optimal hyperparameters, and then you can just apply them to the bigger model and have the ABC parameterization, the maximal update parameterization do all the work for you. So let's see what parameters, which hyperparameters are transferable, and they say here, and we saw this already, so initialization ones, parameter multipliers, and optimization related, especially learning rate. Transferred across our, so we see width here, so this means we can transfer these hyperparameters across these different scaling dimensions. So we do have the theoretical guarantee for width, and we already saw that we don't have the same theoretical guarantee for depth or batch size or any of these, but they empirically, so that's what asterisk means here, the empirically shows that this roughly remains true even when scaling across these different, like scaling dimensions such as depth. What is not transferable are regularization techniques such as dropout, weight decay, because intuitive explanation would be the parameterization only cares about the model, not about the actual data set size, and because of that you cannot count in the regularization correctly. Okay, let's continue here. Worth mentioning, they say that in this work, we primarily focus on hyperparameter transfer with respect to training loss. In settings where regularization is not the bottleneck to test performance, as in all of our experiments here, this also translates to efficacy in terms of test loss. Okay, and that's what we saw in all of these diagrams here, so we had the training loss, and we saw that the training loss remains stable, but if the regularization is not, as they say here, if the regularization is not the bottleneck, then basically even for the test, the test loss will stay similarly stable across various scales. Let's continue here, and let me give you some more intuition for why this whole theory works the way it does. So a bit more math ahead, so brace yourself, let's try and take this one. So the central limit theorem says that if x1 through xn are IID samples, so independent identical distributions, so samples from a zero mean unit variance distribution, so this can be arbitrary distribution, it's just important that we have zero mean and unit variance, and then if we form this novel random variable like this, so basically a sum of all of those random samples, divided by the square root of n, where n is the number of the samples, this converges to a standard Gaussian distribution as n goes to infinity. Therefore, we can say that 1 over square root of n is the right order of scaling factor cn, such that cn times x1 through xn converges to something non-trivial. So in contrast, if we were to change cn to 1 over n, this whole thing would converge to zero, or if cn was 1, then this whole thing will blow up in variance as n goes to infinity. So why would we care about this central limit theorem right now, and this sum and all of this? Well, the answer is kind of simple once you understand it. Basically, let's imagine we have an MLP here, so let's imagine we have our hidden layer, and we have our previous layer, and so how the MLP connectivity pattern looks like is we have, as you can see here, a bunch of connections from previous layer to the next layer. And you can see here that this exactly has that same form as this one here. So we have some weights here, so we have x1, x2, all the way to here, x4, and this will be xn in general when we have much more nodes here. Assuming that the inputs are also zero mean and variance 1, then this multiplication between these two will end up having zero mean and finite variance samples here, and then we sum them up, and that's why we, and now once we sum them up, if we don't want this thing to explode or converge to zero, so if we don't want this to go to zero or infinity in the infinite width limit, then we have to scale this with 1 over square root of n. That's how this mathematics translates to what we care about. Okay, let's continue here, so let's see what's going on here. They say now suppose we would like to minimize the function fn of c, which is defined as an expectation across all of these random samples, f, and then as the input we have this expression c times x1 plus all the way through xn. So we're trying to find that c such that we can minimize this function, and then they say if we reparameterize c as alpha over square root of n, for alpha being a real number, then by the central limit theorem we can see that this converges to basically your Gaussian distribution with variance alpha squared. In the case that alpha is equal to 1, then we converge to standard Gaussian here. Basically any other parameterization would lead to this either exploding or going to zero, and that's not something that we care about, it's not interesting, and that's why we have to kind of choose this type of parameterization inspired by the CLT theorem. Finally they say then for sufficiently large n, the optimal alpha, so alpha n, this star means it's optimal, n means we have n of these samples, and that's the argument of this function, so that optimal alpha n star should be close to alpha big n star for any n big n that's bigger than this small n, and like in particular when n is equal to infinity. And this precisely means that we can transfer the optimal, so cn or alpha n, because those two are kind of connected directly here, they're proportional because this is a constant, so we can transfer from the optimal parameter from a smaller problem to a larger problem. Finally, because the transfer algorithm is simply copying alpha, we say that parameterization alpha over square root of n is the correct parameterization for this problem. So what I've basically said here is that this c is some hyperparameter, whatever that hyperparameter is, and with this parameterization it's going to give us the minimum of this function f, even for smaller as well as for the bigger models, so as n goes to this n which goes to infinity. That's a rough explanation, so this is fairly hard to understand even for me who've been reading a bunch of papers over the many last years, and I can't imagine how this might look to beginners, so just having some writing some blog around this whole work would be super, super beneficial, so if the authors see this video, I strongly encourage them to write a blog and try and explain this with less math and with more pictures. It would be super useful, just like this thing I tried to do here basically. Okay, let's continue on and see the rest of the paper. Okay, I think we're through with most of the math, so this part now is going to be hopefully a lot simpler compared to what we've seen previously. Okay, the point they're trying to make throughout this whole paper is that hyperparameters do not transfer conventionally, and we've seen that in the first graphs. We've seen that you cannot just take a set of hyperparameters you found for a small model and just assume they're going to work for the bigger model, and they say that here, so on the other hand, a non-trivial fraction of papers in deep learning fixes all HPs when comparing against baselines, which reflects an assumption that the optimal HPs should be stable, and my comment to that would be basically it reflects a lack of resources and hoping for the best rather than people thinking this is going to work. That's usually the case, I guess. Okay, so having said that, let's now see some more experiments they've done. On the left-hand side here, we can see the MLP, I think a two-hidden layer MLP, trained with the standard parameterization, so SP stands for standard parameterization, and microP stands for maximal. By the way, I have no idea why they're using microP to describe maximal update parameterization. In any case, we can see here that as they're training with different widths, so going from 256 all the way to 8K192, we can see that the optimum when it comes to learning rate is shifting, like literally for an order of magnitude at least here from minus 7 to minus 9, even more. And this is a log scale, so it's multiple orders of magnitude. On the other hand, we can see that for microP, for their method, it's basically stable here. So this graph is MLP, the one we saw above, so this one here was actually transformers, so they do show that for various architectures, this thing actually works in practice. It's not just a theoretical paper without any results. It actually has awesome results to back up all of this theory, which is one of the nicest parts about the paper. Okay, so let's see that famous maximal update parameterization. Here is what you need to do in practice. You initialize your weights in a particular manner, and the only difference compared to your standard parameterization is a couple of weeks. So here in the third layer, so for the third set of weights, you just have to do n squared instead of n. And this is something you regularly see. This is just 1 over fan in, so the number of neurons entering your neuron. This intuitively makes a lot of sense because if you have, basically, if you have, again, a neuron here and you have n neurons going into this neuron here, then if you, so if you have f in, so fan in, number of neurons here, so it makes sense that you want to initialize these weights as 1 over f in, so as to compensate for the sum of this f in element. So that's the intuitive idea for why these methods work, and what they've shown is that these methods do not work as expected in the infinite width limit, so that's why we have the maximal update parameterization here. Finally, the SGD rates are modified per layer. You can see that the only modifications are that for the first layer you multiply by n here, and for the last layer you multiply by 1 over n, which means we have per layer learning rate in this parameterization. So practically it's very simple to actually implement this, and you don't have to care about this because they actually open sourced a package that does this for you. I'm going to link it down in the video description as well so you can check it out. Here are some empirical validations they've done for different hyperparameters and scaling across width and depth. So as I said, for width this whole theory was developed for the infinite width limit, whereas the depth is not guaranteed theoretically, but you can see that for various different hyperparameters like learning rate schedule and learning rate and these initializations, standard deviation, etc. we can see that the curves are fairly stable across all of these diagrams. I guess this is the only thing that kind of deviates compared to the width analog here, so it doesn't have a clear minimum as up here. Finally, one interesting fact is that, so you're all familiar, I guess, with transformers and how the actual attention mechanism works, and you know that in the original paper they used to divide by square root of D so to avoid the explosion and keep the variance controlled, and it turns out that in this infinite width analysis you actually want to divide by D, not by square root of D, which is a super interesting result considering that if you think, like, the original transformer paper did give a nice explanation for why square root of D works, and now all of a sudden one over D is actually better in this infinite width limit. Yeah, this whole paper feels like magic, but the best thing of all is that it actually works. It's not just a pure theoretical paper without the results to back it up. In these diagrams they show how logits, attention logits, and word embeddings are stable across various widths, as you can see on the x-axis, for the proposed parameterization versus the standard parameterization, where we can see that logits and attention logits start exploding with width, whereas word embeddings are actually constant, and that's why it's problematic to only try, and I think they try to just manipulate, like, a learning rate, and then this becomes stable, but then this diverges or converges to zero or something, and so it's only, yeah, obviously only their parameterization leads to consistently stable results across all of the widths. Okay, let's continue here and let's see what else is interesting. Okay, this one. The transfer across depth only works for the pre-layer norm transformer, and there was a very cool recent paper called DeepNet that addressed this problem, actually. It introduced this deep norm layer, and they managed to train transformers that are super deep, like, 1,000 plus layers deep, as you can see here, and it's a very interesting chart, because you have on the x-axis chronology, like, time, and then on the y-axis, we have the metric of depth, and they are definitely like new soda when it comes to depth on this benchmark. Yeah, so they mentioned somewhere in the paper the following thing. So these guys here find that the pre-norm residual connections, pre-LN, improve the stability of transformers compared to post-norm connections. However, the gradients of pre-LN at bottom layers tend to be larger than at top layers, leading to a degradation in performance compared with post-LN. So here it was already shown that pre-LN improves stability, whereas it loses some of the performance, and here you can see that exactly that's what they're dealing with here. The transfer across definitely works for pre-LN transformers. So the DeepNet paper shows how to reconcile these two worlds, so to take the best out of the two worlds, the performance and the stability, so the stability of pre-LN and the performance of post-LN module, by just applying a couple of tricks, so just kind of multiplying the signal here by alpha, so we have the residual plus the signal, and this is the pre-LN form, as you can see, because we have the sum here before the LN, that's why it's called pre-LN, and just modifying it with alpha here and initializing some of these weights by beta, they manage to get to amazing depths. And so combining this paper, understanding, totally understanding the theory behind DeepNet paper and this maximal object parameterization sounds like a very interesting research idea that promises both depth and both infinite depth and infinite width, and so StackMoreLayers team is winning again. Okay, guys, I thought that this connection with DeepNet paper was quite valuable, so thought mentioning it. So now let's see the results. There's a bunch of tables, I'm going to focus only on a couple of these, so here we have BERT pre-training, here we can see BERT large, and we can see that the model speedup goes for this microtransfer technique at 22x, which means the smaller model takes 22x less time to train compared to the big model, and all in all, the total speedup for the whole hyperparameter tuning procedure takes hundreds and hundreds of times less time to train than the BERT large, than directly training the big model. Interestingly, we can also see that it transfers perfectly, it even achieves better results than training it directly, and that's the consequence of this amazing theory behind this paper. GPT-3, they showed that if we take the smaller 6.7 billion parameter model and train it with microtransfer, you get better results even compared to the 13 billion size model, and that's the consequence of OpenAI not having enough resources to try out different hyperparameter combinations for such a big model. And now with this technique, you can do much more thorough HP search and then just apply the same hyperparameters to the bigger model, and as you can see, they get pretty much better results across most of these datasets. There are some outliers here for some reason, it performs way worse compared to these two on this Lamberta, few shots, not sure what's going on there. Finally, interestingly enough, the same as what DeepNet did for depth, these guys showed that with increasing width, they always get better and better performance, so lower training loss in this example compared to standard parameterization where it's not stable. You can see that the pattern holds up until a certain point depending on the learning rate, and then given enough width, it's going to explode. So here, because the learning rate is bigger, it explodes even sooner than on the middle chart here. And in another example, another chart showing that wider is better, basically you can see that throughout the whole training, we have the training tokens on the x-axis, so this is plotting during the actual training procedure, and you can see that the validation loss is getting smaller and smaller as we're increasing width, although there is some saturation going on. That's pretty much it, guys. This was a super heavy paper, lots of equations, the previous paper is even harder than this one, there is so much math, and it looks like a super impactful work, and it'd be very useful to make some visualizations and make this a bit easier to understand. But yeah, hopefully you found this explanation somewhat useful, I'm aware it was very hand-wavy and not rigorous, but hopefully some of the mental models of how I think about all of this helped you out. If it did, consider sharing the video out and subscribe to this channel, join the Discord community, and until next time, bye-bye.
[{"start": 0.0, "end": 2.7600000000000002, "text": " What's cracking guys? In this video I'm covering"}, {"start": 2.7600000000000002, "end": 6.26, "text": " Tensor Programs 5 Tuning Large Neural Networks"}, {"start": 6.26, "end": 9.06, "text": " by a Zero-Shot Hyperparameter Transfer"}, {"start": 9.06, "end": 14.06, "text": " by Greg Young, Edward Hu and others from Microsoft and OpenAI."}, {"start": 14.06, "end": 18.26, "text": " As a quick note, I've been uploading somewhat less regularly"}, {"start": 18.26, "end": 21.26, "text": " over the last month and a half."}, {"start": 21.26, "end": 23.26, "text": " Basically, I've been moving around London"}, {"start": 23.26, "end": 25.76, "text": " and a lot of other stuff has been going on."}, {"start": 25.76, "end": 30.76, "text": " But going forward, basically I'm going to be much more consistent"}, {"start": 30.76, "end": 35.760000000000005, "text": " and also a very cool series is coming up, so stay tuned for that one."}, {"start": 35.760000000000005, "end": 38.760000000000005, "text": " Without further ado, let's jump back to this paper."}, {"start": 38.760000000000005, "end": 41.760000000000005, "text": " It's one of the most interesting theoretical and practical papers"}, {"start": 41.760000000000005, "end": 43.760000000000005, "text": " I've read in a while."}, {"start": 43.760000000000005, "end": 46.760000000000005, "text": " There's a bunch of theory going behind it,"}, {"start": 46.760000000000005, "end": 49.760000000000005, "text": " as you can tell by this Roman numeral 5 here."}, {"start": 49.760000000000005, "end": 53.760000000000005, "text": " This is basically a fifth paper in this sequence of papers"}, {"start": 53.76, "end": 55.76, "text": " with Tensor Programs papers."}, {"start": 55.76, "end": 57.76, "text": " There's a bunch of theory."}, {"start": 57.76, "end": 60.76, "text": " I'm going to try and give you the necessary background theories,"}, {"start": 60.76, "end": 62.76, "text": " although it's going to be a lot of hand-waving,"}, {"start": 62.76, "end": 64.75999999999999, "text": " but hopefully you're going to get the gist of it."}, {"start": 64.75999999999999, "end": 68.75999999999999, "text": " Let's see what's the main punchline of the paper."}, {"start": 68.75999999999999, "end": 71.75999999999999, "text": " We showed that in the recently discovered"}, {"start": 71.75999999999999, "end": 73.75999999999999, "text": " maximal update parameterization,"}, {"start": 73.75999999999999, "end": 75.75999999999999, "text": " so do remember this concept."}, {"start": 75.75999999999999, "end": 80.75999999999999, "text": " I'm going to try and explain what exactly this thing is in this video."}, {"start": 80.76, "end": 83.76, "text": " So many optimal HPs, hyperparameters,"}, {"start": 83.76, "end": 87.76, "text": " remain stable even as model size changes."}, {"start": 87.76, "end": 88.76, "text": " Okay?"}, {"start": 88.76, "end": 90.76, "text": " So then they go and say,"}, {"start": 90.76, "end": 95.76, "text": " by transferring pre-training HPs from a model of 13 million parameters,"}, {"start": 95.76, "end": 100.76, "text": " we outperform published numbers of birth large 350 million parameters"}, {"start": 100.76, "end": 105.76, "text": " with a total tuning cost equivalent to pre-training birth large ones."}, {"start": 105.76, "end": 107.76, "text": " So this is quite a big deal."}, {"start": 107.76, "end": 110.76, "text": " Basically, what they do, they have a nice diagram here."}, {"start": 110.76, "end": 115.76, "text": " They take a huge model, like let's say birth large or GPT-3,"}, {"start": 115.76, "end": 119.76, "text": " then they scale it down, as you can see on the diagram here."}, {"start": 119.76, "end": 122.76, "text": " They tune this smaller model,"}, {"start": 122.76, "end": 125.76, "text": " so they do the parameter tuning on the smaller models,"}, {"start": 125.76, "end": 130.76, "text": " which is, as you can tell, much more cost-effective, time-effective,"}, {"start": 130.76, "end": 131.76, "text": " and easier."}, {"start": 131.76, "end": 134.76, "text": " It's easier to just kind of sweep the search space."}, {"start": 134.76, "end": 139.76, "text": " And then what they show is that by doing this microparametrization,"}, {"start": 139.76, "end": 143.76, "text": " the thing is those optimal hyperparameters are going to stay optimal"}, {"start": 143.76, "end": 145.76, "text": " even for this much bigger model."}, {"start": 145.76, "end": 148.76, "text": " So you can see here, they can scale it up to the original size,"}, {"start": 148.76, "end": 153.76, "text": " just apply the hyperparameters they found on the smaller model,"}, {"start": 153.76, "end": 156.76, "text": " and it's going to be optimal for the bigger model as well."}, {"start": 156.76, "end": 160.76, "text": " So that's the punchline, and that's super promising,"}, {"start": 160.76, "end": 162.76, "text": " because so far, as you may know,"}, {"start": 162.76, "end": 166.76, "text": " they usually just kind of find hyperparameters directly on the bigger model,"}, {"start": 166.76, "end": 171.76, "text": " and it's nowhere near often exhaustive search, so it's suboptimal."}, {"start": 171.76, "end": 175.76, "text": " Here are some other diagrams where they show that,"}, {"start": 175.76, "end": 178.76, "text": " so on the x-axis, we have learning rate."}, {"start": 178.76, "end": 181.76, "text": " On the y-axis, we have training loss,"}, {"start": 181.76, "end": 186.76, "text": " and we can see that as we are changing widths of our neural networks,"}, {"start": 186.76, "end": 189.76, "text": " basically for the standard parameterization,"}, {"start": 189.76, "end": 192.76, "text": " you can see that the optimum is shifting here."}, {"start": 192.76, "end": 195.76, "text": " So that means that depending on the size of your model,"}, {"start": 195.76, "end": 198.76, "text": " your optimal learning rate is going to be different."}, {"start": 198.76, "end": 204.76, "text": " And here on the right-hand side, you can see that for this novel parameterization,"}, {"start": 204.76, "end": 206.76, "text": " once you find the optimal learning rate,"}, {"start": 206.76, "end": 211.76, "text": " it stays pretty much constant with width for all of these models,"}, {"start": 211.76, "end": 217.76, "text": " and you can see that progressively as we add more and more neurons along the width dimension,"}, {"start": 217.76, "end": 221.76, "text": " we get better performance, i.e., a lower training loss."}, {"start": 221.76, "end": 223.76, "text": " Here, it's not quite the same situation,"}, {"start": 223.76, "end": 227.76, "text": " because you can see that this model that has 2,000 parameters"}, {"start": 227.76, "end": 231.76, "text": " is actually the optimal model here as the lowest loss,"}, {"start": 231.76, "end": 234.76, "text": " so we are not benefiting from having bigger models."}, {"start": 234.76, "end": 236.76, "text": " Now, there is a gotcha here."}, {"start": 236.76, "end": 241.76, "text": " So they actually have a theoretical guarantee that this is going to be the case"}, {"start": 241.76, "end": 246.76, "text": " when you're trying to scale the model along the width dimension."}, {"start": 246.76, "end": 252.76, "text": " But here, Greg Yang, one of the main authors of the paper, says in his Twitter thread,"}, {"start": 252.76, "end": 254.76, "text": " weight can shrink the model only in width,"}, {"start": 254.76, "end": 259.76, "text": " and he says, bad news, there is not much theoretical guarantee for non-width stuff,"}, {"start": 259.76, "end": 263.76, "text": " but good news is we empirically tested transfer across depth,"}, {"start": 263.76, "end": 266.76, "text": " batch size, sequence length, and timestamp,"}, {"start": 266.76, "end": 271.76, "text": " and all of those work within reasonable ranges on pre-LN transformers."}, {"start": 271.76, "end": 275.76, "text": " Okay, so let's understand what kind of source you realize behind this paper,"}, {"start": 275.76, "end": 279.76, "text": " and for that purpose, I have a bunch of snippets from their last paper,"}, {"start": 279.76, "end": 282.76, "text": " the fourth paper in this series of papers,"}, {"start": 282.76, "end": 285.76, "text": " and let's see some definitions and understand what's going on"}, {"start": 285.76, "end": 291.76, "text": " and what type of parameterization is this microparameterization thingy."}, {"start": 291.76, "end": 296.76, "text": " Okay, so they say here, this paper studies a natural class of parameterizations,"}, {"start": 296.76, "end": 299.76, "text": " which we call the ABC parameterization and describe here."}, {"start": 299.76, "end": 302.76, "text": " Consider an L hidden layer perception."}, {"start": 302.76, "end": 307.76, "text": " For weight matrices, W1, which has N times D dimensions,"}, {"start": 307.76, "end": 311.76, "text": " whereas D is basically your input dimensionality,"}, {"start": 311.76, "end": 316.76, "text": " and N is the hidden layer dimensionality."}, {"start": 316.76, "end": 319.76, "text": " Then we have layers W2 through WL,"}, {"start": 319.76, "end": 322.76, "text": " which are all of the N times N dimensionality,"}, {"start": 322.76, "end": 328.76, "text": " which means we keep the same number of neurons across each of the hidden layers."}, {"start": 328.76, "end": 333.76, "text": " We have a non-linearity phi, just a scalar like mapping from real numbers to real numbers,"}, {"start": 333.76, "end": 337.76, "text": " so that's your ReLU, sigmoid, 10H, whatnot."}, {"start": 337.76, "end": 342.76, "text": " Such a neural network on input, Xi, which is D dimensional, as I mentioned,"}, {"start": 342.76, "end": 348.76, "text": " is given by this H1, and you just get that by multiplying W1 with the input."}, {"start": 348.76, "end": 354.76, "text": " So your matrix, you multiply your input with your weights of your first layer."}, {"start": 354.76, "end": 359.76, "text": " Those give you the preactivations, and then after you apply the activation function here,"}, {"start": 359.76, "end": 362.76, "text": " you get the activations, which they denote with X."}, {"start": 362.76, "end": 368.76, "text": " L just stands as the index into the layer in this deep MLP."}, {"start": 368.76, "end": 371.76, "text": " Basically, in order to get the next layer preactivations,"}, {"start": 371.76, "end": 379.76, "text": " you just multiply your activations from your last layer with the next layer's weights here."}, {"start": 379.76, "end": 386.76, "text": " And they say that the network output, also called logits, is denoted as F of Xi,"}, {"start": 386.76, "end": 390.76, "text": " and that's basically, finally, you just apply on the last activations,"}, {"start": 390.76, "end": 396.76, "text": " you apply this linear mapping, and you get your logits, and that's it."}, {"start": 396.76, "end": 402.76, "text": " So now, ABC parameterization is specified by a set of numbers, AL, BL tuples."}, {"start": 402.76, "end": 407.76, "text": " So we have for each of the layers in the MLP, we're going to have one of these tuples."}, {"start": 407.76, "end": 411.76, "text": " And then we have this number C, which is going to be the same across the whole network,"}, {"start": 411.76, "end": 416.76, "text": " because it's going to modulate the learning rate, as we're going to see in a second."}, {"start": 416.76, "end": 420.76, "text": " So let's understand why ABC, what does ABC mean."}, {"start": 420.76, "end": 423.76, "text": " And ABC basically comes from these three statements here."}, {"start": 423.76, "end": 426.76, "text": " So we have the A statement, B statement, and C statement."}, {"start": 426.76, "end": 434.76, "text": " The first one says, we parameterize each weight, so WL, as N raised to the power of minus AL WL,"}, {"start": 434.76, "end": 438.76, "text": " where WLs are the actual learnable, trainable parameters,"}, {"start": 438.76, "end": 446.76, "text": " and N is just basically the width of your layer, and this is just a constant."}, {"start": 446.76, "end": 450.76, "text": " So that means this is going to be for a given network architecture,"}, {"start": 450.76, "end": 455.76, "text": " and for a given set of ALs and these parameters, this is going to be constant,"}, {"start": 455.76, "end": 458.76, "text": " and this is learnable, so W is learnable."}, {"start": 458.76, "end": 461.76, "text": " So that's the A part of this parameterization."}, {"start": 461.76, "end": 468.76, "text": " The B part is we initialize each weight, so they just index using alpha and beta."}, {"start": 468.76, "end": 473.76, "text": " I'm not sure why they're not using something like I and J, which is much more common,"}, {"start": 473.76, "end": 476.76, "text": " but yeah, I guess fancy Greek symbols are cool."}, {"start": 476.76, "end": 481.76, "text": " So they initialize that as a Gaussian that has zero mean,"}, {"start": 481.76, "end": 485.76, "text": " and it has a variance of N raised to the power of minus 2 BL."}, {"start": 485.76, "end": 489.76, "text": " So that's where the BL parameters come into play."}, {"start": 489.76, "end": 496.76, "text": " And finally, the statement C here, the SGD learning rate is this nu times N minus C."}, {"start": 496.76, "end": 499.76, "text": " Let me delete this so you can see it a bit better."}, {"start": 499.76, "end": 505.76, "text": " So if I delete this, you can see we have nu times N raised to the power of minus C"}, {"start": 505.76, "end": 507.76, "text": " for some width independent nu."}, {"start": 507.76, "end": 509.76, "text": " So that's like the master."}, {"start": 509.76, "end": 513.76, "text": " I think they kind of call that the master learning rate."}, {"start": 513.76, "end": 518.76, "text": " Now, as you can understand, this ABC parameterization is very general."}, {"start": 518.76, "end": 525.76, "text": " So depending on how you set these parameters, you can get various different behaviors in this limit."}, {"start": 525.76, "end": 530.76, "text": " And this whole paper is going to be about this infinite width limit analysis."}, {"start": 530.76, "end": 532.76, "text": " So keep that in mind."}, {"start": 532.76, "end": 537.76, "text": " So here they mention a couple of instantiations,"}, {"start": 537.76, "end": 540.76, "text": " like specific instantiations of this ABC parameterization."}, {"start": 540.76, "end": 543.76, "text": " So one of those is NTK parameterization."}, {"start": 543.76, "end": 545.76, "text": " So that's neural tangent kernel."}, {"start": 545.76, "end": 547.76, "text": " And I'm fairly sure most of you don't know this."}, {"start": 547.76, "end": 550.76, "text": " So I'm going to do a small digression and explain what it is,"}, {"start": 550.76, "end": 556.76, "text": " because it's fairly important to understand NTK, to understand the theory behind this paper."}, {"start": 556.76, "end": 563.76, "text": " So NTK has you set A1 to 0, you set AL to 1 half for all of the outer layers."}, {"start": 563.76, "end": 566.76, "text": " So the first layer will have A1 equals to 0,"}, {"start": 566.76, "end": 571.76, "text": " and then all of the outer layers are going to have AL equals to 0.5."}, {"start": 571.76, "end": 576.76, "text": " BL equals 0 for all L, and then C equals 0."}, {"start": 576.76, "end": 581.76, "text": " Similarly, setting some other set of As, Bs, and Cs, you get this mean field parameterization,"}, {"start": 581.76, "end": 584.76, "text": " which is not that important for our discussion at the moment."}, {"start": 584.76, "end": 592.76, "text": " And we have also standard parameterization, whereby standard, they mean just the thing that's currently implemented as a default in PyTorch."}, {"start": 592.76, "end": 598.76, "text": " And I guess TensorFlow has pretty much the same parameterization."}, {"start": 598.76, "end": 601.76, "text": " OK, a brief detour into NTK."}, {"start": 601.76, "end": 602.76, "text": " Bear with me."}, {"start": 602.76, "end": 606.76, "text": " It's going to be, this whole paper is a bit tougher when it comes to theory,"}, {"start": 606.76, "end": 612.76, "text": " but hopefully it's going to pay off and you'll understand stuff a bit more after this video."}, {"start": 612.76, "end": 615.76, "text": " So what's NTK all about?"}, {"start": 615.76, "end": 625.76, "text": " NTK is basically this thing where people notice that if you have like a network with increasingly bigger and bigger width,"}, {"start": 625.76, "end": 629.76, "text": " what happens is during the training, as you're trying to learn on a particular dataset,"}, {"start": 629.76, "end": 635.76, "text": " people observe that the weights of the neural network are changing very, very slightly."}, {"start": 635.76, "end": 642.76, "text": " And extrapolating from there, they understood that we can basically linearize our neural network,"}, {"start": 642.76, "end": 651.76, "text": " which is highly unlinear, by just doing up like a Taylor expansion and just keeping the first term of the Taylor expansion."}, {"start": 651.76, "end": 653.76, "text": " So you basically end up with something like this."}, {"start": 653.76, "end": 665.76, "text": " So you have your highly nonlinear neural network F and it's basically you feed the input X and you parameterize it by some weights W."}, {"start": 665.76, "end": 669.76, "text": " So if you now, if we were to now just do the Taylor expansion, we'll get this."}, {"start": 669.76, "end": 679.76, "text": " We'll get F of X at, we're going to expand it at this, these weights W0, which are basically our initialization weights."}, {"start": 679.76, "end": 683.76, "text": " So now we add the first term, which is the first order derivatives, basically."}, {"start": 683.76, "end": 692.76, "text": " So plus gradient of F at this same point W0."}, {"start": 692.76, "end": 698.76, "text": " So basically gradients of F with respect to neural network parameters."}, {"start": 698.76, "end": 703.76, "text": " And we need to multiply this with W minus W0."}, {"start": 703.76, "end": 707.76, "text": " And we just have to transpose this because this is going to be obviously a vector."}, {"start": 707.76, "end": 710.76, "text": " And so this is a row vector."}, {"start": 710.76, "end": 713.76, "text": " This is a column vector and we get our results."}, {"start": 713.76, "end": 716.76, "text": " So Taylor expansion in and of itself will now have higher order derivatives."}, {"start": 716.76, "end": 722.76, "text": " So basically you'll now have a Hessian here and then third order derivatives, et cetera, et cetera."}, {"start": 722.76, "end": 731.76, "text": " But if in certain conditions you can ignore all of the other terms and this will be a good enough approximation."}, {"start": 731.76, "end": 735.76, "text": " And so you've basically managed to linearize your neural network."}, {"start": 735.76, "end": 740.76, "text": " The approximation becomes better and better as the width goes to infinity."}, {"start": 740.76, "end": 743.76, "text": " So we have smaller and smaller approximation error."}, {"start": 743.76, "end": 746.76, "text": " So where does the kernel come into play?"}, {"start": 746.76, "end": 748.76, "text": " Why a neural tangent kernel?"}, {"start": 748.76, "end": 750.76, "text": " So the trick is the following."}, {"start": 750.76, "end": 761.76, "text": " So if you know your linear regression and if you notice here, this thing, this whole thing here, since we have a set of W0."}, {"start": 761.76, "end": 762.76, "text": " So W0 is kind of constant."}, {"start": 762.76, "end": 764.76, "text": " We initialize an approach somehow."}, {"start": 764.76, "end": 770.76, "text": " And so this is going to map X into some representation."}, {"start": 770.76, "end": 775.76, "text": " So this is going to be your F of X, Phi of X, sorry."}, {"start": 775.76, "end": 779.76, "text": " And basically in linear regression, you usually just have like W's here."}, {"start": 779.76, "end": 783.76, "text": " We have minus W0, but it doesn't change anything."}, {"start": 783.76, "end": 786.76, "text": " So we end up with a linear regression problem."}, {"start": 786.76, "end": 793.76, "text": " Obviously the features are nonlinear, but this is still called a linear regression because basically it's linear in the parameters."}, {"start": 793.76, "end": 796.76, "text": " W. The kernel comes into play if you do."}, {"start": 796.76, "end": 807.76, "text": " So kernel is basically if you take Phi of X and you do an outer product with Phi of X transposed."}, {"start": 807.76, "end": 813.76, "text": " So why would you want to do that is because sometimes it's too expensive to calculate these Phi of X's."}, {"start": 813.76, "end": 816.76, "text": " And so people just use directly this kernel."}, {"start": 816.76, "end": 820.76, "text": " And so this is your neural tangent kernel."}, {"start": 820.76, "end": 822.76, "text": " And I'm going to break it down a little bit more now."}, {"start": 822.76, "end": 835.76, "text": " So neural tangent kernel because, so kernel because it's a kernel, neural because we are basically using neural network to form these nonlinear transformations of our input X."}, {"start": 835.76, "end": 837.76, "text": " So that's the neural part."}, {"start": 837.76, "end": 838.76, "text": " And then we have the tangent part."}, {"start": 838.76, "end": 845.76, "text": " The tangent part is because we are basically creating this linear model, the tangent model of a neural network."}, {"start": 845.76, "end": 859.76, "text": " Now the reason I mentioned this neural tangent kernel, the reason I did this regression is because it turns out that during this neural, when we're in this NTK regime, the network F is changing."}, {"start": 859.76, "end": 860.76, "text": " So we are learning something."}, {"start": 860.76, "end": 864.76, "text": " But the thing is the weights and the features are not changing."}, {"start": 864.76, "end": 868.76, "text": " And that may be super counterintuitive because it is."}, {"start": 868.76, "end": 876.76, "text": " And there is a whole snippet I'm going to explain in a couple of minutes here that is going to kind of give you a rough intuition for why this is happening."}, {"start": 876.76, "end": 885.76, "text": " And because of that, precisely, we don't want to be in this kernel regime because we usually want to do transfer learning."}, {"start": 885.76, "end": 891.76, "text": " And in order to do transfer learning, you need to learn some useful weights."}, {"start": 891.76, "end": 894.76, "text": " You need to modify the weights and then transfer them to other tasks."}, {"start": 894.76, "end": 896.76, "text": " So that's some hand wavy explanation here."}, {"start": 896.76, "end": 898.76, "text": " I'm going to dig into a bit more details."}, {"start": 898.76, "end": 906.76, "text": " But to just kind of drive home the point of how this thing works quickly, if we have a one dimensional function."}, {"start": 906.76, "end": 912.76, "text": " So let's assume we have a one dimensional function here, something like this."}, {"start": 912.76, "end": 913.76, "text": " Let me zoom in a little bit more."}, {"start": 913.76, "end": 918.76, "text": " The reason why this Taylor approximation works is the following."}, {"start": 918.76, "end": 920.76, "text": " So this is your neural network."}, {"start": 920.76, "end": 926.76, "text": " Imagine this is your F, just a single dimensional case, but still it's your nonlinear function."}, {"start": 926.76, "end": 932.76, "text": " And you're trying to to basically describe this function using just these two terms."}, {"start": 932.76, "end": 937.76, "text": " So what you end up doing is if we have if this is w."}, {"start": 937.76, "end": 939.76, "text": " So this here is w zero."}, {"start": 939.76, "end": 941.76, "text": " We're basically doing the following thing."}, {"start": 941.76, "end": 944.76, "text": " So we are describing the network as."}, {"start": 944.76, "end": 956.76, "text": " So this is F of w zero F of w zero, and then we just add this term here, which is the derivative of this function at this point."}, {"start": 956.76, "end": 959.76, "text": " So this is basically this this this tangent here."}, {"start": 959.76, "end": 968.76, "text": " And then if you want to and you can see that for very close points, so for the points are very close to two w zero."}, {"start": 968.76, "end": 970.76, "text": " This approximation that I've just done."}, {"start": 970.76, "end": 973.76, "text": " So this tangent is going to be fairly close to the actual function."}, {"start": 973.76, "end": 976.76, "text": " So let me just change the color here to use green instead."}, {"start": 976.76, "end": 986.76, "text": " You can see the green approximates our function very well when we are in the in the in the neighborhood in the Epsilon neighborhood of w w zero."}, {"start": 986.76, "end": 987.76, "text": " OK, that's enough with NTK."}, {"start": 987.76, "end": 990.76, "text": " Let's get back to the last subroutine."}, {"start": 990.76, "end": 994.76, "text": " And that's explaining ABC parameterization."}, {"start": 994.76, "end": 1002.76, "text": " OK, so it turns out and this is a simple simplistic representation of this of this ABC parameterization space."}, {"start": 1002.76, "end": 1011.76, "text": " But it turns out roughly that we want to be so we want to find ABC parameterization such that we have certain properties."}, {"start": 1011.76, "end": 1018.76, "text": " So first of all, we don't want to be in the part of the space where the network training is unstable in this infinite width limit."}, {"start": 1018.76, "end": 1021.76, "text": " And that's this whole space, this gray area here."}, {"start": 1021.76, "end": 1034.76, "text": " So what an unstable means in this context is that the logits, so the last basically pre activations of the of the network of the output of the network start diverging."}, {"start": 1034.76, "end": 1037.76, "text": " So they start going towards infinity in this in this regime."}, {"start": 1037.76, "end": 1043.76, "text": " So that's bad. We want to have stable converging logits."}, {"start": 1043.76, "end": 1053.76, "text": " Next up, it's not enough that we just like are the logits are not exploding. We also don't want to be in the trivial regime."}, {"start": 1053.76, "end": 1062.76, "text": " Trivial would mean, OK, imagine you have logits are all like set to zero and you're learning your training your network, but nothing is changing."}, {"start": 1062.76, "end": 1064.76, "text": " The weights are just kind of frozen to zero."}, {"start": 1064.76, "end": 1067.76, "text": " And that's stable. OK, because the logits are not exploding."}, {"start": 1067.76, "end": 1073.76, "text": " But obviously it's trivial because we're not learning anything smart. So we don't want to be in that regime either."}, {"start": 1073.76, "end": 1078.76, "text": " So we end up with being inside of this of this polygon shape part of the space."}, {"start": 1078.76, "end": 1081.76, "text": " And again, this is what I explained."}, {"start": 1081.76, "end": 1088.76, "text": " This is the kernel regime where we are actually not learning any any useful features."}, {"start": 1088.76, "end": 1090.76, "text": " So that means we can do any useful transfer learning."}, {"start": 1090.76, "end": 1092.76, "text": " And so we don't want to be here either."}, {"start": 1092.76, "end": 1099.76, "text": " And I want to explain why, as I said, I'm going to give you some intuition why how is this possible that we are learning F but not learning features."}, {"start": 1099.76, "end": 1103.76, "text": " It's kind of confusing. I'm still trying to grasp this theory completely myself."}, {"start": 1103.76, "end": 1108.76, "text": " I'm not an expert here, but like hopefully going to get a better feeling after this video."}, {"start": 1108.76, "end": 1114.76, "text": " So what we end up so we end up wanting to be in this part of the space."}, {"start": 1114.76, "end": 1117.76, "text": " So on these brown lines here, as you can see here."}, {"start": 1117.76, "end": 1124.76, "text": " So these are the lines where we have this feature learning regime where basically we're stable."}, {"start": 1124.76, "end": 1126.76, "text": " So the logits are not exploding."}, {"start": 1126.76, "end": 1128.76, "text": " We are doing something non trivial."}, {"start": 1128.76, "end": 1135.76, "text": " So we are learning features and we also want to be learning the features as much as possible."}, {"start": 1135.76, "end": 1140.76, "text": " But just below like the divergence mode where we were the logical start exploding."}, {"start": 1140.76, "end": 1145.76, "text": " So that's this maximal update parameterization point."}, {"start": 1145.76, "end": 1148.76, "text": " So that's the apex of this of this polygon here."}, {"start": 1148.76, "end": 1157.76, "text": " It turns out that the classical the standard parameterization, as they call it here, is actually unstable in the infinite width limit,"}, {"start": 1157.76, "end": 1166.76, "text": " which is the reason why we have that the instability of the optimal hyperparameters as we are changing the width dimension."}, {"start": 1166.76, "end": 1176.76, "text": " OK, so now brace yourselves. I'm going to try and explain this the intuition behind why why is this possible?"}, {"start": 1176.76, "end": 1182.76, "text": " How can we be learning function F and not learning the actual features?"}, {"start": 1182.76, "end": 1185.76, "text": " So the features are not changing, but the F is changing."}, {"start": 1185.76, "end": 1188.76, "text": " OK, let's see what's going on there."}, {"start": 1188.76, "end": 1193.76, "text": " And this is from the appendix of their fourth paper, which I'm going to link down in the video description."}, {"start": 1193.76, "end": 1199.76, "text": " So how does the function change if the limit does not allow features to evolve?"}, {"start": 1199.76, "end": 1203.76, "text": " Then how does learning occur to answer this question?"}, {"start": 1203.76, "end": 1208.76, "text": " Note that Delta and I'm going to explain all of these symbols and what they mean in a second."}, {"start": 1208.76, "end": 1220.76, "text": " So Delta F T T means we are we are looking at function F after T steps of gradient of SGD of stochastic gradient descent."}, {"start": 1220.76, "end": 1226.76, "text": " Delta means that we are basically looking at the difference between F T and F zero."}, {"start": 1226.76, "end": 1229.76, "text": " So F zero is your neural network."}, {"start": 1229.76, "end": 1237.76, "text": " Sorry, your your logits F represents the logits of your network during the initialization."}, {"start": 1237.76, "end": 1240.76, "text": " So zero steps of SGD. That's the initialization."}, {"start": 1240.76, "end": 1243.76, "text": " So that's equal to V zero, blah, blah, blah, blah, blah."}, {"start": 1243.76, "end": 1250.76, "text": " And before I go there, let me just paint a picture here. What is we what is what is all of this?"}, {"start": 1250.76, "end": 1253.76, "text": " So basically they're just looking at an MLP."}, {"start": 1253.76, "end": 1257.76, "text": " And so we have something like this. So we have an MLP."}, {"start": 1257.76, "end": 1263.76, "text": " So this is your input. We have an MLP, a hidden layer, and we have another projection here."}, {"start": 1263.76, "end": 1268.76, "text": " And here are the logits. This matrix here is called V."}, {"start": 1268.76, "end": 1273.76, "text": " This matrix here is called U. And that's all we care about."}, {"start": 1273.76, "end": 1276.76, "text": " So now let's see. Let's get back here."}, {"start": 1276.76, "end": 1281.76, "text": " Yeah. One thing I forgot to mention. The activations here are called X."}, {"start": 1281.76, "end": 1287.76, "text": " So now we have our terminology, our symbols grounded into the physical network here."}, {"start": 1287.76, "end": 1295.76, "text": " OK, so equals V zero. And this this very fancy expression where they they take so easier is the the the weights of that network."}, {"start": 1295.76, "end": 1305.76, "text": " We during initialization, then they have XT, which are the the activations here basically because it's delta."}, {"start": 1305.76, "end": 1308.76, "text": " That's going to be XT minus X zero."}, {"start": 1308.76, "end": 1311.76, "text": " And then we have plus a couple of other terms."}, {"start": 1311.76, "end": 1317.76, "text": " So instead of me jabbering here, let me just expand this so that we understand what's going on."}, {"start": 1317.76, "end": 1320.76, "text": " We're going to end up with the following. So we zero."}, {"start": 1320.76, "end": 1326.76, "text": " XT minus X zero. OK, so that's the first term here."}, {"start": 1326.76, "end": 1329.76, "text": " Then we have plus we have this part here."}, {"start": 1329.76, "end": 1337.76, "text": " So we have VT minus V zero times X zero."}, {"start": 1337.76, "end": 1350.76, "text": " And then we have this this this last term, which is basically VT minus V zero times XT."}, {"start": 1350.76, "end": 1358.76, "text": " Minus X zero. And if we were to just kind of cancel out the terms here, what we end up with here very trivially is this."}, {"start": 1358.76, "end": 1369.76, "text": " So we have VT XT minus V zero. X zero."}, {"start": 1369.76, "end": 1372.76, "text": " And now the semantics behind this expression is the following."}, {"start": 1372.76, "end": 1378.76, "text": " So here we take the matrix at initialization, so matrix F."}, {"start": 1378.76, "end": 1385.76, "text": " So the weights at initialization, we multiply them with the signal here with the activations at initialization."}, {"start": 1385.76, "end": 1391.76, "text": " And we subtract that from the same the same thing just at step T."}, {"start": 1391.76, "end": 1400.76, "text": " So after T steps of SGD. And so this is kind of quantifying how are the logits of our network evolving throughout training?"}, {"start": 1400.76, "end": 1406.76, "text": " OK, so we want this to change. We want to to to if we are learning something, this thing is changing."}, {"start": 1406.76, "end": 1408.76, "text": " It's evolving. OK, let's continue here."}, {"start": 1408.76, "end": 1414.76, "text": " In short, then the evolution of FT of psi where psi is the the input."}, {"start": 1414.76, "end": 1423.76, "text": " In the NTK limit is predominantly due to V zero delta XT and delta VT X zero."}, {"start": 1423.76, "end": 1430.76, "text": " So why is that? That's because they show that this delta is not changing by a lot in the NTK regime."}, {"start": 1430.76, "end": 1432.76, "text": " So it's a very small quantity."}, {"start": 1432.76, "end": 1438.76, "text": " And that means that this when you have two small quantities multiplied with one another, this is going to rapidly go down to zero."}, {"start": 1438.76, "end": 1446.76, "text": " And so it doesn't it does not contribute to the evolution of our logits."}, {"start": 1446.76, "end": 1448.76, "text": " OK, next up, they give a concrete example."}, {"start": 1448.76, "end": 1451.76, "text": " So let's imagine now that T equals one."}, {"start": 1451.76, "end": 1455.76, "text": " So we've applied only a single SGD update to our network and we get the following expression."}, {"start": 1455.76, "end": 1460.76, "text": " So we get delta F one is going to be this fancy expression."}, {"start": 1460.76, "end": 1468.76, "text": " So all of these and raised to the power of minus two AV blah, blah, blah comes from our ABC parameterization."}, {"start": 1468.76, "end": 1476.76, "text": " If you recall correctly, basically by just plugging in the ABC parameterization into this expression here, we get the following three terms."}, {"start": 1476.76, "end": 1480.76, "text": " So the details here don't don't don't don't worry about the details."}, {"start": 1480.76, "end": 1483.76, "text": " Let's just see the big picture here."}, {"start": 1483.76, "end": 1486.76, "text": " So in NTP, so that's the neural tangent parameterization."}, {"start": 1486.76, "end": 1494.76, "text": " Every AV equals one half. So the term and raised to the power of blah, blah, blah is theta of one."}, {"start": 1494.76, "end": 1496.76, "text": " OK, confusing piece of notation here as well."}, {"start": 1496.76, "end": 1499.76, "text": " I'm going to I'm going to try and dissect that as well."}, {"start": 1499.76, "end": 1504.76, "text": " So basically, that means that this term here is state of one, whatever that means."}, {"start": 1504.76, "end": 1512.76, "text": " And it roughly means that the this random vector, if you take the coordinate, the standard deviation is roughly theta of one."}, {"start": 1512.76, "end": 1518.76, "text": " So that means in the limit, it's going to be upper bounded and lower bounded by by some constant function."}, {"start": 1518.76, "end": 1524.76, "text": " Then they see on the other hand, this expression here and raised to the power of blah, blah, blah."}, {"start": 1524.76, "end": 1531.76, "text": " So that's that's this expression here is of O of one over square root of N."}, {"start": 1531.76, "end": 1535.76, "text": " OK, so that means that as N grows, this is going to go to zero."}, {"start": 1535.76, "end": 1537.76, "text": " It's going to converge to zero."}, {"start": 1537.76, "end": 1540.76, "text": " And finally, they say likewise for this last term."}, {"start": 1540.76, "end": 1546.76, "text": " So for this term here, they kind of manage to reform it into this different shape here."}, {"start": 1546.76, "end": 1549.76, "text": " And they show that this is also theta of one."}, {"start": 1549.76, "end": 1551.76, "text": " No, sorry, that's the term C actually."}, {"start": 1551.76, "end": 1556.76, "text": " So all in all, this term here is going to be theta one over N."}, {"start": 1556.76, "end": 1564.76, "text": " So that's this here. And because we have a sum of N elements that's going to cancel out and this is also theta of one."}, {"start": 1564.76, "end": 1570.76, "text": " And so we end up in the grand scheme of things, we end up that this term is also theta of one."}, {"start": 1570.76, "end": 1573.76, "text": " So we end up with the following."}, {"start": 1573.76, "end": 1581.76, "text": " We end up with this being theta of one, this being theta one and this being one over square root of N, which means this is going to go to zero."}, {"start": 1581.76, "end": 1590.76, "text": " So in summary, even though this delta X as N is approaching infinity is converging towards zero,"}, {"start": 1590.76, "end": 1596.76, "text": " which means that with big enough width of a network, even after a single step, because we have one here,"}, {"start": 1596.76, "end": 1602.76, "text": " even after a single step of this, we don't have any change whatsoever in the features."}, {"start": 1602.76, "end": 1607.76, "text": " So this X stays the same after a single step."}, {"start": 1607.76, "end": 1612.76, "text": " On the other hand, we see that the F, so the logits of the neural network are actually evolving."}, {"start": 1612.76, "end": 1618.76, "text": " And that's why we are able to learn the task because of this weird nonlinear relationship."}, {"start": 1618.76, "end": 1621.76, "text": " Okay, that was the best I could do for now."}, {"start": 1621.76, "end": 1623.76, "text": " Let's not continue."}, {"start": 1623.76, "end": 1625.76, "text": " Hopefully you got something out of that."}, {"start": 1625.76, "end": 1628.76, "text": " So I mentioned this theta expression."}, {"start": 1628.76, "end": 1631.76, "text": " Basically, they define it quite formally here."}, {"start": 1631.76, "end": 1637.76, "text": " Basically, what that means is that your vector X and N is kind of unfortunate here."}, {"start": 1637.76, "end": 1642.76, "text": " It should be subscript because N means Nth element in the sequence and not X raised to the power of N."}, {"start": 1642.76, "end": 1650.76, "text": " So we take the norm from the Nth random vector in our sequence, we square it, we divide it by N,"}, {"start": 1650.76, "end": 1653.76, "text": " and we take a square root of all of that."}, {"start": 1653.76, "end": 1663.76, "text": " And basically this sequence here is theta of N raised to the power of minus A,"}, {"start": 1663.76, "end": 1668.76, "text": " which means we can find by the usual definition of big theta,"}, {"start": 1668.76, "end": 1674.76, "text": " we can find constants A and B such that we can bound that sequence of elements here."}, {"start": 1674.76, "end": 1680.76, "text": " So how I like to think about it is, and it's a great simplification,"}, {"start": 1680.76, "end": 1683.76, "text": " is imagine all of the coordinates here are the same."}, {"start": 1683.76, "end": 1687.76, "text": " Then if you just can expand this, this is going to be X1,"}, {"start": 1687.76, "end": 1693.76, "text": " so the first coordinate of your random vector X raised to the power of 2 plus second coordinate."}, {"start": 1693.76, "end": 1697.76, "text": " And because they're the same, let's just kind of put X1 here, square."}, {"start": 1697.76, "end": 1701.76, "text": " And we'll have, so plus plus plus, so we'll have N of these."}, {"start": 1701.76, "end": 1705.76, "text": " So we end up with NX1 squared."}, {"start": 1705.76, "end": 1709.76, "text": " And so because we have N here, so this thing here cancels out with this one."}, {"start": 1709.76, "end": 1711.76, "text": " And so we end up with something like this."}, {"start": 1711.76, "end": 1715.76, "text": " So you end up with the result that the standard deviation is roughly equal to,"}, {"start": 1715.76, "end": 1719.76, "text": " of a single coordinate of this vector here, X is equal to this."}, {"start": 1719.76, "end": 1727.76, "text": " So that's how you can parse this result that delta F of t is theta of 1."}, {"start": 1727.76, "end": 1730.76, "text": " Okay, let me show you just a couple more results from that paper."}, {"start": 1730.76, "end": 1735.76, "text": " And we're going back to the main paper, which is going to be shorter than the explanation,"}, {"start": 1735.76, "end": 1737.76, "text": " this explanation I just gave you here."}, {"start": 1737.76, "end": 1744.76, "text": " So for L, so for layers L from 0 to big L, which is the depth of your MLP,"}, {"start": 1744.76, "end": 1756.76, "text": " we say that this WL, so that the weights of a particular layer L is updated maximally if this delta WTL XTL-1,"}, {"start": 1756.76, "end": 1761.76, "text": " so basically you take the activations from your previous layer at step t"}, {"start": 1761.76, "end": 1768.76, "text": " and you multiply that with the weights of your L layer at step t of SGD,"}, {"start": 1768.76, "end": 1770.76, "text": " but you just have the delta here."}, {"start": 1770.76, "end": 1776.76, "text": " So basically if this thing has theta of 1 coordinates for some training routine,"}, {"start": 1776.76, "end": 1781.76, "text": " some time that's greater than or equal to 1, and some input Xi."}, {"start": 1781.76, "end": 1785.76, "text": " So what this roughly means is that if you think about what this is,"}, {"start": 1785.76, "end": 1789.76, "text": " you're multiplying with weight matrix, you're multiplying the activations,"}, {"start": 1789.76, "end": 1795.76, "text": " you're getting the pre-activations, so you're basically saying that the pre-activations are evolving as the training progresses."}, {"start": 1795.76, "end": 1799.76, "text": " And then they find this complex expression here that shows that you need,"}, {"start": 1799.76, "end": 1804.76, "text": " so the A, Bs and Cs need to respect this complicated equation here"}, {"start": 1804.76, "end": 1811.76, "text": " if you were to get this maximal update parameterization."}, {"start": 1811.76, "end": 1816.76, "text": " They finally state that this is, there is a unique stable ABC parameterization that has this property,"}, {"start": 1816.76, "end": 1823.76, "text": " and that's what we saw on this diagram here as the apex of this ABC space."}, {"start": 1823.76, "end": 1828.76, "text": " Okay, I'm well aware that this was all very hand-wavy."}, {"start": 1828.76, "end": 1831.76, "text": " It's hard to explain this amount of math in a single video,"}, {"start": 1831.76, "end": 1838.76, "text": " but hopefully you got some understanding of this previous thing that came before this paper."}, {"start": 1838.76, "end": 1843.76, "text": " So yeah, in a nutshell, we have this ABC parameterization."}, {"start": 1843.76, "end": 1850.76, "text": " We somehow find, they somehow find the relationship between A, Bs and Cs"}, {"start": 1850.76, "end": 1858.76, "text": " such that we have this maximal learning of features and being stable and all of these nice properties."}, {"start": 1858.76, "end": 1863.76, "text": " Now it's much easier to just understand what you need to do as a practitioner."}, {"start": 1863.76, "end": 1868.76, "text": " You just need to set a couple of numbers, and this is all going to work magically for you in the background."}, {"start": 1868.76, "end": 1871.76, "text": " Okay, so let's get back to the paper."}, {"start": 1871.76, "end": 1874.76, "text": " So they say here algorithm one is the following."}, {"start": 1874.76, "end": 1878.76, "text": " So they first parameterize target model in maximal update parameterization."}, {"start": 1878.76, "end": 1882.76, "text": " So that means pick A, Bs and Cs in a particular way."}, {"start": 1882.76, "end": 1887.76, "text": " Then you just reduce the model, and because you parameterize your model,"}, {"start": 1887.76, "end": 1891.76, "text": " you'll know how to change your initialization, you'll know how to change your weight,"}, {"start": 1891.76, "end": 1896.76, "text": " your parameter coefficients, and you'll know how to change your learning rate."}, {"start": 1896.76, "end": 1899.76, "text": " Then you tune a smaller version of your model,"}, {"start": 1899.76, "end": 1903.76, "text": " and finally you copy the tuned hyperparameters to the target model, to the big model."}, {"start": 1903.76, "end": 1910.76, "text": " Because of all of this, you can now have all of your compute budget used to search for the optimal,"}, {"start": 1910.76, "end": 1916.76, "text": " for the most optimal hyperparameters, and then you can just apply them to the bigger model"}, {"start": 1916.76, "end": 1921.76, "text": " and have the ABC parameterization, the maximal update parameterization do all the work for you."}, {"start": 1921.76, "end": 1927.76, "text": " So let's see what parameters, which hyperparameters are transferable, and they say here,"}, {"start": 1927.76, "end": 1932.76, "text": " and we saw this already, so initialization ones, parameter multipliers, and optimization related,"}, {"start": 1932.76, "end": 1934.76, "text": " especially learning rate."}, {"start": 1934.76, "end": 1941.76, "text": " Transferred across our, so we see width here, so this means we can transfer these hyperparameters"}, {"start": 1941.76, "end": 1944.76, "text": " across these different scaling dimensions."}, {"start": 1944.76, "end": 1949.76, "text": " So we do have the theoretical guarantee for width, and we already saw that we don't have"}, {"start": 1949.76, "end": 1953.76, "text": " the same theoretical guarantee for depth or batch size or any of these,"}, {"start": 1953.76, "end": 1957.76, "text": " but they empirically, so that's what asterisk means here,"}, {"start": 1957.76, "end": 1966.76, "text": " the empirically shows that this roughly remains true even when scaling across these different,"}, {"start": 1966.76, "end": 1969.76, "text": " like scaling dimensions such as depth."}, {"start": 1969.76, "end": 1974.76, "text": " What is not transferable are regularization techniques such as dropout, weight decay,"}, {"start": 1974.76, "end": 1979.76, "text": " because intuitive explanation would be the parameterization only cares about the model,"}, {"start": 1979.76, "end": 1985.76, "text": " not about the actual data set size, and because of that you cannot count in the regularization correctly."}, {"start": 1985.76, "end": 1990.76, "text": " Okay, let's continue here."}, {"start": 1990.76, "end": 1996.76, "text": " Worth mentioning, they say that in this work, we primarily focus on hyperparameter transfer"}, {"start": 1996.76, "end": 1998.76, "text": " with respect to training loss."}, {"start": 1998.76, "end": 2002.76, "text": " In settings where regularization is not the bottleneck to test performance,"}, {"start": 2002.76, "end": 2008.76, "text": " as in all of our experiments here, this also translates to efficacy in terms of test loss."}, {"start": 2008.76, "end": 2013.76, "text": " Okay, and that's what we saw in all of these diagrams here, so we had the training loss,"}, {"start": 2013.76, "end": 2018.76, "text": " and we saw that the training loss remains stable, but if the regularization is not,"}, {"start": 2018.76, "end": 2022.76, "text": " as they say here, if the regularization is not the bottleneck,"}, {"start": 2022.76, "end": 2030.76, "text": " then basically even for the test, the test loss will stay similarly stable across various scales."}, {"start": 2030.76, "end": 2038.76, "text": " Let's continue here, and let me give you some more intuition for why this whole theory works the way it does."}, {"start": 2038.76, "end": 2044.76, "text": " So a bit more math ahead, so brace yourself, let's try and take this one."}, {"start": 2044.76, "end": 2054.76, "text": " So the central limit theorem says that if x1 through xn are IID samples, so independent identical distributions,"}, {"start": 2054.76, "end": 2060.76, "text": " so samples from a zero mean unit variance distribution, so this can be arbitrary distribution,"}, {"start": 2060.76, "end": 2063.76, "text": " it's just important that we have zero mean and unit variance,"}, {"start": 2063.76, "end": 2071.76, "text": " and then if we form this novel random variable like this, so basically a sum of all of those random samples,"}, {"start": 2071.76, "end": 2075.76, "text": " divided by the square root of n, where n is the number of the samples,"}, {"start": 2075.76, "end": 2083.76, "text": " this converges to a standard Gaussian distribution as n goes to infinity."}, {"start": 2083.76, "end": 2089.76, "text": " Therefore, we can say that 1 over square root of n is the right order of scaling factor cn,"}, {"start": 2089.76, "end": 2096.76, "text": " such that cn times x1 through xn converges to something non-trivial."}, {"start": 2096.76, "end": 2103.76, "text": " So in contrast, if we were to change cn to 1 over n, this whole thing would converge to zero,"}, {"start": 2103.76, "end": 2109.76, "text": " or if cn was 1, then this whole thing will blow up in variance as n goes to infinity."}, {"start": 2109.76, "end": 2115.76, "text": " So why would we care about this central limit theorem right now, and this sum and all of this?"}, {"start": 2115.76, "end": 2120.76, "text": " Well, the answer is kind of simple once you understand it."}, {"start": 2120.76, "end": 2125.76, "text": " Basically, let's imagine we have an MLP here, so let's imagine we have our hidden layer,"}, {"start": 2125.76, "end": 2131.76, "text": " and we have our previous layer, and so how the MLP connectivity pattern looks like is we have,"}, {"start": 2131.76, "end": 2136.76, "text": " as you can see here, a bunch of connections from previous layer to the next layer."}, {"start": 2136.76, "end": 2143.76, "text": " And you can see here that this exactly has that same form as this one here."}, {"start": 2143.76, "end": 2149.76, "text": " So we have some weights here, so we have x1, x2, all the way to here, x4,"}, {"start": 2149.76, "end": 2154.76, "text": " and this will be xn in general when we have much more nodes here."}, {"start": 2154.76, "end": 2162.76, "text": " Assuming that the inputs are also zero mean and variance 1, then this multiplication between these two"}, {"start": 2162.76, "end": 2170.76, "text": " will end up having zero mean and finite variance samples here, and then we sum them up,"}, {"start": 2170.76, "end": 2179.76, "text": " and that's why we, and now once we sum them up, if we don't want this thing to explode or converge to zero,"}, {"start": 2179.76, "end": 2184.76, "text": " so if we don't want this to go to zero or infinity in the infinite width limit,"}, {"start": 2184.76, "end": 2189.76, "text": " then we have to scale this with 1 over square root of n."}, {"start": 2189.76, "end": 2195.76, "text": " That's how this mathematics translates to what we care about."}, {"start": 2195.76, "end": 2200.76, "text": " Okay, let's continue here, so let's see what's going on here."}, {"start": 2200.76, "end": 2204.76, "text": " They say now suppose we would like to minimize the function fn of c,"}, {"start": 2204.76, "end": 2211.76, "text": " which is defined as an expectation across all of these random samples,"}, {"start": 2211.76, "end": 2220.76, "text": " f, and then as the input we have this expression c times x1 plus all the way through xn."}, {"start": 2220.76, "end": 2225.76, "text": " So we're trying to find that c such that we can minimize this function,"}, {"start": 2225.76, "end": 2231.76, "text": " and then they say if we reparameterize c as alpha over square root of n,"}, {"start": 2231.76, "end": 2238.76, "text": " for alpha being a real number, then by the central limit theorem we can see that this converges to basically"}, {"start": 2238.76, "end": 2242.76, "text": " your Gaussian distribution with variance alpha squared."}, {"start": 2242.76, "end": 2247.76, "text": " In the case that alpha is equal to 1, then we converge to standard Gaussian here."}, {"start": 2247.76, "end": 2253.76, "text": " Basically any other parameterization would lead to this either exploding or going to zero,"}, {"start": 2253.76, "end": 2256.76, "text": " and that's not something that we care about, it's not interesting,"}, {"start": 2256.76, "end": 2261.76, "text": " and that's why we have to kind of choose this type of parameterization inspired by the CLT theorem."}, {"start": 2261.76, "end": 2267.76, "text": " Finally they say then for sufficiently large n, the optimal alpha, so alpha n,"}, {"start": 2267.76, "end": 2273.76, "text": " this star means it's optimal, n means we have n of these samples,"}, {"start": 2273.76, "end": 2281.76, "text": " and that's the argument of this function, so that optimal alpha n star should be close to alpha big n star"}, {"start": 2281.76, "end": 2288.76, "text": " for any n big n that's bigger than this small n, and like in particular when n is equal to infinity."}, {"start": 2288.76, "end": 2294.76, "text": " And this precisely means that we can transfer the optimal, so cn or alpha n,"}, {"start": 2294.76, "end": 2299.76, "text": " because those two are kind of connected directly here, they're proportional because this is a constant,"}, {"start": 2299.76, "end": 2304.76, "text": " so we can transfer from the optimal parameter from a smaller problem to a larger problem."}, {"start": 2304.76, "end": 2308.76, "text": " Finally, because the transfer algorithm is simply copying alpha,"}, {"start": 2308.76, "end": 2313.76, "text": " we say that parameterization alpha over square root of n is the correct parameterization for this problem."}, {"start": 2313.76, "end": 2319.76, "text": " So what I've basically said here is that this c is some hyperparameter, whatever that hyperparameter is,"}, {"start": 2319.76, "end": 2325.76, "text": " and with this parameterization it's going to give us the minimum of this function f,"}, {"start": 2325.76, "end": 2333.76, "text": " even for smaller as well as for the bigger models, so as n goes to this n which goes to infinity."}, {"start": 2333.76, "end": 2343.76, "text": " That's a rough explanation, so this is fairly hard to understand even for me who've been reading a bunch of papers over the many last years,"}, {"start": 2343.76, "end": 2346.76, "text": " and I can't imagine how this might look to beginners,"}, {"start": 2346.76, "end": 2353.76, "text": " so just having some writing some blog around this whole work would be super, super beneficial,"}, {"start": 2353.76, "end": 2362.76, "text": " so if the authors see this video, I strongly encourage them to write a blog and try and explain this with less math and with more pictures."}, {"start": 2362.76, "end": 2367.76, "text": " It would be super useful, just like this thing I tried to do here basically."}, {"start": 2367.76, "end": 2372.76, "text": " Okay, let's continue on and see the rest of the paper."}, {"start": 2372.76, "end": 2380.76, "text": " Okay, I think we're through with most of the math, so this part now is going to be hopefully a lot simpler compared to what we've seen previously."}, {"start": 2380.76, "end": 2387.76, "text": " Okay, the point they're trying to make throughout this whole paper is that hyperparameters do not transfer conventionally,"}, {"start": 2387.76, "end": 2391.76, "text": " and we've seen that in the first graphs."}, {"start": 2391.76, "end": 2402.76, "text": " We've seen that you cannot just take a set of hyperparameters you found for a small model and just assume they're going to work for the bigger model,"}, {"start": 2402.76, "end": 2411.76, "text": " and they say that here, so on the other hand, a non-trivial fraction of papers in deep learning fixes all HPs when comparing against baselines,"}, {"start": 2411.76, "end": 2416.76, "text": " which reflects an assumption that the optimal HPs should be stable,"}, {"start": 2416.76, "end": 2425.76, "text": " and my comment to that would be basically it reflects a lack of resources and hoping for the best rather than people thinking this is going to work."}, {"start": 2425.76, "end": 2427.76, "text": " That's usually the case, I guess."}, {"start": 2427.76, "end": 2434.76, "text": " Okay, so having said that, let's now see some more experiments they've done."}, {"start": 2434.76, "end": 2443.76, "text": " On the left-hand side here, we can see the MLP, I think a two-hidden layer MLP, trained with the standard parameterization,"}, {"start": 2443.76, "end": 2448.76, "text": " so SP stands for standard parameterization, and microP stands for maximal."}, {"start": 2448.76, "end": 2457.76, "text": " By the way, I have no idea why they're using microP to describe maximal update parameterization."}, {"start": 2457.76, "end": 2462.76, "text": " In any case, we can see here that as they're training with different widths,"}, {"start": 2462.76, "end": 2468.76, "text": " so going from 256 all the way to 8K192,"}, {"start": 2468.76, "end": 2473.76, "text": " we can see that the optimum when it comes to learning rate is shifting,"}, {"start": 2473.76, "end": 2479.76, "text": " like literally for an order of magnitude at least here from minus 7 to minus 9, even more."}, {"start": 2479.76, "end": 2483.76, "text": " And this is a log scale, so it's multiple orders of magnitude."}, {"start": 2483.76, "end": 2492.76, "text": " On the other hand, we can see that for microP, for their method, it's basically stable here."}, {"start": 2492.76, "end": 2499.76, "text": " So this graph is MLP, the one we saw above, so this one here was actually transformers,"}, {"start": 2499.76, "end": 2504.76, "text": " so they do show that for various architectures, this thing actually works in practice."}, {"start": 2504.76, "end": 2508.76, "text": " It's not just a theoretical paper without any results."}, {"start": 2508.76, "end": 2516.76, "text": " It actually has awesome results to back up all of this theory, which is one of the nicest parts about the paper."}, {"start": 2516.76, "end": 2521.76, "text": " Okay, so let's see that famous maximal update parameterization."}, {"start": 2521.76, "end": 2524.76, "text": " Here is what you need to do in practice."}, {"start": 2524.76, "end": 2532.76, "text": " You initialize your weights in a particular manner, and the only difference compared to your standard parameterization is a couple of weeks."}, {"start": 2532.76, "end": 2539.76, "text": " So here in the third layer, so for the third set of weights, you just have to do n squared instead of n."}, {"start": 2539.76, "end": 2547.76, "text": " And this is something you regularly see. This is just 1 over fan in, so the number of neurons entering your neuron."}, {"start": 2547.76, "end": 2558.76, "text": " This intuitively makes a lot of sense because if you have, basically, if you have, again, a neuron here and you have n neurons going into this neuron here,"}, {"start": 2558.76, "end": 2569.76, "text": " then if you, so if you have f in, so fan in, number of neurons here, so it makes sense that you want to initialize these weights as 1 over f in,"}, {"start": 2569.76, "end": 2572.76, "text": " so as to compensate for the sum of this f in element."}, {"start": 2572.76, "end": 2581.76, "text": " So that's the intuitive idea for why these methods work, and what they've shown is that these methods do not work as expected in the infinite width limit,"}, {"start": 2581.76, "end": 2585.76, "text": " so that's why we have the maximal update parameterization here."}, {"start": 2585.76, "end": 2589.76, "text": " Finally, the SGD rates are modified per layer."}, {"start": 2589.76, "end": 2598.76, "text": " You can see that the only modifications are that for the first layer you multiply by n here, and for the last layer you multiply by 1 over n,"}, {"start": 2598.76, "end": 2603.76, "text": " which means we have per layer learning rate in this parameterization."}, {"start": 2603.76, "end": 2611.76, "text": " So practically it's very simple to actually implement this, and you don't have to care about this because they actually open sourced a package that does this for you."}, {"start": 2611.76, "end": 2616.76, "text": " I'm going to link it down in the video description as well so you can check it out."}, {"start": 2616.76, "end": 2623.76, "text": " Here are some empirical validations they've done for different hyperparameters and scaling across width and depth."}, {"start": 2623.76, "end": 2628.76, "text": " So as I said, for width this whole theory was developed for the infinite width limit,"}, {"start": 2628.76, "end": 2631.76, "text": " whereas the depth is not guaranteed theoretically,"}, {"start": 2631.76, "end": 2642.76, "text": " but you can see that for various different hyperparameters like learning rate schedule and learning rate and these initializations, standard deviation, etc."}, {"start": 2642.76, "end": 2647.76, "text": " we can see that the curves are fairly stable across all of these diagrams."}, {"start": 2647.76, "end": 2661.76, "text": " I guess this is the only thing that kind of deviates compared to the width analog here, so it doesn't have a clear minimum as up here."}, {"start": 2661.76, "end": 2669.76, "text": " Finally, one interesting fact is that, so you're all familiar, I guess, with transformers and how the actual attention mechanism works,"}, {"start": 2669.76, "end": 2678.76, "text": " and you know that in the original paper they used to divide by square root of D so to avoid the explosion and keep the variance controlled,"}, {"start": 2678.76, "end": 2685.76, "text": " and it turns out that in this infinite width analysis you actually want to divide by D, not by square root of D,"}, {"start": 2685.76, "end": 2694.76, "text": " which is a super interesting result considering that if you think, like, the original transformer paper did give a nice explanation for why square root of D works,"}, {"start": 2694.76, "end": 2699.76, "text": " and now all of a sudden one over D is actually better in this infinite width limit."}, {"start": 2699.76, "end": 2704.76, "text": " Yeah, this whole paper feels like magic, but the best thing of all is that it actually works."}, {"start": 2704.76, "end": 2709.76, "text": " It's not just a pure theoretical paper without the results to back it up."}, {"start": 2709.76, "end": 2716.76, "text": " In these diagrams they show how logits, attention logits, and word embeddings are stable across various widths,"}, {"start": 2716.76, "end": 2724.76, "text": " as you can see on the x-axis, for the proposed parameterization versus the standard parameterization,"}, {"start": 2724.76, "end": 2731.76, "text": " where we can see that logits and attention logits start exploding with width, whereas word embeddings are actually constant,"}, {"start": 2731.76, "end": 2736.76, "text": " and that's why it's problematic to only try, and I think they try to just manipulate, like, a learning rate,"}, {"start": 2736.76, "end": 2741.76, "text": " and then this becomes stable, but then this diverges or converges to zero or something,"}, {"start": 2741.76, "end": 2750.76, "text": " and so it's only, yeah, obviously only their parameterization leads to consistently stable results across all of the widths."}, {"start": 2750.76, "end": 2753.76, "text": " Okay, let's continue here and let's see what else is interesting."}, {"start": 2753.76, "end": 2754.76, "text": " Okay, this one."}, {"start": 2754.76, "end": 2758.76, "text": " The transfer across depth only works for the pre-layer norm transformer,"}, {"start": 2758.76, "end": 2763.76, "text": " and there was a very cool recent paper called DeepNet that addressed this problem, actually."}, {"start": 2763.76, "end": 2769.76, "text": " It introduced this deep norm layer, and they managed to train transformers that are super deep,"}, {"start": 2769.76, "end": 2775.76, "text": " like, 1,000 plus layers deep, as you can see here, and it's a very interesting chart,"}, {"start": 2775.76, "end": 2781.76, "text": " because you have on the x-axis chronology, like, time, and then on the y-axis, we have the metric of depth,"}, {"start": 2781.76, "end": 2786.76, "text": " and they are definitely like new soda when it comes to depth on this benchmark."}, {"start": 2786.76, "end": 2793.76, "text": " Yeah, so they mentioned somewhere in the paper the following thing."}, {"start": 2793.76, "end": 2804.76, "text": " So these guys here find that the pre-norm residual connections, pre-LN, improve the stability of transformers compared to post-norm connections."}, {"start": 2804.76, "end": 2809.76, "text": " However, the gradients of pre-LN at bottom layers tend to be larger than at top layers,"}, {"start": 2809.76, "end": 2813.76, "text": " leading to a degradation in performance compared with post-LN."}, {"start": 2813.76, "end": 2820.76, "text": " So here it was already shown that pre-LN improves stability, whereas it loses some of the performance,"}, {"start": 2820.76, "end": 2824.76, "text": " and here you can see that exactly that's what they're dealing with here."}, {"start": 2824.76, "end": 2829.76, "text": " The transfer across definitely works for pre-LN transformers."}, {"start": 2829.76, "end": 2835.76, "text": " So the DeepNet paper shows how to reconcile these two worlds, so to take the best out of the two worlds,"}, {"start": 2835.76, "end": 2842.76, "text": " the performance and the stability, so the stability of pre-LN and the performance of post-LN module,"}, {"start": 2842.76, "end": 2847.76, "text": " by just applying a couple of tricks, so just kind of multiplying the signal here by alpha,"}, {"start": 2847.76, "end": 2852.76, "text": " so we have the residual plus the signal, and this is the pre-LN form, as you can see,"}, {"start": 2852.76, "end": 2856.76, "text": " because we have the sum here before the LN, that's why it's called pre-LN,"}, {"start": 2856.76, "end": 2864.76, "text": " and just modifying it with alpha here and initializing some of these weights by beta,"}, {"start": 2864.76, "end": 2868.76, "text": " they manage to get to amazing depths."}, {"start": 2868.76, "end": 2875.76, "text": " And so combining this paper, understanding, totally understanding the theory behind DeepNet paper"}, {"start": 2875.76, "end": 2881.76, "text": " and this maximal object parameterization sounds like a very interesting research idea"}, {"start": 2881.76, "end": 2886.76, "text": " that promises both depth and both infinite depth and infinite width,"}, {"start": 2886.76, "end": 2889.76, "text": " and so StackMoreLayers team is winning again."}, {"start": 2889.76, "end": 2894.76, "text": " Okay, guys, I thought that this connection with DeepNet paper was quite valuable,"}, {"start": 2894.76, "end": 2897.76, "text": " so thought mentioning it."}, {"start": 2897.76, "end": 2899.76, "text": " So now let's see the results."}, {"start": 2899.76, "end": 2902.76, "text": " There's a bunch of tables, I'm going to focus only on a couple of these,"}, {"start": 2902.76, "end": 2905.76, "text": " so here we have BERT pre-training, here we can see BERT large,"}, {"start": 2905.76, "end": 2912.76, "text": " and we can see that the model speedup goes for this microtransfer technique at 22x,"}, {"start": 2912.76, "end": 2918.76, "text": " which means the smaller model takes 22x less time to train compared to the big model,"}, {"start": 2918.76, "end": 2923.76, "text": " and all in all, the total speedup for the whole hyperparameter tuning procedure"}, {"start": 2923.76, "end": 2928.76, "text": " takes hundreds and hundreds of times less time to train than the BERT large,"}, {"start": 2928.76, "end": 2931.76, "text": " than directly training the big model."}, {"start": 2931.76, "end": 2934.76, "text": " Interestingly, we can also see that it transfers perfectly,"}, {"start": 2934.76, "end": 2937.76, "text": " it even achieves better results than training it directly,"}, {"start": 2937.76, "end": 2942.76, "text": " and that's the consequence of this amazing theory behind this paper."}, {"start": 2942.76, "end": 2949.76, "text": " GPT-3, they showed that if we take the smaller 6.7 billion parameter model"}, {"start": 2949.76, "end": 2957.76, "text": " and train it with microtransfer, you get better results even compared to the 13 billion size model,"}, {"start": 2957.76, "end": 2965.76, "text": " and that's the consequence of OpenAI not having enough resources to try out different hyperparameter combinations for such a big model."}, {"start": 2965.76, "end": 2969.76, "text": " And now with this technique, you can do much more thorough HP search"}, {"start": 2969.76, "end": 2974.76, "text": " and then just apply the same hyperparameters to the bigger model,"}, {"start": 2974.76, "end": 2981.76, "text": " and as you can see, they get pretty much better results across most of these datasets."}, {"start": 2981.76, "end": 2984.76, "text": " There are some outliers here for some reason,"}, {"start": 2984.76, "end": 2992.76, "text": " it performs way worse compared to these two on this Lamberta, few shots, not sure what's going on there."}, {"start": 2992.76, "end": 2997.76, "text": " Finally, interestingly enough, the same as what DeepNet did for depth,"}, {"start": 2997.76, "end": 3003.76, "text": " these guys showed that with increasing width, they always get better and better performance,"}, {"start": 3003.76, "end": 3009.76, "text": " so lower training loss in this example compared to standard parameterization where it's not stable."}, {"start": 3009.76, "end": 3014.76, "text": " You can see that the pattern holds up until a certain point depending on the learning rate,"}, {"start": 3014.76, "end": 3018.76, "text": " and then given enough width, it's going to explode."}, {"start": 3018.76, "end": 3024.76, "text": " So here, because the learning rate is bigger, it explodes even sooner than on the middle chart here."}, {"start": 3024.76, "end": 3031.76, "text": " And in another example, another chart showing that wider is better,"}, {"start": 3031.76, "end": 3036.76, "text": " basically you can see that throughout the whole training, we have the training tokens on the x-axis,"}, {"start": 3036.76, "end": 3040.76, "text": " so this is plotting during the actual training procedure,"}, {"start": 3040.76, "end": 3046.76, "text": " and you can see that the validation loss is getting smaller and smaller as we're increasing width,"}, {"start": 3046.76, "end": 3049.76, "text": " although there is some saturation going on."}, {"start": 3049.76, "end": 3055.76, "text": " That's pretty much it, guys. This was a super heavy paper, lots of equations,"}, {"start": 3055.76, "end": 3059.76, "text": " the previous paper is even harder than this one, there is so much math,"}, {"start": 3059.76, "end": 3068.76, "text": " and it looks like a super impactful work, and it'd be very useful to make some visualizations"}, {"start": 3068.76, "end": 3072.76, "text": " and make this a bit easier to understand."}, {"start": 3072.76, "end": 3075.76, "text": " But yeah, hopefully you found this explanation somewhat useful,"}, {"start": 3075.76, "end": 3078.76, "text": " I'm aware it was very hand-wavy and not rigorous,"}, {"start": 3078.76, "end": 3082.76, "text": " but hopefully some of the mental models of how I think about all of this helped you out."}, {"start": 3082.76, "end": 3087.76, "text": " If it did, consider sharing the video out and subscribe to this channel,"}, {"start": 3087.76, "end": 3091.76, "text": " join the Discord community, and until next time, bye-bye."}]