Ultimate Guide to Diffusion Models | ML Coding Series | Denoising Diffusion Probabilistic Models
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this 3rd video of my ML coding series, we do a deep dive into diffusion models! Diffusion is the powerhouse behind recent text-to-image generation models such as OpenAI's DALL-E 2, Google's Imagen, etc.

I first give you some context by going over 2 seminal diffusion papers:
* Denoising Diffusion Probabilistic Models
* Improved Denoising Diffusion Probabilistic Models

And then we dig deep into the actual code analysis comparing mathematical formulas behind diffusion with actual code implementation.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ DDPM paper: https://arxiv.org/abs/2006.11239
✅ Improved DDPM paper: https://arxiv.org/abs/2102.09672
✅ Improved DDPM code: https://github.com/openai/improved-diffusion
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00:00 (Paper) Denoising Diffusion Probabilistic Models
00:16:00 (Paper) Improved DDPMs
00:23:10 (Coding starts) Training DDPMs
00:24:50 UNet model creation walk-through
00:35:05 Gaussian Diffusion model creation walk-through
00:43:50 Training loop
00:56:08 Computing noise and variance (forward prop through UNet)
01:04:00 Variational lower bound loss
01:17:25 MSE loss
01:19:23 Sampling from diffusion models
01:26:50 Sampling an actual image
01:28:03 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#diffusion #ddpm #coding
 What's up guys, in this video I'm focusing on covering diffusion models. So basically the family of models that are powering some of the most famous AI models over the last couple of months such as I guess the LEED 2, Imagine from Google, Glide from OpenAI as well and many other models. So the idea, the video will be quite ambitious so I'll try and first walk you through two of the seminal papers behind diffusion models and then I'm going to actually go through the code base. So the skimming of the paper just serves the purpose of me showing you the formulas, the mathematical formulas which we can then map and relate to actual code. So it's not going to be a deep dive into the papers per se but hopefully this gives you the necessary context to kind of later on cope with the actual code. Okay, having said that, I'm covering two papers. One is denoising diffusion probabilistic models. That's pretty much the paper that made diffusion models practical and then I'm going to cover improved denoising diffusion probabilistic models from OpenAI and I'm actually going to be covering the code base behind this paper. Okay, but let's start with this one. So this is going to introduce the necessary basics if you haven't already learned anything about diffusion models, hopefully that's going to be some context for you. I did cover the glide paper so I did cover some diffusion models there so do check it out. I'm going to link it somewhere here but this video, having said that, this video is fairly self-contained so yeah, you can continue watching. Okay, so here is how the diffusion model looks like on a high level. So the idea is to start from an image here and then you slowly, gradually start adding Gaussian noise on top of your image. So that's called the forward diffusion process and ultimately you end up with an image such as this one which is, as you can see here, basically a complete noise of an image. And now if you learn how to reverse this process, so this is the forward process here as you can see, if you learn how to reverse this process, so denote it as P theta, if we learn how to do that, then basically if we start a training procedure and we learn to do this for all of the images in our dataset, you eventually learn the underlying data distribution and then you can basically start from a random noise sample and start denoising that image until you end up with a hallucinated new image from your underlying data distribution. So it's going to be a novel image obviously. Okay, so that's a very high level explanation there. Let me now walk you through the formulas. So again, this is how the forward process looks like. This is basically the joint distribution and if we want to sample, if we want to do one step of the reverse process, here is how we do it. So basically we are going to learn a neural network that's going to predict the mu of theta and the sigma of theta, which are basically the mean and the variance of the Gaussian. So now there is a bunch of theory for why this is possible. So they show that if you have steps small enough, basically that means that if your forward process is adding Gaussians, they show that you can also approximate the reverse process as sampling from a Gaussian. So it's not kind of completely obvious why this is the fact, but yeah, we'll have to take it for granted. So the idea is to learn those two. Let's continue here. So here is the actual forward process. You can see how we sample from the forward process here. Basically you kind of downscale the current image and then, so this is how you form the mean. You take the current image, you downscale it and basically then you have this like, covariance matrix and then you just sample from this Gaussian here to end up with the X of t. Okay? So again, we condition on the X t minus one and by sampling from this distribution here, we end up with X t. Okay? So that's basically going from here all the way to here. That's one step. What's next? Now, they show that you can basically train these models by optimizing this variational bound. The idea is, so here we have the log likelihood of our data. We want to obviously maximize. We want to tweak our model such that all of the data points from our data set are super likely under our model. So that's the idea of how we train most of these, like all of these pretty much generative models, not only diffusion models. And now this is a standard thing with variational bounds. Basically you find a surrogate loss, which is like a lower bound basically for this loss here. And then by maximizing it, basically you're certain that the likelihood of your data is going to be at least as big. So that's the main idea there. Now I'm not going to dig into formulas here. We're going to later see how they decompose this into an actual expression that's going to be leveraged later on in the code. But here is the equation for now. Now this is an important, this here is a super important finding. So instead of having to sample every, so during the forward process, instead of having to sample multiple times, and in practice they use thousand steps, they show you can literally sample arbitrary X of t starting from X zero, where X zero is your original image, by sampling from this Gaussian here. So these are some important coefficients we'll be seeing in the code as well. So we have these alpha t's, which are one minus beta t's. And beta is basically, as you can see here, it's basically just like your covariance matrix here. In practice, the original DDPM, so this paper here, used fixed, like fixed schedules, whereas the later papers, such as the one from OpenAI, the improved one, used learnable like schedules. And when I say schedule, I just mean how does beta vary when we go across the whole forward process. Then there is this alpha t bar, which is just a product of all of these alphas, which are defined here, starting from one, all the way to t. So one is the, we start from the, usually one is denoted as the start of the process before the image has been deteriorated, and then as t grows, we're going towards image becoming pure Gaussian noise. So here is the expression, you basically can use square root of this alpha t bar, multiply that, so we multiply the original image here, this is how we form the variance, and then we just sample from this distribution and we get the x of t. So that means we immediately get arbitrarily noisy image. Nice. I mentioned the loss we'll be using, so here is the loss just reshaped into different forms. So this is the form that's going to be actionable, so this is the form we'll be using in the code. So we have L0, LT minus one, and LT, so these are kind of three classes of similar components here. This one is super important, basically KL divergence between this here, as you can see, is the one step of the reverse process, and we're going to do KL divergence with this basically forward process posterior. And then we have this component, which will actually, in this paper, they can ignore it because this is going to be like pure Gaussian, and because they don't learn that the variance is basically, you can ignore this term here, and this one here is basically the negative log likelihood of the image conditioned on the previous step. Anyways, a lot of details I'm going to have to kind of hand wave explain all of this just for you to understand and see the formulas, that's the important part. Okay, so let's see what's up here. So we can actually calculate, that's a cool thing, we can calculate this posterior of the forward process analytically here, and you can see how this mu t tilde is computed, and you can see how beta tilde is computed here. So these expressions are all going to be here in the code, so just kind of take mental notes here. So I'll be comparing the formulas with code like side by side, so that's going to probably be useful for you guys. So anyways, because we have this is a Gaussian, and this is a Gaussian, that means we'll end up when you do the KL divergence between two Gaussians, you end up with simple analytical expressions, we're going to see those a bit later. Okay, so what's up next? So actually here, so here is how these LT minus ones are going to simplify to, so we simplify them to these expressions here. Basically we just find the MSE, so the mean squared error between the means, so this is going to be our learnable one, this is the one we get from the forward posterior. And that can be further, like basically just simple algebra here, because we know that Xt equals this, and that's from the so called nice property here, so that's this property here. So you can kind of imagine that sampling from this Gaussian is actually equivalent to computing the following expression. So if you want to get Xt, you basically do the following. So you do this square root alpha t bar X zero, and you basically add plus square root because this is variance, we want to have like standard deviation, so minus alpha bar t, so it's going to be under square root, and then we just multiply times epsilon, where epsilon is just basically your normal distribution. Okay, so that's the Gaussian with the mean of zero and variance of one. Okay, so let's go back here, and then we just plug in Xt here, so we, sorry, we just calculate X zero here, we just kind of do the algebra, and then we plug it in here, and this is the next expression, and then this simplifies because we know how this is computed up here, so here is how we compute that one, so it's just like bunch of manipulations of symbols, and I'm kind of scheming over it. So you end up with this expression, and then you basically, this is what you want your neural network to learn. So this is going to be your diffusion model, it's going to learn how to predict the noise here. Okay, so let's see, that simplifies to what, basically we want to have the learnable mean needs to be equal to whatever we had here, so we basically want to have this thing, we want it to be like the same as this term here, because then the loss will obviously go to zero, that's what I show here, and if we achieve that, if we learn that mean, then how we sample, how we'll be sampling from the reverse process is again a simple computation, so here's the mean, so that's the same expression as this one here, and then we just add the, basically standard deviation times this vector here, which is going to be sampled from a normal distribution, okay? And ultimately, when you simplify even further that expression, you end up with this type of parameterization, so you can either learn a neural network that's going to predict the mean, or you can learn instead just this term here, just the actual noise, and that's what I end up doing, so this expression, as you can see, is just a mean squared error between the sample noise, so we're basically learning what type of a noise was added to our image, and then we have these weights for each of our terms, LT minus one, so this is again, this is LT minus one, those are the last components we saw, and we're going to see that this weight basically is a function of the step in the diffusion process. Okay, I know this is a lot of formulas, bear with me, it's going to become much easier as the video progresses, because I'm going to start introducing code as well, but let me quickly just explain what this means, so we start from an image here, this is some image, and then basically we add on top of it some noise, it's going to be something inside of there, let me just draw some human being, and then after adding the noise, like this, let me change the color, so let's imagine we added a bunch of noise here, so now your diffusion model is learning this green stuff, and if it learns the green stuff, then you basically know how to go backwards, okay? So you learn the green stuff, the noise, that's the epsilon here, and you know how to denoise your images. Cool, it's kind of magical, it doesn't click, you probably won't understand it immediately, it's not completely straightforward to understand why this works, I'm still struggling to be honest, but the formulas are here, and we're going to follow them for the time being. Okay, I'm going to skip this, and starting from this LT minus one, they basically derive empirically this simplified objective, where they drop the term here, so the term that depends on the basically time step of the diffusion process, and we end up with this one here, and so as you can see, basically we're doing simple MSC loss between the noise, so we're trying to predict this noise added on top of the image, and we do that for various time steps of the diffusion process. So basically what we're doing again is we start from an image here, we add some noise on top of it, we end with an image, and then we keep on doing that for like, let's say a thousand steps, and we end up here, and basically we'll be trying to, at each step of the way, understand which noise was added here, which noise was added here, which noise was added here, and by doing that we learn the reverse process of the diffusion, and that's going to lead us to like a powerful generative model. Cool, let's continue here, let me just show you, so here are some images they get from the model, not that important because you already know that these models are super powerful, so here you can see how the diffusion process looks like in practice, you start with the noisy image, and then gradually keep on denoising it until you get to the sampled image, which is the image sample from the underlying data distribution, okay? Here they show how, depending on from which latent you start from, if you start from a latent like X thousand, so that means thousand steps of diffusion, and then you start three independent reverse processes because they are stochastic, you end up with three different images, but as soon as we start taking like latents that come from later in the reverse process, so let's say X 500, then you can see that three independent reverse processes lead to images that are quite similar. Having seen all of that, now I'm going to quickly walk you through the innovation that the second paper brought, so it's basically building directly on top of the denoising diffusion probabilistic models or DDPMs for short. Let me show you main contributions of this paper, so first of all, they have a learnable variance schedule, so this is how they are going to do it, remember this formula, we're going to see it a bit later, so basically they predict this vector V, and then they kind of do interpolation between beta T and beta tilde T, which is the posterior variance, and this is the forward process variance. Okay, so that's one thing they've done here, so this formula here, the second thing they've done is they use this hybrid loss, so L simple we saw what the one is, that's when you drop the terms that depend on the time step, and then we have the LVLB, which is the variational lower bound, so that's the actual original loss with all of those complex terms, and by just creating this type of weighted average between those two, and using this one to learn the variance, and using this one to learn the mean, basically they show this is the best, this was the best trade off, so there is a lot of experimentation going on here, so it's kind of a lot of hacks put on top of the DTPM in order to make this work, as well as DTPM itself had a bunch of hacks, such as using constant variances instead of learning them, etc., etc., so there is a lot of hacks going on in diffusion models, at least in these earlier papers. Okay, so they say here along the same line of reasoning, we also apply a stop gradient to the mu of theta output for the LVLB term, which basically translates to, so this component is only going to be training this variance expression here. Okay, so that's one thing, so that's the second thing actually, and then the third thing is instead of using linear, like basically noise schedule, so those betas being a simple linear sequence, instead of that they propose this cosine schedule, and what that brings is that you can see that alpha bar theta, alpha bar t, sorry, basically has much more gradual drop compared to linear, and because of that, that determines directly the amount of noise, and thus if you use the linear schedule, so basically using the linear schedule will lead to noisier images earlier on in the forward process of the diffusion. Okay, that's a third thing they do that helps a lot, and then let me show you a couple more expressions here. One very cool thing is using basically the values of the loss for each of the time step to understand how much weight we want to put on top of that time step. So this is kind of middle ground between using the simple objective where all of those constants with all of the LT minus ones are basically constant, like an equal to, I guess, one, then you have the LVLB which had those complex expressions, and finally we have this type of important sampling where depending on the loss, so if, for example, if you're struggling with one particular image in the process, let's say this one, the ith image here, basically what you do is you increase the loss for that particular XT, so you'll put additional focus on trying to predict that noise that was put on top of this image. So that's the idea with this expression here. Let me continue, and we are almost done. Finally this expression 19, not that important, basically what they show is that during training and during sampling you don't have to use the same type, the same length of a diffusion process. For example, your training chain has like 4,000 images whereas you can, during the sampling time you can just have 100 images, 100 latents, and basically they show that how to remap the betas and beta tildes, the posterior variances such that this actually works out nicely, and they get high quality images and they obviously save a lot of computation which is super important because you don't want to sample 4,000 images every time you need to generate an image. That's going to be super expensive. Okay, guys that's pretty much it, couple more things here and we are done with the papers. They show, and this is super interesting, they show the scaling loss for diffusion models and this was back I think in 2020, so if you kind of read this paper you could have expected that they are going to do the same thing as with GPT-3 and that's to scale up these models and that eventually led to Glide and then the Li and the Li2. And you can see here again we have the power law, we see that the FID which is the metric that shows you how high quality your samples are, you can see that with the increasing size we keep on getting smaller and smaller FIDs whereas with NLL which is the negative loss likelihood it does not exactly follow the power law but it's kind of still going down here, so yeah. In any case these two charts are indicative that scaling up diffusion models will probably be a good avenue for future research. Okay let's see the conclusion, the likelihood is improved by learning the variances using our parameterization and hybrid objective. Furthermore we have investigated how DDPM scale with the amount of available training compute and found that more training compute trivially leads to better sample quality and log likelihood. Okay guys, so you hopefully got the gist of diffusion models, we saw how we have this forward process of noising the images and then we have the backward, the reverse process of denoising the images, we learned that and then plus add to that a bunch of hacks to get this thing to work but ultimately they are now the best generality models we have because they have compared to GANs they are much better at covering covering all of the modes of your data distribution whereas we know that GANs suffer from mode collapse, we also have much more like it's much more stable to train diffusion models as compared to GANs so yeah basically diffusion models are currently what GANs were a couple of years ago. Cool guys let's now switch to the code, I'm gonna show you how this thing is actually trained and how we sample from the diffusion models. Before that let me just show you the architecture they're using to actually learn the epsilon so the noise during the diffusion model training so this is a simple unit, this is how the architecture looks like, we're gonna see this in the code, this is less important arguably you could use some other architectures as well, they're just kind of stuck with the unit. Okay let's start doing the training, okay I went on and downloaded the github repo, had to create some modifications to get this thing to work on my single GPU Windows machine, I can kind of submit that to my github, do let me know if you want me to do that, if so I can easily create a like a I'll easily just push this code on to my github repo. So I just created this launch script so I basically used the recommended settings that they've showed in their readme file here so you can kind of take a look at the readme of the repo which we're gonna be using the improved diffusion repo and basically yeah so that's what I've set in my launch and now we can start training here. That's less important we're gonna see what's going on here in a minute. Okay so let me start training, we're first gonna obviously have a bunch of arguments so here they are I'm gonna kind of print them out but don't try and understand what's going on there's too many of them, we'll gradually start analyzing what's going on so we have the Cypher data set which I downloaded using their Cypher script then there is like bunch of other stuff learning rate blah blah blah we'll see all of that later. So I'm gonna kind of ignore the parts that are not crucial to understanding diffusion models which means I'm gonna be ignoring the distributed training, I'm gonna be ignoring the loggers all of that and I'm gonna focus only on diffusion. So here is the first important step we basically take these arguments convert them into dictionary and we start creating the actual unit model and then the diffusion object here. So let's see how the model is gonna be constructed so we basically pass image size is gonna be 64, usually they have lower resolutions because remember the latents are of the same like dimensionality as the X0 as the original image. Contrast that with VAE is contrast that with GANs with any other model the latents are always much of smaller dimensionality whereas here it's a bit more computational intensive and so because of that what they end up doing in practice is training models for smaller images and then training additionally super resolution models and they actually have like here scripts as you can see here basically super res sample super res train I'm gonna skip those because they really work very similarly to how diffusion works I'm gonna focus on these two scripts here the image sample and the image train. Okay anyways number of channels that's gonna be internal dimension of the unit number of res blocks nothing interesting there learn Sigma set to true means we are learning the variances instead of hard coding them instead of using the fixed ones so that's the innovation from the improved diffusion paper class conditioning so we will not be using that but it's very easy to have class conditioning we'll see how they use temporal conditioning so how do we pass the T because T is a scalar how do we pass that into a neural network we're gonna see they are just using simple sinusoids like simple similar embeddings as what the original transformer paper used so and then they can alert I'll show you how they can fuse that into the unit model and we would be doing the same thing with this class conditioning if we were using it which we are not at the moment. Okay checkpointing is just an optimization technique I'm gonna kind of ignore what checkpointing does is during the forward pass instead of storing the activations for every single layer you don't do that and because of that you save bunch of memory but as on the con side you'll have to kind of when you do the back prop in order to calculate the gradients you'll have to do your computations again so you're trading off the memory for time so you'll spend up much more time but you'll save up memory by doing this we'll not be using the checkpointing so yeah just FYI certain layers of unit have basically VAT type of attention so that means each of the token of the image is gonna be attending to each of the other tokens and that's what they show here so 16 means at resolution 16 by 16 they'll be doing the this attention and at 8 by 8 resolution they'll be doing the same thing so when I say 8 by 8 it's inside of the actual unit because you know that unit has that characteristic shape okay so number of heads again just the parameter for that attention layer inside of unit nothing important there we're gonna see how this parameter is used basically this is how we depending on this flag they'll have two different ways of combining of conditioning the images with the time steps so yeah we'll see how this plays out a bit later okay so here is the create model function basically because image size is 64 they specify this how they specify how unit will be constructed then we have this attention DS attention DS basically converts this attention resolution into how many down sampling layers we have to wait inside of the unit before we start using these attentional layers so just another way of specifying when do we start inserting those attention layers into our unit okay not that vital you can ignore that if you didn't understand it so input number of channels obviously three we're dealing with RGB images we specify number of channels here because we are learning Sigma because of that this we end up with six output channels and the first three channels will be predicting the epsilon so that's the noise and the second three channels will be predicting the actual variances so that's why we have six here okay number of blocks blah blah blah nothing special drop out blah blah blah we'll not be using class conditioning so we end up with none here for number of classes we're not using checkpointing okay specifying details of attention etc okay let's enter the constructor I'm going to quickly walk you through unit so we are starting step by step we'll first see how unit works and then we're going to see how it fits into the whole like training loop later on okay so let's see here we just store all of these parameters inside of like internal fields here nothing fancy there so we here we create this layer the sequential layer which consists out of this basically inverse bottleneck shape of MLP this is a simple MLP that's going to be used to transform the sinusoids and before we use them to condition the model we'll see that a bit later okay so we can ignore this because we're not not using classes and now we start adding blocks to form the unit again we have three types of blocks we have the input blocks then we have the middle blocks here and finally we have the output blocks so that corresponds to what we saw in this diagram here basically whoops my one note is glitching so you can see it here so we have we kind of have the first part here the input blocks then we have the middle blocks and then we have the output blocks here that's roughly how how this code is structured so let me now get back to it and let me quickly walk you through there is i'm not going to dig into the actual details there is a couple important things i want to show you how they fuse the temporal information with the image information that's the vital thing i want to show you here okay so there is this wrapper object called time step embed sequential we're going to see that a lot what it does basically is the following depending if the layer inherits from this time step block which is just a simple interface a dummy interface that where where the for a function supports both the x which is the image representation and the embedding the temporal embeddings in that case we'll be we'll be calling the layer passing both arguments whereas if it's not inheriting from time step block then we'll just be passing the x and ignoring like the embeddings so as i said a simple wrapper nothing too interesting okay so the first thing we do is we create this a com2d layer why com2d because this is com and d a generic layer they created and then number of dimensions is two and because of that we end up with a com2d so that's how the unit model starts next up let me show you what is going on here so we have channel multiplication we're going to iterate through this array and then this is the interesting part we start adding residual blocks so the interesting part about residual blocks is actually only the forward function and i'll i'm gonna put a breakpoint here and later on i'm going to show you how this temporal fusion is is happening but for the time being let me quickly step through the constructor of the of the of the resnet block just quickly so we just stored the number of channels the the the number of channels for for the basically temporal vectors blah blah blah nothing fancy here let me see whether there is something interesting i need to focus on here so we have we specify these input layers we specify the embedding layers we'll see how these are used a bit later so bear with me here and we finally have out layers normalization blah blah blah clu is just activation unit there is like a zillion of these activation units and it's not even worth mentioning okay so basically all of the fun will happen later on during the forward prop and that's when i'm gonna kind of step into the resnet block and show you how the how the temporal information is mixed into the network okay so after that sometimes we'll be adding as as i said here so if this thing is if we are if we don't sample four times then we're gonna add the attention block which we haven't at this step so we're gonna skip this part for now and then we add as you can see here to input blocks we just add the layers which we accumulated during the loop and we just wrap it up into this time step embed sequential which is again that useful wrapper we saw a couple minutes ago that's it guys nothing fancy there so i'm gonna kind of skip to middle layer here okay let me ignore this thing here so here we are this middle block consists of a res block attention block and additional res block i can skip all of that let's continue here and finally the output blocks are literally what what we just saw in the input blocks so it's the same pretty much list of objects and here you can see by doing this we are reversing and we're creating a symmetric construction for the output blocks okay so because of that i'm just going to skip all of this and and end up here and here we add the normalization layer which is i think group normalization but again not that important to understand and finally we end up with a con layer and this zero module just zeros out the weights of these kernels i'm not sure why they're doing that if anyone knows that feel free to comment down below why do they initialize some of the layers with all zero weights i'm not completely sure cool again main takeaway from the from the unit model is like it has this very interesting type of a shape and with the input blocks middle blocks and our blocks and the most important thing i want you to remember here is they have this part where they are mixing in the temporal information we're gonna see that a bit later but just keep that in mind for now okay that was the creation of the model we do the same thing for diffusion so we have like i've set 50 steps it was 4000 by default this is just so that i can train a bit quicker on my machine otherwise it's very slow to to actually train this okay so we have learned sigma nothing special there noise schedule is going to be linear so the betas for the forward process are going to be basically sampled as we sample the linear function instead of the cosine we don't use the kl basically this if we were to set this to true we'd be using the variational lower bound loss instead of this we're going to be using the hybrid loss so we'll see all of that a bit later we are not predicting the x start x start is just a starting image the original image from a data set and we will be basically rescaling time step all of this is not that important we'll see what those parameters are a bit later but they're not that interesting okay so here we have we first form the the beta schedule so we pass the like name of the schedule and number of steps and this just returns those betas we saw in the paper in the beginning of the video so let me quickly show you the two like schedules they support one is linear layer as mentioned it's just a simple in space between the beta start bet beta end and we linearly interpolate using the number of diffusion steps here for the for the cosine schedule you have this a bit more complex expression where you create those we saw these equations in the paper i'm going to kind of ignore those for now not that important okay so let's see which loss we pick we are going to use something called rescaled msc mean squared error that's because we are using the hybrid loss we'll see how that plays out a bit later time steps re spacing again this is important only when you want to reduce the number of steps during the sampling process which we will not be using so we can kind of ignore all of that again this is the function that does that type of a logic and basically for our case we can skip this because we end up with let me show you how this all steps look like it's basically just mp our range so you end up with as you can see here zero through 49 because we have 50 steps and so there is nothing no like sub sampling that will be doing in this training okay we pass the betas we'll be model mean type is going to be epsilons we'll be we'll be predicting epsilon instead of predicting x start so in some of the ablations they actually tried predicting the x start which means the original image so instead of trying to predict the noise that the that was added on top of the original image why not try and predict the original image but empirically they showed that it's better to predict the epsilon okay let's continue so what's the model variance type so because this is set to true that means we'll be using the learned range so we'll be learning the variances our last type is as we saw rescaled msc and that's pretty much it now let's step into this space diffusion basically the important piece here is the construction of this Gaussian diffusion object so let's see how that thing is going to look like so that's the Gaussian diffusion class that's important this is the file where all of the magic will happen basically this Gaussian diffusion dot pi that's the most important file in this whole code base so again we specify the mean it's going to be the epsilon the variance which is going to be learned and then the last type rescaled msc blah blah blah and next up we we form the we just basically convert betas into numpy array we do some error checking assert asserting that that's just a like a 1d array and that all of the betas are bigger than zero and smaller or equal than one that's all we grab the number of steps here and here we start creating those equations we saw the coefficients not the equations we saw in the paper now for this formula so i'm just going to put things side by side so it's going to be easier for you to understand what's going on okay let's start so here are the alphas the alphas we saw those coefficients here basically one minus betas that's how we form the alphas and then we have alpha bars which is just the products of alphas and you can see how we form those alphas it's called com prod like a cumulative product we basically just call this mp com prod and we end up with arrays so this thing is still going to be 50 the length of this thing is going to be as you can see here it's going to be 50 that means that by indexing into this array we end up with up like a basically well we basically specify t of this product so it's not like we just calculated the whole product of all of the alphas we actually have well we actually have the array that contains alpha t bars for all of the t's which means zero through 49 okay next up we have these this this parameter called this coefficient called alphas com prod prev like basically what we do here is we what we do the right shifting so we add one as the first element and then we take the first n minus one elements and and pre like append them to this list and we also have this com prod next so where those are used is let me just see i'm fairly sure it's going to be used for this mean yeah it's going to be used for the posterior so it's basically used for this posterior like forward process so for the mean and for the as you can see here for the for the variance okay so we're going to see how we construct those in a second so we we now construct square root of alphas com prod so those are going to be used as you can see here so this let me kind of change the color here and you can see this is now the term we constructed here then we create square root one minus alphas com prod which is going to be let's see whether we have that one somewhere here i don't think so but like yeah it's going to be used somewhere later on okay we do the same thing for log square root blah blah blah there is a lot of these so this is square root reciprocal alphas com prod so that's square root of one over this okay so the square root so this expression here is actually used for the component in the loss here you can see it here as i said for each of these we'll have we'll basically calculate all of the coefficients we have the posterior variance which is again this expression here let me just kind of let me show you that this is indeed the case so we have betas here are the betas we have one minus alphas com prod prep so that's this expression here so that's the numerator of our expression here and then we have denominator which is one minus alphas com prod so that's it as i said they are now literally going through formulas in this paper and calculating all of the necessary coefficients we do the same thing here and then we have this i'm going to show you quickly these two so we have the posterior mean coefficient basically it's equal to um betas as you can see here and then time square root alphas com prod prep so that's this one square root of alphas previous uh and bar okay and then we divide that by one minus this expression here so hopefully i this this convinced you that indeed they're just going on here and calculating all of these coefficients and that's pretty much it i'm going to go back into the full screen here and continue with the code okay so this is again not interesting because this is only vital in the case of we doing a sub sampling during sampling which we will not be doing i using less steps during the sampling procedure which we will not be using so i can kind of safely ignore all of this and finally uh they use uh those betas to construct the uh this gossian diffusion model so this one was just used for for this process here for all practical purposes this init function here is going to construct the same gossian object we just saw so i'm going to skip i'm going to skip everything here and so that's that's it we end up with a gossian diffusion object okay guys so we have unit we saw how they literally go formula by formula and compute all of that uh inside of the constructor of the gossian diffusion object now let's go on further here let's exit all these functions we are back to our main function what they do is they just push the model to gpu in case in case you have gpu uh they just now uh create this schedule sampler so this one is fairly interesting i'm going to show you how this one looks like it basically um specifies how do we want during the training sample uh so which steps i which loss components those lt minus ones do we want to train and so what they use by default is just a uniform sampler which means all of the steps are equally likely but then they also have that other sample which i did mention where depending on the loss of particular lt minus one they'll maybe increase like upscale that weight so that we focus more on uh basically yeah learning that particular uh part of the diffusion process okay let me just kind of go through through here so here is the uniform sampler uh basically here is how that sampler looks like so the weights of the uniform sample are obviously all ones so we have number of steps so we have like 50 ones and then during the sampling what happens is this sample function is called because as you can see here uniform sampler and here is from schedule sampler the subject here and here's what happens so we have weights so this is all ones and then we divide here so we normalize and actually have a probability distribution because the sum needs to be equal to one as you know uh and then we just call mp random choice so we basically do uh this is how we weight all of the uh 50 uh time steps and we just uh basically sample uh whatever the batch size is of those of those time steps so that's what's going to happen and then some conversion here uh nothing fancy so the interesting sampler is that second one which i'll just kind of briefly show you but i will not go into it maybe later if we have enough time so last second moment resampler so this is the one where they basically use the loss history to uh decide on how so you can see here depending on the loss you'll have bigger weights for for those components where the loss is bigger so that's roughly it but we'll not be using this one so i'm going to kind of skip it for the for the sake of time okay let's continue here we have our uniform sampler we now create the data set so basically i'll be just using cipher that doesn't need to be the case they also used image net 64 by 64 in the paper and you can use whatever data set you want and that's it now we have the training loop this is where the actual training is going to happen we pass in the model we put like that's the unit we pass in the diffusion um model on top of it we feed in the data uh batch size micro batch so this is going to be um like uh we're going to be chunking the actual batch into micro batches because we like as i said this is fairly memory intensive that's why they have all of these optimization methods such as checkpointing and micro batching to kind of cope with all of that uh excess memory okay ema so that's your exponential moving average because uh they'll actually be using um uh so ema type of weights for for sampling later on nothing fancy here logging saving resuming checkpointing blah blah fp 16 we don't care about mixed precision training i just want to focus on diffusion uh we pass in our uniform sampler blah blah blah some optimization details like weight decay for the atom w okay we can enter this uh train loop let's see what's going on here so we pass in all of these we kind of store them into the internal fields okay um we have all of this i'm going to kind of skip through all of this nothing interesting okay we have our model parameters we have the master parameters again this is a uh like a consequence of the training this in a distributed fashion of course multiple machines so we for our like for all practical purposes we don't care about the this we just have single set of parameters okay syncing again a consequence of of distributed training we don't care about it they create the atom w optimizer okay uh now they have for each of the ema uh rates specified uh they create uh like a deep copy of the parameters so because we only have one ema rate we'll basically just have a single um like copy of the parameters again not that important for you to understand let's go on here this is distributed uh data parallel a wrapper in pytorch again we'll not be using that so for all practical purposes we can ignore it let's jump into the actual meat of the code of the training code so we first go on to sample a single batch of images and potentially the classic the classes that correspond to those images uh let me quickly walk you through how this load data looks like basically this is gonna collect all of the um images all of the paths from my uh data set so i did go on and kind of download those as i told you so you can see them here and if i kind of step over that uh it's gonna collect all of my images and um let me now show you how it looks like whoops if i copy paste the all files and i print the first let's say two paths you can see 006 png and 13 png and those are exactly those are exactly these two first images here so here is that one well it's super small so you will not see the cipher after all okay um okay then they go on to form this uh image data cell uh where just they just do some resizing cropping but nothing too interesting and here we have the data uh loader with batch sizes of 128 and we finally yield the batch from our data set okay that ends up giving us a batch of size i guess it's gonna be i expect it's gonna be 128 uh 364 64 because it's cipher it's rbg rgb images and 128 is the batch size so let's see whether that's the case uh indeed it is the case and conditioning we don't have anything so it's gonna be just empty dictionary as you can see here okay cool so here's the first step let's now this is the this is where all the magic happens i'm gonna ignore everything afterwards because it's just kind of it's just your common machine learning boilerplate code so i'm gonna ignore all of that so we have a forward forward backward call here okay so we just use zero gradients of our unit we want to clean those gradients before we recompute them again and then do the update okay so here we load the first micro batch so you can see we just sub sample this batch and we end up with like two three sixty four sixty four because our micro batch dimension is two and we did the same thing for conditioning which because it's an empty dictionary we don't really care we just check whether this is a last patch which it's not because we just started the loop and now this is where we do the uniform sampling of the diffusion process okay so this is this is the idea we we now sample t's so the time steps for each of the images in our batch okay so let's do that so we we we do that and we end up with two random t's so 44 and three so that means we'll take the first image we'll basically add noise 44 times in practice we'll just have a single step because of that nice property we saw in the paper and we'll get there and then this second one just has three steps of noise and then we try and predict back the noise so that's going to be the goal okay so this is the main function in this training code this training losses function we'll see it in a moment we just kind of wrap it up using the func func tools partial so that we don't have to pass these parameters every single time so just kind of convenience wrapper there and again we just you can ignore all of this because that's just distributed code nothing interesting let's focus on the compute losses this is the most important function here okay so what's going on first of all we generate noise that has the same shape as our input image uh our input micro batch so because x star is again two three sixty four sixty four we're going to end up with basically normal gossian tensor of the same shape okay so that's again our noise of the same shape as the input images and then we do the q sample so let me remind you what that thing is okay so here is a formula that we are going to use uh we are basically going to do this computation here actually this computation i showed you here so let's see that that's indeed the case so let me just kind of see whether i have a break point here i do have a break point so here we are we are in the in that function you can see that uh this is everything we do so we have this extract into tensor we'll see what that does but we have this square root alpha scum prod which is this part here let me just change the color so okay so we have this part here and then we multiply that element wise with x start which is x here which is this thing here okay and then we add a square root one minus alpha scum prod which is this part here we add up this and we multiply it with with noise as you can see here so we literally are just computing formulas from the paper nothing fancy there okay now i'm gonna one time just show you how this uh extract into tensor function looks like uh for the sake of your um i guess understanding of the code so i'm gonna kind of add a break point there so what we do is the following so we uh take that um so the the array we have like the let's say let's say the alpha t bar or whatnot and we basically just extract using the time steps so time steps are if you recall like we had um whoops whoops time steps we end up with 44.3 so we're gonna end up taking the alpha 44 bar and we're gonna end up taking the alpha 3 bar which are gonna contain uh respectively 44 uh terms uh the product of 44 terms and the product of three terms and then the only thing we do is literally we add these dummy dimensions as many times as possible so that we have the same dimensionality as the image we are doing element wise multiplication with and then we just expand here uh the the vector so we we just want to have the same shape of this of this uh yeah so why are we doing this well because we cannot multiply a scalar with the tensor and because of that we're just basically just copy pasting and broadcasting this this scalar so we end up with a tensor that contains whose all elements contain this particular alpha 44 bar whatnot okay hopefully that was clear enough and now i'm gonna kind of uh let me first uh ignore this i'm gonna not will not be stepping into this function anymore and that's it that's it guys we have we have the x t so we have our noisy version of the image right here okay let's see what the next steps are first of all uh because our loss is uh gonna be hybrid we're gonna ignore this part and we'll see that we'll actually be computing the same thing a bit later down below because hybrid also contains this variational lower bound loss as well okay so here we are we have the our loss type is if you recall rescale msc and we end up now here so this step is supposed to output the epsilon so we need to learn to predict the noise that if you recall uh we basically took that noise and we now we basically using that noise we form the x t's now we're trying to predict especially like particularly this noise so this is the term we'll be trying to learn how to predict those are the green dots i showed you in the one on paper so that's going to be the first three channels the last three channels are going to be about the variance and we'll see those uh uh in a bit uh but for now let me show you how this forward prop is going to work so this is again unit we're going to do something to the time steps so we're going to somehow merge them into this uh together with this uh with the image representation and let me quickly show you why we are doing this why are we passing x and t so this is the particular expression we are now dealing with here we have our unit so this epsilon t in practice so let me again change the color so this thing here is our unit we have this expression is the x t so that's the thing we just calculated so that's the x t again x t and we also pass t which are the time steps and that's why we have this particular expression that's why we are computing by executing this line of code here okay let me let me go back here and let me step uh through this particular thingy uh we can ignore this rep model um basically it serves the purpose of again only if you have the the sub sampling during this the your sampling procedure which will not be doing so none of this really matters and then we do some rescaling nothing that's super interesting just some legacy stuff so that they are comparable with the ddpm original paper etc so here is the actual unit forward pass so we have the new ts so this is just kind of some rescaled versions but still scalars of our original 44 and 3 if we recall these time steps here and now let's let's pass that and this is just empty like the key word arguments are just empty we don't have anything there so we just pass the time steps and we pass the images so let me show you the shape here shape here is again 2364 64 okay now let's step through the forward pass of the uh unit model here is where the magic happens first of all we have this time step embedding which is going to map the scalars into certain uh vectors which are just like heuristics basically we're just using some sinusoids same thing as in the original transformer paper so let's see how that looks like okay so here it is again not gonna dig into the details you can see a bunch of sinusoids cosines what ultimately matters is that we end up instead of with scalars we end up with 228 so we end up with two vectors so now we can we can kind of work with vectors in neural networks it's kind of hard to work with scalars after that okay i stepped into something uh whatever okay so now we have this it was actually the the activation function of this time embed and if you recall time embed is this inverse bottleneck of a of a of a mlp so let me just find that thing quickly so time embed where are you here it is it's just a combination of two linear layers uh whereas the this uh inner most uh like layer has forex the uh dimensionality of the input and output layers hence i call it inverse bottleneck and this is just like this times four is stuck with us since 2017 the original paper people just keep using that again just a simple transformation and we end up with some we'll end up with some uh vector of different dimensionality so let me show you that let's kind of step over here we end up with what we end up embedding vector here being two five twelve okay so that's it that's our time information now let's let's insert that time information somehow so we go through the input blocks remember the first block is going to be like the com layer and then we're going to have the rest blocks so let's see the the first thing so the first thing nothing nothing super interesting because it's just a com layer but if i have hopefully i've added a break point to the rest block let's just find the rest block here uh where are you okay here it is here is the residual block here is where the magic of of merging happens let me show you this so okay here we are we do some processing on the image that's not that important so these in layers are just like normalization some convolutions whatnot and nothing interesting there next up we take our embedding vectors which are again two five twelve right so that's going to be two five twelve and we pass them through the embed layers so let's see what those are again that's just like a linear layer uh that does some additional processing on top of those representations nothing super vital let me show you the shape okay two two fifty six and now what we do is we just add the dummy dimensions until we have the same shape as this as these image features h so h has the shape of as you can see 228 64 64 because of that we're going to keep on adding dummy dimensions to to our temporal tensor until we end up with the same uh number of dimensions you can see here and now because these two are one one we'll basically be broadcasting those so copy pasting the temporal vectors before we combine them with the image features so here's where the magic happens so this is the parameter we saw before the boolean use scale shift norm that's one option how we can do it you can just also just simply add you can simply add the temporal information to the image features and that's how you can condition the model okay but they i guess empirically uh found out that this approach works a bit better instead of just adding up the temporal representations to the image features instead of that let's kind of chunk this so we start with two 256 one one by doing this chunking along the first dimension that means we're going to split this 256 into 128 okay so let me kind of step over there let me show you the scale and shift are going to be 228 11 right the same thing for shift and then this is how they kind of combine the image features with the temporal features one plus scale and then we add a shift and we just have some normalization for the image features beforehand and then we do some um additional processing that's it guys that's everything there is to to to how this unit works i think this is like everything you really need to know i'm gonna remove that breakpoint and continue on here uh and that's pretty much it and now we're gonna keep on doing this input blocks middle blocks apple blocks we're gonna keep on adding and merging the the temporal information and that's it i'm gonna just skip over all of this okay so we've done the for a prop um here we are we are exiting the uh the function so we end up with let's see the shape should be as i said we should have i guess um 264 64 that's the prediction let's see whether that's indeed the case so 264 64 indeed why is that well because basically um uh we we have six channels because the first three channels again are epsilon which are the noise which we're trying to predict and the next three channels are the variance so let's see how that's gonna be used here um later on so because we have uh learned range we're gonna enter this part of the uh this branch here uh we just extract some dimensions so xt is again that's our tensor of noise images 23 64 64 so we're gonna split the output uh into two groups each of with three channels that's what i just explained so model output is gonna have the epsilon so that's gonna contain the noise whereas this is gonna contain the variance and let me now show you the shapes basically it's gonna be 23 64 64 whereas previously we had 26 64 64 okay now what we do here is uh basically we take the epsilon we detach it from the computational pie tort graph which means we'll not be updating uh the the the epsilon whereas we pass the variance like this and why is that well if you recall from the paper and i'll show you that in a second as well we basically just uh in the hybrid loss we use the um um this variational uh bound loss to train the variances whereas we use the simplified a loss objective to train the mean or in this case to train the epsilon which is equivalent because we have that different representations parameterizations so let me show you again the paper and compare this side by side okay so here it is basically as i mentioned we have the hybrid loss consists out of a simple and so the variational lower bound objective and they mentioned here along this same line of reasoning we also apply a stop gradient to the mu of theta or equivalently to the epsilon theta output for the lvlb term so that means this term is gonna we're gonna freeze we're gonna detach from the computational graph that's why we are doing this this part here okay now let's go on and compute the actual uh lvlb terms so we are now computing literally this we're computing this thing here right that's what we are currently doing and later you'll see in a couple seconds we'll be computing the l simple again just executing the formulas from the paper that's it do let me know whether this side by side comparison between formulas and code helps you or not because these videos are super long and take me a lot of time to to create them so any feedback is very much appreciated continuing on here we do that we concatenate those two and then we call this uh vb terms bbd so let me just kind of uh see where i have a breakpoint there yes i do let's go back to the code okay so what we do here is we pass this dummy model why dummy because because the this function here will later be calling this model but we will not actually be doing a for a prop through the unit as you can see this is how we define the model you basically see this is just a like a dummy lambda function that just returns uh so no matter what you pass in it's just going to return r which is the frozen out which is this thing here so we'll see some model like we'll see a for a prop but it's not going to be a for a prop it's just going to be this dummy return of of what we already computed just keep that in mind okay so what we pass we pass the x start which are the original images we pass the x t which are the images that were noised using uh like t number of steps which we have here and now let's kind of step into this uh function and see how it's computed so what we do is we compute the um posterior mean and variance here by doing that we basically compute the um the forward process posterior and we're going to basically do a kl divergence as you can see here between that and between our learned reverse process again let me show you the paper to make this a bit more concrete and then we're going to dig into the code okay guys so here is the expression we are calculating it's nothing more than this so we have the as i said posterior here it is like of the of the uh forward process that's this thing here here we have the learned reverse process so this thing here corresponds to this lines here the q part the posterior of the of the for process corresponds to this line here q posterior mean variance and then we have the kl divergence which is just this part here again this is just this part here kl of those two distributions and those are the l t minus one terms so the variational lower bound terms that's it nothing nothing too fancy now let's kind of dig and see how that's going to be computed so we pass the x start x t and t which are the original image noist image and t the time steps so let me kind of start uh enter this function again so it even says here what it's computing you can see here so it's very nice to add this type of comments that helps a lot to be honest so here it is how we compute this is we take this mean coefficient one we multiply it with x start we take the mean coefficient two we multiply it with x t and that's it we end up with a posterior mean again let me show you this side by side with the paper here it is guys so we saw this already actually so we have again posterior mean coefficient one is just this thing here so we actually computed that if you recall in the constructor of the Gaussian diffusion object so this is the the first coefficient here then we multiply that with x zero as you can see here x start just a different notation and then we have posterior mean coefficient two so that's this part here and we multiply with x t so that's this part here so that's it that's how this formula corresponds to this line here let me now continue here let me step over all of this we end up with the posterior mean we do the same thing with a posterior uh variance so that's just those betas again here are those betas that's what we just computed here that's the posterior variance and the posterior log variance i'm not sure how this one is going to be used we'll see that a bit later hopefully or maybe not okay just some assertions and that's it we computed uh the uh basically this distribution we because it's a Gaussian we can just return the mean and the variance and that's it that's perfectly describing the Gaussian okay now we do the p mean variance so we're now doing um that other step so again that's that's this part here this is what we're trying to compute at the moment here let's see how that's gonna look like uh let me see whether i have a break point inside of here yes i do so we passed uh again model remember here model is just uh dummy it's just going to return what we already computed beforehand so the epsilon and the variances and we pass in x t and t so that's the the noist image and uh time steps okay let's enter here again uh it says here we are basically computing this we are computing the um p mean variance okay so let me step over this nothing fancy there again uh this does nothing this just returns this just returns we're going to see how we are not going to enter the for function we're just going to return here okay so we just end up with the uh tensor we already previously computed so it's 264 64 let's see how how this is going to be computed so because we are actually our model var type so we are we have the we as i said we are we are learning the variances that's why we enter this branch here we split the model output again into the epsilon and variances and now because we are um learned range and not learned we enter the else branch and as you can see here we're now calculated the equation 15 from the improved ddpm so uh paper okay let me show you that side by side okay so here it is guys here's what we are computing this equation 15 so when i said we have variance i was we have variance i was kind of lying because that's not the case that's just a component that when you combine it like this then you end up with the variance okay so let me let me show you what they do here let me kind of remove this uh panel here uh okay so they have the mean log so they have the posterior log variance clipped so that's going to be this part here so that's where the log part comes into play so this is the the part that they compute here it says the mean log and then they compute the max log which is this part here actually because this is betas it's why it's the other way around but okay doesn't matter that much you get the point uh finally we have this um just normalization of the output the output is minus one one they add plus one so that's zero two and then divide dividing by two we bring it into the zero one uh range i guess and then we just do literally this equation here so you can see v times this plus this blah blah same equation so this line is literally this equation here and then we just add the additionally this exponent part so that's this part and we've successfully computed equation 15 okay let's continue on here this is just some clipping we can ignore it for now let me zoom in here uh we are not predicting previous x we are predicting epsilon that's why we enter here and we enter this branch so we first want to predict the the x start uh given the um um epsilon so the noise given the currently noisy images and time step we want to predict the x here i will see why we're using that in a second so let me kind of step into this and show you this part so here we are we're predicting x star from epsilon uh here is how we do that let me again have the equation on the side so we'll be able to understand what's going on here okay guys this is the equation we are computing so we recall that xt is computed like this if we just rearrange the terms and we compute x zero x zero turns out to be this expression here and that's what we are currently computing here okay so here are the square root reciprocal alphas come prods that's literally this square root blah blah this thing here then we multiply that with xt you can see xt here then we have minus uh square root reciprocal minus alpha blah blah so that's literally this term here you can just read it kind of word by word and understand that that's this term here and we multiply that by epsilon and that's it so that's how we calculate the x zero or x start uh let's go back here and now we're going to use that to find the model mean and by model mean i literally mean this expression here no pun intended so this expression here okay so let's see how this is going to work uh posterior mean variance i think this is well yeah this is the expression we already saw uh we already kind of computed this thing here before let me kind of zoom in uh so i can kind of skip through this yeah i'm gonna skip through this part here we're going to ignore this because we already saw it and that's it we end up with our mean and that's what we return back we return back the mean the variance the log variance and the predicted so the x start and after that we just here pass the true mean true log variance that's the the ground truth and we want to minimize the scale divergence so we want to basically be as close as possible to that distribution we pass our mean log variance we calculate the k l do some normalization blah blah okay finally there is this step this is l zero so in case that the time step was zero we do some different computation if you recall from the variational lower bound loss uh let me show you uh let me show you that side by side okay guys so here it is so here are the l t minus ones we were computing so those are the k l divergence we here compute this expression minus log p uh data x zero condition like x one so that's what we compute here this discretized gaussian likelihood blah blah blah that's how we get the negative log likelihood decoder negative log likelihood and that's pretty much it and now we uh compute depending on whether the as i said depending on whether t equals zero if it's equal zero then we have decoder nl that's what we use as the loss otherwise we use k l fairly simple fair enough okay let me kind of zoom in here and that's it we have our first term that's the vb the variational lower bound term and we just do some rescaling that's why the loss is called rescaled msc let's continue i'm going to ignore this part here uh because we'll be extracting epsilon which is noise so we can kind of ignore everything else here so let me just uh make sure this doesn't have any yep doesn't have any break point so we can kind of step over this we end up with target it's just going to be the noise uh and here we just do msc between the model output which are the epsilons that we predicted from the unit and we just do msc here between that and the target which is the noise and that noise let me see whether i can find it whether it's in this function okay and this is the noise that we initially created here before we even started creating these xt's that's it it's fairly actually simple um to be super frank here it's not easy to understand why this exactly works it's kind of still magical to me but i'm slowly learning about these diffusion models more and more i think the first time i kind of took some time to understand these was when i was preparing the glide paper so again do check out that paper i do have some short introduction maybe even better than the one i made in this video when it comes to understanding diffusion models but in any case finally because we have the hybrid loss we just combine the msc and the variational lower bound loss and that's it that's our loss after this we just do the um uh like basically do some waiting uh and for each of the time steps uh and because we use uniform sampling we'll just have ones everywhere uh do some logging blah blah blah and finally we do the computation of the gradients by calling the backward function on our hybrid loss here so that's pretty much it next up we just keep on repeating micro batches blah blah blah uh i'm gonna kind of stop here this is everything you need to know to understand how the how the training procedure works hopefully that was useful now let's quickly dig into the sampling code and then we are done so i'm gonna stop this training here i'm gonna enter the uh sampling script so that's gonna be let me find it here so it's gonna be the image sample function here the script here uh and let's start uh with the main so i'm gonna now start and run this one i'm gonna set the sample configuration from my launchpy in vs code by the way i love vs code he has beautiful design uh amazing debugging experience and i mean i don't know why you would use anything else unless you don't have a choice i guess that's that's a reasonable uh excuse i guess okay let's step over this again bunch of arguments we're gonna ignore distribution blob uh distributional training logging um so i'll skip the model creation as well so i'm just gonna disable all of the breakpoints and i'm gonna step over all of this because we already saw all of that so i'm gonna ignore it now we'll load the actual model they do provide the checkpoints in their github repost to do check it out you can download it and then just set it here so i can have it stored somewhere so it's an image net 64 64 unconditional model trained on 100 million steps i guess so we load it we put it onto to gpu we set it into eval mode and we now start creating the samples okay so because this is not a class conditional model we can ignore all of this part and now this is where the magic happens the sample function first of all there is this paper called ddim i might cover this in the next uh video but for now we're just going to ignore it because this is false we're going to be using the p sample loop this function here that's the one we're going to be using not the ddim it turns out that ddim in a nutshell works better when you only have 50 or less time steps during your sampling procedure as soon as you pass the 50 threshold roughly then this method here operates better and this is just like the the method from the improved ddpm paper the one i showed you in the beginning of the video okay so here's the sampling function this is our desired shape we want to have one three sixty four sixty four images this is the image we want to generate uh this is kind of empty okay let's enter this function and let's see how it works okay before that let me just enable the break points so enable all breakpoints okay now we can enter that here let me enter and here we are so first of all there's going to be this um um p sample loop progressive uh basically this thing is going to keep on producing the samples we pass in the model uh we pass in the desired shape uh noise is none we don't pass anything here we don't specify this blah blah blah this is just boilerplate i'm going to ignore all of that so here we are p sample loop progressive again we just pick a device uh it's going to be gpu in my case because i have uh like gpu here um then we generate the image so here's how we start we literally start with the gaussian noise so this is your your normal distribution so so basically gaussian with mean zero and variance one okay with the desired shape on the desired device let's generate that image now we have some number of time steps which is set to 100 again i think i just modified this for the sake of time otherwise i think default was a bit bigger like 4 000 um in any case so we generate the indices and you can see here we reverse them because now we are doing the uh the reverse process we start with the uh 99 with the index 99 and we go all the way to zero which is the original image when i say original image here this is a generated image from the underlying data distribution that we learned during the training procedure that's the that's the idea there okay let's step over here so let's go over the indices we start with the index 99 so this is going to be first 99 okay we kind of um well you can see here because batch dimension is one nothing fancy happens here but otherwise if you wanted to generate like five images in the batch then you would just copy paste uh 99 five times because for all of the all of the noisy images will be following the same process so that's where we start with with the last time step okay and in this function is where all the magic actually happens everything else was kind of boilerplate let's let's focus on this one so we pass the image which is again in the initial point so now in the initial step it's going to be just pure image pure sorry pure noise we pass the time step uh blah blah blah nothing fancy there let's enter the p sample and here is how the sampling works and actually we already saw this during the training so you pretty much know how this is going to work let me enter here uh this is that part of code where we were literally computing the um we split the model output the model variance then we compute those mean log max log so that's the equation i saw the equation 15 from the paper we get the variance and then later down below what we do is we basically predict the x0 and then use that to to compute model mean so we we already kind of stepped through this code so i'm not going to do that now so yeah because of that let me kind of step over here and ignore this so i'm going to just kind of disable all of the breakpoints let me disable all of the breakpoints let me step to here and let me click f5 to step over okay we end up here uh now i'm going to enable the breakpoints again in any case so we generate the noise again this is your normal distribution uh we generate these masks uh which basically are always going to be one if t is equal is different from zero because we treat the zero the zero step differently if you recall even during the training we had that nl loss uh as opposed to variational lower bound loss similarly here when we when we sample we're going to have a bit different behavior depending on the number of time steps so we just take the mean that's the mean we computed and then we take the log variance we do the xp xp and log cancel out we get the variance we multiply that with the noise and we end up with a sample okay so this is now we are in step 98 okay and we're going to keep on doing this until we get a completely denoised image and at the end once we end up with the zeroth time step this is going to be equal to zero and that means the only result we'll return is the final mean again that's just a detail of how this thing works and we return the sample that's it basically the sample now becomes image that we feed into the next iteration of this p sample function and that's how it pretty much works i'm gonna now stop this because you pretty much saw everything i'm gonna go to launch i'm going to go to sample i'm gonna increase the number of diffusion steps to let's say 500 and i'm gonna run the script again and just show you the results we get this time we're going to disable all of the breakpoints i'm gonna run this and just show you how the results look like let me show you yeah i added this image show plot show line so that we can visualize the actual sample and yeah i'll i'll get back to you as soon as the image is generated okay guys here's the image we got some dog image again do do keep in mind that i only have 500 steps it'd be much better if i used more steps obviously like thousand although there is some bug with this code when you use 4 000 you can check out the issue in their repo basically you get like completely saturated image so that means that the optimum quality of an image is maybe around thousand two thousand or something but like at four thousand the images hit the saturation point and you get like just pretty much junk out of these of this model guys this is pretty much it hopefully you found this format of a video useful do let me know whether you find it useful if so i'll keep on making videos such as this one combining papers combining code putting code in paper side by side trying to make these abstract mathematical ideas a bit more grounded and hopefully i've done a job at doing that in any case if you found this video useful do share it out with others who want to learn more about diffusion processes subscribe to this channel join the discord community you can find the link down below in the video description and until next time bye bye
[{"start": 0.0, "end": 6.04, "text": " What's up guys, in this video I'm focusing on covering diffusion models."}, {"start": 6.04, "end": 11.200000000000001, "text": " So basically the family of models that are powering some of the most famous AI models"}, {"start": 11.200000000000001, "end": 18.64, "text": " over the last couple of months such as I guess the LEED 2, Imagine from Google, Glide from"}, {"start": 18.64, "end": 22.16, "text": " OpenAI as well and many other models."}, {"start": 22.16, "end": 28.72, "text": " So the idea, the video will be quite ambitious so I'll try and first walk you through two"}, {"start": 28.72, "end": 34.44, "text": " of the seminal papers behind diffusion models and then I'm going to actually go through"}, {"start": 34.44, "end": 36.06, "text": " the code base."}, {"start": 36.06, "end": 40.2, "text": " So the skimming of the paper just serves the purpose of me showing you the formulas, the"}, {"start": 40.2, "end": 44.68, "text": " mathematical formulas which we can then map and relate to actual code."}, {"start": 44.68, "end": 49.28, "text": " So it's not going to be a deep dive into the papers per se but hopefully this gives you"}, {"start": 49.28, "end": 53.72, "text": " the necessary context to kind of later on cope with the actual code."}, {"start": 53.72, "end": 56.739999999999995, "text": " Okay, having said that, I'm covering two papers."}, {"start": 56.74, "end": 60.160000000000004, "text": " One is denoising diffusion probabilistic models."}, {"start": 60.160000000000004, "end": 66.36, "text": " That's pretty much the paper that made diffusion models practical and then I'm going to cover"}, {"start": 66.36, "end": 72.36, "text": " improved denoising diffusion probabilistic models from OpenAI and I'm actually going"}, {"start": 72.36, "end": 77.0, "text": " to be covering the code base behind this paper."}, {"start": 77.0, "end": 79.0, "text": " Okay, but let's start with this one."}, {"start": 79.0, "end": 84.36, "text": " So this is going to introduce the necessary basics if you haven't already learned anything"}, {"start": 84.36, "end": 88.84, "text": " about diffusion models, hopefully that's going to be some context for you."}, {"start": 88.84, "end": 93.12, "text": " I did cover the glide paper so I did cover some diffusion models there so do check it"}, {"start": 93.12, "end": 94.12, "text": " out."}, {"start": 94.12, "end": 98.92, "text": " I'm going to link it somewhere here but this video, having said that, this video is fairly"}, {"start": 98.92, "end": 101.96000000000001, "text": " self-contained so yeah, you can continue watching."}, {"start": 101.96000000000001, "end": 107.12, "text": " Okay, so here is how the diffusion model looks like on a high level."}, {"start": 107.12, "end": 114.08, "text": " So the idea is to start from an image here and then you slowly, gradually start adding"}, {"start": 114.08, "end": 116.2, "text": " Gaussian noise on top of your image."}, {"start": 116.2, "end": 122.03999999999999, "text": " So that's called the forward diffusion process and ultimately you end up with an image such"}, {"start": 122.03999999999999, "end": 128.48, "text": " as this one which is, as you can see here, basically a complete noise of an image."}, {"start": 128.48, "end": 133.44, "text": " And now if you learn how to reverse this process, so this is the forward process here as you"}, {"start": 133.44, "end": 142.24, "text": " can see, if you learn how to reverse this process, so denote it as P theta, if we learn"}, {"start": 142.24, "end": 147.04000000000002, "text": " how to do that, then basically if we start a training procedure and we learn to do this"}, {"start": 147.04000000000002, "end": 152.20000000000002, "text": " for all of the images in our dataset, you eventually learn the underlying data distribution"}, {"start": 152.20000000000002, "end": 158.36, "text": " and then you can basically start from a random noise sample and start denoising that image"}, {"start": 158.36, "end": 164.54000000000002, "text": " until you end up with a hallucinated new image from your underlying data distribution."}, {"start": 164.54000000000002, "end": 166.32000000000002, "text": " So it's going to be a novel image obviously."}, {"start": 166.32000000000002, "end": 170.88, "text": " Okay, so that's a very high level explanation there."}, {"start": 170.88, "end": 172.79999999999998, "text": " Let me now walk you through the formulas."}, {"start": 172.79999999999998, "end": 178.32, "text": " So again, this is how the forward process looks like."}, {"start": 178.32, "end": 188.4, "text": " This is basically the joint distribution and if we want to sample, if we want to do one"}, {"start": 188.4, "end": 193.24, "text": " step of the reverse process, here is how we do it."}, {"start": 193.24, "end": 198.84, "text": " So basically we are going to learn a neural network that's going to predict the mu of"}, {"start": 198.84, "end": 204.76, "text": " theta and the sigma of theta, which are basically the mean and the variance of the Gaussian."}, {"start": 204.76, "end": 208.12, "text": " So now there is a bunch of theory for why this is possible."}, {"start": 208.12, "end": 216.52, "text": " So they show that if you have steps small enough, basically that means that if your"}, {"start": 216.52, "end": 222.46, "text": " forward process is adding Gaussians, they show that you can also approximate the reverse"}, {"start": 222.46, "end": 226.7, "text": " process as sampling from a Gaussian."}, {"start": 226.7, "end": 231.72, "text": " So it's not kind of completely obvious why this is the fact, but yeah, we'll have to"}, {"start": 231.72, "end": 233.67999999999998, "text": " take it for granted."}, {"start": 233.67999999999998, "end": 236.16, "text": " So the idea is to learn those two."}, {"start": 236.16, "end": 237.16, "text": " Let's continue here."}, {"start": 237.16, "end": 240.04, "text": " So here is the actual forward process."}, {"start": 240.04, "end": 243.32, "text": " You can see how we sample from the forward process here."}, {"start": 243.32, "end": 249.6, "text": " Basically you kind of downscale the current image and then, so this is how you form the"}, {"start": 249.6, "end": 250.6, "text": " mean."}, {"start": 250.6, "end": 256.68, "text": " You take the current image, you downscale it and basically then you have this like,"}, {"start": 256.68, "end": 264.04, "text": " covariance matrix and then you just sample from this Gaussian here to end up with the"}, {"start": 264.04, "end": 265.04, "text": " X of t."}, {"start": 265.04, "end": 266.04, "text": " Okay?"}, {"start": 266.04, "end": 271.02, "text": " So again, we condition on the X t minus one and by sampling from this distribution here,"}, {"start": 271.02, "end": 272.2, "text": " we end up with X t."}, {"start": 272.2, "end": 273.2, "text": " Okay?"}, {"start": 273.2, "end": 278.08, "text": " So that's basically going from here all the way to here."}, {"start": 278.08, "end": 281.24, "text": " That's one step."}, {"start": 281.24, "end": 282.24, "text": " What's next?"}, {"start": 282.24, "end": 287.12, "text": " Now, they show that you can basically train these models by optimizing this variational"}, {"start": 287.12, "end": 288.12, "text": " bound."}, {"start": 288.12, "end": 292.92, "text": " The idea is, so here we have the log likelihood of our data."}, {"start": 292.92, "end": 294.6, "text": " We want to obviously maximize."}, {"start": 294.6, "end": 300.2, "text": " We want to tweak our model such that all of the data points from our data set are super"}, {"start": 300.2, "end": 302.44, "text": " likely under our model."}, {"start": 302.44, "end": 306.84000000000003, "text": " So that's the idea of how we train most of these, like all of these pretty much generative"}, {"start": 306.84000000000003, "end": 310.16, "text": " models, not only diffusion models."}, {"start": 310.16, "end": 315.12, "text": " And now this is a standard thing with variational bounds."}, {"start": 315.12, "end": 322.84000000000003, "text": " Basically you find a surrogate loss, which is like a lower bound basically for this loss"}, {"start": 322.84000000000003, "end": 323.84000000000003, "text": " here."}, {"start": 323.84000000000003, "end": 329.6, "text": " And then by maximizing it, basically you're certain that the likelihood of your data is"}, {"start": 329.6, "end": 331.82000000000005, "text": " going to be at least as big."}, {"start": 331.82000000000005, "end": 334.40000000000003, "text": " So that's the main idea there."}, {"start": 334.40000000000003, "end": 336.88, "text": " Now I'm not going to dig into formulas here."}, {"start": 336.88, "end": 341.12, "text": " We're going to later see how they decompose this into an actual expression that's going"}, {"start": 341.12, "end": 343.8, "text": " to be leveraged later on in the code."}, {"start": 343.8, "end": 345.84, "text": " But here is the equation for now."}, {"start": 345.84, "end": 349.52, "text": " Now this is an important, this here is a super important finding."}, {"start": 349.52, "end": 354.28, "text": " So instead of having to sample every, so during the forward process, instead of having to"}, {"start": 354.28, "end": 360.6, "text": " sample multiple times, and in practice they use thousand steps, they show you can literally"}, {"start": 360.6, "end": 367.72, "text": " sample arbitrary X of t starting from X zero, where X zero is your original image, by sampling"}, {"start": 367.72, "end": 369.96000000000004, "text": " from this Gaussian here."}, {"start": 369.96000000000004, "end": 373.96000000000004, "text": " So these are some important coefficients we'll be seeing in the code as well."}, {"start": 373.96000000000004, "end": 378.48, "text": " So we have these alpha t's, which are one minus beta t's."}, {"start": 378.48, "end": 384.92, "text": " And beta is basically, as you can see here, it's basically just like your covariance matrix"}, {"start": 384.92, "end": 386.20000000000005, "text": " here."}, {"start": 386.2, "end": 394.52, "text": " In practice, the original DDPM, so this paper here, used fixed, like fixed schedules, whereas"}, {"start": 394.52, "end": 401.28, "text": " the later papers, such as the one from OpenAI, the improved one, used learnable like schedules."}, {"start": 401.28, "end": 408.76, "text": " And when I say schedule, I just mean how does beta vary when we go across the whole forward"}, {"start": 408.76, "end": 409.76, "text": " process."}, {"start": 409.76, "end": 418.08, "text": " Then there is this alpha t bar, which is just a product of all of these alphas, which are"}, {"start": 418.08, "end": 421.88, "text": " defined here, starting from one, all the way to t."}, {"start": 421.88, "end": 427.56, "text": " So one is the, we start from the, usually one is denoted as the start of the process"}, {"start": 427.56, "end": 431.88, "text": " before the image has been deteriorated, and then as t grows, we're going towards image"}, {"start": 431.88, "end": 434.92, "text": " becoming pure Gaussian noise."}, {"start": 434.92, "end": 440.64000000000004, "text": " So here is the expression, you basically can use square root of this alpha t bar, multiply"}, {"start": 440.64000000000004, "end": 445.56, "text": " that, so we multiply the original image here, this is how we form the variance, and then"}, {"start": 445.56, "end": 448.52000000000004, "text": " we just sample from this distribution and we get the x of t."}, {"start": 448.52000000000004, "end": 453.72, "text": " So that means we immediately get arbitrarily noisy image."}, {"start": 453.72, "end": 454.72, "text": " Nice."}, {"start": 454.72, "end": 460.12, "text": " I mentioned the loss we'll be using, so here is the loss just reshaped into different forms."}, {"start": 460.12, "end": 462.88, "text": " So this is the form that's going to be actionable, so this is the form we'll be using in the"}, {"start": 462.88, "end": 463.88, "text": " code."}, {"start": 463.88, "end": 472.0, "text": " So we have L0, LT minus one, and LT, so these are kind of three classes of similar components"}, {"start": 472.0, "end": 474.2, "text": " here."}, {"start": 474.2, "end": 478.96, "text": " This one is super important, basically KL divergence between this here, as you can see,"}, {"start": 478.96, "end": 484.48, "text": " is the one step of the reverse process, and we're going to do KL divergence with this"}, {"start": 484.48, "end": 488.6, "text": " basically forward process posterior."}, {"start": 488.6, "end": 492.28, "text": " And then we have this component, which will actually, in this paper, they can ignore it"}, {"start": 492.28, "end": 500.35999999999996, "text": " because this is going to be like pure Gaussian, and because they don't learn that the variance"}, {"start": 500.35999999999996, "end": 506.0, "text": " is basically, you can ignore this term here, and this one here is basically the negative"}, {"start": 506.0, "end": 511.79999999999995, "text": " log likelihood of the image conditioned on the previous step."}, {"start": 511.79999999999995, "end": 516.68, "text": " Anyways, a lot of details I'm going to have to kind of hand wave explain all of this just"}, {"start": 516.68, "end": 519.8399999999999, "text": " for you to understand and see the formulas, that's the important part."}, {"start": 519.84, "end": 522.6800000000001, "text": " Okay, so let's see what's up here."}, {"start": 522.6800000000001, "end": 528.5600000000001, "text": " So we can actually calculate, that's a cool thing, we can calculate this posterior of"}, {"start": 528.5600000000001, "end": 536.0, "text": " the forward process analytically here, and you can see how this mu t tilde is computed,"}, {"start": 536.0, "end": 539.2800000000001, "text": " and you can see how beta tilde is computed here."}, {"start": 539.2800000000001, "end": 544.64, "text": " So these expressions are all going to be here in the code, so just kind of take mental notes"}, {"start": 544.64, "end": 545.64, "text": " here."}, {"start": 545.64, "end": 550.92, "text": " So I'll be comparing the formulas with code like side by side, so that's going to probably"}, {"start": 550.92, "end": 552.92, "text": " be useful for you guys."}, {"start": 552.92, "end": 558.48, "text": " So anyways, because we have this is a Gaussian, and this is a Gaussian, that means we'll end"}, {"start": 558.48, "end": 563.04, "text": " up when you do the KL divergence between two Gaussians, you end up with simple analytical"}, {"start": 563.04, "end": 566.28, "text": " expressions, we're going to see those a bit later."}, {"start": 566.28, "end": 568.48, "text": " Okay, so what's up next?"}, {"start": 568.48, "end": 572.4399999999999, "text": " So actually here, so here is how these LT minus ones are going to simplify to, so we"}, {"start": 572.44, "end": 577.0600000000001, "text": " simplify them to these expressions here."}, {"start": 577.0600000000001, "end": 583.2, "text": " Basically we just find the MSE, so the mean squared error between the means, so this is"}, {"start": 583.2, "end": 588.9200000000001, "text": " going to be our learnable one, this is the one we get from the forward posterior."}, {"start": 588.9200000000001, "end": 594.98, "text": " And that can be further, like basically just simple algebra here, because we know that"}, {"start": 594.98, "end": 601.0, "text": " Xt equals this, and that's from the so called nice property here, so that's this property"}, {"start": 601.0, "end": 602.0, "text": " here."}, {"start": 602.0, "end": 609.52, "text": " So you can kind of imagine that sampling from this Gaussian is actually equivalent to computing"}, {"start": 609.52, "end": 610.96, "text": " the following expression."}, {"start": 610.96, "end": 614.16, "text": " So if you want to get Xt, you basically do the following."}, {"start": 614.16, "end": 623.0, "text": " So you do this square root alpha t bar X zero, and you basically add plus square root because"}, {"start": 623.0, "end": 628.24, "text": " this is variance, we want to have like standard deviation, so minus alpha bar t, so it's going"}, {"start": 628.24, "end": 634.04, "text": " to be under square root, and then we just multiply times epsilon, where epsilon is just"}, {"start": 634.04, "end": 636.8, "text": " basically your normal distribution."}, {"start": 636.8, "end": 642.6, "text": " Okay, so that's the Gaussian with the mean of zero and variance of one."}, {"start": 642.6, "end": 652.44, "text": " Okay, so let's go back here, and then we just plug in Xt here, so we, sorry, we just calculate"}, {"start": 652.44, "end": 657.52, "text": " X zero here, we just kind of do the algebra, and then we plug it in here, and this is the"}, {"start": 657.52, "end": 663.48, "text": " next expression, and then this simplifies because we know how this is computed up here,"}, {"start": 663.48, "end": 668.24, "text": " so here is how we compute that one, so it's just like bunch of manipulations of symbols,"}, {"start": 668.24, "end": 671.28, "text": " and I'm kind of scheming over it."}, {"start": 671.28, "end": 675.6, "text": " So you end up with this expression, and then you basically, this is what you want your"}, {"start": 675.6, "end": 676.6, "text": " neural network to learn."}, {"start": 676.6, "end": 681.4399999999999, "text": " So this is going to be your diffusion model, it's going to learn how to predict the noise"}, {"start": 681.4399999999999, "end": 682.4399999999999, "text": " here."}, {"start": 682.44, "end": 692.6, "text": " Okay, so let's see, that simplifies to what, basically we want to have the learnable mean"}, {"start": 692.6, "end": 698.48, "text": " needs to be equal to whatever we had here, so we basically want to have this thing, we"}, {"start": 698.48, "end": 704.4000000000001, "text": " want it to be like the same as this term here, because then the loss will obviously go to"}, {"start": 704.4000000000001, "end": 711.72, "text": " zero, that's what I show here, and if we achieve that, if we learn that mean, then how we sample,"}, {"start": 711.72, "end": 716.5600000000001, "text": " how we'll be sampling from the reverse process is again a simple computation, so here's the"}, {"start": 716.5600000000001, "end": 722.36, "text": " mean, so that's the same expression as this one here, and then we just add the, basically"}, {"start": 722.36, "end": 728.88, "text": " standard deviation times this vector here, which is going to be sampled from a normal"}, {"start": 728.88, "end": 731.0, "text": " distribution, okay?"}, {"start": 731.0, "end": 736.96, "text": " And ultimately, when you simplify even further that expression, you end up with this type"}, {"start": 736.96, "end": 740.0400000000001, "text": " of parameterization, so you can either learn a neural network that's going to predict"}, {"start": 740.04, "end": 746.0, "text": " the mean, or you can learn instead just this term here, just the actual noise, and that's"}, {"start": 746.0, "end": 751.3199999999999, "text": " what I end up doing, so this expression, as you can see, is just a mean squared error"}, {"start": 751.3199999999999, "end": 757.9, "text": " between the sample noise, so we're basically learning what type of a noise was added to"}, {"start": 757.9, "end": 764.28, "text": " our image, and then we have these weights for each of our terms, LT minus one, so this"}, {"start": 764.28, "end": 770.0799999999999, "text": " is again, this is LT minus one, those are the last components we saw, and we're going"}, {"start": 770.0799999999999, "end": 775.56, "text": " to see that this weight basically is a function of the step in the diffusion process."}, {"start": 775.56, "end": 779.68, "text": " Okay, I know this is a lot of formulas, bear with me, it's going to become much easier"}, {"start": 779.68, "end": 784.0, "text": " as the video progresses, because I'm going to start introducing code as well, but let"}, {"start": 784.0, "end": 789.1999999999999, "text": " me quickly just explain what this means, so we start from an image here, this is some"}, {"start": 789.2, "end": 796.48, "text": " image, and then basically we add on top of it some noise, it's going to be something"}, {"start": 796.48, "end": 803.32, "text": " inside of there, let me just draw some human being, and then after adding the noise, like"}, {"start": 803.32, "end": 811.2800000000001, "text": " this, let me change the color, so let's imagine we added a bunch of noise here, so now your"}, {"start": 811.2800000000001, "end": 817.72, "text": " diffusion model is learning this green stuff, and if it learns the green stuff, then you"}, {"start": 817.72, "end": 821.24, "text": " basically know how to go backwards, okay?"}, {"start": 821.24, "end": 826.5600000000001, "text": " So you learn the green stuff, the noise, that's the epsilon here, and you know how to denoise"}, {"start": 826.5600000000001, "end": 827.5600000000001, "text": " your images."}, {"start": 827.5600000000001, "end": 834.12, "text": " Cool, it's kind of magical, it doesn't click, you probably won't understand it immediately,"}, {"start": 834.12, "end": 837.76, "text": " it's not completely straightforward to understand why this works, I'm still struggling to be"}, {"start": 837.76, "end": 842.0400000000001, "text": " honest, but the formulas are here, and we're going to follow them for the time being."}, {"start": 842.04, "end": 848.88, "text": " Okay, I'm going to skip this, and starting from this LT minus one, they basically derive"}, {"start": 848.88, "end": 854.0, "text": " empirically this simplified objective, where they drop the term here, so the term that"}, {"start": 854.0, "end": 860.68, "text": " depends on the basically time step of the diffusion process, and we end up with this"}, {"start": 860.68, "end": 867.36, "text": " one here, and so as you can see, basically we're doing simple MSC loss between the noise,"}, {"start": 867.36, "end": 872.6800000000001, "text": " so we're trying to predict this noise added on top of the image, and we do that for various"}, {"start": 872.6800000000001, "end": 874.92, "text": " time steps of the diffusion process."}, {"start": 874.92, "end": 880.4, "text": " So basically what we're doing again is we start from an image here, we add some noise"}, {"start": 880.4, "end": 884.88, "text": " on top of it, we end with an image, and then we keep on doing that for like, let's say"}, {"start": 884.88, "end": 890.8000000000001, "text": " a thousand steps, and we end up here, and basically we'll be trying to, at each step"}, {"start": 890.8000000000001, "end": 894.96, "text": " of the way, understand which noise was added here, which noise was added here, which noise"}, {"start": 894.96, "end": 901.24, "text": " was added here, and by doing that we learn the reverse process of the diffusion, and"}, {"start": 901.24, "end": 905.2800000000001, "text": " that's going to lead us to like a powerful generative model."}, {"start": 905.2800000000001, "end": 909.84, "text": " Cool, let's continue here, let me just show you, so here are some images they get from"}, {"start": 909.84, "end": 915.72, "text": " the model, not that important because you already know that these models are super powerful,"}, {"start": 915.72, "end": 920.2800000000001, "text": " so here you can see how the diffusion process looks like in practice, you start with the"}, {"start": 920.28, "end": 926.36, "text": " noisy image, and then gradually keep on denoising it until you get to the sampled image, which"}, {"start": 926.36, "end": 931.6, "text": " is the image sample from the underlying data distribution, okay?"}, {"start": 931.6, "end": 935.98, "text": " Here they show how, depending on from which latent you start from, if you start from a"}, {"start": 935.98, "end": 940.76, "text": " latent like X thousand, so that means thousand steps of diffusion, and then you start three"}, {"start": 940.76, "end": 944.8399999999999, "text": " independent reverse processes because they are stochastic, you end up with three different"}, {"start": 944.84, "end": 951.12, "text": " images, but as soon as we start taking like latents that come from later in the reverse"}, {"start": 951.12, "end": 958.52, "text": " process, so let's say X 500, then you can see that three independent reverse processes"}, {"start": 958.52, "end": 960.76, "text": " lead to images that are quite similar."}, {"start": 960.76, "end": 965.52, "text": " Having seen all of that, now I'm going to quickly walk you through the innovation that"}, {"start": 965.52, "end": 971.0, "text": " the second paper brought, so it's basically building directly on top of the denoising"}, {"start": 971.0, "end": 975.64, "text": " diffusion probabilistic models or DDPMs for short."}, {"start": 975.64, "end": 982.52, "text": " Let me show you main contributions of this paper, so first of all, they have a learnable"}, {"start": 982.52, "end": 988.04, "text": " variance schedule, so this is how they are going to do it, remember this formula, we're"}, {"start": 988.04, "end": 994.68, "text": " going to see it a bit later, so basically they predict this vector V, and then they"}, {"start": 994.68, "end": 1001.56, "text": " kind of do interpolation between beta T and beta tilde T, which is the posterior variance,"}, {"start": 1001.56, "end": 1004.52, "text": " and this is the forward process variance."}, {"start": 1004.52, "end": 1010.4399999999999, "text": " Okay, so that's one thing they've done here, so this formula here, the second thing they've"}, {"start": 1010.4399999999999, "end": 1017.52, "text": " done is they use this hybrid loss, so L simple we saw what the one is, that's when you drop"}, {"start": 1017.52, "end": 1023.16, "text": " the terms that depend on the time step, and then we have the LVLB, which is the variational"}, {"start": 1023.16, "end": 1030.56, "text": " lower bound, so that's the actual original loss with all of those complex terms, and"}, {"start": 1030.56, "end": 1039.1599999999999, "text": " by just creating this type of weighted average between those two, and using this one to learn"}, {"start": 1039.1599999999999, "end": 1044.3999999999999, "text": " the variance, and using this one to learn the mean, basically they show this is the"}, {"start": 1044.3999999999999, "end": 1049.2, "text": " best, this was the best trade off, so there is a lot of experimentation going on here,"}, {"start": 1049.2, "end": 1055.1200000000001, "text": " so it's kind of a lot of hacks put on top of the DTPM in order to make this work, as"}, {"start": 1055.1200000000001, "end": 1059.8, "text": " well as DTPM itself had a bunch of hacks, such as using constant variances instead of"}, {"start": 1059.8, "end": 1064.2, "text": " learning them, etc., etc., so there is a lot of hacks going on in diffusion models, at"}, {"start": 1064.2, "end": 1066.24, "text": " least in these earlier papers."}, {"start": 1066.24, "end": 1072.2, "text": " Okay, so they say here along the same line of reasoning, we also apply a stop gradient"}, {"start": 1072.2, "end": 1079.0800000000002, "text": " to the mu of theta output for the LVLB term, which basically translates to, so this component"}, {"start": 1079.08, "end": 1083.4399999999998, "text": " is only going to be training this variance expression here."}, {"start": 1083.4399999999998, "end": 1088.32, "text": " Okay, so that's one thing, so that's the second thing actually, and then the third thing is"}, {"start": 1088.32, "end": 1096.24, "text": " instead of using linear, like basically noise schedule, so those betas being a simple linear"}, {"start": 1096.24, "end": 1104.22, "text": " sequence, instead of that they propose this cosine schedule, and what that brings is that"}, {"start": 1104.22, "end": 1112.4, "text": " you can see that alpha bar theta, alpha bar t, sorry, basically has much more gradual"}, {"start": 1112.4, "end": 1119.1000000000001, "text": " drop compared to linear, and because of that, that determines directly the amount of noise,"}, {"start": 1119.1000000000001, "end": 1123.52, "text": " and thus if you use the linear schedule, so basically using the linear schedule will lead"}, {"start": 1123.52, "end": 1129.32, "text": " to noisier images earlier on in the forward process of the diffusion."}, {"start": 1129.32, "end": 1136.24, "text": " Okay, that's a third thing they do that helps a lot, and then let me show you a couple more"}, {"start": 1136.24, "end": 1137.84, "text": " expressions here."}, {"start": 1137.84, "end": 1145.4399999999998, "text": " One very cool thing is using basically the values of the loss for each of the time step"}, {"start": 1145.4399999999998, "end": 1149.08, "text": " to understand how much weight we want to put on top of that time step."}, {"start": 1149.08, "end": 1154.36, "text": " So this is kind of middle ground between using the simple objective where all of those constants"}, {"start": 1154.36, "end": 1161.3999999999999, "text": " with all of the LT minus ones are basically constant, like an equal to, I guess, one,"}, {"start": 1161.3999999999999, "end": 1166.84, "text": " then you have the LVLB which had those complex expressions, and finally we have this type"}, {"start": 1166.84, "end": 1171.78, "text": " of important sampling where depending on the loss, so if, for example, if you're struggling"}, {"start": 1171.78, "end": 1177.4399999999998, "text": " with one particular image in the process, let's say this one, the ith image here, basically"}, {"start": 1177.44, "end": 1184.72, "text": " what you do is you increase the loss for that particular XT, so you'll put additional focus"}, {"start": 1184.72, "end": 1189.6000000000001, "text": " on trying to predict that noise that was put on top of this image."}, {"start": 1189.6000000000001, "end": 1194.56, "text": " So that's the idea with this expression here."}, {"start": 1194.56, "end": 1196.96, "text": " Let me continue, and we are almost done."}, {"start": 1196.96, "end": 1201.56, "text": " Finally this expression 19, not that important, basically what they show is that during training"}, {"start": 1201.56, "end": 1205.92, "text": " and during sampling you don't have to use the same type, the same length of a diffusion"}, {"start": 1205.92, "end": 1206.92, "text": " process."}, {"start": 1206.92, "end": 1212.5600000000002, "text": " For example, your training chain has like 4,000 images whereas you can, during the sampling"}, {"start": 1212.5600000000002, "end": 1220.76, "text": " time you can just have 100 images, 100 latents, and basically they show that how to remap"}, {"start": 1220.76, "end": 1228.16, "text": " the betas and beta tildes, the posterior variances such that this actually works out nicely,"}, {"start": 1228.16, "end": 1232.44, "text": " and they get high quality images and they obviously save a lot of computation which"}, {"start": 1232.44, "end": 1235.96, "text": " is super important because you don't want to sample 4,000 images every time you need"}, {"start": 1235.96, "end": 1236.96, "text": " to generate an image."}, {"start": 1236.96, "end": 1238.88, "text": " That's going to be super expensive."}, {"start": 1238.88, "end": 1244.96, "text": " Okay, guys that's pretty much it, couple more things here and we are done with the papers."}, {"start": 1244.96, "end": 1249.4, "text": " They show, and this is super interesting, they show the scaling loss for diffusion models"}, {"start": 1249.4, "end": 1254.8400000000001, "text": " and this was back I think in 2020, so if you kind of read this paper you could have expected"}, {"start": 1254.8400000000001, "end": 1259.6000000000001, "text": " that they are going to do the same thing as with GPT-3 and that's to scale up these models"}, {"start": 1259.6000000000001, "end": 1264.3600000000001, "text": " and that eventually led to Glide and then the Li and the Li2."}, {"start": 1264.36, "end": 1269.84, "text": " And you can see here again we have the power law, we see that the FID which is the metric"}, {"start": 1269.84, "end": 1273.76, "text": " that shows you how high quality your samples are, you can see that with the increasing"}, {"start": 1273.76, "end": 1279.36, "text": " size we keep on getting smaller and smaller FIDs whereas with NLL which is the negative"}, {"start": 1279.36, "end": 1285.04, "text": " loss likelihood it does not exactly follow the power law but it's kind of still going"}, {"start": 1285.04, "end": 1286.62, "text": " down here, so yeah."}, {"start": 1286.62, "end": 1291.9199999999998, "text": " In any case these two charts are indicative that scaling up diffusion models will probably"}, {"start": 1291.92, "end": 1296.48, "text": " be a good avenue for future research."}, {"start": 1296.48, "end": 1302.3200000000002, "text": " Okay let's see the conclusion, the likelihood is improved by learning the variances using"}, {"start": 1302.3200000000002, "end": 1306.48, "text": " our parameterization and hybrid objective."}, {"start": 1306.48, "end": 1310.6000000000001, "text": " Furthermore we have investigated how DDPM scale with the amount of available training"}, {"start": 1310.6000000000001, "end": 1314.5600000000002, "text": " compute and found that more training compute trivially leads to better sample quality and"}, {"start": 1314.5600000000002, "end": 1315.5600000000002, "text": " log likelihood."}, {"start": 1315.5600000000002, "end": 1321.2, "text": " Okay guys, so you hopefully got the gist of diffusion models, we saw how we have this"}, {"start": 1321.2, "end": 1325.48, "text": " forward process of noising the images and then we have the backward, the reverse process"}, {"start": 1325.48, "end": 1333.48, "text": " of denoising the images, we learned that and then plus add to that a bunch of hacks to"}, {"start": 1333.48, "end": 1338.0800000000002, "text": " get this thing to work but ultimately they are now the best generality models we have"}, {"start": 1338.0800000000002, "end": 1342.88, "text": " because they have compared to GANs they are much better at covering covering all of the"}, {"start": 1342.88, "end": 1349.2, "text": " modes of your data distribution whereas we know that GANs suffer from mode collapse, we"}, {"start": 1349.2, "end": 1356.44, "text": " also have much more like it's much more stable to train diffusion models as compared to GANs"}, {"start": 1356.44, "end": 1362.24, "text": " so yeah basically diffusion models are currently what GANs were a couple of years ago."}, {"start": 1362.24, "end": 1366.2, "text": " Cool guys let's now switch to the code, I'm gonna show you how this thing is actually"}, {"start": 1366.2, "end": 1370.68, "text": " trained and how we sample from the diffusion models."}, {"start": 1370.68, "end": 1375.16, "text": " Before that let me just show you the architecture they're using to actually learn the epsilon"}, {"start": 1375.16, "end": 1381.92, "text": " so the noise during the diffusion model training so this is a simple unit, this is how the"}, {"start": 1381.92, "end": 1386.3600000000001, "text": " architecture looks like, we're gonna see this in the code, this is less important arguably"}, {"start": 1386.3600000000001, "end": 1390.28, "text": " you could use some other architectures as well, they're just kind of stuck with the"}, {"start": 1390.28, "end": 1391.28, "text": " unit."}, {"start": 1391.28, "end": 1399.3200000000002, "text": " Okay let's start doing the training, okay I went on and downloaded the github repo, had"}, {"start": 1399.3200000000002, "end": 1404.64, "text": " to create some modifications to get this thing to work on my single GPU Windows machine,"}, {"start": 1404.64, "end": 1411.0400000000002, "text": " I can kind of submit that to my github, do let me know if you want me to do that, if"}, {"start": 1411.0400000000002, "end": 1420.0800000000002, "text": " so I can easily create a like a I'll easily just push this code on to my github repo."}, {"start": 1420.0800000000002, "end": 1425.48, "text": " So I just created this launch script so I basically used the recommended settings that"}, {"start": 1425.48, "end": 1430.5200000000002, "text": " they've showed in their readme file here so you can kind of take a look at the readme"}, {"start": 1430.52, "end": 1435.6, "text": " of the repo which we're gonna be using the improved diffusion repo and basically yeah"}, {"start": 1435.6, "end": 1439.76, "text": " so that's what I've set in my launch and now we can start training here."}, {"start": 1439.76, "end": 1443.24, "text": " That's less important we're gonna see what's going on here in a minute."}, {"start": 1443.24, "end": 1450.28, "text": " Okay so let me start training, we're first gonna obviously have a bunch of arguments"}, {"start": 1450.28, "end": 1454.6399999999999, "text": " so here they are I'm gonna kind of print them out but don't try and understand what's going"}, {"start": 1454.6399999999999, "end": 1460.16, "text": " on there's too many of them, we'll gradually start analyzing what's going on so we have"}, {"start": 1460.16, "end": 1466.8000000000002, "text": " the Cypher data set which I downloaded using their Cypher script then there is like bunch"}, {"start": 1466.8000000000002, "end": 1470.42, "text": " of other stuff learning rate blah blah blah we'll see all of that later."}, {"start": 1470.42, "end": 1474.48, "text": " So I'm gonna kind of ignore the parts that are not crucial to understanding diffusion"}, {"start": 1474.48, "end": 1478.6000000000001, "text": " models which means I'm gonna be ignoring the distributed training, I'm gonna be ignoring"}, {"start": 1478.6000000000001, "end": 1482.52, "text": " the loggers all of that and I'm gonna focus only on diffusion."}, {"start": 1482.52, "end": 1488.76, "text": " So here is the first important step we basically take these arguments convert them into dictionary"}, {"start": 1488.76, "end": 1496.52, "text": " and we start creating the actual unit model and then the diffusion object here."}, {"start": 1496.52, "end": 1501.04, "text": " So let's see how the model is gonna be constructed so we basically pass image size is gonna be"}, {"start": 1501.04, "end": 1507.12, "text": " 64, usually they have lower resolutions because remember the latents are of the same like"}, {"start": 1507.12, "end": 1513.2, "text": " dimensionality as the X0 as the original image."}, {"start": 1513.2, "end": 1517.72, "text": " Contrast that with VAE is contrast that with GANs with any other model the latents are"}, {"start": 1517.72, "end": 1522.92, "text": " always much of smaller dimensionality whereas here it's a bit more computational intensive"}, {"start": 1522.92, "end": 1528.3, "text": " and so because of that what they end up doing in practice is training models for smaller"}, {"start": 1528.3, "end": 1533.24, "text": " images and then training additionally super resolution models and they actually have like"}, {"start": 1533.24, "end": 1538.8, "text": " here scripts as you can see here basically super res sample super res train I'm gonna"}, {"start": 1538.8, "end": 1544.16, "text": " skip those because they really work very similarly to how diffusion works I'm gonna focus on"}, {"start": 1544.16, "end": 1546.84, "text": " these two scripts here the image sample and the image train."}, {"start": 1546.84, "end": 1553.6799999999998, "text": " Okay anyways number of channels that's gonna be internal dimension of the unit number of"}, {"start": 1553.6799999999998, "end": 1558.1999999999998, "text": " res blocks nothing interesting there learn Sigma set to true means we are learning the"}, {"start": 1558.1999999999998, "end": 1564.1999999999998, "text": " variances instead of hard coding them instead of using the fixed ones so that's the innovation"}, {"start": 1564.1999999999998, "end": 1571.04, "text": " from the improved diffusion paper class conditioning so we will not be using that but it's very"}, {"start": 1571.04, "end": 1575.56, "text": " easy to have class conditioning we'll see how they use temporal conditioning so how"}, {"start": 1575.56, "end": 1579.8799999999999, "text": " do we pass the T because T is a scalar how do we pass that into a neural network we're"}, {"start": 1579.8799999999999, "end": 1584.72, "text": " gonna see they are just using simple sinusoids like simple similar embeddings as what the"}, {"start": 1584.72, "end": 1589.56, "text": " original transformer paper used so and then they can alert I'll show you how they can"}, {"start": 1589.56, "end": 1595.96, "text": " fuse that into the unit model and we would be doing the same thing with this class conditioning"}, {"start": 1595.96, "end": 1598.6799999999998, "text": " if we were using it which we are not at the moment."}, {"start": 1598.6799999999998, "end": 1605.36, "text": " Okay checkpointing is just an optimization technique I'm gonna kind of ignore what checkpointing"}, {"start": 1605.36, "end": 1609.12, "text": " does is during the forward pass instead of storing the activations for every single"}, {"start": 1609.12, "end": 1614.9599999999998, "text": " layer you don't do that and because of that you save bunch of memory but as on the con"}, {"start": 1614.9599999999998, "end": 1620.4799999999998, "text": " side you'll have to kind of when you do the back prop in order to calculate the gradients"}, {"start": 1620.4799999999998, "end": 1625.32, "text": " you'll have to do your computations again so you're trading off the memory for time"}, {"start": 1625.32, "end": 1629.24, "text": " so you'll spend up much more time but you'll save up memory by doing this we'll not be"}, {"start": 1629.24, "end": 1638.04, "text": " using the checkpointing so yeah just FYI certain layers of unit have basically VAT type of"}, {"start": 1638.04, "end": 1643.3, "text": " attention so that means each of the token of the image is gonna be attending to each"}, {"start": 1643.3, "end": 1650.92, "text": " of the other tokens and that's what they show here so 16 means at resolution 16 by 16 they'll"}, {"start": 1650.92, "end": 1656.76, "text": " be doing the this attention and at 8 by 8 resolution they'll be doing the same thing"}, {"start": 1656.76, "end": 1662.84, "text": " so when I say 8 by 8 it's inside of the actual unit because you know that unit has that characteristic"}, {"start": 1662.84, "end": 1669.0, "text": " shape okay so number of heads again just the parameter for that attention layer inside"}, {"start": 1669.0, "end": 1674.48, "text": " of unit nothing important there we're gonna see how this parameter is used basically this"}, {"start": 1674.48, "end": 1680.72, "text": " is how we depending on this flag they'll have two different ways of combining of conditioning"}, {"start": 1680.72, "end": 1688.44, "text": " the images with the time steps so yeah we'll see how this plays out a bit later okay so"}, {"start": 1688.44, "end": 1694.48, "text": " here is the create model function basically because image size is 64 they specify this"}, {"start": 1694.48, "end": 1700.3600000000001, "text": " how they specify how unit will be constructed then we have this attention DS attention DS"}, {"start": 1700.3600000000001, "end": 1706.38, "text": " basically converts this attention resolution into how many down sampling layers we have"}, {"start": 1706.38, "end": 1712.2800000000002, "text": " to wait inside of the unit before we start using these attentional layers so just another"}, {"start": 1712.2800000000002, "end": 1717.8400000000001, "text": " way of specifying when do we start inserting those attention layers into our unit okay"}, {"start": 1717.8400000000001, "end": 1722.8000000000002, "text": " not that vital you can ignore that if you didn't understand it so input number of channels"}, {"start": 1722.8000000000002, "end": 1728.2800000000002, "text": " obviously three we're dealing with RGB images we specify number of channels here because"}, {"start": 1728.2800000000002, "end": 1734.1200000000001, "text": " we are learning Sigma because of that this we end up with six output channels and the"}, {"start": 1734.12, "end": 1738.9599999999998, "text": " first three channels will be predicting the epsilon so that's the noise and the second"}, {"start": 1738.9599999999998, "end": 1744.52, "text": " three channels will be predicting the actual variances so that's why we have six here okay"}, {"start": 1744.52, "end": 1749.9799999999998, "text": " number of blocks blah blah blah nothing special drop out blah blah blah we'll not be using"}, {"start": 1749.9799999999998, "end": 1754.04, "text": " class conditioning so we end up with none here for number of classes we're not using"}, {"start": 1754.04, "end": 1761.36, "text": " checkpointing okay specifying details of attention etc okay let's enter the constructor I'm going"}, {"start": 1761.36, "end": 1765.32, "text": " to quickly walk you through unit so we are starting step by step we'll first see how"}, {"start": 1765.32, "end": 1770.6399999999999, "text": " unit works and then we're going to see how it fits into the whole like training loop"}, {"start": 1770.6399999999999, "end": 1779.6799999999998, "text": " later on okay so let's see here we just store all of these parameters inside of like internal"}, {"start": 1779.6799999999998, "end": 1786.1999999999998, "text": " fields here nothing fancy there so we here we create this layer the sequential layer"}, {"start": 1786.2, "end": 1793.52, "text": " which consists out of this basically inverse bottleneck shape of MLP this is a simple MLP"}, {"start": 1793.52, "end": 1801.0, "text": " that's going to be used to transform the sinusoids and before we use them to condition the model"}, {"start": 1801.0, "end": 1805.64, "text": " we'll see that a bit later okay so we can ignore this because we're not not using classes"}, {"start": 1805.64, "end": 1811.56, "text": " and now we start adding blocks to form the unit again we have three types of blocks we"}, {"start": 1811.56, "end": 1817.8, "text": " have the input blocks then we have the middle blocks here and finally we have the output"}, {"start": 1817.8, "end": 1825.56, "text": " blocks so that corresponds to what we saw in this diagram here basically whoops my one"}, {"start": 1825.56, "end": 1831.28, "text": " note is glitching so you can see it here so we have we kind of have the first part here"}, {"start": 1831.28, "end": 1835.36, "text": " the input blocks then we have the middle blocks and then we have the output blocks here that's"}, {"start": 1835.36, "end": 1841.32, "text": " roughly how how this code is structured so let me now get back to it and let me quickly"}, {"start": 1841.32, "end": 1845.04, "text": " walk you through there is i'm not going to dig into the actual details there is a couple"}, {"start": 1845.04, "end": 1849.6, "text": " important things i want to show you how they fuse the temporal information with the image"}, {"start": 1849.6, "end": 1854.8799999999999, "text": " information that's the vital thing i want to show you here okay so there is this wrapper"}, {"start": 1854.8799999999999, "end": 1860.22, "text": " object called time step embed sequential we're going to see that a lot what it does basically"}, {"start": 1860.22, "end": 1864.84, "text": " is the following depending if the layer inherits from this time step block which is just a"}, {"start": 1864.84, "end": 1869.24, "text": " simple interface a dummy interface that where where the for a function supports both the"}, {"start": 1869.24, "end": 1874.56, "text": " x which is the image representation and the embedding the temporal embeddings in that"}, {"start": 1874.56, "end": 1881.28, "text": " case we'll be we'll be calling the layer passing both arguments whereas if it's not inheriting"}, {"start": 1881.28, "end": 1886.6, "text": " from time step block then we'll just be passing the x and ignoring like the embeddings so"}, {"start": 1886.6, "end": 1892.88, "text": " as i said a simple wrapper nothing too interesting okay so the first thing we do is we create"}, {"start": 1892.88, "end": 1898.56, "text": " this a com2d layer why com2d because this is com and d a generic layer they created"}, {"start": 1898.56, "end": 1904.08, "text": " and then number of dimensions is two and because of that we end up with a com2d so that's how"}, {"start": 1904.08, "end": 1910.6399999999999, "text": " the unit model starts next up let me show you what is going on here so we have channel"}, {"start": 1910.6399999999999, "end": 1915.76, "text": " multiplication we're going to iterate through this array and then this is the interesting"}, {"start": 1915.76, "end": 1922.08, "text": " part we start adding residual blocks so the interesting part about residual blocks is"}, {"start": 1922.08, "end": 1927.8, "text": " actually only the forward function and i'll i'm gonna put a breakpoint here and later"}, {"start": 1927.8, "end": 1934.48, "text": " on i'm going to show you how this temporal fusion is is happening but for the time being"}, {"start": 1934.48, "end": 1939.44, "text": " let me quickly step through the constructor of the of the of the resnet block just quickly"}, {"start": 1939.44, "end": 1946.68, "text": " so we just stored the number of channels the the the number of channels for for the basically"}, {"start": 1946.68, "end": 1952.68, "text": " temporal vectors blah blah blah nothing fancy here let me see whether there is something"}, {"start": 1952.68, "end": 1958.8400000000001, "text": " interesting i need to focus on here so we have we specify these input layers we specify"}, {"start": 1958.8400000000001, "end": 1964.04, "text": " the embedding layers we'll see how these are used a bit later so bear with me here and"}, {"start": 1964.04, "end": 1969.52, "text": " we finally have out layers normalization blah blah blah clu is just activation unit there"}, {"start": 1969.52, "end": 1976.16, "text": " is like a zillion of these activation units and it's not even worth mentioning okay so"}, {"start": 1976.16, "end": 1981.28, "text": " basically all of the fun will happen later on during the forward prop and that's when"}, {"start": 1981.28, "end": 1986.56, "text": " i'm gonna kind of step into the resnet block and show you how the how the temporal information"}, {"start": 1986.56, "end": 1993.6, "text": " is mixed into the network okay so after that sometimes we'll be adding as as i said here"}, {"start": 1993.6, "end": 2000.08, "text": " so if this thing is if we are if we don't sample four times then we're gonna add the"}, {"start": 2000.08, "end": 2005.28, "text": " attention block which we haven't at this step so we're gonna skip this part for now and"}, {"start": 2005.28, "end": 2010.84, "text": " then we add as you can see here to input blocks we just add the layers which we accumulated"}, {"start": 2010.84, "end": 2016.08, "text": " during the loop and we just wrap it up into this time step embed sequential which is again"}, {"start": 2016.08, "end": 2021.6799999999998, "text": " that useful wrapper we saw a couple minutes ago that's it guys nothing fancy there so"}, {"start": 2021.6799999999998, "end": 2028.48, "text": " i'm gonna kind of skip to middle layer here okay let me ignore this thing here so here"}, {"start": 2028.48, "end": 2035.12, "text": " we are this middle block consists of a res block attention block and additional res block"}, {"start": 2035.12, "end": 2041.6399999999999, "text": " i can skip all of that let's continue here and finally the output blocks are literally"}, {"start": 2041.6399999999999, "end": 2047.36, "text": " what what we just saw in the input blocks so it's the same pretty much list of objects"}, {"start": 2047.36, "end": 2052.8399999999997, "text": " and here you can see by doing this we are reversing and we're creating a symmetric construction"}, {"start": 2052.8399999999997, "end": 2057.64, "text": " for the output blocks okay so because of that i'm just going to skip all of this and and"}, {"start": 2057.64, "end": 2064.56, "text": " end up here and here we add the normalization layer which is i think group normalization"}, {"start": 2064.56, "end": 2068.56, "text": " but again not that important to understand and finally we end up with a con layer and"}, {"start": 2068.56, "end": 2075.06, "text": " this zero module just zeros out the weights of these kernels i'm not sure why they're"}, {"start": 2075.06, "end": 2079.96, "text": " doing that if anyone knows that feel free to comment down below why do they initialize"}, {"start": 2079.96, "end": 2084.7999999999997, "text": " some of the layers with all zero weights i'm not completely sure cool again main takeaway"}, {"start": 2084.7999999999997, "end": 2091.12, "text": " from the from the unit model is like it has this very interesting type of a shape and"}, {"start": 2091.12, "end": 2095.12, "text": " with the input blocks middle blocks and our blocks and the most important thing i want"}, {"start": 2095.12, "end": 2099.88, "text": " you to remember here is they have this part where they are mixing in the temporal information"}, {"start": 2099.88, "end": 2104.3599999999997, "text": " we're gonna see that a bit later but just keep that in mind for now okay that was the"}, {"start": 2104.3599999999997, "end": 2110.88, "text": " creation of the model we do the same thing for diffusion so we have like i've set 50"}, {"start": 2110.88, "end": 2116.6, "text": " steps it was 4000 by default this is just so that i can train a bit quicker on my machine"}, {"start": 2116.6, "end": 2121.96, "text": " otherwise it's very slow to to actually train this okay so we have learned sigma nothing"}, {"start": 2121.96, "end": 2126.0, "text": " special there noise schedule is going to be linear so the betas for the forward process"}, {"start": 2126.0, "end": 2132.88, "text": " are going to be basically sampled as we sample the linear function instead of the cosine"}, {"start": 2132.88, "end": 2138.16, "text": " we don't use the kl basically this if we were to set this to true we'd be using the variational"}, {"start": 2138.16, "end": 2142.2799999999997, "text": " lower bound loss instead of this we're going to be using the hybrid loss so we'll see all"}, {"start": 2142.28, "end": 2146.96, "text": " of that a bit later we are not predicting the x start x start is just a starting image"}, {"start": 2146.96, "end": 2153.0, "text": " the original image from a data set and we will be basically rescaling time step all"}, {"start": 2153.0, "end": 2157.7200000000003, "text": " of this is not that important we'll see what those parameters are a bit later but they're"}, {"start": 2157.7200000000003, "end": 2164.1200000000003, "text": " not that interesting okay so here we have we first form the the beta schedule so we"}, {"start": 2164.1200000000003, "end": 2170.0400000000004, "text": " pass the like name of the schedule and number of steps and this just returns those betas"}, {"start": 2170.04, "end": 2176.16, "text": " we saw in the paper in the beginning of the video so let me quickly show you the two like"}, {"start": 2176.16, "end": 2181.48, "text": " schedules they support one is linear layer as mentioned it's just a simple in space between"}, {"start": 2181.48, "end": 2187.8, "text": " the beta start bet beta end and we linearly interpolate using the number of diffusion"}, {"start": 2187.8, "end": 2192.8, "text": " steps here for the for the cosine schedule you have this a bit more complex expression"}, {"start": 2192.8, "end": 2199.04, "text": " where you create those we saw these equations in the paper i'm going to kind of ignore those"}, {"start": 2199.04, "end": 2204.92, "text": " for now not that important okay so let's see which loss we pick we are going to use something"}, {"start": 2204.92, "end": 2212.32, "text": " called rescaled msc mean squared error that's because we are using the hybrid loss we'll"}, {"start": 2212.32, "end": 2219.64, "text": " see how that plays out a bit later time steps re spacing again this is important only when"}, {"start": 2219.64, "end": 2224.88, "text": " you want to reduce the number of steps during the sampling process which we will not be"}, {"start": 2224.88, "end": 2229.92, "text": " using so we can kind of ignore all of that again this is the function that does that"}, {"start": 2229.92, "end": 2236.8, "text": " type of a logic and basically for our case we can skip this because we end up with let"}, {"start": 2236.8, "end": 2241.6, "text": " me show you how this all steps look like it's basically just mp our range so you end up"}, {"start": 2241.6, "end": 2248.96, "text": " with as you can see here zero through 49 because we have 50 steps and so there is nothing no"}, {"start": 2248.96, "end": 2256.0, "text": " like sub sampling that will be doing in this training okay we pass the betas we'll be model"}, {"start": 2256.0, "end": 2260.52, "text": " mean type is going to be epsilons we'll be we'll be predicting epsilon instead of predicting"}, {"start": 2260.52, "end": 2265.04, "text": " x start so in some of the ablations they actually tried predicting the x start which means the"}, {"start": 2265.04, "end": 2270.32, "text": " original image so instead of trying to predict the noise that the that was added on top of"}, {"start": 2270.32, "end": 2274.48, "text": " the original image why not try and predict the original image but empirically they showed"}, {"start": 2274.48, "end": 2280.36, "text": " that it's better to predict the epsilon okay let's continue so what's the model variance"}, {"start": 2280.36, "end": 2287.28, "text": " type so because this is set to true that means we'll be using the learned range so we'll"}, {"start": 2287.28, "end": 2293.28, "text": " be learning the variances our last type is as we saw rescaled msc and that's pretty much"}, {"start": 2293.28, "end": 2299.72, "text": " it now let's step into this space diffusion basically the important piece here is the"}, {"start": 2299.72, "end": 2304.04, "text": " construction of this Gaussian diffusion object so let's see how that thing is going to look"}, {"start": 2304.04, "end": 2309.08, "text": " like so that's the Gaussian diffusion class that's important this is the file where all"}, {"start": 2309.08, "end": 2313.12, "text": " of the magic will happen basically this Gaussian diffusion dot pi that's the most important"}, {"start": 2313.12, "end": 2319.08, "text": " file in this whole code base so again we specify the mean it's going to be the epsilon the"}, {"start": 2319.08, "end": 2323.44, "text": " variance which is going to be learned and then the last type rescaled msc blah blah"}, {"start": 2323.44, "end": 2329.92, "text": " blah and next up we we form the we just basically convert betas into numpy array we do some"}, {"start": 2329.92, "end": 2335.0, "text": " error checking assert asserting that that's just a like a 1d array and that all of the"}, {"start": 2335.0, "end": 2342.08, "text": " betas are bigger than zero and smaller or equal than one that's all we grab the number"}, {"start": 2342.08, "end": 2347.12, "text": " of steps here and here we start creating those equations we saw the coefficients not the"}, {"start": 2347.12, "end": 2350.48, "text": " equations we saw in the paper now for this formula so i'm just going to put things side"}, {"start": 2350.48, "end": 2354.6800000000003, "text": " by side so it's going to be easier for you to understand what's going on okay let's start"}, {"start": 2354.68, "end": 2360.3199999999997, "text": " so here are the alphas the alphas we saw those coefficients here basically one minus betas"}, {"start": 2360.3199999999997, "end": 2364.2799999999997, "text": " that's how we form the alphas and then we have alpha bars which is just the products"}, {"start": 2364.2799999999997, "end": 2369.3999999999996, "text": " of alphas and you can see how we form those alphas it's called com prod like a cumulative"}, {"start": 2369.3999999999996, "end": 2375.3999999999996, "text": " product we basically just call this mp com prod and we end up with arrays so this thing"}, {"start": 2375.3999999999996, "end": 2380.56, "text": " is still going to be 50 the length of this thing is going to be as you can see here it's"}, {"start": 2380.56, "end": 2389.32, "text": " going to be 50 that means that by indexing into this array we end up with up like a basically"}, {"start": 2389.32, "end": 2395.56, "text": " well we basically specify t of this product so it's not like we just calculated the whole"}, {"start": 2395.56, "end": 2402.44, "text": " product of all of the alphas we actually have well we actually have the array that contains"}, {"start": 2402.44, "end": 2410.04, "text": " alpha t bars for all of the t's which means zero through 49 okay next up we have these"}, {"start": 2410.04, "end": 2417.0, "text": " this this parameter called this coefficient called alphas com prod prev like basically"}, {"start": 2417.0, "end": 2423.96, "text": " what we do here is we what we do the right shifting so we add one as the first element"}, {"start": 2423.96, "end": 2432.4, "text": " and then we take the first n minus one elements and and pre like append them to this list"}, {"start": 2432.4, "end": 2438.6, "text": " and we also have this com prod next so where those are used is let me just see i'm fairly"}, {"start": 2438.6, "end": 2444.96, "text": " sure it's going to be used for this mean yeah it's going to be used for the posterior so"}, {"start": 2444.96, "end": 2452.36, "text": " it's basically used for this posterior like forward process so for the mean and for the"}, {"start": 2452.36, "end": 2457.16, "text": " as you can see here for the for the variance okay so we're going to see how we construct"}, {"start": 2457.16, "end": 2462.2799999999997, "text": " those in a second so we we now construct square root of alphas com prod so those are going"}, {"start": 2462.28, "end": 2470.0, "text": " to be used as you can see here so this let me kind of change the color here and you can"}, {"start": 2470.0, "end": 2477.5600000000004, "text": " see this is now the term we constructed here then we create square root one minus alphas"}, {"start": 2477.5600000000004, "end": 2483.48, "text": " com prod which is going to be let's see whether we have that one somewhere here i don't think"}, {"start": 2483.48, "end": 2492.0400000000004, "text": " so but like yeah it's going to be used somewhere later on okay we do the same thing for log"}, {"start": 2492.04, "end": 2498.48, "text": " square root blah blah blah there is a lot of these so this is square root reciprocal"}, {"start": 2498.48, "end": 2504.16, "text": " alphas com prod so that's square root of one over this okay so the square root so this"}, {"start": 2504.16, "end": 2511.2, "text": " expression here is actually used for the component in the loss here you can see it here as i"}, {"start": 2511.2, "end": 2517.32, "text": " said for each of these we'll have we'll basically calculate all of the coefficients we have"}, {"start": 2517.32, "end": 2524.2000000000003, "text": " the posterior variance which is again this expression here let me just kind of let me"}, {"start": 2524.2000000000003, "end": 2528.88, "text": " show you that this is indeed the case so we have betas here are the betas we have one"}, {"start": 2528.88, "end": 2534.56, "text": " minus alphas com prod prep so that's this expression here so that's the numerator of"}, {"start": 2534.56, "end": 2541.56, "text": " our expression here and then we have denominator which is one minus alphas com prod so that's"}, {"start": 2541.56, "end": 2546.84, "text": " it as i said they are now literally going through formulas in this paper and calculating"}, {"start": 2546.84, "end": 2552.48, "text": " all of the necessary coefficients we do the same thing here and then we have this i'm"}, {"start": 2552.48, "end": 2557.56, "text": " going to show you quickly these two so we have the posterior mean coefficient basically"}, {"start": 2557.56, "end": 2566.28, "text": " it's equal to um betas as you can see here and then time square root alphas com prod"}, {"start": 2566.28, "end": 2571.92, "text": " prep so that's this one square root of alphas previous uh and bar okay and then we divide"}, {"start": 2571.92, "end": 2578.7200000000003, "text": " that by one minus this expression here so hopefully i this this convinced you that indeed"}, {"start": 2578.7200000000003, "end": 2582.84, "text": " they're just going on here and calculating all of these coefficients and that's pretty"}, {"start": 2582.84, "end": 2587.04, "text": " much it i'm going to go back into the full screen here and continue with the code okay"}, {"start": 2587.04, "end": 2593.7400000000002, "text": " so this is again not interesting because this is only vital in the case of we doing a sub"}, {"start": 2593.7400000000002, "end": 2599.98, "text": " sampling during sampling which we will not be doing i using less steps during the sampling"}, {"start": 2599.98, "end": 2605.6, "text": " procedure which we will not be using so i can kind of safely ignore all of this and"}, {"start": 2605.6, "end": 2613.36, "text": " finally uh they use uh those betas to construct the uh this gossian diffusion model so this"}, {"start": 2613.36, "end": 2619.76, "text": " one was just used for for this process here for all practical purposes this init function"}, {"start": 2619.76, "end": 2624.56, "text": " here is going to construct the same gossian object we just saw so i'm going to skip i'm"}, {"start": 2624.56, "end": 2630.04, "text": " going to skip everything here and so that's that's it we end up with a gossian diffusion"}, {"start": 2630.04, "end": 2636.4, "text": " object okay guys so we have unit we saw how they literally go formula by formula and compute"}, {"start": 2636.4, "end": 2641.84, "text": " all of that uh inside of the constructor of the gossian diffusion object now let's go"}, {"start": 2641.84, "end": 2647.52, "text": " on further here let's exit all these functions we are back to our main function what they"}, {"start": 2647.52, "end": 2654.12, "text": " do is they just push the model to gpu in case in case you have gpu uh they just now uh create"}, {"start": 2654.12, "end": 2659.44, "text": " this schedule sampler so this one is fairly interesting i'm going to show you how this"}, {"start": 2659.44, "end": 2666.52, "text": " one looks like it basically um specifies how do we want during the training sample uh so"}, {"start": 2666.52, "end": 2674.24, "text": " which steps i which loss components those lt minus ones do we want to train and so what"}, {"start": 2674.24, "end": 2678.72, "text": " they use by default is just a uniform sampler which means all of the steps are equally likely"}, {"start": 2678.72, "end": 2683.2, "text": " but then they also have that other sample which i did mention where depending on the"}, {"start": 2683.2, "end": 2689.3599999999997, "text": " loss of particular lt minus one they'll maybe increase like upscale that weight so that"}, {"start": 2689.3599999999997, "end": 2695.2, "text": " we focus more on uh basically yeah learning that particular uh part of the diffusion process"}, {"start": 2695.2, "end": 2701.8399999999997, "text": " okay let me just kind of go through through here so here is the uniform sampler uh basically"}, {"start": 2701.8399999999997, "end": 2706.3399999999997, "text": " here is how that sampler looks like so the weights of the uniform sample are obviously"}, {"start": 2706.3399999999997, "end": 2711.3999999999996, "text": " all ones so we have number of steps so we have like 50 ones and then during the sampling"}, {"start": 2711.4, "end": 2716.0, "text": " what happens is this sample function is called because as you can see here uniform sampler"}, {"start": 2716.0, "end": 2722.44, "text": " and here is from schedule sampler the subject here and here's what happens so we have weights"}, {"start": 2722.44, "end": 2726.4, "text": " so this is all ones and then we divide here so we normalize and actually have a probability"}, {"start": 2726.4, "end": 2731.48, "text": " distribution because the sum needs to be equal to one as you know uh and then we just call"}, {"start": 2731.48, "end": 2738.92, "text": " mp random choice so we basically do uh this is how we weight all of the uh 50 uh time"}, {"start": 2738.92, "end": 2745.76, "text": " steps and we just uh basically sample uh whatever the batch size is of those of those time steps"}, {"start": 2745.76, "end": 2750.12, "text": " so that's what's going to happen and then some conversion here uh nothing fancy so the"}, {"start": 2750.12, "end": 2754.7200000000003, "text": " interesting sampler is that second one which i'll just kind of briefly show you but i will"}, {"start": 2754.7200000000003, "end": 2759.7200000000003, "text": " not go into it maybe later if we have enough time so last second moment resampler so this"}, {"start": 2759.7200000000003, "end": 2765.84, "text": " is the one where they basically use the loss history to uh decide on how so you can see"}, {"start": 2765.84, "end": 2770.28, "text": " here depending on the loss you'll have bigger weights for for those components where the"}, {"start": 2770.28, "end": 2773.84, "text": " loss is bigger so that's roughly it but we'll not be using this one so i'm going to kind"}, {"start": 2773.84, "end": 2781.08, "text": " of skip it for the for the sake of time okay let's continue here we have our uniform sampler"}, {"start": 2781.08, "end": 2785.76, "text": " we now create the data set so basically i'll be just using cipher that doesn't need to"}, {"start": 2785.76, "end": 2791.88, "text": " be the case they also used image net 64 by 64 in the paper and you can use whatever data"}, {"start": 2791.88, "end": 2797.2400000000002, "text": " set you want and that's it now we have the training loop this is where the actual training"}, {"start": 2797.2400000000002, "end": 2801.8, "text": " is going to happen we pass in the model we put like that's the unit we pass in the diffusion"}, {"start": 2801.8, "end": 2807.6400000000003, "text": " um model on top of it we feed in the data uh batch size micro batch so this is going"}, {"start": 2807.6400000000003, "end": 2812.4, "text": " to be um like uh we're going to be chunking the actual batch into micro batches because"}, {"start": 2812.4, "end": 2818.12, "text": " we like as i said this is fairly memory intensive that's why they have all of these optimization"}, {"start": 2818.12, "end": 2823.3599999999997, "text": " methods such as checkpointing and micro batching to kind of cope with all of that uh excess"}, {"start": 2823.3599999999997, "end": 2829.2799999999997, "text": " memory okay ema so that's your exponential moving average because uh they'll actually"}, {"start": 2829.2799999999997, "end": 2837.52, "text": " be using um uh so ema type of weights for for sampling later on nothing fancy here logging"}, {"start": 2837.52, "end": 2842.56, "text": " saving resuming checkpointing blah blah fp 16 we don't care about mixed precision training"}, {"start": 2842.56, "end": 2847.64, "text": " i just want to focus on diffusion uh we pass in our uniform sampler blah blah blah some"}, {"start": 2847.64, "end": 2853.56, "text": " optimization details like weight decay for the atom w okay we can enter this uh train"}, {"start": 2853.56, "end": 2857.52, "text": " loop let's see what's going on here so we pass in all of these we kind of store them"}, {"start": 2857.52, "end": 2863.7599999999998, "text": " into the internal fields okay um we have all of this i'm going to kind of skip through"}, {"start": 2863.7599999999998, "end": 2870.04, "text": " all of this nothing interesting okay we have our model parameters we have the master parameters"}, {"start": 2870.04, "end": 2875.44, "text": " again this is a uh like a consequence of the training this in a distributed fashion of"}, {"start": 2875.44, "end": 2881.44, "text": " course multiple machines so we for our like for all practical purposes we don't care about"}, {"start": 2881.44, "end": 2888.32, "text": " the this we just have single set of parameters okay syncing again a consequence of of distributed"}, {"start": 2888.32, "end": 2894.48, "text": " training we don't care about it they create the atom w optimizer okay uh now they have"}, {"start": 2894.48, "end": 2901.2400000000002, "text": " for each of the ema uh rates specified uh they create uh like a deep copy of the parameters"}, {"start": 2901.24, "end": 2907.24, "text": " so because we only have one ema rate we'll basically just have a single um like copy"}, {"start": 2907.24, "end": 2912.56, "text": " of the parameters again not that important for you to understand let's go on here this"}, {"start": 2912.56, "end": 2917.6, "text": " is distributed uh data parallel a wrapper in pytorch again we'll not be using that"}, {"start": 2917.6, "end": 2922.8399999999997, "text": " so for all practical purposes we can ignore it let's jump into the actual meat of the"}, {"start": 2922.8399999999997, "end": 2930.7599999999998, "text": " code of the training code so we first go on to sample a single batch of images and potentially"}, {"start": 2930.76, "end": 2936.48, "text": " the classic the classes that correspond to those images uh let me quickly walk you through"}, {"start": 2936.48, "end": 2944.0400000000004, "text": " how this load data looks like basically this is gonna collect all of the um images all"}, {"start": 2944.0400000000004, "end": 2949.5200000000004, "text": " of the paths from my uh data set so i did go on and kind of download those as i told"}, {"start": 2949.5200000000004, "end": 2954.7200000000003, "text": " you so you can see them here and if i kind of step over that uh it's gonna collect all"}, {"start": 2954.72, "end": 2963.2, "text": " of my images and um let me now show you how it looks like whoops if i copy paste the all"}, {"start": 2963.2, "end": 2970.7599999999998, "text": " files and i print the first let's say two paths you can see 006 png and 13 png and those"}, {"start": 2970.7599999999998, "end": 2977.16, "text": " are exactly those are exactly these two first images here so here is that one well it's"}, {"start": 2977.16, "end": 2982.68, "text": " super small so you will not see the cipher after all okay um okay then they go on to"}, {"start": 2982.68, "end": 2989.44, "text": " form this uh image data cell uh where just they just do some resizing cropping but nothing"}, {"start": 2989.44, "end": 2996.0, "text": " too interesting and here we have the data uh loader with batch sizes of 128 and we finally"}, {"start": 2996.0, "end": 3003.96, "text": " yield the batch from our data set okay that ends up giving us a batch of size i guess"}, {"start": 3003.96, "end": 3015.16, "text": " it's gonna be i expect it's gonna be 128 uh 364 64 because it's cipher it's rbg rgb images"}, {"start": 3015.16, "end": 3021.12, "text": " and 128 is the batch size so let's see whether that's the case uh indeed it is the case and"}, {"start": 3021.12, "end": 3024.52, "text": " conditioning we don't have anything so it's gonna be just empty dictionary as you can"}, {"start": 3024.52, "end": 3029.56, "text": " see here okay cool so here's the first step let's now this is the this is where all the"}, {"start": 3029.56, "end": 3033.32, "text": " magic happens i'm gonna ignore everything afterwards because it's just kind of it's"}, {"start": 3033.32, "end": 3037.56, "text": " just your common machine learning boilerplate code so i'm gonna ignore all of that so we"}, {"start": 3037.56, "end": 3044.2400000000002, "text": " have a forward forward backward call here okay so we just use zero gradients of our"}, {"start": 3044.2400000000002, "end": 3049.56, "text": " unit we want to clean those gradients before we recompute them again and then do the update"}, {"start": 3049.56, "end": 3055.6000000000004, "text": " okay so here we load the first micro batch so you can see we just sub sample this batch"}, {"start": 3055.6, "end": 3063.2799999999997, "text": " and we end up with like two three sixty four sixty four because our micro batch dimension is two and we"}, {"start": 3063.2799999999997, "end": 3067.2799999999997, "text": " did the same thing for conditioning which because it's an empty dictionary we don't really care"}, {"start": 3067.92, "end": 3072.88, "text": " we just check whether this is a last patch which it's not because we just started the loop and"}, {"start": 3073.7599999999998, "end": 3079.92, "text": " now this is where we do the uniform sampling of the diffusion process okay so this is this is the"}, {"start": 3079.92, "end": 3088.96, "text": " idea we we now sample t's so the time steps for each of the images in our batch okay so let's do"}, {"start": 3088.96, "end": 3096.96, "text": " that so we we we do that and we end up with two random t's so 44 and three so that means we'll"}, {"start": 3096.96, "end": 3103.6, "text": " take the first image we'll basically add noise 44 times in practice we'll just have a single step"}, {"start": 3103.6, "end": 3108.88, "text": " because of that nice property we saw in the paper and we'll get there and then this second one just"}, {"start": 3108.88, "end": 3113.44, "text": " has three steps of noise and then we try and predict back the noise so that's going to be the"}, {"start": 3113.44, "end": 3119.36, "text": " goal okay so this is the main function in this training code this training losses function we'll"}, {"start": 3119.36, "end": 3124.7200000000003, "text": " see it in a moment we just kind of wrap it up using the func func tools partial so that we don't"}, {"start": 3124.7200000000003, "end": 3131.52, "text": " have to pass these parameters every single time so just kind of convenience wrapper there and again"}, {"start": 3131.52, "end": 3137.2000000000003, "text": " we just you can ignore all of this because that's just distributed code nothing interesting let's"}, {"start": 3137.2, "end": 3142.3999999999996, "text": " focus on the compute losses this is the most important function here okay so what's going on"}, {"start": 3143.3599999999997, "end": 3151.6, "text": " first of all we generate noise that has the same shape as our input image uh our input micro batch"}, {"start": 3151.6, "end": 3159.2799999999997, "text": " so because x star is again two three sixty four sixty four we're going to end up with basically"}, {"start": 3159.2799999999997, "end": 3166.08, "text": " normal gossian tensor of the same shape okay so that's again our noise of the same shape as the"}, {"start": 3166.08, "end": 3171.52, "text": " input images and then we do the q sample so let me remind you what that thing is okay so here is"}, {"start": 3171.52, "end": 3177.12, "text": " a formula that we are going to use uh we are basically going to do this computation here"}, {"start": 3177.12, "end": 3181.84, "text": " actually this computation i showed you here so let's see that that's indeed the case so let me"}, {"start": 3181.84, "end": 3187.04, "text": " just kind of see whether i have a break point here i do have a break point so here we are we are in"}, {"start": 3187.04, "end": 3194.56, "text": " the in that function you can see that uh this is everything we do so we have this extract into"}, {"start": 3194.56, "end": 3199.84, "text": " tensor we'll see what that does but we have this square root alpha scum prod which is this part here"}, {"start": 3199.84, "end": 3204.4, "text": " let me just change the color so okay so we have this part here"}, {"start": 3206.72, "end": 3212.24, "text": " and then we multiply that element wise with x start which is x here which is this thing here"}, {"start": 3212.24, "end": 3220.0, "text": " okay and then we add a square root one minus alpha scum prod which is this part here"}, {"start": 3220.0, "end": 3229.12, "text": " we add up this and we multiply it with with noise as you can see here so we literally are just"}, {"start": 3229.12, "end": 3236.0, "text": " computing formulas from the paper nothing fancy there okay now i'm gonna one time just show you"}, {"start": 3236.0, "end": 3243.92, "text": " how this uh extract into tensor function looks like uh for the sake of your um i guess understanding"}, {"start": 3243.92, "end": 3251.28, "text": " of the code so i'm gonna kind of add a break point there so what we do is the following so we uh take"}, {"start": 3251.28, "end": 3258.56, "text": " that um so the the array we have like the let's say let's say the alpha t bar or whatnot and we"}, {"start": 3258.56, "end": 3265.76, "text": " basically just extract using the time steps so time steps are if you recall like we had um whoops"}, {"start": 3265.76, "end": 3274.4, "text": " whoops time steps we end up with 44.3 so we're gonna end up taking the alpha 44 bar and we're"}, {"start": 3274.4, "end": 3282.5600000000004, "text": " gonna end up taking the alpha 3 bar which are gonna contain uh respectively 44 uh terms uh the"}, {"start": 3282.5600000000004, "end": 3288.6400000000003, "text": " product of 44 terms and the product of three terms and then the only thing we do is literally"}, {"start": 3288.6400000000003, "end": 3294.4, "text": " we add these dummy dimensions as many times as possible so that we have the same dimensionality"}, {"start": 3294.4, "end": 3301.04, "text": " as the image we are doing element wise multiplication with and then we just expand here"}, {"start": 3301.6, "end": 3308.88, "text": " uh the the vector so we we just want to have the same shape of this of this uh yeah so why are we"}, {"start": 3308.88, "end": 3314.08, "text": " doing this well because we cannot multiply a scalar with the tensor and because of that we're just"}, {"start": 3314.08, "end": 3318.8, "text": " basically just copy pasting and broadcasting this this scalar so we end up with a tensor that"}, {"start": 3318.8, "end": 3325.6800000000003, "text": " contains whose all elements contain this particular alpha 44 bar whatnot okay hopefully that was clear"}, {"start": 3325.6800000000003, "end": 3334.0800000000004, "text": " enough and now i'm gonna kind of uh let me first uh ignore this i'm gonna not will not be stepping"}, {"start": 3334.0800000000004, "end": 3340.6400000000003, "text": " into this function anymore and that's it that's it guys we have we have the x t so we have our"}, {"start": 3340.6400000000003, "end": 3347.92, "text": " noisy version of the image right here okay let's see what the next steps are first of all uh because"}, {"start": 3347.92, "end": 3354.4, "text": " our loss is uh gonna be hybrid we're gonna ignore this part and we'll see that we'll actually be"}, {"start": 3354.4, "end": 3359.6, "text": " computing the same thing a bit later down below because hybrid also contains this variational"}, {"start": 3359.6, "end": 3366.16, "text": " lower bound loss as well okay so here we are we have the our loss type is if you recall rescale"}, {"start": 3366.16, "end": 3373.52, "text": " msc and we end up now here so this step is supposed to output the epsilon so we need to learn to"}, {"start": 3373.52, "end": 3381.44, "text": " predict the noise that if you recall uh we basically took that noise and we now we basically using that"}, {"start": 3381.44, "end": 3387.12, "text": " noise we form the x t's now we're trying to predict especially like particularly this noise so this is"}, {"start": 3387.12, "end": 3391.04, "text": " the term we'll be trying to learn how to predict those are the green dots i showed you in the one"}, {"start": 3391.04, "end": 3394.96, "text": " on paper so that's going to be the first three channels the last three channels are going to be"}, {"start": 3394.96, "end": 3402.88, "text": " about the variance and we'll see those uh uh in a bit uh but for now let me show you how this forward"}, {"start": 3402.88, "end": 3408.48, "text": " prop is going to work so this is again unit we're going to do something to the time steps so we're"}, {"start": 3408.48, "end": 3414.32, "text": " going to somehow merge them into this uh together with this uh with the image representation and let"}, {"start": 3414.32, "end": 3419.44, "text": " me quickly show you why we are doing this why are we passing x and t so this is the particular"}, {"start": 3419.44, "end": 3426.08, "text": " expression we are now dealing with here we have our unit so this epsilon t in practice so let me"}, {"start": 3426.08, "end": 3434.88, "text": " again change the color so this thing here is our unit we have this expression is the x t so that's"}, {"start": 3434.88, "end": 3443.2, "text": " the thing we just calculated so that's the x t again x t and we also pass t which are the time"}, {"start": 3443.2, "end": 3447.68, "text": " steps and that's why we have this particular expression that's why we are computing by"}, {"start": 3447.68, "end": 3452.72, "text": " executing this line of code here okay let me let me go back here and let me step uh through this"}, {"start": 3452.72, "end": 3461.8399999999997, "text": " particular thingy uh we can ignore this rep model um basically it serves the purpose of again"}, {"start": 3462.3999999999996, "end": 3468.0, "text": " only if you have the the sub sampling during this the your sampling procedure which will not be"}, {"start": 3468.0, "end": 3473.7599999999998, "text": " doing so none of this really matters and then we do some rescaling nothing that's super interesting"}, {"start": 3473.7599999999998, "end": 3479.12, "text": " just some legacy stuff so that they are comparable with the ddpm original paper etc so here is the"}, {"start": 3479.12, "end": 3486.08, "text": " actual unit forward pass so we have the new ts so this is just kind of some rescaled versions"}, {"start": 3486.64, "end": 3494.4, "text": " but still scalars of our original 44 and 3 if we recall these time steps here and now let's let's"}, {"start": 3494.4, "end": 3499.52, "text": " pass that and this is just empty like the key word arguments are just empty we don't have anything"}, {"start": 3499.52, "end": 3504.24, "text": " there so we just pass the time steps and we pass the images so let me show you the shape here"}, {"start": 3504.24, "end": 3511.8399999999997, "text": " shape here is again 2364 64 okay now let's step through the forward pass of the uh unit model"}, {"start": 3513.2799999999997, "end": 3519.52, "text": " here is where the magic happens first of all we have this time step embedding which is going to"}, {"start": 3520.8799999999997, "end": 3527.6, "text": " map the scalars into certain uh vectors which are just like heuristics basically we're just using"}, {"start": 3527.6, "end": 3532.16, "text": " some sinusoids same thing as in the original transformer paper so let's see how that looks"}, {"start": 3532.16, "end": 3537.92, "text": " like okay so here it is again not gonna dig into the details you can see a bunch of sinusoids"}, {"start": 3537.92, "end": 3545.44, "text": " cosines what ultimately matters is that we end up instead of with scalars we end up with 228"}, {"start": 3545.44, "end": 3550.08, "text": " so we end up with two vectors so now we can we can kind of work with vectors in neural networks"}, {"start": 3550.08, "end": 3557.8399999999997, "text": " it's kind of hard to work with scalars after that okay i stepped into something uh whatever"}, {"start": 3557.84, "end": 3563.04, "text": " okay so now we have this it was actually the the activation function of this time"}, {"start": 3563.04, "end": 3570.2400000000002, "text": " embed and if you recall time embed is this inverse bottleneck of a of a of a mlp so let me just find"}, {"start": 3570.2400000000002, "end": 3577.04, "text": " that thing quickly so time embed where are you here it is it's just a combination of two linear"}, {"start": 3577.04, "end": 3585.2000000000003, "text": " layers uh whereas the this uh inner most uh like layer has forex the uh dimensionality of the input"}, {"start": 3585.2, "end": 3591.7599999999998, "text": " and output layers hence i call it inverse bottleneck and this is just like this times four is stuck with"}, {"start": 3591.7599999999998, "end": 3598.7999999999997, "text": " us since 2017 the original paper people just keep using that again just a simple transformation"}, {"start": 3598.7999999999997, "end": 3604.64, "text": " and we end up with some we'll end up with some uh vector of different dimensionality so let me"}, {"start": 3604.64, "end": 3610.64, "text": " show you that let's kind of step over here we end up with what we end up embedding vector here being"}, {"start": 3610.64, "end": 3616.7999999999997, "text": " two five twelve okay so that's it that's our time information now let's let's insert that time"}, {"start": 3616.7999999999997, "end": 3621.7599999999998, "text": " information somehow so we go through the input blocks remember the first block is going to be"}, {"start": 3621.7599999999998, "end": 3625.8399999999997, "text": " like the com layer and then we're going to have the rest blocks so let's see the the first thing"}, {"start": 3625.8399999999997, "end": 3629.6, "text": " so the first thing nothing nothing super interesting because it's just a com layer"}, {"start": 3630.4, "end": 3637.68, "text": " but if i have hopefully i've added a break point to the rest block let's just find the rest block here"}, {"start": 3637.68, "end": 3644.7999999999997, "text": " uh where are you okay here it is here is the residual block here is where the magic of of"}, {"start": 3644.7999999999997, "end": 3652.3199999999997, "text": " merging happens let me show you this so okay here we are we do some processing on the image that's"}, {"start": 3652.3199999999997, "end": 3658.16, "text": " not that important so these in layers are just like normalization some convolutions whatnot"}, {"start": 3658.64, "end": 3663.68, "text": " and nothing interesting there next up we take our embedding vectors which are again two five twelve"}, {"start": 3663.68, "end": 3671.2, "text": " right so that's going to be two five twelve and we pass them through the embed layers so let's see"}, {"start": 3671.2, "end": 3677.2799999999997, "text": " what those are again that's just like a linear layer uh that does some additional processing on"}, {"start": 3677.2799999999997, "end": 3684.96, "text": " top of those representations nothing super vital let me show you the shape okay two two fifty six"}, {"start": 3685.68, "end": 3692.48, "text": " and now what we do is we just add the dummy dimensions until we have the same shape as this"}, {"start": 3692.48, "end": 3700.08, "text": " as these image features h so h has the shape of as you can see 228 64 64 because of that we're"}, {"start": 3700.08, "end": 3706.56, "text": " going to keep on adding dummy dimensions to to our temporal tensor until we end up with the same"}, {"start": 3708.32, "end": 3714.96, "text": " uh number of dimensions you can see here and now because these two are one one we'll basically be"}, {"start": 3714.96, "end": 3719.76, "text": " broadcasting those so copy pasting the temporal vectors before we combine them with the image"}, {"start": 3719.76, "end": 3724.5600000000004, "text": " features so here's where the magic happens so this is the parameter we saw before the boolean use"}, {"start": 3724.5600000000004, "end": 3730.1600000000003, "text": " scale shift norm that's one option how we can do it you can just also just simply add you can simply"}, {"start": 3730.1600000000003, "end": 3735.84, "text": " add the temporal information to the image features and that's how you can condition the model okay"}, {"start": 3736.96, "end": 3742.88, "text": " but they i guess empirically uh found out that this approach works a bit better instead of just"}, {"start": 3742.88, "end": 3749.2000000000003, "text": " adding up the temporal representations to the image features instead of that let's kind of chunk"}, {"start": 3749.2, "end": 3756.3199999999997, "text": " this so we start with two 256 one one by doing this chunking along the first dimension that means"}, {"start": 3756.3199999999997, "end": 3761.52, "text": " we're going to split this 256 into 128 okay so let me kind of step over there let me show you the"}, {"start": 3761.52, "end": 3768.3999999999996, "text": " scale and shift are going to be 228 11 right the same thing for shift and then this is how they"}, {"start": 3768.3999999999996, "end": 3773.2, "text": " kind of combine the image features with the temporal features one plus scale and then we"}, {"start": 3773.2, "end": 3778.0, "text": " add a shift and we just have some normalization for the image features beforehand and then we do some"}, {"start": 3778.0, "end": 3784.56, "text": " um additional processing that's it guys that's everything there is to to to how this unit works"}, {"start": 3784.56, "end": 3789.84, "text": " i think this is like everything you really need to know i'm gonna remove that breakpoint and"}, {"start": 3789.84, "end": 3797.12, "text": " continue on here uh and that's pretty much it and now we're gonna keep on doing this input blocks"}, {"start": 3797.12, "end": 3802.32, "text": " middle blocks apple blocks we're gonna keep on adding and merging the the temporal information"}, {"start": 3802.32, "end": 3809.44, "text": " and that's it i'm gonna just skip over all of this okay so we've done the for a prop um here we are"}, {"start": 3809.44, "end": 3816.6400000000003, "text": " we are exiting the uh the function so we end up with let's see the shape should be as i said we"}, {"start": 3816.6400000000003, "end": 3827.36, "text": " should have i guess um 264 64 that's the prediction let's see whether that's indeed the case so 264"}, {"start": 3827.36, "end": 3835.6800000000003, "text": " 64 indeed why is that well because basically um uh we we have six channels because the first three"}, {"start": 3835.6800000000003, "end": 3840.6400000000003, "text": " channels again are epsilon which are the noise which we're trying to predict and the next three"}, {"start": 3840.6400000000003, "end": 3848.4, "text": " channels are the variance so let's see how that's gonna be used here um later on so because we have"}, {"start": 3848.4, "end": 3855.36, "text": " uh learned range we're gonna enter this part of the uh this branch here uh we just extract some"}, {"start": 3855.36, "end": 3864.2400000000002, "text": " dimensions so xt is again that's our tensor of noise images 23 64 64 so we're gonna split the"}, {"start": 3864.2400000000002, "end": 3870.4, "text": " output uh into two groups each of with three channels that's what i just explained so model"}, {"start": 3870.4, "end": 3874.88, "text": " output is gonna have the epsilon so that's gonna contain the noise whereas this is gonna contain"}, {"start": 3874.88, "end": 3881.44, "text": " the variance and let me now show you the shapes basically it's gonna be 23 64 64 whereas previously"}, {"start": 3881.44, "end": 3890.88, "text": " we had 26 64 64 okay now what we do here is uh basically we take the epsilon we detach it from"}, {"start": 3890.88, "end": 3899.12, "text": " the computational pie tort graph which means we'll not be updating uh the the the epsilon"}, {"start": 3899.12, "end": 3904.16, "text": " whereas we pass the variance like this and why is that well if you recall from the paper and"}, {"start": 3904.16, "end": 3910.32, "text": " i'll show you that in a second as well we basically just uh in the hybrid loss we use the um"}, {"start": 3910.32, "end": 3917.28, "text": " um this variational uh bound loss to train the variances whereas we use the simplified"}, {"start": 3917.28, "end": 3922.8, "text": " a loss objective to train the mean or in this case to train the epsilon which is equivalent"}, {"start": 3922.8, "end": 3927.28, "text": " because we have that different representations parameterizations so let me show you again the"}, {"start": 3927.28, "end": 3933.76, "text": " paper and compare this side by side okay so here it is basically as i mentioned we have the hybrid"}, {"start": 3933.76, "end": 3940.7200000000003, "text": " loss consists out of a simple and so the variational lower bound objective and they mentioned here"}, {"start": 3940.7200000000003, "end": 3947.5200000000004, "text": " along this same line of reasoning we also apply a stop gradient to the mu of theta or equivalently"}, {"start": 3947.5200000000004, "end": 3953.5200000000004, "text": " to the epsilon theta output for the lvlb term so that means this term is gonna we're gonna freeze"}, {"start": 3953.5200000000004, "end": 3957.76, "text": " we're gonna detach from the computational graph that's why we are doing this this part here okay"}, {"start": 3957.76, "end": 3963.6000000000004, "text": " now let's go on and compute the actual uh lvlb terms so we are now computing literally this"}, {"start": 3963.6000000000004, "end": 3968.32, "text": " we're computing this thing here right that's what we are currently doing and later you'll see in a"}, {"start": 3968.32, "end": 3973.76, "text": " couple seconds we'll be computing the l simple again just executing the formulas from the paper"}, {"start": 3973.76, "end": 3980.0, "text": " that's it do let me know whether this side by side comparison between formulas and code helps you or"}, {"start": 3980.0, "end": 3986.0, "text": " not because these videos are super long and take me a lot of time to to create them so any feedback"}, {"start": 3986.0, "end": 3993.52, "text": " is very much appreciated continuing on here we do that we concatenate those two and then we call"}, {"start": 3993.52, "end": 4000.24, "text": " this uh vb terms bbd so let me just kind of uh see where i have a breakpoint there yes i do"}, {"start": 4001.76, "end": 4010.16, "text": " let's go back to the code okay so what we do here is we pass this dummy model why dummy because"}, {"start": 4010.16, "end": 4016.3199999999997, "text": " because the this function here will later be calling this model but we will not actually be"}, {"start": 4016.3199999999997, "end": 4021.6, "text": " doing a for a prop through the unit as you can see this is how we define the model you basically"}, {"start": 4021.6, "end": 4027.52, "text": " see this is just a like a dummy lambda function that just returns uh so no matter what you pass"}, {"start": 4027.52, "end": 4032.72, "text": " in it's just going to return r which is the frozen out which is this thing here so we'll see some"}, {"start": 4033.6, "end": 4037.68, "text": " model like we'll see a for a prop but it's not going to be a for a prop it's just going to be"}, {"start": 4037.68, "end": 4043.7599999999998, "text": " this dummy return of of what we already computed just keep that in mind okay so what we pass we"}, {"start": 4043.7599999999998, "end": 4049.6, "text": " pass the x start which are the original images we pass the x t which are the images that were noised"}, {"start": 4049.6, "end": 4057.3599999999997, "text": " using uh like t number of steps which we have here and now let's kind of step into this uh"}, {"start": 4057.3599999999997, "end": 4064.96, "text": " function and see how it's computed so what we do is we compute the um posterior mean and variance"}, {"start": 4064.96, "end": 4072.7200000000003, "text": " here by doing that we basically compute the um the forward process posterior and we're going to"}, {"start": 4073.52, "end": 4080.96, "text": " basically do a kl divergence as you can see here between that and between our learned reverse"}, {"start": 4080.96, "end": 4084.8, "text": " process again let me show you the paper to make this a bit more concrete and then we're going to"}, {"start": 4084.8, "end": 4089.92, "text": " dig into the code okay guys so here is the expression we are calculating it's nothing more"}, {"start": 4089.92, "end": 4097.12, "text": " than this so we have the as i said posterior here it is like of the of the uh forward process"}, {"start": 4097.84, "end": 4106.0, "text": " that's this thing here here we have the learned reverse process so this thing here corresponds to"}, {"start": 4106.0, "end": 4114.0, "text": " this lines here the q part the posterior of the of the for process corresponds to this line here"}, {"start": 4114.0, "end": 4119.84, "text": " q posterior mean variance and then we have the kl divergence which is just this part here again"}, {"start": 4119.84, "end": 4125.28, "text": " this is just this part here kl of those two distributions and those are the l t minus one"}, {"start": 4125.28, "end": 4132.48, "text": " terms so the variational lower bound terms that's it nothing nothing too fancy now let's kind of dig"}, {"start": 4132.48, "end": 4139.12, "text": " and see how that's going to be computed so we pass the x start x t and t which are the original"}, {"start": 4139.12, "end": 4148.16, "text": " image noist image and t the time steps so let me kind of start uh enter this function again so it"}, {"start": 4148.16, "end": 4153.76, "text": " even says here what it's computing you can see here so it's very nice to add this type of comments"}, {"start": 4153.76, "end": 4161.12, "text": " that helps a lot to be honest so here it is how we compute this is we take this mean coefficient"}, {"start": 4161.12, "end": 4167.04, "text": " one we multiply it with x start we take the mean coefficient two we multiply it with x t and that's"}, {"start": 4167.04, "end": 4171.84, "text": " it we end up with a posterior mean again let me show you this side by side with the paper here it"}, {"start": 4171.84, "end": 4178.96, "text": " is guys so we saw this already actually so we have again posterior mean coefficient one is just this"}, {"start": 4178.96, "end": 4184.24, "text": " thing here so we actually computed that if you recall in the constructor of the Gaussian diffusion"}, {"start": 4185.04, "end": 4190.8, "text": " object so this is the the first coefficient here then we multiply that with x zero as you can see"}, {"start": 4190.8, "end": 4196.24, "text": " here x start just a different notation and then we have posterior mean coefficient two so that's"}, {"start": 4196.24, "end": 4202.16, "text": " this part here and we multiply with x t so that's this part here so that's it that's how this formula"}, {"start": 4202.16, "end": 4209.76, "text": " corresponds to this line here let me now continue here let me step over all of this we end up with"}, {"start": 4209.76, "end": 4217.12, "text": " the posterior mean we do the same thing with a posterior uh variance so that's just those betas"}, {"start": 4217.12, "end": 4222.88, "text": " again here are those betas that's what we just computed here that's the posterior variance and"}, {"start": 4222.88, "end": 4228.24, "text": " the posterior log variance i'm not sure how this one is going to be used we'll see that a bit later"}, {"start": 4228.24, "end": 4235.6, "text": " hopefully or maybe not okay just some assertions and that's it we computed uh the uh basically"}, {"start": 4235.6, "end": 4240.16, "text": " this distribution we because it's a Gaussian we can just return the mean and the variance and that's"}, {"start": 4240.16, "end": 4247.12, "text": " it that's perfectly describing the Gaussian okay now we do the p mean variance so we're now doing"}, {"start": 4247.12, "end": 4253.2, "text": " um that other step so again that's that's this part here this is what we're trying to compute at the"}, {"start": 4253.2, "end": 4259.599999999999, "text": " moment here let's see how that's gonna look like uh let me see whether i have a break point inside"}, {"start": 4259.599999999999, "end": 4266.5599999999995, "text": " of here yes i do so we passed uh again model remember here model is just uh dummy it's just"}, {"start": 4266.5599999999995, "end": 4271.84, "text": " going to return what we already computed beforehand so the epsilon and the variances and we pass in x"}, {"start": 4271.84, "end": 4281.04, "text": " t and t so that's the the noist image and uh time steps okay let's enter here again uh it says here"}, {"start": 4281.04, "end": 4289.12, "text": " we are basically computing this we are computing the um p mean variance okay so let me step over"}, {"start": 4289.12, "end": 4296.24, "text": " this nothing fancy there again uh this does nothing this just returns this just returns"}, {"start": 4296.24, "end": 4299.68, "text": " we're going to see how we are not going to enter the for function we're just going to return here"}, {"start": 4299.68, "end": 4307.360000000001, "text": " okay so we just end up with the uh tensor we already previously computed so it's 264 64 let's"}, {"start": 4307.360000000001, "end": 4315.360000000001, "text": " see how how this is going to be computed so because we are actually our model var type so we are we"}, {"start": 4315.360000000001, "end": 4319.76, "text": " have the we as i said we are we are learning the variances that's why we enter this branch here"}, {"start": 4319.76, "end": 4329.360000000001, "text": " we split the model output again into the epsilon and variances and now because we are um learned"}, {"start": 4329.360000000001, "end": 4334.400000000001, "text": " range and not learned we enter the else branch and as you can see here we're now calculated the"}, {"start": 4334.400000000001, "end": 4339.6, "text": " equation 15 from the improved ddpm so uh paper okay let me show you that side by side okay so"}, {"start": 4339.6, "end": 4345.2, "text": " here it is guys here's what we are computing this equation 15 so when i said we have variance i was"}, {"start": 4345.2, "end": 4349.92, "text": " we have variance i was kind of lying because that's not the case that's just a component"}, {"start": 4349.92, "end": 4355.04, "text": " that when you combine it like this then you end up with the variance okay so let me let me show you"}, {"start": 4355.04, "end": 4362.16, "text": " what they do here let me kind of remove this uh panel here uh okay so they have the mean log so"}, {"start": 4362.16, "end": 4368.5599999999995, "text": " they have the posterior log variance clipped so that's going to be this part here so that's where"}, {"start": 4368.56, "end": 4375.120000000001, "text": " the log part comes into play so this is the the part that they compute here it says the mean log"}, {"start": 4375.120000000001, "end": 4378.96, "text": " and then they compute the max log which is this part here"}, {"start": 4381.4400000000005, "end": 4386.400000000001, "text": " actually because this is betas it's why it's the other way around but okay doesn't matter that much"}, {"start": 4386.400000000001, "end": 4393.92, "text": " you get the point uh finally we have this um just normalization of the output the output is minus one"}, {"start": 4393.92, "end": 4399.36, "text": " one they add plus one so that's zero two and then divide dividing by two we bring it into the zero"}, {"start": 4399.36, "end": 4405.92, "text": " one uh range i guess and then we just do literally this equation here so you can see v times this"}, {"start": 4405.92, "end": 4412.64, "text": " plus this blah blah same equation so this line is literally this equation here and then we just add"}, {"start": 4412.64, "end": 4418.24, "text": " the additionally this exponent part so that's this part and we've successfully computed equation 15"}, {"start": 4418.24, "end": 4424.96, "text": " okay let's continue on here this is just some clipping we can ignore it for now let me zoom in"}, {"start": 4424.96, "end": 4431.679999999999, "text": " here uh we are not predicting previous x we are predicting epsilon that's why we enter here"}, {"start": 4432.88, "end": 4440.16, "text": " and we enter this branch so we first want to predict the the x start uh given the um"}, {"start": 4440.16, "end": 4448.96, "text": " um epsilon so the noise given the currently noisy images and time step we want to predict"}, {"start": 4448.96, "end": 4454.88, "text": " the x here i will see why we're using that in a second so let me kind of step into this and show"}, {"start": 4454.88, "end": 4461.76, "text": " you this part so here we are we're predicting x star from epsilon uh here is how we do that"}, {"start": 4463.44, "end": 4467.84, "text": " let me again have the equation on the side so we'll be able to understand what's going on here"}, {"start": 4467.84, "end": 4473.28, "text": " okay guys this is the equation we are computing so we recall that xt is computed like this"}, {"start": 4474.4800000000005, "end": 4480.400000000001, "text": " if we just rearrange the terms and we compute x zero x zero turns out to be this expression here"}, {"start": 4481.4400000000005, "end": 4486.64, "text": " and that's what we are currently computing here okay so here are the square root reciprocal"}, {"start": 4486.64, "end": 4492.64, "text": " alphas come prods that's literally this square root blah blah this thing here then we multiply"}, {"start": 4492.64, "end": 4501.52, "text": " that with xt you can see xt here then we have minus uh square root reciprocal minus alpha blah blah"}, {"start": 4501.52, "end": 4507.52, "text": " so that's literally this term here you can just read it kind of word by word and understand that"}, {"start": 4507.52, "end": 4513.04, "text": " that's this term here and we multiply that by epsilon and that's it so that's how we calculate"}, {"start": 4513.04, "end": 4525.12, "text": " the x zero or x start uh let's go back here and now we're going to use that to find the model mean"}, {"start": 4525.12, "end": 4530.48, "text": " and by model mean i literally mean this expression here no pun intended so this expression here okay"}, {"start": 4531.36, "end": 4539.12, "text": " so let's see how this is going to work uh posterior mean variance i think this is well yeah this is"}, {"start": 4539.12, "end": 4544.5599999999995, "text": " the expression we already saw uh we already kind of computed this thing here before let me kind of"}, {"start": 4544.5599999999995, "end": 4551.28, "text": " zoom in uh so i can kind of skip through this yeah i'm gonna skip through this part here we're going"}, {"start": 4551.28, "end": 4557.68, "text": " to ignore this because we already saw it and that's it we end up with our mean and that's what we"}, {"start": 4557.68, "end": 4563.76, "text": " return back we return back the mean the variance the log variance and the predicted so the x start"}, {"start": 4563.76, "end": 4571.04, "text": " and after that we just here pass the true mean true log variance that's the the ground truth"}, {"start": 4571.68, "end": 4576.64, "text": " and we want to minimize the scale divergence so we want to basically be as close as possible to"}, {"start": 4576.64, "end": 4582.4800000000005, "text": " that distribution we pass our mean log variance we calculate the k l do some normalization blah"}, {"start": 4582.4800000000005, "end": 4587.92, "text": " blah okay finally there is this step this is l zero so in case that the time step was zero we do"}, {"start": 4587.92, "end": 4593.280000000001, "text": " some different computation if you recall from the variational lower bound loss uh let me show you"}, {"start": 4593.28, "end": 4597.44, "text": " uh let me show you that side by side okay guys so here it is so here are the l t minus ones we"}, {"start": 4597.44, "end": 4604.88, "text": " were computing so those are the k l divergence we here compute this expression minus log p uh"}, {"start": 4605.759999999999, "end": 4611.2, "text": " data x zero condition like x one so that's what we compute here this discretized gaussian likelihood"}, {"start": 4611.2, "end": 4616.4, "text": " blah blah blah that's how we get the negative log likelihood decoder negative log likelihood"}, {"start": 4616.4, "end": 4623.5199999999995, "text": " and that's pretty much it and now we uh compute depending on whether the as i said depending on"}, {"start": 4623.5199999999995, "end": 4630.719999999999, "text": " whether t equals zero if it's equal zero then we have decoder nl that's what we use as the loss"}, {"start": 4630.719999999999, "end": 4638.48, "text": " otherwise we use k l fairly simple fair enough okay let me kind of zoom in here and that's it"}, {"start": 4638.48, "end": 4643.36, "text": " we have our first term that's the vb the variational lower bound term and we just do some"}, {"start": 4643.36, "end": 4651.28, "text": " rescaling that's why the loss is called rescaled msc let's continue i'm going to ignore this part"}, {"start": 4651.28, "end": 4659.28, "text": " here uh because we'll be extracting epsilon which is noise so we can kind of ignore everything else"}, {"start": 4659.28, "end": 4664.16, "text": " here so let me just uh make sure this doesn't have any yep doesn't have any break point so we can"}, {"start": 4664.16, "end": 4671.04, "text": " kind of step over this we end up with target it's just going to be the noise uh and here we just do"}, {"start": 4671.04, "end": 4677.84, "text": " msc between the model output which are the epsilons that we predicted from the unit and we just do msc"}, {"start": 4677.84, "end": 4682.72, "text": " here between that and the target which is the noise and that noise let me see whether i can find"}, {"start": 4682.72, "end": 4688.0, "text": " it whether it's in this function okay and this is the noise that we initially created here before"}, {"start": 4688.0, "end": 4695.36, "text": " we even started creating these xt's that's it it's fairly actually simple um to be super frank here"}, {"start": 4695.36, "end": 4701.599999999999, "text": " it's not easy to understand why this exactly works it's kind of still magical to me but i'm slowly"}, {"start": 4701.599999999999, "end": 4706.16, "text": " learning about these diffusion models more and more i think the first time i kind of took some"}, {"start": 4706.16, "end": 4710.88, "text": " time to understand these was when i was preparing the glide paper so again do check out that paper"}, {"start": 4710.88, "end": 4715.839999999999, "text": " i do have some short introduction maybe even better than the one i made in this video when it"}, {"start": 4715.839999999999, "end": 4722.48, "text": " comes to understanding diffusion models but in any case finally because we have the hybrid loss we"}, {"start": 4722.48, "end": 4729.04, "text": " just combine the msc and the variational lower bound loss and that's it that's our loss after this"}, {"start": 4729.04, "end": 4736.24, "text": " we just do the um uh like basically do some waiting uh and for each of the time steps"}, {"start": 4737.04, "end": 4743.44, "text": " uh and because we use uniform sampling we'll just have ones everywhere uh do some logging blah blah"}, {"start": 4743.44, "end": 4747.5199999999995, "text": " blah and finally we do the computation of the gradients by calling the backward function"}, {"start": 4747.52, "end": 4754.56, "text": " on our hybrid loss here so that's pretty much it next up we just keep on repeating micro batches"}, {"start": 4754.56, "end": 4760.56, "text": " blah blah blah uh i'm gonna kind of stop here this is everything you need to know to understand how"}, {"start": 4760.56, "end": 4766.0, "text": " the how the training procedure works hopefully that was useful now let's quickly dig into the"}, {"start": 4766.0, "end": 4773.120000000001, "text": " sampling code and then we are done so i'm gonna stop this training here i'm gonna enter the"}, {"start": 4773.12, "end": 4780.24, "text": " uh sampling script so that's gonna be let me find it here so it's gonna be the image sample function"}, {"start": 4780.24, "end": 4786.72, "text": " here the script here uh and let's start uh with the main so i'm gonna now start and run this one"}, {"start": 4787.84, "end": 4793.5199999999995, "text": " i'm gonna set the sample configuration from my launchpy in vs code by the way i love vs code"}, {"start": 4793.5199999999995, "end": 4798.96, "text": " he has beautiful design uh amazing debugging experience and i mean i don't know why you would"}, {"start": 4798.96, "end": 4804.8, "text": " use anything else unless you don't have a choice i guess that's that's a reasonable uh excuse i guess"}, {"start": 4805.76, "end": 4810.24, "text": " okay let's step over this again bunch of arguments we're gonna ignore distribution"}, {"start": 4810.24, "end": 4819.2, "text": " blob uh distributional training logging um so i'll skip the model creation as well so i'm just gonna"}, {"start": 4819.2, "end": 4825.2, "text": " disable all of the breakpoints and i'm gonna step over all of this because we already saw all of"}, {"start": 4825.2, "end": 4830.5599999999995, "text": " that so i'm gonna ignore it now we'll load the actual model they do provide the checkpoints"}, {"start": 4831.44, "end": 4837.76, "text": " in their github repost to do check it out you can download it and then just set it here so i can"}, {"start": 4837.76, "end": 4844.639999999999, "text": " have it stored somewhere so it's an image net 64 64 unconditional model trained on 100 million steps"}, {"start": 4844.639999999999, "end": 4853.84, "text": " i guess so we load it we put it onto to gpu we set it into eval mode and we now start creating the"}, {"start": 4853.84, "end": 4859.360000000001, "text": " samples okay so because this is not a class conditional model we can ignore all of this part"}, {"start": 4859.360000000001, "end": 4867.68, "text": " and now this is where the magic happens the sample function first of all there is this paper called"}, {"start": 4867.68, "end": 4873.2, "text": " ddim i might cover this in the next uh video but for now we're just going to ignore it because this"}, {"start": 4873.2, "end": 4878.64, "text": " is false we're going to be using the p sample loop this function here that's the one we're going to"}, {"start": 4878.64, "end": 4886.400000000001, "text": " be using not the ddim it turns out that ddim in a nutshell works better when you only have 50 or"}, {"start": 4886.400000000001, "end": 4891.84, "text": " less time steps during your sampling procedure as soon as you pass the 50 threshold roughly"}, {"start": 4891.84, "end": 4898.88, "text": " then this method here operates better and this is just like the the method from the improved"}, {"start": 4898.88, "end": 4904.240000000001, "text": " ddpm paper the one i showed you in the beginning of the video okay so here's the sampling function"}, {"start": 4904.24, "end": 4910.639999999999, "text": " this is our desired shape we want to have one three sixty four sixty four images this is the image we"}, {"start": 4910.639999999999, "end": 4916.639999999999, "text": " want to generate uh this is kind of empty okay let's enter this function and let's see how it"}, {"start": 4916.639999999999, "end": 4923.12, "text": " works okay before that let me just enable the break points so enable all breakpoints okay now"}, {"start": 4923.12, "end": 4930.48, "text": " we can enter that here let me enter and here we are so first of all there's going to be this um"}, {"start": 4930.48, "end": 4937.679999999999, "text": " um p sample loop progressive uh basically this thing is going to keep on producing the samples"}, {"start": 4938.5599999999995, "end": 4945.759999999999, "text": " we pass in the model uh we pass in the desired shape uh noise is none we don't pass anything here"}, {"start": 4945.759999999999, "end": 4950.08, "text": " we don't specify this blah blah blah this is just boilerplate i'm going to ignore all of that"}, {"start": 4950.08, "end": 4957.599999999999, "text": " so here we are p sample loop progressive again we just pick a device uh it's going to be gpu"}, {"start": 4957.6, "end": 4964.96, "text": " in my case because i have uh like gpu here um then we generate the image so here's how we start we"}, {"start": 4964.96, "end": 4970.320000000001, "text": " literally start with the gaussian noise so this is your your normal distribution so so basically"}, {"start": 4970.320000000001, "end": 4977.120000000001, "text": " gaussian with mean zero and variance one okay with the desired shape on the desired device"}, {"start": 4977.92, "end": 4984.0, "text": " let's generate that image now we have some number of time steps which is set to 100"}, {"start": 4984.0, "end": 4989.52, "text": " again i think i just modified this for the sake of time otherwise i think default was a bit bigger"}, {"start": 4989.52, "end": 4995.36, "text": " like 4 000 um in any case so we generate the indices and you can see here we reverse them"}, {"start": 4995.36, "end": 5003.2, "text": " because now we are doing the uh the reverse process we start with the uh 99 with the index 99 and we"}, {"start": 5003.2, "end": 5008.48, "text": " go all the way to zero which is the original image when i say original image here this is a"}, {"start": 5008.48, "end": 5013.6, "text": " generated image from the underlying data distribution that we learned during the training procedure"}, {"start": 5013.6, "end": 5020.08, "text": " that's the that's the idea there okay let's step over here so let's go over the indices we start"}, {"start": 5020.08, "end": 5028.64, "text": " with the index 99 so this is going to be first 99 okay we kind of um well you can see here because"}, {"start": 5028.64, "end": 5033.76, "text": " batch dimension is one nothing fancy happens here but otherwise if you wanted to generate like five"}, {"start": 5033.76, "end": 5039.52, "text": " images in the batch then you would just copy paste uh 99 five times because for all of the all of the"}, {"start": 5039.52, "end": 5044.080000000001, "text": " noisy images will be following the same process so that's where we start with with the last time"}, {"start": 5044.080000000001, "end": 5049.200000000001, "text": " step okay and in this function is where all the magic actually happens everything else was kind"}, {"start": 5049.200000000001, "end": 5055.120000000001, "text": " of boilerplate let's let's focus on this one so we pass the image which is again in the initial"}, {"start": 5055.120000000001, "end": 5060.8, "text": " point so now in the initial step it's going to be just pure image pure sorry pure noise we pass the"}, {"start": 5060.8, "end": 5066.72, "text": " time step uh blah blah blah nothing fancy there let's enter the p sample and here is how the"}, {"start": 5066.72, "end": 5071.84, "text": " sampling works and actually we already saw this during the training so you pretty much know how"}, {"start": 5071.84, "end": 5078.240000000001, "text": " this is going to work let me enter here uh this is that part of code where we were literally"}, {"start": 5078.240000000001, "end": 5086.400000000001, "text": " computing the um we split the model output the model variance then we compute those mean log max"}, {"start": 5086.400000000001, "end": 5091.52, "text": " log so that's the equation i saw the equation 15 from the paper we get the variance and then later"}, {"start": 5091.52, "end": 5100.72, "text": " down below what we do is we basically predict the x0 and then use that to to compute model mean so we"}, {"start": 5100.72, "end": 5106.96, "text": " we already kind of stepped through this code so i'm not going to do that now so yeah because of"}, {"start": 5106.96, "end": 5113.52, "text": " that let me kind of step over here and ignore this so i'm going to just kind of disable all of the"}, {"start": 5113.52, "end": 5122.8, "text": " breakpoints let me disable all of the breakpoints let me step to here and let me click f5 to step"}, {"start": 5122.8, "end": 5132.240000000001, "text": " over okay we end up here uh now i'm going to enable the breakpoints again in any case so we"}, {"start": 5132.240000000001, "end": 5139.68, "text": " generate the noise again this is your normal distribution uh we generate these masks uh which"}, {"start": 5139.68, "end": 5146.8, "text": " basically are always going to be one if t is equal is different from zero because we treat the zero"}, {"start": 5146.8, "end": 5153.4400000000005, "text": " the zero step differently if you recall even during the training we had that nl loss uh as opposed to"}, {"start": 5153.4400000000005, "end": 5159.04, "text": " variational lower bound loss similarly here when we when we sample we're going to have a bit different"}, {"start": 5159.04, "end": 5165.68, "text": " behavior depending on the number of time steps so we just take the mean that's the mean we computed"}, {"start": 5165.68, "end": 5172.0, "text": " and then we take the log variance we do the xp xp and log cancel out we get the variance we"}, {"start": 5172.0, "end": 5179.4400000000005, "text": " multiply that with the noise and we end up with a sample okay so this is now we are in step 98"}, {"start": 5179.4400000000005, "end": 5185.84, "text": " okay and we're going to keep on doing this until we get a completely denoised image and at the end"}, {"start": 5185.84, "end": 5192.64, "text": " once we end up with the zeroth time step this is going to be equal to zero and that means the only"}, {"start": 5192.64, "end": 5198.64, "text": " result we'll return is the final mean again that's just a detail of how this thing works and we return"}, {"start": 5198.64, "end": 5205.92, "text": " the sample that's it basically the sample now becomes image that we feed into the next iteration"}, {"start": 5205.92, "end": 5211.68, "text": " of this p sample function and that's how it pretty much works i'm gonna now stop this because you"}, {"start": 5211.68, "end": 5217.84, "text": " pretty much saw everything i'm gonna go to launch i'm going to go to sample i'm gonna increase the"}, {"start": 5217.84, "end": 5225.04, "text": " number of diffusion steps to let's say 500 and i'm gonna run the script again and just show you the"}, {"start": 5225.04, "end": 5230.400000000001, "text": " results we get this time we're going to disable all of the breakpoints i'm gonna run this and"}, {"start": 5230.400000000001, "end": 5236.16, "text": " just show you how the results look like let me show you yeah i added this image show plot show"}, {"start": 5237.12, "end": 5243.360000000001, "text": " line so that we can visualize the actual sample and yeah i'll i'll get back to you as soon as the"}, {"start": 5243.36, "end": 5252.719999999999, "text": " image is generated okay guys here's the image we got some dog image again do do keep in mind that"}, {"start": 5252.719999999999, "end": 5261.2, "text": " i only have 500 steps it'd be much better if i used more steps obviously like thousand although"}, {"start": 5261.2, "end": 5265.839999999999, "text": " there is some bug with this code when you use 4 000 you can check out the issue in their repo"}, {"start": 5266.639999999999, "end": 5272.96, "text": " basically you get like completely saturated image so that means that the optimum quality of an image"}, {"start": 5272.96, "end": 5277.92, "text": " is maybe around thousand two thousand or something but like at four thousand the images hit the"}, {"start": 5277.92, "end": 5284.4, "text": " saturation point and you get like just pretty much junk out of these of this model guys this is pretty"}, {"start": 5284.4, "end": 5290.64, "text": " much it hopefully you found this format of a video useful do let me know whether you find it useful"}, {"start": 5291.44, "end": 5297.6, "text": " if so i'll keep on making videos such as this one combining papers combining code putting code in"}, {"start": 5297.6, "end": 5306.400000000001, "text": " paper side by side trying to make these abstract mathematical ideas a bit more grounded and hopefully"}, {"start": 5306.400000000001, "end": 5311.92, "text": " i've done a job at doing that in any case if you found this video useful do share it out with others"}, {"start": 5311.92, "end": 5317.68, "text": " who want to learn more about diffusion processes subscribe to this channel join the discord"}, {"start": 5317.68, "end": 5335.200000000001, "text": " community you can find the link down below in the video description and until next time bye bye"}]