Diffusion Models Beat GANs on Image Synthesis | ML Coding Series | Part 2
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

4th video in the ML coding series! In this one I continue explaining diffusion models! I cover the "Diffusion Models Beat GANs on Image Synthesis" paper and the code behind it.

I focus on how classifier guidance works. I cover both the training of the noise-aware classifier as well as the actual sampling (the mean shift method). I also walk you through a minor bug in their code.

Let me know how you find this format!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ GitHub: https://github.com/openai/guided-diffusion
✅ My issue: https://github.com/openai/guided-diffusion/issues/51
✅ Paper: https://arxiv.org/abs/2105.05233
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00:00 Intro
00:01:30 Paper overview part - U-Net architecture improvements
00:05:38 Classifier guidance explained
00:15:18 Intuition behind classifier guidance
00:20:10 Scaling classifier guidance
00:24:10 Diversity vs quality tradeoff and future work
00:26:15 Coding part - training a noise-aware classifier
00:35:35 Main training loop
00:44:26 Visualizing timestep conditioning
00:46:00 Sampling using classifier guidance
00:52:35 Core of the sampling logic
00:59:20 Shifting the mean - classifier guidance
01:05:03 Minor bug in their code and my GitHub issue
01:07:53 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#diffusion #generativemodeling #coding
 What's up guys in this video I'm continuing with the machine learning coding series and I'll be covering a follow-up paper to the previous video. So in the last one we covered the improved DDPMs or denoising diffusion probabilistic models paper in this one I'm gonna cover the follow-up work called the diffusion models speed cans on image synthesis Paper and the associated code. So the idea will be to To again for me to give you a brief context like 10-15 minutes short overview of the most salient parts of the paper and After that we're gonna dig into the code and analyze two scripts So one is this classifier train and the other one is classifier sample. So basically I want to focus on The guiding so how do we guide? diffusion using pre trained noise classifiers So understand what that means in 10-15 minutes In any case what I what I've done is I cloned this repo so the guided diffusion from open AI They have like everything all the instructions you'll need I had to do some modifications again If I get enough enough votes down in the in the comments, I'll just push my my changes So I had to make minor changes to make this to work on my machine in any case you can you can basically download various checkpoints here and just follow the Instructions they they have in the readme which are super nice and yeah, you should be ready to go without further Do let's first start with the paper. Okay, so as I said, it's diffusion models beat cans on image synthesis and Two main innovations here are they improve the unit architecture from their previous work So from the improved DDP M's paper and the second improvement is they find a way how to condition an additionally condition the model so that they can achieve higher quality so they can trade off the Diversity versus the quality of a sample which is something that previous generative models such as GANs used to achieve using this truncation Truncation trick as well as some autoregressive models can simply modify the temperature and thus Control the like diversity versus the quality of the of the sample So we'll see in this paper how they achieve the same thing for for the class of generative models called diffusion models so I strongly suggest you you watch my previous video if you haven't because I'm gonna kind of assume some knowledge from that one But yeah, you can you can continue watching this one as well. Okay, so So they say here we hypothesize that the gap between diffusion models and GANs stems from at least two factors So first one that the model architecture is used by recent GAN literature have been heavily explored and refined And we saw the same thing happening with in the Comnex paper if you recall that one Where like basically they people showed like people focused a lot on transformers and there was a lot of innovation that happened on that side of the spectrum whereas the CNNs were not as as explored and then what Comnex showed that if you take some of the innovations that were Well invented in the context of transformers and you adopt that into CNNs you get much better models And so similarly here they took some ideas from other GAN papers And they took those ideas and then integrated them into the fusion models into the architecture and that Improved the results quite quite nicely Anyways, so that's the first thing the second thing is that GANs are able to trade off diversity for fidelity Producing high quality samples, but not covering the whole distribution. So that's the second thing I mentioned previously So they do that using this truncation trick. You can check out the the style GAN paper I think that's that's the one where I recall that they have that they're using the the truncation trick in any case Let me quickly show you the Dimensions across which they they kind of explored And how to improve the architecture so one is increasing depth versus width holding model size relatively constant Increasing the number of attention heads using attention at 32 times 32 resolution as well as 16 By 16 and not and 8 by 8 Rather than only at 16 by 16 using big GAN residual block for up sampling and down sampling the activations And rescaling residual connections with one over square root of two. Okay. So those are some of the ideas they they kind of played with And you can see here some of the curves here after you Take the best of out of those changes after a lot of experimentation I guess they end up having the the the best architecture here So you can see here number of channels 160 number of blocks two blah blah blah Anyways, not that important. I'm going to focus on the um like guiding the diffusion much more than on the architecture So here again just to to wrap this up the architecture part in the rest of the paper We use this final improved model architecture as our default Variable width with two residual blocks per resolution multiple heads with 64 channels per head attention at 32 16 and 8 resolutions Big GAN residual blocks for up and down sampling and adaptive group normalization for injecting time step and Class embeddings into residual blocks you might recall this one from my previous video So this adaptive group normalization for injecting So this one adaptive for injecting time step and class embeddings So do check it out if you want to know the the details how that is how that works Okay, so this is the important part So classifier guidance, that's what we want to understand a bit better They meant to be a little bit more complex That's what we want to understand a bit better. They mentioned here. We already incorporate class information into normalization layers So if you recall from the previous video basically the same way we we incorporate the the time step information Through that same pathway They incorporate the class information. So that means they already had class condition diffusion models But the thing is they show in this paper that using this classifier guidance is complementary to the class conditioning And because of that they achieve the best results With with class conditioning obviously so with classifier conditioning So let's let's see what it is basically So here we explore a different approach compared to the one I just mentioned so exploiting a classifier p Uh y uh conditioned on x so basically you input x and you get like a distribution across labels That's how you're going to implement this abstract distribution here To improve a diffusion generator So blah blah blah the previous paper show one way to achieve this wherein a pre-trained diffusion model can be conditioned using the Gradients of a classifier. Okay, that's the important part. So so in particular we can train a classifier p parameterized by parameters phi and conditioned on x of t And t so xt is basically the the if you recall from the that this is the forward so again diffusion uh, just a quick reminder is you start with uh, like a Fresh image. Let's let's uh, let me draw it like here. So we have a fresh image here And then what you do is you gradually stop adding more and more noise so Basically here you had like some image and then you have here the same image And basically you start adding gaussian noise So you're going to add some noise here and then you you keep repeating that that that process uh multiple times Uh, basically, uh, like for example 1000 4000 are our numbers which makes sense And so at the end well actually at the end you'll end up with a completely noised image So at the end you'll just have a pure gaussian noise here I'm just gonna like draw it like this basically it's pure noise and what you're trying to learn is how to Go backwards. So this is the forward part. This is the forward direction and we want to learn how to go backwards So we want to learn how to start from noise and then gradually basically generate the image from the underlying Probability distribution which we have learned during the training procedure. Okay? So x of t is basically the t-th step of this forward process. So we we take the x-th image here So there's going to be like some image here and that's like t-th time step This is going to be this is usually denoted as 0-th. This is usually denoted as big t And that's or t minus 1 whatever in any case So that's that's what this x of t is and t is just a scalar which we're gonna obviously Encode using the usually they use the like sinusoid embeddings same as for the original transformer paper Okay, so in particular we can train a classifier which I just explained on noisy images And then use gradients of that classifier as you can see here So to guide the diffusion sampling process towards an arbitrary class label y So the idea here is This gradient here tells you how you need to tweak the current image So x of t how you need to tweak it such that you maximize the class label of your interest So let's say you want to you want to create a dog image Then you want to make sure that so this will tell you how to tweak the image such that the dog class is maximized i.e. how to yeah basically how to tweak the diffusion process such that Basically, you start guiding it towards this part of the space where the dog images lie in We're gonna see those details in a second, but like that's that's that's a rough intuition for now. Okay, let's continue here Here just briefly they briefly mentioned the notation they're gonna omit the t part because it's kind of Redundant in the sense that well just for concise and sake basically. It's not redundant It's just easier to write it down like that. Okay, so So here's how we're gonna do this. We start with the diffusion model with an unconditional reverse Noising process so p parameters by theta and the idea is to given x t plus one So that's the image with more noise You want to predict the next step of the reverse process as I previously explained Here, okay. So that's that's this part. We want to learn how to go how to you how to go from more noise to less noise in any case To condition this on label y it suffices to sample each transition according to This distribution here. So if we want to sample from Class conditions so you can see here. We additionally have the class information here It can be shown and they have a like a whole derivation in the appendix h So if you want to dig into the maths you can go there. I'm gonna kind of assume this to be like true Like as an axiom because otherwise will take too much time for me to explain the maths And in any in any case you can see here. So that is just an arbitrary context Constant and we we already have this part here. So this is the unconditioned Reverse process Reverse process which we learned and here this is going to be the classifier that we're going to train On the noisy images where by noisy images, I mean Images taken from arbitrary time step of the forward diffusion process. Okay, that's the basic idea So if you take this as as given Now let's kind of derive how can we actually sample from this distribution on the right hand side because as they mentioned here Uh, it is typically interactable to sample from this distribution exactly Okay, so that's that's the that's the idea Now let's see how we can break this down and make it simple and easy to sample from They say here Well, we basically know that our our reverse process can be parameterized like this This is what we saw in the previous paper. So we got we're going to learn how to predict mu and covariance matrix Now if we take log of this expression here since this is like e raised to blah blah blah something That's the gaussian definition you end up with this expression here. Okay, so we're working our way here so we are basically trying to understand how we can break this down and now we're going to see how we can break this down and that's going to be on the next page and then we're going to have a simple expression for for uh, like a Sampling from a class conditioned distribution. Okay So under certain assumptions which they mentioned here so we can assume that the log of this of our classifier has low curvature compared to the Basically inverse of this covariance matrix, which are which we're going to predict from the unconditional Diffusion model. So this assumption is reasonable in the limit of infinite diffusion steps blah blah blah where the Well, I guess this is for binius norm Converges to zero and in this case we can approximate this a log of the classifier using a Taylor expansion around mu and mu is basically the Mean of the x of t. So basically let me go back here So that's the same u as this one here. Okay, so that's that's the Basically the mean of the distribution from which we're going to sample x of t. That's it Fairly simple and if we do that they end up with this expression here This component here is going to be very very important and under this like Basically approximation and then derivation here. We can finally get the the the the final expression here So we just take the log of the distribution We just saw above and we can omit z because that is a constant So we don't care about it and you can see that after some derivation here We end up with this expression here So again, we can ignore a c4 is just a constant So we basically if we want to sample from that complex intractable distribution here What we in practice need to do is just sample from this gaussian here So as you can see the Gaussian just has a mean a shifted mean so we have mu which is from the unconditional Diffusion model and we just simply add This offset here. So we have the covariance matrix. We just basically do like we Multiply it with the g and g is just the as you can see here the gradient of our classifier For x of t equals mu. So i'm going to give you some intuition for behind these formulas But first let me just tell you this so so we have thus found that the conditional transition operator So that's the the one we saw above here Can be approximated by a gaussian similar to the unconditional transition operator But with its mean shifted by sigma g Okay, so let me try and Give you some intuition for why this works. It's going to be a very hand-wavy explanation But like I think the intuition is is completely correct on some level of abstraction. It's a correct So let's just say that we have a condition But I think the intuition is is completely correct on some level of abstraction. It's it's a correct Hand-wavy explanation. Okay. So g again it tells you how you should tweak X of t so the the image such that you can maximize certain class y. Okay, so Let's see what we do. So we we pass in we have our model and Let me kind of draw it like this. So let me just change the color here So we have an image That's x t plus one. So that's the x t plus one image. We pass that image into our diffusion model So that's our diffusion model here and out comes what outcomes mu? and out comes this Covariance matrix and these two together basically parameterize the Gaussian distribution from which we would sample the Next so x of t so from from this from this distribution here. We sample x of t Okay, so we can kind of visualize the Gaussian as just Like a point in a in a in a multi-dimensional space for the sake of explanation I'm just gonna assume we're we're dealing with 3d space But obviously this is going to be much higher dimensionality mu is actually the dimension the same dimensionality as our our image so basically, uh, it's gonna be um, well h Times w times three so that's the dimension of our image and similarly for for sigma But just for the sake of understanding what's going on with this shift here. So the the sigma g part I'm gonna kind of assume a coordinate system like this and Like assume we have a like a Gaussian distribution. The mean is here. So this is the mu. Okay and uh, we have some we basically have some some variance So under normal circumstances if we just have like an unconditional diffusion model We would basically sample a point some from from this distribution somewhere from from this distribution Like maybe maybe this would be our next uh, this would be our x of t. Okay, but Because we we additionally have this information coming from from the classifier What it does is the following so now you take this mu here Let me change the color Okay, so so you take the mu and you pass that as an image to like our classifier So we're gonna have something like this. So we have an image That's gonna be the same as this one here. So that's gonna be mu We pass that inside of our classifier This is our classifier And out comes uh, well for image net they train obviously it has like thousand classes And We basically learn how we should tweak this image So the classifier tells us how we should tweak the image such that a particular class is maximized maybe this one Okay, and with that information with that gradient, so so those blue dots. I'm just gonna denote that as g With that information which is of the same dimensionality as the mu we're going to shift this mu So we're going to shift this maybe here And we're going to end up here. So our new distribution Is going to be actually here instead of here. Okay, and now we have and we have the same variance So i'm going to kind of change the color because otherwise you will not be able to see anything So we shifted our original color And we're not going to be able to see anything so we shifted our original distribution and now we are somewhere here Okay, it's kind of messy, but you get the point so this Shift vector we just saw there. So that's this blue one is just this expression here. So Sigma big sigma times g And this is the like the hand way we explanation what's going on So the classifier knows where you should kind of move this image In which direction should you move this image such such that you maximize the class of interest and by just doing that by just kind of moving the The mean in the direction of the gradient of the classifier You basically increase the probability of the class label here and that's that's basically everything you need to understand. Okay Uh, hopefully that's clear enough. That's how we can combine again The classifier here and the unconditional diffusion model here to form images from a particular class Again, i'm kind of lying here because as I said, they are still class conditioning this model here But using the classifier we additionally complementary, uh improve, uh the uh, well this class conditioned, uh generation Cool, uh, that's it. Let me briefly, um Explain a couple more things. Um, aside from using the the the classifier The classifier gradient they additionally scale it and this is just empirically they found out that this gives them much better results So here they mentioned our classifier architecture is simply the down sampling trunk of the unit model With an attention pool at the 8 times 8 layer to produce the final output We train these classifiers on the same noising distribution as the corresponding diffusion model Then they say in initial experiments with unconditional image map models We found it necessary to scale the classifier gradients by constant factor larger than one So to understand the effect of the scaling classifier gradients note that if we scale by s So this is the what we denoted as g previously. This is these are the gradients of our classifier So if we do this you can show quite easily that that's equivalent to this expression here So well, basically, um, you can convince yourself This is true because log of a product is log of sums and because this is delta x We don't care about the this part here and then we would end up with the same expression Basically, these two are equivalent Okay, and you can see that we are basically just raising this to the power of s which which means we're going to sharpen this distribution So let me continue reading what they said here So so z is an arbitrary constant and as a result the conditioning process is still theoretically grounded in our renormalized classifier distribution proportional to P raised to the power of s when s is bigger than one. This is distribution becomes sharper than The original distribution. Okay in the above derivations. We assume that the underlying diffusion model was unconditional modeling P of x it is also possible to train conditional diffusion models And use classifier guidance in the exact same way table 4 shows that the sample quality of both unconditional and conditional models can be greatly improved by classifier guidance So this is the the thing I was mentioning we we can both have conditional model plus the classifier guidance because they are complementary We see that with a high enough scale the guided Unconditional model can get quite close to the fid of an unguided conditional model Although training directly with the class label still helps finally guiding a conditional model further improves fid So that's what I mentioned multiple times using the class conditional model plus using the guiding is what gave them the best Result on various metrics such as fid is inception score Precision recall, etc, etc. Okay Um quickly to let me just quickly address this because we're going to see it in the code later on So how they train the so why do they need to train the model? Uh, the the the classifier on on noisy images because they say here we train these classifiers on the same Noising distribution as the corresponding diffusion model Well Because we're going to be using it on images on x of t so x of t which can be super noisy And if we just train our classifier on normal images normal when I say normal I mean images without any noise Then the model will probably struggle because these images would be out of distribution for the model And then it would not give us correct class for a noisy version of the original image because of that we want to train model For all so basically on the whole process. So this is our diffusion Like for example, if we want to train the model Our diffusion, uh, like forward process something like this. So this is the step zero. This is the step Let's say big t minus one We want to train on all of these images So we'll be randomly sampling images Um, and uh making sure that the model knows how to predict the correct class And uh, because of that this is kind of bothersome and some of the later papers such as glide Introduce this concept of a classifier free guidance. I'm gonna, um, well tell you more about that one Uh in one of the future videos. Okay, let's go back here Uh here we can see that the trade-off we mentioned in the beginning of the video And that's that we want to trade out between diversity and quality of the sample They show here using various metrics that that's indeed the case So you can see here as we increase this gradient scale. So that's the hyper parameter s Uh, you can see that the recall is uh falling down which means that the diversity the the the mode coverage Of our generative model is falling down But the precision which basically um, well, uh, it's a proxy for the image quality Goes up and you can see similarly here is goes up which means that the image quality is increasing And f id is is also going up which means that the diversity is is uh, basically, uh, diminishing Okay, that's pretty much it. I'm gonna skip all of this and i'm just gonna briefly, uh Uh mention a couple sentences here and then we're gonna jump into the code Our proposed classifier guidance technique is currently limited to labeled datasets. So keep that in mind We do have to train it on on on labeled, uh images um Well, the effectiveness of classifier guidance demonstrates that we can obtain powerful generative models from the gradients of a classification function This could be used to condition pre-trained models in a plethora of ways for example by conditioning an image generator with a text caption Using a noisy version of clip and because of this sentence After reading this paper You could have assumed that open ai would start working on something like glide or dali models and they did indeed And finally it also suggests that large unlabeled datasets could be leveraged in the future to pre-train powerful diffusion models that can later be improved By using a classifier with desirable properties. Okay again You could have if you read this paper You could have anticipated open ai is going to try and just scale up scale up diffusion models Uh such as what they've done with with dali and when I say dali I basically mean dali Two because dali one didn't even use diffusion models. Okay, let's jump to the code now um, I I did expect this Uh paper overview part to be a bit shorter But like do let me know whether you find this type of combination useful or not Just feel free to comment down below. What do you think about this format? In any case, let's now Start working on on the classifier train script. So this is how we trained This is how we train this this noisy classifier Uh that I just mentioned so let me start and walk you through the actual script for training it So i'm gonna obviously again, uh abstract a lot of the details because otherwise this would not be tractable uh for for a video That has an ambition to be less than an hour long. Uh, so just some Setting up some distributed stuff, which we don't care about logging blah blah blah Um this part part we do care about because I just want to briefly show you basically how the classifier is constructed We did sign the paper that it's simply the first part of the unit model up to the bottleneck part and then we just stitch Attention pooling and we ditch the the second part of the unit model. That's how we that's how they set up the uh, the classifier The architecture so let's kind of go inside here. So this just kind of gives you some Um default arguments i'm gonna try and zoom in a little bit in case you don't see this I'm gonna close this part. Okay, so let's step inside this Line of code and let's see how the classifier works Okay, so here's the classifier Image size 128 128 blah blah. Those are just some uh hyper parameters obviously can be different Uh, we are not using fp16 Uh, there is some width depth Various hyper parameters, uh, we saw this is going to be um, well basically this is going to enable Uh the attention to be at resolutions 32 times 32 in the latent space as well as 16 by 16 and 8 by 8 Um, this is the scale shift norm that I showed you in the previous video uh, this basically, um enables us to Integrate to incorporate the the class as well as the time step information into the model um In any case i'm going to scheme over those details Um, let me just kind of go through here because we have image size 128 We have some specification for the number of channels in the unit And finally, we just uh transform, uh this from the resolution. So so remember this is like, um, Basically 6 32 16 and 8 after we step through this, uh for loop We're basically and end up with the same information But just like and now it it tells us how many down sampling steps you need to take before you apply the attention Uh, we saw that in the previous video as well. So i'm gonna just kind of skip over this Blah blah blah and here here it comes. So here's the encoder unit model Basically, let me see what interesting is here. So we have the output number of channels is hard coded to 2000 because this is trained on on on image net Uh, i'm gonna skip over all of those not that important Uh classifier pool is going to be attention. We're gonna see that in a second. So let me Go to the construction constructor here So i'm gonna again just scheme over they're just kind of setting all of these arguments to um local variables here Uh, nothing nothing fancy there Uh, we construct the uh time embedding. This is the the mlp that we use to transform The sinusoidal embeddings for our time step time steps um Nothing, nothing interesting really we can just kind of scheme over all of this. So Um, if we recall the from the previous video we had the same structure So we initially just have number of input block. We just have the conv, uh, basically 2d Uh, so the convolution as the first processing layer then we have this uh part where we uh add various Red residual residual blocks and attention blocks and then we have the middle block And the main difference is we don't have here the output blocks. We just have as you can see here We just have the pooling operator. So let me go To there obviously these details are not that important. The only important part lies in the residual block where you can see how the um How the information such as time steps and class information is incorporated. I'm going to show you that in a second I'm going to kind of skip over all of this again. You can see here that uh Sometimes we're going to add attention blocks and that's going to happen precisely for the resolutions 32 16 and 8 And there are these wrapper functions which just enable us to uh, sometimes pass the temporal information Sometimes just pass the the feature vectors. Okay, in any case i'm going to skip all of this because i'm just um There's too much information there and it's not that vital for your understanding It's just like the architectural details of the unit model. I'm just going to show you this one So here is the let me just kind of uh remind you from the previous video uh There is this part where we embed so here we embed the uh temporal information and then because this is set to true Use scale shift norm what we're gonna do is we're gonna take the temporal information here as in a form of a scale and shift And we are going to combine that this is how we merge the temporal information Uh into into the uh features here and this is just as a reminder For the what happened in the previous video. Let me just see whether they have class information here Um doesn't seem to be the case so embedding They passed embedding here No, I don't think this is going to have well, it doesn't make any sense to have uh like a class Um conditioning in a classifier. So this is going to make sense for the diffusion model and in any case Okay, let's continue here so we Uh, we have the the unit part and now we just end up uh with this um Sequential model that has some normalization some activation function and finally this attention pooling 2d 2d this is basically just going to take the output features from the unit model And using a tension mechanism is going to form a resultant feature vector which we're going to then use to predict the the logits and you can see that the output channels here is Thousands is going to give us a thousand Thousand dimensional vector which is going to be just the logits that come out of this classifier and that's it guys Um, I kind of schemed a lot of details because they are not really that important. So Uh, you just need to have like a rough mental model how the architecture looks like it has this this basically encoder type of a structure it has this attention pooling at the end and That's pretty much it Okay So I skipped the the gaussian diffusion model here because we saw that in the previous video I just want to focus on the differences compared to the previous video Uh, so here we just uh shift the model onto the gpu Uh, we create the uniform sampler here. So this one is just going to help us sample from the forward Process we're just gonna pick the random uh time step using the uniform distribution Obviously you can kind of bias this and and form various different distributions One of those which I showed you in the previous video was depending on the loss You're gonna focus more on those time steps where we have higher loss, which makes sense intuitively Um, okay, so we can skip over all of these parts. We don't want to resume from the checkpoint We just have some mixed precision trainer wrapper here. We don't care about that part. We have the distributed Data parallel object that's for distributed training, but we don't care about it either I'm gonna skip over all of that. We have some data loading Basically, we're gonna have batch size 256 image size 120 128 Uh, we have the data directory which I downloaded before the video started and that's it We just create the training and validation datasets I'm not gonna dig into the actual details of how that works because it's just like a side thing Um, so logging, addmw optimizer. We're not resuming from a checkpoint blah blah blah And here this this is going to be the main function to understand how to classify a string So we have forward backward log. I've added a breakpoint to that function. We're gonna step into it a bit later So here we defined it And now here we have we step for a number of iterations. Okay, so we're gonna do this for a number of steps So as specified by the number of iterations there I've just set it to 30 because I want to have a short training here just for for Debugging purposes in any case just some logging nothing interesting I can skip all of this and this is where the whole magic of the classifier training happens after that. We just do the Optimization step and we do the same logic for on the validation. We just do the forward backward on the validation Set and finally we just do some logging here So basically this is where the whole magic is in so the forward backward log function is everything that we care about So let's step into it So we first gonna grab like a batch of images and associated labels. Let me show you the shapes As always that's a very important thing to do So let's see what batch looks like so if I do batch shape So we can see here Bad shape so we have 200 256 images RGB images of resolution 128 128 We have the labels as well. So after I do this, so basically this extra here Just contains this y And let's see the shape So shape is 256. Obviously we have for each of the images we have associated label If I were to step over this And print labels We can see there is a bunch of labels here because i'm using cipher 10 even though this is This is image net training That's why we only see labels from zero through nine, but we don't care about this detail for the Purpose of understanding how the classified training looks like. Okay, so we we put the as you can see here We put the labels onto our target device, which is gpu in my case same with images here And now what we do is we do we take the uniform sampler and we just sample 256 time steps. That's it. So we now have let's see if I were to print t you can see Uh, like it's a it's a pile of numbers Of the following length. Let me just print this so we have 256 time steps for each of the images in the batch and it's just randomly sampled between uh zero and And and 999 after that we just do the q sample So we take the image we take the time step and we just do the noising process So we create from the x0 image we create we create the x t image. I'm gonna quickly remind you I showed you this in the previous video. So i'm going to just quickly show you how this thing looks like So here it is. Uh, there is literally a formula which I showed you in the previous paper, which is fairly easy you just take uh these Uh specific constants here you multiply it with the x start x start is the x0 the original image And we just add noise According to this square blah blah. Again, there's some content constant. Let me try to find a formula. It's going to be easier Okay, guys, here is the formula from the previous video. So here is how we sample So given x0 the original image, here's how we get x of t We can just basically sample from this distribution here or equivalently. We just compute this expression here So you can see here we multiply by some constant constant We multiply the x0 or x start as we saw in the code We just add up on on top of that. We add this constant times epsilon, which is just your noise from your normal Distribution now, let's go back to the code. Let me see whether you can you can see resemblance between this formula here And between this thing here. So again constant times x start We add on top of that some constant times noise and that's how we get xt and that's it That's because that's why i'm going to just skip over this function and we have finally we now have a batch of Noisy xt samples and that's what we're going to use to train our classifier Uh on and again the reason we are doing this is because otherwise if we didn't have these noisy images if we just trained on the original images then the classifier wouldn't be able to To estimate the gradients correctly for for an arbitrary noisy image and that's why we want to train it on noisy images Okay So here now we just um instead of training on the whole batch This is just for for for memory optimization stuff. Uh, we're gonna kind of chunk the batch size That's 256 into smaller batches so-called micro batches Okay, and here it is and now we just call we just pass so we do a for pass through our Our classifier again, the the classifier is basically just a unit model Uh, well basically the first portion of the unit model Uh, and first of all, let me just show you the shapes here. I think it's gonna be four No, it's gonna be actually just one. Okay, so the micro batch size is one So we basically pass just a single image through the classifier as an optimization Thingy and we pass in the corresponding Like basically time step. So again, if this is like five That means this is going to be x5 and we pass that through the model and we get back to logits. Okay, let me uh step through there and here we end up in the forward pass of our model so that's the that's the That's the classifier and we do the time step embedding which is just going to embed it Like using the the sinusoidal embeddings and then we just pass it through the mlp. And so we ultimately end up with Uh something that's going to be of shape one 512. Yeah, okay So and then we just do forward pass through the input blocks through the middle blocks and finally we pass through the attention Uh pooling layer and that's it depending on the module some of them are going to encode the temporal Uh, the time step information into our features some of the other modules will not do that such as the conf2d layer But in any case, that's pretty much it so I can now get back to the classifier training Uh, we can skip all the way to the loss here. So let's Skip skip over and here we are. So we got some logits and now basically we just do a cross entropy For the label we have and that's it. That's our loss. We just do the cross entropy loss Which is basically just a standard classifier training loss Okay, and now we just collect some information such as the accuracy at one accuracy at five We do some logging blah blah blah. We do the mean on top of the loss And basically here because we have micro batches What they do is in the in the zeroth micro batch We just clear up the gradients and then we keep on accumulating the gradients for each of the micro batches And we don't do the update until we do we go through all of the micro batches inside of uh, Instead of inside of a single mini batch and that's just again, uh minor optimization detail But we saw everything we needed to know basically that the main thing is that we have this Q-samp we call this function here, which makes us train not on the original images, but instead we train on x Uh t images so noisy versions of the images. Um Okay, guys, that's it. Uh just for um being detailed here, I want to show you the part with merging the um The temporal information I want to show you that part here So res block i'm going to enter there and i'm going to set i'm going to set a breakpoint inside of the forward function here Uh, just so that I can show you how we merge the information from the embedding here Okay, so i'm going to set a breakpoint here And let's now step let's do another pass through another micro batch and here we are So we have the embedding and we are gonna hit The res block at one point of time and here we are. So here is the um part where we merge the temporal information I want to show you this once more Again, it's fairly simple. We just do some additional processing. So let's see what these In m layers are so again, it's just like a like a basically a linear layer and an activation function Nothing fancy there And we end up with embedding Here of the shape 1 to 56 and now we just as you can see here We start adding dummy dimensions such that we can merge this temporal information into the Um image features, okay, so we can see here because this is set to true we enter this branch here. We have some layers Here and we split them into two parts. The first part is the normalization layer and then the rest And you can see this out there. I think it's just like a sequential. Yeah, you can see here how it's constructed Um, not that important. So the important part is here. We take our embedding out So this vector and we split it into two parts one is scale one is shift And I guess it's going to be 128 is going to be the shape here So one 128 and we added the dummy dimensions And so now if we do this we're going to have some broadcasting going on And we are going to going to merge them with the image features and this image features are going to be of shape As you can see here 128 128 128 and that's it guys I'm going to briefly show you how this looks like inside of my one note So again fairly simple thing. We have so we have something that came out of the So we have a volume that came out of the unit model. That's going to look like something like this So here is the our volume So those are the image features From the latent space of our encoder And we saw that that's going to be like this is 128 Number of channels and we saw that this is also 128 so the height and the width all of this is 128 Whereas on the other hand we just have this is our vector. This is our temporal embedding So this is how it looks like it's only going to be 128 here But it's only one one here and then what broadcasting is going to do Is basically just copy paste this multiple times so that we end up with something like this So that we end up with multiple copies here here here Etc so we're just going to copy paste it here and then after we do that we can freely add it up So we can freely do addition on top of the image features and that's how we incorporate the temporal information Cool. Let's go back here I'm going to stop the execution of the classifier train script. We saw how to train the classifier Do let me know whether you found this explanation useful whether I can improve something any feedback is always appreciated And also if you if you like this video share it out with your friends Let's now focus on the sampling function. So I have prepared Some arguments again. I just took the arguments from the read me and I created I just kind of extracted them here So here are the arguments i'm using but you can find all of that You can find all of that basically in the read me file itself So i'm going to go back here and i'm going to now focus on the sampling function. Okay guys, let's run this script and Again, i'm going to scheme a lot of the i'm going to skip explaining a lot of details Which are irrelevant to understanding the actual Well, uh got in this particular case We want to see how the classifier guidance works like and that's what i'm going to focus on So what this function is going to return is the model which is going to be the unit model That's going to predict the epsilon. So the the noise And diffusion just contains bunch of those constants necessary to do the diffusion process So i'm going to skip over all of that We're just going to load a certain checkpoint We like as I said, I have a model path here So, let me just see whether I can show you yeah, so I went ahead and downloaded the model Into this models directory inside of my root directory and Everything is already explained in the read me so you don't have to worry about it So we just push the model to the gpu We convert it into fp16 Again, just some Optimization details. I'm not going to focus on that We create the classifier. So that's again our encoder The the the half portion of the unit You can see here and we already Schemed through that code previously in the in the training script So i'm going to just ignore all of that and uh, whoops Well, obviously not so i'll have to just ask Let me just kind of do the Disable all the points all the breakpoints and then i'm going to click f5 and here we are Now i'm going to enable all of the breakpoints again And we just load the weights We load the state dictionary from the classifier path So I went ahead and downloaded this this model again And I put it inside of the models directory. That's it Nothing smart there We push the model the classifier on the to the gpu as well And we set it inside into the eval mode That's it. Now this is going to be the most important part This is where the gradients are calculated for the classifier I'm going to analyze that a bit later once we hit once we enter that function So here is where it's defined later. We're going to see how it basically works So that's it. We just define this function as well The model function and Now we're going to do a bunch of iterations until we create the number of samples that we can specify And you can see here I specified 100 but that's just arbitrary numbers so not that important Okay, so we create random numbers so random classes and depending on the batch size We have like four random classes being generated here So let me show you what those are If I go to debug console it's going to be a bunch of random classes If I go to debug console if I print that we can see we have four random classes And we don't I don't know directly what these correspond to But like let's say these are just classes 33 918 etc Okay, we don't care about the actual semantics behind the numbers We are going to store those the class information inside of this model Like keyword arguments dictionary So I'm gonna go there and because we're not using this ddim again ddim is just just it's just like This optimize it like this sampling method whereby if you have less than 50 diffusion steps They showed in that paper that they achieve better results compared to the original ddpm paper So only in the mode where you only sample for less than 50 steps roughly then ddim makes sense But here I think we have like I think we set like thousand steps so we're going to use the p sample loop instead of the ddim sample loop I'm not going to focus on the ddim part because there is a lot of formulas It's hard to explain the logic but like on the high level just treating it as a black box When you would want to use it is when you have less than 50 diffusion steps Okay, that's it Here is where the whole magic happens After that there's just some normalization going on permutation blah blah blah And we just collect the images here and finally yeah basically just a bunch of boilerplate code This is where the main logic is in so we're going to step inside of here We basically specify here the target shape so our images are going to be 64 by 64 Later on they have the up sampling modules which we're also going to ignore But let's focus on this part here Okay so we pass in the model keyword arguments that have the class information We pass these special functions so the cond fn which we define here is going to calculate the gradients we're going to see how that comes into the play in a second So here is the p sample loop what it does is we're just going to keep on generating samples I'm just going to directly jump inside of this function Because everything else is just like basically boilerplate code So here we generate the random noise image Okay so we have the desired shape and the desired shape is as you can see here Four because we have four images in the batch and here we have the resolution of our target image And we just generate we just sample from a normal distribution here And then we just generate indices So depending on the number of time steps so we have 250 here We just generate these indices and remember that's because we want to start from the Completely noisy image so from the index 249 and we're going to work our way all the way to zero Where we're going to end up with completely denoised images from our underlying data Distribution which we learned during the training but this is now the sampling part Okay so indices let's see what t is it should be 249 as I said And here is where the whole magic happens in this p sample step So after 250 steps we are going to end up with our final image And that's one of the drawbacks of current diffusion models We need much more compute compared to like say GANs where you just need a single four pass And you end up with a like a very realistic image very crispy image Here we have much more we need much more compute okay So let's step inside of the p sample function and the main function here is going to be this p mean Variance that's just basically our reverse process that we learned After we calculate the out variable here we're going to generate noise again We're going to do the conditioning I'm going to get here so this is going to be important part here This is where we do the conditioning this is where we do the where we shift the mean Via the sigma times gradients expression and then we just form the image here okay So the this is the this is going to contain the x t minus one If the if we condition on the x t or alternatively if we condition off on x t plus one This is going to be x t in any case it's just like a single step closer to the final image Okay let's jump into the p mean variance here it is this is what we calculate as I said our Reverse process and let's kind of quickly scheme over over this part so I'm going to Set a break point here here we are we enter here and now we're going to pass what we're going to Pass a couple things first we're going to pass the the the noise we just sampled so that's going to Be I guess uh four images right so four three sixty four sixty four we're going to pass the time steps So those are going to be the two forty nine two forty nine two forty nine for each image in the batch And we're going to additionally pass the class information because if you recall From the paper explanation we are not only going to use classifier to guide We are also going to use class information to condition the model so those two do the same Thing but are kind of complementary so don't be confused by that okay let's step there we hit the Basically the this model fn that we saw previously and because class conditioning is set to true We're going to pass x so the noise images the time steps and the conditioning information all of that Let's now quickly step into the unit model into the forward pass and see how the class information Is incorporated into uh with the image features and hint hint it's literally the same thing as with The temporal information pretty much we're going to see that in a second so here it is we have the Time steps we're going to embed those so sinusoidal embeddings then some learned stuff here and we end Up with a vector that's of shape i guess let's see what it is um it's going to be four seven sixty Eight because we're dealing with uh batches with four images okay and now because we have the the Classes set here the only thing we're going to do with our classes so here here are classes so we Have four classes here we're going to um embed them using this label embed oh i hate these warnings I should have toggled them off there is some config uh line that you can set to lon jason if you if you Don't want to see those but yeah i forgot to do that uh so let's see what this this label embed Does okay so you can see here is just a simple embedding layer and it's going to specify uh it's Going to have some number of dimensions for each of the classes nothing fancy there you can literally Treat that as um let's see what it gives us basically uh so if we do this and i do shape Yeah you can see it's going to transform uh our four scalars into four vectors of dimensionality 768 and then we're just going to simply add that on top of the temporal embedding and then everything Else proceeds as as usual we're just going to keep on incorporating that information so both the class Information as well as the temporal information directly merge them with the image features Using the logic i showed you with the scaling and the offset we saw in the res block and that's it I'm going to skip over all of this uh we don't care about it and now let's continue on so we are Here and we ended up with the model outputs if sigma is learned this should be four six Uh something right so four six because six because we all we predict the epsilon so that's the noise And we also predict the sigma so that's the the covariance um and now let's continue here We saw all of this logic in the previous video so i'm going to kind of skim through this We split the model outputs into two parts so again the epsilon and the variance and then Well actually it's not the variance it's it's the v vector let me show you in a second formula So again if you recall from the previous video this is the formula we used so we are predicting the v's here And then we're going to form the variance by doing this expression here so this is the expression calculation we just saw So we calculate all of those we get the variance ultimately here and now we need to predict the mean So here is how we predict the mean i'm going to skim through all of this here We just need to first predict the x start so that's the x0 And then we use this posterior mean variance to get the model mean so this is all of this is same as in the previous video pretty much So we got the mean we have the variance and we finally return all of those variables so the mean Uh variance model log variance etc etc by the way let me just quickly show you what those are So here are we in the one note explanation I showed you in the beginning of the video So this is what we currently generated we generated the Um these two so we have the mean and we have the sigma and now we need to do this shift part So we need to do the shift part and then we can sample our our our image and that's it Fairly simple. Okay. Let me go back here. I'm going to return all of these variables I'm going to return all of those We generate we sample from a normal distribution We have this masking stuff going on Uh that um because we have if you recall we have a discrepancy for t equals zero time step versus all of the other steps Uh things look a bit different, but like I guess that's just a minor detail again. This is where the whole magic happens This is where the shifting of the mean happens using the sigma using the gradients of our classifier Uh, and also like just a note, uh, there is a bug here I submitted an issue and the authors of open from opening I basically confirmed that's indeed Uh an issue i'm gonna um explain you a bit later Uh what that exactly is but the funny thing is it did not even matter It's like that's the thing with uh training machine learning models Sometimes you have a bug but the models are forgiving and you will just get Something working maybe a bit suboptimal, but still there is no way you can figure out there is a bug It's not like with software systems where something when you when you when you like a basically develop a traditional software system If something does not work You will know that something will crash and the program will not work. So yeah ml ml is a bit different. Okay, so Here we form a new mean and a new mean is formed by adding On top of our old mean we just add the sigma times the gradient. So again, that's that's that's the whole idea So here again, we just have uh, we pass the um, we have the conditional con function which we defined in the main script we pass our Uh means the mean and the and the covariance we just calculated uh here above So this basically contains the x t and this is the x t plus one and this is the time step in any case Let's step inside of this function and let's see what's going on Okay So first we calculate the gradient and then as we recall from the equation we just add on our new on our old mean we add the The variance we calculated times the gradient and that's it. So that's that's easy. Let's now see that this part here Let's step into this function here. Okay, so First thing we do is we take our noisy image Uh, we detach it from the computational graph We said it requires graph to true so that we can calculate the gradients for that x because that's what we ultimately care about right, we want to have the gradients of x with respect to the log Probability of of our target class from the classifier. Okay, so We pass it through the classifier. We get the logits And unfortunately, I stepped inside of the classifier. We don't really care about it. Let's just kind of um Continue here. We don't care about it. So we have logits now we just form the log softmax So we have the log props So first, let's see what what the dimensionality of log props is So if we do shape here we have four because if we have four images and we have thousand lodges because we're dealing with image And and so what we do here is we basically extract the target. So why is again remember? Why is the the target? So those are the ground truth classes. We use those to take only those ledges logits. So we have the selected ones so now we if I print this these are the The um, uh target classes. So so these are the logits we need to maximize if we want to have An image from that particular class if that makes sense And so what we do here was just and this is a part that kind of confused me We sum up all of these uh different, um, like, um Logits, sorry not logits the log props the log probabilities And then we find the gradient with respect to the input. So the input is again a batch So it's going to be four three sixty four sixty four and we just multiply so this is the classifier scale I mentioned So that's denoted as s in the paper. So let me show you that in a second here Let's go back to the paper and let's find the uh, that part Somewhere here. So I think it's down here. Okay, so here's here is the part We're just computing we have the s Which is on the right hand side in the code and we have the gradients here. So again, let's go back to the Uh code here. So this is the s and this is the gradient we are we are computing and the reason this works is because um If you do a sum you'll end up with so our loss here is going to be L1 so that's like l1 is just my uh, shorthand notation for log probability for the first image So for the for the log probability for the first image plus l2 plus l3 plus l4 And if we do a gradient with respect to x because x1 so the first image from this batch only Influences this particular loss. Okay x2. So that's the second image from from the batch only influences the second one Okay, so because they only influence the log probes that correspond to to their index and because of that if we do gradient here Uh, each of the images here will have only the so basically the first image will have um l1 dx The second image will be uh l2 dx etc. Etc. Okay So basically we'll return a batch of gradients where each of the uh images inside of that batch Is basically gradients with respect to the corresponding loss. So if I go back here We basically end up with a gradient here and the gradient is going to be of the same shape as the images So here 436464 and that's pretty much it Okay, so now we just do the the the logic we saw in the paper and guys that's pretty much it We're done. We now form Uh using the new mean and the log variance We just sample by doing this expression by just adding the log variance times the noise And that's how we get the uh next image in the reverse process Now this just repeats on and on but this was the main this was the gist of the paper So where is the bug I mentioned? I'm going to show you the issue in a second But like basically what happens here is they passed xt plus one and t plus one So these are the uh, they should have passed the out mean Instead of xt plus one because this is xt and this is xt plus one and so It might be a bit subtle. So let's let's let's try and see whether I can explain it So as you can see here they pass the the the the the xt plus one. So if I go back here Uh, you can see that then they call this conditional function with xt plus one and t plus one So we end up using xt plus one t plus one here Uh, so that means That we calculate we pass xt plus one t plus one here. We get the logits, but if you recall from the actual paper Uh, we do not need to we we need to condition on on x. Let me just find the the expression here blah blah blah so You can see here That we want to condition on xt and not on xt plus one. Otherwise the gradients we get here Are again from the xt plus one not from the xt and all of that kind of messes up the logic I just explained it here. So mu Mu is from xt and we want to use that same mu and pass it inside of the classifier and not the previous image And despite all of this, it's very subtle. I don't even know how I noticed it. I was fairly I guess pedantic stepping through this code Uh They they get the same results pretty much. They didn't even notice it They noticed it post-hoc after the paper was published or something. I'm going to show you the issue in a second So here is the issue I opened up I think a couple of days ago um So I think it's closed already so bugs blah blah blah. So here's my issue So I created an issue here as you can see here I think I found two bugs So the first one we shouldn't shouldn't we pass the out of mean so the x of t instead of x of t plus one here Uh, and similarly the the smaller time step instead of t Uh, and then I linked to the particular line of code which is the same line of code I showed to you guys So that's here And uh, basically, um, one of the authors replied yes, this is indeed a slight bug we which we noticed shortly after releasing our work However, we did try ablating using the correct formula and found that it did not noticeably change results again That's the tricky part about training machine learning models And then the second part I was kind of confused with the summation of of of those losses But it turned out it's quite. Um, it makes a lot of sense. Uh, it was just a minor Uh, mistake on my on my side anyway in any case I can kind of link this issue if you want to dig deeper into it Uh, I'll have it down in the video description So guys, that's pretty much it. Um, I showed you how the how the paper what the paper introduced the two main things are Improving the unit architecture. The second thing is uh, adding this classifier guidance technique And then I showed you how to train the classifier Uh in in this classifier transcript we saw how to train the classifier This classifier transcript we saw how to train it on on basically noisy images and then use that very same classifier Inside of this classifier sample script to basically shift the mean and thus Like sample images from the class conditioned diffusion model There is a lot of details Hopefully you picked up something useful from this video if you did Uh, consider subscribing also share the video out and until next time. Bye. Bye
[{"start": 0.0, "end": 4.44, "text": " What's up guys in this video I'm continuing with the machine learning coding series and"}, {"start": 4.76, "end": 11.06, "text": " I'll be covering a follow-up paper to the previous video. So in the last one we covered the improved"}, {"start": 11.8, "end": 16.36, "text": " DDPMs or denoising diffusion probabilistic models paper in this one"}, {"start": 16.36, "end": 22.02, "text": " I'm gonna cover the follow-up work called the diffusion models speed cans on image synthesis"}, {"start": 22.72, "end": 26.12, "text": " Paper and the associated code. So the idea will be to"}, {"start": 26.12, "end": 33.52, "text": " To again for me to give you a brief context like 10-15 minutes short overview of the most salient parts of the paper and"}, {"start": 34.4, "end": 38.0, "text": " After that we're gonna dig into the code and analyze two scripts"}, {"start": 38.0, "end": 43.6, "text": " So one is this classifier train and the other one is classifier sample. So basically I want to focus on"}, {"start": 44.120000000000005, "end": 46.56, "text": " The guiding so how do we guide?"}, {"start": 47.28, "end": 50.8, "text": " diffusion using pre trained noise classifiers"}, {"start": 51.480000000000004, "end": 54.64, "text": " So understand what that means in 10-15 minutes"}, {"start": 54.64, "end": 61.76, "text": " In any case what I what I've done is I cloned this repo so the guided diffusion from open AI"}, {"start": 62.64, "end": 68.68, "text": " They have like everything all the instructions you'll need I had to do some modifications again"}, {"start": 68.68, "end": 75.24000000000001, "text": " If I get enough enough votes down in the in the comments, I'll just push my my changes"}, {"start": 75.24000000000001, "end": 81.04, "text": " So I had to make minor changes to make this to work on my machine in any case you can you can basically download"}, {"start": 81.04, "end": 84.4, "text": " various checkpoints here and just follow the"}, {"start": 85.12, "end": 91.68, "text": " Instructions they they have in the readme which are super nice and yeah, you should be ready to go without further"}, {"start": 91.68, "end": 98.28, "text": " Do let's first start with the paper. Okay, so as I said, it's diffusion models beat cans on image synthesis and"}, {"start": 98.80000000000001, "end": 103.80000000000001, "text": " Two main innovations here are they improve the unit architecture from their previous work"}, {"start": 103.80000000000001, "end": 109.04, "text": " So from the improved DDP M's paper and the second improvement is they find a way how to"}, {"start": 109.04, "end": 116.04, "text": " condition an additionally condition the model so that they can achieve higher quality so they can trade off the"}, {"start": 116.56, "end": 123.28, "text": " Diversity versus the quality of a sample which is something that previous generative models such as GANs used to achieve"}, {"start": 123.76, "end": 125.76, "text": " using this truncation"}, {"start": 126.24000000000001, "end": 131.92000000000002, "text": " Truncation trick as well as some autoregressive models can simply modify the temperature and thus"}, {"start": 132.4, "end": 136.72, "text": " Control the like diversity versus the quality of the of the sample"}, {"start": 136.72, "end": 143.04, "text": " So we'll see in this paper how they achieve the same thing for for the class of generative models called diffusion models"}, {"start": 143.84, "end": 151.44, "text": " so I strongly suggest you you watch my previous video if you haven't because I'm gonna kind of assume some knowledge from that one"}, {"start": 152.07999999999998, "end": 156.4, "text": " But yeah, you can you can continue watching this one as well. Okay, so"}, {"start": 157.04, "end": 162.96, "text": " So they say here we hypothesize that the gap between diffusion models and GANs stems from at least two factors"}, {"start": 162.96, "end": 170.16, "text": " So first one that the model architecture is used by recent GAN literature have been heavily explored and refined"}, {"start": 170.8, "end": 175.60000000000002, "text": " And we saw the same thing happening with in the Comnex paper if you recall that one"}, {"start": 176.24, "end": 182.8, "text": " Where like basically they people showed like people focused a lot on transformers and there was a lot of innovation that happened"}, {"start": 183.28, "end": 186.4, "text": " on that side of the spectrum whereas the CNNs were"}, {"start": 186.4, "end": 192.8, "text": " not as as explored and then what Comnex showed that if you take some of the innovations that were"}, {"start": 193.36, "end": 198.96, "text": " Well invented in the context of transformers and you adopt that into CNNs you get much better models"}, {"start": 198.96, "end": 203.36, "text": " And so similarly here they took some ideas from other GAN papers"}, {"start": 203.36, "end": 208.8, "text": " And they took those ideas and then integrated them into the fusion models into the architecture and that"}, {"start": 209.36, "end": 211.92000000000002, "text": " Improved the results quite quite nicely"}, {"start": 211.92, "end": 218.0, "text": " Anyways, so that's the first thing the second thing is that GANs are able to trade off diversity for fidelity"}, {"start": 218.48, "end": 224.48, "text": " Producing high quality samples, but not covering the whole distribution. So that's the second thing I mentioned previously"}, {"start": 224.48, "end": 228.79999999999998, "text": " So they do that using this truncation trick. You can check out the the style GAN paper"}, {"start": 228.79999999999998, "end": 235.83999999999997, "text": " I think that's that's the one where I recall that they have that they're using the the truncation trick in any case"}, {"start": 237.04, "end": 239.04, "text": " Let me quickly show you the"}, {"start": 239.04, "end": 242.07999999999998, "text": " Dimensions across which they they kind of explored"}, {"start": 243.12, "end": 248.95999999999998, "text": " And how to improve the architecture so one is increasing depth versus width holding model size relatively constant"}, {"start": 249.51999999999998, "end": 255.92, "text": " Increasing the number of attention heads using attention at 32 times 32 resolution as well as 16"}, {"start": 256.64, "end": 259.12, "text": " By 16 and not and 8 by 8"}, {"start": 259.68, "end": 265.28, "text": " Rather than only at 16 by 16 using big GAN residual block for up sampling and down sampling the activations"}, {"start": 265.28, "end": 272.23999999999995, "text": " And rescaling residual connections with one over square root of two. Okay. So those are some of the ideas they they kind of played with"}, {"start": 273.2, "end": 276.32, "text": " And you can see here some of the curves here after you"}, {"start": 277.03999999999996, "end": 280.71999999999997, "text": " Take the best of out of those changes after a lot of experimentation"}, {"start": 280.96, "end": 285.28, "text": " I guess they end up having the the the best architecture here"}, {"start": 285.28, "end": 289.2, "text": " So you can see here number of channels 160 number of blocks two blah blah blah"}, {"start": 289.76, "end": 291.35999999999996, "text": " Anyways, not that important. I'm going to focus on the"}, {"start": 291.36, "end": 295.52000000000004, "text": " um like guiding the diffusion much more than on the architecture"}, {"start": 296.16, "end": 300.96000000000004, "text": " So here again just to to wrap this up the architecture part in the rest of the paper"}, {"start": 300.96000000000004, "end": 303.68, "text": " We use this final improved model architecture as our default"}, {"start": 304.08000000000004, "end": 311.28000000000003, "text": " Variable width with two residual blocks per resolution multiple heads with 64 channels per head attention at 32 16 and 8 resolutions"}, {"start": 311.68, "end": 316.8, "text": " Big GAN residual blocks for up and down sampling and adaptive group normalization for injecting time step and"}, {"start": 316.8, "end": 322.16, "text": " Class embeddings into residual blocks you might recall this one from my previous video"}, {"start": 322.16, "end": 325.04, "text": " So this adaptive group normalization for injecting"}, {"start": 325.6, "end": 328.72, "text": " So this one adaptive for injecting time step and class embeddings"}, {"start": 329.04, "end": 333.2, "text": " So do check it out if you want to know the the details how that is how that works"}, {"start": 334.08000000000004, "end": 336.08000000000004, "text": " Okay, so this is the important part"}, {"start": 336.48, "end": 340.72, "text": " So classifier guidance, that's what we want to understand a bit better"}, {"start": 341.6, "end": 343.6, "text": " They meant to be a little bit more complex"}, {"start": 343.6, "end": 350.8, "text": " That's what we want to understand a bit better. They mentioned here. We already incorporate class information into normalization layers"}, {"start": 351.12, "end": 356.08000000000004, "text": " So if you recall from the previous video basically the same way we we"}, {"start": 356.94, "end": 358.72, "text": " incorporate the"}, {"start": 358.72, "end": 360.72, "text": " the time step information"}, {"start": 360.72, "end": 362.72, "text": " Through that same pathway"}, {"start": 362.72, "end": 368.32000000000005, "text": " They incorporate the class information. So that means they already had class condition diffusion models"}, {"start": 368.32, "end": 376.32, "text": " But the thing is they show in this paper that using this classifier guidance is complementary to the class conditioning"}, {"start": 376.71999999999997, "end": 379.44, "text": " And because of that they achieve the best results"}, {"start": 380.56, "end": 384.32, "text": " With with class conditioning obviously so with classifier conditioning"}, {"start": 385.28, "end": 387.28, "text": " So let's let's see what it is"}, {"start": 387.36, "end": 388.71999999999997, "text": " basically"}, {"start": 388.71999999999997, "end": 393.76, "text": " So here we explore a different approach compared to the one I just mentioned so exploiting a classifier p"}, {"start": 393.76, "end": 399.76, "text": " Uh y uh conditioned on x so basically you input x and you get like a distribution across labels"}, {"start": 400.24, "end": 403.03999999999996, "text": " That's how you're going to implement this abstract"}, {"start": 404.4, "end": 406.0, "text": " distribution here"}, {"start": 406.0, "end": 408.0, "text": " To improve a diffusion generator"}, {"start": 408.08, "end": 415.36, "text": " So blah blah blah the previous paper show one way to achieve this wherein a pre-trained diffusion model can be conditioned using the"}, {"start": 415.42, "end": 420.9, "text": " Gradients of a classifier. Okay, that's the important part. So so in particular we can train a classifier"}, {"start": 420.9, "end": 423.7, "text": " p parameterized by parameters phi"}, {"start": 424.73999999999995, "end": 425.85999999999996, "text": " and"}, {"start": 425.85999999999996, "end": 427.85999999999996, "text": " conditioned on x of t"}, {"start": 428.09999999999997, "end": 434.9, "text": " And t so xt is basically the the if you recall from the that this is the forward so again diffusion"}, {"start": 435.38, "end": 439.7, "text": " uh, just a quick reminder is you start with uh, like a"}, {"start": 440.26, "end": 445.29999999999995, "text": " Fresh image. Let's let's uh, let me draw it like here. So we have a fresh image here"}, {"start": 446.09999999999997, "end": 449.38, "text": " And then what you do is you gradually stop adding more and more noise"}, {"start": 449.38, "end": 450.58, "text": " so"}, {"start": 450.58, "end": 456.18, "text": " Basically here you had like some image and then you have here the same image"}, {"start": 456.65999999999997, "end": 460.1, "text": " And basically you start adding gaussian noise"}, {"start": 460.1, "end": 465.62, "text": " So you're going to add some noise here and then you you keep repeating that that that process"}, {"start": 466.18, "end": 468.18, "text": " uh multiple times"}, {"start": 468.18, "end": 470.65999999999997, "text": " Uh, basically, uh, like for example"}, {"start": 471.21999999999997, "end": 474.02, "text": " 1000 4000 are our numbers which makes sense"}, {"start": 474.02, "end": 479.53999999999996, "text": " And so at the end well actually at the end you'll end up with a completely noised image"}, {"start": 479.53999999999996, "end": 482.65999999999997, "text": " So at the end you'll just have a pure gaussian noise here"}, {"start": 482.65999999999997, "end": 487.94, "text": " I'm just gonna like draw it like this basically it's pure noise and what you're trying to learn is how to"}, {"start": 488.26, "end": 494.26, "text": " Go backwards. So this is the forward part. This is the forward direction and we want to learn how to go backwards"}, {"start": 494.26, "end": 500.26, "text": " So we want to learn how to start from noise and then gradually basically generate the image from the underlying"}, {"start": 500.26, "end": 505.62, "text": " Probability distribution which we have learned during the training procedure. Okay?"}, {"start": 506.42, "end": 512.42, "text": " So x of t is basically the t-th step of this forward process. So we we take the x-th image here"}, {"start": 512.42, "end": 519.3, "text": " So there's going to be like some image here and that's like t-th time step"}, {"start": 519.3, "end": 523.7, "text": " This is going to be this is usually denoted as 0-th. This is usually denoted as big t"}, {"start": 523.7, "end": 527.3, "text": " And that's or t minus 1 whatever in any case"}, {"start": 527.3, "end": 534.5, "text": " So that's that's what this x of t is and t is just a scalar which we're gonna obviously"}, {"start": 535.4599999999999, "end": 542.8199999999999, "text": " Encode using the usually they use the like sinusoid embeddings same as for the original transformer paper"}, {"start": 542.9, "end": 548.18, "text": " Okay, so in particular we can train a classifier which I just explained on noisy images"}, {"start": 548.8199999999999, "end": 551.8599999999999, "text": " And then use gradients of that classifier as you can see here"}, {"start": 551.86, "end": 557.46, "text": " So to guide the diffusion sampling process towards an arbitrary class label y"}, {"start": 557.46, "end": 559.46, "text": " So the idea here is"}, {"start": 559.46, "end": 564.26, "text": " This gradient here tells you how you need to tweak the current image"}, {"start": 564.26, "end": 570.9, "text": " So x of t how you need to tweak it such that you maximize the class label of your interest"}, {"start": 570.9, "end": 574.26, "text": " So let's say you want to you want to create a dog image"}, {"start": 574.26, "end": 580.66, "text": " Then you want to make sure that so this will tell you how to tweak the image such that the dog class is"}, {"start": 580.66, "end": 587.2199999999999, "text": " maximized i.e. how to yeah basically how to tweak the diffusion process such that"}, {"start": 587.78, "end": 593.54, "text": " Basically, you start guiding it towards this part of the space where the dog images lie in"}, {"start": 594.26, "end": 600.5, "text": " We're gonna see those details in a second, but like that's that's that's a rough intuition for now. Okay, let's"}, {"start": 601.38, "end": 602.98, "text": " continue here"}, {"start": 602.98, "end": 607.9399999999999, "text": " Here just briefly they briefly mentioned the notation they're gonna omit the t part because it's kind of"}, {"start": 607.94, "end": 612.9000000000001, "text": " Redundant in the sense that well just for concise and sake basically. It's not redundant"}, {"start": 612.9000000000001, "end": 615.94, "text": " It's just easier to write it down like that. Okay, so"}, {"start": 616.5, "end": 621.7, "text": " So here's how we're gonna do this. We start with the diffusion model with an unconditional reverse"}, {"start": 622.2600000000001, "end": 628.2600000000001, "text": " Noising process so p parameters by theta and the idea is to given x t plus one"}, {"start": 628.2600000000001, "end": 630.5, "text": " So that's the image with more noise"}, {"start": 630.5, "end": 635.3000000000001, "text": " You want to predict the next step of the reverse process as I previously explained"}, {"start": 635.3, "end": 642.26, "text": " Here, okay. So that's that's this part. We want to learn how to go how to you how to go from more noise to less noise"}, {"start": 643.38, "end": 645.38, "text": " in any case"}, {"start": 646.02, "end": 651.54, "text": " To condition this on label y it suffices to sample each transition according to"}, {"start": 652.0999999999999, "end": 656.0999999999999, "text": " This distribution here. So if we want to sample from"}, {"start": 656.9, "end": 660.0999999999999, "text": " Class conditions so you can see here. We additionally have the class information here"}, {"start": 660.1, "end": 664.74, "text": " It can be shown and they have a like a whole derivation in the appendix h"}, {"start": 664.74, "end": 670.02, "text": " So if you want to dig into the maths you can go there. I'm gonna kind of assume this to be like true"}, {"start": 670.66, "end": 674.82, "text": " Like as an axiom because otherwise will take too much time for me to explain the maths"}, {"start": 675.38, "end": 679.46, "text": " And in any in any case you can see here. So that is just an arbitrary context"}, {"start": 680.1, "end": 685.46, "text": " Constant and we we already have this part here. So this is the unconditioned"}, {"start": 687.3000000000001, "end": 689.3000000000001, "text": " Reverse process"}, {"start": 689.3, "end": 694.9799999999999, "text": " Reverse process which we learned and here this is going to be the classifier that we're going to train"}, {"start": 695.4599999999999, "end": 698.42, "text": " On the noisy images where by noisy images, I mean"}, {"start": 699.14, "end": 704.66, "text": " Images taken from arbitrary time step of the forward diffusion process. Okay, that's the basic idea"}, {"start": 704.66, "end": 706.66, "text": " So if you take this as as given"}, {"start": 707.2199999999999, "end": 714.02, "text": " Now let's kind of derive how can we actually sample from this distribution on the right hand side because as they mentioned here"}, {"start": 714.26, "end": 718.5799999999999, "text": " Uh, it is typically interactable to sample from this distribution exactly"}, {"start": 718.58, "end": 720.9000000000001, "text": " Okay, so that's that's the that's the idea"}, {"start": 722.5, "end": 726.82, "text": " Now let's see how we can break this down and make it simple and easy to sample from"}, {"start": 728.4200000000001, "end": 729.94, "text": " They say here"}, {"start": 729.94, "end": 735.3000000000001, "text": " Well, we basically know that our our reverse process can be parameterized like this"}, {"start": 735.38, "end": 741.86, "text": " This is what we saw in the previous paper. So we got we're going to learn how to predict mu and covariance matrix"}, {"start": 741.86, "end": 747.94, "text": " Now if we take log of this expression here since this is like e raised to blah blah blah something"}, {"start": 747.94, "end": 753.54, "text": " That's the gaussian definition you end up with this expression here. Okay, so we're working our way here"}, {"start": 753.54, "end": 756.34, "text": " so we are basically trying to understand how we can break this down and"}, {"start": 756.98, "end": 759.38, "text": " now we're going to see how we can break this down and"}, {"start": 759.94, "end": 765.46, "text": " that's going to be on the next page and then we're going to have a simple expression for for uh, like a"}, {"start": 766.26, "end": 769.38, "text": " Sampling from a class conditioned distribution. Okay"}, {"start": 769.38, "end": 773.3, "text": " So under certain assumptions which they mentioned here so we can assume that the log"}, {"start": 774.02, "end": 777.9399999999999, "text": " of this of our classifier has low curvature compared to the"}, {"start": 778.5, "end": 783.62, "text": " Basically inverse of this covariance matrix, which are which we're going to predict from the unconditional"}, {"start": 784.42, "end": 790.42, "text": " Diffusion model. So this assumption is reasonable in the limit of infinite diffusion steps blah blah blah where the"}, {"start": 791.14, "end": 793.14, "text": " Well, I guess this is for binius norm"}, {"start": 793.54, "end": 798.02, "text": " Converges to zero and in this case we can approximate this a log of"}, {"start": 798.02, "end": 800.8199999999999, "text": " the classifier using a Taylor expansion around"}, {"start": 801.38, "end": 803.38, "text": " mu and mu is basically the"}, {"start": 803.86, "end": 808.18, "text": " Mean of the x of t. So basically let me go back here"}, {"start": 809.46, "end": 813.54, "text": " So that's the same u as this one here. Okay, so that's that's the"}, {"start": 814.34, "end": 818.9, "text": " Basically the mean of the distribution from which we're going to sample x of t. That's it"}, {"start": 819.62, "end": 822.98, "text": " Fairly simple and if we do that they end up with this expression here"}, {"start": 822.98, "end": 828.82, "text": " This component here is going to be very very important and under this like"}, {"start": 829.7, "end": 837.22, "text": " Basically approximation and then derivation here. We can finally get the the the the final expression here"}, {"start": 837.22, "end": 840.34, "text": " So we just take the log of the distribution"}, {"start": 840.34, "end": 843.94, "text": " We just saw above and we can omit z because that is a constant"}, {"start": 843.94, "end": 848.02, "text": " So we don't care about it and you can see that after some derivation here"}, {"start": 848.02, "end": 850.02, "text": " We end up with this expression here"}, {"start": 850.02, "end": 852.66, "text": " So again, we can ignore a c4 is just a constant"}, {"start": 852.66, "end": 857.86, "text": " So we basically if we want to sample from that complex intractable distribution here"}, {"start": 858.34, "end": 862.18, "text": " What we in practice need to do is just sample from this gaussian here"}, {"start": 862.18, "end": 868.66, "text": " So as you can see the Gaussian just has a mean a shifted mean so we have mu which is from the unconditional"}, {"start": 869.38, "end": 871.78, "text": " Diffusion model and we just simply add"}, {"start": 872.8199999999999, "end": 876.26, "text": " This offset here. So we have the covariance matrix. We just"}, {"start": 876.26, "end": 879.7, "text": " basically do like we"}, {"start": 880.26, "end": 885.9399999999999, "text": " Multiply it with the g and g is just the as you can see here the gradient of our classifier"}, {"start": 886.9, "end": 892.5, "text": " For x of t equals mu. So i'm going to give you some intuition for behind these formulas"}, {"start": 892.5, "end": 897.7, "text": " But first let me just tell you this so so we have thus found that the conditional transition operator"}, {"start": 897.7, "end": 900.02, "text": " So that's the the one we saw above here"}, {"start": 900.02, "end": 905.3, "text": " Can be approximated by a gaussian similar to the unconditional transition operator"}, {"start": 905.3, "end": 908.74, "text": " But with its mean shifted by sigma g"}, {"start": 909.3, "end": 911.78, "text": " Okay, so let me try and"}, {"start": 912.5, "end": 917.06, "text": " Give you some intuition for why this works. It's going to be a very hand-wavy explanation"}, {"start": 917.06, "end": 922.9, "text": " But like I think the intuition is is completely correct on some level of abstraction. It's a correct"}, {"start": 922.9, "end": 924.98, "text": " So let's just say that we have a condition"}, {"start": 924.98, "end": 930.1, "text": " But I think the intuition is is completely correct on some level of abstraction. It's it's a correct"}, {"start": 931.14, "end": 935.86, "text": " Hand-wavy explanation. Okay. So g again it tells you how you should tweak"}, {"start": 936.82, "end": 942.34, "text": " X of t so the the image such that you can maximize certain class y. Okay, so"}, {"start": 943.38, "end": 947.62, "text": " Let's see what we do. So we we pass in we have our model and"}, {"start": 948.26, "end": 952.1, "text": " Let me kind of draw it like this. So let me just change the color here"}, {"start": 952.1, "end": 954.1, "text": " So we have an image"}, {"start": 954.66, "end": 962.02, "text": " That's x t plus one. So that's the x t plus one image. We pass that image into our diffusion model"}, {"start": 962.02, "end": 966.4200000000001, "text": " So that's our diffusion model here and out comes what outcomes mu?"}, {"start": 967.38, "end": 969.38, "text": " and out comes this"}, {"start": 969.86, "end": 978.34, "text": " Covariance matrix and these two together basically parameterize the Gaussian distribution from which we would sample the"}, {"start": 978.34, "end": 984.02, "text": " Next so x of t so from from this from this distribution here. We sample x of t"}, {"start": 985.14, "end": 988.82, "text": " Okay, so we can kind of visualize the Gaussian as just"}, {"start": 989.5400000000001, "end": 994.1, "text": " Like a point in a in a in a multi-dimensional space for the sake of explanation"}, {"start": 994.1, "end": 996.98, "text": " I'm just gonna assume we're we're dealing with 3d space"}, {"start": 996.98, "end": 1004.1800000000001, "text": " But obviously this is going to be much higher dimensionality mu is actually the dimension the same dimensionality as our our image"}, {"start": 1004.18, "end": 1007.6999999999999, "text": " so basically, uh, it's gonna be um,"}, {"start": 1008.3399999999999, "end": 1009.3, "text": " well"}, {"start": 1009.3, "end": 1010.0999999999999, "text": " h"}, {"start": 1010.0999999999999, "end": 1015.06, "text": " Times w times three so that's the dimension of our image and similarly for for sigma"}, {"start": 1015.38, "end": 1021.14, "text": " But just for the sake of understanding what's going on with this shift here. So the the sigma g part"}, {"start": 1021.54, "end": 1024.26, "text": " I'm gonna kind of assume a coordinate system like this and"}, {"start": 1024.98, "end": 1030.02, "text": " Like assume we have a like a Gaussian distribution. The mean is here. So this is the mu. Okay"}, {"start": 1030.02, "end": 1034.5, "text": " and uh, we have some we basically have some some variance"}, {"start": 1036.02, "end": 1040.74, "text": " So under normal circumstances if we just have like an unconditional diffusion model"}, {"start": 1040.98, "end": 1046.5, "text": " We would basically sample a point some from from this distribution somewhere from from this distribution"}, {"start": 1046.5, "end": 1051.62, "text": " Like maybe maybe this would be our next uh, this would be our x of t. Okay, but"}, {"start": 1052.26, "end": 1056.42, "text": " Because we we additionally have this information coming from from the classifier"}, {"start": 1056.42, "end": 1060.8200000000002, "text": " What it does is the following so now you take this mu here"}, {"start": 1061.7, "end": 1063.7, "text": " Let me change the color"}, {"start": 1064.1000000000001, "end": 1069.38, "text": " Okay, so so you take the mu and you pass that as an image to like our classifier"}, {"start": 1069.38, "end": 1072.3400000000001, "text": " So we're gonna have something like this. So we have an image"}, {"start": 1072.9, "end": 1075.94, "text": " That's gonna be the same as this one here. So that's gonna be mu"}, {"start": 1076.74, "end": 1079.3000000000002, "text": " We pass that inside of our classifier"}, {"start": 1080.02, "end": 1082.02, "text": " This is our classifier"}, {"start": 1082.02, "end": 1087.78, "text": " And out comes uh, well for image net they train obviously it has like thousand classes"}, {"start": 1089.3799999999999, "end": 1090.26, "text": " And"}, {"start": 1090.26, "end": 1093.3799999999999, "text": " We basically learn how we should tweak this image"}, {"start": 1094.18, "end": 1100.18, "text": " So the classifier tells us how we should tweak the image such that a particular class is maximized maybe this one"}, {"start": 1100.5, "end": 1106.66, "text": " Okay, and with that information with that gradient, so so those blue dots. I'm just gonna denote that as g"}, {"start": 1106.66, "end": 1112.02, "text": " With that information which is of the same dimensionality as the mu we're going to shift this mu"}, {"start": 1112.02, "end": 1114.5800000000002, "text": " So we're going to shift this maybe here"}, {"start": 1115.8600000000001, "end": 1118.9, "text": " And we're going to end up here. So our new distribution"}, {"start": 1119.6200000000001, "end": 1125.14, "text": " Is going to be actually here instead of here. Okay, and now we have and we have the same variance"}, {"start": 1125.14, "end": 1128.5800000000002, "text": " So i'm going to kind of change the color because otherwise you will not be able to see anything"}, {"start": 1129.5400000000002, "end": 1132.42, "text": " So we shifted our original color"}, {"start": 1132.42, "end": 1139.38, "text": " And we're not going to be able to see anything so we shifted our original distribution and now we are somewhere here"}, {"start": 1140.26, "end": 1142.9, "text": " Okay, it's kind of messy, but you get the point so this"}, {"start": 1143.6200000000001, "end": 1149.54, "text": " Shift vector we just saw there. So that's this blue one is just this expression here. So"}, {"start": 1150.9, "end": 1152.98, "text": " Sigma big sigma times g"}, {"start": 1155.3000000000002, "end": 1158.3400000000001, "text": " And this is the like the hand way we explanation what's going on"}, {"start": 1158.34, "end": 1162.74, "text": " So the classifier knows where you should kind of move this image"}, {"start": 1163.62, "end": 1170.1799999999998, "text": " In which direction should you move this image such such that you maximize the class of interest and by just doing that by just kind of"}, {"start": 1170.58, "end": 1172.74, "text": " moving the"}, {"start": 1172.74, "end": 1175.8799999999999, "text": " The mean in the direction of the gradient of the classifier"}, {"start": 1176.26, "end": 1184.1799999999998, "text": " You basically increase the probability of the class label here and that's that's basically everything you need to understand. Okay"}, {"start": 1184.18, "end": 1188.26, "text": " Uh, hopefully that's clear enough. That's how we can combine again"}, {"start": 1189.0600000000002, "end": 1196.9, "text": " The classifier here and the unconditional diffusion model here to form images from a particular class"}, {"start": 1197.6200000000001, "end": 1202.5, "text": " Again, i'm kind of lying here because as I said, they are still class conditioning this model here"}, {"start": 1202.5, "end": 1211.14, "text": " But using the classifier we additionally complementary, uh improve, uh the uh, well this class conditioned, uh generation"}, {"start": 1211.14, "end": 1213.6200000000001, "text": " Cool, uh, that's it. Let me briefly, um"}, {"start": 1214.3400000000001, "end": 1219.38, "text": " Explain a couple more things. Um, aside from using the the the classifier"}, {"start": 1219.38, "end": 1226.3400000000001, "text": " The classifier gradient they additionally scale it and this is just empirically they found out that this gives them much better results"}, {"start": 1226.66, "end": 1231.94, "text": " So here they mentioned our classifier architecture is simply the down sampling trunk of the unit model"}, {"start": 1232.26, "end": 1236.66, "text": " With an attention pool at the 8 times 8 layer to produce the final output"}, {"start": 1236.66, "end": 1241.14, "text": " We train these classifiers on the same noising distribution as the corresponding diffusion model"}, {"start": 1241.7, "end": 1245.22, "text": " Then they say in initial experiments with unconditional image map models"}, {"start": 1245.46, "end": 1250.9, "text": " We found it necessary to scale the classifier gradients by constant factor larger than one"}, {"start": 1250.9, "end": 1255.8600000000001, "text": " So to understand the effect of the scaling classifier gradients note that if we scale by s"}, {"start": 1256.3400000000001, "end": 1261.8600000000001, "text": " So this is the what we denoted as g previously. This is these are the gradients of our classifier"}, {"start": 1261.86, "end": 1267.3, "text": " So if we do this you can show quite easily that that's equivalent to this expression here"}, {"start": 1267.3, "end": 1271.06, "text": " So well, basically, um, you can convince yourself"}, {"start": 1271.06, "end": 1276.82, "text": " This is true because log of a product is log of sums and because this is delta x"}, {"start": 1276.82, "end": 1281.54, "text": " We don't care about the this part here and then we would end up with the same expression"}, {"start": 1281.54, "end": 1283.06, "text": " Basically, these two are equivalent"}, {"start": 1283.06, "end": 1288.6599999999999, "text": " Okay, and you can see that we are basically just raising this to the power of s which which means we're going to sharpen this distribution"}, {"start": 1288.66, "end": 1292.1000000000001, "text": " So let me continue reading what they said here"}, {"start": 1292.1000000000001, "end": 1299.5400000000002, "text": " So so z is an arbitrary constant and as a result the conditioning process is still theoretically grounded in our renormalized classifier distribution proportional to"}, {"start": 1300.18, "end": 1306.18, "text": " P raised to the power of s when s is bigger than one. This is distribution becomes sharper than"}, {"start": 1306.98, "end": 1313.6200000000001, "text": " The original distribution. Okay in the above derivations. We assume that the underlying diffusion model was unconditional modeling"}, {"start": 1313.62, "end": 1318.1799999999998, "text": " P of x it is also possible to train conditional diffusion models"}, {"start": 1318.8999999999999, "end": 1328.34, "text": " And use classifier guidance in the exact same way table 4 shows that the sample quality of both unconditional and conditional models can be greatly improved by classifier guidance"}, {"start": 1328.34, "end": 1335.3799999999999, "text": " So this is the the thing I was mentioning we we can both have conditional model plus the classifier guidance because they are complementary"}, {"start": 1335.3799999999999, "end": 1338.1799999999998, "text": " We see that with a high enough scale the guided"}, {"start": 1338.18, "end": 1344.02, "text": " Unconditional model can get quite close to the fid of an unguided conditional model"}, {"start": 1344.02, "end": 1349.94, "text": " Although training directly with the class label still helps finally guiding a conditional model further improves fid"}, {"start": 1349.94, "end": 1356.1000000000001, "text": " So that's what I mentioned multiple times using the class conditional model plus using the guiding is what gave them the best"}, {"start": 1356.74, "end": 1360.42, "text": " Result on various metrics such as fid is inception score"}, {"start": 1361.22, "end": 1363.22, "text": " Precision recall, etc, etc. Okay"}, {"start": 1363.22, "end": 1367.6200000000001, "text": " Um quickly to let me just quickly address this because we're going to see it in the code later on"}, {"start": 1367.94, "end": 1371.54, "text": " So how they train the so why do they need to train the model?"}, {"start": 1371.54, "end": 1377.46, "text": " Uh, the the the classifier on on noisy images because they say here we train these classifiers on the same"}, {"start": 1377.46, "end": 1379.7, "text": " Noising distribution as the corresponding diffusion model"}, {"start": 1380.5, "end": 1381.14, "text": " Well"}, {"start": 1381.14, "end": 1388.58, "text": " Because we're going to be using it on images on x of t so x of t which can be super noisy"}, {"start": 1388.58, "end": 1394.98, "text": " And if we just train our classifier on normal images normal when I say normal I mean images without any noise"}, {"start": 1395.62, "end": 1400.34, "text": " Then the model will probably struggle because these images would be out of distribution for the model"}, {"start": 1401.22, "end": 1408.26, "text": " And then it would not give us correct class for a noisy version of the original image because of that we want to train model"}, {"start": 1408.26, "end": 1411.86, "text": " For all so basically on the whole process. So this is our diffusion"}, {"start": 1413.22, "end": 1415.9399999999998, "text": " Like for example, if we want to train the model"}, {"start": 1415.94, "end": 1422.74, "text": " Our diffusion, uh, like forward process something like this. So this is the step zero. This is the step"}, {"start": 1423.14, "end": 1425.14, "text": " Let's say big t minus one"}, {"start": 1425.94, "end": 1428.66, "text": " We want to train on all of these images"}, {"start": 1430.42, "end": 1432.42, "text": " So we'll be randomly sampling images"}, {"start": 1432.74, "end": 1437.22, "text": " Um, and uh making sure that the model knows how to predict the correct class"}, {"start": 1437.6200000000001, "end": 1443.06, "text": " And uh, because of that this is kind of bothersome and some of the later papers such as glide"}, {"start": 1443.06, "end": 1448.74, "text": " Introduce this concept of a classifier free guidance. I'm gonna, um, well tell you more about that one"}, {"start": 1448.74, "end": 1451.78, "text": " Uh in one of the future videos. Okay, let's go back here"}, {"start": 1452.82, "end": 1457.1399999999999, "text": " Uh here we can see that the trade-off we mentioned in the beginning of the video"}, {"start": 1457.46, "end": 1461.86, "text": " And that's that we want to trade out between diversity and quality of the sample"}, {"start": 1462.34, "end": 1465.46, "text": " They show here using various metrics that that's indeed the case"}, {"start": 1465.46, "end": 1469.3799999999999, "text": " So you can see here as we increase this gradient scale. So that's the hyper parameter s"}, {"start": 1469.38, "end": 1476.3400000000001, "text": " Uh, you can see that the recall is uh falling down which means that the diversity the the the mode coverage"}, {"start": 1476.66, "end": 1478.66, "text": " Of our generative model is falling down"}, {"start": 1479.0600000000002, "end": 1484.1000000000001, "text": " But the precision which basically um, well, uh, it's a proxy for the image quality"}, {"start": 1484.5800000000002, "end": 1490.8200000000002, "text": " Goes up and you can see similarly here is goes up which means that the image quality is increasing"}, {"start": 1491.14, "end": 1498.18, "text": " And f id is is also going up which means that the diversity is is uh, basically, uh, diminishing"}, {"start": 1498.18, "end": 1503.54, "text": " Okay, that's pretty much it. I'm gonna skip all of this and i'm just gonna briefly, uh"}, {"start": 1504.02, "end": 1506.9, "text": " Uh mention a couple sentences here and then we're gonna jump into the code"}, {"start": 1507.6200000000001, "end": 1512.42, "text": " Our proposed classifier guidance technique is currently limited to labeled datasets. So keep that in mind"}, {"start": 1512.5, "end": 1515.54, "text": " We do have to train it on on on labeled, uh images"}, {"start": 1516.5800000000002, "end": 1517.6200000000001, "text": " um"}, {"start": 1517.6200000000001, "end": 1524.1000000000001, "text": " Well, the effectiveness of classifier guidance demonstrates that we can obtain powerful generative models from the gradients of a classification function"}, {"start": 1524.1, "end": 1531.62, "text": " This could be used to condition pre-trained models in a plethora of ways for example by conditioning an image generator with a text caption"}, {"start": 1531.9399999999998, "end": 1535.9399999999998, "text": " Using a noisy version of clip and because of this sentence"}, {"start": 1536.74, "end": 1537.78, "text": " After reading this paper"}, {"start": 1537.78, "end": 1544.5, "text": " You could have assumed that open ai would start working on something like glide or dali models and they did indeed"}, {"start": 1545.3, "end": 1552.8999999999999, "text": " And finally it also suggests that large unlabeled datasets could be leveraged in the future to pre-train powerful diffusion models that can later be improved"}, {"start": 1552.9, "end": 1557.46, "text": " By using a classifier with desirable properties. Okay again"}, {"start": 1558.26, "end": 1560.02, "text": " You could have if you read this paper"}, {"start": 1560.02, "end": 1565.46, "text": " You could have anticipated open ai is going to try and just scale up scale up diffusion models"}, {"start": 1565.8600000000001, "end": 1570.9, "text": " Uh such as what they've done with with dali and when I say dali I basically mean dali"}, {"start": 1571.38, "end": 1576.66, "text": " Two because dali one didn't even use diffusion models. Okay, let's jump to the code now"}, {"start": 1576.98, "end": 1578.98, "text": " um, I I did expect this"}, {"start": 1578.98, "end": 1582.34, "text": " Uh paper overview part to be a bit shorter"}, {"start": 1582.58, "end": 1587.46, "text": " But like do let me know whether you find this type of combination useful or not"}, {"start": 1588.02, "end": 1591.54, "text": " Just feel free to comment down below. What do you think about this format?"}, {"start": 1592.18, "end": 1594.18, "text": " In any case, let's now"}, {"start": 1594.66, "end": 1599.22, "text": " Start working on on the classifier train script. So this is how we trained"}, {"start": 1599.8600000000001, "end": 1602.74, "text": " This is how we train this this noisy classifier"}, {"start": 1602.74, "end": 1608.5, "text": " Uh that I just mentioned so let me start and walk you through the actual script for training it"}, {"start": 1609.6200000000001, "end": 1615.8, "text": " So i'm gonna obviously again, uh abstract a lot of the details because otherwise this would not be tractable"}, {"start": 1616.26, "end": 1618.26, "text": " uh for for a video"}, {"start": 1619.06, "end": 1623.14, "text": " That has an ambition to be less than an hour long. Uh, so just some"}, {"start": 1624.1, "end": 1628.1, "text": " Setting up some distributed stuff, which we don't care about logging blah blah blah"}, {"start": 1628.1, "end": 1634.82, "text": " Um this part part we do care about because I just want to briefly show you basically how the classifier is constructed"}, {"start": 1635.78, "end": 1642.1, "text": " We did sign the paper that it's simply the first part of the unit model up to the bottleneck part and then we just stitch"}, {"start": 1642.6599999999999, "end": 1648.98, "text": " Attention pooling and we ditch the the second part of the unit model. That's how we that's how they set up the uh, the classifier"}, {"start": 1649.2199999999998, "end": 1654.98, "text": " The architecture so let's kind of go inside here. So this just kind of gives you some"}, {"start": 1654.98, "end": 1659.94, "text": " Um default arguments i'm gonna try and zoom in a little bit in case you don't see this"}, {"start": 1660.34, "end": 1664.02, "text": " I'm gonna close this part. Okay, so let's step inside this"}, {"start": 1665.3, "end": 1667.8600000000001, "text": " Line of code and let's see how the classifier works"}, {"start": 1668.66, "end": 1670.66, "text": " Okay, so here's the classifier"}, {"start": 1670.82, "end": 1676.58, "text": " Image size 128 128 blah blah. Those are just some uh hyper parameters obviously can be different"}, {"start": 1677.06, "end": 1679.06, "text": " Uh, we are not using fp16"}, {"start": 1679.94, "end": 1681.94, "text": " Uh, there is some width depth"}, {"start": 1681.94, "end": 1688.66, "text": " Various hyper parameters, uh, we saw this is going to be um, well basically this is going to enable"}, {"start": 1688.8200000000002, "end": 1696.66, "text": " Uh the attention to be at resolutions 32 times 32 in the latent space as well as 16 by 16 and 8 by 8"}, {"start": 1697.14, "end": 1700.8200000000002, "text": " Um, this is the scale shift norm that I showed you in the previous video"}, {"start": 1701.14, "end": 1704.18, "text": " uh, this basically, um enables us to"}, {"start": 1704.8200000000002, "end": 1709.8600000000001, "text": " Integrate to incorporate the the class as well as the time step information into the model"}, {"start": 1710.42, "end": 1711.54, "text": " um"}, {"start": 1711.54, "end": 1713.78, "text": " In any case i'm going to scheme over those details"}, {"start": 1714.26, "end": 1718.58, "text": " Um, let me just kind of go through here because we have image size 128"}, {"start": 1718.58, "end": 1722.1, "text": " We have some specification for the number of channels in the unit"}, {"start": 1722.82, "end": 1729.62, "text": " And finally, we just uh transform, uh this from the resolution. So so remember this is like, um,"}, {"start": 1730.1, "end": 1734.74, "text": " Basically 6 32 16 and 8 after we step through this, uh for loop"}, {"start": 1734.98, "end": 1737.7, "text": " We're basically and end up with the same information"}, {"start": 1737.7, "end": 1743.6200000000001, "text": " But just like and now it it tells us how many down sampling steps you need to take before you apply the attention"}, {"start": 1743.94, "end": 1747.78, "text": " Uh, we saw that in the previous video as well. So i'm gonna just kind of skip over this"}, {"start": 1748.26, "end": 1751.8600000000001, "text": " Blah blah blah and here here it comes. So here's the encoder unit model"}, {"start": 1753.14, "end": 1760.18, "text": " Basically, let me see what interesting is here. So we have the output number of channels is hard coded to 2000 because this is"}, {"start": 1760.98, "end": 1762.98, "text": " trained on on on image net"}, {"start": 1762.98, "end": 1767.8600000000001, "text": " Uh, i'm gonna skip over all of those not that important"}, {"start": 1768.58, "end": 1772.82, "text": " Uh classifier pool is going to be attention. We're gonna see that in a second. So let me"}, {"start": 1773.8600000000001, "end": 1776.02, "text": " Go to the construction constructor here"}, {"start": 1776.5, "end": 1783.22, "text": " So i'm gonna again just scheme over they're just kind of setting all of these arguments to um local"}, {"start": 1783.94, "end": 1785.6200000000001, "text": " variables here"}, {"start": 1785.6200000000001, "end": 1787.7, "text": " Uh, nothing nothing fancy there"}, {"start": 1787.7, "end": 1793.94, "text": " Uh, we construct the uh time embedding. This is the the mlp that we use to transform"}, {"start": 1794.42, "end": 1798.42, "text": " The sinusoidal embeddings for our time step time steps"}, {"start": 1798.9, "end": 1799.8600000000001, "text": " um"}, {"start": 1799.8600000000001, "end": 1803.78, "text": " Nothing, nothing interesting really we can just kind of scheme over all of this. So"}, {"start": 1804.3400000000001, "end": 1807.94, "text": " Um, if we recall the from the previous video we had the same structure"}, {"start": 1807.94, "end": 1813.22, "text": " So we initially just have number of input block. We just have the conv, uh, basically 2d"}, {"start": 1813.22, "end": 1817.3, "text": " Uh, so the convolution as the first processing layer"}, {"start": 1817.6200000000001, "end": 1819.38, "text": " then we have this uh"}, {"start": 1819.38, "end": 1822.34, "text": " part where we uh add various"}, {"start": 1822.88, "end": 1827.78, "text": " Red residual residual blocks and attention blocks and then we have the middle block"}, {"start": 1828.18, "end": 1832.34, "text": " And the main difference is we don't have here the output blocks. We just have as you can see here"}, {"start": 1832.34, "end": 1834.82, "text": " We just have the pooling operator. So let me go"}, {"start": 1835.7, "end": 1841.14, "text": " To there obviously these details are not that important. The only important part lies in the residual block"}, {"start": 1841.14, "end": 1843.14, "text": " where you can see how the um"}, {"start": 1843.5400000000002, "end": 1850.18, "text": " How the information such as time steps and class information is incorporated. I'm going to show you that in a second"}, {"start": 1850.26, "end": 1854.18, "text": " I'm going to kind of skip over all of this again. You can see here that uh"}, {"start": 1854.18, "end": 1860.9, "text": " Sometimes we're going to add attention blocks and that's going to happen precisely for the resolutions 32 16 and 8"}, {"start": 1861.6200000000001, "end": 1868.26, "text": " And there are these wrapper functions which just enable us to uh, sometimes pass the temporal information"}, {"start": 1868.26, "end": 1874.9, "text": " Sometimes just pass the the feature vectors. Okay, in any case i'm going to skip all of this because i'm just um"}, {"start": 1875.3, "end": 1878.98, "text": " There's too much information there and it's not that vital for your understanding"}, {"start": 1878.98, "end": 1882.82, "text": " It's just like the architectural details of the unit model. I'm just going to show you this one"}, {"start": 1883.06, "end": 1886.98, "text": " So here is the let me just kind of uh remind you from the previous video"}, {"start": 1888.18, "end": 1889.22, "text": " uh"}, {"start": 1889.22, "end": 1896.42, "text": " There is this part where we embed so here we embed the uh temporal information and then because this is set to true"}, {"start": 1896.42, "end": 1903.0600000000002, "text": " Use scale shift norm what we're gonna do is we're gonna take the temporal information here as in a form of a scale and shift"}, {"start": 1903.3000000000002, "end": 1907.54, "text": " And we are going to combine that this is how we merge the temporal information"}, {"start": 1907.94, "end": 1912.5800000000002, "text": " Uh into into the uh features here and this is just as a reminder"}, {"start": 1913.14, "end": 1918.74, "text": " For the what happened in the previous video. Let me just see whether they have class information here"}, {"start": 1919.46, "end": 1922.5800000000002, "text": " Um doesn't seem to be the case so embedding"}, {"start": 1923.3000000000002, "end": 1925.3000000000002, "text": " They passed embedding here"}, {"start": 1925.3, "end": 1931.46, "text": " No, I don't think this is going to have well, it doesn't make any sense to have uh like a class"}, {"start": 1932.26, "end": 1938.1, "text": " Um conditioning in a classifier. So this is going to make sense for the diffusion model and in any case"}, {"start": 1938.1, "end": 1940.1, "text": " Okay, let's continue here"}, {"start": 1940.1, "end": 1941.3, "text": " so we"}, {"start": 1941.3, "end": 1946.8999999999999, "text": " Uh, we have the the unit part and now we just end up uh with this um"}, {"start": 1947.46, "end": 1952.4199999999998, "text": " Sequential model that has some normalization some activation function and finally this attention pooling 2d"}, {"start": 1952.42, "end": 1956.9, "text": " 2d this is basically just going to take the output features from the unit model"}, {"start": 1957.94, "end": 1962.02, "text": " And using a tension mechanism is going to form a resultant"}, {"start": 1962.5800000000002, "end": 1970.18, "text": " feature vector which we're going to then use to predict the the logits and you can see that the output channels here is"}, {"start": 1970.18, "end": 1972.18, "text": " Thousands is going to give us a thousand"}, {"start": 1972.8200000000002, "end": 1978.66, "text": " Thousand dimensional vector which is going to be just the logits that come out of this classifier and that's it guys"}, {"start": 1978.66, "end": 1984.1000000000001, "text": " Um, I kind of schemed a lot of details because they are not really that important. So"}, {"start": 1984.8200000000002, "end": 1989.78, "text": " Uh, you just need to have like a rough mental model how the architecture looks like it has this this basically"}, {"start": 1990.42, "end": 1995.78, "text": " encoder type of a structure it has this attention pooling at the end and"}, {"start": 1997.3000000000002, "end": 1998.3400000000001, "text": " That's pretty much it"}, {"start": 1998.3400000000001, "end": 1998.5800000000002, "text": " Okay"}, {"start": 1998.5800000000002, "end": 2003.8600000000001, "text": " So I skipped the the gaussian diffusion model here because we saw that in the previous video"}, {"start": 2003.8600000000001, "end": 2007.5400000000002, "text": " I just want to focus on the differences compared to the previous video"}, {"start": 2007.54, "end": 2011.8799999999999, "text": " Uh, so here we just uh shift the model onto the gpu"}, {"start": 2012.42, "end": 2018.84, "text": " Uh, we create the uniform sampler here. So this one is just going to help us sample from the forward"}, {"start": 2019.62, "end": 2024.6599999999999, "text": " Process we're just gonna pick the random uh time step using the uniform distribution"}, {"start": 2024.74, "end": 2028.34, "text": " Obviously you can kind of bias this and and form various different distributions"}, {"start": 2028.58, "end": 2031.78, "text": " One of those which I showed you in the previous video was depending on the loss"}, {"start": 2031.78, "end": 2037.78, "text": " You're gonna focus more on those time steps where we have higher loss, which makes sense intuitively"}, {"start": 2038.42, "end": 2043.22, "text": " Um, okay, so we can skip over all of these parts. We don't want to resume from the checkpoint"}, {"start": 2043.62, "end": 2048.5, "text": " We just have some mixed precision trainer wrapper here. We don't care about that part. We have the distributed"}, {"start": 2049.06, "end": 2054.18, "text": " Data parallel object that's for distributed training, but we don't care about it either"}, {"start": 2055.06, "end": 2058.74, "text": " I'm gonna skip over all of that. We have some data loading"}, {"start": 2058.74, "end": 2063.4599999999996, "text": " Basically, we're gonna have batch size 256 image size 120 128"}, {"start": 2063.7799999999997, "end": 2069.22, "text": " Uh, we have the data directory which I downloaded before the video started and that's it"}, {"start": 2069.22, "end": 2071.7, "text": " We just create the training and validation datasets"}, {"start": 2071.7, "end": 2076.8199999999997, "text": " I'm not gonna dig into the actual details of how that works because it's just like a side thing"}, {"start": 2077.3799999999997, "end": 2083.14, "text": " Um, so logging, addmw optimizer. We're not resuming from a checkpoint blah blah blah"}, {"start": 2083.3799999999997, "end": 2087.22, "text": " And here this this is going to be the main function to understand how to classify a string"}, {"start": 2087.22, "end": 2092.8999999999996, "text": " So we have forward backward log. I've added a breakpoint to that function. We're gonna step into it a bit later"}, {"start": 2093.14, "end": 2094.8199999999997, "text": " So here we defined it"}, {"start": 2094.8199999999997, "end": 2101.4599999999996, "text": " And now here we have we step for a number of iterations. Okay, so we're gonna do this for a number of steps"}, {"start": 2101.54, "end": 2104.02, "text": " So as specified by the number of iterations there"}, {"start": 2104.18, "end": 2109.2999999999997, "text": " I've just set it to 30 because I want to have a short training here just for for"}, {"start": 2110.02, "end": 2113.8599999999997, "text": " Debugging purposes in any case just some logging nothing interesting"}, {"start": 2113.86, "end": 2120.58, "text": " I can skip all of this and this is where the whole magic of the classifier training happens after that. We just do the"}, {"start": 2121.6, "end": 2129.2200000000003, "text": " Optimization step and we do the same logic for on the validation. We just do the forward backward on the validation"}, {"start": 2130.6600000000003, "end": 2134.02, "text": " Set and finally we just do some logging here"}, {"start": 2134.34, "end": 2139.86, "text": " So basically this is where the whole magic is in so the forward backward log function is everything that we care about"}, {"start": 2140.1800000000003, "end": 2142.1800000000003, "text": " So let's step into it"}, {"start": 2142.18, "end": 2148.02, "text": " So we first gonna grab like a batch of images and associated labels. Let me show you the"}, {"start": 2148.8999999999996, "end": 2150.02, "text": " shapes"}, {"start": 2150.02, "end": 2152.02, "text": " As always that's a very important thing to do"}, {"start": 2152.8199999999997, "end": 2156.4199999999996, "text": " So let's see what batch looks like so if I do batch shape"}, {"start": 2157.7799999999997, "end": 2159.46, "text": " So we can see here"}, {"start": 2159.46, "end": 2162.5, "text": " Bad shape so we have 200 256 images"}, {"start": 2163.22, "end": 2166.3599999999997, "text": " RGB images of resolution 128 128"}, {"start": 2166.36, "end": 2172.28, "text": " We have the labels as well. So after I do this, so basically this extra here"}, {"start": 2172.92, "end": 2174.92, "text": " Just contains this y"}, {"start": 2175.56, "end": 2177.32, "text": " And let's see the shape"}, {"start": 2177.32, "end": 2182.1200000000003, "text": " So shape is 256. Obviously we have for each of the images we have associated label"}, {"start": 2182.6800000000003, "end": 2184.92, "text": " If I were to step over this"}, {"start": 2186.52, "end": 2188.6, "text": " And print labels"}, {"start": 2188.6, "end": 2193.88, "text": " We can see there is a bunch of labels here because i'm using cipher 10 even though this is"}, {"start": 2193.88, "end": 2195.88, "text": " This is image net training"}, {"start": 2196.44, "end": 2202.76, "text": " That's why we only see labels from zero through nine, but we don't care about this detail for the"}, {"start": 2203.32, "end": 2209.08, "text": " Purpose of understanding how the classified training looks like. Okay, so we we put the as you can see here"}, {"start": 2209.08, "end": 2214.44, "text": " We put the labels onto our target device, which is gpu in my case same with images here"}, {"start": 2215.0, "end": 2220.36, "text": " And now what we do is we do we take the uniform sampler and we just sample"}, {"start": 2220.36, "end": 2227.26, "text": " 256 time steps. That's it. So we now have let's see if I were to print t you can see"}, {"start": 2227.7400000000002, "end": 2229.9, "text": " Uh, like it's a it's a pile of numbers"}, {"start": 2230.7000000000003, "end": 2234.06, "text": " Of the following length. Let me just print this so we have 256"}, {"start": 2234.6200000000003, "end": 2241.82, "text": " time steps for each of the images in the batch and it's just randomly sampled between uh zero and"}, {"start": 2242.3, "end": 2246.78, "text": " And and 999 after that we just do the q sample"}, {"start": 2246.78, "end": 2251.26, "text": " So we take the image we take the time step and we just do the noising process"}, {"start": 2251.5, "end": 2257.6600000000003, "text": " So we create from the x0 image we create we create the x t image. I'm gonna quickly remind you"}, {"start": 2257.7400000000002, "end": 2262.2200000000003, "text": " I showed you this in the previous video. So i'm going to just quickly show you how this thing looks like"}, {"start": 2262.5400000000004, "end": 2268.38, "text": " So here it is. Uh, there is literally a formula which I showed you in the previous paper, which is fairly easy"}, {"start": 2268.6200000000003, "end": 2270.6200000000003, "text": " you just take uh these"}, {"start": 2270.62, "end": 2277.8199999999997, "text": " Uh specific constants here you multiply it with the x start x start is the x0 the original image"}, {"start": 2278.06, "end": 2280.06, "text": " And we just add noise"}, {"start": 2280.54, "end": 2286.8599999999997, "text": " According to this square blah blah. Again, there's some content constant. Let me try to find a formula. It's going to be easier"}, {"start": 2287.2599999999998, "end": 2290.94, "text": " Okay, guys, here is the formula from the previous video. So here is how we sample"}, {"start": 2291.18, "end": 2294.38, "text": " So given x0 the original image, here's how we get x of t"}, {"start": 2294.7, "end": 2300.54, "text": " We can just basically sample from this distribution here or equivalently. We just compute this expression here"}, {"start": 2300.54, "end": 2303.1, "text": " So you can see here we multiply by some constant constant"}, {"start": 2303.18, "end": 2306.54, "text": " We multiply the x0 or x start as we saw in the code"}, {"start": 2306.86, "end": 2311.9, "text": " We just add up on on top of that. We add this constant times epsilon, which is just your"}, {"start": 2312.54, "end": 2314.54, "text": " noise from your normal"}, {"start": 2314.84, "end": 2321.02, "text": " Distribution now, let's go back to the code. Let me see whether you can you can see resemblance between this formula here"}, {"start": 2321.66, "end": 2325.42, "text": " And between this thing here. So again constant times x start"}, {"start": 2325.42, "end": 2330.54, "text": " We add on top of that some constant times noise and that's how we get xt and that's it"}, {"start": 2330.54, "end": 2336.62, "text": " That's because that's why i'm going to just skip over this function and we have finally we now have a batch of"}, {"start": 2337.26, "end": 2342.0, "text": " Noisy xt samples and that's what we're going to use to train our classifier"}, {"start": 2342.2200000000003, "end": 2348.46, "text": " Uh on and again the reason we are doing this is because otherwise if we didn't have these noisy images if we just trained on"}, {"start": 2348.46, "end": 2351.7400000000002, "text": " the original images then the classifier wouldn't be able to"}, {"start": 2351.74, "end": 2358.8599999999997, "text": " To estimate the gradients correctly for for an arbitrary noisy image and that's why we want to train it on noisy images"}, {"start": 2359.2599999999998, "end": 2360.3799999999997, "text": " Okay"}, {"start": 2360.3799999999997, "end": 2364.14, "text": " So here now we just um instead of training on the whole batch"}, {"start": 2364.2999999999997, "end": 2369.9799999999996, "text": " This is just for for for memory optimization stuff. Uh, we're gonna kind of chunk the batch size"}, {"start": 2369.9799999999996, "end": 2373.74, "text": " That's 256 into smaller batches so-called micro batches"}, {"start": 2373.8999999999996, "end": 2380.2999999999997, "text": " Okay, and here it is and now we just call we just pass so we do a for pass through our"}, {"start": 2380.3, "end": 2384.46, "text": " Our classifier again, the the classifier is basically just a unit model"}, {"start": 2384.78, "end": 2387.5, "text": " Uh, well basically the first portion of the unit model"}, {"start": 2387.82, "end": 2391.82, "text": " Uh, and first of all, let me just show you the shapes here. I think it's gonna be four"}, {"start": 2392.38, "end": 2395.42, "text": " No, it's gonna be actually just one. Okay, so the micro batch size is one"}, {"start": 2395.5800000000004, "end": 2399.6800000000003, "text": " So we basically pass just a single image through the classifier as an optimization"}, {"start": 2401.1000000000004, "end": 2403.6000000000004, "text": " Thingy and we pass in the corresponding"}, {"start": 2404.3, "end": 2407.5800000000004, "text": " Like basically time step. So again, if this is like five"}, {"start": 2407.58, "end": 2413.5, "text": " That means this is going to be x5 and we pass that through the model and we get back to logits. Okay, let me"}, {"start": 2414.06, "end": 2416.06, "text": " uh step through there"}, {"start": 2416.7, "end": 2420.46, "text": " and here we end up in the forward pass of our"}, {"start": 2421.02, "end": 2423.02, "text": " model so that's the that's the"}, {"start": 2423.02, "end": 2427.5, "text": " That's the classifier and we do the time step embedding which is just going to embed it"}, {"start": 2427.98, "end": 2434.62, "text": " Like using the the sinusoidal embeddings and then we just pass it through the mlp. And so we ultimately end up with"}, {"start": 2434.62, "end": 2437.8199999999997, "text": " Uh something that's going to be of shape one"}, {"start": 2438.8599999999997, "end": 2440.46, "text": " 512. Yeah, okay"}, {"start": 2440.46, "end": 2446.7799999999997, "text": " So and then we just do forward pass through the input blocks through the middle blocks and finally we pass through the attention"}, {"start": 2447.18, "end": 2453.1, "text": " Uh pooling layer and that's it depending on the module some of them are going to encode the temporal"}, {"start": 2453.42, "end": 2460.14, "text": " Uh, the time step information into our features some of the other modules will not do that such as the conf2d layer"}, {"start": 2460.14, "end": 2465.98, "text": " But in any case, that's pretty much it so I can now get back to the classifier training"}, {"start": 2466.3799999999997, "end": 2469.74, "text": " Uh, we can skip all the way to the loss here. So let's"}, {"start": 2470.3799999999997, "end": 2476.14, "text": " Skip skip over and here we are. So we got some logits and now basically we just do a cross entropy"}, {"start": 2476.7, "end": 2481.3399999999997, "text": " For the label we have and that's it. That's our loss. We just do the cross entropy loss"}, {"start": 2481.9, "end": 2484.8599999999997, "text": " Which is basically just a standard classifier training loss"}, {"start": 2484.86, "end": 2490.46, "text": " Okay, and now we just collect some information such as the accuracy at one accuracy at five"}, {"start": 2490.78, "end": 2495.1800000000003, "text": " We do some logging blah blah blah. We do the mean on top of the loss"}, {"start": 2495.6600000000003, "end": 2499.02, "text": " And basically here because we have micro batches"}, {"start": 2499.6600000000003, "end": 2503.42, "text": " What they do is in the in the zeroth micro batch"}, {"start": 2503.42, "end": 2510.06, "text": " We just clear up the gradients and then we keep on accumulating the gradients for each of the micro batches"}, {"start": 2510.06, "end": 2515.82, "text": " And we don't do the update until we do we go through all of the micro batches inside of uh,"}, {"start": 2516.06, "end": 2522.14, "text": " Instead of inside of a single mini batch and that's just again, uh minor optimization detail"}, {"start": 2522.38, "end": 2527.2599999999998, "text": " But we saw everything we needed to know basically that the main thing is that we have this"}, {"start": 2527.58, "end": 2535.2599999999998, "text": " Q-samp we call this function here, which makes us train not on the original images, but instead we train on"}, {"start": 2535.9, "end": 2536.86, "text": " x"}, {"start": 2536.86, "end": 2540.3, "text": " Uh t images so noisy versions of the images. Um"}, {"start": 2541.02, "end": 2544.2200000000003, "text": " Okay, guys, that's it. Uh just for um"}, {"start": 2544.78, "end": 2548.2200000000003, "text": " being detailed here, I want to show you the part with merging the"}, {"start": 2548.94, "end": 2549.9, "text": " um"}, {"start": 2549.9, "end": 2552.6200000000003, "text": " The temporal information I want to show you that part here"}, {"start": 2552.6200000000003, "end": 2560.1400000000003, "text": " So res block i'm going to enter there and i'm going to set i'm going to set a breakpoint inside of the forward function here"}, {"start": 2560.7000000000003, "end": 2564.38, "text": " Uh, just so that I can show you how we merge the information from the embedding here"}, {"start": 2564.38, "end": 2566.86, "text": " Okay, so i'm going to set a breakpoint here"}, {"start": 2567.9, "end": 2572.86, "text": " And let's now step let's do another pass through another micro batch and here we are"}, {"start": 2573.58, "end": 2576.54, "text": " So we have the embedding and we are gonna hit"}, {"start": 2577.26, "end": 2583.9, "text": " The res block at one point of time and here we are. So here is the um part where we merge the temporal information"}, {"start": 2583.9, "end": 2585.9, "text": " I want to show you this once more"}, {"start": 2585.9, "end": 2589.82, "text": " Again, it's fairly simple. We just do some additional processing. So let's see what these"}, {"start": 2589.82, "end": 2597.1000000000004, "text": " In m layers are so again, it's just like a like a basically a linear layer and an activation function"}, {"start": 2597.5800000000004, "end": 2599.42, "text": " Nothing fancy there"}, {"start": 2599.42, "end": 2601.42, "text": " And we end up with embedding"}, {"start": 2601.7400000000002, "end": 2607.02, "text": " Here of the shape 1 to 56 and now we just as you can see here"}, {"start": 2607.02, "end": 2611.84, "text": " We start adding dummy dimensions such that we can merge this temporal information"}, {"start": 2612.46, "end": 2614.46, "text": " into the"}, {"start": 2614.46, "end": 2621.98, "text": " Um image features, okay, so we can see here because this is set to true we enter this branch here. We have some layers"}, {"start": 2622.7, "end": 2627.58, "text": " Here and we split them into two parts. The first part is the normalization layer and then the rest"}, {"start": 2627.9, "end": 2632.62, "text": " And you can see this out there. I think it's just like a sequential. Yeah, you can see here how it's constructed"}, {"start": 2633.1, "end": 2637.9, "text": " Um, not that important. So the important part is here. We take our embedding out"}, {"start": 2638.46, "end": 2642.38, "text": " So this vector and we split it into two parts one is scale one is shift"}, {"start": 2642.38, "end": 2647.82, "text": " And I guess it's going to be 128 is going to be the shape here"}, {"start": 2648.46, "end": 2651.42, "text": " So one 128 and we added the dummy dimensions"}, {"start": 2651.42, "end": 2655.1800000000003, "text": " And so now if we do this we're going to have some broadcasting going on"}, {"start": 2655.1800000000003, "end": 2661.34, "text": " And we are going to going to merge them with the image features and this image features are going to be of shape"}, {"start": 2661.82, "end": 2665.82, "text": " As you can see here 128 128 128 and that's it guys"}, {"start": 2665.82, "end": 2669.02, "text": " I'm going to briefly show you how this looks like inside of my one note"}, {"start": 2669.02, "end": 2676.14, "text": " So again fairly simple thing. We have so we have something that came out of the"}, {"start": 2677.1, "end": 2682.22, "text": " So we have a volume that came out of the unit model. That's going to look like something like this"}, {"start": 2682.78, "end": 2684.78, "text": " So here is the our volume"}, {"start": 2685.34, "end": 2687.66, "text": " So those are the image features"}, {"start": 2688.78, "end": 2690.78, "text": " From the latent space of our encoder"}, {"start": 2691.5, "end": 2694.72, "text": " And we saw that that's going to be like this is 128"}, {"start": 2694.72, "end": 2702.0, "text": " Number of channels and we saw that this is also 128 so the height and the width all of this is 128"}, {"start": 2702.64, "end": 2708.0, "text": " Whereas on the other hand we just have this is our vector. This is our temporal embedding"}, {"start": 2709.7599999999998, "end": 2713.3599999999997, "text": " So this is how it looks like it's only going to be 128 here"}, {"start": 2715.6, "end": 2720.56, "text": " But it's only one one here and then what broadcasting is going to do"}, {"start": 2720.56, "end": 2725.68, "text": " Is basically just copy paste this multiple times so that we end up with something like this"}, {"start": 2726.32, "end": 2730.16, "text": " So that we end up with multiple copies here here here"}, {"start": 2731.04, "end": 2736.56, "text": " Etc so we're just going to copy paste it here and then after we do that we can freely add it up"}, {"start": 2736.56, "end": 2742.0, "text": " So we can freely do addition on top of the image features and that's how we incorporate the temporal information"}, {"start": 2743.12, "end": 2745.12, "text": " Cool. Let's go back here"}, {"start": 2745.12, "end": 2750.7999999999997, "text": " I'm going to stop the execution of the classifier train script. We saw how to train the classifier"}, {"start": 2751.12, "end": 2756.88, "text": " Do let me know whether you found this explanation useful whether I can improve something any feedback is always appreciated"}, {"start": 2757.44, "end": 2760.16, "text": " And also if you if you like this video share it out with your friends"}, {"start": 2761.6, "end": 2765.52, "text": " Let's now focus on the sampling function. So I have prepared"}, {"start": 2766.16, "end": 2773.2799999999997, "text": " Some arguments again. I just took the arguments from the read me and I created I just kind of extracted them here"}, {"start": 2773.28, "end": 2776.32, "text": " So here are the arguments i'm using but you can find all of that"}, {"start": 2776.32, "end": 2780.0800000000004, "text": " You can find all of that basically in the read me file itself"}, {"start": 2780.0800000000004, "end": 2786.2400000000002, "text": " So i'm going to go back here and i'm going to now focus on the sampling function. Okay guys, let's run this script"}, {"start": 2787.0400000000004, "end": 2789.0400000000004, "text": " and"}, {"start": 2789.0400000000004, "end": 2793.0400000000004, "text": " Again, i'm going to scheme a lot of the i'm going to skip explaining a lot of details"}, {"start": 2793.0400000000004, "end": 2796.1600000000003, "text": " Which are irrelevant to understanding the actual"}, {"start": 2796.96, "end": 2799.6800000000003, "text": " Well, uh got in this particular case"}, {"start": 2799.68, "end": 2804.48, "text": " We want to see how the classifier guidance works like and that's what i'm going to focus on"}, {"start": 2805.2, "end": 2809.2799999999997, "text": " So what this function is going to return is the model which is going to be the unit model"}, {"start": 2809.2799999999997, "end": 2812.48, "text": " That's going to predict the epsilon. So the the noise"}, {"start": 2813.2799999999997, "end": 2818.72, "text": " And diffusion just contains bunch of those constants necessary to do the diffusion process"}, {"start": 2818.72, "end": 2821.12, "text": " So i'm going to skip over all of that"}, {"start": 2821.7599999999998, "end": 2824.96, "text": " We're just going to load a certain checkpoint"}, {"start": 2824.96, "end": 2829.28, "text": " We like as I said, I have a model path here"}, {"start": 2829.76, "end": 2834.56, "text": " So, let me just see whether I can show you yeah, so I went ahead and downloaded the model"}, {"start": 2834.56, "end": 2839.52, "text": " Into this models directory inside of my root directory and"}, {"start": 2839.52, "end": 2843.84, "text": " Everything is already explained in the read me so you don't have to worry about it"}, {"start": 2843.84, "end": 2846.88, "text": " So we just push the model to the gpu"}, {"start": 2846.88, "end": 2848.88, "text": " We convert it into fp16"}, {"start": 2849.68, "end": 2851.68, "text": " Again, just some"}, {"start": 2851.68, "end": 2854.72, "text": " Optimization details. I'm not going to focus on that"}, {"start": 2854.72, "end": 2857.68, "text": " We create the classifier. So that's again our encoder"}, {"start": 2857.68, "end": 2860.7999999999997, "text": " The the the half portion of the unit"}, {"start": 2861.7599999999998, "end": 2863.8399999999997, "text": " You can see here and we already"}, {"start": 2865.44, "end": 2869.44, "text": " Schemed through that code previously in the in the training script"}, {"start": 2869.44, "end": 2872.72, "text": " So i'm going to just ignore all of that and uh, whoops"}, {"start": 2872.72, "end": 2876.0, "text": " Well, obviously not so i'll have to just ask"}, {"start": 2876.64, "end": 2878.24, "text": " Let me just kind of do the"}, {"start": 2878.24, "end": 2883.4399999999996, "text": " Disable all the points all the breakpoints and then i'm going to click f5 and here we are"}, {"start": 2883.4399999999996, "end": 2886.4799999999996, "text": " Now i'm going to enable all of the breakpoints again"}, {"start": 2887.3599999999997, "end": 2889.3599999999997, "text": " And we just load the weights"}, {"start": 2889.3599999999997, "end": 2892.16, "text": " We load the state dictionary from the classifier path"}, {"start": 2892.16, "end": 2896.16, "text": " So I went ahead and downloaded this this model again"}, {"start": 2896.16, "end": 2898.8799999999997, "text": " And I put it inside of the models directory. That's it"}, {"start": 2899.4399999999996, "end": 2900.64, "text": " Nothing smart there"}, {"start": 2900.64, "end": 2904.4799999999996, "text": " We push the model the classifier on the to the gpu as well"}, {"start": 2904.48, "end": 2908.4, "text": " And we set it inside into the eval mode"}, {"start": 2908.4, "end": 2911.68, "text": " That's it. Now this is going to be the most important part"}, {"start": 2911.68, "end": 2916.2400000000002, "text": " This is where the gradients are calculated for the classifier"}, {"start": 2916.2400000000002, "end": 2920.08, "text": " I'm going to analyze that a bit later once we hit once we enter that function"}, {"start": 2920.08, "end": 2925.12, "text": " So here is where it's defined later. We're going to see how it basically works"}, {"start": 2925.12, "end": 2928.32, "text": " So that's it. We just define this function as well"}, {"start": 2928.32, "end": 2930.4, "text": " The model function and"}, {"start": 2930.4, "end": 2937.6, "text": " Now we're going to do a bunch of iterations until we create the number of samples that we can specify"}, {"start": 2938.48, "end": 2943.04, "text": " And you can see here I specified 100 but that's just arbitrary numbers so not that important"}, {"start": 2943.04, "end": 2949.84, "text": " Okay, so we create random numbers so random classes and depending on the batch size"}, {"start": 2949.84, "end": 2952.1600000000003, "text": " We have like four random classes being generated here"}, {"start": 2952.1600000000003, "end": 2953.76, "text": " So let me show you what those are"}, {"start": 2953.76, "end": 2957.84, "text": " If I go to debug console it's going to be a bunch of random classes"}, {"start": 2957.84, "end": 2963.84, "text": " If I go to debug console if I print that we can see we have four random classes"}, {"start": 2964.48, "end": 2968.0, "text": " And we don't I don't know directly what these correspond to"}, {"start": 2968.0, "end": 2972.56, "text": " But like let's say these are just classes 33 918 etc"}, {"start": 2972.56, "end": 2976.1600000000003, "text": " Okay, we don't care about the actual semantics behind the numbers"}, {"start": 2976.1600000000003, "end": 2980.2400000000002, "text": " We are going to store those the class information inside of this model"}, {"start": 2981.04, "end": 2984.4, "text": " Like keyword arguments dictionary"}, {"start": 2984.4, "end": 2990.7200000000003, "text": " So I'm gonna go there and because we're not using this ddim again ddim is just just it's just like"}, {"start": 2990.7200000000003, "end": 2997.2000000000003, "text": " This optimize it like this sampling method whereby if you have less than 50 diffusion steps"}, {"start": 2997.84, "end": 3004.32, "text": " They showed in that paper that they achieve better results compared to the original ddpm paper"}, {"start": 3004.32, "end": 3010.32, "text": " So only in the mode where you only sample for less than 50 steps roughly then ddim makes sense"}, {"start": 3010.32, "end": 3014.96, "text": " But here I think we have like I think we set like thousand steps so we're going to use the"}, {"start": 3014.96, "end": 3018.0800000000004, "text": " p sample loop instead of the ddim sample loop"}, {"start": 3018.0800000000004, "end": 3021.76, "text": " I'm not going to focus on the ddim part because there is a lot of formulas"}, {"start": 3021.76, "end": 3025.92, "text": " It's hard to explain the logic but like on the high level just treating it as a black box"}, {"start": 3026.48, "end": 3030.88, "text": " When you would want to use it is when you have less than 50 diffusion steps"}, {"start": 3030.88, "end": 3031.76, "text": " Okay, that's it"}, {"start": 3032.56, "end": 3034.6400000000003, "text": " Here is where the whole magic happens"}, {"start": 3034.6400000000003, "end": 3039.44, "text": " After that there's just some normalization going on permutation blah blah blah"}, {"start": 3039.44, "end": 3047.04, "text": " And we just collect the images here and finally yeah basically just a bunch of boilerplate code"}, {"start": 3047.04, "end": 3051.6, "text": " This is where the main logic is in so we're going to step inside of here"}, {"start": 3052.8, "end": 3060.16, "text": " We basically specify here the target shape so our images are going to be 64 by 64"}, {"start": 3060.16, "end": 3064.4, "text": " Later on they have the up sampling modules which we're also going to ignore"}, {"start": 3064.4, "end": 3066.0, "text": " But let's focus on this part here"}, {"start": 3066.0, "end": 3073.28, "text": " Okay so we pass in the model keyword arguments that have the class information"}, {"start": 3073.92, "end": 3078.88, "text": " We pass these special functions so the cond fn which we define here is going to calculate the"}, {"start": 3078.88, "end": 3082.8, "text": " gradients we're going to see how that comes into the play in a second"}, {"start": 3082.8, "end": 3088.4, "text": " So here is the p sample loop what it does is we're just going to keep on generating samples"}, {"start": 3088.96, "end": 3092.72, "text": " I'm just going to directly jump inside of this function"}, {"start": 3092.72, "end": 3096.72, "text": " Because everything else is just like basically boilerplate code"}, {"start": 3098.3999999999996, "end": 3103.2, "text": " So here we generate the random noise image"}, {"start": 3103.2, "end": 3107.9199999999996, "text": " Okay so we have the desired shape and the desired shape is as you can see here"}, {"start": 3108.48, "end": 3113.12, "text": " Four because we have four images in the batch and here we have the resolution of our target image"}, {"start": 3113.12, "end": 3118.16, "text": " And we just generate we just sample from a normal distribution here"}, {"start": 3119.4399999999996, "end": 3122.08, "text": " And then we just generate indices"}, {"start": 3122.08, "end": 3125.36, "text": " So depending on the number of time steps so we have 250 here"}, {"start": 3126.0, "end": 3130.56, "text": " We just generate these indices and remember that's because we want to start from the"}, {"start": 3130.56, "end": 3136.72, "text": " Completely noisy image so from the index 249 and we're going to work our way all the way to zero"}, {"start": 3136.72, "end": 3141.7599999999998, "text": " Where we're going to end up with completely denoised images from our underlying data"}, {"start": 3141.7599999999998, "end": 3145.84, "text": " Distribution which we learned during the training but this is now the sampling part"}, {"start": 3145.84, "end": 3154.4, "text": " Okay so indices let's see what t is it should be 249 as I said"}, {"start": 3154.4, "end": 3159.04, "text": " And here is where the whole magic happens in this p sample step"}, {"start": 3159.04, "end": 3164.1600000000003, "text": " So after 250 steps we are going to end up with our final image"}, {"start": 3164.1600000000003, "end": 3167.6000000000004, "text": " And that's one of the drawbacks of current diffusion models"}, {"start": 3167.6000000000004, "end": 3172.7200000000003, "text": " We need much more compute compared to like say GANs where you just need a single four pass"}, {"start": 3172.72, "end": 3177.68, "text": " And you end up with a like a very realistic image very crispy image"}, {"start": 3177.68, "end": 3180.48, "text": " Here we have much more we need much more compute okay"}, {"start": 3182.08, "end": 3187.9199999999996, "text": " So let's step inside of the p sample function and the main function here is going to be this p mean"}, {"start": 3187.9199999999996, "end": 3193.8399999999997, "text": " Variance that's just basically our reverse process that we learned"}, {"start": 3194.72, "end": 3200.48, "text": " After we calculate the out variable here we're going to generate noise again"}, {"start": 3200.48, "end": 3203.68, "text": " We're going to do the conditioning I'm going to get here so this is going to be important part here"}, {"start": 3203.68, "end": 3208.16, "text": " This is where we do the conditioning this is where we do the where we shift the mean"}, {"start": 3209.2, "end": 3217.44, "text": " Via the sigma times gradients expression and then we just form the image here okay"}, {"start": 3217.44, "end": 3222.8, "text": " So the this is the this is going to contain the x t minus one"}, {"start": 3224.16, "end": 3229.28, "text": " If the if we condition on the x t or alternatively if we condition off on x t plus one"}, {"start": 3229.28, "end": 3234.4, "text": " This is going to be x t in any case it's just like a single step closer to the final image"}, {"start": 3235.84, "end": 3241.84, "text": " Okay let's jump into the p mean variance here it is this is what we calculate as I said our"}, {"start": 3241.84, "end": 3247.1200000000003, "text": " Reverse process and let's kind of quickly scheme over over this part so I'm going to"}, {"start": 3247.1200000000003, "end": 3254.96, "text": " Set a break point here here we are we enter here and now we're going to pass what we're going to"}, {"start": 3254.96, "end": 3260.16, "text": " Pass a couple things first we're going to pass the the the noise we just sampled so that's going to"}, {"start": 3260.16, "end": 3267.92, "text": " Be I guess uh four images right so four three sixty four sixty four we're going to pass the time steps"}, {"start": 3267.92, "end": 3273.36, "text": " So those are going to be the two forty nine two forty nine two forty nine for each image in the batch"}, {"start": 3273.92, "end": 3278.4, "text": " And we're going to additionally pass the class information because if you recall"}, {"start": 3278.4, "end": 3282.64, "text": " From the paper explanation we are not only going to use classifier to guide"}, {"start": 3282.64, "end": 3288.24, "text": " We are also going to use class information to condition the model so those two do the same"}, {"start": 3288.24, "end": 3296.0, "text": " Thing but are kind of complementary so don't be confused by that okay let's step there we hit the"}, {"start": 3296.64, "end": 3303.3599999999997, "text": " Basically the this model fn that we saw previously and because class conditioning is set to true"}, {"start": 3303.3599999999997, "end": 3310.7999999999997, "text": " We're going to pass x so the noise images the time steps and the conditioning information all of that"}, {"start": 3310.8, "end": 3316.4, "text": " Let's now quickly step into the unit model into the forward pass and see how the class information"}, {"start": 3316.4, "end": 3323.2000000000003, "text": " Is incorporated into uh with the image features and hint hint it's literally the same thing as with"}, {"start": 3323.2000000000003, "end": 3327.76, "text": " The temporal information pretty much we're going to see that in a second so here it is we have the"}, {"start": 3327.76, "end": 3334.6400000000003, "text": " Time steps we're going to embed those so sinusoidal embeddings then some learned stuff here and we end"}, {"start": 3334.64, "end": 3341.92, "text": " Up with a vector that's of shape i guess let's see what it is um it's going to be four seven sixty"}, {"start": 3341.92, "end": 3348.24, "text": " Eight because we're dealing with uh batches with four images okay and now because we have the the"}, {"start": 3348.24, "end": 3354.64, "text": " Classes set here the only thing we're going to do with our classes so here here are classes so we"}, {"start": 3354.64, "end": 3362.56, "text": " Have four classes here we're going to um embed them using this label embed oh i hate these warnings"}, {"start": 3362.56, "end": 3368.72, "text": " I should have toggled them off there is some config uh line that you can set to lon jason if you if you"}, {"start": 3368.72, "end": 3375.6, "text": " Don't want to see those but yeah i forgot to do that uh so let's see what this this label embed"}, {"start": 3375.6, "end": 3382.72, "text": " Does okay so you can see here is just a simple embedding layer and it's going to specify uh it's"}, {"start": 3382.72, "end": 3388.56, "text": " Going to have some number of dimensions for each of the classes nothing fancy there you can literally"}, {"start": 3388.56, "end": 3395.68, "text": " Treat that as um let's see what it gives us basically uh so if we do this and i do shape"}, {"start": 3396.24, "end": 3402.64, "text": " Yeah you can see it's going to transform uh our four scalars into four vectors of dimensionality"}, {"start": 3402.64, "end": 3409.7599999999998, "text": " 768 and then we're just going to simply add that on top of the temporal embedding and then everything"}, {"start": 3409.7599999999998, "end": 3415.84, "text": " Else proceeds as as usual we're just going to keep on incorporating that information so both the class"}, {"start": 3415.84, "end": 3420.7200000000003, "text": " Information as well as the temporal information directly merge them with the image features"}, {"start": 3420.7200000000003, "end": 3426.8, "text": " Using the logic i showed you with the scaling and the offset we saw in the res block and that's it"}, {"start": 3427.6000000000004, "end": 3434.2400000000002, "text": " I'm going to skip over all of this uh we don't care about it and now let's continue on so we are"}, {"start": 3434.2400000000002, "end": 3441.36, "text": " Here and we ended up with the model outputs if sigma is learned this should be four six"}, {"start": 3441.36, "end": 3447.52, "text": " Uh something right so four six because six because we all we predict the epsilon so that's the noise"}, {"start": 3447.52, "end": 3454.08, "text": " And we also predict the sigma so that's the the covariance um and now let's continue here"}, {"start": 3454.08, "end": 3457.6, "text": " We saw all of this logic in the previous video so i'm going to kind of skim through this"}, {"start": 3457.6, "end": 3463.6, "text": " We split the model outputs into two parts so again the epsilon and the variance and then"}, {"start": 3464.1600000000003, "end": 3468.8, "text": " Well actually it's not the variance it's it's the v vector let me show you in a second formula"}, {"start": 3468.8, "end": 3475.28, "text": " So again if you recall from the previous video this is the formula we used so we are predicting the v's here"}, {"start": 3475.28, "end": 3480.8, "text": " And then we're going to form the variance by doing this expression here so this is the expression calculation we just saw"}, {"start": 3480.8, "end": 3486.32, "text": " So we calculate all of those we get the variance ultimately here and now we need to predict the mean"}, {"start": 3486.32, "end": 3489.84, "text": " So here is how we predict the mean i'm going to skim through all of this here"}, {"start": 3490.8, "end": 3494.32, "text": " We just need to first predict the x start so that's the x0"}, {"start": 3494.32, "end": 3501.36, "text": " And then we use this posterior mean variance to get the model mean so this is all of this is same as in the previous video pretty much"}, {"start": 3501.36, "end": 3507.36, "text": " So we got the mean we have the variance and we finally return all of those variables so the mean"}, {"start": 3508.48, "end": 3513.6000000000004, "text": " Uh variance model log variance etc etc by the way let me just quickly show you what those are"}, {"start": 3514.0800000000004, "end": 3518.2400000000002, "text": " So here are we in the one note explanation I showed you in the beginning of the video"}, {"start": 3518.48, "end": 3521.36, "text": " So this is what we currently generated we generated the"}, {"start": 3521.36, "end": 3528.08, "text": " Um these two so we have the mean and we have the sigma and now we need to do this shift part"}, {"start": 3528.2400000000002, "end": 3532.4, "text": " So we need to do the shift part and then we can sample our our our image and that's it"}, {"start": 3532.96, "end": 3537.52, "text": " Fairly simple. Okay. Let me go back here. I'm going to return all of these variables"}, {"start": 3538.08, "end": 3540.08, "text": " I'm going to return all of those"}, {"start": 3540.2400000000002, "end": 3544.26, "text": " We generate we sample from a normal distribution"}, {"start": 3545.36, "end": 3547.36, "text": " We have this masking stuff going on"}, {"start": 3547.36, "end": 3554.96, "text": " Uh that um because we have if you recall we have a discrepancy for t equals zero time step versus all of the other steps"}, {"start": 3555.28, "end": 3561.36, "text": " Uh things look a bit different, but like I guess that's just a minor detail again. This is where the whole magic happens"}, {"start": 3561.52, "end": 3567.04, "text": " This is where the shifting of the mean happens using the sigma using the gradients of our classifier"}, {"start": 3567.52, "end": 3570.8, "text": " Uh, and also like just a note, uh, there is a bug here"}, {"start": 3570.96, "end": 3575.52, "text": " I submitted an issue and the authors of open from opening I basically confirmed that's indeed"}, {"start": 3575.52, "end": 3578.56, "text": " Uh an issue i'm gonna um explain you a bit later"}, {"start": 3578.56, "end": 3582.56, "text": " Uh what that exactly is but the funny thing is it did not even matter"}, {"start": 3582.56, "end": 3585.52, "text": " It's like that's the thing with uh training machine learning models"}, {"start": 3585.52, "end": 3589.2, "text": " Sometimes you have a bug but the models are forgiving and you will just get"}, {"start": 3589.68, "end": 3594.8, "text": " Something working maybe a bit suboptimal, but still there is no way you can figure out there is a bug"}, {"start": 3594.8, "end": 3601.12, "text": " It's not like with software systems where something when you when you when you like a basically develop a traditional software system"}, {"start": 3601.12, "end": 3602.48, "text": " If something does not work"}, {"start": 3602.48, "end": 3609.2, "text": " You will know that something will crash and the program will not work. So yeah ml ml is a bit different. Okay, so"}, {"start": 3610.16, "end": 3615.2, "text": " Here we form a new mean and a new mean is formed by adding"}, {"start": 3615.68, "end": 3622.08, "text": " On top of our old mean we just add the sigma times the gradient. So again, that's that's that's the whole idea"}, {"start": 3622.4, "end": 3630.2400000000002, "text": " So here again, we just have uh, we pass the um, we have the conditional con function which we defined in the main script"}, {"start": 3630.24, "end": 3632.24, "text": " we pass our"}, {"start": 3632.24, "end": 3637.7599999999998, "text": " Uh means the mean and the and the covariance we just calculated uh here above"}, {"start": 3638.24, "end": 3643.68, "text": " So this basically contains the x t and this is the x t plus one and this is the time step in any case"}, {"start": 3643.7599999999998, "end": 3646.3999999999996, "text": " Let's step inside of this function and let's see what's going on"}, {"start": 3646.56, "end": 3646.8799999999997, "text": " Okay"}, {"start": 3646.8799999999997, "end": 3652.08, "text": " So first we calculate the gradient and then as we recall from the equation"}, {"start": 3652.3999999999996, "end": 3655.8399999999997, "text": " we just add on our new on our old mean we add the"}, {"start": 3655.84, "end": 3662.0, "text": " The variance we calculated times the gradient and that's it. So that's that's easy. Let's now see that this part here"}, {"start": 3662.08, "end": 3664.4, "text": " Let's step into this function here. Okay, so"}, {"start": 3665.2000000000003, "end": 3667.2000000000003, "text": " First thing we do is we take our"}, {"start": 3668.32, "end": 3669.76, "text": " noisy image"}, {"start": 3669.76, "end": 3671.76, "text": " Uh, we detach it from the computational graph"}, {"start": 3671.76, "end": 3677.76, "text": " We said it requires graph to true so that we can calculate the gradients for that x because that's what we ultimately care about"}, {"start": 3677.76, "end": 3681.28, "text": " right, we want to have the gradients of x with respect to the"}, {"start": 3682.2400000000002, "end": 3683.84, "text": " log"}, {"start": 3683.84, "end": 3688.08, "text": " Probability of of our target class from the classifier. Okay, so"}, {"start": 3688.7200000000003, "end": 3691.28, "text": " We pass it through the classifier. We get the logits"}, {"start": 3692.48, "end": 3698.4, "text": " And unfortunately, I stepped inside of the classifier. We don't really care about it. Let's just kind of um"}, {"start": 3699.04, "end": 3704.98, "text": " Continue here. We don't care about it. So we have logits now we just form the log softmax"}, {"start": 3705.84, "end": 3707.76, "text": " So we have the log props"}, {"start": 3707.76, "end": 3710.88, "text": " So first, let's see what what the dimensionality of log props is"}, {"start": 3710.88, "end": 3717.2000000000003, "text": " So if we do shape here we have four because if we have four images and we have thousand lodges because we're dealing with image"}, {"start": 3717.2000000000003, "end": 3724.48, "text": " And and so what we do here is we basically extract the target. So why is again remember? Why is the the target?"}, {"start": 3725.04, "end": 3732.1600000000003, "text": " So those are the ground truth classes. We use those to take only those ledges logits. So we have the selected ones"}, {"start": 3732.1600000000003, "end": 3734.8, "text": " so now we if I print this these are the"}, {"start": 3734.8, "end": 3741.1200000000003, "text": " The um, uh target classes. So so these are the logits we need to maximize if we want to have"}, {"start": 3741.84, "end": 3744.6400000000003, "text": " An image from that particular class if that makes sense"}, {"start": 3744.96, "end": 3748.32, "text": " And so what we do here was just and this is a part that kind of confused me"}, {"start": 3748.5600000000004, "end": 3752.1600000000003, "text": " We sum up all of these uh different, um, like, um"}, {"start": 3753.04, "end": 3756.9, "text": " Logits, sorry not logits the log props the log probabilities"}, {"start": 3757.28, "end": 3762.8, "text": " And then we find the gradient with respect to the input. So the input is again a batch"}, {"start": 3762.8, "end": 3769.28, "text": " So it's going to be four three sixty four sixty four and we just multiply so this is the classifier scale I mentioned"}, {"start": 3769.28, "end": 3773.1200000000003, "text": " So that's denoted as s in the paper. So let me show you that in a second here"}, {"start": 3773.44, "end": 3777.6000000000004, "text": " Let's go back to the paper and let's find the uh, that part"}, {"start": 3778.2400000000002, "end": 3783.04, "text": " Somewhere here. So I think it's down here. Okay, so here's here is the part"}, {"start": 3783.04, "end": 3785.04, "text": " We're just computing we have the s"}, {"start": 3785.76, "end": 3791.92, "text": " Which is on the right hand side in the code and we have the gradients here. So again, let's go back to the"}, {"start": 3791.92, "end": 3799.36, "text": " Uh code here. So this is the s and this is the gradient we are we are computing and the reason this works is because um"}, {"start": 3799.92, "end": 3803.92, "text": " If you do a sum you'll end up with so our loss here is going to be"}, {"start": 3804.56, "end": 3810.96, "text": " L1 so that's like l1 is just my uh, shorthand notation for log probability for the first image"}, {"start": 3811.36, "end": 3816.64, "text": " So for the for the log probability for the first image plus l2 plus l3 plus l4"}, {"start": 3816.64, "end": 3822.96, "text": " And if we do a gradient with respect to x because x1 so the first image from this batch only"}, {"start": 3823.2, "end": 3830.16, "text": " Influences this particular loss. Okay x2. So that's the second image from from the batch only influences the second one"}, {"start": 3830.64, "end": 3839.2799999999997, "text": " Okay, so because they only influence the log probes that correspond to to their index and because of that if we do gradient here"}, {"start": 3839.92, "end": 3845.8399999999997, "text": " Uh, each of the images here will have only the so basically the first image"}, {"start": 3845.84, "end": 3847.84, "text": " will have um"}, {"start": 3847.84, "end": 3849.1200000000003, "text": " l1"}, {"start": 3849.1200000000003, "end": 3850.2400000000002, "text": " dx"}, {"start": 3850.2400000000002, "end": 3855.44, "text": " The second image will be uh l2 dx etc. Etc. Okay"}, {"start": 3855.44, "end": 3861.84, "text": " So basically we'll return a batch of gradients where each of the uh images inside of that batch"}, {"start": 3862.2400000000002, "end": 3867.52, "text": " Is basically gradients with respect to the corresponding loss. So if I go back here"}, {"start": 3868.56, "end": 3873.76, "text": " We basically end up with a gradient here and the gradient is going to be of the same shape as the images"}, {"start": 3873.76, "end": 3878.0, "text": " So here 436464 and that's pretty much it"}, {"start": 3878.1600000000003, "end": 3884.5600000000004, "text": " Okay, so now we just do the the the logic we saw in the paper and guys that's pretty much it"}, {"start": 3884.96, "end": 3886.96, "text": " We're done. We now form"}, {"start": 3887.2000000000003, "end": 3889.92, "text": " Uh using the new mean and the log variance"}, {"start": 3890.2400000000002, "end": 3895.0400000000004, "text": " We just sample by doing this expression by just adding the log variance times the noise"}, {"start": 3895.36, "end": 3899.6800000000003, "text": " And that's how we get the uh next image in the reverse process"}, {"start": 3899.68, "end": 3903.52, "text": " Now this just repeats on and on but this was the main this was the gist of the paper"}, {"start": 3903.8399999999997, "end": 3907.2799999999997, "text": " So where is the bug I mentioned? I'm going to show you the issue in a second"}, {"start": 3907.2799999999997, "end": 3911.2799999999997, "text": " But like basically what happens here is they passed xt plus one and t plus one"}, {"start": 3911.6, "end": 3915.2, "text": " So these are the uh, they should have passed the out mean"}, {"start": 3915.68, "end": 3920.96, "text": " Instead of xt plus one because this is xt and this is xt plus one and so"}, {"start": 3921.44, "end": 3925.52, "text": " It might be a bit subtle. So let's let's let's try and see whether I can explain it"}, {"start": 3925.52, "end": 3932.08, "text": " So as you can see here they pass the the the the the xt plus one. So if I go back here"}, {"start": 3932.64, "end": 3938.32, "text": " Uh, you can see that then they call this conditional function with xt plus one and t plus one"}, {"start": 3938.88, "end": 3940.64, "text": " So we end up"}, {"start": 3940.64, "end": 3943.12, "text": " using xt plus one t plus one here"}, {"start": 3943.68, "end": 3945.68, "text": " Uh, so that means"}, {"start": 3945.68, "end": 3952.8, "text": " That we calculate we pass xt plus one t plus one here. We get the logits, but if you recall from the actual paper"}, {"start": 3952.8, "end": 3958.7200000000003, "text": " Uh, we do not need to we we need to condition on on x. Let me just find the the expression here"}, {"start": 3959.44, "end": 3960.96, "text": " blah blah blah"}, {"start": 3960.96, "end": 3962.0800000000004, "text": " so"}, {"start": 3962.0800000000004, "end": 3964.0, "text": " You can see here"}, {"start": 3964.0, "end": 3969.1200000000003, "text": " That we want to condition on xt and not on xt plus one. Otherwise the gradients we get here"}, {"start": 3970.0800000000004, "end": 3975.84, "text": " Are again from the xt plus one not from the xt and all of that kind of messes up the logic"}, {"start": 3975.84, "end": 3977.84, "text": " I just explained it here. So mu"}, {"start": 3977.84, "end": 3984.2400000000002, "text": " Mu is from xt and we want to use that same mu and pass it inside of the classifier and not the previous image"}, {"start": 3984.8, "end": 3992.88, "text": " And despite all of this, it's very subtle. I don't even know how I noticed it. I was fairly I guess pedantic stepping through this code"}, {"start": 3993.84, "end": 3995.04, "text": " Uh"}, {"start": 3995.04, "end": 3997.6000000000004, "text": " They they get the same results pretty much. They didn't even notice it"}, {"start": 3997.6000000000004, "end": 4001.76, "text": " They noticed it post-hoc after the paper was published or something. I'm going to show you the issue in a second"}, {"start": 4001.76, "end": 4003.76, "text": " So here is the issue"}, {"start": 4003.76, "end": 4006.4, "text": " I opened up I think a couple of days ago"}, {"start": 4006.4, "end": 4008.4, "text": " um"}, {"start": 4008.4, "end": 4012.48, "text": " So I think it's closed already so bugs blah blah blah. So here's my issue"}, {"start": 4012.48, "end": 4014.48, "text": " So I created an issue here as you can see here"}, {"start": 4014.48, "end": 4016.0, "text": " I think I found two bugs"}, {"start": 4016.0, "end": 4022.8, "text": " So the first one we shouldn't shouldn't we pass the out of mean so the x of t instead of x of t plus one here"}, {"start": 4022.8, "end": 4026.48, "text": " Uh, and similarly the the smaller time step instead of t"}, {"start": 4026.48, "end": 4032.48, "text": " Uh, and then I linked to the particular line of code which is the same line of code I showed to you guys"}, {"start": 4032.48, "end": 4034.48, "text": " So that's here"}, {"start": 4034.48, "end": 4043.68, "text": " And uh, basically, um, one of the authors replied yes, this is indeed a slight bug we which we noticed shortly after releasing our work"}, {"start": 4043.68, "end": 4050.0, "text": " However, we did try ablating using the correct formula and found that it did not noticeably change results again"}, {"start": 4050.0, "end": 4052.8, "text": " That's the tricky part about training machine learning models"}, {"start": 4052.8, "end": 4057.68, "text": " And then the second part I was kind of confused with the summation of of of those losses"}, {"start": 4057.68, "end": 4062.08, "text": " But it turned out it's quite. Um, it makes a lot of sense. Uh, it was just a minor"}, {"start": 4062.08, "end": 4068.4, "text": " Uh, mistake on my on my side anyway in any case I can kind of link this issue if you want to dig deeper into it"}, {"start": 4068.4, "end": 4070.4, "text": " Uh, I'll have it down in the video description"}, {"start": 4070.4, "end": 4078.48, "text": " So guys, that's pretty much it. Um, I showed you how the how the paper what the paper introduced the two main things are"}, {"start": 4078.48, "end": 4084.08, "text": " Improving the unit architecture. The second thing is uh, adding this classifier guidance technique"}, {"start": 4084.08, "end": 4086.4, "text": " And then I showed you how to train the classifier"}, {"start": 4086.4, "end": 4090.4, "text": " Uh in in this classifier transcript we saw how to train the classifier"}, {"start": 4090.4, "end": 4097.76, "text": " This classifier transcript we saw how to train it on on basically noisy images and then use that very same classifier"}, {"start": 4098.88, "end": 4104.88, "text": " Inside of this classifier sample script to basically shift the mean and thus"}, {"start": 4106.56, "end": 4111.36, "text": " Like sample images from the class conditioned diffusion model"}, {"start": 4112.4, "end": 4114.32, "text": " There is a lot of details"}, {"start": 4114.32, "end": 4117.52, "text": " Hopefully you picked up something useful from this video if you did"}, {"start": 4117.52, "end": 4122.580000000001, "text": " Uh, consider subscribing also share the video out and until next time. Bye. Bye"}]