Stable Diffusion: High-Resolution Image Synthesis with Latent Diffusion Models | ML Coding Series
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

If you want to understand how stable diffusion exactly works behind the scenes this video is for you. I do a deep dive into the code behind Stable Diffusion explaining:
1. First stage autoencoder training (autoencoder with KL regularization)
2. Latent Diffusion Model training (UNet + conditioning model)
3. Sampling using PLMS scheduler

Stable diffusion directly builds upon the "High-Resolution Image Synthesis with Latent Diffusion Models" paper so I do a deep dive into the code behind this paper. Let me know how you like this one - feedback is welcome as always!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Stable diffusion repo: https://github.com/CompVis/stable-diffusion
✅ LDM repo: https://github.com/CompVis/latent-diffusion

✅ LDM paper: https://arxiv.org/abs/2112.10752
✅ VQ-GAN (taming transformers) paper: https://arxiv.org/abs/2012.09841
✅ PLMS paper: https://arxiv.org/abs/2202.09778

✅ Imagenette dataset: https://github.com/fastai/imagenette
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00:00 Intro: why is Stable Diffusion important
00:03:50 Background knowledge: VQ-GAN, LDM, PLMS papers
00:09:20 Setup for a minimal code walk-through
00:13:30 Autoencoder with KL regularization training
00:17:15 LPIPS (perceptual loss) with discriminator loss
00:21:30 Loading ImageNet data and PyTorch Lightning training loop
00:26:35 Forward pass through the autoencoder
00:30:12 Loss calculation
00:32:08 Perceptual loss
00:36:30 KL and GAN generator loss
00:40:55 Discriminator loss
00:42:45 Summarizing the autoencoder training
00:45:44 LDM training
00:57:00 Encoding the image into the latent space
01:00:12 Forward pass through the LDM
01:01:22 LDM loss
01:04:08 Integrating conditioning via cross attention
01:10:34 Sampling using PLMS
01:16:02 CLIP
01:19:20 Classifier free guidance
01:22:19 Sampling code
01:26:42 Diffusion connection to differential equations (PLMS paper)
01:37:00 Quick glimpse into the safety check function
01:39:20 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#stablediffusion #latentdiffusion #imagesynthesis
 What's up guys, Alex here. In this video we are doing a deep dive into stable diffusion model and by the end of this video you'll understand exactly how the training of both stages works, how the sampling works, how to generate images, etc. Having said that, a week ago the weights for the stable diffusion models were published, which is super exciting. So we know that since the beginning of this year with the release of the Dali 2 paper, we had a Cambrian explosion of various image generation models such as, well, Dali 2, we had Mid Journey, we had Imagine, Party, we had Dali Mini, which is the open source implementation of Dali version 1, etc. etc. And the reason why this model is so important is, well, there are multiple reasons. One of them is the images are super high quality and additionally you have much less constraints. So that means you can generate images of human faces. You can also, because the code is open source, remove the safety features and generate whatever you want, although I don't encourage you definitely to share malicious images across the internet, but if you want to experiment you can do that as well. So that's kind of cool. Additionally, and very importantly, you can run this model directly on your machine, even if you only have a GPU that has 8GB of VRAM. So I personally have RTX 2080 on my laptop and I'm able to run this in Flow 16 without a problem and generate awesome images in a couple of seconds. So that's again very cool. So it's much faster, it requires much less memory and it has less constraints and it's high quality. That's why stable diffusion is so interesting. Okay, so I want to showcase a couple of very cool examples that some digital artists such as Xander here have been creating. You can see how cool these videos are and they were created using stable diffusion. You can see that the name of this piece is called Voyage Through Time and what Xander did, and you can kind of go through the video and I strongly encourage you to check out this video, it's amazing, it's really like mind blowing, like that we can create this on like literally consumer grade GPUs and enjoy this piece of art. Okay, so here's an example of the prompts and the seeds that Xander had to create to generate this cool video. So you can see there is a lot of like seed is being used as a hyperparameter, you have to tweak the seeds, you have to tweak the prompts, do some prompt engineering and at the end you end up with something as cool as this video. Okay, so I said I also managed to run this stable diffusion on my GPU that has 8GB of VRAM, I was generating using Float 16 precision, but you can see the images are super high quality as well. So yeah, these images were generated using a prompt, a painting of an AI robot having an epiphany moment. And additionally, I'll basically release a script that I use to generate some cool interpolation, so basically the idea is to you pick, you generate a diverse set of images such as the set you've seen here, and you can pick two images you like and then basically do interpolation between them in the latent space of the model. So that was inspired by Karpathy's gist, but yeah, I'm going to share that script very soon and also cover it in a different video. So you can see here how it looks like interpolation between basically I'm interpolating between this image here and this image here, and I'm going to now show you how the procedure kind of looks like. So here is how the image is being morphed as we are approaching the target image, and you can see there are some like jumps in the latent space, but all in all, let me kind of move this faster, you can see how cool it is. And yeah, I'm going to share the script a bit later. So having said that, let's jump into the code and well, first, let me show you some prerequisite knowledge here. I'll have three papers which I'll consult. I will not do deep dives of the papers in this video. So one of those papers is a teaming transformers basically for high resolution image synthesis that introduced the VQGAN paper. I previously covered that paper on my on my YouTube channel, so do check it out if you want to have a thorough understanding of how that works. I'm going to consult some of the formulas later when I show you the code. Next up, we have the high resolution image synthesis with latent diffusion models or LDMs, and that's the paper that's behind the stable diffusion model basically. So LDMs is what's powering the stable diffusion. And finally, I'm going to briefly consult this paper, pseudo numerical methods for diffusion models on manifolds that introduced the PLMS. So the pseudo linear multi-step scheduler that makes stable diffusion fast and high quality so you can literally have only 50 steps of diffusion model and still generate very high quality images. OK, so I'm going to briefly walk you through some ideas in this paper, like super briefly. And if you want to learn more about diffusion models, I have a whole diffusion playlist, so do check it out. I've been doing both the paper overviews as well as code walkthroughs of the original repos. So do check those out if you want to have a deeper understanding of how diffusion models work. Here I'm going to do mostly a diff between what has changed compared to those older models. OK, let's start here. So to enable diffusion model training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pre-trained autoencoders. So this is literally the main diff compared to all of the other papers. They train instead of working in the image space and doing the forward diffusion process and then learning the reverse process in the image space. Instead, they do that in the latent space. OK, let me just quickly show you the difference in the loss. So here it is. So here is the original basically construction, how the original loss for the DPM model looked like. You basically do you sample images here, you sample noise from the from the normal distribution, you sample different time steps and you literally just do MSC loss. So the mean squared error loss between the noise and what you predict. So this is so epsilon theta is modeled usually as a unit architecture. So using the unit architecture. And so what you do is take the X of t, which is the original input image, plus t steps of diffusion being applied to it. So you kind of noise it and then you pass this noisy version and the time step that's that. So basically the time step information and then you need to to predict the noise that was literally used to noise that image in the forward process, if that makes sense. So this is just a hopefully a recap for for most of you. And then you just keep on repeating this until you train this unit to predict the noise. And then later you can just use it to denoise the well pure noise images such that you can generate cool images. OK, so this is the only difference between LDM. So that this paper and the previous work, they are literally just working in the latent space of this encoder. And you can see everything else remains the same. So literally the difference is the following. So instead of working in the image space, you're going to whoops, you're going to have an like an encoder that's going to train being is going to be trained using the VQ similar to VQ GAN paper. You end up input the image. You end up with a latent here and then everything else remains the same. You're now using this latent to train your diffusion models. So here is the like a snippet from the DDPM paper. So now instead of using X of T will not be an image here. It will it will be instead. Sorry, X of zero will not be an image is going to be instead like a latent representation of the image. And that's pretty much it. Really, really. That's all there is to it. That's the only difference between this paper and and the previous art. So here is how the how the diagram looks like. So we will the code walkthrough will contain the three three parts. First part, I'm going to show you how to train the autoencoder here. It's going to be the first part of the video. The second part, I'm going to show you how to train the unit. So the fusion model such that you can so you can see here we first end up in the latent space. Then we do the forward diffusion process such that we end up with Z of T and then we diffuse it. We will learn how to predict the noise that was added during this diffusion process here. We learn how to predict it as the output. And well, that's it. Everything else is diffusion magic. And also we'll have I'm going to show you how they are using it. We'll also be learning the conditioning model. So we'll mostly be focusing on text as well as classes from the image net data set. OK, that's it. And the third part of the video will be about basically how to sample once we train these models. If you didn't understand everything because you lack some background, feel free to continue watching this video. I think that the code will be fairly self-explanatory. OK, so if you want to follow along what I'm doing here, you'll have to do a couple of steps. Obviously, you have to clone the original repo to the stable diffusion repo here. Just create the con the environment following the instructions here under the requirements section. And after that, go ahead and download like such that we can have a minimal setup and just get something running. You don't have to download the original image net data set, which is huge. You can instead go to this like fast eyes image net. However, we pronounce this thing like GitHub repo and download the smallest version possible. So literally 160 pixels is going to download only 10 classes from ImageNet and is going to make us well set up for it for the training procedure. Basically, this older repo latent diffusion, which preceded their stable diffusion repo, which I'm going to also link in the description, contains the necessary instructions for how you can unpack the data set and where you need to place it such that the script can recognize and the data can be loaded. OK, guys, so let me jump now into the actual code. Here we are. A couple more things we need to sort out. So first of all, obviously, we need to set some input arguments. There is only a couple of them that we care about. First one is we want to pass the like the configuration file for the auto encoder. So first we're going to train the auto encoder and then you need to pass the T flag, meaning we want to train it and then GPUs. So I'm passing zero comma because I only have a single GPU and the index of the GPUs here. If you have multiple GPUs, feel free to add one, two, three, whatever, how many GPUs you have there. OK, final thing, because this is a research code base after all, there are some bugs. And so I had to kind of sort them out before getting this to work on a Windows machine. OK, so let me open up my diff tool here. So I just have to do this diff tool D and I'm going to open up the differences to the code I made. So first of all, this is not a bug. This is just like a small modifications you have to do if you want to train this on a single GPU, if you want to be able to do a walkthrough on a VRAM limited system. So basically set batch size to one instead of default 12. Otherwise, you'll get CUDA out of memory exceptions. That's the first week you have to do. The second one is actually a bug. So you have to go to data, image and file. I mean, it's a bug if you're on Windows. So the thing is they kind of hard coded the slash here, assuming that that's how you split the path on an arbitrary system, which is not the case for Windows. So it's much better to use OS.SAP, which is going to resolve automatically depending on the operating system into the correct character sequence for Windows. It's going to be double backslash. And so this now works. Otherwise, you'll have some errors and the training will not work. OK, and the final tweak is in the main script. So if we go here to the main script, you can see I had to make a couple of changes. First of all, I had to set the number of workers to zero because otherwise I was again getting some errors. That's one tweak. The second tweak is set the shuffle to false here for the train data loader. The reason being is we are using, if you recall, we're using that super small subsample of the image net. And so if you just keep if you just do the shuffling, it might happen that you take some super big index and try to index into our data set and we don't have that image. And then you're going to have the well index out of range exception or something. OK, so that's the second tweak I had to do. The third tweak is comment on the DDP in case you only have a single GPU. So that's the distributed data parallel like object from from by Torch. I don't need that. So I had to comment it out. And finally, I had to comment out the signal parts because that does not work on Windows. And I didn't want to bother figuring out how to fix it on Windows because I'm just want to I just want to do a walkthrough and explain you guys how this training procedure looks like. That's it. We are ready. Let's get into the code. I'm going to start in the main file. So main file is the is where the training magic happens. So let me go to the main function here. OK, so here we are. I'm going to set the breakpoint here and let's start debugging this thing. OK, so just hit the training. We're going to use the information we passed in our launch Jason. If you're using this code, if not, you just need to pass those arguments somehow. Here we are. And again, I'm going to focus only only on on important parts. So I'm going to scheme everything else like we're just doing some parsing, blah, blah, blah. We can skip literally everything. We're just creating some directories, doing some configurations. I'm going to skip here. This is a salient point. So here we are doing the loading from the config file. So let me show you briefly how the config file looks like. So if you go here, if you find the configs directory and then you find under the other encoder, you'll find the one we're using. So that's this one. And you can see everything you need to instantiate various sub modules of the of the LDM is here. So we have the auto encoder here. We have the last we are using. We'll see what this is. It's basically a perceptual loss and the parameters we use to construct that loss. We have information to specify such that we can construct our data loaders. Some callbacks for PyTorch lightning, which is the framework they are using, which is built on top of PyTorch. If you never heard of it and that's it, like some information about accumulating radiance. Nothing, nothing important. Let's go back to the basically main function. That's all you need to see. OK, now I'm going to keep skipping all of this accelerators, GPUs, nothing fancy. I'm going to skip all the way here to a model. So that's where we instantiate the actual model. OK, so you can see here we're going to now construct the auto encoder KL. So let me just see what I've enabled all of the breakpoints. And if so, let me do F10 and I'm going to hit the init function of the auto encoder. OK, a couple of steps here. We because it's an auto encoder, obviously we have to instantiate first the encoder and then decoder. So let's see how those look like. Basically, let me just see what this DDConfig is. Yeah, OK, so it just contains the necessary parameters to construct the encoder. OK, so let's let's go into the encoder encoder. Nothing, nothing really fundamental. Basically, bunch of com layers, bunch of there are some interesting attention layers, which basically what I do is in the latent space of the encoder. So they take the image features and they just do the VAT type of like self attention. So you can you can kind of unroll them and then just do simple like a self attention logic. Where every token attends to every other token, that's it. And then there are some down sampling layers, which are again just a combination like basically you can see here, it's just a com like a layer with the stride of two, what not. That's it. I'm going to jump over all of these again. Just a simple encoder model. Nothing fancy there. OK, let's continue. Let's exit this part and now let's enter the decoder. Similar story. Nothing fundamental to understand here. Bunch of com layers, bunch of res blocks, and then there is some up sampling. I'm going to hit that five. Let's exit this model. That's it. The architecture is not the interesting part of this of training the other encoder. OK, now this is the interesting part. Here is where we instantiate this LP IPS with discriminator loss. It's a mouthful. It's basically a perceptual loss combined with adversarial loss. So let's step into it. OK, so here it is. Here we construct it again. I'm going to focus only on the important parts. We have some loss coefficients. So depending on which component of the loss we are looking at, we'll have a different weight. But most of all, let me let me show you this. So if you're not familiar with this perceptual loss concept, I think it has been introduced at least since the Neural Stuller Transfer Paper. So that was the those were the first papers I saw were using the perceptual loss. So the idea is to basically let me step into it to basically take a pre trained VGG 16 network and then pass your image and grab some representation, intermediate representation from the VGG 16 network and then basically do MSC in the latent space, in that representation space instead of in the image space. And by doing that, you can kind of compare like the semantics of your input images and not not like focus on on some maybe superficial noisy details in the image space. That's the basic idea. Additionally, there are these netlin layers, which what they do is they reduce the number of channels from, let's say, 64 to one. And then you can kind of collapse them and do the loss logic. We'll see in a couple of minutes what that exactly means once we start doing the actual forward prop through the models. Now we're just instantiating. So this is enough for you to understand. So we are loading the model here and then we literally grab the pre trained model from the teaming. So this is their older repos. So teaming transformers, which introduced the VQGAM paper, and they basically have a checkpoint there of the of the that's going to initialize this LP IPS loss. OK, so let me just kind of go and do that. And you can see here loaded pre trained loss from this file here, VGG PTH. OK, so now what happens is we just set gradients to false everywhere. And that's pretty much it. So let me now hit F5 and we are out of that function. OK, now the second interesting part is discriminator. So that's going to be used in the adversarial loss component of the final loss. Again, I'm not sure it's worth even digging through it, but it's literally just a bunch of you can see here. So there is some batch norm going on, calm layers, leaky rail use, calm layers, leaky rail use, nothing fundamental there. I'm just going to skip over it. And it's again going to down sample because this is a patch based discriminator. You can kind of check. You can see here that's the patch can discriminator. There was a first described in the PIX2PIX paper. You can check out this link if you want to check it out. But the main difference is instead of having a scalar and then say scalar telling you whether this image is real or fake, which GAN networks do GAN discriminators do here. You will instead have literally for a patch you have maybe 32 by 32 scalars and all of those will tell you. So a particular scalar will tell you whether that patch is real or fake. And that just kind of gives you more information to train. So more information for for for your model to train. That's it. OK, let's exit the discriminator. Again, we have some hinge loss. We'll see how that comes into into play a bit later. We have this weights again. OK, we'll see all of that a bit later. That's it. Now we define some more calm layers. Blah, blah, blah. Nothing fancy. Monitor just tells us which loss are we monitoring. And in this case, validation reconstruction loss is something we care about. OK, that's it. Now I'm going to skip again. We some biases logging blah, blah, blah. There is some model checkpointing. We don't care about model checkpointing in this logic. There is, as you can see, it's kind of fairly well, researchy code base. Some callbacks, logging images, learning rates, blah, blah, blah. Kuda callbacks. Nothing, nothing fundamental for our understanding of how the how the stable diffusion is trained. So I'm going to skip all of that until until I guess until the data part. OK, so it's going to skip until the data part. So this is where we load the image net data. And again, if you've set if you downloaded the the image net and E.T.T.E. data set and you've placed it into the correct directory, then everything is going to work as expected here. Plus the minor tweak I made with the OS separator, if you recall that from a couple of minutes ago. OK, so again, I'm going to ignore all of this because it's just going to prepare the data. Nothing, nothing fancy there. So I'm going to do this and click F5 and wait for our data to be loaded. You can see there is some filtering going on. Blah, blah, blah. And we have the data ready. OK, so that's it. We have you can see here train data and validation data. You can see the numbers are super, super big. And that's definitely not the number of images we have in our small image net. And that's the reason why you have to set a shuffle to false. Otherwise, you'll you'll get the index out of range exception. Again, this is just what is needed. What is needed to get a minimal setup up and running. If you actually want to train this and obviously you want to have a full image net, you want to have multiple GPUs, et cetera, et cetera. But if you just want to step through and understand what's going on, this is more than enough. OK, again, I'm going to skip across all of these parts. Not interesting checkpointing, blah, blah, blah, debugging signals. I'm going to enable breakpoint here, hit F5, and this is where the magic will start going on. So I'm going to now enable all the breakpoints and let's start digging into this code. OK, so F10, we end up in this on pre-trained routine start. So that's, again, something that PyTorch Lightning defines for you. PyTorch Lightning is very cool if you're a researcher and you're doing something that's fairly has a fairly common structure in the sense that you don't have to think about zeroing your gradients. You don't have to think about calling the optimizer step. You don't have to think about all of those details. You just have to define a couple of functions that the PyTorch Lightning API requires you to. And then everything kind of works out of the box automatically. OK, this part is not interesting. We're just creating some directories for configs and logging. Nothing interesting really. And finally, we hit the validation data loader. Now, you may be confused. Why are we starting with validation? How does that make sense? And that's, again, PyTorch Lightning detail. What the framework does, and I think this is fairly brilliant, is it first literally loads only one or two batches of data in the validation loop just to make sure that the validation works so that you don't have to waste a bunch of time in the training loop only to find out that your validation loop is broken and then you have to start from scratch. So instead, what I do is literally just verify that validation works. And after that, you resume the actual training and then validation. Everything else remains according to the usual sequence. OK, so because of that, I'm going to disable all of the breakpoints here. I'm just going to just enable the one in the train loader, hit F5, and wait until this validation data set basically check is completed. Here it is. We can see we are hitting the train data loader. Again, that's something that PyTorch Lightning requires you to define. Let's continue here. And let me just see whether I've enabled all the breakpoints. OK, so now we're going to first OK, again, some some some PyTorch Lightning stuff. OK, so here we are. So now what's happening is we are loading the data from our training data set. And here we are. We end up with this example. We've done some pre-processing. You can see here we fetch the image, blah, blah, blah. And we end up with the example, which is a dictionary that has multiple keys. So we have image and other ImageNet idiosyncratic like keys. So let me show you some of those. So example image is obviously our input image. So it's processed such that we have 256, 256, and three channels. We also have if I do example, let's say class label. Whoops. I need to make it a string. So class label is going to be, as you can see, label zero. So our data set only has 10 labels. We are not using the full ImageNet. That's why zero is highly probable. Then let me show you one more human label. Obviously, just a human readable label of this ImageNet class. So think of think of whatever that is. And that's it. OK, and this is where it starts getting interesting. So so here we are. We have a batch that was provided to us by PyTorch Lightning. Again, we have image in there. The shape is familiar 256, 256. And now we do this get input. It's just going to fetch the image, do some permutation, make sure that the memory is contiguous, blah, blah, blah. And now we're doing self and we pass inputs. So this is a fancy way of saying call the forward method of this class. And let's see. So we are dealing with we're dealing with all the encoders here, obviously. And the forward class is here. So let me just do F10 and we hit this part. So first part is obviously we do the encoding and then we do the sampling and then we do the decoding. OK, so let's dig into the decoder. Here it is. Here is the decoding logic, some down sampling, blah, blah, blah. Again, I'm going to skip everything here because as I already told you, encoder logic is fairly simple. We just do a forward pass through it and we end up with a latent space representation. So what's the dimensionality of this representation? You can see it's 6, 64, 64. And we expected three because if you recall, let me show you the. Well, we are using such a config where we expect 64, 64, 3. Six is there because we are actually returning the mean and the standard deviation or the variance here. And then we are sampling before we pass that sample to the decoder. So that's just a detail worth mentioning. They did experiment with different types of auto encoders. One of them we are currently working with is the KL regularized auto encoder. They were also playing with quantized versions, but we really don't care about all of those details. There is just too many things going on. OK, some processing of that representation. We end up with moments when they say moment, they literally mean zero within first moment. Like so the mean and the and the and the variance. So now we pass that into this diagonal Gaussian distribution. So we're going to basically form distribution here. We're going to name it posterior. So let me kind of enter there. You can see we just chunk. We just split into two parts that representation and we end up with mean and log variance. So that's what we're actually returning. And then some clamping exponentiation such that we end up with standard deviation variance. And that's it. OK, just a Gaussian. That's all. After that, we have to sample and sampling is simply because Gaussians are such a such a nice mathematical object. You basically just have to take the mean and add to it standard deviation multiplied by the normal noise from the normal Gaussian. That's it. OK, and we end up with the representation. So now let's see that should now have three channels, I guess. So three sixty four sixty four as expected. OK, so now let's pass this into the decoder stage. Here we are. We do some processing, just a conv layer decoder. What it does is, again, just up sampling, conf attention again, trivial stuff. I'm going to skip over that. That's kind of common knowledge. OK, so we end up with output output should be because this is an autoencoder should have the same shape as the input image. So that's two fifty six, two fifty six, three. Let's just print out the shape so that we are. Yeah, OK, we can we can see here that the shape is as expected. Let's continue. We now return that decoded output and we return the posterior object, which is the Gaussian. And that's this part. OK, so now you'll see there are two if statements here. One is when the optimizer index is zero and the other one is when the optimizer index is one. The reason we have those parameters is because we have two different optimizers. Let me just see whether where is that function. So configure optimizers. I didn't show you this one, so we didn't step into it. But simply, simply put, we have two optimizers. One is Adam and that Adam is going to be optimizing encoders and decoders here. And the second optimizer is going to be optimizing the discriminator weights. So we saw that when we instantiated the discriminator, I told you that it's going to be trainable. And here the other optimizer cares about updating discriminator weights, whereas this can be treated as a generator. So the autoencoder part is the generator. The discriminator is obviously discriminator. And that's that's it. OK, so let's go back here. So PyTorch Lightning makes it easy for us to do this type of gain loss computation. So first we'll step into this branch and then it will literally return called the training step again with optimizer index set to one. And then we'll train the discriminator. So that's everything is kind of handled for us by the framework. OK, this is where the whole. Brain of this autoencoder training is going to happen. So let's dig into this code. This is very important case. We get the last layer that's going to be used for some lambda calculation. You'll see that in a second. So here we are. We are inside of the loss. Let's see how the loss looks like for the other encoder. So first of all, we have the reconstruction loss. It's simply as you can see here, we subtract the inputs, which are the inputs image. So let's let me just kind of make sure that this is non-py or by Torch tensor, I guess. Yeah. And then let's just see the shape. The shape should be 256. Yeah, everything's fine there. We have reconstructions. We literally just subtract them. And we obviously want to make this difference as small as possible. So that makes a lot of sense. We're just doing simple image space like emmits type of a loss. OK, now we have perceptual loss. This is the interesting part. So we pass the inputs and the reconstructions again. But this time we do not compare them in the image space. Instead, we compare them in the latent space of the VGG of the pre-trained VGG network. So let me show you that thing as well. OK, so here we are. So we are again in this LP IPS loss and we are in the forward step. So first thing they do is they have the scaling layer and that scaling is just going to subtract some mean. And so let me just find that for a second. So scaling layer is here. So literally there is some shift and scale. So computations going on. I'm not sure whether this is from ImageNet statistics or not. Like if somebody knows, let me know. I think this is from ImageNet and that will make sense since we are now using ImageNet data set as well. So that would make sense. We just kind of normalize our input tensors and then we pass them through the VGG. And so we can see here what the VGG forward passing entails. So literally we're going to return some intermediate representations and we're going to wrap those up into this named top object and return it back. So let me hit F10. We are here. We're going to return all of that. We're going to hit it again because we have two computations, both for the input as well as well as for the reconstruction. Now we kind of cluster all of these layers that are going to reduce the number of channels together, just some syntactic sugar. And finally, let's see. Here is where the magic is happening. So what we do is we normalize the tensor, normalize just going to basically divide it by the L2 norm. OK, and then we do, as you can see here again, we just do we subtract them and we do the square. So it's an MSC loss without the M part because we're not doing the mean part yet. We're just doing this and we repeat that for all of the representations because we will have like five representations from the VGG. So that's why we'll have five iterations of this loop. And now we're going to break out of it. And now we're here. OK, so now what happens is we are going to pass those differences into these layers that are going to reduce the number of channels. So let me let me let me let me explain what I mean by that. So here here is this diff. So here is the difference in the that we that we've done among the features and the shape is going to be what? So here is the shape after we apply this layer. So after we apply this, we'll just end up with a single channel. So if I put zero here and if I put zero here and if I just do the shape, you can see we have one one two fifty six to fifty six. OK, so that's the sole purpose of this of this layer. And finally, we do the special spatial average, which is simply a mean a mean across dimensions. And so we end up with a scalar here. And that's it. That's that's how the perceptual loss looks like. It's fairly simple. And I'm going to just keep on stepping over five times there and then I'm going to keep the five. And that's it. So now we end up with this this array of well, it's actually. Yeah, we've accumulated we've aggregated the values here by doing the sum operator and we end up with a single number. And that's the perceptual loss. That's it, guys. The only part that's kind of new here for me personally are these layers. You could as well skip that part and simply do like a mean operation across these differences. So just MSC loss directly in the feature space. And that's it. But this is some type of modification. I'm not sure why and where there are some ablations, but that's the perceptual loss. OK, finally, we formed the reconstruction loss as a weighted sum of the reconstruction loss from the image space. Plus, we grab the perceptual loss here and we have some weights one point zero here. OK, so they are equally weighted. OK, next up, since this is zero, this is going to be zero. Let me just kind of double check. And because of that exponent raised to power zero is one. So it's just going to be a neutral operation. So this is doing nothing. And this is also going to skip here. And here we just do some rescaling. That's it. Summation, blah, blah, blah. And that's our final reconstruction loss. That's the loss there. Now we do the because remember, we return the posterior, which is the Gaussian from the latent space of the odd encoder. And we're just going to compute the KL divergence. So that's going to be the regularizer component. So I'm going to hit F10 here. You can see it's simply computing the KL divergence using the mean information variance and log variance information. OK, and we end up with a KL loss there. Some rescaling again. And now we enter the branch where we train the generator, I assume. Let's see. OK, all is good so far. OK, so we passed the reconstructions, which are the fake images through the discriminator and we get logits fake. OK. And then what we do is we have a minus and we do a mean across those weights. So let me show you the shape as I told you. That's a special discriminator. That's like a patch based discriminator. So that means that the shape is going to be maybe I think it's 32 times 32 or something. So let me see that. So it's 30 30. So that's how many patches we have. And we just do a mean across them. And by putting a minus here, we are literally forcing we will be tweaking the generator weights in such a way such that the discriminator gives a high value, which means it thinks, quote unquote, that those images are real. So, again, that's just your your your GAN standard GAN GAN stuff. Not nothing, nothing super complex there. If you're familiar with GANs. OK. And now there is some adaptive weight calculation. Let me see. So what's going to happen there? OK. Yeah. So I remember this is going to make make sure that we we are waiting the GAN part of the loss and the reconstruction loss appropriately. Let me show you the paper formula for this one is going to make a bit more sense. OK, guys, here we are. This is the VQ GAN paper I showed that in the beginning of the video. And this is basically the formula we're computing. We are taking the gradients of the reconstruction loss with respect to the last layer of the decoder weights. We divide that by the gradients of the GAN loss with respect to the weights of the last layer of the decoder again. What's the reasoning behind this? The reasoning is the following. If the gradients are super big for the reconstruction loss and they are smaller for the GAN loss, then this is going to be a big number, which is going to going to put a bigger weight on the GAN loss. So by doing that, we make sure that the network is learning from both losses and that one loss is not overwhelming the other loss when it comes to the contribution to the gradients. So that's the rough logic. And you can see that that's exactly what we're computing here. So you can see NLL loss is the reconstruction loss of the last layer weights. And that's it. We get the gradients there. Then we compute the gradients for the G loss, which is the GAN loss. That's it. And now we just normalize them. We do the norm. We divide them as well. And that's it. This is clamping, blah, blah, blah, times some weight. And that's it. That's the that's that weight. OK, I'm now going to return back to the code here. Now, this is going to be zero for the initial 50,000 iterations or something. The reason being they don't want to they first want to train the autoencoder and ignore the GAN loss such that they can form some representations and for the stability sake and only then slowly start using the GAN loss. So let me let me show you what I mean by that. Let me enter here. You can see that until we pass the global threshold. So I'm going to enter here. You'll see that the weight will be zero. So the weight will be set to zero until the global step crosses some certain threshold. And because of that, this is zero. And you can see it's used here. So that means this thing is kind of toggled off for the good portion of the of the of the beginning of the training. So that means we only use the KL regular regularize the loss here and we use the reconstruction loss here as the final loss. That's it. I know this was a mouthful, but like hopefully it makes sense. Now we just do some blah, blah, blah accumulation of those and we return back the loss. And that's it. We do some logging. We return the autoencoder loss. And that's it. Now we're going to hit the training step again. This time we're going to be training not the generator, but instead we are going to be training the discriminator. Let's see how that's going to look like again. Blah, blah, blah. I'm going to skip across these steps. I'm going to do basically wrong disabled break points. We're going to do a four pass again. This is kind of suboptimal. The there must be some way to optimize this such that we don't have to do because we are basically sending the same images here again. And that doesn't make much sense. And now let me let me return. Let me enable back all the break points. Now we enter this this branch here. OK, so again, we have a loss here. All of the inputs are the same. What's different is that let me just enter here. So we're going to have the same parts here. Reconstruction loss. Everything else remains the same. So the interesting part that we care about is here. So we want to go here. So I'm going to disable break points, enable just this one, hit F5 and we enter this branch this time. OK, so here we are. Now we pass the inputs to discriminator to get the real images and we pass the reconstructions to get the fake images, the largest of the fake images. OK, again, we just have this factor which is going to be zero initially, which means this loss will not be enabled in the first part of the training. Later, it's going to gradually kick in. And basically what we do here is a hinge loss between the largest of the real and the fake images. And basically that's it. That's that's the gain loss. So this is going to train the discriminator such that the discriminator learns the difference between real and fake images. Consequently, that's going to lead to better or the encoder because we're losing that we are using that discriminator to train the order. Guys, that's it. That was the training of the order. Hopefully that was interesting and made sense. Now I'm going to just stop this training because that's pretty much it. OK, I've hit F5 just to show you that now we're just going to iterate across patches and keep on repeating the same things we've just seen. So that's why I'm going to stop this training right now. We've seen how the order encoder training looks like. I'm going to show you quickly the formulas from the VQGAN paper just to to consolidate the knowledge here. And then we're going to step into understanding how the diffusion, the unit model is being trained. OK, guys, quickly coming to the paper. Here are the formulas for the VQGAN paper. So originally how the VQGAN was trained was it had these code books of discrete basically vectors and they had the reconstruction loss component. You can see here plus these losses here were called commitment losses and they were used to train the code book and the encoder. What changed in the VQGAN is that they started using so you can see here they're using instead of L2 loss, they use a perceptual loss and they introduce an adversarial training procedure with a patch based discriminator. So that's everything we've seen so far. And you can see that the final loss looks like this. So there is the GAN component weighted by this lambda. We've seen all of these. And there is this component here that basically consists out of the reconstruction loss and the perceptual loss. And additionally, in this in the latent diffusion model, they've introduced the KL divergence regularization. So basically. Bottom line is the order encoder used for the LDM paper is a small modification of what they've already done. The same authors in the VQGAN paper. That's it. Let's go back to the code. Wondering for a second about why these losses make sense, even though they are, they are not by any stretch of imagination, probably an optimal solution to how we should be training our models. Let's just think about it for a second. So we have the reconstruction loss in the image space and we have the perceptual loss. Those basically make sure that we are we are reconstructing images correctly so that we can we learn how to not lose information when we go through the bottleneck part. OK, then we have the GAN component, which makes sure that the images look very realistic. So that's additionally kind of enforcing the reconstruction. And finally, we have the KL divergence loss that's just going to be regularizing the latent space of our order encoder such that we can later be able to smoothly go through that space and be able to have meaningful representations. So that's the basic idea. Like we are. Yeah. OK, having said that, let's go to the long Jason. Let's modify the argument such that we are now training. We're now training the diffusion part and not the order. So I'm just going to remove this part here, remove this space here and paste this back here. So that's everything you need to do. Now we're training the diffusion model. So let's go back here. The difference will now be that they are not using with this config. They're not using the same order encoder. They're using a different one with the with the quantization. But like that doesn't matter. We're going to focus on on important parts only. OK, so I'm going to hit F hit the training here and let's start analyzing the code again. I'm going to focus only on instantiating the models and on the training loop. That's it. I'm going to skip everything else because we've seen all of that. We've seen the config as well. So let's just go to the model instantiation. So let's go here. And now let's let me just make sure that everything is enabled. I think it already is. But yeah. OK, so here we are late in diffusion like object. We're starting to instantiate everything we need. So one of those things is going to be unit architecture. OK, so let's start here. Let's see what's going on. We can ignore all of this. So here is the first interesting part. We're going to be initializing the superclass and the superclass is this DDPM. OK, so the DPM is the denoising diffusion probabilistic model. So that's the original diffusion paper that made diffusion kind of practical. OK, so let's enter there. So let's see what's going on there. OK, so here we are predicting you can see we're running in EPS prediction mode, which means we're predicting that noise instead of. Well, there are some other things you can predict like X zero, et cetera. OK, we can skip all of this. We can skip all of this. Now there is this diffusion wrapper. And that's where we actually start making the unit. So here you can see here unit model is constructed here. So let's construct the unit. Let's see how it looks like again in some of my previous videos. I've been going through in a lot of detail through how unit is constructed so you can go through that if you want. Here I'm just going to kind of put the scheme. And by the way, I love the I love the statements in this in this code base. Fool, you forgot to include the dimension of your cross attention conditioning. Very cool. Yeah, you can you can tell it's a production code. OK, so let's let's continue here. I think we can skip across all of these details. The important parts are these time step in bad sequential objects. What they basically make sure is that we can later pass time step information or conditional information into various sub modules. So let me kind of click F12 there, enter the definition. You can see that depending on the layer type, they'll sometimes be passing the conditioning information, sometimes the embedded information, sometimes just the input features, image features, and that's it. So it might be interesting to for me to just show you one small thing. And that's the following. So there are these blocks that integrate the conditional information, and I think those might be interesting. So here spatial transformer. So that's the module that's going to be integrating the conditional information into the unit. So I'm going to hit F12 there and I'm just going to add I'm just going to add like basically a break point there. And let's hit F10. Let's continue everything else. We don't care about it. Really, we can just construct the unit and let's go to the end here. So this is the end of the unit definition. Quite a long definition definition, as you can tell. So I'm going to skip over it and that's it. So we're using cross attention to do the conditioning, the conditional information integration. So, yeah. OK, so let's continue here. OK, that's it. Counting the parameters, nothing fancy. We don't care about that as well. That's just the exponential moving average. Not the fundamental part why this model is working. So I'm going to skip across all of these. Now we're registering the schedule. So this is the important part. And I've covered how this exactly works. Like I've been doing side by side comparison of formulas and of code. So do check out. I'm going to link those video cards somewhere here. But the diffusion playlist is the best place to start if you want to understand a bit better why those work. Otherwise, this video would be like five hours long or something. So let me let me enter here. Let me just show you how this roughly looks like. So you can see here a bunch of those alphas and alpha like the cumulative products and all of those variations of the formulas. Basically nothing is learnable here. These are just the weights of the scheduler that we need to get diffusion to work. So I'm going to skip across all of these and that's it. That's an important part. But like something I've covered previously and just bunch of formulas, you wouldn't get any insight from me going through it. So, yeah, I'm going to skip over that. OK, we are back in the latent diffusion. So we generated the unit we generated the schedule. Now let's continue and see what else is interesting here. I'm going to skip across all of these. So because now we are training the model in a holistic fashion, we obviously have to instantiate the first stage. And by first stage, they mean the auto encoder. So let's again just briefly go through this one. This time we are forming this VQ model and not the auto encoder KL. So that's a that's a difference. So let's just do that. So here we are. We are instantiating the VQ model. Here we are. And that's going to call the VQ model here. So let's just kind of start entering there. OK, so here we are. Let's see what's the main difference. We still have the encoder. I'm going to just toggle off all the breakpoints. We have the encoder. We have the decoder. Nothing have changed there. The only difference is so we have a loss, which is going to be identity in this this time because we are not training the auto encoder. We'll just be we'll be just loading the pre-trained weights. So let me enable all the breakpoints and let's enter this part. So let's see what's going on there. Some beddings, blah, blah, blah. Well, it's not blah, blah, blah. Embeddings are actually what's important in this model. So this is a codebook. You can see there is sixteen thousand three hundred eighty four codebook vectors and each of those has four dimensionality of four. And that's what's being used to do the quantizations later in the forward step. We'll see how it looks like a bit later. OK, so that's the quantized part. Now we have the conv layer, same as with the auto encoder KL. Nothing has changed there. Blah, blah, blah. We can skip that. We can skip all of this. And now we initialize from the pre-trained checkpoint. I'm just going to scheme over all of that. We are just basically doing the initialization of the auto encoder because remember how the whole logic works. You first pre-trained the auto encoder and then you basically freeze it and you use its latent space. And now you train the unit and the conditional model and everything. That's that's it. That's why we are loading the weights here. OK, and that's it. Now we do some set the eval mode and basically now we set the gradients to false everywhere and we can just continue on with the execution here. Let me hit F5. We exit the condition, instantiate the first stage function and now we instantiate the conditional stage. So this time let's enter this one. I think we are just going to have a class information. So you can see here a class embedder is the type of a conditioning model that will be instantiating here. So let's enter there. You can see it's simply thousand classes because we are dealing with image and embedding dimension and then literally just does the embedding in the forward pass. That's it. That's how the conditional stage model looks like. Let me remind you what. So that's basically let me show you the diagram here. That's this part in the image. So this part here is what we've just instantiated and this is the unit. OK, so we are in the stage two. OK, let's go back here. Let me keep on stepping over here and that's it, guys. Now I'm going to skip across. This is the main function again. I'm going to skip everything here. I'm going to also skip the data because that's again just image net and I'm going to stop at the trainer fit here. So hitting F5 waiting for everything to. Oops. I'm going to have to disable the break points and only then will this work. So disable and just enable this one. Hit the five. Get to trainer fit and then we're going to start start analyzing how this works. OK, so enabling the break points and we're going to hit the validation batch as usually. OK, no, actually. OK, what I've done here is I've added a break point to the configure optimizers this time. We're just using Adam W here and nothing else is important. I'm going to skip over this again. PyTorch lightning stuff. So this is not important. Here is the validation loader. So I'm going to disable all break points and just end up here hitting F5. We're going to end up in the training part of the of the training. OK, here we are. I'm going to step over this and now I'm going to enable the break points. Let's just go through this idiosyncratic part again. And now we're loading the data. OK, so we are loading the data. Everything remains the same. We have example that has keys such as images and labels, blah, blah, blah. I stepped through that part and we end up in the important part. And that's the training step. Again, that's a function that PyTorch lightning requires you to define. You can see that we have our batch here and everything is the same as when we were training our encoder. So batch image shape. You can see it's a 256 256 three channel image. OK, let's stop. Start stepping through this shared step. That's the first part. So we call this get input and I think that's just going to grab us the OK. So it grabs the back the image. So you can see here we just grab the image. And we end up with so X shape. So we have 256 256 and three. All of that is as usual. Now we just push it to the GPU. OK, so now here's what we do. So we encode using the first stage. So that means we don't want to deal with images anymore. When we're training the diffusion in the LVM in the latent diffusion models, instead we want to deal with the latent space. That's why we call the encode first stage. So let's do F10 here. Here's the encoded first stage. And here's what it does. It basically calls first stage model. It just calls the encode function of that model. Here is how it looks like. So it's just going to call the encoder. So let's basically let's hit F10 and we are here. So we are in the encoder. Everything remains. That's the same as our order encoder from the first part of the video. So I'm going to hit F5. Just bunch of calm layers, rest blocks and down sampling stuff. So F10. So we end up representation here. So we have now H shape. We have 32 32 4. OK, four is the number of of latent channels and this is the spatial dimensionality. Now we do some processing with a calm layer and we return back that representation. And that's it. That's the encoder posterior. You can see that the shape here is this. So it's not anymore. It's not a Gaussian distribution because these types of all the encoder models work a bit differently, but the logic is fairly similar. So now let's call this get first stage encoding. Let's see what this is. So I'm going to hit F12 just to see. OK, so I'm going to enter here. We can see it's not this object, so we will not sample from it. Instead, because it's a tensor, we simply just map, create this type of variable name binding and we just scale with some constant factor. Let me see what that number is and how it was defined. I'm not sure about it. OK, so it's one. OK, so we can we can just ignore all that. So let's continue. So now we have our representation. So that's the latent representation now because we have conditioning and the conditioning key is, I think, cross attention or something. So no, it's it's class label, but we are going to integrate using the cross attention logic. So let's step over here. And what we do is we just passed a batch because the batch contains, if you recall, a bunch of keys and among them, it contains the label. So that's how they've implemented this. Basically, they pass more data than is needed, but we'll see how that's going to be integrated a bit later. So let's see what's going on going on here. So we just map H.C. to C. So that's again the batch information. And then we're going to skip all of this. And finally, we we return back. So this is the latent representation and this is the conditioning information and a bit more stuff because it's the batch information. OK, and finally, we return back all of that. So that's the first part of the shared step function. Again, recall that we are currently in the latent diffusion model. Blah, blah, blah. If I scroll all the way up here, you'll see that. Oh, my God. Oh, my God. Latent diffusion. OK, so let's let's go back here. Now we do the forward prop through the diffusion model. So let's see how that looks like. F10. Here we are in the forward prop. We generate some time steps randomly. Basically, this is going to be thousand. So we generate randomly the time steps information. And now we do the conditioning. So because this is trainable, we get the learned conditioning. So let's see what's going to happen there. Basically, we just call the forward pass and we pass C. So C is as you can see, C is still a batch information. So if I enter inside of the forward function of the class embedder, you can see that we are now going to extract the key. So the class label from here is going to be some label of the image. So it's zero. And then we're going to embed it using the embedding table here. So we're going to return some representation that's like what has should have however many dimensions distinct had. I think it was hundred twenty eight or something. So C shape five hundred twelve. OK, so we return that back. We return that back. We turn the C and here we are. So we have the C now. And finally, we pass the image. We pass the sorry, this is not the image. X should be with was X. X should be. X should be the latent representation, right? Yeah, it is. So we passed the latent representation. We passed the conditioning information. We passed the T and we compute the losses. So this is basically what we saw here. We are literally randomly sampling like the these latent representations. Noise and time steps. That's it. Let's go back to the VS code. OK, so now we have the P losses. This is where the whole magic of the training happens. We sample some normal noise. So here we are. So we sample the normal noise and then we do the Q sample. So let's not do the noising process. So we start from our pure latent representation and we add up key steps of noise on top of it. So we simulate that. If you recall from my previous videos, there is a formula that makes us capable of doing that in a single step by just combining the start the start representation with the noise and using these basically non learnable parameters from the scheduler. We end up with a noisy version and now we just apply. We passed the noisy, the T and the conditioning. So that's literally that's literally this formula here. That's literally we are passing that T T and C. So let me show you that version here so you can see this formula three in the paper in the LDM paper. We pass these variables here and we are now passing that through the unit such that we can get a noise as the output. OK, so let's go back to the code. Let's hit F10 and enter the apply model function. So here what happens is just some variable backing. Nothing fundamental there. We just pass that vector that was 512 dimensional. Whoops, actually have to extract the first because we packed it into a list. Just some details. Nothing. Yeah, we are passing the same information. So that's the conditional vector. OK, and here we are. Now we pass. We call the this should be the unit. This should be the unit. Let me just if I do type on this object and this code is so nice I can do this debugging so easily. So diffusion wrapper, which contains the unit model and the scheduler. OK, so if I do F10, here we are. We enter the diffusion wrapper and because we have conditioning key set to cross attention, we're going to call the fusion model, pass the conditioning, pass the time steps, pass the latent representation. And this is going to be automatically handled and integrated by the cross attention. If I click F10, we should be in the forward pass of the unit model. Let's see whether that's indeed the case. And yeah, you can see here this is the definition of unit. We are in the right spot. And so let's now continue. So we just embed the time step information. So now we end up with some processing on top of those temporal representations. You can see that this is what the dimensionality of the time temporal information now is. OK, and now we start integrating. Now we start literally going through the unit. And you can see here we always pass the representation, the temporal embedding and the conditioning information. And now that's where the trick comes. So this is where the special object I mentioned a couple of like 20 minutes ago or something is going to play, come into the picture because it's going to know exactly what to pass. So I'm going to put a breakpoint here. I'm going to click F10 and you can see we immediately hit this one. And now, depending on what instance is this layer, whether it's a spatial transformer or temporal block or just a pure block, it's going to call one of the three versions of the layer. And that's that's it. Like now I'm going to hit F5 and that's going to hit to the it's going to hit the spatial transformer part. OK. So we hit a special transformer. And this is where the contextual information. So this should be five hundred twelve. This is the yeah. So this is the conditioning information from the label. And you can see that you basically now do simple transformer logic with the conditioning additionally here. So we just pass. You can see here just some projections, blah, blah, blah. We rearrange our representation such that it's suitable for transformers. We have batch size. We have sequence size, basically flattening out the height and the width. And we have the number of channels. OK. And now we just pass and do the cross attention with the basically with the transformer blocks. So let me do this. And that's it. Guys, that's it. That's it. That's the that's the whole logic. I'm going to hit F5 again and I'll have to remove this breakpoint. And now let's get out of here. Let's get out of this function. Let's just exit this function. OK. And I'm going to put a breakpoint here. Hit F5 again. Exit this function. And basically now I'm going to hit F5 again. And we are exiting the this is the unit for a pass. We're basically exiting the unit for a pass. And that's it. So let's exit here and let's see what the output shape is. It should be the same as the input latent representation. So it's going to be. Whoops. Let's let's hit F10. Now we have out. I'm not sure whether this is going to. Yeah. So we have the same. You can see the same shape as the input representation we passed. But this is now the prediction. This is now the noise that was put on top of that input latent representation. So, again, simply what we've done here, we have a unit model. We passed the input late representation. We have some time step information, some conditioning information. We use the time step information to noise the input latent. We combine them with the with the conditioning information. We pass all of that through the unit. We do a forward pass and we predict back the noise. And this is where we are at the moment. We have noise. And let's now see what's going on. Going on. We're going to return that noise. And here is how the loss is going to look like in the case where we use Epsilon prediction as parameterization. You can see that exactly this noise that was used to noise the initial representation to get the noisy version is now going to be the one the variable that we're trying to predict. So that's that's that's it. Like that's as simple as that. And this is just going to be literally just the MSC loss or something like that. So so, yeah, let me just do here. Yeah, it was literally just L2 loss. Nothing nothing other than that. And that's it. As simple as that. Again, I think this log bar is set to zero. So it will not influence. Yeah, it will not literally change anything by doing this. We don't do we don't change the loss. And we just now do some waiting. And that's it. I'm not sure why we need the VLB loss, the lower bound loss, because it's going to be the same computation as what we had up there. So look, this thing here is the same as this thing here. So if I do have 11 and enter this will again will again hit the L2 loss branch. So we just compute the L2 loss again and we return that log. OK, I'm not sure why we're computing this, because it literally gives us the same results. If I were to print loss simple here and if I were to print loss VLB, so the variational lower bound, we get the same values and we compute literally the same lines here. There is the only difference is we have a different weight here for this loss. And then because this is zero, this will not even have the impact on the final loss. That part is kind of confusing. And yeah, that's pretty much it. After I do this, there is additionally some some like logic with the with EMA here. But yeah, nothing, nothing, nothing vital there. OK, so we can we can skip over all of this and that's it. And now we just keep on repeating the batch after batch. So that's it. Just a forward pass through the unit. And then we basically do L2 loss between the predicted noise versus the noise that we used to noise our input representation. Fairly simple. Literally the formula we saw in the paper just being played out here in the code. So that's this formula here. We are randomly sampling these noise and conditioning label. OK, that one is correlated, obviously, with the image. And then we just use the latent representation of that image. So Formula 3 is everything that we have in the stage two of training this system here. After we've done this, we literally can now use the pre-trained weights and start sampling. So let me show you how we can sample using using this this code. So I'm going to stop this and let's go to the long script. And this time we're going to be using the text to image script, this one. And the only argument you have to pass is this PLMS. So that's going to be the scheduler we are using. You can also use the I.M. but like this one has higher quality and they showed in the paper that the I.M. is just a special case of the PLMS scheduler. OK, so let's open up the text to image and let's start. OK, guys, let's pick the correct configuration here and I'm going to hit run and we'll soon start executing the sampling script. So this one is used using a textual prompt. You can now generate the images. And that's that's how these text condition image generation models became popular. You can kind of tweak the prompts and generate corresponding images. OK, so here's a here's a prompt I'm currently using a painting of an AI having an epiphany moment. That's the prompt we're using. You specify the output directory, whether you want to. This is not important. Skipping all of these number of diffusion steps. That's important. We will be using five in this example just to make sure this is going to execute very, very quickly. I'm going to store the PLMS such that we want to use that scheduler. This one is also not important. This is if you want to have basically always start from the same latent. And that's going to kind of constrain your outputs to be less diverse as a consequence of that. We don't need that. We don't care about this variable as well. This is how many images I'm generating. Just nine. The input image dimensions, five, twelve, five, twelve. The latent number of channels is four. The sampling factor is eight. Number of samples is one. Blah, blah, blah. We can skip all of this. There's too many. Oh, my God. OK, so I'm just going to hit F5 and get here. We can skip this part because it's false. We see everything. We load the configuration. So let's see what that's going to be. OK, we want inference. So let me see. So let's go to configs. Let's go to configs here. Let's go to stable diffusion. We want inference. This is how that thing looks like. So, again, it's literally just going to specify the latent model, specify the unit, specify all the parameters, all the encoder, pretty much everything. And we're going to be using clip this time, not the class conditional information for the conditioning stage. That's it. It's going to need that all of this is specified inside of a config file. A couple of things you need to do if you want to follow along. First up is you have to go to this like page and download the weights. And the best ones are V1,4 or you can play with V1,3 as well. Basically, go ahead, download those, put them in the corresponding directory. And that's it. After that, there is a couple more things. So I had to, again, create some tweaks. So one of the tweaks is the following. So let's go to latent diffusion. I'm using this image net config file. I had to specify. So here's where you specify the checkpoint path. So wherever you download, you can see you need to specify the checkpoint path there. And there is the batch size. I also reduced it from 64 to 1. Otherwise, I'm getting good out of memory exceptions. OK, so that's one thing. And then you have to also modify this text to image. So basically what I had to do is to set. And this is very dirty just to make sure that I'm getting this to work on my computer. But there are better ways to do this. And it's much better for you to use the diffusers library than to do what I've done here because they actually have some parts where they accumulate in FP32 instead of doing everything in FP16 as I'm doing. So as a consequence, I'm probably getting a bit lower quality images, but it works and I can step through my code. So that's one tweak. And then what else? Let me see what I had to change. That's important to get this to work. Set the number of samples to one. Otherwise, we batch size of three. I was getting out of memory exceptions as well. OK, so here's the checkpoint information. You don't need this part. And and finally, I have explicitly made it such that we are dealing with flow 60 here and not with mixed precision because I was getting I was hitting errors. If I recall correctly, if I don't put the FP16 here explicitly and finally, I've just for the sake of speed, I commented out the check safety functionality that basically checks whether you have not safe for work or other problematic content. OK, having said that, let's go back to the text to image script and we can now continue. That's everything you need to know. OK, we now load the model. So we load the diffusion model this time. Obviously, I'm going to disable. We've seen all of this. So actually, we're going to enter this part and I'm only only going to show you the difference. And that's loading the clip model. So I'm going to hit F10. And let's just see how the clip model is being used to basically create the conditioning information from the input prompt. And that's going to be the only interesting part. OK, so I'm now going to toggle on all the breakpoints, hit F10. And here we are. So we are now creating the conditioning stage. Instantiated from config. We have frozen clip embedder. So let's hit F10. So here it is. So basically what I do is they use hugging faces, pre trained clip tokenizer and pre trained text model. And that's it. Like everything else, I've covered clip in one of my previous videos, so I'm going to link it somewhere here. You can you can go and check it out if you want to understand how clip exactly works. I also covered papers. So, yeah, there is plenty of information about clip on my channel. That's it. I'm going to leave the forward function, the breakpoint there and let's exit here. And we are done. We're done. We just set the gradients to false because we don't want to train this. We're now in the sampling stage. So I'm going to go back to the text to image and I'm just going to put a breakpoint here. I'm going to basically disable everything and just leave that breakpoint here. Let's exit the load model from config function and continue from there. OK, so pushing the model to GPU, we instantiate the PLMS sampler when it comes to well, when it comes to any function, it's not that complicated. We literally just store the LDM model here. So this is going to be just the let me show you this. So basically type is going to be LDM. So you can see here late in diffusion and we have thousand steps were used to train the model. That's an important information to form the schedule and schedule is going to be like a linear schedule. OK, so I'm going to show you more once we get to the actual forward pass sampling. They additionally have this border mark tool that basically makes sure that it encodes a watermark that's invisible to humanize basically into the image such that we know that it was machine generated image. And later we can use that watermark to exclude those samples from our training data if we decide to do that. I guess that's one of the main reasons they were they were doing that as well as to catch someone who is generating images and not giving proper credit to stable diffusion. I assume. OK, so here's a prompt. We have a painting of an AI having epiphany moment and we form some output directories. Nothing interesting there. And finally, here is so this is the interesting part. We're going to start the sampling here. So I'm going to hit F5 exit that part. And here it is. So we we have a single problem. So this loop is going to be kind of trivial. The first part we do is we get the learned conditioning from the for the empty prompt. So this is the again a classifier free guidance technique. So because our scale is seven point five, so that's the guidance scale. That's why we basically entered this part. And now let's let me just make sure this is all enabled. Let's enter this and let's see how clip works here. So what we do is we encode the prompt and the prompt in this particular case is an empty prompt. So encoding is just a forward pass through the frozen clipping better. So we just tokenize the text. You can see here we get batching codings. So if I were to since because it's an empty prompt, it's going to be fairly trivial. Let's see the shape here. OK, so it's going to be a trivial encoding. Basically, all of the numbers are the same. Those are the this is the beginning of sentence token. And these are the end of sentence. OK, we can kind of validate that by doing the following. So tokenizer decode, I think, was the name. And then we just pass this number. And let's see what it is to start of text. And the one with ending in seven is the end of text. That's it. That's how the representation is. The sequence of IDs is formed. We push them to GPU and we just passed them now to the transformer, which is basically the textual part of the clip model. And we end up with a final representation that's seventy seven five or something. So seventy seven and six seven hundred sixty eight. That's that's the final representation that came from clip. And we're going to use that to condition the unit model. That's the idea. OK, so let's get back here. Let's get back here. We're returning the information and that's it. So that's going to be used to condition the unit. Now we do the same thing just with the with the actual prompt. So I'm going to disable the breakpoints here and we do the same thing. So I'm just going to skip that again. C is going to be same shape as you see. You see standing for basically unconditional conditioning. And finally, here is where the sampling starts. After the sampling, we have a simple we just pass the final latent representation through the decoder, which is just a set of layers, attention layers, etc, etc. And up sampling, do some clamping, put the put the final representation on the CPU. When I say final presentation, I mean image, disimplementation, blah, blah, blah. And then we can store the image here. And that's everything. Everything else is kind of arranging the images into into grids. So this is where the gist of the logic is. So in the sample part, so I'm going to enable all the breakpoints and let's enter this part. Let's hit F5. And here we are. So we enter the sample part. And because we have conditioning, let's see what we do. We do some error checking there. We make a schedule. So make a schedule is just going to make sure that we set the appropriate constants. So let me let me go into this. So here here we are. So first of all, let's see. So we have uniform discretization. Number of steps is five because I said this is just dumb. Usually what you want to use like 50 is OK, 200 if you want to get a bit better results. But there is a saturated saturation going on. Definitely. So 50 is completely fine. So this is the actual number we use during the training. That's a vital information for the scheduler so that we can construct the the the final set of time steps. I'm going to disable these and we're going to end up as you can see here. So one two one four one six one and eight one. That means that's how we uniformly sampled our thousand time stamps into only five time steps. That's it. OK, next up, we grab the original alphas from our diffusion model here. We create some lambda functions. And then, as I said, we just start creating these non learnable constants. And I'm going to skip across all of these. Because it's really hard to explain this without taking one hour or something. So, yeah. OK, so finally, we have those constants in place. We formed our time steps. Now let's enter the logic here. So enable breakpoints. Let's hit F5 and enter this part. So here we are. We are starting to generate the image right now. So here we generate the initial random basically noise tensor, which is going to be sixty four sixty four four, because that's the size of the latent space of this particular LDM. And we start there and then we we form the time steps. So you can see here one two one, etc. And we just reverse the time steps here because when we are generating, we want to start from the end. We're starting from the noise image and we're just time going in reverse until we generate the actual image from our data distribution that we learned. OK, so running running PLM sampling with five time steps. Let's continue there. Basically, we we create the time steps there. The first one should be, I guess, eight or one or something. So TSA to one. And then we grab the next ones because the PLM logic needs it. So it's going to be, I guess, six or one. Yeah. And then let's continue. So now we end up doing this. We call the P sample PLM. The code is fairly messy and complicated. So, yeah, I apologize for not being able to explain this a bit more clearly, but I'm giving my best here. So stick with me. So, OK, let's see how the final logic looks like. So I'm setting a breakpoint here. We're going to hit that line a bit later. So for now, we just grab these constants, alphas, sigmas, etc, etc. I'm also going to put the breakpoint here. So we're going to enter that function. And here is the final logic that the PLM sampler does. So we get the model output and that's basically a forward pass through the through the unit. So let's let's go there. Let's see what's going on there. So here it is because we're doing the other classifier free guidance. We have to repeat our input representation, which is currently just a pure noise. Our time steps as well. We have to concatenate both our unconditional conditioning and the conditioning from the actual prompt. We do a forward pass through the unit model. So I'm going to hear just disable breakpoints due to forward pass because nothing insightful is happening there. And finally, we get the output representations here, which we then combine. So we combine the noise from the when we when we condition the unit with the unconditional conditioning and we also pass the basically the information that we got when using the actual conditioning here. And that's that's how we form our final noise prediction. OK, after that. So, OK, it had to take some time to figure out how to connect this code with the formulas from the paper. So let me show you side by side comparison of the code and formulas. And let's start and figure out what's going on here. OK, guys, so here is the paper. So I opened up the pseudo numerical methods for the future models on many folks paper. It's a mouthful. Even the title is hard to comprehend. So let's get back to this statement here. Oops, let me just find it. Basically, so we see four branches here. And the reason that is is because they are using this linear multi step method. And they say here here we cannot use linear multi step initially because the linear multi step method cannot start automatically, which needs at least three previous steps information to generate results. So we use the rune Jakuta method to compute the first three steps results and then use the linear multi step method to calculate the remaining. OK, so I've done some annotations here so that we can find the precise formula for each of these branches. So let's start like that. OK, so maybe I'll start start with the fourth branch. So the final step, once we have had at least three steps of this of this sampling step, basically we'll end up in this hitting this branch every single time. So let's go to Formula 12 and let's kind of convince ourselves that this makes sense. OK, so here is the Formula 12. You can see that the first step is to calculate the epsilon theta. So this is what we've done here in this step. So get model outputs we get we have our epsilon. So the noise prediction here. The next step is, as you can see here, to calculate this epsilon prime and you can see it's one over 24 blah blah blah. Some some some expression there you can see that corresponds directly to this one here. So fifty five E t minus fifty nine E t minus delta, etc. etc. So you can kind of see this corresponds to this. So here is the third step, this five function, which they call, I think, transfer function or something like that. Basically, here it is defined, but we will not use it for for for this explanation. So let me just get back here. So what I've done is basically I figure out that this function here, which will get to in a couple of seconds. So let me just enable the break points for a second here. So let's enable all of the break points. Let me do this. So this basically function corresponds to Formula 8. So that's this formula here. We're going to convince ourselves in that in a couple of seconds. OK, but let me let me get back here. So we saw that this expression here makes sense. Now, let's make sense out of the other branches. So for this one, I could not find the corresponding expression in the paper. So if anyone knows what the heck is going on, feel free to comment down below. I'm listening. So for the for for this branch here, I found the Formula 23 to correspond to this one. So let's let me let me find that one. So 23 is in the appendix of the paper. It's kind of hard to, yeah, even find the correspondence between these, let alone have some intuition. And I guess even the authors of this paper don't have the intuition. It's more of a OK, we associated this differential equation with this diffusion process. And we just then can automatically pick up the tools that we already have from a long and rich history of solving differential equations and apply those to solve diffusion. But like intuition wise, I'm not sure anyone understand what's going on here. Like I might be wrong, but that's that's my current understanding of the things. OK, so I said Formula 23. Here it is. So you can see we calculate Epsilon again and then we have one over half three Epsilon minus Epsilon old. And you can see that exact formula here. So we are computing that result here. And then finally, let me go to equation 22. That's the first branch of this of this complex branching. So first on a high level, this let me kind of step over here. So this function here get X previous and prediction X zero is calculating whatever phi is. We'll see what it is in a second. So once we have the results, so we have this X prev, then we feed that into the neural network and we again grab the find the noise. And that corresponds to this this step here. So whatever is output from the second step, we feed it back into our neural network. We feed in the T next as you can see here. So that's why we have T plus Delta here and we get back the results here. OK, and then finally we grab those results from this step and we just add them up with the Epsilon from the previous step and we divide by two. So that's this part here and we end up with the E T prime. And then after that, we're going to again call the five function here. So now I guess it boils down to figure out what this what this five function is. So let's step inside of this function. Let me show you what's going on here. So again, these are just some non learnable expressions. I'm going to skip those. But let me find the formula eight, which directly I found that that one corresponds directly to this to this code here. So let me find that. OK, so formula eight. Here it is. So let's convince ourselves that this makes sense. So here we have X minus square root one minus 80 times E T Epsilon. So we can see that corresponds to this expression here. We basically have X minus square root one minus this cumulative sum product of alphas. And then we multiply that, as you can see here, with with the output of our neural network. And then we divide all of that by the square root of 80 here. And that's pretty much this first term next up. So that's the first part. And then we calculate the second part. The second part is, as you can see here, it's computing this part here. So one minus a previous minus sigma squared square root of all of that times the Epsilon. And you can see that's precisely this term here. So let me now continue stepping. Now we calculate the noise. And finally, let's see how all of that is combined together. We have a previous square root. So that's this part times whatever we predicted up there, which was the first term, plus this term here. And then finally, plus the noise. So that's this part here. Let me just see what the value of sigma t is. And it's zero. OK, so this this part will actually be ignored. So as you can see here, this part, the noise, because it's multiplied by sigma t. Let me just kind of go into the debug console here. Let's convince ourselves sigma t is zero. And that makes sense because here in the paper, they mention it somewhere. Let me just find it that they only care about the case where sigma is equal to zero. Let me just find that one. OK, I found it. I didn't highlight it initially, so it was hard to find it. So they say here, therefore, our work concentrate on the case where sigma equals zero. OK, let me read this for you. So here sigma controls the ratio of random noise because it's modulating the noise, as you can see here in Formula 8. If sigma equals one, equation eight represents the reverse process of DDPMs. So those are the denoising diffusion probabilistic models. So if sigma equals zero, this equation represents the reverse process of DDIMs. OK, and only when sigma equals zero, this equation removes the random item and becomes a discrete form of a certain ODE, ordinary differential equation. Theoretically, the numerical methods that can be used on differential equations with random items are limited. So that's why we want to escape. We want to set sigma to zero because there are richer set of tools when we are not dealing with that random term. These authors here have done enough research in this case. Empirically, they have shown that DDIMs have a better acceleration effect when the number of total steps is relatively small. Therefore, our work concentrates on the case where sigma equals zero. OK, so guys, that's that's pretty much it. I'm going to open up the code here. As I said, I cannot provide you with much more intuition than this. So let me kind of step through all of this. I'm going to set a breakpoint here. Let me remove the disabled breakpoints. Let me enable this one. Let's hit F5 and let's get back to the function. OK, guys, so here we are. We now take that output. We basically sum it up with the last prediction from the network here, divide by two as per the formulas we saw previously. Next up, we compute the five function again, and that's it. So we return back the X previous, which is the basically as well as the noise, which is the next step in the reverse diffusion process. So we are slowly getting to the pure image. And that's pretty much it. I think we're not going to keep on iterating here. As you can see, we append the Epsilon to this to this array of fold Epsilon, which is used, if you remember the four branches. So this is where we collect the old Epsilon's and then we pass them inside here. And that's it. So if we start some circular array logic, callbacks are not important. So I'm going to skip all of that. And now we just keep on iterating and that's it. So we're going to have in this particular example, five steps, because that's how I've basically configured it. But like in general, you have like 50 steps or 200 steps if you want to have a bit better, basically quality of the image generation. But I'm going to stop this here. And basically I'm going to stop it here. And I'm going to finally just show you briefly the safety function they added that might be interesting to some of you. So let me show you how that functions. Let me just find it basically in the text to image here somewhere we have. OK, here. So here is the check safety function. Basically, what happens is once you have generated the image and you've done this clamping, blah, blah, blah, you convert it into an umpire array. Basically you have an image. So now they call this check safety for you and the check safety function. What it does is cause this safety feature extractor. So let's see what it does. Basically, that's some pre-trained model from hugging face hub. And then they call this safety checker, which is also basically, as you can see here, some pre-trained model from from from hugging face. And so what's interesting here is so they do some checks whether whether you have a basically not safe for work concept inside of the image. And if so, for that particular image, they load the replacement. And I think this one is fairly cool. So basically what I do in this repo is they take the image, they load this image here. I'm going to show you what it is in a second. Basically, they're recalling us. So let's let's find the assets folder. And then under the assets, we are looking for Rick. And that's going to be this one. OK, so they basically load this image in case you have not safe for work content and they get it back. Now, the problem is I was playing with this code base a bit. And even though I was not generating anything explicit or anything, I was still getting this this function being triggered. And so, yeah, it's not perfect. That's that's the point. So we can see the definition. Actually, I found it on the in the diffusers library. Here you can you can see the safety checker, how all of those functions are defined here. And you can see you can kind of go through this if you if you if you care about it. But bottom line is I couldn't find how the models were trained. I guess that's by design because they don't want you to know how to hack the model. Although I don't know where that's the best position on this topic, but I guess it's very highly debatable. So, yeah, you can kind of go through this code and basically explore it at your own pace. If you're guys, this was a super long video. Like we saw a lot of things we saw how to how to basically train and sample from from these class of LMS, latent diffusion models. So we saw how to first train this autoencoder who like whose way whose way to basically then freeze and use the latent space in the second stage where we train the LMS. So basically the unit plus the conditioning model. And finally, we saw how to sample from these models. And you saw that you saw that some of the formulas and the connections with differential equations make it kind of hard to have a clear intuition. But like I'm curious to know and hear whether and how you understand how diffusion models work. So if you have any intuitive type of an explanation, feel free to comment down below. I'll try and read all of those because I'm super curious. In any case, if you like this video, share it out. Subscribe to this channel. And until next time, bye bye.
[{"start": 0.0, "end": 16.0, "text": " What's up guys, Alex here. In this video we are doing a deep dive into stable diffusion model and by the end of this video you'll understand exactly how the training of both stages works, how the sampling works, how to generate images, etc."}, {"start": 16.0, "end": 45.0, "text": " Having said that, a week ago the weights for the stable diffusion models were published, which is super exciting. So we know that since the beginning of this year with the release of the Dali 2 paper, we had a Cambrian explosion of various image generation models such as, well, Dali 2, we had Mid Journey, we had Imagine, Party, we had Dali Mini, which is the open source implementation of Dali version 1, etc. etc."}, {"start": 45.0, "end": 60.0, "text": " And the reason why this model is so important is, well, there are multiple reasons. One of them is the images are super high quality and additionally you have much less constraints. So that means you can generate images of human faces."}, {"start": 60.0, "end": 73.0, "text": " You can also, because the code is open source, remove the safety features and generate whatever you want, although I don't encourage you definitely to share malicious images across the internet, but if you want to experiment you can do that as well."}, {"start": 73.0, "end": 94.0, "text": " So that's kind of cool. Additionally, and very importantly, you can run this model directly on your machine, even if you only have a GPU that has 8GB of VRAM. So I personally have RTX 2080 on my laptop and I'm able to run this in Flow 16 without a problem and generate awesome images in a couple of seconds."}, {"start": 94.0, "end": 103.0, "text": " So that's again very cool. So it's much faster, it requires much less memory and it has less constraints and it's high quality. That's why stable diffusion is so interesting."}, {"start": 103.0, "end": 115.0, "text": " Okay, so I want to showcase a couple of very cool examples that some digital artists such as Xander here have been creating. You can see how cool these videos are and they were created using stable diffusion."}, {"start": 115.0, "end": 134.0, "text": " You can see that the name of this piece is called Voyage Through Time and what Xander did, and you can kind of go through the video and I strongly encourage you to check out this video, it's amazing, it's really like mind blowing, like that we can create this on like literally consumer grade GPUs and enjoy this piece of art."}, {"start": 134.0, "end": 154.0, "text": " Okay, so here's an example of the prompts and the seeds that Xander had to create to generate this cool video. So you can see there is a lot of like seed is being used as a hyperparameter, you have to tweak the seeds, you have to tweak the prompts, do some prompt engineering and at the end you end up with something as cool as this video."}, {"start": 154.0, "end": 166.0, "text": " Okay, so I said I also managed to run this stable diffusion on my GPU that has 8GB of VRAM, I was generating using Float 16 precision, but you can see the images are super high quality as well."}, {"start": 166.0, "end": 174.0, "text": " So yeah, these images were generated using a prompt, a painting of an AI robot having an epiphany moment."}, {"start": 174.0, "end": 196.0, "text": " And additionally, I'll basically release a script that I use to generate some cool interpolation, so basically the idea is to you pick, you generate a diverse set of images such as the set you've seen here, and you can pick two images you like and then basically do interpolation between them in the latent space of the model."}, {"start": 196.0, "end": 204.0, "text": " So that was inspired by Karpathy's gist, but yeah, I'm going to share that script very soon and also cover it in a different video."}, {"start": 204.0, "end": 215.0, "text": " So you can see here how it looks like interpolation between basically I'm interpolating between this image here and this image here, and I'm going to now show you how the procedure kind of looks like."}, {"start": 215.0, "end": 228.0, "text": " So here is how the image is being morphed as we are approaching the target image, and you can see there are some like jumps in the latent space, but all in all, let me kind of move this faster, you can see how cool it is."}, {"start": 228.0, "end": 231.0, "text": " And yeah, I'm going to share the script a bit later."}, {"start": 231.0, "end": 240.0, "text": " So having said that, let's jump into the code and well, first, let me show you some prerequisite knowledge here."}, {"start": 240.0, "end": 253.0, "text": " I'll have three papers which I'll consult. I will not do deep dives of the papers in this video. So one of those papers is a teaming transformers basically for high resolution image synthesis that introduced the VQGAN paper."}, {"start": 253.0, "end": 261.0, "text": " I previously covered that paper on my on my YouTube channel, so do check it out if you want to have a thorough understanding of how that works."}, {"start": 261.0, "end": 265.0, "text": " I'm going to consult some of the formulas later when I show you the code."}, {"start": 265.0, "end": 273.0, "text": " Next up, we have the high resolution image synthesis with latent diffusion models or LDMs, and that's the paper that's behind the stable diffusion model basically."}, {"start": 273.0, "end": 286.0, "text": " So LDMs is what's powering the stable diffusion. And finally, I'm going to briefly consult this paper, pseudo numerical methods for diffusion models on manifolds that introduced the PLMS."}, {"start": 286.0, "end": 301.0, "text": " So the pseudo linear multi-step scheduler that makes stable diffusion fast and high quality so you can literally have only 50 steps of diffusion model and still generate very high quality images."}, {"start": 301.0, "end": 306.0, "text": " OK, so I'm going to briefly walk you through some ideas in this paper, like super briefly."}, {"start": 306.0, "end": 318.0, "text": " And if you want to learn more about diffusion models, I have a whole diffusion playlist, so do check it out. I've been doing both the paper overviews as well as code walkthroughs of the original repos."}, {"start": 318.0, "end": 322.0, "text": " So do check those out if you want to have a deeper understanding of how diffusion models work."}, {"start": 322.0, "end": 328.0, "text": " Here I'm going to do mostly a diff between what has changed compared to those older models."}, {"start": 328.0, "end": 341.0, "text": " OK, let's start here. So to enable diffusion model training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pre-trained autoencoders."}, {"start": 341.0, "end": 352.0, "text": " So this is literally the main diff compared to all of the other papers. They train instead of working in the image space and doing the forward diffusion process and then learning the reverse process in the image space."}, {"start": 352.0, "end": 355.0, "text": " Instead, they do that in the latent space."}, {"start": 355.0, "end": 360.0, "text": " OK, let me just quickly show you the difference in the loss."}, {"start": 360.0, "end": 367.0, "text": " So here it is. So here is the original basically construction, how the original loss for the DPM model looked like."}, {"start": 367.0, "end": 377.0, "text": " You basically do you sample images here, you sample noise from the from the normal distribution, you sample different time steps and you literally just do MSC loss."}, {"start": 377.0, "end": 388.0, "text": " So the mean squared error loss between the noise and what you predict. So this is so epsilon theta is modeled usually as a unit architecture."}, {"start": 388.0, "end": 398.0, "text": " So using the unit architecture. And so what you do is take the X of t, which is the original input image, plus t steps of diffusion being applied to it."}, {"start": 398.0, "end": 414.0, "text": " So you kind of noise it and then you pass this noisy version and the time step that's that. So basically the time step information and then you need to to predict the noise that was literally used to noise that image in the forward process, if that makes sense."}, {"start": 414.0, "end": 422.0, "text": " So this is just a hopefully a recap for for most of you. And then you just keep on repeating this until you train this unit to predict the noise."}, {"start": 422.0, "end": 430.0, "text": " And then later you can just use it to denoise the well pure noise images such that you can generate cool images."}, {"start": 430.0, "end": 440.0, "text": " OK, so this is the only difference between LDM. So that this paper and the previous work, they are literally just working in the latent space of this encoder."}, {"start": 440.0, "end": 447.0, "text": " And you can see everything else remains the same. So literally the difference is the following."}, {"start": 447.0, "end": 460.0, "text": " So instead of working in the image space, you're going to whoops, you're going to have an like an encoder that's going to train being is going to be trained using the VQ similar to VQ GAN paper."}, {"start": 460.0, "end": 469.0, "text": " You end up input the image. You end up with a latent here and then everything else remains the same. You're now using this latent to train your diffusion models."}, {"start": 469.0, "end": 477.0, "text": " So here is the like a snippet from the DDPM paper. So now instead of using X of T will not be an image here."}, {"start": 477.0, "end": 485.0, "text": " It will it will be instead. Sorry, X of zero will not be an image is going to be instead like a latent representation of the image."}, {"start": 485.0, "end": 493.0, "text": " And that's pretty much it. Really, really. That's all there is to it. That's the only difference between this paper and and the previous art."}, {"start": 493.0, "end": 501.0, "text": " So here is how the how the diagram looks like. So we will the code walkthrough will contain the three three parts."}, {"start": 501.0, "end": 505.0, "text": " First part, I'm going to show you how to train the autoencoder here. It's going to be the first part of the video."}, {"start": 505.0, "end": 517.0, "text": " The second part, I'm going to show you how to train the unit. So the fusion model such that you can so you can see here we first end up in the latent space."}, {"start": 517.0, "end": 523.0, "text": " Then we do the forward diffusion process such that we end up with Z of T and then we diffuse it."}, {"start": 523.0, "end": 527.0, "text": " We will learn how to predict the noise that was added during this diffusion process here."}, {"start": 527.0, "end": 533.0, "text": " We learn how to predict it as the output. And well, that's it. Everything else is diffusion magic."}, {"start": 533.0, "end": 539.0, "text": " And also we'll have I'm going to show you how they are using it. We'll also be learning the conditioning model."}, {"start": 539.0, "end": 545.0, "text": " So we'll mostly be focusing on text as well as classes from the image net data set."}, {"start": 545.0, "end": 552.0, "text": " OK, that's it. And the third part of the video will be about basically how to sample once we train these models."}, {"start": 552.0, "end": 558.0, "text": " If you didn't understand everything because you lack some background, feel free to continue watching this video."}, {"start": 558.0, "end": 561.0, "text": " I think that the code will be fairly self-explanatory."}, {"start": 561.0, "end": 568.0, "text": " OK, so if you want to follow along what I'm doing here, you'll have to do a couple of steps."}, {"start": 568.0, "end": 572.0, "text": " Obviously, you have to clone the original repo to the stable diffusion repo here."}, {"start": 572.0, "end": 578.0, "text": " Just create the con the environment following the instructions here under the requirements section."}, {"start": 578.0, "end": 585.0, "text": " And after that, go ahead and download like such that we can have a minimal setup and just get something running."}, {"start": 585.0, "end": 589.0, "text": " You don't have to download the original image net data set, which is huge."}, {"start": 589.0, "end": 593.0, "text": " You can instead go to this like fast eyes image net."}, {"start": 593.0, "end": 599.0, "text": " However, we pronounce this thing like GitHub repo and download the smallest version possible."}, {"start": 599.0, "end": 610.0, "text": " So literally 160 pixels is going to download only 10 classes from ImageNet and is going to make us well set up for it for the training procedure."}, {"start": 610.0, "end": 615.0, "text": " Basically, this older repo latent diffusion, which preceded their stable diffusion repo,"}, {"start": 615.0, "end": 622.0, "text": " which I'm going to also link in the description, contains the necessary instructions for how you can unpack the data set"}, {"start": 622.0, "end": 628.0, "text": " and where you need to place it such that the script can recognize and the data can be loaded."}, {"start": 628.0, "end": 632.0, "text": " OK, guys, so let me jump now into the actual code."}, {"start": 632.0, "end": 635.0, "text": " Here we are. A couple more things we need to sort out."}, {"start": 635.0, "end": 639.0, "text": " So first of all, obviously, we need to set some input arguments."}, {"start": 639.0, "end": 642.0, "text": " There is only a couple of them that we care about."}, {"start": 642.0, "end": 649.0, "text": " First one is we want to pass the like the configuration file for the auto encoder."}, {"start": 649.0, "end": 656.0, "text": " So first we're going to train the auto encoder and then you need to pass the T flag, meaning we want to train it and then GPUs."}, {"start": 656.0, "end": 663.0, "text": " So I'm passing zero comma because I only have a single GPU and the index of the GPUs here."}, {"start": 663.0, "end": 670.0, "text": " If you have multiple GPUs, feel free to add one, two, three, whatever, how many GPUs you have there."}, {"start": 670.0, "end": 675.0, "text": " OK, final thing, because this is a research code base after all, there are some bugs."}, {"start": 675.0, "end": 679.0, "text": " And so I had to kind of sort them out before getting this to work on a Windows machine."}, {"start": 679.0, "end": 683.0, "text": " OK, so let me open up my diff tool here."}, {"start": 683.0, "end": 689.0, "text": " So I just have to do this diff tool D and I'm going to open up the differences to the code I made."}, {"start": 689.0, "end": 692.0, "text": " So first of all, this is not a bug."}, {"start": 692.0, "end": 697.0, "text": " This is just like a small modifications you have to do if you want to train this on a single GPU,"}, {"start": 697.0, "end": 701.0, "text": " if you want to be able to do a walkthrough on a VRAM limited system."}, {"start": 701.0, "end": 705.0, "text": " So basically set batch size to one instead of default 12."}, {"start": 705.0, "end": 709.0, "text": " Otherwise, you'll get CUDA out of memory exceptions."}, {"start": 709.0, "end": 711.0, "text": " That's the first week you have to do."}, {"start": 711.0, "end": 716.0, "text": " The second one is actually a bug. So you have to go to data, image and file."}, {"start": 716.0, "end": 718.0, "text": " I mean, it's a bug if you're on Windows."}, {"start": 718.0, "end": 721.0, "text": " So the thing is they kind of hard coded the slash here,"}, {"start": 721.0, "end": 726.0, "text": " assuming that that's how you split the path on an arbitrary system, which is not the case for Windows."}, {"start": 726.0, "end": 732.0, "text": " So it's much better to use OS.SAP, which is going to resolve automatically depending on the operating system"}, {"start": 732.0, "end": 735.0, "text": " into the correct character sequence for Windows."}, {"start": 735.0, "end": 737.0, "text": " It's going to be double backslash."}, {"start": 737.0, "end": 742.0, "text": " And so this now works. Otherwise, you'll have some errors and the training will not work."}, {"start": 742.0, "end": 746.0, "text": " OK, and the final tweak is in the main script."}, {"start": 746.0, "end": 750.0, "text": " So if we go here to the main script, you can see I had to make a couple of changes."}, {"start": 750.0, "end": 756.0, "text": " First of all, I had to set the number of workers to zero because otherwise I was again getting some errors."}, {"start": 756.0, "end": 758.0, "text": " That's one tweak."}, {"start": 758.0, "end": 763.0, "text": " The second tweak is set the shuffle to false here for the train data loader."}, {"start": 763.0, "end": 770.0, "text": " The reason being is we are using, if you recall, we're using that super small subsample of the image net."}, {"start": 770.0, "end": 773.0, "text": " And so if you just keep if you just do the shuffling,"}, {"start": 773.0, "end": 779.0, "text": " it might happen that you take some super big index and try to index into our data set and we don't have that image."}, {"start": 779.0, "end": 784.0, "text": " And then you're going to have the well index out of range exception or something."}, {"start": 784.0, "end": 787.0, "text": " OK, so that's the second tweak I had to do."}, {"start": 787.0, "end": 791.0, "text": " The third tweak is comment on the DDP in case you only have a single GPU."}, {"start": 791.0, "end": 797.0, "text": " So that's the distributed data parallel like object from from by Torch."}, {"start": 797.0, "end": 799.0, "text": " I don't need that. So I had to comment it out."}, {"start": 799.0, "end": 804.0, "text": " And finally, I had to comment out the signal parts because that does not work on Windows."}, {"start": 804.0, "end": 813.0, "text": " And I didn't want to bother figuring out how to fix it on Windows because I'm just want to I just want to do a walkthrough and explain you guys how this training procedure looks like."}, {"start": 813.0, "end": 817.0, "text": " That's it. We are ready. Let's get into the code."}, {"start": 817.0, "end": 819.0, "text": " I'm going to start in the main file."}, {"start": 819.0, "end": 822.0, "text": " So main file is the is where the training magic happens."}, {"start": 822.0, "end": 825.0, "text": " So let me go to the main function here."}, {"start": 825.0, "end": 831.0, "text": " OK, so here we are. I'm going to set the breakpoint here and let's start debugging this thing."}, {"start": 831.0, "end": 834.0, "text": " OK, so just hit the training."}, {"start": 834.0, "end": 838.0, "text": " We're going to use the information we passed in our launch Jason."}, {"start": 838.0, "end": 842.0, "text": " If you're using this code, if not, you just need to pass those arguments somehow."}, {"start": 842.0, "end": 847.0, "text": " Here we are. And again, I'm going to focus only only on on important parts."}, {"start": 847.0, "end": 852.0, "text": " So I'm going to scheme everything else like we're just doing some parsing, blah, blah, blah."}, {"start": 852.0, "end": 854.0, "text": " We can skip literally everything."}, {"start": 854.0, "end": 857.0, "text": " We're just creating some directories, doing some configurations."}, {"start": 857.0, "end": 860.0, "text": " I'm going to skip here. This is a salient point."}, {"start": 860.0, "end": 863.0, "text": " So here we are doing the loading from the config file."}, {"start": 863.0, "end": 867.0, "text": " So let me show you briefly how the config file looks like."}, {"start": 867.0, "end": 874.0, "text": " So if you go here, if you find the configs directory and then you find under the other encoder, you'll find the one we're using."}, {"start": 874.0, "end": 883.0, "text": " So that's this one. And you can see everything you need to instantiate various sub modules of the of the LDM is here."}, {"start": 883.0, "end": 886.0, "text": " So we have the auto encoder here. We have the last we are using."}, {"start": 886.0, "end": 892.0, "text": " We'll see what this is. It's basically a perceptual loss and the parameters we use to construct that loss."}, {"start": 892.0, "end": 899.0, "text": " We have information to specify such that we can construct our data loaders."}, {"start": 899.0, "end": 904.0, "text": " Some callbacks for PyTorch lightning, which is the framework they are using, which is built on top of PyTorch."}, {"start": 904.0, "end": 910.0, "text": " If you never heard of it and that's it, like some information about accumulating radiance."}, {"start": 910.0, "end": 915.0, "text": " Nothing, nothing important. Let's go back to the basically main function."}, {"start": 915.0, "end": 922.0, "text": " That's all you need to see. OK, now I'm going to keep skipping all of this accelerators, GPUs, nothing fancy."}, {"start": 922.0, "end": 927.0, "text": " I'm going to skip all the way here to a model. So that's where we instantiate the actual model."}, {"start": 927.0, "end": 932.0, "text": " OK, so you can see here we're going to now construct the auto encoder KL."}, {"start": 932.0, "end": 935.0, "text": " So let me just see what I've enabled all of the breakpoints."}, {"start": 935.0, "end": 940.0, "text": " And if so, let me do F10 and I'm going to hit the init function of the auto encoder."}, {"start": 940.0, "end": 947.0, "text": " OK, a couple of steps here. We because it's an auto encoder, obviously we have to instantiate first the encoder and then decoder."}, {"start": 947.0, "end": 954.0, "text": " So let's see how those look like. Basically, let me just see what this DDConfig is."}, {"start": 954.0, "end": 959.0, "text": " Yeah, OK, so it just contains the necessary parameters to construct the encoder."}, {"start": 959.0, "end": 964.0, "text": " OK, so let's let's go into the encoder encoder. Nothing, nothing really fundamental."}, {"start": 964.0, "end": 976.0, "text": " Basically, bunch of com layers, bunch of there are some interesting attention layers, which basically what I do is in the latent space of the encoder."}, {"start": 976.0, "end": 982.0, "text": " So they take the image features and they just do the VAT type of like self attention."}, {"start": 982.0, "end": 991.0, "text": " So you can you can kind of unroll them and then just do simple like a self attention logic."}, {"start": 991.0, "end": 994.0, "text": " Where every token attends to every other token, that's it."}, {"start": 994.0, "end": 1005.0, "text": " And then there are some down sampling layers, which are again just a combination like basically you can see here, it's just a com like a layer with the stride of two, what not."}, {"start": 1005.0, "end": 1009.0, "text": " That's it. I'm going to jump over all of these again."}, {"start": 1009.0, "end": 1015.0, "text": " Just a simple encoder model. Nothing fancy there."}, {"start": 1015.0, "end": 1018.0, "text": " OK, let's continue. Let's exit this part and now let's enter the decoder."}, {"start": 1018.0, "end": 1022.0, "text": " Similar story. Nothing fundamental to understand here."}, {"start": 1022.0, "end": 1026.0, "text": " Bunch of com layers, bunch of res blocks, and then there is some up sampling."}, {"start": 1026.0, "end": 1029.0, "text": " I'm going to hit that five. Let's exit this model."}, {"start": 1029.0, "end": 1034.0, "text": " That's it. The architecture is not the interesting part of this of training the other encoder."}, {"start": 1034.0, "end": 1042.0, "text": " OK, now this is the interesting part. Here is where we instantiate this LP IPS with discriminator loss."}, {"start": 1042.0, "end": 1046.0, "text": " It's a mouthful. It's basically a perceptual loss combined with adversarial loss."}, {"start": 1046.0, "end": 1049.0, "text": " So let's step into it. OK, so here it is."}, {"start": 1049.0, "end": 1052.0, "text": " Here we construct it again. I'm going to focus only on the important parts."}, {"start": 1052.0, "end": 1058.0, "text": " We have some loss coefficients. So depending on which component of the loss we are looking at, we'll have a different weight."}, {"start": 1058.0, "end": 1060.0, "text": " But most of all, let me let me show you this."}, {"start": 1060.0, "end": 1068.0, "text": " So if you're not familiar with this perceptual loss concept, I think it has been introduced at least since the Neural Stuller Transfer Paper."}, {"start": 1068.0, "end": 1072.0, "text": " So that was the those were the first papers I saw were using the perceptual loss."}, {"start": 1072.0, "end": 1085.0, "text": " So the idea is to basically let me step into it to basically take a pre trained VGG 16 network and then pass your image and grab some representation,"}, {"start": 1085.0, "end": 1094.0, "text": " intermediate representation from the VGG 16 network and then basically do MSC in the latent space, in that representation space instead of in the image space."}, {"start": 1094.0, "end": 1105.0, "text": " And by doing that, you can kind of compare like the semantics of your input images and not not like focus on on some maybe superficial noisy details in the image space."}, {"start": 1105.0, "end": 1113.0, "text": " That's the basic idea. Additionally, there are these netlin layers, which what they do is they reduce the number of channels from, let's say, 64 to one."}, {"start": 1113.0, "end": 1115.0, "text": " And then you can kind of collapse them and do the loss logic."}, {"start": 1115.0, "end": 1121.0, "text": " We'll see in a couple of minutes what that exactly means once we start doing the actual forward prop through the models."}, {"start": 1121.0, "end": 1125.0, "text": " Now we're just instantiating. So this is enough for you to understand."}, {"start": 1125.0, "end": 1133.0, "text": " So we are loading the model here and then we literally grab the pre trained model from the teaming."}, {"start": 1133.0, "end": 1135.0, "text": " So this is their older repos."}, {"start": 1135.0, "end": 1145.0, "text": " So teaming transformers, which introduced the VQGAM paper, and they basically have a checkpoint there of the of the that's going to initialize this LP IPS loss."}, {"start": 1145.0, "end": 1148.0, "text": " OK, so let me just kind of go and do that."}, {"start": 1148.0, "end": 1153.0, "text": " And you can see here loaded pre trained loss from this file here, VGG PTH."}, {"start": 1153.0, "end": 1158.0, "text": " OK, so now what happens is we just set gradients to false everywhere."}, {"start": 1158.0, "end": 1161.0, "text": " And that's pretty much it."}, {"start": 1161.0, "end": 1166.0, "text": " So let me now hit F5 and we are out of that function."}, {"start": 1166.0, "end": 1169.0, "text": " OK, now the second interesting part is discriminator."}, {"start": 1169.0, "end": 1175.0, "text": " So that's going to be used in the adversarial loss component of the final loss."}, {"start": 1175.0, "end": 1182.0, "text": " Again, I'm not sure it's worth even digging through it, but it's literally just a bunch of you can see here."}, {"start": 1182.0, "end": 1189.0, "text": " So there is some batch norm going on, calm layers, leaky rail use, calm layers, leaky rail use, nothing fundamental there."}, {"start": 1189.0, "end": 1191.0, "text": " I'm just going to skip over it."}, {"start": 1191.0, "end": 1196.0, "text": " And it's again going to down sample because this is a patch based discriminator."}, {"start": 1196.0, "end": 1200.0, "text": " You can kind of check. You can see here that's the patch can discriminator."}, {"start": 1200.0, "end": 1203.0, "text": " There was a first described in the PIX2PIX paper."}, {"start": 1203.0, "end": 1206.0, "text": " You can check out this link if you want to check it out."}, {"start": 1206.0, "end": 1216.0, "text": " But the main difference is instead of having a scalar and then say scalar telling you whether this image is real or fake, which GAN networks do GAN discriminators do here."}, {"start": 1216.0, "end": 1223.0, "text": " You will instead have literally for a patch you have maybe 32 by 32 scalars and all of those will tell you."}, {"start": 1223.0, "end": 1228.0, "text": " So a particular scalar will tell you whether that patch is real or fake."}, {"start": 1228.0, "end": 1231.0, "text": " And that just kind of gives you more information to train."}, {"start": 1231.0, "end": 1234.0, "text": " So more information for for for your model to train."}, {"start": 1234.0, "end": 1238.0, "text": " That's it. OK, let's exit the discriminator."}, {"start": 1238.0, "end": 1241.0, "text": " Again, we have some hinge loss."}, {"start": 1241.0, "end": 1244.0, "text": " We'll see how that comes into into play a bit later."}, {"start": 1244.0, "end": 1247.0, "text": " We have this weights again."}, {"start": 1247.0, "end": 1249.0, "text": " OK, we'll see all of that a bit later. That's it."}, {"start": 1249.0, "end": 1253.0, "text": " Now we define some more calm layers. Blah, blah, blah."}, {"start": 1253.0, "end": 1257.0, "text": " Nothing fancy. Monitor just tells us which loss are we monitoring."}, {"start": 1257.0, "end": 1261.0, "text": " And in this case, validation reconstruction loss is something we care about."}, {"start": 1261.0, "end": 1264.0, "text": " OK, that's it. Now I'm going to skip again."}, {"start": 1264.0, "end": 1268.0, "text": " We some biases logging blah, blah, blah."}, {"start": 1268.0, "end": 1271.0, "text": " There is some model checkpointing."}, {"start": 1271.0, "end": 1274.0, "text": " We don't care about model checkpointing in this logic."}, {"start": 1274.0, "end": 1279.0, "text": " There is, as you can see, it's kind of fairly well, researchy code base."}, {"start": 1279.0, "end": 1284.0, "text": " Some callbacks, logging images, learning rates, blah, blah, blah."}, {"start": 1284.0, "end": 1290.0, "text": " Kuda callbacks. Nothing, nothing fundamental for our understanding of how the how the stable diffusion is trained."}, {"start": 1290.0, "end": 1296.0, "text": " So I'm going to skip all of that until until I guess until the data part."}, {"start": 1296.0, "end": 1298.0, "text": " OK, so it's going to skip until the data part."}, {"start": 1298.0, "end": 1301.0, "text": " So this is where we load the image net data."}, {"start": 1301.0, "end": 1312.0, "text": " And again, if you've set if you downloaded the the image net and E.T.T.E. data set and you've placed it into the correct directory,"}, {"start": 1312.0, "end": 1314.0, "text": " then everything is going to work as expected here."}, {"start": 1314.0, "end": 1320.0, "text": " Plus the minor tweak I made with the OS separator, if you recall that from a couple of minutes ago."}, {"start": 1320.0, "end": 1325.0, "text": " OK, so again, I'm going to ignore all of this because it's just going to prepare the data."}, {"start": 1325.0, "end": 1328.0, "text": " Nothing, nothing fancy there."}, {"start": 1328.0, "end": 1333.0, "text": " So I'm going to do this and click F5 and wait for our data to be loaded."}, {"start": 1333.0, "end": 1335.0, "text": " You can see there is some filtering going on."}, {"start": 1335.0, "end": 1338.0, "text": " Blah, blah, blah. And we have the data ready. OK, so that's it."}, {"start": 1338.0, "end": 1341.0, "text": " We have you can see here train data and validation data."}, {"start": 1341.0, "end": 1343.0, "text": " You can see the numbers are super, super big."}, {"start": 1343.0, "end": 1347.0, "text": " And that's definitely not the number of images we have in our small image net."}, {"start": 1347.0, "end": 1350.0, "text": " And that's the reason why you have to set a shuffle to false."}, {"start": 1350.0, "end": 1353.0, "text": " Otherwise, you'll you'll get the index out of range exception."}, {"start": 1353.0, "end": 1356.0, "text": " Again, this is just what is needed."}, {"start": 1356.0, "end": 1359.0, "text": " What is needed to get a minimal setup up and running."}, {"start": 1359.0, "end": 1362.0, "text": " If you actually want to train this and obviously you want to have a full image net,"}, {"start": 1362.0, "end": 1364.0, "text": " you want to have multiple GPUs, et cetera, et cetera."}, {"start": 1364.0, "end": 1368.0, "text": " But if you just want to step through and understand what's going on, this is more than enough."}, {"start": 1368.0, "end": 1371.0, "text": " OK, again, I'm going to skip across all of these parts."}, {"start": 1371.0, "end": 1376.0, "text": " Not interesting checkpointing, blah, blah, blah, debugging signals."}, {"start": 1376.0, "end": 1383.0, "text": " I'm going to enable breakpoint here, hit F5, and this is where the magic will start going on."}, {"start": 1383.0, "end": 1388.0, "text": " So I'm going to now enable all the breakpoints and let's start digging into this code."}, {"start": 1388.0, "end": 1393.0, "text": " OK, so F10, we end up in this on pre-trained routine start."}, {"start": 1393.0, "end": 1397.0, "text": " So that's, again, something that PyTorch Lightning defines for you."}, {"start": 1397.0, "end": 1403.0, "text": " PyTorch Lightning is very cool if you're a researcher and you're doing something that's fairly has a fairly common structure"}, {"start": 1403.0, "end": 1406.0, "text": " in the sense that you don't have to think about zeroing your gradients."}, {"start": 1406.0, "end": 1409.0, "text": " You don't have to think about calling the optimizer step."}, {"start": 1409.0, "end": 1411.0, "text": " You don't have to think about all of those details."}, {"start": 1411.0, "end": 1418.0, "text": " You just have to define a couple of functions that the PyTorch Lightning API requires you to."}, {"start": 1418.0, "end": 1421.0, "text": " And then everything kind of works out of the box automatically."}, {"start": 1421.0, "end": 1425.0, "text": " OK, this part is not interesting."}, {"start": 1425.0, "end": 1428.0, "text": " We're just creating some directories for configs and logging."}, {"start": 1428.0, "end": 1430.0, "text": " Nothing interesting really."}, {"start": 1430.0, "end": 1433.0, "text": " And finally, we hit the validation data loader."}, {"start": 1433.0, "end": 1437.0, "text": " Now, you may be confused. Why are we starting with validation?"}, {"start": 1437.0, "end": 1441.0, "text": " How does that make sense? And that's, again, PyTorch Lightning detail."}, {"start": 1441.0, "end": 1449.0, "text": " What the framework does, and I think this is fairly brilliant, is it first literally loads only one or two batches of data"}, {"start": 1449.0, "end": 1453.0, "text": " in the validation loop just to make sure that the validation works"}, {"start": 1453.0, "end": 1456.0, "text": " so that you don't have to waste a bunch of time in the training loop"}, {"start": 1456.0, "end": 1461.0, "text": " only to find out that your validation loop is broken and then you have to start from scratch."}, {"start": 1461.0, "end": 1465.0, "text": " So instead, what I do is literally just verify that validation works."}, {"start": 1465.0, "end": 1469.0, "text": " And after that, you resume the actual training and then validation."}, {"start": 1469.0, "end": 1474.0, "text": " Everything else remains according to the usual sequence."}, {"start": 1474.0, "end": 1480.0, "text": " OK, so because of that, I'm going to disable all of the breakpoints here."}, {"start": 1480.0, "end": 1493.0, "text": " I'm just going to just enable the one in the train loader, hit F5, and wait until this validation data set basically check is completed."}, {"start": 1493.0, "end": 1497.0, "text": " Here it is. We can see we are hitting the train data loader."}, {"start": 1497.0, "end": 1501.0, "text": " Again, that's something that PyTorch Lightning requires you to define."}, {"start": 1501.0, "end": 1507.0, "text": " Let's continue here. And let me just see whether I've enabled all the breakpoints."}, {"start": 1507.0, "end": 1514.0, "text": " OK, so now we're going to first OK, again, some some some PyTorch Lightning stuff."}, {"start": 1514.0, "end": 1519.0, "text": " OK, so here we are. So now what's happening is we are loading the data from our training data set."}, {"start": 1519.0, "end": 1524.0, "text": " And here we are. We end up with this example. We've done some pre-processing."}, {"start": 1524.0, "end": 1527.0, "text": " You can see here we fetch the image, blah, blah, blah."}, {"start": 1527.0, "end": 1531.0, "text": " And we end up with the example, which is a dictionary that has multiple keys."}, {"start": 1531.0, "end": 1538.0, "text": " So we have image and other ImageNet idiosyncratic like keys."}, {"start": 1538.0, "end": 1542.0, "text": " So let me show you some of those. So example image is obviously our input image."}, {"start": 1542.0, "end": 1547.0, "text": " So it's processed such that we have 256, 256, and three channels."}, {"start": 1547.0, "end": 1552.0, "text": " We also have if I do example, let's say class label. Whoops."}, {"start": 1552.0, "end": 1558.0, "text": " I need to make it a string. So class label is going to be, as you can see, label zero."}, {"start": 1558.0, "end": 1563.0, "text": " So our data set only has 10 labels. We are not using the full ImageNet."}, {"start": 1563.0, "end": 1570.0, "text": " That's why zero is highly probable. Then let me show you one more human label."}, {"start": 1570.0, "end": 1576.0, "text": " Obviously, just a human readable label of this ImageNet class."}, {"start": 1576.0, "end": 1579.0, "text": " So think of think of whatever that is. And that's it."}, {"start": 1579.0, "end": 1582.0, "text": " OK, and this is where it starts getting interesting. So so here we are."}, {"start": 1582.0, "end": 1586.0, "text": " We have a batch that was provided to us by PyTorch Lightning."}, {"start": 1586.0, "end": 1591.0, "text": " Again, we have image in there. The shape is familiar 256, 256."}, {"start": 1591.0, "end": 1597.0, "text": " And now we do this get input. It's just going to fetch the image, do some permutation,"}, {"start": 1597.0, "end": 1599.0, "text": " make sure that the memory is contiguous, blah, blah, blah."}, {"start": 1599.0, "end": 1602.0, "text": " And now we're doing self and we pass inputs."}, {"start": 1602.0, "end": 1606.0, "text": " So this is a fancy way of saying call the forward method of this class."}, {"start": 1606.0, "end": 1612.0, "text": " And let's see. So we are dealing with we're dealing with all the encoders here, obviously."}, {"start": 1612.0, "end": 1618.0, "text": " And the forward class is here. So let me just do F10 and we hit this part."}, {"start": 1618.0, "end": 1623.0, "text": " So first part is obviously we do the encoding and then we do the sampling and then we do the decoding."}, {"start": 1623.0, "end": 1628.0, "text": " OK, so let's dig into the decoder. Here it is."}, {"start": 1628.0, "end": 1631.0, "text": " Here is the decoding logic, some down sampling, blah, blah, blah."}, {"start": 1631.0, "end": 1636.0, "text": " Again, I'm going to skip everything here because as I already told you, encoder logic is fairly simple."}, {"start": 1636.0, "end": 1641.0, "text": " We just do a forward pass through it and we end up with a latent space representation."}, {"start": 1641.0, "end": 1645.0, "text": " So what's the dimensionality of this representation?"}, {"start": 1645.0, "end": 1652.0, "text": " You can see it's 6, 64, 64. And we expected three because if you recall, let me show you the."}, {"start": 1652.0, "end": 1657.0, "text": " Well, we are using such a config where we expect 64, 64, 3."}, {"start": 1657.0, "end": 1664.0, "text": " Six is there because we are actually returning the mean and the standard deviation or the variance here."}, {"start": 1664.0, "end": 1668.0, "text": " And then we are sampling before we pass that sample to the decoder."}, {"start": 1668.0, "end": 1673.0, "text": " So that's just a detail worth mentioning. They did experiment with different types of auto encoders."}, {"start": 1673.0, "end": 1678.0, "text": " One of them we are currently working with is the KL regularized auto encoder."}, {"start": 1678.0, "end": 1683.0, "text": " They were also playing with quantized versions, but we really don't care about all of those details."}, {"start": 1683.0, "end": 1687.0, "text": " There is just too many things going on. OK, some processing of that representation."}, {"start": 1687.0, "end": 1692.0, "text": " We end up with moments when they say moment, they literally mean zero within first moment."}, {"start": 1692.0, "end": 1700.0, "text": " Like so the mean and the and the and the variance. So now we pass that into this diagonal Gaussian distribution."}, {"start": 1700.0, "end": 1705.0, "text": " So we're going to basically form distribution here. We're going to name it posterior."}, {"start": 1705.0, "end": 1708.0, "text": " So let me kind of enter there. You can see we just chunk."}, {"start": 1708.0, "end": 1713.0, "text": " We just split into two parts that representation and we end up with mean and log variance."}, {"start": 1713.0, "end": 1720.0, "text": " So that's what we're actually returning. And then some clamping exponentiation such that we end up with standard deviation variance."}, {"start": 1720.0, "end": 1723.0, "text": " And that's it. OK, just a Gaussian. That's all."}, {"start": 1723.0, "end": 1731.0, "text": " After that, we have to sample and sampling is simply because Gaussians are such a such a nice mathematical object."}, {"start": 1731.0, "end": 1739.0, "text": " You basically just have to take the mean and add to it standard deviation multiplied by the normal noise from the normal Gaussian."}, {"start": 1739.0, "end": 1743.0, "text": " That's it. OK, and we end up with the representation."}, {"start": 1743.0, "end": 1749.0, "text": " So now let's see that should now have three channels, I guess. So three sixty four sixty four as expected."}, {"start": 1749.0, "end": 1754.0, "text": " OK, so now let's pass this into the decoder stage. Here we are."}, {"start": 1754.0, "end": 1757.0, "text": " We do some processing, just a conv layer decoder."}, {"start": 1757.0, "end": 1763.0, "text": " What it does is, again, just up sampling, conf attention again, trivial stuff."}, {"start": 1763.0, "end": 1767.0, "text": " I'm going to skip over that. That's kind of common knowledge."}, {"start": 1767.0, "end": 1774.0, "text": " OK, so we end up with output output should be because this is an autoencoder should have the same shape as the input image."}, {"start": 1774.0, "end": 1780.0, "text": " So that's two fifty six, two fifty six, three. Let's just print out the shape so that we are."}, {"start": 1780.0, "end": 1785.0, "text": " Yeah, OK, we can we can see here that the shape is as expected."}, {"start": 1785.0, "end": 1793.0, "text": " Let's continue. We now return that decoded output and we return the posterior object, which is the Gaussian."}, {"start": 1793.0, "end": 1800.0, "text": " And that's this part. OK, so now you'll see there are two if statements here."}, {"start": 1800.0, "end": 1806.0, "text": " One is when the optimizer index is zero and the other one is when the optimizer index is one."}, {"start": 1806.0, "end": 1812.0, "text": " The reason we have those parameters is because we have two different optimizers."}, {"start": 1812.0, "end": 1815.0, "text": " Let me just see whether where is that function. So configure optimizers."}, {"start": 1815.0, "end": 1818.0, "text": " I didn't show you this one, so we didn't step into it."}, {"start": 1818.0, "end": 1822.0, "text": " But simply, simply put, we have two optimizers."}, {"start": 1822.0, "end": 1826.0, "text": " One is Adam and that Adam is going to be optimizing encoders and decoders here."}, {"start": 1826.0, "end": 1830.0, "text": " And the second optimizer is going to be optimizing the discriminator weights."}, {"start": 1830.0, "end": 1835.0, "text": " So we saw that when we instantiated the discriminator, I told you that it's going to be trainable."}, {"start": 1835.0, "end": 1842.0, "text": " And here the other optimizer cares about updating discriminator weights, whereas this can be treated as a generator."}, {"start": 1842.0, "end": 1847.0, "text": " So the autoencoder part is the generator. The discriminator is obviously discriminator."}, {"start": 1847.0, "end": 1852.0, "text": " And that's that's it. OK, so let's go back here."}, {"start": 1852.0, "end": 1859.0, "text": " So PyTorch Lightning makes it easy for us to do this type of gain loss computation."}, {"start": 1859.0, "end": 1867.0, "text": " So first we'll step into this branch and then it will literally return called the training step again with optimizer index set to one."}, {"start": 1867.0, "end": 1872.0, "text": " And then we'll train the discriminator. So that's everything is kind of handled for us by the framework."}, {"start": 1872.0, "end": 1875.0, "text": " OK, this is where the whole."}, {"start": 1875.0, "end": 1879.0, "text": " Brain of this autoencoder training is going to happen."}, {"start": 1879.0, "end": 1883.0, "text": " So let's dig into this code. This is very important case."}, {"start": 1883.0, "end": 1887.0, "text": " We get the last layer that's going to be used for some lambda calculation."}, {"start": 1887.0, "end": 1890.0, "text": " You'll see that in a second. So here we are. We are inside of the loss."}, {"start": 1890.0, "end": 1893.0, "text": " Let's see how the loss looks like for the other encoder."}, {"start": 1893.0, "end": 1900.0, "text": " So first of all, we have the reconstruction loss. It's simply as you can see here, we subtract the inputs, which are the inputs image."}, {"start": 1900.0, "end": 1907.0, "text": " So let's let me just kind of make sure that this is non-py or by Torch tensor, I guess."}, {"start": 1907.0, "end": 1912.0, "text": " Yeah. And then let's just see the shape. The shape should be 256."}, {"start": 1912.0, "end": 1916.0, "text": " Yeah, everything's fine there. We have reconstructions. We literally just subtract them."}, {"start": 1916.0, "end": 1919.0, "text": " And we obviously want to make this difference as small as possible."}, {"start": 1919.0, "end": 1927.0, "text": " So that makes a lot of sense. We're just doing simple image space like emmits type of a loss."}, {"start": 1927.0, "end": 1930.0, "text": " OK, now we have perceptual loss. This is the interesting part."}, {"start": 1930.0, "end": 1934.0, "text": " So we pass the inputs and the reconstructions again."}, {"start": 1934.0, "end": 1942.0, "text": " But this time we do not compare them in the image space. Instead, we compare them in the latent space of the VGG of the pre-trained VGG network."}, {"start": 1942.0, "end": 1946.0, "text": " So let me show you that thing as well. OK, so here we are."}, {"start": 1946.0, "end": 1952.0, "text": " So we are again in this LP IPS loss and we are in the forward step."}, {"start": 1952.0, "end": 1958.0, "text": " So first thing they do is they have the scaling layer and that scaling is just going to subtract some mean."}, {"start": 1958.0, "end": 1962.0, "text": " And so let me just find that for a second. So scaling layer is here."}, {"start": 1962.0, "end": 1968.0, "text": " So literally there is some shift and scale. So computations going on."}, {"start": 1968.0, "end": 1971.0, "text": " I'm not sure whether this is from ImageNet statistics or not."}, {"start": 1971.0, "end": 1979.0, "text": " Like if somebody knows, let me know. I think this is from ImageNet and that will make sense since we are now using ImageNet data set as well."}, {"start": 1979.0, "end": 1986.0, "text": " So that would make sense. We just kind of normalize our input tensors and then we pass them through the VGG."}, {"start": 1986.0, "end": 1992.0, "text": " And so we can see here what the VGG forward passing entails."}, {"start": 1992.0, "end": 2001.0, "text": " So literally we're going to return some intermediate representations and we're going to wrap those up into this named top object and return it back."}, {"start": 2001.0, "end": 2005.0, "text": " So let me hit F10. We are here. We're going to return all of that."}, {"start": 2005.0, "end": 2011.0, "text": " We're going to hit it again because we have two computations, both for the input as well as well as for the reconstruction."}, {"start": 2011.0, "end": 2020.0, "text": " Now we kind of cluster all of these layers that are going to reduce the number of channels together, just some syntactic sugar."}, {"start": 2020.0, "end": 2025.0, "text": " And finally, let's see. Here is where the magic is happening."}, {"start": 2025.0, "end": 2031.0, "text": " So what we do is we normalize the tensor, normalize just going to basically divide it by the L2 norm."}, {"start": 2031.0, "end": 2038.0, "text": " OK, and then we do, as you can see here again, we just do we subtract them and we do the square."}, {"start": 2038.0, "end": 2042.0, "text": " So it's an MSC loss without the M part because we're not doing the mean part yet."}, {"start": 2042.0, "end": 2050.0, "text": " We're just doing this and we repeat that for all of the representations because we will have like five representations from the VGG."}, {"start": 2050.0, "end": 2054.0, "text": " So that's why we'll have five iterations of this loop. And now we're going to break out of it."}, {"start": 2054.0, "end": 2065.0, "text": " And now we're here. OK, so now what happens is we are going to pass those differences into these layers that are going to reduce the number of channels."}, {"start": 2065.0, "end": 2069.0, "text": " So let me let me let me let me explain what I mean by that. So here here is this diff."}, {"start": 2069.0, "end": 2076.0, "text": " So here is the difference in the that we that we've done among the features and the shape is going to be what?"}, {"start": 2076.0, "end": 2084.0, "text": " So here is the shape after we apply this layer. So after we apply this, we'll just end up with a single channel."}, {"start": 2084.0, "end": 2093.0, "text": " So if I put zero here and if I put zero here and if I just do the shape, you can see we have one one two fifty six to fifty six."}, {"start": 2093.0, "end": 2096.0, "text": " OK, so that's the sole purpose of this of this layer."}, {"start": 2096.0, "end": 2103.0, "text": " And finally, we do the special spatial average, which is simply a mean a mean across dimensions."}, {"start": 2103.0, "end": 2107.0, "text": " And so we end up with a scalar here. And that's it. That's that's how the perceptual loss looks like."}, {"start": 2107.0, "end": 2114.0, "text": " It's fairly simple. And I'm going to just keep on stepping over five times there and then I'm going to keep the five."}, {"start": 2114.0, "end": 2121.0, "text": " And that's it. So now we end up with this this array of well, it's actually."}, {"start": 2121.0, "end": 2128.0, "text": " Yeah, we've accumulated we've aggregated the values here by doing the sum operator and we end up with a single number."}, {"start": 2128.0, "end": 2135.0, "text": " And that's the perceptual loss. That's it, guys. The only part that's kind of new here for me personally are these layers."}, {"start": 2135.0, "end": 2142.0, "text": " You could as well skip that part and simply do like a mean operation across these differences."}, {"start": 2142.0, "end": 2148.0, "text": " So just MSC loss directly in the feature space. And that's it. But this is some type of modification."}, {"start": 2148.0, "end": 2152.0, "text": " I'm not sure why and where there are some ablations, but that's the perceptual loss."}, {"start": 2152.0, "end": 2159.0, "text": " OK, finally, we formed the reconstruction loss as a weighted sum of the reconstruction loss from the image space."}, {"start": 2159.0, "end": 2164.0, "text": " Plus, we grab the perceptual loss here and we have some weights one point zero here."}, {"start": 2164.0, "end": 2170.0, "text": " OK, so they are equally weighted. OK, next up, since this is zero, this is going to be zero."}, {"start": 2170.0, "end": 2175.0, "text": " Let me just kind of double check. And because of that exponent raised to power zero is one."}, {"start": 2175.0, "end": 2179.0, "text": " So it's just going to be a neutral operation. So this is doing nothing."}, {"start": 2179.0, "end": 2185.0, "text": " And this is also going to skip here. And here we just do some rescaling. That's it."}, {"start": 2185.0, "end": 2191.0, "text": " Summation, blah, blah, blah. And that's our final reconstruction loss. That's the loss there."}, {"start": 2191.0, "end": 2198.0, "text": " Now we do the because remember, we return the posterior, which is the Gaussian from the latent space of the odd encoder."}, {"start": 2198.0, "end": 2203.0, "text": " And we're just going to compute the KL divergence. So that's going to be the regularizer component."}, {"start": 2203.0, "end": 2212.0, "text": " So I'm going to hit F10 here. You can see it's simply computing the KL divergence using the mean information variance and log variance information."}, {"start": 2212.0, "end": 2217.0, "text": " OK, and we end up with a KL loss there. Some rescaling again."}, {"start": 2217.0, "end": 2224.0, "text": " And now we enter the branch where we train the generator, I assume. Let's see. OK, all is good so far."}, {"start": 2224.0, "end": 2233.0, "text": " OK, so we passed the reconstructions, which are the fake images through the discriminator and we get logits fake. OK."}, {"start": 2233.0, "end": 2239.0, "text": " And then what we do is we have a minus and we do a mean across those weights."}, {"start": 2239.0, "end": 2245.0, "text": " So let me show you the shape as I told you. That's a special discriminator. That's like a patch based discriminator."}, {"start": 2245.0, "end": 2252.0, "text": " So that means that the shape is going to be maybe I think it's 32 times 32 or something. So let me see that."}, {"start": 2252.0, "end": 2257.0, "text": " So it's 30 30. So that's how many patches we have. And we just do a mean across them."}, {"start": 2257.0, "end": 2268.0, "text": " And by putting a minus here, we are literally forcing we will be tweaking the generator weights in such a way such that the discriminator gives a high value,"}, {"start": 2268.0, "end": 2273.0, "text": " which means it thinks, quote unquote, that those images are real."}, {"start": 2273.0, "end": 2281.0, "text": " So, again, that's just your your your GAN standard GAN GAN stuff. Not nothing, nothing super complex there. If you're familiar with GANs."}, {"start": 2281.0, "end": 2288.0, "text": " OK. And now there is some adaptive weight calculation. Let me see. So what's going to happen there?"}, {"start": 2288.0, "end": 2300.0, "text": " OK. Yeah. So I remember this is going to make make sure that we we are waiting the GAN part of the loss and the reconstruction loss appropriately."}, {"start": 2300.0, "end": 2305.0, "text": " Let me show you the paper formula for this one is going to make a bit more sense. OK, guys, here we are."}, {"start": 2305.0, "end": 2312.0, "text": " This is the VQ GAN paper I showed that in the beginning of the video. And this is basically the formula we're computing."}, {"start": 2312.0, "end": 2320.0, "text": " We are taking the gradients of the reconstruction loss with respect to the last layer of the decoder weights."}, {"start": 2320.0, "end": 2328.0, "text": " We divide that by the gradients of the GAN loss with respect to the weights of the last layer of the decoder again."}, {"start": 2328.0, "end": 2337.0, "text": " What's the reasoning behind this? The reasoning is the following. If the gradients are super big for the reconstruction loss and they are smaller for the GAN loss,"}, {"start": 2337.0, "end": 2343.0, "text": " then this is going to be a big number, which is going to going to put a bigger weight on the GAN loss."}, {"start": 2343.0, "end": 2355.0, "text": " So by doing that, we make sure that the network is learning from both losses and that one loss is not overwhelming the other loss when it comes to the contribution to the gradients."}, {"start": 2355.0, "end": 2360.0, "text": " So that's the rough logic. And you can see that that's exactly what we're computing here."}, {"start": 2360.0, "end": 2367.0, "text": " So you can see NLL loss is the reconstruction loss of the last layer weights. And that's it. We get the gradients there."}, {"start": 2367.0, "end": 2374.0, "text": " Then we compute the gradients for the G loss, which is the GAN loss. That's it. And now we just normalize them."}, {"start": 2374.0, "end": 2382.0, "text": " We do the norm. We divide them as well. And that's it. This is clamping, blah, blah, blah, times some weight. And that's it. That's the that's that weight."}, {"start": 2382.0, "end": 2391.0, "text": " OK, I'm now going to return back to the code here. Now, this is going to be zero for the initial 50,000 iterations or something."}, {"start": 2391.0, "end": 2404.0, "text": " The reason being they don't want to they first want to train the autoencoder and ignore the GAN loss such that they can form some representations and for the stability sake and only then slowly start using the GAN loss."}, {"start": 2404.0, "end": 2412.0, "text": " So let me let me show you what I mean by that. Let me enter here. You can see that until we pass the global threshold. So I'm going to enter here."}, {"start": 2412.0, "end": 2420.0, "text": " You'll see that the weight will be zero. So the weight will be set to zero until the global step crosses some certain threshold."}, {"start": 2420.0, "end": 2431.0, "text": " And because of that, this is zero. And you can see it's used here. So that means this thing is kind of toggled off for the good portion of the of the of the beginning of the training."}, {"start": 2431.0, "end": 2440.0, "text": " So that means we only use the KL regular regularize the loss here and we use the reconstruction loss here as the final loss. That's it."}, {"start": 2440.0, "end": 2448.0, "text": " I know this was a mouthful, but like hopefully it makes sense. Now we just do some blah, blah, blah accumulation of those and we return back the loss."}, {"start": 2448.0, "end": 2455.0, "text": " And that's it. We do some logging. We return the autoencoder loss. And that's it. Now we're going to hit the training step again."}, {"start": 2455.0, "end": 2465.0, "text": " This time we're going to be training not the generator, but instead we are going to be training the discriminator. Let's see how that's going to look like again."}, {"start": 2465.0, "end": 2476.0, "text": " Blah, blah, blah. I'm going to skip across these steps. I'm going to do basically wrong disabled break points. We're going to do a four pass again."}, {"start": 2476.0, "end": 2486.0, "text": " This is kind of suboptimal. The there must be some way to optimize this such that we don't have to do because we are basically sending the same images here again."}, {"start": 2486.0, "end": 2495.0, "text": " And that doesn't make much sense. And now let me let me return. Let me enable back all the break points. Now we enter this this branch here."}, {"start": 2495.0, "end": 2506.0, "text": " OK, so again, we have a loss here. All of the inputs are the same. What's different is that let me just enter here. So we're going to have the same parts here."}, {"start": 2506.0, "end": 2513.0, "text": " Reconstruction loss. Everything else remains the same. So the interesting part that we care about is here. So we want to go here."}, {"start": 2513.0, "end": 2521.0, "text": " So I'm going to disable break points, enable just this one, hit F5 and we enter this branch this time. OK, so here we are."}, {"start": 2521.0, "end": 2531.0, "text": " Now we pass the inputs to discriminator to get the real images and we pass the reconstructions to get the fake images, the largest of the fake images."}, {"start": 2531.0, "end": 2542.0, "text": " OK, again, we just have this factor which is going to be zero initially, which means this loss will not be enabled in the first part of the training."}, {"start": 2542.0, "end": 2551.0, "text": " Later, it's going to gradually kick in. And basically what we do here is a hinge loss between the largest of the real and the fake images."}, {"start": 2551.0, "end": 2562.0, "text": " And basically that's it. That's that's the gain loss. So this is going to train the discriminator such that the discriminator learns the difference between real and fake images."}, {"start": 2562.0, "end": 2569.0, "text": " Consequently, that's going to lead to better or the encoder because we're losing that we are using that discriminator to train the order."}, {"start": 2569.0, "end": 2578.0, "text": " Guys, that's it. That was the training of the order. Hopefully that was interesting and made sense."}, {"start": 2578.0, "end": 2589.0, "text": " Now I'm going to just stop this training because that's pretty much it. OK, I've hit F5 just to show you that now we're just going to iterate across patches and keep on repeating the same things we've just seen."}, {"start": 2589.0, "end": 2596.0, "text": " So that's why I'm going to stop this training right now. We've seen how the order encoder training looks like."}, {"start": 2596.0, "end": 2602.0, "text": " I'm going to show you quickly the formulas from the VQGAN paper just to to consolidate the knowledge here."}, {"start": 2602.0, "end": 2606.0, "text": " And then we're going to step into understanding how the diffusion, the unit model is being trained."}, {"start": 2606.0, "end": 2611.0, "text": " OK, guys, quickly coming to the paper. Here are the formulas for the VQGAN paper."}, {"start": 2611.0, "end": 2624.0, "text": " So originally how the VQGAN was trained was it had these code books of discrete basically vectors and they had the reconstruction loss component."}, {"start": 2624.0, "end": 2635.0, "text": " You can see here plus these losses here were called commitment losses and they were used to train the code book and the encoder."}, {"start": 2635.0, "end": 2648.0, "text": " What changed in the VQGAN is that they started using so you can see here they're using instead of L2 loss, they use a perceptual loss and they introduce an adversarial training procedure with a patch based discriminator."}, {"start": 2648.0, "end": 2654.0, "text": " So that's everything we've seen so far. And you can see that the final loss looks like this."}, {"start": 2654.0, "end": 2659.0, "text": " So there is the GAN component weighted by this lambda. We've seen all of these."}, {"start": 2659.0, "end": 2665.0, "text": " And there is this component here that basically consists out of the reconstruction loss and the perceptual loss."}, {"start": 2665.0, "end": 2673.0, "text": " And additionally, in this in the latent diffusion model, they've introduced the KL divergence regularization."}, {"start": 2673.0, "end": 2682.0, "text": " So basically. Bottom line is the order encoder used for the LDM paper is a small modification of what they've already done."}, {"start": 2682.0, "end": 2686.0, "text": " The same authors in the VQGAN paper. That's it. Let's go back to the code."}, {"start": 2686.0, "end": 2698.0, "text": " Wondering for a second about why these losses make sense, even though they are, they are not by any stretch of imagination, probably an optimal solution to how we should be training our models."}, {"start": 2698.0, "end": 2704.0, "text": " Let's just think about it for a second. So we have the reconstruction loss in the image space and we have the perceptual loss."}, {"start": 2704.0, "end": 2714.0, "text": " Those basically make sure that we are we are reconstructing images correctly so that we can we learn how to not lose information when we go through the bottleneck part."}, {"start": 2714.0, "end": 2720.0, "text": " OK, then we have the GAN component, which makes sure that the images look very realistic."}, {"start": 2720.0, "end": 2724.0, "text": " So that's additionally kind of enforcing the reconstruction."}, {"start": 2724.0, "end": 2741.0, "text": " And finally, we have the KL divergence loss that's just going to be regularizing the latent space of our order encoder such that we can later be able to smoothly go through that space and be able to have meaningful representations."}, {"start": 2741.0, "end": 2744.0, "text": " So that's the basic idea. Like we are. Yeah."}, {"start": 2744.0, "end": 2758.0, "text": " OK, having said that, let's go to the long Jason. Let's modify the argument such that we are now training. We're now training the diffusion part and not the order."}, {"start": 2758.0, "end": 2765.0, "text": " So I'm just going to remove this part here, remove this space here and paste this back here."}, {"start": 2765.0, "end": 2771.0, "text": " So that's everything you need to do. Now we're training the diffusion model."}, {"start": 2771.0, "end": 2775.0, "text": " So let's go back here. The difference will now be that they are not using with this config."}, {"start": 2775.0, "end": 2780.0, "text": " They're not using the same order encoder. They're using a different one with the with the quantization."}, {"start": 2780.0, "end": 2784.0, "text": " But like that doesn't matter. We're going to focus on on important parts only."}, {"start": 2784.0, "end": 2791.0, "text": " OK, so I'm going to hit F hit the training here and let's start analyzing the code again."}, {"start": 2791.0, "end": 2795.0, "text": " I'm going to focus only on instantiating the models and on the training loop."}, {"start": 2795.0, "end": 2801.0, "text": " That's it. I'm going to skip everything else because we've seen all of that. We've seen the config as well."}, {"start": 2801.0, "end": 2805.0, "text": " So let's just go to the model instantiation. So let's go here."}, {"start": 2805.0, "end": 2810.0, "text": " And now let's let me just make sure that everything is enabled. I think it already is."}, {"start": 2810.0, "end": 2815.0, "text": " But yeah. OK, so here we are late in diffusion like object."}, {"start": 2815.0, "end": 2822.0, "text": " We're starting to instantiate everything we need. So one of those things is going to be unit architecture."}, {"start": 2822.0, "end": 2827.0, "text": " OK, so let's start here. Let's see what's going on. We can ignore all of this."}, {"start": 2827.0, "end": 2834.0, "text": " So here is the first interesting part. We're going to be initializing the superclass and the superclass is this DDPM."}, {"start": 2834.0, "end": 2838.0, "text": " OK, so the DPM is the denoising diffusion probabilistic model."}, {"start": 2838.0, "end": 2843.0, "text": " So that's the original diffusion paper that made diffusion kind of practical."}, {"start": 2843.0, "end": 2847.0, "text": " OK, so let's enter there. So let's see what's going on there."}, {"start": 2847.0, "end": 2856.0, "text": " OK, so here we are predicting you can see we're running in EPS prediction mode, which means we're predicting that noise instead of."}, {"start": 2856.0, "end": 2862.0, "text": " Well, there are some other things you can predict like X zero, et cetera. OK, we can skip all of this."}, {"start": 2862.0, "end": 2870.0, "text": " We can skip all of this. Now there is this diffusion wrapper. And that's where we actually start making the unit."}, {"start": 2870.0, "end": 2875.0, "text": " So here you can see here unit model is constructed here. So let's construct the unit."}, {"start": 2875.0, "end": 2878.0, "text": " Let's see how it looks like again in some of my previous videos."}, {"start": 2878.0, "end": 2884.0, "text": " I've been going through in a lot of detail through how unit is constructed so you can go through that if you want."}, {"start": 2884.0, "end": 2890.0, "text": " Here I'm just going to kind of put the scheme. And by the way, I love the I love the statements in this in this code base."}, {"start": 2890.0, "end": 2895.0, "text": " Fool, you forgot to include the dimension of your cross attention conditioning. Very cool."}, {"start": 2895.0, "end": 2899.0, "text": " Yeah, you can you can tell it's a production code. OK, so let's let's continue here."}, {"start": 2899.0, "end": 2903.0, "text": " I think we can skip across all of these details."}, {"start": 2903.0, "end": 2908.0, "text": " The important parts are these time step in bad sequential objects."}, {"start": 2908.0, "end": 2918.0, "text": " What they basically make sure is that we can later pass time step information or conditional information into various sub modules."}, {"start": 2918.0, "end": 2922.0, "text": " So let me kind of click F12 there, enter the definition."}, {"start": 2922.0, "end": 2927.0, "text": " You can see that depending on the layer type, they'll sometimes be passing the conditioning information,"}, {"start": 2927.0, "end": 2934.0, "text": " sometimes the embedded information, sometimes just the input features, image features, and that's it."}, {"start": 2934.0, "end": 2942.0, "text": " So it might be interesting to for me to just show you one small thing."}, {"start": 2942.0, "end": 2949.0, "text": " And that's the following. So there are these blocks that integrate the conditional information, and I think those might be interesting."}, {"start": 2949.0, "end": 2955.0, "text": " So here spatial transformer. So that's the module that's going to be integrating the conditional information into the unit."}, {"start": 2955.0, "end": 2962.0, "text": " So I'm going to hit F12 there and I'm just going to add I'm just going to add like basically a break point there."}, {"start": 2962.0, "end": 2968.0, "text": " And let's hit F10. Let's continue everything else. We don't care about it."}, {"start": 2968.0, "end": 2973.0, "text": " Really, we can just construct the unit and let's go to the end here."}, {"start": 2973.0, "end": 2979.0, "text": " So this is the end of the unit definition. Quite a long definition definition, as you can tell."}, {"start": 2979.0, "end": 2982.0, "text": " So I'm going to skip over it and that's it."}, {"start": 2982.0, "end": 2989.0, "text": " So we're using cross attention to do the conditioning, the conditional information integration."}, {"start": 2989.0, "end": 2993.0, "text": " So, yeah. OK, so let's continue here. OK, that's it."}, {"start": 2993.0, "end": 2999.0, "text": " Counting the parameters, nothing fancy. We don't care about that as well."}, {"start": 2999.0, "end": 3006.0, "text": " That's just the exponential moving average. Not the fundamental part why this model is working."}, {"start": 3006.0, "end": 3010.0, "text": " So I'm going to skip across all of these. Now we're registering the schedule."}, {"start": 3010.0, "end": 3014.0, "text": " So this is the important part. And I've covered how this exactly works."}, {"start": 3014.0, "end": 3020.0, "text": " Like I've been doing side by side comparison of formulas and of code."}, {"start": 3020.0, "end": 3023.0, "text": " So do check out. I'm going to link those video cards somewhere here."}, {"start": 3023.0, "end": 3029.0, "text": " But the diffusion playlist is the best place to start if you want to understand a bit better why those work."}, {"start": 3029.0, "end": 3031.0, "text": " Otherwise, this video would be like five hours long or something."}, {"start": 3031.0, "end": 3038.0, "text": " So let me let me enter here. Let me just show you how this roughly looks like."}, {"start": 3038.0, "end": 3047.0, "text": " So you can see here a bunch of those alphas and alpha like the cumulative products and all of those variations of the formulas."}, {"start": 3047.0, "end": 3054.0, "text": " Basically nothing is learnable here. These are just the weights of the scheduler that we need to get diffusion to work."}, {"start": 3054.0, "end": 3057.0, "text": " So I'm going to skip across all of these and that's it."}, {"start": 3057.0, "end": 3065.0, "text": " That's an important part. But like something I've covered previously and just bunch of formulas, you wouldn't get any insight from me going through it."}, {"start": 3065.0, "end": 3070.0, "text": " So, yeah, I'm going to skip over that. OK, we are back in the latent diffusion."}, {"start": 3070.0, "end": 3075.0, "text": " So we generated the unit we generated the schedule."}, {"start": 3075.0, "end": 3079.0, "text": " Now let's continue and see what else is interesting here."}, {"start": 3079.0, "end": 3087.0, "text": " I'm going to skip across all of these. So because now we are training the model in a holistic fashion, we obviously have to instantiate the first stage."}, {"start": 3087.0, "end": 3092.0, "text": " And by first stage, they mean the auto encoder. So let's again just briefly go through this one."}, {"start": 3092.0, "end": 3098.0, "text": " This time we are forming this VQ model and not the auto encoder KL. So that's a that's a difference."}, {"start": 3098.0, "end": 3106.0, "text": " So let's just do that. So here we are. We are instantiating the VQ model."}, {"start": 3106.0, "end": 3112.0, "text": " Here we are. And that's going to call the VQ model here. So let's just kind of start entering there."}, {"start": 3112.0, "end": 3117.0, "text": " OK, so here we are. Let's see what's the main difference. We still have the encoder."}, {"start": 3117.0, "end": 3123.0, "text": " I'm going to just toggle off all the breakpoints. We have the encoder. We have the decoder. Nothing have changed there."}, {"start": 3123.0, "end": 3131.0, "text": " The only difference is so we have a loss, which is going to be identity in this this time because we are not training the auto encoder."}, {"start": 3131.0, "end": 3139.0, "text": " We'll just be we'll be just loading the pre-trained weights. So let me enable all the breakpoints and let's enter this part."}, {"start": 3139.0, "end": 3145.0, "text": " So let's see what's going on there. Some beddings, blah, blah, blah."}, {"start": 3145.0, "end": 3149.0, "text": " Well, it's not blah, blah, blah. Embeddings are actually what's important in this model. So this is a codebook."}, {"start": 3149.0, "end": 3157.0, "text": " You can see there is sixteen thousand three hundred eighty four codebook vectors and each of those has four dimensionality of four."}, {"start": 3157.0, "end": 3164.0, "text": " And that's what's being used to do the quantizations later in the forward step. We'll see how it looks like a bit later."}, {"start": 3164.0, "end": 3169.0, "text": " OK, so that's the quantized part. Now we have the conv layer, same as with the auto encoder KL."}, {"start": 3169.0, "end": 3175.0, "text": " Nothing has changed there. Blah, blah, blah. We can skip that. We can skip all of this."}, {"start": 3175.0, "end": 3182.0, "text": " And now we initialize from the pre-trained checkpoint. I'm just going to scheme over all of that."}, {"start": 3182.0, "end": 3190.0, "text": " We are just basically doing the initialization of the auto encoder because remember how the whole logic works."}, {"start": 3190.0, "end": 3196.0, "text": " You first pre-trained the auto encoder and then you basically freeze it and you use its latent space."}, {"start": 3196.0, "end": 3199.0, "text": " And now you train the unit and the conditional model and everything. That's that's it."}, {"start": 3199.0, "end": 3204.0, "text": " That's why we are loading the weights here. OK, and that's it."}, {"start": 3204.0, "end": 3217.0, "text": " Now we do some set the eval mode and basically now we set the gradients to false everywhere and we can just continue on with the execution here."}, {"start": 3217.0, "end": 3225.0, "text": " Let me hit F5. We exit the condition, instantiate the first stage function and now we instantiate the conditional stage."}, {"start": 3225.0, "end": 3232.0, "text": " So this time let's enter this one. I think we are just going to have a class information."}, {"start": 3232.0, "end": 3240.0, "text": " So you can see here a class embedder is the type of a conditioning model that will be instantiating here."}, {"start": 3240.0, "end": 3252.0, "text": " So let's enter there. You can see it's simply thousand classes because we are dealing with image and embedding dimension and then literally just does the embedding in the forward pass."}, {"start": 3252.0, "end": 3258.0, "text": " That's it. That's how the conditional stage model looks like. Let me remind you what."}, {"start": 3258.0, "end": 3266.0, "text": " So that's basically let me show you the diagram here. That's this part in the image."}, {"start": 3266.0, "end": 3272.0, "text": " So this part here is what we've just instantiated and this is the unit. OK, so we are in the stage two."}, {"start": 3272.0, "end": 3278.0, "text": " OK, let's go back here. Let me keep on stepping over here and that's it, guys."}, {"start": 3278.0, "end": 3283.0, "text": " Now I'm going to skip across. This is the main function again. I'm going to skip everything here."}, {"start": 3283.0, "end": 3290.0, "text": " I'm going to also skip the data because that's again just image net and I'm going to stop at the trainer fit here."}, {"start": 3290.0, "end": 3299.0, "text": " So hitting F5 waiting for everything to. Oops. I'm going to have to disable the break points and only then will this work."}, {"start": 3299.0, "end": 3308.0, "text": " So disable and just enable this one. Hit the five. Get to trainer fit and then we're going to start start analyzing how this works."}, {"start": 3308.0, "end": 3315.0, "text": " OK, so enabling the break points and we're going to hit the validation batch as usually."}, {"start": 3315.0, "end": 3321.0, "text": " OK, no, actually. OK, what I've done here is I've added a break point to the configure optimizers this time."}, {"start": 3321.0, "end": 3330.0, "text": " We're just using Adam W here and nothing else is important. I'm going to skip over this again."}, {"start": 3330.0, "end": 3334.0, "text": " PyTorch lightning stuff. So this is not important. Here is the validation loader."}, {"start": 3334.0, "end": 3340.0, "text": " So I'm going to disable all break points and just end up here hitting F5."}, {"start": 3340.0, "end": 3345.0, "text": " We're going to end up in the training part of the of the training. OK, here we are."}, {"start": 3345.0, "end": 3353.0, "text": " I'm going to step over this and now I'm going to enable the break points."}, {"start": 3353.0, "end": 3359.0, "text": " Let's just go through this idiosyncratic part again. And now we're loading the data."}, {"start": 3359.0, "end": 3362.0, "text": " OK, so we are loading the data. Everything remains the same."}, {"start": 3362.0, "end": 3368.0, "text": " We have example that has keys such as images and labels, blah, blah, blah."}, {"start": 3368.0, "end": 3371.0, "text": " I stepped through that part and we end up in the important part."}, {"start": 3371.0, "end": 3376.0, "text": " And that's the training step. Again, that's a function that PyTorch lightning requires you to define."}, {"start": 3376.0, "end": 3383.0, "text": " You can see that we have our batch here and everything is the same as when we were training our encoder."}, {"start": 3383.0, "end": 3390.0, "text": " So batch image shape. You can see it's a 256 256 three channel image."}, {"start": 3390.0, "end": 3396.0, "text": " OK, let's stop. Start stepping through this shared step. That's the first part."}, {"start": 3396.0, "end": 3402.0, "text": " So we call this get input and I think that's just going to grab us the OK."}, {"start": 3402.0, "end": 3408.0, "text": " So it grabs the back the image. So you can see here we just grab the image."}, {"start": 3408.0, "end": 3416.0, "text": " And we end up with so X shape. So we have 256 256 and three. All of that is as usual."}, {"start": 3416.0, "end": 3421.0, "text": " Now we just push it to the GPU. OK, so now here's what we do."}, {"start": 3421.0, "end": 3426.0, "text": " So we encode using the first stage. So that means we don't want to deal with images anymore."}, {"start": 3426.0, "end": 3433.0, "text": " When we're training the diffusion in the LVM in the latent diffusion models, instead we want to deal with the latent space."}, {"start": 3433.0, "end": 3438.0, "text": " That's why we call the encode first stage. So let's do F10 here."}, {"start": 3438.0, "end": 3444.0, "text": " Here's the encoded first stage. And here's what it does. It basically calls first stage model."}, {"start": 3444.0, "end": 3449.0, "text": " It just calls the encode function of that model. Here is how it looks like."}, {"start": 3449.0, "end": 3457.0, "text": " So it's just going to call the encoder. So let's basically let's hit F10 and we are here."}, {"start": 3457.0, "end": 3463.0, "text": " So we are in the encoder. Everything remains. That's the same as our order encoder from the first part of the video."}, {"start": 3463.0, "end": 3470.0, "text": " So I'm going to hit F5. Just bunch of calm layers, rest blocks and down sampling stuff. So F10."}, {"start": 3470.0, "end": 3477.0, "text": " So we end up representation here. So we have now H shape. We have 32 32 4."}, {"start": 3477.0, "end": 3482.0, "text": " OK, four is the number of of latent channels and this is the spatial dimensionality."}, {"start": 3482.0, "end": 3488.0, "text": " Now we do some processing with a calm layer and we return back that representation. And that's it."}, {"start": 3488.0, "end": 3493.0, "text": " That's the encoder posterior. You can see that the shape here is this."}, {"start": 3493.0, "end": 3501.0, "text": " So it's not anymore. It's not a Gaussian distribution because these types of all the encoder models work a bit differently, but the logic is fairly similar."}, {"start": 3501.0, "end": 3507.0, "text": " So now let's call this get first stage encoding. Let's see what this is. So I'm going to hit F12 just to see."}, {"start": 3507.0, "end": 3514.0, "text": " OK, so I'm going to enter here. We can see it's not this object, so we will not sample from it."}, {"start": 3514.0, "end": 3526.0, "text": " Instead, because it's a tensor, we simply just map, create this type of variable name binding and we just scale with some constant factor."}, {"start": 3526.0, "end": 3532.0, "text": " Let me see what that number is and how it was defined. I'm not sure about it."}, {"start": 3532.0, "end": 3538.0, "text": " OK, so it's one. OK, so we can we can just ignore all that. So let's continue. So now we have our representation."}, {"start": 3538.0, "end": 3545.0, "text": " So that's the latent representation now because we have conditioning and the conditioning key is, I think, cross attention or something."}, {"start": 3545.0, "end": 3550.0, "text": " So no, it's it's class label, but we are going to integrate using the cross attention logic."}, {"start": 3550.0, "end": 3561.0, "text": " So let's step over here. And what we do is we just passed a batch because the batch contains, if you recall, a bunch of keys and among them, it contains the label."}, {"start": 3561.0, "end": 3570.0, "text": " So that's how they've implemented this. Basically, they pass more data than is needed, but we'll see how that's going to be integrated a bit later."}, {"start": 3570.0, "end": 3581.0, "text": " So let's see what's going on going on here. So we just map H.C. to C. So that's again the batch information. And then we're going to skip all of this."}, {"start": 3581.0, "end": 3590.0, "text": " And finally, we we return back. So this is the latent representation and this is the conditioning information and a bit more stuff because it's the batch information."}, {"start": 3590.0, "end": 3596.0, "text": " OK, and finally, we return back all of that. So that's the first part of the shared step function."}, {"start": 3596.0, "end": 3603.0, "text": " Again, recall that we are currently in the latent diffusion model. Blah, blah, blah."}, {"start": 3603.0, "end": 3610.0, "text": " If I scroll all the way up here, you'll see that. Oh, my God. Oh, my God. Latent diffusion."}, {"start": 3610.0, "end": 3617.0, "text": " OK, so let's let's go back here. Now we do the forward prop through the diffusion model. So let's see how that looks like."}, {"start": 3617.0, "end": 3628.0, "text": " F10. Here we are in the forward prop. We generate some time steps randomly. Basically, this is going to be thousand. So we generate randomly the time steps information."}, {"start": 3628.0, "end": 3637.0, "text": " And now we do the conditioning. So because this is trainable, we get the learned conditioning."}, {"start": 3637.0, "end": 3648.0, "text": " So let's see what's going to happen there. Basically, we just call the forward pass and we pass C. So C is as you can see, C is still a batch information."}, {"start": 3648.0, "end": 3656.0, "text": " So if I enter inside of the forward function of the class embedder, you can see that we are now going to extract the key."}, {"start": 3656.0, "end": 3661.0, "text": " So the class label from here is going to be some label of the image. So it's zero."}, {"start": 3661.0, "end": 3671.0, "text": " And then we're going to embed it using the embedding table here. So we're going to return some representation that's like what has should have however many dimensions distinct had."}, {"start": 3671.0, "end": 3678.0, "text": " I think it was hundred twenty eight or something. So C shape five hundred twelve. OK, so we return that back."}, {"start": 3678.0, "end": 3685.0, "text": " We return that back. We turn the C and here we are. So we have the C now. And finally, we pass the image."}, {"start": 3685.0, "end": 3694.0, "text": " We pass the sorry, this is not the image. X should be with was X. X should be."}, {"start": 3694.0, "end": 3699.0, "text": " X should be the latent representation, right? Yeah, it is. So we passed the latent representation."}, {"start": 3699.0, "end": 3703.0, "text": " We passed the conditioning information. We passed the T and we compute the losses."}, {"start": 3703.0, "end": 3714.0, "text": " So this is basically what we saw here. We are literally randomly sampling like the these latent representations."}, {"start": 3714.0, "end": 3721.0, "text": " Noise and time steps. That's it. Let's go back to the VS code. OK, so now we have the P losses."}, {"start": 3721.0, "end": 3726.0, "text": " This is where the whole magic of the training happens. We sample some normal noise."}, {"start": 3726.0, "end": 3733.0, "text": " So here we are. So we sample the normal noise and then we do the Q sample. So let's not do the noising process."}, {"start": 3733.0, "end": 3741.0, "text": " So we start from our pure latent representation and we add up key steps of noise on top of it."}, {"start": 3741.0, "end": 3759.0, "text": " So we simulate that. If you recall from my previous videos, there is a formula that makes us capable of doing that in a single step by just combining the start the start representation with the noise and using these basically non learnable parameters from the scheduler."}, {"start": 3759.0, "end": 3765.0, "text": " We end up with a noisy version and now we just apply. We passed the noisy, the T and the conditioning."}, {"start": 3765.0, "end": 3772.0, "text": " So that's literally that's literally this formula here. That's literally we are passing that T T and C."}, {"start": 3772.0, "end": 3777.0, "text": " So let me show you that version here so you can see this formula three in the paper in the LDM paper."}, {"start": 3777.0, "end": 3784.0, "text": " We pass these variables here and we are now passing that through the unit such that we can get a noise as the output."}, {"start": 3784.0, "end": 3796.0, "text": " OK, so let's go back to the code. Let's hit F10 and enter the apply model function. So here what happens is just some variable backing."}, {"start": 3796.0, "end": 3803.0, "text": " Nothing fundamental there. We just pass that vector that was 512 dimensional."}, {"start": 3803.0, "end": 3808.0, "text": " Whoops, actually have to extract the first because we packed it into a list. Just some details."}, {"start": 3808.0, "end": 3814.0, "text": " Nothing. Yeah, we are passing the same information. So that's the conditional vector. OK, and here we are."}, {"start": 3814.0, "end": 3818.0, "text": " Now we pass. We call the this should be the unit. This should be the unit."}, {"start": 3818.0, "end": 3825.0, "text": " Let me just if I do type on this object and this code is so nice I can do this debugging so easily."}, {"start": 3825.0, "end": 3828.0, "text": " So diffusion wrapper, which contains the unit model and the scheduler."}, {"start": 3828.0, "end": 3839.0, "text": " OK, so if I do F10, here we are. We enter the diffusion wrapper and because we have conditioning key set to cross attention,"}, {"start": 3839.0, "end": 3845.0, "text": " we're going to call the fusion model, pass the conditioning, pass the time steps, pass the latent representation."}, {"start": 3845.0, "end": 3848.0, "text": " And this is going to be automatically handled and integrated by the cross attention."}, {"start": 3848.0, "end": 3854.0, "text": " If I click F10, we should be in the forward pass of the unit model. Let's see whether that's indeed the case."}, {"start": 3854.0, "end": 3858.0, "text": " And yeah, you can see here this is the definition of unit. We are in the right spot."}, {"start": 3858.0, "end": 3865.0, "text": " And so let's now continue. So we just embed the time step information."}, {"start": 3865.0, "end": 3870.0, "text": " So now we end up with some processing on top of those temporal representations."}, {"start": 3870.0, "end": 3880.0, "text": " You can see that this is what the dimensionality of the time temporal information now is."}, {"start": 3880.0, "end": 3886.0, "text": " OK, and now we start integrating. Now we start literally going through the unit."}, {"start": 3886.0, "end": 3895.0, "text": " And you can see here we always pass the representation, the temporal embedding and the conditioning information."}, {"start": 3895.0, "end": 3906.0, "text": " And now that's where the trick comes. So this is where the special object I mentioned a couple of like 20 minutes ago or something is going to play,"}, {"start": 3906.0, "end": 3912.0, "text": " come into the picture because it's going to know exactly what to pass. So I'm going to put a breakpoint here."}, {"start": 3912.0, "end": 3916.0, "text": " I'm going to click F10 and you can see we immediately hit this one."}, {"start": 3916.0, "end": 3923.0, "text": " And now, depending on what instance is this layer, whether it's a spatial transformer or temporal block or just a pure block,"}, {"start": 3923.0, "end": 3927.0, "text": " it's going to call one of the three versions of the layer. And that's that's it."}, {"start": 3927.0, "end": 3934.0, "text": " Like now I'm going to hit F5 and that's going to hit to the it's going to hit the spatial transformer part. OK."}, {"start": 3934.0, "end": 3938.0, "text": " So we hit a special transformer. And this is where the contextual information."}, {"start": 3938.0, "end": 3942.0, "text": " So this should be five hundred twelve. This is the yeah."}, {"start": 3942.0, "end": 3945.0, "text": " So this is the conditioning information from the label."}, {"start": 3945.0, "end": 3952.0, "text": " And you can see that you basically now do simple transformer logic with the conditioning additionally here."}, {"start": 3952.0, "end": 3955.0, "text": " So we just pass. You can see here just some projections, blah, blah, blah."}, {"start": 3955.0, "end": 3961.0, "text": " We rearrange our representation such that it's suitable for transformers."}, {"start": 3961.0, "end": 3966.0, "text": " We have batch size. We have sequence size, basically flattening out the height and the width."}, {"start": 3966.0, "end": 3968.0, "text": " And we have the number of channels. OK."}, {"start": 3968.0, "end": 3977.0, "text": " And now we just pass and do the cross attention with the basically with the transformer blocks."}, {"start": 3977.0, "end": 3981.0, "text": " So let me do this. And that's it. Guys, that's it. That's it."}, {"start": 3981.0, "end": 3984.0, "text": " That's the that's the whole logic."}, {"start": 3984.0, "end": 3987.0, "text": " I'm going to hit F5 again and I'll have to remove this breakpoint."}, {"start": 3987.0, "end": 3991.0, "text": " And now let's get out of here. Let's get out of this function."}, {"start": 3991.0, "end": 3994.0, "text": " Let's just exit this function. OK."}, {"start": 3994.0, "end": 3998.0, "text": " And I'm going to put a breakpoint here. Hit F5 again."}, {"start": 3998.0, "end": 4003.0, "text": " Exit this function. And basically now I'm going to hit F5 again."}, {"start": 4003.0, "end": 4007.0, "text": " And we are exiting the this is the unit for a pass."}, {"start": 4007.0, "end": 4010.0, "text": " We're basically exiting the unit for a pass. And that's it."}, {"start": 4010.0, "end": 4015.0, "text": " So let's exit here and let's see what the output shape is."}, {"start": 4015.0, "end": 4020.0, "text": " It should be the same as the input latent representation."}, {"start": 4020.0, "end": 4024.0, "text": " So it's going to be. Whoops. Let's let's hit F10."}, {"start": 4024.0, "end": 4027.0, "text": " Now we have out. I'm not sure whether this is going to."}, {"start": 4027.0, "end": 4032.0, "text": " Yeah. So we have the same. You can see the same shape as the input representation we passed."}, {"start": 4032.0, "end": 4039.0, "text": " But this is now the prediction. This is now the noise that was put on top of that input latent representation."}, {"start": 4039.0, "end": 4042.0, "text": " So, again, simply what we've done here, we have a unit model."}, {"start": 4042.0, "end": 4048.0, "text": " We passed the input late representation. We have some time step information, some conditioning information."}, {"start": 4048.0, "end": 4052.0, "text": " We use the time step information to noise the input latent."}, {"start": 4052.0, "end": 4057.0, "text": " We combine them with the with the conditioning information. We pass all of that through the unit."}, {"start": 4057.0, "end": 4060.0, "text": " We do a forward pass and we predict back the noise."}, {"start": 4060.0, "end": 4064.0, "text": " And this is where we are at the moment. We have noise."}, {"start": 4064.0, "end": 4067.0, "text": " And let's now see what's going on. Going on."}, {"start": 4067.0, "end": 4078.0, "text": " We're going to return that noise. And here is how the loss is going to look like in the case where we use Epsilon prediction as parameterization."}, {"start": 4078.0, "end": 4090.0, "text": " You can see that exactly this noise that was used to noise the initial representation to get the noisy version is now going to be the one the variable that we're trying to predict."}, {"start": 4090.0, "end": 4093.0, "text": " So that's that's that's it. Like that's as simple as that."}, {"start": 4093.0, "end": 4098.0, "text": " And this is just going to be literally just the MSC loss or something like that."}, {"start": 4098.0, "end": 4104.0, "text": " So so, yeah, let me just do here. Yeah, it was literally just L2 loss."}, {"start": 4104.0, "end": 4108.0, "text": " Nothing nothing other than that. And that's it. As simple as that."}, {"start": 4108.0, "end": 4113.0, "text": " Again, I think this log bar is set to zero. So it will not influence."}, {"start": 4113.0, "end": 4119.0, "text": " Yeah, it will not literally change anything by doing this. We don't do we don't change the loss."}, {"start": 4119.0, "end": 4124.0, "text": " And we just now do some waiting. And that's it."}, {"start": 4124.0, "end": 4131.0, "text": " I'm not sure why we need the VLB loss, the lower bound loss, because it's going to be the same computation as what we had up there."}, {"start": 4131.0, "end": 4141.0, "text": " So look, this thing here is the same as this thing here. So if I do have 11 and enter this will again will again hit the L2 loss branch."}, {"start": 4141.0, "end": 4144.0, "text": " So we just compute the L2 loss again and we return that log."}, {"start": 4144.0, "end": 4149.0, "text": " OK, I'm not sure why we're computing this, because it literally gives us the same results."}, {"start": 4149.0, "end": 4159.0, "text": " If I were to print loss simple here and if I were to print loss VLB, so the variational lower bound, we get the same values and we compute literally the same lines here."}, {"start": 4159.0, "end": 4163.0, "text": " There is the only difference is we have a different weight here for this loss."}, {"start": 4163.0, "end": 4168.0, "text": " And then because this is zero, this will not even have the impact on the final loss."}, {"start": 4168.0, "end": 4173.0, "text": " That part is kind of confusing. And yeah, that's pretty much it."}, {"start": 4173.0, "end": 4182.0, "text": " After I do this, there is additionally some some like logic with the with EMA here."}, {"start": 4182.0, "end": 4189.0, "text": " But yeah, nothing, nothing, nothing vital there."}, {"start": 4189.0, "end": 4192.0, "text": " OK, so we can we can skip over all of this and that's it."}, {"start": 4192.0, "end": 4194.0, "text": " And now we just keep on repeating the batch after batch."}, {"start": 4194.0, "end": 4206.0, "text": " So that's it. Just a forward pass through the unit. And then we basically do L2 loss between the predicted noise versus the noise that we used to noise our input representation."}, {"start": 4206.0, "end": 4212.0, "text": " Fairly simple. Literally the formula we saw in the paper just being played out here in the code."}, {"start": 4212.0, "end": 4221.0, "text": " So that's this formula here. We are randomly sampling these noise and conditioning label."}, {"start": 4221.0, "end": 4227.0, "text": " OK, that one is correlated, obviously, with the image. And then we just use the latent representation of that image."}, {"start": 4227.0, "end": 4234.0, "text": " So Formula 3 is everything that we have in the stage two of training this system here."}, {"start": 4234.0, "end": 4239.0, "text": " After we've done this, we literally can now use the pre-trained weights and start sampling."}, {"start": 4239.0, "end": 4243.0, "text": " So let me show you how we can sample using using this this code."}, {"start": 4243.0, "end": 4254.0, "text": " So I'm going to stop this and let's go to the long script. And this time we're going to be using the text to image script, this one."}, {"start": 4254.0, "end": 4260.0, "text": " And the only argument you have to pass is this PLMS. So that's going to be the scheduler we are using."}, {"start": 4260.0, "end": 4269.0, "text": " You can also use the I.M. but like this one has higher quality and they showed in the paper that the I.M. is just a special case of the PLMS scheduler."}, {"start": 4269.0, "end": 4274.0, "text": " OK, so let's open up the text to image and let's start."}, {"start": 4274.0, "end": 4283.0, "text": " OK, guys, let's pick the correct configuration here and I'm going to hit run and we'll soon start executing the sampling script."}, {"start": 4283.0, "end": 4287.0, "text": " So this one is used using a textual prompt. You can now generate the images."}, {"start": 4287.0, "end": 4292.0, "text": " And that's that's how these text condition image generation models became popular."}, {"start": 4292.0, "end": 4296.0, "text": " You can kind of tweak the prompts and generate corresponding images."}, {"start": 4296.0, "end": 4302.0, "text": " OK, so here's a here's a prompt I'm currently using a painting of an AI having an epiphany moment."}, {"start": 4302.0, "end": 4306.0, "text": " That's the prompt we're using. You specify the output directory, whether you want to."}, {"start": 4306.0, "end": 4311.0, "text": " This is not important. Skipping all of these number of diffusion steps."}, {"start": 4311.0, "end": 4318.0, "text": " That's important. We will be using five in this example just to make sure this is going to execute very, very quickly."}, {"start": 4318.0, "end": 4322.0, "text": " I'm going to store the PLMS such that we want to use that scheduler."}, {"start": 4322.0, "end": 4329.0, "text": " This one is also not important. This is if you want to have basically always start from the same latent."}, {"start": 4329.0, "end": 4335.0, "text": " And that's going to kind of constrain your outputs to be less diverse as a consequence of that."}, {"start": 4335.0, "end": 4338.0, "text": " We don't need that. We don't care about this variable as well."}, {"start": 4338.0, "end": 4342.0, "text": " This is how many images I'm generating. Just nine."}, {"start": 4342.0, "end": 4345.0, "text": " The input image dimensions, five, twelve, five, twelve."}, {"start": 4345.0, "end": 4353.0, "text": " The latent number of channels is four. The sampling factor is eight. Number of samples is one."}, {"start": 4353.0, "end": 4356.0, "text": " Blah, blah, blah. We can skip all of this. There's too many."}, {"start": 4356.0, "end": 4362.0, "text": " Oh, my God. OK, so I'm just going to hit F5 and get here."}, {"start": 4362.0, "end": 4369.0, "text": " We can skip this part because it's false. We see everything. We load the configuration."}, {"start": 4369.0, "end": 4372.0, "text": " So let's see what that's going to be. OK, we want inference."}, {"start": 4372.0, "end": 4376.0, "text": " So let me see. So let's go to configs. Let's go to configs here."}, {"start": 4376.0, "end": 4382.0, "text": " Let's go to stable diffusion. We want inference. This is how that thing looks like."}, {"start": 4382.0, "end": 4393.0, "text": " So, again, it's literally just going to specify the latent model, specify the unit, specify all the parameters, all the encoder, pretty much everything."}, {"start": 4393.0, "end": 4399.0, "text": " And we're going to be using clip this time, not the class conditional information for the conditioning stage."}, {"start": 4399.0, "end": 4403.0, "text": " That's it. It's going to need that all of this is specified inside of a config file."}, {"start": 4403.0, "end": 4406.0, "text": " A couple of things you need to do if you want to follow along."}, {"start": 4406.0, "end": 4412.0, "text": " First up is you have to go to this like page and download the weights."}, {"start": 4412.0, "end": 4418.0, "text": " And the best ones are V1,4 or you can play with V1,3 as well."}, {"start": 4418.0, "end": 4424.0, "text": " Basically, go ahead, download those, put them in the corresponding directory. And that's it."}, {"start": 4424.0, "end": 4431.0, "text": " After that, there is a couple more things. So I had to, again, create some tweaks."}, {"start": 4431.0, "end": 4435.0, "text": " So one of the tweaks is the following. So let's go to latent diffusion."}, {"start": 4435.0, "end": 4442.0, "text": " I'm using this image net config file. I had to specify. So here's where you specify the checkpoint path."}, {"start": 4442.0, "end": 4448.0, "text": " So wherever you download, you can see you need to specify the checkpoint path there."}, {"start": 4448.0, "end": 4456.0, "text": " And there is the batch size. I also reduced it from 64 to 1. Otherwise, I'm getting good out of memory exceptions."}, {"start": 4456.0, "end": 4462.0, "text": " OK, so that's one thing. And then you have to also modify this text to image."}, {"start": 4462.0, "end": 4471.0, "text": " So basically what I had to do is to set. And this is very dirty just to make sure that I'm getting this to work on my computer."}, {"start": 4471.0, "end": 4473.0, "text": " But there are better ways to do this."}, {"start": 4473.0, "end": 4486.0, "text": " And it's much better for you to use the diffusers library than to do what I've done here because they actually have some parts where they accumulate in FP32 instead of doing everything in FP16 as I'm doing."}, {"start": 4486.0, "end": 4493.0, "text": " So as a consequence, I'm probably getting a bit lower quality images, but it works and I can step through my code."}, {"start": 4493.0, "end": 4502.0, "text": " So that's one tweak. And then what else? Let me see what I had to change. That's important to get this to work."}, {"start": 4502.0, "end": 4510.0, "text": " Set the number of samples to one. Otherwise, we batch size of three. I was getting out of memory exceptions as well."}, {"start": 4510.0, "end": 4515.0, "text": " OK, so here's the checkpoint information. You don't need this part."}, {"start": 4515.0, "end": 4527.0, "text": " And and finally, I have explicitly made it such that we are dealing with flow 60 here and not with mixed precision because I was getting I was hitting errors."}, {"start": 4527.0, "end": 4545.0, "text": " If I recall correctly, if I don't put the FP16 here explicitly and finally, I've just for the sake of speed, I commented out the check safety functionality that basically checks whether you have not safe for work or other problematic content."}, {"start": 4545.0, "end": 4553.0, "text": " OK, having said that, let's go back to the text to image script and we can now continue. That's everything you need to know."}, {"start": 4553.0, "end": 4562.0, "text": " OK, we now load the model. So we load the diffusion model this time. Obviously, I'm going to disable. We've seen all of this."}, {"start": 4562.0, "end": 4567.0, "text": " So actually, we're going to enter this part and I'm only only going to show you the difference."}, {"start": 4567.0, "end": 4580.0, "text": " And that's loading the clip model. So I'm going to hit F10. And let's just see how the clip model is being used to basically create the conditioning information from the input prompt."}, {"start": 4580.0, "end": 4589.0, "text": " And that's going to be the only interesting part. OK, so I'm now going to toggle on all the breakpoints, hit F10."}, {"start": 4589.0, "end": 4599.0, "text": " And here we are. So we are now creating the conditioning stage. Instantiated from config. We have frozen clip embedder. So let's hit F10."}, {"start": 4599.0, "end": 4610.0, "text": " So here it is. So basically what I do is they use hugging faces, pre trained clip tokenizer and pre trained text model. And that's it."}, {"start": 4610.0, "end": 4615.0, "text": " Like everything else, I've covered clip in one of my previous videos, so I'm going to link it somewhere here."}, {"start": 4615.0, "end": 4621.0, "text": " You can you can go and check it out if you want to understand how clip exactly works. I also covered papers."}, {"start": 4621.0, "end": 4627.0, "text": " So, yeah, there is plenty of information about clip on my channel. That's it."}, {"start": 4627.0, "end": 4633.0, "text": " I'm going to leave the forward function, the breakpoint there and let's exit here."}, {"start": 4633.0, "end": 4641.0, "text": " And we are done. We're done. We just set the gradients to false because we don't want to train this. We're now in the sampling stage."}, {"start": 4641.0, "end": 4646.0, "text": " So I'm going to go back to the text to image and I'm just going to put a breakpoint here."}, {"start": 4646.0, "end": 4651.0, "text": " I'm going to basically disable everything and just leave that breakpoint here."}, {"start": 4651.0, "end": 4657.0, "text": " Let's exit the load model from config function and continue from there."}, {"start": 4657.0, "end": 4670.0, "text": " OK, so pushing the model to GPU, we instantiate the PLMS sampler when it comes to well, when it comes to any function, it's not that complicated."}, {"start": 4670.0, "end": 4676.0, "text": " We literally just store the LDM model here. So this is going to be just the let me show you this."}, {"start": 4676.0, "end": 4687.0, "text": " So basically type is going to be LDM. So you can see here late in diffusion and we have thousand steps were used to train the model."}, {"start": 4687.0, "end": 4693.0, "text": " That's an important information to form the schedule and schedule is going to be like a linear schedule."}, {"start": 4693.0, "end": 4700.0, "text": " OK, so I'm going to show you more once we get to the actual forward pass sampling."}, {"start": 4700.0, "end": 4716.0, "text": " They additionally have this border mark tool that basically makes sure that it encodes a watermark that's invisible to humanize basically into the image such that we know that it was machine generated image."}, {"start": 4716.0, "end": 4724.0, "text": " And later we can use that watermark to exclude those samples from our training data if we decide to do that."}, {"start": 4724.0, "end": 4734.0, "text": " I guess that's one of the main reasons they were they were doing that as well as to catch someone who is generating images and not giving proper credit to stable diffusion."}, {"start": 4734.0, "end": 4744.0, "text": " I assume. OK, so here's a prompt. We have a painting of an AI having epiphany moment and we form some output directories."}, {"start": 4744.0, "end": 4750.0, "text": " Nothing interesting there. And finally, here is so this is the interesting part."}, {"start": 4750.0, "end": 4755.0, "text": " We're going to start the sampling here. So I'm going to hit F5 exit that part."}, {"start": 4755.0, "end": 4761.0, "text": " And here it is. So we we have a single problem. So this loop is going to be kind of trivial."}, {"start": 4761.0, "end": 4768.0, "text": " The first part we do is we get the learned conditioning from the for the empty prompt."}, {"start": 4768.0, "end": 4774.0, "text": " So this is the again a classifier free guidance technique."}, {"start": 4774.0, "end": 4780.0, "text": " So because our scale is seven point five, so that's the guidance scale. That's why we basically entered this part."}, {"start": 4780.0, "end": 4788.0, "text": " And now let's let me just make sure this is all enabled. Let's enter this and let's see how clip works here."}, {"start": 4788.0, "end": 4795.0, "text": " So what we do is we encode the prompt and the prompt in this particular case is an empty prompt."}, {"start": 4795.0, "end": 4803.0, "text": " So encoding is just a forward pass through the frozen clipping better. So we just tokenize the text."}, {"start": 4803.0, "end": 4813.0, "text": " You can see here we get batching codings. So if I were to since because it's an empty prompt, it's going to be fairly trivial."}, {"start": 4813.0, "end": 4818.0, "text": " Let's see the shape here. OK, so it's going to be a trivial encoding. Basically, all of the numbers are the same."}, {"start": 4818.0, "end": 4823.0, "text": " Those are the this is the beginning of sentence token. And these are the end of sentence."}, {"start": 4823.0, "end": 4834.0, "text": " OK, we can kind of validate that by doing the following. So tokenizer decode, I think, was the name. And then we just pass this number."}, {"start": 4834.0, "end": 4841.0, "text": " And let's see what it is to start of text. And the one with ending in seven is the end of text. That's it."}, {"start": 4841.0, "end": 4847.0, "text": " That's how the representation is. The sequence of IDs is formed."}, {"start": 4847.0, "end": 4854.0, "text": " We push them to GPU and we just passed them now to the transformer, which is basically the textual part of the clip model."}, {"start": 4854.0, "end": 4864.0, "text": " And we end up with a final representation that's seventy seven five or something. So seventy seven and six seven hundred sixty eight."}, {"start": 4864.0, "end": 4870.0, "text": " That's that's the final representation that came from clip. And we're going to use that to condition the unit model."}, {"start": 4870.0, "end": 4876.0, "text": " That's the idea. OK, so let's get back here. Let's get back here. We're returning the information and that's it."}, {"start": 4876.0, "end": 4883.0, "text": " So that's going to be used to condition the unit. Now we do the same thing just with the with the actual prompt."}, {"start": 4883.0, "end": 4888.0, "text": " So I'm going to disable the breakpoints here and we do the same thing. So I'm just going to skip that again."}, {"start": 4888.0, "end": 4898.0, "text": " C is going to be same shape as you see. You see standing for basically unconditional conditioning."}, {"start": 4898.0, "end": 4908.0, "text": " And finally, here is where the sampling starts. After the sampling, we have a simple we just pass the final latent representation through the decoder,"}, {"start": 4908.0, "end": 4917.0, "text": " which is just a set of layers, attention layers, etc, etc. And up sampling, do some clamping, put the put the final representation on the CPU."}, {"start": 4917.0, "end": 4923.0, "text": " When I say final presentation, I mean image, disimplementation, blah, blah, blah. And then we can store the image here."}, {"start": 4923.0, "end": 4929.0, "text": " And that's everything. Everything else is kind of arranging the images into into grids."}, {"start": 4929.0, "end": 4938.0, "text": " So this is where the gist of the logic is. So in the sample part, so I'm going to enable all the breakpoints and let's enter this part."}, {"start": 4938.0, "end": 4947.0, "text": " Let's hit F5. And here we are. So we enter the sample part. And because we have conditioning, let's see what we do."}, {"start": 4947.0, "end": 4955.0, "text": " We do some error checking there. We make a schedule. So make a schedule is just going to make sure that we set the appropriate constants."}, {"start": 4955.0, "end": 4961.0, "text": " So let me let me go into this. So here here we are. So first of all, let's see."}, {"start": 4961.0, "end": 4969.0, "text": " So we have uniform discretization. Number of steps is five because I said this is just dumb."}, {"start": 4969.0, "end": 4978.0, "text": " Usually what you want to use like 50 is OK, 200 if you want to get a bit better results. But there is a saturated saturation going on."}, {"start": 4978.0, "end": 4983.0, "text": " Definitely. So 50 is completely fine. So this is the actual number we use during the training."}, {"start": 4983.0, "end": 4992.0, "text": " That's a vital information for the scheduler so that we can construct the the the final set of time steps."}, {"start": 4992.0, "end": 5000.0, "text": " I'm going to disable these and we're going to end up as you can see here. So one two one four one six one and eight one."}, {"start": 5000.0, "end": 5006.0, "text": " That means that's how we uniformly sampled our thousand time stamps into only five time steps. That's it."}, {"start": 5006.0, "end": 5011.0, "text": " OK, next up, we grab the original alphas from our diffusion model here."}, {"start": 5011.0, "end": 5020.0, "text": " We create some lambda functions. And then, as I said, we just start creating these non learnable constants."}, {"start": 5020.0, "end": 5029.0, "text": " And I'm going to skip across all of these. Because it's really hard to explain this without taking one hour or something."}, {"start": 5029.0, "end": 5034.0, "text": " So, yeah. OK, so finally, we have those constants in place. We formed our time steps."}, {"start": 5034.0, "end": 5041.0, "text": " Now let's enter the logic here. So enable breakpoints. Let's hit F5 and enter this part."}, {"start": 5041.0, "end": 5054.0, "text": " So here we are. We are starting to generate the image right now. So here we generate the initial random basically noise tensor,"}, {"start": 5054.0, "end": 5060.0, "text": " which is going to be sixty four sixty four four, because that's the size of the latent space of this particular LDM."}, {"start": 5060.0, "end": 5066.0, "text": " And we start there and then we we form the time steps. So you can see here one two one, etc."}, {"start": 5066.0, "end": 5073.0, "text": " And we just reverse the time steps here because when we are generating, we want to start from the end."}, {"start": 5073.0, "end": 5080.0, "text": " We're starting from the noise image and we're just time going in reverse until we generate the actual image from our data distribution that we learned."}, {"start": 5080.0, "end": 5087.0, "text": " OK, so running running PLM sampling with five time steps. Let's continue there."}, {"start": 5087.0, "end": 5096.0, "text": " Basically, we we create the time steps there. The first one should be, I guess, eight or one or something. So TSA to one."}, {"start": 5096.0, "end": 5103.0, "text": " And then we grab the next ones because the PLM logic needs it. So it's going to be, I guess, six or one."}, {"start": 5103.0, "end": 5110.0, "text": " Yeah. And then let's continue. So now we end up doing this. We call the P sample PLM."}, {"start": 5110.0, "end": 5118.0, "text": " The code is fairly messy and complicated. So, yeah, I apologize for not being able to explain this a bit more clearly, but I'm giving my best here."}, {"start": 5118.0, "end": 5125.0, "text": " So stick with me. So, OK, let's see how the final logic looks like. So I'm setting a breakpoint here."}, {"start": 5125.0, "end": 5135.0, "text": " We're going to hit that line a bit later. So for now, we just grab these constants, alphas, sigmas, etc, etc."}, {"start": 5135.0, "end": 5143.0, "text": " I'm also going to put the breakpoint here. So we're going to enter that function. And here is the final logic that the PLM sampler does."}, {"start": 5143.0, "end": 5151.0, "text": " So we get the model output and that's basically a forward pass through the through the unit."}, {"start": 5151.0, "end": 5158.0, "text": " So let's let's go there. Let's see what's going on there. So here it is because we're doing the other classifier free guidance."}, {"start": 5158.0, "end": 5163.0, "text": " We have to repeat our input representation, which is currently just a pure noise."}, {"start": 5163.0, "end": 5170.0, "text": " Our time steps as well. We have to concatenate both our unconditional conditioning and the conditioning from the actual prompt."}, {"start": 5170.0, "end": 5180.0, "text": " We do a forward pass through the unit model. So I'm going to hear just disable breakpoints due to forward pass because nothing insightful is happening there."}, {"start": 5180.0, "end": 5184.0, "text": " And finally, we get the output representations here, which we then combine."}, {"start": 5184.0, "end": 5197.0, "text": " So we combine the noise from the when we when we condition the unit with the unconditional conditioning and we also pass the basically the information that we got when using the actual conditioning here."}, {"start": 5197.0, "end": 5203.0, "text": " And that's that's how we form our final noise prediction. OK, after that."}, {"start": 5203.0, "end": 5210.0, "text": " So, OK, it had to take some time to figure out how to connect this code with the formulas from the paper."}, {"start": 5210.0, "end": 5216.0, "text": " So let me show you side by side comparison of the code and formulas. And let's start and figure out what's going on here."}, {"start": 5216.0, "end": 5224.0, "text": " OK, guys, so here is the paper. So I opened up the pseudo numerical methods for the future models on many folks paper."}, {"start": 5224.0, "end": 5231.0, "text": " It's a mouthful. Even the title is hard to comprehend. So let's get back to this statement here."}, {"start": 5231.0, "end": 5241.0, "text": " Oops, let me just find it. Basically, so we see four branches here. And the reason that is is because they are using this linear multi step method."}, {"start": 5241.0, "end": 5253.0, "text": " And they say here here we cannot use linear multi step initially because the linear multi step method cannot start automatically, which needs at least three previous steps information to generate results."}, {"start": 5253.0, "end": 5262.0, "text": " So we use the rune Jakuta method to compute the first three steps results and then use the linear multi step method to calculate the remaining."}, {"start": 5262.0, "end": 5269.0, "text": " OK, so I've done some annotations here so that we can find the precise formula for each of these branches."}, {"start": 5269.0, "end": 5274.0, "text": " So let's start like that. OK, so maybe I'll start start with the fourth branch."}, {"start": 5274.0, "end": 5286.0, "text": " So the final step, once we have had at least three steps of this of this sampling step, basically we'll end up in this hitting this branch every single time."}, {"start": 5286.0, "end": 5292.0, "text": " So let's go to Formula 12 and let's kind of convince ourselves that this makes sense."}, {"start": 5292.0, "end": 5298.0, "text": " OK, so here is the Formula 12. You can see that the first step is to calculate the epsilon theta."}, {"start": 5298.0, "end": 5307.0, "text": " So this is what we've done here in this step. So get model outputs we get we have our epsilon. So the noise prediction here."}, {"start": 5307.0, "end": 5314.0, "text": " The next step is, as you can see here, to calculate this epsilon prime and you can see it's one over 24 blah blah blah."}, {"start": 5314.0, "end": 5318.0, "text": " Some some some expression there you can see that corresponds directly to this one here."}, {"start": 5318.0, "end": 5326.0, "text": " So fifty five E t minus fifty nine E t minus delta, etc. etc. So you can kind of see this corresponds to this."}, {"start": 5326.0, "end": 5333.0, "text": " So here is the third step, this five function, which they call, I think, transfer function or something like that."}, {"start": 5333.0, "end": 5339.0, "text": " Basically, here it is defined, but we will not use it for for for this explanation."}, {"start": 5339.0, "end": 5350.0, "text": " So let me just get back here. So what I've done is basically I figure out that this function here, which will get to in a couple of seconds."}, {"start": 5350.0, "end": 5357.0, "text": " So let me just enable the break points for a second here. So let's enable all of the break points. Let me do this."}, {"start": 5357.0, "end": 5367.0, "text": " So this basically function corresponds to Formula 8. So that's this formula here."}, {"start": 5367.0, "end": 5372.0, "text": " We're going to convince ourselves in that in a couple of seconds. OK, but let me let me get back here."}, {"start": 5372.0, "end": 5378.0, "text": " So we saw that this expression here makes sense. Now, let's make sense out of the other branches."}, {"start": 5378.0, "end": 5382.0, "text": " So for this one, I could not find the corresponding expression in the paper."}, {"start": 5382.0, "end": 5387.0, "text": " So if anyone knows what the heck is going on, feel free to comment down below. I'm listening."}, {"start": 5387.0, "end": 5394.0, "text": " So for the for for this branch here, I found the Formula 23 to correspond to this one."}, {"start": 5394.0, "end": 5400.0, "text": " So let's let me let me find that one. So 23 is in the appendix of the paper."}, {"start": 5400.0, "end": 5407.0, "text": " It's kind of hard to, yeah, even find the correspondence between these, let alone have some intuition."}, {"start": 5407.0, "end": 5410.0, "text": " And I guess even the authors of this paper don't have the intuition."}, {"start": 5410.0, "end": 5416.0, "text": " It's more of a OK, we associated this differential equation with this diffusion process."}, {"start": 5416.0, "end": 5426.0, "text": " And we just then can automatically pick up the tools that we already have from a long and rich history of solving differential equations and apply those to solve diffusion."}, {"start": 5426.0, "end": 5430.0, "text": " But like intuition wise, I'm not sure anyone understand what's going on here."}, {"start": 5430.0, "end": 5434.0, "text": " Like I might be wrong, but that's that's my current understanding of the things."}, {"start": 5434.0, "end": 5438.0, "text": " OK, so I said Formula 23. Here it is."}, {"start": 5438.0, "end": 5445.0, "text": " So you can see we calculate Epsilon again and then we have one over half three Epsilon minus Epsilon old."}, {"start": 5445.0, "end": 5450.0, "text": " And you can see that exact formula here. So we are computing that result here."}, {"start": 5450.0, "end": 5454.0, "text": " And then finally, let me go to equation 22."}, {"start": 5454.0, "end": 5458.0, "text": " That's the first branch of this of this complex branching."}, {"start": 5458.0, "end": 5463.0, "text": " So first on a high level, this let me kind of step over here."}, {"start": 5463.0, "end": 5469.0, "text": " So this function here get X previous and prediction X zero is calculating whatever phi is."}, {"start": 5469.0, "end": 5480.0, "text": " We'll see what it is in a second. So once we have the results, so we have this X prev, then we feed that into the neural network and we again grab the find the noise."}, {"start": 5480.0, "end": 5489.0, "text": " And that corresponds to this this step here. So whatever is output from the second step, we feed it back into our neural network."}, {"start": 5489.0, "end": 5496.0, "text": " We feed in the T next as you can see here. So that's why we have T plus Delta here and we get back the results here."}, {"start": 5496.0, "end": 5505.0, "text": " OK, and then finally we grab those results from this step and we just add them up with the Epsilon from the previous step and we divide by two."}, {"start": 5505.0, "end": 5509.0, "text": " So that's this part here and we end up with the E T prime."}, {"start": 5509.0, "end": 5513.0, "text": " And then after that, we're going to again call the five function here."}, {"start": 5513.0, "end": 5518.0, "text": " So now I guess it boils down to figure out what this what this five function is."}, {"start": 5518.0, "end": 5523.0, "text": " So let's step inside of this function. Let me show you what's going on here."}, {"start": 5523.0, "end": 5528.0, "text": " So again, these are just some non learnable expressions."}, {"start": 5528.0, "end": 5537.0, "text": " I'm going to skip those. But let me find the formula eight, which directly I found that that one corresponds directly to this to this code here."}, {"start": 5537.0, "end": 5544.0, "text": " So let me find that. OK, so formula eight. Here it is."}, {"start": 5544.0, "end": 5552.0, "text": " So let's convince ourselves that this makes sense. So here we have X minus square root one minus 80 times E T Epsilon."}, {"start": 5552.0, "end": 5556.0, "text": " So we can see that corresponds to this expression here."}, {"start": 5556.0, "end": 5563.0, "text": " We basically have X minus square root one minus this cumulative sum product of alphas."}, {"start": 5563.0, "end": 5569.0, "text": " And then we multiply that, as you can see here, with with the output of our neural network."}, {"start": 5569.0, "end": 5574.0, "text": " And then we divide all of that by the square root of 80 here."}, {"start": 5574.0, "end": 5577.0, "text": " And that's pretty much this first term next up."}, {"start": 5577.0, "end": 5581.0, "text": " So that's the first part. And then we calculate the second part."}, {"start": 5581.0, "end": 5585.0, "text": " The second part is, as you can see here, it's computing this part here."}, {"start": 5585.0, "end": 5592.0, "text": " So one minus a previous minus sigma squared square root of all of that times the Epsilon."}, {"start": 5592.0, "end": 5597.0, "text": " And you can see that's precisely this term here. So let me now continue stepping."}, {"start": 5597.0, "end": 5604.0, "text": " Now we calculate the noise. And finally, let's see how all of that is combined together."}, {"start": 5604.0, "end": 5614.0, "text": " We have a previous square root. So that's this part times whatever we predicted up there, which was the first term, plus this term here."}, {"start": 5614.0, "end": 5619.0, "text": " And then finally, plus the noise. So that's this part here."}, {"start": 5619.0, "end": 5626.0, "text": " Let me just see what the value of sigma t is. And it's zero. OK, so this this part will actually be ignored."}, {"start": 5626.0, "end": 5630.0, "text": " So as you can see here, this part, the noise, because it's multiplied by sigma t."}, {"start": 5630.0, "end": 5637.0, "text": " Let me just kind of go into the debug console here. Let's convince ourselves sigma t is zero."}, {"start": 5637.0, "end": 5642.0, "text": " And that makes sense because here in the paper, they mention it somewhere."}, {"start": 5642.0, "end": 5650.0, "text": " Let me just find it that they only care about the case where sigma is equal to zero."}, {"start": 5650.0, "end": 5655.0, "text": " Let me just find that one. OK, I found it. I didn't highlight it initially, so it was hard to find it."}, {"start": 5655.0, "end": 5661.0, "text": " So they say here, therefore, our work concentrate on the case where sigma equals zero."}, {"start": 5661.0, "end": 5668.0, "text": " OK, let me read this for you. So here sigma controls the ratio of random noise because it's modulating the noise, as you can see here in Formula 8."}, {"start": 5668.0, "end": 5674.0, "text": " If sigma equals one, equation eight represents the reverse process of DDPMs."}, {"start": 5674.0, "end": 5677.0, "text": " So those are the denoising diffusion probabilistic models."}, {"start": 5677.0, "end": 5682.0, "text": " So if sigma equals zero, this equation represents the reverse process of DDIMs."}, {"start": 5682.0, "end": 5692.0, "text": " OK, and only when sigma equals zero, this equation removes the random item and becomes a discrete form of a certain ODE, ordinary differential equation."}, {"start": 5692.0, "end": 5698.0, "text": " Theoretically, the numerical methods that can be used on differential equations with random items are limited."}, {"start": 5698.0, "end": 5706.0, "text": " So that's why we want to escape. We want to set sigma to zero because there are richer set of tools when we are not dealing with that random term."}, {"start": 5706.0, "end": 5715.0, "text": " These authors here have done enough research in this case. Empirically, they have shown that DDIMs have a better acceleration effect when the number of total steps is relatively small."}, {"start": 5715.0, "end": 5719.0, "text": " Therefore, our work concentrates on the case where sigma equals zero."}, {"start": 5719.0, "end": 5725.0, "text": " OK, so guys, that's that's pretty much it. I'm going to open up the code here."}, {"start": 5725.0, "end": 5729.0, "text": " As I said, I cannot provide you with much more intuition than this."}, {"start": 5729.0, "end": 5735.0, "text": " So let me kind of step through all of this. I'm going to set a breakpoint here."}, {"start": 5735.0, "end": 5739.0, "text": " Let me remove the disabled breakpoints. Let me enable this one."}, {"start": 5739.0, "end": 5744.0, "text": " Let's hit F5 and let's get back to the function."}, {"start": 5744.0, "end": 5747.0, "text": " OK, guys, so here we are. We now take that output."}, {"start": 5747.0, "end": 5755.0, "text": " We basically sum it up with the last prediction from the network here, divide by two as per the formulas we saw previously."}, {"start": 5755.0, "end": 5759.0, "text": " Next up, we compute the five function again, and that's it."}, {"start": 5759.0, "end": 5770.0, "text": " So we return back the X previous, which is the basically as well as the noise, which is the next step in the reverse diffusion process."}, {"start": 5770.0, "end": 5774.0, "text": " So we are slowly getting to the pure image. And that's pretty much it."}, {"start": 5774.0, "end": 5777.0, "text": " I think we're not going to keep on iterating here."}, {"start": 5777.0, "end": 5785.0, "text": " As you can see, we append the Epsilon to this to this array of fold Epsilon, which is used, if you remember the four branches."}, {"start": 5785.0, "end": 5790.0, "text": " So this is where we collect the old Epsilon's and then we pass them inside here."}, {"start": 5790.0, "end": 5796.0, "text": " And that's it. So if we start some circular array logic, callbacks are not important."}, {"start": 5796.0, "end": 5800.0, "text": " So I'm going to skip all of that. And now we just keep on iterating and that's it."}, {"start": 5800.0, "end": 5806.0, "text": " So we're going to have in this particular example, five steps, because that's how I've basically configured it."}, {"start": 5806.0, "end": 5814.0, "text": " But like in general, you have like 50 steps or 200 steps if you want to have a bit better, basically quality of the image generation."}, {"start": 5814.0, "end": 5820.0, "text": " But I'm going to stop this here. And basically I'm going to stop it here."}, {"start": 5820.0, "end": 5827.0, "text": " And I'm going to finally just show you briefly the safety function they added that might be interesting to some of you."}, {"start": 5827.0, "end": 5836.0, "text": " So let me show you how that functions. Let me just find it basically in the text to image here somewhere we have."}, {"start": 5836.0, "end": 5838.0, "text": " OK, here. So here is the check safety function."}, {"start": 5838.0, "end": 5845.0, "text": " Basically, what happens is once you have generated the image and you've done this clamping, blah, blah, blah, you convert it into an umpire array."}, {"start": 5845.0, "end": 5851.0, "text": " Basically you have an image. So now they call this check safety for you and the check safety function."}, {"start": 5851.0, "end": 5856.0, "text": " What it does is cause this safety feature extractor. So let's see what it does."}, {"start": 5856.0, "end": 5861.0, "text": " Basically, that's some pre-trained model from hugging face hub."}, {"start": 5861.0, "end": 5871.0, "text": " And then they call this safety checker, which is also basically, as you can see here, some pre-trained model from from from hugging face."}, {"start": 5871.0, "end": 5880.0, "text": " And so what's interesting here is so they do some checks whether whether you have a basically not safe for work concept inside of the image."}, {"start": 5880.0, "end": 5887.0, "text": " And if so, for that particular image, they load the replacement. And I think this one is fairly cool."}, {"start": 5887.0, "end": 5893.0, "text": " So basically what I do in this repo is they take the image, they load this image here."}, {"start": 5893.0, "end": 5897.0, "text": " I'm going to show you what it is in a second. Basically, they're recalling us."}, {"start": 5897.0, "end": 5904.0, "text": " So let's let's find the assets folder. And then under the assets, we are looking for Rick."}, {"start": 5904.0, "end": 5914.0, "text": " And that's going to be this one. OK, so they basically load this image in case you have not safe for work content and they get it back."}, {"start": 5914.0, "end": 5917.0, "text": " Now, the problem is I was playing with this code base a bit."}, {"start": 5917.0, "end": 5925.0, "text": " And even though I was not generating anything explicit or anything, I was still getting this this function being triggered."}, {"start": 5925.0, "end": 5928.0, "text": " And so, yeah, it's not perfect. That's that's the point."}, {"start": 5928.0, "end": 5933.0, "text": " So we can see the definition. Actually, I found it on the in the diffusers library."}, {"start": 5933.0, "end": 5939.0, "text": " Here you can you can see the safety checker, how all of those functions are defined here."}, {"start": 5939.0, "end": 5943.0, "text": " And you can see you can kind of go through this if you if you if you care about it."}, {"start": 5943.0, "end": 5949.0, "text": " But bottom line is I couldn't find how the models were trained."}, {"start": 5949.0, "end": 5953.0, "text": " I guess that's by design because they don't want you to know how to hack the model."}, {"start": 5953.0, "end": 5960.0, "text": " Although I don't know where that's the best position on this topic, but I guess it's very highly debatable."}, {"start": 5960.0, "end": 5965.0, "text": " So, yeah, you can kind of go through this code and basically explore it at your own pace."}, {"start": 5965.0, "end": 5968.0, "text": " If you're guys, this was a super long video."}, {"start": 5968.0, "end": 5977.0, "text": " Like we saw a lot of things we saw how to how to basically train and sample from from these class of LMS, latent diffusion models."}, {"start": 5977.0, "end": 5988.0, "text": " So we saw how to first train this autoencoder who like whose way whose way to basically then freeze and use the latent space in the second stage where we train the LMS."}, {"start": 5988.0, "end": 5992.0, "text": " So basically the unit plus the conditioning model."}, {"start": 5992.0, "end": 5995.0, "text": " And finally, we saw how to sample from these models."}, {"start": 5995.0, "end": 6003.0, "text": " And you saw that you saw that some of the formulas and the connections with differential equations make it kind of hard to have a clear intuition."}, {"start": 6003.0, "end": 6009.0, "text": " But like I'm curious to know and hear whether and how you understand how diffusion models work."}, {"start": 6009.0, "end": 6014.0, "text": " So if you have any intuitive type of an explanation, feel free to comment down below."}, {"start": 6014.0, "end": 6017.0, "text": " I'll try and read all of those because I'm super curious."}, {"start": 6017.0, "end": 6019.0, "text": " In any case, if you like this video, share it out."}, {"start": 6019.0, "end": 6021.0, "text": " Subscribe to this channel."}, {"start": 6021.0, "end": 6035.0, "text": " And until next time, bye bye."}]