OpenAI GLIDE (Diffusion) | ML Coding series | Towards Photorealistic Image Generation and Editing
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

5th video of the ML coding series! In this one I cover OpenAI's GLIDE model from the "GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models" paper.

It directly builds upon the "Diffusion Models Beat GANs on Image Synthesis" paper that I've covered in the previous video of the series. It is also a direct precursor to DALL-E 2!

I explain classifier-free guidance as well as CLIP guidance in this one.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2112.10741
✅ GitHub: https://github.com/openai/glide-text2im
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00:00 [paper] GLIDE - quick overview
00:03:30 Classifier free guidance explained
00:07:25 CLIP guidance explained
00:10:49 Training details, etc.
00:14:20 [coding] CLIP guidance script
00:26:25 Generating CLIP text target embedding
00:32:32 Reverse process (p_sample_loop) 
00:34:35 How is text conditioning implemented
00:43:00 CLIP guidance code
00:46:26 Classifier-free guidance script
00:51:47 Main logic
00:56:35 Showing generated images
00:58:23 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#glide #diffusion #imagesynthesis
 What's cracking guys in this video? I'm continuing on with the diffusion model tracking the machine learning coding series and this one I'll be covering glide, so that's a model that was a precursor to delete to Coming from open AI and I've previously covered the paper in great depth So if you want to have a deep dive into the paper itself, you can check it out I'm gonna link it somewhere here But in this video, I'll obviously be focusing on analyzing understanding the code I have three Jupyter notebooks prepared for you guys. And so we'll be going through those So what I've done is as always adjust I've downloaded I've cloned the repo here. You can find it I'll kind of link it in the video description But before we go there, I first want to Walk you through the paper as usually just to give you all of the necessary context Hopefully that's gonna be like 10 15 minutes quick overview compared to the deep dive. I've already previously made so without further ado let's start understanding how glide works and Then we'll we'll start doing the coding part cool so First things first as I said glide is a precursor to like a Dali model So they mentioned here when evaluated by human judges our samples are preferred to those from Dali. So this is the version 1 87% of the time when evaluated for photorealism and 69% of the time when evaluated for caption similarity The Dali version 1 actually did not even use diffusion models But the lead to did and it built directly up on glide. So yeah, we're slowly building our way towards more complex models Cool, you can see the some of the images here again here instead of compared to my previous video where I showed you the guided DDPM a model where you can could condition But using only like a class and for example image net class here Glide can actually cope with text with natural language So that's much much cooler and you can see some very cool examples here like a corgi varying a red bowtie and a purple party hat you can see that like you can composite different concepts together in a plausible way you can bind attributes such as a Purple party hat to to certain objects and red bowtie. So it's a fairly Powerful model and as I said, it was a precursor to delete you and we all know what delete you can now make Aside from image generation. It can also do image in painting You can see here you put a mask here and then you set some natural language prompt here zebras roaming in the field and Outcomes an image that's modified similarly here a girl hugging a corgi and a pedestal You can see how this dog here is is swapped by a corgi Etc etc. You can see a bunch of examples there So the model itself stands for guided language to image diffusion for generation and editing So generation is the the part that we all know about and anything is the in painting part cool so there is not this background about DDP M's are gonna skip all of that because Hopefully you watch some of my previous videos. This one is gonna build heavily upon those So do check out the whole series if you haven't so far You can also iteratively do this image in painting so you can see here how slowly they are adding They first had the corgi image and they add the table here then they add the ways here, etc. Etc So you can cannot iteratively improve your images Okay, so the parts which I want to focus on in this video are mostly about clip guidance and Classifier free guidance so I want to focus on those and I'll probably if I have enough time focus on in painting as well So without further ado, let's first introduce the classifier free guidance. So in the previous video, we saw how we can do classifier based guidance where you use a pre trained Well classifier and you use its gradients So the classifier was trained on noisy images. That's an important detail and then you just use the gradients from that classifier to kind of guide your your class conditional diffusion generation Here the cool thing about classifier free guidance is you do not have to train a separate classifier on pre trained on noisy images It's it's much easier and and that's better Okay, so let me try to briefly explain how classifier free guidance works Again, as I just mentioned, this method does not require a separate classifier That's a big plus for for this method compared to the classifier guidance And then they say here so for classifier free guidance the label in a class conditional diffusion model is replaced with a null label with a fixed probability during training This is I think usually 10 or 20 percent during sampling the output of the model is extrapolated further in the direction of the model The output of the model is extrapolated further in the direction of the class conditional model and away from the null conditional model like so And you can see here the expression we basically have the one where we condition on those on the on the null label And then we just add this this basically this difference between the class conditional and the null conditional and we just use this guidance scale so the parameter s to well to to to modulate how much we want to go in in that direction And how I think about this in my head the mental model I have is basically you have this thing here tells you gives you the the noise in the unconditional case So it's going to give you some noise in some is going to be some point in the end dimensional space where and is basically the number of dimensions in your image So the resolution times the number of channels and so now once you have that point you then plot this point of where the model thinks the class conditional noise should be So if it outputs point like here then you can form the vector by just kind of subtracting one from another and you get something like this And then what what this final expression is going to be about is OK the model outputs the null conditioned noise and it's here And we know that if you want to move towards that particular class on upon which we are conditioning we need to move in this direction And because of that we just add that vector and we modulated with the with the scalar s to basically well to to start generating images that are from that particular class So that's the I literally have a geometric interpretation now whether this is completely correct or not I don't know but like I think I think it does sound like a like a good mental model In any case let's now see how we can replace the simple labels by the natural language because that's what we care about when it comes to to glide So what what they do is so to implement classifier free guidance with generic text prompts we sometimes replace text captions with an empty sequence which we also refer to as this Well zero crossed I guess during the training we then guide towards the caption using the modified prediction blah blah blah OK so you can see that everything remains the same except that we are now conditioning on caption Like and not on the simple label and that's pretty much it so the second thing they do is they use clip for guidance because here you cannot have like a thousand classes or whatnot because languages obviously has infinite possibilities of compositing And yeah there is infinite number of sentences I guess so you have to use something like clip and the only difference compared to the classifier guidance we saw in the previous video is that here you're looking for a gradient of the product between the caption and the image So let me read this for you perturbing the so when applied to the future models these techniques are similar to classifier guidance perturbing the reverse process mean along the gradient of the text image product with respect to the image OK So what that means is basically if you recall how we how we've done the classifier guidance we output like a mean and then on top of that we shift that mean by the scale version of the covariance matrix times the gradient of our classifier So now the only thing that differs in this in this clip guidance is that this these gradients here because we are now dealing with text and not with with classes are going to be this thing here So basically you encode a certain caption and you want to make sure you want to find you want to learn how to tweak the input image such that such that the product between the image feature from that image and the text embedding here is maximized And that means that the image is belonging more and more to that class or differently said it's becoming better and better described by that particular caption So that's that's all like that's a small tweak that we have to make in order to to make the clip thing and the text conditioning to work Again let me just quickly draw a picture here it might be a bit easier to understand so we have an input image here and we have our clip model here OK So here's clip So now we now have some type of a prompt like I don't know like maybe something simple like a dog OK So we encode that thing we pass it through the clip again we pass it through clip here so this same model basically and outcomes outcomes some some in there here so this is going to be an embedding that corresponds to this prompt here Next up we pass the image through the clip and we get a different embedding here OK This is going to be the image embedding let me maybe choose a different color just for the sake of like not to come in order not to confuse this So here we have a prompt for the So we have the text embedding and here we have the image embedding and now the gradients are going to tell us how we should tweak the image how should we tweak the image such that this vector is much closer in the image text embedding space OK Hopefully that was clear enough Now let me briefly tell you a couple of things about their training so they mentioned here for our main experiments we train a 3.5 billion parameter text conditional diffusion model at 64 times 64 resolution and another 1.5 billion parameter text conditional up sampling diffusion model to increase the resolution to 256 times 256 For clip guidance we also train a noise aware 64 times 64 VAT clip model OK So the only difference compared to the previous video on the guided DDPMs is basically here we have additional transformer that's going to be used to text condition our diffusion model instead of just doing a simple class conditioning It's not that like big of a difference to be honest so a couple more details here to condition on the text we first encoded into sequence of key tokens and feed these tokens into transformer model OK The output of this transformer is used in two ways so first the final token embedding is used in place of a class embedding in the ADM model so that's the model from the previous paper again the paper is called I think diffusion models beat GANs on image synthesis And second the last layer of token embeddings a sequence of key feature vectors is separately projected to the dimensionality of each attention layer throughout the ADM model so the fusion model and then concatenated to the attention context at each layer OK So that's how we integrate this textual conditioning information into the ADM into the diffusion model That's an additional detail I want you to understand OK Finally here they mentioned how like let me read this for you so after the initial training run we fine tuned our base models to support unconditional image generation this training procedure is exactly like pre training except 20% of token of text token sequences are replaced with the empty sequence this way the model retains its ability to generate text conditional outputs but can also generate images unconditionally and we need this ability so that we can such that we can well use the the the basically the classifier free guidance effectively OK guys that's pretty much it I'm going to skip everything else again the results are basically better when we use classifier free guidance compared to clip guidance so that's one of the reasons this is so cool I'm gonna skip all of this this is not that vital and finally worth mentioning the only model that they've open source is the so called glide filtered where they say here we filtered out training images containing people to reduce the capabilities of the model in many people sent you problematic use cases the model is also way smaller compared to their biggest version and they also filter out some other images such as like hate symbols et cetera et cetera so just worth having that in mind finally the model such as glide was still struggling a lot with with certain well let's call it weird phrases whereas the lead to became much better with these still not perfect but but better than then then what glide could achieve here we can see an example a mouse hunting a lion and we can see that the model does does not know that there does not understand the direction so here it appears as if the lion was hunting the mouse and not the other way around so it understands the association but not the the the causal relationship here cool okay that's pretty much it now let's go back to decode let's understand how this looks in actual code okay I'm gonna first start with this simple clip guided script so we're going to use the clip the pre trained clip model on on that's like noise up there and we're going to use it to to guide our generation so let me run this thing and again as usually I'm going to skip over everything that's not core to understanding what I'm trying to explain in a particular script here we want to understand how clip is used to guide the generation process so we just grab the devices and I'm going to have I have a GPU so it's going to be a GPU here some options default options for the model and diffusion we saw this in the first video like they're obviously are reusing a lot of the code, which is a good thing. And so I'm going to scheme across all of this. So time a step through spacing. So what it does is if the model was trained on like 1000 steps, you can now instead during the sampling procedure have only hundred steps and still get good results, good high quality samples. Then we're going to form the model and the diffusion here. So model is basically a unit model and diffusion is contains all the constants that are necessary for the diffusion. Okay, so we'll need to step into this part because the model is a bit different compared to previous papers. So the model now contains also the transformer because of the text conditioning. So let's see how that looks like diffusion itself looks pretty much the same as in previous videos. So I'm going to skip over that. But we're going to jump into this create model function. Let me just hit F5 and let's get into the function itself. Okay, so a couple of details here depending on the image size. We're going to set up the unit in a particular way number number of channels is going to be different compared to depending on the image resolution. Again, we've seen this multiple times and they just have these attention blocks for particular resolutions. Not that important. And finally, we're going to have this text to image unit. So again, it's just a simple unit plus the transformer. That's everything that Glide changed compared to previous papers. Okay, now we're going to form the text to image unit here. There is a lot of parameters. I don't want to explain every single one of those. There is also the encoder part. I'm going to quickly show you what this is. Basically, they have a pre trained BPE encoder that are going that they are going to use. And I explained this in the open AI clip video. So I'm going to link it somewhere here. You can check it out if you want to understand a bit better what's going on here. How the actual tokenization works, how that magic looks like. Then you can check out that video. But here I'm just going to skip all of that and hit that five to enter the constructor of the text to image unit. Okay, so let's see what's different here. We call the constructor of the super class and that's going to be just a simple unit model. I'm not going to show you the unit because we already saw that in previous videos. As I said, I have to kind of depend on previous videos because otherwise this would be three or four hours long video. I kid you not. Okay, so we construct the we construct the unit model here. I'm going to skip over everything here. I'm just going to hit F5 and we have our unit constructed. Let's go outside here. And now, so this is the different part compared to previous papers. They have a transformer. So they form a transformer that has a particular context 128 tokens, certain width 512 dimensions, number of layers 16, etc, etc. Then they have they use some layer norm, not that important. They create some token embedding layer here. So this one is basically going to embed our tokens into certain well into a certain vector space. And then we also have positional embeddings. We don't need more than 128 because that's going to be the max context that this transformer is going to use. And we finally have this transformer projection linear layer. We're going to see how that one is used a bit later. Okay, finally, there is this padding embedding. We'll see how that all fits together a bit later. Okay, so cash stuff, not that vital. I'm going to skip all of that. And finally, I'm going to skip the diffusion model because we've seen that already. I'll have to skip this. And here we are. So again, we have a unit with transformer and we have our diffusion model here. So basically those two together form form a diffusion model. Okay, let's go back into the main script. We just do some some boilerplate for machine learning, just in conversion to FP 16. We put the model onto GPU will load the actual weights, and then we print the number of parameters here. Okay, we repeat the same thing for the up sampling model. I'm going to skip all of this because we will not be focusing on the up sampling model. So I'm going to kind of skip over all of this until we get back into the main script. I should have toggled off the break points. But yeah, I guess next time. Cool. I think we are there almost. Basically, okay, so we created the up sampling model. Now we create the clip model. Okay, let's see how clip looks like. So clip, as you can see here, I'm going to just step into here. We grab some default config path so they have this config YAML file that specifies certain properties of our model. So of the clip model so you can see it here again too many details to cover. I'm just going to continue on execution in the in the create clip function. So here we are. We create a simple tokenizer so clip will have a different a bit different tokenizer compared to the diffusion model. Again, we can treat it as a black box for all practical purposes. Okay. Now, as I already said, I covered clip in one of the previous videos. If you really want to understand how the code works and how clip works to check it out or or also check out my paper video on clip. I've done that as well. I can link those somewhere here. We can briefly jump into the text encoder. Let me see whether I have basically. Okay, so let's kind of let me let me quickly show you how this text encoder is going to look like. I'm going to skip some parts so they just form the causal mask here because well does that make sense. Yeah, it does because we are here dealing with with text. But then again, we are not generating text. So I guess you could use bi-directional you could use non causal masks as well here. But in any case, let me get a scheme over all of this. So now they start stacking these blocks. The first block is just going to be a text embedding block, which means you're going to take tokens and then you're going to embed them into certain space. I think I'm going to show you this in the in the I'm going to just put a well, I don't want to even put a breakpoint here. It's just a simple embedding layer. I'm going to skip that. And then they form they had a bunch of transformer blocks and then at the end they just extract the basically embedding that's above the last token in the sequence. So that's how clip works. I'm going to have to skip all of that and let's continue. So we have the text encoders. Basically, it's a simple transformer. It first embeds the tokens, then it processes them using the transformer blocks. And then at the end, it just extracts, as I said, the last embedding. That's it. Next up, we have image encoder. Image encoder is VAT. So they're using vision transformer. Again, I'll have to just treat this as a black box. Let me see whether there's something interesting here. OK, this time they're using a non causal mask, as you can see here, which makes sense for images. Let's see what else. So again, similar structure. They form these blocks. The first one is going to be image embedding. So we're going to basically do the patchify like layer of the VAT, which means you're going to treat image as a bunch of patches. And then you're going to embed those patches into certain into certain well, vector space. And then you're going to just treat those as tokens. And then it's pretty much the same as if you were to just use transformer. And you can see here they do have a bunch of transformer blocks. And at the end, they have image feature extractor that's going to basically extract. Let me just see what's going on there. OK, we're going to I'm going to put a like a simple break point in the four functions. We're going to see that a bit later once we start using clip actually. OK, so that's it. We have the image, you go to we have the text encoder. We form this logic scale. That's a learnable parameter. We might see how it's used a bit later. And finally, they form the actual clip model. So he has a bunch of helper functions. What we care about is, well, for now, the I can just keep the function. That's not that important. We're going to see everything a bit later during the actual four pass. OK, so now we just load the checkpoints for the image encoder. We load the checkpoint for the text encoder. This is where the actual fun starts. We're going to take the prompt and oil painting of a corgi. We're going to encode that and we're going to generate an image that's going to have that's going to be well described by this prompt. So the guidance scale is going to be three. So that's the S parameter that we use to multiply the gradients coming from the clip model. We have some temperature that's not that important. And now we just tokenize. Again, we're going to treat this as a black box. We take the prompt and we talk and I said using the diffusion models tokenizer into tokens. OK, so let me show you how that thing is going to look like. It's going to be as you can see here, there is seven tokens in total. And then there's this special utility function that's going to just pad the these tokens such that we fill in the context for the transformer. That's a part of our diffusion model. OK, let me show you what that means. In practice, that means we're going to have I think we are using hundred twenty eight. So that's this text context parameter. So basically if I were to print the number of tokens, you can see there is hundred twenty eight tokens now. And the mask just tells you which ones are actually legit. And the first seven ones are actually true tokens. Everything else is just padding, which is specified by setting false for all of those other tokens. OK, so so the reason we're doing this is so that we can basically take multiple prompts and do this in a single for a pass. And that's why we cannot pad to a fixed dimension. OK, so now we form this model keyword arguments dictionary. We just store the tokens of exercise is just one. So that means we're going to we won't do anything smart here. We just store the tokens and the mask. And this is going to be later used to condition the diffusion model. We're going to see that in a second. OK, so there is this condition function. I'm just going to enable basically all of the breakpoints. And this is going to we're going to hit this function a bit later. It's going to be very important. We are actually going to have to enter this function. So let me click F10 and enter this one. So we're going to form this. Well, basically the embedding for the prompts and then we're going to return the this conditioning function. OK, so let's see what's going to happen here. So we just basically pass prompts through the textual pathway of the clip model. Nothing fancy there. We've seen this multiple times. We've seen this in my previous clip video. So we've called this in code prompts that's just going to, as you can see here, iterate through the list of prompts. We only have a single prompt, so nothing fancy will happen there. We're going to treat the tokenizer as a black box. So this is going to return us some sequence of tokens. Let me let me show you what we're going to get here. So if I run this, we have like, as you can see here, six tokens and then we're going to call again this padded tokens and length function, which is going to just pad everything. OK, I kind of entered here. Let me just skip over this. We don't want to go there. So again, we end up with these subtokens, which are going to be just padded versions. Whoops. So if I just grab the subtokens here and if I print this, you can see we have the same tokens as above. Hmm. For some reason, there is some difference here. Let me just see. Oh, no. OK, we have the start of sentence token here and the end of sentence token here. And we have our tokens here. And then we have additionally the padding. So it's a bit different. As I said, Clip has a bit different tokenizer compared to the tokenizer that we have in our diffusion model. And now we're just going to append that and we are done. We just converted to torch tensors and we are back here. So now we have basically tokens and their associated lengths. Next up, we're going to pass these tokens through the text pathway. So that's again just going to embed the tokens, then apply a bunch of transformer blocks. And ultimately, we are going to extract a particular a particular token. So let's see how the extractor part works. So again, this is the this is the forward pass of our of our text encoder. And we don't care about most of these parts because that's just going to do as I said, just going to do the embedding and then number of transformer blocks. And ultimately, we'll have this. I think this is the part that we care about. Let's see how this is going to work. So we call the text extractor here. And here it is. So the text feature extractor. Let's see what it does. Basically, as you can see here, text length tells us how big how many tokens do we have fed into the transformer. And it's going to be equal to seven. So that tells you that that's the index of the last token. And then we just gather. So we let me show you what text is. So text is a print shape here. So we have 77 tokens. Some of those tokens are obviously padding. Some of those tokens are start and end of sentence tokens and 512 is just a dimensionality. So what we do by by doing the gather, we are basically going to extract the last token here. So that means we're going to extract the seventh, the well, the eighth token. Let's step over there and let me convince you that that's indeed the case if I were to take the text. And then if I were to take the seventh here and let's take the first three elements, you can see them here. And now let's let me show you the let me just see what the shape of this thing is. OK, so let's now take zero zero. And let's take also first three elements. And those. Yeah, as you can see, these are the same because we literally just extracted whatever the embeddings are on top of that seventh token. That's pretty much it. And finally, we pass that X through the layer norm. And finally, we just return the basically the mapped version of that particular token. So let's see what F is. I think it's just a simple. Yeah, fine. It's just a basically a simple linear layer with some tweaks. But I will not get into those details because as you can imagine, this is a very huge code base and I cannot cover everything. OK, so let's go outside here and we end up with H being what H shape is going to be one five hundred twelve. So we have a single token that is supposed to be our representation of our input prompt. OK, continuing on here, just some asserts. And that's it. We have our ZT tokens. That's the same thing. So again, one five twelve. And we're going to now normalize it just using the well, I guess this is just L2 norm as as previously as we've seen previously in the open clip video. OK, so that's the CT is that the and now we return this con function. I have a breakpoint here. So we're going to see it a bit later when the time comes. But for now, let's continue on with the main script. So a recap, because this was fairly overwhelming potentially. So we formed our diffusion models and we loaded the actual checkpoints here. We can ignore the sampling model. Then we we loaded the clip model. We have a prompt and we basically encoded a prompt here. We have a mask and tokens for that prompt. And now we ended up creating this this embedding vector for our prompt using the clip model. OK, so that's it. Now we're going to go through the P sample loop. So that's going to be the reverse process of our diffusion model. Let's see what we're going to do here. We want to generate an image that's going to be of this particular resolution. So it's going to be a sixty four sixty four RGB. And let's see what's interesting here. We're going to use the tokens and mask to condition the model. So that's going to be different compared to our previous videos with the DPMs. In any case, let's now jump into this P sample loop. Again, I kind of skipped the P sample loop because everything it does is just basically iterates through this dysfunction loop progressive and starts generating multiple samples. But I think we are generating just a single sample so that it doesn't even matter. So we can just kind of skip over all of this. And now the actual fun starts. So we start as we as you may know from the completely like a normal distribution. We can just sample our image from a normal distribution. That means we have initially completely noisy image. And then we're going to slowly work our way from there and denoise that image until we get the image that corresponds to the prompt that we are using for this for this program. OK, so we'll have a hundred steps of the reverse process and just some DQDM stuff for for monitoring the progress. But we're going to now do hundred times in order to hundred denoising steps so that we end up with an image. OK, so initially T is going to be 99. As you can see here. And now the whole fun happens in this P sample function. So we're going to pass the fusion model so that's unit with transformer. We're going to pass the noisy image time step and a couple of other basically. Parameters here and let's see how the sample is going to look like. OK, so this is the funny part. OK, here let's see how the text information is used to condition the model. That's going to be the interesting part. OK, let me step into here. So that's going to be again. That's going to be the text to image unit. And what we're going to do is first we're going to embed the time steps. So first sinusoidal embeddings and then we just map them using simple linear layers here. And this is the interesting and different part. We're going to pass the tokens and the mask through the transformer part of our what's the name again of this thing? Text to image unit. So let's see how that thing is going to look like. Let me set a break point here. So it's going to be simply a transformer. So I don't think anything fancy will happen here. We get the embeddings, we do the positionally encoding, blah, blah, blah. We apply the transformer here. Let me just see whether there is something interesting. OK, so here is the interesting part. We are returning, as you recall from the paper, not just like the embedding for the last token, we're also returning all of the tokens, all of the embedding vectors from the previous layer or something like that. Let me just see what exactly is going on here. So we passed it through the transformer. We get this XF out and then we applied this layer norm. And as you can see here, so let me just show you the shape of XF out first. So the shape of this thing is going to be 128, 512. OK, and so we are going to do the following. We're going to take the last token and then map it using this transformer projection layer. And that's going to be simply, I assume, yeah, a linear layer. OK, and that's how we add we end up with XF projection, which is going to be 11512 in shape, I assume. Let me just verify that's indeed the case. Because the linear layer actually has a different output dimensionality. That's why we end up with 768. Not that important, but in any case. And we take here, we just do some permutation, but we take all of the tokens. And that's the part from the paper. So again, this is going to be 512 tokens. And this is the part we saw in the paper. Let me let me show you somewhere here. They mentioned it. So the output of this transformer is used in two ways. First, the final token embedding is used in place of a class embedding in the ADM model. Second, the last layer of token embeddings, a sequence of K feature vectors, is separately projected to the dimensionality of each attention layer throughout the ADM model and then concatenated to the attention context of each layer. So that's we're using this twofold. And that's why we're returning this dictionary containing both of these. OK, let's continue on. And now you can see we are adding to the temporal embedding. We're just having this this this projection. So this is this simple embedding vector as per the as the paper already mentioned. So we're doing this in the same way as if we were if we had just a simple class information and not text. So that's everything remains the same as in previous papers. But the different part is going to be how do we use this XF out. And we see here that we are passing out XF out throughout various modules of our unit model. So that's the difference. That's how we are conditioning using the temporal information plus the textual information. OK, let's go here. Finally, the whole magic is going to happen here. Let's find a module that's actually integrating the text information. I'm not sure which ones are used. Let me just double check that. So let's go to unit model here and let's find the model that's going to be using the textual information. I think it's going to be either textual block. Yeah, I think it's going to be the textual block. So this is the part is going to be using the textual information. The rest blocks. So here is the residual block. That one is only going to be using the temporal information as previously we saw this in in previous videos as well. So let's just step over. Whoops. Where am I? OK, so the first module is not the attention module. So we're not hitting the we'll have to kind of keep on doing this or I can alternatively hit F5 until we hit the first attention block where we're going to finally integrate this text information. OK, so we can verify that this is 512, 128 as expected. And you can hear see how it's integrated into the attention. So what we do with those tokens is we apply this encoder key value. So that's just going to be a convolutional like a layer applied. So we do some processing here first and then we end up with a shape that's going to be what one seven sixty eight hundred twenty eight hundred twenty eight tokens of this dimensionality. And then we're going to pass that into the attention and attention is this QKB attention object. Let me get kind of enter into it and let's just add a break point there. So let's just see how the information is integrated. We can see here. So we care about this encoder encoder variable. We form the queries, keys and values the usual way using the image features. And now we're trying to integrate the textual features and how they do this is as you can see, they grab they just do some reshaping and then they concatenate those textual basically keys to image keys. And they concatenate the textual values to image values. And then they just do the attention. So it's kind of bothersome. This is obviously not the only way you can do it, but it's basically exactly what the paper here said. So the output of this transformer is using two ways. Blah, blah, blah. And then second, the last layer of token embeddings, a sequence of key feature vectors is separately projected to the dimensionality of each attention layer. Throughout the ADI model and then concatenated to the attention context at each layer. So we just saw that that's indeed the case. And let me just continue here. We don't need to go through the this is the implementation of a transformer. I'm going to skip all of this, assuming you already know how this thing works. And that's it. So basically we saw how we are taking the textual information stored in this dictionary here, temporal information and passing all of that through UNET and forming the model output model. This is again going to contain the Epsilon. So the noise plus the covariance matrix is going to be one six, sixty four, sixty four, six, because we have three channels predicting the Epsilon and three channels predicting Sigma or the covariance matrix. OK, so next up. So everything else remains the same as in the first DDPM, the coding video I've done. The only difference here was the textual basically conditioning. OK, so we split the output into two parts, the Epsilon and the model var values. Again, there is this V vector that's actually predicted and then we form the model variance by by using these expressions here. Not as important. We saw that multiple times in previous videos. And finally, we predict. So we now pass the noisy image, the time steps and the output. And then we just predict the X zero here. Just some clamping. And finally, we predict the model mean. So everything here remains the same as in the original, as in the improved DDPM paper. So I'm going to skip all of this. And now this is where the difference will will will kick in. So here we just a sample from a normal distribution. We create this non zero mask. And this is the interesting part. So we now want to see how the conditioning is done using the clip model. Again, this is very similar in structure to the guided DDPM that I've covered in my previous video. So let's step inside here. I'm going to hit F 10. We are going to form the gradient and then we're just going to shift the predicted mean by the as you can see here. So Sigma, the covariance times the gradient. OK, and the grading itself already contains the scaling coefficient. So that's why we don't see it here. OK, let's hit F 10. And here are we in the conditioning function. So we previously saw how we created this textual embedding. Now we're going to do the fun part. So we're going to take the input noisy image, detach it from the computational graph and set requires graded true because we want to compute the gradients for this particular image. We're going to embed this image using the image pathway of the clip. So such that we end up with embedding image embedding here. So that's again, let me show you what we're doing there. We're doing the drawing I showed you a couple of minutes ago. So I think it's somewhere here. So we are just now forming this red embedding vector. And now we're going to basically, well, find the gradients of the product between the text embedding vector and the image embedding vector. OK, let's go back here. Let's step over image embeddings. So nothing. Well, it's again just walking me through the I have a bunch of break points here, but we can kind of ignore all of those safely. And what's this? My God, my God, I'm hitting every single I'm hitting every single break point here. Nothing interesting here. Nothing interesting here. So we form the embedding vector. We normalize it. And here we are. So we have the embedding vector for the image. And now we form the product and we additionally have this logic scale. So we form the loss and we just then apply. We just find the gradient of that loss and that gradient is now going to be returned. We just multiply with the gradient scale. So it's going to be three because that's the parameter we set in the main script. And we just return that gradient. But everything. So basically this is what we saw in the paper. And this is the only difference compared to your regular classifier guidance we saw in the previous video on diffusion models beat GANs on image synthesis paper. So this gradient is going to move our image towards the part of the space where that image resembles the prompt much more closely compared to if we were to just do it unconditionally. OK, let's step over here. We have the gradients. We find the new mean. And that's it, guys. Then we just sample from that new mean. Again, we have this this standard expression. We just have mean plus sigma and we just multiply by the by the noise here. This is how we sample from the Gaussian. And that's it. And this now just keeps on repeating. We're now going to repeat the same process for the next time step, which is going to be 98 and all the way to zero. And that's pretty much it. OK, guys, next up, I want to show you how the classifier free guidance works. So let me run the script text to image and let's start. A lot of the structure is shared between this script and the previous one. So I'll just be skipping even more than before. So we're creating the model and the diffusion here. Let me disable the break points such that we are just stepping over all of this. So we have modeling, diffusion, blah, blah, blah, shifting into GPU, loading the checkpoint, counting the number of parameters. Then we are forming the up sampling model here, which we don't care about. So we're going to skip all of that until we hit the prompt here and all painting of accordi. And now let's start. So same hyper parameters as before, same encoding of the tokens and then adding the padding and the mask. And this is where it's now starting to become a bit different. This is where the structure starts to deviate from the previous script. So now we encode the null prompt. So that's the empty prompt here, as you can see. And that's the part. I'll show you again in the paper. So let's find the OK, here it is. Here's the equation. So we have to pass the empty prompt and then we have to pass captions. So that means we'll effectively have two passes through the diffusion model. Or in this particular implementation, what I've done is they've sacrificed some memory for a single pass for time. And so they basically do a single pass, as we'll soon see, and they compute both of these expressions here. So both conditional caption as well as as on the empty well, null caption or empty caption. OK, so let's go back to the code. Here we can see that we now tokenize the empty empty sequence and we end up with tokens. We just contain pretty much just the I think end of tokens. So all of these, as you can see here, just contain the padding token. I think this is the padding token. Let me double check. That's indeed the case. So if I were to enter here, so this padded tokens, let me just find that thing. I think it's going to be somewhere in here. So there is a tokenizer part and we care about the BPE file. And here it is. So you can see here. Because we have passed empty tokens, we're basically going to pad the end token for the full context. Basically, that's what happens. We have end token as the result here. That's that's what these this fifty thousand two fifty six represents. It's just a pet token. OK. Now we're going to store both the tokens as well as the this unconditional tokens. And we're also going to store mask as well as the unconditional mask. So this mask is kind of boring. I assume it's going to have all forces. Yeah, as you could have expected. And now let's continue. So that's the key word arguments that we're going to use to condition our model during the four pass. OK, so here we have an interesting function. So let me now enable all of the breakpoints. We'll enter this one a bit later. This is where the whole magic happens when it comes to computing the classifier free guidance. And again, we have the P sample loop standard stuff. We want to generate an image that's going to be I think this is going to be sixty four sixty four resolution. Yep, that's the case. And let's continue here. We are passing the conditioning information here. And finally, you can see here that we are extracting batch size, which is one, whereas the full batch size here is going to be two. And the reason is this is the optimization they've done such that they have a single four pass and they end up with the with the result, even though they have to compute the and the output both for the empty token string as well as for the actual prompt we're using. OK, let's now dig into the code again. I'm skipping some parts because it's just boilerplate, some for loops and stuff. So we are generating the noisy image we're sampling from the I guess normal distribution here. We are forming the indices again. We have how many steps? Hundred. We are logging the progress. Nothing fancy there. And now we're starting the actual diffusion reverse process. OK, so here is the P sample function. This is the important function. Really, nothing has changed. So the model is going to be different here. That's the one that was defined in the main script. We're going to hit it a bit later. So you're going to see it otherwise. Nothing interesting here. No conditioning here because we are doing the classifier free guidance. And this is the conditional information. OK, we're going to see all of that in a second. Let me step over all of this. And this is where the magic of the classifier free guidance is going to kick in. So here we hit the model function that we define in the main script here. So let's see what we're doing. So we take the input images and we just grab the first part. So again, remember that we have two images here. So let me show you the shape. So we have two. So we're going to take the first part only. Because only the first part is going to contain the actual information of interest. And that's again the consequence of the way of them wanted to have a single forward pass, but like sacrifice the memory. And that's why there is this weird burden that we as someone analyzing this code base have to kind of suffer because of that optimization trick. But like it's fairly simple. Basically, you just take the first part of the of the image. We we know just as you can see here, we form this combined variable, which is just basically copy paste. We copy paste that first portion. And now we pass that basically together with time steps and together with conditioning information. We pass all of that through the unit with the transformer. So I'm going to disable the break points and just skip over this whole step over this this this function. Because we already saw this in the first script. We are basically just integrating the temporal information, integrating the textual information into the attention context, etc, etc. And as the output, we get the epsilons and the sigmas for both images. So let's see the model output. Basically, it should be 264, 64, because we have two images, six because we have both the epsilon and as well as the covariance matrix, so the sigma. And now we're going to split those, as you can see here, into epsilons and and and well, they call it rest. And then finally, we do another splitting into conditional epsilon and unconditional epsilon, because as you recall, the first image is going to be conditioned here with the conditional information. The second one is just going to have the tokens for the for the empty prompt. That's why we have these two variables here. So these two, again, are just are just these two variables here. And the epsilon conditioned on the empty prompt as well as the epsilon conditioned on the on the caption on the prompt. OK, let's go back to the code. And now here's the step. Here's the actual formula. We have the unconditional epsilon. We add up the scaled version of this difference. And that's it. That's the whole magic. And now we just again do this. Well, as I said, it's a consequence of justice of the optimization they've implemented. And then we return all of that. And that's it. Everything else remains pretty much the same. We split into we form the model variance. We form the let's see where there is something different here. We just predict the X start or the X zero, which is the I guess the newest image. And then we do some pre-processing. So nothing fancy there again will only care. So this model output contains copy pasted same results. So we only care. And let me kind of show you that's indeed the case. So the shape of this thing is two, three, sixty four, sixty four. If I were to take all zeros, so that means I'm going to just grab a single element from the first batch. If I were to take the second, the same element from the second batch, you can see they are the same because that's how, again, the optimization they're doing is playing out. OK, so all in all, at all times, only the zeros image contains the results of interest. The second one is just basically a consequence of the optimization. Now they form the mean and then they return all of these variables back. And that's it. Everything else remains the same. We just we don't have the conditioning now and we just form the final sample by sampling from this distribution sample. Here is still two, two, three, sixty four, sixty four. But only the first part contains the sort of zeros index of the batch dimension contains the actual information that we care about. And that's it, guys. I'm going to now go back into the script. I'm going to disable all of the breakpoints and I'm going to set a breakpoint to show images here. Now let's hit F5 and let's see how this is going to be generated. You can see here the progress is being tracked using the TQDM library, which is so cool. And now let me show you the image. So if I click F10 here, let me show you the image. So we have a corgi, as you can see here. So what was the prompt? Our prompt was, let me just find it, an oil painting of a corgi. And you can see indeed this is a low resolution version of that particular prompt because the models are already pre-trained for us. That's why we are getting these cool results. Keep in mind that OpenAI has not open source the training code, so we can only play with the sampling and generation in general and not with training. Let me quickly show you what's going to happen after we do the up sampling. I'm going to set another breakpoint here. Let me hit F5 here. And now we're generating the up sampled version. So if I hit F10 here, let me just see whether that's going to exit the program or not. Okay, unfortunately that exited the whole program, so I'm going to have to set something here like print blah blah, for example. And now let's hit run again. And we're going to sample both the corgi as well as the up sampled version of the corgi, although it may take some time. Cool, guys. That's pretty much it. I know there is a lot of information in this video. Do let me know if you have any feedback. Did you find this one useful or not? All of that's going to help me modify my future videos. And hopefully you learned something from this. Hopefully you like this combination of using paper, using code. There is a lot of information, but I think it might be useful for all of you. So here's the corgi again. This is the low res version. And now we're going to see the up sampled version. So here is the up sampled version. So that's it. I'm going to quickly also show you how the inpainting script looks like. Let me move it here. Let me try and run it. Let me see where the show images are. So I'm going to set a break point there and then there is the up sampling part. But yeah, I think we can just set a couple of break points here to visualize what's going on. So let me hit run and let's see what's going on. So first things first after the loading has happened. So again, the logic is actually not that complicated. It's a bit harder to form a mental model for how this inpainting works, but I encourage you to check out the code at your own pace and understand what's going on. So here is the image we are trying to impaint. So we basically have this grass image and then we masked all of the rows here. So I'm going to exit here. Let's hit F5. Now we're going to generate impaint this particular image. And let me now hit this. We're going to hit this break point here after 100 steps. So let me show you what we get as a result. We get a corgi on this on this on this on the grass. And that's because the prompt we are using is let me just find a prompt a corgi in a field, which makes sense. And finally, if I were to hit well F5 here, well, we'll just get the super like the up sampled version. But that's pretty much it. Again, guys, do let me know whether you find this video interesting and useful. Any feedback is super appreciated. And do subscribe to this channel if you liked this video. Share it out with your friends. And until next time, bye bye.
[{"start": 0.0, "end": 7.26, "text": " What's cracking guys in this video? I'm continuing on with the diffusion model tracking the machine learning coding series and this one"}, {"start": 7.26, "end": 11.4, "text": " I'll be covering glide, so that's a model that was a precursor to delete to"}, {"start": 11.96, "end": 16.78, "text": " Coming from open AI and I've previously covered the paper in great depth"}, {"start": 16.78, "end": 20.28, "text": " So if you want to have a deep dive into the paper itself, you can check it out"}, {"start": 20.28, "end": 22.28, "text": " I'm gonna link it somewhere here"}, {"start": 22.36, "end": 26.8, "text": " But in this video, I'll obviously be focusing on analyzing understanding the code"}, {"start": 26.8, "end": 32.36, "text": " I have three Jupyter notebooks prepared for you guys. And so we'll be going through those"}, {"start": 33.04, "end": 40.44, "text": " So what I've done is as always adjust I've downloaded I've cloned the repo here. You can find it"}, {"start": 40.44, "end": 43.040000000000006, "text": " I'll kind of link it in the video description"}, {"start": 43.6, "end": 46.480000000000004, "text": " But before we go there, I first want to"}, {"start": 47.2, "end": 53.28, "text": " Walk you through the paper as usually just to give you all of the necessary context"}, {"start": 53.28, "end": 60.36, "text": " Hopefully that's gonna be like 10 15 minutes quick overview compared to the deep dive. I've already previously made so without further ado"}, {"start": 60.36, "end": 63.68, "text": " let's start understanding how glide works and"}, {"start": 64.72, "end": 68.6, "text": " Then we'll we'll start doing the coding part cool"}, {"start": 69.28, "end": 70.4, "text": " so"}, {"start": 70.4, "end": 75.6, "text": " First things first as I said glide is a precursor to like a Dali model"}, {"start": 75.6, "end": 82.6, "text": " So they mentioned here when evaluated by human judges our samples are preferred to those from Dali. So this is the version 1"}, {"start": 83.47999999999999, "end": 89.97999999999999, "text": " 87% of the time when evaluated for photorealism and 69% of the time when evaluated for caption similarity"}, {"start": 90.32, "end": 93.91999999999999, "text": " The Dali version 1 actually did not even use diffusion models"}, {"start": 94.39999999999999, "end": 102.08, "text": " But the lead to did and it built directly up on glide. So yeah, we're slowly building our way towards more complex models"}, {"start": 102.08, "end": 109.64, "text": " Cool, you can see the some of the images here again here instead of compared to my previous video where I showed you the guided"}, {"start": 109.8, "end": 112.72, "text": " DDPM a model where you can could condition"}, {"start": 113.24, "end": 118.6, "text": " But using only like a class and for example image net class here"}, {"start": 119.0, "end": 121.88, "text": " Glide can actually cope with text with natural language"}, {"start": 121.88, "end": 128.88, "text": " So that's much much cooler and you can see some very cool examples here like a corgi varying a red bowtie and a purple"}, {"start": 128.88, "end": 131.12, "text": " party hat you can see that like you can"}, {"start": 131.12, "end": 133.04, "text": " composite"}, {"start": 133.04, "end": 137.92000000000002, "text": " different concepts together in a plausible way you can bind attributes such as a"}, {"start": 138.44, "end": 143.56, "text": " Purple party hat to to certain objects and red bowtie. So it's a fairly"}, {"start": 144.36, "end": 150.96, "text": " Powerful model and as I said, it was a precursor to delete you and we all know what delete you can now make"}, {"start": 151.56, "end": 155.84, "text": " Aside from image generation. It can also do image in painting"}, {"start": 155.84, "end": 162.28, "text": " You can see here you put a mask here and then you set some natural language prompt here zebras roaming in the field and"}, {"start": 162.48000000000002, "end": 167.6, "text": " Outcomes an image that's modified similarly here a girl hugging a corgi and a pedestal"}, {"start": 167.6, "end": 171.48000000000002, "text": " You can see how this dog here is is swapped by a corgi"}, {"start": 171.8, "end": 174.24, "text": " Etc etc. You can see a bunch of examples there"}, {"start": 174.24, "end": 178.8, "text": " So the model itself stands for guided language to image diffusion for generation and editing"}, {"start": 178.8, "end": 183.92000000000002, "text": " So generation is the the part that we all know about and anything is the in painting part cool"}, {"start": 183.92, "end": 187.44, "text": " so there is not this background about DDP M's are gonna skip all of that because"}, {"start": 188.16, "end": 192.83999999999997, "text": " Hopefully you watch some of my previous videos. This one is gonna build heavily upon those"}, {"start": 192.83999999999997, "end": 195.29999999999998, "text": " So do check out the whole series if you haven't so far"}, {"start": 195.72, "end": 201.6, "text": " You can also iteratively do this image in painting so you can see here how slowly they are adding"}, {"start": 201.6, "end": 206.35999999999999, "text": " They first had the corgi image and they add the table here then they add the ways here, etc. Etc"}, {"start": 206.35999999999999, "end": 209.35999999999999, "text": " So you can cannot iteratively improve your images"}, {"start": 209.36, "end": 216.36, "text": " Okay, so the parts which I want to focus on in this video are mostly about clip guidance and"}, {"start": 216.36, "end": 222.36, "text": " Classifier free guidance so I want to focus on those and I'll probably if I have enough time focus on in painting as well"}, {"start": 222.36, "end": 229.36, "text": " So without further ado, let's first introduce the classifier free guidance. So in the previous video, we saw how we can do classifier"}, {"start": 229.36, "end": 232.36, "text": " based guidance where you use a pre trained"}, {"start": 232.36, "end": 235.36, "text": " Well classifier and you use its gradients"}, {"start": 235.36, "end": 242.36, "text": " So the classifier was trained on noisy images. That's an important detail and then you just use the gradients from that classifier to kind of guide"}, {"start": 242.36, "end": 244.36, "text": " your your"}, {"start": 244.36, "end": 247.36, "text": " class conditional diffusion generation"}, {"start": 247.36, "end": 256.36, "text": " Here the cool thing about classifier free guidance is you do not have to train a separate classifier on pre trained on noisy images"}, {"start": 256.36, "end": 259.36, "text": " It's it's much easier and and that's better"}, {"start": 259.36, "end": 262.36, "text": " Okay, so let me try to briefly explain how classifier free guidance works"}, {"start": 262.36, "end": 268.36, "text": " Again, as I just mentioned, this method does not require a separate classifier"}, {"start": 268.36, "end": 272.36, "text": " That's a big plus for for this method compared to the classifier guidance"}, {"start": 272.36, "end": 282.36, "text": " And then they say here so for classifier free guidance the label in a class conditional diffusion model is replaced with a null label with a fixed probability during training"}, {"start": 282.36, "end": 289.36, "text": " This is I think usually 10 or 20 percent during sampling the output of the model is extrapolated further in the direction of the model"}, {"start": 289.36, "end": 299.36, "text": " The output of the model is extrapolated further in the direction of the class conditional model and away from the null conditional model like so"}, {"start": 299.36, "end": 306.36, "text": " And you can see here the expression we basically have the one where we condition on those on the on the null label"}, {"start": 306.36, "end": 322.36, "text": " And then we just add this this basically this difference between the class conditional and the null conditional and we just use this guidance scale so the parameter s to well to to to modulate how much we want to go in in that direction"}, {"start": 322.36, "end": 332.36, "text": " And how I think about this in my head the mental model I have is basically you have this thing here tells you gives you the the noise in the unconditional case"}, {"start": 332.36, "end": 341.36, "text": " So it's going to give you some noise in some is going to be some point in the end dimensional space where and is basically the number of dimensions in your image"}, {"start": 341.36, "end": 358.36, "text": " So the resolution times the number of channels and so now once you have that point you then plot this point of where the model thinks the class conditional noise should be"}, {"start": 358.36, "end": 367.36, "text": " So if it outputs point like here then you can form the vector by just kind of subtracting one from another and you get something like this"}, {"start": 367.36, "end": 377.36, "text": " And then what what this final expression is going to be about is OK the model outputs the null conditioned noise and it's here"}, {"start": 377.36, "end": 384.36, "text": " And we know that if you want to move towards that particular class on upon which we are conditioning we need to move in this direction"}, {"start": 384.36, "end": 395.36, "text": " And because of that we just add that vector and we modulated with the with the scalar s to basically well to to start generating images that are from that particular class"}, {"start": 395.36, "end": 406.36, "text": " So that's the I literally have a geometric interpretation now whether this is completely correct or not I don't know but like I think I think it does sound like a like a good mental model"}, {"start": 406.36, "end": 416.36, "text": " In any case let's now see how we can replace the simple labels by the natural language because that's what we care about when it comes to to glide"}, {"start": 416.36, "end": 427.36, "text": " So what what they do is so to implement classifier free guidance with generic text prompts we sometimes replace text captions with an empty sequence which we also refer to as this"}, {"start": 427.36, "end": 440.36, "text": " Well zero crossed I guess during the training we then guide towards the caption using the modified prediction blah blah blah OK so you can see that everything remains the same except that we are now conditioning on caption"}, {"start": 440.36, "end": 461.36, "text": " Like and not on the simple label and that's pretty much it so the second thing they do is they use clip for guidance because here you cannot have like a thousand classes or whatnot because languages obviously has infinite possibilities of compositing"}, {"start": 461.36, "end": 476.36, "text": " And yeah there is infinite number of sentences I guess so you have to use something like clip and the only difference compared to the classifier guidance we saw in the previous video is that here you're looking for a gradient of the product between the caption and the image"}, {"start": 476.36, "end": 491.36, "text": " So let me read this for you perturbing the so when applied to the future models these techniques are similar to classifier guidance perturbing the reverse process mean along the gradient of the text image product with respect to the image OK"}, {"start": 491.36, "end": 511.36, "text": " So what that means is basically if you recall how we how we've done the classifier guidance we output like a mean and then on top of that we shift that mean by the scale version of the covariance matrix times the gradient of our classifier"}, {"start": 511.36, "end": 524.36, "text": " So now the only thing that differs in this in this clip guidance is that this these gradients here because we are now dealing with text and not with with classes are going to be this thing here"}, {"start": 524.36, "end": 547.36, "text": " So basically you encode a certain caption and you want to make sure you want to find you want to learn how to tweak the input image such that such that the product between the image feature from that image and the text embedding here is maximized"}, {"start": 547.36, "end": 567.36, "text": " And that means that the image is belonging more and more to that class or differently said it's becoming better and better described by that particular caption So that's that's all like that's a small tweak that we have to make in order to to make the clip thing and the text conditioning to work"}, {"start": 567.36, "end": 582.36, "text": " Again let me just quickly draw a picture here it might be a bit easier to understand so we have an input image here and we have our clip model here OK So here's clip"}, {"start": 582.36, "end": 607.36, "text": " So now we now have some type of a prompt like I don't know like maybe something simple like a dog OK So we encode that thing we pass it through the clip again we pass it through clip here so this same model basically and outcomes outcomes some some in there here so this is going to be an embedding that corresponds to this prompt here"}, {"start": 607.36, "end": 627.36, "text": " Next up we pass the image through the clip and we get a different embedding here OK This is going to be the image embedding let me maybe choose a different color just for the sake of like not to come in order not to confuse this So here we have a prompt for the"}, {"start": 627.36, "end": 648.36, "text": " So we have the text embedding and here we have the image embedding and now the gradients are going to tell us how we should tweak the image how should we tweak the image such that this vector is much closer in the image text embedding space OK Hopefully that was clear enough"}, {"start": 648.36, "end": 668.36, "text": " Now let me briefly tell you a couple of things about their training so they mentioned here for our main experiments we train a 3.5 billion parameter text conditional diffusion model at 64 times 64 resolution and another 1.5 billion parameter text conditional up sampling diffusion model to increase the resolution to 256 times 256"}, {"start": 668.36, "end": 688.36, "text": " For clip guidance we also train a noise aware 64 times 64 VAT clip model OK So the only difference compared to the previous video on the guided DDPMs is basically here we have additional transformer that's going to be used to text condition our diffusion model instead of just doing a simple class conditioning"}, {"start": 688.36, "end": 717.36, "text": " It's not that like big of a difference to be honest so a couple more details here to condition on the text we first encoded into sequence of key tokens and feed these tokens into transformer model OK The output of this transformer is used in two ways so first the final token embedding is used in place of a class embedding in the ADM model so that's the model from the previous paper again the paper is called I think diffusion models beat GANs on image synthesis"}, {"start": 717.36, "end": 739.36, "text": " And second the last layer of token embeddings a sequence of key feature vectors is separately projected to the dimensionality of each attention layer throughout the ADM model so the fusion model and then concatenated to the attention context at each layer OK So that's how we integrate this textual conditioning information into the ADM into the diffusion model"}, {"start": 739.36, "end": 769.36, "text": " That's an additional detail I want you to understand OK Finally here they mentioned how like let me read this for you so after the initial training run we fine tuned our base models to support unconditional image generation this training procedure is exactly like pre training except 20% of token of text token sequences are replaced with the empty sequence this way the model retains its ability to generate text conditional outputs but can also generate images unconditionally and we need this ability so that we can"}, {"start": 769.36, "end": 791.36, "text": " such that we can well use the the the basically the classifier free guidance effectively OK guys that's pretty much it I'm going to skip everything else again the results are basically better when we use classifier free guidance compared to clip guidance so that's one of the reasons this is so cool"}, {"start": 791.36, "end": 811.36, "text": " I'm gonna skip all of this this is not that vital and finally worth mentioning the only model that they've open source is the so called glide filtered where they say here we filtered out training images containing people to reduce the capabilities of the model in many people sent you problematic use cases"}, {"start": 811.36, "end": 840.36, "text": " the model is also way smaller compared to their biggest version and they also filter out some other images such as like hate symbols et cetera et cetera so just worth having that in mind finally the model such as glide was still struggling a lot with with certain well let's call it weird phrases whereas the lead to became much better with these still not perfect but but better than then then what glide could achieve"}, {"start": 840.36, "end": 860.36, "text": " here we can see an example a mouse hunting a lion and we can see that the model does does not know that there does not understand the direction so here it appears as if the lion was hunting the mouse and not the other way around so it understands the association but not the the the causal relationship here cool"}, {"start": 860.36, "end": 887.36, "text": " okay that's pretty much it now let's go back to decode let's understand how this looks in actual code okay I'm gonna first start with this simple clip guided script so we're going to use the clip the pre trained clip model on on that's like noise up there and we're going to use it to to guide our generation so let me run this thing"}, {"start": 887.36, "end": 909.36, "text": " and again as usually I'm going to skip over everything that's not core to understanding what I'm trying to explain in a particular script here we want to understand how clip is used to guide the generation process so we just grab the devices and I'm going to have I have a GPU so it's going to be a GPU here"}, {"start": 909.36, "end": 938.36, "text": " some options default options for the model and diffusion we saw this in the first video like they're obviously are reusing a lot of the code, which is a good thing. And so I'm going to scheme across all of this. So time a step through spacing. So what it does is if the model was trained on like 1000 steps, you can now instead during the sampling procedure have only hundred steps and still get good results, good high quality samples."}, {"start": 938.36, "end": 961.36, "text": " Then we're going to form the model and the diffusion here. So model is basically a unit model and diffusion is contains all the constants that are necessary for the diffusion. Okay, so we'll need to step into this part because the model is a bit different compared to previous papers. So the model now contains also the transformer because of the text conditioning."}, {"start": 961.36, "end": 984.36, "text": " So let's see how that looks like diffusion itself looks pretty much the same as in previous videos. So I'm going to skip over that. But we're going to jump into this create model function. Let me just hit F5 and let's get into the function itself. Okay, so a couple of details here depending on the image size."}, {"start": 984.36, "end": 1003.36, "text": " We're going to set up the unit in a particular way number number of channels is going to be different compared to depending on the image resolution. Again, we've seen this multiple times and they just have these attention blocks for particular resolutions. Not that important."}, {"start": 1003.36, "end": 1024.3600000000001, "text": " And finally, we're going to have this text to image unit. So again, it's just a simple unit plus the transformer. That's everything that Glide changed compared to previous papers. Okay, now we're going to form the text to image unit here. There is a lot of parameters. I don't want to explain every single one of those."}, {"start": 1024.36, "end": 1044.36, "text": " There is also the encoder part. I'm going to quickly show you what this is. Basically, they have a pre trained BPE encoder that are going that they are going to use. And I explained this in the open AI clip video. So I'm going to link it somewhere here. You can check it out if you want to understand a bit better what's going on here."}, {"start": 1044.36, "end": 1059.36, "text": " How the actual tokenization works, how that magic looks like. Then you can check out that video. But here I'm just going to skip all of that and hit that five to enter the constructor of the text to image unit. Okay, so let's see what's different here."}, {"start": 1059.36, "end": 1076.36, "text": " We call the constructor of the super class and that's going to be just a simple unit model. I'm not going to show you the unit because we already saw that in previous videos. As I said, I have to kind of depend on previous videos because otherwise this would be three or four hours long video."}, {"start": 1076.36, "end": 1095.36, "text": " I kid you not. Okay, so we construct the we construct the unit model here. I'm going to skip over everything here. I'm just going to hit F5 and we have our unit constructed. Let's go outside here."}, {"start": 1095.36, "end": 1112.36, "text": " And now, so this is the different part compared to previous papers. They have a transformer. So they form a transformer that has a particular context 128 tokens, certain width 512 dimensions, number of layers 16, etc, etc."}, {"start": 1112.36, "end": 1127.36, "text": " Then they have they use some layer norm, not that important. They create some token embedding layer here. So this one is basically going to embed our tokens into certain well into a certain vector space."}, {"start": 1127.36, "end": 1136.36, "text": " And then we also have positional embeddings. We don't need more than 128 because that's going to be the max context that this transformer is going to use."}, {"start": 1136.36, "end": 1147.36, "text": " And we finally have this transformer projection linear layer. We're going to see how that one is used a bit later. Okay, finally, there is this padding embedding. We'll see how that all fits together a bit later."}, {"start": 1147.36, "end": 1159.36, "text": " Okay, so cash stuff, not that vital. I'm going to skip all of that. And finally, I'm going to skip the diffusion model because we've seen that already. I'll have to skip this."}, {"start": 1159.36, "end": 1173.36, "text": " And here we are. So again, we have a unit with transformer and we have our diffusion model here. So basically those two together form form a diffusion model."}, {"start": 1173.36, "end": 1189.36, "text": " Okay, let's go back into the main script. We just do some some boilerplate for machine learning, just in conversion to FP 16. We put the model onto GPU will load the actual weights, and then we print the number of parameters here."}, {"start": 1189.36, "end": 1203.36, "text": " Okay, we repeat the same thing for the up sampling model. I'm going to skip all of this because we will not be focusing on the up sampling model. So I'm going to kind of skip over all of this until we get back into the main script."}, {"start": 1203.36, "end": 1211.36, "text": " I should have toggled off the break points. But yeah, I guess next time."}, {"start": 1211.36, "end": 1212.36, "text": " Cool."}, {"start": 1212.36, "end": 1226.36, "text": " I think we are there almost. Basically, okay, so we created the up sampling model. Now we create the clip model. Okay, let's see how clip looks like. So clip, as you can see here, I'm going to just step into here."}, {"start": 1226.36, "end": 1246.36, "text": " We grab some default config path so they have this config YAML file that specifies certain properties of our model. So of the clip model so you can see it here again too many details to cover. I'm just going to continue on execution in the in the create clip function."}, {"start": 1246.36, "end": 1260.36, "text": " So here we are. We create a simple tokenizer so clip will have a different a bit different tokenizer compared to the diffusion model. Again, we can treat it as a black box for all practical purposes."}, {"start": 1260.36, "end": 1261.36, "text": " Okay."}, {"start": 1261.36, "end": 1276.36, "text": " Now, as I already said, I covered clip in one of the previous videos. If you really want to understand how the code works and how clip works to check it out or or also check out my paper video on clip. I've done that as well. I can link those somewhere here."}, {"start": 1276.36, "end": 1288.36, "text": " We can briefly jump into the text encoder. Let me see whether I have basically. Okay, so let's kind of let me let me quickly show you how this text encoder is going to look like."}, {"start": 1288.36, "end": 1306.36, "text": " I'm going to skip some parts so they just form the causal mask here because well does that make sense. Yeah, it does because we are here dealing with with text. But then again, we are not generating text. So I guess you could use bi-directional you could use non causal masks as well here."}, {"start": 1306.36, "end": 1320.36, "text": " But in any case, let me get a scheme over all of this. So now they start stacking these blocks. The first block is just going to be a text embedding block, which means you're going to take tokens and then you're going to embed them into certain space."}, {"start": 1320.36, "end": 1331.36, "text": " I think I'm going to show you this in the in the I'm going to just put a well, I don't want to even put a breakpoint here. It's just a simple embedding layer. I'm going to skip that."}, {"start": 1331.36, "end": 1345.36, "text": " And then they form they had a bunch of transformer blocks and then at the end they just extract the basically embedding that's above the last token in the sequence."}, {"start": 1345.36, "end": 1354.36, "text": " So that's how clip works. I'm going to have to skip all of that and let's continue. So we have the text encoders. Basically, it's a simple transformer."}, {"start": 1354.36, "end": 1365.36, "text": " It first embeds the tokens, then it processes them using the transformer blocks. And then at the end, it just extracts, as I said, the last embedding. That's it."}, {"start": 1365.36, "end": 1380.36, "text": " Next up, we have image encoder. Image encoder is VAT. So they're using vision transformer. Again, I'll have to just treat this as a black box. Let me see whether there's something interesting here."}, {"start": 1380.36, "end": 1388.36, "text": " OK, this time they're using a non causal mask, as you can see here, which makes sense for images."}, {"start": 1388.36, "end": 1401.36, "text": " Let's see what else. So again, similar structure. They form these blocks. The first one is going to be image embedding. So we're going to basically do the patchify like layer of the VAT, which means you're going to treat image as a bunch of patches."}, {"start": 1401.36, "end": 1410.36, "text": " And then you're going to embed those patches into certain into certain well, vector space. And then you're going to just treat those as tokens."}, {"start": 1410.36, "end": 1418.36, "text": " And then it's pretty much the same as if you were to just use transformer. And you can see here they do have a bunch of transformer blocks."}, {"start": 1418.36, "end": 1426.36, "text": " And at the end, they have image feature extractor that's going to basically extract. Let me just see what's going on there."}, {"start": 1426.36, "end": 1434.36, "text": " OK, we're going to I'm going to put a like a simple break point in the four functions. We're going to see that a bit later once we start using clip actually."}, {"start": 1434.36, "end": 1444.36, "text": " OK, so that's it. We have the image, you go to we have the text encoder. We form this logic scale. That's a learnable parameter."}, {"start": 1444.36, "end": 1453.36, "text": " We might see how it's used a bit later. And finally, they form the actual clip model. So he has a bunch of helper functions."}, {"start": 1453.36, "end": 1460.36, "text": " What we care about is, well, for now, the I can just keep the function. That's not that important."}, {"start": 1460.36, "end": 1468.36, "text": " We're going to see everything a bit later during the actual four pass. OK, so now we just load the checkpoints for the image encoder."}, {"start": 1468.36, "end": 1474.36, "text": " We load the checkpoint for the text encoder. This is where the actual fun starts."}, {"start": 1474.36, "end": 1478.36, "text": " We're going to take the prompt and oil painting of a corgi."}, {"start": 1478.36, "end": 1487.36, "text": " We're going to encode that and we're going to generate an image that's going to have that's going to be well described by this prompt."}, {"start": 1487.36, "end": 1495.36, "text": " So the guidance scale is going to be three. So that's the S parameter that we use to multiply the gradients coming from the clip model."}, {"start": 1495.36, "end": 1501.36, "text": " We have some temperature that's not that important. And now we just tokenize. Again, we're going to treat this as a black box."}, {"start": 1501.36, "end": 1511.36, "text": " We take the prompt and we talk and I said using the diffusion models tokenizer into tokens. OK, so let me show you how that thing is going to look like."}, {"start": 1511.36, "end": 1515.36, "text": " It's going to be as you can see here, there is seven tokens in total."}, {"start": 1515.36, "end": 1526.36, "text": " And then there's this special utility function that's going to just pad the these tokens such that we fill in the context for the transformer."}, {"start": 1526.36, "end": 1534.36, "text": " That's a part of our diffusion model. OK, let me show you what that means. In practice, that means we're going to have I think we are using hundred twenty eight."}, {"start": 1534.36, "end": 1544.36, "text": " So that's this text context parameter. So basically if I were to print the number of tokens, you can see there is hundred twenty eight tokens now."}, {"start": 1544.36, "end": 1551.36, "text": " And the mask just tells you which ones are actually legit. And the first seven ones are actually true tokens."}, {"start": 1551.36, "end": 1563.36, "text": " Everything else is just padding, which is specified by setting false for all of those other tokens. OK, so so the reason we're doing this is so that we can basically take multiple prompts and do this in a single for a pass."}, {"start": 1563.36, "end": 1571.36, "text": " And that's why we cannot pad to a fixed dimension. OK, so now we form this model keyword arguments dictionary."}, {"start": 1571.36, "end": 1577.36, "text": " We just store the tokens of exercise is just one. So that means we're going to we won't do anything smart here."}, {"start": 1577.36, "end": 1585.36, "text": " We just store the tokens and the mask. And this is going to be later used to condition the diffusion model. We're going to see that in a second."}, {"start": 1585.36, "end": 1592.36, "text": " OK, so there is this condition function. I'm just going to enable basically all of the breakpoints."}, {"start": 1592.36, "end": 1597.36, "text": " And this is going to we're going to hit this function a bit later. It's going to be very important."}, {"start": 1597.36, "end": 1603.36, "text": " We are actually going to have to enter this function. So let me click F10 and enter this one."}, {"start": 1603.36, "end": 1611.36, "text": " So we're going to form this. Well, basically the embedding for the prompts and then we're going to return the this conditioning function."}, {"start": 1611.36, "end": 1619.36, "text": " OK, so let's see what's going to happen here. So we just basically pass prompts through the textual pathway of the clip model."}, {"start": 1619.36, "end": 1624.36, "text": " Nothing fancy there. We've seen this multiple times. We've seen this in my previous clip video."}, {"start": 1624.36, "end": 1633.36, "text": " So we've called this in code prompts that's just going to, as you can see here, iterate through the list of prompts."}, {"start": 1633.36, "end": 1638.36, "text": " We only have a single prompt, so nothing fancy will happen there. We're going to treat the tokenizer as a black box."}, {"start": 1638.36, "end": 1645.36, "text": " So this is going to return us some sequence of tokens. Let me let me show you what we're going to get here."}, {"start": 1645.36, "end": 1654.36, "text": " So if I run this, we have like, as you can see here, six tokens and then we're going to call again this padded tokens and length function,"}, {"start": 1654.36, "end": 1663.36, "text": " which is going to just pad everything. OK, I kind of entered here. Let me just skip over this. We don't want to go there."}, {"start": 1663.36, "end": 1670.36, "text": " So again, we end up with these subtokens, which are going to be just padded versions. Whoops."}, {"start": 1670.36, "end": 1677.36, "text": " So if I just grab the subtokens here and if I print this, you can see we have the same tokens as above."}, {"start": 1677.36, "end": 1686.36, "text": " Hmm. For some reason, there is some difference here. Let me just see. Oh, no. OK, we have the start of sentence token here"}, {"start": 1686.36, "end": 1692.36, "text": " and the end of sentence token here. And we have our tokens here. And then we have additionally the padding."}, {"start": 1692.36, "end": 1701.36, "text": " So it's a bit different. As I said, Clip has a bit different tokenizer compared to the tokenizer that we have in our diffusion model."}, {"start": 1701.36, "end": 1708.36, "text": " And now we're just going to append that and we are done. We just converted to torch tensors and we are back here."}, {"start": 1708.36, "end": 1720.36, "text": " So now we have basically tokens and their associated lengths. Next up, we're going to pass these tokens through the text pathway."}, {"start": 1720.36, "end": 1725.36, "text": " So that's again just going to embed the tokens, then apply a bunch of transformer blocks."}, {"start": 1725.36, "end": 1733.36, "text": " And ultimately, we are going to extract a particular a particular token. So let's see how the extractor part works."}, {"start": 1733.36, "end": 1741.36, "text": " So again, this is the this is the forward pass of our of our text encoder."}, {"start": 1741.36, "end": 1749.36, "text": " And we don't care about most of these parts because that's just going to do as I said, just going to do the embedding and then number of transformer blocks."}, {"start": 1749.36, "end": 1755.36, "text": " And ultimately, we'll have this. I think this is the part that we care about. Let's see how this is going to work."}, {"start": 1755.36, "end": 1763.36, "text": " So we call the text extractor here. And here it is. So the text feature extractor. Let's see what it does."}, {"start": 1763.36, "end": 1773.36, "text": " Basically, as you can see here, text length tells us how big how many tokens do we have fed into the transformer."}, {"start": 1773.36, "end": 1779.36, "text": " And it's going to be equal to seven. So that tells you that that's the index of the last token."}, {"start": 1779.36, "end": 1786.36, "text": " And then we just gather. So we let me show you what text is. So text is a print shape here."}, {"start": 1786.36, "end": 1792.36, "text": " So we have 77 tokens. Some of those tokens are obviously padding."}, {"start": 1792.36, "end": 1798.36, "text": " Some of those tokens are start and end of sentence tokens and 512 is just a dimensionality."}, {"start": 1798.36, "end": 1803.36, "text": " So what we do by by doing the gather, we are basically going to extract the last token here."}, {"start": 1803.36, "end": 1809.36, "text": " So that means we're going to extract the seventh, the well, the eighth token."}, {"start": 1809.36, "end": 1815.36, "text": " Let's step over there and let me convince you that that's indeed the case if I were to take the text."}, {"start": 1815.36, "end": 1823.36, "text": " And then if I were to take the seventh here and let's take the first three elements, you can see them here."}, {"start": 1823.36, "end": 1830.36, "text": " And now let's let me show you the let me just see what the shape of this thing is. OK, so let's now take zero zero."}, {"start": 1830.36, "end": 1842.36, "text": " And let's take also first three elements. And those. Yeah, as you can see, these are the same because we literally just extracted whatever the embeddings are on top of that seventh token."}, {"start": 1842.36, "end": 1849.36, "text": " That's pretty much it. And finally, we pass that X through the layer norm."}, {"start": 1849.36, "end": 1855.36, "text": " And finally, we just return the basically the mapped version of that particular token."}, {"start": 1855.36, "end": 1863.36, "text": " So let's see what F is. I think it's just a simple. Yeah, fine. It's just a basically a simple linear layer with some tweaks."}, {"start": 1863.36, "end": 1870.36, "text": " But I will not get into those details because as you can imagine, this is a very huge code base and I cannot cover everything."}, {"start": 1870.36, "end": 1885.36, "text": " OK, so let's go outside here and we end up with H being what H shape is going to be one five hundred twelve. So we have a single token that is supposed to be our representation of our input prompt."}, {"start": 1885.36, "end": 1894.36, "text": " OK, continuing on here, just some asserts. And that's it. We have our ZT tokens. That's the same thing."}, {"start": 1894.36, "end": 1908.36, "text": " So again, one five twelve. And we're going to now normalize it just using the well, I guess this is just L2 norm as as previously as we've seen previously in the open clip video."}, {"start": 1908.36, "end": 1915.36, "text": " OK, so that's the CT is that the and now we return this con function. I have a breakpoint here."}, {"start": 1915.36, "end": 1921.36, "text": " So we're going to see it a bit later when the time comes. But for now, let's continue on with the main script."}, {"start": 1921.36, "end": 1930.36, "text": " So a recap, because this was fairly overwhelming potentially. So we formed our diffusion models and we loaded the actual checkpoints here."}, {"start": 1930.36, "end": 1935.36, "text": " We can ignore the sampling model. Then we we loaded the clip model."}, {"start": 1935.36, "end": 1943.36, "text": " We have a prompt and we basically encoded a prompt here. We have a mask and tokens for that prompt."}, {"start": 1943.36, "end": 1951.36, "text": " And now we ended up creating this this embedding vector for our prompt using the clip model. OK, so that's it."}, {"start": 1951.36, "end": 1959.36, "text": " Now we're going to go through the P sample loop. So that's going to be the reverse process of our diffusion model."}, {"start": 1959.36, "end": 1965.36, "text": " Let's see what we're going to do here. We want to generate an image that's going to be of this particular resolution."}, {"start": 1965.36, "end": 1976.36, "text": " So it's going to be a sixty four sixty four RGB. And let's see what's interesting here. We're going to use the tokens and mask to condition the model."}, {"start": 1976.36, "end": 1981.36, "text": " So that's going to be different compared to our previous videos with the DPMs."}, {"start": 1981.36, "end": 1987.36, "text": " In any case, let's now jump into this P sample loop."}, {"start": 1987.36, "end": 1999.36, "text": " Again, I kind of skipped the P sample loop because everything it does is just basically iterates through this dysfunction loop progressive and starts generating multiple samples."}, {"start": 1999.36, "end": 2005.36, "text": " But I think we are generating just a single sample so that it doesn't even matter. So we can just kind of skip over all of this."}, {"start": 2005.36, "end": 2015.36, "text": " And now the actual fun starts. So we start as we as you may know from the completely like a normal distribution."}, {"start": 2015.36, "end": 2020.36, "text": " We can just sample our image from a normal distribution. That means we have initially completely noisy image."}, {"start": 2020.36, "end": 2032.36, "text": " And then we're going to slowly work our way from there and denoise that image until we get the image that corresponds to the prompt that we are using for this for this program."}, {"start": 2032.36, "end": 2045.36, "text": " OK, so we'll have a hundred steps of the reverse process and just some DQDM stuff for for monitoring the progress."}, {"start": 2045.36, "end": 2052.3599999999997, "text": " But we're going to now do hundred times in order to hundred denoising steps so that we end up with an image."}, {"start": 2052.3599999999997, "end": 2060.3599999999997, "text": " OK, so initially T is going to be 99. As you can see here. And now the whole fun happens in this P sample function."}, {"start": 2060.36, "end": 2071.36, "text": " So we're going to pass the fusion model so that's unit with transformer. We're going to pass the noisy image time step and a couple of other basically."}, {"start": 2071.36, "end": 2079.36, "text": " Parameters here and let's see how the sample is going to look like. OK, so this is the funny part."}, {"start": 2079.36, "end": 2085.36, "text": " OK, here let's see how the text information is used to condition the model. That's going to be the interesting part."}, {"start": 2085.36, "end": 2096.36, "text": " OK, let me step into here. So that's going to be again. That's going to be the text to image unit."}, {"start": 2096.36, "end": 2101.36, "text": " And what we're going to do is first we're going to embed the time steps."}, {"start": 2101.36, "end": 2107.36, "text": " So first sinusoidal embeddings and then we just map them using simple linear layers here."}, {"start": 2107.36, "end": 2118.36, "text": " And this is the interesting and different part. We're going to pass the tokens and the mask through the transformer part of our what's the name again of this thing?"}, {"start": 2118.36, "end": 2123.36, "text": " Text to image unit. So let's see how that thing is going to look like."}, {"start": 2123.36, "end": 2130.36, "text": " Let me set a break point here. So it's going to be simply a transformer. So I don't think anything fancy will happen here."}, {"start": 2130.36, "end": 2140.36, "text": " We get the embeddings, we do the positionally encoding, blah, blah, blah. We apply the transformer here. Let me just see whether there is something interesting."}, {"start": 2140.36, "end": 2153.36, "text": " OK, so here is the interesting part. We are returning, as you recall from the paper, not just like the embedding for the last token, we're also returning all of the tokens, all of the embedding vectors from the previous layer or something like that."}, {"start": 2153.36, "end": 2163.36, "text": " Let me just see what exactly is going on here. So we passed it through the transformer. We get this XF out and then we applied this layer norm."}, {"start": 2163.36, "end": 2172.36, "text": " And as you can see here, so let me just show you the shape of XF out first. So the shape of this thing is going to be 128, 512."}, {"start": 2172.36, "end": 2185.36, "text": " OK, and so we are going to do the following. We're going to take the last token and then map it using this transformer projection layer. And that's going to be simply, I assume, yeah, a linear layer."}, {"start": 2185.36, "end": 2197.36, "text": " OK, and that's how we add we end up with XF projection, which is going to be 11512 in shape, I assume. Let me just verify that's indeed the case."}, {"start": 2197.36, "end": 2204.36, "text": " Because the linear layer actually has a different output dimensionality. That's why we end up with 768. Not that important, but in any case."}, {"start": 2204.36, "end": 2214.36, "text": " And we take here, we just do some permutation, but we take all of the tokens. And that's the part from the paper."}, {"start": 2214.36, "end": 2223.36, "text": " So again, this is going to be 512 tokens. And this is the part we saw in the paper. Let me let me show you somewhere here."}, {"start": 2223.36, "end": 2231.36, "text": " They mentioned it. So the output of this transformer is used in two ways. First, the final token embedding is used in place of a class embedding in the ADM model."}, {"start": 2231.36, "end": 2243.36, "text": " Second, the last layer of token embeddings, a sequence of K feature vectors, is separately projected to the dimensionality of each attention layer throughout the ADM model and then concatenated to the attention context of each layer."}, {"start": 2243.36, "end": 2257.36, "text": " So that's we're using this twofold. And that's why we're returning this dictionary containing both of these. OK, let's continue on. And now you can see we are adding to the temporal embedding."}, {"start": 2257.36, "end": 2265.36, "text": " We're just having this this this projection. So this is this simple embedding vector as per the as the paper already mentioned."}, {"start": 2265.36, "end": 2276.36, "text": " So we're doing this in the same way as if we were if we had just a simple class information and not text. So that's everything remains the same as in previous papers."}, {"start": 2276.36, "end": 2285.36, "text": " But the different part is going to be how do we use this XF out. And we see here that we are passing out XF out throughout various modules of our unit model."}, {"start": 2285.36, "end": 2293.36, "text": " So that's the difference. That's how we are conditioning using the temporal information plus the textual information. OK, let's go here."}, {"start": 2293.36, "end": 2301.36, "text": " Finally, the whole magic is going to happen here. Let's find a module that's actually integrating the text information."}, {"start": 2301.36, "end": 2313.36, "text": " I'm not sure which ones are used. Let me just double check that. So let's go to unit model here and let's find the model that's going to be using the textual information."}, {"start": 2313.36, "end": 2322.36, "text": " I think it's going to be either textual block. Yeah, I think it's going to be the textual block. So this is the part is going to be using the textual information."}, {"start": 2322.36, "end": 2335.36, "text": " The rest blocks. So here is the residual block. That one is only going to be using the temporal information as previously we saw this in in previous videos as well."}, {"start": 2335.36, "end": 2343.36, "text": " So let's just step over. Whoops. Where am I? OK, so the first module is not the attention module."}, {"start": 2343.36, "end": 2355.36, "text": " So we're not hitting the we'll have to kind of keep on doing this or I can alternatively hit F5 until we hit the first attention block where we're going to finally integrate this text information."}, {"start": 2355.36, "end": 2365.36, "text": " OK, so we can verify that this is 512, 128 as expected. And you can hear see how it's integrated into the attention."}, {"start": 2365.36, "end": 2377.36, "text": " So what we do with those tokens is we apply this encoder key value. So that's just going to be a convolutional like a layer applied."}, {"start": 2377.36, "end": 2389.36, "text": " So we do some processing here first and then we end up with a shape that's going to be what one seven sixty eight hundred twenty eight hundred twenty eight tokens of this dimensionality."}, {"start": 2389.36, "end": 2397.36, "text": " And then we're going to pass that into the attention and attention is this QKB attention object."}, {"start": 2397.36, "end": 2405.36, "text": " Let me get kind of enter into it and let's just add a break point there. So let's just see how the information is integrated."}, {"start": 2405.36, "end": 2410.36, "text": " We can see here. So we care about this encoder encoder variable."}, {"start": 2410.36, "end": 2433.36, "text": " We form the queries, keys and values the usual way using the image features. And now we're trying to integrate the textual features and how they do this is as you can see, they grab they just do some reshaping and then they concatenate those textual basically keys to image keys."}, {"start": 2433.36, "end": 2444.36, "text": " And they concatenate the textual values to image values. And then they just do the attention. So it's kind of bothersome."}, {"start": 2444.36, "end": 2450.36, "text": " This is obviously not the only way you can do it, but it's basically exactly what the paper here said."}, {"start": 2450.36, "end": 2461.36, "text": " So the output of this transformer is using two ways. Blah, blah, blah. And then second, the last layer of token embeddings, a sequence of key feature vectors is separately projected to the dimensionality of each attention layer."}, {"start": 2461.36, "end": 2470.36, "text": " Throughout the ADI model and then concatenated to the attention context at each layer. So we just saw that that's indeed the case."}, {"start": 2470.36, "end": 2480.36, "text": " And let me just continue here. We don't need to go through the this is the implementation of a transformer. I'm going to skip all of this, assuming you already know how this thing works."}, {"start": 2480.36, "end": 2494.36, "text": " And that's it. So basically we saw how we are taking the textual information stored in this dictionary here, temporal information and passing all of that through UNET and forming the model output model."}, {"start": 2494.36, "end": 2510.36, "text": " This is again going to contain the Epsilon. So the noise plus the covariance matrix is going to be one six, sixty four, sixty four, six, because we have three channels predicting the Epsilon and three channels predicting Sigma or the covariance matrix."}, {"start": 2510.36, "end": 2531.36, "text": " OK, so next up. So everything else remains the same as in the first DDPM, the coding video I've done. The only difference here was the textual basically conditioning. OK, so we split the output into two parts, the Epsilon and the model var values."}, {"start": 2531.36, "end": 2544.36, "text": " Again, there is this V vector that's actually predicted and then we form the model variance by by using these expressions here. Not as important. We saw that multiple times in previous videos."}, {"start": 2544.36, "end": 2556.36, "text": " And finally, we predict. So we now pass the noisy image, the time steps and the output. And then we just predict the X zero here."}, {"start": 2556.36, "end": 2567.36, "text": " Just some clamping. And finally, we predict the model mean. So everything here remains the same as in the original, as in the improved DDPM paper."}, {"start": 2567.36, "end": 2580.36, "text": " So I'm going to skip all of this. And now this is where the difference will will will kick in. So here we just a sample from a normal distribution."}, {"start": 2580.36, "end": 2587.36, "text": " We create this non zero mask. And this is the interesting part. So we now want to see how the conditioning is done using the clip model."}, {"start": 2587.36, "end": 2596.36, "text": " Again, this is very similar in structure to the guided DDPM that I've covered in my previous video. So let's step inside here."}, {"start": 2596.36, "end": 2610.36, "text": " I'm going to hit F 10. We are going to form the gradient and then we're just going to shift the predicted mean by the as you can see here. So Sigma, the covariance times the gradient."}, {"start": 2610.36, "end": 2617.36, "text": " OK, and the grading itself already contains the scaling coefficient. So that's why we don't see it here."}, {"start": 2617.36, "end": 2626.36, "text": " OK, let's hit F 10. And here are we in the conditioning function. So we previously saw how we created this textual embedding."}, {"start": 2626.36, "end": 2641.36, "text": " Now we're going to do the fun part. So we're going to take the input noisy image, detach it from the computational graph and set requires graded true because we want to compute the gradients for this particular image."}, {"start": 2641.36, "end": 2649.36, "text": " We're going to embed this image using the image pathway of the clip. So such that we end up with embedding image embedding here."}, {"start": 2649.36, "end": 2658.36, "text": " So that's again, let me show you what we're doing there. We're doing the drawing I showed you a couple of minutes ago. So I think it's somewhere here."}, {"start": 2658.36, "end": 2670.36, "text": " So we are just now forming this red embedding vector. And now we're going to basically, well, find the gradients of the product between the text embedding vector and the image embedding vector."}, {"start": 2670.36, "end": 2685.36, "text": " OK, let's go back here. Let's step over image embeddings. So nothing. Well, it's again just walking me through the I have a bunch of break points here, but we can kind of ignore all of those safely."}, {"start": 2685.36, "end": 2696.36, "text": " And what's this? My God, my God, I'm hitting every single I'm hitting every single break point here. Nothing interesting here. Nothing interesting here."}, {"start": 2696.36, "end": 2703.36, "text": " So we form the embedding vector. We normalize it. And here we are. So we have the embedding vector for the image."}, {"start": 2703.36, "end": 2711.36, "text": " And now we form the product and we additionally have this logic scale. So we form the loss and we just then apply."}, {"start": 2711.36, "end": 2719.36, "text": " We just find the gradient of that loss and that gradient is now going to be returned. We just multiply with the gradient scale."}, {"start": 2719.36, "end": 2726.36, "text": " So it's going to be three because that's the parameter we set in the main script. And we just return that gradient. But everything."}, {"start": 2726.36, "end": 2739.36, "text": " So basically this is what we saw in the paper. And this is the only difference compared to your regular classifier guidance we saw in the previous video on diffusion models beat GANs on image synthesis paper."}, {"start": 2739.36, "end": 2754.36, "text": " So this gradient is going to move our image towards the part of the space where that image resembles the prompt much more closely compared to if we were to just do it unconditionally."}, {"start": 2754.36, "end": 2760.36, "text": " OK, let's step over here. We have the gradients. We find the new mean. And that's it, guys."}, {"start": 2760.36, "end": 2771.36, "text": " Then we just sample from that new mean. Again, we have this this standard expression. We just have mean plus sigma and we just multiply by the by the noise here."}, {"start": 2771.36, "end": 2778.36, "text": " This is how we sample from the Gaussian. And that's it. And this now just keeps on repeating."}, {"start": 2778.36, "end": 2787.36, "text": " We're now going to repeat the same process for the next time step, which is going to be 98 and all the way to zero. And that's pretty much it."}, {"start": 2787.36, "end": 2799.36, "text": " OK, guys, next up, I want to show you how the classifier free guidance works. So let me run the script text to image and let's start."}, {"start": 2799.36, "end": 2806.36, "text": " A lot of the structure is shared between this script and the previous one. So I'll just be skipping even more than before."}, {"start": 2806.36, "end": 2814.36, "text": " So we're creating the model and the diffusion here. Let me disable the break points such that we are just stepping over all of this."}, {"start": 2814.36, "end": 2823.36, "text": " So we have modeling, diffusion, blah, blah, blah, shifting into GPU, loading the checkpoint, counting the number of parameters."}, {"start": 2823.36, "end": 2833.36, "text": " Then we are forming the up sampling model here, which we don't care about. So we're going to skip all of that until we hit the prompt here and all painting of accordi."}, {"start": 2833.36, "end": 2844.36, "text": " And now let's start. So same hyper parameters as before, same encoding of the tokens and then adding the padding and the mask."}, {"start": 2844.36, "end": 2852.36, "text": " And this is where it's now starting to become a bit different. This is where the structure starts to deviate from the previous script."}, {"start": 2852.36, "end": 2860.36, "text": " So now we encode the null prompt. So that's the empty prompt here, as you can see. And that's the part."}, {"start": 2860.36, "end": 2868.36, "text": " I'll show you again in the paper. So let's find the OK, here it is. Here's the equation."}, {"start": 2868.36, "end": 2874.36, "text": " So we have to pass the empty prompt and then we have to pass captions."}, {"start": 2874.36, "end": 2878.36, "text": " So that means we'll effectively have two passes through the diffusion model."}, {"start": 2878.36, "end": 2884.36, "text": " Or in this particular implementation, what I've done is they've sacrificed some memory for a single pass for time."}, {"start": 2884.36, "end": 2891.36, "text": " And so they basically do a single pass, as we'll soon see, and they compute both of these expressions here."}, {"start": 2891.36, "end": 2899.36, "text": " So both conditional caption as well as as on the empty well, null caption or empty caption."}, {"start": 2899.36, "end": 2909.36, "text": " OK, so let's go back to the code. Here we can see that we now tokenize the empty empty sequence and we end up with tokens."}, {"start": 2909.36, "end": 2918.36, "text": " We just contain pretty much just the I think end of tokens. So all of these, as you can see here, just contain the padding token."}, {"start": 2918.36, "end": 2922.36, "text": " I think this is the padding token. Let me double check. That's indeed the case."}, {"start": 2922.36, "end": 2929.36, "text": " So if I were to enter here, so this padded tokens, let me just find that thing."}, {"start": 2929.36, "end": 2938.36, "text": " I think it's going to be somewhere in here. So there is a tokenizer part and we care about the BPE file."}, {"start": 2938.36, "end": 2952.36, "text": " And here it is. So you can see here. Because we have passed empty tokens, we're basically going to pad the end token for the full context."}, {"start": 2952.36, "end": 2956.36, "text": " Basically, that's what happens. We have end token as the result here."}, {"start": 2956.36, "end": 2961.36, "text": " That's that's what these this fifty thousand two fifty six represents. It's just a pet token."}, {"start": 2961.36, "end": 2970.36, "text": " OK. Now we're going to store both the tokens as well as the this unconditional tokens."}, {"start": 2970.36, "end": 2974.36, "text": " And we're also going to store mask as well as the unconditional mask."}, {"start": 2974.36, "end": 2978.36, "text": " So this mask is kind of boring. I assume it's going to have all forces."}, {"start": 2978.36, "end": 2983.36, "text": " Yeah, as you could have expected. And now let's continue."}, {"start": 2983.36, "end": 2992.36, "text": " So that's the key word arguments that we're going to use to condition our model during the four pass."}, {"start": 2992.36, "end": 2999.36, "text": " OK, so here we have an interesting function. So let me now enable all of the breakpoints."}, {"start": 2999.36, "end": 3007.36, "text": " We'll enter this one a bit later. This is where the whole magic happens when it comes to computing the classifier free guidance."}, {"start": 3007.36, "end": 3017.36, "text": " And again, we have the P sample loop standard stuff. We want to generate an image that's going to be I think this is going to be sixty four sixty four resolution."}, {"start": 3017.36, "end": 3022.36, "text": " Yep, that's the case. And let's continue here. We are passing the conditioning information here."}, {"start": 3022.36, "end": 3033.36, "text": " And finally, you can see here that we are extracting batch size, which is one, whereas the full batch size here is going to be two."}, {"start": 3033.36, "end": 3042.36, "text": " And the reason is this is the optimization they've done such that they have a single four pass and they end up with the with the result,"}, {"start": 3042.36, "end": 3052.36, "text": " even though they have to compute the and the output both for the empty token string as well as for the actual prompt we're using."}, {"start": 3052.36, "end": 3062.36, "text": " OK, let's now dig into the code again. I'm skipping some parts because it's just boilerplate, some for loops and stuff."}, {"start": 3062.36, "end": 3069.36, "text": " So we are generating the noisy image we're sampling from the I guess normal distribution here."}, {"start": 3069.36, "end": 3076.36, "text": " We are forming the indices again. We have how many steps? Hundred. We are logging the progress. Nothing fancy there."}, {"start": 3076.36, "end": 3084.36, "text": " And now we're starting the actual diffusion reverse process. OK, so here is the P sample function. This is the important function."}, {"start": 3084.36, "end": 3090.36, "text": " Really, nothing has changed. So the model is going to be different here. That's the one that was defined in the main script."}, {"start": 3090.36, "end": 3097.36, "text": " We're going to hit it a bit later. So you're going to see it otherwise. Nothing interesting here."}, {"start": 3097.36, "end": 3103.36, "text": " No conditioning here because we are doing the classifier free guidance. And this is the conditional information."}, {"start": 3103.36, "end": 3107.36, "text": " OK, we're going to see all of that in a second. Let me step over all of this."}, {"start": 3107.36, "end": 3112.36, "text": " And this is where the magic of the classifier free guidance is going to kick in."}, {"start": 3112.36, "end": 3119.36, "text": " So here we hit the model function that we define in the main script here. So let's see what we're doing."}, {"start": 3119.36, "end": 3127.36, "text": " So we take the input images and we just grab the first part. So again, remember that we have two images here."}, {"start": 3127.36, "end": 3133.36, "text": " So let me show you the shape. So we have two. So we're going to take the first part only."}, {"start": 3133.36, "end": 3138.36, "text": " Because only the first part is going to contain the actual information of interest."}, {"start": 3138.36, "end": 3148.36, "text": " And that's again the consequence of the way of them wanted to have a single forward pass, but like sacrifice the memory."}, {"start": 3148.36, "end": 3160.36, "text": " And that's why there is this weird burden that we as someone analyzing this code base have to kind of suffer because of that optimization trick."}, {"start": 3160.36, "end": 3167.36, "text": " But like it's fairly simple. Basically, you just take the first part of the of the image."}, {"start": 3167.36, "end": 3173.36, "text": " We we know just as you can see here, we form this combined variable, which is just basically copy paste."}, {"start": 3173.36, "end": 3182.36, "text": " We copy paste that first portion. And now we pass that basically together with time steps and together with conditioning information."}, {"start": 3182.36, "end": 3198.36, "text": " We pass all of that through the unit with the transformer. So I'm going to disable the break points and just skip over this whole step over this this this function."}, {"start": 3198.36, "end": 3210.36, "text": " Because we already saw this in the first script. We are basically just integrating the temporal information, integrating the textual information into the attention context, etc, etc."}, {"start": 3210.36, "end": 3217.36, "text": " And as the output, we get the epsilons and the sigmas for both images. So let's see the model output."}, {"start": 3217.36, "end": 3229.36, "text": " Basically, it should be 264, 64, because we have two images, six because we have both the epsilon and as well as the covariance matrix, so the sigma."}, {"start": 3229.36, "end": 3236.36, "text": " And now we're going to split those, as you can see here, into epsilons and and and well, they call it rest."}, {"start": 3236.36, "end": 3244.36, "text": " And then finally, we do another splitting into conditional epsilon and unconditional epsilon, because as you recall,"}, {"start": 3244.36, "end": 3252.36, "text": " the first image is going to be conditioned here with the conditional information."}, {"start": 3252.36, "end": 3256.36, "text": " The second one is just going to have the tokens for the for the empty prompt."}, {"start": 3256.36, "end": 3264.36, "text": " That's why we have these two variables here. So these two, again, are just are just these two variables here."}, {"start": 3264.36, "end": 3275.36, "text": " And the epsilon conditioned on the empty prompt as well as the epsilon conditioned on the on the caption on the prompt."}, {"start": 3275.36, "end": 3281.36, "text": " OK, let's go back to the code. And now here's the step. Here's the actual formula."}, {"start": 3281.36, "end": 3287.36, "text": " We have the unconditional epsilon. We add up the scaled version of this difference. And that's it."}, {"start": 3287.36, "end": 3295.36, "text": " That's the whole magic. And now we just again do this. Well, as I said, it's a consequence of justice of the optimization they've implemented."}, {"start": 3295.36, "end": 3300.36, "text": " And then we return all of that. And that's it. Everything else remains pretty much the same."}, {"start": 3300.36, "end": 3308.36, "text": " We split into we form the model variance. We form the let's see where there is something different here."}, {"start": 3308.36, "end": 3317.36, "text": " We just predict the X start or the X zero, which is the I guess the newest image. And then we do some pre-processing."}, {"start": 3317.36, "end": 3325.36, "text": " So nothing fancy there again will only care. So this model output contains copy pasted same results."}, {"start": 3325.36, "end": 3333.36, "text": " So we only care. And let me kind of show you that's indeed the case. So the shape of this thing is two, three, sixty four, sixty four."}, {"start": 3333.36, "end": 3339.36, "text": " If I were to take all zeros, so that means I'm going to just grab a single element from the first batch."}, {"start": 3339.36, "end": 3350.36, "text": " If I were to take the second, the same element from the second batch, you can see they are the same because that's how, again, the optimization they're doing is playing out."}, {"start": 3350.36, "end": 3357.36, "text": " OK, so all in all, at all times, only the zeros image contains the results of interest."}, {"start": 3357.36, "end": 3367.36, "text": " The second one is just basically a consequence of the optimization. Now they form the mean and then they return all of these variables back."}, {"start": 3367.36, "end": 3380.36, "text": " And that's it. Everything else remains the same. We just we don't have the conditioning now and we just form the final sample by sampling from this distribution sample."}, {"start": 3380.36, "end": 3394.36, "text": " Here is still two, two, three, sixty four, sixty four. But only the first part contains the sort of zeros index of the batch dimension contains the actual information that we care about."}, {"start": 3394.36, "end": 3406.36, "text": " And that's it, guys. I'm going to now go back into the script. I'm going to disable all of the breakpoints and I'm going to set a breakpoint to show images here."}, {"start": 3406.36, "end": 3419.36, "text": " Now let's hit F5 and let's see how this is going to be generated. You can see here the progress is being tracked using the TQDM library, which is so cool."}, {"start": 3419.36, "end": 3426.36, "text": " And now let me show you the image. So if I click F10 here, let me show you the image. So we have a corgi, as you can see here."}, {"start": 3426.36, "end": 3442.36, "text": " So what was the prompt? Our prompt was, let me just find it, an oil painting of a corgi. And you can see indeed this is a low resolution version of that particular prompt because the models are already pre-trained for us."}, {"start": 3442.36, "end": 3457.36, "text": " That's why we are getting these cool results. Keep in mind that OpenAI has not open source the training code, so we can only play with the sampling and generation in general and not with training."}, {"start": 3457.36, "end": 3467.36, "text": " Let me quickly show you what's going to happen after we do the up sampling. I'm going to set another breakpoint here. Let me hit F5 here."}, {"start": 3467.36, "end": 3480.36, "text": " And now we're generating the up sampled version. So if I hit F10 here, let me just see whether that's going to exit the program or not."}, {"start": 3480.36, "end": 3489.36, "text": " Okay, unfortunately that exited the whole program, so I'm going to have to set something here like print blah blah, for example."}, {"start": 3489.36, "end": 3501.36, "text": " And now let's hit run again. And we're going to sample both the corgi as well as the up sampled version of the corgi, although it may take some time."}, {"start": 3501.36, "end": 3511.36, "text": " Cool, guys. That's pretty much it. I know there is a lot of information in this video. Do let me know if you have any feedback. Did you find this one useful or not?"}, {"start": 3511.36, "end": 3522.36, "text": " All of that's going to help me modify my future videos. And hopefully you learned something from this. Hopefully you like this combination of using paper, using code."}, {"start": 3522.36, "end": 3530.36, "text": " There is a lot of information, but I think it might be useful for all of you. So here's the corgi again. This is the low res version."}, {"start": 3530.36, "end": 3536.36, "text": " And now we're going to see the up sampled version. So here is the up sampled version. So that's it."}, {"start": 3536.36, "end": 3548.36, "text": " I'm going to quickly also show you how the inpainting script looks like. Let me move it here. Let me try and run it. Let me see where the show images are."}, {"start": 3548.36, "end": 3560.36, "text": " So I'm going to set a break point there and then there is the up sampling part. But yeah, I think we can just set a couple of break points here to visualize what's going on."}, {"start": 3560.36, "end": 3574.36, "text": " So let me hit run and let's see what's going on. So first things first after the loading has happened. So again, the logic is actually not that complicated."}, {"start": 3574.36, "end": 3583.36, "text": " It's a bit harder to form a mental model for how this inpainting works, but I encourage you to check out the code at your own pace and understand what's going on."}, {"start": 3583.36, "end": 3593.36, "text": " So here is the image we are trying to impaint. So we basically have this grass image and then we masked all of the rows here."}, {"start": 3593.36, "end": 3602.36, "text": " So I'm going to exit here. Let's hit F5. Now we're going to generate impaint this particular image."}, {"start": 3602.36, "end": 3611.36, "text": " And let me now hit this. We're going to hit this break point here after 100 steps. So let me show you what we get as a result."}, {"start": 3611.36, "end": 3624.36, "text": " We get a corgi on this on this on this on the grass. And that's because the prompt we are using is let me just find a prompt a corgi in a field, which makes sense."}, {"start": 3624.36, "end": 3636.36, "text": " And finally, if I were to hit well F5 here, well, we'll just get the super like the up sampled version. But that's pretty much it."}, {"start": 3636.36, "end": 3643.36, "text": " Again, guys, do let me know whether you find this video interesting and useful. Any feedback is super appreciated."}, {"start": 3643.36, "end": 3666.36, "text": " And do subscribe to this channel if you liked this video. Share it out with your friends. And until next time, bye bye."}]