OpenAI GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I cover a new paper from OpenAI - "GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models" where they combine diffusion models with transformers to outperform their older DALL-E model.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ GLIDE paper: https://arxiv.org/abs/2112.10741
✅ GLIDE code: https://github.com/openai/glide-text2im

Learning about diffusion models
Papers:
✅ Seminal (2015): https://arxiv.org/pdf/1503.03585.pdf
✅ DDPM (2020): https://arxiv.org/pdf/2006.11239.pdf
✅ OpenAI (1): https://arxiv.org/pdf/2102.09672.pdf
✅ OpenAI (2): https://arxiv.org/pdf/2105.05233.pdf

Blogs:
✅ Score-based models: https://yang-song.github.io/blog/2021/score/
✅ Diffusion models: https://lilianweng.github.io/lil-log/2021/07/11/diffusion-models.html
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro to GLIDE - results 
04:00 Intro to diffusion models
07:10 Inpainting and other awesome results
11:05 Diffusion models in depth
20:45 VAE inspired loss
31:30 GLIDE pipeline (diffusion + transformers)
34:15 Guided diffusion
38:00 Classifier-free guidance
42:25 CLIP guidance
45:25 Comparison with other models
48:30 Safety considerations
49:25 Failure cases
51:40 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Kulsoom Abdullah
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#glide #openai #diffusionmodels
 What's up guys in this video I'm covering Glide towards photorealistic image generation and editing with text guided diffusion models by the awesome OpenAI team right here and this is yet another iteration of OpenAI's efforts to improve upon these basically text conditioned image generation image synthesis models and you're probably familiar with one of their previous models which is namely Dali which had amazing results but as we're going to soon see Glide has even better results and they kind of verified that using human judges so we're gonna see the results a bit later but first with those of you who are not even familiar with Dali let me let me show you what the hype is all about this is what like Glide produces so Glide produces these images conditioned on these prompts here so we have a prompt here a hedgehog using a calculator and we can see an image that like Glide generated which is fairly reasonable I mean you as a human probably wouldn't be able to do a much better job although there's some there is some blur here which may be to the fact that they're using diffusion models but we're gonna see and discuss those a bit later then we have a like another example a corgi wearing a red bowtie and you can see here we have a this attribute binding we basically have the model has to understand that red corresponds to bowtie and then we have purple the color purple corresponds to this party hat so the model first has to kind of hallucinate and create those and then the plate to place them on the right spot in the image and make it look like a realistic and I think they do an awesome job and by the way I don't know what like what's up with corgis I can see corgis everywhere I guess some of the open AI members really likes corgis or it's there kind of in-house pet or something in other cool picture here robots meditating in a vipa sauna retreat fairly sure you cannot find this image anywhere but it'd be nice to see the the nearest neighbor from the training data set from all of these generated images although yeah this looks very very awesome if you ask me I'm gonna just cherry pick a couple more examples here and all of these are also cherry picked already they're using something called guidance like classifier free if I'm not wrong let me just check it out here classifier free guidance yeah that's right so here we can see a high quality oil painting of a psychedelic hamster dragon and this is totally wicked like psychedelic okay we can see kind of the colors are there then we have hamster hamster is also there dragon we can see these these wings in the background which kind of reminds us of the dragon high quality well I'm not sure about that part but like all it definitely has this oil painting vibe to it so again amazing amazing rendering of this of this prompt here many other cool results we have in other corgi here and finally we have this thing here so a red cube on top of a blue cube and I'd love to see some more complex query some more complex prompt where the model we have to do more serious relational reasoning because as far as I remember in the Dali paper they tried and and and constructed those and the model the Dali failed like didn't manage to either count well or fail to do the attribute binding correctly like associate the correct color with the shapes etc so it'd be cool to kind of further probe this model but that's gonna be hard because they only released a smaller model which is trained on a filter data set so it's not not nearly as potent as the one they have in house behind I guess yeah not sure whether they have an API for the for this glide model in any case let's try and understand the magic behind behind this model so first things first let me just kind of mention here when evaluated by human judges our samples are preferred to those from Dali 87% of the time when evaluated for photorealism and 69% of the time when evaluated for caption similarity so that's cool so that means that this model definitely improved upon Dali and the main like component that that brought that improvement I'd say is these diffusion models and you're probably familiar with some other generative models such as GANs generative other serial networks or VAE's variational autoencoders or even flow based models but if you're models are probably new for for most of you because only in 2020 did people like manage to get them to work to be in the sense they are they are now kind of you can you can sample very fast like relatively speaking and you can train them and they just work there is a nice relationship between diffusion models and VAE's we're gonna see that the actual loss for the diffusion models is inspired by these VAE models and also there is another class of generative models called score based models and I'm gonna link a lot of resources down in the video description so do check it out so there is a nice connection with that family as well I don't have it here but yeah you can check it out in the video description and another thing I want to point out here is that let's let's see the like a difference at least on this block diagram the main difference between like all of these models and and diffusion models and you can see let me try and use my fancy new touchpad basically you can see that the latents here the latents here are completely like they are of the same dimensionality as the as the input image whereas usually when you have something like VAE so it's an autoencoder that means you're basically down you're down sampling you're you're reducing the dimensionality of your input data into some latent vector here usually denoted as Z and this one is way like lower dimensional compared to the input input data you're trying to model here on the other side you're you're basically you're basically keeping the dimensionality the same as the as your input and you can imagine that that can cause some problems training this model and sampling from this generative model etc but we're gonna see how people manage to to work their way around it kinda I say kinda because before we even dig into details let me just show you I mean there are obviously some limitations and they pointed those out somewhere here let me try and find limitations okay our unoptimized model takes 15 seconds to sample one image on a single a hundred GPU and we're going to understand why this is basically the reason is that we have to do like 20 forward passes in order to render an image whereas using something like GAN or VAE you do a single forward pass and you've got yourself an image so that's that's a limitation to keep in mind and yeah so let's get back up here okay before I even start digging deeper into how diffusion models work which is going to be very ungrateful job for me to do because it'll probably take a whole video to do it so I'm gonna do a high level overview first let's see just the what the model can do on a high level what the model is capable of so here we can see some in painting capabilities and thing to keep in mind here is that they had to do some fine-tuning so this is not zero shot as as your Dali model or some of the other like images rendered in this paper this like model was fine-tuned for this precise task of in painting and you can see the awesome results so they they have this mask and the model inputs to the kind of I think that they feed that the three channeled RGB image they also feed this this mask as a separate channel and I think they have a couple more channels in that in painting model but that's not that important right now what's important is you can see that results are fairly stunning again here a girl hugging a corgi on a pedestal you can see it rendered corgi pretty awesome like just look at this this like hand wrapping around the body of this corgi I mean the results are if you if you told me like a couple of years ago that AI models will be able to do this I'd say it's it's like almost impossible and now I think people are just kind of used to AI being able to do this and we're setting even higher and higher goals and so now this is not like nothing special but yeah we need to appreciate the progress that happened since 2012 and of course earlier than that but yeah with the Alex net the things really started skyrocketing same thing here you can see the red hair you can see how the the ways of flowers appears I especially find this one fascinating the hat lies perfectly like the perspective the the color maybe the color is a bit brighter you can see the hopefully can see it on your screen the color is a bit brighter compared to this shirt here so yeah but still fairly stunning results and yeah small knit here glad stands for a guided language to image diffusion for generation and editing you can understand why that is we can both generate we can add it as you can see here we have language that's conditioned we are basically conditioning our our diffusion model on some language prompt and we basically generate images using the diffusion models so that's kind of trying to deconstruct that that complex sentence here are some work like awesome capabilities of Clyde model you can see here that first they kind of generate using the cozy living room they zero shot generate this image of the cozy living room and then they put a mouse like a mask here and then they say a painting of a corgi on the wall above a couch and here comes corgi again and then they put a mask again and then they say around coffee table in front of a couch and here comes the table and it's round and that's I mean fairly awesome and then again they put this mask and they see a vase of flowers on the coffee table and again we see the results rendered and finally here they say they put a mask here and they say a couch in the corner of her room and now I think this is a failure case or I may be missing something out but basically what happened is that we just got ourself a window here but I don't see like a couch in the corner of her room or maybe or maybe it kind of cropped this part so now this is this is like officially in the corner of a room although we don't see the wall here so but but anyways that's maybe me being too nitpicky here but yeah the results are very cool here this just shows some combination using this as the edit and here they kind of like instead of putting this generic mask with a generic shape they kind of help the model a bit more by using both this is kind of semantic segmentation in a way and they put a approximate shape and so it's easier to generate the picture here you can see I can see the results yourself here okay so that's what glide can do now let me try and demystify how this thing actually works and it's gonna be very very hard to try and explain how these diffusion models work very quickly but yeah let me try it if you think the explanation was not good enough I can maybe try just kind of upload my comment and I can I can create another video explaining just the diffusion models okay so the the process itself these diffusion models were inspired by some non-equilibrium statistical physics I know it sounds very very complicated because it is but like back in 2015 the first paper appeared and they managed to to basically originated the idea of the diffusion models but only in 2020 do we get the models to to work nicely to generate nice samples and to be like faster and yeah so here is the the main idea you have this diffusion process whereas you take your input image and what you do is you basically add add a certain amount of noise from a specific family like usually people use Gaussian because cautions are from this thing called exponential family which means they have all of these nice properties when you add up cautions when you do product with them you end up with a Gaussian like the structure of the Gaussian remains and it's easy to compute and that's why people are using Gossians all around like in VAE's as well for the for the priors etc etc so basically they have nice computational properties so we add up some noise and then we do we continue doing that until we get to like a picture which is basically pure Gaussian noise now wouldn't it be nice if we could reverse the process and starting from a Gaussian noise so just sampling this Gaussian noise picture reverse the process and generate our data so that would be cool so again we have this conditional distribution so X of t conditioned on the X of t minus 1 which is timestamp the previous timestamp and we basically know that because that's described by this formula here and I'm gonna quickly explain how this thing works in a second and then what's problematic is that we cannot it's not easy to figure out the reverse distribution here so so basically not a reverse distribution by this conditional distribution where you're trying to figure out the t minus 1 condition on the X of t so you're trying to denoise the image here so what diffusion models are trying to do is construct this p of theta which is going to be a neural network in practice to try and model this this unknown distribution here and we're gonna see a couple of nice properties of this distribution which will allow us to actually do that and okay before we get there let me try and dissect this this formula here so we can see that the notation is kind of cringy I love this notation better when you have like for example let me just take the touchpad here so I like it better when you have something like X of t and then you're sampling from the normal distribution or whatever distribution like so like having this squiggly thing but yeah just be aware how this notation works this just means we're sampling X of t from like the Gaussian distribution denoted by n and this is the mu so this thing here this part here is the mu so that's the mean and this thing here is just the covariance matrix okay so let's try and dissect this so we have X of t minus 1 multiplied by this thing here and beta t is something called I think it's called noise level something I forgot the exact name but whatever basically it's between 0 and 1 which means that this thing here under square root is gonna be less than 1 and that means that we're gonna contract this X of t minus 1 okay so let me try and visualize this thing right here okay so we have and for the sake of clarity I'm gonna assume that X of t minus 1 which is our image is actually only three dimensional data point so that I can actually visualize what's happening otherwise if this is thousand by thousand that's million dimensions which means I cannot draw a million dimensions on my one node screen here so here is the coordinate system so let's say that X of t minus 1 is somewhere here and so this is the origin point here it's kind of let me kind of try and connect it here so what this thing here does is it's contracting because it's more than one which means then the mu the mean of this distribution let me take a different color here let me try and take the green one it's gonna kind of push this point from here all the way to somewhere here okay so this is gonna be whoops it's gonna be well I'm still trying to get used to this novel touchpad okay so we have the point now here and now this beta is gonna be we're gonna have a diagonal I guess isotropic covariance matrix which means we're gonna have I'm gonna denote that as some small circle around this point so we're gonna have some Gaussian distribution centered around this point here and we're gonna sample our X of t from that distribution here and so let's say we sample it let me try and change the color here let me take the blue color so let's say we sample data point here so that's gonna be our so we started with with here so we had a X t minus one now we are here so this is visually representing in this 3d space what's happening in this image space where we're getting more noisy and noisy images so maybe from this image we got this one although this will take more steps obviously but yeah you get a point now let's see what happens in the limit the thing with these betas is that in the original paper in I think in the paper from 2020 they were not learnable they just took and created a schedule that started with some larger betas which are still smaller than one and then slowly anneal them towards zero okay linearly I think so we had a linear schedule and that means when beta t gets down to zero what happens actually it's the other way around it starts with very small because we want to have like lesser noise so it starts with zero and then it's a meal towards one okay that will make more sense so what happens when it gets to one that means this is going to zero so this thing here let me change the color so I'm gonna take the red pen here and so basically this thing is gonna be a kneeled it's gonna go to zero and that means that this thing here is gonna be contracted all the way down here until we get to the origin point so that means that after a lot of steps our data points are gonna be basically drawn from from this from this like distribution which is basically your your standard Gaussian distribution the normal distribution because we'll have a mu of zero because this is zero so that means this thing is gonna be zero and this thing is gonna be one which means we have identity matrix and we are basically sampling from a noise pure noise distribution and that's what I explained here so that that's basically this image here so that's at the limit of this diffusion process now you might think that this would be very slow if we had to sample like if we had to sample and get X 100 we'd have to repeat this sampling procedure hundred times but that's not the case luckily for us because you can you can form you can because you can see here Q X of T conditions of X of zero say X of zero is the original image from our data set the actual images we have in our data set we can form like a distribution so that we can directly sample and get and kind of skip these intermediate steps and like get the X 100 or X 200 from a single from a single like a step and you can see it here again these alpha T with this hat thing is a product of alpha of these coefficients here and they are 1 minus beta and you can imagine because all of these are smaller than 1 when you multiple a bunch of numbers that are smaller than 1 this thing is going to converge in the limit it's going to go down to zero to zero okay it's going to down to zero so that means what we end up with is nothing here and we end up with one here which means this thing is going to be equal to this small epsilon here which is as you can see here the normal distribution so that's what happens again in the in the limit of this diffusion process now this is very important they state here and this is this is actually a conclusion from 1949 I think they say note that Q so this is conditional distribution we are we're trying to model approaches a diagonal Gaussian distribution as capital T approaches infinity and this is the number of steps in our diffusion process in correspondingly this this noise level in the limit goes down to zero so it's sufficient to train a neural network to predict a mean mu theta and a diagonal covariance matrix Sigma theta and so this will be the this is this thing here is the distribution we're trying to learn in order to model the actual the true distribution here so this one the unknown one okay and the reason why it's possible is because we know that under these conditions the actual true distribution is going to have that same shape and then we can just kind of learn and kind of fit this caution on top on top of that distribution now if you're familiar with VA is this may be this whole like a diagram here may be familiar and if I were to swap if I were to swap this and kind of change it to and denote it as Z which is usual notation VA is because X one through T are the latents remember I mentioned that the latents here in this model are of the same dimension so these are the latents these noisy images here are the latents of our of our image here and basically if I were to denote these as Z so this one here as Z you may notice this formula from the VA is construction and here we're just trying to like construct something that's called evidence lower bound and it's fairly self-explanatory although it may be super confusing when you hear it the first time so why is that why is it called elbow or evidence lower bound so first we have this thing here is called evidence so this is evidence or I think you kept include the log one so this basically you want to maximize this you want to have your data points from your data set have high probability under your fitted distributions under your learned distribution P of theta so that's what machine learning is models are oftentimes trying to do we're trying to maximize the probability of the data points under our our model okay now because log is monotonic it actually does not matter whether you're optimizing you're not going to change the loss function by adding a log you're gonna change some computational properties and the convergence speed etc so it's kind of usually needs to have log inside but so if we have this why don't we try and maximize it directly well the problem is it's intractable so you can try and break this thing down and you end up with something with a complex integral that looks something like this we have these P theta effects zero and we'll have the prior so the probability of the actual latent and we'll have to basically integrate all over all of those latents as you can see here we have to sample the latents to Z the Zets we have to have their probabilities we have to to construct this to basically compute this integral which is going to be it's simply intractable so we cannot directly compute this thing here so this is equal to the probability theta X zero you'd have to do this thing here in order to get this this probability here so that's that's out of the question we can do that so that's why people are constructing these lower bounds and basically this is trivial by definition because KL divergence is always greater or equal than zero it's equal only in the case when the two distributions perfectly match which we're trying to do by training these beta t to match the the P distribution to to match the real Q distribution so after doing some algebra here I'm not gonna dig into a lot of details we end up with this formula here and this is what we call the the elbow the like like variance lower bound like loss and you can see here that basically if we tried and minimized this thing we know that it's gonna be that the actual thing we care about is gonna be even lower so what's called lower bound if it's actually higher because usually you don't have the minus and then you're trying to maximize this thing and this thing is gonna be smaller or equal and that means as you're maximizing it you know that the actual thing you care about which is the the the evidence here so this thing here is gonna be at least as big as your loss so this is also called surrogate loss because you're trying to find different loss it's kind of going to bound your actual thing you care about okay so long story short what happens is they kind of further have to break down this this equation so that it's actually tractable and computable so they just end up in a sum of various scale divergences and at the end they end up with a simple MSC loss between the the the mu theta so that's the the thing we're trying to learn here the mu theta of our P theta distribution and they just need to do MSC loss but what happens is they actually instead of figuring out the mu they actually just figure out the noise level that happens in that happened in a specific step and let me try and break that down in a second oops by the way I realized I forgot to put the condition this on that so you have to condition so it's P of X zero given that and we see we look at the probability of that so what's the probability of that and then what's the probability of this X zero being so what's the probability of X zero given that that so we have to integrate all of that to get the actual probability of X zero so it's a small mistake hopefully you can see what's happening there anyways on the high level just it's hard it's intractable that's everything I wanted to keep in mind that's it you don't have to know it is the probability details here okay so let me try and break this down why why they're why they're what they're doing here so instead of figuring out mu so that means instead of let's let's say let me go back to this picture here so let's say this is X of this is X of t and this was X of t minus 1 this point here was X of t minus 1 this one is X of t so instead of trying to predict given X of t trying to predict this this mu here so that that's the point from where we generated in the first place this point X of t instead of doing that they figure out the noise which is the same thing in the end we're gonna see that but it's up it's a bit different so let's say we had an image here so let's say we had an image here and it had some amount of noise like it had a couple of noise points here I'm gonna reduce the amount of noise obviously to make it easier to explain so we want to basically from there on we want to understand which portions of the noise were added going from step t minus 1 to step t we want to understand which of these noise points were added so maybe we figure out okay these were the noise points that was the noise that was added and basically we end up with something like this we end up with a image here that has these two but we we cannot get got rid of this of this noise here and you can imagine by doing this we are reducing the noise and at the end we're gonna end up with the final picture that's noise free image from our data distribution and that's awesome so that's what's happening they're trying to reduce and understand what was the noise that was added between the two steps that's that's everything I want you to understand here and that's what they do exactly here so you can see here we have this L simple so the simple loss because they had some simplifications here but this turned out to work quite nice empirically so let's see how this L simple works so we have expectation over the time step goes from 1 to capital T which is the size of our diffusion chain let's call it that way we are sampling X 0 from our from the true distribution so this is a fancy way of saying we take a data sample we take an image from our data set and we sample epsilon from the like a normal like Gaussian distribution right here okay so we're gonna what happens in practice how this model is going to be trained is the following let me let me change the color here we're gonna sample X 0 which means we take so we sample let me do it like this oops so we sample X 0 here so that means we take an image then we sample T so that mean maybe hundred in one single like training pass so we have we have hundred we have a specific image and because we have this specific we have this this cool thing that we got here that means we can kind of jump over we don't have to do hundred sampling steps we can just directly calculate the X of t because of this formula here so that means now we have X of t okay so now we have X of t which means we can feed it here we can feed X of t into our epsilon theta model we can also feed the timestamp and how do you feed a scalar you may ask into neural network well please remember transformers you're basically what happens they're using the same thing as with transformers using these basically Fourier basically these sinusoids and the index into a table of precomputed sinusoids and they just take a single row of there and they kind of append it and add it to the model that's that's how you convert scalar into into into vector check out my video on the transformer paper if you didn't understand that and now you're just trying to regress this this epsilon that's it so that's how the whole procedure works and once you train the model once you have this epsilon theta model trained how you actually generate the mu theta is using this formula which kind of is simple after a couple of simple algebraic manipulations you end up with this formula and now you can just input X of t input the predicted epsilon theta and you end up with the mu theta which is the mu theta for the X of t minus 1 so that's the the the mean from where the less noisy version of the image came from so remember going to our visualization here that means we found a mu up the diffusion chain where the image was less noisy okay that was my best attempt at trying to make these diffusion models a bit more clear it took me a couple of days to wrap my head around this I had to read four papers I had to read a couple of blocks and I'm still until you start and until you try and implement this thing yourself you're gonna miss like out on the details but the high level pictures here you have this process of adding noise you have this ability to reverse the process because of this nice property that the the actual conditional distribution so the X t condition X t minus 1 sorry the the X t minus 1 condition X t is also of the Gaussian form and that makes life a lot easier and yeah basically you're trying to figure out what part of the current image is the actual noise and that's how you're denoising with every single step and finally you end up with the image from your probability distribution which is hopefully as close as possible to the actual data distribution if you train your model correctly and that was the the hardest part to understand about this paper now after you know that now let's let's see a couple more details there's some guidance they're using in order to to to be able to not just generate unconditional images but take the piece of text like I as I showed you here you take a piece of text you condition the model additionally and then you generate an image and not just generate an image so here let me let me kind of dig in deeper here so for our main experiments we train a 3.5 billion parameter text conditional diffusion model at 64 times 64 resolution and another 1.5 billion parameter text conditional up sampling diffusion model to increase the resolution to 256 256 for clip guidance we also train noise aware blah blah blah VAT blah blah blah that's not that important what's probably worth mentioning here is that it's noise aware which means you have you cannot take an off-the-shelf VAT you have to retrain it with noisy images noisy samples to get the correct results more important than that how do you condition on text and the answer is simple transformers so to condition on the text we first encode it into a sequence of K tokens and feed these tokens into a transformer so then they say the output of the transformer is used in two ways they first the final token embedding is used in place of a class embedding in the ADM model which is the diffusion model I was just explaining second the last layer of token embeddings a sequence of K feature vectors is separately projected to the dimensionality of each attention layer throughout the ADM model basically in this is very architecture specific what they do is they take the output of the transform model so again we have a text here we have some text here like I don't know whatever some text and then you pass that through a transformer the almighty transformer here so that's T for transformer outcome the feature vectors and you basically use these feature vectors and you input them in a smart way into the diffusion model so this is the diffusion model let me kind of draw a box here so something like this diffusion model you condition D on on these feature vectors and that's the final pipeline that's how the conditional thing works so the training itself remains the same you still have you still have this diffusion thing you just now you'll at all times you'll have the captions you'll have the associated caption with each of these images you have a caption here you have a caption here which will be the same throughout this whole chain and basically you're gonna use that information additionally and train the the the yeah the the text condition version of this of this model okay so what turned out to be the case is that you can you can even further improve the results by having a dedicated classifier basically manipulate your your your your training procedure here and let me try and explain how that works so for for a moment forget that we are working with text and think of this like amnest we are trying to just let's say we have ten classes and we still have passing the text through transformer we just take up I don't know like a one hot encoding of our class and we input that instead of to condition to condition the model so what I do is they take the that they said they basically take the data set and they train a classifier so they have a classifier here like I just you know this see that's gonna be probably some convolutional neural network or something or VAT I know and you take the noisy versions of your sample and you teach this model to predict the class you teach you to predict the correct class like maybe it's gonna be C1 like dog or whatnot and now because this model knows how to given an image it can predict the correct class now you can use that model to guide the the training procedure and improve the samples we've seen this with clip guided VQ VA models and some other generative models basically you can see that all throughout Twitter yeah nowadays but yeah in a nutshell what happens how this is implemented is we are going to remember we're trying to predict the mu we're just gonna have some some scalar coefficient here which is basically an amplify the influence of the gradients with respect to the input image of the log of p phi y condition on XT so this is looks very complicated but this is basically our classifier so this is basically our classifier and we're trying to so what happens here on a semantic level is that hey we know because we have the gradient we know how to tweak X of t how to change the pixels of X of t in order to maximize the the the probability of the class so that means so that's something you can see in deep dream you can check out deep dreams we're not familiar with it but basically let's say you haven't like let me just kind of draw an image here so you have an image here and you'll know how to tweak the particular pixels so that a particular class here let's let me denote it as a like a softmax distribution or something like discrete one or whatnot so you know this is the correct class for example just change the color so you know this is a correct class and what you do is you tweak this so that you kind of boost that particular class that's the whole point that's what gradients are there for and because you have that information you can use that to tweak the mu so if we were to get back to the drawing I showed you up here let me just try and find it here that means instead of this time instead of going right so instead of going from from from here to here the classifier is smart enough to tell you hey if you want to create a class of a dog instead of going here so instead of going from here to here it's better that you move a little bit towards here and so this is the better path according to the classifier so this thing here it's gonna be a better class and yeah I mean that's that's what happened that's what the formula is telling us I didn't explain you how the actual formula was derived but yeah that's gonna be enough for now so that's one way of how they are conditioning the guiding the generation process of the for the particular class and there is another thing they do is classifier free guidance the idea here is not to have a dedicated classifier because you can imagine it's very expensive to train a separate dedicated classifier on the noisy samples this time so you cannot take an off-the-shelf like a model you have to train it on the noisy data and that's expensive you wanna you want to avoid that if possible and that's where the classifier free basically part of this text comes from so classifier free we do not want to have a classifier if we don't have to so let me let me walk you through this because it's fairly important since they got better results using this classifier free approach compared to using a clip as the guidance model so they say here instead of training a separate classifier model we choose to train an unconditional denoising diffusion model P theta of Z the notation is a bit different but bear with me yeah parameterized to a score estimator a e theta of ZL this is the thing we just saw this is the noise we're trying to to learn together with the conditional model P theta is that condition and C parameterized to the yeah basically the e theta ZL C model so that's basically our model we're just additionally conditioning that that model we're learning so we use a single neural network to parameterize both models where for the unconditioned model we can simply input zeros for the class identifier C when predicting the score IE we just put the C equals to zero for that model and we jointly train the unconditional and conditional models simply by randomly setting C to the unconditional class identifier so we have a like they basically have a same network and sometimes with some probability there they are basically zeroing the class and that's how they're training the unconditional and the conditional like a diffusion model at the same time and finally we then perform sampling using the following linear combination of the conditional and unconditional score estimates and I'm fairly sure this formula is broken that this here is actually the correct formula so I'm gonna now jump to this paper and it says that in order to get the final estimate we are going to take the the prediction that we got by using the like unconditioned version plus S minus the this difference here so what this means geometrically is the following so we have this distinct e theta conditioned on the zero that means it's unconditioned so it says hey if you want to generate a cool denoised image go so we are here go in this direction okay so now this thing here the difference tells us hey this thing the the the condition model is smarter because it learned how to generate conditioned images so it's gonna say hey instead of there go here okay and so you can imagine that the difference vector here is gonna be the green thing so we are from this we are subtracting this so that means we're gonna end up with this vector here and so what this is telling you is hey instead of going with the blue or with the red one we're gonna go first with the red one so that's the unconditioned generator and then we're gonna do so we can now amplify this difference vector by how much we want so there may be like two or something that means hey we want to do it like this so go twice there and so basically we end up going going in this direction or something like that so that's the the high-level idea of how this should be supposed to work yeah there is obviously a lot of theory for why this is yeah I think I think this was mostly empirical finding somebody correct me if I'm wrong but yeah and now I was constantly talking about like simple class conditioning because remember we were working with text here the version they're actually using is like conditioning on captions and this it means just empty empty text sequence and that's that's yeah we sometimes replace text captions with an empty sequence during training so that's how we we get into the from the class conditions world into the text condition world okay also also here I explained to you how the guided diffusion would work in the case where we are conditioning on a class but remember again we're working with text so they are going to use clip and they say here they are protruding the reverse process mean along the gradient of the text image top product with respect to the image so that's here this may be a bit overwhelming perturbing the reverse process mean is what I explained up there so we are as you can see we're protruding the mean with the gradients of whatever is this and in the case of clip it's gonna be we're bringing me along the gradient of the text image dot product with respect to the image go ahead and check out my clip video if you haven't watched that one if you're not familiar with clip but here briefly what clip does is you're basically gonna have like an image here and that's so let me just change the color here so the image and the the texts the the the captions are gonna be embedded in the joint space and so what happens is you have an image here and you have some pieces of text here and you're gonna do dot product because they're in the same space that means they're like I don't know like three-dimensional vector or three-dimensional vector it's gonna usually be higher dimensionality but for the sake of argument and you're gonna do dot product between them now if you're trying to generate a specific caption like the hamster the dragon hamster from from the example I showed you in the beginning what you want to do is again so this is an image here you'll want to tweak the pixels such that this dot product between these two such that this dot product goes up okay and that's that's these you're basically that's those are the gradients here that's what this formula is capturing so instead of in the case of class conditioning you're trying to maximize just the probability of a class here you're trying to maximize the match between the image and a particular caption for which you're trying to generate this image so remember you're given the caption and you want to generate a specific image and so you're using this clip model to learn how to change the image so that you can better describe and generate the the corresponding image for this particular caption so that's it okay there was a lot of details yeah let me know whether this this format was too too too many details or not but yeah now we're gonna just see the results and that's pretty much it thing worth noticing is that they they made sure that the training compute is roughly equal to that used to train the Lee so that they can fairly compare both models and you can read this part yourself if you're if you're curious to know more I basically this day basically describe how the how the model is fine-tuned for the image painting task in painting task and how some additional details about the guidance procedure here they compare glide using class class classifier free guidance clip guidance dali some GAN versions and the real images you can see the results and let me just kind of maybe focus on this column here and you can see that using classifier free guidance is creates better images arguably compared to the clip guidance and better than then the Lee although I took a single column buddy I think we can agree that a group of elephants walking in a muddy water this also looks better than this one that's for sure because it's kind of like a blurry and that has to do with the fact that they are using in the background a discretized version of the VAE model that's how the Lee works you can again check my video if you're not familiar with the Lee so those are some some some comparison results here what they've done is they've compared I'm gonna skip these curves but basically they compared how classifier free guidance compares to clip guidance and it turns out that clip classifier free is also better looking at these metrics except for this one where the one of the metrics was actually clip score so what happened is that basically the the adversarial examples made it look like the the clip guidance is better but it's actually not the classifier free guidance is better so they say here we hypothesize that clip guidance is finding adversarial examples for the evaluation clip model rather than actually a performing classifier free guidance when it comes to matching the prompt I guess that would be my first hypothesis as well for why this thing happens you can see here like the clip guidance model has for the higher score it has lower FID which means basically that if lower FID is better so that means that this curve the orange curve in this particular case is is the better one okay so they did some comparisons with the Lee model they they tried really hard to make glide as bad as possible and the Lee is as good as possible and still they got always better results you can see here everything above 50% means that glide is better than the Lee and that's the case as you can see across all of the columns and all of the rows so this row means that they're using clip to rerank the output images of the Lee and they use I think like 512 or something temps from the Lee and then they take the best one using clip models so the clip model is ranking the images for a particular caption and that's how they improve from no ranking to the Lee we ranked and then here what they do is they just blur the output of glide by passing it through the VQ VAE model if I'm not wrong from the Dali and that's make that makes it blur and that's why we saw up there we saw up there that this Dali image here is way more blurred compared to the output of glide so then they pass the glide image through that same model that caused this and they still compare them and it's still better compared to the Lee so that's very very encouraging result okay finally some safety considerations a section they as I said they open source the smaller version of a model that was trained in this filter data set so they say here we filtered out training images containing people to reduce the capabilities of the modeling many people centric problematic use cases okay that's one of the attempts they try to do so it's a 300 million it's a way smaller model remember that the big one is like in the billions of parameters yeah ballpark and they say here that we also probed glide the filtered version so the small filtered version for some forms of bias and found that it retains and may even amplify biases in the data set for example when asked to generate toys for girls our model produces more pink toys and stuffed animals than it does for the prompt toys for boys I mean this thing is kind of funny like I mean of course that that's gonna happen because the way our society works is that girls are playing with more pink toys when they are young I mean this kind of seems weird to me the model is just reproducing whatever is present in the data set I mean we can all agree on this so yeah let me know your thoughts on this particular statement here I'd love to hear your thoughts on this and finally some some failure cases and illustration of a cat that has eight legs yeah you can see it failed I mean maybe the model here thought that these are four legs here and then there are two tails so this is a Chernobyl cat or something so that's six and then these ears have some features that look like like maybe like look like like so that's like eight here but yeah for this one that that line of reasoning does not work yeah bunch of failure cases here it's kind of interesting like a mouse hunting a lion I'd agree you maybe maybe here this mouse is actually hunting this lion that may be the case also here maybe the the line is dead and the mouse is actually eating the lion but it just yeah went into the mouth to just check double check some of the I don't know some of the markers so that the line is healthy or something I don't know a car with triangular wheels here this is I guess the best attempt that the model got and this looks like something that may actually yeah appear in a world in real world data sets but yeah it's not perfect and I found some other failure cases down in the appendix like let me just kind of try and find it here so the fine-tune model they say golden necklace they do this this impeding you can see here they put a mask here and you can see what the model the model doesn't even produce the necklace so there are some failure cases obviously it's not perfect also I mentioned the the fact that the Lee was failing with these like relational reasoning types of tasks and here they just put a simple one so I guess they really cherry picked the best out of this model and we cannot test it ourselves so that's kind of that kind of sucks but other than that this is fairly fairly awesome work I love the fact that they are pushing diffusion models I'm not sure whether diffusion models are ever gonna be as fast as as GANs or V ease when sampling for this very simple reason of the fact that the whole diffusion model thing rests on this fact so this distribution approaches the Gaussian only when you have a bunch of steps and when you have beta t going to zero if you manage to find a solution around this it's not going to be diffusion model anymore or it's gonna be something different yeah it's not the diffusion anymore I guess in any case hopefully you found this video useful again I still don't have a camera hopefully in a couple of next weeks I'm gonna get it and let me know if you found this video yeah interesting useful and leave some comments feedback and yeah subscribe to this channel share it with friends and until next time bye bye
[{"start": 0.0, "end": 4.54, "text": " What's up guys in this video I'm covering Glide towards photorealistic"}, {"start": 4.54, "end": 10.32, "text": " image generation and editing with text guided diffusion models by the awesome"}, {"start": 10.32, "end": 17.48, "text": " OpenAI team right here and this is yet another iteration of OpenAI's efforts to"}, {"start": 17.48, "end": 23.32, "text": " improve upon these basically text conditioned image generation image"}, {"start": 23.32, "end": 27.12, "text": " synthesis models and you're probably familiar with one of their previous"}, {"start": 27.12, "end": 33.08, "text": " models which is namely Dali which had amazing results but as we're going to"}, {"start": 33.08, "end": 38.2, "text": " soon see Glide has even better results and they kind of verified that using"}, {"start": 38.2, "end": 42.160000000000004, "text": " human judges so we're gonna see the results a bit later but first with those"}, {"start": 42.160000000000004, "end": 45.400000000000006, "text": " of you who are not even familiar with Dali let me let me show you what the"}, {"start": 45.400000000000006, "end": 50.52, "text": " hype is all about this is what like Glide produces so Glide produces these"}, {"start": 50.52, "end": 56.760000000000005, "text": " images conditioned on these prompts here so we have a prompt here a hedgehog"}, {"start": 56.76, "end": 62.58, "text": " using a calculator and we can see an image that like Glide generated which is"}, {"start": 62.58, "end": 66.67999999999999, "text": " fairly reasonable I mean you as a human probably wouldn't be able to do a much"}, {"start": 66.67999999999999, "end": 71.48, "text": " better job although there's some there is some blur here which may be to the"}, {"start": 71.48, "end": 74.64, "text": " fact that they're using diffusion models but we're gonna see and discuss those a"}, {"start": 74.64, "end": 79.88, "text": " bit later then we have a like another example a corgi wearing a red bowtie"}, {"start": 79.88, "end": 84.08, "text": " and you can see here we have a this attribute binding we basically have the"}, {"start": 84.08, "end": 88.28, "text": " model has to understand that red corresponds to bowtie and then we"}, {"start": 88.28, "end": 93.56, "text": " have purple the color purple corresponds to this party hat so the model first has"}, {"start": 93.56, "end": 98.4, "text": " to kind of hallucinate and create those and then the plate to place them on the"}, {"start": 98.4, "end": 103.03999999999999, "text": " right spot in the image and make it look like a realistic and I think they do an"}, {"start": 103.03999999999999, "end": 106.75999999999999, "text": " awesome job and by the way I don't know what like what's up with corgis I can"}, {"start": 106.75999999999999, "end": 112.08, "text": " see corgis everywhere I guess some of the open AI members really likes corgis"}, {"start": 112.08, "end": 117.36, "text": " or it's there kind of in-house pet or something in other cool picture here"}, {"start": 117.36, "end": 123.2, "text": " robots meditating in a vipa sauna retreat fairly sure you cannot find this"}, {"start": 123.2, "end": 128.36, "text": " image anywhere but it'd be nice to see the the nearest neighbor from the"}, {"start": 128.36, "end": 131.64, "text": " training data set from all of these generated images although yeah this"}, {"start": 131.64, "end": 136.96, "text": " looks very very awesome if you ask me I'm gonna just cherry pick a couple more"}, {"start": 136.96, "end": 141.16, "text": " examples here and all of these are also cherry picked already they're using"}, {"start": 141.16, "end": 145.64, "text": " something called guidance like classifier free if I'm not wrong let me"}, {"start": 145.64, "end": 149.48, "text": " just check it out here classifier free guidance yeah that's right so here we"}, {"start": 149.48, "end": 154.92, "text": " can see a high quality oil painting of a psychedelic hamster dragon and this is"}, {"start": 154.92, "end": 158.76, "text": " totally wicked like psychedelic okay we can see kind of the colors are there"}, {"start": 158.76, "end": 163.56, "text": " then we have hamster hamster is also there dragon we can see these these"}, {"start": 163.56, "end": 168.48, "text": " wings in the background which kind of reminds us of the dragon high quality"}, {"start": 168.48, "end": 171.95999999999998, "text": " well I'm not sure about that part but like all it definitely has this oil"}, {"start": 171.95999999999998, "end": 178.6, "text": " painting vibe to it so again amazing amazing rendering of this of this prompt"}, {"start": 178.6, "end": 187.6, "text": " here many other cool results we have in other corgi here and finally we have"}, {"start": 187.6, "end": 193.16, "text": " this thing here so a red cube on top of a blue cube and I'd love to see some"}, {"start": 193.16, "end": 197.39999999999998, "text": " more complex query some more complex prompt where the model we have to do"}, {"start": 197.4, "end": 202.08, "text": " more serious relational reasoning because as far as I remember in the"}, {"start": 202.08, "end": 206.56, "text": " Dali paper they tried and and and constructed those and the model the Dali"}, {"start": 206.56, "end": 211.8, "text": " failed like didn't manage to either count well or fail to do the attribute"}, {"start": 211.8, "end": 216.16, "text": " binding correctly like associate the correct color with the shapes etc so it'd"}, {"start": 216.16, "end": 221.36, "text": " be cool to kind of further probe this model but that's gonna be hard because"}, {"start": 221.36, "end": 226.12, "text": " they only released a smaller model which is trained on a filter data set so it's"}, {"start": 226.12, "end": 231.84, "text": " not not nearly as potent as the one they have in house behind I guess yeah"}, {"start": 231.84, "end": 238.52, "text": " not sure whether they have an API for the for this glide model in any case"}, {"start": 238.52, "end": 243.56, "text": " let's try and understand the magic behind behind this model so first things"}, {"start": 243.56, "end": 248.20000000000002, "text": " first let me just kind of mention here when evaluated by human judges our"}, {"start": 248.20000000000002, "end": 253.8, "text": " samples are preferred to those from Dali 87% of the time when evaluated for"}, {"start": 253.8, "end": 260.24, "text": " photorealism and 69% of the time when evaluated for caption similarity so"}, {"start": 260.24, "end": 264.04, "text": " that's cool so that means that this model definitely improved upon Dali and"}, {"start": 264.04, "end": 269.40000000000003, "text": " the main like component that that brought that improvement I'd say is"}, {"start": 269.40000000000003, "end": 272.72, "text": " these diffusion models and you're probably familiar with some other"}, {"start": 272.72, "end": 277.36, "text": " generative models such as GANs generative other serial networks or VAE's"}, {"start": 277.36, "end": 282.52, "text": " variational autoencoders or even flow based models but if you're models are"}, {"start": 282.52, "end": 288.68, "text": " probably new for for most of you because only in 2020 did people like manage to"}, {"start": 288.68, "end": 293.35999999999996, "text": " get them to work to be in the sense they are they are now kind of you can you can"}, {"start": 293.35999999999996, "end": 298.64, "text": " sample very fast like relatively speaking and you can train them and they"}, {"start": 298.64, "end": 302.76, "text": " just work there is a nice relationship between diffusion models and VAE's we're"}, {"start": 302.76, "end": 308.12, "text": " gonna see that the actual loss for the diffusion models is inspired by these VAE"}, {"start": 308.12, "end": 311.96, "text": " models and also there is another class of generative models called score based"}, {"start": 311.96, "end": 315.35999999999996, "text": " models and I'm gonna link a lot of resources down in the video description"}, {"start": 315.35999999999996, "end": 319.91999999999996, "text": " so do check it out so there is a nice connection with that family as well I"}, {"start": 319.91999999999996, "end": 322.96, "text": " don't have it here but yeah you can check it out in the video description"}, {"start": 322.96, "end": 329.96, "text": " and another thing I want to point out here is that let's let's see the like a"}, {"start": 329.96, "end": 334.0, "text": " difference at least on this block diagram the main difference between like"}, {"start": 334.0, "end": 337.88, "text": " all of these models and and diffusion models and you can see let me try and"}, {"start": 337.88, "end": 344.4, "text": " use my fancy new touchpad basically you can see that the latents here the"}, {"start": 344.4, "end": 349.84, "text": " latents here are completely like they are of the same dimensionality as the as"}, {"start": 349.84, "end": 354.8, "text": " the input image whereas usually when you have something like VAE so it's an"}, {"start": 354.8, "end": 359.68, "text": " autoencoder that means you're basically down you're down sampling you're you're"}, {"start": 359.68, "end": 364.76, "text": " reducing the dimensionality of your input data into some latent vector here"}, {"start": 364.76, "end": 369.48, "text": " usually denoted as Z and this one is way like lower dimensional compared to the"}, {"start": 369.48, "end": 374.59999999999997, "text": " input input data you're trying to model here on the other side you're you're"}, {"start": 374.59999999999997, "end": 381.36, "text": " basically you're basically keeping the dimensionality the same as the as your"}, {"start": 381.36, "end": 385.92, "text": " input and you can imagine that that can cause some problems training this model"}, {"start": 385.92, "end": 388.92, "text": " and sampling from this generative model etc but we're gonna see how people"}, {"start": 388.92, "end": 395.16, "text": " manage to to work their way around it kinda I say kinda because before we even"}, {"start": 395.16, "end": 398.16, "text": " dig into details let me just show you I mean there are obviously some"}, {"start": 398.16, "end": 402.36, "text": " limitations and they pointed those out somewhere here let me try and find"}, {"start": 402.36, "end": 408.24, "text": " limitations okay our unoptimized model takes 15 seconds to sample one image on"}, {"start": 408.24, "end": 414.04, "text": " a single a hundred GPU and we're going to understand why this is basically the"}, {"start": 414.04, "end": 419.16, "text": " reason is that we have to do like 20 forward passes in order to render an"}, {"start": 419.16, "end": 424.24, "text": " image whereas using something like GAN or VAE you do a single forward pass and"}, {"start": 424.24, "end": 428.72, "text": " you've got yourself an image so that's that's a limitation to keep in mind and"}, {"start": 428.72, "end": 436.28000000000003, "text": " yeah so let's get back up here okay before I even start digging deeper into"}, {"start": 436.28000000000003, "end": 440.8, "text": " how diffusion models work which is going to be very ungrateful job for me to do"}, {"start": 440.8, "end": 444.24, "text": " because it'll probably take a whole video to do it so I'm gonna do a high"}, {"start": 444.24, "end": 449.44, "text": " level overview first let's see just the what the model can do on a high level"}, {"start": 449.44, "end": 453.88, "text": " what the model is capable of so here we can see some in painting capabilities"}, {"start": 453.88, "end": 458.32, "text": " and thing to keep in mind here is that they had to do some fine-tuning so this"}, {"start": 458.32, "end": 463.88, "text": " is not zero shot as as your Dali model or some of the other like images rendered"}, {"start": 463.88, "end": 468.6, "text": " in this paper this like model was fine-tuned for this precise task of in"}, {"start": 468.6, "end": 473.64000000000004, "text": " painting and you can see the awesome results so they they have this mask and"}, {"start": 473.64000000000004, "end": 478.48, "text": " the model inputs to the kind of I think that they feed that the three channeled"}, {"start": 478.48, "end": 482.64000000000004, "text": " RGB image they also feed this this mask as a separate channel and I think they"}, {"start": 482.64000000000004, "end": 485.20000000000005, "text": " have a couple more channels in that in painting model but that's not that"}, {"start": 485.20000000000005, "end": 489.40000000000003, "text": " important right now what's important is you can see that results are fairly"}, {"start": 489.40000000000003, "end": 494.28000000000003, "text": " stunning again here a girl hugging a corgi on a pedestal you can see it"}, {"start": 494.28, "end": 499.28, "text": " rendered corgi pretty awesome like just look at this this like hand wrapping"}, {"start": 499.28, "end": 504.55999999999995, "text": " around the body of this corgi I mean the results are if you if you told me like a"}, {"start": 504.55999999999995, "end": 509.35999999999996, "text": " couple of years ago that AI models will be able to do this I'd say it's it's"}, {"start": 509.35999999999996, "end": 513.52, "text": " like almost impossible and now I think people are just kind of used to AI being"}, {"start": 513.52, "end": 516.88, "text": " able to do this and we're setting even higher and higher goals and so now this"}, {"start": 516.88, "end": 521.4399999999999, "text": " is not like nothing special but yeah we need to appreciate the progress that"}, {"start": 521.44, "end": 525.6, "text": " happened since 2012 and of course earlier than that but yeah with the Alex"}, {"start": 525.6, "end": 530.24, "text": " net the things really started skyrocketing same thing here you can see"}, {"start": 530.24, "end": 535.0, "text": " the red hair you can see how the the ways of flowers appears I especially"}, {"start": 535.0, "end": 539.24, "text": " find this one fascinating the hat lies perfectly like the perspective the the"}, {"start": 539.24, "end": 543.0400000000001, "text": " color maybe the color is a bit brighter you can see the hopefully can see it on"}, {"start": 543.0400000000001, "end": 548.0, "text": " your screen the color is a bit brighter compared to this shirt here so yeah but"}, {"start": 548.0, "end": 553.72, "text": " still fairly stunning results and yeah small knit here glad stands for a guided"}, {"start": 553.72, "end": 557.92, "text": " language to image diffusion for generation and editing you can understand"}, {"start": 557.92, "end": 561.6, "text": " why that is we can both generate we can add it as you can see here we have"}, {"start": 561.6, "end": 564.76, "text": " language that's conditioned we are basically conditioning our our diffusion"}, {"start": 564.76, "end": 569.9, "text": " model on some language prompt and we basically generate images using the"}, {"start": 569.9, "end": 573.72, "text": " diffusion models so that's kind of trying to deconstruct that that complex"}, {"start": 573.72, "end": 579.0, "text": " sentence here are some work like awesome capabilities of Clyde model you can see"}, {"start": 579.0, "end": 585.6, "text": " here that first they kind of generate using the cozy living room they zero"}, {"start": 585.6, "end": 589.72, "text": " shot generate this image of the cozy living room and then they put a mouse"}, {"start": 589.72, "end": 594.48, "text": " like a mask here and then they say a painting of a corgi on the wall above a"}, {"start": 594.48, "end": 598.96, "text": " couch and here comes corgi again and then they put a mask again and then they"}, {"start": 598.96, "end": 602.84, "text": " say around coffee table in front of a couch and here comes the table and it's"}, {"start": 602.84, "end": 608.0, "text": " round and that's I mean fairly awesome and then again they put this mask and"}, {"start": 608.0, "end": 612.6, "text": " they see a vase of flowers on the coffee table and again we see the results"}, {"start": 612.6, "end": 616.88, "text": " rendered and finally here they say they put a mask here and they say a couch in"}, {"start": 616.88, "end": 620.96, "text": " the corner of her room and now I think this is a failure case or I may be"}, {"start": 620.96, "end": 624.88, "text": " missing something out but basically what happened is that we just got ourself a"}, {"start": 624.88, "end": 629.6800000000001, "text": " window here but I don't see like a couch in the corner of her room or maybe or"}, {"start": 629.68, "end": 634.7199999999999, "text": " maybe it kind of cropped this part so now this is this is like officially in"}, {"start": 634.7199999999999, "end": 638.56, "text": " the corner of a room although we don't see the wall here so but but anyways"}, {"start": 638.56, "end": 645.4, "text": " that's maybe me being too nitpicky here but yeah the results are very cool here"}, {"start": 645.4, "end": 649.3599999999999, "text": " this just shows some combination using this as the edit and here they kind of"}, {"start": 649.3599999999999, "end": 653.68, "text": " like instead of putting this generic mask with a generic shape they kind of"}, {"start": 653.68, "end": 658.8, "text": " help the model a bit more by using both this is kind of semantic segmentation in"}, {"start": 658.8, "end": 662.52, "text": " a way and they put a approximate shape and so it's easier to generate the"}, {"start": 662.52, "end": 666.4799999999999, "text": " picture here you can see I can see the results yourself here okay so that's"}, {"start": 666.4799999999999, "end": 673.16, "text": " what glide can do now let me try and demystify how this thing actually works"}, {"start": 673.16, "end": 679.04, "text": " and it's gonna be very very hard to try and explain how these diffusion models"}, {"start": 679.04, "end": 684.3199999999999, "text": " work very quickly but yeah let me try it if you think the explanation was not"}, {"start": 684.3199999999999, "end": 688.4399999999999, "text": " good enough I can maybe try just kind of upload my comment and I can I can create"}, {"start": 688.44, "end": 693.9200000000001, "text": " another video explaining just the diffusion models okay so the the process"}, {"start": 693.9200000000001, "end": 697.8000000000001, "text": " itself these diffusion models were inspired by some non-equilibrium"}, {"start": 697.8000000000001, "end": 702.44, "text": " statistical physics I know it sounds very very complicated because it is but"}, {"start": 702.44, "end": 708.84, "text": " like back in 2015 the first paper appeared and they managed to to basically"}, {"start": 708.84, "end": 713.44, "text": " originated the idea of the diffusion models but only in 2020 do we get the"}, {"start": 713.44, "end": 718.32, "text": " models to to work nicely to generate nice samples and to be like faster and"}, {"start": 718.32, "end": 724.72, "text": " yeah so here is the the main idea you have this diffusion process whereas you"}, {"start": 724.72, "end": 730.24, "text": " take your input image and what you do is you basically add add a certain amount"}, {"start": 730.24, "end": 734.2800000000001, "text": " of noise from a specific family like usually people use Gaussian because"}, {"start": 734.2800000000001, "end": 738.84, "text": " cautions are from this thing called exponential family which means they have"}, {"start": 738.84, "end": 742.2, "text": " all of these nice properties when you add up cautions when you do product with"}, {"start": 742.2, "end": 746.48, "text": " them you end up with a Gaussian like the structure of the Gaussian remains and"}, {"start": 746.48, "end": 750.08, "text": " it's easy to compute and that's why people are using Gossians all around"}, {"start": 750.08, "end": 755.08, "text": " like in VAE's as well for the for the priors etc etc so basically they have"}, {"start": 755.08, "end": 760.5600000000001, "text": " nice computational properties so we add up some noise and then we do we continue"}, {"start": 760.5600000000001, "end": 765.4000000000001, "text": " doing that until we get to like a picture which is basically pure Gaussian"}, {"start": 765.4000000000001, "end": 770.48, "text": " noise now wouldn't it be nice if we could reverse the process and starting"}, {"start": 770.48, "end": 774.96, "text": " from a Gaussian noise so just sampling this Gaussian noise picture reverse the"}, {"start": 774.96, "end": 778.8000000000001, "text": " process and generate our data so that would be cool so again we have this"}, {"start": 778.8000000000001, "end": 784.9200000000001, "text": " conditional distribution so X of t conditioned on the X of t minus 1 which"}, {"start": 784.9200000000001, "end": 790.44, "text": " is timestamp the previous timestamp and we basically know that because that's"}, {"start": 790.44, "end": 795.0600000000001, "text": " described by this formula here and I'm gonna quickly explain how this thing"}, {"start": 795.06, "end": 800.7199999999999, "text": " works in a second and then what's problematic is that we cannot it's not easy to"}, {"start": 800.7199999999999, "end": 805.5999999999999, "text": " figure out the reverse distribution here so so basically not a reverse"}, {"start": 805.5999999999999, "end": 808.92, "text": " distribution by this conditional distribution where you're trying to"}, {"start": 808.92, "end": 813.5999999999999, "text": " figure out the t minus 1 condition on the X of t so you're trying to denoise"}, {"start": 813.5999999999999, "end": 819.3599999999999, "text": " the image here so what diffusion models are trying to do is construct this p of"}, {"start": 819.36, "end": 825.4, "text": " theta which is going to be a neural network in practice to try and model"}, {"start": 825.4, "end": 828.92, "text": " this this unknown distribution here and we're gonna see a couple of nice"}, {"start": 828.92, "end": 833.8000000000001, "text": " properties of this distribution which will allow us to actually do that and"}, {"start": 833.8000000000001, "end": 839.8000000000001, "text": " okay before we get there let me try and dissect this this formula here so we can"}, {"start": 839.8000000000001, "end": 845.32, "text": " see that the notation is kind of cringy I love this notation better when you have"}, {"start": 845.32, "end": 851.5200000000001, "text": " like for example let me just take the touchpad here so I like it better when"}, {"start": 851.5200000000001, "end": 856.0400000000001, "text": " you have something like X of t and then you're sampling from the normal"}, {"start": 856.0400000000001, "end": 859.48, "text": " distribution or whatever distribution like so like having this squiggly thing"}, {"start": 859.48, "end": 863.5600000000001, "text": " but yeah just be aware how this notation works this just means we're sampling X"}, {"start": 863.5600000000001, "end": 870.2, "text": " of t from like the Gaussian distribution denoted by n and this is the mu so this"}, {"start": 870.2, "end": 874.8000000000001, "text": " thing here this part here is the mu so that's the mean and this thing here is"}, {"start": 874.8, "end": 880.16, "text": " just the covariance matrix okay so let's try and dissect this so we have X of t"}, {"start": 880.16, "end": 886.0, "text": " minus 1 multiplied by this thing here and beta t is something called I think"}, {"start": 886.0, "end": 889.0799999999999, "text": " it's called noise level something I forgot the exact name but whatever"}, {"start": 889.0799999999999, "end": 893.16, "text": " basically it's between 0 and 1 which means that this thing here under"}, {"start": 893.16, "end": 897.92, "text": " square root is gonna be less than 1 and that means that we're gonna"}, {"start": 897.92, "end": 902.4399999999999, "text": " contract this X of t minus 1 okay so let me try and visualize this thing right"}, {"start": 902.44, "end": 909.96, "text": " here okay so we have and for the sake of clarity I'm gonna assume that X of t"}, {"start": 909.96, "end": 914.4000000000001, "text": " minus 1 which is our image is actually only three dimensional data point so"}, {"start": 914.4000000000001, "end": 917.8800000000001, "text": " that I can actually visualize what's happening otherwise if this is thousand"}, {"start": 917.8800000000001, "end": 921.08, "text": " by thousand that's million dimensions which means I cannot draw a million"}, {"start": 921.08, "end": 926.7600000000001, "text": " dimensions on my one node screen here so here is the coordinate system so let's"}, {"start": 926.7600000000001, "end": 931.9200000000001, "text": " say that X of t minus 1 is somewhere here and so this is the origin point"}, {"start": 931.92, "end": 937.16, "text": " here it's kind of let me kind of try and connect it here so what this thing here"}, {"start": 937.16, "end": 942.76, "text": " does is it's contracting because it's more than one which means then the mu"}, {"start": 942.76, "end": 946.56, "text": " the mean of this distribution let me take a different color here let me try"}, {"start": 946.56, "end": 951.24, "text": " and take the green one it's gonna kind of push this point from here all the way"}, {"start": 951.24, "end": 955.36, "text": " to somewhere here okay so this is gonna be whoops it's gonna be well I'm still"}, {"start": 955.36, "end": 959.4, "text": " trying to get used to this novel touchpad okay so we have the point now"}, {"start": 959.4, "end": 964.8, "text": " here and now this beta is gonna be we're gonna have a diagonal I guess"}, {"start": 964.8, "end": 968.9399999999999, "text": " isotropic covariance matrix which means we're gonna have I'm gonna denote that"}, {"start": 968.9399999999999, "end": 972.4599999999999, "text": " as some small circle around this point so we're gonna have some Gaussian"}, {"start": 972.4599999999999, "end": 977.1999999999999, "text": " distribution centered around this point here and we're gonna sample our X of t"}, {"start": 977.1999999999999, "end": 982.9599999999999, "text": " from that distribution here and so let's say we sample it let me try and change"}, {"start": 982.9599999999999, "end": 986.92, "text": " the color here let me take the blue color so let's say we sample data point"}, {"start": 986.92, "end": 990.88, "text": " here so that's gonna be our so we started with with here so we had a X t"}, {"start": 990.88, "end": 995.5999999999999, "text": " minus one now we are here so this is visually representing in this 3d space"}, {"start": 995.5999999999999, "end": 999.56, "text": " what's happening in this image space where we're getting more noisy and noisy"}, {"start": 999.56, "end": 1003.92, "text": " images so maybe from this image we got this one although this will take more"}, {"start": 1003.92, "end": 1007.4399999999999, "text": " steps obviously but yeah you get a point now let's see what happens in the limit"}, {"start": 1007.4399999999999, "end": 1012.52, "text": " the thing with these betas is that in the original paper in I think in the"}, {"start": 1012.52, "end": 1018.6, "text": " paper from 2020 they were not learnable they just took and created a schedule"}, {"start": 1018.6, "end": 1022.04, "text": " that started with some larger betas which are still smaller than one and"}, {"start": 1022.04, "end": 1027.6, "text": " then slowly anneal them towards zero okay linearly I think so we had a linear"}, {"start": 1027.6, "end": 1033.96, "text": " schedule and that means when beta t gets down to zero what happens actually it's"}, {"start": 1033.96, "end": 1038.32, "text": " the other way around it starts with very small because we want to have like"}, {"start": 1038.32, "end": 1043.04, "text": " lesser noise so it starts with zero and then it's a meal towards one okay that"}, {"start": 1043.04, "end": 1046.3999999999999, "text": " will make more sense so what happens when it gets to one that means this is"}, {"start": 1046.3999999999999, "end": 1050.8, "text": " going to zero so this thing here let me change the color so I'm gonna take the"}, {"start": 1050.8, "end": 1055.2, "text": " red pen here and so basically this thing is gonna be a kneeled it's gonna go to"}, {"start": 1055.2, "end": 1061.36, "text": " zero and that means that this thing here is gonna be contracted all the way down"}, {"start": 1061.36, "end": 1067.96, "text": " here until we get to the origin point so that means that after a lot of steps"}, {"start": 1067.96, "end": 1074.72, "text": " our data points are gonna be basically drawn from from this from this like"}, {"start": 1074.72, "end": 1080.2, "text": " distribution which is basically your your standard Gaussian distribution the"}, {"start": 1080.2, "end": 1084.5, "text": " normal distribution because we'll have a mu of zero because this is zero so that"}, {"start": 1084.5, "end": 1089.6000000000001, "text": " means this thing is gonna be zero and this thing is gonna be one which means"}, {"start": 1089.6000000000001, "end": 1093.4, "text": " we have identity matrix and we are basically sampling from a noise pure"}, {"start": 1093.4, "end": 1096.48, "text": " noise distribution and that's what I explained here so that that's basically"}, {"start": 1096.48, "end": 1101.3600000000001, "text": " this image here so that's at the limit of this diffusion process now you might"}, {"start": 1101.3600000000001, "end": 1107.3600000000001, "text": " think that this would be very slow if we had to sample like if we had to sample"}, {"start": 1107.3600000000001, "end": 1113.28, "text": " and get X 100 we'd have to repeat this sampling procedure hundred times but"}, {"start": 1113.28, "end": 1118.48, "text": " that's not the case luckily for us because you can you can form you can"}, {"start": 1118.48, "end": 1122.8, "text": " because you can see here Q X of T conditions of X of zero say X of zero is"}, {"start": 1122.8, "end": 1126.16, "text": " the original image from our data set the actual images we have in our data set"}, {"start": 1126.16, "end": 1131.68, "text": " we can form like a distribution so that we can directly sample and get and kind"}, {"start": 1131.68, "end": 1137.2, "text": " of skip these intermediate steps and like get the X 100 or X 200 from a"}, {"start": 1137.2, "end": 1144.0400000000002, "text": " single from a single like a step and you can see it here again these alpha T with"}, {"start": 1144.0400000000002, "end": 1149.44, "text": " this hat thing is a product of alpha of these coefficients here and they are 1"}, {"start": 1149.44, "end": 1153.28, "text": " minus beta and you can imagine because all of these are smaller than 1 when you"}, {"start": 1153.28, "end": 1156.36, "text": " multiple a bunch of numbers that are smaller than 1 this thing is going to"}, {"start": 1156.36, "end": 1160.6399999999999, "text": " converge in the limit it's going to go down to zero to zero okay it's going to"}, {"start": 1160.6399999999999, "end": 1166.04, "text": " down to zero so that means what we end up with is nothing here and we end up"}, {"start": 1166.04, "end": 1170.8, "text": " with one here which means this thing is going to be equal to this small epsilon"}, {"start": 1170.8, "end": 1175.52, "text": " here which is as you can see here the normal distribution so that's what"}, {"start": 1175.52, "end": 1180.8, "text": " happens again in the in the limit of this diffusion process now this is very"}, {"start": 1180.8, "end": 1187.2, "text": " important they state here and this is this is actually a conclusion from 1949"}, {"start": 1187.2, "end": 1192.44, "text": " I think they say note that Q so this is conditional distribution we are we're"}, {"start": 1192.44, "end": 1198.12, "text": " trying to model approaches a diagonal Gaussian distribution as capital T"}, {"start": 1198.12, "end": 1201.68, "text": " approaches infinity and this is the number of steps in our diffusion"}, {"start": 1201.68, "end": 1206.44, "text": " process in correspondingly this this noise level in the limit goes down to"}, {"start": 1206.44, "end": 1211.2, "text": " zero so it's sufficient to train a neural network to predict a mean mu"}, {"start": 1211.2, "end": 1218.0800000000002, "text": " theta and a diagonal covariance matrix Sigma theta and so this will be the this"}, {"start": 1218.0800000000002, "end": 1223.24, "text": " is this thing here is the distribution we're trying to learn in order to model"}, {"start": 1223.24, "end": 1228.72, "text": " the actual the true distribution here so this one the unknown one okay and the"}, {"start": 1228.72, "end": 1234.48, "text": " reason why it's possible is because we know that under these conditions the"}, {"start": 1234.48, "end": 1239.16, "text": " actual true distribution is going to have that same shape and then we can"}, {"start": 1239.16, "end": 1243.24, "text": " just kind of learn and kind of fit this caution on top on top of that"}, {"start": 1243.24, "end": 1249.32, "text": " distribution now if you're familiar with VA is this may be this whole like a"}, {"start": 1249.32, "end": 1256.2, "text": " diagram here may be familiar and if I were to swap if I were to swap this and"}, {"start": 1256.2, "end": 1262.16, "text": " kind of change it to and denote it as Z which is usual notation VA is because X"}, {"start": 1262.16, "end": 1267.28, "text": " one through T are the latents remember I mentioned that the latents here in this"}, {"start": 1267.28, "end": 1271.8400000000001, "text": " model are of the same dimension so these are the latents these noisy images here"}, {"start": 1271.8400000000001, "end": 1278.0, "text": " are the latents of our of our image here and basically if I were to denote these"}, {"start": 1278.0, "end": 1282.8000000000002, "text": " as Z so this one here as Z you may notice this formula from the VA is"}, {"start": 1282.8000000000002, "end": 1286.76, "text": " construction and here we're just trying to like construct something that's"}, {"start": 1286.76, "end": 1291.3200000000002, "text": " called evidence lower bound and it's fairly self-explanatory although it may"}, {"start": 1291.32, "end": 1295.1599999999999, "text": " be super confusing when you hear it the first time so why is that why is it"}, {"start": 1295.1599999999999, "end": 1299.1599999999999, "text": " called elbow or evidence lower bound so first we have this thing here is called"}, {"start": 1299.1599999999999, "end": 1303.8799999999999, "text": " evidence so this is evidence or I think you kept include the log one so this"}, {"start": 1303.8799999999999, "end": 1307.24, "text": " basically you want to maximize this you want to have your data points from your"}, {"start": 1307.24, "end": 1313.52, "text": " data set have high probability under your fitted distributions under your"}, {"start": 1313.52, "end": 1318.4399999999998, "text": " learned distribution P of theta so that's what machine learning is models"}, {"start": 1318.44, "end": 1322.92, "text": " are oftentimes trying to do we're trying to maximize the probability of the data"}, {"start": 1322.92, "end": 1328.64, "text": " points under our our model okay now because log is monotonic it actually"}, {"start": 1328.64, "end": 1332.6000000000001, "text": " does not matter whether you're optimizing you're not going to change"}, {"start": 1332.6000000000001, "end": 1335.6000000000001, "text": " the loss function by adding a log you're gonna change some computational"}, {"start": 1335.6000000000001, "end": 1339.16, "text": " properties and the convergence speed etc so it's kind of usually needs to have"}, {"start": 1339.16, "end": 1344.2, "text": " log inside but so if we have this why don't we try and maximize it directly"}, {"start": 1344.2, "end": 1347.6000000000001, "text": " well the problem is it's intractable so you can try and break this thing down"}, {"start": 1347.6, "end": 1351.04, "text": " and you end up with something with a complex integral that looks something"}, {"start": 1351.04, "end": 1360.3999999999999, "text": " like this we have these P theta effects zero and we'll have the prior so the"}, {"start": 1360.3999999999999, "end": 1365.32, "text": " probability of the actual latent and we'll have to basically integrate all"}, {"start": 1365.32, "end": 1369.52, "text": " over all of those latents as you can see here we have to sample the latents to"}, {"start": 1369.52, "end": 1374.7199999999998, "text": " Z the Zets we have to have their probabilities we have to to construct"}, {"start": 1374.72, "end": 1379.04, "text": " this to basically compute this integral which is going to be it's simply"}, {"start": 1379.04, "end": 1382.76, "text": " intractable so we cannot directly compute this thing here so this is equal"}, {"start": 1382.76, "end": 1389.08, "text": " to the probability theta X zero you'd have to do this thing here in order to"}, {"start": 1389.08, "end": 1393.08, "text": " get this this probability here so that's that's out of the question we can do"}, {"start": 1393.08, "end": 1396.96, "text": " that so that's why people are constructing these lower bounds and"}, {"start": 1396.96, "end": 1403.0, "text": " basically this is trivial by definition because KL divergence is always greater"}, {"start": 1403.0, "end": 1407.84, "text": " or equal than zero it's equal only in the case when the two distributions"}, {"start": 1407.84, "end": 1413.04, "text": " perfectly match which we're trying to do by training these beta t to match the"}, {"start": 1413.04, "end": 1419.36, "text": " the P distribution to to match the real Q distribution so after doing some"}, {"start": 1419.36, "end": 1424.12, "text": " algebra here I'm not gonna dig into a lot of details we end up with this"}, {"start": 1424.12, "end": 1430.6, "text": " formula here and this is what we call the the elbow the like like variance"}, {"start": 1430.6, "end": 1437.56, "text": " lower bound like loss and you can see here that basically if we tried and"}, {"start": 1437.56, "end": 1444.28, "text": " minimized this thing we know that it's gonna be that the actual thing we care"}, {"start": 1444.28, "end": 1449.56, "text": " about is gonna be even lower so what's called lower bound if it's actually"}, {"start": 1449.56, "end": 1452.84, "text": " higher because usually you don't have the minus and then you're trying to"}, {"start": 1452.84, "end": 1457.84, "text": " maximize this thing and this thing is gonna be smaller or equal and that means"}, {"start": 1457.84, "end": 1461.6, "text": " as you're maximizing it you know that the actual thing you care about which is"}, {"start": 1461.6, "end": 1467.32, "text": " the the the evidence here so this thing here is gonna be at least as big as"}, {"start": 1467.32, "end": 1471.56, "text": " your loss so this is also called surrogate loss because you're trying to"}, {"start": 1471.56, "end": 1475.84, "text": " find different loss it's kind of going to bound your actual thing you care"}, {"start": 1475.84, "end": 1483.36, "text": " about okay so long story short what happens is they kind of further have to"}, {"start": 1483.36, "end": 1487.1599999999999, "text": " break down this this equation so that it's actually tractable and computable"}, {"start": 1487.16, "end": 1492.16, "text": " so they just end up in a sum of various scale divergences and at the end they"}, {"start": 1492.16, "end": 1498.48, "text": " end up with a simple MSC loss between the the the mu theta so that's the the"}, {"start": 1498.48, "end": 1502.6000000000001, "text": " thing we're trying to learn here the mu theta of our P theta distribution and"}, {"start": 1502.6000000000001, "end": 1506.8000000000002, "text": " they just need to do MSC loss but what happens is they actually instead of"}, {"start": 1506.8000000000002, "end": 1512.0, "text": " figuring out the mu they actually just figure out the noise level that happens"}, {"start": 1512.0, "end": 1515.68, "text": " in that happened in a specific step and let me try and break that down in a"}, {"start": 1515.68, "end": 1520.44, "text": " second oops by the way I realized I forgot to put the condition this on that"}, {"start": 1520.44, "end": 1526.1200000000001, "text": " so you have to condition so it's P of X zero given that and we see we look at the"}, {"start": 1526.1200000000001, "end": 1529.24, "text": " probability of that so what's the probability of that and then what's the"}, {"start": 1529.24, "end": 1536.3200000000002, "text": " probability of this X zero being so what's the probability of X zero given"}, {"start": 1536.3200000000002, "end": 1539.52, "text": " that that so we have to integrate all of that to get the actual probability of X"}, {"start": 1539.52, "end": 1542.48, "text": " zero so it's a small mistake hopefully you can see what's happening there"}, {"start": 1542.48, "end": 1547.0, "text": " anyways on the high level just it's hard it's intractable that's everything I"}, {"start": 1547.0, "end": 1550.4, "text": " wanted to keep in mind that's it you don't have to know it is the probability"}, {"start": 1550.4, "end": 1556.16, "text": " details here okay so let me try and break this down why why they're why"}, {"start": 1556.16, "end": 1560.44, "text": " they're what they're doing here so instead of figuring out mu so that means"}, {"start": 1560.44, "end": 1564.76, "text": " instead of let's let's say let me go back to this picture here so let's say"}, {"start": 1564.76, "end": 1574.32, "text": " this is X of this is X of t and this was X of t minus 1 this point here was X of"}, {"start": 1574.32, "end": 1580.4, "text": " t minus 1 this one is X of t so instead of trying to predict given X of t trying"}, {"start": 1580.4, "end": 1584.84, "text": " to predict this this mu here so that that's the point from where we"}, {"start": 1584.84, "end": 1589.36, "text": " generated in the first place this point X of t instead of doing that they figure"}, {"start": 1589.36, "end": 1592.8, "text": " out the noise which is the same thing in the end we're gonna see that but it's up"}, {"start": 1592.8, "end": 1597.3999999999999, "text": " it's a bit different so let's say we had an image here so let's say we had an"}, {"start": 1597.3999999999999, "end": 1602.04, "text": " image here and it had some amount of noise like it had a couple of noise"}, {"start": 1602.04, "end": 1605.76, "text": " points here I'm gonna reduce the amount of noise obviously to make it easier to"}, {"start": 1605.76, "end": 1612.24, "text": " explain so we want to basically from there on we want to understand which"}, {"start": 1612.24, "end": 1619.8, "text": " portions of the noise were added going from step t minus 1 to step t we want"}, {"start": 1619.8, "end": 1625.3999999999999, "text": " to understand which of these noise points were added so maybe we figure out"}, {"start": 1625.3999999999999, "end": 1629.28, "text": " okay these were the noise points that was the noise that was added and"}, {"start": 1629.28, "end": 1634.76, "text": " basically we end up with something like this we end up with a image here that"}, {"start": 1634.76, "end": 1639.56, "text": " has these two but we we cannot get got rid of this of this noise here and you"}, {"start": 1639.56, "end": 1642.72, "text": " can imagine by doing this we are reducing the noise and at the end we're"}, {"start": 1642.72, "end": 1647.12, "text": " gonna end up with the final picture that's noise free image from our data"}, {"start": 1647.12, "end": 1650.0, "text": " distribution and that's awesome so that's what's happening they're trying"}, {"start": 1650.0, "end": 1653.76, "text": " to reduce and understand what was the noise that was added between the two"}, {"start": 1653.76, "end": 1658.1599999999999, "text": " steps that's that's everything I want you to understand here and that's what"}, {"start": 1658.1599999999999, "end": 1662.9199999999998, "text": " they do exactly here so you can see here we have this L simple so the simple loss"}, {"start": 1662.9199999999998, "end": 1667.04, "text": " because they had some simplifications here but this turned out to work quite"}, {"start": 1667.04, "end": 1672.9599999999998, "text": " nice empirically so let's see how this L simple works so we have expectation over"}, {"start": 1672.96, "end": 1678.04, "text": " the time step goes from 1 to capital T which is the size of our diffusion chain"}, {"start": 1678.04, "end": 1683.2, "text": " let's call it that way we are sampling X 0 from our from the true distribution so"}, {"start": 1683.2, "end": 1686.78, "text": " this is a fancy way of saying we take a data sample we take an image from our"}, {"start": 1686.78, "end": 1692.8, "text": " data set and we sample epsilon from the like a normal like Gaussian distribution"}, {"start": 1692.8, "end": 1697.64, "text": " right here okay so we're gonna what happens in practice how this model is"}, {"start": 1697.64, "end": 1701.3600000000001, "text": " going to be trained is the following let me let me change the color here we're"}, {"start": 1701.36, "end": 1706.36, "text": " gonna sample X 0 which means we take so we sample let me do it like this oops so"}, {"start": 1706.36, "end": 1712.0, "text": " we sample X 0 here so that means we take an image then we sample T so that"}, {"start": 1712.0, "end": 1718.32, "text": " mean maybe hundred in one single like training pass so we have we have hundred"}, {"start": 1718.32, "end": 1722.9599999999998, "text": " we have a specific image and because we have this specific we have this this"}, {"start": 1722.9599999999998, "end": 1727.52, "text": " cool thing that we got here that means we can kind of jump over we don't have"}, {"start": 1727.52, "end": 1732.76, "text": " to do hundred sampling steps we can just directly calculate the X of t because of"}, {"start": 1732.76, "end": 1738.08, "text": " this formula here so that means now we have X of t okay so now we have X of t"}, {"start": 1738.08, "end": 1743.96, "text": " which means we can feed it here we can feed X of t into our epsilon theta model"}, {"start": 1743.96, "end": 1749.24, "text": " we can also feed the timestamp and how do you feed a scalar you may ask into"}, {"start": 1749.24, "end": 1753.52, "text": " neural network well please remember transformers you're basically what"}, {"start": 1753.52, "end": 1757.48, "text": " happens they're using the same thing as with transformers using these basically"}, {"start": 1757.48, "end": 1763.16, "text": " Fourier basically these sinusoids and the index into a table of precomputed"}, {"start": 1763.16, "end": 1767.56, "text": " sinusoids and they just take a single row of there and they kind of append it"}, {"start": 1767.56, "end": 1772.1200000000001, "text": " and add it to the model that's that's how you convert scalar into into into"}, {"start": 1772.1200000000001, "end": 1775.76, "text": " vector check out my video on the transformer paper if you didn't"}, {"start": 1775.76, "end": 1779.1200000000001, "text": " understand that and now you're just trying to regress this this epsilon"}, {"start": 1779.1200000000001, "end": 1785.1200000000001, "text": " that's it so that's how the whole procedure works and once you train the"}, {"start": 1785.12, "end": 1789.6, "text": " model once you have this epsilon theta model trained how you actually generate"}, {"start": 1789.6, "end": 1795.28, "text": " the mu theta is using this formula which kind of is simple after a couple of"}, {"start": 1795.28, "end": 1798.32, "text": " simple algebraic manipulations you end up with this formula and now you can"}, {"start": 1798.32, "end": 1803.0, "text": " just input X of t input the predicted epsilon theta and you end up with the"}, {"start": 1803.0, "end": 1808.9599999999998, "text": " mu theta which is the mu theta for the X of t minus 1 so that's the the the"}, {"start": 1808.96, "end": 1815.64, "text": " mean from where the less noisy version of the image came from so remember"}, {"start": 1815.64, "end": 1821.8400000000001, "text": " going to our visualization here that means we found a mu up the diffusion"}, {"start": 1821.8400000000001, "end": 1827.6000000000001, "text": " chain where the image was less noisy okay that was my best attempt at trying"}, {"start": 1827.6000000000001, "end": 1832.72, "text": " to make these diffusion models a bit more clear it took me a couple of days"}, {"start": 1832.72, "end": 1837.16, "text": " to wrap my head around this I had to read four papers I had to read a couple"}, {"start": 1837.16, "end": 1841.8400000000001, "text": " of blocks and I'm still until you start and until you try and implement this"}, {"start": 1841.8400000000001, "end": 1845.52, "text": " thing yourself you're gonna miss like out on the details but the high level"}, {"start": 1845.52, "end": 1851.68, "text": " pictures here you have this process of adding noise you have this ability to"}, {"start": 1851.68, "end": 1857.24, "text": " reverse the process because of this nice property that the the actual"}, {"start": 1857.24, "end": 1864.0600000000002, "text": " conditional distribution so the X t condition X t minus 1 sorry the the X t"}, {"start": 1864.06, "end": 1870.32, "text": " minus 1 condition X t is also of the Gaussian form and that makes life a lot"}, {"start": 1870.32, "end": 1875.44, "text": " easier and yeah basically you're trying to figure out what part of the current"}, {"start": 1875.44, "end": 1879.12, "text": " image is the actual noise and that's how you're denoising with every single step"}, {"start": 1879.12, "end": 1883.56, "text": " and finally you end up with the image from your probability distribution"}, {"start": 1883.56, "end": 1887.6, "text": " which is hopefully as close as possible to the actual data distribution if you"}, {"start": 1887.6, "end": 1892.0, "text": " train your model correctly and that was the the hardest part to understand about"}, {"start": 1892.0, "end": 1897.16, "text": " this paper now after you know that now let's let's see a couple more details"}, {"start": 1897.16, "end": 1902.52, "text": " there's some guidance they're using in order to to to be able to not just"}, {"start": 1902.52, "end": 1907.32, "text": " generate unconditional images but take the piece of text like I as I showed you"}, {"start": 1907.32, "end": 1910.66, "text": " here you take a piece of text you condition the model additionally and then"}, {"start": 1910.66, "end": 1916.52, "text": " you generate an image and not just generate an image so here let me let me"}, {"start": 1916.52, "end": 1922.32, "text": " kind of dig in deeper here so for our main experiments we train a 3.5 billion"}, {"start": 1922.32, "end": 1927.32, "text": " parameter text conditional diffusion model at 64 times 64 resolution and"}, {"start": 1927.32, "end": 1931.8799999999999, "text": " another 1.5 billion parameter text conditional up sampling diffusion model"}, {"start": 1931.8799999999999, "end": 1936.36, "text": " to increase the resolution to 256 256 for clip guidance we also train noise"}, {"start": 1936.36, "end": 1940.48, "text": " aware blah blah blah VAT blah blah blah that's not that important what's"}, {"start": 1940.48, "end": 1945.24, "text": " probably worth mentioning here is that it's noise aware which means you have"}, {"start": 1945.24, "end": 1950.64, "text": " you cannot take an off-the-shelf VAT you have to retrain it with noisy images"}, {"start": 1950.64, "end": 1955.6, "text": " noisy samples to get the correct results more important than that how do you"}, {"start": 1955.6, "end": 1959.08, "text": " condition on text and the answer is simple transformers so to condition on"}, {"start": 1959.08, "end": 1963.1200000000001, "text": " the text we first encode it into a sequence of K tokens and feed these"}, {"start": 1963.1200000000001, "end": 1967.28, "text": " tokens into a transformer so then they say the output of the transformer is"}, {"start": 1967.28, "end": 1971.48, "text": " used in two ways they first the final token embedding is used in place of a"}, {"start": 1971.48, "end": 1974.84, "text": " class embedding in the ADM model which is the diffusion model I was just"}, {"start": 1974.84, "end": 1979.4399999999998, "text": " explaining second the last layer of token embeddings a sequence of K feature"}, {"start": 1979.4399999999998, "end": 1983.08, "text": " vectors is separately projected to the dimensionality of each attention layer"}, {"start": 1983.08, "end": 1988.8, "text": " throughout the ADM model basically in this is very architecture specific what"}, {"start": 1988.8, "end": 1995.08, "text": " they do is they take the output of the transform model so again we have a text"}, {"start": 1995.08, "end": 2000.12, "text": " here we have some text here like I don't know whatever some text and then you"}, {"start": 2000.12, "end": 2005.2399999999998, "text": " pass that through a transformer the almighty transformer here so that's T"}, {"start": 2005.2399999999998, "end": 2010.0, "text": " for transformer outcome the feature vectors and you basically use these"}, {"start": 2010.0, "end": 2014.4799999999998, "text": " feature vectors and you input them in a smart way into the diffusion model so"}, {"start": 2014.4799999999998, "end": 2019.36, "text": " this is the diffusion model let me kind of draw a box here so something like"}, {"start": 2019.36, "end": 2025.9599999999998, "text": " this diffusion model you condition D on on these feature vectors and that's the"}, {"start": 2025.96, "end": 2030.16, "text": " final pipeline that's how the conditional thing works so the training itself"}, {"start": 2030.16, "end": 2035.08, "text": " remains the same you still have you still have this diffusion thing you just"}, {"start": 2035.08, "end": 2039.6000000000001, "text": " now you'll at all times you'll have the captions you'll have the associated"}, {"start": 2039.6000000000001, "end": 2042.92, "text": " caption with each of these images you have a caption here you have a caption"}, {"start": 2042.92, "end": 2047.3600000000001, "text": " here which will be the same throughout this whole chain and basically you're"}, {"start": 2047.3600000000001, "end": 2052.8, "text": " gonna use that information additionally and train the the the yeah the the text"}, {"start": 2052.8, "end": 2056.76, "text": " condition version of this of this model okay so what turned out to be the case"}, {"start": 2056.76, "end": 2061.0800000000004, "text": " is that you can you can even further improve the results by having a"}, {"start": 2061.0800000000004, "end": 2067.36, "text": " dedicated classifier basically manipulate your your your your training"}, {"start": 2067.36, "end": 2072.28, "text": " procedure here and let me try and explain how that works so for for a"}, {"start": 2072.28, "end": 2077.4, "text": " moment forget that we are working with text and think of this like amnest we"}, {"start": 2077.4, "end": 2082.8, "text": " are trying to just let's say we have ten classes and we still have passing the"}, {"start": 2082.8, "end": 2086.84, "text": " text through transformer we just take up I don't know like a one hot encoding of"}, {"start": 2086.84, "end": 2092.56, "text": " our class and we input that instead of to condition to condition the model so"}, {"start": 2092.56, "end": 2097.12, "text": " what I do is they take the that they said they basically take the data set"}, {"start": 2097.12, "end": 2101.2000000000003, "text": " and they train a classifier so they have a classifier here like I just you know"}, {"start": 2101.2000000000003, "end": 2103.76, "text": " this see that's gonna be probably some convolutional neural network or"}, {"start": 2103.76, "end": 2109.2400000000002, "text": " something or VAT I know and you take the noisy versions of your sample and you"}, {"start": 2109.2400000000002, "end": 2113.48, "text": " teach this model to predict the class you teach you to predict the correct"}, {"start": 2113.48, "end": 2118.36, "text": " class like maybe it's gonna be C1 like dog or whatnot and now because this"}, {"start": 2118.36, "end": 2124.0400000000004, "text": " model knows how to given an image it can predict the correct class now you can"}, {"start": 2124.0400000000004, "end": 2130.4, "text": " use that model to guide the the training procedure and improve the samples we've"}, {"start": 2130.4, "end": 2134.56, "text": " seen this with clip guided VQ VA models and some other generative models"}, {"start": 2134.56, "end": 2139.96, "text": " basically you can see that all throughout Twitter yeah nowadays but yeah in a"}, {"start": 2139.96, "end": 2144.2000000000003, "text": " nutshell what happens how this is implemented is we are going to remember"}, {"start": 2144.2000000000003, "end": 2147.48, "text": " we're trying to predict the mu we're just gonna have some some scalar"}, {"start": 2147.48, "end": 2152.52, "text": " coefficient here which is basically an amplify the influence of the gradients"}, {"start": 2152.52, "end": 2158.7200000000003, "text": " with respect to the input image of the log of p phi y condition on XT so this"}, {"start": 2158.72, "end": 2163.2, "text": " is looks very complicated but this is basically our classifier so this is"}, {"start": 2163.2, "end": 2166.9199999999996, "text": " basically our classifier and we're trying to so what happens here on a"}, {"start": 2166.9199999999996, "end": 2172.0, "text": " semantic level is that hey we know because we have the gradient we know how"}, {"start": 2172.0, "end": 2179.24, "text": " to tweak X of t how to change the pixels of X of t in order to maximize the the"}, {"start": 2179.24, "end": 2183.08, "text": " the probability of the class so that means so that's something you can see"}, {"start": 2183.08, "end": 2185.9599999999996, "text": " in deep dream you can check out deep dreams we're not familiar with it but"}, {"start": 2185.96, "end": 2189.44, "text": " basically let's say you haven't like let me just kind of draw an image here so"}, {"start": 2189.44, "end": 2194.36, "text": " you have an image here and you'll know how to tweak the particular pixels so"}, {"start": 2194.36, "end": 2199.04, "text": " that a particular class here let's let me denote it as a like a softmax"}, {"start": 2199.04, "end": 2204.12, "text": " distribution or something like discrete one or whatnot so you know this is the"}, {"start": 2204.12, "end": 2209.16, "text": " correct class for example just change the color so you know this is a correct"}, {"start": 2209.16, "end": 2216.3999999999996, "text": " class and what you do is you tweak this so that you kind of boost that particular"}, {"start": 2216.3999999999996, "end": 2221.16, "text": " class that's the whole point that's what gradients are there for and because you"}, {"start": 2221.16, "end": 2226.3199999999997, "text": " have that information you can use that to tweak the mu so if we were to get"}, {"start": 2226.3199999999997, "end": 2233.0, "text": " back to the drawing I showed you up here let me just try and find it here that"}, {"start": 2233.0, "end": 2239.16, "text": " means instead of this time instead of going right so instead of going from from"}, {"start": 2239.16, "end": 2244.48, "text": " from here to here the classifier is smart enough to tell you hey if you want"}, {"start": 2244.48, "end": 2249.16, "text": " to create a class of a dog instead of going here so instead of going from here"}, {"start": 2249.16, "end": 2254.96, "text": " to here it's better that you move a little bit towards here and so this is"}, {"start": 2254.96, "end": 2259.8, "text": " the better path according to the classifier so this thing here it's gonna"}, {"start": 2259.8, "end": 2263.8, "text": " be a better class and yeah I mean that's that's what happened that's what the"}, {"start": 2263.8, "end": 2267.48, "text": " formula is telling us I didn't explain you how the actual formula was derived"}, {"start": 2267.48, "end": 2273.0800000000004, "text": " but yeah that's gonna be enough for now so that's one way of how they are"}, {"start": 2273.0800000000004, "end": 2279.84, "text": " conditioning the guiding the generation process of the for the particular class"}, {"start": 2279.84, "end": 2285.4, "text": " and there is another thing they do is classifier free guidance the idea here"}, {"start": 2285.4, "end": 2289.2400000000002, "text": " is not to have a dedicated classifier because you can imagine it's very"}, {"start": 2289.24, "end": 2293.72, "text": " expensive to train a separate dedicated classifier on the noisy samples this"}, {"start": 2293.72, "end": 2297.3199999999997, "text": " time so you cannot take an off-the-shelf like a model you have to"}, {"start": 2297.3199999999997, "end": 2301.3199999999997, "text": " train it on the noisy data and that's expensive you wanna you want to avoid"}, {"start": 2301.3199999999997, "end": 2307.7999999999997, "text": " that if possible and that's where the classifier free basically part of this"}, {"start": 2307.7999999999997, "end": 2312.02, "text": " text comes from so classifier free we do not want to have a classifier if we"}, {"start": 2312.02, "end": 2316.12, "text": " don't have to so let me let me walk you through this because it's fairly"}, {"start": 2316.12, "end": 2319.92, "text": " important since they got better results using this classifier free approach"}, {"start": 2319.92, "end": 2326.16, "text": " compared to using a clip as the guidance model so they say here instead of"}, {"start": 2326.16, "end": 2330.4, "text": " training a separate classifier model we choose to train an unconditional"}, {"start": 2330.4, "end": 2335.12, "text": " denoising diffusion model P theta of Z the notation is a bit different but bear"}, {"start": 2335.12, "end": 2342.22, "text": " with me yeah parameterized to a score estimator a e theta of ZL this is the"}, {"start": 2342.22, "end": 2346.16, "text": " thing we just saw this is the noise we're trying to to learn together with"}, {"start": 2346.16, "end": 2351.8399999999997, "text": " the conditional model P theta is that condition and C parameterized to the"}, {"start": 2351.8399999999997, "end": 2357.7599999999998, "text": " yeah basically the e theta ZL C model so that's basically our model we're just"}, {"start": 2357.7599999999998, "end": 2361.8799999999997, "text": " additionally conditioning that that model we're learning so we use a single"}, {"start": 2361.8799999999997, "end": 2366.56, "text": " neural network to parameterize both models where for the unconditioned"}, {"start": 2366.56, "end": 2371.2, "text": " model we can simply input zeros for the class identifier C when predicting the"}, {"start": 2371.2, "end": 2376.2799999999997, "text": " score IE we just put the C equals to zero for that model and we jointly train"}, {"start": 2376.2799999999997, "end": 2380.2, "text": " the unconditional and conditional models simply by randomly setting C to the"}, {"start": 2380.2, "end": 2385.48, "text": " unconditional class identifier so we have a like they basically have a same"}, {"start": 2385.48, "end": 2389.3599999999997, "text": " network and sometimes with some probability there they are basically"}, {"start": 2389.3599999999997, "end": 2393.68, "text": " zeroing the class and that's how they're training the unconditional and the"}, {"start": 2393.68, "end": 2399.2799999999997, "text": " conditional like a diffusion model at the same time and finally we then"}, {"start": 2399.28, "end": 2402.48, "text": " perform sampling using the following linear combination of the conditional"}, {"start": 2402.48, "end": 2406.52, "text": " and unconditional score estimates and I'm fairly sure this formula is broken"}, {"start": 2406.52, "end": 2412.2400000000002, "text": " that this here is actually the correct formula so I'm gonna now jump to this"}, {"start": 2412.2400000000002, "end": 2419.52, "text": " paper and it says that in order to get the final estimate we are going to take"}, {"start": 2419.52, "end": 2426.5600000000004, "text": " the the prediction that we got by using the like unconditioned version plus S"}, {"start": 2426.56, "end": 2432.04, "text": " minus the this difference here so what this means geometrically is the"}, {"start": 2432.04, "end": 2438.7599999999998, "text": " following so we have this distinct e theta conditioned on the zero that means"}, {"start": 2438.7599999999998, "end": 2443.84, "text": " it's unconditioned so it says hey if you want to generate a cool denoised image"}, {"start": 2443.84, "end": 2449.08, "text": " go so we are here go in this direction okay so now this thing here the"}, {"start": 2449.08, "end": 2456.24, "text": " difference tells us hey this thing the the the condition model is smarter"}, {"start": 2456.24, "end": 2460.4399999999996, "text": " because it learned how to generate conditioned images so it's gonna say hey"}, {"start": 2460.4399999999996, "end": 2466.2799999999997, "text": " instead of there go here okay and so you can imagine that the difference vector"}, {"start": 2466.2799999999997, "end": 2473.12, "text": " here is gonna be the green thing so we are from this we are subtracting this so"}, {"start": 2473.12, "end": 2478.8399999999997, "text": " that means we're gonna end up with this vector here and so what this is telling"}, {"start": 2478.8399999999997, "end": 2484.16, "text": " you is hey instead of going with the blue or with the red one we're gonna go"}, {"start": 2484.16, "end": 2489.6, "text": " first with the red one so that's the unconditioned generator and then we're"}, {"start": 2489.6, "end": 2495.04, "text": " gonna do so we can now amplify this difference vector by how much we want so"}, {"start": 2495.04, "end": 2500.2, "text": " there may be like two or something that means hey we want to do it like this so"}, {"start": 2500.2, "end": 2504.7999999999997, "text": " go twice there and so basically we end up going going in this direction or"}, {"start": 2504.7999999999997, "end": 2509.56, "text": " something like that so that's the the high-level idea of how this should be"}, {"start": 2509.56, "end": 2513.3199999999997, "text": " supposed to work yeah there is obviously a lot of theory for why this is yeah I"}, {"start": 2513.32, "end": 2517.32, "text": " think I think this was mostly empirical finding somebody correct me if I'm wrong"}, {"start": 2517.32, "end": 2523.1200000000003, "text": " but yeah and now I was constantly talking about like simple class"}, {"start": 2523.1200000000003, "end": 2528.52, "text": " conditioning because remember we were working with text here the version"}, {"start": 2528.52, "end": 2532.92, "text": " they're actually using is like conditioning on captions and this it"}, {"start": 2532.92, "end": 2538.2400000000002, "text": " means just empty empty text sequence and that's that's yeah we sometimes replace"}, {"start": 2538.2400000000002, "end": 2542.48, "text": " text captions with an empty sequence during training so that's how we we get"}, {"start": 2542.48, "end": 2548.64, "text": " into the from the class conditions world into the text condition world okay also"}, {"start": 2548.64, "end": 2555.4, "text": " also here I explained to you how the guided diffusion would work in the case"}, {"start": 2555.4, "end": 2559.2, "text": " where we are conditioning on a class but remember again we're working with text"}, {"start": 2559.2, "end": 2564.04, "text": " so they are going to use clip and they say here they are protruding the reverse"}, {"start": 2564.04, "end": 2569.6, "text": " process mean along the gradient of the text image top product with respect to"}, {"start": 2569.6, "end": 2573.64, "text": " the image so that's here this may be a bit overwhelming perturbing the reverse"}, {"start": 2573.64, "end": 2577.52, "text": " process mean is what I explained up there so we are as you can see we're"}, {"start": 2577.52, "end": 2583.0, "text": " protruding the mean with the gradients of whatever is this and in the case of"}, {"start": 2583.0, "end": 2587.48, "text": " clip it's gonna be we're bringing me along the gradient of the text image dot"}, {"start": 2587.48, "end": 2592.52, "text": " product with respect to the image go ahead and check out my clip video if you"}, {"start": 2592.52, "end": 2596.92, "text": " haven't watched that one if you're not familiar with clip but here briefly what"}, {"start": 2596.92, "end": 2604.08, "text": " clip does is you're basically gonna have like an image here and that's so let me"}, {"start": 2604.08, "end": 2609.7200000000003, "text": " just change the color here so the image and the the texts the the the captions"}, {"start": 2609.7200000000003, "end": 2614.52, "text": " are gonna be embedded in the joint space and so what happens is you have an image"}, {"start": 2614.52, "end": 2619.08, "text": " here and you have some pieces of text here and you're gonna do dot product"}, {"start": 2619.08, "end": 2622.48, "text": " because they're in the same space that means they're like I don't know like"}, {"start": 2622.48, "end": 2624.56, "text": " three-dimensional vector or three-dimensional vector it's gonna"}, {"start": 2624.56, "end": 2628.56, "text": " usually be higher dimensionality but for the sake of argument and you're gonna do"}, {"start": 2628.56, "end": 2633.24, "text": " dot product between them now if you're trying to generate a specific caption"}, {"start": 2633.24, "end": 2638.68, "text": " like the hamster the dragon hamster from from the example I showed you in the"}, {"start": 2638.68, "end": 2643.84, "text": " beginning what you want to do is again so this is an image here you'll want to"}, {"start": 2643.84, "end": 2650.12, "text": " tweak the pixels such that this dot product between these two such that this"}, {"start": 2650.12, "end": 2655.56, "text": " dot product goes up okay and that's that's these you're basically that's"}, {"start": 2655.56, "end": 2659.6, "text": " those are the gradients here that's what this formula is capturing so instead of"}, {"start": 2659.6, "end": 2663.0, "text": " in the case of class conditioning you're trying to maximize just the probability"}, {"start": 2663.0, "end": 2666.88, "text": " of a class here you're trying to maximize the match between the image and"}, {"start": 2666.88, "end": 2670.48, "text": " a particular caption for which you're trying to generate this image so"}, {"start": 2670.48, "end": 2674.12, "text": " remember you're given the caption and you want to generate a specific image and"}, {"start": 2674.12, "end": 2679.68, "text": " so you're using this clip model to learn how to change the image so that you can"}, {"start": 2679.68, "end": 2684.12, "text": " better describe and generate the the corresponding image for this particular"}, {"start": 2684.12, "end": 2689.16, "text": " caption so that's it okay there was a lot of details yeah let me know whether"}, {"start": 2689.16, "end": 2693.3599999999997, "text": " this this format was too too too many details or not but yeah now we're gonna"}, {"start": 2693.3599999999997, "end": 2698.12, "text": " just see the results and that's pretty much it thing worth noticing is that"}, {"start": 2698.12, "end": 2703.16, "text": " they they made sure that the training compute is roughly equal to that used"}, {"start": 2703.16, "end": 2710.24, "text": " to train the Lee so that they can fairly compare both models and you can read"}, {"start": 2710.24, "end": 2715.04, "text": " this part yourself if you're if you're curious to know more I basically this"}, {"start": 2715.04, "end": 2719.7999999999997, "text": " day basically describe how the how the model is fine-tuned for the image"}, {"start": 2719.7999999999997, "end": 2724.3999999999996, "text": " painting task in painting task and how some additional details about the"}, {"start": 2724.3999999999996, "end": 2731.3199999999997, "text": " guidance procedure here they compare glide using class class classifier free"}, {"start": 2731.32, "end": 2736.44, "text": " guidance clip guidance dali some GAN versions and the real images you can see"}, {"start": 2736.44, "end": 2743.1800000000003, "text": " the results and let me just kind of maybe focus on this column here and you"}, {"start": 2743.1800000000003, "end": 2748.04, "text": " can see that using classifier free guidance is creates better images"}, {"start": 2748.04, "end": 2754.76, "text": " arguably compared to the clip guidance and better than then the Lee although I"}, {"start": 2754.76, "end": 2758.28, "text": " took a single column buddy I think we can agree that a group of elephants"}, {"start": 2758.28, "end": 2762.6000000000004, "text": " walking in a muddy water this also looks better than this one that's for sure"}, {"start": 2762.6000000000004, "end": 2767.6000000000004, "text": " because it's kind of like a blurry and that has to do with the fact that they"}, {"start": 2767.6000000000004, "end": 2771.84, "text": " are using in the background a discretized version of the VAE model"}, {"start": 2771.84, "end": 2775.0, "text": " that's how the Lee works you can again check my video if you're not familiar"}, {"start": 2775.0, "end": 2780.96, "text": " with the Lee so those are some some some comparison results here what they've"}, {"start": 2780.96, "end": 2784.2200000000003, "text": " done is they've compared I'm gonna skip these curves but basically they"}, {"start": 2784.22, "end": 2790.3599999999997, "text": " compared how classifier free guidance compares to clip guidance and it turns"}, {"start": 2790.3599999999997, "end": 2795.9199999999996, "text": " out that clip classifier free is also better looking at these metrics except"}, {"start": 2795.9199999999996, "end": 2801.08, "text": " for this one where the one of the metrics was actually clip score so what"}, {"start": 2801.08, "end": 2806.52, "text": " happened is that basically the the adversarial examples made it look like"}, {"start": 2806.52, "end": 2810.48, "text": " the the clip guidance is better but it's actually not the classifier free guidance"}, {"start": 2810.48, "end": 2813.7999999999997, "text": " is better so they say here we hypothesize that clip guidance is finding"}, {"start": 2813.8, "end": 2818.0, "text": " adversarial examples for the evaluation clip model rather than actually a"}, {"start": 2818.0, "end": 2821.76, "text": " performing classifier free guidance when it comes to matching the prompt I guess"}, {"start": 2821.76, "end": 2825.88, "text": " that would be my first hypothesis as well for why this thing happens you can"}, {"start": 2825.88, "end": 2832.8, "text": " see here like the clip guidance model has for the higher score it has lower FID"}, {"start": 2832.8, "end": 2838.36, "text": " which means basically that if lower FID is better so that means that this curve"}, {"start": 2838.36, "end": 2843.5, "text": " the orange curve in this particular case is is the better one okay so they did"}, {"start": 2843.5, "end": 2851.44, "text": " some comparisons with the Lee model they they tried really hard to make glide as"}, {"start": 2851.44, "end": 2856.0, "text": " bad as possible and the Lee is as good as possible and still they got always"}, {"start": 2856.0, "end": 2860.4, "text": " better results you can see here everything above 50% means that glide is"}, {"start": 2860.4, "end": 2863.76, "text": " better than the Lee and that's the case as you can see across all of the columns"}, {"start": 2863.76, "end": 2870.24, "text": " and all of the rows so this row means that they're using clip to rerank the"}, {"start": 2870.24, "end": 2875.3199999999997, "text": " output images of the Lee and they use I think like 512 or something temps from"}, {"start": 2875.3199999999997, "end": 2878.7599999999998, "text": " the Lee and then they take the best one using clip models so the clip model is"}, {"start": 2878.7599999999998, "end": 2882.4799999999996, "text": " ranking the images for a particular caption and that's how they improve from"}, {"start": 2882.4799999999996, "end": 2887.58, "text": " no ranking to the Lee we ranked and then here what they do is they just blur the"}, {"start": 2887.58, "end": 2892.4799999999996, "text": " output of glide by passing it through the VQ VAE model if I'm not wrong from"}, {"start": 2892.4799999999996, "end": 2897.24, "text": " the Dali and that's make that makes it blur and that's why we saw up there we"}, {"start": 2897.24, "end": 2901.4799999999996, "text": " saw up there that this Dali image here is way more blurred compared to the"}, {"start": 2901.4799999999996, "end": 2907.08, "text": " output of glide so then they pass the glide image through that same model that"}, {"start": 2907.08, "end": 2910.52, "text": " caused this and they still compare them and it's still better compared to the"}, {"start": 2910.52, "end": 2914.4799999999996, "text": " Lee so that's very very encouraging result okay finally some safety"}, {"start": 2914.4799999999996, "end": 2919.56, "text": " considerations a section they as I said they open source the smaller version of"}, {"start": 2919.56, "end": 2923.2, "text": " a model that was trained in this filter data set so they say here we filtered"}, {"start": 2923.2, "end": 2927.6, "text": " out training images containing people to reduce the capabilities of the modeling"}, {"start": 2927.6, "end": 2931.3199999999997, "text": " many people centric problematic use cases okay that's one of the attempts"}, {"start": 2931.3199999999997, "end": 2936.3999999999996, "text": " they try to do so it's a 300 million it's a way smaller model remember that"}, {"start": 2936.3999999999996, "end": 2943.48, "text": " the big one is like in the billions of parameters yeah ballpark and they say"}, {"start": 2943.48, "end": 2947.16, "text": " here that we also probed glide the filtered version so the small filtered"}, {"start": 2947.16, "end": 2951.24, "text": " version for some forms of bias and found that it retains and may even amplify"}, {"start": 2951.24, "end": 2955.52, "text": " biases in the data set for example when asked to generate toys for girls our"}, {"start": 2955.52, "end": 2959.8399999999997, "text": " model produces more pink toys and stuffed animals than it does for the"}, {"start": 2959.8399999999997, "end": 2966.2799999999997, "text": " prompt toys for boys I mean this thing is kind of funny like I mean of course"}, {"start": 2966.2799999999997, "end": 2972.3999999999996, "text": " that that's gonna happen because the way our society works is that girls are"}, {"start": 2972.3999999999996, "end": 2978.4799999999996, "text": " playing with more pink toys when they are young I mean this kind of seems"}, {"start": 2978.48, "end": 2982.2, "text": " weird to me the model is just reproducing whatever is present in the"}, {"start": 2982.2, "end": 2985.88, "text": " data set I mean we can all agree on this so yeah let me know your thoughts on"}, {"start": 2985.88, "end": 2989.2, "text": " this particular statement here I'd love to hear your thoughts on this and"}, {"start": 2989.2, "end": 2994.32, "text": " finally some some failure cases and illustration of a cat that has eight"}, {"start": 2994.32, "end": 3001.84, "text": " legs yeah you can see it failed I mean maybe the model here thought that these"}, {"start": 3001.84, "end": 3006.72, "text": " are four legs here and then there are two tails so this is a Chernobyl cat or"}, {"start": 3006.72, "end": 3011.3199999999997, "text": " something so that's six and then these ears have some features that look like"}, {"start": 3011.3199999999997, "end": 3015.9199999999996, "text": " like maybe like look like like so that's like eight here but yeah for this one"}, {"start": 3015.9199999999996, "end": 3021.48, "text": " that that line of reasoning does not work yeah bunch of failure cases here"}, {"start": 3021.48, "end": 3027.16, "text": " it's kind of interesting like a mouse hunting a lion I'd agree you maybe maybe"}, {"start": 3027.16, "end": 3032.52, "text": " here this mouse is actually hunting this lion that may be the case also here"}, {"start": 3032.52, "end": 3036.92, "text": " maybe the the line is dead and the mouse is actually eating the lion but it just"}, {"start": 3036.92, "end": 3040.7599999999998, "text": " yeah went into the mouth to just check double check some of the I don't know"}, {"start": 3040.7599999999998, "end": 3045.32, "text": " some of the markers so that the line is healthy or something I don't know a car"}, {"start": 3045.32, "end": 3049.28, "text": " with triangular wheels here this is I guess the best attempt that the model"}, {"start": 3049.28, "end": 3056.64, "text": " got and this looks like something that may actually yeah appear in a world in"}, {"start": 3056.64, "end": 3063.2, "text": " real world data sets but yeah it's not perfect and I found some other failure"}, {"start": 3063.2, "end": 3068.64, "text": " cases down in the appendix like let me just kind of try and find it here so the"}, {"start": 3068.64, "end": 3073.7999999999997, "text": " fine-tune model they say golden necklace they do this this impeding you can see"}, {"start": 3073.7999999999997, "end": 3077.16, "text": " here they put a mask here and you can see what the model the model doesn't"}, {"start": 3077.16, "end": 3080.8799999999997, "text": " even produce the necklace so there are some failure cases obviously it's not"}, {"start": 3080.88, "end": 3088.04, "text": " perfect also I mentioned the the fact that the Lee was failing with these like"}, {"start": 3088.04, "end": 3092.48, "text": " relational reasoning types of tasks and here they just put a simple one so I"}, {"start": 3092.48, "end": 3096.56, "text": " guess they really cherry picked the best out of this model and we cannot test it"}, {"start": 3096.56, "end": 3101.1600000000003, "text": " ourselves so that's kind of that kind of sucks but other than that this is fairly"}, {"start": 3101.1600000000003, "end": 3106.6, "text": " fairly awesome work I love the fact that they are pushing diffusion models I'm"}, {"start": 3106.6, "end": 3111.62, "text": " not sure whether diffusion models are ever gonna be as fast as as GANs or V"}, {"start": 3111.62, "end": 3116.68, "text": " ease when sampling for this very simple reason of the fact that the whole"}, {"start": 3116.68, "end": 3122.7999999999997, "text": " diffusion model thing rests on this fact so this distribution approaches the"}, {"start": 3122.7999999999997, "end": 3128.6, "text": " Gaussian only when you have a bunch of steps and when you have beta t going to"}, {"start": 3128.6, "end": 3132.2799999999997, "text": " zero if you manage to find a solution around this it's not going to be"}, {"start": 3132.28, "end": 3136.84, "text": " diffusion model anymore or it's gonna be something different yeah it's not the"}, {"start": 3136.84, "end": 3142.1600000000003, "text": " diffusion anymore I guess in any case hopefully you found this video useful"}, {"start": 3142.1600000000003, "end": 3146.0800000000004, "text": " again I still don't have a camera hopefully in a couple of next weeks I'm"}, {"start": 3146.0800000000004, "end": 3150.6400000000003, "text": " gonna get it and let me know if you found this video yeah interesting useful"}, {"start": 3150.6400000000003, "end": 3154.5600000000004, "text": " and leave some comments feedback and yeah subscribe to this channel share it"}, {"start": 3154.56, "end": 3171.84, "text": " with friends and until next time bye bye"}]