Get Started With Stable Diffusion! (Code, HF Spaces, Diffusers Notebooks)
🚀 Sign up for Cohere using my link and get 150$ credits 🚀
https://os.cohere.ai/register?utm_source=influencer&utm_medium=&utm_campaign=ALEKSA

👨‍👩‍👧‍👦 And check out Cohere's Discord community! 👨‍👩‍👧‍👦
https://discord.gg/co-mmunity

In this video I show you 3 ways to get started with Stable diffusion:
1. Using HuggingFace Spaces (super slow, but super easy)
2. Using diffusers Colab notebooks (mid-ground)
3. Running it locally (my code, most control, and flexibility)

Watch out for my deep dive into stable diffusion if you want to learn how it works behind the scenes! (coming soon)

Update: deep dive into Stable Diffusion video is out: https://www.youtube.com/watch?v=f6PtJKdey8E

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ My code: https://github.com/gordicaleksa/stable_diffusion_playground
✅ My older GAN projects: https://github.com/gordicaleksa/pytorch-GANs

✅ Diffusers notebook: https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_diffusion.ipynb
✅ HF Space: https://huggingface.co/spaces/stabilityai/stable-diffusion

Misc:
✅ Xander's awesome video: https://twitter.com/xsteenbrugge/status/1558508866463219712
✅ Model weights: https://huggingface.co/CompVis/stable-diffusion-v1-4
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
00:55 (sponsored) - Cohere's NLP toolkit
03:35 First option: HF Spaces
05:30 Second option: Diffusers Colab notebooks
15:10 Third option (more control): my code
16:10 Setup
20:40 How to use the script to generate images
28:00 How to reproduce the image from metadata
31:37 Patching the diffusers lib
33:45 How to interpolate between images
38:10 Results
39:20 Outro (share your images on Twitter/LI!)

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#stablediffusion #generativeart #coding
 What's up guys, Alex here. In this video, I'm showing you how to get started with stable diffusion. I have another video where I do a super deep dive into how everything works. I take the original code base and I literally step through the code and explain you how the training works, how the sampling works, how to generate images, how to have complete power over your generation basically. So if you're curious to learn more about that, I'm gonna link it somewhere down in the video description once the video is published. So in this video, I show you three ways to get started with stable diffusion. The first one being using the hugging face spaces, the second one being using diffusers notebooks, and the third one is using my code I developed inspired by Karpathis recent gist and basically you can see some of the images I managed to generate with that code base. Stick around to the second part of the video where I literally walk you through how to exactly get started and set it up on your own machine. But before we go there, I wanna thank Cohere for sponsoring this video. They're basically an amazing way to get started building NLP applications. Getting started is as easy as just doing pip install Cohere as you can see here and they also have a very cool no code prototyping platform where you can just explore what features they offer and then get started very easily. So here's an example of generation where you can do for example, hashtag generation. They also support embeddings, classifier stuff and getting started is like super easy. You just do export code, you copy paste the snippet here and you additionally just have to create an API key which you can do in the dashboard here. So it literally takes a couple of minutes to get started. If you wanna learn more about Cohere, I strongly suggest checking out some of the blogs where they use Cohere to generate super cool applications. So this one by Jay Elmer, the same author who kind of famous Illustrator transformer blog basically walks you through how you can use Cohere to generate insights from various hacker news posts and there is also a very nice web app that goes behind it. Finally, the documentation is amazing. You can learn just about the machine learning as much as you can learn about Cohere product. So I really like that type of contribution. By signing up by using the link down below, you can get started with $75 and that's enough in their own words to generate like the whole works of Shakespeare for free. So yeah, check them out, support the channel and now let's get back to the video. Having said that you must have seen the news, basically a week ago, the stable diffusion weights were published and now you can generate awesome images such as these ones here. And what you can probably notice immediately is that with stable diffusion, you can generate human faces, you have much more control and the model is less constrained than the other alternatives, I think, to the best of my knowledge. And basically you can run it much faster and you can even run it on your own machine even if you have only eight gigabytes of VRAM. I'm gonna show you how to do that in a couple of minutes. People have already been using a stable diffusion to generate very cool art. So here is an example you must have seen from Twitter where Xander generated this video called Voyage Through Time using various prompts and prompt engineering and tweaking the seeds. You can see, just yeah, check out the video at your own pace, it's an amazing piece of art. So you can see how he managed to generate this video by basically tweaking the prompts, creating a narrative, tweaking the seeds and then stitching all the images together. Okay, so having said that, how do you get started with stable diffusion and how do you generate your own images? So the easiest way you can do is basically use this hugging faces like a space. So this is literally like a web application. You just come here, you don't need to know anything about how stuff works, you just enter a prompt. So I literally just entered this prompt a couple of minutes ago, a horse with glasses. I hit generate image and a couple of minutes later, you get this as the output. So a couple of minutes later statement is the problem. So you literally have to wait a lot depending on the current load on the server. So basically that's the main drawback of this approach. You do have some control, some advanced options you can pick how many images you want, you can pick how many steps of the diffusion process you want. As a general rule of thumb, the more you have, the better the quality. They kind of kept it to 50 here. So you should probably stick it at 50. Well, but if you obviously want the results faster then you need to reduce it. But 50 is kind of recommended value. Then the guidance scale basically is a trade off between how high quality the images, how closely is it following your prompt versus how diverse the inputs are. So like usually people use 7.5 or between three and 10. And basically the more you increase the scale, the higher quality the images will be, they will closely follow the prompt, but you'll get less diverse set of output results. And finally, the seed is also just enables you to control the diversity of your sample. That's it, that's the simplest approach, but it takes a lot of time. So if we try to input something like an AI robot having an epiphany moment, and if I hit generate image, you can see that it will take at least, so 152 seconds current estimated time. So it's two and a half minutes. So if you're kind of a patient to wait, then go ahead. The second best approach is just use the colab. So there is a lot of cool colabs. I linked directly from the diffusers library, which is Hugging Face's new library that basically is trying to get the latest and greatest diffusion models available to general public. So the first thing we need to do here is to connect to the GPU. It requires a lot of RAM, so you just kind of acknowledge that, okay. You make sure that you get the GPU and you make sure by either just checking it here so we can see it here, basically we have GPU enabled. So that's one way to figure out that you have the GPU. The second one is just hit here and then you have this part change runtime type. You can see that the GPU is set as the hardware accelerator by default. Okay, so let's kind of run this one. Just hit run anyway, and you get some output here. The main point is if it's working, that means you have a GPU. Okay, so now we're gonna install some libraries such as diffusers, transformers, SciPy, and some other libraries for processing text. And after that, we'll have to log in to be able to use the model weights. So you basically have to accept an agreement before you use this notebook. So I'm gonna hit run here and then run here and it will ask for credentials. So basically what I have to do is go to this page and then basically just copy the token. You need to have an account and just log in. If you don't have, just create an account on Hogan face and then you can just kind of copy paste that information and you will be able to log in successfully, okay? So that's it. So now the first cell, what it does is basically creates, instantiates this stable diffusion pipeline. You can see we're using this model V14. I'm gonna quickly show you what it means. So you can see here on this link, basically that V14 says it's V12 plus 225,000 steps at certain resolution on lion aesthetics data set, blah, blah, blah. So that means that's pretty much the best checkpoint. So that one, you can use either that one or V13. You can kind of play if you want with both. So having said that, let's go back to the notebook. This thing is still loading. The second part you have to notice here is that they're using FP16 because basically if we were to run in full precision, that's FP32, you would probably get CUDA out of memory exceptions, even on a colab that has 16 gigabytes of VRAM. So I did play and set, like just delete this part here so you can just delete these two, but I was getting out of memory exceptions if I'm using 512, 512 resolution. So probably the best thing is to just leave it FP16. Later, once I show you the script that you can run on your own machine, if you have a better GPU, then it's recommended to just remove this part and then you'll have even higher quality generated images. Okay, so the final argument here is just the authentication token that's just needed for us to load the weights and that's pretty much it. Okay, once that's loaded, I'm just gonna hit this cell here. It's going to push the loaded weights onto the GPU and now we can start generating the images. So here you can see basically, let me zoom in a little bit here and remove this part here. So we can specify a prompt here, a photograph of an astronaut riding a horse, and then you basically pass the prompt to the pipe and you grab the sample, the zero sample, which will be the image. Then we can save it and display the image and you can see the astronaut here. Okay, so I'm gonna run this now and let's see what we get. So we probably won't have the same image as the one that was there and you will see how long it takes to generate an image. So currently, this is the speed. I think we have 50 steps by default. So now we're at 28, we need to wait a bit more and then we'll get the image out. There is a lot more parameters you can tune. Again, I super encourage you to check out my deep dive if you wanna really understand how to delicately control stable diffusion, that's gonna be probably your best bet. Okay, so we see an image here and the second thing you can do, if you kind of, so if I were to hit Shift Enter again here and generate yet another image, because we didn't set any seeds, because of that, we'll basically generate a completely different image every time we run the cell. So if you don't like that behavior, if you want to have consistency between the runs, then you have to do a small trick, so you can see here a different image. You have to do a small trick and that's to basically set the seed here of your torch generator and then pass that generator to the pipe. So here in this example, if I were to run this, every time you rerun the cell, you're gonna get the same image output, assuming that you're not changing the prompt obviously. So that's the idea with the generator part. So in the background, what happens is, it's going to be used to generate like a specific latent, which is just a noise vector and every time you run it because you've passed the generator, you'll end up with the same noise vector and because the noise vector is the same, the output image is gonna be the same. So here you can see the results are consistent with what we had before I rerun the cell. Okay, cool. Again, you can control the number of inference steps. I already mentioned in the previous here, like in the space here that 50 is probably recommended. You don't wanna go lower than that. You can also see the images that were generated, which are fairly cool. But yeah, so here we have 15 and that's why we get, you can see here much lower quality image compared to the one above. That's pretty much it. So here you can check out this notebook at your own pace. This just creates some utility functions where you can generate multiple images in a grid. Same here, but like the logic remains the same. You can also play with the height and width parameters, whereby, well, you can get a non-square type of an image and there are some rules, like you need to have multiples of eight for your image width and height and you probably don't wanna go lower than 512. Okay, so then the rest of the notebook is a deep dive into how the thing works. But I already told you I have a video that's almost two hours long explaining exactly how stable diffusion is trained, how sampling works, how everything works. So do check out that one. Okay, there's a couple more notebooks. I'm gonna exit here. There's a couple more cool notebooks you can check out. So this one, image to image pipeline, basically enables you to start from a particular image and then using a prompt, guide that image towards a certain, basically appearance. So you can see here that if we take the prompt, a fantasy landscape trending on ArtStation, you get this output here. Okay, so I think worth mentioning here is that you can see, like probably, if you've never seen stable diffusion before, this part might be kinda confusing, like why would you put the trending on ArtStation? And that's the whole art of basically prompt engineering where you find certain words, certain sentences that trigger the model to generate better output or output with specific properties. So people kinda, we globally as a community try a lot of prompts and then somebody figures out a good prompt and then they share it and then the community learns and then we're slowly coevolving with the model in a way. I'm gonna leave you to explore this notebook at your own pace, but those are the basic notebooks you have at your disposal. There is this one as well where you can control exactly the type of a seed you're using. So what I mean by that is the following. So here, basically they generate four latents. So those are those types of noise vectors I mentioned in this cell here and they generate four images here. And then basically, if you like one of them, you can pick that same latent and you can reconstruct the image or generate like use different prompts and have a similar type of appearance in the image. Let me show you what I mean by that. So if we just pick this same latent, you can see here. So they pick seed one because we have four seeds. This corresponds to this image here. Whoops. So it corresponds to this image here, to this one here. So if we pick that same seed and regenerate the latent, you can see we get the same image out. But if you now change the prompt, but reuse the same latent, you can see that you get a similar posture to this dog here, but just a different breed. Similarly here, Labrador in the style of Van Gogh. And you can see that you have some features of the output image, some like course level structures in this particular example are very similar. So namely the posture. And you can even go wild and use like, you don't have to use a dog. You can be as creative as you want. Okay, so that's everything that's at your disposal. Basically, that's the two easiest methods you can use is use the space or just use the notebooks to generate images. But if you want more control, if you want more speed, if you have better hardware locally, I strongly recommend you do the third option. And that's to use the script I generated. And now I'm gonna walk you through the process of how you can set it up and get started generating your own images on your own machine. And also do some fancy things like interpolation, etc. The script itself was inspired by Carpathi script. So this is this gist he made basically using spherical interpolation to generate basically a video such as this one, or similar videos. So you basically pick two images, and then you smoothly interpolate in the latent space of the diffusion model, you do interpolation, and then you get the corresponding images out. And you can see how they are slowly morphing from the source image into the target image. Okay, that's the basic idea. Okay, guys, let's see how we can go through this notebook, how we can set up this on our local machine. So here's what you need to do. So I'm gonna assume, so I do have basically a Windows machine, but it should be similar no matter whether you're using Linux or Mac OS or whatnot. So first thing you need to do is basically find a place where you wanna store this. So I'm gonna randomly create a new folder here on desktop, and name it stable diffusion playground. And I'm going to open it up. I'm going to open up a git bash here. So basically you want to clone this particular repo into this directory. So you can do git clone, and then just copy the URL you copied from the website here, from the GitHub. Okay, so that's one way. You can also just do it here. You should have the option to just download, but people usually use console to do this. So if you hit clone, you're going to clone, as you can see here, the repository. Now let's follow the second step. So you can just follow the steps here. Everything should be self-explanatory. Okay, so now just navigate with your conda prompt to the, so navigate with the anaconda to basically this directory. So let's do cd to that directory, and we're here. And basically the second step, the third step is to just do, as you can see here, conda and create. And that's going to automatically create basically a new environment for you. So because I already have an environment, I'm just going to basically do rename it, and then I'm going to rerun it here. So conda and create, and this is going to create a new environment from scratch. So I'm just doing this to make sure that if I'm not hitting any errors, then probably most other people using Windows will not. So hopefully others as well. If you notice any problems, feel free to submit an issue here. I'll be tracking issues over the next couple of days. I mean, the script is fairly simple, so hopefully it's going to be useful and easy to set up. Okay guys, so here it is, the script executed. I can just do the conda activate, blah, blah, blah. We enter the SD playground. And then the next step is to, you will have to run Hugging Face CLI login the first time before you start running the script, because otherwise you will not be able to access the model weights. So this is everything you would do. Basically copy this command, paste it here, execute it, and then it will probably ask you to again, copy that token from the website. You can just copy paste it here, and everything is going to work as expected. Okay guys, so now let's open up an ID of your choice. I'm going to open up VS Code. That's my favorite ID by far. So let me try and open up the corresponding directory. So we just open up this one, I guess. And I just trust the author, blah, blah, blah. I say trust the author because the author, I am the author, man. What's up with this? Okay, so generate images. Here is the main script. I'm going to walk you through it right now. Before that, let me just create, basically, let me just create a JSON, long JSON for the Python file. You don't even need to do this because we will not be passing any arguments through the long JSON. This is where you would usually set the arguments in case you need them. But here I don't need them, so I'm just going to leave them as empty here, okay? So this is a script. Basically, I grabbed this function from Karpathy. Basically, I already previously implemented it when I was developing my GAN projects. You can check out my code there, but this looks neater and supports Torch and NumPy conversion, so I just kind of copy-pasted his version. But it's a fairly similar thing to implement. You can just take a look at the Wiki page and copy-paste everything to your, I mean, translate from human description to code. Fairly trivial. Okay, so I'm going to set a breakpoint here. I'm going to pick a corresponding interpreter we just created, so st-playground-2. So you pick the playground, so we pick the environment we just created. And that should make us, that should make this work out of the box. Sometimes you have to restart VS Code or whatever ID you use in order for the changes to be, and for the interpreter to be correctly loaded. But let's see whether now it's going to crash or not. So I'm going to basically just run, hit run here, and hopefully if everything has worked correctly, I'm going to just hit this breakpoint and everything should work. Okay, let's see whether that's the case, and it is. Awesome. So I'm going to now walk you through the script. It's fairly simple. The first thing I will do is actually I'm going to exit. I'm going to exit the script. I'm going to set generate diverse here. So generate diverse is going to start generating diverse set of images with the prompt you specify. So here is the prompt I've specified, a painting of an AI robot having an epiphany moment. I'm going to save that to the output directory AI epiphany. I'm going to use 50 steps, 7.5 guidance scale, and I'm going to generate, let's say only one image. For the sake of this video, I don't want to make it too slow. And everything else you can kind of leave. You can even put none here. The width and height are 512. I'm using FP16 because my machine only has eight gigabytes of VRAM, and I can basically, I'm going to set none here. This is, I'm going to explain you briefly what this is. Basically, it helps you with reproducibility of this script, but we don't need it now. So I'm just going to leave all of this as none. So I'm going to put a comma there, and that's pretty much it. Okay, so let's now rerun the script in the generate mode. So I'm going to hit run here, and let's see what's going to happen. Okay, so first things first, you do need the GPU to run this, otherwise it's going to take like painfully long time, and it's better for you to use the hugging face space than to wait here. Yeah, so basically, first thing first is we want to make sure that the height and width are a multiply of eight. Okay, so device is CUDA, we just set the seed here such that every time if I rerun this, I'll have the same latent generated. If you don't want that, you can just set seed to none, then every time you rerun this, you'll get a completely new set of images. Okay, so I do some logistics here. Basically, I create a specific output file structure such that we are generating images in a suitable location. I'm going to show you what I mean by that in a second. So let me do it like this. So here is the directory, and that's where we'll be generating. You can see I generated these samples, so that's where the images will be stored. We have meta, that's where the metadata that we use to generate the image will be stored. And finally latent, that's going to be the actual latent representation that was fed into the pipeline to generate the image. Okay, let's create the scheduler. Again, if you don't know what this is, I suggest you watch my deep dive, then everything will be clear. So we pick again the B14 version. We pick because FP16 is set to true, we're going to use the FP16 weights. We passed basically the scheduler, and we finally have the pipe. So we'll take some time for the pipe to be constructed because it has to load a bunch of weights. I'm going to show you in a second, let me show you how this is going to function. Basically performance tab, and you can see how the memory is slowly increasing. And after I execute the two device, you can see how the GPU will start, the GPU consumption, the VRAM consumption will increase. That's going to hopefully happen very soon. So let's take a look. So I'm going to do F10 again here, F10. And now you can see how the pipe was basically pushed onto the GPU. You can see a spike here in the VRAM consumption. Okay guys, so let's ignore that now. I'm going to maximize this window again. So we enter the branch with the generation. And because I've set the number of images to one, we'll have only a single image generated. So let's see what we do. The first step is we generate the latent. This is kind of hard coded because that's how the, this is what pipe is also expecting. Like in general, you don't want to hard code the dimensions like this, but in this case, it just works. Okay, we generate the latent. And now I use the AutoCast, which is going to use Mixed Precision. And in my particular case, it's just going to do, I guess, FP16 pretty much for all computations. I'm not sure whether it's going to do FP32 accumulation for some of the operations, but basically that's it. Okay, so now we run the pipe. We pass the prompt, we pass the number of steps, the latent and the guidance scale. And out comes the image. So as you can see so far, so good. We didn't have any errors with the script. Again, I encourage you, if you find any errors, basically submit an issue to my GitHub repo. That would be super useful. Okay, let's continue here. Here is the image. We now just basically store the image. So let's see how that's going to happen. So we have samples here. After I run this, we have this neat generate name function that's going to automatically detect all of the images inside of this folder and make sure not to overwrite the existing images. Instead, it's gonna increment the file name and just store it there. So I'm gonna show you the script a bit later, but for now, let's treat it as a black box. So here is the image. I'm gonna put extra large icons so you can see the results. So here is the first result, amazing result. Next up, I'm gonna save the metadata. So when I say metadata, I mean, let's save the prompt information, how many steps we used to generate this image and what guidance scale we used to generate this image. I'm gonna improve this script over the next days. So feel free to contribute, feel free to create a pull request. I'm going to be tracking what's going on with the repo over the next days. So let me open up the meta directory here, and I'm expecting to generate a meta file there. So if I hit F10, we can see here there is a JSON. If I open it up, you can see that's basically storing the prompt information, the number of steps, and everything else. I kinda have, wait, let me move this thingy. Let me try and move this thingy. Okay, so we have the scale information there. Okay, so basically that's it. Now, finally, I store the latents. So that's the latent that was used to generate this image. So here's the latent, I just push it onto the CPU, I convert it into NumPy, and then I store it here. So that's it. Let's hit F10. Here is the latent, and okay, now I can step and exit the script. So that's it. That was how the script looks like in this generation mode. There is a couple of different modes. The second one you can use is reproduce mode. So here, and this is more for playing and understanding the reproducibility than something that will help you with your art generation. So what this does is basically now it expects me to pass, as you can see here, so now it expects me to pass the source latent path and the metadata path. And using the latent and the metadata, it's going to regenerate the same image from scratch. Okay, so that's the idea. So let's do that. I'm going to grab this latent. I'm gonna just copy this path here. I'm going to put that path here. So here it is. I'm just gonna do it like this. Okay, comment this out. I probably have, this sucks on Windows. I'm not sure whether, oh my God, am I doing this really live? Okay, so let me put the backslash here so that this actually works. Maybe it can work even without it, but yeah. Okay, now I'm gonna copy the metadata path as well. So just, oops, shift, right click, copy as path, and I'm gonna specify that thing here. Okay, and that should be pretty much it. Now I just have to, well, let's see whether this is gonna crash or not. I'm curious, to be honest. So this should now work. I have reproduce mode and we can just click save here and let me run this script again. So if I run this, okay guys, I did have to add the additional backslash after all, otherwise it's gonna make me, it's gonna complain. Okay, so again, same logic here. I'm gonna skip over all of this. We generate the pipe, blah, blah, blah. So let me just put the breakpoint here and let's hit F5 and let's get to this point in the code. So again, you'll see how we're gonna reuse this particular metadata here and this particular latent here to generate the very same image that we saw here. And additionally, I have a math plot live plot function being called, so this is gonna pop up after this script is done with the execution. Okay, let's see whether we are there. Here we are, we hit this breakpoint. So I make sure that we are passing the needed paths. So I open up the metadata. You can see that the metadata contains exactly the information that was used to generate the image a couple of minutes ago. So I load the latent here and now we end up, let me show you the shape of that thingy. So init.shape, we end up with 464.64. So that's the expected size of the latent space for the diffusion model for this particular one. And now let's pass the same metadata, that latent, and let's generate a NumPy image. So that's why I put output NumPy output type to NumPy, basically can be anything because the way how a pipe is implemented in the background, you just need to specify anything other than then pill and then it will generate a NumPy image as the output. Okay, so here we are. Finally, let's generate the image. So image show, I'm gonna show you the image we generated here and you can see that, okay guys, so that failed because there was a very sneaky bug behind this code. Not behind my code, but behind the version of the diffusers library I'm using. So namely, if you take a look at the main branch, the latest commit in this pipeline stable diffusion file, so that's basically the call function behind the stable diffusion pipeline object, you can see that there is this lines 102 to 113. You can see that the latent is being used here. So if we pass the latent, then basically, we are going to use it. Otherwise, if we pass none, then we generate it on the fly and there is a latent as well here as the argument to this call function. Okay, on the other side, I had to manually add the latent here and I had to add this piece of code here. So let me go back. So I had to copy paste that part and I had to copy paste that part and now after adding that, manually patching the 0.2.4 version of the library I currently have and that's the latest pip version they have. So they still haven't pushed the latest changes into that pip package. So after you do this fix, now everything works. So basically, I went ahead and generated another image. So let me show you what I mean by that. So I went ahead and generated this image here and now if I pick its metadata, so we have its metadata 0.0002, same for the latent. So if I go and specify that here, 0002, and if I just set the reproduce mode, if I hit run, now I'm gonna let it execute and I'm gonna show you the results as soon as we get to this break point here. Okay guys, the moment of truth. Let's basically display the image and you can see that's exactly the same image as what we had here. So that's the same image as what we had here. Let me do a side by side comparison and we have perfect reproducibility of this thingy. Okay, that's very cool. The last thing I wanna show you is the interpolation execution mode. So that's the funny part where we basically, so I'm gonna do interpolate here and here in that case, there is a couple of options. So that's the third branch of the code here. Basically, if you don't specify the paths, then we just generate two random latent vectors. Those two latent vectors correspond to two images and then we're gonna smoothly, spherically interpolate between those images and generate everything in between. So that's if you don't specify the latency. If you do specify the latency, if you find two images that you like, just find basically the numpy, the associated numpy, the latents and pass the paths here and then you can generate, you can basically, well, you can have more control over between which images are you interpolating. That's the basic idea. Okay, so now I'm going to do this route. You can always play with this one, but I'm gonna specify both latents and let's then interpolate and let me show you what's going on. So let's do the following thing. So I'm going to interpolate between this robot and this robot here. So to do that, we need to set one as the source, one as the target. So because this is one is already set as the source, I'm gonna take this one and put it as a target. So basically that means we need to take its corresponding latent and that's gonna be easy because of the structure I'm using. So that means I just have to do the following. I just have to take the source latent, which is 002. I need to paste it here. I need to set it to one instead. And now we have that robot set as the target and that's pretty much it. The metadata will be ignored and I think we just need to, yeah, that's pretty much it. Now we're gonna have source latent. Let me just check. And we have the target latent. We load them and then we are gonna be generating the images. Cool, I'm gonna set a break point here. I'm gonna hit execute and I'm gonna get back to you once we hit this break point. Okay guys, so let's enter. Hopefully we have both paths specified correctly. I'm gonna load those latents. We save the metadata again. And we save these latents, which is probably less relevant in this example because I already had them saved. So probably worth updating the script there. And now let's see what I'm doing. So basically you can see here, I'm doing a lint space. So between zero and one, I want to generate 50 images. So I had to set that number of images to 50 here. And additionally prepend this zero just as a simple hack. And the reason being is I want to, for the first iteration, I want to generate the target image such that I'm confident that the target image I've chosen is the correct one. And then I'm gonna generate the source one and then it's gonna do the smooth interpolation from source to target, if that makes sense. Okay, so that's why in the first iteration we enter here, we grab the target images latent. And now I'm gonna generate it. So let's just do this same logic. We just passed a prompt, we passed the number of inference steps, we passed the latent, the guidance scale and everything. And then we're gonna save the image. So let me show you that indeed, so if I recall correctly, we use which image did we use as the source, just a second. So we use as the target, we are using the image one. Okay, so the target is this one. So I'm expecting to generate yet another image that looks like that. So let's verify that. So I'm gonna hit here. And if I save, okay, I actually get the different image. And the reason is because of the bug I had, basically those metadata that were saved for these first two images because of the bug, it was actually corresponding to this image. So now the second one should be this one. And if that's not correct, then something weird is going on, but I'm fairly sure that's not gonna happen. So I'm gonna hit F5 and get back to you once this image is generated. So that's the source. Okay, so let's do F10. And you can see here, okay, now we're starting the interpolation. I'll get back to you once all of the images are generated, we'll see a smooth interpolation between this image as a source and this image as a target. Okay, guys, I've let it run and here you can see the results. Fairly cool. So here is again the source image. Here is the target image. Again, apologize for the bug there, but yeah. You will not have to, by the way, sort it out because as soon as the patch is submitted to the diffusers library, you can just use the script without having to do the patch. But for now, if you try and use the script, you will have to apply the patch before that. Okay, so let's see how this is gonna look like. Let's start with this image and let's slowly go to the right. You can see how the image is morphing. It's amazing. And then it's slowly morphing to the final robot and that's it. So there are some jumps in the latent space. That's always the case. And I had to use like maybe 200 or more images if I were to have a smoother interpolation over these couple of last steps. So you can see there are some sudden jumps here. But yeah, guys, that's pretty much it. I showed you how to use the script. There is the small gotcha I just explained about having to patch this file, but everything else should work as expected. Hopefully you like this video. If you try and generate some cool images with this script, feel free to tag me on Twitter or on LinkedIn and I'll be happy to see your images and reply. And yeah, again, I'll be updating the script for the next couple of days, especially if somebody submits an issue, a bug somebody finds or something, I'm gonna try and address it as soon as possible. Having said that, if you like this video, share it out with your friends, try and generate cool stuff yourself and subscribe to this YouTube channel, of course. Until next time, bye bye. Often when you miss an issue, I try and watch it again.
[{"start": 0.0, "end": 2.12, "text": " What's up guys, Alex here."}, {"start": 2.12, "end": 5.08, "text": " In this video, I'm showing you how to get started"}, {"start": 5.08, "end": 6.5200000000000005, "text": " with stable diffusion."}, {"start": 6.5200000000000005, "end": 9.24, "text": " I have another video where I do a super deep dive"}, {"start": 9.24, "end": 10.6, "text": " into how everything works."}, {"start": 10.6, "end": 12.6, "text": " I take the original code base"}, {"start": 12.6, "end": 14.42, "text": " and I literally step through the code"}, {"start": 14.42, "end": 15.9, "text": " and explain you how the training works,"}, {"start": 15.9, "end": 17.88, "text": " how the sampling works, how to generate images,"}, {"start": 17.88, "end": 22.52, "text": " how to have complete power over your generation basically."}, {"start": 22.52, "end": 24.36, "text": " So if you're curious to learn more about that,"}, {"start": 24.36, "end": 27.0, "text": " I'm gonna link it somewhere down in the video description"}, {"start": 27.0, "end": 28.400000000000002, "text": " once the video is published."}, {"start": 28.4, "end": 31.0, "text": " So in this video, I show you three ways to get started"}, {"start": 31.0, "end": 32.0, "text": " with stable diffusion."}, {"start": 32.0, "end": 35.04, "text": " The first one being using the hugging face spaces,"}, {"start": 35.04, "end": 38.7, "text": " the second one being using diffusers notebooks,"}, {"start": 38.7, "end": 41.06, "text": " and the third one is using my code"}, {"start": 41.06, "end": 44.66, "text": " I developed inspired by Karpathis recent gist"}, {"start": 44.66, "end": 46.519999999999996, "text": " and basically you can see some of the images"}, {"start": 46.519999999999996, "end": 49.92, "text": " I managed to generate with that code base."}, {"start": 49.92, "end": 52.0, "text": " Stick around to the second part of the video"}, {"start": 52.0, "end": 53.36, "text": " where I literally walk you through"}, {"start": 53.36, "end": 55.18, "text": " how to exactly get started and set it up"}, {"start": 55.18, "end": 56.26, "text": " on your own machine."}, {"start": 56.26, "end": 58.559999999999995, "text": " But before we go there, I wanna thank Cohere"}, {"start": 58.559999999999995, "end": 59.72, "text": " for sponsoring this video."}, {"start": 59.72, "end": 61.96, "text": " They're basically an amazing way to get started"}, {"start": 61.96, "end": 64.48, "text": " building NLP applications."}, {"start": 64.48, "end": 66.64, "text": " Getting started is as easy as just doing"}, {"start": 66.64, "end": 69.1, "text": " pip install Cohere as you can see here"}, {"start": 69.1, "end": 73.24, "text": " and they also have a very cool no code prototyping platform"}, {"start": 73.24, "end": 76.38, "text": " where you can just explore what features they offer"}, {"start": 76.38, "end": 78.1, "text": " and then get started very easily."}, {"start": 78.1, "end": 79.92, "text": " So here's an example of generation"}, {"start": 79.92, "end": 83.32, "text": " where you can do for example, hashtag generation."}, {"start": 83.32, "end": 86.75999999999999, "text": " They also support embeddings, classifier stuff"}, {"start": 86.75999999999999, "end": 89.36, "text": " and getting started is like super easy."}, {"start": 89.36, "end": 92.83999999999999, "text": " You just do export code, you copy paste the snippet here"}, {"start": 92.83999999999999, "end": 96.28, "text": " and you additionally just have to create an API key"}, {"start": 96.28, "end": 98.61999999999999, "text": " which you can do in the dashboard here."}, {"start": 98.61999999999999, "end": 101.0, "text": " So it literally takes a couple of minutes to get started."}, {"start": 101.0, "end": 103.63999999999999, "text": " If you wanna learn more about Cohere,"}, {"start": 103.63999999999999, "end": 105.8, "text": " I strongly suggest checking out some of the blogs"}, {"start": 105.8, "end": 108.94, "text": " where they use Cohere to generate super cool applications."}, {"start": 108.94, "end": 111.19999999999999, "text": " So this one by Jay Elmer, the same author"}, {"start": 111.2, "end": 114.36, "text": " who kind of famous Illustrator transformer blog"}, {"start": 114.36, "end": 116.72, "text": " basically walks you through how you can use Cohere"}, {"start": 116.72, "end": 120.2, "text": " to generate insights from various hacker news posts"}, {"start": 120.2, "end": 124.12, "text": " and there is also a very nice web app that goes behind it."}, {"start": 124.12, "end": 126.56, "text": " Finally, the documentation is amazing."}, {"start": 126.56, "end": 130.16, "text": " You can learn just about the machine learning"}, {"start": 130.16, "end": 132.56, "text": " as much as you can learn about Cohere product."}, {"start": 132.56, "end": 136.6, "text": " So I really like that type of contribution."}, {"start": 136.6, "end": 140.12, "text": " By signing up by using the link down below,"}, {"start": 140.12, "end": 142.8, "text": " you can get started with $75"}, {"start": 142.8, "end": 145.6, "text": " and that's enough in their own words to generate"}, {"start": 145.6, "end": 148.56, "text": " like the whole works of Shakespeare for free."}, {"start": 148.56, "end": 151.28, "text": " So yeah, check them out, support the channel"}, {"start": 151.28, "end": 152.96, "text": " and now let's get back to the video."}, {"start": 152.96, "end": 154.8, "text": " Having said that you must have seen the news,"}, {"start": 154.8, "end": 157.82, "text": " basically a week ago, the stable diffusion weights"}, {"start": 157.82, "end": 160.88, "text": " were published and now you can generate awesome images"}, {"start": 160.88, "end": 162.38, "text": " such as these ones here."}, {"start": 162.38, "end": 164.92000000000002, "text": " And what you can probably notice immediately"}, {"start": 164.92000000000002, "end": 168.12, "text": " is that with stable diffusion, you can generate human faces,"}, {"start": 168.12, "end": 171.32, "text": " you have much more control and the model is less constrained"}, {"start": 171.32, "end": 173.54, "text": " than the other alternatives, I think,"}, {"start": 173.54, "end": 174.96, "text": " to the best of my knowledge."}, {"start": 174.96, "end": 178.0, "text": " And basically you can run it much faster"}, {"start": 178.0, "end": 180.12, "text": " and you can even run it on your own machine"}, {"start": 180.12, "end": 182.8, "text": " even if you have only eight gigabytes of VRAM."}, {"start": 182.8, "end": 185.1, "text": " I'm gonna show you how to do that in a couple of minutes."}, {"start": 185.1, "end": 187.88, "text": " People have already been using a stable diffusion"}, {"start": 187.88, "end": 189.20000000000002, "text": " to generate very cool art."}, {"start": 189.20000000000002, "end": 191.88, "text": " So here is an example you must have seen from Twitter"}, {"start": 191.88, "end": 196.04000000000002, "text": " where Xander generated this video called Voyage Through Time"}, {"start": 196.04, "end": 198.92, "text": " using various prompts and prompt engineering"}, {"start": 198.92, "end": 200.17999999999998, "text": " and tweaking the seeds."}, {"start": 200.17999999999998, "end": 202.92, "text": " You can see, just yeah, check out the video"}, {"start": 202.92, "end": 205.28, "text": " at your own pace, it's an amazing piece of art."}, {"start": 205.28, "end": 208.64, "text": " So you can see how he managed to generate this video"}, {"start": 208.64, "end": 210.76, "text": " by basically tweaking the prompts,"}, {"start": 210.76, "end": 212.79999999999998, "text": " creating a narrative, tweaking the seeds"}, {"start": 212.79999999999998, "end": 215.68, "text": " and then stitching all the images together."}, {"start": 215.68, "end": 217.0, "text": " Okay, so having said that,"}, {"start": 217.0, "end": 218.95999999999998, "text": " how do you get started with stable diffusion"}, {"start": 218.95999999999998, "end": 221.34, "text": " and how do you generate your own images?"}, {"start": 221.34, "end": 224.56, "text": " So the easiest way you can do is basically"}, {"start": 224.56, "end": 227.8, "text": " use this hugging faces like a space."}, {"start": 227.8, "end": 230.58, "text": " So this is literally like a web application."}, {"start": 230.58, "end": 233.14000000000001, "text": " You just come here, you don't need to know anything"}, {"start": 233.14000000000001, "end": 235.6, "text": " about how stuff works, you just enter a prompt."}, {"start": 235.6, "end": 237.2, "text": " So I literally just entered this prompt"}, {"start": 237.2, "end": 239.36, "text": " a couple of minutes ago, a horse with glasses."}, {"start": 239.36, "end": 242.96, "text": " I hit generate image and a couple of minutes later,"}, {"start": 242.96, "end": 244.96, "text": " you get this as the output."}, {"start": 244.96, "end": 248.12, "text": " So a couple of minutes later statement is the problem."}, {"start": 248.12, "end": 249.64000000000001, "text": " So you literally have to wait a lot"}, {"start": 249.64000000000001, "end": 252.36, "text": " depending on the current load on the server."}, {"start": 252.36, "end": 256.36, "text": " So basically that's the main drawback of this approach."}, {"start": 256.36, "end": 258.68, "text": " You do have some control, some advanced options"}, {"start": 258.68, "end": 260.8, "text": " you can pick how many images you want,"}, {"start": 260.8, "end": 262.34000000000003, "text": " you can pick how many steps"}, {"start": 262.34000000000003, "end": 264.1, "text": " of the diffusion process you want."}, {"start": 264.1, "end": 265.32, "text": " As a general rule of thumb,"}, {"start": 265.32, "end": 267.88, "text": " the more you have, the better the quality."}, {"start": 267.88, "end": 269.84000000000003, "text": " They kind of kept it to 50 here."}, {"start": 269.84000000000003, "end": 272.28000000000003, "text": " So you should probably stick it at 50."}, {"start": 272.28000000000003, "end": 274.96000000000004, "text": " Well, but if you obviously want the results faster"}, {"start": 274.96000000000004, "end": 275.92, "text": " then you need to reduce it."}, {"start": 275.92, "end": 277.96000000000004, "text": " But 50 is kind of recommended value."}, {"start": 277.96000000000004, "end": 280.1, "text": " Then the guidance scale basically is a trade off"}, {"start": 280.1, "end": 282.8, "text": " between how high quality the images,"}, {"start": 282.8, "end": 285.52000000000004, "text": " how closely is it following your prompt"}, {"start": 285.52000000000004, "end": 287.6, "text": " versus how diverse the inputs are."}, {"start": 287.6, "end": 292.52000000000004, "text": " So like usually people use 7.5 or between three and 10."}, {"start": 292.52000000000004, "end": 296.6, "text": " And basically the more you increase the scale,"}, {"start": 296.6, "end": 298.68, "text": " the higher quality the images will be,"}, {"start": 298.68, "end": 300.40000000000003, "text": " they will closely follow the prompt,"}, {"start": 300.40000000000003, "end": 303.6, "text": " but you'll get less diverse set of output results."}, {"start": 303.6, "end": 306.52000000000004, "text": " And finally, the seed is also just enables you"}, {"start": 306.52000000000004, "end": 308.72, "text": " to control the diversity of your sample."}, {"start": 308.72, "end": 310.6, "text": " That's it, that's the simplest approach,"}, {"start": 310.6, "end": 312.18, "text": " but it takes a lot of time."}, {"start": 312.18, "end": 315.72, "text": " So if we try to input something like an AI robot"}, {"start": 317.66, "end": 321.28000000000003, "text": " having an epiphany moment,"}, {"start": 321.28000000000003, "end": 323.48, "text": " and if I hit generate image,"}, {"start": 323.48, "end": 325.20000000000005, "text": " you can see that it will take at least,"}, {"start": 325.20000000000005, "end": 328.52000000000004, "text": " so 152 seconds current estimated time."}, {"start": 328.52000000000004, "end": 330.02000000000004, "text": " So it's two and a half minutes."}, {"start": 330.96000000000004, "end": 334.32000000000005, "text": " So if you're kind of a patient to wait, then go ahead."}, {"start": 334.32000000000005, "end": 336.52000000000004, "text": " The second best approach is just use the colab."}, {"start": 336.52000000000004, "end": 338.32000000000005, "text": " So there is a lot of cool colabs."}, {"start": 338.32, "end": 340.71999999999997, "text": " I linked directly from the diffusers library,"}, {"start": 340.71999999999997, "end": 343.44, "text": " which is Hugging Face's new library"}, {"start": 343.44, "end": 345.92, "text": " that basically is trying to get the latest"}, {"start": 345.92, "end": 350.54, "text": " and greatest diffusion models available to general public."}, {"start": 350.54, "end": 352.78, "text": " So the first thing we need to do here"}, {"start": 352.78, "end": 355.56, "text": " is to connect to the GPU."}, {"start": 355.56, "end": 356.92, "text": " It requires a lot of RAM,"}, {"start": 356.92, "end": 359.64, "text": " so you just kind of acknowledge that, okay."}, {"start": 359.64, "end": 362.08, "text": " You make sure that you get the GPU"}, {"start": 362.08, "end": 366.0, "text": " and you make sure by either just checking it here"}, {"start": 366.0, "end": 368.96, "text": " so we can see it here,"}, {"start": 368.96, "end": 372.8, "text": " basically we have GPU enabled."}, {"start": 372.8, "end": 375.76, "text": " So that's one way to figure out that you have the GPU."}, {"start": 375.76, "end": 377.56, "text": " The second one is just hit here"}, {"start": 377.56, "end": 381.12, "text": " and then you have this part change runtime type."}, {"start": 381.12, "end": 382.88, "text": " You can see that the GPU is set"}, {"start": 382.88, "end": 385.12, "text": " as the hardware accelerator by default."}, {"start": 385.12, "end": 388.28, "text": " Okay, so let's kind of run this one."}, {"start": 388.28, "end": 392.64, "text": " Just hit run anyway, and you get some output here."}, {"start": 392.64, "end": 395.24, "text": " The main point is if it's working,"}, {"start": 395.24, "end": 397.12, "text": " that means you have a GPU."}, {"start": 397.12, "end": 398.84000000000003, "text": " Okay, so now we're gonna install some libraries"}, {"start": 398.84000000000003, "end": 401.76, "text": " such as diffusers, transformers, SciPy,"}, {"start": 401.76, "end": 405.0, "text": " and some other libraries for processing text."}, {"start": 405.0, "end": 407.84000000000003, "text": " And after that, we'll have to log in"}, {"start": 407.84000000000003, "end": 411.16, "text": " to be able to use the model weights."}, {"start": 411.16, "end": 413.28000000000003, "text": " So you basically have to accept an agreement"}, {"start": 413.28000000000003, "end": 414.92, "text": " before you use this notebook."}, {"start": 414.92, "end": 418.08, "text": " So I'm gonna hit run here and then run here"}, {"start": 418.08, "end": 420.02, "text": " and it will ask for credentials."}, {"start": 420.02, "end": 423.34000000000003, "text": " So basically what I have to do is go to this page"}, {"start": 423.34, "end": 428.34, "text": " and then basically just copy the token."}, {"start": 428.52, "end": 430.59999999999997, "text": " You need to have an account and just log in."}, {"start": 430.59999999999997, "end": 433.47999999999996, "text": " If you don't have, just create an account on Hogan face"}, {"start": 433.47999999999996, "end": 436.56, "text": " and then you can just kind of copy paste that information"}, {"start": 436.56, "end": 440.67999999999995, "text": " and you will be able to log in successfully, okay?"}, {"start": 440.67999999999995, "end": 441.79999999999995, "text": " So that's it."}, {"start": 441.79999999999995, "end": 444.67999999999995, "text": " So now the first cell, what it does is basically"}, {"start": 444.67999999999995, "end": 448.91999999999996, "text": " creates, instantiates this stable diffusion pipeline."}, {"start": 448.91999999999996, "end": 452.15999999999997, "text": " You can see we're using this model V14."}, {"start": 452.16, "end": 454.16, "text": " I'm gonna quickly show you what it means."}, {"start": 454.16, "end": 457.56, "text": " So you can see here on this link,"}, {"start": 457.56, "end": 462.56, "text": " basically that V14 says it's V12 plus 225,000 steps"}, {"start": 465.04, "end": 468.6, "text": " at certain resolution on lion aesthetics data set,"}, {"start": 468.6, "end": 469.44000000000005, "text": " blah, blah, blah."}, {"start": 469.44000000000005, "end": 471.84000000000003, "text": " So that means that's pretty much the best checkpoint."}, {"start": 471.84000000000003, "end": 475.36, "text": " So that one, you can use either that one or V13."}, {"start": 475.36, "end": 477.56, "text": " You can kind of play if you want with both."}, {"start": 478.92, "end": 481.64000000000004, "text": " So having said that, let's go back to the notebook."}, {"start": 481.64, "end": 483.84, "text": " This thing is still loading."}, {"start": 483.84, "end": 485.71999999999997, "text": " The second part you have to notice here is that"}, {"start": 485.71999999999997, "end": 490.71999999999997, "text": " they're using FP16 because basically if we were to run"}, {"start": 491.08, "end": 495.32, "text": " in full precision, that's FP32, you would probably get"}, {"start": 495.32, "end": 498.52, "text": " CUDA out of memory exceptions, even on a colab"}, {"start": 498.52, "end": 501.44, "text": " that has 16 gigabytes of VRAM."}, {"start": 501.44, "end": 505.36, "text": " So I did play and set, like just delete this part here"}, {"start": 505.36, "end": 506.84, "text": " so you can just delete these two,"}, {"start": 506.84, "end": 509.71999999999997, "text": " but I was getting out of memory exceptions"}, {"start": 509.72, "end": 513.08, "text": " if I'm using 512, 512 resolution."}, {"start": 513.08, "end": 516.48, "text": " So probably the best thing is to just leave it FP16."}, {"start": 516.48, "end": 519.8000000000001, "text": " Later, once I show you the script that you can run"}, {"start": 519.8000000000001, "end": 522.9200000000001, "text": " on your own machine, if you have a better GPU,"}, {"start": 522.9200000000001, "end": 526.2, "text": " then it's recommended to just remove this part"}, {"start": 526.2, "end": 530.5600000000001, "text": " and then you'll have even higher quality generated images."}, {"start": 530.5600000000001, "end": 534.24, "text": " Okay, so the final argument here is just"}, {"start": 534.24, "end": 537.88, "text": " the authentication token that's just needed for us"}, {"start": 537.88, "end": 540.6, "text": " to load the weights and that's pretty much it."}, {"start": 540.6, "end": 543.4, "text": " Okay, once that's loaded, I'm just gonna hit this cell here."}, {"start": 543.4, "end": 548.4, "text": " It's going to push the loaded weights onto the GPU"}, {"start": 548.64, "end": 551.2, "text": " and now we can start generating the images."}, {"start": 551.2, "end": 553.08, "text": " So here you can see basically, let me zoom in"}, {"start": 553.08, "end": 556.68, "text": " a little bit here and remove this part here."}, {"start": 556.68, "end": 559.72, "text": " So we can specify a prompt here, a photograph"}, {"start": 559.72, "end": 563.08, "text": " of an astronaut riding a horse, and then you basically"}, {"start": 563.08, "end": 565.92, "text": " pass the prompt to the pipe and you grab the sample,"}, {"start": 565.92, "end": 568.1999999999999, "text": " the zero sample, which will be the image."}, {"start": 568.1999999999999, "end": 569.8399999999999, "text": " Then we can save it and display the image"}, {"start": 569.8399999999999, "end": 571.88, "text": " and you can see the astronaut here."}, {"start": 571.88, "end": 575.4399999999999, "text": " Okay, so I'm gonna run this now and let's see what we get."}, {"start": 575.4399999999999, "end": 577.3199999999999, "text": " So we probably won't have the same image"}, {"start": 577.3199999999999, "end": 580.3199999999999, "text": " as the one that was there and you will see"}, {"start": 580.3199999999999, "end": 582.1999999999999, "text": " how long it takes to generate an image."}, {"start": 582.1999999999999, "end": 584.28, "text": " So currently, this is the speed."}, {"start": 584.28, "end": 587.24, "text": " I think we have 50 steps by default."}, {"start": 587.24, "end": 590.28, "text": " So now we're at 28, we need to wait a bit more"}, {"start": 590.28, "end": 592.8, "text": " and then we'll get the image out."}, {"start": 592.8, "end": 595.0799999999999, "text": " There is a lot more parameters you can tune."}, {"start": 595.08, "end": 597.96, "text": " Again, I super encourage you to check out my deep dive"}, {"start": 597.96, "end": 601.5600000000001, "text": " if you wanna really understand how to delicately control"}, {"start": 601.5600000000001, "end": 604.64, "text": " stable diffusion, that's gonna be probably your best bet."}, {"start": 605.64, "end": 609.12, "text": " Okay, so we see an image here and the second thing"}, {"start": 609.12, "end": 612.6800000000001, "text": " you can do, if you kind of, so if I were to hit"}, {"start": 612.6800000000001, "end": 616.12, "text": " Shift Enter again here and generate yet another image,"}, {"start": 616.12, "end": 619.08, "text": " because we didn't set any seeds, because of that,"}, {"start": 619.08, "end": 622.12, "text": " we'll basically generate a completely different image"}, {"start": 622.12, "end": 624.1600000000001, "text": " every time we run the cell."}, {"start": 624.16, "end": 626.0, "text": " So if you don't like that behavior,"}, {"start": 626.0, "end": 628.92, "text": " if you want to have consistency between the runs,"}, {"start": 628.92, "end": 630.6, "text": " then you have to do a small trick,"}, {"start": 630.6, "end": 632.7199999999999, "text": " so you can see here a different image."}, {"start": 632.7199999999999, "end": 635.7199999999999, "text": " You have to do a small trick and that's to basically"}, {"start": 635.7199999999999, "end": 638.92, "text": " set the seed here of your torch generator"}, {"start": 638.92, "end": 641.04, "text": " and then pass that generator to the pipe."}, {"start": 641.04, "end": 644.4, "text": " So here in this example, if I were to run this,"}, {"start": 644.4, "end": 647.0799999999999, "text": " every time you rerun the cell,"}, {"start": 647.0799999999999, "end": 650.0, "text": " you're gonna get the same image output,"}, {"start": 650.0, "end": 652.72, "text": " assuming that you're not changing the prompt obviously."}, {"start": 652.72, "end": 655.9200000000001, "text": " So that's the idea with the generator part."}, {"start": 655.9200000000001, "end": 657.48, "text": " So in the background, what happens is,"}, {"start": 657.48, "end": 662.1600000000001, "text": " it's going to be used to generate like a specific latent,"}, {"start": 662.1600000000001, "end": 664.84, "text": " which is just a noise vector and every time you run it"}, {"start": 664.84, "end": 667.0, "text": " because you've passed the generator,"}, {"start": 667.0, "end": 669.24, "text": " you'll end up with the same noise vector"}, {"start": 669.24, "end": 671.24, "text": " and because the noise vector is the same,"}, {"start": 671.24, "end": 672.88, "text": " the output image is gonna be the same."}, {"start": 672.88, "end": 675.08, "text": " So here you can see the results are consistent"}, {"start": 675.08, "end": 678.1600000000001, "text": " with what we had before I rerun the cell."}, {"start": 678.1600000000001, "end": 680.64, "text": " Okay, cool."}, {"start": 680.64, "end": 682.96, "text": " Again, you can control the number of inference steps."}, {"start": 682.96, "end": 685.4399999999999, "text": " I already mentioned in the previous here,"}, {"start": 686.36, "end": 689.92, "text": " like in the space here that 50 is probably recommended."}, {"start": 689.92, "end": 692.0, "text": " You don't wanna go lower than that."}, {"start": 692.0, "end": 694.08, "text": " You can also see the images that were generated,"}, {"start": 694.08, "end": 695.96, "text": " which are fairly cool."}, {"start": 695.96, "end": 699.04, "text": " But yeah, so here we have 15 and that's why we get,"}, {"start": 699.04, "end": 701.76, "text": " you can see here much lower quality image"}, {"start": 701.76, "end": 703.6, "text": " compared to the one above."}, {"start": 704.6, "end": 706.16, "text": " That's pretty much it."}, {"start": 706.16, "end": 709.56, "text": " So here you can check out this notebook at your own pace."}, {"start": 709.56, "end": 711.8399999999999, "text": " This just creates some utility functions"}, {"start": 711.8399999999999, "end": 714.5999999999999, "text": " where you can generate multiple images in a grid."}, {"start": 714.5999999999999, "end": 718.3199999999999, "text": " Same here, but like the logic remains the same."}, {"start": 718.3199999999999, "end": 721.3199999999999, "text": " You can also play with the height and width parameters,"}, {"start": 721.3199999999999, "end": 726.3199999999999, "text": " whereby, well, you can get a non-square type of an image"}, {"start": 727.0799999999999, "end": 728.0, "text": " and there are some rules,"}, {"start": 728.0, "end": 731.0799999999999, "text": " like you need to have multiples of eight"}, {"start": 731.0799999999999, "end": 733.5999999999999, "text": " for your image width and height"}, {"start": 733.5999999999999, "end": 736.28, "text": " and you probably don't wanna go lower than 512."}, {"start": 736.28, "end": 738.1999999999999, "text": " Okay, so then the rest of the notebook"}, {"start": 738.2, "end": 741.44, "text": " is a deep dive into how the thing works."}, {"start": 741.44, "end": 743.24, "text": " But I already told you I have a video"}, {"start": 743.24, "end": 745.08, "text": " that's almost two hours long"}, {"start": 745.08, "end": 747.96, "text": " explaining exactly how stable diffusion is trained,"}, {"start": 747.96, "end": 749.72, "text": " how sampling works, how everything works."}, {"start": 749.72, "end": 751.72, "text": " So do check out that one."}, {"start": 751.72, "end": 753.4000000000001, "text": " Okay, there's a couple more notebooks."}, {"start": 753.4000000000001, "end": 754.6800000000001, "text": " I'm gonna exit here."}, {"start": 754.6800000000001, "end": 756.9200000000001, "text": " There's a couple more cool notebooks you can check out."}, {"start": 756.9200000000001, "end": 759.44, "text": " So this one, image to image pipeline,"}, {"start": 759.44, "end": 762.76, "text": " basically enables you to start from a particular image"}, {"start": 762.76, "end": 764.24, "text": " and then using a prompt,"}, {"start": 764.24, "end": 768.52, "text": " guide that image towards a certain, basically appearance."}, {"start": 768.52, "end": 771.28, "text": " So you can see here that if we take the prompt,"}, {"start": 771.28, "end": 774.92, "text": " a fantasy landscape trending on ArtStation,"}, {"start": 774.92, "end": 777.2, "text": " you get this output here."}, {"start": 777.2, "end": 779.32, "text": " Okay, so I think worth mentioning here"}, {"start": 779.32, "end": 781.48, "text": " is that you can see, like probably,"}, {"start": 781.48, "end": 784.6800000000001, "text": " if you've never seen stable diffusion before,"}, {"start": 784.6800000000001, "end": 786.28, "text": " this part might be kinda confusing,"}, {"start": 786.28, "end": 789.44, "text": " like why would you put the trending on ArtStation?"}, {"start": 789.44, "end": 793.96, "text": " And that's the whole art of basically prompt engineering"}, {"start": 793.96, "end": 798.96, "text": " where you find certain words, certain sentences"}, {"start": 799.2, "end": 801.6800000000001, "text": " that trigger the model to generate better output"}, {"start": 801.6800000000001, "end": 804.76, "text": " or output with specific properties."}, {"start": 804.76, "end": 808.48, "text": " So people kinda, we globally as a community"}, {"start": 808.48, "end": 810.8000000000001, "text": " try a lot of prompts and then somebody figures out"}, {"start": 810.8000000000001, "end": 812.2, "text": " a good prompt and then they share it"}, {"start": 812.2, "end": 813.4000000000001, "text": " and then the community learns"}, {"start": 813.4000000000001, "end": 817.2800000000001, "text": " and then we're slowly coevolving with the model in a way."}, {"start": 818.36, "end": 819.88, "text": " I'm gonna leave you to explore this notebook"}, {"start": 819.88, "end": 822.76, "text": " at your own pace, but those are the basic notebooks"}, {"start": 822.76, "end": 824.3199999999999, "text": " you have at your disposal."}, {"start": 824.3199999999999, "end": 828.16, "text": " There is this one as well where you can control"}, {"start": 828.16, "end": 831.08, "text": " exactly the type of a seed you're using."}, {"start": 831.08, "end": 833.0, "text": " So what I mean by that is the following."}, {"start": 833.0, "end": 837.2, "text": " So here, basically they generate four latents."}, {"start": 837.2, "end": 840.6, "text": " So those are those types of noise vectors I mentioned"}, {"start": 840.6, "end": 844.3199999999999, "text": " in this cell here and they generate four images here."}, {"start": 844.3199999999999, "end": 847.08, "text": " And then basically, if you like one of them,"}, {"start": 847.08, "end": 848.92, "text": " you can pick that same latent"}, {"start": 848.92, "end": 850.92, "text": " and you can reconstruct the image"}, {"start": 850.92, "end": 854.36, "text": " or generate like use different prompts"}, {"start": 854.36, "end": 856.7199999999999, "text": " and have a similar type of appearance in the image."}, {"start": 856.7199999999999, "end": 858.36, "text": " Let me show you what I mean by that."}, {"start": 858.36, "end": 862.0, "text": " So if we just pick this same latent, you can see here."}, {"start": 862.0, "end": 865.24, "text": " So they pick seed one because we have four seeds."}, {"start": 865.24, "end": 867.88, "text": " This corresponds to this image here."}, {"start": 867.88, "end": 868.8399999999999, "text": " Whoops."}, {"start": 868.8399999999999, "end": 873.4399999999999, "text": " So it corresponds to this image here, to this one here."}, {"start": 873.4399999999999, "end": 876.7199999999999, "text": " So if we pick that same seed and regenerate the latent,"}, {"start": 876.7199999999999, "end": 879.28, "text": " you can see we get the same image out."}, {"start": 879.28, "end": 881.56, "text": " But if you now change the prompt,"}, {"start": 881.56, "end": 883.68, "text": " but reuse the same latent,"}, {"start": 883.68, "end": 886.04, "text": " you can see that you get a similar posture"}, {"start": 886.04, "end": 889.24, "text": " to this dog here, but just a different breed."}, {"start": 889.24, "end": 893.1999999999999, "text": " Similarly here, Labrador in the style of Van Gogh."}, {"start": 893.1999999999999, "end": 896.66, "text": " And you can see that you have some features"}, {"start": 896.66, "end": 900.12, "text": " of the output image, some like course level structures"}, {"start": 900.12, "end": 902.56, "text": " in this particular example are very similar."}, {"start": 902.56, "end": 904.68, "text": " So namely the posture."}, {"start": 904.68, "end": 907.4399999999999, "text": " And you can even go wild and use like,"}, {"start": 907.4399999999999, "end": 908.52, "text": " you don't have to use a dog."}, {"start": 908.52, "end": 910.48, "text": " You can be as creative as you want."}, {"start": 910.48, "end": 913.04, "text": " Okay, so that's everything that's at your disposal."}, {"start": 913.04, "end": 917.5799999999999, "text": " Basically, that's the two easiest methods you can use"}, {"start": 917.5799999999999, "end": 919.8, "text": " is use the space or just use the notebooks"}, {"start": 919.8, "end": 921.0799999999999, "text": " to generate images."}, {"start": 921.0799999999999, "end": 923.36, "text": " But if you want more control, if you want more speed,"}, {"start": 923.36, "end": 925.8, "text": " if you have better hardware locally,"}, {"start": 925.8, "end": 928.6, "text": " I strongly recommend you do the third option."}, {"start": 928.6, "end": 931.56, "text": " And that's to use the script I generated."}, {"start": 931.56, "end": 933.16, "text": " And now I'm gonna walk you through the process"}, {"start": 933.16, "end": 935.48, "text": " of how you can set it up and get started"}, {"start": 935.48, "end": 938.4399999999999, "text": " generating your own images on your own machine."}, {"start": 938.44, "end": 941.6400000000001, "text": " And also do some fancy things like interpolation, etc."}, {"start": 941.6400000000001, "end": 944.36, "text": " The script itself was inspired by Carpathi script."}, {"start": 944.36, "end": 947.32, "text": " So this is this gist he made basically"}, {"start": 947.32, "end": 950.6, "text": " using spherical interpolation to generate"}, {"start": 950.6, "end": 954.6800000000001, "text": " basically a video such as this one, or similar videos."}, {"start": 954.6800000000001, "end": 956.6800000000001, "text": " So you basically pick two images,"}, {"start": 956.6800000000001, "end": 960.12, "text": " and then you smoothly interpolate in the latent space"}, {"start": 960.12, "end": 962.5200000000001, "text": " of the diffusion model, you do interpolation,"}, {"start": 962.5200000000001, "end": 965.32, "text": " and then you get the corresponding images out."}, {"start": 965.32, "end": 968.2, "text": " And you can see how they are slowly morphing"}, {"start": 968.2, "end": 971.0600000000001, "text": " from the source image into the target image."}, {"start": 971.0600000000001, "end": 973.6600000000001, "text": " Okay, that's the basic idea."}, {"start": 973.6600000000001, "end": 976.44, "text": " Okay, guys, let's see how we can go through this notebook,"}, {"start": 976.44, "end": 979.5600000000001, "text": " how we can set up this on our local machine."}, {"start": 979.5600000000001, "end": 980.94, "text": " So here's what you need to do."}, {"start": 980.94, "end": 984.86, "text": " So I'm gonna assume, so I do have basically a Windows machine,"}, {"start": 984.86, "end": 988.2, "text": " but it should be similar no matter whether you're using Linux"}, {"start": 988.2, "end": 990.48, "text": " or Mac OS or whatnot."}, {"start": 990.48, "end": 994.0400000000001, "text": " So first thing you need to do is basically find a place"}, {"start": 994.0400000000001, "end": 995.5200000000001, "text": " where you wanna store this."}, {"start": 995.52, "end": 1000.52, "text": " So I'm gonna randomly create a new folder here on desktop,"}, {"start": 1001.6, "end": 1006.6, "text": " and name it stable diffusion playground."}, {"start": 1006.72, "end": 1008.68, "text": " And I'm going to open it up."}, {"start": 1008.68, "end": 1012.24, "text": " I'm going to open up a git bash here."}, {"start": 1012.24, "end": 1017.24, "text": " So basically you want to clone this particular repo"}, {"start": 1017.5, "end": 1018.6, "text": " into this directory."}, {"start": 1018.6, "end": 1022.02, "text": " So you can do git clone, and then just copy the URL"}, {"start": 1022.02, "end": 1025.04, "text": " you copied from the website here, from the GitHub."}, {"start": 1025.04, "end": 1026.3999999999999, "text": " Okay, so that's one way."}, {"start": 1026.3999999999999, "end": 1028.0, "text": " You can also just do it here."}, {"start": 1028.96, "end": 1032.6, "text": " You should have the option to just download,"}, {"start": 1032.6, "end": 1035.8, "text": " but people usually use console to do this."}, {"start": 1035.8, "end": 1038.68, "text": " So if you hit clone, you're going to clone,"}, {"start": 1038.68, "end": 1040.6399999999999, "text": " as you can see here, the repository."}, {"start": 1040.6399999999999, "end": 1041.94, "text": " Now let's follow the second step."}, {"start": 1041.94, "end": 1043.52, "text": " So you can just follow the steps here."}, {"start": 1043.52, "end": 1045.7, "text": " Everything should be self-explanatory."}, {"start": 1045.7, "end": 1050.7, "text": " Okay, so now just navigate with your conda prompt to the,"}, {"start": 1050.7, "end": 1055.7, "text": " so navigate with the anaconda to basically this directory."}, {"start": 1057.5800000000002, "end": 1062.38, "text": " So let's do cd to that directory, and we're here."}, {"start": 1062.38, "end": 1065.98, "text": " And basically the second step, the third step is to just do,"}, {"start": 1065.98, "end": 1069.02, "text": " as you can see here, conda and create."}, {"start": 1069.02, "end": 1071.42, "text": " And that's going to automatically create"}, {"start": 1071.42, "end": 1075.16, "text": " basically a new environment for you."}, {"start": 1075.16, "end": 1076.8600000000001, "text": " So because I already have an environment,"}, {"start": 1076.86, "end": 1080.9799999999998, "text": " I'm just going to basically do rename it,"}, {"start": 1080.9799999999998, "end": 1083.34, "text": " and then I'm going to rerun it here."}, {"start": 1083.34, "end": 1087.1399999999999, "text": " So conda and create, and this is going to create"}, {"start": 1087.1399999999999, "end": 1089.62, "text": " a new environment from scratch."}, {"start": 1089.62, "end": 1092.1399999999999, "text": " So I'm just doing this to make sure that"}, {"start": 1092.1399999999999, "end": 1093.82, "text": " if I'm not hitting any errors,"}, {"start": 1093.82, "end": 1098.26, "text": " then probably most other people using Windows will not."}, {"start": 1098.26, "end": 1099.8999999999999, "text": " So hopefully others as well."}, {"start": 1099.8999999999999, "end": 1101.86, "text": " If you notice any problems,"}, {"start": 1101.86, "end": 1104.5, "text": " feel free to submit an issue here."}, {"start": 1104.5, "end": 1107.42, "text": " I'll be tracking issues over the next couple of days."}, {"start": 1107.42, "end": 1109.34, "text": " I mean, the script is fairly simple,"}, {"start": 1109.34, "end": 1114.24, "text": " so hopefully it's going to be useful and easy to set up."}, {"start": 1114.24, "end": 1116.9, "text": " Okay guys, so here it is, the script executed."}, {"start": 1116.9, "end": 1121.5, "text": " I can just do the conda activate, blah, blah, blah."}, {"start": 1121.5, "end": 1123.74, "text": " We enter the SD playground."}, {"start": 1123.74, "end": 1126.62, "text": " And then the next step is to,"}, {"start": 1126.62, "end": 1130.56, "text": " you will have to run Hugging Face CLI login"}, {"start": 1130.56, "end": 1135.22, "text": " the first time before you start running the script,"}, {"start": 1135.22, "end": 1136.62, "text": " because otherwise you will not be able"}, {"start": 1136.62, "end": 1138.06, "text": " to access the model weights."}, {"start": 1138.06, "end": 1140.8, "text": " So this is everything you would do."}, {"start": 1140.8, "end": 1144.3799999999999, "text": " Basically copy this command, paste it here, execute it,"}, {"start": 1144.3799999999999, "end": 1146.6599999999999, "text": " and then it will probably ask you to again,"}, {"start": 1146.6599999999999, "end": 1148.8999999999999, "text": " copy that token from the website."}, {"start": 1148.8999999999999, "end": 1150.36, "text": " You can just copy paste it here,"}, {"start": 1150.36, "end": 1152.46, "text": " and everything is going to work as expected."}, {"start": 1152.46, "end": 1156.86, "text": " Okay guys, so now let's open up an ID of your choice."}, {"start": 1156.86, "end": 1158.02, "text": " I'm going to open up VS Code."}, {"start": 1158.02, "end": 1160.5, "text": " That's my favorite ID by far."}, {"start": 1160.5, "end": 1165.5, "text": " So let me try and open up the corresponding directory."}, {"start": 1166.34, "end": 1169.46, "text": " So we just open up this one, I guess."}, {"start": 1169.46, "end": 1172.74, "text": " And I just trust the author, blah, blah, blah."}, {"start": 1172.74, "end": 1174.7, "text": " I say trust the author because the author,"}, {"start": 1174.7, "end": 1176.08, "text": " I am the author, man."}, {"start": 1176.08, "end": 1177.7, "text": " What's up with this?"}, {"start": 1177.7, "end": 1179.38, "text": " Okay, so generate images."}, {"start": 1179.38, "end": 1181.94, "text": " Here is the main script."}, {"start": 1181.94, "end": 1184.5, "text": " I'm going to walk you through it right now."}, {"start": 1184.5, "end": 1187.34, "text": " Before that, let me just create, basically,"}, {"start": 1187.34, "end": 1191.26, "text": " let me just create a JSON,"}, {"start": 1191.26, "end": 1193.6, "text": " long JSON for the Python file."}, {"start": 1193.6, "end": 1194.8999999999999, "text": " You don't even need to do this"}, {"start": 1194.8999999999999, "end": 1197.54, "text": " because we will not be passing any arguments"}, {"start": 1197.54, "end": 1198.56, "text": " through the long JSON."}, {"start": 1198.56, "end": 1201.5, "text": " This is where you would usually set the arguments"}, {"start": 1201.5, "end": 1203.0, "text": " in case you need them."}, {"start": 1203.0, "end": 1204.06, "text": " But here I don't need them,"}, {"start": 1204.06, "end": 1207.6, "text": " so I'm just going to leave them as empty here, okay?"}, {"start": 1207.6, "end": 1209.02, "text": " So this is a script."}, {"start": 1209.02, "end": 1212.8999999999999, "text": " Basically, I grabbed this function from Karpathy."}, {"start": 1212.8999999999999, "end": 1215.86, "text": " Basically, I already previously implemented it"}, {"start": 1215.86, "end": 1217.8999999999999, "text": " when I was developing my GAN projects."}, {"start": 1217.8999999999999, "end": 1220.1, "text": " You can check out my code there,"}, {"start": 1220.1, "end": 1224.12, "text": " but this looks neater and supports Torch"}, {"start": 1224.12, "end": 1225.3, "text": " and NumPy conversion,"}, {"start": 1225.3, "end": 1227.1, "text": " so I just kind of copy-pasted his version."}, {"start": 1227.1, "end": 1230.1799999999998, "text": " But it's a fairly similar thing to implement."}, {"start": 1230.1799999999998, "end": 1232.06, "text": " You can just take a look at the Wiki page"}, {"start": 1232.06, "end": 1235.9399999999998, "text": " and copy-paste everything to your,"}, {"start": 1235.9399999999998, "end": 1239.4599999999998, "text": " I mean, translate from human description to code."}, {"start": 1239.4599999999998, "end": 1240.74, "text": " Fairly trivial."}, {"start": 1240.74, "end": 1243.6999999999998, "text": " Okay, so I'm going to set a breakpoint here."}, {"start": 1243.7, "end": 1247.6200000000001, "text": " I'm going to pick a corresponding interpreter"}, {"start": 1247.6200000000001, "end": 1250.7, "text": " we just created, so st-playground-2."}, {"start": 1250.7, "end": 1252.22, "text": " So you pick the playground,"}, {"start": 1252.22, "end": 1255.46, "text": " so we pick the environment we just created."}, {"start": 1255.46, "end": 1257.78, "text": " And that should make us,"}, {"start": 1259.02, "end": 1261.14, "text": " that should make this work out of the box."}, {"start": 1261.14, "end": 1263.44, "text": " Sometimes you have to restart VS Code"}, {"start": 1263.44, "end": 1267.26, "text": " or whatever ID you use in order for the changes to be,"}, {"start": 1267.26, "end": 1270.82, "text": " and for the interpreter to be correctly loaded."}, {"start": 1270.82, "end": 1273.8999999999999, "text": " But let's see whether now it's going to crash or not."}, {"start": 1273.8999999999999, "end": 1277.8999999999999, "text": " So I'm going to basically just run, hit run here,"}, {"start": 1277.8999999999999, "end": 1280.4199999999998, "text": " and hopefully if everything has worked correctly,"}, {"start": 1280.4199999999998, "end": 1283.26, "text": " I'm going to just hit this breakpoint"}, {"start": 1283.26, "end": 1284.86, "text": " and everything should work."}, {"start": 1284.86, "end": 1287.4199999999998, "text": " Okay, let's see whether that's the case, and it is."}, {"start": 1287.4199999999998, "end": 1288.26, "text": " Awesome."}, {"start": 1288.26, "end": 1291.26, "text": " So I'm going to now walk you through the script."}, {"start": 1292.4199999999998, "end": 1293.7, "text": " It's fairly simple."}, {"start": 1293.7, "end": 1297.82, "text": " The first thing I will do is actually I'm going to exit."}, {"start": 1297.82, "end": 1299.82, "text": " I'm going to exit the script."}, {"start": 1299.82, "end": 1302.26, "text": " I'm going to set generate diverse here."}, {"start": 1302.26, "end": 1305.3799999999999, "text": " So generate diverse is going to start generating"}, {"start": 1305.3799999999999, "end": 1308.12, "text": " diverse set of images with the prompt you specify."}, {"start": 1308.12, "end": 1309.86, "text": " So here is the prompt I've specified,"}, {"start": 1309.86, "end": 1313.74, "text": " a painting of an AI robot having an epiphany moment."}, {"start": 1313.74, "end": 1317.78, "text": " I'm going to save that to the output directory AI epiphany."}, {"start": 1317.78, "end": 1322.58, "text": " I'm going to use 50 steps, 7.5 guidance scale,"}, {"start": 1322.58, "end": 1325.1399999999999, "text": " and I'm going to generate, let's say only one image."}, {"start": 1325.1399999999999, "end": 1329.1, "text": " For the sake of this video, I don't want to make it too slow."}, {"start": 1329.1, "end": 1331.74, "text": " And everything else you can kind of leave."}, {"start": 1331.74, "end": 1334.02, "text": " You can even put none here."}, {"start": 1334.02, "end": 1335.98, "text": " The width and height are 512."}, {"start": 1335.98, "end": 1339.62, "text": " I'm using FP16 because my machine only has"}, {"start": 1339.62, "end": 1343.78, "text": " eight gigabytes of VRAM, and I can basically,"}, {"start": 1343.78, "end": 1345.54, "text": " I'm going to set none here."}, {"start": 1345.54, "end": 1348.3, "text": " This is, I'm going to explain you briefly what this is."}, {"start": 1348.3, "end": 1351.26, "text": " Basically, it helps you with reproducibility"}, {"start": 1351.26, "end": 1353.7199999999998, "text": " of this script, but we don't need it now."}, {"start": 1353.7199999999998, "end": 1357.02, "text": " So I'm just going to leave all of this as none."}, {"start": 1357.02, "end": 1360.46, "text": " So I'm going to put a comma there,"}, {"start": 1360.46, "end": 1361.54, "text": " and that's pretty much it."}, {"start": 1361.54, "end": 1365.74, "text": " Okay, so let's now rerun the script in the generate mode."}, {"start": 1365.74, "end": 1368.06, "text": " So I'm going to hit run here,"}, {"start": 1368.06, "end": 1369.9, "text": " and let's see what's going to happen."}, {"start": 1369.9, "end": 1372.46, "text": " Okay, so first things first, you do need the GPU"}, {"start": 1372.46, "end": 1374.78, "text": " to run this, otherwise it's going to take"}, {"start": 1374.78, "end": 1377.94, "text": " like painfully long time, and it's better for you"}, {"start": 1377.94, "end": 1381.86, "text": " to use the hugging face space than to wait here."}, {"start": 1381.86, "end": 1385.42, "text": " Yeah, so basically, first thing first is we want"}, {"start": 1385.42, "end": 1388.66, "text": " to make sure that the height and width are a multiply of eight."}, {"start": 1389.8200000000002, "end": 1394.1000000000001, "text": " Okay, so device is CUDA, we just set the seed here"}, {"start": 1394.1000000000001, "end": 1396.3600000000001, "text": " such that every time if I rerun this,"}, {"start": 1396.3600000000001, "end": 1398.46, "text": " I'll have the same latent generated."}, {"start": 1398.46, "end": 1401.02, "text": " If you don't want that, you can just set seed to none,"}, {"start": 1401.02, "end": 1402.72, "text": " then every time you rerun this,"}, {"start": 1402.72, "end": 1405.5600000000002, "text": " you'll get a completely new set of images."}, {"start": 1405.5600000000002, "end": 1407.5, "text": " Okay, so I do some logistics here."}, {"start": 1407.5, "end": 1411.3400000000001, "text": " Basically, I create a specific output file structure"}, {"start": 1411.3400000000001, "end": 1414.8000000000002, "text": " such that we are generating images in a suitable location."}, {"start": 1414.8, "end": 1416.74, "text": " I'm going to show you what I mean by that in a second."}, {"start": 1416.74, "end": 1418.86, "text": " So let me do it like this."}, {"start": 1418.86, "end": 1423.4199999999998, "text": " So here is the directory,"}, {"start": 1423.4199999999998, "end": 1426.46, "text": " and that's where we'll be generating."}, {"start": 1426.46, "end": 1429.54, "text": " You can see I generated these samples,"}, {"start": 1429.54, "end": 1432.3799999999999, "text": " so that's where the images will be stored."}, {"start": 1432.3799999999999, "end": 1435.06, "text": " We have meta, that's where the metadata that we use"}, {"start": 1435.06, "end": 1436.84, "text": " to generate the image will be stored."}, {"start": 1436.84, "end": 1439.18, "text": " And finally latent, that's going to be the actual"}, {"start": 1439.18, "end": 1442.4199999999998, "text": " latent representation that was fed into the pipeline"}, {"start": 1442.4199999999998, "end": 1444.5, "text": " to generate the image."}, {"start": 1444.5, "end": 1446.46, "text": " Okay, let's create the scheduler."}, {"start": 1446.46, "end": 1448.1, "text": " Again, if you don't know what this is,"}, {"start": 1448.1, "end": 1450.5, "text": " I suggest you watch my deep dive,"}, {"start": 1450.5, "end": 1452.22, "text": " then everything will be clear."}, {"start": 1452.22, "end": 1455.1, "text": " So we pick again the B14 version."}, {"start": 1455.1, "end": 1457.18, "text": " We pick because FP16 is set to true,"}, {"start": 1457.18, "end": 1460.08, "text": " we're going to use the FP16 weights."}, {"start": 1460.08, "end": 1463.14, "text": " We passed basically the scheduler,"}, {"start": 1463.14, "end": 1464.34, "text": " and we finally have the pipe."}, {"start": 1464.34, "end": 1467.1, "text": " So we'll take some time for the pipe to be constructed"}, {"start": 1467.1, "end": 1469.22, "text": " because it has to load a bunch of weights."}, {"start": 1469.22, "end": 1470.38, "text": " I'm going to show you in a second,"}, {"start": 1470.38, "end": 1473.46, "text": " let me show you how this is going to function."}, {"start": 1473.46, "end": 1475.54, "text": " Basically performance tab,"}, {"start": 1475.54, "end": 1478.06, "text": " and you can see how the memory is slowly increasing."}, {"start": 1478.06, "end": 1481.74, "text": " And after I execute the two device,"}, {"start": 1481.74, "end": 1483.54, "text": " you can see how the GPU will start,"}, {"start": 1484.42, "end": 1488.26, "text": " the GPU consumption, the VRAM consumption will increase."}, {"start": 1488.26, "end": 1491.18, "text": " That's going to hopefully happen very soon."}, {"start": 1491.18, "end": 1492.9, "text": " So let's take a look."}, {"start": 1492.9, "end": 1495.78, "text": " So I'm going to do F10 again here, F10."}, {"start": 1495.78, "end": 1499.46, "text": " And now you can see how the pipe was basically pushed"}, {"start": 1499.46, "end": 1500.4, "text": " onto the GPU."}, {"start": 1500.4, "end": 1503.6200000000001, "text": " You can see a spike here in the VRAM consumption."}, {"start": 1503.6200000000001, "end": 1506.74, "text": " Okay guys, so let's ignore that now."}, {"start": 1506.74, "end": 1509.94, "text": " I'm going to maximize this window again."}, {"start": 1509.94, "end": 1513.38, "text": " So we enter the branch with the generation."}, {"start": 1513.38, "end": 1516.14, "text": " And because I've set the number of images to one,"}, {"start": 1516.14, "end": 1518.3000000000002, "text": " we'll have only a single image generated."}, {"start": 1518.3000000000002, "end": 1519.14, "text": " So let's see what we do."}, {"start": 1519.14, "end": 1521.74, "text": " The first step is we generate the latent."}, {"start": 1521.74, "end": 1525.52, "text": " This is kind of hard coded because that's how the,"}, {"start": 1525.52, "end": 1527.7800000000002, "text": " this is what pipe is also expecting."}, {"start": 1527.7800000000002, "end": 1529.98, "text": " Like in general, you don't want to hard code"}, {"start": 1529.98, "end": 1534.24, "text": " the dimensions like this, but in this case, it just works."}, {"start": 1535.26, "end": 1536.88, "text": " Okay, we generate the latent."}, {"start": 1536.88, "end": 1539.18, "text": " And now I use the AutoCast,"}, {"start": 1539.18, "end": 1541.14, "text": " which is going to use Mixed Precision."}, {"start": 1541.14, "end": 1543.54, "text": " And in my particular case, it's just going to do, I guess,"}, {"start": 1543.54, "end": 1546.22, "text": " FP16 pretty much for all computations."}, {"start": 1546.22, "end": 1549.3, "text": " I'm not sure whether it's going to do FP32 accumulation"}, {"start": 1549.3, "end": 1553.84, "text": " for some of the operations, but basically that's it."}, {"start": 1553.84, "end": 1556.1, "text": " Okay, so now we run the pipe."}, {"start": 1556.1, "end": 1558.42, "text": " We pass the prompt, we pass the number of steps,"}, {"start": 1558.42, "end": 1559.8600000000001, "text": " the latent and the guidance scale."}, {"start": 1559.86, "end": 1561.26, "text": " And out comes the image."}, {"start": 1562.78, "end": 1564.54, "text": " So as you can see so far, so good."}, {"start": 1564.54, "end": 1566.9599999999998, "text": " We didn't have any errors with the script."}, {"start": 1566.9599999999998, "end": 1570.86, "text": " Again, I encourage you, if you find any errors,"}, {"start": 1570.86, "end": 1573.34, "text": " basically submit an issue to my GitHub repo."}, {"start": 1573.34, "end": 1574.86, "text": " That would be super useful."}, {"start": 1574.86, "end": 1576.4599999999998, "text": " Okay, let's continue here."}, {"start": 1576.4599999999998, "end": 1577.6799999999998, "text": " Here is the image."}, {"start": 1578.82, "end": 1581.5, "text": " We now just basically store the image."}, {"start": 1581.5, "end": 1582.9799999999998, "text": " So let's see how that's going to happen."}, {"start": 1582.9799999999998, "end": 1584.24, "text": " So we have samples here."}, {"start": 1584.24, "end": 1588.84, "text": " After I run this, we have this neat generate name function"}, {"start": 1588.84, "end": 1592.58, "text": " that's going to automatically detect all of the images"}, {"start": 1592.58, "end": 1595.9399999999998, "text": " inside of this folder and make sure not to overwrite"}, {"start": 1595.9399999999998, "end": 1596.98, "text": " the existing images."}, {"start": 1596.98, "end": 1599.3, "text": " Instead, it's gonna increment the file name"}, {"start": 1599.3, "end": 1600.6999999999998, "text": " and just store it there."}, {"start": 1600.6999999999998, "end": 1602.78, "text": " So I'm gonna show you the script a bit later,"}, {"start": 1602.78, "end": 1604.82, "text": " but for now, let's treat it as a black box."}, {"start": 1604.82, "end": 1606.06, "text": " So here is the image."}, {"start": 1606.06, "end": 1609.06, "text": " I'm gonna put extra large icons so you can see the results."}, {"start": 1609.06, "end": 1612.1799999999998, "text": " So here is the first result, amazing result."}, {"start": 1613.1399999999999, "end": 1615.1399999999999, "text": " Next up, I'm gonna save the metadata."}, {"start": 1615.1399999999999, "end": 1616.98, "text": " So when I say metadata, I mean,"}, {"start": 1616.98, "end": 1618.4199999999998, "text": " let's save the prompt information,"}, {"start": 1618.42, "end": 1620.8600000000001, "text": " how many steps we used to generate this image"}, {"start": 1620.8600000000001, "end": 1624.1000000000001, "text": " and what guidance scale we used to generate this image."}, {"start": 1624.1000000000001, "end": 1626.6200000000001, "text": " I'm gonna improve this script over the next days."}, {"start": 1626.6200000000001, "end": 1628.1000000000001, "text": " So feel free to contribute,"}, {"start": 1628.1000000000001, "end": 1629.8200000000002, "text": " feel free to create a pull request."}, {"start": 1629.8200000000002, "end": 1631.42, "text": " I'm going to be tracking what's going on"}, {"start": 1631.42, "end": 1633.42, "text": " with the repo over the next days."}, {"start": 1633.42, "end": 1636.3200000000002, "text": " So let me open up the meta directory here,"}, {"start": 1636.3200000000002, "end": 1639.1000000000001, "text": " and I'm expecting to generate a meta file there."}, {"start": 1639.1000000000001, "end": 1642.88, "text": " So if I hit F10, we can see here there is a JSON."}, {"start": 1642.88, "end": 1645.46, "text": " If I open it up, you can see that's basically"}, {"start": 1645.46, "end": 1648.54, "text": " storing the prompt information, the number of steps,"}, {"start": 1648.54, "end": 1649.9, "text": " and everything else."}, {"start": 1649.9, "end": 1652.94, "text": " I kinda have, wait, let me move this thingy."}, {"start": 1652.94, "end": 1654.26, "text": " Let me try and move this thingy."}, {"start": 1654.26, "end": 1655.78, "text": " Okay, so we have the scale information there."}, {"start": 1655.78, "end": 1659.3, "text": " Okay, so basically that's it."}, {"start": 1659.3, "end": 1661.52, "text": " Now, finally, I store the latents."}, {"start": 1661.52, "end": 1664.78, "text": " So that's the latent that was used to generate this image."}, {"start": 1664.78, "end": 1667.4, "text": " So here's the latent, I just push it onto the CPU,"}, {"start": 1667.4, "end": 1670.14, "text": " I convert it into NumPy, and then I store it here."}, {"start": 1670.14, "end": 1670.98, "text": " So that's it."}, {"start": 1672.06, "end": 1673.6200000000001, "text": " Let's hit F10."}, {"start": 1673.62, "end": 1677.6599999999999, "text": " Here is the latent, and okay, now I can step"}, {"start": 1678.7399999999998, "end": 1680.6599999999999, "text": " and exit the script."}, {"start": 1680.6599999999999, "end": 1681.5, "text": " So that's it."}, {"start": 1681.5, "end": 1686.1999999999998, "text": " That was how the script looks like in this generation mode."}, {"start": 1686.1999999999998, "end": 1688.58, "text": " There is a couple of different modes."}, {"start": 1688.58, "end": 1692.1, "text": " The second one you can use is reproduce mode."}, {"start": 1692.1, "end": 1695.54, "text": " So here, and this is more for playing"}, {"start": 1695.54, "end": 1697.62, "text": " and understanding the reproducibility"}, {"start": 1697.62, "end": 1701.32, "text": " than something that will help you with your art generation."}, {"start": 1701.32, "end": 1704.98, "text": " So what this does is basically now it expects me to pass,"}, {"start": 1704.98, "end": 1709.86, "text": " as you can see here, so now it expects me to pass"}, {"start": 1709.86, "end": 1712.62, "text": " the source latent path and the metadata path."}, {"start": 1712.62, "end": 1714.86, "text": " And using the latent and the metadata,"}, {"start": 1714.86, "end": 1717.62, "text": " it's going to regenerate the same image from scratch."}, {"start": 1717.62, "end": 1718.9399999999998, "text": " Okay, so that's the idea."}, {"start": 1718.9399999999998, "end": 1720.46, "text": " So let's do that."}, {"start": 1720.46, "end": 1722.82, "text": " I'm going to grab this latent."}, {"start": 1722.82, "end": 1724.78, "text": " I'm gonna just copy this path here."}, {"start": 1725.82, "end": 1729.3, "text": " I'm going to put that path here."}, {"start": 1729.3, "end": 1731.86, "text": " So here it is."}, {"start": 1731.86, "end": 1734.4199999999998, "text": " I'm just gonna do it like this."}, {"start": 1734.4199999999998, "end": 1737.1399999999999, "text": " Okay, comment this out."}, {"start": 1737.1399999999999, "end": 1738.98, "text": " I probably have, this sucks on Windows."}, {"start": 1738.98, "end": 1741.8999999999999, "text": " I'm not sure whether, oh my God,"}, {"start": 1741.8999999999999, "end": 1745.62, "text": " am I doing this really live?"}, {"start": 1745.62, "end": 1750.46, "text": " Okay, so let me put the backslash here"}, {"start": 1750.46, "end": 1752.94, "text": " so that this actually works."}, {"start": 1752.94, "end": 1755.9199999999998, "text": " Maybe it can work even without it, but yeah."}, {"start": 1755.92, "end": 1759.54, "text": " Okay, now I'm gonna copy the metadata path as well."}, {"start": 1759.54, "end": 1764.54, "text": " So just, oops, shift, right click, copy as path,"}, {"start": 1765.22, "end": 1768.14, "text": " and I'm gonna specify that thing here."}, {"start": 1768.14, "end": 1772.54, "text": " Okay, and that should be pretty much it."}, {"start": 1772.54, "end": 1775.42, "text": " Now I just have to, well, let's see whether"}, {"start": 1775.42, "end": 1776.7, "text": " this is gonna crash or not."}, {"start": 1776.7, "end": 1777.98, "text": " I'm curious, to be honest."}, {"start": 1777.98, "end": 1779.3400000000001, "text": " So this should now work."}, {"start": 1779.3400000000001, "end": 1783.22, "text": " I have reproduce mode and we can just click save here"}, {"start": 1783.22, "end": 1784.9, "text": " and let me run this script again."}, {"start": 1784.9, "end": 1787.22, "text": " So if I run this, okay guys,"}, {"start": 1787.22, "end": 1790.5800000000002, "text": " I did have to add the additional backslash after all,"}, {"start": 1790.5800000000002, "end": 1794.5, "text": " otherwise it's gonna make me, it's gonna complain."}, {"start": 1794.5, "end": 1796.3000000000002, "text": " Okay, so again, same logic here."}, {"start": 1796.3000000000002, "end": 1798.9, "text": " I'm gonna skip over all of this."}, {"start": 1798.9, "end": 1800.8600000000001, "text": " We generate the pipe, blah, blah, blah."}, {"start": 1800.8600000000001, "end": 1802.7800000000002, "text": " So let me just put the breakpoint here"}, {"start": 1802.7800000000002, "end": 1806.7800000000002, "text": " and let's hit F5 and let's get to this point in the code."}, {"start": 1806.7800000000002, "end": 1810.7800000000002, "text": " So again, you'll see how we're gonna reuse"}, {"start": 1810.7800000000002, "end": 1813.14, "text": " this particular metadata here"}, {"start": 1813.14, "end": 1815.5, "text": " and this particular latent here"}, {"start": 1815.5, "end": 1819.3400000000001, "text": " to generate the very same image that we saw here."}, {"start": 1819.3400000000001, "end": 1822.0800000000002, "text": " And additionally, I have a math plot live plot function"}, {"start": 1822.0800000000002, "end": 1824.3000000000002, "text": " being called, so this is gonna pop up"}, {"start": 1824.3000000000002, "end": 1829.16, "text": " after this script is done with the execution."}, {"start": 1829.16, "end": 1832.0200000000002, "text": " Okay, let's see whether we are there."}, {"start": 1832.0200000000002, "end": 1834.8600000000001, "text": " Here we are, we hit this breakpoint."}, {"start": 1834.8600000000001, "end": 1838.8200000000002, "text": " So I make sure that we are passing the needed paths."}, {"start": 1838.8200000000002, "end": 1840.38, "text": " So I open up the metadata."}, {"start": 1840.38, "end": 1842.3400000000001, "text": " You can see that the metadata contains"}, {"start": 1842.34, "end": 1844.34, "text": " exactly the information that was used"}, {"start": 1844.34, "end": 1848.02, "text": " to generate the image a couple of minutes ago."}, {"start": 1848.02, "end": 1851.6599999999999, "text": " So I load the latent here and now we end up,"}, {"start": 1851.6599999999999, "end": 1853.78, "text": " let me show you the shape of that thingy."}, {"start": 1853.78, "end": 1858.04, "text": " So init.shape, we end up with 464.64."}, {"start": 1858.04, "end": 1861.1, "text": " So that's the expected size of the latent space"}, {"start": 1861.1, "end": 1863.9399999999998, "text": " for the diffusion model for this particular one."}, {"start": 1863.9399999999998, "end": 1867.78, "text": " And now let's pass the same metadata, that latent,"}, {"start": 1867.78, "end": 1869.82, "text": " and let's generate a NumPy image."}, {"start": 1869.82, "end": 1872.24, "text": " So that's why I put output NumPy output type"}, {"start": 1872.24, "end": 1874.66, "text": " to NumPy, basically can be anything"}, {"start": 1874.66, "end": 1879.22, "text": " because the way how a pipe is implemented in the background,"}, {"start": 1879.22, "end": 1883.9, "text": " you just need to specify anything other than then pill"}, {"start": 1883.9, "end": 1887.14, "text": " and then it will generate a NumPy image as the output."}, {"start": 1887.14, "end": 1889.3, "text": " Okay, so here we are."}, {"start": 1889.3, "end": 1891.38, "text": " Finally, let's generate the image."}, {"start": 1891.38, "end": 1894.78, "text": " So image show, I'm gonna show you the image"}, {"start": 1894.78, "end": 1898.08, "text": " we generated here and you can see that,"}, {"start": 1898.08, "end": 1899.94, "text": " okay guys, so that failed because there was"}, {"start": 1899.94, "end": 1903.22, "text": " a very sneaky bug behind this code."}, {"start": 1903.22, "end": 1905.74, "text": " Not behind my code, but behind the version"}, {"start": 1905.74, "end": 1907.9, "text": " of the diffusers library I'm using."}, {"start": 1907.9, "end": 1911.8200000000002, "text": " So namely, if you take a look at the main branch,"}, {"start": 1911.8200000000002, "end": 1915.0, "text": " the latest commit in this pipeline stable diffusion file,"}, {"start": 1915.0, "end": 1917.7, "text": " so that's basically the call function"}, {"start": 1917.7, "end": 1920.1200000000001, "text": " behind the stable diffusion pipeline object,"}, {"start": 1920.1200000000001, "end": 1924.42, "text": " you can see that there is this lines 102 to 113."}, {"start": 1924.42, "end": 1927.42, "text": " You can see that the latent is being used here."}, {"start": 1927.42, "end": 1931.6200000000001, "text": " So if we pass the latent, then basically,"}, {"start": 1931.6200000000001, "end": 1933.94, "text": " we are going to use it."}, {"start": 1933.94, "end": 1937.8600000000001, "text": " Otherwise, if we pass none, then we generate it on the fly"}, {"start": 1937.8600000000001, "end": 1940.1000000000001, "text": " and there is a latent as well here"}, {"start": 1940.1000000000001, "end": 1943.26, "text": " as the argument to this call function."}, {"start": 1943.26, "end": 1947.8600000000001, "text": " Okay, on the other side, I had to manually add the latent"}, {"start": 1947.8600000000001, "end": 1951.18, "text": " here and I had to add this piece of code here."}, {"start": 1951.18, "end": 1952.1000000000001, "text": " So let me go back."}, {"start": 1952.1000000000001, "end": 1953.8200000000002, "text": " So I had to copy paste that part"}, {"start": 1953.8200000000002, "end": 1955.5, "text": " and I had to copy paste that part"}, {"start": 1955.5, "end": 1960.5, "text": " and now after adding that, manually patching the 0.2.4"}, {"start": 1960.66, "end": 1962.74, "text": " version of the library I currently have"}, {"start": 1962.74, "end": 1964.52, "text": " and that's the latest pip version they have."}, {"start": 1964.52, "end": 1966.54, "text": " So they still haven't pushed the latest changes"}, {"start": 1966.54, "end": 1969.1, "text": " into that pip package."}, {"start": 1969.1, "end": 1971.64, "text": " So after you do this fix, now everything works."}, {"start": 1971.64, "end": 1975.7, "text": " So basically, I went ahead and generated another image."}, {"start": 1975.7, "end": 1977.14, "text": " So let me show you what I mean by that."}, {"start": 1977.14, "end": 1980.58, "text": " So I went ahead and generated this image here"}, {"start": 1980.58, "end": 1983.86, "text": " and now if I pick its metadata,"}, {"start": 1983.86, "end": 1988.54, "text": " so we have its metadata 0.0002, same for the latent."}, {"start": 1988.54, "end": 1993.54, "text": " So if I go and specify that here, 0002,"}, {"start": 1994.2199999999998, "end": 1999.06, "text": " and if I just set the reproduce mode, if I hit run,"}, {"start": 1999.06, "end": 2001.78, "text": " now I'm gonna let it execute and I'm gonna show you"}, {"start": 2001.78, "end": 2006.78, "text": " the results as soon as we get to this break point here."}, {"start": 2007.6999999999998, "end": 2009.4799999999998, "text": " Okay guys, the moment of truth."}, {"start": 2009.4799999999998, "end": 2012.76, "text": " Let's basically display the image"}, {"start": 2012.76, "end": 2015.22, "text": " and you can see that's exactly the same image"}, {"start": 2015.22, "end": 2016.98, "text": " as what we had here."}, {"start": 2016.98, "end": 2019.5, "text": " So that's the same image as what we had here."}, {"start": 2019.5, "end": 2021.46, "text": " Let me do a side by side comparison"}, {"start": 2021.46, "end": 2026.34, "text": " and we have perfect reproducibility of this thingy."}, {"start": 2026.34, "end": 2028.98, "text": " Okay, that's very cool."}, {"start": 2028.98, "end": 2030.86, "text": " The last thing I wanna show you"}, {"start": 2030.86, "end": 2033.7, "text": " is the interpolation execution mode."}, {"start": 2033.7, "end": 2036.6, "text": " So that's the funny part where we basically,"}, {"start": 2036.6, "end": 2041.06, "text": " so I'm gonna do interpolate here and here in that case,"}, {"start": 2041.06, "end": 2042.54, "text": " there is a couple of options."}, {"start": 2042.54, "end": 2045.1, "text": " So that's the third branch of the code here."}, {"start": 2045.1, "end": 2047.86, "text": " Basically, if you don't specify the paths,"}, {"start": 2047.86, "end": 2050.98, "text": " then we just generate two random latent vectors."}, {"start": 2050.98, "end": 2053.7799999999997, "text": " Those two latent vectors correspond to two images"}, {"start": 2053.7799999999997, "end": 2056.7, "text": " and then we're gonna smoothly, spherically interpolate"}, {"start": 2056.7, "end": 2059.62, "text": " between those images and generate everything in between."}, {"start": 2059.62, "end": 2061.34, "text": " So that's if you don't specify the latency."}, {"start": 2061.34, "end": 2062.7799999999997, "text": " If you do specify the latency,"}, {"start": 2062.7799999999997, "end": 2064.82, "text": " if you find two images that you like,"}, {"start": 2064.82, "end": 2068.5, "text": " just find basically the numpy, the associated numpy,"}, {"start": 2068.5, "end": 2070.86, "text": " the latents and pass the paths here"}, {"start": 2070.86, "end": 2073.54, "text": " and then you can generate, you can basically,"}, {"start": 2073.54, "end": 2075.58, "text": " well, you can have more control over"}, {"start": 2075.58, "end": 2078.06, "text": " between which images are you interpolating."}, {"start": 2078.06, "end": 2079.1800000000003, "text": " That's the basic idea."}, {"start": 2079.1800000000003, "end": 2082.42, "text": " Okay, so now I'm going to do this route."}, {"start": 2082.42, "end": 2084.5, "text": " You can always play with this one,"}, {"start": 2084.5, "end": 2087.06, "text": " but I'm gonna specify both latents"}, {"start": 2087.06, "end": 2088.94, "text": " and let's then interpolate"}, {"start": 2088.94, "end": 2090.86, "text": " and let me show you what's going on."}, {"start": 2090.86, "end": 2093.1400000000003, "text": " So let's do the following thing."}, {"start": 2093.1400000000003, "end": 2095.78, "text": " So I'm going to interpolate between this robot"}, {"start": 2095.78, "end": 2097.2000000000003, "text": " and this robot here."}, {"start": 2097.2, "end": 2101.4399999999996, "text": " So to do that, we need to set one as the source,"}, {"start": 2101.4399999999996, "end": 2102.48, "text": " one as the target."}, {"start": 2102.48, "end": 2104.96, "text": " So because this is one is already set as the source,"}, {"start": 2104.96, "end": 2107.3199999999997, "text": " I'm gonna take this one and put it as a target."}, {"start": 2107.3199999999997, "end": 2109.3599999999997, "text": " So basically that means we need to take"}, {"start": 2109.3599999999997, "end": 2112.8999999999996, "text": " its corresponding latent and that's gonna be easy"}, {"start": 2112.8999999999996, "end": 2115.04, "text": " because of the structure I'm using."}, {"start": 2115.04, "end": 2116.8399999999997, "text": " So that means I just have to do the following."}, {"start": 2116.8399999999997, "end": 2121.8399999999997, "text": " I just have to take the source latent, which is 002."}, {"start": 2122.6, "end": 2126.3199999999997, "text": " I need to paste it here."}, {"start": 2126.32, "end": 2128.52, "text": " I need to set it to one instead."}, {"start": 2128.52, "end": 2131.8, "text": " And now we have that robot set as the target"}, {"start": 2131.8, "end": 2133.6400000000003, "text": " and that's pretty much it."}, {"start": 2133.6400000000003, "end": 2136.1600000000003, "text": " The metadata will be ignored"}, {"start": 2136.1600000000003, "end": 2139.2000000000003, "text": " and I think we just need to, yeah, that's pretty much it."}, {"start": 2139.2000000000003, "end": 2141.32, "text": " Now we're gonna have source latent."}, {"start": 2141.32, "end": 2142.84, "text": " Let me just check."}, {"start": 2142.84, "end": 2146.34, "text": " And we have the target latent."}, {"start": 2146.34, "end": 2150.2000000000003, "text": " We load them and then we are gonna be generating the images."}, {"start": 2150.2000000000003, "end": 2152.44, "text": " Cool, I'm gonna set a break point here."}, {"start": 2152.44, "end": 2155.84, "text": " I'm gonna hit execute and I'm gonna get back to you"}, {"start": 2155.84, "end": 2157.7200000000003, "text": " once we hit this break point."}, {"start": 2157.7200000000003, "end": 2159.92, "text": " Okay guys, so let's enter."}, {"start": 2159.92, "end": 2163.48, "text": " Hopefully we have both paths specified correctly."}, {"start": 2163.48, "end": 2166.06, "text": " I'm gonna load those latents."}, {"start": 2167.06, "end": 2169.52, "text": " We save the metadata again."}, {"start": 2171.1600000000003, "end": 2173.1200000000003, "text": " And we save these latents,"}, {"start": 2173.1200000000003, "end": 2177.32, "text": " which is probably less relevant in this example"}, {"start": 2177.32, "end": 2178.92, "text": " because I already had them saved."}, {"start": 2178.92, "end": 2181.6400000000003, "text": " So probably worth updating the script there."}, {"start": 2181.6400000000003, "end": 2183.46, "text": " And now let's see what I'm doing."}, {"start": 2183.46, "end": 2185.2000000000003, "text": " So basically you can see here,"}, {"start": 2185.2, "end": 2186.2799999999997, "text": " I'm doing a lint space."}, {"start": 2186.2799999999997, "end": 2189.3999999999996, "text": " So between zero and one, I want to generate 50 images."}, {"start": 2189.3999999999996, "end": 2192.6, "text": " So I had to set that number of images to 50 here."}, {"start": 2192.6, "end": 2195.74, "text": " And additionally prepend this zero just as a simple hack."}, {"start": 2195.74, "end": 2199.4199999999996, "text": " And the reason being is I want to, for the first iteration,"}, {"start": 2199.4199999999996, "end": 2201.68, "text": " I want to generate the target image"}, {"start": 2201.68, "end": 2206.2, "text": " such that I'm confident that the target image I've chosen"}, {"start": 2206.2, "end": 2207.3199999999997, "text": " is the correct one."}, {"start": 2207.3199999999997, "end": 2209.2, "text": " And then I'm gonna generate the source one"}, {"start": 2209.2, "end": 2211.6, "text": " and then it's gonna do the smooth interpolation"}, {"start": 2211.6, "end": 2213.6, "text": " from source to target, if that makes sense."}, {"start": 2213.6, "end": 2217.64, "text": " Okay, so that's why in the first iteration we enter here,"}, {"start": 2217.64, "end": 2221.02, "text": " we grab the target images latent."}, {"start": 2221.02, "end": 2222.7599999999998, "text": " And now I'm gonna generate it."}, {"start": 2222.7599999999998, "end": 2224.86, "text": " So let's just do this same logic."}, {"start": 2224.86, "end": 2226.14, "text": " We just passed a prompt,"}, {"start": 2226.14, "end": 2227.96, "text": " we passed the number of inference steps,"}, {"start": 2227.96, "end": 2230.36, "text": " we passed the latent, the guidance scale and everything."}, {"start": 2230.36, "end": 2231.9, "text": " And then we're gonna save the image."}, {"start": 2231.9, "end": 2234.7599999999998, "text": " So let me show you that indeed,"}, {"start": 2235.92, "end": 2238.62, "text": " so if I recall correctly,"}, {"start": 2238.62, "end": 2242.48, "text": " we use which image did we use as the source,"}, {"start": 2242.48, "end": 2244.06, "text": " just a second."}, {"start": 2244.06, "end": 2248.08, "text": " So we use as the target, we are using the image one."}, {"start": 2248.08, "end": 2249.76, "text": " Okay, so the target is this one."}, {"start": 2249.76, "end": 2253.16, "text": " So I'm expecting to generate yet another image"}, {"start": 2253.16, "end": 2255.04, "text": " that looks like that."}, {"start": 2255.04, "end": 2257.2400000000002, "text": " So let's verify that."}, {"start": 2257.2400000000002, "end": 2258.76, "text": " So I'm gonna hit here."}, {"start": 2258.76, "end": 2263.12, "text": " And if I save, okay, I actually get the different image."}, {"start": 2263.12, "end": 2265.4, "text": " And the reason is because of the bug I had,"}, {"start": 2265.4, "end": 2267.58, "text": " basically those metadata that were saved"}, {"start": 2267.58, "end": 2270.36, "text": " for these first two images because of the bug,"}, {"start": 2270.36, "end": 2272.96, "text": " it was actually corresponding to this image."}, {"start": 2272.96, "end": 2276.1, "text": " So now the second one should be this one."}, {"start": 2276.1, "end": 2277.34, "text": " And if that's not correct,"}, {"start": 2277.34, "end": 2279.26, "text": " then something weird is going on,"}, {"start": 2279.26, "end": 2281.0, "text": " but I'm fairly sure that's not gonna happen."}, {"start": 2281.0, "end": 2283.88, "text": " So I'm gonna hit F5 and get back to you"}, {"start": 2283.88, "end": 2286.02, "text": " once this image is generated."}, {"start": 2287.0, "end": 2288.1, "text": " So that's the source."}, {"start": 2288.1, "end": 2290.6800000000003, "text": " Okay, so let's do F10."}, {"start": 2290.6800000000003, "end": 2292.6800000000003, "text": " And you can see here, okay,"}, {"start": 2292.6800000000003, "end": 2294.36, "text": " now we're starting the interpolation."}, {"start": 2294.36, "end": 2296.7200000000003, "text": " I'll get back to you once all of the images are generated,"}, {"start": 2296.7200000000003, "end": 2298.08, "text": " we'll see a smooth interpolation"}, {"start": 2298.08, "end": 2299.56, "text": " between this image as a source"}, {"start": 2299.56, "end": 2300.84, "text": " and this image as a target."}, {"start": 2300.84, "end": 2302.48, "text": " Okay, guys, I've let it run"}, {"start": 2302.48, "end": 2304.92, "text": " and here you can see the results."}, {"start": 2304.92, "end": 2306.24, "text": " Fairly cool."}, {"start": 2306.24, "end": 2308.56, "text": " So here is again the source image."}, {"start": 2308.56, "end": 2310.24, "text": " Here is the target image."}, {"start": 2310.24, "end": 2313.64, "text": " Again, apologize for the bug there, but yeah."}, {"start": 2313.64, "end": 2315.96, "text": " You will not have to, by the way, sort it out"}, {"start": 2315.96, "end": 2317.9, "text": " because as soon as the patch is submitted"}, {"start": 2317.9, "end": 2319.24, "text": " to the diffusers library,"}, {"start": 2319.24, "end": 2322.2799999999997, "text": " you can just use the script without having to do the patch."}, {"start": 2322.2799999999997, "end": 2324.4, "text": " But for now, if you try and use the script,"}, {"start": 2324.4, "end": 2327.36, "text": " you will have to apply the patch before that."}, {"start": 2327.36, "end": 2330.28, "text": " Okay, so let's see how this is gonna look like."}, {"start": 2331.1200000000003, "end": 2333.32, "text": " Let's start with this image"}, {"start": 2333.32, "end": 2335.32, "text": " and let's slowly go to the right."}, {"start": 2335.32, "end": 2337.1800000000003, "text": " You can see how the image is morphing."}, {"start": 2338.2000000000003, "end": 2339.26, "text": " It's amazing."}, {"start": 2340.42, "end": 2345.32, "text": " And then it's slowly morphing to the final robot"}, {"start": 2345.32, "end": 2346.1600000000003, "text": " and that's it."}, {"start": 2346.1600000000003, "end": 2348.1200000000003, "text": " So there are some jumps in the latent space."}, {"start": 2348.1200000000003, "end": 2349.76, "text": " That's always the case."}, {"start": 2349.76, "end": 2353.96, "text": " And I had to use like maybe 200 or more images"}, {"start": 2353.96, "end": 2356.76, "text": " if I were to have a smoother interpolation"}, {"start": 2356.76, "end": 2358.36, "text": " over these couple of last steps."}, {"start": 2358.36, "end": 2360.36, "text": " So you can see there are some sudden jumps here."}, {"start": 2360.36, "end": 2363.28, "text": " But yeah, guys, that's pretty much it."}, {"start": 2363.28, "end": 2365.38, "text": " I showed you how to use the script."}, {"start": 2365.38, "end": 2367.5200000000004, "text": " There is the small gotcha I just explained"}, {"start": 2367.5200000000004, "end": 2369.36, "text": " about having to patch this file,"}, {"start": 2369.36, "end": 2373.0400000000004, "text": " but everything else should work as expected."}, {"start": 2373.0400000000004, "end": 2374.88, "text": " Hopefully you like this video."}, {"start": 2374.88, "end": 2378.2000000000003, "text": " If you try and generate some cool images with this script,"}, {"start": 2378.2000000000003, "end": 2380.6400000000003, "text": " feel free to tag me on Twitter or on LinkedIn"}, {"start": 2380.6400000000003, "end": 2384.0, "text": " and I'll be happy to see your images and reply."}, {"start": 2384.0, "end": 2386.6000000000004, "text": " And yeah, again, I'll be updating the script"}, {"start": 2386.6, "end": 2387.8399999999997, "text": " for the next couple of days,"}, {"start": 2387.8399999999997, "end": 2389.96, "text": " especially if somebody submits an issue,"}, {"start": 2389.96, "end": 2392.02, "text": " a bug somebody finds or something,"}, {"start": 2392.02, "end": 2395.48, "text": " I'm gonna try and address it as soon as possible."}, {"start": 2395.48, "end": 2397.6, "text": " Having said that, if you like this video,"}, {"start": 2397.6, "end": 2399.52, "text": " share it out with your friends,"}, {"start": 2399.52, "end": 2402.04, "text": " try and generate cool stuff yourself"}, {"start": 2402.04, "end": 2405.0, "text": " and subscribe to this YouTube channel, of course."}, {"start": 2405.0, "end": 2406.64, "text": " Until next time, bye bye."}, {"start": 2406.64, "end": 2418.46, "text": " Often when you miss an issue, I try and watch it again."}]