DALL-E: Zero-Shot Text-to-Image Generation | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I cover DALL-E or "Zero-Shot Text-to-Image Generation" paper by OpenAI team.

They train a VQ-VAE to learn compressed image representations and then they train an autoregressive transformer on top of that discrete latent space and BPEd text.

The model learns to combine distinct concepts in a plausible way, image to image capabilities emerge, etc.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2102.12092

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 

00:00 What is DALL-E?
03:25 VQ-VAE blur problems
05:15 transformers, transformers, transformers!
07:10 Stage 1 and Stage 2 explained
07:30 Stage 1 VQ-VAE recap
10:00 Stage 2 autoregressive transformer
10:45 Some notes on ELBO
13:05 VQ-VAE modifications
17:20 Stage 2 in-depth
23:00 Results
24:25 Engineering, engineering, engineering
25:40 Automatic filtering via CLIP
27:40 More results
32:00 Additional image to image translation examples

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️


Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#dalle #openai #generativemodeling
 What's up? In this video I'm covering zero shot text to image generation from the OpenAI team or Dali for short. And this work came out a couple of months ago, but I didn't see anybody covering the paper, so I thought walking you through and explaining in detail how this thing works. And let me just briefly explain what Dali is all about. As you can see here, Dali can synthesize these awesome images using just textual prompts. So on the left side here we have... so this is the prompt you're using and inputting into Dali. And so a taper made of accordion, a taper with the texture of an accordion. And you can see that Dali is able to remarkably mix up these concepts into something that looks like a plausible drawing that a human would. So a human could combine these concepts as well. So here we can see some kind of like a merge between the accordion and taper. Here we can see that the accordion has... so this handle is kind of in the taper shaped head. And yeah, it's pretty awesome. Some other examples involve this one. So an illustration of a baby hedgehog in a Christmas sweater walking a dog. And just like focusing maybe on this one, it looks super amazing. Like you can imagine this being used to generate ad hoc logos or gifts, whatever. So huge potential use cases for this model. Obviously it does make some mistakes at moments. So here we can see just a small hedgehog instead of a dog. So here the face is kind of blurry. So it's not perfect, but it's a huge leap forward. And I mean, I'm loving it. Here we have a neon sign that reads backprop, a neon sign that reads backprop, backprop, neon sign. So obviously you can see there is a lot of like prompt engineering. This is something that somebody called it on Twitter, like programming 3.0. So basically you have programming 1.0, that's your classical software engineering. Then you had programming 2.0, that's machine learning, where your data is your program. And here you're finally starting to just kind of hack the prompts in order to get the desired result from your model. And it looks like an emerging paradigm. And we'll be seeing more and more of this with GPT models as well, et cetera. So you can see results are pretty cool. You can see backprop here rendered. And I mean, pretty awesome. And finally, there are some like this emerging property of image translation as well. So basically they see here, so the exact same cat on the top as a sketch on the bottom. And in this example here, you can see it's, I mean, it got perfectly right. So it kind of just extracted the edges from this picture of a cat here and looks pretty awesome. On the other examples, it's not that perfect. But yeah, I mean, the very fact that it can actually do this without being explicitly trained to do these kind of stuff. So this is just an emerging property, same as we had in GPT models where we had we basically trained the model to predict the next token. We got like something like machine translation as a side effect, et cetera. So yeah, same thing happens here with the lead. Basically, and having mentioned GPT, you can assume that transformers and this is open AI. So you can just assume there is a transformer behind this. And we'll get to that in a moment. Let me just kind of touch on the one of the components of this of this of this model. So basically we have so Dali consists out of two parts. So one is VQVA model, which, by the way, I've covered in one of my previous videos, as well as VQGAM paper, which is super similar to Dali paper, except for the GAM component. So I do recommend you check those out. I'll just link them somewhere here. But having said that, I'll briefly explain what VQVA is in this video. So here I just want to point out one thing. So VQVA is basically an auto encoder like so it's a VA with like a quantized discrete latent space. And it has this like encoder like structure. Right. So we have a bottleneck there and then we reconstruct back the image. So this is your input image goes here. They reconstructed the image. Let's call it R comes out here. So here you can see that this is the input image and after it goes through the whole VQVA pipeline outcomes, this kind of blurry image. So the global composition is still here. Everything is still here. But you can see that it's blurry. And that's something you don't see with GANs. GANs have crispy, clear output. And here as well. So we have this text here and it's kind of blurred out here in the in the reconstruction. And similarly for these squiggly lines, some of them even get lost. So you can see here we focus on this one. You can see that like it kind of totally disappeared. OK. So it's not perfect. But like the reason they do this is so they can model because it's really hard to model the image directly in the in the in the pixel space. That's the whole idea. And I'll get to that in a moment. OK. So I mentioned transformers. Let's see. So this is open and I don't forget. So when compute model size and data are scaled carefully or regressive transformers. So this is the original transformer paper, which I've covered in one of my videos. You can check it out as well. You have achieved impressive results in several domains such as text. So this is your these are the GPT family of models. So GPT one, two and three images. So this is your image GPT from OpenAI and audio. So this is Jukebox paper from OpenAI as well. So you can see a trend here. What they do is they take a different modality. So, for example, text. They take a transformer. They scale the data. They scale the transformer and they get awesome results. And then we have images. Similar thing happened with images. So basically they also used a transformer. They modeled images directly in the pixel space. And the consequence was that they could only generate up to, I think, two like 64 times 64 images. Don't get don't I may be wrong here, but I think it's something like that. So in any case, it's a super small resolution compared to two. Nowadays we have generative models such as style game V2, where we have images that are like a million pixels, et cetera. So and finally we have audio. Again, they used a similar approach to this paper. So again, I think they use VQVAE and a transformer. They scaled it up and they generate they managed to generate some nice like music, et cetera. OK, so question is they ask is could data set size and model size be the limiting factor of current approaches? So here we're modeling both the text and the image images jointly. And so they say they train 12 billion parameter or aggressive transformer on 250 million image text pairs. So that's it. So data plus big transformers and you get awesome results. That's that's the main like kind of leitmotif here. OK, so briefly explaining the details of this like pipeline. They have two stages in the first stage. They train the VQVAE model in the second stage. They train this all regressive transformer directly in the latent discrete latent space of the VQVAE. So let's kind of just briefly touch on it. As I said, I have a whole video on this. You can check it out. But let me just kind of briefly explain it. So how you train this thing is you have an image, you pass it through this. This will be some kind of a CNN, some kind of encoder outcome. These these vectors you'll have, for example, this will be like one vector here. And just this will be the other dimensions along this side. And what you do is you kind of snap that that so these will be just like a collection of feature maps. Right. So you just take this vector and you'll find the closest one in this in this codebook of vectors. So this is just your embedding table. You find the closest one according to L2 distance and you snap it to that one. So you can see, for example, if this one was maybe the closest codebook vector was one here, you can see we snap it to one. So we have just a simple one here. And then later you can just kind of index using these indices into the codebook vector and you can actually find the actual vector. So you'll put E1 here. So that's this vector. So this vector will replace this one. And then you pass these these vectors, these quantized vectors into the coder, which is also just some kind of transposed convolutions. And you get the CNL, you get the reconstructed image back. So the loss is fairly simple. They just have reconstruction loss. So they used MSC. So that's mean squared error. Plus, they have some loss that encourages encourages the codebook vectors to be close to these vectors and these vectors to be close to codebook vectors. So the one difficulty is it's hard to it's not hard. It's impossible to actually backprop through this thing because of this quantization step. And so what they do is they whatever the gradients for these vectors here are, they just copy paste those. As you can see this by this red line, you just copy paste those into the encoder. And then because this part is differentiable, you can just backprop through the CNN and figure out all of the other gradients. But that's a difficulty. So this part, because of the quantization, that's one of the main things they had to solve. And yeah, that's that's in a nutshell how it works. So in my VQVAE video, I even went through some code. So if you're still confused how this exactly work, you can check it out. But that's the whole like a like a rough idea of how VQVAE works. So once you train this model, so what you have is you have you train these embedding vectors, you train the encoder, you train the decoder. And so now the second step is this transformer thing. So we concatenate up to 256 pp encoded. So that's by pairing coding text tokens with the 32 by 32. That's 1024 image tokens and train an autoregressive transformer to model the joint distribution over the text and image tokens. So this 32 by 32, let me just show you show you where that comes up. So basically this means that this in here will have 32 vectors here and 32 here. So we'll have 1024 tokens like in total. And these are image tokens. So those kind of capture the information about the image and then using decoder, you can just decode them back into the pixel space. OK, there's this part about where dimension elbow. I think it's it will be more confusing than than than good to explain this. But if you ignore why for a second, so if we ignore the labels, the captions, sorry, if we ignore the captions, this is basically your your VAE objective. So just elbow. Basically, this is the LN. So of p of x given that that's your reconstruction loss. And then depending how you decide to model your your data. So if you take Gaussian here, so basically what will happen is because Gaussian has some some some structure or something like this. So we have C, we have e raised to the power of blah, blah, blah. We have basically x minus mu squared over something. Sigma's doesn't matter. But basically when you do LN of this, this is a constant. So we don't care. This is E. These two are so LN and E are just inverse of each other. So you end up with x minus mu squared, which is your basically MSC loss. So that's how you end up with MSC loss, even though you have this abstraction here. And I know this part confused me. So hopefully this will like give you some epiphany moment. And similarly here, this is just a KL divergence between the approximate posterior and your prior. So that's basically your your your like VAE objective. And you're trying to to maximize this part because in turn, because it's a lower bound, that means you're you're going to maximize the like the likelihood of your data. So you just kind of want to maximize the reconstruction probability and you want to minimize the KL divergence between those two. That's just in a nutshell. I don't want to get into maths. As I said, the video has a better explanation of this of this component, as well as some links to some cool blocks, which you can check out at your own pace and understand the mathematics behind this. But the confusing part is you actually won't see the KL part, especially in VQVAE model. The reason being is that the KL divergence is constant in this model. So it's kind of confusing to you. It's a nice thing to think about it like this. But like for practice from a practical standpoint, it's just confusing. OK, so a couple of details where VQGAN and like VQVAE papers differ from DALI. So I mentioned the straight through estimator. So that's this part. Let me go back here. So this is called the straight through estimator. This red line, this is basically your you copy pasting the gradients from the decoder over to the encoder so that you can back prop through the encoder and train your model. OK, so that's the straight through gradient. Like a gradient estimator. And here, so they say we instead use the GUMBO softmax relaxation, replacing the expectation over blah, blah, blah, blah, blah. All in all, this is a modification they made as well as the likelihood for the P sub theta is evaluated using the log Laplace distribution. So I mentioned previously I mentioned this Gaussian here. So Gaussian leads to you having MSE as an objective in the loss function. This thing, because they're using log Laplace, they'll have some different objective. And let me briefly go to the appendix and show you what I mean by this. This is kind of different compared to other papers. And I think it's pretty neat idea. So the L1 and L2 reconstruction objectives are commonly used when training VA's. So L2 was what we saw. So MSE is basically L2 squared. And so these objectives correspond to using Laplace and Gaussian distributions for the this term. And that's what I mentioned. So I explained the thing with Gaussian. Similarly, it goes with Laplace. You'll have the same logic behind it. So you'll just end up with instead of square, you just won't have the square component. And there is a strange mismatch in this in this modeling choice. Pixel values lie within a bounding interval. So your pixels will be in the like 0, 1 range, right? That's your normal RGB image. And they say here, but both of these distributions are supported by the entire real line. Hence, some amount of likelihood will be placed outside the admissible range of pixel values. And so I just kind of draw like based at the distributions here. So your output pixel values will be pretty much here from 0 to 1, no matter how you parameterize this Laplace. So if you put mew to be equal to 0.5, so somewhere here, you're still going to have some non zero probability being given to values which are outside of the 0, 1 range. So this is the range we care about, right? The 0, 1 range. And these distributions, these assumptions that we have that the output picks like random variables are model as Gaussians or Laplace simply do not make that much sense if you think about it. And that's why they replace this. So they say here. So we represent the variant of Laplace distribution, which is also supported by a bounded interval, blah, blah, blah. So this PDF is defined on 0, 1 and is given by this this PDF. So what it did is they just applied they say somewhere here a sigmoid. Let me just a sec. So yeah. So we consider the PDF for random variable obtained by applying the sigmoid function to Laplace distributed random variable. And so instead of using emacy, what you'll now use to in order to do to maximize the likelihood the likelihood of your data. So to increase to decrease the reconstruction loss, you just you'll just do log of this. So just Ellen of this term and whatever pops out. That's what your objective will be like. And so that's in contrast with emacy loss. OK. And just this is the sigmoid. You have domain goes from minus infinity to plus infinity. And we have the codomain, the image being from zero to one. And that's the reason why the random variable will now be constrained to be between zero and one. Like contrast that to Laplace, as I mentioned, it will have values which are outside of the bounded region we are interested in. So that was just a small, small glimpse into these details, which they they kind of change compared to the original paper, VQVA paper. Let me now get back to where we ended it. I'll skip the explanation of the gumbell softmax relaxation. I think it's just a detail and there is a whole paper behind this idea. So I have to skip it. But like in a nutshell, they replaced the straight through gradient estimator with this novel gumbell softmax relaxation method. And now let's jump into stage two. OK. So I mentioned that after we train the VQVA, we have a discrete space. And now we want to learn how to train a transformer on top of that, on top of those image tokens. And they say here, given a text image pair, we BPE encode the lower case caption using at most 256 tokens with walk up size of 16000 and encode the image using 32 by 32 tokens with walk up size 8K. OK. Finally, the text and image tokens are concatenated and modeled or aggressively as a single stream of data. Let me try and break this down for you if it's already not clear. But like what I do is so imagine you have you have a caption. So you have some like I'll draw something like this. So we have a caption that's your some that's some text and you'll have associated image with that caption. So something like this will have a squiggly man here, random stuff, whatever. So that's your image. And so what happens now is the following. So they first lowercase this. So let me just denote that as L, then they do BPE. So that means they'll basically tokenize the words into into some smaller units. And what they have is they have this walk up size of 16K. So that just basically means you have a huge table here and that table contains contains 16K vectors which are learned. OK, so now what this thing how this thing will work like is the following. So let's say you have a sentence like I don't know like monkey. So monkey, monkey, king. And then we'll have whatever like just three dots. OK, so this is your input sentence. And so BPE will just kind of break down your words maybe into some sub words like I know you'll have maybe mon you'll have key. You'll have I know maybe key I and K.I. And so what happens is you'll just take this sub word. You'll find the appropriate vector and you'll replace your sub word with this vector and et cetera, et cetera. And so what they say is they have max of 256 tokens. So that means if we have more than that, we'll just kind of truncate. If we have less, they have some special like tokens which they'll train for that particular purpose. Where when we have like those when we have like a less number than 256. But it's a minor detail. And so that's what happens with the text. So we have we end up with 256 tokens here. OK, now the following thing happens with the image. So this one just gets fed into the encoder of the VQVAE, which was previously trained in stage one and outcome 32 by 32 tokens. So basically this is 32 by 32 and you're kind of going to unroll these. And so this will be your your target. So basically you'll just put start of sentence token here and then you'll basically all start of image some special token. And then you'll just unroll this thing here. OK, so just unroll it here. And that's your input into your transformer. And what the output will be is the following. So this very same thing just shifted by one to the left. OK, so you'll have something like this and then you'll have end of sentence token or end of image in this case. OK, so that's it. And so now how the whole thing works is you have a huge transformer. I think they used I think like GPT-2 or something with some sparse attention maps. And so you'll have a transformer here and then you have your usual thing. You just pass this in, you transform it through the multiple layers of transformer and you'll want to predict these. So you'll want to predict this thing from the start of sentence token. So in this case, these are image tokens. But just imagine for a second this was like a textual token. In that case, if this was text like Monkey King, we'd have M here and we'd have M here. And then the second letter would be O. So we'd have O here. So as you can see, you want to predict from the like this special token, you want to predict M. From M, you want to predict O because O is the next one. So you're just trying to predict the next token, simple stuff pretty much. And you can do this in parallel. And that's how they train this whole thing. So this thing here will be 256 and this thing here will be 1024. And you just train this transformer to kind of learn how to do this next token prediction. And then how you actually use this later in inference is you just prepend this caption and you prepend this special token like this one here. And then you just start sampling from the model until you get 32 by 32 tokens. You just pass that through the decoder and out comes a novel image. So that caption can be obviously completely novel. So the model has never seen the caption during training like the examples I showed you with taper and hedgehogs. So yeah, that's how it works in a nutshell. Now, let me just show you some results they have here. So they compare obviously the Dali paper with different GAN baselines. So like DFGAN, DMGAN, TENSIONGAN, etc. And here we have, so this is the ground truth caption, a very cute cat lying by a big bike. And here we have the actual validation images. So the images that go with these captions. So that's the ground truth. And if you take a look at the results, you can see two things. So first thing is Dali has better outputs. Like it's just if we focus on this thing here, a living room with a TV on top of a stand with a guitars sitting next to you. And basically you can see this looks really cool, much better than GANs. The main difference is like VQVAE is, as we saw in the beginning of the video, has blurry output. That's why we have blurry kind of blurry results here, even though their composition is much better compared to GANs. Where GANs are kind of the chat programmers of the generative modeling world because they're super confident. They just output some values super crisp, even though it's complete BS. And so on the other hand, we have VQVAEs, which have better compositionality, but they have blurry results. That's something you can take off from this image. OK, let's now continue. One thing I want to mention is, and they actually explicitly mentioned it here, is getting to model the model to train in a 16 bit precision past one billion parameters. And if you remember, they have 12 billion parameters, so that they have 12 billion parameter transformer without diverging was the most challenging part of this project. This just makes you appreciate engineering so much because if you think about it, the ideas in this paper are nothing brand novel. Like VQGAN has super similar ideas. Jukebox, all of these papers are kind of variations of each other. There is nothing super novel here from the research standpoint, in my honest opinion. But like from the engineering standpoint, there is a lot of things you have to solve in order to get this to work. So one of the things was that they had to train this in 16 bit precision and they had problems with overflow. So they had to devise multiple techniques like this one here, and they also had to do some engineering hacks in order to kind of avoid the memory bottlenecks they had on their GPUs. So they couldn't fit the whole model. So they had to do this charting between multiple nodes, etc. So just keep that in mind. I wouldn't say there is anything super novel research wise in this paper. But just when you scale it up, when you collect the data, when you invest just a bunch of money, a bunch of engineering, like just a bunch of dev like ours, you get some awesome results. And I think that's a valuable lesson. We don't always have to just invent novel ideas. Sometimes we just have to scale up and kind of push to the limits the current work we have. What I have here is, if you're familiar with CLIP, if you're not, I also create a video on CLIP. You can check it out. But basically what I do here is they have this automatic way of filtering the output from the Dali. So, for example, you generate you generate 512 images from the Dali. So we have a bunch of images. This is one. This is second one. This is third one, etc. So you generate a bunch of them. And then you have this model called CLIP, which will let me just draw it like like a circle. So CLIP will just basically you input an image and you input a caption. So, for example, this one, a group of urinals is near the trees and CLIP will tell you. So this is CLIP. CLIP will tell you how likely is this image to be under that caption. Let's call it that way. So that means you can do explicit sorting of these images according to the scores that CLIP model gave them. And you just take you cherry pick the ones with high scores. And you can see the results here are pretty astounding. So if you just have if you just change a single image, you can see it's pretty random. You don't see even the urinals. Then when you have when you generate eight images and you pick the best one, the best one, according to CLIP, you get better results. And finally, if you have 512 images, this one looks much better. So this one looks way better compared to the ones below. OK, so, yeah, this just works. It's just a heuristic to kind of get some better results from these generating models. Same goes from some different captions. And that's the that's the rough idea. Again, check out the CLIP video if you're interested in learning about details, how that exactly works. OK, some additional results. They compare this to this DF can baseline. Yep, they just get better realism and accuracy results compared to the baseline. Nothing fancy there. Here they show some results, some zero shot samples on this CUB data set. This CUB data set basically consists out of birds. And the thing is that the original that huge data set I mentioned that the Dali model uses has like a bunch of images. But the distribution of those images and captions is completely different compared to this data set. And they show some not that great performance on this on this on this CUB data set. You can see some birds here. It's not perfect. And especially when you focus on these quantitative results, you can see that Dali is much worse compared to other baselines on that data set, because, as I said, it's out of distribution in a way so it doesn't perform that well. What they do here on these diagrams is they plot the inception score and the FID. So that's for shades, inception, distance. Those are just some metrics which people use in order to kind of try and quantify how good are the generated images, because the problem itself is ill-posed, same as Newell style transfer, same as deep dream, those kind of images. You kind of you don't have a clear cut way to determine which image is better. So these are some heuristics like IS and FID. And what they do is on the next axis, they have a blur kernel radius. So what they do is they kind of whatever is generated out of out of Dali and other baselines like GANs, they just kind of blurred them with increasingly bigger filters. So you have initially you'll have some so you have an image here. And hopefully you're familiar with this. This is just some basic image processing technique. So you can create a kernel like a Gaussian and you just kind of like slide over the image and apply the like the you just do the kernel operation, the same as in CNNs. You just do like element wise multiplication. You add them up. You kind of map that to an output pixel. And if you do that with a Gaussian kernel, it'll just kind of blur the image. And what they then do is they kind of start increasing these kernels. And so they have a bigger one. And so the bigger the kernel, the more blurrier the image will be. And they show that when they kind of blur the image, they get better results. So this is the Dali and you can see that higher is better. And so the bigger the blur kernel is, the better results we get from the Dali compared to other baselines. Here, FID should be smaller so you can see again that even here after just increasing the blur radius to one, we have better results compared to baselines. Why do this? The reason is, as I mentioned at the beginning of the video, VQVAs have problems because the outputs are not crispy. And so they suspect that these metrics may be penalizing them because of that. So in order to get on an even ground, they take the GAN output, they blur it so as to compare to kind of make it fair to Dali. And then they kind of calculate the FIDs and IS. I mean, this is kind of a way to, I guess, measure the, like, how high quality the composition of the images and put less focus on the actual details. I guess that's the whole point of this. On FID, they show that the inception score is much lower, which is a bad thing. You want to have high inception score. And they show that the FID is higher. So that means it's having problems with the CUP data set. As I already mentioned, CUP is kind of different. The distribution of data is different there. Okay. Here, pretty obvious fact. That's that when we increase the sample size for re-ranking, so that's the clip part, the more so if you have 512 generated images, that's going to be way better than you generating just a single image. As you can see here, the inception score goes up. The FID goes down. And so that means there is a sweet spot, maybe around 128 generated images where you can then automatically select the best one. And this just improves the results on both metrics, FID and IS. Okay. Those were the results. I guess that's pretty much it. Let me just go back to the appendix and show you one more thing. One more thing. So, yeah, you can see the patterns in the transformer. They have just different patterns there, not just simple causal masks. And this was introduced in their sparse transformer papers. You can check that out if you wish so. And I just want to show you some additional results on the image to image translation somewhere here. Okay. So the exact same cat on the top color, on the top colored red on the bottom. So you can see here again perfectly image to image translation, even though this task, so you now know how the Dali model is trained. It's just trained to first train VQVAE. Then you train the autoregressive transformer to just predict the next image token. And you can see that out comes this ability to do this kind of stuff as well. So here they mentioned two panel image of the exact same cat on the top, a photo of the cat on the bottom, the cat with sunglasses. And you can see again like just placing some sunglasses onto the cat. Really awesome results. I love this. Here with postcard, the exact same cat on the top as a postage stamp on the bottom. Pretty awesome results. And with that, I leave you here. Okay. That was it. Hopefully you like this video. Let me know what you think. Subscribe, share out this content if you like it. That's the best way you can support me. And until next time, bye bye.
[{"start": 0.0, "end": 9.0, "text": " What's up? In this video I'm covering zero shot text to image generation from the OpenAI team or Dali for short."}, {"start": 9.0, "end": 15.0, "text": " And this work came out a couple of months ago, but I didn't see anybody covering the paper,"}, {"start": 15.0, "end": 20.0, "text": " so I thought walking you through and explaining in detail how this thing works."}, {"start": 20.0, "end": 24.0, "text": " And let me just briefly explain what Dali is all about."}, {"start": 24.0, "end": 31.0, "text": " As you can see here, Dali can synthesize these awesome images using just textual prompts."}, {"start": 31.0, "end": 38.0, "text": " So on the left side here we have... so this is the prompt you're using and inputting into Dali."}, {"start": 38.0, "end": 43.0, "text": " And so a taper made of accordion, a taper with the texture of an accordion."}, {"start": 43.0, "end": 53.0, "text": " And you can see that Dali is able to remarkably mix up these concepts into something that looks like a plausible drawing that a human would."}, {"start": 53.0, "end": 55.0, "text": " So a human could combine these concepts as well."}, {"start": 55.0, "end": 60.0, "text": " So here we can see some kind of like a merge between the accordion and taper."}, {"start": 60.0, "end": 66.0, "text": " Here we can see that the accordion has... so this handle is kind of in the taper shaped head."}, {"start": 66.0, "end": 69.0, "text": " And yeah, it's pretty awesome."}, {"start": 69.0, "end": 71.0, "text": " Some other examples involve this one."}, {"start": 71.0, "end": 77.0, "text": " So an illustration of a baby hedgehog in a Christmas sweater walking a dog."}, {"start": 77.0, "end": 81.0, "text": " And just like focusing maybe on this one, it looks super amazing."}, {"start": 81.0, "end": 89.0, "text": " Like you can imagine this being used to generate ad hoc logos or gifts, whatever."}, {"start": 89.0, "end": 93.0, "text": " So huge potential use cases for this model."}, {"start": 93.0, "end": 95.0, "text": " Obviously it does make some mistakes at moments."}, {"start": 95.0, "end": 99.0, "text": " So here we can see just a small hedgehog instead of a dog."}, {"start": 99.0, "end": 102.0, "text": " So here the face is kind of blurry."}, {"start": 102.0, "end": 106.0, "text": " So it's not perfect, but it's a huge leap forward."}, {"start": 106.0, "end": 108.0, "text": " And I mean, I'm loving it."}, {"start": 108.0, "end": 115.0, "text": " Here we have a neon sign that reads backprop, a neon sign that reads backprop, backprop, neon sign."}, {"start": 115.0, "end": 119.0, "text": " So obviously you can see there is a lot of like prompt engineering."}, {"start": 119.0, "end": 124.0, "text": " This is something that somebody called it on Twitter, like programming 3.0."}, {"start": 124.0, "end": 127.0, "text": " So basically you have programming 1.0, that's your classical software engineering."}, {"start": 127.0, "end": 132.0, "text": " Then you had programming 2.0, that's machine learning, where your data is your program."}, {"start": 132.0, "end": 141.0, "text": " And here you're finally starting to just kind of hack the prompts in order to get the desired result from your model."}, {"start": 141.0, "end": 143.0, "text": " And it looks like an emerging paradigm."}, {"start": 143.0, "end": 147.0, "text": " And we'll be seeing more and more of this with GPT models as well, et cetera."}, {"start": 147.0, "end": 150.0, "text": " So you can see results are pretty cool."}, {"start": 150.0, "end": 152.0, "text": " You can see backprop here rendered."}, {"start": 152.0, "end": 154.0, "text": " And I mean, pretty awesome."}, {"start": 154.0, "end": 161.0, "text": " And finally, there are some like this emerging property of image translation as well."}, {"start": 161.0, "end": 169.0, "text": " So basically they see here, so the exact same cat on the top as a sketch on the bottom."}, {"start": 169.0, "end": 175.0, "text": " And in this example here, you can see it's, I mean, it got perfectly right."}, {"start": 175.0, "end": 181.0, "text": " So it kind of just extracted the edges from this picture of a cat here and looks pretty awesome."}, {"start": 181.0, "end": 183.0, "text": " On the other examples, it's not that perfect."}, {"start": 183.0, "end": 189.0, "text": " But yeah, I mean, the very fact that it can actually do this without being explicitly trained to do these kind of stuff."}, {"start": 189.0, "end": 196.0, "text": " So this is just an emerging property, same as we had in GPT models where we had we basically trained the model to predict the next token."}, {"start": 196.0, "end": 201.0, "text": " We got like something like machine translation as a side effect, et cetera."}, {"start": 201.0, "end": 204.0, "text": " So yeah, same thing happens here with the lead."}, {"start": 204.0, "end": 209.0, "text": " Basically, and having mentioned GPT, you can assume that transformers and this is open AI."}, {"start": 209.0, "end": 212.0, "text": " So you can just assume there is a transformer behind this."}, {"start": 212.0, "end": 214.0, "text": " And we'll get to that in a moment."}, {"start": 214.0, "end": 220.0, "text": " Let me just kind of touch on the one of the components of this of this of this model."}, {"start": 220.0, "end": 223.0, "text": " So basically we have so Dali consists out of two parts."}, {"start": 223.0, "end": 235.0, "text": " So one is VQVA model, which, by the way, I've covered in one of my previous videos, as well as VQGAM paper, which is super similar to Dali paper, except for the GAM component."}, {"start": 235.0, "end": 237.0, "text": " So I do recommend you check those out."}, {"start": 237.0, "end": 239.0, "text": " I'll just link them somewhere here."}, {"start": 239.0, "end": 243.0, "text": " But having said that, I'll briefly explain what VQVA is in this video."}, {"start": 243.0, "end": 246.0, "text": " So here I just want to point out one thing."}, {"start": 246.0, "end": 254.0, "text": " So VQVA is basically an auto encoder like so it's a VA with like a quantized discrete latent space."}, {"start": 254.0, "end": 257.0, "text": " And it has this like encoder like structure."}, {"start": 257.0, "end": 261.0, "text": " Right. So we have a bottleneck there and then we reconstruct back the image."}, {"start": 261.0, "end": 263.0, "text": " So this is your input image goes here."}, {"start": 263.0, "end": 264.0, "text": " They reconstructed the image."}, {"start": 264.0, "end": 267.0, "text": " Let's call it R comes out here."}, {"start": 267.0, "end": 275.0, "text": " So here you can see that this is the input image and after it goes through the whole VQVA pipeline outcomes, this kind of blurry image."}, {"start": 275.0, "end": 277.0, "text": " So the global composition is still here."}, {"start": 277.0, "end": 279.0, "text": " Everything is still here."}, {"start": 279.0, "end": 281.0, "text": " But you can see that it's blurry."}, {"start": 281.0, "end": 283.0, "text": " And that's something you don't see with GANs."}, {"start": 283.0, "end": 286.0, "text": " GANs have crispy, clear output."}, {"start": 286.0, "end": 287.0, "text": " And here as well."}, {"start": 287.0, "end": 291.0, "text": " So we have this text here and it's kind of blurred out here in the in the reconstruction."}, {"start": 291.0, "end": 296.0, "text": " And similarly for these squiggly lines, some of them even get lost."}, {"start": 296.0, "end": 298.0, "text": " So you can see here we focus on this one."}, {"start": 298.0, "end": 301.0, "text": " You can see that like it kind of totally disappeared."}, {"start": 301.0, "end": 303.0, "text": " OK. So it's not perfect."}, {"start": 303.0, "end": 311.0, "text": " But like the reason they do this is so they can model because it's really hard to model the image directly in the in the in the pixel space."}, {"start": 311.0, "end": 312.0, "text": " That's the whole idea."}, {"start": 312.0, "end": 314.0, "text": " And I'll get to that in a moment."}, {"start": 314.0, "end": 316.0, "text": " OK. So I mentioned transformers."}, {"start": 316.0, "end": 317.0, "text": " Let's see."}, {"start": 317.0, "end": 318.0, "text": " So this is open and I don't forget."}, {"start": 318.0, "end": 324.0, "text": " So when compute model size and data are scaled carefully or regressive transformers."}, {"start": 324.0, "end": 327.0, "text": " So this is the original transformer paper, which I've covered in one of my videos."}, {"start": 327.0, "end": 329.0, "text": " You can check it out as well."}, {"start": 329.0, "end": 332.0, "text": " You have achieved impressive results in several domains such as text."}, {"start": 332.0, "end": 335.0, "text": " So this is your these are the GPT family of models."}, {"start": 335.0, "end": 338.0, "text": " So GPT one, two and three images."}, {"start": 338.0, "end": 341.0, "text": " So this is your image GPT from OpenAI and audio."}, {"start": 341.0, "end": 345.0, "text": " So this is Jukebox paper from OpenAI as well."}, {"start": 345.0, "end": 346.0, "text": " So you can see a trend here."}, {"start": 346.0, "end": 349.0, "text": " What they do is they take a different modality."}, {"start": 349.0, "end": 351.0, "text": " So, for example, text."}, {"start": 351.0, "end": 352.0, "text": " They take a transformer."}, {"start": 352.0, "end": 353.0, "text": " They scale the data."}, {"start": 353.0, "end": 356.0, "text": " They scale the transformer and they get awesome results."}, {"start": 356.0, "end": 357.0, "text": " And then we have images."}, {"start": 357.0, "end": 359.0, "text": " Similar thing happened with images."}, {"start": 359.0, "end": 361.0, "text": " So basically they also used a transformer."}, {"start": 361.0, "end": 364.0, "text": " They modeled images directly in the pixel space."}, {"start": 364.0, "end": 370.0, "text": " And the consequence was that they could only generate up to, I think, two like 64 times 64 images."}, {"start": 370.0, "end": 374.0, "text": " Don't get don't I may be wrong here, but I think it's something like that."}, {"start": 374.0, "end": 378.0, "text": " So in any case, it's a super small resolution compared to two."}, {"start": 378.0, "end": 382.0, "text": " Nowadays we have generative models such as style game V2, where we have images that are like"}, {"start": 382.0, "end": 385.0, "text": " a million pixels, et cetera."}, {"start": 385.0, "end": 388.0, "text": " So and finally we have audio."}, {"start": 388.0, "end": 391.0, "text": " Again, they used a similar approach to this paper."}, {"start": 391.0, "end": 394.0, "text": " So again, I think they use VQVAE and a transformer."}, {"start": 394.0, "end": 400.0, "text": " They scaled it up and they generate they managed to generate some nice like music, et cetera."}, {"start": 400.0, "end": 408.0, "text": " OK, so question is they ask is could data set size and model size be the limiting factor of current approaches?"}, {"start": 408.0, "end": 412.0, "text": " So here we're modeling both the text and the image images jointly."}, {"start": 412.0, "end": 419.0, "text": " And so they say they train 12 billion parameter or aggressive transformer on 250 million image text pairs."}, {"start": 419.0, "end": 423.0, "text": " So that's it. So data plus big transformers and you get awesome results."}, {"start": 423.0, "end": 428.0, "text": " That's that's the main like kind of leitmotif here."}, {"start": 428.0, "end": 435.0, "text": " OK, so briefly explaining the details of this like pipeline."}, {"start": 435.0, "end": 442.0, "text": " They have two stages in the first stage. They train the VQVAE model in the second stage."}, {"start": 442.0, "end": 448.0, "text": " They train this all regressive transformer directly in the latent discrete latent space of the VQVAE."}, {"start": 448.0, "end": 450.0, "text": " So let's kind of just briefly touch on it."}, {"start": 450.0, "end": 453.0, "text": " As I said, I have a whole video on this. You can check it out."}, {"start": 453.0, "end": 455.0, "text": " But let me just kind of briefly explain it."}, {"start": 455.0, "end": 459.0, "text": " So how you train this thing is you have an image, you pass it through this."}, {"start": 459.0, "end": 463.0, "text": " This will be some kind of a CNN, some kind of encoder outcome."}, {"start": 463.0, "end": 469.0, "text": " These these vectors you'll have, for example, this will be like one vector here."}, {"start": 469.0, "end": 474.0, "text": " And just this will be the other dimensions along this side."}, {"start": 474.0, "end": 480.0, "text": " And what you do is you kind of snap that that so these will be just like a collection of feature maps."}, {"start": 480.0, "end": 486.0, "text": " Right. So you just take this vector and you'll find the closest one in this in this codebook of vectors."}, {"start": 486.0, "end": 488.0, "text": " So this is just your embedding table."}, {"start": 488.0, "end": 493.0, "text": " You find the closest one according to L2 distance and you snap it to that one."}, {"start": 493.0, "end": 500.0, "text": " So you can see, for example, if this one was maybe the closest codebook vector was one here, you can see we snap it to one."}, {"start": 500.0, "end": 502.0, "text": " So we have just a simple one here."}, {"start": 502.0, "end": 510.0, "text": " And then later you can just kind of index using these indices into the codebook vector and you can actually find the actual vector."}, {"start": 510.0, "end": 512.0, "text": " So you'll put E1 here. So that's this vector."}, {"start": 512.0, "end": 514.0, "text": " So this vector will replace this one."}, {"start": 514.0, "end": 523.0, "text": " And then you pass these these vectors, these quantized vectors into the coder, which is also just some kind of transposed convolutions."}, {"start": 523.0, "end": 527.0, "text": " And you get the CNL, you get the reconstructed image back."}, {"start": 527.0, "end": 530.0, "text": " So the loss is fairly simple."}, {"start": 530.0, "end": 532.0, "text": " They just have reconstruction loss."}, {"start": 532.0, "end": 533.0, "text": " So they used MSC."}, {"start": 533.0, "end": 534.0, "text": " So that's mean squared error."}, {"start": 534.0, "end": 543.0, "text": " Plus, they have some loss that encourages encourages the codebook vectors to be close to these vectors and these vectors to be close to codebook vectors."}, {"start": 543.0, "end": 548.0, "text": " So the one difficulty is it's hard to it's not hard."}, {"start": 548.0, "end": 554.0, "text": " It's impossible to actually backprop through this thing because of this quantization step."}, {"start": 554.0, "end": 561.0, "text": " And so what they do is they whatever the gradients for these vectors here are, they just copy paste those."}, {"start": 561.0, "end": 565.0, "text": " As you can see this by this red line, you just copy paste those into the encoder."}, {"start": 565.0, "end": 571.0, "text": " And then because this part is differentiable, you can just backprop through the CNN and figure out all of the other gradients."}, {"start": 571.0, "end": 573.0, "text": " But that's a difficulty."}, {"start": 573.0, "end": 577.0, "text": " So this part, because of the quantization, that's one of the main things they had to solve."}, {"start": 577.0, "end": 580.0, "text": " And yeah, that's that's in a nutshell how it works."}, {"start": 580.0, "end": 583.0, "text": " So in my VQVAE video, I even went through some code."}, {"start": 583.0, "end": 586.0, "text": " So if you're still confused how this exactly work, you can check it out."}, {"start": 586.0, "end": 591.0, "text": " But that's the whole like a like a rough idea of how VQVAE works."}, {"start": 591.0, "end": 599.0, "text": " So once you train this model, so what you have is you have you train these embedding vectors, you train the encoder, you train the decoder."}, {"start": 599.0, "end": 603.0, "text": " And so now the second step is this transformer thing."}, {"start": 603.0, "end": 606.0, "text": " So we concatenate up to 256 pp encoded."}, {"start": 606.0, "end": 611.0, "text": " So that's by pairing coding text tokens with the 32 by 32."}, {"start": 611.0, "end": 619.0, "text": " That's 1024 image tokens and train an autoregressive transformer to model the joint distribution over the text and image tokens."}, {"start": 619.0, "end": 623.0, "text": " So this 32 by 32, let me just show you show you where that comes up."}, {"start": 623.0, "end": 630.0, "text": " So basically this means that this in here will have 32 vectors here and 32 here."}, {"start": 630.0, "end": 635.0, "text": " So we'll have 1024 tokens like in total."}, {"start": 635.0, "end": 637.0, "text": " And these are image tokens."}, {"start": 637.0, "end": 646.0, "text": " So those kind of capture the information about the image and then using decoder, you can just decode them back into the pixel space."}, {"start": 646.0, "end": 649.0, "text": " OK, there's this part about where dimension elbow."}, {"start": 649.0, "end": 654.0, "text": " I think it's it will be more confusing than than than good to explain this."}, {"start": 654.0, "end": 664.0, "text": " But if you ignore why for a second, so if we ignore the labels, the captions, sorry, if we ignore the captions, this is basically your your VAE objective."}, {"start": 664.0, "end": 665.0, "text": " So just elbow."}, {"start": 665.0, "end": 667.0, "text": " Basically, this is the LN."}, {"start": 667.0, "end": 673.0, "text": " So of p of x given that that's your reconstruction loss."}, {"start": 673.0, "end": 677.0, "text": " And then depending how you decide to model your your data."}, {"start": 677.0, "end": 684.0, "text": " So if you take Gaussian here, so basically what will happen is because Gaussian has some some some structure or something like this."}, {"start": 684.0, "end": 688.0, "text": " So we have C, we have e raised to the power of blah, blah, blah."}, {"start": 688.0, "end": 695.0, "text": " We have basically x minus mu squared over something."}, {"start": 695.0, "end": 697.0, "text": " Sigma's doesn't matter."}, {"start": 697.0, "end": 699.0, "text": " But basically when you do LN of this, this is a constant."}, {"start": 699.0, "end": 700.0, "text": " So we don't care."}, {"start": 700.0, "end": 701.0, "text": " This is E."}, {"start": 701.0, "end": 705.0, "text": " These two are so LN and E are just inverse of each other."}, {"start": 705.0, "end": 710.0, "text": " So you end up with x minus mu squared, which is your basically MSC loss."}, {"start": 710.0, "end": 715.0, "text": " So that's how you end up with MSC loss, even though you have this abstraction here."}, {"start": 715.0, "end": 717.0, "text": " And I know this part confused me."}, {"start": 717.0, "end": 721.0, "text": " So hopefully this will like give you some epiphany moment."}, {"start": 721.0, "end": 729.0, "text": " And similarly here, this is just a KL divergence between the approximate posterior and your prior."}, {"start": 729.0, "end": 733.0, "text": " So that's basically your your your like VAE objective."}, {"start": 733.0, "end": 742.0, "text": " And you're trying to to maximize this part because in turn, because it's a lower bound, that means you're you're going to maximize the like the likelihood of your data."}, {"start": 742.0, "end": 750.0, "text": " So you just kind of want to maximize the reconstruction probability and you want to minimize the KL divergence between those two."}, {"start": 750.0, "end": 752.0, "text": " That's just in a nutshell."}, {"start": 752.0, "end": 753.0, "text": " I don't want to get into maths."}, {"start": 753.0, "end": 762.0, "text": " As I said, the video has a better explanation of this of this component, as well as some links to some cool blocks, which you can check out at your own pace and understand the mathematics behind this."}, {"start": 762.0, "end": 769.0, "text": " But the confusing part is you actually won't see the KL part, especially in VQVAE model."}, {"start": 769.0, "end": 773.0, "text": " The reason being is that the KL divergence is constant in this model."}, {"start": 773.0, "end": 775.0, "text": " So it's kind of confusing to you."}, {"start": 775.0, "end": 777.0, "text": " It's a nice thing to think about it like this."}, {"start": 777.0, "end": 782.0, "text": " But like for practice from a practical standpoint, it's just confusing."}, {"start": 782.0, "end": 792.0, "text": " OK, so a couple of details where VQGAN and like VQVAE papers differ from DALI."}, {"start": 792.0, "end": 795.0, "text": " So I mentioned the straight through estimator."}, {"start": 795.0, "end": 797.0, "text": " So that's this part. Let me go back here."}, {"start": 797.0, "end": 799.0, "text": " So this is called the straight through estimator."}, {"start": 799.0, "end": 808.0, "text": " This red line, this is basically your you copy pasting the gradients from the decoder over to the encoder so that you can back prop through the encoder and train your model."}, {"start": 808.0, "end": 810.0, "text": " OK, so that's the straight through gradient."}, {"start": 810.0, "end": 812.0, "text": " Like a gradient estimator."}, {"start": 812.0, "end": 820.0, "text": " And here, so they say we instead use the GUMBO softmax relaxation, replacing the expectation over blah, blah, blah, blah, blah."}, {"start": 820.0, "end": 830.0, "text": " All in all, this is a modification they made as well as the likelihood for the P sub theta is evaluated using the log Laplace distribution."}, {"start": 830.0, "end": 833.0, "text": " So I mentioned previously I mentioned this Gaussian here."}, {"start": 833.0, "end": 839.0, "text": " So Gaussian leads to you having MSE as an objective in the loss function."}, {"start": 839.0, "end": 843.0, "text": " This thing, because they're using log Laplace, they'll have some different objective."}, {"start": 843.0, "end": 848.0, "text": " And let me briefly go to the appendix and show you what I mean by this."}, {"start": 848.0, "end": 850.0, "text": " This is kind of different compared to other papers."}, {"start": 850.0, "end": 852.0, "text": " And I think it's pretty neat idea."}, {"start": 852.0, "end": 857.0, "text": " So the L1 and L2 reconstruction objectives are commonly used when training VA's."}, {"start": 857.0, "end": 861.0, "text": " So L2 was what we saw. So MSE is basically L2 squared."}, {"start": 861.0, "end": 869.0, "text": " And so these objectives correspond to using Laplace and Gaussian distributions for the this term."}, {"start": 869.0, "end": 872.0, "text": " And that's what I mentioned. So I explained the thing with Gaussian."}, {"start": 872.0, "end": 876.0, "text": " Similarly, it goes with Laplace. You'll have the same logic behind it."}, {"start": 876.0, "end": 880.0, "text": " So you'll just end up with instead of square, you just won't have the square component."}, {"start": 880.0, "end": 885.0, "text": " And there is a strange mismatch in this in this modeling choice."}, {"start": 885.0, "end": 891.0, "text": " Pixel values lie within a bounding interval. So your pixels will be in the like 0, 1 range, right?"}, {"start": 891.0, "end": 894.0, "text": " That's your normal RGB image."}, {"start": 894.0, "end": 898.0, "text": " And they say here, but both of these distributions are supported by the entire real line."}, {"start": 898.0, "end": 903.0, "text": " Hence, some amount of likelihood will be placed outside the admissible range of pixel values."}, {"start": 903.0, "end": 908.0, "text": " And so I just kind of draw like based at the distributions here."}, {"start": 908.0, "end": 914.0, "text": " So your output pixel values will be pretty much here from 0 to 1, no matter how you parameterize this Laplace."}, {"start": 914.0, "end": 925.0, "text": " So if you put mew to be equal to 0.5, so somewhere here, you're still going to have some non zero probability being given to values which are outside of the 0, 1 range."}, {"start": 925.0, "end": 929.0, "text": " So this is the range we care about, right? The 0, 1 range."}, {"start": 929.0, "end": 940.0, "text": " And these distributions, these assumptions that we have that the output picks like random variables are model as Gaussians or Laplace simply do not make that much sense if you think about it."}, {"start": 940.0, "end": 945.0, "text": " And that's why they replace this. So they say here."}, {"start": 945.0, "end": 952.0, "text": " So we represent the variant of Laplace distribution, which is also supported by a bounded interval, blah, blah, blah."}, {"start": 952.0, "end": 957.0, "text": " So this PDF is defined on 0, 1 and is given by this this PDF."}, {"start": 957.0, "end": 961.0, "text": " So what it did is they just applied they say somewhere here a sigmoid."}, {"start": 961.0, "end": 963.0, "text": " Let me just a sec. So yeah."}, {"start": 963.0, "end": 971.0, "text": " So we consider the PDF for random variable obtained by applying the sigmoid function to Laplace distributed random variable."}, {"start": 971.0, "end": 981.0, "text": " And so instead of using emacy, what you'll now use to in order to do to maximize the likelihood the likelihood of your data."}, {"start": 981.0, "end": 986.0, "text": " So to increase to decrease the reconstruction loss, you just you'll just do log of this."}, {"start": 986.0, "end": 989.0, "text": " So just Ellen of this term and whatever pops out."}, {"start": 989.0, "end": 991.0, "text": " That's what your objective will be like."}, {"start": 991.0, "end": 994.0, "text": " And so that's in contrast with emacy loss."}, {"start": 994.0, "end": 996.0, "text": " OK. And just this is the sigmoid."}, {"start": 996.0, "end": 999.0, "text": " You have domain goes from minus infinity to plus infinity."}, {"start": 999.0, "end": 1003.0, "text": " And we have the codomain, the image being from zero to one."}, {"start": 1003.0, "end": 1008.0, "text": " And that's the reason why the random variable will now be constrained to be between zero and one."}, {"start": 1008.0, "end": 1015.0, "text": " Like contrast that to Laplace, as I mentioned, it will have values which are outside of the bounded region we are interested in."}, {"start": 1015.0, "end": 1026.0, "text": " So that was just a small, small glimpse into these details, which they they kind of change compared to the original paper, VQVA paper."}, {"start": 1026.0, "end": 1030.0, "text": " Let me now get back to where we ended it."}, {"start": 1030.0, "end": 1035.0, "text": " I'll skip the explanation of the gumbell softmax relaxation."}, {"start": 1035.0, "end": 1039.0, "text": " I think it's just a detail and there is a whole paper behind this idea."}, {"start": 1039.0, "end": 1040.0, "text": " So I have to skip it."}, {"start": 1040.0, "end": 1049.0, "text": " But like in a nutshell, they replaced the straight through gradient estimator with this novel gumbell softmax relaxation method."}, {"start": 1049.0, "end": 1052.0, "text": " And now let's jump into stage two."}, {"start": 1052.0, "end": 1057.0, "text": " OK. So I mentioned that after we train the VQVA, we have a discrete space."}, {"start": 1057.0, "end": 1064.0, "text": " And now we want to learn how to train a transformer on top of that, on top of those image tokens."}, {"start": 1064.0, "end": 1082.0, "text": " And they say here, given a text image pair, we BPE encode the lower case caption using at most 256 tokens with walk up size of 16000 and encode the image using 32 by 32 tokens with walk up size 8K."}, {"start": 1082.0, "end": 1089.0, "text": " OK. Finally, the text and image tokens are concatenated and modeled or aggressively as a single stream of data."}, {"start": 1089.0, "end": 1094.0, "text": " Let me try and break this down for you if it's already not clear."}, {"start": 1094.0, "end": 1099.0, "text": " But like what I do is so imagine you have you have a caption."}, {"start": 1099.0, "end": 1101.0, "text": " So you have some like I'll draw something like this."}, {"start": 1101.0, "end": 1107.0, "text": " So we have a caption that's your some that's some text and you'll have associated image with that caption."}, {"start": 1107.0, "end": 1115.0, "text": " So something like this will have a squiggly man here, random stuff, whatever."}, {"start": 1115.0, "end": 1118.0, "text": " So that's your image. And so what happens now is the following."}, {"start": 1118.0, "end": 1120.0, "text": " So they first lowercase this."}, {"start": 1120.0, "end": 1124.0, "text": " So let me just denote that as L, then they do BPE."}, {"start": 1124.0, "end": 1131.0, "text": " So that means they'll basically tokenize the words into into some smaller units."}, {"start": 1131.0, "end": 1134.0, "text": " And what they have is they have this walk up size of 16K."}, {"start": 1134.0, "end": 1145.0, "text": " So that just basically means you have a huge table here and that table contains contains 16K vectors which are learned."}, {"start": 1145.0, "end": 1149.0, "text": " OK, so now what this thing how this thing will work like is the following."}, {"start": 1149.0, "end": 1152.0, "text": " So let's say you have a sentence like I don't know like monkey."}, {"start": 1152.0, "end": 1158.0, "text": " So monkey, monkey, king."}, {"start": 1158.0, "end": 1162.0, "text": " And then we'll have whatever like just three dots."}, {"start": 1162.0, "end": 1164.0, "text": " OK, so this is your input sentence."}, {"start": 1164.0, "end": 1171.0, "text": " And so BPE will just kind of break down your words maybe into some sub words like I know you'll have maybe mon you'll have key."}, {"start": 1171.0, "end": 1177.0, "text": " You'll have I know maybe key I and K.I. And so what happens is you'll just take this sub word."}, {"start": 1177.0, "end": 1184.0, "text": " You'll find the appropriate vector and you'll replace your sub word with this vector and et cetera, et cetera."}, {"start": 1184.0, "end": 1188.0, "text": " And so what they say is they have max of 256 tokens."}, {"start": 1188.0, "end": 1192.0, "text": " So that means if we have more than that, we'll just kind of truncate."}, {"start": 1192.0, "end": 1198.0, "text": " If we have less, they have some special like tokens which they'll train for that particular purpose."}, {"start": 1198.0, "end": 1203.0, "text": " Where when we have like those when we have like a less number than 256."}, {"start": 1203.0, "end": 1207.0, "text": " But it's a minor detail. And so that's what happens with the text."}, {"start": 1207.0, "end": 1211.0, "text": " So we have we end up with 256 tokens here."}, {"start": 1211.0, "end": 1214.0, "text": " OK, now the following thing happens with the image."}, {"start": 1214.0, "end": 1225.0, "text": " So this one just gets fed into the encoder of the VQVAE, which was previously trained in stage one and outcome 32 by 32 tokens."}, {"start": 1225.0, "end": 1232.0, "text": " So basically this is 32 by 32 and you're kind of going to unroll these."}, {"start": 1232.0, "end": 1235.0, "text": " And so this will be your your target."}, {"start": 1235.0, "end": 1244.0, "text": " So basically you'll just put start of sentence token here and then you'll basically all start of image some special token."}, {"start": 1244.0, "end": 1246.0, "text": " And then you'll just unroll this thing here."}, {"start": 1246.0, "end": 1249.0, "text": " OK, so just unroll it here."}, {"start": 1249.0, "end": 1251.0, "text": " And that's your input into your transformer."}, {"start": 1251.0, "end": 1254.0, "text": " And what the output will be is the following."}, {"start": 1254.0, "end": 1258.0, "text": " So this very same thing just shifted by one to the left."}, {"start": 1258.0, "end": 1267.0, "text": " OK, so you'll have something like this and then you'll have end of sentence token or end of image in this case."}, {"start": 1267.0, "end": 1269.0, "text": " OK, so that's it."}, {"start": 1269.0, "end": 1274.0, "text": " And so now how the whole thing works is you have a huge transformer."}, {"start": 1274.0, "end": 1280.0, "text": " I think they used I think like GPT-2 or something with some sparse attention maps."}, {"start": 1280.0, "end": 1286.0, "text": " And so you'll have a transformer here and then you have your usual thing."}, {"start": 1286.0, "end": 1294.0, "text": " You just pass this in, you transform it through the multiple layers of transformer and you'll want to predict these."}, {"start": 1294.0, "end": 1299.0, "text": " So you'll want to predict this thing from the start of sentence token."}, {"start": 1299.0, "end": 1302.0, "text": " So in this case, these are image tokens."}, {"start": 1302.0, "end": 1305.0, "text": " But just imagine for a second this was like a textual token."}, {"start": 1305.0, "end": 1311.0, "text": " In that case, if this was text like Monkey King, we'd have M here and we'd have M here."}, {"start": 1311.0, "end": 1314.0, "text": " And then the second letter would be O."}, {"start": 1314.0, "end": 1316.0, "text": " So we'd have O here."}, {"start": 1316.0, "end": 1321.0, "text": " So as you can see, you want to predict from the like this special token, you want to predict M."}, {"start": 1321.0, "end": 1324.0, "text": " From M, you want to predict O because O is the next one."}, {"start": 1324.0, "end": 1328.0, "text": " So you're just trying to predict the next token, simple stuff pretty much."}, {"start": 1328.0, "end": 1330.0, "text": " And you can do this in parallel."}, {"start": 1330.0, "end": 1332.0, "text": " And that's how they train this whole thing."}, {"start": 1332.0, "end": 1341.0, "text": " So this thing here will be 256 and this thing here will be 1024."}, {"start": 1341.0, "end": 1347.0, "text": " And you just train this transformer to kind of learn how to do this next token prediction."}, {"start": 1347.0, "end": 1356.0, "text": " And then how you actually use this later in inference is you just prepend this caption and you prepend this special token like this one here."}, {"start": 1356.0, "end": 1361.0, "text": " And then you just start sampling from the model until you get 32 by 32 tokens."}, {"start": 1361.0, "end": 1365.0, "text": " You just pass that through the decoder and out comes a novel image."}, {"start": 1365.0, "end": 1368.0, "text": " So that caption can be obviously completely novel."}, {"start": 1368.0, "end": 1376.0, "text": " So the model has never seen the caption during training like the examples I showed you with taper and hedgehogs."}, {"start": 1376.0, "end": 1379.0, "text": " So yeah, that's how it works in a nutshell."}, {"start": 1379.0, "end": 1384.0, "text": " Now, let me just show you some results they have here."}, {"start": 1384.0, "end": 1392.0, "text": " So they compare obviously the Dali paper with different GAN baselines."}, {"start": 1392.0, "end": 1396.0, "text": " So like DFGAN, DMGAN, TENSIONGAN, etc."}, {"start": 1396.0, "end": 1402.0, "text": " And here we have, so this is the ground truth caption, a very cute cat lying by a big bike."}, {"start": 1402.0, "end": 1404.0, "text": " And here we have the actual validation images."}, {"start": 1404.0, "end": 1406.0, "text": " So the images that go with these captions."}, {"start": 1406.0, "end": 1407.0, "text": " So that's the ground truth."}, {"start": 1407.0, "end": 1411.0, "text": " And if you take a look at the results, you can see two things."}, {"start": 1411.0, "end": 1414.0, "text": " So first thing is Dali has better outputs."}, {"start": 1414.0, "end": 1422.0, "text": " Like it's just if we focus on this thing here, a living room with a TV on top of a stand with a guitars sitting next to you."}, {"start": 1422.0, "end": 1427.0, "text": " And basically you can see this looks really cool, much better than GANs."}, {"start": 1427.0, "end": 1434.0, "text": " The main difference is like VQVAE is, as we saw in the beginning of the video, has blurry output."}, {"start": 1434.0, "end": 1440.0, "text": " That's why we have blurry kind of blurry results here, even though their composition is much better compared to GANs."}, {"start": 1440.0, "end": 1446.0, "text": " Where GANs are kind of the chat programmers of the generative modeling world because they're super confident."}, {"start": 1446.0, "end": 1450.0, "text": " They just output some values super crisp, even though it's complete BS."}, {"start": 1450.0, "end": 1456.0, "text": " And so on the other hand, we have VQVAEs, which have better compositionality, but they have blurry results."}, {"start": 1456.0, "end": 1460.0, "text": " That's something you can take off from this image."}, {"start": 1460.0, "end": 1463.0, "text": " OK, let's now continue."}, {"start": 1463.0, "end": 1474.0, "text": " One thing I want to mention is, and they actually explicitly mentioned it here, is getting to model the model to train in a 16 bit precision past one billion parameters."}, {"start": 1474.0, "end": 1485.0, "text": " And if you remember, they have 12 billion parameters, so that they have 12 billion parameter transformer without diverging was the most challenging part of this project."}, {"start": 1485.0, "end": 1492.0, "text": " This just makes you appreciate engineering so much because if you think about it, the ideas in this paper are nothing brand novel."}, {"start": 1492.0, "end": 1495.0, "text": " Like VQGAN has super similar ideas."}, {"start": 1495.0, "end": 1499.0, "text": " Jukebox, all of these papers are kind of variations of each other."}, {"start": 1499.0, "end": 1504.0, "text": " There is nothing super novel here from the research standpoint, in my honest opinion."}, {"start": 1504.0, "end": 1509.0, "text": " But like from the engineering standpoint, there is a lot of things you have to solve in order to get this to work."}, {"start": 1509.0, "end": 1515.0, "text": " So one of the things was that they had to train this in 16 bit precision and they had problems with overflow."}, {"start": 1515.0, "end": 1526.0, "text": " So they had to devise multiple techniques like this one here, and they also had to do some engineering hacks in order to kind of avoid the memory bottlenecks they had on their GPUs."}, {"start": 1526.0, "end": 1528.0, "text": " So they couldn't fit the whole model."}, {"start": 1528.0, "end": 1532.0, "text": " So they had to do this charting between multiple nodes, etc."}, {"start": 1532.0, "end": 1534.0, "text": " So just keep that in mind."}, {"start": 1534.0, "end": 1538.0, "text": " I wouldn't say there is anything super novel research wise in this paper."}, {"start": 1538.0, "end": 1548.0, "text": " But just when you scale it up, when you collect the data, when you invest just a bunch of money, a bunch of engineering, like just a bunch of dev like ours, you get some awesome results."}, {"start": 1548.0, "end": 1550.0, "text": " And I think that's a valuable lesson."}, {"start": 1550.0, "end": 1552.0, "text": " We don't always have to just invent novel ideas."}, {"start": 1552.0, "end": 1558.0, "text": " Sometimes we just have to scale up and kind of push to the limits the current work we have."}, {"start": 1558.0, "end": 1564.0, "text": " What I have here is, if you're familiar with CLIP, if you're not, I also create a video on CLIP."}, {"start": 1564.0, "end": 1565.0, "text": " You can check it out."}, {"start": 1565.0, "end": 1571.0, "text": " But basically what I do here is they have this automatic way of filtering the output from the Dali."}, {"start": 1571.0, "end": 1576.0, "text": " So, for example, you generate you generate 512 images from the Dali."}, {"start": 1576.0, "end": 1578.0, "text": " So we have a bunch of images."}, {"start": 1578.0, "end": 1580.0, "text": " This is one. This is second one."}, {"start": 1580.0, "end": 1582.0, "text": " This is third one, etc."}, {"start": 1582.0, "end": 1583.0, "text": " So you generate a bunch of them."}, {"start": 1583.0, "end": 1588.0, "text": " And then you have this model called CLIP, which will let me just draw it like like a circle."}, {"start": 1588.0, "end": 1593.0, "text": " So CLIP will just basically you input an image and you input a caption."}, {"start": 1593.0, "end": 1598.0, "text": " So, for example, this one, a group of urinals is near the trees and CLIP will tell you."}, {"start": 1598.0, "end": 1599.0, "text": " So this is CLIP."}, {"start": 1599.0, "end": 1604.0, "text": " CLIP will tell you how likely is this image to be under that caption."}, {"start": 1604.0, "end": 1605.0, "text": " Let's call it that way."}, {"start": 1605.0, "end": 1613.0, "text": " So that means you can do explicit sorting of these images according to the scores that CLIP model gave them."}, {"start": 1613.0, "end": 1616.0, "text": " And you just take you cherry pick the ones with high scores."}, {"start": 1616.0, "end": 1619.0, "text": " And you can see the results here are pretty astounding."}, {"start": 1619.0, "end": 1624.0, "text": " So if you just have if you just change a single image, you can see it's pretty random."}, {"start": 1624.0, "end": 1625.0, "text": " You don't see even the urinals."}, {"start": 1625.0, "end": 1631.0, "text": " Then when you have when you generate eight images and you pick the best one, the best one, according to CLIP, you get better results."}, {"start": 1631.0, "end": 1635.0, "text": " And finally, if you have 512 images, this one looks much better."}, {"start": 1635.0, "end": 1638.0, "text": " So this one looks way better compared to the ones below."}, {"start": 1638.0, "end": 1641.0, "text": " OK, so, yeah, this just works."}, {"start": 1641.0, "end": 1648.0, "text": " It's just a heuristic to kind of get some better results from these generating models."}, {"start": 1648.0, "end": 1650.0, "text": " Same goes from some different captions."}, {"start": 1650.0, "end": 1652.0, "text": " And that's the that's the rough idea."}, {"start": 1652.0, "end": 1657.0, "text": " Again, check out the CLIP video if you're interested in learning about details, how that exactly works."}, {"start": 1657.0, "end": 1660.0, "text": " OK, some additional results."}, {"start": 1660.0, "end": 1664.0, "text": " They compare this to this DF can baseline."}, {"start": 1664.0, "end": 1670.0, "text": " Yep, they just get better realism and accuracy results compared to the baseline."}, {"start": 1670.0, "end": 1672.0, "text": " Nothing fancy there."}, {"start": 1672.0, "end": 1677.0, "text": " Here they show some results, some zero shot samples on this CUB data set."}, {"start": 1677.0, "end": 1680.0, "text": " This CUB data set basically consists out of birds."}, {"start": 1680.0, "end": 1687.0, "text": " And the thing is that the original that huge data set I mentioned that the Dali model uses has like a bunch of images."}, {"start": 1687.0, "end": 1692.0, "text": " But the distribution of those images and captions is completely different compared to this data set."}, {"start": 1692.0, "end": 1697.0, "text": " And they show some not that great performance on this on this on this CUB data set."}, {"start": 1697.0, "end": 1699.0, "text": " You can see some birds here."}, {"start": 1699.0, "end": 1700.0, "text": " It's not perfect."}, {"start": 1700.0, "end": 1713.0, "text": " And especially when you focus on these quantitative results, you can see that Dali is much worse compared to other baselines on that data set, because, as I said, it's out of distribution in a way so it doesn't perform that well."}, {"start": 1713.0, "end": 1719.0, "text": " What they do here on these diagrams is they plot the inception score and the FID."}, {"start": 1719.0, "end": 1721.0, "text": " So that's for shades, inception, distance."}, {"start": 1721.0, "end": 1733.0, "text": " Those are just some metrics which people use in order to kind of try and quantify how good are the generated images, because the problem itself is ill-posed, same as Newell style transfer, same as deep dream, those kind of images."}, {"start": 1733.0, "end": 1738.0, "text": " You kind of you don't have a clear cut way to determine which image is better."}, {"start": 1738.0, "end": 1741.0, "text": " So these are some heuristics like IS and FID."}, {"start": 1741.0, "end": 1745.0, "text": " And what they do is on the next axis, they have a blur kernel radius."}, {"start": 1745.0, "end": 1755.0, "text": " So what they do is they kind of whatever is generated out of out of Dali and other baselines like GANs, they just kind of blurred them with increasingly bigger filters."}, {"start": 1755.0, "end": 1759.0, "text": " So you have initially you'll have some so you have an image here."}, {"start": 1759.0, "end": 1760.0, "text": " And hopefully you're familiar with this."}, {"start": 1760.0, "end": 1763.0, "text": " This is just some basic image processing technique."}, {"start": 1763.0, "end": 1774.0, "text": " So you can create a kernel like a Gaussian and you just kind of like slide over the image and apply the like the you just do the kernel operation, the same as in CNNs."}, {"start": 1774.0, "end": 1777.0, "text": " You just do like element wise multiplication."}, {"start": 1777.0, "end": 1779.0, "text": " You add them up. You kind of map that to an output pixel."}, {"start": 1779.0, "end": 1783.0, "text": " And if you do that with a Gaussian kernel, it'll just kind of blur the image."}, {"start": 1783.0, "end": 1787.0, "text": " And what they then do is they kind of start increasing these kernels."}, {"start": 1787.0, "end": 1790.0, "text": " And so they have a bigger one."}, {"start": 1790.0, "end": 1794.0, "text": " And so the bigger the kernel, the more blurrier the image will be."}, {"start": 1794.0, "end": 1800.0, "text": " And they show that when they kind of blur the image, they get better results."}, {"start": 1800.0, "end": 1804.0, "text": " So this is the Dali and you can see that higher is better."}, {"start": 1804.0, "end": 1810.0, "text": " And so the bigger the blur kernel is, the better results we get from the Dali compared to other baselines."}, {"start": 1810.0, "end": 1821.0, "text": " Here, FID should be smaller so you can see again that even here after just increasing the blur radius to one, we have better results compared to baselines."}, {"start": 1821.0, "end": 1822.0, "text": " Why do this?"}, {"start": 1822.0, "end": 1830.0, "text": " The reason is, as I mentioned at the beginning of the video, VQVAs have problems because the outputs are not crispy."}, {"start": 1830.0, "end": 1834.0, "text": " And so they suspect that these metrics may be penalizing them because of that."}, {"start": 1834.0, "end": 1844.0, "text": " So in order to get on an even ground, they take the GAN output, they blur it so as to compare to kind of make it fair to Dali."}, {"start": 1844.0, "end": 1847.0, "text": " And then they kind of calculate the FIDs and IS."}, {"start": 1847.0, "end": 1857.0, "text": " I mean, this is kind of a way to, I guess, measure the, like, how high quality the composition of the images and put less focus on the actual details."}, {"start": 1857.0, "end": 1859.0, "text": " I guess that's the whole point of this."}, {"start": 1859.0, "end": 1864.0, "text": " On FID, they show that the inception score is much lower, which is a bad thing."}, {"start": 1864.0, "end": 1866.0, "text": " You want to have high inception score."}, {"start": 1866.0, "end": 1868.0, "text": " And they show that the FID is higher."}, {"start": 1868.0, "end": 1871.0, "text": " So that means it's having problems with the CUP data set."}, {"start": 1871.0, "end": 1873.0, "text": " As I already mentioned, CUP is kind of different."}, {"start": 1873.0, "end": 1875.0, "text": " The distribution of data is different there."}, {"start": 1875.0, "end": 1877.0, "text": " Okay."}, {"start": 1877.0, "end": 1879.0, "text": " Here, pretty obvious fact."}, {"start": 1879.0, "end": 1891.0, "text": " That's that when we increase the sample size for re-ranking, so that's the clip part, the more so if you have 512 generated images, that's going to be way better than you generating just a single image."}, {"start": 1891.0, "end": 1893.0, "text": " As you can see here, the inception score goes up."}, {"start": 1893.0, "end": 1894.0, "text": " The FID goes down."}, {"start": 1894.0, "end": 1902.0, "text": " And so that means there is a sweet spot, maybe around 128 generated images where you can then automatically select the best one."}, {"start": 1902.0, "end": 1907.0, "text": " And this just improves the results on both metrics, FID and IS."}, {"start": 1907.0, "end": 1908.0, "text": " Okay."}, {"start": 1908.0, "end": 1909.0, "text": " Those were the results."}, {"start": 1909.0, "end": 1910.0, "text": " I guess that's pretty much it."}, {"start": 1910.0, "end": 1915.0, "text": " Let me just go back to the appendix and show you one more thing."}, {"start": 1915.0, "end": 1916.0, "text": " One more thing."}, {"start": 1916.0, "end": 1918.0, "text": " So, yeah, you can see the patterns in the transformer."}, {"start": 1918.0, "end": 1924.0, "text": " They have just different patterns there, not just simple causal masks."}, {"start": 1924.0, "end": 1926.0, "text": " And this was introduced in their sparse transformer papers."}, {"start": 1926.0, "end": 1929.0, "text": " You can check that out if you wish so."}, {"start": 1929.0, "end": 1934.0, "text": " And I just want to show you some additional results on the image to image translation somewhere here."}, {"start": 1934.0, "end": 1935.0, "text": " Okay."}, {"start": 1935.0, "end": 1940.0, "text": " So the exact same cat on the top color, on the top colored red on the bottom."}, {"start": 1940.0, "end": 1948.0, "text": " So you can see here again perfectly image to image translation, even though this task, so you now know how the Dali model is trained."}, {"start": 1948.0, "end": 1951.0, "text": " It's just trained to first train VQVAE."}, {"start": 1951.0, "end": 1955.0, "text": " Then you train the autoregressive transformer to just predict the next image token."}, {"start": 1955.0, "end": 1960.0, "text": " And you can see that out comes this ability to do this kind of stuff as well."}, {"start": 1960.0, "end": 1968.0, "text": " So here they mentioned two panel image of the exact same cat on the top, a photo of the cat on the bottom, the cat with sunglasses."}, {"start": 1968.0, "end": 1972.0, "text": " And you can see again like just placing some sunglasses onto the cat."}, {"start": 1972.0, "end": 1974.0, "text": " Really awesome results."}, {"start": 1974.0, "end": 1975.0, "text": " I love this."}, {"start": 1975.0, "end": 1981.0, "text": " Here with postcard, the exact same cat on the top as a postage stamp on the bottom."}, {"start": 1981.0, "end": 1982.0, "text": " Pretty awesome results."}, {"start": 1982.0, "end": 1985.0, "text": " And with that, I leave you here."}, {"start": 1985.0, "end": 1986.0, "text": " Okay. That was it."}, {"start": 1986.0, "end": 1987.0, "text": " Hopefully you like this video."}, {"start": 1987.0, "end": 1989.0, "text": " Let me know what you think."}, {"start": 1989.0, "end": 1991.0, "text": " Subscribe, share out this content if you like it."}, {"start": 1991.0, "end": 1993.0, "text": " That's the best way you can support me."}, {"start": 1993.0, "end": 2012.0, "text": " And until next time, bye bye."}]