VQ-GAN: Taming Transformers for High-Resolution Image Synthesis | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I cover VQ-GAN or Taming Transformers for High-Resolution Image Synthesis.

It uses modified VQ-VAEs and a powerful transformer (GPT-2) to synthesize high-res images.

An important modification of VQ-VAE they brought are:
1) changing MSE for perceptual loss
2) adding adversarial loss which makes the images way more crispy compared to the original VQ-VAE which had blurry outputs.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2012.09841
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
01:50 A high-level VQ-GAN overview
04:00 Perceptual loss
05:10 Patch-based adversarial loss
06:45 Sequence prediction via GPT
09:50 Generating high-res images
12:45 Loss explained in depth
16:15 Training the transformer
17:50 Conditioning transformer
20:45 Comparisons and results
22:00 Sampling strategies
23:00 Comparisons and results continued
25:00 Rejection sampling with ResNet or CLIP
26:45 Receptive field effects
28:30 Comparisons with DALL-E

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️


Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#vqvae #imagesynthesis #gpt
 What's up in this video I'm covering VQGAN or taming transformers for high resolution image synthesis paper by Patrick Esser, Robin Rhombach and Bjorn Omer from Heidelberg University. So as you can see it's all about image synthesis and what's exciting about this paper is they are the first to use transformers and achieve high resolution images as the one you can see here on the screen. And the thing about this paper is it directly builds off of VQVAE and that's a model that was previously developed by DeepMind and I've covered in my last video so do check it out if you don't know anything about it. But like aside from this paper, so aside from VQGAN, Dali from OpenAI also uses VQVAE as the backbone. So that's an important building block currently for these generative models and yeah, do check that video out. Okay, having said that let's see what this paper is all about. So I mentioned transformers and they say here we demonstrate how combining the effectiveness of the inductive bias of CNNs with the expressivity of transformers enables them to model and thereby synthesize higher resolution images. So this CNN part is just the encoder part of the VQVAE as we'll soon see. Okay. They say here that we hypothesize that low-level image structure is well described by local connectivity, i.e. convolutional architecture, whereas this structural assumption ceases to be effective on higher semantic levels. So the idea is to extract certain features which are kind of representative of what image constitutes of and then kind of use transformer to do this sequence token prediction as we'll soon see in order to generate and in order to generate like images which are globally coherent and that's where the attention pattern of the transformers which basically treats the underlying data as the fully connected graph comes like useful. Okay, so having said that let's see the high-level overview of the architecture. Okay, so first things first as you can probably recognize this bottom block here is VQVAE and the training itself of this VQGAN paper consists out of two stages. So very similar to VQVAE. Basically you first train the VQVAE module, so that's this one, and then you train a prior so in the VQVAE paper they use pixelCNN, here they're using transformers or more precisely they're using GPT-2 model from OpenAI. Okay, if you watch my previous video you'll see that this structure overall is very similar to the original VQVAE paper. So what I've done additionally here is they swapped pixelCNN and they're not now using state-of-the-art sequence modeling model and that's transformer GPT-2 in this case and they also improved upon the reconstruction loss. They are not using MSC, they're using perceptual loss plus adversarial loss instead. So there is a lot of overlap between the two projects, but like this project got a much better much better like imagery and high-res imagery. So yeah, let's see how it works. So again, you have the VQVAE component here. What you do is you have your image, you encode it, you get some latent representations and then you snap these latent vectors onto these codebook vectors so that you have this discrete representation here which you then feed back into the decoder in order to reconstruct the image. So that's the high level picture of how that works. And again, like just as a reminder this system is trained in two stages. The stage one is you learn to train this VQVAE model, you learn how to kind of create these high quality latent vectors and then once you have that you freeze everything, you freeze the decoder, you freeze the encoder, you freeze the codebook and you train the transformer across this discrete latent space. Okay, that's the high level idea of how the system is trained. So now let's see what are the differences between VQVAE and this paper. So the main difference here in the stage one is they are not using MSC. So they are instead using perceptual loss plus adversarial loss. So perceptual loss just it comes from the, to the best of my knowledge, it comes from the neural stall transfer literature. And so what you do is you basically take an image. So you'll take this original image, you'll feed it into a pre-trained CNN like maybe VGG net or something. So this is your CNN. You feed in the original image. So let's denote that as X and you feed in the reconstructed image and let's denote that as X hat. And what you then do, you just take a certain layer from this VGG, from this pre-trained CNN and you extract those feature maps. So that will be something like this. Okay, so that's gonna be, these are the channels height and width. Okay, and you do the same thing for the original image. So you also extract those features there and what you do then instead is you just apply MSC loss, but this time not in the image space, but across this feature space. That's the only difference. And the reason that functions is because the classifier already learned how to extract some valuable features and it's just unreasonably effective. Okay, and aside from that, they also have this adversarial component. So as you may know, the original VQVAE had a problem with blur and we already know that GANs kind of improved that. They make much more crispier image. So it was like actually the original authors of VQVAE paper also like suggested using GANs in the future work. So here it is. So they're using PatchGAN. So what it does, so what's the difference between that and your regular GAN? You usually have just a single output like full, like you have like something like a sigmoid activation function and then you're trying to train the discriminator to discriminate between real data and fake data. So here it's the same thing instead, but like the only difference is you're taking patches, like overlapping patches, like maybe this thing and then you'll have this and you'll have another patch and you have like four patches together and you can see it will, the discriminator will output some values here. So if this image was a real image, we'd expect it to output, for example, all of these should be R or 1, whatever that, like whatever the value, however you decide to encode the real values. And so basically train, let's say these are like 0 to 1 values, you'll just do your minus loss. So basically your cross entropy and you're going to train this patch-based discriminator the usual way. Okay, so I'll get into a bit more details about the GAN loss, but this is roughly how it works like. So they have patch-based GAN, all of that taken together, so the perceptual loss plus this GAN loss, so these two together give much more crispier image as a reconstruction here from this VQVA pipeline. Okay, the second step, once you train that, once you have these code vectors, what you do is you just train the transformer on a sequence prediction task. So how the procedure looks like, you take an image, when you're training your transformer, you feed it into the encoder, which is frozen, you get some sequence, as you can see here, of discrete symbols, and these symbols are just indices into this codebook vector table, and you now try and train your transformer as a basically, as a sequence predictor. So how that looks like is, this will be your target sequence, so this 14233, etc. And what you do is you'll have some special token, like a start of the sequence token, and then you'll just shift this whole array by one to the right, so you'll have one, 42, you have three, etc. And now you just feed this input into your transformer, you do a couple of layers of transformer the usual way, and then you basically do cross-entropy again to train and backprop through your transformer to train it. So here is an example we have here at 22, and so we know because of this here that the next symbol is 57, so how it will be trained is we'll find like element 57, so it may be here, we'll find its probability that, let me know that as like pi sub 57, and we'll just do minus log of that probability, okay? And so in order for this to, for the loss to get down to zero, you want to push p 57 to one, because we know that's the true symbol as you can see here. So just usual sequence prediction stuff, nothing special there. So if you understand transformers, if you understand GANs, and if you understand VQVAEs, this is super trivial to understand, just connecting those components, you get the system, and voila. Okay, so that was the high level overview. Now let's, let me just show you one more thing. So there is one thing I need to know, to mention here, and that's that compared to OpenAI, these guys are from Heidelberg, so I guess they didn't have as much compute as OpenAI, obviously, and so they're constrained to have only 16 by 16 discrete latent space here, so their transformer can only work with these many tokens, okay? So that's 256, and so that means you can't have a super big image in the input, because what I've also noticed is, if they go, if they kind of decimate the spatial extent, like the spatial dimension of this image, by more than 16x, so what happens is the reconstruction starts degrading severely. So they are kind of limited to train the system when a 256 times 256 images, okay? Because of these two things I just mentioned. So this constrains you to, because reconstruction loss will suffer if you start, like, reducing the spatial dimension by a lot, and also, transformer cannot have more than 16 by 16, so take those two together, and you get, like, that their images are 256 by 256. So now, how do they get the high res images is pretty simple, so they just do sliding window across this discrete space, so let me just show you that. They have it somewhere here, okay, so what they do is because, obviously, we have only, we can only fit 256 tokens into the transformer, they just do this sliding window across, and that's how they generate more than 256 tokens, and then they just plug that in into the decoder, which is a fully convolutional neural network, so we can just accept variable number tokens. So that's how I understood this works, if, I may be wrong, like, comment down if you think there's something else they're doing, but yeah, they also mentioned here that, so there is an assumption, implicit assumption they have in order for this to work, and they say here, our wiki again ensures that the available context is still sufficient to faithfully model images, as long as either the statistics of the data set are approximately spatially invariant, or spatial conditioning information is available. So, in practice, they mentioned that even if that's kind of violated, there is this CocoGAM paper where they show they can take these smaller patches and kind of combine them into a bigger image without any seams, and that sounds really good. Okay, so that's the whole, how the whole system looks like. Again, once, what I haven't mentioned is once you train the system, how they generate novel images is, so you just prompt your transformer with a special symbol, it will generate some token, like three, I don't know, then you feed three back into the input because it's all regressive, outcome some novel symbol, like maybe 21, and again, you put 21 here and you repeat, okay, until you get 16 by 16 tokens here or more, and then you just feed that through this decoder, which was frozen from the stage one of the training, and out comes the image. Okay, so that's it. Now let's start digging into the details. First thing, I want to contrast this work with ImageGPT, and they mentioned here, so previous work, so this is ImageGPT, which applied transformers to image generation, demonstrated promising results for images up to a size of 64 by 64 pixels, but due to the quadratically increasing cost in sequence length, cannot simply be scaled to higher resolutions. So as you may know, like transformers have that, like, quadratic complexity because every token is attending to every other token, so you have n square complexity, and so this ImageGPT paper, instead of working in the discrete latent space, like here, they were working directly in the image space, and because they are working in the image space, they cannot generate, like, images of that high of a resolution, like when we have a decoder and working when we are regressing, when we are predicting next tokens in the discrete latent space. So that's what that approach was bound to fail, but like, you can see the inception of this very same idea was already there and in the VQVA paper, so yeah. Okay, so let me kind of explain the loss components in a bit more detail. So I assume you already are familiar with VQVAE, so again, we have reconstruction loss in the original paper, so this is just the L2 squared, so your MSU loss, so what this term does, and what this term does is the following. You take those codebook vectors and you push them towards the encoded vectors, so this E of X is just this thing here, so these are the E of X vectors, okay, and SG just means stop gradient, so you freeze those and you push the code vectors towards those and you also then freeze the encoded vectors and you push the, no sorry, you actually, you freeze the codebook vectors and you push the encoded vectors towards those, and by doing that, you get, that's the VQVAE in a nutshell, and now the interesting thing this paper does is they change this, they're not using MSC anymore, they sit here, so we replace the L2 loss used in the original paper for the reconstruction loss by a perceptual loss and introduce an adversarial training procedure with a patch-based generator, discriminator. So here is your usual adversarial loss, so we have this equation here and what happens is, so just a small recap, basically discriminator wants to maximize this function, this term here, and generator wants to minimize it. In order for the discriminator to maximize this, what it needs to do is, so this is discriminator and you want to push it to one, because for real images, right, for X, because log of one will be zero and that's maximizing this term here, and here, because these are fake images, X hat is fake, i.e. generated images, so you want to make sure that the discriminator outputs zero here, so that means it can discriminate between the fake and the real images, and when this goes to zero, log of one is zero, so that means we've just maximized this whole equation, okay? On the other hand, the generator wants to trick the discriminator into thinking that the images it generates are real, so it wants to instead, like, minimize this thing and it minimizes it by just pushing this to one. When you have this going to one, log of zero tends to minus infinity, and so basically that's minimizing this equation, so that's that's just a minimax game, like, setup of your regular GAN training, and the final loss of the VQVAE of this modified VQVAE model is you just want to, as you can see, we have E, that's encoder, G generator, and Z, the codebook table, we want to minimize, so we minimize according to, like, tweaking those parameters, we minimize this VQ loss, and then we also actually have the GAN component, which is maximized by discriminator and minimized by these three components here, okay? So the generator actually tries to minimize this GAN component and that's that's it, that's your, that's the whole loss function. Additionally, they have this lambda weight, which seems a bit arbitrary, to be honest, I didn't see any ablations on this, but, like, they found an adaptive way to kind of tweak this lambda, which is a coefficient that stands with this GAN component of the loss function, okay? So that's the first stage of the training, and the second part, as I already mentioned, is training the transformer. So they say here, so thus, after choosing some ordering of the indices in S, where S is just, that's your token sequence, so that's, this is S, all of these symbols are S, and the ordering they mentioned here is the following thing. So let me just clarify this part a bit. So you can, you usually take the rest of the order, so what you'll usually do, so you have this table, let me just redraw it here for clarity, what you'll do is you need to somehow impose an order, because this is a matrix, and there is no inherent, like, linear, like, you have to linearize this, right? And usually people just do, like, rest order, so that's how you're going to predict the tokens. So, because in order to have next token defined, you need to kind of make this assumption of a rest order, but I also tried doing spirals, like outgoing spirals and in-going spirals, and the rest order turned out to work the best, so they kind of stuck with it and used it as the other papers prior to this one. Okay, so, so after choosing some ordering of the indices in S, image generation can be formulated as autoregressive next index prediction. Given previous indices, the transformer learns to predict the distribution of possible next indices, i.e. this term here, to compute the likelihood of the full representation, so the joint is just modeled as a product of these conditionals, okay? And finally, in order to train the transformer, you just apply the cross entropy already mentioned. Okay, now, interesting part is this one. If you want to condition this, and they condition the transformer on multi- like on very different modalities, so you can condition this transformer on a bunch of things. You can condition it on a class, you can condition it on a like image that has semantic segmentation applied to it, etc. So, they show here, let me show you just some examples. So, here, as you can see, you have this image which is semantically segmented, and you're trying to, given that as a condition, you're trying to generate some novel images here, as you can see on the right. Okay, so let me show you how they kind of make that work. So, they say here, if the conditioning information C has spatial extent, so as that semantic masks, we first learn in other VQ again to obtain again an index-based representation R with the newly obtained codebook Z. Due to the autoregressive structure of the transformer, we can then simply prepend R to S. So, if you don't understand that, let me kind of clarify it. I think it's fairly simple. Now, let's see how this procedure will look like. Basically, you'll have an additional VQ again, okay? So, this is your encoder one, you have a discrete latent space, you have a decoder. So, you train this thing on those semantic masks, so this is decoder one, okay? So, you train this thing on your like semantic masks, and once it's trained, you just freeze it up, and then you do the following thing. So, you have your target VQ again, so this one will just be encoding the regular images, and what you'll do is the following during the training of this one. So, you take the input image you encoded, you get the target sequence, so this is your target sequence, and how your input will be formed is the following. So, you take a semantic mask, you'll just get some tokens here, you flatten it out, and you prepend it here, and now what you do is you just take your special token, you place it here, you take your sequence here, you just shift it by one position to the right, and this is your input to the transformer, this is your target, and you again train it on a simple sequence prediction problem. So, that's how they'll train this model, and then later during generation, obviously, you're just going to prepend the semantic mask and the prompting mask token, and you're then just going to sample from your transformer to get the output image, which you'll just feed through the transformer. So, the output sequence, which you'll then just feed to the decoder, okay? So, yeah, that was pretty much a thorough explanation of how this system works. Hopefully, that was useful. I went into a bit more details than usually, so let me know whether you find this kind of in-depth explanation useful, or should I just concentrate on like high-level explanation and just stick to the key points of the paper, okay? So, that was it. Now, let's see some experiments and some comparisons. Okay, in this table, they just compare and kind of justify the use of transformer instead of some other or regressive model, like as I mentioned, the QVAE originally used PixelCNN. PixelSnail is just an improvement from that very same family, and you can see here that because this one trains 2x faster, they have two comparisons. So, they have a comparison where they train GPT-2 for the same amount of time as P-Snail, and basically, you can see we have an improvement here because these numbers are lower. Lower is better for NLL, and again, when you have the same number of steps, it's even better. So, that kind of justifies the usage of transformers, although I think at this point of time, it's pretty obvious that transformers are better than those types of models. Okay, here are some results. Basically, they can do multiple things, and that's the premise of this paper. You'll later see that they are on-pair with other methods, but they don't try and say that they are better. They are just on-pair, but they are really diverse. They can apply this to many different modalities. So, one thing you can do is you can take this image here, you can encode it and use it to prompt the transformer in order to generate by sampling different images. And when I say sampling, again, when you take this as the context and you're trying to predict the next token, okay, so this token will have some distribution output from the transformer, right? And so, there are multiple ways you can do sampling from distribution. One is to do a greedy approach, so you just take the token that has the highest probability. One is to do, and that's what these folks did, is top-k sampling. That means, for example, if k equals 3, you'll take the three tokens with highest probability, so that would be these three here. And then you just re-rate them so that they sum up to 1, and you'll just sample according to their weights. And that's how you avoid sampling these super small probability tokens, which would be the case if you were using your regular sampling. So, not a greedy, not top-k, but just a regular sampling where you're taking into consideration all of the tokens with their corresponding probabilities. So, that's one task you can, aside from semantic masks, you can use depth masks. As you can see, they get these weird pigeons. They scared the shit out of me, but like, yeah, I don't know. And additionally, you can see they can kind of condition on these pose kind of images, and you get these as output. Like, I think this girl broke her leg, but yeah, okay, she'll survive. And finally, you have class conditioning. So, basically, they conditioned on the class label of a frog, and they generate these novel, scary frogs. Okay, and additionally, they have something called rejection sampling. We'll get to that in a moment, but like using that, they can achieve much better. They can get much better images than these here. I think these were not obtained by using this rejection sampling. Okay, let's see other results. So, again, these are some awesome images they can obtain using semantic segmentation masks. Here on the right, you can see again, like depth images, semantic masks, edges, super resolution, a bunch of different tasks they can apply this on. As I mentioned, they say they're on par with other methods. So, for this semantic image synthesis, they compare with the spade. It's even a bit better. So, spade is a bit better than their approach, but yeah. Okay, finally, comparing to other methods on the face synthesis, face image synthesis task, looking at the FIDs, we can see that this paper is a bit worse than this PGGAN, and it's a lot worse than StyleGAN2, looking at the FID metrics, so fresh-ase inception distance metric. They are way worse compared to StyleGAN, but as I said, they're not claiming they're better. In that sense, they're just more general, and yeah, and they're the first to have used transformers to generate these higher-res images. Okay, similarly here, if we focus on FID, we can see that without... So, keeping this acceptance rate, so that's the rejection sampling I was mentioning about. So, if you keep it at one, you can see that they arrive at FIDs of maybe 15, and that's worse than many of these methods here. Okay, but if you start applying this rejection sampling, they drop the FID down to five, which is much better or on par with these methods, but they can also use rejection sampling, so I guess this is not that fair of a comparison. By the way, they use ResNet, they say it's somewhere here, they use ResNet 101 to do this rejection sampling, so here, but instead, if you'd use something better, like a clip model from OpenAI, they can achieve even better results, and you'll see those all around, like on Twitter, people using VQGAN plus clip to generate nice images, and so the whole trick with those rejection sampling is the following. So, you output a couple of images from your... You sample a couple of images from your... Like, from VQGAN, and what you then do is you feed these into, for example, clip, and you prompt clip with a specific class, so for example, you have class one, whatever, that may be like a frog or whatever, and what clip will do, it will say how probable this picture is to be of a class one, so maybe this one will be, I don't know, 0.7, this one will be 0.85 or something, this one will be, I don't know, 0.6, and so you can just sort them, sort the images according to these scores, and just pick the top three or top five, whatever, and what it turns out, it turns out this kind of is an automatic way of cherry picking, so it additionally improves upon the results that this paper achieved, reported. Okay, what I show here is how that down sampling in the encoder affects the results, so what I do is they keep the latent space at 16 by 16, and then what I do is you can see here the number of times that the spatial extent of the image has been reduced, so 16 means that the original image was 256 by 256, one means basically input image was 16 by 16, and that means that during the training, they actually have to take a crop of 16 by 16 here and directly train the transformer on that, and as you can see, the receptive field is much, much, much smaller when you have f1, and because of that, you get these weird Picasso-looking images, and I really think, like, somebody can earn some money by just producing these and selling them as artwork. If somebody earns some money, consider becoming a patron of the epiphany, yeah, that'd be nice. Okay, when they use f2, so that's a bit higher receptive field, you can see it kind of gets better, but this really freaks me out, and then additionally, it gets better and better, and finally, when you have 16, assuming these are 256 images, meaning basically you had a receptive field was spanning the whole image, and that's why we get much better results here. Additionally, these are much faster to generate, so we have a huge improvement in speed up compared to when you're trying to regress this in the pixel space, because here you'll have to regress all of the pixels, whereas here you'll just have to regress the 16 by 16 in the latent space and then decode it, so that's why we get a huge speed up. Okay, I think that's pretty much it. Let me just show you a couple more things that's kind of interesting. So here they show that they are better than Delle and VQVAE2 when we look at the FIDs of the reconstructed images. So you just, they just compare, they take the validation images, for example, so that's this FID slash VAL, so they pass it through the encoder and then decoder, and then they just calculate the FID, and so this just shows that the reconstruction is coming from this VQGAN paper are better than Delle and VQVAE, as we can expect, because as I mentioned, they are blurry because they're not using adversarial component, neither did they use perceptual loss. Okay, one more interesting thing I want to show you in the appendix. So basically here is that visualizable, I just explained. You have Delle here, you have VQGAN here, you can see these images are much more blurry compared to VQGAN, which has adversarial component, so that's really cool. GANs are like famous for making images much more crispy. They have many more diagrams and pixTiv included in the appendix, it's huge, it's got maybe 50 pages, you can check it out, some of the images there, but that was in a nutshell everything I wanted to cover today. If you found this video useful, share it out, subscribe, and until next time, bye-bye!
[{"start": 0.0, "end": 7.58, "text": " What's up in this video I'm covering VQGAN or taming transformers for high resolution image synthesis paper by Patrick Esser,"}, {"start": 7.86, "end": 11.78, "text": " Robin Rhombach and Bjorn Omer from Heidelberg University."}, {"start": 11.9, "end": 18.34, "text": " So as you can see it's all about image synthesis and what's exciting about this paper is they are the first to use"}, {"start": 18.5, "end": 23.04, "text": " transformers and achieve high resolution images as the one you can see here on the screen."}, {"start": 23.04, "end": 28.2, "text": " And the thing about this paper is it directly builds off of VQVAE and"}, {"start": 28.2, "end": 33.56, "text": " that's a model that was previously developed by DeepMind and I've covered in my last video"}, {"start": 33.56, "end": 38.66, "text": " so do check it out if you don't know anything about it. But like aside from this paper, so aside from VQGAN,"}, {"start": 38.879999999999995, "end": 42.76, "text": " Dali from OpenAI also uses VQVAE as the backbone."}, {"start": 42.879999999999995, "end": 48.739999999999995, "text": " So that's an important building block currently for these generative models and yeah, do check that video out."}, {"start": 49.16, "end": 51.8, "text": " Okay, having said that let's see what this paper is all about."}, {"start": 51.8, "end": 54.760000000000005, "text": " So I mentioned transformers and they say here"}, {"start": 54.76, "end": 60.76, "text": " we demonstrate how combining the effectiveness of the inductive bias of CNNs with the expressivity of transformers"}, {"start": 61.3, "end": 64.67999999999999, "text": " enables them to model and thereby synthesize higher resolution images."}, {"start": 64.67999999999999, "end": 70.03999999999999, "text": " So this CNN part is just the encoder part of the VQVAE as we'll soon see. Okay."}, {"start": 70.7, "end": 77.12, "text": " They say here that we hypothesize that low-level image structure is well described by local connectivity, i.e."}, {"start": 77.74, "end": 84.47999999999999, "text": " convolutional architecture, whereas this structural assumption ceases to be effective on higher semantic levels."}, {"start": 84.48, "end": 91.30000000000001, "text": " So the idea is to extract certain features which are kind of representative of what image constitutes of and then kind of use"}, {"start": 91.58, "end": 99.62, "text": " transformer to do this sequence token prediction as we'll soon see in order to generate and in order to generate like images"}, {"start": 99.62, "end": 105.9, "text": " which are globally coherent and that's where the attention pattern of the transformers which basically"}, {"start": 106.46000000000001, "end": 111.36, "text": " treats the underlying data as the fully connected graph comes like useful."}, {"start": 111.36, "end": 116.82, "text": " Okay, so having said that let's see the high-level overview of the architecture."}, {"start": 117.48, "end": 123.6, "text": " Okay, so first things first as you can probably recognize this bottom block here is VQVAE and"}, {"start": 124.24, "end": 127.08, "text": " the training itself of this VQGAN paper"}, {"start": 127.75999999999999, "end": 133.84, "text": " consists out of two stages. So very similar to VQVAE. Basically you first train the VQVAE"}, {"start": 134.32, "end": 137.76, "text": " module, so that's this one, and then you train a prior"}, {"start": 137.76, "end": 144.35999999999999, "text": " so in the VQVAE paper they use pixelCNN, here they're using transformers or more precisely"}, {"start": 144.35999999999999, "end": 148.32, "text": " they're using GPT-2 model from OpenAI. Okay, if you watch my previous video"}, {"start": 148.56, "end": 153.44, "text": " you'll see that this structure overall is very similar to the original VQVAE paper."}, {"start": 153.44, "end": 158.92, "text": " So what I've done additionally here is they swapped pixelCNN and they're not now using state-of-the-art sequence modeling"}, {"start": 159.48, "end": 165.12, "text": " model and that's transformer GPT-2 in this case and they also improved upon the reconstruction loss. They are not using MSC,"}, {"start": 165.12, "end": 167.84, "text": " they're using perceptual loss plus adversarial loss instead."}, {"start": 167.92000000000002, "end": 175.56, "text": " So there is a lot of overlap between the two projects, but like this project got a much better much better like imagery and high-res imagery."}, {"start": 175.56, "end": 179.88, "text": " So yeah, let's see how it works. So again, you have the VQVAE component here."}, {"start": 179.88, "end": 184.52, "text": " What you do is you have your image, you encode it, you get some latent"}, {"start": 184.72, "end": 192.04000000000002, "text": " representations and then you snap these latent vectors onto these codebook vectors so that you have this discrete representation here"}, {"start": 192.04, "end": 195.45999999999998, "text": " which you then feed back into the decoder in order to reconstruct the image."}, {"start": 195.48, "end": 201.07999999999998, "text": " So that's the high level picture of how that works. And again, like just as a reminder"}, {"start": 201.64, "end": 207.07999999999998, "text": " this system is trained in two stages. The stage one is you learn to train this VQVAE model,"}, {"start": 207.07999999999998, "end": 211.51999999999998, "text": " you learn how to kind of create these high quality"}, {"start": 212.64, "end": 217.0, "text": " latent vectors and then once you have that you freeze everything, you freeze the"}, {"start": 217.0, "end": 223.4, "text": " decoder, you freeze the encoder, you freeze the codebook and you train the transformer across this discrete latent space."}, {"start": 223.48, "end": 226.24, "text": " Okay, that's the high level idea of how the system is trained."}, {"start": 226.32, "end": 230.52, "text": " So now let's see what are the differences between VQVAE and this paper."}, {"start": 230.52, "end": 236.16, "text": " So the main difference here in the stage one is they are not using MSC."}, {"start": 237.0, "end": 240.88, "text": " So they are instead using perceptual loss plus adversarial loss."}, {"start": 240.88, "end": 244.6, "text": " So perceptual loss just it comes from the, to the best of my knowledge,"}, {"start": 244.6, "end": 250.38, "text": " it comes from the neural stall transfer literature. And so what you do is you basically take an image."}, {"start": 250.38, "end": 256.4, "text": " So you'll take this original image, you'll feed it into a pre-trained CNN like maybe"}, {"start": 257.32, "end": 262.18, "text": " VGG net or something. So this is your CNN. You feed in the original image."}, {"start": 262.18, "end": 268.48, "text": " So let's denote that as X and you feed in the reconstructed image and let's denote that as X hat."}, {"start": 268.48, "end": 274.04, "text": " And what you then do, you just take a certain layer from this VGG, from this pre-trained CNN"}, {"start": 274.04, "end": 279.74, "text": " and you extract those feature maps. So that will be something like this. Okay, so that's gonna be,"}, {"start": 280.22, "end": 286.20000000000005, "text": " these are the channels height and width. Okay, and you do the same thing for the original image."}, {"start": 286.20000000000005, "end": 292.36, "text": " So you also extract those features there and what you do then instead is you just apply MSC loss,"}, {"start": 292.36, "end": 297.76, "text": " but this time not in the image space, but across this feature space. That's the only difference."}, {"start": 297.76, "end": 303.96000000000004, "text": " And the reason that functions is because the classifier already learned how to extract some valuable features"}, {"start": 303.96, "end": 311.15999999999997, "text": " and it's just unreasonably effective. Okay, and aside from that, they also have this adversarial component."}, {"start": 311.15999999999997, "end": 319.88, "text": " So as you may know, the original VQVAE had a problem with blur and we already know that GANs kind of improved that."}, {"start": 319.88, "end": 325.79999999999995, "text": " They make much more crispier image. So it was like actually the original authors of VQVAE paper also"}, {"start": 325.79999999999995, "end": 331.88, "text": " like suggested using GANs in the future work. So here it is. So they're using PatchGAN."}, {"start": 331.88, "end": 337.8, "text": " So what it does, so what's the difference between that and your regular GAN? You usually have just a single output"}, {"start": 337.8, "end": 343.8, "text": " like full, like you have like something like a sigmoid activation function and then you're trying to train the discriminator"}, {"start": 343.8, "end": 349.08, "text": " to discriminate between real data and fake data. So here it's the same thing instead, but like the only difference is"}, {"start": 349.08, "end": 355.4, "text": " you're taking patches, like overlapping patches, like maybe this thing and then you'll have this and you'll have another patch"}, {"start": 355.4, "end": 361.64, "text": " and you have like four patches together and you can see it will, the discriminator will output some values here."}, {"start": 361.64, "end": 369.47999999999996, "text": " So if this image was a real image, we'd expect it to output, for example, all of these should be R or 1, whatever that,"}, {"start": 369.47999999999996, "end": 376.44, "text": " like whatever the value, however you decide to encode the real values. And so basically train, let's say these are like"}, {"start": 376.44, "end": 383.56, "text": " 0 to 1 values, you'll just do your minus loss. So basically your cross entropy and you're going to train this patch-based discriminator"}, {"start": 383.56, "end": 390.2, "text": " the usual way. Okay, so I'll get into a bit more details about the GAN loss, but this is roughly how it works like."}, {"start": 390.2, "end": 398.59999999999997, "text": " So they have patch-based GAN, all of that taken together, so the perceptual loss plus this GAN loss, so these two together"}, {"start": 399.47999999999996, "end": 407.0, "text": " give much more crispier image as a reconstruction here from this VQVA pipeline. Okay, the second step, once you train that,"}, {"start": 407.0, "end": 413.56, "text": " once you have these code vectors, what you do is you just train the transformer on a sequence prediction task."}, {"start": 413.56, "end": 419.56, "text": " So how the procedure looks like, you take an image, when you're training your transformer, you feed it into the encoder,"}, {"start": 419.56, "end": 427.64, "text": " which is frozen, you get some sequence, as you can see here, of discrete symbols, and these symbols are just indices into this"}, {"start": 427.64, "end": 435.72, "text": " codebook vector table, and you now try and train your transformer as a basically, as a sequence predictor."}, {"start": 435.72, "end": 445.0, "text": " So how that looks like is, this will be your target sequence, so this 14233, etc. And what you do is you'll have some"}, {"start": 445.0, "end": 450.68, "text": " special token, like a start of the sequence token, and then you'll just shift this whole array by one to the right,"}, {"start": 450.68, "end": 457.72, "text": " so you'll have one, 42, you have three, etc. And now you just feed this input into your transformer,"}, {"start": 458.84, "end": 465.72, "text": " you do a couple of layers of transformer the usual way, and then you basically do cross-entropy"}, {"start": 466.52, "end": 472.84, "text": " again to train and backprop through your transformer to train it. So here is an example we have here at 22,"}, {"start": 472.84, "end": 480.03999999999996, "text": " and so we know because of this here that the next symbol is 57, so how it will be trained is we'll find"}, {"start": 480.03999999999996, "end": 488.91999999999996, "text": " like element 57, so it may be here, we'll find its probability that, let me know that as like pi sub 57,"}, {"start": 489.64, "end": 496.52, "text": " and we'll just do minus log of that probability, okay? And so in order for this to, for the loss to get"}, {"start": 496.52, "end": 503.88, "text": " down to zero, you want to push p 57 to one, because we know that's the true symbol as you can see here."}, {"start": 503.88, "end": 509.56, "text": " So just usual sequence prediction stuff, nothing special there. So if you understand transformers,"}, {"start": 509.56, "end": 514.52, "text": " if you understand GANs, and if you understand VQVAEs, this is super trivial to understand, just"}, {"start": 514.52, "end": 521.0, "text": " connecting those components, you get the system, and voila. Okay, so that was the high level overview."}, {"start": 521.0, "end": 527.0, "text": " Now let's, let me just show you one more thing. So there is one thing I need to know, to mention here,"}, {"start": 527.0, "end": 532.52, "text": " and that's that compared to OpenAI, these guys are from Heidelberg, so I guess they didn't have"}, {"start": 532.52, "end": 538.92, "text": " as much compute as OpenAI, obviously, and so they're constrained to have only 16 by 16 discrete"}, {"start": 538.92, "end": 544.68, "text": " latent space here, so their transformer can only work with these many tokens, okay? So that's 256,"}, {"start": 544.68, "end": 551.0, "text": " and so that means you can't have a super big image in the input, because what I've also noticed is,"}, {"start": 551.0, "end": 556.04, "text": " if they go, if they kind of decimate the spatial extent, like the spatial dimension of this image,"}, {"start": 556.04, "end": 563.3199999999999, "text": " by more than 16x, so what happens is the reconstruction starts degrading severely."}, {"start": 563.3199999999999, "end": 571.64, "text": " So they are kind of limited to train the system when a 256 times 256 images, okay? Because of these"}, {"start": 571.64, "end": 576.52, "text": " two things I just mentioned. So this constrains you to, because reconstruction loss will suffer"}, {"start": 576.52, "end": 582.52, "text": " if you start, like, reducing the spatial dimension by a lot, and also, transformer cannot have more"}, {"start": 582.52, "end": 589.0, "text": " than 16 by 16, so take those two together, and you get, like, that their images are 256 by 256."}, {"start": 589.0, "end": 595.0, "text": " So now, how do they get the high res images is pretty simple, so they just do sliding window"}, {"start": 595.0, "end": 600.28, "text": " across this discrete space, so let me just show you that. They have it somewhere here, okay, so"}, {"start": 600.28, "end": 607.48, "text": " what they do is because, obviously, we have only, we can only fit 256 tokens into the transformer,"}, {"start": 607.48, "end": 613.56, "text": " they just do this sliding window across, and that's how they generate more than 256 tokens,"}, {"start": 613.56, "end": 618.04, "text": " and then they just plug that in into the decoder, which is a fully convolutional neural network,"}, {"start": 618.04, "end": 622.8399999999999, "text": " so we can just accept variable number tokens. So that's how I understood this works, if,"}, {"start": 622.8399999999999, "end": 627.64, "text": " I may be wrong, like, comment down if you think there's something else they're doing, but yeah,"}, {"start": 627.64, "end": 632.92, "text": " they also mentioned here that, so there is an assumption, implicit assumption they have in"}, {"start": 632.92, "end": 637.16, "text": " order for this to work, and they say here, our wiki again ensures that the available context is"}, {"start": 637.16, "end": 642.28, "text": " still sufficient to faithfully model images, as long as either the statistics of the data set are"}, {"start": 642.28, "end": 648.12, "text": " approximately spatially invariant, or spatial conditioning information is available. So, in"}, {"start": 648.12, "end": 653.48, "text": " practice, they mentioned that even if that's kind of violated, there is this CocoGAM paper where"}, {"start": 653.48, "end": 659.48, "text": " they show they can take these smaller patches and kind of combine them into a bigger image"}, {"start": 659.48, "end": 665.72, "text": " without any seams, and that sounds really good. Okay, so that's the whole, how the whole system"}, {"start": 665.72, "end": 671.72, "text": " looks like. Again, once, what I haven't mentioned is once you train the system, how they generate"}, {"start": 671.72, "end": 678.6800000000001, "text": " novel images is, so you just prompt your transformer with a special symbol, it will generate some token,"}, {"start": 678.68, "end": 683.64, "text": " like three, I don't know, then you feed three back into the input because it's all regressive, outcome"}, {"start": 683.64, "end": 689.9599999999999, "text": " some novel symbol, like maybe 21, and again, you put 21 here and you repeat, okay, until you get"}, {"start": 689.9599999999999, "end": 695.88, "text": " 16 by 16 tokens here or more, and then you just feed that through this decoder, which was frozen"}, {"start": 695.88, "end": 701.56, "text": " from the stage one of the training, and out comes the image. Okay, so that's it. Now let's start"}, {"start": 701.56, "end": 707.9599999999999, "text": " digging into the details. First thing, I want to contrast this work with ImageGPT, and they"}, {"start": 707.96, "end": 712.0400000000001, "text": " mentioned here, so previous work, so this is ImageGPT, which applied transformers to image"}, {"start": 712.0400000000001, "end": 718.6800000000001, "text": " generation, demonstrated promising results for images up to a size of 64 by 64 pixels, but due"}, {"start": 718.6800000000001, "end": 724.44, "text": " to the quadratically increasing cost in sequence length, cannot simply be scaled to higher resolutions."}, {"start": 724.44, "end": 729.88, "text": " So as you may know, like transformers have that, like, quadratic complexity because every token is"}, {"start": 729.88, "end": 735.1600000000001, "text": " attending to every other token, so you have n square complexity, and so this ImageGPT paper,"}, {"start": 735.16, "end": 740.4399999999999, "text": " instead of working in the discrete latent space, like here, they were working directly in the"}, {"start": 740.4399999999999, "end": 744.12, "text": " image space, and because they are working in the image space, they cannot generate, like, images of"}, {"start": 744.12, "end": 749.0799999999999, "text": " that high of a resolution, like when we have a decoder and working when we are regressing,"}, {"start": 749.0799999999999, "end": 753.88, "text": " when we are predicting next tokens in the discrete latent space. So that's what that"}, {"start": 753.88, "end": 759.72, "text": " approach was bound to fail, but like, you can see the inception of this very same idea was already"}, {"start": 759.72, "end": 769.0, "text": " there and in the VQVA paper, so yeah. Okay, so let me kind of explain the loss components in a bit"}, {"start": 769.0, "end": 775.64, "text": " more detail. So I assume you already are familiar with VQVAE, so again, we have reconstruction loss"}, {"start": 775.64, "end": 783.0, "text": " in the original paper, so this is just the L2 squared, so your MSU loss, so what this term does,"}, {"start": 783.0, "end": 788.44, "text": " and what this term does is the following. You take those codebook vectors and you push them"}, {"start": 788.44, "end": 795.32, "text": " towards the encoded vectors, so this E of X is just this thing here, so these are the E of X vectors,"}, {"start": 795.32, "end": 800.7600000000001, "text": " okay, and SG just means stop gradient, so you freeze those and you push the code vectors towards"}, {"start": 800.7600000000001, "end": 807.1600000000001, "text": " those and you also then freeze the encoded vectors and you push the, no sorry, you actually, you"}, {"start": 807.1600000000001, "end": 813.08, "text": " freeze the codebook vectors and you push the encoded vectors towards those, and by doing that, you get,"}, {"start": 813.08, "end": 819.32, "text": " that's the VQVAE in a nutshell, and now the interesting thing this paper does is they change"}, {"start": 819.32, "end": 825.5600000000001, "text": " this, they're not using MSC anymore, they sit here, so we replace the L2 loss used in the original"}, {"start": 825.5600000000001, "end": 831.32, "text": " paper for the reconstruction loss by a perceptual loss and introduce an adversarial training"}, {"start": 831.32, "end": 837.8000000000001, "text": " procedure with a patch-based generator, discriminator. So here is your usual adversarial loss, so we have"}, {"start": 837.8, "end": 844.92, "text": " this equation here and what happens is, so just a small recap, basically discriminator wants to"}, {"start": 844.92, "end": 851.16, "text": " maximize this function, this term here, and generator wants to minimize it. In order for"}, {"start": 851.16, "end": 857.24, "text": " the discriminator to maximize this, what it needs to do is, so this is discriminator and you want to"}, {"start": 857.24, "end": 865.4799999999999, "text": " push it to one, because for real images, right, for X, because log of one will be zero and that's"}, {"start": 865.48, "end": 872.12, "text": " maximizing this term here, and here, because these are fake images, X hat is fake, i.e. generated"}, {"start": 872.12, "end": 877.72, "text": " images, so you want to make sure that the discriminator outputs zero here, so that means it can discriminate"}, {"start": 877.72, "end": 883.16, "text": " between the fake and the real images, and when this goes to zero, log of one is zero, so that means"}, {"start": 883.16, "end": 889.88, "text": " we've just maximized this whole equation, okay? On the other hand, the generator wants to trick the"}, {"start": 889.88, "end": 895.88, "text": " discriminator into thinking that the images it generates are real, so it wants to instead, like,"}, {"start": 895.88, "end": 901.16, "text": " minimize this thing and it minimizes it by just pushing this to one. When you have this going to"}, {"start": 901.16, "end": 908.36, "text": " one, log of zero tends to minus infinity, and so basically that's minimizing this equation, so that's"}, {"start": 908.36, "end": 915.24, "text": " that's just a minimax game, like, setup of your regular GAN training, and the final loss of the"}, {"start": 915.24, "end": 921.5600000000001, "text": " VQVAE of this modified VQVAE model is you just want to, as you can see, we have E, that's encoder,"}, {"start": 921.5600000000001, "end": 928.12, "text": " G generator, and Z, the codebook table, we want to minimize, so we minimize according to, like,"}, {"start": 928.12, "end": 935.48, "text": " tweaking those parameters, we minimize this VQ loss, and then we also actually have"}, {"start": 936.6, "end": 942.52, "text": " the GAN component, which is maximized by discriminator and minimized by these three"}, {"start": 942.52, "end": 948.6, "text": " components here, okay? So the generator actually tries to minimize this GAN component and that's"}, {"start": 948.6, "end": 953.96, "text": " that's it, that's your, that's the whole loss function. Additionally, they have this lambda"}, {"start": 953.96, "end": 960.1999999999999, "text": " weight, which seems a bit arbitrary, to be honest, I didn't see any ablations on this, but, like, they"}, {"start": 960.1999999999999, "end": 966.68, "text": " found an adaptive way to kind of tweak this lambda, which is a coefficient that stands with this GAN"}, {"start": 966.68, "end": 973.56, "text": " component of the loss function, okay? So that's the first stage of the training, and the second"}, {"start": 973.56, "end": 980.04, "text": " part, as I already mentioned, is training the transformer. So they say here, so thus, after"}, {"start": 980.04, "end": 986.68, "text": " choosing some ordering of the indices in S, where S is just, that's your token sequence, so that's,"}, {"start": 986.68, "end": 993.56, "text": " this is S, all of these symbols are S, and the ordering they mentioned here is the following"}, {"start": 993.56, "end": 999.8, "text": " thing. So let me just clarify this part a bit. So you can, you usually take the rest of the order,"}, {"start": 999.8, "end": 1004.68, "text": " so what you'll usually do, so you have this table, let me just redraw it here for clarity,"}, {"start": 1005.7199999999999, "end": 1010.92, "text": " what you'll do is you need to somehow impose an order, because this is a matrix, and there is no"}, {"start": 1011.56, "end": 1016.1999999999999, "text": " inherent, like, linear, like, you have to linearize this, right? And usually people just do, like,"}, {"start": 1016.1999999999999, "end": 1022.04, "text": " rest order, so that's how you're going to predict the tokens. So, because in order to have next"}, {"start": 1022.04, "end": 1027.56, "text": " token defined, you need to kind of make this assumption of a rest order, but I also tried"}, {"start": 1027.56, "end": 1035.56, "text": " doing spirals, like outgoing spirals and in-going spirals, and the rest order turned out to work"}, {"start": 1035.56, "end": 1042.6, "text": " the best, so they kind of stuck with it and used it as the other papers prior to this one. Okay, so,"}, {"start": 1043.56, "end": 1048.2, "text": " so after choosing some ordering of the indices in S, image generation can be formulated as"}, {"start": 1048.2, "end": 1053.32, "text": " autoregressive next index prediction. Given previous indices, the transformer learns to predict"}, {"start": 1053.32, "end": 1059.32, "text": " the distribution of possible next indices, i.e. this term here, to compute the likelihood of the"}, {"start": 1059.32, "end": 1066.1200000000001, "text": " full representation, so the joint is just modeled as a product of these conditionals, okay? And"}, {"start": 1066.1200000000001, "end": 1070.1200000000001, "text": " finally, in order to train the transformer, you just apply the cross entropy already mentioned."}, {"start": 1070.1200000000001, "end": 1076.3600000000001, "text": " Okay, now, interesting part is this one. If you want to condition this, and they condition the"}, {"start": 1076.36, "end": 1081.32, "text": " transformer on multi- like on very different modalities, so you can condition this transformer"}, {"start": 1081.32, "end": 1085.24, "text": " on a bunch of things. You can condition it on a class, you can condition it on a like image that"}, {"start": 1085.24, "end": 1090.6799999999998, "text": " has semantic segmentation applied to it, etc. So, they show here, let me show you just some examples."}, {"start": 1090.6799999999998, "end": 1096.4399999999998, "text": " So, here, as you can see, you have this image which is semantically segmented, and you're trying to,"}, {"start": 1096.4399999999998, "end": 1100.12, "text": " given that as a condition, you're trying to generate some novel images here, as you can see"}, {"start": 1100.12, "end": 1107.56, "text": " on the right. Okay, so let me show you how they kind of make that work. So, they say here, if the"}, {"start": 1107.56, "end": 1113.8, "text": " conditioning information C has spatial extent, so as that semantic masks, we first learn in other"}, {"start": 1113.8, "end": 1123.32, "text": " VQ again to obtain again an index-based representation R with the newly obtained codebook Z. Due to the"}, {"start": 1123.32, "end": 1129.0, "text": " autoregressive structure of the transformer, we can then simply prepend R to S. So, if you"}, {"start": 1129.0, "end": 1133.4, "text": " don't understand that, let me kind of clarify it. I think it's fairly simple. Now, let's see how"}, {"start": 1133.4, "end": 1138.36, "text": " this procedure will look like. Basically, you'll have an additional VQ again, okay? So, this is your"}, {"start": 1138.36, "end": 1144.12, "text": " encoder one, you have a discrete latent space, you have a decoder. So, you train this thing on those"}, {"start": 1144.12, "end": 1149.96, "text": " semantic masks, so this is decoder one, okay? So, you train this thing on your like semantic masks,"}, {"start": 1149.96, "end": 1154.04, "text": " and once it's trained, you just freeze it up, and then you do the following thing. So, you have your"}, {"start": 1154.04, "end": 1161.56, "text": " target VQ again, so this one will just be encoding the regular images, and what you'll do is the"}, {"start": 1161.56, "end": 1166.6, "text": " following during the training of this one. So, you take the input image you encoded, you get the"}, {"start": 1166.6, "end": 1172.28, "text": " target sequence, so this is your target sequence, and how your input will be formed is the following."}, {"start": 1172.28, "end": 1178.92, "text": " So, you take a semantic mask, you'll just get some tokens here, you flatten it out, and you prepend"}, {"start": 1178.92, "end": 1185.5600000000002, "text": " it here, and now what you do is you just take your special token, you place it here,"}, {"start": 1186.44, "end": 1192.6000000000001, "text": " you take your sequence here, you just shift it by one position to the right, and this is your"}, {"start": 1192.6000000000001, "end": 1198.52, "text": " input to the transformer, this is your target, and you again train it on a simple sequence prediction"}, {"start": 1198.52, "end": 1204.2, "text": " problem. So, that's how they'll train this model, and then later during generation, obviously,"}, {"start": 1204.2, "end": 1210.8400000000001, "text": " you're just going to prepend the semantic mask and the prompting mask token, and you're then just"}, {"start": 1210.8400000000001, "end": 1215.72, "text": " going to sample from your transformer to get the output image, which you'll just feed through the"}, {"start": 1215.72, "end": 1219.96, "text": " transformer. So, the output sequence, which you'll then just feed to the decoder, okay?"}, {"start": 1221.0, "end": 1227.64, "text": " So, yeah, that was pretty much a thorough explanation of how this system works. Hopefully,"}, {"start": 1227.64, "end": 1232.6000000000001, "text": " that was useful. I went into a bit more details than usually, so let me know whether you find"}, {"start": 1232.6, "end": 1237.3999999999999, "text": " this kind of in-depth explanation useful, or should I just concentrate on like high-level explanation"}, {"start": 1237.3999999999999, "end": 1244.28, "text": " and just stick to the key points of the paper, okay? So, that was it. Now, let's see some"}, {"start": 1244.28, "end": 1250.36, "text": " experiments and some comparisons. Okay, in this table, they just compare and kind of justify"}, {"start": 1250.36, "end": 1255.8, "text": " the use of transformer instead of some other or regressive model, like as I mentioned,"}, {"start": 1255.8, "end": 1261.08, "text": " the QVAE originally used PixelCNN. PixelSnail is just an improvement from that very same family,"}, {"start": 1261.08, "end": 1267.24, "text": " and you can see here that because this one trains 2x faster, they have two comparisons. So, they have"}, {"start": 1267.24, "end": 1273.32, "text": " a comparison where they train GPT-2 for the same amount of time as P-Snail, and basically, you can"}, {"start": 1273.32, "end": 1277.6399999999999, "text": " see we have an improvement here because these numbers are lower. Lower is better for NLL,"}, {"start": 1277.6399999999999, "end": 1283.0, "text": " and again, when you have the same number of steps, it's even better. So, that kind of justifies the"}, {"start": 1283.0, "end": 1287.8799999999999, "text": " usage of transformers, although I think at this point of time, it's pretty obvious that transformers"}, {"start": 1287.88, "end": 1296.1200000000001, "text": " are better than those types of models. Okay, here are some results. Basically, they can do"}, {"start": 1296.1200000000001, "end": 1301.96, "text": " multiple things, and that's the premise of this paper. You'll later see that they are on-pair"}, {"start": 1301.96, "end": 1306.6000000000001, "text": " with other methods, but they don't try and say that they are better. They are just on-pair,"}, {"start": 1306.6000000000001, "end": 1311.0800000000002, "text": " but they are really diverse. They can apply this to many different modalities. So, one thing you"}, {"start": 1311.0800000000002, "end": 1317.3200000000002, "text": " can do is you can take this image here, you can encode it and use it to prompt the transformer"}, {"start": 1317.32, "end": 1324.84, "text": " in order to generate by sampling different images. And when I say sampling, again, when you take this"}, {"start": 1324.84, "end": 1330.28, "text": " as the context and you're trying to predict the next token, okay, so this token will have some"}, {"start": 1330.28, "end": 1337.6399999999999, "text": " distribution output from the transformer, right? And so, there are multiple ways you can do sampling"}, {"start": 1337.6399999999999, "end": 1342.28, "text": " from distribution. One is to do a greedy approach, so you just take the token that has the highest"}, {"start": 1342.28, "end": 1349.8, "text": " probability. One is to do, and that's what these folks did, is top-k sampling. That means, for"}, {"start": 1349.8, "end": 1354.6, "text": " example, if k equals 3, you'll take the three tokens with highest probability, so that would be"}, {"start": 1356.2, "end": 1362.36, "text": " these three here. And then you just re-rate them so that they sum up to 1, and you'll just sample"}, {"start": 1362.36, "end": 1368.92, "text": " according to their weights. And that's how you avoid sampling these super small probability"}, {"start": 1368.92, "end": 1374.2, "text": " tokens, which would be the case if you were using your regular sampling. So, not a greedy, not top-k,"}, {"start": 1374.2, "end": 1378.76, "text": " but just a regular sampling where you're taking into consideration all of the tokens with their"}, {"start": 1378.76, "end": 1385.72, "text": " corresponding probabilities. So, that's one task you can, aside from semantic masks, you can use"}, {"start": 1385.72, "end": 1391.0800000000002, "text": " depth masks. As you can see, they get these weird pigeons. They scared the shit out of me, but like,"}, {"start": 1391.0800000000002, "end": 1397.72, "text": " yeah, I don't know. And additionally, you can see they can kind of condition on these"}, {"start": 1397.72, "end": 1406.52, "text": " pose kind of images, and you get these as output. Like, I think this girl broke her"}, {"start": 1406.52, "end": 1411.8, "text": " leg, but yeah, okay, she'll survive. And finally, you have class conditioning. So, basically,"}, {"start": 1411.8, "end": 1418.92, "text": " they conditioned on the class label of a frog, and they generate these novel, scary frogs. Okay,"}, {"start": 1418.92, "end": 1424.28, "text": " and additionally, they have something called rejection sampling. We'll get to that in a moment,"}, {"start": 1424.28, "end": 1429.6399999999999, "text": " but like using that, they can achieve much better. They can get much better images than these here."}, {"start": 1429.6399999999999, "end": 1437.72, "text": " I think these were not obtained by using this rejection sampling. Okay, let's see other results."}, {"start": 1437.72, "end": 1443.8, "text": " So, again, these are some awesome images they can obtain using semantic segmentation masks. Here on"}, {"start": 1443.8, "end": 1449.72, "text": " the right, you can see again, like depth images, semantic masks, edges, super resolution, a bunch"}, {"start": 1449.72, "end": 1454.92, "text": " of different tasks they can apply this on. As I mentioned, they say they're on par with other"}, {"start": 1454.92, "end": 1459.96, "text": " methods. So, for this semantic image synthesis, they compare with the spade. It's even a bit"}, {"start": 1459.96, "end": 1466.92, "text": " better. So, spade is a bit better than their approach, but yeah. Okay, finally, comparing"}, {"start": 1466.92, "end": 1473.72, "text": " to other methods on the face synthesis, face image synthesis task, looking at the FIDs, we can see"}, {"start": 1473.72, "end": 1481.56, "text": " that this paper is a bit worse than this PGGAN, and it's a lot worse than StyleGAN2, looking at"}, {"start": 1481.56, "end": 1487.56, "text": " the FID metrics, so fresh-ase inception distance metric. They are way worse compared to StyleGAN,"}, {"start": 1487.56, "end": 1492.84, "text": " but as I said, they're not claiming they're better. In that sense, they're just more general,"}, {"start": 1492.84, "end": 1496.92, "text": " and yeah, and they're the first to have used transformers to generate these higher-res images."}, {"start": 1496.92, "end": 1505.24, "text": " Okay, similarly here, if we focus on FID, we can see that without... So, keeping this acceptance"}, {"start": 1505.24, "end": 1510.3600000000001, "text": " rate, so that's the rejection sampling I was mentioning about. So, if you keep it at one,"}, {"start": 1510.3600000000001, "end": 1516.6000000000001, "text": " you can see that they arrive at FIDs of maybe 15, and that's worse than many of these methods here."}, {"start": 1516.6000000000001, "end": 1521.64, "text": " Okay, but if you start applying this rejection sampling, they drop the FID down to five, which"}, {"start": 1521.64, "end": 1526.04, "text": " is much better or on par with these methods, but they can also use rejection sampling, so I guess"}, {"start": 1526.04, "end": 1531.32, "text": " this is not that fair of a comparison. By the way, they use ResNet, they say it's somewhere here,"}, {"start": 1531.32, "end": 1537.6399999999999, "text": " they use ResNet 101 to do this rejection sampling, so here, but instead, if you'd use"}, {"start": 1538.28, "end": 1543.1599999999999, "text": " something better, like a clip model from OpenAI, they can achieve even better results, and you'll"}, {"start": 1543.1599999999999, "end": 1550.2, "text": " see those all around, like on Twitter, people using VQGAN plus clip to generate nice images,"}, {"start": 1550.2, "end": 1554.6, "text": " and so the whole trick with those rejection sampling is the following. So, you output"}, {"start": 1554.6, "end": 1561.0, "text": " a couple of images from your... You sample a couple of images from your... Like, from VQGAN,"}, {"start": 1561.0, "end": 1567.24, "text": " and what you then do is you feed these into, for example, clip, and you prompt clip with a specific"}, {"start": 1567.24, "end": 1572.1999999999998, "text": " class, so for example, you have class one, whatever, that may be like a frog or whatever,"}, {"start": 1572.1999999999998, "end": 1579.08, "text": " and what clip will do, it will say how probable this picture is to be of a class one, so maybe"}, {"start": 1579.08, "end": 1586.04, "text": " this one will be, I don't know, 0.7, this one will be 0.85 or something, this one will be, I don't"}, {"start": 1586.04, "end": 1593.24, "text": " know, 0.6, and so you can just sort them, sort the images according to these scores, and just pick the"}, {"start": 1593.24, "end": 1600.4399999999998, "text": " top three or top five, whatever, and what it turns out, it turns out this kind of is an automatic way"}, {"start": 1600.4399999999998, "end": 1606.04, "text": " of cherry picking, so it additionally improves upon the results that this paper achieved, reported."}, {"start": 1606.04, "end": 1613.3999999999999, "text": " Okay, what I show here is how that down sampling in the encoder affects the results, so what I do"}, {"start": 1613.3999999999999, "end": 1620.76, "text": " is they keep the latent space at 16 by 16, and then what I do is you can see here the number of"}, {"start": 1620.76, "end": 1626.12, "text": " times that the spatial extent of the image has been reduced, so 16 means that the original image"}, {"start": 1626.12, "end": 1634.76, "text": " was 256 by 256, one means basically input image was 16 by 16, and that means that during the training,"}, {"start": 1634.76, "end": 1642.28, "text": " they actually have to take a crop of 16 by 16 here and directly train the transformer on that,"}, {"start": 1642.28, "end": 1647.64, "text": " and as you can see, the receptive field is much, much, much smaller when you have f1, and because"}, {"start": 1647.64, "end": 1654.12, "text": " of that, you get these weird Picasso-looking images, and I really think, like, somebody can earn some"}, {"start": 1654.12, "end": 1658.92, "text": " money by just producing these and selling them as artwork. If somebody earns some money, consider"}, {"start": 1658.92, "end": 1666.92, "text": " becoming a patron of the epiphany, yeah, that'd be nice. Okay, when they use f2, so that's a bit higher"}, {"start": 1667.88, "end": 1673.8000000000002, "text": " receptive field, you can see it kind of gets better, but this really freaks me out, and then"}, {"start": 1673.8000000000002, "end": 1678.8400000000001, "text": " additionally, it gets better and better, and finally, when you have 16, assuming these are 256"}, {"start": 1678.8400000000001, "end": 1683.64, "text": " images, meaning basically you had a receptive field was spanning the whole image, and that's"}, {"start": 1683.64, "end": 1689.0800000000002, "text": " why we get much better results here. Additionally, these are much faster to generate, so we have a"}, {"start": 1689.0800000000002, "end": 1694.76, "text": " huge improvement in speed up compared to when you're trying to regress this in the pixel space,"}, {"start": 1694.76, "end": 1701.24, "text": " because here you'll have to regress all of the pixels, whereas here you'll just have to regress"}, {"start": 1701.8000000000002, "end": 1707.5600000000002, "text": " the 16 by 16 in the latent space and then decode it, so that's why we get a huge speed up. Okay,"}, {"start": 1707.56, "end": 1713.3999999999999, "text": " I think that's pretty much it. Let me just show you a couple more things that's kind of interesting."}, {"start": 1713.3999999999999, "end": 1720.44, "text": " So here they show that they are better than Delle and VQVAE2 when we look at the FIDs of the"}, {"start": 1720.44, "end": 1724.76, "text": " reconstructed images. So you just, they just compare, they take the validation images, for"}, {"start": 1724.76, "end": 1730.9199999999998, "text": " example, so that's this FID slash VAL, so they pass it through the encoder and then decoder,"}, {"start": 1730.9199999999998, "end": 1735.8799999999999, "text": " and then they just calculate the FID, and so this just shows that the reconstruction is coming from"}, {"start": 1735.88, "end": 1741.16, "text": " this VQGAN paper are better than Delle and VQVAE, as we can expect, because as I mentioned, they are"}, {"start": 1741.16, "end": 1745.88, "text": " blurry because they're not using adversarial component, neither did they use perceptual loss."}, {"start": 1745.88, "end": 1753.88, "text": " Okay, one more interesting thing I want to show you in the appendix. So basically here is that"}, {"start": 1753.88, "end": 1759.96, "text": " visualizable, I just explained. You have Delle here, you have VQGAN here, you can see these images"}, {"start": 1759.96, "end": 1765.24, "text": " are much more blurry compared to VQGAN, which has adversarial component, so that's really"}, {"start": 1765.24, "end": 1773.8, "text": " cool. GANs are like famous for making images much more crispy. They have many more diagrams and"}, {"start": 1773.8, "end": 1779.0, "text": " pixTiv included in the appendix, it's huge, it's got maybe 50 pages, you can check it out, some of"}, {"start": 1779.0, "end": 1783.4, "text": " the images there, but that was in a nutshell everything I wanted to cover today. If you found"}, {"start": 1783.4, "end": 1800.76, "text": " this video useful, share it out, subscribe, and until next time, bye-bye!"}]