DALL-E mini explained | min(DALL-E) | Craiyon | ML Coding Series
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In the 6th video of the ML coding series I start explaining the DALL-E mini project - the open-source implementation of DALL-E. I start with its minimal port into PyTorch called min(DALL-E).

I first give you the necessary context by walking you through the VQ-GAN, BART, GLU, and DALL-E papers as well as the Weights & Biases report on DALL-E mini, and then I dig into the actual code!

Let me know how you like this video!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ min-dalle code: https://github.com/kuprel/min-dalle

My previous relevant videos:
✅ VQ-GAN: https://www.youtube.com/watch?v=j2PXES-liuc
✅ VQ-VAE: https://www.youtube.com/watch?v=VZFVUrYcig0
✅ DALL-E: https://www.youtube.com/watch?v=jMqLTPcA9CQ

✅ Weights & Biases DALL-E mini report: https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-Mini-Explained-with-Demo--Vmlldzo4NjIxODA
✅ Rivers Have Wings tweet: https://twitter.com/RiversHaveWings/status/1478093658716966912
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00:00 Intro
00:02:12 VQGAN overview
00:08:42 Conditioning in VQGAN
00:14:00 BART transformer
00:18:25 DALL-E 1 overview
00:24:13 DALL-E mini Weights & Biases report
00:30:35 [code] min-dalle
00:34:23 Text tokenizer
00:41:50 BART encoder
00:44:22 GLU explained (paper + code)
00:51:22 BART decoder
00:58:35 Image latent vector autoregressive generation
01:05:10 Super conditioning, top-k sampling
01:09:53 VQGAN decoder
01:15:05 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#dallemini #imagesynthesis #mindalle
 What's up guys in this video I'm continuing on with the machine learning coding series and in this one I'll be covering the Lee mini. So that's the open source implementation of the of open AI's Dali model and when actually say I'll be covering the Lee mini what I mean I'll be cover covering this minimal port of the Lee mini that's inference only and that's Basically written in pytorch and the reason is it's much easier to go through this one and to set it up on my Windows machine, so we'll start with this one, but in the next video. I'll actually be covering the full the original Jack's code base of the Lee mini which is much more involved But this video will give you enough context to know what's going on Okay, so as Usually I'll first give you the additional context in the form of explaining the papers behind this And one thing that's very important to understand is that the Lee mini is not actually an implementation of the Lee model It's more of a mixture and let me open up my one note So it's more of a mixture of this paper called VQ GANs or taming transformers for high resolution image synthesis So that's the paper that introduced the VQ GANs And I've covered this one and all of these papers actually pretty much in in my previous videos So I'll be linking those as we go along So this is the VQ GAN paper, and they also leverage this model which is encoder decoder transformer model called BART And they use some modification of BART. They use this glue from Noam Shazir. We'll see all of those details a bit later and Here is the original the Lee paper, and then I'm gonna kind of contrast What the Lee mini is with the actual the Lee paper? They also have like bits and biases also have a report which will like literally skim through I'm gonna show you Some of the they actually have very nice training and inference Diagrams here how the Lee mini works and they also have Comparisons between the Lee and the Lee mini and you can see there is quite a lot of bullet points here So I'm gonna go through all of those a bit later. Okay, so let's first start with with actual papers I'm gonna skim. I'm gonna literally just give you the gist because otherwise there's like four papers That's that would take too much time. So let me go straight to it So this is the VQ GAN paper and as I said, I previously already covered it So if you want a deep dive go ahead and check it out. I'm gonna link it somewhere here Okay guys, so again on the high level Keeping a high level because I've covered this in great depth in a separate video We have two components of VQ GAN basically One is the learning the visual Codebook here and that's this component here and Then the second part is after you learn the visual codebook. Then you basically train a transformer model to just to autoregressively model the latent space of visual vectors here So this is the transformer part. So basically as I said, we have the first part here and we have the transformer part here So how this is trained is very similar to VQ VAE again That's a paper I've covered before and a couple of differences are that they are using this thing called reconstruction loss And the second thing is they're using discriminator here So basically adversarial approach and because of that the images from VQ GAN are much crispier Compared to what you get using VQ VAEs, which is something that actually the lead version one was using Quickly running through the pipeline here. What you do is you feed an image here you then encode it into a Lower result into a lower dimensional latent space here So you end up with some vectors as you can tell here and then what what what you do is you kind of snap them onto the closest visual Vector that you learn in this in this codebook so you can see that for example for this vector here the closest vector In as you can see here L2 by using the L2 as the definition of closeness You snap it onto a vector number one. So that means this one was closest to this one to this vector Number one from the visual codebook. Okay So once you do that for all of the vectors from your low dimensional space here You snap them onto discrete vectors here then you pass them through a convolutional decoder to get the image out and Then again, I'm going to show you the losses in a second But you train them using a couple of losses such as a reconstruction loss the perceptual loss and the and the Adversarial loss here and then the transformer part is I guess very self-explanatory Once you have this whole pathway here that allows you to map your data set of images into a data set of basically tokens because you can flip now this in our Raster order and you end up with you can directly map your image into a sequence of tokens Then you basically can just train your transformer on this new data set of Basically sequences and that's how you learn how to well the data distribution of interest Okay, so that's that's it on a high level here a couple of formulas I'm going to show you so basically as I said, what do you do is you snap your your vectors? So Basically and I have some glitch with my one node so you can see you'll you'll be seeing this thing kind of staying here Unfortunately, but yeah for whatever reason one out is glitching today So here is the vector that you estimate then you snap it on to the closest codebook vector here That's that the quantization step here. So you first encoded then you quantize it and then you have the Basically decoder part and that's how you get the reconstructed image X hat out and with that notation under our belt here is the actual loss The weak you lost is just as you can see here are some of couple of losses One of them is reconstruction loss, which is L2 in this case here And then there there are a couple of these losses which they I think they call commitment loss This one so the idea as you can see here is you put a stop gradient? in this particular case here on on top of your codebook vector and then you try and and and make sure to Make that to make your vectors closer to the codebook and here is the opposite here You you kind of put a stop gradient operator on top of your estimated Vectors and then you try and make your codebook closer to the output of your encoder as you can see here So that's the that's that and then the VQ again actually adds a couple of modifications So first one is they replaced the L2 loss as you can see here for the perceptual loss and They introduce another serial training procedure as well with the patch based discriminator So you don't have a single scalar you have as you can as you could saw see on the on the image here You actually have basically you're doing a patch based discrimination of whether that patch is real or or false Stemming from real or false image and that helps additionally that can increases the bandwidth of the learning signal compared to just using a classical scalar Okay, and that's pretty much it. That's what we can changes So we have the VQ component and then we have now the GAN component and again The difference here is that they are using the perceptual loss if you're not familiar with what that is Check out some of my neural style transfer videos But basically the idea is to take like a pre-trained convolutional neural network such as like a maybe VGG pre-trained on like image net Or something and then you instead of doing the L2 in the pixel space of the of your reconstructed image you actually feed it inside of that like a Network and then you do L2 in the latent space of that network where the features are much more abstract and you're you're basically trying to find Well, you're basically capturing the semantics much more than than than high level noise in the picture space Again refer to my video if you want to know more details But this will be enough more than enough for the for the for the code walkthrough Okay, guys, so let me show you one more thing in this paper and that's conditional synthesis So dimension here. So if the conditioning information C has spatial extent we first learn in other VQ GAN to obtain again an index-based representation as you can see here So you'll have what they say here you they'll have A height times width of these vectors where the set of vectors has is this big there is like big Z sub C Number of those vectors. So that's the size of the code book for that Condition conditional information. So they're there in concrete. Let me give you a concrete example here This could be maybe like semantic segmentation Maps that you can then basically learn a VQ GAN on top of that type of a data set. Okay Well blind then you have this newly obtained code book and then due to autoregressive structure of the transformer we can then simply prepend R to S which is the Well, that's your your your your image from your target data set and R is stems from the conditioning information and then basically you do the Negative log likelihood to these entries. So in case that was not that clear Let me quickly draw it here. So again, imagine you have Basically some semantic segmentation Masks here. So maybe something like this. I'm gonna kind of try and draw it somehow so This is some type of a semantic segmentation image and you have a big data set of such images Big data set of such images and now what you do is basically train a VQ GAN Such that you'll end up with a particular code book For that vector for for that data set so this is going to be your your code book For this particular Image here, so you then encode it you end up with basically Well, you end up with HC times WC vectors you then flatten them out and You end up with a sequence and This sequence that which they denote as R is something you can prepend while training on your on your data set Okay. So now obviously for each of these images you have an actual image you care about which is the Image from your from the data set you're trying to for which you're trying to learn the data distribution off So that there will be some image here, which is actually original image And this is its semantic segmentation and then you do the same thing here. So you'll you first learn the VQ GAN component So something like this and I'm really terrible at drawing today And and then you encode it here You flatten out this sequence. I'm going to denote it with a different color. So we now have a sequence here And now how you're going to train is you're going to prepend this vector R so we'll you'll grab the vector R from the semantic segmentation map and Here you have the target image which they denote as S And now you just collect these tuples for your whole data set So you basically collect this and then you learn how to to model that type of uh of of of sequence And so how do you later when you want to generate? A new image you literally you literally grab an arbitrary semantic segmentation map And you you pass it through the encoder here you fetch the This vector R and then you pass it through the Decoder you fetch the uh this vector R and then you put it inside of your transformer So here you'll have a transformer You put the R here And then you ought to aggressively keep on Generating the sequence and then you pass that through the decoder here So once you generate that sequence here you pass it through the decoder and out comes the image So this will maybe be in practice like 256 tokens Which you can then rearrange into 16 by 16 Tokens something like this And then that's that's like that's the that's the low uh dimensional representation And then you can as I said you pass it through the decoder and you get your image out So why I took some time to explain this is because um Dali and and uh dali mini, uh use a very similar approach instead of uh, and instead of using the uh Semantic segmentation masks they're actually using text but the logic is super similar. So just instead of um uh doing this whole thing you um Pass a piece of text that's associated with your image Here and then you learn how to model them together and that's it. So literally by doing that you learn the um You learn the data distribution uh p x y and that's where the bart paper comes into picture. So that's that's where this paper comes into play I'm gonna explain to you. Whoops Um, i'm gonna explain to you in a second like it's a it's a very simple It's basically like an encoder decoder transformer. There is not too much to say there Okay So again how bart will come into play here is instead of doing this pipeline with semantic segmentation masks You'll instead have uh an image And you'll have associated caption here. So some piece of text here. Okay, so have some piece of text here And you'll basically feed that text through uh encoder part of of uh of bart So let me kind of denote the uh bart Such as this like we have encoder part of bart and we hope we have the decoder part Of bart decoder is obviously attending Uh the encoder part that's how the transformers work. And so you you grab this text here You basically feed it directly into the encoder part and out comes something from the decoder Out comes this thing here and now you use this output of the encoder as r and that's it And now you just again you you'll now have these tuples of uh, this conditioning vectors And your uh images from your data set and you just learn how to model the p x y So the the distribution of your uh, like images and uh and captions and by the way, this is literally this this These two boxes is everything you need to have like that's a mental model of bart I have when i'm when i'm discussing it. So the only thing we need to See to understand how bart works compared to all of the other transformers is I guess this thing here so basically in your regular, uh, for example in inbert model so inbert What the pre-training objective looked like is you masked out so it's first is bidirectional So it's only encoder part of the of the uh encoder decoder original transformer structure Whereas bart is both encoder as well as the decoder So you're trying to just you just kind of mask out some of the code So you're trying to just you just kind of mask out certain tokens and I think like you do that with like 10 20 percent Probability for each of the tokens And then you try to predict what the token at that particular position was so it's a simple, uh, like a auto encoding Objective, uh, and what's different with bart is you can kind of model You can kind of take a span of of tokens and then mask them out with a single Mask token and then make sure to predict the original sequence here So because of this thing because of you can mask it out here However, um many tokens you want and then predict them here you have additional flexibility Which you don't have with a pure encoder, um, like transformers such as such as bert And I think they mentioned it somewhere here. Let me find that piece of text So the objective is called text in filling and they mentioned here a number of text spans are sampled with span lengths drawn from a post Poisson distribution with lambda Parameter equals three and each span is replaced with a single mask token zero length spans correspond to the insertion of mask tokens and Okay, it's inspired blah blah blah So text in filling teaches the model to predict how many tokens are missing from a span And that's all like it's literally encoder decoder transformer with this particular denoising objective And they show a bit later here how it compares against the baselines using different types of Like retraining objectives such as just token masking a la bert or token deletion or text in filling So a bunch of different sentence shuffling document rotation, etc, etc, and they compare that against the Language model so that's just predicting the next token or mask language models such as birth, etc, etc, and they show Well competitive results Okay, that's the bird part the bard part and then there's a small modification called glue Which i'll show you a bit later. So glue variance improved transformer from this author noam schazer Who is actually the co-author of the original transformer paper. So it's a very prolific author But before that, let me just quickly skim over over dali version one Which is which came from the paper zero shot text image generation again Super similar pipeline to vq again Again, they mentioned here they have two stages. So in the first stage they say here we train a discrete variational autoencoder to compress each of the 256 256 rgb images into 32 by 32 grid of image tokens each elements of which can assume 8192 possible values so that that means that the codebook size is this big and this reduces the context size of the transformer by a factor of 192 without a large degradation in visual quality because if you divide basically 256 times 256 times 3 because you have rgb image into 232 times 32 you end up with 192. Okay and The same authors of this paper. Well, I think this is from yeah, this is from open.ai So basically they they they previously trained this model called image gpt and in that model They were training they were autoregressively learning how to model the the images in the pixel space, which is much more complex because Transformers have quadratic complexity and obviously the image space can can grow like up to like Well thousand by thousand images and so it becomes very quickly impossible to model in the pixel space So that's why this reduction this mapping into a lower dimensional space here. Okay So again, that's stage one and it's literally the same as vq again without the Perceptual loss and without the adversarial component and then they say here In stage two we concatenate up to 256 vpe By pair encoding text tokens with the 32 times 32 image tokens that come from the stage one and Train an autoregressive transformer to model the joint distribution over text and image tokens. So basically what I said here Is that so again they they train the vqvae which you can think as as same as vqgan So yeah, they have the encoder part. They have the decoder part They kind of have a low dimensional representation here and so what they did what they do instead so so they can kind of unroll this in a raster order to end up with a Sequence of tokens s so to get r so to get the conditioning vector And I don't know what why they're using r like I guess c c makes more sense But i'll just keep on using the same uh terminology So to get the r component That you'll use to to prepend here And then train or progressively train a transformer on top of it Uh, they just use as as you saw here They just have uh, basically they take a piece of text and they bpe encoded so you start with the text And then you apply the tokenizer which gives you a sequence of tokens It just gives you a sequence of tokens and then I assume they just embed those using like a shallow like a simple Well, you have an embedding table you map that into tokens and then you you place that directly here to start um modeling whereas Um, as you saw the lini is using bart encoder. So that means they'll have additional Layers of processing before they they output this r vector. So the the conditioning vector Okay, guys, that's that's pretty much it. Um, let me quickly show you what they say here There is a lot of complexity hidden in here As I said, I covered this one the lee one, uh, you know in a separate video so you can check it out They use stuff such as gumbell softmax relaxation To train this model, etc, etc. And also it's worth Keeping in mind. I think they mentioned it somewhere here Let me just show you this so getting the model To train in 16-bit precision past 1 billion parameters without diverging was the most challenging part of this project so training this um Big model well back in back a couple of years ago. This was considered a big model Was was extremely hard and um, that's something I think boris the the main developer of the um, the lee mini Project is also struggling with but he's using jacks and he he has like free tpus From google so that kind of helps Because tpus have they did mention somewhere here that tpus have some less problems than than than gpus in certain aspects Let me quickly show you one more thing again Uh, let me just explain this conditioning part So given a text image pair we bpe encode with lowercase captions using at most 256 tokens with woke up size 16 384 and encode the image using 32 times 32 tokens with woke up size 8192 The image tokens are obtained using argmax sampling from the dvee Via encoded logits without adding any gumble noise. Uh, finally the text and image tokens are concatenated and model Regressively as a single stream of data. So that's something we saw multiple times already Okay, guys, so that's it. It's a very simple model as you saw here Uh, there is a lot of overlap between the ideas between vq gam paper and the lee Uh version one Um, they took the ideas from bar. They took some other ideas Uh boris is experimenting a lot as soon as a new paper comes out such as the deep net paper With novel initializations. He tries an experiment with it. Okay, so let me now show you the uh report that Uh boris wrote with the other other co-authors Again They have a very nice drawing of the training pipeline of the lee mini Model as you can see here again, they have the Bart encoder so you as you can see you you first have these Puppets of images and associated captions then you pass the caption through the bart encoder you get the the conditioning vector here You take the image you pass it through the vq again encoder you get the image tokens you feed them here So you can see here vq again, uh tokens And then you pass all of that through a bar decoder and you learn how to autoregressively model these sequences Uh simple stuff as for the inference we can see it here Uh, so you grab a piece of text you pass it through the bart encoder You feed it into the decoder and then you autoregressively generate the image tokens Uh, you can generate multiple Uh series of these image tokens because uh, the process of generation is inherently stochastic Then you pass all of those through the vq again Uh decoder you get some associated images So the token sequence from the low dimensional embedding space is projected into Uh the image in the pixel space And then you can use something like clip to filter out the best images So you again feed the same caption to clip and finds the image that That is best described by this particular caption you can see in this particular example White snow covered mountain under blue sky during daytime clip found that this image here is the best this is is uh Well best described by this by this particular caption. Okay That's it. Let me quickly Walk you through and I encourage you to check out the whole report In this in this awesome, uh weight and biases report, but let me show you some some differences that the uh authors Of the lee Mini model themselves mentioned so we are grateful for the research and pre-trained models published by open.ai which we're essentially building our model Uh, not all the uh, the details on the leak our public knowledge But here what we consider to be the main differences So the lee uses a 12 billion parameter version of gpt3 In comparison our model is 27 times smaller with about 0.4 billion parameters Now this report was written a while ago. And so this was the the lee mini Like they now have the lee mega as well And also I think there is a mistake here because i'm fairly sure the original dali Paper used gpt2 not gpt3 Which are fairly similar but still Um, then they say we heavily leverage pre-trained models such as vq-gan barding coder and clip We mentioned all of those so far Uh, and while open.ai had to train all their models from scratch our model architecture takes into account pre-trained models available And their efficiency The lee encodes images using a larger number of tokens So they use 32 times 32 whereas the uh, dali mini use uh uses 16 uh by 16 from a smaller vocab And the lee uses a vqvae while we use a vq-gan Okay, then they say the lee encodes text using fewer tokens at most 256 Uh, whereas they where the lee mini uses 1024 And the lee reads text and images as a single stream of data while we split them between these uh, well bard encoder and decoder Uh, this also lets us use independent vocab for text and images The lee reads the text through an autoregressive model while we use a bi-directional encoder Okay, so I made a small mistake when I was explaining how the lee wevon works. I'm gonna Show you what I mean by that in a second, but let me finish here So the lee was trained on 250 million pairs of image and text while we used only 15 million pairs Okay So the difference is scale obviously the model scale the dataset scale etc, etc But also a lot of difference in the architecture and the training method like they don't have the gumbo softmax They don't have the vqvae. They're using vq-gan. They're using bard instead of using gpt, etc, etc So there is so the lee mini is um, well not really the lee model It's a it's a mixture of of vq-gans and bard and everything else So so yeah, but still and that's why they are now called crayon not the lee mini But in any case, let me quickly go back to notebook and then we're jumping to code So I think I mentioned somewhere here that they bpe encode Text and then they just learn how to autoregressively Model that and well, I kind of did correctly explain it but not quite so what they do is so let me just draw this again here so we have the Conditioning part and we have the target sequence So what they do how they learn this is they feed all of these into the transformer And then you learn how to predict all of the tokens. So basically here So yeah again as a target you just take this same sequence and you shift it leftwards by by one uh by one step and that's what you're trying to to predict so this is what you're trying to predict and then here ultimately you have the uh end of Image token in this particular case because in this particular example the green part Are the image tokens from the vq vae portion? Okay, so that's it The main thing I want to stress here is that the the embeddings for the conditioning part are not shallow they're using uh deep transformer layers because as you as you can as you know, like the uh transformer works by Cross attending um previous tokens here. So it will be attending to deeper representations of this particular Text here. Okay guys this there was a bit more than I wanted to To give you when it comes to context and and paper overviews, but it is what it is Is let's now dig into the actual code. Okay, so here we are in my vs code I'm gonna open up the i'm gonna hit debug here and we're gonna slowly start stepping over the code again I'm gonna skip all of the unnecessary details. I just want to show you The similarity between what we just saw theoretically and the actual code implementation again this uh I'm showing you the code from a mindaly uh repo and uh, they only have the inference portion of the pipeline and they're um, It's it's a port in into pytorch and it's whereas the original repo was written in jax We have some arguments here as you can see, uh, so i'm using the mega version and not the mini version Uh fp16 set to false so that we have higher quality images, but the footprint will be bigger So I have a 8 gigabyte tpu and uh Otherwise, you'll have problems executing this thing. So let me show you uh, we'll see how the um GPU memory is slowly increasing as we're going through the code There is a couple of optimization in the code as well such as as soon as the encoder Is created and and the image is encoded they uh dispose of the encoder So that's controlled by this parameter. Let me see whether They have it here. I don't think it's in here. They have a parameter that literally controls that that that behavior Okay. So again, we have uh, we're using mega we have a piece of text I just inputted a random caption here a photo of a funny dog. Let's see what we get There is some seed there is a grid size. We're just using we're just generating a single image top cape for top case sampling Is set to 256? Uh image path, uh, so this is going to be directly dumped into the root root of of this repo so the image will be saved there Uh, the models are uh, well already downloaded into this pre-trained I went ahead and downloaded them beforehand so that we don't have to wait and again as I said, we're using fp16 So let's continue here. So first off we're gonna generate the we're gonna create this Uh mindaly model and then we are gonna do this generation process. So let's see what happens here So again is reusable is set to false That's the parameter that's going to control that optimization behavior. We'll see that in a second. So let me continue here So here it is Nothing fancy there just storing these variables we have number of text tokens that's gonna specify the the maximal size of the caption Number of layers is going to be 24 for mega Uh, as you can see the difference between mega and mini is just in the number of layers number of attention heads Also the size of the embedding space The size of the glue embedding space we'll get once we get there I'll briefly show you the paper as well for glue and then the text vocab count Uh, and the image vocab count kind of differ between these two models in any case, we just uh form the paths here for the uh Well by dali here, they are literally referring to the bart part of the dali model. So that's kind of um Potentially confusing. Um, okay, so Basically, i'm gonna skip over all of this. We have a vocab json and merges text. So that's gonna Basically, uh be used to form the tokenizer the text tokenizer. Okay Okay, and now we just form some some um paths which contain the the pre-trained models weights And now we hit in initialize tokenizer. Let me just see what I have. Yep. I have a breakpoint here. So Basically, it's already downloaded so we can ignore all of this Uh, and now we just open the vocab and here here it is So let's see what's the size of the vocab. So length of the vocab is 50,265 50,265 Now we go and read this merge merges file that contains the basically all of the merges that happen through during the training of the bpe uh tokenizer Do check out my open ai clip video if you if you're confused by the tokenizer details i've covered that they are in a bit greater depth So let's continue here. We just stored the vocab. We loaded a couple of seconds ago We basically grabbed the the pairs from this Like uh merges object and we split them into well into doubles To get the pair. So let me show you an example of what the pair looked like so Pairs and i'm gonna print the first three pairs. You can see Inn characters are merged together Uh, and this weird I don't know how to pronounce this one, etc. Etc. So there is there's going to be a lot of those bpe merges Stored in the in the pairs this rank from pair is basically going to associate integer From zero to the length of pairs with each of the pairs and that's going to be used during the tokenization step I'm going to quickly Later show you how the tokenization Uh works on a very high level but i'm gonna skip this part here because it's fairly intricate to to to describe And uh, you can you can read on bpe yourself. We can just treat the tokenizer as a black box. That's fairly Common structure between all of these models. So i'm not gonna keep on explaining it every single time I did cover it in an open ai's clip, uh video okay, so Continuing on because the easy reusable flag is set. We're not going to beforehand create encoder decoder and the uh, The tokenizer which is the vqgen Uh decoder because otherwise the memory would literally explode so I don't have enough memory in my machine to do that So we're going to skip over that and now we have the generate image part Okay, so we have the model here and let's now generate the image again the encoder and everything else will now be built Uh on the fly so we have the text again. We have the photo of a funny dog blah blah same parameters before Let's see what this does. So there is this um, uh generator, uh function that they uh, Developed let me that's not that vital. So we're just gonna generate the stream and then grab doing next We're gonna grab a single image from the stream, but let's dig into the actual function Okay, nothing happened there because it's a generator So once then this line once we hit this line when they go f10, we're gonna enter the actual function. Okay, so Similarly here everything interesting happens in the generate draw stream part. So i'm gonna ignore everything else here So now let's see what's going on. Okay. So first off we um Depending on the grid size we estimate the number of images because the grid size is one for us We just generate a single image. So this grid they basically this project offers an Like an option to generate like a grid of multiple images here because of memory constraints I'm just dealing with single image so we can ignore the grid every single time Okay, so because we are in a verbose mode We are printing out some details here tokenizing text etc. Etc and now we hit the tokenization part So now we tokenize our input caption before we feed it into the bart encoder so let's see what's going on here, so they grab the Like a separation token the cls token And the unknown token here, uh, there's some processing that they do on on top of the text So they do lower casing and then they encode it as ascii with this error set to ignore and then they do decoding So this is just going to do some simple primitive processing of text Nothing fancy there. Let me let me print the text for you here Let's see what happened. Basically nothing happened with this particular sentence because it was fairly simple And then you literally kind of split the sentence into words And splitting is happening. So basically you use spaces to split the sentence into words And then each of the word of the words is going to be split into the sub words here Because the this get byte pair encoding which i'll skip height It's fairly complex to explain but it's just gonna split the word into bpe Like sub words and then each of the sub words is basically associated with the like with the specific index integer from from the vocab here and that's it on a high level and then we just kind of Bracket that with the cls token in the beginning and that this uh see a separation token at the at the end and that's how we form Uh, basically a sequence of tokens for our particular text Okay, then they do optionally here the cropping but because we are less than 64 tokens. I think this is set to 64 Let me see what this number is, but i'm fairly sure it was 64 Yep. So if if it's like a longer than that length then they crop the the the tokens But for us, it's not Okay, so just some printing because it's verbose mode blah blah blah Uh now As you can see, this is the interesting part. So they they create this variable Text tokens which is initialized as all ones and it has you can be confused by a couple of details here So first why why is once is because one is a bad token in this vocab? Uh, and that's kind of implicit here because this is as I said just a port from the original repo So sometimes sometimes things are hard to understand unless you well unless you understand what's going on So here here am I trying to explain to you that so that's one detail The second important detail is there is this number two? and why they do that is because they have a basically analog thing to Classify our free guidance that we saw in diffusion model So you can basically do the same thing for the class of autoregressive generative models and that's why they have two so as you see here they create the first sequence will just contain the cls and the Separator token so it's going to be literally empty sequence everything else going to be pad paddings So that's what we do on this line. And then on the second line we add the actual tokens. Okay So let me show you what this thing is so text tokens If I print text tokens zero and then first maybe seven tokens You can see it's an empty sequence because this is a cls. This is a Separator token and everything else is padding and the second one contains the actual Uh tokens for our caption. Okay, that's it Now we just convert that into tensor with a long data type We put it onto gpu gpu and we continue on okay Now here is the optimization I was mentioning because we're using this not reusable We're because we said this flag is reusable to false. We now on the fly initialize the encoder So the encoder is going to be the vq-gan encoder. So let me step inside of here and let's see what's going on So again, uh, because I already downloaded the model We can skip over all of these steps and we're going to focus on how this dali birth encoder is constructed So the interesting part as usually is going to be in the forward pass of the of the model and After we construct it you can see here. We just load the parameters Of the pre-trained model that was trained in the actual dali mini repository We're gonna load those parameters and then we're gonna push to do this this model onto the gpu, okay So let's see how dali birth encoder looks like Here we are. We have the the vocab size of 50k something we form the embedding tables for the Well, that's the the text vocab Because encoder again remember has the text vocab and the decoder will be generating the image token So it will have a separate image vocab Okay, so we generate that table the embedding vectors will be 2048 we then generate the positional Encodings here. This is going to be only 64 because if you recall before We had the cropping so that we make sure that the number of tokens that we feed into the bart Encoder is always up to 64 never more than that. And that's why we never need to go Uh above 64 positional encodings. Okay now we start forming the basically Encoder layers and we'll have 24 of these for the Bart encoder for dali mega I'm gonna hit go here and put a small breakpoint In an encoder layer, which is a simple transformer Uh block nothing fancy. The interesting part is this glue. So i'm gonna set uh one breakpoint there Uh, and that's it. So let's now continue execution here So we end up in the encoder layer as you can see, it's just a simple transformer Block we have the encoder. We have the self-attention module we have the some layer norms and we have glue which is just a modification of the the uh, your regular like um mlp that's used of the feed forward, uh, like a layer of the transformer block, okay So let's see how glue looks like here is glue It's a very simple modification by noam scherzer And i'm gonna now open up the paper side by side to show you What this is and to make sense out of it, but it's a very experimental finding So there is nothing. Uh, well fundamental understanding you can get from this. Okay guys here it is side by side here are the equations from the glue paper by noam scherzer And you can see that this layer here implements exactly this line here So the idea here was they they kind of like he ablated a couple of um, he made a couple of modifications Why not using different activation functions instead of rally? Why not using gelu or swish etc? Etc. Why not combine him combine them in a bit different ways using um Similarly to the gated linear units so you can see so Like the lee mini ended up using this variant here And let me just kind of convince you that that's indeed the case So let's see so we have we get the input x this is x and then we pass it through layer norm So ignoring that part we we pass it we then pass it through a linear layer so that's that corresponds to basically this bracket here and after that we we pass it through Gelu, so that's this part the sigma of this so that's this term And then we have v which is formed from x as well from the input. We also pass it through the linear layer so that's that's this part the x times big v and finally we do element wise product so hadamard product here and then we pass that product through a layer norm and So that's that's that's basically what happened here and finally we we do a feed forward through a linear layer So that's the w that's the that's the sorry. That's the w2 here And that's thus you can see that this simple forward pass of this glue model implements exactly This ffn glue from norm shimseyer's paper and that's it. Now. Let's go back to the code I'm gonna Remove this break point. So it's a simple simple simple modification instead of having just like a like a feed forward Like a basically a linear layer for all but a relu followed by a linear layer. It's a bit more intricate Uh, and they showed experimentally that this gives a better performance But there is no ultimately no theory behind why this should work and that's it. That's I guess deep learning Let's continue I'm gonna now We're now back into encoder layer and again encoder layer is simply a transformer block So we have as you can see here we have some um layer norm and then attention and then norm and then the residual connection and then Then we pass through the feed forward part and then we have again the residual connection and we return back the variable So that's it. So i'm gonna Basically ignore all of these break points and let's form the 24 layers Of this birth of bart Encoder so i'm gonna hit f5 i'm gonna form 24 layers and then We are now here We form additional layer norm Additional layer norm and then we form uh these token indices which are going to be used to index the positional embeddings here and that's it and we just Do times two because remember we are doing the classifier free Trick for all progressive models. That's why we have Batch size of two here Okay That's it. That's the that's the initialization of the bart encoder. Now we are loading the weights here and then we kind of Push those weights into the encoder structure and then we delete the parameters now You can already see that this is pushing my gpu somewhat. Well, well actually not that much Okay, so i'm surprised. This is actually not that big of a memory footprint I think some of the later components will be spiking it up. But yeah, let's continue so After the initialization of the encoder, let's see what else happens Okay, and now we grab the text tokens and we pass them through the bart encoder as we saw in the paper part of the video Okay, let's step into it so we're gonna hit the for function of the Bart encoder here So let's see what happens. So we ask where the text tokens are not equal to one and one is again remember They're not using a constant here. They're just using one to represent the pad tokens It's kind of hard coded and dirty but it works Okay, so not equal to one That's the actual content. That's where attention mask is going to say true So let's now do something like this. Let me print this you'll see that It basically for the second component of the text tokens where we actually passed the the the caption tokens We have a bunch of truths here But for the first one we only have true true for the first two because those are the cls and the separator token Okay now what we do is we pass those tokens into the We embed them and then we just add the positional encodings using the these uh, well the the pose Those tokens indices that were previously constructed and that's it. That's simple stuff there And now we just pass it through the layer norm and then we keep on processing those embeddings Via the transformer Bart transformer layers and that's it. I'm just gonna hit that five here We're gonna do a forward pass through the transformer and we end up with the final representation here called encoder state Okay, and because again, this is the optimization I was mentioning. We now delete the weights of the encoder Let's see how the the memory footprint is gonna reduce. Okay, so the memory footprint went up You can see it here and after we hit delete and empty cache, you're gonna see how it kind of dips down Hopefully, I'm not sure what's going on Why is my gpu not reacting here? Not sure in any case Let's continue so now we have the initialized decoder part i'm gonna hit Step over and we enter the init decoder So again, it's already downloaded so we don't care about that part Here we construct it and again, we'll load the weights a similar structure as as with the encoder Now we are passing the image vocab, etc. So let's enter the constructor here So let's see what's the difference? So here we form again the embedding table this time. We have the image vocab size plus one Because let me just think here because they also have the they pass this this beginning of sentence token And that's the additional token that they need in addition to the image to image code book So I think they mentioned this somewhere in the in the report Let me show you this so basically It's called bos token Okay, so they say here at inference time blah blah. So a bos token is fed through the bar decoder And yeah, that's so that's basically why there is plus one as I said not not the cleanest way to do something But yeah, this is just a port of the original repo Now we have we form the table for the position in encodings and there is only 256 of those because as you recall from the paper part It's going to the latent space is going to be 16 times 16 of those basically vectors That are then going to be fed into dvq again decoder. Okay, let's continue here So we now generate a bunch of decoder layers And the coder layers have additionally the cross attention component because they are the cross attention component because this is again encoder decoder transformer so, let me see whether it makes sense to To kind of show you that part Let me just think Okay, i'm gonna put a break point into the mid part here and we can maybe do a single pass Here because they're doing something interesting With this attention state we'll see that a bit later And here as well i'm going to put the break point here And now let's continue. Okay, so let's hit the first decoder layer It's again Simply there is a self-attention part There is the decoder there is a cross attention part and there is the feed forward part So the glue variant in this particular case, so that's pretty much it So i'm going to now remove the break point from the init And let's just generate a bunch of these layers layers oops Let's now do that and we do that for 24 times again. I think let me just print the Layer count 24. Okay, so there is 24 decoder layers as well Okay, we have some layer norms nothing fancy and finally we have we take the final embedding vectors and we map them into the The space that has this dimensionality because remember we now want to sample those image tokens That's why we need to to to go from 2048 which was the internal model dimension. We want to go into into this dimension here Okay, let's continue on here And that's it. Okay, that's pretty much it. We load the weights again 24 layers of decoder blocks and finally then we we map into the image vocab Size dimensionality space output space so that we can sample from it later Okay Again, we delete the parameters. We put the decoder on top. We push it to our to my gpu. So let's see whether Something changes here. Nope Uh, it's a bit different behavior compared to what I've seen before usually there is a spike um Bigger than this one happening and i'm tracking this uh slot here by the way Okay In any case now comes the interesting part. So here we are going to now start sampling from the decoder Those image tokens and this one is already trained So everything it samples is going to be meaningful and then we're just going to pass it through the vq GAN decoder and that's the that's the end of the program. Okay, let's go here. So um We're dealing with float 32. So this doesn't make much sense, but with float 30, uh float 16 Uh, this kind of puts it puts all the operations inside of this context in the float float 16 regime which helps us save some memory So in the case when you have multiple images Uh, this expanded indices make sense because you will not replicate The the textual captions for multiple images because you'll be generating multiple images for one Particular caption, but we don't care about that here so we just basically Grab the uh encoder state and the and the and the text tokens and those are of dimensionality So this is going to be 264 if you recall and this one is going to be I guess uh to 2000 something 2048 because that's the Output of the encoder, right? So to oh actually yeah. Yeah 264 2048 because we just embedded this into this dimensionality here. Okay, so we formed the mask again This is the padding token. So mask is going to be true for for the non-pad tokens Uh, here is the we form this attention state which is gonna contain the Keys and values from each of the layers and for each of the positions in the bart decoder That's just how they decided to implement this and finally image tokens is gonna contain That's the output array that's gonna contain two 256 tokens Plus one because the first one will be the I think yeah the bos Uh token the beginning of uh sentence token. So let's continue we're gonna initialize this um Vector initially as you can see here with this value here, which is going to be the beginning of sentence uh tokens id so let me show you what That looks like i'm going to take the first five elements and all of them are going to be equal like this So that's how how they initially Um, well initialize the image tokens and then later we're gonna keep on as we produce the image tokens We're gonna feed them back into this image tokens array. Okay So nothing fancy there Uh continuing on we form these uh token indices And we have some settings like temperature is set to one top k 256 superconditioning factor Which is used for the classifier free, uh Uh guidance part and that's it. Those are the settings and now we start iterating basically a for loop 256 times we're gonna generate a novel image vector Uh, and uh, basically then we're gonna feed that into the vq-gan decoder as I said multiple times already so We care about this part of the loop because this one we'll enter this one once we have 256 tokens Um, we'll enter the image grid from tokens function So let's focus on this part here. So what happens here as you can see whatever we take the image So i is initially zero, which means we initially feed the b.os So the beginning of of of sentence token And we feed the output result into into the first slot of this image tokens And that's how we start populating this the image tokens Array again, its dimensionality is one plus 256 It's precisely because of this behavior here We pass the tokens we pass the attention mask the encoder state the attention state Which is going to contain the keys and values from the decoder And that's it and we we pass the token indices. So we basically pass Where we are in the generation? uh In the generation sequence, okay in the generated sequence. Okay. Now let's step inside of the uh forward function of the bar decoder so We grab the number of images uh, we then basically Make sure that the batch dimension is corresponds to to the super conditioning Thingy, so it's going to be two Okay, so it's going to be two there We grab the the list of previous tokens uh, so previous tokens dimensionality is Uh one so that's going to be initially just a b.os token and then Because we only have a single image. This is just going to replicate Uh the previous tokens we're going to have two b.os tokens And that's it Again, the reason being the super conditioning That they are doing so the classifier free guidance thingy for autoregressive models Uh some clamping. I don't think this actually is needed And then so now we do the embedding but remember this time. This is the these are the image embeddings Okay, so these are the as you can see here. These are this is the embedding table This is a separate vocabulary compared to the text that we use for the bar encoder part. Okay, so we do the encoding Then we add the positional Encoding here And now we have the coder state. It's going to be I guess What two? One 2000 something maybe Yeah, no two two two thousand forty eight makes sense Okay, because we have b.os a single token we have two because of super conditioning and we just embedded them into this Dimensionality here. So all of that makes sense. Let's continue on here Now we add the additional dimensions now. It's going to be 21 2048 and now we just iterate through the decoder layers of this decoder So we pass the decoder state The encoder state because we have the cross attention. Remember we passed the attention state. So attention state is going to be Basically um number of layers uh, and then four because we're going to keep keys and values and times two because we have the superconditioning and then finally we're going to have 256 and 2048 so again 24 because we have 24 layers four because we are saving key and value For each of the layers but times two because we are using Uh two sequences the one with uh empty sequence and the one with the actual caption because of superconditioning and then 256 is because we have 256 image tokens in the decoder and 2048 because that's the model dimension. Okay, just kind of breaking down. The shape is always useful we pass the attention mask for the Input caption and we pass the token now, let's hit the forward function here So here we just generate the attention mask depending on where we are in the generation process So because token token index currently is zero We are we're just we're we're just have passed the b.os. So the beginning of sentence token And that's why the self-attention mask will currently just be active in the first In the zero slot and everything else is false. So you can see here That the first will be true and then everything else is false is false and then we just kind of make sure that we have batch dimension of two because of superconditioning And then we just pass the equator state through the layer normalization. Nothing fancy there and now we pass it through the Attention part. So i'm gonna skip the attention part is fairly. Um Well, actually, let me see This is where we will store the keys and values. Let me show you this part So let me just see whether I have uh Okay, so we have a self-attention we have a break point there so i'm gonna hit That break point and let's see what we do. So we store the keys and values And let's see what their dimensionality is. So we have two one 2048 and the same for for a value vector We basically concatenate them and then we store it inside of this attention state And that's it. And now grabbing from the attention state We're going to grab all of the previous keys up to this token Which in this particular case is only these keys and values because we are just starting the generation But that's how the how the logic is implemented using this attention state We're basically fetching all of the previous keys and values and that's it And now we just pass it through the we just do the attention logic and we return back here That's it now we have the uh cross attention part we passed the decoder and encoder Uh, and that's it. Everything else is standard transformer logic. We pass it now through the glue and that's a single step through the Decoder layer i'm gonna now remove the break points here I'm gonna remove the break point here as well And let's continue on here I'm gonna hit f5 we went through all of the layers and now we pass it through uh, again Uh, this final l n is just like layer norm Uh, and finally here is where the mapping happens So here we're gonna take the vector that's 2 1 2048 and we're gonna map that into the corresponding into the image space. So basically now the logits Will be of 2 1 16 000 416 Because that's the basically the image space Dimensionality we grab the settings such as top k temperature is super conditioning vector now this part is kind of um Not completely clear because it's so so hard coded. So let me try and uh, break it down so first we grab all of the um Our long the batch dimension we grab everything and that makes sense we take minus one because we only only only want to we only care about the last the logis from the from the Um from the last token because we're trying to sample from there that makes sense, but this part doesn't make sense. Why is it? um this number Which is smaller than the number here And my hypothesis is that well it might be that the original vq. Gan checkpoint they used Um had a bigger vocab compared to their target fine tuning After they've done the fine tuning of of the gans not completely clear About about this part in any case. So here is the super conditioning part So we grab the logits and as you can see here we grab now we grab the so these are the logis that correspond To the empty sequence and these are the logis that correspond to the actual caption And we do the logic we saw in the diffusion models as well and that's how we combine the logis to To form the final Supercondition distribution here. Okay, so I think let me just check this Fairly sure so this is this reverse half wings Twitter profile I think was the first one to have suggested this type of super conditioning for Uh, ultra-regressive models. So so she said you can apply a similar trick to classify our free guidance to other regressive transformers To sample from a synthetic supercondition distribution Uh blah blah blah. I trained to try this and you can see how well, basically you trade off the variance, uh, with the um Quality of the images And finally, this is what we do So you take the unconditional logits and you just basically add up on top of that This difference times the conditional scale. So this is the same expression. We are now using in in the code Okay, let me go back to the code here So that's basically if you cannot decompose this you get that same expression um Now we do some this is the top case sampling part So we do we sort and we sort in a descending fashion such that the first Logit will be the biggest one and then they start descending Okay, and then is kept so we take the original logits and we find basically the 256th biggest logits here because here's a sorted one and because we have 256 here, so we're going to grab the 255th Biggest logit and use that as a threshold and all of those logits which are bigger than that threshold are kept That's how the top case sampling works Then we do some uh, this is just for for stabilization purposes temperature Exponential and then we just multiply with uh with this mask. So again, we are just keeping those logits that are that are Basically big enough where big enough is defined by the top k parameter. Okay, and Finally, we just sample from the distribution and we get the image token. Okay, so we now have image token here That's the that's the token we sampled and in the next iteration of this loop Basically now we're gonna feed that token as the input and then we're gonna keep on autoregressively generating the image tokens So now i'm gonna try and skip all of this So i'm gonna go and disable all breakpoints and i'm gonna enable only this one here I'm gonna hit f5 and i'm gonna let it run until we generate all of the 256 image tokens basically Okay, here we are uh now let's enter the Vq.GAN decoder As you can see here, we only pass we ignore the first token in the image tokens because that was the BOS So that was the beginning of sentence token. So we ignore it. So let's jump here. I'm gonna uh enable all of the breakpoints Let's see where I have a breakpoint here. Yes, I do So let's continue doing this Okay, here we are We delete the decoder that's gonna hopefully release some memory and empty we empty the cache So here finally we have this tip in memory This is what i'm used to and i'm not sure why that didn't happen previously when we were deleting the encoder Um, yeah, that was weird in any case now we can see it here. Okay, so we now initialize the um Detokenizer or the Vq.GAN decoder So let's see what I have. Yeah, I have a breakpoint there again similar structure as before Uh, we create the tokenizer object and then we just load the parameters. So let's see how it looks like Again, we pass it the vocab count and the embedding count So it's gonna have 256 tokens and this is the vocab count which corresponds To the this is the same Size as what the bar decoder had right? Okay, we form the embedding table Uh, just some processing layers and decoder. I'm gonna ignore how the architecture looks like It's basically a bunch of like up sampling layers and attention blocks and resonant blocks Nothing super, um I guess Informative so we just have resonant blocks some attention blocks which literally do attention uh treating image tokens as as well Treating images as as sequences and thus we can just apply attention Um, just some up sampling As I said nothing worth digging deeper into to be honest Uh, you can go and check it out at your own pace if you want. Okay, so we now load the parameters And we push the the tokenizer that we can again on the gpu And now we do the forward pass through the uh, this is the interesting part Not that complex but still interesting. So we have z that these are 256 tokens. So is that Shape is 256 26 again, i'm not sure why they're doing the clamping because we we we are certain that this will be inside of these boundaries already So there is some weird thing happening there Uh, that's a consequence of that cropping we saw in the when the with the logits so that that might explain it Okay, grid size is one because uh, as I said, we're generating just a single image and now there is a difference whether um Well, basically the the shape of this, um, how do we feed the the the image tokens into the actual convolutional decoder part? And we're going to enter this branch because the seamless is set to to false Uh, and we just embed using the uh embedding table embed our tokens and I assume this is the same Embedding table as the one that we had in the bart decoder. That's that should be shared weights and now we just Do this view thingy and we end up with 16 16 256 And that's precisely the latent space volume that we saw in the papers that we're going to now feed into dvq again uh convolutional, uh decoder part, okay Some permutation some processing with the cnn Basically, this is a like a single, uh channel cnn Sorry single, uh, the spatial extent of the cnn is one times one and then we just up sample This this image and we end up finally with uh z of shape Uh 256 256 and three channels and that's our image and now we do some clipping Uh renormalization back into the 0255 range And that's it Some again manipulations on the shape and let's see what we whoops. Let's see what we end up with here So 256 256 3 so that's our rgb image and that's it guys basically, we now uh push the image we convert it into uh Unsigned integer, uh 8-bit format. We push it onto cpu from the kuda and we Basically convert it into non-py and now we basically yield back Uh pil image pill image and that's it and then we just save the image and that's it Now there is also this print ascii from image function. That's very neat actually So again ascii from image makes sense when you're running this from a console. It's just gonna print the image Um in the ascii format, I guess. Okay, that's it Uh, let me uh step over this and we are pretty much done That's the end of the program. Let me now show you the generated image I should have it somewhere here Let me open it up And here it is. So here is the uh, what was our caption? Our caption was um, a photo of a funny dog And here is a dog. I'm not sure whether it's funny, but Uh, basically, that's it. Let me try a different caption and i'll get back to you once we once once it's done Maybe like something a bit more complex. This might break the dali mega But it's worth a shot a photo of a funny dog. Uh, maybe riding a bicycle Bicycle Let me run this one And let's see what we get. Okay guys, here is the actual image It's kind of weird You can kind of see it's trying to merge the concept of a dog with a concept of a bicycle But it's still not that expressive and big to achieve that so scale is vital for these types of models And that's it. Uh, do let me know whether you found this video useful as as usually Uh, leave down in the comments below if you have any feedback for me, uh, and until next time. Bye. Bye
[{"start": 0.0, "end": 5.62, "text": " What's up guys in this video I'm continuing on with the machine learning coding series and in this one"}, {"start": 5.62, "end": 12.26, "text": " I'll be covering the Lee mini. So that's the open source implementation of the of open AI's"}, {"start": 12.64, "end": 17.1, "text": " Dali model and when actually say I'll be covering the Lee mini what I mean"}, {"start": 17.1, "end": 22.78, "text": " I'll be cover covering this minimal port of the Lee mini that's inference only and that's"}, {"start": 22.78, "end": 29.54, "text": " Basically written in pytorch and the reason is it's much easier to go through this one and to set it up on my"}, {"start": 29.92, "end": 35.78, "text": " Windows machine, so we'll start with this one, but in the next video. I'll actually be covering the full the original"}, {"start": 36.56, "end": 40.480000000000004, "text": " Jack's code base of the Lee mini which is much more involved"}, {"start": 40.96, "end": 43.92, "text": " But this video will give you enough context to know what's going on"}, {"start": 45.2, "end": 47.0, "text": " Okay, so as"}, {"start": 47.0, "end": 53.04, "text": " Usually I'll first give you the additional context in the form of explaining the papers behind this"}, {"start": 53.64, "end": 60.6, "text": " And one thing that's very important to understand is that the Lee mini is not actually an implementation of the Lee model"}, {"start": 60.6, "end": 64.03999999999999, "text": " It's more of a mixture and let me open up my one note"}, {"start": 64.28, "end": 70.96000000000001, "text": " So it's more of a mixture of this paper called VQ GANs or taming transformers for high resolution image synthesis"}, {"start": 70.96000000000001, "end": 73.2, "text": " So that's the paper that introduced the VQ GANs"}, {"start": 73.2, "end": 77.92, "text": " And I've covered this one and all of these papers actually pretty much in in my previous videos"}, {"start": 77.92, "end": 80.08, "text": " So I'll be linking those as we go along"}, {"start": 80.44, "end": 88.44, "text": " So this is the VQ GAN paper, and they also leverage this model which is encoder decoder transformer model called BART"}, {"start": 89.24000000000001, "end": 96.4, "text": " And they use some modification of BART. They use this glue from Noam Shazir. We'll see all of those details a bit later and"}, {"start": 97.48, "end": 101.0, "text": " Here is the original the Lee paper, and then I'm gonna kind of contrast"}, {"start": 101.0, "end": 106.0, "text": " What the Lee mini is with the actual the Lee paper?"}, {"start": 107.24, "end": 114.12, "text": " They also have like bits and biases also have a report which will like literally skim through I'm gonna show you"}, {"start": 114.8, "end": 118.44, "text": " Some of the they actually have very nice training and inference"}, {"start": 119.12, "end": 122.02, "text": " Diagrams here how the Lee mini works and they also have"}, {"start": 122.84, "end": 128.96, "text": " Comparisons between the Lee and the Lee mini and you can see there is quite a lot of bullet points here"}, {"start": 128.96, "end": 134.38, "text": " So I'm gonna go through all of those a bit later. Okay, so let's first start with with actual papers"}, {"start": 135.56, "end": 140.88, "text": " I'm gonna skim. I'm gonna literally just give you the gist because otherwise there's like four papers"}, {"start": 140.88, "end": 145.08, "text": " That's that would take too much time. So let me go straight to it"}, {"start": 145.44, "end": 149.10000000000002, "text": " So this is the VQ GAN paper and as I said, I previously already covered it"}, {"start": 149.10000000000002, "end": 153.04000000000002, "text": " So if you want a deep dive go ahead and check it out. I'm gonna link it somewhere here"}, {"start": 154.0, "end": 156.0, "text": " Okay guys, so again on the high level"}, {"start": 156.0, "end": 161.12, "text": " Keeping a high level because I've covered this in great depth in a separate video"}, {"start": 161.64, "end": 164.92, "text": " We have two components of VQ GAN basically"}, {"start": 166.0, "end": 168.66, "text": " One is the learning the visual"}, {"start": 169.48, "end": 171.68, "text": " Codebook here and that's this"}, {"start": 172.64, "end": 174.16, "text": " component here and"}, {"start": 174.16, "end": 181.08, "text": " Then the second part is after you learn the visual codebook. Then you basically train a transformer model"}, {"start": 181.08, "end": 186.68, "text": " to just to autoregressively model the latent space of visual vectors here"}, {"start": 186.68, "end": 194.0, "text": " So this is the transformer part. So basically as I said, we have the first part here and we have the transformer part here"}, {"start": 195.04000000000002, "end": 199.28, "text": " So how this is trained is very similar to VQ VAE again"}, {"start": 199.28, "end": 205.92000000000002, "text": " That's a paper I've covered before and a couple of differences are that they are using this thing called reconstruction loss"}, {"start": 205.92, "end": 210.39999999999998, "text": " And the second thing is they're using discriminator here"}, {"start": 210.39999999999998, "end": 216.51999999999998, "text": " So basically adversarial approach and because of that the images from VQ GAN are much crispier"}, {"start": 217.16, "end": 222.64, "text": " Compared to what you get using VQ VAEs, which is something that actually the lead version one"}, {"start": 223.39999999999998, "end": 225.35999999999999, "text": " was using"}, {"start": 225.35999999999999, "end": 231.51999999999998, "text": " Quickly running through the pipeline here. What you do is you feed an image here you then encode it into a"}, {"start": 231.52, "end": 236.16, "text": " Lower result into a lower dimensional latent space here"}, {"start": 236.28, "end": 243.20000000000002, "text": " So you end up with some vectors as you can tell here and then what what what you do is you kind of snap them"}, {"start": 243.52, "end": 245.52, "text": " onto the closest"}, {"start": 245.68, "end": 247.52, "text": " visual"}, {"start": 247.52, "end": 253.68, "text": " Vector that you learn in this in this codebook so you can see that for example for this vector here the closest vector"}, {"start": 254.28, "end": 259.44, "text": " In as you can see here L2 by using the L2 as the definition of closeness"}, {"start": 259.44, "end": 267.0, "text": " You snap it onto a vector number one. So that means this one was closest to this one to this vector"}, {"start": 267.8, "end": 270.78, "text": " Number one from the visual codebook. Okay"}, {"start": 271.36, "end": 276.2, "text": " So once you do that for all of the vectors from your low dimensional space here"}, {"start": 276.2, "end": 278.8, "text": " You snap them onto discrete vectors here"}, {"start": 278.84, "end": 283.72, "text": " then you pass them through a convolutional decoder to get the image out and"}, {"start": 284.15999999999997, "end": 287.14, "text": " Then again, I'm going to show you the losses in a second"}, {"start": 287.14, "end": 294.0, "text": " But you train them using a couple of losses such as a reconstruction loss the perceptual loss and the and the"}, {"start": 294.47999999999996, "end": 298.52, "text": " Adversarial loss here and then the transformer part is I guess very self-explanatory"}, {"start": 299.64, "end": 305.96, "text": " Once you have this whole pathway here that allows you to map your data set of images"}, {"start": 306.47999999999996, "end": 309.36, "text": " into a data set of basically"}, {"start": 309.88, "end": 312.8, "text": " tokens because you can flip now this in our"}, {"start": 312.8, "end": 318.44, "text": " Raster order and you end up with you can directly map your image into a sequence of tokens"}, {"start": 318.56, "end": 324.08, "text": " Then you basically can just train your transformer on this new data set of"}, {"start": 324.68, "end": 331.62, "text": " Basically sequences and that's how you learn how to well the data distribution of interest"}, {"start": 332.40000000000003, "end": 336.88, "text": " Okay, so that's that's it on a high level here a couple of formulas"}, {"start": 336.88, "end": 342.0, "text": " I'm going to show you so basically as I said, what do you do is you snap your your vectors? So"}, {"start": 342.0, "end": 349.16, "text": " Basically and I have some glitch with my one node so you can see you'll you'll be seeing this thing kind of staying here"}, {"start": 349.64, "end": 352.88, "text": " Unfortunately, but yeah for whatever reason one out is glitching today"}, {"start": 353.28, "end": 359.3, "text": " So here is the vector that you estimate then you snap it on to the closest codebook vector here"}, {"start": 360.04, "end": 366.92, "text": " That's that the quantization step here. So you first encoded then you quantize it and then you have the"}, {"start": 366.92, "end": 371.8, "text": " Basically decoder part and that's how you get the reconstructed image"}, {"start": 372.2, "end": 376.76, "text": " X hat out and with that notation under our belt here is the actual loss"}, {"start": 377.32, "end": 381.0, "text": " The weak you lost is just as you can see here are some of couple of losses"}, {"start": 381.0, "end": 384.36, "text": " One of them is reconstruction loss, which is L2 in this case here"}, {"start": 384.36, "end": 388.8, "text": " And then there there are a couple of these losses which they I think they call commitment loss"}, {"start": 389.32, "end": 393.12, "text": " This one so the idea as you can see here is you put a stop gradient?"}, {"start": 393.12, "end": 400.52, "text": " in this particular case here on on top of your codebook vector and then you try and and and make sure to"}, {"start": 400.8, "end": 406.08, "text": " Make that to make your vectors closer to the codebook and here is the opposite here"}, {"start": 406.08, "end": 410.46, "text": " You you kind of put a stop gradient operator on top of your estimated"}, {"start": 411.56, "end": 417.8, "text": " Vectors and then you try and make your codebook closer to the output of your encoder as you can see here"}, {"start": 417.8, "end": 424.28000000000003, "text": " So that's the that's that and then the VQ again actually adds a couple of modifications"}, {"start": 424.6, "end": 430.96000000000004, "text": " So first one is they replaced the L2 loss as you can see here for the"}, {"start": 432.40000000000003, "end": 434.08000000000004, "text": " perceptual loss and"}, {"start": 434.08000000000004, "end": 438.48, "text": " They introduce another serial training procedure as well with the patch based discriminator"}, {"start": 438.48, "end": 444.56, "text": " So you don't have a single scalar you have as you can as you could saw see on the on the image here"}, {"start": 444.56, "end": 451.6, "text": " You actually have basically you're doing a patch based discrimination of whether that patch is real or or false"}, {"start": 452.84, "end": 458.84000000000003, "text": " Stemming from real or false image and that helps additionally that can increases the bandwidth of the learning signal"}, {"start": 459.4, "end": 461.04, "text": " compared to just using a"}, {"start": 461.04, "end": 462.96, "text": " classical scalar"}, {"start": 462.96, "end": 466.16, "text": " Okay, and that's pretty much it. That's what we can changes"}, {"start": 466.16, "end": 471.88, "text": " So we have the VQ component and then we have now the GAN component and again"}, {"start": 471.88, "end": 476.2, "text": " The difference here is that they are using the perceptual loss if you're not familiar with what that is"}, {"start": 476.52, "end": 478.8, "text": " Check out some of my neural style transfer videos"}, {"start": 478.92, "end": 486.08, "text": " But basically the idea is to take like a pre-trained convolutional neural network such as like a maybe VGG pre-trained on like image net"}, {"start": 486.08, "end": 493.4, "text": " Or something and then you instead of doing the L2 in the pixel space of the of your reconstructed image"}, {"start": 493.52, "end": 496.2, "text": " you actually feed it inside of that like a"}, {"start": 496.2, "end": 504.2, "text": " Network and then you do L2 in the latent space of that network where the features are much more abstract and you're you're basically"}, {"start": 505.2, "end": 507.2, "text": " trying to find"}, {"start": 507.32, "end": 513.0, "text": " Well, you're basically capturing the semantics much more than than than high level noise in the picture space"}, {"start": 513.8, "end": 516.8, "text": " Again refer to my video if you want to know more details"}, {"start": 516.84, "end": 521.6, "text": " But this will be enough more than enough for the for the for the code walkthrough"}, {"start": 521.6, "end": 526.76, "text": " Okay, guys, so let me show you one more thing in this paper and that's conditional"}, {"start": 527.36, "end": 528.5600000000001, "text": " synthesis"}, {"start": 528.5600000000001, "end": 537.0400000000001, "text": " So dimension here. So if the conditioning information C has spatial extent we first learn in other VQ GAN to obtain again an index-based"}, {"start": 537.96, "end": 539.96, "text": " representation as you can see here"}, {"start": 540.96, "end": 544.12, "text": " So you'll have what they say here you they'll have"}, {"start": 544.12, "end": 552.16, "text": " A height times width of these vectors where the set of vectors has is this big there is like"}, {"start": 552.8, "end": 554.8, "text": " big Z sub C"}, {"start": 555.28, "end": 558.92, "text": " Number of those vectors. So that's the size of the code book for that"}, {"start": 560.2, "end": 566.32, "text": " Condition conditional information. So they're there in concrete. Let me give you a concrete example here"}, {"start": 566.32, "end": 568.32, "text": " This could be maybe like"}, {"start": 568.6, "end": 570.6, "text": " semantic segmentation"}, {"start": 570.6, "end": 576.96, "text": " Maps that you can then basically learn a VQ GAN on top of that type of a data set. Okay"}, {"start": 578.08, "end": 583.4, "text": " Well blind then you have this newly obtained code book and then due to autoregressive structure of the transformer"}, {"start": 583.4, "end": 587.62, "text": " we can then simply prepend R to S which is the"}, {"start": 588.36, "end": 592.8000000000001, "text": " Well, that's your your your your image from your target data set and R is"}, {"start": 593.28, "end": 597.22, "text": " stems from the conditioning information and then basically you do the"}, {"start": 597.22, "end": 602.26, "text": " Negative log likelihood to these entries. So in case that was not that clear"}, {"start": 602.26, "end": 605.46, "text": " Let me quickly draw it here. So again, imagine you have"}, {"start": 606.1, "end": 608.4200000000001, "text": " Basically some semantic segmentation"}, {"start": 609.22, "end": 614.0600000000001, "text": " Masks here. So maybe something like this. I'm gonna kind of try and"}, {"start": 614.82, "end": 616.82, "text": " draw it somehow so"}, {"start": 618.82, "end": 624.4200000000001, "text": " This is some type of a semantic segmentation image and you have a big data set of such images"}, {"start": 624.42, "end": 629.9, "text": " Big data set of such images and now what you do is basically train a VQ GAN"}, {"start": 631.5, "end": 635.6999999999999, "text": " Such that you'll end up with a particular code book"}, {"start": 637.4599999999999, "end": 641.78, "text": " For that vector for for that data set so this is going to be your your code book"}, {"start": 643.2199999999999, "end": 645.2199999999999, "text": " For this particular"}, {"start": 645.86, "end": 650.62, "text": " Image here, so you then encode it you end up with basically"}, {"start": 650.62, "end": 652.62, "text": " Well, you end up with"}, {"start": 653.02, "end": 659.34, "text": " HC times WC vectors you then flatten them out and"}, {"start": 659.82, "end": 661.82, "text": " You end up with a sequence and"}, {"start": 662.3, "end": 667.82, "text": " This sequence that which they denote as R is something you can prepend while training"}, {"start": 668.38, "end": 670.38, "text": " on your on your data set"}, {"start": 670.62, "end": 676.94, "text": " Okay. So now obviously for each of these images you have an actual image you care about which is the"}, {"start": 676.94, "end": 682.7, "text": " Image from your from the data set you're trying to for which you're trying to learn the data distribution off"}, {"start": 683.1, "end": 687.1800000000001, "text": " So that there will be some image here, which is actually original image"}, {"start": 687.1800000000001, "end": 694.0600000000001, "text": " And this is its semantic segmentation and then you do the same thing here. So you'll you first learn the VQ GAN component"}, {"start": 695.2600000000001, "end": 698.3000000000001, "text": " So something like this and I'm really terrible at drawing today"}, {"start": 699.0200000000001, "end": 701.9000000000001, "text": " And and then you encode it here"}, {"start": 701.9, "end": 707.66, "text": " You flatten out this sequence. I'm going to denote it with a different color. So we now have a sequence here"}, {"start": 709.26, "end": 712.62, "text": " And now how you're going to train is you're going to prepend this vector R"}, {"start": 713.26, "end": 717.9, "text": " so we'll you'll grab the vector R from the semantic segmentation map and"}, {"start": 718.38, "end": 721.98, "text": " Here you have the target image which they denote as S"}, {"start": 722.6999999999999, "end": 725.98, "text": " And now you just collect these tuples for your whole data set"}, {"start": 725.98, "end": 732.54, "text": " So you basically collect this and then you learn how to to model that type of uh of of of sequence"}, {"start": 732.86, "end": 735.82, "text": " And so how do you later when you want to generate?"}, {"start": 736.46, "end": 742.38, "text": " A new image you literally you literally grab an arbitrary semantic segmentation"}, {"start": 743.26, "end": 744.46, "text": " map"}, {"start": 744.46, "end": 748.14, "text": " And you you pass it through the encoder here"}, {"start": 748.62, "end": 750.3000000000001, "text": " you fetch the"}, {"start": 750.3000000000001, "end": 753.26, "text": " This vector R and then you pass it through the"}, {"start": 753.26, "end": 759.5, "text": " Decoder you fetch the uh this vector R and then you put it inside of your transformer"}, {"start": 760.62, "end": 762.62, "text": " So here you'll have a transformer"}, {"start": 763.5, "end": 765.5, "text": " You put the R here"}, {"start": 765.9, "end": 767.9, "text": " And then you ought to aggressively"}, {"start": 768.62, "end": 770.62, "text": " keep on"}, {"start": 771.34, "end": 774.9399999999999, "text": " Generating the sequence and then you pass that through the decoder here"}, {"start": 775.26, "end": 780.46, "text": " So once you generate that sequence here you pass it through the decoder and out comes the image"}, {"start": 780.46, "end": 783.5, "text": " So this will maybe be in practice like 256 tokens"}, {"start": 784.14, "end": 787.26, "text": " Which you can then rearrange into 16 by 16"}, {"start": 788.38, "end": 790.38, "text": " Tokens something like this"}, {"start": 792.3000000000001, "end": 797.58, "text": " And then that's that's like that's the that's the low uh dimensional representation"}, {"start": 797.58, "end": 801.5, "text": " And then you can as I said you pass it through the decoder and you get your image out"}, {"start": 801.5, "end": 805.4200000000001, "text": " So why I took some time to explain this is because um"}, {"start": 805.42, "end": 811.8199999999999, "text": " Dali and and uh dali mini, uh use a very similar approach instead of uh, and instead of using the uh"}, {"start": 812.38, "end": 818.9399999999999, "text": " Semantic segmentation masks they're actually using text but the logic is super similar. So just instead of um"}, {"start": 819.74, "end": 822.54, "text": " uh doing this whole thing you um"}, {"start": 823.3399999999999, "end": 826.38, "text": " Pass a piece of text that's associated with your image"}, {"start": 826.86, "end": 833.0999999999999, "text": " Here and then you learn how to model them together and that's it. So literally by doing that you learn the um"}, {"start": 833.1, "end": 835.1, "text": " You learn the data distribution"}, {"start": 835.82, "end": 837.82, "text": " uh p"}, {"start": 837.82, "end": 844.0600000000001, "text": " x y and that's where the bart paper comes into picture. So that's that's where this paper comes into play"}, {"start": 844.5400000000001, "end": 847.1, "text": " I'm gonna explain to you. Whoops"}, {"start": 847.66, "end": 851.34, "text": " Um, i'm gonna explain to you in a second like it's a it's a very simple"}, {"start": 851.34, "end": 855.1, "text": " It's basically like an encoder decoder transformer. There is not too much to say there"}, {"start": 855.34, "end": 855.98, "text": " Okay"}, {"start": 855.98, "end": 861.74, "text": " So again how bart will come into play here is instead of doing this pipeline with semantic segmentation masks"}, {"start": 861.74, "end": 863.74, "text": " You'll instead have uh an image"}, {"start": 864.78, "end": 870.3, "text": " And you'll have associated caption here. So some piece of text here. Okay, so have some piece of text here"}, {"start": 871.98, "end": 878.54, "text": " And you'll basically feed that text through uh encoder part of of uh of bart"}, {"start": 879.02, "end": 881.58, "text": " So let me kind of denote the uh bart"}, {"start": 882.22, "end": 887.82, "text": " Such as this like we have encoder part of bart and we hope we have the decoder part"}, {"start": 887.82, "end": 891.34, "text": " Of bart decoder is obviously attending"}, {"start": 891.98, "end": 896.94, "text": " Uh the encoder part that's how the transformers work. And so you you grab this text here"}, {"start": 897.9000000000001, "end": 902.86, "text": " You basically feed it directly into the encoder part and out comes something from the decoder"}, {"start": 903.58, "end": 909.1800000000001, "text": " Out comes this thing here and now you use this output of the encoder as r and that's it"}, {"start": 909.1800000000001, "end": 915.1, "text": " And now you just again you you'll now have these tuples of uh, this conditioning vectors"}, {"start": 915.1, "end": 920.94, "text": " And your uh images from your data set and you just learn how to model the p x y"}, {"start": 920.94, "end": 928.38, "text": " So the the distribution of your uh, like images and uh and captions and by the way, this is literally this this"}, {"start": 928.62, "end": 932.38, "text": " These two boxes is everything you need to have like that's a mental model of bart"}, {"start": 932.38, "end": 936.0600000000001, "text": " I have when i'm when i'm discussing it. So the only thing we need to"}, {"start": 937.1, "end": 940.94, "text": " See to understand how bart works compared to all of the other transformers is"}, {"start": 940.94, "end": 949.74, "text": " I guess this thing here so basically in your regular, uh, for example in inbert model so inbert"}, {"start": 951.0200000000001, "end": 956.46, "text": " What the pre-training objective looked like is you masked out so it's first is bidirectional"}, {"start": 956.46, "end": 961.9000000000001, "text": " So it's only encoder part of the of the uh encoder decoder original transformer structure"}, {"start": 961.9000000000001, "end": 964.62, "text": " Whereas bart is both encoder as well as the decoder"}, {"start": 965.1, "end": 968.1400000000001, "text": " So you're trying to just you just kind of mask out some of the code"}, {"start": 968.14, "end": 973.26, "text": " So you're trying to just you just kind of mask out certain tokens and I think like you do that with like"}, {"start": 973.9, "end": 975.9, "text": " 10 20 percent"}, {"start": 976.92, "end": 978.92, "text": " Probability for each of the tokens"}, {"start": 979.42, "end": 986.06, "text": " And then you try to predict what the token at that particular position was so it's a simple, uh, like a auto encoding"}, {"start": 987.5, "end": 992.14, "text": " Objective, uh, and what's different with bart is you can kind of model"}, {"start": 992.14, "end": 998.3, "text": " You can kind of take a span of of tokens and then mask them out with a single"}, {"start": 998.6999999999999, "end": 1002.14, "text": " Mask token and then make sure to predict the original sequence here"}, {"start": 1002.46, "end": 1005.74, "text": " So because of this thing because of you can mask it out here"}, {"start": 1005.8199999999999, "end": 1011.1, "text": " However, um many tokens you want and then predict them here you have additional flexibility"}, {"start": 1011.1, "end": 1015.98, "text": " Which you don't have with a pure encoder, um, like transformers such as such as bert"}, {"start": 1016.22, "end": 1019.42, "text": " And I think they mentioned it somewhere here. Let me find that piece of text"}, {"start": 1019.42, "end": 1027.18, "text": " So the objective is called text in filling and they mentioned here a number of text spans are sampled with span lengths drawn from a post"}, {"start": 1027.34, "end": 1028.7, "text": " Poisson"}, {"start": 1028.7, "end": 1030.46, "text": " distribution with lambda"}, {"start": 1030.46, "end": 1036.54, "text": " Parameter equals three and each span is replaced with a single mask token zero length spans"}, {"start": 1037.08, "end": 1039.6599999999999, "text": " correspond to the insertion of mask tokens and"}, {"start": 1040.7, "end": 1042.3799999999999, "text": " Okay, it's inspired blah blah blah"}, {"start": 1042.3799999999999, "end": 1047.34, "text": " So text in filling teaches the model to predict how many tokens are missing from a span"}, {"start": 1047.34, "end": 1052.4599999999998, "text": " And that's all like it's literally encoder decoder transformer with this particular"}, {"start": 1053.34, "end": 1055.34, "text": " denoising objective"}, {"start": 1055.4199999999998, "end": 1059.98, "text": " And they show a bit later here how it compares against the baselines"}, {"start": 1061.34, "end": 1063.34, "text": " using different types of"}, {"start": 1063.34, "end": 1069.6599999999999, "text": " Like retraining objectives such as just token masking a la bert or token deletion or text in filling"}, {"start": 1069.6599999999999, "end": 1074.3799999999999, "text": " So a bunch of different sentence shuffling document rotation, etc, etc, and they compare"}, {"start": 1075.1799999999998, "end": 1076.9399999999998, "text": " that against the"}, {"start": 1076.94, "end": 1083.18, "text": " Language model so that's just predicting the next token or mask language models such as birth, etc, etc, and they show"}, {"start": 1084.38, "end": 1086.3, "text": " Well competitive results"}, {"start": 1086.3, "end": 1092.06, "text": " Okay, that's the bird part the bard part and then there's a small modification called glue"}, {"start": 1092.06, "end": 1097.18, "text": " Which i'll show you a bit later. So glue variance improved transformer from this author noam schazer"}, {"start": 1097.98, "end": 1102.78, "text": " Who is actually the co-author of the original transformer paper. So it's a very prolific author"}, {"start": 1102.78, "end": 1107.58, "text": " But before that, let me just quickly skim over over dali version one"}, {"start": 1108.06, "end": 1111.34, "text": " Which is which came from the paper zero shot text image generation"}, {"start": 1112.1399999999999, "end": 1113.74, "text": " again"}, {"start": 1113.74, "end": 1116.3, "text": " Super similar pipeline to vq again"}, {"start": 1116.86, "end": 1123.82, "text": " Again, they mentioned here they have two stages. So in the first stage they say here we train a discrete variational autoencoder"}, {"start": 1124.54, "end": 1126.54, "text": " to compress each of the"}, {"start": 1126.62, "end": 1131.42, "text": " 256 256 rgb images into 32 by 32 grid of image tokens"}, {"start": 1131.42, "end": 1134.94, "text": " each elements of which can assume"}, {"start": 1136.92, "end": 1144.94, "text": " 8192 possible values so that that means that the codebook size is this big and this reduces the context size of the transformer"}, {"start": 1145.26, "end": 1150.6200000000001, "text": " by a factor of 192 without a large degradation in visual quality because if you divide"}, {"start": 1151.18, "end": 1152.8400000000001, "text": " basically"}, {"start": 1152.8400000000001, "end": 1158.6200000000001, "text": " 256 times 256 times 3 because you have rgb image into"}, {"start": 1158.62, "end": 1163.34, "text": " 232 times 32 you end up with 192. Okay"}, {"start": 1164.2199999999998, "end": 1165.26, "text": " and"}, {"start": 1165.26, "end": 1170.6999999999998, "text": " The same authors of this paper. Well, I think this is from yeah, this is from open.ai"}, {"start": 1170.86, "end": 1176.6999999999998, "text": " So basically they they they previously trained this model called image gpt and in that model"}, {"start": 1176.6999999999998, "end": 1179.9799999999998, "text": " They were training they were autoregressively learning how to model"}, {"start": 1180.6999999999998, "end": 1184.9399999999998, "text": " the the images in the pixel space, which is much more complex because"}, {"start": 1184.94, "end": 1191.5800000000002, "text": " Transformers have quadratic complexity and obviously the image space can can grow like up to like"}, {"start": 1192.14, "end": 1197.98, "text": " Well thousand by thousand images and so it becomes very quickly impossible to model in the pixel space"}, {"start": 1197.98, "end": 1203.26, "text": " So that's why this reduction this mapping into a lower dimensional space here. Okay"}, {"start": 1203.98, "end": 1205.98, "text": " So again, that's stage one"}, {"start": 1205.98, "end": 1209.18, "text": " and it's literally the same as vq again without the"}, {"start": 1209.74, "end": 1213.74, "text": " Perceptual loss and without the adversarial component and then they say here"}, {"start": 1213.74, "end": 1218.54, "text": " In stage two we concatenate up to 256 vpe"}, {"start": 1218.6200000000001, "end": 1222.54, "text": " By pair encoding text tokens with the 32 times 32"}, {"start": 1223.1, "end": 1225.98, "text": " image tokens that come from the stage one and"}, {"start": 1226.46, "end": 1232.7, "text": " Train an autoregressive transformer to model the joint distribution over text and image tokens. So basically what I said here"}, {"start": 1233.42, "end": 1239.5, "text": " Is that so again they they train the vqvae which you can think as as same as vqgan"}, {"start": 1239.5, "end": 1244.22, "text": " So yeah, they have the encoder part. They have the decoder part"}, {"start": 1245.18, "end": 1248.3, "text": " They kind of have a low dimensional representation here"}, {"start": 1249.58, "end": 1253.5, "text": " and so what they did what they do instead so so they can kind of"}, {"start": 1254.3, "end": 1257.1, "text": " unroll this in a raster order to end up with a"}, {"start": 1259.18, "end": 1263.82, "text": " Sequence of tokens s so to get r so to get the conditioning vector"}, {"start": 1264.06, "end": 1267.82, "text": " And I don't know what why they're using r like I guess c c makes more sense"}, {"start": 1267.82, "end": 1270.9399999999998, "text": " But i'll just keep on using the same uh terminology"}, {"start": 1271.8999999999999, "end": 1273.8999999999999, "text": " So to get the r component"}, {"start": 1274.3799999999999, "end": 1276.62, "text": " That you'll use to to prepend here"}, {"start": 1277.34, "end": 1280.3, "text": " And then train or progressively train a transformer on top of it"}, {"start": 1280.78, "end": 1283.6599999999999, "text": " Uh, they just use as as you saw here"}, {"start": 1283.8999999999999, "end": 1290.22, "text": " They just have uh, basically they take a piece of text and they bpe encoded so you start with the text"}, {"start": 1290.86, "end": 1294.86, "text": " And then you apply the tokenizer which gives you a sequence of tokens"}, {"start": 1294.86, "end": 1301.82, "text": " It just gives you a sequence of tokens and then I assume they just embed those using like a shallow like a simple"}, {"start": 1302.1399999999999, "end": 1308.78, "text": " Well, you have an embedding table you map that into tokens and then you you place that directly here to start"}, {"start": 1309.4199999999998, "end": 1311.4199999999998, "text": " um modeling whereas"}, {"start": 1311.4199999999998, "end": 1316.3, "text": " Um, as you saw the lini is using bart encoder. So that means they'll have additional"}, {"start": 1316.9399999999998, "end": 1322.54, "text": " Layers of processing before they they output this r vector. So the the conditioning vector"}, {"start": 1322.54, "end": 1327.8999999999999, "text": " Okay, guys, that's that's pretty much it. Um, let me quickly show you what they say here"}, {"start": 1327.98, "end": 1330.94, "text": " There is a lot of complexity hidden in here"}, {"start": 1331.02, "end": 1335.8999999999999, "text": " As I said, I covered this one the lee one, uh, you know in a separate video so you can check it out"}, {"start": 1335.98, "end": 1338.78, "text": " They use stuff such as gumbell softmax relaxation"}, {"start": 1339.5, "end": 1343.18, "text": " To train this model, etc, etc. And also it's worth"}, {"start": 1343.82, "end": 1346.3799999999999, "text": " Keeping in mind. I think they mentioned it somewhere here"}, {"start": 1347.5, "end": 1350.1399999999999, "text": " Let me just show you this so getting the model"}, {"start": 1350.14, "end": 1357.18, "text": " To train in 16-bit precision past 1 billion parameters without diverging was the most challenging part of this project"}, {"start": 1357.5, "end": 1359.5, "text": " so training this um"}, {"start": 1359.74, "end": 1363.66, "text": " Big model well back in back a couple of years ago. This was considered a big model"}, {"start": 1364.22, "end": 1371.42, "text": " Was was extremely hard and um, that's something I think boris the the main developer of the um, the lee mini"}, {"start": 1371.9, "end": 1377.66, "text": " Project is also struggling with but he's using jacks and he he has like free tpus"}, {"start": 1377.66, "end": 1379.66, "text": " From google so that kind of helps"}, {"start": 1379.98, "end": 1387.3400000000001, "text": " Because tpus have they did mention somewhere here that tpus have some less problems than than than gpus in certain aspects"}, {"start": 1388.22, "end": 1390.78, "text": " Let me quickly show you one more thing again"}, {"start": 1391.26, "end": 1393.5, "text": " Uh, let me just explain this conditioning part"}, {"start": 1393.5, "end": 1401.42, "text": " So given a text image pair we bpe encode with lowercase captions using at most 256 tokens with woke up size"}, {"start": 1401.98, "end": 1404.8600000000001, "text": " 16 384 and encode the image using"}, {"start": 1404.86, "end": 1407.9199999999998, "text": " 32 times 32 tokens with woke up size 8192"}, {"start": 1412.78, "end": 1416.1599999999999, "text": " The image tokens are obtained using argmax sampling from the dvee"}, {"start": 1417.58, "end": 1424.78, "text": " Via encoded logits without adding any gumble noise. Uh, finally the text and image tokens are concatenated and model"}, {"start": 1425.34, "end": 1429.34, "text": " Regressively as a single stream of data. So that's something we saw multiple times already"}, {"start": 1429.34, "end": 1434.54, "text": " Okay, guys, so that's it. It's a very simple model as you saw here"}, {"start": 1435.5, "end": 1439.58, "text": " Uh, there is a lot of overlap between the ideas between vq gam paper and the lee"}, {"start": 1440.1399999999999, "end": 1441.6599999999999, "text": " Uh version one"}, {"start": 1441.6599999999999, "end": 1445.02, "text": " Um, they took the ideas from bar. They took some other ideas"}, {"start": 1445.1799999999998, "end": 1450.1399999999999, "text": " Uh boris is experimenting a lot as soon as a new paper comes out such as the deep net paper"}, {"start": 1450.54, "end": 1456.6999999999998, "text": " With novel initializations. He tries an experiment with it. Okay, so let me now show you the uh report that"}, {"start": 1456.7, "end": 1460.48, "text": " Uh boris wrote with the other other co-authors"}, {"start": 1462.06, "end": 1463.3400000000001, "text": " Again"}, {"start": 1463.3400000000001, "end": 1467.5800000000002, "text": " They have a very nice drawing of the training pipeline of the lee"}, {"start": 1468.38, "end": 1469.3400000000001, "text": " mini"}, {"start": 1469.3400000000001, "end": 1471.3400000000001, "text": " Model as you can see here"}, {"start": 1471.9, "end": 1473.9, "text": " again, they have the"}, {"start": 1474.54, "end": 1478.46, "text": " Bart encoder so you as you can see you you first have these"}, {"start": 1479.1000000000001, "end": 1485.26, "text": " Puppets of images and associated captions then you pass the caption through the bart encoder you get the"}, {"start": 1485.26, "end": 1487.26, "text": " the conditioning vector here"}, {"start": 1487.9, "end": 1493.02, "text": " You take the image you pass it through the vq again encoder you get the image tokens you feed them here"}, {"start": 1493.1, "end": 1495.5, "text": " So you can see here vq again, uh tokens"}, {"start": 1496.06, "end": 1502.78, "text": " And then you pass all of that through a bar decoder and you learn how to autoregressively model these sequences"}, {"start": 1503.34, "end": 1507.18, "text": " Uh simple stuff as for the inference we can see it here"}, {"start": 1507.9, "end": 1512.06, "text": " Uh, so you grab a piece of text you pass it through the bart encoder"}, {"start": 1512.06, "end": 1518.3799999999999, "text": " You feed it into the decoder and then you autoregressively generate the image tokens"}, {"start": 1519.1, "end": 1521.1, "text": " Uh, you can generate multiple"}, {"start": 1521.34, "end": 1527.98, "text": " Uh series of these image tokens because uh, the process of generation is inherently stochastic"}, {"start": 1528.62, "end": 1530.78, "text": " Then you pass all of those through the vq again"}, {"start": 1531.1799999999998, "end": 1534.3799999999999, "text": " Uh decoder you get some associated images"}, {"start": 1534.78, "end": 1539.4199999999998, "text": " So the token sequence from the low dimensional embedding space is projected into"}, {"start": 1539.42, "end": 1542.38, "text": " Uh the image in the pixel space"}, {"start": 1543.74, "end": 1545.74, "text": " And then you can use something like clip"}, {"start": 1546.22, "end": 1548.54, "text": " to filter out the best images"}, {"start": 1548.78, "end": 1554.22, "text": " So you again feed the same caption to clip and finds the image that"}, {"start": 1554.7, "end": 1559.3400000000001, "text": " That is best described by this particular caption you can see in this particular example"}, {"start": 1559.5800000000002, "end": 1565.42, "text": " White snow covered mountain under blue sky during daytime clip found that this image here is the best"}, {"start": 1565.9, "end": 1567.98, "text": " this is is uh"}, {"start": 1567.98, "end": 1570.94, "text": " Well best described by this by this particular caption. Okay"}, {"start": 1571.9, "end": 1573.58, "text": " That's it. Let me quickly"}, {"start": 1573.58, "end": 1576.22, "text": " Walk you through and I encourage you to check out the whole report"}, {"start": 1577.1, "end": 1584.54, "text": " In this in this awesome, uh weight and biases report, but let me show you some some differences that the uh authors"}, {"start": 1585.18, "end": 1586.38, "text": " Of the lee"}, {"start": 1586.38, "end": 1593.5, "text": " Mini model themselves mentioned so we are grateful for the research and pre-trained models published by open.ai which we're essentially building our model"}, {"start": 1593.5, "end": 1598.22, "text": " Uh, not all the uh, the details on the leak our public knowledge"}, {"start": 1598.22, "end": 1601.1, "text": " But here what we consider to be the main differences"}, {"start": 1601.18, "end": 1606.16, "text": " So the lee uses a 12 billion parameter version of gpt3"}, {"start": 1606.86, "end": 1612.38, "text": " In comparison our model is 27 times smaller with about 0.4 billion parameters"}, {"start": 1612.54, "end": 1617.42, "text": " Now this report was written a while ago. And so this was the the lee mini"}, {"start": 1618.14, "end": 1620.14, "text": " Like they now have the lee mega as well"}, {"start": 1620.14, "end": 1624.94, "text": " And also I think there is a mistake here because i'm fairly sure the original dali"}, {"start": 1625.5800000000002, "end": 1627.5800000000002, "text": " Paper used gpt2 not gpt3"}, {"start": 1628.7800000000002, "end": 1630.7800000000002, "text": " Which are fairly similar but still"}, {"start": 1630.7800000000002, "end": 1636.14, "text": " Um, then they say we heavily leverage pre-trained models such as vq-gan barding coder and clip"}, {"start": 1636.46, "end": 1638.46, "text": " We mentioned all of those so far"}, {"start": 1638.46, "end": 1645.5800000000002, "text": " Uh, and while open.ai had to train all their models from scratch our model architecture takes into account pre-trained models available"}, {"start": 1645.98, "end": 1647.8200000000002, "text": " And their efficiency"}, {"start": 1647.82, "end": 1650.86, "text": " The lee encodes images using a larger number of tokens"}, {"start": 1651.26, "end": 1656.46, "text": " So they use 32 times 32 whereas the uh, dali mini use uh uses"}, {"start": 1657.1599999999999, "end": 1660.54, "text": " 16 uh by 16 from a smaller vocab"}, {"start": 1661.26, "end": 1664.46, "text": " And the lee uses a vqvae while we use a vq-gan"}, {"start": 1664.9399999999998, "end": 1670.3999999999999, "text": " Okay, then they say the lee encodes text using fewer tokens at most 256"}, {"start": 1671.1, "end": 1674.08, "text": " Uh, whereas they where the lee mini uses 1024"}, {"start": 1674.08, "end": 1681.84, "text": " And the lee reads text and images as a single stream of data while we split them between these uh, well bard encoder and decoder"}, {"start": 1682.56, "end": 1686.24, "text": " Uh, this also lets us use independent vocab for text and images"}, {"start": 1686.8799999999999, "end": 1691.4399999999998, "text": " The lee reads the text through an autoregressive model while we use a bi-directional encoder"}, {"start": 1691.52, "end": 1696.08, "text": " Okay, so I made a small mistake when I was explaining how the lee wevon works. I'm gonna"}, {"start": 1696.72, "end": 1700.3999999999999, "text": " Show you what I mean by that in a second, but let me finish here"}, {"start": 1700.4, "end": 1707.2, "text": " So the lee was trained on 250 million pairs of image and text while we used only 15 million pairs"}, {"start": 1707.44, "end": 1707.68, "text": " Okay"}, {"start": 1707.68, "end": 1712.48, "text": " So the difference is scale obviously the model scale the dataset scale etc, etc"}, {"start": 1712.64, "end": 1717.52, "text": " But also a lot of difference in the architecture and the training method like they don't have the gumbo softmax"}, {"start": 1717.6000000000001, "end": 1722.96, "text": " They don't have the vqvae. They're using vq-gan. They're using bard instead of using gpt, etc, etc"}, {"start": 1722.96, "end": 1727.3600000000001, "text": " So there is so the lee mini is um, well not really the lee model"}, {"start": 1727.36, "end": 1731.28, "text": " It's a it's a mixture of of vq-gans and bard and everything else"}, {"start": 1731.6, "end": 1736.4799999999998, "text": " So so yeah, but still and that's why they are now called crayon not the lee mini"}, {"start": 1737.04, "end": 1740.3999999999999, "text": " But in any case, let me quickly go back to notebook and then we're jumping to code"}, {"start": 1741.1999999999998, "end": 1744.8799999999999, "text": " So I think I mentioned somewhere here that they bpe encode"}, {"start": 1745.6799999999998, "end": 1749.2199999999998, "text": " Text and then they just learn how to autoregressively"}, {"start": 1750.56, "end": 1755.1999999999998, "text": " Model that and well, I kind of did correctly explain it but not quite"}, {"start": 1755.2, "end": 1757.68, "text": " so what they do is so let me just"}, {"start": 1758.48, "end": 1760.8, "text": " draw this again here so we have the"}, {"start": 1762.16, "end": 1764.56, "text": " Conditioning part and we have the target"}, {"start": 1765.42, "end": 1767.42, "text": " sequence"}, {"start": 1768.0800000000002, "end": 1773.8600000000001, "text": " So what they do how they learn this is they feed all of these into the transformer"}, {"start": 1775.04, "end": 1778.64, "text": " And then you learn how to predict all of the tokens. So basically here"}, {"start": 1778.96, "end": 1784.0, "text": " So yeah again as a target you just take this same sequence and you shift it leftwards by by one"}, {"start": 1784.0, "end": 1787.44, "text": " uh by one step and that's what you're trying to"}, {"start": 1788.32, "end": 1790.0, "text": " to predict"}, {"start": 1790.0, "end": 1794.16, "text": " so this is what you're trying to predict and then here ultimately you have the"}, {"start": 1794.88, "end": 1796.88, "text": " uh end of"}, {"start": 1797.84, "end": 1801.76, "text": " Image token in this particular case because in this particular example the green part"}, {"start": 1802.4, "end": 1807.2, "text": " Are the image tokens from the vq vae portion? Okay, so that's it"}, {"start": 1807.44, "end": 1813.2, "text": " The main thing I want to stress here is that the the embeddings for the conditioning part are not shallow they're using"}, {"start": 1813.2, "end": 1815.3600000000001, "text": " uh deep transformer layers"}, {"start": 1816.0, "end": 1820.16, "text": " because as you as you can as you know, like the uh transformer works by"}, {"start": 1821.6000000000001, "end": 1829.04, "text": " Cross attending um previous tokens here. So it will be attending to deeper representations of this particular"}, {"start": 1829.6000000000001, "end": 1833.04, "text": " Text here. Okay guys this there was a bit more than I wanted to"}, {"start": 1833.52, "end": 1838.32, "text": " To give you when it comes to context and and paper overviews, but it is what it is"}, {"start": 1838.32, "end": 1842.96, "text": " Is let's now dig into the actual code. Okay, so here we are in my vs code"}, {"start": 1844.0, "end": 1850.3999999999999, "text": " I'm gonna open up the i'm gonna hit debug here and we're gonna slowly start stepping over the code again"}, {"start": 1850.48, "end": 1853.9199999999998, "text": " I'm gonna skip all of the unnecessary details. I just want to show you"}, {"start": 1854.56, "end": 1859.28, "text": " The similarity between what we just saw theoretically and the actual code"}, {"start": 1859.8999999999999, "end": 1861.8999999999999, "text": " implementation again this uh"}, {"start": 1862.1599999999999, "end": 1864.56, "text": " I'm showing you the code from a mindaly"}, {"start": 1864.56, "end": 1870.48, "text": " uh repo and uh, they only have the inference portion of the pipeline and they're um,"}, {"start": 1870.72, "end": 1876.8799999999999, "text": " It's it's a port in into pytorch and it's whereas the original repo was written in jax"}, {"start": 1877.52, "end": 1883.44, "text": " We have some arguments here as you can see, uh, so i'm using the mega version and not the mini version"}, {"start": 1884.0, "end": 1889.52, "text": " Uh fp16 set to false so that we have higher quality images, but the footprint will be bigger"}, {"start": 1889.76, "end": 1892.32, "text": " So I have a 8 gigabyte tpu"}, {"start": 1892.96, "end": 1894.1599999999999, "text": " and uh"}, {"start": 1894.16, "end": 1897.52, "text": " Otherwise, you'll have problems executing this thing. So let me show you"}, {"start": 1898.4, "end": 1900.4, "text": " uh, we'll see how the um"}, {"start": 1901.28, "end": 1904.4, "text": " GPU memory is slowly increasing as we're going through the code"}, {"start": 1904.5600000000002, "end": 1909.0400000000002, "text": " There is a couple of optimization in the code as well such as as soon as the encoder"}, {"start": 1909.44, "end": 1914.88, "text": " Is created and and the image is encoded they uh dispose of the encoder"}, {"start": 1915.44, "end": 1919.3600000000001, "text": " So that's controlled by this parameter. Let me see whether"}, {"start": 1919.36, "end": 1927.04, "text": " They have it here. I don't think it's in here. They have a parameter that literally controls that that that behavior"}, {"start": 1927.4399999999998, "end": 1931.52, "text": " Okay. So again, we have uh, we're using mega we have a piece of text"}, {"start": 1931.52, "end": 1935.6, "text": " I just inputted a random caption here a photo of a funny dog. Let's see what we get"}, {"start": 1936.1599999999999, "end": 1942.56, "text": " There is some seed there is a grid size. We're just using we're just generating a single image top cape for top case sampling"}, {"start": 1942.9599999999998, "end": 1944.9599999999998, "text": " Is set to 256?"}, {"start": 1944.96, "end": 1949.44, "text": " Uh image path, uh, so this is going to be directly dumped into the root"}, {"start": 1950.24, "end": 1953.52, "text": " root of of this repo so the image will be"}, {"start": 1955.04, "end": 1956.56, "text": " saved there"}, {"start": 1956.56, "end": 1960.56, "text": " Uh, the models are uh, well already downloaded into this pre-trained"}, {"start": 1960.8, "end": 1966.8, "text": " I went ahead and downloaded them beforehand so that we don't have to wait and again as I said, we're using fp16"}, {"start": 1966.8, "end": 1971.68, "text": " So let's continue here. So first off we're gonna generate the we're gonna create this"}, {"start": 1971.68, "end": 1978.16, "text": " Uh mindaly model and then we are gonna do this generation process. So let's see what happens here"}, {"start": 1978.3200000000002, "end": 1980.3200000000002, "text": " So again is reusable is set to false"}, {"start": 1981.04, "end": 1986.8, "text": " That's the parameter that's going to control that optimization behavior. We'll see that in a second. So let me continue here"}, {"start": 1987.76, "end": 1989.76, "text": " So here it is"}, {"start": 1989.8400000000001, "end": 1992.0800000000002, "text": " Nothing fancy there just storing these"}, {"start": 1992.94, "end": 1994.4, "text": " variables"}, {"start": 1994.4, "end": 1996.4, "text": " we have"}, {"start": 1996.4, "end": 1999.68, "text": " number of text tokens that's gonna specify the"}, {"start": 1999.68, "end": 2000.88, "text": " the"}, {"start": 2000.88, "end": 2002.88, "text": " maximal size of the caption"}, {"start": 2003.52, "end": 2006.48, "text": " Number of layers is going to be 24 for mega"}, {"start": 2007.04, "end": 2012.72, "text": " Uh, as you can see the difference between mega and mini is just in the number of layers number of attention heads"}, {"start": 2013.1200000000001, "end": 2015.52, "text": " Also the size of the embedding space"}, {"start": 2016.4, "end": 2019.44, "text": " The size of the glue embedding space we'll get once we get there"}, {"start": 2019.6000000000001, "end": 2024.48, "text": " I'll briefly show you the paper as well for glue and then the text vocab count"}, {"start": 2024.64, "end": 2028.4, "text": " Uh, and the image vocab count kind of differ between these two models"}, {"start": 2028.4, "end": 2032.88, "text": " in any case, we just uh form the paths here for the uh"}, {"start": 2033.68, "end": 2040.72, "text": " Well by dali here, they are literally referring to the bart part of the dali model. So that's kind of um"}, {"start": 2041.9, "end": 2044.5600000000002, "text": " Potentially confusing. Um, okay, so"}, {"start": 2045.74, "end": 2051.44, "text": " Basically, i'm gonna skip over all of this. We have a vocab json and merges text. So that's gonna"}, {"start": 2052.08, "end": 2057.12, "text": " Basically, uh be used to form the tokenizer the text tokenizer. Okay"}, {"start": 2057.12, "end": 2063.44, "text": " Okay, and now we just form some some um paths which contain the the pre-trained models weights"}, {"start": 2064.88, "end": 2071.04, "text": " And now we hit in initialize tokenizer. Let me just see what I have. Yep. I have a breakpoint here. So"}, {"start": 2071.7599999999998, "end": 2074.7999999999997, "text": " Basically, it's already downloaded so we can ignore all of this"}, {"start": 2075.3599999999997, "end": 2079.3599999999997, "text": " Uh, and now we just open the vocab and here here it is"}, {"start": 2079.44, "end": 2084.74, "text": " So let's see what's the size of the vocab. So length of the vocab is 50,265"}, {"start": 2084.74, "end": 2086.9799999999996, "text": " 50,265"}, {"start": 2086.9799999999996, "end": 2091.22, "text": " Now we go and read this merge merges file"}, {"start": 2091.8599999999997, "end": 2094.58, "text": " that contains the basically all of the"}, {"start": 2095.3799999999997, "end": 2098.58, "text": " merges that happen through during the training of the bpe"}, {"start": 2099.3599999999997, "end": 2101.3599999999997, "text": " uh tokenizer"}, {"start": 2101.7799999999997, "end": 2108.66, "text": " Do check out my open ai clip video if you if you're confused by the tokenizer details i've covered that they are in a bit"}, {"start": 2109.4599999999996, "end": 2110.8199999999997, "text": " greater depth"}, {"start": 2110.82, "end": 2116.42, "text": " So let's continue here. We just stored the vocab. We loaded a couple of seconds ago"}, {"start": 2116.98, "end": 2119.94, "text": " We basically grabbed the the pairs from this"}, {"start": 2120.98, "end": 2125.78, "text": " Like uh merges object and we split them into well into doubles"}, {"start": 2126.34, "end": 2129.86, "text": " To get the pair. So let me show you an example of what the pair looked like so"}, {"start": 2130.34, "end": 2133.7000000000003, "text": " Pairs and i'm gonna print the first three pairs. You can see"}, {"start": 2134.56, "end": 2137.2200000000003, "text": " Inn characters are merged together"}, {"start": 2137.22, "end": 2143.2999999999997, "text": " Uh, and this weird I don't know how to pronounce this one, etc. Etc. So there is there's going to be a lot of those"}, {"start": 2144.18, "end": 2145.4599999999996, "text": " bpe"}, {"start": 2145.4599999999996, "end": 2146.74, "text": " merges"}, {"start": 2146.74, "end": 2150.98, "text": " Stored in the in the pairs this rank from pair is basically going to associate"}, {"start": 2151.8599999999997, "end": 2153.06, "text": " integer"}, {"start": 2153.06, "end": 2159.22, "text": " From zero to the length of pairs with each of the pairs and that's going to be used during the tokenization step"}, {"start": 2159.54, "end": 2161.22, "text": " I'm going to quickly"}, {"start": 2161.22, "end": 2163.22, "text": " Later show you how the tokenization"}, {"start": 2163.22, "end": 2169.4599999999996, "text": " Uh works on a very high level but i'm gonna skip this part here because it's fairly intricate to to to describe"}, {"start": 2169.9399999999996, "end": 2176.5, "text": " And uh, you can you can read on bpe yourself. We can just treat the tokenizer as a black box. That's fairly"}, {"start": 2177.14, "end": 2182.4199999999996, "text": " Common structure between all of these models. So i'm not gonna keep on explaining it every single time"}, {"start": 2182.74, "end": 2185.62, "text": " I did cover it in an open ai's clip, uh video"}, {"start": 2186.3399999999997, "end": 2188.02, "text": " okay, so"}, {"start": 2188.02, "end": 2194.66, "text": " Continuing on because the easy reusable flag is set. We're not going to beforehand create encoder decoder and the uh,"}, {"start": 2194.74, "end": 2196.74, "text": " The tokenizer which is the vqgen"}, {"start": 2197.22, "end": 2203.38, "text": " Uh decoder because otherwise the memory would literally explode so I don't have enough memory in my machine to do that"}, {"start": 2203.54, "end": 2206.34, "text": " So we're going to skip over that and now we have the generate image part"}, {"start": 2206.42, "end": 2213.46, "text": " Okay, so we have the model here and let's now generate the image again the encoder and everything else will now be built"}, {"start": 2213.46, "end": 2219.46, "text": " Uh on the fly so we have the text again. We have the photo of a funny dog blah blah same parameters before"}, {"start": 2219.78, "end": 2225.54, "text": " Let's see what this does. So there is this um, uh generator, uh function that they uh,"}, {"start": 2226.18, "end": 2232.34, "text": " Developed let me that's not that vital. So we're just gonna generate the stream and then grab doing next"}, {"start": 2232.34, "end": 2236.82, "text": " We're gonna grab a single image from the stream, but let's dig into the actual function"}, {"start": 2236.98, "end": 2239.7, "text": " Okay, nothing happened there because it's a generator"}, {"start": 2239.7, "end": 2246.74, "text": " So once then this line once we hit this line when they go f10, we're gonna enter the actual function. Okay, so"}, {"start": 2247.54, "end": 2254.18, "text": " Similarly here everything interesting happens in the generate draw stream part. So i'm gonna ignore everything else here"}, {"start": 2255.22, "end": 2258.8199999999997, "text": " So now let's see what's going on. Okay. So first off we um"}, {"start": 2259.62, "end": 2264.58, "text": " Depending on the grid size we estimate the number of images because the grid size is one for us"}, {"start": 2264.58, "end": 2270.18, "text": " We just generate a single image. So this grid they basically this project offers an"}, {"start": 2270.74, "end": 2276.02, "text": " Like an option to generate like a grid of multiple images here because of memory constraints"}, {"start": 2276.02, "end": 2280.18, "text": " I'm just dealing with single image so we can ignore the grid every single time"}, {"start": 2281.14, "end": 2283.62, "text": " Okay, so because we are in a verbose mode"}, {"start": 2283.62, "end": 2289.06, "text": " We are printing out some details here tokenizing text etc. Etc and now we hit the tokenization part"}, {"start": 2289.06, "end": 2294.74, "text": " So now we tokenize our input caption before we feed it into the bart encoder"}, {"start": 2294.98, "end": 2297.38, "text": " so let's see what's going on here, so they grab the"}, {"start": 2298.82, "end": 2301.86, "text": " Like a separation token the cls token"}, {"start": 2302.58, "end": 2307.22, "text": " And the unknown token here, uh, there's some processing that they do on on top of the text"}, {"start": 2307.22, "end": 2314.18, "text": " So they do lower casing and then they encode it as ascii with this error set to ignore and then they do decoding"}, {"start": 2314.2599999999998, "end": 2317.46, "text": " So this is just going to do some simple primitive processing of text"}, {"start": 2317.46, "end": 2320.1, "text": " Nothing fancy there. Let me let me print the text for you here"}, {"start": 2321.3, "end": 2327.38, "text": " Let's see what happened. Basically nothing happened with this particular sentence because it was fairly simple"}, {"start": 2327.94, "end": 2332.98, "text": " And then you literally kind of split the sentence into words"}, {"start": 2334.02, "end": 2338.9, "text": " And splitting is happening. So basically you use spaces to split the sentence into words"}, {"start": 2339.38, "end": 2343.7, "text": " And then each of the word of the words is going to be split into the sub words here"}, {"start": 2343.7, "end": 2348.02, "text": " Because the this get byte pair encoding which i'll skip height"}, {"start": 2348.4199999999996, "end": 2353.22, "text": " It's fairly complex to explain but it's just gonna split the word into bpe"}, {"start": 2353.8599999999997, "end": 2361.22, "text": " Like sub words and then each of the sub words is basically associated with the like with the specific index"}, {"start": 2361.7799999999997, "end": 2365.62, "text": " integer from from the vocab here and that's it on a high level and then we just kind of"}, {"start": 2365.9399999999996, "end": 2368.8999999999996, "text": " Bracket that with the cls token in the beginning and that this"}, {"start": 2368.9, "end": 2374.7400000000002, "text": " uh see a separation token at the at the end and that's how we form"}, {"start": 2375.14, "end": 2378.42, "text": " Uh, basically a sequence of tokens for our particular text"}, {"start": 2378.82, "end": 2385.46, "text": " Okay, then they do optionally here the cropping but because we are less than 64 tokens. I think this is set to 64"}, {"start": 2386.7400000000002, "end": 2390.1, "text": " Let me see what this number is, but i'm fairly sure it was 64"}, {"start": 2390.6600000000003, "end": 2397.3, "text": " Yep. So if if it's like a longer than that length then they crop the the"}, {"start": 2397.3, "end": 2399.3, "text": " the tokens"}, {"start": 2399.46, "end": 2400.98, "text": " But for us, it's not"}, {"start": 2400.98, "end": 2404.9, "text": " Okay, so just some printing because it's verbose mode blah blah blah"}, {"start": 2405.3, "end": 2406.9, "text": " Uh now"}, {"start": 2406.9, "end": 2410.34, "text": " As you can see, this is the interesting part. So they they create this variable"}, {"start": 2411.3, "end": 2417.86, "text": " Text tokens which is initialized as all ones and it has you can be confused by a couple of details here"}, {"start": 2417.94, "end": 2423.38, "text": " So first why why is once is because one is a bad token in this vocab?"}, {"start": 2423.38, "end": 2429.06, "text": " Uh, and that's kind of implicit here because this is as I said just a port from the original repo"}, {"start": 2429.1400000000003, "end": 2433.86, "text": " So sometimes sometimes things are hard to understand unless you well unless you understand what's going on"}, {"start": 2434.42, "end": 2438.1, "text": " So here here am I trying to explain to you that so that's one detail"}, {"start": 2438.1800000000003, "end": 2440.58, "text": " The second important detail is there is this number two?"}, {"start": 2441.54, "end": 2446.5, "text": " and why they do that is because they have a basically analog thing to"}, {"start": 2447.2200000000003, "end": 2450.02, "text": " Classify our free guidance that we saw in diffusion model"}, {"start": 2450.02, "end": 2455.38, "text": " So you can basically do the same thing for the class of autoregressive generative models and that's why they have two"}, {"start": 2455.7, "end": 2461.86, "text": " so as you see here they create the first sequence will just contain the cls and the"}, {"start": 2463.7, "end": 2469.3, "text": " Separator token so it's going to be literally empty sequence everything else going to be pad paddings"}, {"start": 2469.46, "end": 2475.14, "text": " So that's what we do on this line. And then on the second line we add the actual tokens. Okay"}, {"start": 2475.78, "end": 2478.42, "text": " So let me show you what this thing is"}, {"start": 2478.42, "end": 2480.42, "text": " so text tokens"}, {"start": 2480.7400000000002, "end": 2486.1800000000003, "text": " If I print text tokens zero and then first maybe seven tokens"}, {"start": 2486.1800000000003, "end": 2491.2200000000003, "text": " You can see it's an empty sequence because this is a cls. This is a"}, {"start": 2492.02, "end": 2496.34, "text": " Separator token and everything else is padding and the second one contains the actual"}, {"start": 2496.7400000000002, "end": 2499.06, "text": " Uh tokens for our caption. Okay, that's it"}, {"start": 2499.38, "end": 2503.54, "text": " Now we just convert that into tensor with a long data type"}, {"start": 2503.86, "end": 2505.86, "text": " We put it onto gpu"}, {"start": 2505.86, "end": 2508.58, "text": " gpu and we continue on okay"}, {"start": 2509.46, "end": 2514.9, "text": " Now here is the optimization I was mentioning because we're using this not reusable"}, {"start": 2515.7000000000003, "end": 2521.46, "text": " We're because we said this flag is reusable to false. We now on the fly initialize the encoder"}, {"start": 2521.6200000000003, "end": 2527.6200000000003, "text": " So the encoder is going to be the vq-gan encoder. So let me step inside of here and let's see what's going on"}, {"start": 2527.94, "end": 2530.9, "text": " So again, uh, because I already downloaded the model"}, {"start": 2530.9, "end": 2537.38, "text": " We can skip over all of these steps and we're going to focus on how this dali birth encoder is constructed"}, {"start": 2538.1800000000003, "end": 2543.62, "text": " So the interesting part as usually is going to be in the forward pass of the of the model"}, {"start": 2544.34, "end": 2545.62, "text": " and"}, {"start": 2545.62, "end": 2548.6, "text": " After we construct it you can see here. We just load the parameters"}, {"start": 2549.38, "end": 2554.26, "text": " Of the pre-trained model that was trained in the actual dali mini"}, {"start": 2555.36, "end": 2556.9, "text": " repository"}, {"start": 2556.9, "end": 2562.02, "text": " We're gonna load those parameters and then we're gonna push to do this this model onto"}, {"start": 2562.6600000000003, "end": 2564.58, "text": " the gpu, okay"}, {"start": 2564.58, "end": 2568.34, "text": " So let's see how dali birth encoder looks like"}, {"start": 2568.98, "end": 2576.34, "text": " Here we are. We have the the vocab size of 50k something we form the embedding tables for the"}, {"start": 2577.3, "end": 2579.3, "text": " Well, that's the the text vocab"}, {"start": 2579.62, "end": 2585.78, "text": " Because encoder again remember has the text vocab and the decoder will be generating the image token"}, {"start": 2585.78, "end": 2587.78, "text": " So it will have a separate"}, {"start": 2588.02, "end": 2590.02, "text": " image vocab"}, {"start": 2590.34, "end": 2594.02, "text": " Okay, so we generate that table the embedding vectors will be"}, {"start": 2594.88, "end": 2597.7000000000003, "text": " 2048 we then generate the positional"}, {"start": 2598.34, "end": 2602.26, "text": " Encodings here. This is going to be only 64 because if you recall before"}, {"start": 2602.5800000000004, "end": 2607.2200000000003, "text": " We had the cropping so that we make sure that the number of tokens that we feed into the bart"}, {"start": 2607.3, "end": 2612.5800000000004, "text": " Encoder is always up to 64 never more than that. And that's why we never need to go"}, {"start": 2612.58, "end": 2616.02, "text": " Uh above 64 positional encodings. Okay"}, {"start": 2616.58, "end": 2618.58, "text": " now we start forming the"}, {"start": 2619.62, "end": 2621.14, "text": " basically"}, {"start": 2621.14, "end": 2624.42, "text": " Encoder layers and we'll have 24 of these"}, {"start": 2625.06, "end": 2626.18, "text": " for the"}, {"start": 2626.18, "end": 2628.42, "text": " Bart encoder for dali mega"}, {"start": 2629.2999999999997, "end": 2633.4, "text": " I'm gonna hit go here and put a small breakpoint"}, {"start": 2634.34, "end": 2637.56, "text": " In an encoder layer, which is a simple transformer"}, {"start": 2637.56, "end": 2643.4, "text": " Uh block nothing fancy. The interesting part is this glue. So i'm gonna set uh one"}, {"start": 2644.44, "end": 2646.2799999999997, "text": " breakpoint there"}, {"start": 2646.2799999999997, "end": 2650.68, "text": " Uh, and that's it. So let's now continue execution here"}, {"start": 2651.88, "end": 2656.06, "text": " So we end up in the encoder layer as you can see, it's just a simple transformer"}, {"start": 2656.7599999999998, "end": 2660.2, "text": " Block we have the encoder. We have the self-attention module"}, {"start": 2660.52, "end": 2665.48, "text": " we have the some layer norms and we have glue which is just a modification of the"}, {"start": 2665.48, "end": 2667.56, "text": " the uh, your regular"}, {"start": 2668.68, "end": 2675.48, "text": " like um mlp that's used of the feed forward, uh, like a layer of the transformer block, okay"}, {"start": 2676.6, "end": 2679.48, "text": " So let's see how glue looks like here is glue"}, {"start": 2680.36, "end": 2684.14, "text": " It's a very simple modification by noam scherzer"}, {"start": 2684.52, "end": 2688.44, "text": " And i'm gonna now open up the paper side by side to show you"}, {"start": 2689.16, "end": 2694.2, "text": " What this is and to make sense out of it, but it's a very experimental finding"}, {"start": 2694.2, "end": 2701.48, "text": " So there is nothing. Uh, well fundamental understanding you can get from this. Okay guys here it is side by side"}, {"start": 2702.04, "end": 2704.04, "text": " here are the equations from the"}, {"start": 2704.3599999999997, "end": 2706.3599999999997, "text": " glue paper by noam scherzer"}, {"start": 2706.52, "end": 2711.48, "text": " And you can see that this layer here implements exactly this line here"}, {"start": 2711.8799999999997, "end": 2717.64, "text": " So the idea here was they they kind of like he ablated a couple of um, he made a couple of modifications"}, {"start": 2717.7999999999997, "end": 2722.7599999999998, "text": " Why not using different activation functions instead of rally? Why not using gelu or swish etc?"}, {"start": 2722.76, "end": 2727.5600000000004, "text": " Etc. Why not combine him combine them in a bit different ways using um"}, {"start": 2728.1200000000003, "end": 2731.1600000000003, "text": " Similarly to the gated linear units so you can see so"}, {"start": 2732.0400000000004, "end": 2734.6800000000003, "text": " Like the lee mini ended up using this variant here"}, {"start": 2735.5600000000004, "end": 2738.5200000000004, "text": " And let me just kind of convince you that that's indeed the case"}, {"start": 2739.0800000000004, "end": 2745.88, "text": " So let's see so we have we get the input x this is x and then we pass it through layer norm"}, {"start": 2746.6800000000003, "end": 2751.0, "text": " So ignoring that part we we pass it we then pass it through a linear layer"}, {"start": 2751.0, "end": 2753.16, "text": " so that's that corresponds to"}, {"start": 2753.88, "end": 2759.72, "text": " basically this bracket here and after that we we pass it through"}, {"start": 2760.12, "end": 2765.0, "text": " Gelu, so that's this part the sigma of this so that's this term"}, {"start": 2765.56, "end": 2772.68, "text": " And then we have v which is formed from x as well from the input. We also pass it through the linear layer"}, {"start": 2772.76, "end": 2775.56, "text": " so that's that's this part the x times"}, {"start": 2776.44, "end": 2778.36, "text": " big v"}, {"start": 2778.36, "end": 2780.36, "text": " and finally we"}, {"start": 2780.36, "end": 2782.1200000000003, "text": " do"}, {"start": 2782.1200000000003, "end": 2785.7200000000003, "text": " element wise product so hadamard product here"}, {"start": 2786.52, "end": 2789.56, "text": " and then we pass that product through"}, {"start": 2790.36, "end": 2792.36, "text": " a layer norm and"}, {"start": 2792.84, "end": 2799.96, "text": " So that's that's that's basically what happened here and finally we we do a feed forward through a linear layer"}, {"start": 2800.04, "end": 2803.7200000000003, "text": " So that's the w that's the that's the sorry. That's the w2 here"}, {"start": 2803.72, "end": 2811.08, "text": " And that's thus you can see that this simple forward pass of this glue model implements exactly"}, {"start": 2811.56, "end": 2817.9599999999996, "text": " This ffn glue from norm shimseyer's paper and that's it. Now. Let's go back to the code"}, {"start": 2818.9199999999996, "end": 2820.12, "text": " I'm gonna"}, {"start": 2820.12, "end": 2826.52, "text": " Remove this break point. So it's a simple simple simple modification instead of having just like a like a feed forward"}, {"start": 2827.08, "end": 2832.9199999999996, "text": " Like a basically a linear layer for all but a relu followed by a linear layer. It's a bit more intricate"}, {"start": 2832.92, "end": 2838.04, "text": " Uh, and they showed experimentally that this gives a better performance"}, {"start": 2838.04, "end": 2844.52, "text": " But there is no ultimately no theory behind why this should work and that's it. That's I guess deep learning"}, {"start": 2845.08, "end": 2846.84, "text": " Let's continue"}, {"start": 2846.84, "end": 2848.84, "text": " I'm gonna now"}, {"start": 2849.08, "end": 2854.2000000000003, "text": " We're now back into encoder layer and again encoder layer is simply a transformer block"}, {"start": 2854.2000000000003, "end": 2856.84, "text": " So we have as you can see here we have some"}, {"start": 2856.84, "end": 2862.6800000000003, "text": " um layer norm and then attention and then norm and then the residual connection and then"}, {"start": 2863.96, "end": 2869.6400000000003, "text": " Then we pass through the feed forward part and then we have again the residual connection and we return back the variable"}, {"start": 2869.6400000000003, "end": 2871.2400000000002, "text": " So that's it. So i'm gonna"}, {"start": 2871.2400000000002, "end": 2875.7200000000003, "text": " Basically ignore all of these break points and let's form the 24 layers"}, {"start": 2876.44, "end": 2878.6000000000004, "text": " Of this birth of bart"}, {"start": 2879.4, "end": 2881.4, "text": " Encoder so i'm gonna hit f5"}, {"start": 2882.28, "end": 2884.28, "text": " i'm gonna form 24 layers"}, {"start": 2885.1600000000003, "end": 2886.76, "text": " and then"}, {"start": 2886.76, "end": 2888.6000000000004, "text": " We are now here"}, {"start": 2888.6000000000004, "end": 2890.6000000000004, "text": " We form additional layer norm"}, {"start": 2891.0800000000004, "end": 2898.44, "text": " Additional layer norm and then we form uh these token indices which are going to be used to index the positional embeddings"}, {"start": 2898.92, "end": 2900.1200000000003, "text": " here"}, {"start": 2900.1200000000003, "end": 2902.1200000000003, "text": " and that's it and we just"}, {"start": 2902.92, "end": 2906.5200000000004, "text": " Do times two because remember we are doing the classifier free"}, {"start": 2907.4, "end": 2910.1200000000003, "text": " Trick for all progressive models. That's why we have"}, {"start": 2911.1600000000003, "end": 2912.84, "text": " Batch size of two here"}, {"start": 2912.84, "end": 2914.36, "text": " Okay"}, {"start": 2914.36, "end": 2921.0, "text": " That's it. That's the that's the initialization of the bart encoder. Now we are loading the weights here"}, {"start": 2921.8, "end": 2923.2400000000002, "text": " and then"}, {"start": 2923.2400000000002, "end": 2925.2400000000002, "text": " we kind of"}, {"start": 2925.4, "end": 2929.88, "text": " Push those weights into the encoder structure and then we delete the parameters now"}, {"start": 2929.88, "end": 2935.6400000000003, "text": " You can already see that this is pushing my gpu somewhat. Well, well actually not that much"}, {"start": 2935.8, "end": 2941.88, "text": " Okay, so i'm surprised. This is actually not that big of a memory footprint"}, {"start": 2941.88, "end": 2946.76, "text": " I think some of the later components will be spiking it up. But yeah, let's continue"}, {"start": 2948.12, "end": 2949.88, "text": " so"}, {"start": 2949.88, "end": 2952.84, "text": " After the initialization of the encoder, let's see what else happens"}, {"start": 2953.4, "end": 2961.1600000000003, "text": " Okay, and now we grab the text tokens and we pass them through the bart encoder as we saw in the paper part of the video"}, {"start": 2961.56, "end": 2964.2000000000003, "text": " Okay, let's step into it"}, {"start": 2965.08, "end": 2967.48, "text": " so we're gonna hit the for function of the"}, {"start": 2968.36, "end": 2970.36, "text": " Bart encoder here"}, {"start": 2970.36, "end": 2977.48, "text": " So let's see what happens. So we ask where the text tokens are not equal to one and one is again remember"}, {"start": 2978.04, "end": 2982.2000000000003, "text": " They're not using a constant here. They're just using one to represent the pad tokens"}, {"start": 2982.52, "end": 2984.92, "text": " It's kind of hard coded and dirty but it works"}, {"start": 2985.4, "end": 2987.32, "text": " Okay, so not equal"}, {"start": 2987.32, "end": 2989.0, "text": " to one"}, {"start": 2989.0, "end": 2994.1200000000003, "text": " That's the actual content. That's where attention mask is going to say true"}, {"start": 2994.52, "end": 2998.52, "text": " So let's now do something like this. Let me print this you'll see that"}, {"start": 2998.52, "end": 3005.16, "text": " It basically for the second component of the text tokens where we actually passed the the the caption tokens"}, {"start": 3005.32, "end": 3007.08, "text": " We have a bunch of truths here"}, {"start": 3007.08, "end": 3013.16, "text": " But for the first one we only have true true for the first two because those are the cls and the separator token"}, {"start": 3013.32, "end": 3014.44, "text": " Okay"}, {"start": 3014.44, "end": 3017.4, "text": " now what we do is we pass those tokens into the"}, {"start": 3018.12, "end": 3025.0, "text": " We embed them and then we just add the positional encodings using the these uh, well the the pose"}, {"start": 3025.0, "end": 3031.0, "text": " Those tokens indices that were previously constructed and that's it. That's simple stuff there"}, {"start": 3031.64, "end": 3035.56, "text": " And now we just pass it through the layer norm and then we keep on"}, {"start": 3036.9, "end": 3039.16, "text": " processing those embeddings"}, {"start": 3039.96, "end": 3045.56, "text": " Via the transformer Bart transformer layers and that's it. I'm just gonna hit that five here"}, {"start": 3045.88, "end": 3052.52, "text": " We're gonna do a forward pass through the transformer and we end up with the final representation here called encoder state"}, {"start": 3052.52, "end": 3058.2, "text": " Okay, and because again, this is the optimization I was mentioning. We now delete the weights of the encoder"}, {"start": 3058.2, "end": 3063.56, "text": " Let's see how the the memory footprint is gonna reduce. Okay, so the memory footprint went up"}, {"start": 3063.56, "end": 3070.68, "text": " You can see it here and after we hit delete and empty cache, you're gonna see how it kind of dips down"}, {"start": 3071.56, "end": 3073.72, "text": " Hopefully, I'm not sure what's going on"}, {"start": 3075.24, "end": 3079.92, "text": " Why is my gpu not reacting here? Not sure in any case"}, {"start": 3079.92, "end": 3086.2400000000002, "text": " Let's continue so now we have the initialized decoder part i'm gonna hit"}, {"start": 3087.76, "end": 3090.96, "text": " Step over and we enter the init decoder"}, {"start": 3091.44, "end": 3095.6800000000003, "text": " So again, it's already downloaded so we don't care about that part"}, {"start": 3096.2400000000002, "end": 3101.36, "text": " Here we construct it and again, we'll load the weights a similar structure as as with the encoder"}, {"start": 3102.08, "end": 3107.12, "text": " Now we are passing the image vocab, etc. So let's enter the constructor here"}, {"start": 3107.12, "end": 3109.6, "text": " So let's see what's the difference?"}, {"start": 3110.16, "end": 3116.56, "text": " So here we form again the embedding table this time. We have the image vocab size plus one"}, {"start": 3117.2, "end": 3124.0, "text": " Because let me just think here because they also have the they pass this this beginning of sentence token"}, {"start": 3124.72, "end": 3128.72, "text": " And that's the additional token that they need in addition to the"}, {"start": 3129.68, "end": 3131.68, "text": " image to image code book"}, {"start": 3132.08, "end": 3135.04, "text": " So I think they mentioned this somewhere in the in the report"}, {"start": 3135.04, "end": 3137.6, "text": " Let me show you this so basically"}, {"start": 3138.08, "end": 3140.08, "text": " It's called bos token"}, {"start": 3140.48, "end": 3145.84, "text": " Okay, so they say here at inference time blah blah. So a bos token is fed through the bar decoder"}, {"start": 3148.16, "end": 3155.36, "text": " And yeah, that's so that's basically why there is plus one as I said not not the cleanest way to do something"}, {"start": 3155.36, "end": 3158.4, "text": " But yeah, this is just a port of the original repo"}, {"start": 3158.96, "end": 3163.04, "text": " Now we have we form the table for the position"}, {"start": 3163.04, "end": 3165.2799999999997, "text": " in encodings and there is only"}, {"start": 3166.08, "end": 3170.08, "text": " 256 of those because as you recall from the paper part"}, {"start": 3170.64, "end": 3174.96, "text": " It's going to the latent space is going to be 16 times 16 of those"}, {"start": 3175.68, "end": 3177.12, "text": " basically"}, {"start": 3177.12, "end": 3178.72, "text": " vectors"}, {"start": 3178.72, "end": 3183.04, "text": " That are then going to be fed into dvq again decoder. Okay, let's continue here"}, {"start": 3183.36, "end": 3185.84, "text": " So we now generate a bunch of decoder layers"}, {"start": 3186.4, "end": 3190.88, "text": " And the coder layers have additionally the cross attention component because they are"}, {"start": 3190.88, "end": 3195.28, "text": " the cross attention component because this is again encoder decoder transformer"}, {"start": 3195.92, "end": 3198.1600000000003, "text": " so, let me see whether it makes sense to"}, {"start": 3200.7200000000003, "end": 3202.7200000000003, "text": " To kind of show you that part"}, {"start": 3203.12, "end": 3204.88, "text": " Let me just think"}, {"start": 3204.88, "end": 3210.96, "text": " Okay, i'm gonna put a break point into the mid part here and we can maybe do a single pass"}, {"start": 3212.08, "end": 3214.7400000000002, "text": " Here because they're doing something interesting"}, {"start": 3215.36, "end": 3218.08, "text": " With this attention state we'll see that a bit later"}, {"start": 3218.08, "end": 3220.88, "text": " And here as well i'm going to put the break point here"}, {"start": 3221.2799999999997, "end": 3225.68, "text": " And now let's continue. Okay, so let's hit the first decoder layer"}, {"start": 3226.64, "end": 3228.0, "text": " It's again"}, {"start": 3228.0, "end": 3230.0, "text": " Simply there is a self-attention part"}, {"start": 3230.0, "end": 3234.72, "text": " There is the decoder there is a cross attention part and there is the feed forward part"}, {"start": 3234.72, "end": 3238.24, "text": " So the glue variant in this particular case, so that's pretty much it"}, {"start": 3238.24, "end": 3241.6, "text": " So i'm going to now remove the break point from the init"}, {"start": 3242.24, "end": 3245.7599999999998, "text": " And let's just generate a bunch of these layers"}, {"start": 3245.76, "end": 3247.36, "text": " layers"}, {"start": 3247.36, "end": 3248.48, "text": " oops"}, {"start": 3248.48, "end": 3254.48, "text": " Let's now do that and we do that for 24 times again. I think let me just print the"}, {"start": 3255.44, "end": 3259.6000000000004, "text": " Layer count 24. Okay, so there is 24 decoder layers as well"}, {"start": 3261.2000000000003, "end": 3267.2000000000003, "text": " Okay, we have some layer norms nothing fancy and finally we have we take the final"}, {"start": 3267.6000000000004, "end": 3270.8, "text": " embedding vectors and we map them into the"}, {"start": 3270.8, "end": 3278.1600000000003, "text": " The space that has this dimensionality because remember we now want to sample those image tokens"}, {"start": 3278.1600000000003, "end": 3280.6400000000003, "text": " That's why we need to to to go from"}, {"start": 3281.5800000000004, "end": 3288.4, "text": " 2048 which was the internal model dimension. We want to go into into this dimension here"}, {"start": 3288.7200000000003, "end": 3291.28, "text": " Okay, let's continue on here"}, {"start": 3292.0, "end": 3298.8, "text": " And that's it. Okay, that's pretty much it. We load the weights again 24 layers of decoder"}, {"start": 3298.8, "end": 3300.48, "text": " blocks"}, {"start": 3300.48, "end": 3303.28, "text": " and finally then we we map into the"}, {"start": 3304.2400000000002, "end": 3305.84, "text": " image vocab"}, {"start": 3305.84, "end": 3311.2000000000003, "text": " Size dimensionality space output space so that we can sample from it later"}, {"start": 3312.1600000000003, "end": 3313.44, "text": " Okay"}, {"start": 3313.44, "end": 3320.96, "text": " Again, we delete the parameters. We put the decoder on top. We push it to our to my gpu. So let's see whether"}, {"start": 3321.6000000000004, "end": 3323.6000000000004, "text": " Something changes here. Nope"}, {"start": 3323.6, "end": 3328.7999999999997, "text": " Uh, it's a bit different behavior compared to what I've seen before usually there is a spike"}, {"start": 3329.52, "end": 3330.4, "text": " um"}, {"start": 3330.4, "end": 3334.88, "text": " Bigger than this one happening and i'm tracking this uh slot here by the way"}, {"start": 3335.52, "end": 3336.88, "text": " Okay"}, {"start": 3336.88, "end": 3343.2, "text": " In any case now comes the interesting part. So here we are going to now start sampling from the decoder"}, {"start": 3343.44, "end": 3346.4, "text": " Those image tokens and this one is already trained"}, {"start": 3346.48, "end": 3350.7999999999997, "text": " So everything it samples is going to be meaningful and then we're just going to pass it through the vq"}, {"start": 3350.8, "end": 3356.5600000000004, "text": " GAN decoder and that's the that's the end of the program. Okay, let's go here. So um"}, {"start": 3357.6800000000003, "end": 3363.6800000000003, "text": " We're dealing with float 32. So this doesn't make much sense, but with float 30, uh float 16"}, {"start": 3364.1600000000003, "end": 3371.2000000000003, "text": " Uh, this kind of puts it puts all the operations inside of this context in the float float 16 regime which helps us save"}, {"start": 3371.84, "end": 3373.2000000000003, "text": " some memory"}, {"start": 3373.2000000000003, "end": 3375.6000000000004, "text": " So in the case when you have multiple images"}, {"start": 3375.6, "end": 3380.16, "text": " Uh, this expanded indices make sense because you will not replicate"}, {"start": 3380.64, "end": 3386.7999999999997, "text": " The the textual captions for multiple images because you'll be generating multiple images for one"}, {"start": 3387.36, "end": 3390.0, "text": " Particular caption, but we don't care about that here"}, {"start": 3390.64, "end": 3392.64, "text": " so we just basically"}, {"start": 3393.52, "end": 3399.36, "text": " Grab the uh encoder state and the and the and the text tokens and those are of dimensionality"}, {"start": 3399.44, "end": 3401.7599999999998, "text": " So this is going to be 264 if you recall"}, {"start": 3402.4, "end": 3404.4, "text": " and this one is going to be I guess"}, {"start": 3404.4, "end": 3406.4, "text": " uh to"}, {"start": 3406.46, "end": 3408.46, "text": " 2000 something 2048"}, {"start": 3409.12, "end": 3411.12, "text": " because that's the"}, {"start": 3411.52, "end": 3417.2000000000003, "text": " Output of the encoder, right? So to oh actually yeah. Yeah 264"}, {"start": 3418.06, "end": 3423.76, "text": " 2048 because we just embedded this into this dimensionality here. Okay, so we formed the mask again"}, {"start": 3423.84, "end": 3426.0, "text": " This is the padding token. So mask is going to be"}, {"start": 3426.7200000000003, "end": 3428.48, "text": " true"}, {"start": 3428.48, "end": 3430.4, "text": " for for the"}, {"start": 3430.4, "end": 3432.4, "text": " non-pad tokens"}, {"start": 3432.4, "end": 3437.2000000000003, "text": " Uh, here is the we form this attention state which is gonna contain the"}, {"start": 3437.84, "end": 3443.76, "text": " Keys and values from each of the layers and for each of the positions in the bart decoder"}, {"start": 3443.92, "end": 3449.12, "text": " That's just how they decided to implement this and finally image tokens is gonna contain"}, {"start": 3449.52, "end": 3451.92, "text": " That's the output array that's gonna contain two"}, {"start": 3452.7000000000003, "end": 3454.7000000000003, "text": " 256 tokens"}, {"start": 3455.2000000000003, "end": 3460.56, "text": " Plus one because the first one will be the I think yeah the bos"}, {"start": 3460.56, "end": 3467.04, "text": " Uh token the beginning of uh sentence token. So let's continue we're gonna initialize"}, {"start": 3467.84, "end": 3469.84, "text": " this um"}, {"start": 3470.08, "end": 3476.96, "text": " Vector initially as you can see here with this value here, which is going to be the beginning of sentence"}, {"start": 3477.44, "end": 3480.32, "text": " uh tokens id so let me show you what"}, {"start": 3481.12, "end": 3485.92, "text": " That looks like i'm going to take the first five elements and all of them are going to be equal like this"}, {"start": 3485.92, "end": 3487.92, "text": " So that's how how they initially"}, {"start": 3487.92, "end": 3494.08, "text": " Um, well initialize the image tokens and then later we're gonna keep on as we produce the image tokens"}, {"start": 3494.08, "end": 3496.88, "text": " We're gonna feed them back into this image tokens array. Okay"}, {"start": 3497.52, "end": 3499.2000000000003, "text": " So nothing fancy there"}, {"start": 3499.2000000000003, "end": 3503.28, "text": " Uh continuing on we form these uh token indices"}, {"start": 3503.92, "end": 3508.42, "text": " And we have some settings like temperature is set to one top k 256"}, {"start": 3509.02, "end": 3511.02, "text": " superconditioning factor"}, {"start": 3512.08, "end": 3514.32, "text": " Which is used for the classifier free, uh"}, {"start": 3514.32, "end": 3520.56, "text": " Uh guidance part and that's it. Those are the settings and now we start iterating basically a for loop"}, {"start": 3521.42, "end": 3523.84, "text": " 256 times we're gonna generate a novel"}, {"start": 3524.48, "end": 3526.1600000000003, "text": " image vector"}, {"start": 3526.1600000000003, "end": 3532.4, "text": " Uh, and uh, basically then we're gonna feed that into the vq-gan decoder as I said multiple times already"}, {"start": 3533.28, "end": 3534.88, "text": " so"}, {"start": 3534.88, "end": 3540.96, "text": " We care about this part of the loop because this one we'll enter this one once we have 256 tokens"}, {"start": 3540.96, "end": 3544.0, "text": " Um, we'll enter the image grid from tokens function"}, {"start": 3544.88, "end": 3550.56, "text": " So let's focus on this part here. So what happens here as you can see whatever we take the image"}, {"start": 3550.64, "end": 3554.2400000000002, "text": " So i is initially zero, which means we initially feed the b.os"}, {"start": 3554.32, "end": 3556.7200000000003, "text": " So the beginning of of of sentence token"}, {"start": 3557.36, "end": 3562.32, "text": " And we feed the output result into into the first slot of this image tokens"}, {"start": 3562.56, "end": 3565.6, "text": " And that's how we start populating this the image tokens"}, {"start": 3566.32, "end": 3570.32, "text": " Array again, its dimensionality is one plus 256"}, {"start": 3570.32, "end": 3573.28, "text": " It's precisely because of this behavior here"}, {"start": 3574.2400000000002, "end": 3578.4, "text": " We pass the tokens we pass the attention mask the encoder state the attention state"}, {"start": 3578.48, "end": 3581.2000000000003, "text": " Which is going to contain the keys and values from the decoder"}, {"start": 3581.76, "end": 3586.0, "text": " And that's it and we we pass the token indices. So we basically pass"}, {"start": 3586.6400000000003, "end": 3588.6400000000003, "text": " Where we are in the generation?"}, {"start": 3588.6400000000003, "end": 3589.92, "text": " uh"}, {"start": 3589.92, "end": 3596.32, "text": " In the generation sequence, okay in the generated sequence. Okay. Now let's step inside of the uh forward function"}, {"start": 3597.2000000000003, "end": 3599.2000000000003, "text": " of the bar decoder"}, {"start": 3599.2, "end": 3600.48, "text": " so"}, {"start": 3600.48, "end": 3602.48, "text": " We grab the number of images"}, {"start": 3602.56, "end": 3604.56, "text": " uh, we then basically"}, {"start": 3605.68, "end": 3610.66, "text": " Make sure that the batch dimension is corresponds to to the super conditioning"}, {"start": 3611.68, "end": 3613.68, "text": " Thingy, so it's going to be two"}, {"start": 3614.08, "end": 3616.08, "text": " Okay, so it's going to be two there"}, {"start": 3616.3999999999996, "end": 3619.2, "text": " We grab the the list of previous tokens"}, {"start": 3620.3999999999996, "end": 3623.2799999999997, "text": " uh, so previous tokens dimensionality is"}, {"start": 3624.24, "end": 3627.8399999999997, "text": " Uh one so that's going to be initially just a b.os token"}, {"start": 3627.84, "end": 3629.84, "text": " and then"}, {"start": 3629.92, "end": 3633.1200000000003, "text": " Because we only have a single image. This is just going to replicate"}, {"start": 3633.76, "end": 3637.04, "text": " Uh the previous tokens we're going to have two b.os tokens"}, {"start": 3638.08, "end": 3639.36, "text": " And that's it"}, {"start": 3639.36, "end": 3641.44, "text": " Again, the reason being the super conditioning"}, {"start": 3642.1600000000003, "end": 3645.76, "text": " That they are doing so the classifier free guidance thingy for autoregressive models"}, {"start": 3646.32, "end": 3648.48, "text": " Uh some clamping. I don't think this actually"}, {"start": 3649.44, "end": 3651.1200000000003, "text": " is needed"}, {"start": 3651.12, "end": 3657.8399999999997, "text": " And then so now we do the embedding but remember this time. This is the these are the image embeddings"}, {"start": 3658.0, "end": 3661.68, "text": " Okay, so these are the as you can see here. These are this is the embedding table"}, {"start": 3662.16, "end": 3669.2, "text": " This is a separate vocabulary compared to the text that we use for the bar encoder part. Okay, so we do the encoding"}, {"start": 3669.7599999999998, "end": 3671.7599999999998, "text": " Then we add the positional"}, {"start": 3672.3199999999997, "end": 3673.92, "text": " Encoding here"}, {"start": 3673.92, "end": 3676.16, "text": " And now we have the coder state. It's going to be I guess"}, {"start": 3677.2, "end": 3679.2, "text": " What two?"}, {"start": 3679.2, "end": 3681.2, "text": " One"}, {"start": 3681.66, "end": 3683.2799999999997, "text": " 2000 something maybe"}, {"start": 3683.2799999999997, "end": 3686.24, "text": " Yeah, no two two two thousand forty eight makes sense"}, {"start": 3686.48, "end": 3692.7999999999997, "text": " Okay, because we have b.os a single token we have two because of super conditioning and we just embedded them into this"}, {"start": 3693.8399999999997, "end": 3697.68, "text": " Dimensionality here. So all of that makes sense. Let's continue on here"}, {"start": 3698.3999999999996, "end": 3700.7999999999997, "text": " Now we add the additional dimensions now. It's going to be"}, {"start": 3700.8, "end": 3708.0800000000004, "text": " 21 2048 and now we just iterate through the decoder"}, {"start": 3709.44, "end": 3711.44, "text": " layers of this decoder"}, {"start": 3711.6800000000003, "end": 3714.1600000000003, "text": " So we pass the decoder state"}, {"start": 3714.7200000000003, "end": 3721.2000000000003, "text": " The encoder state because we have the cross attention. Remember we passed the attention state. So attention state is going to be"}, {"start": 3722.94, "end": 3724.1600000000003, "text": " Basically"}, {"start": 3724.1600000000003, "end": 3726.1600000000003, "text": " um number of layers"}, {"start": 3726.16, "end": 3732.0, "text": " uh, and then four because we're going to keep keys and values and times two because we have the"}, {"start": 3732.62, "end": 3736.96, "text": " superconditioning and then finally we're going to have 256 and"}, {"start": 3737.8199999999997, "end": 3745.44, "text": " 2048 so again 24 because we have 24 layers four because we are saving key and value"}, {"start": 3746.3999999999996, "end": 3750.08, "text": " For each of the layers but times two because we are using"}, {"start": 3750.08, "end": 3757.36, "text": " Uh two sequences the one with uh empty sequence and the one with the actual caption because of superconditioning and then 256 is because we have"}, {"start": 3757.66, "end": 3760.56, "text": " 256 image tokens in the decoder and"}, {"start": 3761.2599999999998, "end": 3766.3199999999997, "text": " 2048 because that's the model dimension. Okay, just kind of breaking down. The shape is always useful"}, {"start": 3766.96, "end": 3769.2799999999997, "text": " we pass the attention mask for the"}, {"start": 3771.44, "end": 3776.0, "text": " Input caption and we pass the token now, let's hit the forward function here"}, {"start": 3776.0, "end": 3781.44, "text": " So here we just generate the attention mask depending on where we are in the generation process"}, {"start": 3781.44, "end": 3784.48, "text": " So because token token index currently is zero"}, {"start": 3784.96, "end": 3790.0, "text": " We are we're just we're we're just have passed the b.os. So the beginning of sentence token"}, {"start": 3790.32, "end": 3794.32, "text": " And that's why the self-attention mask will currently just be active in the first"}, {"start": 3794.88, "end": 3798.96, "text": " In the zero slot and everything else is false. So you can see here"}, {"start": 3799.92, "end": 3803.28, "text": " That the first will be true and then everything else is false"}, {"start": 3803.28, "end": 3809.92, "text": " is false and then we just kind of make sure that we have batch dimension of two because of superconditioning"}, {"start": 3810.88, "end": 3816.4, "text": " And then we just pass the equator state through the layer normalization. Nothing fancy there"}, {"start": 3817.6800000000003, "end": 3819.6800000000003, "text": " and now we pass it through the"}, {"start": 3819.9, "end": 3823.84, "text": " Attention part. So i'm gonna skip the attention part is fairly. Um"}, {"start": 3824.96, "end": 3826.96, "text": " Well, actually, let me see"}, {"start": 3827.0400000000004, "end": 3830.32, "text": " This is where we will store the keys and values. Let me show you this part"}, {"start": 3830.32, "end": 3833.04, "text": " So let me just see whether I have uh"}, {"start": 3836.0, "end": 3839.76, "text": " Okay, so we have a self-attention we have a break point there so i'm gonna hit"}, {"start": 3840.96, "end": 3844.6400000000003, "text": " That break point and let's see what we do. So we store the keys and values"}, {"start": 3845.92, "end": 3850.0, "text": " And let's see what their dimensionality is. So we have two one"}, {"start": 3850.94, "end": 3854.48, "text": " 2048 and the same for for a value vector"}, {"start": 3855.04, "end": 3859.28, "text": " We basically concatenate them and then we store it inside of this attention state"}, {"start": 3859.28, "end": 3862.48, "text": " And that's it. And now grabbing from the attention state"}, {"start": 3862.88, "end": 3866.5600000000004, "text": " We're going to grab all of the previous keys up to this token"}, {"start": 3866.96, "end": 3871.84, "text": " Which in this particular case is only these keys and values because we are just starting the generation"}, {"start": 3872.1600000000003, "end": 3875.92, "text": " But that's how the how the logic is implemented using this attention state"}, {"start": 3875.92, "end": 3880.2400000000002, "text": " We're basically fetching all of the previous keys and values and that's it"}, {"start": 3880.2400000000002, "end": 3885.1200000000003, "text": " And now we just pass it through the we just do the attention logic and we return back here"}, {"start": 3886.48, "end": 3888.48, "text": " That's it now we have the"}, {"start": 3888.48, "end": 3891.76, "text": " uh cross attention part we passed the decoder and encoder"}, {"start": 3892.56, "end": 3900.72, "text": " Uh, and that's it. Everything else is standard transformer logic. We pass it now through the glue and that's a single step through the"}, {"start": 3901.2, "end": 3904.32, "text": " Decoder layer i'm gonna now remove the break points here"}, {"start": 3905.44, "end": 3907.84, "text": " I'm gonna remove the break point here as well"}, {"start": 3908.8, "end": 3910.8, "text": " And let's continue on here"}, {"start": 3910.8, "end": 3917.84, "text": " I'm gonna hit f5 we went through all of the layers and now we pass it through uh, again"}, {"start": 3917.84, "end": 3921.76, "text": " Uh, this final l n is just like layer norm"}, {"start": 3922.2400000000002, "end": 3925.2000000000003, "text": " Uh, and finally here is where the mapping happens"}, {"start": 3925.52, "end": 3928.7200000000003, "text": " So here we're gonna take the vector that's 2 1"}, {"start": 3929.34, "end": 3936.5, "text": " 2048 and we're gonna map that into the corresponding into the image space. So basically now the logits"}, {"start": 3937.92, "end": 3941.7000000000003, "text": " Will be of 2 1 16 000 416"}, {"start": 3942.56, "end": 3946.0, "text": " Because that's the basically the image space"}, {"start": 3946.0, "end": 3952.9, "text": " Dimensionality we grab the settings such as top k temperature is super conditioning vector"}, {"start": 3953.54, "end": 3955.54, "text": " now this part is kind of"}, {"start": 3956.1, "end": 3957.06, "text": " um"}, {"start": 3957.06, "end": 3962.74, "text": " Not completely clear because it's so so hard coded. So let me try and uh, break it down"}, {"start": 3963.06, "end": 3965.7, "text": " so first we grab all of the um"}, {"start": 3966.5, "end": 3970.1, "text": " Our long the batch dimension we grab everything and that makes sense"}, {"start": 3970.1, "end": 3976.3399999999997, "text": " we take minus one because we only only only want to we only care about the last the logis from the from the"}, {"start": 3976.98, "end": 3983.7, "text": " Um from the last token because we're trying to sample from there that makes sense, but this part doesn't make sense. Why is it?"}, {"start": 3984.18, "end": 3986.18, "text": " um this number"}, {"start": 3986.18, "end": 3988.42, "text": " Which is smaller than the number here"}, {"start": 3989.38, "end": 3996.66, "text": " And my hypothesis is that well it might be that the original vq. Gan checkpoint they used"}, {"start": 3996.66, "end": 4001.54, "text": " Um had a bigger vocab compared to their target fine tuning"}, {"start": 4002.1, "end": 4006.2599999999998, "text": " After they've done the fine tuning of of the gans not completely clear"}, {"start": 4006.8999999999996, "end": 4011.06, "text": " About about this part in any case. So here is the super conditioning part"}, {"start": 4011.06, "end": 4017.7799999999997, "text": " So we grab the logits and as you can see here we grab now we grab the so these are the logis that correspond"}, {"start": 4018.18, "end": 4022.74, "text": " To the empty sequence and these are the logis that correspond to the actual caption"}, {"start": 4022.74, "end": 4028.66, "text": " And we do the logic we saw in the diffusion models as well and that's how we combine the logis to"}, {"start": 4029.2999999999997, "end": 4031.2999999999997, "text": " To form the final"}, {"start": 4031.8399999999997, "end": 4036.5, "text": " Supercondition distribution here. Okay, so I think let me just check this"}, {"start": 4037.9399999999996, "end": 4041.2999999999997, "text": " Fairly sure so this is this reverse half wings"}, {"start": 4041.8599999999997, "end": 4048.02, "text": " Twitter profile I think was the first one to have suggested this type of super conditioning for"}, {"start": 4048.02, "end": 4055.4, "text": " Uh, ultra-regressive models. So so she said you can apply a similar trick to classify our free guidance to other regressive transformers"}, {"start": 4055.4, "end": 4057.88, "text": " To sample from a synthetic supercondition distribution"}, {"start": 4058.52, "end": 4061.48, "text": " Uh blah blah blah. I trained to try this and you can see how"}, {"start": 4062.2, "end": 4065.16, "text": " well, basically you trade off the variance, uh,"}, {"start": 4065.8, "end": 4067.8, "text": " with the um"}, {"start": 4068.2, "end": 4070.12, "text": " Quality of the images"}, {"start": 4070.12, "end": 4071.96, "text": " And finally, this is what we do"}, {"start": 4072.04, "end": 4076.84, "text": " So you take the unconditional logits and you just basically add up on top of that"}, {"start": 4076.84, "end": 4082.52, "text": " This difference times the conditional scale. So this is the same expression. We are now using in in the code"}, {"start": 4082.6000000000004, "end": 4084.6000000000004, "text": " Okay, let me go back to the code here"}, {"start": 4085.6400000000003, "end": 4089.6400000000003, "text": " So that's basically if you cannot decompose this you get that same"}, {"start": 4090.1800000000003, "end": 4091.08, "text": " expression"}, {"start": 4091.08, "end": 4091.88, "text": " um"}, {"start": 4091.88, "end": 4094.6000000000004, "text": " Now we do some this is the top case sampling part"}, {"start": 4094.84, "end": 4099.96, "text": " So we do we sort and we sort in a descending fashion such that the first"}, {"start": 4100.360000000001, "end": 4104.360000000001, "text": " Logit will be the biggest one and then they start descending"}, {"start": 4104.36, "end": 4111.88, "text": " Okay, and then is kept so we take the original logits and we find basically the"}, {"start": 4112.58, "end": 4117.5599999999995, "text": " 256th biggest logits here because here's a sorted one and because we have"}, {"start": 4118.339999999999, "end": 4120.339999999999, "text": " 256 here, so we're going to grab the"}, {"start": 4121.46, "end": 4122.92, "text": " 255th"}, {"start": 4122.92, "end": 4129.48, "text": " Biggest logit and use that as a threshold and all of those logits which are bigger than that threshold are kept"}, {"start": 4129.639999999999, "end": 4131.639999999999, "text": " That's how the top case sampling works"}, {"start": 4131.64, "end": 4134.4400000000005, "text": " Then we do some uh, this is just for for"}, {"start": 4135.54, "end": 4137.46, "text": " stabilization purposes"}, {"start": 4137.46, "end": 4138.68, "text": " temperature"}, {"start": 4138.68, "end": 4144.76, "text": " Exponential and then we just multiply with uh with this mask. So again, we are just keeping those logits that are that are"}, {"start": 4145.4800000000005, "end": 4150.84, "text": " Basically big enough where big enough is defined by the top k parameter. Okay, and"}, {"start": 4151.8, "end": 4158.12, "text": " Finally, we just sample from the distribution and we get the image token. Okay, so we now have image token here"}, {"start": 4158.12, "end": 4163.4, "text": " That's the that's the token we sampled and in the next iteration of this"}, {"start": 4164.36, "end": 4165.48, "text": " loop"}, {"start": 4165.48, "end": 4172.84, "text": " Basically now we're gonna feed that token as the input and then we're gonna keep on autoregressively generating the image tokens"}, {"start": 4172.84, "end": 4176.36, "text": " So now i'm gonna try and skip all of this"}, {"start": 4176.76, "end": 4182.84, "text": " So i'm gonna go and disable all breakpoints and i'm gonna enable only this one here"}, {"start": 4182.84, "end": 4188.92, "text": " I'm gonna hit f5 and i'm gonna let it run until we generate all of the 256"}, {"start": 4189.400000000001, "end": 4191.400000000001, "text": " image tokens basically"}, {"start": 4192.04, "end": 4193.56, "text": " Okay, here we are"}, {"start": 4193.56, "end": 4195.56, "text": " uh now let's enter the"}, {"start": 4196.52, "end": 4198.52, "text": " Vq.GAN decoder"}, {"start": 4198.76, "end": 4203.8, "text": " As you can see here, we only pass we ignore the first token in the image tokens because that was the BOS"}, {"start": 4203.88, "end": 4209.96, "text": " So that was the beginning of sentence token. So we ignore it. So let's jump here. I'm gonna"}, {"start": 4209.96, "end": 4213.02, "text": " uh enable all of the breakpoints"}, {"start": 4214.06, "end": 4216.54, "text": " Let's see where I have a breakpoint here. Yes, I do"}, {"start": 4217.18, "end": 4219.18, "text": " So let's continue doing this"}, {"start": 4219.74, "end": 4221.74, "text": " Okay, here we are"}, {"start": 4222.22, "end": 4227.5, "text": " We delete the decoder that's gonna hopefully release some memory and empty we empty the cache"}, {"start": 4228.3, "end": 4231.1, "text": " So here finally we have this tip in memory"}, {"start": 4231.1, "end": 4237.18, "text": " This is what i'm used to and i'm not sure why that didn't happen previously when we were deleting the encoder"}, {"start": 4237.18, "end": 4244.780000000001, "text": " Um, yeah, that was weird in any case now we can see it here. Okay, so we now initialize the um"}, {"start": 4245.400000000001, "end": 4247.58, "text": " Detokenizer or the Vq.GAN decoder"}, {"start": 4248.700000000001, "end": 4253.26, "text": " So let's see what I have. Yeah, I have a breakpoint there again similar structure as before"}, {"start": 4253.740000000001, "end": 4258.860000000001, "text": " Uh, we create the tokenizer object and then we just load the parameters. So let's see"}, {"start": 4259.5, "end": 4261.26, "text": " how it looks like"}, {"start": 4261.26, "end": 4265.26, "text": " Again, we pass it the vocab count and the embedding count"}, {"start": 4265.26, "end": 4270.860000000001, "text": " So it's gonna have 256 tokens and this is the vocab count which corresponds"}, {"start": 4271.66, "end": 4273.66, "text": " To the this is the same"}, {"start": 4274.54, "end": 4280.7, "text": " Size as what the bar decoder had right? Okay, we form the embedding table"}, {"start": 4281.58, "end": 4286.780000000001, "text": " Uh, just some processing layers and decoder. I'm gonna ignore how the architecture looks like"}, {"start": 4286.780000000001, "end": 4292.62, "text": " It's basically a bunch of like up sampling layers and attention blocks and resonant blocks"}, {"start": 4292.62, "end": 4295.099999999999, "text": " Nothing super, um"}, {"start": 4295.66, "end": 4297.26, "text": " I guess"}, {"start": 4297.26, "end": 4301.9, "text": " Informative so we just have resonant blocks some attention blocks which literally do attention"}, {"start": 4302.38, "end": 4303.98, "text": " uh treating"}, {"start": 4303.98, "end": 4306.3, "text": " image tokens as as well"}, {"start": 4307.74, "end": 4311.82, "text": " Treating images as as sequences and thus we can just apply attention"}, {"start": 4312.38, "end": 4314.38, "text": " Um, just some up sampling"}, {"start": 4315.099999999999, "end": 4319.0199999999995, "text": " As I said nothing worth digging deeper into to be honest"}, {"start": 4319.02, "end": 4324.860000000001, "text": " Uh, you can go and check it out at your own pace if you want. Okay, so we now load the parameters"}, {"start": 4325.660000000001, "end": 4330.46, "text": " And we push the the tokenizer that we can again on the gpu"}, {"start": 4331.02, "end": 4335.26, "text": " And now we do the forward pass through the uh, this is the interesting part"}, {"start": 4336.06, "end": 4342.22, "text": " Not that complex but still interesting. So we have z that these are 256 tokens. So is that"}, {"start": 4343.1, "end": 4345.1, "text": " Shape is 256"}, {"start": 4345.1, "end": 4353.58, "text": " 26 again, i'm not sure why they're doing the clamping because we we we are certain that this will be inside of these boundaries already"}, {"start": 4353.820000000001, "end": 4355.820000000001, "text": " So there is some weird thing happening there"}, {"start": 4356.22, "end": 4362.620000000001, "text": " Uh, that's a consequence of that cropping we saw in the when the with the logits so that that might explain it"}, {"start": 4363.820000000001, "end": 4369.18, "text": " Okay, grid size is one because uh, as I said, we're generating just a single image and now there is a difference whether"}, {"start": 4370.46, "end": 4371.5, "text": " um"}, {"start": 4371.5, "end": 4380.54, "text": " Well, basically the the shape of this, um, how do we feed the the the image tokens into the actual convolutional decoder part?"}, {"start": 4380.78, "end": 4384.14, "text": " And we're going to enter this branch because the seamless is set to to false"}, {"start": 4384.62, "end": 4391.74, "text": " Uh, and we just embed using the uh embedding table embed our tokens and I assume this is the same"}, {"start": 4392.62, "end": 4398.14, "text": " Embedding table as the one that we had in the bart decoder. That's that should be shared weights"}, {"start": 4399.02, "end": 4400.62, "text": " and now we just"}, {"start": 4400.62, "end": 4403.099999999999, "text": " Do this view thingy and we end up with"}, {"start": 4404.2, "end": 4406.2, "text": " 16 16 256"}, {"start": 4406.78, "end": 4414.54, "text": " And that's precisely the latent space volume that we saw in the papers that we're going to now feed into dvq again"}, {"start": 4414.92, "end": 4417.74, "text": " uh convolutional, uh decoder part, okay"}, {"start": 4418.54, "end": 4421.099999999999, "text": " Some permutation some processing with the cnn"}, {"start": 4422.3, "end": 4425.58, "text": " Basically, this is a like a single, uh channel cnn"}, {"start": 4425.58, "end": 4432.0599999999995, "text": " Sorry single, uh, the spatial extent of the cnn is one times one and then we just up sample"}, {"start": 4432.54, "end": 4435.98, "text": " This this image and we end up finally with"}, {"start": 4437.18, "end": 4439.18, "text": " uh z of shape"}, {"start": 4440.38, "end": 4445.5, "text": " Uh 256 256 and three channels and that's our image and now we do some clipping"}, {"start": 4446.04, "end": 4449.5, "text": " Uh renormalization back into the 0255 range"}, {"start": 4450.0599999999995, "end": 4451.5, "text": " And that's it"}, {"start": 4451.5, "end": 4457.58, "text": " Some again manipulations on the shape and let's see what we whoops. Let's see what we end up with here"}, {"start": 4458.3, "end": 4463.5, "text": " So 256 256 3 so that's our rgb image and that's it guys"}, {"start": 4464.14, "end": 4468.06, "text": " basically, we now uh push the image we convert it into uh"}, {"start": 4468.7, "end": 4473.5, "text": " Unsigned integer, uh 8-bit format. We push it onto cpu from the kuda"}, {"start": 4474.22, "end": 4475.98, "text": " and we"}, {"start": 4475.98, "end": 4479.9, "text": " Basically convert it into non-py and now we basically yield back"}, {"start": 4479.9, "end": 4485.82, "text": " Uh pil image pill image and that's it and then we just save the image and that's it"}, {"start": 4485.82, "end": 4490.62, "text": " Now there is also this print ascii from image function. That's very neat actually"}, {"start": 4490.62, "end": 4496.139999999999, "text": " So again ascii from image makes sense when you're running this from a console. It's just gonna print the image"}, {"start": 4497.58, "end": 4501.179999999999, "text": " Um in the ascii format, I guess. Okay, that's it"}, {"start": 4501.66, "end": 4506.139999999999, "text": " Uh, let me uh step over this and we are pretty much done"}, {"start": 4506.14, "end": 4509.660000000001, "text": " That's the end of the program. Let me now show you the generated image"}, {"start": 4511.740000000001, "end": 4513.740000000001, "text": " I should have it somewhere here"}, {"start": 4514.22, "end": 4516.22, "text": " Let me open it up"}, {"start": 4516.22, "end": 4524.06, "text": " And here it is. So here is the uh, what was our caption? Our caption was um, a photo of a funny dog"}, {"start": 4524.62, "end": 4527.900000000001, "text": " And here is a dog. I'm not sure whether it's funny, but"}, {"start": 4528.54, "end": 4535.1, "text": " Uh, basically, that's it. Let me try a different caption and i'll get back to you once we once once it's done"}, {"start": 4535.1, "end": 4539.5, "text": " Maybe like something a bit more complex. This might break the dali mega"}, {"start": 4540.06, "end": 4545.660000000001, "text": " But it's worth a shot a photo of a funny dog. Uh, maybe riding a bicycle"}, {"start": 4546.860000000001, "end": 4548.46, "text": " Bicycle"}, {"start": 4548.46, "end": 4550.46, "text": " Let me run this one"}, {"start": 4550.700000000001, "end": 4554.780000000001, "text": " And let's see what we get. Okay guys, here is the actual image"}, {"start": 4554.780000000001, "end": 4556.3, "text": " It's kind of weird"}, {"start": 4556.3, "end": 4561.660000000001, "text": " You can kind of see it's trying to merge the concept of a dog with a concept of a bicycle"}, {"start": 4561.66, "end": 4569.18, "text": " But it's still not that expressive and big to achieve that so scale is vital for these types of models"}, {"start": 4569.74, "end": 4574.78, "text": " And that's it. Uh, do let me know whether you found this video useful as as usually"}, {"start": 4574.78, "end": 4592.139999999999, "text": " Uh, leave down in the comments below if you have any feedback for me, uh, and until next time. Bye. Bye"}]