OpenAI CLIP | Machine Learning Coding Series
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

Kicking off a series of videos where I'll be going through the actual code of many of the papers I've covered over the last few years!

In this video I do a code walkthrough of OpenAI's CLIP model from the "Learning Transferable Visual Models From Natural Language Supervision" paper.

Let me know what you'd like me to cover next!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ GitHub: https://github.com/openai/CLIP
✅ Paper: https://arxiv.org/abs/2103.00020

Learn about Byte-Pair Encoding:
✅ https://en.wikipedia.org/wiki/Byte_pair_encoding
✅ https://leimao.github.io/blog/Byte-Pair-Encoding/
✅ Video: https://www.youtube.com/watch?v=tOMjTCO0htA&ab_channel=FromLanguagestoInformation
✅ https://www.youtube.com/watch?v=zjaRNfvNMTs&ab_channel=AbhishekThakur

Unicode:
✅ https://dmitripavlutin.com/what-every-javascript-developer-should-know-about-unicode/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00:00 Intro
00:02:00 High level overview: Interacting with CLIP
00:26:11 High level overview: Prompt engineering for ImageNet
00:40:25 Deep dive starts: vocabulary and byte-pair encoding
00:49:00 Vision Transformer & Text Transformer explained
01:02:00 Tokenization walkthrough
01:09:25 Encoding the image
01:15:15 Encoding the text
01:23:15 Learning a linear probe
01:27:00 Tokenization of the (brain emoji)
01:29:56 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Kevin Stone
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#CLIP #contrastive #codewalkthrough
 What's cracking guys? In this video I'm doing something a bit different. So I'll be kicking off this series of videos where I'll be going through various, through the code of various papers that I've covered over the last months. And so I'll be starting with OpenAI's clip. So basically I cloned this repository OpenAI's clip implementation, created a conda environment locally and I'll be just debugging and going through VS code and showing you how how clip works. But if you don't want to go through the hustle of creating the environment, you can just use their basically like colab here. The environment is already set up, you can kind of copy paste some of the code from here into the colab and that's how you can kind of avoid having to go through the conda environment creation, which I'll not be explaining how to do in this video. So having said that, let's jump into the actual code. So first I'll go through the Jupyter notebooks to give you a high level understanding of how clip works. And then we'll literally step through the code and see how the details of clip are implemented. So vision transformers, transformers, contrastive loss, all of that. So no, actually the training details are not implemented in the OpenAI's like open source code. There is a separate repository that does that. So I'll be skipping that part, but it's probably less relevant for what I'm trying to explain you here. And that's how clip works in general. OK, so I did create a video covering clips. So do check that one out if you want to get the theoretical understanding. I'll not be combining paper and code right now. I'll just be focusing on code. So if you haven't watched that one, go ahead and watch it. I'll link it somewhere here. So yeah, do check it out. Anyways, let's start here. So let's basically import a couple of things here. No, thanks. VS code is fairly annoying with these prompts, but it's probably the nicest environment to be debugging the code, both visually and just like there's so many cool features, which you'll hopefully see over this video. So I imported necessary dependencies, numpy, torch, etc. So let's see which models we have available. So clip basically, again, how clip is constructed. You have the visual like pathway, you have the textual pathway, so to speak. So the visual one will encode the image into a vector. The textual side will basically take a caption and embed it into a vector. And basically then you just want to see the similarity between the dot product between normalized embeddings. That's clip on a high level. So as you can see, they've been using various different visual baselines. So here are some modified versions of Fresnets. They also use VAT. We'll be using VAT in this Jupyter notebook. So I can also kind of go through the definition and show you how this looks. So basically they've defined this models dictionary. It's a key value dictionary, obviously. So we have the key here and we have the associated URL. So that's where the model is stored. So here and basically what this function shows us is all of the available, like pre-trained models, basically. Okay, let's get back here. Let me now run this cell. So first what it does is again, it's going to load this VAT. So the big version, the patches will be 32 by 32. And it's going to basically give us the model and this pre-processing like a structure that contains the PyTorch transforms that we'll be using to transform the images like cropping and stuff. We'll see that in a second. So let's run this cell and let's see what's going on here. Okay, so it's going to count the number of parameters in the model. We can see here we have 151 millions of parameters in this VAT. The input resolution that the VAT is expecting is 224 by 224 and they've artificially set this context length to 77. And so that means the number of tokens that will go into the, like text encoding pathway will be 77 tokens always. So that's kind of fixed. The woke up size is, as you can see here, 49,000 roughly. And they are using the BPE encoding. So that's the byte pairing coding. I'll link some of the resources when it comes to BPE, which you can check out because I will not get into the details. Otherwise, I could just create a video explaining how that thing works. Okay, so this is some metadata about the model. This is, okay, let me just run this to see which transformation we're using. So we have resize to 224 because again, images will be of arbitrary aspect ratios and arbitrary resolutions. We want to make it basically of the format that the VAT is expecting and that's 224 by 224. So we're going to resize the image to 224, then do the central crop, then convert the image into RGB, convert it into tensor, and then do the normalization. So these are just, I assume this is just ImageNet, like mean and standard deviations. You literally go through the ImageNet training set and you collect these statistics. You find the mean for all of the images, you find the standard deviation, and then you use that to center the input images before feeding them into the VAT portion of the clip. Okay, so let's continue. So we are first going to tokenize the sentence, hello world, and we'll later see how this tokenization works. So as I said, it's using BPE in the background. If we run this, we expect to get like a list of integers basically. So you can see here, here is the result. So this is the BPE magic we somehow get. So this is going to be the start of sentence token, the end of sentence token, and these are the subtokens that BPE kind of broke this sentence down into these. And it has all of these because, as I said, it's going to have 77 tokens because that's the constraint they artificially are using. So let me just kind of maybe store this into a temporary variable and then print the shape of this thing. It's going to be 7, yeah, as you can see, it's 77. Okay, so let's get to this part here. What they do here is they create this dictionary of descriptions. So this is going to be a name of the image and this is going to be associated caption. So we'll be using these names to just fetch the images from, I think, scikit-learn or something. You'll see that in a second. So let me run this cell. Let's form the dictionary and now let's get here. Okay, so what's going on here? So as you can see, we're going to iterate through this scikit-image. So scikit-image package has certain images that come pre-installed with it. So we're going to iterate through those file names. And if the file name ends with PNG or JPEG, meaning if the file name is an image, we're going to enter this loop. Then we're going to just grab the actual name here. And if it's not in the description, so we're going to only extract these eight images here. If it's not in the description, we'll just kind of skip it. If it's in the description, then we're going to grab the path of that image, open it up as a pillow image and convert it into RGB. Then we're going to just show all of those on a nice figure here. I will not get into the Matplotlib details here. And we're just going to store the pill images as well as the processed images and associated captions. So this text is going to have so descriptions is here. We're going to have the keywords here. So basically we'll fetch the associated captions. Let me run this enough rambling. So you can see here the images we got from the scikit image package. And well, unfortunately, let me see whether I can change the color here. Yeah, I'm not sure whether you see the associated caption, but like I can read one for you, basically a portrait of an astronaut with the American flag. This is this one. And then a person looking at a camera on a tripod, blah, blah, blah. So we now have the data set as clip expects. So it's a small eight image data set image and associated caption. And now let's see how clip is going to use these. So let's continue here. First of all, we're going to stack the processed images. So remember images are the ones where we've applied the processing. So the cropping and conversion to tensor normalization, all of that goodies. Now we're going to stack it. We're going to convert it into into into like a big tensor, which then we're going to move to to CUDA. So that means basically pushing this tensor onto your GPU. So let me run this. And let me now create another cell here. Whoops. That's not what I wanted to do. Let me just basically create a cell. Okay, mixing up shortcuts from Colab. So I'll just have to manually add a code cell here. What I want to show you is just the shape. So always understanding the shapes of your object is super important. Let me run this. So 8, 3, 2, 24, 2, 24. So that's your image. And because we have eight of those, we stacked eight of those. We have like a torch tensor like this. And similarly, Remy do the same thing for text tokens. So what we've done here is we've gone through the captions. Okay, so that's the text. And then we just create this artificial prompt. So this is and then we append the captions, the ones we had here. So this is a page of text about segmentation, etc. etc. And then we're going to tokenize that which means we'll get those 77 context tokens. So let me just run this again. Let's see what we get. So we have 8, 77. So 8 because we have 8 captions and 77 because that's the basically context length that the transformer textual pathway in the clip is expecting. Okay. Finally, what we're going to do is we're going to patch. We're going to pass the image tensor to the clip model. So model is just clip and we're going to call this encode image function. We'll see later what exactly does but basically just a forward prop of the images through your VAT and as the output will get I guess something probably the shape will be something like 8 and then I think it's either 512 or 756 or something like that. We're going to see that in a second. And here what we're expecting is similarly will pass these text tokens and I expect we'll have something like well, I guess 8 512 because we need to be able to compare those do dot product between the textual and visual embeddings. So let me run this and then let me test the hypothesis whether I got the shapes right from like by the heart. Okay, let me add an additional code cell here and let's just print the shapes of these objects. So shape will be 8 512. That's correct. And then this must be the same. So let me run this. So 8 512. Okay. So again, pretty simple here because this is the transformer will be just taking the whatever the feature vector is above the end of sentence token. So we just fetch that vector and that's what we're going to be using as a embedding for that particular caption. Again, we'll see the details a bit later. That's it. Now the next steps are normalization. So we'll normalize all of these embedding vectors. So visual and textual embedding vectors and then we're going to do matrix multiply which is going to do dot products between all of these combinations of visual and textual embeddings and that's going to give us the similarities between those images and all of these captions. So let me run this first like this and then I'm going to analyze what we are getting out of here. So similarity should be 8 by 8. So I'm expecting shape 8 by 8 because we want to know for each image. What's the similarity score for that image and all of the other captions. So let's see whether that's correct. So if I print similarity shape here and if I run this it's 8 by 8 as expected. Okay, so what's this thing about? The best thing to do is to just kind of because we are we have the interactivity here in Jupyter so why not use that fact. So let me just print this and see what we get here. Basically what this is going to do is find the I assume this is going to do L2 norm. So it's going to find the L2 norm for each of the embedding vectors and then just divide as you can see here, we're just going to divide the vectors by their L2. So basically if we have a vector X what we do is we convert it into X divided by L2 norm. How can I denote that? Maybe like this L2 norm of that X and that's just your regularization that normalization that that makes it makes it so that the norm will now be 1. So all of your vectors will be of norm 1. Okay, let's run this cell and see what we get. Well because I've already done the because I've already done the I already ran the cell. I'm getting basically all of them are already 1. So not that interesting. So let me just rerun this for a second. So let me rerun the image features and text features get the original ones and now before I run this cell, let me see what we get here. So you can see here the associated L2 norms for each of the eight visual embedding vectors and we just divide by those L2 norms and we normalize the vectors by doing that. We do the similar thing for basically for for the text. So let me see if I can go to definition and see whether okay, no definition found for this. So I guess we're going to use just some good old Google search. So let's see what's going on here. So norm by torch. I think it should be like basically okay for a binius norm. Let me see what's the default here. So the order of norm default is for binius norm, which is just a fancy way of saying L2 for for matrix. It's going to do the square root and then basically squared of all of the elements and then you just do the summation. So yeah, we can quickly verify that hypothesis. Just let's create some artificial vector. So let's let me maybe create something like this. So we have like a dummy dummy tensor. My pi torch skills are bit rusty. So hopefully I get everything correctly here. So torch tensor or just zeros. Can we do zeros? Well that will not be that interesting. Let me just think what we can do here. Okay, so I'm gonna just do once and then I'm gonna take like size. Let's say we just have a vector of size 2. So there's going to be one one if it's L2 norm we expect square root of 2 which is roughly 1.41. So if I now pass dummy tensor here and let's find the norm probably dimension and all of this is not that important. But yeah, you see basically we have L2 here and that's pretty much it. Now this thing is probably interesting. We convert that at tensors. We just push them to CPU, convert them to non-pi and then we do matrix multiplication. So why does this make sense? And we have to transpose here. So why does that make sense? Well because the following basically if we think about it text features is what 8 512 and then we do matrix multiplication with a matrix that's 512 8 because we do the transpose here operation and that's gonna obviously give us if you do your matrix multiplication 8 by 8. So what you're doing is you're taking a vector like you're taking like a textual embedding and by multiplying what you're doing is you're doing the products because you're multiplying the row with the column and the column is the visual embedding vector. So if you do all of that you're basically getting the similarities between this particular text vector with all of these image vectors. Hopefully that makes sense. We can finally print we can print the similarity and do let me know whether you find this type of explanation useful. Any feedback is super useful because this is the first time I'm doing something like this. So as you can see all of these are similarities so they are not normalized which means if I were to do a summation will this work x is minus one if I print this. So as you can see it's not one because we haven't applied softmax or anything. So that's what we get here. Okay let's continue. Let's see what we have here. So descriptions that's going to be our dictionaries we'll get I guess eight here which is going to plot something. Let's see what we're doing here. So original images we're going to plot the image and then we are going to plot the scores. Okay let's just run the cell. It's going to be easier to understand what's going on once we plotted. So as you can see so what they've what they've done here is basically they have they plot the images here they plot the captions here and then they just plot this similarity matrix. So we are basically visualizing the similarity matrix that I just printed a couple of cells before and you can see that as expected because this is a train clip model the biggest similarity is between the image and its correct caption and that's why we have this diagonal here. That's pretty much it. So yeah I guess I'll just ignore the details of this code because it's just idiosyncrasies of Matplotlib. What I'm going to focus on is on the semantics and on what clip model is doing. So this is it. Let's see what else. So now we are going to basically load a cipher hundred data set. So that's going to probably take a while. Oh okay I already have it cached so that's why it was so fast. So we have a cipher hundred data set which is just your image data set that has a hundred classes hence the suffix hundred here. And so what we're going to do is we're going to iterate through the classes so through all of those classes and we are going to form a caption out of that class. So if you read the paper this is what I've been doing this is how you can do the zero shot image classification. So we form a caption like this. This is a photo of and then goes a label for example horse dog or whatnot. So I'm not sure what the cipher 10 classes are. We can print that. I can kind of add a cell here and then print cipher 10. Let's see what's going on here. If I run this. Okay. Apple aquarium fish baby bear beaver bad blah blah blah. Nice. So we're going to as I said form captions from the classes like this and then do the tokenization. So again the shape of this should be I guess hundred seventy seven because we have hundred captions hundred labels and hundred captions and seventy seven is the expected context length. So let me run this and let me print the text tokens shape just to be sure. So yeah as expected. And now what we're going to do is the same thing as before past the textual these these these like tensors. Okay this is not a tensor. This is just as a wait. What's the what's the type of this. It's interesting. What. Okay. It says here it's a torch tensor. We're going to pass that we're going to encode a text. So we are going to end up here with I assume hundred five twelve because all of those captions are going to be embedded into a vector. So we get hundred five twelve. Then we do the normalization as before. So that means all of the norms of those embedding vectors going to be equal to one. Whereas by norm I mean L2 norm in particular. And then what we do is again image features text feature. So we're not teaching the images. So we're still using those eight images from before. And we're just trying to find the most similar class from Cypher hundred for those images. And again this thing is just going to calculate pair by similarity score between each of the images each of the eight images and each of the hundred captions. We have to multiply by 100 just for numerical stability. And then we apply softmax. So that that means that now we'll basically have a sum sum would be equal to one for a particular image. And so the sum of the similarity scores for this particular image and all of the captions needs to be equal to one. OK. So let me run this. And finally what we do is once we find those. So let me maybe let me print this. So if I do this if I take the text props and they print those let's first bring the shape the shape should be what the shape should be eight times hundred I guess. Yeah. And if I print the actual text props it's going to be just not that intelligible because bunch of small numbers. So this here this vector here is basically the similarity score for the first image and all of the hundred classes of the Cypher 10 Cypher 100. And now we're trying to find the maximum because that's the high similarity. And we find that by doing the top case. So we find the five highest similarity scores for each of the images. And that's what we get here. So let me now print the top props and the top labels. So let me just bring the top labels actually. So we're going to get. So these are the indices of the classes from Cypher 100. They have the highest similarity score for the associated image. So hopefully that makes sense. OK. So having done that let me run this final cell which is going to do the following. So we are going to go through the original images. We're going to plot that image. We're going to take the probability scores. OK. And we're just going to do bar plots with those scores and with the associated images. So again let me run this because this is too much detail to analyze in a video. But basically this is what happens. We just plot the images and the probabilities as well as the labels we just found in the cell above. So this is what this magic here does. You can see that this image is the closest class in Cypher 100 is woman. The closest class for this picture here is man. The closest class for the cat images sweet pepper. OK. We have a false like positive there. And now we have cop here. OK. That's fairly decent motorcycle cattle. I guess that's the closest thing they have. Let me see where they have a horse. So I guess the horse would be the closest label. I'm going to do the following. So I'm going to print his horse in Cypher 10 classes. And it's not. So we don't have a horse label. So that means that this is probably the closest label we got to hear a motorcycle make sense again bed. So this is completely gibberish unless there is some word better something inside of this image. Let me zoom in. Whoops. Let us first determine my blah blah blah. No this is probably just false positive. And finally here we have a rocket which is also a good class. That's it. That's the first Jupiter notebook. Do let me know how you're liking this video so far. Comment down below if you find it useful. OK now let's go to the second Jupiter notebook. This one we're going to go fast through it because it's much simpler I assume. Let me run the cell. OK let me again available models. We've seen that we're just going to load the VAT big 32 by 32. That size is that's that model. So that's the same model we used in the previous Jupiter again 151 million 224 2424 context length is 77 for the transformer woke up sizes like this. So here what I've done is the following. So they have a thousand image net classes with a small caveat. So they've actually modified some of the classes such that clip has easier time basically recognizing those images. So I think they mentioned it here. Let me kind of read it through. So subset of these class names are modified from the default image and class names source from any athletes image and simple labels. These edits were made by trial and error and concentrated on the lowest performing classes according to top one and top five accuracy on the image net training set for the for these variations of the models. So these weeks improved up one by one point five percent on the ATB 32 over using the default class names Alec which is the author of the Alec Redford assume from the clip paper got bored somewhere along the way as gain started to diminish and never finished updating tweaking the list. He also didn't revisit this with the better performing models or any of the VAT models he thinks it's likely in other zero point five percent to one percent up one could be gained from further work here. It'd be interesting to more rigorously study understand this. So this is super interesting. So this means that by tweaking the classes and then embedding them into those templates that the caption templates we saw. So let me show you what I mean here. So this is the template we are using. So by just tweaking that we are forming better text embeddings and then consequently those make for a better classifiers your shot classifier on top of our image embedding vectors. So let me quickly explain what I mean by that before proceeding here. So imagine we call up we've done here we basically took hundred classes we embedded them into captions we have hundred like strings now we formed embedding vectors for each of the captions and now we have hundred hundred seventy. So we basically have this we have sorry we have after embedding we end up with hundred five twelve. So let me explain why this is so cool. I'm going to open up my one note for a second here and let me just draw you a couple of things here. OK. So let's see what we've done. So we have we embedded our image here into an embedding vector that's like five hundred twelve dimensions we have five hundred and twelve dimensions here. Then we saw what we've done with the with the with the classes we've we formed basically a collection of these embedding vectors. So this is hundred we have hundred and we have five twelve here as well. So now because of the way how we calculate the similarity scores we basically multiply with your dot product between this one and this one with your dot product between the image vector again and the next and the next embedding vector the next representation of a class and the next one next one. And what do you do if you think about it what do you do when you do dot products with these weights here. This represents the weights of ad hoc MLP of a zero shot MLP that's used as a prediction head to predict the classes. So again if you do dot product between this and this what you're effectively doing is the following. So you're doing this and you form one weight and then you take this one. So let me change the color here. Then you do this you take this vector you do dot product between this and this and you effectively do the following. So you just I'm kind of assuming this has four dimensions but it has five twelve so just bear with me here. So you do this and then you do the same thing for the third vector. So you're going to do dot product which is going to do this. And as you can see this is basically the weights of this particular MLP here. So you ad hoc construct the prediction head and by doing that you can take arbitrary data set you can literally take arbitrary data set here. Take the classes form those captions embed those captions through just kind of do a forward pass through the transformer. You get additional MLP. So now you have MLP so this this is going to be a set of end classes with dimensionality of five twelve and you can basically use this to now classify this image into end classes. So that's super cool and that's what a white clip is such an awesome model because of all of these zero shot capabilities. So having said that let's go back to the code. Let me go back to the where we stop here. So now you can appreciate why why tweaking this can improve that prediction had weights and thus give us higher top one and top five accuracy on arbitrary data set data set. So that's why they've been doing that. OK. So they give an example of what the tweaks some of the tweaks they've done. So some of the examples beyond the crane crane so crane is going to be so one crane can be construction crane. The other one is actually like a bird so bird crane. So that's why they did those modifications and then they've done stuff like mail convert that to fingernail or like metal nail depending. Yes I guess. OK clip interprets nail as fingernail but we actually want it to interpret it as a metal nail and that's why they explicitly change the class to metal nail. And similarly image net kite classes refer to the bird of prey not the flying toy. So we change kite to kite bird of prey etc etc. OK. So that's that's it. Now they have instead of using that single template we've been using here this is a photo of they also experimented with trying different templates. You see here a bad photo of a photo of many a sculpture of. And so there is a creative set of these here. I think there is like literally at least 100 of them. Let me let me see if you run this. So we have 80 templates. So they devised 80 of these templates. And now they're going to do some type of in sampling to get better results for for image net. Let's see how how that works. Let's quickly go through the Russian rationale behind doing this again. A similar intuition guided trial and error based on the image and training set was used for templates. This list is pretty haphazard and was gradually made expanded over the course of about a year of the project and was revisited tweak every few months. A surprising weird thing was adding templates intended to help image net our performance specifying different possible renditions of an object improved standard image net accuracy to after the 80 templates were locked for the paper we ran sequential forward selection over the list of 80 templates. The search terminated after in sampling seven templates and selected them in the order below. So combining these seven templates gave them the best results. So then they say speculating we think it's interesting to see different scales large and small. That's why they have templates like this working very nicely a photo of the large a difficult view. That's why they have a bad photo of the and abstract versions origami video game art were all selected for. But we haven't studied this in any detail. This subset performs a bit better than the full 80 ensemble reported in the paper especially for the smaller models. So what I mean by ensemble let me let me kind of break that down. So we're going to see that now. So let me load the image net data set here. I think it's cash so shouldn't take too much. So we're going to have the image net and here is what what they are doing. So basically they form a zero shot classifier using the following technique. So we pass all of the image and templates and we pass the image and classes. And so let's see what they do. So they go through all of the classes and then they go through all of the templates. So that means that a single class will this time be embedded using not a single template but 80 of those and then they tokenize those the encoders which means we'll hear after doing the encoding we'll end up with what we'll end up with because we have 80 templates we'll end up with 80 77 right. I think let me think here. So we're doing the tokenization no sorry. So here we'll end up with 80 77. So tokenizer is going to give us this. We can kind of print those shapes later on and see whether that's true. And now after doing the embeddings they get 80 512 here. And now the smart part is after doing the normalization which is which is like standard thing to do they form the final representation for that class by averaging across all of these embeddings for all of these different templates. And then additionally they divide by the by the norm of the class embedding to make sure it's again has L2 norm equals to one. And then they use. So now they just stack those zero shot weights. So this is going to be the zero shot classifier I explained in the one note a couple of minutes ago and that's it. So this is going to be our classifier zero shot weights. So if I run this it might take a while but it's such a neat idea. Basically in sampling is a very very famous technique like you people use that to trade off between higher performance but like somewhat obviously longer inference more memory etc. But just by kind of combining the majority like logic just doing this majority voting type of logic we get higher quality embeddings. And and yeah. OK this run now we can print what we get here again we're expecting I guess a shape of 1000 512 because we have 1000 image and classes we have embedding vectors of size 512. So let me just print the zero shot weights shape here. So if I print whoops if I print the shape here let me see whether my hypothesis was correct. So 512 1000 OK it's just not it's stacked in a different way so that's why we get this shape shape. OK we can maybe quickly just print these shapes here so text shape and I'm going to add print class embeddings here the shape as well. And then we're just going to break out of this whole logic so that this is run quickly. So whoops I'm getting OK because of the break statement everything crashed but like we are verifying that this makes sense so 80 77 80 512. Let me now remove these statements and I'll have to rerun this to get the actual zero shot weights. So after that while that's running let me walk you through this part here. So we have the accuracy I will not go through how the accuracy is calculated you can go through that piece of code on your own. So what we're going to do is we're going to here go through the image net batches of image net images. We're going to push those to CUDA the associated labels as well. So when I say to CUDA I mean to a GPU if you have one. We are going to then encode the images so there's going to be 32 512 so 32 because we specified the batch size somewhere here so batch size going to be 32 and then 512 because as I said the visual embedding vector is going to be a 512 dimensionality. Then we just do the regular normalization here and then we find the similarity. So basically this is again this is your prediction head that was ad hoc constructed ad hoc. So this is the zero shot classifier head and we get the logits so and then after we pass the logits and the target we'll see how good do we perform here when it comes to accuracy top one and top five accuracy. So let me run this cell and let me run this one. This might take a while so maybe I'll just kind of skip to the end. Okay so this thing ran and we can see top one accuracy 55 top five 83.4 by just having by just doing this zero shot classification. Again because we're passing logits just a minor detail obviously we'll have some soft max here or something. Let me let me see how this logic here works. So we just grab the outputs we find. Okay we don't even have to do the soft max we just need to find the top five or whatever the case is five in our case top five logits which is going to correlate with the top five highest probabilities if we were to do soft max which we will not because that would be redundant and then just your usual logic. It's going to convolute it because it's a one liner but it just kind of calculates the top one and top five accuracy. Okay that's it. Do let me know whether you found this Jupyter walkthrough useful and now I'm going to dig into going through some details of how everything works behind the scenes. Hopefully you'll find this interesting as well. So let's start with this playground. So I just created a simple playground.py file then I just copy pasted pretty much some of the examples here from the readme file of their official repository and now we're going to walk through this particular example and see the details behind how the imaging code tokenization all of that works. So without further ado let me run the debugger here. So let me maybe zoom in a little bit more here hopefully you can see everything nicely. So what's going on is that this tokenizer is just being created initially before we even hit this first line in simple. We have a tokenizer somewhere here. Let me just find it. It's kind of defined I think in model or somewhere. The object is independently defined. Let me find it. Oh my god. Oh my god. Okay tokenizer. Yeah here. So here is the construct. So basically the constructor is called before we even hit the first line of our function in the playground. So again let me just kind of show you. I have the main function I'm calling the simple here and simple is this one. So you'd expect this to be the first line that we hit but that's not the case. We instead hit this one because as I already explained why that's going on. Okay so tokenizer. What's going on? So I'll have to kind of skim through some of the details because it gets fairly complicated and I also don't understand every single detail as well to be honest. So this bytes to unicode what it does is just creates like a dictionary between 0 to 255 which are like values that a byte can have and then it maps it onto a unicode string. There is some convoluted logic here of which ranges they take etc. So that's why I'm saying I'm going to kind of skip some of these details but like we'll analyze what we get as the output. So we'll treat it like as a semi black box. So if that's even a word. Okay let me go to debug console and this is why Visual Studio Code is such an awesome environment. Let me show you what we have here. So if I print this we get as you can see here we get basically so from 0 to 255 we get the keys and then associated unicode characters. Okay that's pretty much it. Now the byte encoder is going to do the opposite. It's just going to map from the unicode character onto the byte here. So I'm just going to kind of skim over that. So F10 then we open up so there is this BPE path. So there is basically if I open it up here I think it stores somewhere in here. Let me let me let me see where it is. Just a second. Okay so here it is. So this is this file BP simple vocab 16E6 takes that. So that's basically your BPE. That's the byte pair encoding as I said I will not get into the details. So what they've done they had some data set. I'm not sure how I can track back the actual data set that was used to train this BPE. And then you do the BPE logic which is basically finding the pair of tokens with the highest frequency and then merging those two tokens and adding that token to the vocab and then just repeating those merges until we have the final vocabulary that that's the BPE vocab. So that's kind of already done for us. That's why we can just read this and then basically decode. Consider thinking like decode as a UTF-8 and then just split on the new line characters. And then I'll show you in a second what this merges are doing. So the merges are doing the following. So let me first close this so we have more space. So merges is a list. So I'm going to just show you the first five for example. So you can see here. So here these are the merges that were happening. So that means this this this whoops why is this okay. So this means this particular these two tokens were the most frequent tokens in that corpus of text at some point of time during the BPE algorithm. So if you don't know what BPE is I strongly suggest you just kind of find some article. I'm going to link it down in the YouTube description actually in the video description and you can check it out. And so this will make sense my explanations. So so yeah we have these merges now and I'm going to go over this. And finally we start forming the vocab. So first of all we're going to grab the the the the Unicode strings that we formed up there. So let me go F10. So that means the vocab size will be 256. So if I print the vocab size here it's going to be 256. Now the following again this is BPE idosyncrasy. So basically we have to to concat this this end of word like token. So if I click we'll just do we'll just basically form new tokens in our vocab by concatenating all of the existing like tokens with this particular end of word token. OK. So if I click F10 if I print the length now it's going to be 512 because we just duplicate everything and then we go through the merges and we because all of those merges basically are now a unique token in the BPE vocab. That's why we do this logic here. So we basically join. So that means we're going to join these. So we're going to have in instead of instead of so I can maybe let me see whether I can kind of step through this quickly or just do something like this. OK. So if I do this if I do F10 and let me print you the merge here directly in the debug console. So as you can see we're going to add the in to the vocab. And again that's BPE idosyncrasy. I'm going to skip over all of that. Click F5 here and just get to the next line here. OK. Let me collapse this. So bear with me. It's a bit more. I'll summarize the findings in a second. So we extend with these two special tokens. Start of text and of text. So it's like start of sentence and of sentence tokens. So if I add all that now we have a pretty big vocab. So I think it's almost 50. Yeah. 49,408. That's it. Now we form the encoder by basically zipping the those tokens from the vocab with the associated indices and we do the decoder. So that's going to basically convert from from from a token into an int and vice versa the decoder will do from int into into the token. So if I do F10 here just step over we get all of this and then they form the special variable BP ranks. It's going to take those merges. So and and and kind of store them. We'll see why that's useful a bit later. And then they have this cache again some optimization detail that the BP will need otherwise will be super slow. And finally this pattern that they're going to use to kind of parse the input text. We'll get to that in a second. OK. So as a summary we have a vocabulary formed using the BP. Lots of details. You don't need to know all of these. What is important is that we have the encoder and decoder and all of these are some other implementation details. Important ones but we can we can kind of kind of squint and ignore them. OK. Let me go to our next breakpoint. OK. So we are now in the first line of the simple method. OK. So we just do the standard stuff. We see whether we have like a GPU device and I do have to do and I do happen to have a GPU. So that's that's cool. Now we'll load the device. So now we'll have some some some break point. That's interesting. So basically before the video I put some salient break points on certain parts of the code base that I want to dig to kind of go through with you. The code base is fairly big so you can kind of do that in your own time. But I'm going to focus on the most salient parts of the code base. OK. So the first one being the construction of the clip model. So context length. That's a 77. That's a number we've been seeing all around. And that's because that's a that's a parameter that's passed during the construction of the clip model. So let's see what's going on here. So we're going to form the VAT model here. So I'll hit the VAT construction right now. So we have image net. The image resolution is 24 through to the internal vectors will be 768. We'll have 12 layers 12 heads and the embedding dimension at the output is 512. So even though the inner dimensions are a bit larger. So that's here 768. The output as you can recall is 512. We saw that in the Jupiter. So let me click F5 here and go to the vision transformer basically in it function. So we're just going to store some of these dimensions. So this is everything that vision transformer needs. This is the conv layer that that you have to this is the patchy file layer basically of your vision transformer. As you can see here the pet size is 32. So we set the kernel size to 32 and we set a stride the same exact width. And the reason is is because we want to take non overlapping patches of the image and then embed them from the initial dimension of three into 768. So let me quickly go through that in the one note maybe it will help you understand what's going on. Again what is going on is the following. So if you have an image here. This is our input image. We have three channels. It's going to be three dimensions here so this is three and then this first conv layer is doing the following. It's going to take the kernel put it here and map this whole volume into a vector into a corresponding vector that's going to be of dimension 768. Then we're going to move here. So let me change the color. We'll then move here and then embed that into additional vector here 768. And we're going to repeat the whole procedure until we hit the end of the image. So now we have here etc. etc. You do that for the whole image and you end up with what you end up with whatever the number of patches is and it's going to be 224. We divide that by what by 32. That's going to give us I think 16. So we're going to end up with 16 by 16 of these 768 vectors. So we're going to end up with like 16, 16, 768 volumes here. So let's kind of verify that hypothesis. Well we'll get to that a bit later but that's what this layer is doing. Hopefully this helped. Let's continue. So if I hit f10 we just form this scale. This is just for normalization. Basically what we do here is we form the CLS embedding. So that's the CLS token of our vision transformer. That's something that BERT is also using and many other transformer models. And we are basically going to it's going to be a 768 dimensional vector. We form the positional embeddings. It's going to be of the size. So basically 16 squared. That's the number of tokens we'll have. So that's like 256 right? And plus one because we have CLS. And then the width is 768 because we want to basically add those positional encodings on top of each of the embedding vectors. So if you're confused by the means is basically let me again go back here. So what it does is it's going to construct it's going to construct embedding table here. So we'll have an embedding table here. The dimensionality will be again 16 times 16 plus one for all of these vectors here. So let me kind of try and draw them some somehow here. So this is going to be our output volume 16 16. And this is going to be this dimensionality here 768. Okay. And we'll have corresponding embedding vectors. And now we'll basically be learning these and then adding them on top of these ones. So basically what will happen is that this vector here, this position vector here will always be added to this vector here. So to the first vector here. So we take this one and we combine it. We'll sum it up. We'll do the summation between those two. And that's going to happen in the forward pass of the VAT. Okay. Let's go back to the code. That's that. We have some layer norms. We form some transformers. And then yeah, so because transformer will also be used in this second, I dubbed it as a textual pathway. Not sure whether anyone else is using that terminology, but like let me kind of just quickly scheme over what transformer is doing. So if I click this, we'll get to the transformer logic. So what transformer does is fairly simple. We just specify the width, which is going to be also 768. We have number of layers. We just form a sequential layer. So we're going to form these residual attention blocks. And what they do is, I mean, your regular transformer logic. So let me show you what it does. We have the multi-head attention head. Wait, multi-head attention layer. And then we have the layer norm. And then we have the MLP. So we have the linear, followed by Jalu, followed by linear. And you can see this is your regular expansion ratio of four. Usually the innermost part of that MLP in that particular transformer layer is usually 4x bigger than the input part. Okay? And so then we construct the second LN layer and we optionally have attention mask in case we want to have a causal transformer model. So the vision transformer will be using non-causal transformer model because you want to attend each token, each patch needs to attend to each other patch. And in the case of a textual transformer, we'll be using a causal one. So that's why we have this attention part. And now what the forward pass of the transformer does is basically applies the attention. So sorry, this is just a particular layer of the transformer, not the whole transformer. So it does the attention and then just basically, okay, of course, layer norms, attention, residual connections, and then there is the MLP part. So that's your transformer logic here. And finally, we just kind of do a forward pass through all of the 12 of these transformer layers. And that's your transformer model. Okay? So that's a quick scheme, like a quick scheme over what's going on there. Anyways, let's proceed here. I'm going to ignore all of this and we are back to vision transformer. We form all of this. And finally, we have the projection layer that's going to project from 768, which is the inner model dimension into our desired output dimension, which is 512. That's it. I'm going to now hit run and get to our next step. So this is another point I wanted to make. Basically, this is the attention mask that's going to be formed for the causal transformers. If I hit step over and if I print the mask here, let's see what we get. So as you can see here is that your classical mask that's used to create a causal attention pattern. Okay? Let's continue. Okay. So we're back in the clip in this function before we created the visual parts. That's the vision transformer. And now we just create the transformer here. So as you see here, this particular transformer is going to have the build attention mask. It's going to be a causal one, whereas we do not pass the same argument to the vision transformer. So that one's going to be non-causal. Okay? And then we have the vocab size. So that's the token. So we saw that this is the BPE vocab we just created. And we're going to obviously create a token embedding table, which is going to be used by the transformer, by the textual pathway to basically embed the tokenized text. Okay? And we'll also have a positional embedding also for the transformer. And that's going to be 77 transformer width is 5. Okay. So that's going to be the positional embeddings. So again, it may be quickly worth going through that again here. So that was vision transformer. Now let me show you how this thing will work. So we have vocab size. We'll have a table. So we'll have a table here. That's going to be whatever the vocab size is, which is like 49k or something. And we'll have this size being, so I forgot what it is. Let me check what exactly that part is. It's going to be transformer width. Okay. So it's going to be 512. Okay. Let's go back here. That thing is going to be 512. And then we have additional table that's going to be like context length, which is going to be 77. And then also it's going to be 512. So those are going to be, this is the, basically the table we use to embed our tokens. And this is the positional, the positional encoding table. So what it means in practice is the following. So imagine you have a piece of text. You take the text, you tokenize it, you convert it into a list of integers. So let's imagine you started with like a hello world. And then what you do is you first tokenize it. So that means it's going to be somehow split. We'll see that logic a bit later. Maybe some like here, here, here. And then we're going to somehow map it into integer. So all of those tokens will have corresponding integers. And then that means we're going to end up with something like this. We'll end up with like, I don't know, like 23, 47, 56, 37. And we now use this to index this embedding table to get the associated vector. So 23 means, hey, let's grab token, let's grab the embedding vector that has index 23. And that's going to be your representation for this particular token. Okay. Then we say, let's take this 47. Let's take the, this token here and let's find its corresponding embedding vector. So that's this one. And then we put it here, et cetera, et cetera. And then once we form this and remember we're going to pad this with all zeros. So this is going to have bunch of zeros here. Zero, zero, zero, zeros, because we want to have the size of this is going to be 77. Whoops. Sorry, let me redraw that. That was some terrible drawing. So if I do this, basically this whole thing is going to be 77. And now we just add. So now we're going to add these positionally, positional embeddings on top of these vectors here. That's how this whole, this is how all of these structures come together. Hopefully that kind of clarifies it. Okay. Let's go back to the text, to the code. We formed the positional embedding table. We formed the layer norm. We finally have the text projection, which is going to be used to transform from the transformer width into the embedding width, which happened to have the same dimensionality. So this transformer is actually using internally 512, not 768 as this vision transformer. Okay. And there is this allotted scale. I'm not sure why these exact numbers kind of seems like, yeah, magic numbers, but yeah. And then just some initialization and that's it. Okay. Let me click F5. Let's get to this next salient break point. And that's here. Okay. So we made the image, which is going to be just this clip image, just some random diagram. We're going to do the pre-processing and basically we end up with a batch of size one. So if I were to print the image shape, it's going to be probably, okay, before we even do it, let's kind of do the predictions. It's going to be one, I guess, three, two, two, four, two, two, four, because this is the format that PyTorch usually does. So you have this channel dimension here instead of here. And if I print the shape, we're going to get, yeah, exactly that. Okay. So next up, we need to do the tokenization of these captions here. So a diagram, a dog, and a cat. And let me quickly recap what we've seen so far. I showed you how the clip is constructed. We saw how the vision transformer and the transformer models are instantiated and created. We've seen the image pre-processing here. And finally, let's kind of jump into this part here with the tokenization. So let me click F5. So we pass our text here and we need to tokenize them. And when I say tokenize, we literally need to transform to basically split these captions into sub words and then map those into corresponding integers, depending on our vocab mapping. Okay. So let me kind of go through this. First of all, we want to tokenize these special tokens because as you can see here, whatever the text is, we'll be trading through our captions. We're going to encode them. So this is going to give us a list of integers. And then we basically put the start of text token in the beginning and end of text token at the end. And now let's kind of jump into the actual encode function. What it does is, well, fairly complicated. I have to say the tokenization part, even though it's always abstracted away from us when you read the papers, et cetera, et cetera, I found this to be the hardest part of this whole code base for understanding. And yeah, just like the level of details is amazing. Having said that, let's go through this. Okay. So this line is basically going to clean the text. I won't go into the details of the actual cleaning. They're using some external libraries to do that. And then they just lowercase the whole caption. And that's the whole processing that's going on here. Next up, they are using this pattern. So this regex expression they compiled in the tokenizer to find the next piece of text from the caption, which I don't fully understand. If somebody understands the exact details of why this particular... So let me show you the pattern again. So here's the pattern, fairly complicated, not sure why all of these details. There is no comments, so it's kind of dark magic. But yeah, we'll have to roll with it. We can kind of treat it as a black box for now. Okay. So what happens here is the token is encoded using the UTF-8 encoding, which for this simple... So currently token is this small letter A, which is going to be encoded as a single byte. But some of the characters might be encoded into two, three, or four bytes. That's how UTF-8 works. Well, I'm not sure about the three, whether there are characters that have three bytes. Although, yeah, I'm fairly sure that's the case. I'm going to show you a bit later when you use emoji, this is going to give us four bytes, the encoding into UTF-8. But for this simple example, it's going to be a single byte. And then we use this dictionary byte encoder to fetch the corresponding... Basically to catch the corresponding Unicode character. So if I do F10 here, we're going to end up with this... Again, this is identity mapping for this particular token. We'll see later that it's going to be a bit more complicated for some other Unicode characters. So what happens now is the BPE magic. It's going to basically give us back... It's going to... In the general case, it's going to split this token into BPE tokens. In this case, it's going to be small a, again, just trivial. And then we're going to encode that using the encoder. So again, remember encoder just contains our vocabulary. The keys are the tokens and the values are the associated IDs, I guess. If I click F10 here, we just end up with BPE tokens being... As you can see here, we just encode 320. That's going to be the ID that corresponds to small letter a. If I continue the loop here, we get diagram. Again, this is going to be encoded. And finally, we're going to end up with, I think... Yeah, again, just a single... There is a single token that corresponds to diagram in our vocabulary. And that's going to be it. So now we basically repeat this whole procedure for the other captions. And let me kind of skip this and click F5. So let me go over this. And here we are. So we end up with all tokens being these realists. So as you can see, it's always a beginning and ending with a start and end of text tokens. And then in between is the tokenization that we get from our BPE tokenizer. Okay, let's continue here. So again, if you didn't quite understand everything that's going on here, don't worry because I did not also understand every single detail, to be honest. Okay, let's go on here. So what we do is we just pre-allocate a tensor of a corresponding... Of a following shape. Basically we want to have the context length, 77 here, and then we want to have whatever the number of captions is along the zeroth axis. So we end up with a tensor that's like, I guess, 377. Okay, let me go click F10. So just for sanity here, the shape is going to be 377. And here we're just going to fill it in. Basically we're just going to fill in the tokens that we got from the encoding here, and we're going to fill into this tensor. So I'm going to skip over all of these parts, and let me show you how the resulting tensor looks like. Basically, as you can see here, it begins with these IDs for our tokens, and then it's just like padded with zeros until we have the full 77 tokens worth of context. That's it. Let me step over here and let's continue. So that was a tokenization. We ended up with a tensor here. Again, the shape is 377. We have an image. The image is like this. And now we want to encode the image and we want to encode the text. And that means passing them through the visual transformer and through the transformer respectively. So let me show you how that is going to work. Here I think we're going to go through the... So visual is again a vision transformer. There is some type of conversion going on here. Basically we want to make sure that whatever the weights of our com1, so that's the patchify there, whatever the type of that thing is, we want to convert our image into the same type. So whether it's float32 or float16. In this case, I'm fairly sure it's going to be float32. Not that important, but let me just kind of see. Okay, it's actually float16. Yeah, because they have during the model loading, they have this conversion into FP16 function being called. So that explains it. Anyway, so let me go to the forward pass. Again, what's going to happen here is the logic I explained in my OneNote. Let me remind you if you forgot about that. So this is the logic that's going to happen. Basically we're going to patchify and extract these vectors and we're going to end up with like I think 16, 16, 7, 68. That should be the tensor we end up with. So if I click F10, we're going to patchify our tensor and let's see what the shape is. Or maybe we'll have additionally prepended one because we have a batch of one. So yeah, so it's basically one, whoops, it's one, 7, 68, 7, 7. So there is some basically permutation going on, but we kind of got the shape right. Okay, so now there is the reshaping going on here. We're going to, let me just kind of step through all of this, not explain all of the details, just some permutation and reshape going on. So we now just flatten the 7, 7 so we end up with all of the tokens being flattened out. So this is the batch size, the number of tokens. I don't know why it's expanding this constantly. Okay, so batch size, number of tokens, and finally the size of the embedding vector. Cool. So far so good. What we do here is we now take the class token, which is a learnable embedding vector, and we are going to basically concatenate it with this vector X here. Okay, so let me show you what I mean by that. So if I take the self class embedding, this is going to be just, the shape should be 1, 7, 6, 8, I think. So the shape of this is, the shape is actually just 7, 6, 8. And then once we sum it up with zeros, with a zero vector here, this is going to convert it into whatever the batch size is, and then 1, and then we'll end up with 7, 6, 8 here as well. I guess this is just a way to broadcast the CLS token. I can kind of copy paste this whole thing. I'm going to store it into a temporary variable. Oops, did I copy paste it? Yeah, I did. It just kind of overflowed. I can maybe remove this part with D type and device. Let me see where it's going to work. Maybe or maybe not. I think, yeah, it's important to actually preserve this part. So I'm going to add, do it like this. TMP and TMP shape is going to be 1, 1, 7, 6, 8. So basically we've done broadcasting, and then once you concatenate that with the X, which is of shape 1, 4, 9, 7, 6, 8, what is that going to do is let me think. Yeah, basically we're going to end up with 1, 5, 7, 6, 8, because we're going to concatenate the CLS to prepend the CLS to our token sequence. That should be the case here. So if I click F10, we end up with X shape 1, 5, 7, 6, 8, as I said. And then we just add the positional embeddings. So that's this step here. Oops, let me show you that. So that's going to be basically, let me just find it. That's this part, adding the positional encodings onto the vectors there. Let me step over that. Then there is some LNNorm going on, some permutation of the dimensions, because that's how this transformer expects it. Then we just process basically these. So now you can literally treat this as a simple NLP problem. We have a bunch of tokens with the corresponding channel dimensions. We just pass that through the transformer, and then we do the permutation again here. And that's it. So Vision Transformer is basically Patchify plus Transformer. That's all there is to it. And finally, we just want to project onto the common dimensionality of this joint multimodal space between the visual part and the textual part. And that's why we have this self-projection. So it's going to project us from basically 756 to 512. So as you can see here, that's the corresponding dimensions. And if we run this, we end up with what? We end up with a single vector that has 512 dimensions, because here we extracted some of the other dimensions. So basically what happened is we took our image and we ended up with a single vector that has 512 dimensions. Let's continue. That's the encoding image function. Now for the encoding text function. Again, let me know whether this is too much detail for you. Any feedback is super appreciated. The first time I'm doing this, so it will be super useful to hear some feedback. Embedding text, what's going to happen here is, so we have token embedding. So let's see what's going to happen here. Okay, so basically we are going to use the text tensor to index into our vocab, into our token embedding table. Okay. And let me remind you what this table is about. So token embedding is just a vocab size transformer width. So it's going to be just our basically embedding table. If I enter here, if I run this code, we end up with bit size number of tokens. Sorry, the size of the context, which is 77 and then 512. So let me bring the shape. So we started with this. We started with 377 and after embedding all of those IDs, and I'm butchering the terminology here, sorry for that. I'm not used to explaining these types of codes out loud, but yeah. Then we just add the self-positional embeddings, which are going to be of the shape 77, 512, I assume. So let me remind myself here, 77, 512. Yeah. So we just add them up. So that's the step we saw again. We saw that in the one note here. So basically that's this step. You have this embedding table and you're going to just add it up. You're just going to kind of put it here. So after the embedding has happened, you just kind of align it with your tokens. So it's going to be 77 and you just kind of add the corresponding vectors. You just add these two, you add these two, et cetera, et cetera. Okay. That's what happens. That's all. Let me step over. We do the, again, the permutation. We pass through the transformer. Again, return back into the shape that's desired. Then we pass through the layer normalization. And finally, there is this interesting piece of code. So what's this going to do is extract those tokens that are above the end of text token. Let me break that down for you. Again, let me show you what I mean by that. So if I do text argmax, we're going to end up with basically a vector of threes. Okay. And that's because the text tensor itself. So we have zero, one, two, three. You can see that the end of text token has the highest associated ID and that's why this argmax is going to work. So basically what we do is we extract the location of where this end of text token is in our tensor. And then we use those positions to extract the tokens from our vector x. So let me again show you the shape of x. So x is 3, 77, 5, 12. And what we're going to do out of all of these 77 tokens, we extract just a single one. And that's it. And then we do the projection into the final 512 dimension. So actually it's already in the correct dimension, but still they do this projection just to increase the capacity of the model, I guess. Let me step over this and we end up with x shape being 3, 5, 12. Okay. So that's it. We passed in three captions and we got out three vectors that all have 512 dimensions. That's everything we need. We can just do a simple normalization and then do the dot product to get the similarity score. Okay. Let's go back here. Now this function is kind of silly because we will not be using this, but if we just pass this model, if we just call the model, it's going to do exactly this. So let me show you what I mean by that. So if we go and if I click F5, so the for function again just does the encode image and encode text. So I'm just going to skip over those two because we've seen it already. So I'm just going to skip over all of that. Whoops. Let me just, my God, this was a mistake. Let me go back here. Let me go back here. Let me just put a break point here and click F5, skip all of this. Okay. So we are here. So we've done, we have, again, we have the image features, which is going to be 1, 5, 12. That's our vector. And then we're going to have text features, which is going to be 3, 5, 12. So that's a three vectors for the three captions. Now we do the normalization. We've seen this in the Drupiter already. So I'm just going to skip and assume you understand all of this. And finally, we, we take this, this is a learnable parameter that we did. I kind of wasn't sure what this magic number 0.7 is. Let me show you what I mean by that. So here is that logit scale. They kind of make it learnable and they initialize it like this. I'm not sure why this exact number, if anyone knows, feel free to comment down below. I guess it doesn't matter that much, but probably knowing how these NLP things work, it probably matters a lot. Like it's a, it's a, it's a difference between this, this training loss spiking or not a single, a single change of a variable can do that. So let's step over here. And what we do is we find the similarity matrix between the image vectors and the text vectors. And we just do the multiplication here because I guess they're going to pass it later on into soft max. That's why they're doing this. So in any case, we get the logits per image and logits per text, which you can get by just transposing so you don't have to do again, text features and do matrix multiplication with image features transpose. You can just transpose result. You can, you can kind of verify that that's going to be correct. Okay. Let's click F10. So we get the logits. And now, as I said, we do the soft max along the last dimension. So dimension minus one, and that's going to give us the probabilities. So logits, whoops, let me copy paste this. This should be, I guess, one three shape or something. Yep. And then if I kind of just print it, so here are the results. As you can see, they are not normalized. So after doing soft max here, if I click F10 here, we end up with probabilities. So probabilities, and we find that the caption number one or zero, depending on how you index your arrays, is going to be the one that corresponds to our image. And we'll see that that makes a lot of sense. So okay, I'm just going to exit this. And okay, I think, yeah, this is actually the end of the program. So we got the result that the zeroth caption is the best one. Let's see what it is. Well, because the zeroth caption is this caption here, it's a diagram. And let me show you the corresponding image. I think it's somewhere here. So clip image, where are you? Where are you? Okay, clip image here. So as you can see, it's a diagram of the clip infrastructure, of the clip architecture, and thus this makes a lot of sense. Okay, guys, this was pretty much it. This was a very, very long video. I hope you found something useful out of it. Before we wrap up, I just want to show you one more thing. So there is one more cool function I found in the in the reading file, and that's the linear probe. So everything is the same. So what we do, we download the cipher 100. What we then do is we would pass through the data set, and we encode all of the images into these feature vectors. So basically, what that means is we end up with, at the end, we end up with a bunch of like training features and associated labels, and same for the test data set. And then instead of doing this zero shop classifier, you can instead just train a simple logistic regression on top of the train feature. So again, train features, let me show you the shape. Shape is going to be whatever the number of data points in the training data set is, 512. So this is the shape. And then because we have the data set and we have the associated labels, this is a supervised data set will have basically an associated ground truth labels. And using these, you can train a linear probe. So that's this classifier here. Instead of doing what? Instead of doing the zero shot, where you would basically take the class, you would embed it into the template, and then you would encode that. And that's how you create those ad hoc classifiers. If you recall what I showed you before here. So that's basically this logic here. You could do zero shot, but obviously it's much better perf wise to do like a linear probe. Although again, you would you do have to train to get that. Whereas zero shot obviously is zero shot. Hence you don't have to do the training. Okay, so these variables are just found on some validation set. And then once you train the classifier, you can then see how it performs on your test features and report the accuracy. That's it. That's clip for you. Let me try and maybe quickly recap what we've seen. I'm going to walk you through the function here again. We load the clip model, we saw that clip model consists out of the vision transformer and the transformer. Aside from the training parameters, those models have these associated tables. Basically it's going to the division transformer is going to have this associated position encoding table plus the CLS token that's going to be learnable. Transformer is going to have this positional encoding table as well, as well as the vocab table. So that's the one that we use to embed our tokens here or the well tokens or IDs. We have the integers, so maybe better to refer to those as IDs, although people call those tokens as well. And that's that part. So after we create the clip, we can then do the tokenization. And this is arguably the hardest part. In my humble opinion, it's kind of dark magic. You have to process this text and you have to decide how to kind of split it using those reg ex expressions. And then after you do the reg ex expression, you use this PP to kind of split it into into sub words. And after that, you have to do the encoding into the corresponding integers. And that's how you get the text. Okay. Finally, once we have those two, we can basically pass them through the vision transformer and we can pass them through our transformer and get associated vectors. And then we can just do simple dot product after we normalize them to get the scores and thus similarity. That's it. I realized I want to show you one thing quickly. As I said, this is the hardest part of this tokenization. So I'm just going to add this emoji here and let's run the program again. So let me focus. Let me just remove these breakpoints and let me run this again. So let me debug the file. Let me show you what happens when we pass the brain emoji. Okay. Here we are. Let me go through the tokenize function again. Again we encode the special tokens and this is where the magic happens. So let me enter it into this function encode. So this is the hardest part of the whole code base if you ask me. I'm still trying to understand every detail here. We do some cleaning. Cleaning will not change this. This captions will end up with the same caption here. So we end up with the same caption. Nothing changed. Then we're going to first grab token A. So that's nothing interesting there. I'm just going to skip that. Finally this is the interesting part. So now we have a token brain. So this is emoji and this is your this is a legit Unicode character and there is a unique code point that Unicode assigns to each of the characters you can encounter wherever. So in whatever language you always have a Unicode is this this very nice. Well how would they call it like encoding principle to to map each of the characters Unicode characters onto unique code points which is just a unique integer. Okay. And so what happens here is once you encode this particular token you're going to end up with four bytes. So this particular token once you encoded using UTF eight it's going to give you four bytes. So I'll show you what I mean by that. If I take this and just print it here we end up with four bytes. And so that means we have four iterations here and then we're going to use this mapping table to fetch the associated to fetch the associated Unicode character and then we're going to just join them. And so we'll end up with some very weird token. Let me do show you that. So here's what this token looks like. Very weird. This is the representation we'll use to represent our brain emoji. And then now we have to do the BPE token part. So this is going to split this this character into sub words. So let me show you what we get. So this is what how he decides to split this particular character. And after that we just encode all of those. So that means we'll end up with two integers. So as you can see that the brain will be encoded using two integers. So let me show you what the BPE tokens are. So these two numbers eight seven nine two and five ten. That's what we how we encode the brain. As I said fairly fairly magic. I think they're referring to it as a dark horse behind the NLP like success because nobody talks about organization but it's fairly complicated to understand and to kind of capture all of those edge cases. Anyways enough rambling guys. Hope you found this video useful. If you did comment down below. Let me know how whether I can improve something or what not. Consider subscribing to this channel. Share the video art and until next time. Bye bye.
[{"start": 0.0, "end": 3.36, "text": " What's cracking guys? In this video I'm doing something a bit different."}, {"start": 3.36, "end": 9.72, "text": " So I'll be kicking off this series of videos where I'll be going through various,"}, {"start": 9.72, "end": 13.72, "text": " through the code of various papers that I've covered over the last months."}, {"start": 13.72, "end": 17.72, "text": " And so I'll be starting with OpenAI's clip."}, {"start": 17.72, "end": 24.92, "text": " So basically I cloned this repository OpenAI's clip implementation,"}, {"start": 24.92, "end": 32.04, "text": " created a conda environment locally and I'll be just debugging and going through VS code"}, {"start": 32.04, "end": 34.040000000000006, "text": " and showing you how how clip works."}, {"start": 34.040000000000006, "end": 37.44, "text": " But if you don't want to go through the hustle of creating the environment,"}, {"start": 37.44, "end": 42.84, "text": " you can just use their basically like colab here."}, {"start": 42.84, "end": 48.32000000000001, "text": " The environment is already set up, you can kind of copy paste some of the code from here into the colab"}, {"start": 48.32000000000001, "end": 53.040000000000006, "text": " and that's how you can kind of avoid having to go through the conda environment creation,"}, {"start": 53.04, "end": 56.64, "text": " which I'll not be explaining how to do in this video."}, {"start": 56.64, "end": 62.64, "text": " So having said that, let's jump into the actual code."}, {"start": 62.64, "end": 68.64, "text": " So first I'll go through the Jupyter notebooks to give you a high level understanding of how clip works."}, {"start": 68.64, "end": 75.24, "text": " And then we'll literally step through the code and see how the details of clip are implemented."}, {"start": 75.24, "end": 80.84, "text": " So vision transformers, transformers, contrastive loss, all of that."}, {"start": 80.84, "end": 89.04, "text": " So no, actually the training details are not implemented in the OpenAI's like open source code."}, {"start": 89.04, "end": 91.44, "text": " There is a separate repository that does that."}, {"start": 91.44, "end": 97.84, "text": " So I'll be skipping that part, but it's probably less relevant for what I'm trying to explain you here."}, {"start": 97.84, "end": 101.24000000000001, "text": " And that's how clip works in general."}, {"start": 101.24000000000001, "end": 105.24000000000001, "text": " OK, so I did create a video covering clips."}, {"start": 105.24000000000001, "end": 109.44, "text": " So do check that one out if you want to get the theoretical understanding."}, {"start": 109.44, "end": 113.64, "text": " I'll not be combining paper and code right now."}, {"start": 113.64, "end": 115.03999999999999, "text": " I'll just be focusing on code."}, {"start": 115.03999999999999, "end": 117.64, "text": " So if you haven't watched that one, go ahead and watch it."}, {"start": 117.64, "end": 119.03999999999999, "text": " I'll link it somewhere here."}, {"start": 119.03999999999999, "end": 121.03999999999999, "text": " So yeah, do check it out."}, {"start": 121.03999999999999, "end": 122.44, "text": " Anyways, let's start here."}, {"start": 122.44, "end": 126.64, "text": " So let's basically import a couple of things here."}, {"start": 126.64, "end": 127.64, "text": " No, thanks."}, {"start": 127.64, "end": 130.44, "text": " VS code is fairly annoying with these prompts,"}, {"start": 130.44, "end": 134.64, "text": " but it's probably the nicest environment to be debugging the code,"}, {"start": 134.64, "end": 137.24, "text": " both visually and just like there's so many cool features,"}, {"start": 137.24, "end": 140.64000000000001, "text": " which you'll hopefully see over this video."}, {"start": 140.64000000000001, "end": 146.84, "text": " So I imported necessary dependencies, numpy, torch, etc."}, {"start": 146.84, "end": 150.84, "text": " So let's see which models we have available."}, {"start": 150.84, "end": 154.44, "text": " So clip basically, again, how clip is constructed."}, {"start": 154.44, "end": 160.04000000000002, "text": " You have the visual like pathway, you have the textual pathway, so to speak."}, {"start": 160.04000000000002, "end": 163.84, "text": " So the visual one will encode the image into a vector."}, {"start": 163.84, "end": 170.64000000000001, "text": " The textual side will basically take a caption and embed it into a vector."}, {"start": 170.64000000000001, "end": 173.84, "text": " And basically then you just want to see the similarity"}, {"start": 173.84, "end": 176.84, "text": " between the dot product between normalized embeddings."}, {"start": 176.84, "end": 178.44, "text": " That's clip on a high level."}, {"start": 178.44, "end": 184.44, "text": " So as you can see, they've been using various different visual baselines."}, {"start": 184.44, "end": 187.44, "text": " So here are some modified versions of Fresnets."}, {"start": 187.44, "end": 188.84, "text": " They also use VAT."}, {"start": 188.84, "end": 192.24, "text": " We'll be using VAT in this Jupyter notebook."}, {"start": 192.24, "end": 196.24, "text": " So I can also kind of go through the definition and show you how this looks."}, {"start": 196.24, "end": 200.44, "text": " So basically they've defined this models dictionary."}, {"start": 200.44, "end": 202.44, "text": " It's a key value dictionary, obviously."}, {"start": 202.44, "end": 207.44, "text": " So we have the key here and we have the associated URL."}, {"start": 207.44, "end": 209.44, "text": " So that's where the model is stored."}, {"start": 209.44, "end": 217.24, "text": " So here and basically what this function shows us is all of the available,"}, {"start": 217.24, "end": 220.24, "text": " like pre-trained models, basically."}, {"start": 220.24, "end": 223.24, "text": " Okay, let's get back here."}, {"start": 223.24, "end": 225.84, "text": " Let me now run this cell."}, {"start": 225.84, "end": 229.84, "text": " So first what it does is again, it's going to load this VAT."}, {"start": 229.84, "end": 234.64000000000001, "text": " So the big version, the patches will be 32 by 32."}, {"start": 234.64000000000001, "end": 239.64000000000001, "text": " And it's going to basically give us the model and this pre-processing"}, {"start": 239.64000000000001, "end": 244.64000000000001, "text": " like a structure that contains the PyTorch transforms that we'll be using"}, {"start": 244.64000000000001, "end": 247.04000000000002, "text": " to transform the images like cropping and stuff."}, {"start": 247.04000000000002, "end": 248.44, "text": " We'll see that in a second."}, {"start": 248.44, "end": 253.44, "text": " So let's run this cell and let's see what's going on here."}, {"start": 253.44, "end": 256.24, "text": " Okay, so it's going to count the number of parameters in the model."}, {"start": 256.24, "end": 261.64, "text": " We can see here we have 151 millions of parameters in this VAT."}, {"start": 261.64, "end": 266.84, "text": " The input resolution that the VAT is expecting is 224 by 224"}, {"start": 266.84, "end": 271.24, "text": " and they've artificially set this context length to 77."}, {"start": 271.24, "end": 275.04, "text": " And so that means the number of tokens that will go into the,"}, {"start": 275.04, "end": 279.64000000000004, "text": " like text encoding pathway will be 77 tokens always."}, {"start": 279.64000000000004, "end": 281.04, "text": " So that's kind of fixed."}, {"start": 281.04, "end": 287.24, "text": " The woke up size is, as you can see here, 49,000 roughly."}, {"start": 287.24, "end": 289.84000000000003, "text": " And they are using the BPE encoding."}, {"start": 289.84000000000003, "end": 292.04, "text": " So that's the byte pairing coding."}, {"start": 292.04, "end": 294.44, "text": " I'll link some of the resources when it comes to BPE,"}, {"start": 294.44, "end": 297.04, "text": " which you can check out because I will not get into the details."}, {"start": 297.04, "end": 301.64000000000004, "text": " Otherwise, I could just create a video explaining how that thing works."}, {"start": 301.64, "end": 305.64, "text": " Okay, so this is some metadata about the model."}, {"start": 305.64, "end": 309.64, "text": " This is, okay, let me just run this to see which transformation we're using."}, {"start": 309.64, "end": 313.24, "text": " So we have resize to 224 because again,"}, {"start": 313.24, "end": 317.64, "text": " images will be of arbitrary aspect ratios and arbitrary resolutions."}, {"start": 317.64, "end": 324.24, "text": " We want to make it basically of the format that the VAT is expecting and that's 224 by 224."}, {"start": 324.24, "end": 328.64, "text": " So we're going to resize the image to 224, then do the central crop,"}, {"start": 328.64, "end": 333.84, "text": " then convert the image into RGB, convert it into tensor, and then do the normalization."}, {"start": 333.84, "end": 341.03999999999996, "text": " So these are just, I assume this is just ImageNet, like mean and standard deviations."}, {"start": 341.03999999999996, "end": 346.64, "text": " You literally go through the ImageNet training set and you collect these statistics."}, {"start": 346.64, "end": 349.44, "text": " You find the mean for all of the images, you find the standard deviation,"}, {"start": 349.44, "end": 358.44, "text": " and then you use that to center the input images before feeding them into the VAT portion of the clip."}, {"start": 358.44, "end": 361.64, "text": " Okay, so let's continue."}, {"start": 361.64, "end": 365.64, "text": " So we are first going to tokenize the sentence, hello world,"}, {"start": 365.64, "end": 369.24, "text": " and we'll later see how this tokenization works."}, {"start": 369.24, "end": 372.24, "text": " So as I said, it's using BPE in the background."}, {"start": 372.24, "end": 376.84, "text": " If we run this, we expect to get like a list of integers basically."}, {"start": 376.84, "end": 379.24, "text": " So you can see here, here is the result."}, {"start": 379.24, "end": 382.24, "text": " So this is the BPE magic we somehow get."}, {"start": 382.24, "end": 387.24, "text": " So this is going to be the start of sentence token, the end of sentence token,"}, {"start": 387.24, "end": 395.24, "text": " and these are the subtokens that BPE kind of broke this sentence down into these."}, {"start": 395.24, "end": 400.24, "text": " And it has all of these because, as I said, it's going to have 77 tokens"}, {"start": 400.24, "end": 403.44, "text": " because that's the constraint they artificially are using."}, {"start": 403.44, "end": 407.64, "text": " So let me just kind of maybe store this into a temporary variable"}, {"start": 407.64, "end": 410.64, "text": " and then print the shape of this thing."}, {"start": 410.64, "end": 413.84000000000003, "text": " It's going to be 7, yeah, as you can see, it's 77."}, {"start": 413.84, "end": 418.64, "text": " Okay, so let's get to this part here."}, {"start": 418.64, "end": 424.44, "text": " What they do here is they create this dictionary of descriptions."}, {"start": 424.44, "end": 431.03999999999996, "text": " So this is going to be a name of the image and this is going to be associated caption."}, {"start": 431.03999999999996, "end": 435.84, "text": " So we'll be using these names to just fetch the images from,"}, {"start": 435.84, "end": 439.23999999999995, "text": " I think, scikit-learn or something."}, {"start": 439.23999999999995, "end": 440.44, "text": " You'll see that in a second."}, {"start": 440.44, "end": 442.03999999999996, "text": " So let me run this cell."}, {"start": 442.04, "end": 445.44, "text": " Let's form the dictionary and now let's get here."}, {"start": 445.44, "end": 448.04, "text": " Okay, so what's going on here?"}, {"start": 448.04, "end": 455.44, "text": " So as you can see, we're going to iterate through this scikit-image."}, {"start": 455.44, "end": 461.84000000000003, "text": " So scikit-image package has certain images that come pre-installed with it."}, {"start": 461.84000000000003, "end": 463.64000000000004, "text": " So we're going to iterate through those file names."}, {"start": 463.64000000000004, "end": 471.44, "text": " And if the file name ends with PNG or JPEG, meaning if the file name is an image,"}, {"start": 471.44, "end": 473.64, "text": " we're going to enter this loop."}, {"start": 473.64, "end": 478.64, "text": " Then we're going to just grab the actual name here."}, {"start": 478.64, "end": 483.84, "text": " And if it's not in the description, so we're going to only extract these eight images here."}, {"start": 483.84, "end": 488.04, "text": " If it's not in the description, we'll just kind of skip it."}, {"start": 488.04, "end": 493.24, "text": " If it's in the description, then we're going to grab the path of that image,"}, {"start": 493.24, "end": 498.64, "text": " open it up as a pillow image and convert it into RGB."}, {"start": 498.64, "end": 504.44, "text": " Then we're going to just show all of those on a nice figure here."}, {"start": 504.44, "end": 507.64, "text": " I will not get into the Matplotlib details here."}, {"start": 507.64, "end": 514.4399999999999, "text": " And we're just going to store the pill images as well as the processed images and associated captions."}, {"start": 514.4399999999999, "end": 517.84, "text": " So this text is going to have so descriptions is here."}, {"start": 517.84, "end": 520.04, "text": " We're going to have the keywords here."}, {"start": 520.04, "end": 522.84, "text": " So basically we'll fetch the associated captions."}, {"start": 522.84, "end": 526.24, "text": " Let me run this enough rambling."}, {"start": 526.24, "end": 534.04, "text": " So you can see here the images we got from the scikit image package."}, {"start": 534.04, "end": 540.44, "text": " And well, unfortunately, let me see whether I can change the color here."}, {"start": 540.44, "end": 543.24, "text": " Yeah, I'm not sure whether you see the associated caption,"}, {"start": 543.24, "end": 548.24, "text": " but like I can read one for you, basically a portrait of an astronaut with the American flag."}, {"start": 548.24, "end": 549.24, "text": " This is this one."}, {"start": 549.24, "end": 552.84, "text": " And then a person looking at a camera on a tripod, blah, blah, blah."}, {"start": 552.84, "end": 556.24, "text": " So we now have the data set as clip expects."}, {"start": 556.24, "end": 560.44, "text": " So it's a small eight image data set image and associated caption."}, {"start": 560.44, "end": 565.84, "text": " And now let's see how clip is going to use these."}, {"start": 565.84, "end": 568.0400000000001, "text": " So let's continue here."}, {"start": 568.0400000000001, "end": 572.0400000000001, "text": " First of all, we're going to stack the processed images."}, {"start": 572.0400000000001, "end": 575.64, "text": " So remember images are the ones where we've applied the processing."}, {"start": 575.64, "end": 580.24, "text": " So the cropping and conversion to tensor normalization, all of that goodies."}, {"start": 580.24, "end": 582.0400000000001, "text": " Now we're going to stack it."}, {"start": 582.04, "end": 586.8399999999999, "text": " We're going to convert it into into into like a big tensor,"}, {"start": 586.8399999999999, "end": 589.24, "text": " which then we're going to move to to CUDA."}, {"start": 589.24, "end": 595.64, "text": " So that means basically pushing this tensor onto your GPU."}, {"start": 595.64, "end": 598.8399999999999, "text": " So let me run this."}, {"start": 598.8399999999999, "end": 601.04, "text": " And let me now create another cell here."}, {"start": 601.04, "end": 603.4399999999999, "text": " Whoops."}, {"start": 603.4399999999999, "end": 604.8399999999999, "text": " That's not what I wanted to do."}, {"start": 604.8399999999999, "end": 609.3399999999999, "text": " Let me just basically create a cell."}, {"start": 609.3399999999999, "end": 611.64, "text": " Okay, mixing up shortcuts from Colab."}, {"start": 611.64, "end": 615.24, "text": " So I'll just have to manually add a code cell here."}, {"start": 615.24, "end": 617.04, "text": " What I want to show you is just the shape."}, {"start": 617.04, "end": 622.04, "text": " So always understanding the shapes of your object is super important."}, {"start": 622.04, "end": 623.04, "text": " Let me run this."}, {"start": 623.04, "end": 626.4399999999999, "text": " So 8, 3, 2, 24, 2, 24."}, {"start": 626.4399999999999, "end": 628.14, "text": " So that's your image."}, {"start": 628.14, "end": 631.34, "text": " And because we have eight of those, we stacked eight of those."}, {"start": 631.34, "end": 634.84, "text": " We have like a torch tensor like this."}, {"start": 634.84, "end": 638.9399999999999, "text": " And similarly, Remy do the same thing for text tokens."}, {"start": 638.94, "end": 642.34, "text": " So what we've done here is we've gone through the captions."}, {"start": 642.34, "end": 644.34, "text": " Okay, so that's the text."}, {"start": 644.34, "end": 647.84, "text": " And then we just create this artificial prompt."}, {"start": 647.84, "end": 653.0400000000001, "text": " So this is and then we append the captions, the ones we had here."}, {"start": 653.0400000000001, "end": 656.24, "text": " So this is a page of text about segmentation, etc."}, {"start": 656.24, "end": 656.94, "text": " etc."}, {"start": 656.94, "end": 662.5400000000001, "text": " And then we're going to tokenize that which means we'll get those 77 context tokens."}, {"start": 662.5400000000001, "end": 664.84, "text": " So let me just run this again."}, {"start": 664.84, "end": 665.6400000000001, "text": " Let's see what we get."}, {"start": 665.6400000000001, "end": 667.34, "text": " So we have 8, 77."}, {"start": 667.34, "end": 673.0400000000001, "text": " So 8 because we have 8 captions and 77 because that's the basically context length"}, {"start": 673.0400000000001, "end": 677.64, "text": " that the transformer textual pathway in the clip is expecting."}, {"start": 677.64, "end": 679.44, "text": " Okay."}, {"start": 679.44, "end": 682.44, "text": " Finally, what we're going to do is we're going to patch."}, {"start": 682.44, "end": 686.74, "text": " We're going to pass the image tensor to the clip model."}, {"start": 686.74, "end": 690.64, "text": " So model is just clip and we're going to call this encode image function."}, {"start": 690.64, "end": 696.0400000000001, "text": " We'll see later what exactly does but basically just a forward prop of the images"}, {"start": 696.04, "end": 702.04, "text": " through your VAT and as the output will get I guess something probably the shape will"}, {"start": 702.04, "end": 709.74, "text": " be something like 8 and then I think it's either 512 or 756 or something like that."}, {"start": 709.74, "end": 711.54, "text": " We're going to see that in a second."}, {"start": 711.54, "end": 718.74, "text": " And here what we're expecting is similarly will pass these text tokens"}, {"start": 718.74, "end": 724.3399999999999, "text": " and I expect we'll have something like well, I guess 8 512 because we need to be able"}, {"start": 724.34, "end": 729.44, "text": " to compare those do dot product between the textual and visual embeddings."}, {"start": 729.44, "end": 734.84, "text": " So let me run this and then let me test the hypothesis whether I got the shapes right"}, {"start": 734.84, "end": 738.34, "text": " from like by the heart."}, {"start": 738.34, "end": 746.24, "text": " Okay, let me add an additional code cell here and let's just print the shapes of these objects."}, {"start": 746.24, "end": 749.74, "text": " So shape will be 8 512."}, {"start": 749.74, "end": 750.44, "text": " That's correct."}, {"start": 750.44, "end": 751.94, "text": " And then this must be the same."}, {"start": 751.94, "end": 753.34, "text": " So let me run this."}, {"start": 753.34, "end": 755.14, "text": " So 8 512."}, {"start": 755.14, "end": 755.5400000000001, "text": " Okay."}, {"start": 755.5400000000001, "end": 762.34, "text": " So again, pretty simple here because this is the transformer will be just taking the"}, {"start": 762.34, "end": 767.64, "text": " whatever the feature vector is above the end of sentence token."}, {"start": 767.64, "end": 773.44, "text": " So we just fetch that vector and that's what we're going to be using as a embedding for"}, {"start": 773.44, "end": 774.74, "text": " that particular caption."}, {"start": 774.74, "end": 777.44, "text": " Again, we'll see the details a bit later."}, {"start": 777.44, "end": 778.64, "text": " That's it."}, {"start": 778.64, "end": 782.5400000000001, "text": " Now the next steps are normalization."}, {"start": 782.54, "end": 786.64, "text": " So we'll normalize all of these embedding vectors."}, {"start": 786.64, "end": 793.14, "text": " So visual and textual embedding vectors and then we're going to do matrix multiply which"}, {"start": 793.14, "end": 801.04, "text": " is going to do dot products between all of these combinations of visual and textual embeddings"}, {"start": 801.04, "end": 807.64, "text": " and that's going to give us the similarities between those images and all of these captions."}, {"start": 807.64, "end": 816.84, "text": " So let me run this first like this and then I'm going to analyze what we are getting out"}, {"start": 816.84, "end": 817.24, "text": " of here."}, {"start": 817.24, "end": 819.34, "text": " So similarity should be 8 by 8."}, {"start": 819.34, "end": 824.4399999999999, "text": " So I'm expecting shape 8 by 8 because we want to know for each image."}, {"start": 824.4399999999999, "end": 827.74, "text": " What's the similarity score for that image and all of the other captions."}, {"start": 827.74, "end": 829.9399999999999, "text": " So let's see whether that's correct."}, {"start": 829.9399999999999, "end": 836.9399999999999, "text": " So if I print similarity shape here and if I run this it's 8 by 8 as expected."}, {"start": 836.94, "end": 840.74, "text": " Okay, so what's this thing about?"}, {"start": 840.74, "end": 845.84, "text": " The best thing to do is to just kind of because we are we have the interactivity here in Jupyter"}, {"start": 845.84, "end": 848.74, "text": " so why not use that fact."}, {"start": 848.74, "end": 851.74, "text": " So let me just print this and see what we get here."}, {"start": 851.74, "end": 857.34, "text": " Basically what this is going to do is find the I assume this is going to do L2 norm."}, {"start": 857.34, "end": 863.6400000000001, "text": " So it's going to find the L2 norm for each of the embedding vectors and then just divide"}, {"start": 863.64, "end": 868.14, "text": " as you can see here, we're just going to divide the vectors by their L2."}, {"start": 868.14, "end": 874.84, "text": " So basically if we have a vector X what we do is we convert it into X divided by L2 norm."}, {"start": 874.84, "end": 875.9399999999999, "text": " How can I denote that?"}, {"start": 875.9399999999999, "end": 883.24, "text": " Maybe like this L2 norm of that X and that's just your regularization that normalization"}, {"start": 883.24, "end": 887.34, "text": " that that makes it makes it so that the norm will now be 1."}, {"start": 887.34, "end": 890.9399999999999, "text": " So all of your vectors will be of norm 1."}, {"start": 890.94, "end": 895.94, "text": " Okay, let's run this cell and see what we get."}, {"start": 895.94, "end": 902.44, "text": " Well because I've already done the because I've already done the I already ran the cell."}, {"start": 902.44, "end": 906.24, "text": " I'm getting basically all of them are already 1."}, {"start": 906.24, "end": 907.74, "text": " So not that interesting."}, {"start": 907.74, "end": 909.74, "text": " So let me just rerun this for a second."}, {"start": 909.74, "end": 914.94, "text": " So let me rerun the image features and text features get the original ones and now before"}, {"start": 914.94, "end": 917.74, "text": " I run this cell, let me see what we get here."}, {"start": 917.74, "end": 923.14, "text": " So you can see here the associated L2 norms for each of the eight visual embedding vectors"}, {"start": 923.14, "end": 927.84, "text": " and we just divide by those L2 norms and we normalize the vectors by doing that."}, {"start": 927.84, "end": 933.44, "text": " We do the similar thing for basically for for the text."}, {"start": 933.44, "end": 937.74, "text": " So let me see if I can go to definition and see whether okay, no definition found for this."}, {"start": 937.74, "end": 942.14, "text": " So I guess we're going to use just some good old Google search."}, {"start": 942.14, "end": 943.74, "text": " So let's see what's going on here."}, {"start": 943.74, "end": 946.54, "text": " So norm by torch."}, {"start": 946.54, "end": 950.4399999999999, "text": " I think it should be like basically okay for a binius norm."}, {"start": 950.4399999999999, "end": 953.4399999999999, "text": " Let me see what's the default here."}, {"start": 953.4399999999999, "end": 958.74, "text": " So the order of norm default is for binius norm, which is just a fancy way of saying"}, {"start": 958.74, "end": 960.74, "text": " L2 for for matrix."}, {"start": 960.74, "end": 966.24, "text": " It's going to do the square root and then basically squared of all of the elements and"}, {"start": 966.24, "end": 968.4399999999999, "text": " then you just do the summation."}, {"start": 968.4399999999999, "end": 971.8399999999999, "text": " So yeah, we can quickly verify that hypothesis."}, {"start": 971.8399999999999, "end": 975.54, "text": " Just let's create some artificial vector."}, {"start": 975.54, "end": 980.9399999999999, "text": " So let's let me maybe create something like this."}, {"start": 980.9399999999999, "end": 986.4399999999999, "text": " So we have like a dummy dummy tensor."}, {"start": 986.4399999999999, "end": 987.8399999999999, "text": " My pi torch skills are bit rusty."}, {"start": 987.8399999999999, "end": 990.24, "text": " So hopefully I get everything correctly here."}, {"start": 990.24, "end": 995.9399999999999, "text": " So torch tensor or just zeros."}, {"start": 995.9399999999999, "end": 997.8399999999999, "text": " Can we do zeros?"}, {"start": 997.8399999999999, "end": 999.9399999999999, "text": " Well that will not be that interesting."}, {"start": 999.9399999999999, "end": 1001.54, "text": " Let me just think what we can do here."}, {"start": 1001.54, "end": 1007.04, "text": " Okay, so I'm gonna just do once and then I'm gonna take like size."}, {"start": 1007.04, "end": 1009.4399999999999, "text": " Let's say we just have a vector of size 2."}, {"start": 1009.4399999999999, "end": 1014.54, "text": " So there's going to be one one if it's L2 norm we expect square root of 2 which is roughly"}, {"start": 1014.54, "end": 1016.3399999999999, "text": " 1.41."}, {"start": 1016.3399999999999, "end": 1024.1399999999999, "text": " So if I now pass dummy tensor here and let's find the norm probably dimension and all of"}, {"start": 1024.1399999999999, "end": 1025.84, "text": " this is not that important."}, {"start": 1025.84, "end": 1031.04, "text": " But yeah, you see basically we have L2 here and that's pretty much it."}, {"start": 1031.04, "end": 1033.6399999999999, "text": " Now this thing is probably interesting."}, {"start": 1033.6399999999999, "end": 1035.6399999999999, "text": " We convert that at tensors."}, {"start": 1035.6399999999999, "end": 1040.94, "text": " We just push them to CPU, convert them to non-pi and then we do matrix multiplication."}, {"start": 1040.94, "end": 1042.34, "text": " So why does this make sense?"}, {"start": 1042.34, "end": 1043.54, "text": " And we have to transpose here."}, {"start": 1043.54, "end": 1044.6399999999999, "text": " So why does that make sense?"}, {"start": 1044.6399999999999, "end": 1052.94, "text": " Well because the following basically if we think about it text features is what 8 512"}, {"start": 1052.94, "end": 1060.1399999999999, "text": " and then we do matrix multiplication with a matrix that's 512 8 because we do the transpose"}, {"start": 1060.14, "end": 1066.64, "text": " here operation and that's gonna obviously give us if you do your matrix multiplication"}, {"start": 1066.64, "end": 1067.64, "text": " 8 by 8."}, {"start": 1067.64, "end": 1073.94, "text": " So what you're doing is you're taking a vector like you're taking like a textual embedding"}, {"start": 1073.94, "end": 1078.3400000000001, "text": " and by multiplying what you're doing is you're doing the products because you're multiplying"}, {"start": 1078.3400000000001, "end": 1082.8400000000001, "text": " the row with the column and the column is the visual embedding vector."}, {"start": 1082.84, "end": 1090.6399999999999, "text": " So if you do all of that you're basically getting the similarities between this particular"}, {"start": 1090.6399999999999, "end": 1093.24, "text": " text vector with all of these image vectors."}, {"start": 1093.24, "end": 1094.4399999999998, "text": " Hopefully that makes sense."}, {"start": 1094.4399999999998, "end": 1103.28, "text": " We can finally print we can print the similarity and do let me know whether you find this type"}, {"start": 1103.28, "end": 1104.58, "text": " of explanation useful."}, {"start": 1104.58, "end": 1107.84, "text": " Any feedback is super useful because this is the first time I'm doing something like"}, {"start": 1107.84, "end": 1108.84, "text": " this."}, {"start": 1108.84, "end": 1116.1399999999999, "text": " So as you can see all of these are similarities so they are not normalized which means if"}, {"start": 1116.1399999999999, "end": 1123.1, "text": " I were to do a summation will this work x is minus one if I print this."}, {"start": 1123.1, "end": 1127.98, "text": " So as you can see it's not one because we haven't applied softmax or anything."}, {"start": 1127.98, "end": 1129.74, "text": " So that's what we get here."}, {"start": 1129.74, "end": 1131.78, "text": " Okay let's continue."}, {"start": 1131.78, "end": 1132.78, "text": " Let's see what we have here."}, {"start": 1132.78, "end": 1139.62, "text": " So descriptions that's going to be our dictionaries we'll get I guess eight here which is going"}, {"start": 1139.62, "end": 1140.98, "text": " to plot something."}, {"start": 1140.98, "end": 1142.34, "text": " Let's see what we're doing here."}, {"start": 1142.34, "end": 1152.06, "text": " So original images we're going to plot the image and then we are going to plot the scores."}, {"start": 1152.06, "end": 1153.3799999999999, "text": " Okay let's just run the cell."}, {"start": 1153.3799999999999, "end": 1157.1, "text": " It's going to be easier to understand what's going on once we plotted."}, {"start": 1157.1, "end": 1161.46, "text": " So as you can see so what they've what they've done here is basically they have they plot"}, {"start": 1161.46, "end": 1167.46, "text": " the images here they plot the captions here and then they just plot this similarity matrix."}, {"start": 1167.46, "end": 1172.1000000000001, "text": " So we are basically visualizing the similarity matrix that I just printed a couple of cells"}, {"start": 1172.1000000000001, "end": 1178.8600000000001, "text": " before and you can see that as expected because this is a train clip model the biggest similarity"}, {"start": 1178.8600000000001, "end": 1185.94, "text": " is between the image and its correct caption and that's why we have this diagonal here."}, {"start": 1185.94, "end": 1187.8, "text": " That's pretty much it."}, {"start": 1187.8, "end": 1193.6599999999999, "text": " So yeah I guess I'll just ignore the details of this code because it's just idiosyncrasies"}, {"start": 1193.6599999999999, "end": 1195.78, "text": " of Matplotlib."}, {"start": 1195.78, "end": 1199.9199999999998, "text": " What I'm going to focus on is on the semantics and on what clip model is doing."}, {"start": 1199.9199999999998, "end": 1201.98, "text": " So this is it."}, {"start": 1201.98, "end": 1203.12, "text": " Let's see what else."}, {"start": 1203.12, "end": 1209.6599999999999, "text": " So now we are going to basically load a cipher hundred data set."}, {"start": 1209.6599999999999, "end": 1211.86, "text": " So that's going to probably take a while."}, {"start": 1211.86, "end": 1215.6599999999999, "text": " Oh okay I already have it cached so that's why it was so fast."}, {"start": 1215.66, "end": 1220.3400000000001, "text": " So we have a cipher hundred data set which is just your image data set that has a hundred"}, {"start": 1220.3400000000001, "end": 1223.9, "text": " classes hence the suffix hundred here."}, {"start": 1223.9, "end": 1228.02, "text": " And so what we're going to do is we're going to iterate through the classes so through"}, {"start": 1228.02, "end": 1232.42, "text": " all of those classes and we are going to form a caption out of that class."}, {"start": 1232.42, "end": 1235.66, "text": " So if you read the paper this is what I've been doing this is how you can do the zero"}, {"start": 1235.66, "end": 1238.0800000000002, "text": " shot image classification."}, {"start": 1238.0800000000002, "end": 1239.78, "text": " So we form a caption like this."}, {"start": 1239.78, "end": 1244.26, "text": " This is a photo of and then goes a label for example horse dog or whatnot."}, {"start": 1244.26, "end": 1246.06, "text": " So I'm not sure what the cipher 10 classes are."}, {"start": 1246.06, "end": 1247.46, "text": " We can print that."}, {"start": 1247.46, "end": 1253.66, "text": " I can kind of add a cell here and then print cipher 10."}, {"start": 1253.66, "end": 1254.94, "text": " Let's see what's going on here."}, {"start": 1254.94, "end": 1255.94, "text": " If I run this."}, {"start": 1255.94, "end": 1256.94, "text": " Okay."}, {"start": 1256.94, "end": 1261.66, "text": " Apple aquarium fish baby bear beaver bad blah blah blah."}, {"start": 1261.66, "end": 1262.66, "text": " Nice."}, {"start": 1262.66, "end": 1266.26, "text": " So we're going to as I said form captions from the classes like this and then do the"}, {"start": 1266.26, "end": 1267.26, "text": " tokenization."}, {"start": 1267.26, "end": 1272.3, "text": " So again the shape of this should be I guess hundred seventy seven because we have hundred"}, {"start": 1272.3, "end": 1278.3799999999999, "text": " captions hundred labels and hundred captions and seventy seven is the expected context"}, {"start": 1278.3799999999999, "end": 1279.3799999999999, "text": " length."}, {"start": 1279.3799999999999, "end": 1284.7, "text": " So let me run this and let me print the text tokens shape just to be sure."}, {"start": 1284.7, "end": 1287.8999999999999, "text": " So yeah as expected."}, {"start": 1287.8999999999999, "end": 1295.02, "text": " And now what we're going to do is the same thing as before past the textual these these"}, {"start": 1295.02, "end": 1298.02, "text": " these like tensors."}, {"start": 1298.02, "end": 1299.02, "text": " Okay this is not a tensor."}, {"start": 1299.02, "end": 1300.7, "text": " This is just as a wait."}, {"start": 1300.7, "end": 1302.26, "text": " What's the what's the type of this."}, {"start": 1302.26, "end": 1303.26, "text": " It's interesting."}, {"start": 1303.26, "end": 1304.26, "text": " What."}, {"start": 1304.26, "end": 1305.26, "text": " Okay."}, {"start": 1305.26, "end": 1306.26, "text": " It says here it's a torch tensor."}, {"start": 1306.26, "end": 1308.06, "text": " We're going to pass that we're going to encode a text."}, {"start": 1308.06, "end": 1314.74, "text": " So we are going to end up here with I assume hundred five twelve because all of those captions"}, {"start": 1314.74, "end": 1317.54, "text": " are going to be embedded into a vector."}, {"start": 1317.54, "end": 1319.48, "text": " So we get hundred five twelve."}, {"start": 1319.48, "end": 1321.72, "text": " Then we do the normalization as before."}, {"start": 1321.72, "end": 1326.3799999999999, "text": " So that means all of the norms of those embedding vectors going to be equal to one."}, {"start": 1326.3799999999999, "end": 1330.34, "text": " Whereas by norm I mean L2 norm in particular."}, {"start": 1330.34, "end": 1332.9399999999998, "text": " And then what we do is again image features text feature."}, {"start": 1332.9399999999998, "end": 1334.54, "text": " So we're not teaching the images."}, {"start": 1334.54, "end": 1337.4199999999998, "text": " So we're still using those eight images from before."}, {"start": 1337.4199999999998, "end": 1344.1, "text": " And we're just trying to find the most similar class from Cypher hundred for those images."}, {"start": 1344.1, "end": 1349.6999999999998, "text": " And again this thing is just going to calculate pair by similarity score between each of the"}, {"start": 1349.6999999999998, "end": 1354.26, "text": " images each of the eight images and each of the hundred captions."}, {"start": 1354.26, "end": 1358.1, "text": " We have to multiply by 100 just for numerical stability."}, {"start": 1358.1, "end": 1359.78, "text": " And then we apply softmax."}, {"start": 1359.78, "end": 1365.94, "text": " So that that means that now we'll basically have a sum sum would be equal to one for a"}, {"start": 1365.94, "end": 1367.62, "text": " particular image."}, {"start": 1367.62, "end": 1373.1, "text": " And so the sum of the similarity scores for this particular image and all of the captions"}, {"start": 1373.1, "end": 1375.3, "text": " needs to be equal to one."}, {"start": 1375.3, "end": 1376.3, "text": " OK."}, {"start": 1376.3, "end": 1380.58, "text": " So let me run this."}, {"start": 1380.58, "end": 1383.1, "text": " And finally what we do is once we find those."}, {"start": 1383.1, "end": 1385.18, "text": " So let me maybe let me print this."}, {"start": 1385.18, "end": 1393.18, "text": " So if I do this if I take the text props and they print those let's first bring the shape"}, {"start": 1393.18, "end": 1399.46, "text": " the shape should be what the shape should be eight times hundred I guess."}, {"start": 1399.46, "end": 1400.46, "text": " Yeah."}, {"start": 1400.46, "end": 1406.42, "text": " And if I print the actual text props it's going to be just not that intelligible because"}, {"start": 1406.42, "end": 1408.1000000000001, "text": " bunch of small numbers."}, {"start": 1408.1000000000001, "end": 1413.8200000000002, "text": " So this here this vector here is basically the similarity score for the first image and"}, {"start": 1413.82, "end": 1418.54, "text": " all of the hundred classes of the Cypher 10 Cypher 100."}, {"start": 1418.54, "end": 1422.98, "text": " And now we're trying to find the maximum because that's the high similarity."}, {"start": 1422.98, "end": 1425.3799999999999, "text": " And we find that by doing the top case."}, {"start": 1425.3799999999999, "end": 1431.4199999999998, "text": " So we find the five highest similarity scores for each of the images."}, {"start": 1431.4199999999998, "end": 1432.4199999999998, "text": " And that's what we get here."}, {"start": 1432.4199999999998, "end": 1436.86, "text": " So let me now print the top props and the top labels."}, {"start": 1436.86, "end": 1438.62, "text": " So let me just bring the top labels actually."}, {"start": 1438.62, "end": 1439.82, "text": " So we're going to get."}, {"start": 1439.82, "end": 1444.58, "text": " So these are the indices of the classes from Cypher 100."}, {"start": 1444.58, "end": 1447.82, "text": " They have the highest similarity score for the associated image."}, {"start": 1447.82, "end": 1449.9399999999998, "text": " So hopefully that makes sense."}, {"start": 1449.9399999999998, "end": 1451.5, "text": " OK."}, {"start": 1451.5, "end": 1458.34, "text": " So having done that let me run this final cell which is going to do the following."}, {"start": 1458.34, "end": 1463.86, "text": " So we are going to go through the original images."}, {"start": 1463.86, "end": 1466.8999999999999, "text": " We're going to plot that image."}, {"start": 1466.9, "end": 1471.46, "text": " We're going to take the probability scores."}, {"start": 1471.46, "end": 1473.14, "text": " OK."}, {"start": 1473.14, "end": 1477.6200000000001, "text": " And we're just going to do bar plots with those scores and with the associated images."}, {"start": 1477.6200000000001, "end": 1482.3400000000001, "text": " So again let me run this because this is too much detail to analyze in a video."}, {"start": 1482.3400000000001, "end": 1485.5800000000002, "text": " But basically this is what happens."}, {"start": 1485.5800000000002, "end": 1492.3400000000001, "text": " We just plot the images and the probabilities as well as the labels we just found in the"}, {"start": 1492.3400000000001, "end": 1493.3400000000001, "text": " cell above."}, {"start": 1493.3400000000001, "end": 1496.02, "text": " So this is what this magic here does."}, {"start": 1496.02, "end": 1503.1399999999999, "text": " You can see that this image is the closest class in Cypher 100 is woman."}, {"start": 1503.1399999999999, "end": 1507.34, "text": " The closest class for this picture here is man."}, {"start": 1507.34, "end": 1510.86, "text": " The closest class for the cat images sweet pepper."}, {"start": 1510.86, "end": 1511.86, "text": " OK."}, {"start": 1511.86, "end": 1516.54, "text": " We have a false like positive there."}, {"start": 1516.54, "end": 1518.02, "text": " And now we have cop here."}, {"start": 1518.02, "end": 1519.02, "text": " OK."}, {"start": 1519.02, "end": 1521.22, "text": " That's fairly decent motorcycle cattle."}, {"start": 1521.22, "end": 1522.98, "text": " I guess that's the closest thing they have."}, {"start": 1522.98, "end": 1524.82, "text": " Let me see where they have a horse."}, {"start": 1524.82, "end": 1527.26, "text": " So I guess the horse would be the closest label."}, {"start": 1527.26, "end": 1529.06, "text": " I'm going to do the following."}, {"start": 1529.06, "end": 1535.34, "text": " So I'm going to print his horse in Cypher 10 classes."}, {"start": 1535.34, "end": 1536.34, "text": " And it's not."}, {"start": 1536.34, "end": 1537.34, "text": " So we don't have a horse label."}, {"start": 1537.34, "end": 1542.3799999999999, "text": " So that means that this is probably the closest label we got to hear a motorcycle make sense"}, {"start": 1542.3799999999999, "end": 1543.58, "text": " again bed."}, {"start": 1543.58, "end": 1548.4199999999998, "text": " So this is completely gibberish unless there is some word better something inside of this"}, {"start": 1548.4199999999998, "end": 1550.7, "text": " image."}, {"start": 1550.7, "end": 1551.7, "text": " Let me zoom in."}, {"start": 1551.7, "end": 1552.7, "text": " Whoops."}, {"start": 1552.7, "end": 1558.66, "text": " Let us first determine my blah blah blah."}, {"start": 1558.66, "end": 1560.66, "text": " No this is probably just false positive."}, {"start": 1560.66, "end": 1564.22, "text": " And finally here we have a rocket which is also a good class."}, {"start": 1564.22, "end": 1565.22, "text": " That's it."}, {"start": 1565.22, "end": 1566.22, "text": " That's the first Jupiter notebook."}, {"start": 1566.22, "end": 1569.26, "text": " Do let me know how you're liking this video so far."}, {"start": 1569.26, "end": 1572.3, "text": " Comment down below if you find it useful."}, {"start": 1572.3, "end": 1575.74, "text": " OK now let's go to the second Jupiter notebook."}, {"start": 1575.74, "end": 1579.82, "text": " This one we're going to go fast through it because it's much simpler I assume."}, {"start": 1579.82, "end": 1583.3799999999999, "text": " Let me run the cell."}, {"start": 1583.3799999999999, "end": 1586.26, "text": " OK let me again available models."}, {"start": 1586.26, "end": 1592.54, "text": " We've seen that we're just going to load the VAT big 32 by 32."}, {"start": 1592.54, "end": 1594.82, "text": " That size is that's that model."}, {"start": 1594.82, "end": 1600.6599999999999, "text": " So that's the same model we used in the previous Jupiter again 151 million 224 2424 context"}, {"start": 1600.6599999999999, "end": 1605.1, "text": " length is 77 for the transformer woke up sizes like this."}, {"start": 1605.1, "end": 1607.26, "text": " So here what I've done is the following."}, {"start": 1607.26, "end": 1613.3799999999999, "text": " So they have a thousand image net classes with a small caveat."}, {"start": 1613.3799999999999, "end": 1621.7, "text": " So they've actually modified some of the classes such that clip has easier time basically recognizing"}, {"start": 1621.7, "end": 1622.7, "text": " those images."}, {"start": 1622.7, "end": 1624.86, "text": " So I think they mentioned it here."}, {"start": 1624.86, "end": 1626.08, "text": " Let me kind of read it through."}, {"start": 1626.08, "end": 1630.2, "text": " So subset of these class names are modified from the default image and class names source"}, {"start": 1630.2, "end": 1634.52, "text": " from any athletes image and simple labels."}, {"start": 1634.52, "end": 1639.76, "text": " These edits were made by trial and error and concentrated on the lowest performing classes"}, {"start": 1639.76, "end": 1646.06, "text": " according to top one and top five accuracy on the image net training set for the for"}, {"start": 1646.06, "end": 1647.98, "text": " these variations of the models."}, {"start": 1647.98, "end": 1653.7, "text": " So these weeks improved up one by one point five percent on the ATB 32 over using the"}, {"start": 1653.7, "end": 1660.22, "text": " default class names Alec which is the author of the Alec Redford assume from the clip paper"}, {"start": 1660.22, "end": 1665.18, "text": " got bored somewhere along the way as gain started to diminish and never finished updating"}, {"start": 1665.18, "end": 1666.5, "text": " tweaking the list."}, {"start": 1666.5, "end": 1671.38, "text": " He also didn't revisit this with the better performing models or any of the VAT models"}, {"start": 1671.38, "end": 1676.3, "text": " he thinks it's likely in other zero point five percent to one percent up one could be"}, {"start": 1676.3, "end": 1677.82, "text": " gained from further work here."}, {"start": 1677.82, "end": 1682.0, "text": " It'd be interesting to more rigorously study understand this."}, {"start": 1682.0, "end": 1683.32, "text": " So this is super interesting."}, {"start": 1683.32, "end": 1687.9, "text": " So this means that by tweaking the classes and then embedding them into those templates"}, {"start": 1687.9, "end": 1690.1200000000001, "text": " that the caption templates we saw."}, {"start": 1690.12, "end": 1692.6, "text": " So let me show you what I mean here."}, {"start": 1692.6, "end": 1696.0, "text": " So this is the template we are using."}, {"start": 1696.0, "end": 1702.8999999999999, "text": " So by just tweaking that we are forming better text embeddings and then consequently those"}, {"start": 1702.8999999999999, "end": 1708.8799999999999, "text": " make for a better classifiers your shot classifier on top of our image embedding vectors."}, {"start": 1708.8799999999999, "end": 1712.5, "text": " So let me quickly explain what I mean by that before proceeding here."}, {"start": 1712.5, "end": 1719.86, "text": " So imagine we call up we've done here we basically took hundred classes we embedded them into"}, {"start": 1719.86, "end": 1727.1399999999999, "text": " captions we have hundred like strings now we formed embedding vectors for each of the"}, {"start": 1727.1399999999999, "end": 1730.62, "text": " captions and now we have hundred hundred seventy."}, {"start": 1730.62, "end": 1737.82, "text": " So we basically have this we have sorry we have after embedding we end up with hundred"}, {"start": 1737.82, "end": 1738.86, "text": " five twelve."}, {"start": 1738.86, "end": 1741.06, "text": " So let me explain why this is so cool."}, {"start": 1741.06, "end": 1745.58, "text": " I'm going to open up my one note for a second here and let me just draw you a couple of"}, {"start": 1745.58, "end": 1746.58, "text": " things here."}, {"start": 1746.58, "end": 1747.58, "text": " OK."}, {"start": 1747.58, "end": 1748.58, "text": " So let's see what we've done."}, {"start": 1748.58, "end": 1752.9399999999998, "text": " So we have we embedded our image here into an embedding vector that's like five hundred"}, {"start": 1752.9399999999998, "end": 1757.26, "text": " twelve dimensions we have five hundred and twelve dimensions here."}, {"start": 1757.26, "end": 1763.22, "text": " Then we saw what we've done with the with the with the classes we've we formed basically"}, {"start": 1763.22, "end": 1766.26, "text": " a collection of these embedding vectors."}, {"start": 1766.26, "end": 1771.8999999999999, "text": " So this is hundred we have hundred and we have five twelve here as well."}, {"start": 1771.8999999999999, "end": 1776.74, "text": " So now because of the way how we calculate the similarity scores we basically multiply"}, {"start": 1776.74, "end": 1780.66, "text": " with your dot product between this one and this one with your dot product between the"}, {"start": 1780.66, "end": 1786.34, "text": " image vector again and the next and the next embedding vector the next representation of"}, {"start": 1786.34, "end": 1788.86, "text": " a class and the next one next one."}, {"start": 1788.86, "end": 1793.42, "text": " And what do you do if you think about it what do you do when you do dot products with these"}, {"start": 1793.42, "end": 1794.42, "text": " weights here."}, {"start": 1794.42, "end": 1800.22, "text": " This represents the weights of ad hoc MLP of a zero shot MLP that's used as a prediction"}, {"start": 1800.22, "end": 1802.14, "text": " head to predict the classes."}, {"start": 1802.14, "end": 1806.34, "text": " So again if you do dot product between this and this what you're effectively doing is"}, {"start": 1806.34, "end": 1807.34, "text": " the following."}, {"start": 1807.34, "end": 1812.6999999999998, "text": " So you're doing this and you form one weight and then you take this one."}, {"start": 1812.6999999999998, "end": 1814.82, "text": " So let me change the color here."}, {"start": 1814.82, "end": 1819.1, "text": " Then you do this you take this vector you do dot product between this and this and you"}, {"start": 1819.1, "end": 1820.54, "text": " effectively do the following."}, {"start": 1820.54, "end": 1824.6999999999998, "text": " So you just I'm kind of assuming this has four dimensions but it has five twelve so"}, {"start": 1824.6999999999998, "end": 1826.26, "text": " just bear with me here."}, {"start": 1826.26, "end": 1831.3, "text": " So you do this and then you do the same thing for the third vector."}, {"start": 1831.3, "end": 1834.52, "text": " So you're going to do dot product which is going to do this."}, {"start": 1834.52, "end": 1838.98, "text": " And as you can see this is basically the weights of this particular MLP here."}, {"start": 1838.98, "end": 1845.98, "text": " So you ad hoc construct the prediction head and by doing that you can take arbitrary data"}, {"start": 1845.98, "end": 1850.48, "text": " set you can literally take arbitrary data set here."}, {"start": 1850.48, "end": 1857.42, "text": " Take the classes form those captions embed those captions through just kind of do a forward"}, {"start": 1857.42, "end": 1859.54, "text": " pass through the transformer."}, {"start": 1859.54, "end": 1862.0, "text": " You get additional MLP."}, {"start": 1862.0, "end": 1867.04, "text": " So now you have MLP so this this is going to be a set of end classes with dimensionality"}, {"start": 1867.04, "end": 1873.5, "text": " of five twelve and you can basically use this to now classify this image into end classes."}, {"start": 1873.5, "end": 1878.54, "text": " So that's super cool and that's what a white clip is such an awesome model because of all"}, {"start": 1878.54, "end": 1880.74, "text": " of these zero shot capabilities."}, {"start": 1880.74, "end": 1883.98, "text": " So having said that let's go back to the code."}, {"start": 1883.98, "end": 1887.74, "text": " Let me go back to the where we stop here."}, {"start": 1887.74, "end": 1894.14, "text": " So now you can appreciate why why tweaking this can improve that prediction had weights"}, {"start": 1894.14, "end": 1899.8, "text": " and thus give us higher top one and top five accuracy on arbitrary data set data set."}, {"start": 1899.8, "end": 1901.7, "text": " So that's why they've been doing that."}, {"start": 1901.7, "end": 1902.7, "text": " OK."}, {"start": 1902.7, "end": 1905.46, "text": " So they give an example of what the tweaks some of the tweaks they've done."}, {"start": 1905.46, "end": 1910.14, "text": " So some of the examples beyond the crane crane so crane is going to be so one crane can be"}, {"start": 1910.14, "end": 1911.14, "text": " construction crane."}, {"start": 1911.14, "end": 1913.66, "text": " The other one is actually like a bird so bird crane."}, {"start": 1913.66, "end": 1919.14, "text": " So that's why they did those modifications and then they've done stuff like mail convert"}, {"start": 1919.14, "end": 1923.5800000000002, "text": " that to fingernail or like metal nail depending."}, {"start": 1923.5800000000002, "end": 1925.66, "text": " Yes I guess."}, {"start": 1925.66, "end": 1929.74, "text": " OK clip interprets nail as fingernail but we actually want it to interpret it as a metal"}, {"start": 1929.74, "end": 1934.38, "text": " nail and that's why they explicitly change the class to metal nail."}, {"start": 1934.38, "end": 1939.18, "text": " And similarly image net kite classes refer to the bird of prey not the flying toy."}, {"start": 1939.18, "end": 1943.02, "text": " So we change kite to kite bird of prey etc etc."}, {"start": 1943.02, "end": 1944.02, "text": " OK."}, {"start": 1944.02, "end": 1945.06, "text": " So that's that's it."}, {"start": 1945.06, "end": 1949.62, "text": " Now they have instead of using that single template we've been using here this is a photo"}, {"start": 1949.62, "end": 1953.74, "text": " of they also experimented with trying different templates."}, {"start": 1953.74, "end": 1957.86, "text": " You see here a bad photo of a photo of many a sculpture of."}, {"start": 1957.86, "end": 1961.1, "text": " And so there is a creative set of these here."}, {"start": 1961.1, "end": 1963.66, "text": " I think there is like literally at least 100 of them."}, {"start": 1963.66, "end": 1965.82, "text": " Let me let me see if you run this."}, {"start": 1965.82, "end": 1967.22, "text": " So we have 80 templates."}, {"start": 1967.22, "end": 1970.9, "text": " So they devised 80 of these templates."}, {"start": 1970.9, "end": 1977.5400000000002, "text": " And now they're going to do some type of in sampling to get better results for for image"}, {"start": 1977.5400000000002, "end": 1978.5400000000002, "text": " net."}, {"start": 1978.5400000000002, "end": 1980.3400000000001, "text": " Let's see how how that works."}, {"start": 1980.3400000000001, "end": 1984.18, "text": " Let's quickly go through the Russian rationale behind doing this again."}, {"start": 1984.18, "end": 1988.98, "text": " A similar intuition guided trial and error based on the image and training set was used"}, {"start": 1988.98, "end": 1990.22, "text": " for templates."}, {"start": 1990.22, "end": 1995.5, "text": " This list is pretty haphazard and was gradually made expanded over the course of about a year"}, {"start": 1995.5, "end": 1998.74, "text": " of the project and was revisited tweak every few months."}, {"start": 1998.74, "end": 2004.42, "text": " A surprising weird thing was adding templates intended to help image net our performance"}, {"start": 2004.42, "end": 2008.9, "text": " specifying different possible renditions of an object improved standard image net accuracy"}, {"start": 2008.9, "end": 2015.98, "text": " to after the 80 templates were locked for the paper we ran sequential forward selection"}, {"start": 2015.98, "end": 2018.06, "text": " over the list of 80 templates."}, {"start": 2018.06, "end": 2023.34, "text": " The search terminated after in sampling seven templates and selected them in the order below."}, {"start": 2023.34, "end": 2026.32, "text": " So combining these seven templates gave them the best results."}, {"start": 2026.32, "end": 2030.7, "text": " So then they say speculating we think it's interesting to see different scales large"}, {"start": 2030.7, "end": 2031.7, "text": " and small."}, {"start": 2031.7, "end": 2037.54, "text": " That's why they have templates like this working very nicely a photo of the large a difficult"}, {"start": 2037.54, "end": 2038.54, "text": " view."}, {"start": 2038.54, "end": 2042.74, "text": " That's why they have a bad photo of the and abstract versions origami video game art were"}, {"start": 2042.74, "end": 2044.06, "text": " all selected for."}, {"start": 2044.06, "end": 2046.34, "text": " But we haven't studied this in any detail."}, {"start": 2046.34, "end": 2051.64, "text": " This subset performs a bit better than the full 80 ensemble reported in the paper especially"}, {"start": 2051.64, "end": 2053.36, "text": " for the smaller models."}, {"start": 2053.36, "end": 2056.52, "text": " So what I mean by ensemble let me let me kind of break that down."}, {"start": 2056.52, "end": 2058.88, "text": " So we're going to see that now."}, {"start": 2058.88, "end": 2064.7000000000003, "text": " So let me load the image net data set here."}, {"start": 2064.7000000000003, "end": 2067.98, "text": " I think it's cash so shouldn't take too much."}, {"start": 2067.98, "end": 2071.42, "text": " So we're going to have the image net and here is what what they are doing."}, {"start": 2071.42, "end": 2079.9, "text": " So basically they form a zero shot classifier using the following technique."}, {"start": 2079.9, "end": 2085.3, "text": " So we pass all of the image and templates and we pass the image and classes."}, {"start": 2085.3, "end": 2086.6, "text": " And so let's see what they do."}, {"start": 2086.6, "end": 2091.98, "text": " So they go through all of the classes and then they go through all of the templates."}, {"start": 2091.98, "end": 2096.82, "text": " So that means that a single class will this time be embedded using not a single template"}, {"start": 2096.82, "end": 2103.56, "text": " but 80 of those and then they tokenize those the encoders which means we'll hear after"}, {"start": 2103.56, "end": 2108.9, "text": " doing the encoding we'll end up with what we'll end up with because we have 80 templates"}, {"start": 2108.9, "end": 2113.04, "text": " we'll end up with 80 77 right."}, {"start": 2113.04, "end": 2115.7000000000003, "text": " I think let me think here."}, {"start": 2115.7000000000003, "end": 2118.5, "text": " So we're doing the tokenization no sorry."}, {"start": 2118.5, "end": 2121.76, "text": " So here we'll end up with 80 77."}, {"start": 2121.76, "end": 2125.36, "text": " So tokenizer is going to give us this."}, {"start": 2125.36, "end": 2128.98, "text": " We can kind of print those shapes later on and see whether that's true."}, {"start": 2128.98, "end": 2135.7200000000003, "text": " And now after doing the embeddings they get 80 512 here."}, {"start": 2135.72, "end": 2140.4599999999996, "text": " And now the smart part is after doing the normalization which is which is like standard"}, {"start": 2140.4599999999996, "end": 2148.3399999999997, "text": " thing to do they form the final representation for that class by averaging across all of"}, {"start": 2148.3399999999997, "end": 2151.08, "text": " these embeddings for all of these different templates."}, {"start": 2151.08, "end": 2156.02, "text": " And then additionally they divide by the by the norm of the class embedding to make sure"}, {"start": 2156.02, "end": 2160.7, "text": " it's again has L2 norm equals to one."}, {"start": 2160.7, "end": 2162.3599999999997, "text": " And then they use."}, {"start": 2162.3599999999997, "end": 2164.8999999999996, "text": " So now they just stack those zero shot weights."}, {"start": 2164.9, "end": 2169.14, "text": " So this is going to be the zero shot classifier I explained in the one note a couple of minutes"}, {"start": 2169.14, "end": 2170.98, "text": " ago and that's it."}, {"start": 2170.98, "end": 2173.7400000000002, "text": " So this is going to be our classifier zero shot weights."}, {"start": 2173.7400000000002, "end": 2179.62, "text": " So if I run this it might take a while but it's such a neat idea."}, {"start": 2179.62, "end": 2184.94, "text": " Basically in sampling is a very very famous technique like you people use that to trade"}, {"start": 2184.94, "end": 2190.5, "text": " off between higher performance but like somewhat obviously longer inference more memory etc."}, {"start": 2190.5, "end": 2197.22, "text": " But just by kind of combining the majority like logic just doing this majority voting"}, {"start": 2197.22, "end": 2202.3, "text": " type of logic we get higher quality embeddings."}, {"start": 2202.3, "end": 2204.38, "text": " And and yeah."}, {"start": 2204.38, "end": 2214.38, "text": " OK this run now we can print what we get here again we're expecting I guess a shape of 1000"}, {"start": 2214.38, "end": 2221.2200000000003, "text": " 512 because we have 1000 image and classes we have embedding vectors of size 512."}, {"start": 2221.2200000000003, "end": 2225.6400000000003, "text": " So let me just print the zero shot weights shape here."}, {"start": 2225.6400000000003, "end": 2232.6600000000003, "text": " So if I print whoops if I print the shape here let me see whether my hypothesis was"}, {"start": 2232.6600000000003, "end": 2234.12, "text": " correct."}, {"start": 2234.12, "end": 2240.46, "text": " So 512 1000 OK it's just not it's stacked in a different way so that's why we get this"}, {"start": 2240.46, "end": 2242.82, "text": " shape shape."}, {"start": 2242.82, "end": 2255.34, "text": " OK we can maybe quickly just print these shapes here so text shape and I'm going to add print"}, {"start": 2255.34, "end": 2259.7000000000003, "text": " class embeddings here the shape as well."}, {"start": 2259.7000000000003, "end": 2265.2400000000002, "text": " And then we're just going to break out of this whole logic so that this is run quickly."}, {"start": 2265.2400000000002, "end": 2272.1000000000004, "text": " So whoops I'm getting OK because of the break statement everything crashed but like we are"}, {"start": 2272.1, "end": 2276.7, "text": " verifying that this makes sense so 80 77 80 512."}, {"start": 2276.7, "end": 2283.3399999999997, "text": " Let me now remove these statements and I'll have to rerun this to get the actual zero"}, {"start": 2283.3399999999997, "end": 2284.5, "text": " shot weights."}, {"start": 2284.5, "end": 2288.42, "text": " So after that while that's running let me walk you through this part here."}, {"start": 2288.42, "end": 2292.96, "text": " So we have the accuracy I will not go through how the accuracy is calculated you can go"}, {"start": 2292.96, "end": 2296.56, "text": " through that piece of code on your own."}, {"start": 2296.56, "end": 2302.56, "text": " So what we're going to do is we're going to here go through the image net batches of image"}, {"start": 2302.56, "end": 2303.56, "text": " net images."}, {"start": 2303.56, "end": 2309.2599999999998, "text": " We're going to push those to CUDA the associated labels as well."}, {"start": 2309.2599999999998, "end": 2312.7, "text": " So when I say to CUDA I mean to a GPU if you have one."}, {"start": 2312.7, "end": 2321.9, "text": " We are going to then encode the images so there's going to be 32 512 so 32 because we"}, {"start": 2321.9, "end": 2328.58, "text": " specified the batch size somewhere here so batch size going to be 32 and then 512 because"}, {"start": 2328.58, "end": 2335.7400000000002, "text": " as I said the visual embedding vector is going to be a 512 dimensionality."}, {"start": 2335.7400000000002, "end": 2340.78, "text": " Then we just do the regular normalization here and then we find the similarity."}, {"start": 2340.78, "end": 2347.06, "text": " So basically this is again this is your prediction head that was ad hoc constructed ad hoc."}, {"start": 2347.06, "end": 2355.46, "text": " So this is the zero shot classifier head and we get the logits so and then after we pass"}, {"start": 2355.46, "end": 2364.1, "text": " the logits and the target we'll see how good do we perform here when it comes to accuracy"}, {"start": 2364.1, "end": 2365.2999999999997, "text": " top one and top five accuracy."}, {"start": 2365.2999999999997, "end": 2368.56, "text": " So let me run this cell and let me run this one."}, {"start": 2368.56, "end": 2372.9, "text": " This might take a while so maybe I'll just kind of skip to the end."}, {"start": 2372.9, "end": 2381.06, "text": " Okay so this thing ran and we can see top one accuracy 55 top five 83.4 by just having"}, {"start": 2381.06, "end": 2383.82, "text": " by just doing this zero shot classification."}, {"start": 2383.82, "end": 2387.62, "text": " Again because we're passing logits just a minor detail obviously we'll have some soft"}, {"start": 2387.62, "end": 2388.82, "text": " max here or something."}, {"start": 2388.82, "end": 2392.7000000000003, "text": " Let me let me see how this logic here works."}, {"start": 2392.7000000000003, "end": 2394.7400000000002, "text": " So we just grab the outputs we find."}, {"start": 2394.7400000000002, "end": 2402.06, "text": " Okay we don't even have to do the soft max we just need to find the top five or whatever"}, {"start": 2402.06, "end": 2408.2999999999997, "text": " the case is five in our case top five logits which is going to correlate with the top five"}, {"start": 2408.2999999999997, "end": 2412.48, "text": " highest probabilities if we were to do soft max which we will not because that would be"}, {"start": 2412.48, "end": 2416.74, "text": " redundant and then just your usual logic."}, {"start": 2416.74, "end": 2420.62, "text": " It's going to convolute it because it's a one liner but it just kind of calculates the"}, {"start": 2420.62, "end": 2423.2599999999998, "text": " top one and top five accuracy."}, {"start": 2423.2599999999998, "end": 2424.2599999999998, "text": " Okay that's it."}, {"start": 2424.2599999999998, "end": 2430.42, "text": " Do let me know whether you found this Jupyter walkthrough useful and now I'm going to dig"}, {"start": 2430.42, "end": 2436.62, "text": " into going through some details of how everything works behind the scenes."}, {"start": 2436.62, "end": 2439.14, "text": " Hopefully you'll find this interesting as well."}, {"start": 2439.14, "end": 2440.82, "text": " So let's start with this playground."}, {"start": 2440.82, "end": 2447.66, "text": " So I just created a simple playground.py file then I just copy pasted pretty much some of"}, {"start": 2447.66, "end": 2454.58, "text": " the examples here from the readme file of their official repository and now we're going"}, {"start": 2454.58, "end": 2460.7, "text": " to walk through this particular example and see the details behind how the imaging code"}, {"start": 2460.7, "end": 2462.24, "text": " tokenization all of that works."}, {"start": 2462.24, "end": 2467.58, "text": " So without further ado let me run the debugger here."}, {"start": 2467.58, "end": 2473.14, "text": " So let me maybe zoom in a little bit more here hopefully you can see everything nicely."}, {"start": 2473.14, "end": 2478.9, "text": " So what's going on is that this tokenizer is just being created initially before we"}, {"start": 2478.9, "end": 2481.7, "text": " even hit this first line in simple."}, {"start": 2481.7, "end": 2484.8199999999997, "text": " We have a tokenizer somewhere here."}, {"start": 2484.8199999999997, "end": 2487.18, "text": " Let me just find it."}, {"start": 2487.18, "end": 2490.9399999999996, "text": " It's kind of defined I think in model or somewhere."}, {"start": 2490.9399999999996, "end": 2492.7799999999997, "text": " The object is independently defined."}, {"start": 2492.7799999999997, "end": 2493.7799999999997, "text": " Let me find it."}, {"start": 2493.7799999999997, "end": 2494.7799999999997, "text": " Oh my god."}, {"start": 2494.7799999999997, "end": 2495.7799999999997, "text": " Oh my god."}, {"start": 2495.7799999999997, "end": 2496.7799999999997, "text": " Okay tokenizer."}, {"start": 2496.7799999999997, "end": 2497.7799999999997, "text": " Yeah here."}, {"start": 2497.7799999999997, "end": 2498.7799999999997, "text": " So here is the construct."}, {"start": 2498.7799999999997, "end": 2504.48, "text": " So basically the constructor is called before we even hit the first line of our function"}, {"start": 2504.48, "end": 2505.48, "text": " in the playground."}, {"start": 2505.48, "end": 2506.8999999999996, "text": " So again let me just kind of show you."}, {"start": 2506.8999999999996, "end": 2511.3199999999997, "text": " I have the main function I'm calling the simple here and simple is this one."}, {"start": 2511.32, "end": 2515.42, "text": " So you'd expect this to be the first line that we hit but that's not the case."}, {"start": 2515.42, "end": 2519.7400000000002, "text": " We instead hit this one because as I already explained why that's going on."}, {"start": 2519.7400000000002, "end": 2521.42, "text": " Okay so tokenizer."}, {"start": 2521.42, "end": 2522.42, "text": " What's going on?"}, {"start": 2522.42, "end": 2528.44, "text": " So I'll have to kind of skim through some of the details because it gets fairly complicated"}, {"start": 2528.44, "end": 2532.52, "text": " and I also don't understand every single detail as well to be honest."}, {"start": 2532.52, "end": 2539.1400000000003, "text": " So this bytes to unicode what it does is just creates like a dictionary between 0 to 255"}, {"start": 2539.14, "end": 2548.06, "text": " which are like values that a byte can have and then it maps it onto a unicode string."}, {"start": 2548.06, "end": 2553.04, "text": " There is some convoluted logic here of which ranges they take etc."}, {"start": 2553.04, "end": 2557.14, "text": " So that's why I'm saying I'm going to kind of skip some of these details but like we'll"}, {"start": 2557.14, "end": 2559.96, "text": " analyze what we get as the output."}, {"start": 2559.96, "end": 2563.42, "text": " So we'll treat it like as a semi black box."}, {"start": 2563.42, "end": 2565.24, "text": " So if that's even a word."}, {"start": 2565.24, "end": 2569.54, "text": " Okay let me go to debug console and this is why Visual Studio Code is such an awesome"}, {"start": 2569.54, "end": 2572.2599999999998, "text": " environment."}, {"start": 2572.2599999999998, "end": 2574.14, "text": " Let me show you what we have here."}, {"start": 2574.14, "end": 2582.3799999999997, "text": " So if I print this we get as you can see here we get basically so from 0 to 255 we get the"}, {"start": 2582.3799999999997, "end": 2587.06, "text": " keys and then associated unicode characters."}, {"start": 2587.06, "end": 2589.7599999999998, "text": " Okay that's pretty much it."}, {"start": 2589.7599999999998, "end": 2592.4599999999996, "text": " Now the byte encoder is going to do the opposite."}, {"start": 2592.46, "end": 2600.92, "text": " It's just going to map from the unicode character onto the byte here."}, {"start": 2600.92, "end": 2603.96, "text": " So I'm just going to kind of skim over that."}, {"start": 2603.96, "end": 2607.2, "text": " So F10 then we open up so there is this BPE path."}, {"start": 2607.2, "end": 2611.98, "text": " So there is basically if I open it up here I think it stores somewhere in here."}, {"start": 2611.98, "end": 2614.42, "text": " Let me let me let me see where it is."}, {"start": 2614.42, "end": 2615.42, "text": " Just a second."}, {"start": 2615.42, "end": 2616.46, "text": " Okay so here it is."}, {"start": 2616.46, "end": 2623.0, "text": " So this is this file BP simple vocab 16E6 takes that."}, {"start": 2623.0, "end": 2626.42, "text": " So that's basically your BPE."}, {"start": 2626.42, "end": 2630.5, "text": " That's the byte pair encoding as I said I will not get into the details."}, {"start": 2630.5, "end": 2632.42, "text": " So what they've done they had some data set."}, {"start": 2632.42, "end": 2639.14, "text": " I'm not sure how I can track back the actual data set that was used to train this BPE."}, {"start": 2639.14, "end": 2645.58, "text": " And then you do the BPE logic which is basically finding the pair of tokens with the highest"}, {"start": 2645.58, "end": 2649.86, "text": " frequency and then merging those two tokens and adding that token to the vocab and then"}, {"start": 2649.86, "end": 2657.5, "text": " just repeating those merges until we have the final vocabulary that that's the BPE vocab."}, {"start": 2657.5, "end": 2660.74, "text": " So that's kind of already done for us."}, {"start": 2660.74, "end": 2665.86, "text": " That's why we can just read this and then basically decode."}, {"start": 2665.86, "end": 2672.1, "text": " Consider thinking like decode as a UTF-8 and then just split on the new line characters."}, {"start": 2672.1, "end": 2676.22, "text": " And then I'll show you in a second what this merges are doing."}, {"start": 2676.22, "end": 2678.18, "text": " So the merges are doing the following."}, {"start": 2678.18, "end": 2681.7799999999997, "text": " So let me first close this so we have more space."}, {"start": 2681.7799999999997, "end": 2684.06, "text": " So merges is a list."}, {"start": 2684.06, "end": 2686.96, "text": " So I'm going to just show you the first five for example."}, {"start": 2686.96, "end": 2688.38, "text": " So you can see here."}, {"start": 2688.38, "end": 2690.46, "text": " So here these are the merges that were happening."}, {"start": 2690.46, "end": 2694.94, "text": " So that means this this this whoops why is this okay."}, {"start": 2694.94, "end": 2701.18, "text": " So this means this particular these two tokens were the most frequent tokens in that corpus"}, {"start": 2701.18, "end": 2705.3399999999997, "text": " of text at some point of time during the BPE algorithm."}, {"start": 2705.3399999999997, "end": 2709.54, "text": " So if you don't know what BPE is I strongly suggest you just kind of find some article."}, {"start": 2709.54, "end": 2713.7, "text": " I'm going to link it down in the YouTube description actually in the video description and you"}, {"start": 2713.7, "end": 2714.7, "text": " can check it out."}, {"start": 2714.7, "end": 2717.22, "text": " And so this will make sense my explanations."}, {"start": 2717.22, "end": 2723.1, "text": " So so yeah we have these merges now and I'm going to go over this."}, {"start": 2723.1, "end": 2725.62, "text": " And finally we start forming the vocab."}, {"start": 2725.62, "end": 2732.46, "text": " So first of all we're going to grab the the the the Unicode strings that we formed up"}, {"start": 2732.46, "end": 2733.46, "text": " there."}, {"start": 2733.46, "end": 2734.46, "text": " So let me go F10."}, {"start": 2734.46, "end": 2736.94, "text": " So that means the vocab size will be 256."}, {"start": 2736.94, "end": 2741.08, "text": " So if I print the vocab size here it's going to be 256."}, {"start": 2741.08, "end": 2744.7599999999998, "text": " Now the following again this is BPE idosyncrasy."}, {"start": 2744.7599999999998, "end": 2751.22, "text": " So basically we have to to concat this this end of word like token."}, {"start": 2751.22, "end": 2760.18, "text": " So if I click we'll just do we'll just basically form new tokens in our vocab by concatenating"}, {"start": 2760.18, "end": 2765.06, "text": " all of the existing like tokens with this particular end of word token."}, {"start": 2765.06, "end": 2766.06, "text": " OK."}, {"start": 2766.06, "end": 2771.74, "text": " So if I click F10 if I print the length now it's going to be 512 because we just duplicate"}, {"start": 2771.74, "end": 2776.3799999999997, "text": " everything and then we go through the merges and we because all of those merges basically"}, {"start": 2776.3799999999997, "end": 2780.04, "text": " are now a unique token in the BPE vocab."}, {"start": 2780.04, "end": 2781.42, "text": " That's why we do this logic here."}, {"start": 2781.42, "end": 2782.68, "text": " So we basically join."}, {"start": 2782.68, "end": 2784.5, "text": " So that means we're going to join these."}, {"start": 2784.5, "end": 2789.66, "text": " So we're going to have in instead of instead of so I can maybe let me see whether I can"}, {"start": 2789.66, "end": 2794.9, "text": " kind of step through this quickly or just do something like this."}, {"start": 2794.9, "end": 2795.9, "text": " OK."}, {"start": 2795.9, "end": 2802.14, "text": " So if I do this if I do F10 and let me print you the merge here directly in the debug console."}, {"start": 2802.14, "end": 2805.22, "text": " So as you can see we're going to add the in to the vocab."}, {"start": 2805.22, "end": 2807.2599999999998, "text": " And again that's BPE idosyncrasy."}, {"start": 2807.2599999999998, "end": 2809.42, "text": " I'm going to skip over all of that."}, {"start": 2809.42, "end": 2812.66, "text": " Click F5 here and just get to the next line here."}, {"start": 2812.66, "end": 2813.66, "text": " OK."}, {"start": 2813.66, "end": 2816.62, "text": " Let me collapse this."}, {"start": 2816.62, "end": 2817.62, "text": " So bear with me."}, {"start": 2817.62, "end": 2818.62, "text": " It's a bit more."}, {"start": 2818.62, "end": 2821.66, "text": " I'll summarize the findings in a second."}, {"start": 2821.66, "end": 2823.66, "text": " So we extend with these two special tokens."}, {"start": 2823.66, "end": 2824.66, "text": " Start of text and of text."}, {"start": 2824.66, "end": 2827.7400000000002, "text": " So it's like start of sentence and of sentence tokens."}, {"start": 2827.7400000000002, "end": 2831.14, "text": " So if I add all that now we have a pretty big vocab."}, {"start": 2831.14, "end": 2833.34, "text": " So I think it's almost 50."}, {"start": 2833.34, "end": 2834.34, "text": " Yeah."}, {"start": 2834.34, "end": 2835.34, "text": " 49,408."}, {"start": 2835.34, "end": 2837.12, "text": " That's it."}, {"start": 2837.12, "end": 2843.9, "text": " Now we form the encoder by basically zipping the those tokens from the vocab with the associated"}, {"start": 2843.9, "end": 2846.3399999999997, "text": " indices and we do the decoder."}, {"start": 2846.3399999999997, "end": 2852.02, "text": " So that's going to basically convert from from from a token into an int and vice versa"}, {"start": 2852.02, "end": 2855.58, "text": " the decoder will do from int into into the token."}, {"start": 2855.58, "end": 2862.2999999999997, "text": " So if I do F10 here just step over we get all of this and then they form the special"}, {"start": 2862.2999999999997, "end": 2864.8199999999997, "text": " variable BP ranks."}, {"start": 2864.8199999999997, "end": 2866.7999999999997, "text": " It's going to take those merges."}, {"start": 2866.8, "end": 2869.86, "text": " So and and and kind of store them."}, {"start": 2869.86, "end": 2872.1400000000003, "text": " We'll see why that's useful a bit later."}, {"start": 2872.1400000000003, "end": 2877.5, "text": " And then they have this cache again some optimization detail that the BP will need otherwise will"}, {"start": 2877.5, "end": 2878.88, "text": " be super slow."}, {"start": 2878.88, "end": 2885.86, "text": " And finally this pattern that they're going to use to kind of parse the input text."}, {"start": 2885.86, "end": 2886.86, "text": " We'll get to that in a second."}, {"start": 2886.86, "end": 2887.86, "text": " OK."}, {"start": 2887.86, "end": 2891.7000000000003, "text": " So as a summary we have a vocabulary formed using the BP."}, {"start": 2891.7000000000003, "end": 2892.7000000000003, "text": " Lots of details."}, {"start": 2892.7000000000003, "end": 2894.6600000000003, "text": " You don't need to know all of these."}, {"start": 2894.66, "end": 2899.22, "text": " What is important is that we have the encoder and decoder and all of these are some other"}, {"start": 2899.22, "end": 2901.2999999999997, "text": " implementation details."}, {"start": 2901.2999999999997, "end": 2905.2999999999997, "text": " Important ones but we can we can kind of kind of squint and ignore them."}, {"start": 2905.2999999999997, "end": 2906.2999999999997, "text": " OK."}, {"start": 2906.2999999999997, "end": 2908.3799999999997, "text": " Let me go to our next breakpoint."}, {"start": 2908.3799999999997, "end": 2909.3799999999997, "text": " OK."}, {"start": 2909.3799999999997, "end": 2911.54, "text": " So we are now in the first line of the simple method."}, {"start": 2911.54, "end": 2912.54, "text": " OK."}, {"start": 2912.54, "end": 2914.06, "text": " So we just do the standard stuff."}, {"start": 2914.06, "end": 2919.7799999999997, "text": " We see whether we have like a GPU device and I do have to do and I do happen to have a"}, {"start": 2919.7799999999997, "end": 2920.7799999999997, "text": " GPU."}, {"start": 2920.7799999999997, "end": 2921.8999999999996, "text": " So that's that's cool."}, {"start": 2921.8999999999996, "end": 2923.04, "text": " Now we'll load the device."}, {"start": 2923.04, "end": 2926.3, "text": " So now we'll have some some some break point."}, {"start": 2926.3, "end": 2927.3, "text": " That's interesting."}, {"start": 2927.3, "end": 2932.66, "text": " So basically before the video I put some salient break points on certain parts of the code"}, {"start": 2932.66, "end": 2936.7799999999997, "text": " base that I want to dig to kind of go through with you."}, {"start": 2936.7799999999997, "end": 2940.82, "text": " The code base is fairly big so you can kind of do that in your own time."}, {"start": 2940.82, "end": 2943.7799999999997, "text": " But I'm going to focus on the most salient parts of the code base."}, {"start": 2943.7799999999997, "end": 2944.7799999999997, "text": " OK."}, {"start": 2944.7799999999997, "end": 2949.42, "text": " So the first one being the construction of the clip model."}, {"start": 2949.42, "end": 2950.94, "text": " So context length."}, {"start": 2950.94, "end": 2952.1, "text": " That's a 77."}, {"start": 2952.1, "end": 2954.1, "text": " That's a number we've been seeing all around."}, {"start": 2954.1, "end": 2959.06, "text": " And that's because that's a that's a parameter that's passed during the construction of the"}, {"start": 2959.06, "end": 2960.06, "text": " clip model."}, {"start": 2960.06, "end": 2961.54, "text": " So let's see what's going on here."}, {"start": 2961.54, "end": 2966.54, "text": " So we're going to form the VAT model here."}, {"start": 2966.54, "end": 2968.7599999999998, "text": " So I'll hit the VAT construction right now."}, {"start": 2968.7599999999998, "end": 2970.9, "text": " So we have image net."}, {"start": 2970.9, "end": 2978.46, "text": " The image resolution is 24 through to the internal vectors will be 768."}, {"start": 2978.46, "end": 2987.12, "text": " We'll have 12 layers 12 heads and the embedding dimension at the output is 512."}, {"start": 2987.12, "end": 2989.98, "text": " So even though the inner dimensions are a bit larger."}, {"start": 2989.98, "end": 2993.3, "text": " So that's here 768."}, {"start": 2993.3, "end": 2996.18, "text": " The output as you can recall is 512."}, {"start": 2996.18, "end": 2997.7400000000002, "text": " We saw that in the Jupiter."}, {"start": 2997.7400000000002, "end": 3007.18, "text": " So let me click F5 here and go to the vision transformer basically in it function."}, {"start": 3007.18, "end": 3009.68, "text": " So we're just going to store some of these dimensions."}, {"start": 3009.68, "end": 3012.14, "text": " So this is everything that vision transformer needs."}, {"start": 3012.14, "end": 3016.44, "text": " This is the conv layer that that you have to this is the patchy file layer basically"}, {"start": 3016.44, "end": 3018.3999999999996, "text": " of your vision transformer."}, {"start": 3018.3999999999996, "end": 3020.58, "text": " As you can see here the pet size is 32."}, {"start": 3020.58, "end": 3027.46, "text": " So we set the kernel size to 32 and we set a stride the same exact width."}, {"start": 3027.46, "end": 3033.66, "text": " And the reason is is because we want to take non overlapping patches of the image and then"}, {"start": 3033.66, "end": 3039.62, "text": " embed them from the initial dimension of three into 768."}, {"start": 3039.62, "end": 3044.06, "text": " So let me quickly go through that in the one note maybe it will help you understand what's"}, {"start": 3044.06, "end": 3045.94, "text": " going on."}, {"start": 3045.94, "end": 3049.3399999999997, "text": " Again what is going on is the following."}, {"start": 3049.3399999999997, "end": 3053.8599999999997, "text": " So if you have an image here."}, {"start": 3053.8599999999997, "end": 3055.2599999999998, "text": " This is our input image."}, {"start": 3055.2599999999998, "end": 3057.1, "text": " We have three channels."}, {"start": 3057.1, "end": 3065.54, "text": " It's going to be three dimensions here so this is three and then this first conv layer"}, {"start": 3065.54, "end": 3066.54, "text": " is doing the following."}, {"start": 3066.54, "end": 3074.02, "text": " It's going to take the kernel put it here and map this whole volume into a vector into"}, {"start": 3074.02, "end": 3080.66, "text": " a corresponding vector that's going to be of dimension 768."}, {"start": 3080.66, "end": 3082.9, "text": " Then we're going to move here."}, {"start": 3082.9, "end": 3085.38, "text": " So let me change the color."}, {"start": 3085.38, "end": 3091.38, "text": " We'll then move here and then embed that into additional vector here 768."}, {"start": 3091.38, "end": 3096.2200000000003, "text": " And we're going to repeat the whole procedure until we hit the end of the image."}, {"start": 3096.2200000000003, "end": 3097.58, "text": " So now we have here etc. etc."}, {"start": 3097.58, "end": 3103.5, "text": " You do that for the whole image and you end up with what you end up with whatever the"}, {"start": 3103.5, "end": 3106.5, "text": " number of patches is and it's going to be 224."}, {"start": 3106.5, "end": 3109.82, "text": " We divide that by what by 32."}, {"start": 3109.82, "end": 3112.3, "text": " That's going to give us I think 16."}, {"start": 3112.3, "end": 3117.46, "text": " So we're going to end up with 16 by 16 of these 768 vectors."}, {"start": 3117.46, "end": 3125.94, "text": " So we're going to end up with like 16, 16, 768 volumes here."}, {"start": 3125.94, "end": 3129.34, "text": " So let's kind of verify that hypothesis."}, {"start": 3129.34, "end": 3132.7000000000003, "text": " Well we'll get to that a bit later but that's what this layer is doing."}, {"start": 3132.7000000000003, "end": 3134.98, "text": " Hopefully this helped."}, {"start": 3134.98, "end": 3135.98, "text": " Let's continue."}, {"start": 3135.98, "end": 3140.7400000000002, "text": " So if I hit f10 we just form this scale."}, {"start": 3140.74, "end": 3143.66, "text": " This is just for normalization."}, {"start": 3143.66, "end": 3146.3399999999997, "text": " Basically what we do here is we form the CLS embedding."}, {"start": 3146.3399999999997, "end": 3149.9399999999996, "text": " So that's the CLS token of our vision transformer."}, {"start": 3149.9399999999996, "end": 3154.4399999999996, "text": " That's something that BERT is also using and many other transformer models."}, {"start": 3154.4399999999996, "end": 3160.9399999999996, "text": " And we are basically going to it's going to be a 768 dimensional vector."}, {"start": 3160.9399999999996, "end": 3163.6, "text": " We form the positional embeddings."}, {"start": 3163.6, "end": 3165.7799999999997, "text": " It's going to be of the size."}, {"start": 3165.7799999999997, "end": 3168.6, "text": " So basically 16 squared."}, {"start": 3168.6, "end": 3170.74, "text": " That's the number of tokens we'll have."}, {"start": 3170.74, "end": 3173.66, "text": " So that's like 256 right?"}, {"start": 3173.66, "end": 3175.86, "text": " And plus one because we have CLS."}, {"start": 3175.86, "end": 3181.12, "text": " And then the width is 768 because we want to basically add those positional encodings"}, {"start": 3181.12, "end": 3185.7999999999997, "text": " on top of each of the embedding vectors."}, {"start": 3185.7999999999997, "end": 3191.8199999999997, "text": " So if you're confused by the means is basically let me again go back here."}, {"start": 3191.8199999999997, "end": 3197.38, "text": " So what it does is it's going to construct it's going to construct embedding table here."}, {"start": 3197.38, "end": 3200.94, "text": " So we'll have an embedding table here."}, {"start": 3200.94, "end": 3209.54, "text": " The dimensionality will be again 16 times 16 plus one for all of these vectors here."}, {"start": 3209.54, "end": 3213.7200000000003, "text": " So let me kind of try and draw them some somehow here."}, {"start": 3213.7200000000003, "end": 3219.5, "text": " So this is going to be our output volume 16 16."}, {"start": 3219.5, "end": 3224.42, "text": " And this is going to be this dimensionality here 768."}, {"start": 3224.42, "end": 3225.7000000000003, "text": " Okay."}, {"start": 3225.7, "end": 3229.1, "text": " And we'll have corresponding embedding vectors."}, {"start": 3229.1, "end": 3234.14, "text": " And now we'll basically be learning these and then adding them on top of these ones."}, {"start": 3234.14, "end": 3240.54, "text": " So basically what will happen is that this vector here, this position vector here will"}, {"start": 3240.54, "end": 3242.54, "text": " always be added to this vector here."}, {"start": 3242.54, "end": 3243.9399999999996, "text": " So to the first vector here."}, {"start": 3243.9399999999996, "end": 3247.46, "text": " So we take this one and we combine it."}, {"start": 3247.46, "end": 3250.1, "text": " We'll sum it up."}, {"start": 3250.1, "end": 3253.4199999999996, "text": " We'll do the summation between those two."}, {"start": 3253.42, "end": 3256.02, "text": " And that's going to happen in the forward pass of the VAT."}, {"start": 3256.02, "end": 3257.02, "text": " Okay."}, {"start": 3257.02, "end": 3258.98, "text": " Let's go back to the code."}, {"start": 3258.98, "end": 3260.2200000000003, "text": " That's that."}, {"start": 3260.2200000000003, "end": 3261.54, "text": " We have some layer norms."}, {"start": 3261.54, "end": 3263.06, "text": " We form some transformers."}, {"start": 3263.06, "end": 3269.58, "text": " And then yeah, so because transformer will also be used in this second, I dubbed it as"}, {"start": 3269.58, "end": 3271.04, "text": " a textual pathway."}, {"start": 3271.04, "end": 3275.1, "text": " Not sure whether anyone else is using that terminology, but like let me kind of just"}, {"start": 3275.1, "end": 3277.2000000000003, "text": " quickly scheme over what transformer is doing."}, {"start": 3277.2000000000003, "end": 3282.34, "text": " So if I click this, we'll get to the transformer logic."}, {"start": 3282.34, "end": 3285.78, "text": " So what transformer does is fairly simple."}, {"start": 3285.78, "end": 3289.54, "text": " We just specify the width, which is going to be also 768."}, {"start": 3289.54, "end": 3290.9, "text": " We have number of layers."}, {"start": 3290.9, "end": 3292.48, "text": " We just form a sequential layer."}, {"start": 3292.48, "end": 3296.02, "text": " So we're going to form these residual attention blocks."}, {"start": 3296.02, "end": 3300.46, "text": " And what they do is, I mean, your regular transformer logic."}, {"start": 3300.46, "end": 3302.58, "text": " So let me show you what it does."}, {"start": 3302.58, "end": 3305.6200000000003, "text": " We have the multi-head attention head."}, {"start": 3305.6200000000003, "end": 3308.6000000000004, "text": " Wait, multi-head attention layer."}, {"start": 3308.6000000000004, "end": 3310.7000000000003, "text": " And then we have the layer norm."}, {"start": 3310.7000000000003, "end": 3311.7000000000003, "text": " And then we have the MLP."}, {"start": 3311.7, "end": 3315.8999999999996, "text": " So we have the linear, followed by Jalu, followed by linear."}, {"start": 3315.8999999999996, "end": 3320.62, "text": " And you can see this is your regular expansion ratio of four."}, {"start": 3320.62, "end": 3325.74, "text": " Usually the innermost part of that MLP in that particular transformer layer is usually"}, {"start": 3325.74, "end": 3329.02, "text": " 4x bigger than the input part."}, {"start": 3329.02, "end": 3330.02, "text": " Okay?"}, {"start": 3330.02, "end": 3335.58, "text": " And so then we construct the second LN layer and we optionally have attention mask in case"}, {"start": 3335.58, "end": 3338.8199999999997, "text": " we want to have a causal transformer model."}, {"start": 3338.82, "end": 3344.7000000000003, "text": " So the vision transformer will be using non-causal transformer model because you want to attend"}, {"start": 3344.7000000000003, "end": 3348.9, "text": " each token, each patch needs to attend to each other patch."}, {"start": 3348.9, "end": 3353.5, "text": " And in the case of a textual transformer, we'll be using a causal one."}, {"start": 3353.5, "end": 3355.1800000000003, "text": " So that's why we have this attention part."}, {"start": 3355.1800000000003, "end": 3364.42, "text": " And now what the forward pass of the transformer does is basically applies the attention."}, {"start": 3364.42, "end": 3369.54, "text": " So sorry, this is just a particular layer of the transformer, not the whole transformer."}, {"start": 3369.54, "end": 3376.9, "text": " So it does the attention and then just basically, okay, of course, layer norms, attention, residual"}, {"start": 3376.9, "end": 3379.54, "text": " connections, and then there is the MLP part."}, {"start": 3379.54, "end": 3382.02, "text": " So that's your transformer logic here."}, {"start": 3382.02, "end": 3388.86, "text": " And finally, we just kind of do a forward pass through all of the 12 of these transformer"}, {"start": 3388.86, "end": 3389.86, "text": " layers."}, {"start": 3389.86, "end": 3390.86, "text": " And that's your transformer model."}, {"start": 3390.86, "end": 3391.86, "text": " Okay?"}, {"start": 3391.86, "end": 3395.98, "text": " So that's a quick scheme, like a quick scheme over what's going on there."}, {"start": 3395.98, "end": 3399.7400000000002, "text": " Anyways, let's proceed here."}, {"start": 3399.7400000000002, "end": 3402.78, "text": " I'm going to ignore all of this and we are back to vision transformer."}, {"start": 3402.78, "end": 3404.82, "text": " We form all of this."}, {"start": 3404.82, "end": 3409.38, "text": " And finally, we have the projection layer that's going to project from 768, which is"}, {"start": 3409.38, "end": 3413.7400000000002, "text": " the inner model dimension into our desired output dimension, which is 512."}, {"start": 3413.7400000000002, "end": 3414.7400000000002, "text": " That's it."}, {"start": 3414.7400000000002, "end": 3419.1, "text": " I'm going to now hit run and get to our next step."}, {"start": 3419.1, "end": 3423.58, "text": " So this is another point I wanted to make."}, {"start": 3423.58, "end": 3430.14, "text": " Basically, this is the attention mask that's going to be formed for the causal transformers."}, {"start": 3430.14, "end": 3436.42, "text": " If I hit step over and if I print the mask here, let's see what we get."}, {"start": 3436.42, "end": 3442.54, "text": " So as you can see here is that your classical mask that's used to create a causal attention"}, {"start": 3442.54, "end": 3443.54, "text": " pattern."}, {"start": 3443.54, "end": 3444.54, "text": " Okay?"}, {"start": 3444.54, "end": 3445.54, "text": " Let's continue."}, {"start": 3445.54, "end": 3446.54, "text": " Okay."}, {"start": 3446.54, "end": 3451.9, "text": " So we're back in the clip in this function before we created the visual parts."}, {"start": 3451.9, "end": 3453.34, "text": " That's the vision transformer."}, {"start": 3453.34, "end": 3457.46, "text": " And now we just create the transformer here."}, {"start": 3457.46, "end": 3463.46, "text": " So as you see here, this particular transformer is going to have the build attention mask."}, {"start": 3463.46, "end": 3467.98, "text": " It's going to be a causal one, whereas we do not pass the same argument to the vision"}, {"start": 3467.98, "end": 3468.98, "text": " transformer."}, {"start": 3468.98, "end": 3469.98, "text": " So that one's going to be non-causal."}, {"start": 3469.98, "end": 3471.1, "text": " Okay?"}, {"start": 3471.1, "end": 3472.38, "text": " And then we have the vocab size."}, {"start": 3472.38, "end": 3473.7, "text": " So that's the token."}, {"start": 3473.7, "end": 3478.9399999999996, "text": " So we saw that this is the BPE vocab we just created."}, {"start": 3478.9399999999996, "end": 3485.5, "text": " And we're going to obviously create a token embedding table, which is going to be used"}, {"start": 3485.5, "end": 3493.74, "text": " by the transformer, by the textual pathway to basically embed the tokenized text."}, {"start": 3493.74, "end": 3494.7799999999997, "text": " Okay?"}, {"start": 3494.7799999999997, "end": 3500.7999999999997, "text": " And we'll also have a positional embedding also for the transformer."}, {"start": 3500.8, "end": 3504.6400000000003, "text": " And that's going to be 77 transformer width is 5."}, {"start": 3504.6400000000003, "end": 3505.6400000000003, "text": " Okay."}, {"start": 3505.6400000000003, "end": 3508.9, "text": " So that's going to be the positional embeddings."}, {"start": 3508.9, "end": 3512.94, "text": " So again, it may be quickly worth going through that again here."}, {"start": 3512.94, "end": 3513.94, "text": " So that was vision transformer."}, {"start": 3513.94, "end": 3516.78, "text": " Now let me show you how this thing will work."}, {"start": 3516.78, "end": 3518.0600000000004, "text": " So we have vocab size."}, {"start": 3518.0600000000004, "end": 3520.44, "text": " We'll have a table."}, {"start": 3520.44, "end": 3523.3, "text": " So we'll have a table here."}, {"start": 3523.3, "end": 3529.78, "text": " That's going to be whatever the vocab size is, which is like 49k or something."}, {"start": 3529.78, "end": 3533.7400000000002, "text": " And we'll have this size being, so I forgot what it is."}, {"start": 3533.7400000000002, "end": 3535.78, "text": " Let me check what exactly that part is."}, {"start": 3535.78, "end": 3537.38, "text": " It's going to be transformer width."}, {"start": 3537.38, "end": 3538.38, "text": " Okay."}, {"start": 3538.38, "end": 3539.38, "text": " So it's going to be 512."}, {"start": 3539.38, "end": 3540.38, "text": " Okay."}, {"start": 3540.38, "end": 3541.38, "text": " Let's go back here."}, {"start": 3541.38, "end": 3544.1000000000004, "text": " That thing is going to be 512."}, {"start": 3544.1000000000004, "end": 3548.26, "text": " And then we have additional table that's going to be like context length, which is going"}, {"start": 3548.26, "end": 3550.0600000000004, "text": " to be 77."}, {"start": 3550.0600000000004, "end": 3552.7400000000002, "text": " And then also it's going to be 512."}, {"start": 3552.7400000000002, "end": 3559.1400000000003, "text": " So those are going to be, this is the, basically the table we use to embed our tokens."}, {"start": 3559.14, "end": 3565.2599999999998, "text": " And this is the positional, the positional encoding table."}, {"start": 3565.2599999999998, "end": 3566.74, "text": " So what it means in practice is the following."}, {"start": 3566.74, "end": 3568.94, "text": " So imagine you have a piece of text."}, {"start": 3568.94, "end": 3574.3399999999997, "text": " You take the text, you tokenize it, you convert it into a list of integers."}, {"start": 3574.3399999999997, "end": 3579.7, "text": " So let's imagine you started with like a hello world."}, {"start": 3579.7, "end": 3583.3399999999997, "text": " And then what you do is you first tokenize it."}, {"start": 3583.3399999999997, "end": 3586.94, "text": " So that means it's going to be somehow split."}, {"start": 3586.94, "end": 3588.2799999999997, "text": " We'll see that logic a bit later."}, {"start": 3588.28, "end": 3590.6200000000003, "text": " Maybe some like here, here, here."}, {"start": 3590.6200000000003, "end": 3592.6000000000004, "text": " And then we're going to somehow map it into integer."}, {"start": 3592.6000000000004, "end": 3596.02, "text": " So all of those tokens will have corresponding integers."}, {"start": 3596.02, "end": 3598.78, "text": " And then that means we're going to end up with something like this."}, {"start": 3598.78, "end": 3608.1400000000003, "text": " We'll end up with like, I don't know, like 23, 47, 56, 37."}, {"start": 3608.1400000000003, "end": 3616.42, "text": " And we now use this to index this embedding table to get the associated vector."}, {"start": 3616.42, "end": 3625.86, "text": " So 23 means, hey, let's grab token, let's grab the embedding vector that has index 23."}, {"start": 3625.86, "end": 3629.38, "text": " And that's going to be your representation for this particular token."}, {"start": 3629.38, "end": 3630.38, "text": " Okay."}, {"start": 3630.38, "end": 3633.06, "text": " Then we say, let's take this 47."}, {"start": 3633.06, "end": 3639.06, "text": " Let's take the, this token here and let's find its corresponding embedding vector."}, {"start": 3639.06, "end": 3640.3, "text": " So that's this one."}, {"start": 3640.3, "end": 3643.14, "text": " And then we put it here, et cetera, et cetera."}, {"start": 3643.14, "end": 3647.7, "text": " And then once we form this and remember we're going to pad this with all zeros."}, {"start": 3647.7, "end": 3650.9, "text": " So this is going to have bunch of zeros here."}, {"start": 3650.9, "end": 3657.2599999999998, "text": " Zero, zero, zero, zeros, because we want to have the size of this is going to be 77."}, {"start": 3657.2599999999998, "end": 3658.8199999999997, "text": " Whoops."}, {"start": 3658.8199999999997, "end": 3661.9, "text": " Sorry, let me redraw that."}, {"start": 3661.9, "end": 3665.3399999999997, "text": " That was some terrible drawing."}, {"start": 3665.3399999999997, "end": 3671.2599999999998, "text": " So if I do this, basically this whole thing is going to be 77."}, {"start": 3671.2599999999998, "end": 3673.02, "text": " And now we just add."}, {"start": 3673.02, "end": 3677.82, "text": " So now we're going to add these positionally, positional embeddings on top of these vectors"}, {"start": 3677.82, "end": 3678.82, "text": " here."}, {"start": 3678.82, "end": 3682.1, "text": " That's how this whole, this is how all of these structures come together."}, {"start": 3682.1, "end": 3683.54, "text": " Hopefully that kind of clarifies it."}, {"start": 3683.54, "end": 3684.54, "text": " Okay."}, {"start": 3684.54, "end": 3689.88, "text": " Let's go back to the text, to the code."}, {"start": 3689.88, "end": 3692.94, "text": " We formed the positional embedding table."}, {"start": 3692.94, "end": 3694.94, "text": " We formed the layer norm."}, {"start": 3694.94, "end": 3699.62, "text": " We finally have the text projection, which is going to be used to transform from the"}, {"start": 3699.62, "end": 3703.2599999999998, "text": " transformer width into the embedding width, which happened to have the same dimensionality."}, {"start": 3703.2599999999998, "end": 3709.2599999999998, "text": " So this transformer is actually using internally 512, not 768 as this vision transformer."}, {"start": 3709.2599999999998, "end": 3710.7799999999997, "text": " Okay."}, {"start": 3710.7799999999997, "end": 3712.22, "text": " And there is this allotted scale."}, {"start": 3712.22, "end": 3717.9, "text": " I'm not sure why these exact numbers kind of seems like, yeah, magic numbers, but yeah."}, {"start": 3717.9, "end": 3720.2999999999997, "text": " And then just some initialization and that's it."}, {"start": 3720.2999999999997, "end": 3721.2999999999997, "text": " Okay."}, {"start": 3721.2999999999997, "end": 3722.48, "text": " Let me click F5."}, {"start": 3722.48, "end": 3726.22, "text": " Let's get to this next salient break point."}, {"start": 3726.22, "end": 3727.22, "text": " And that's here."}, {"start": 3727.22, "end": 3728.22, "text": " Okay."}, {"start": 3728.22, "end": 3732.66, "text": " So we made the image, which is going to be just this clip image, just some random diagram."}, {"start": 3732.66, "end": 3739.06, "text": " We're going to do the pre-processing and basically we end up with a batch of size one."}, {"start": 3739.06, "end": 3746.22, "text": " So if I were to print the image shape, it's going to be probably, okay, before we even"}, {"start": 3746.22, "end": 3748.14, "text": " do it, let's kind of do the predictions."}, {"start": 3748.14, "end": 3752.7799999999997, "text": " It's going to be one, I guess, three, two, two, four, two, two, four, because this is"}, {"start": 3752.7799999999997, "end": 3754.4199999999996, "text": " the format that PyTorch usually does."}, {"start": 3754.42, "end": 3759.02, "text": " So you have this channel dimension here instead of here."}, {"start": 3759.02, "end": 3762.9, "text": " And if I print the shape, we're going to get, yeah, exactly that."}, {"start": 3762.9, "end": 3763.9, "text": " Okay."}, {"start": 3763.9, "end": 3769.1800000000003, "text": " So next up, we need to do the tokenization of these captions here."}, {"start": 3769.1800000000003, "end": 3771.9, "text": " So a diagram, a dog, and a cat."}, {"start": 3771.9, "end": 3774.54, "text": " And let me quickly recap what we've seen so far."}, {"start": 3774.54, "end": 3777.32, "text": " I showed you how the clip is constructed."}, {"start": 3777.32, "end": 3784.94, "text": " We saw how the vision transformer and the transformer models are instantiated and created."}, {"start": 3784.94, "end": 3787.54, "text": " We've seen the image pre-processing here."}, {"start": 3787.54, "end": 3792.98, "text": " And finally, let's kind of jump into this part here with the tokenization."}, {"start": 3792.98, "end": 3795.0800000000004, "text": " So let me click F5."}, {"start": 3795.0800000000004, "end": 3799.02, "text": " So we pass our text here and we need to tokenize them."}, {"start": 3799.02, "end": 3805.38, "text": " And when I say tokenize, we literally need to transform to basically split these captions"}, {"start": 3805.38, "end": 3813.1400000000003, "text": " into sub words and then map those into corresponding integers, depending on our vocab mapping."}, {"start": 3813.1400000000003, "end": 3814.1400000000003, "text": " Okay."}, {"start": 3814.1400000000003, "end": 3816.98, "text": " So let me kind of go through this."}, {"start": 3816.98, "end": 3825.5, "text": " First of all, we want to tokenize these special tokens because as you can see here, whatever"}, {"start": 3825.5, "end": 3827.54, "text": " the text is, we'll be trading through our captions."}, {"start": 3827.54, "end": 3828.6600000000003, "text": " We're going to encode them."}, {"start": 3828.6600000000003, "end": 3831.7000000000003, "text": " So this is going to give us a list of integers."}, {"start": 3831.7, "end": 3837.9399999999996, "text": " And then we basically put the start of text token in the beginning and end of text token"}, {"start": 3837.9399999999996, "end": 3839.6, "text": " at the end."}, {"start": 3839.6, "end": 3843.66, "text": " And now let's kind of jump into the actual encode function."}, {"start": 3843.66, "end": 3847.62, "text": " What it does is, well, fairly complicated."}, {"start": 3847.62, "end": 3854.1, "text": " I have to say the tokenization part, even though it's always abstracted away from us"}, {"start": 3854.1, "end": 3859.3399999999997, "text": " when you read the papers, et cetera, et cetera, I found this to be the hardest part of this"}, {"start": 3859.34, "end": 3863.58, "text": " whole code base for understanding."}, {"start": 3863.58, "end": 3866.94, "text": " And yeah, just like the level of details is amazing."}, {"start": 3866.94, "end": 3870.82, "text": " Having said that, let's go through this."}, {"start": 3870.82, "end": 3871.82, "text": " Okay."}, {"start": 3871.82, "end": 3874.1800000000003, "text": " So this line is basically going to clean the text."}, {"start": 3874.1800000000003, "end": 3877.1800000000003, "text": " I won't go into the details of the actual cleaning."}, {"start": 3877.1800000000003, "end": 3880.06, "text": " They're using some external libraries to do that."}, {"start": 3880.06, "end": 3883.3, "text": " And then they just lowercase the whole caption."}, {"start": 3883.3, "end": 3886.06, "text": " And that's the whole processing that's going on here."}, {"start": 3886.06, "end": 3889.02, "text": " Next up, they are using this pattern."}, {"start": 3889.02, "end": 3896.02, "text": " So this regex expression they compiled in the tokenizer to find the next piece of text"}, {"start": 3896.02, "end": 3900.42, "text": " from the caption, which I don't fully understand."}, {"start": 3900.42, "end": 3904.54, "text": " If somebody understands the exact details of why this particular..."}, {"start": 3904.54, "end": 3906.18, "text": " So let me show you the pattern again."}, {"start": 3906.18, "end": 3912.78, "text": " So here's the pattern, fairly complicated, not sure why all of these details."}, {"start": 3912.78, "end": 3915.22, "text": " There is no comments, so it's kind of dark magic."}, {"start": 3915.22, "end": 3916.82, "text": " But yeah, we'll have to roll with it."}, {"start": 3916.82, "end": 3919.54, "text": " We can kind of treat it as a black box for now."}, {"start": 3919.54, "end": 3920.6200000000003, "text": " Okay."}, {"start": 3920.6200000000003, "end": 3929.6200000000003, "text": " So what happens here is the token is encoded using the UTF-8 encoding, which for this simple..."}, {"start": 3929.6200000000003, "end": 3935.2400000000002, "text": " So currently token is this small letter A, which is going to be encoded as a single byte."}, {"start": 3935.2400000000002, "end": 3941.2200000000003, "text": " But some of the characters might be encoded into two, three, or four bytes."}, {"start": 3941.2200000000003, "end": 3942.7000000000003, "text": " That's how UTF-8 works."}, {"start": 3942.7, "end": 3947.14, "text": " Well, I'm not sure about the three, whether there are characters that have three bytes."}, {"start": 3947.14, "end": 3949.5, "text": " Although, yeah, I'm fairly sure that's the case."}, {"start": 3949.5, "end": 3953.8199999999997, "text": " I'm going to show you a bit later when you use emoji, this is going to give us four bytes,"}, {"start": 3953.8199999999997, "end": 3955.3799999999997, "text": " the encoding into UTF-8."}, {"start": 3955.3799999999997, "end": 3958.7799999999997, "text": " But for this simple example, it's going to be a single byte."}, {"start": 3958.7799999999997, "end": 3964.18, "text": " And then we use this dictionary byte encoder to fetch the corresponding..."}, {"start": 3964.18, "end": 3966.9399999999996, "text": " Basically to catch the corresponding Unicode character."}, {"start": 3966.9399999999996, "end": 3970.3799999999997, "text": " So if I do F10 here, we're going to end up with this..."}, {"start": 3970.38, "end": 3973.9, "text": " Again, this is identity mapping for this particular token."}, {"start": 3973.9, "end": 3979.06, "text": " We'll see later that it's going to be a bit more complicated for some other Unicode characters."}, {"start": 3979.06, "end": 3982.1800000000003, "text": " So what happens now is the BPE magic."}, {"start": 3982.1800000000003, "end": 3985.1, "text": " It's going to basically give us back..."}, {"start": 3985.1, "end": 3986.1, "text": " It's going to..."}, {"start": 3986.1, "end": 3990.5, "text": " In the general case, it's going to split this token into BPE tokens."}, {"start": 3990.5, "end": 3995.04, "text": " In this case, it's going to be small a, again, just trivial."}, {"start": 3995.04, "end": 3998.1400000000003, "text": " And then we're going to encode that using the encoder."}, {"start": 3998.14, "end": 4001.74, "text": " So again, remember encoder just contains our vocabulary."}, {"start": 4001.74, "end": 4010.7799999999997, "text": " The keys are the tokens and the values are the associated IDs, I guess."}, {"start": 4010.7799999999997, "end": 4015.42, "text": " If I click F10 here, we just end up with BPE tokens being..."}, {"start": 4015.42, "end": 4019.14, "text": " As you can see here, we just encode 320."}, {"start": 4019.14, "end": 4023.2999999999997, "text": " That's going to be the ID that corresponds to small letter a."}, {"start": 4023.2999999999997, "end": 4027.42, "text": " If I continue the loop here, we get diagram."}, {"start": 4027.42, "end": 4030.98, "text": " Again, this is going to be encoded."}, {"start": 4030.98, "end": 4034.7000000000003, "text": " And finally, we're going to end up with, I think..."}, {"start": 4034.7000000000003, "end": 4036.58, "text": " Yeah, again, just a single..."}, {"start": 4036.58, "end": 4041.58, "text": " There is a single token that corresponds to diagram in our vocabulary."}, {"start": 4041.58, "end": 4042.62, "text": " And that's going to be it."}, {"start": 4042.62, "end": 4049.02, "text": " So now we basically repeat this whole procedure for the other captions."}, {"start": 4049.02, "end": 4053.66, "text": " And let me kind of skip this and click F5."}, {"start": 4053.66, "end": 4056.1800000000003, "text": " So let me go over this."}, {"start": 4056.1800000000003, "end": 4057.1800000000003, "text": " And here we are."}, {"start": 4057.18, "end": 4060.58, "text": " So we end up with all tokens being these realists."}, {"start": 4060.58, "end": 4066.18, "text": " So as you can see, it's always a beginning and ending with a start and end of text tokens."}, {"start": 4066.18, "end": 4072.74, "text": " And then in between is the tokenization that we get from our BPE tokenizer."}, {"start": 4072.74, "end": 4076.3399999999997, "text": " Okay, let's continue here."}, {"start": 4076.3399999999997, "end": 4080.46, "text": " So again, if you didn't quite understand everything that's going on here, don't worry because"}, {"start": 4080.46, "end": 4083.4199999999996, "text": " I did not also understand every single detail, to be honest."}, {"start": 4083.4199999999996, "end": 4085.3999999999996, "text": " Okay, let's go on here."}, {"start": 4085.4, "end": 4092.02, "text": " So what we do is we just pre-allocate a tensor of a corresponding..."}, {"start": 4092.02, "end": 4095.14, "text": " Of a following shape."}, {"start": 4095.14, "end": 4100.64, "text": " Basically we want to have the context length, 77 here, and then we want to have whatever"}, {"start": 4100.64, "end": 4104.9400000000005, "text": " the number of captions is along the zeroth axis."}, {"start": 4104.9400000000005, "end": 4110.3, "text": " So we end up with a tensor that's like, I guess, 377."}, {"start": 4110.3, "end": 4113.5, "text": " Okay, let me go click F10."}, {"start": 4113.5, "end": 4120.86, "text": " So just for sanity here, the shape is going to be 377."}, {"start": 4120.86, "end": 4122.1, "text": " And here we're just going to fill it in."}, {"start": 4122.1, "end": 4128.32, "text": " Basically we're just going to fill in the tokens that we got from the encoding here,"}, {"start": 4128.32, "end": 4131.3, "text": " and we're going to fill into this tensor."}, {"start": 4131.3, "end": 4136.42, "text": " So I'm going to skip over all of these parts, and let me show you how the resulting tensor"}, {"start": 4136.42, "end": 4137.42, "text": " looks like."}, {"start": 4137.42, "end": 4144.86, "text": " Basically, as you can see here, it begins with these IDs for our tokens, and then it's"}, {"start": 4144.86, "end": 4151.9, "text": " just like padded with zeros until we have the full 77 tokens worth of context."}, {"start": 4151.9, "end": 4152.9, "text": " That's it."}, {"start": 4152.9, "end": 4155.34, "text": " Let me step over here and let's continue."}, {"start": 4155.34, "end": 4156.34, "text": " So that was a tokenization."}, {"start": 4156.34, "end": 4159.46, "text": " We ended up with a tensor here."}, {"start": 4159.46, "end": 4163.06, "text": " Again, the shape is 377."}, {"start": 4163.06, "end": 4165.1, "text": " We have an image."}, {"start": 4165.1, "end": 4166.46, "text": " The image is like this."}, {"start": 4166.46, "end": 4169.78, "text": " And now we want to encode the image and we want to encode the text."}, {"start": 4169.78, "end": 4175.46, "text": " And that means passing them through the visual transformer and through the transformer respectively."}, {"start": 4175.46, "end": 4178.94, "text": " So let me show you how that is going to work."}, {"start": 4178.94, "end": 4180.94, "text": " Here I think we're going to go through the..."}, {"start": 4180.94, "end": 4184.06, "text": " So visual is again a vision transformer."}, {"start": 4184.06, "end": 4187.34, "text": " There is some type of conversion going on here."}, {"start": 4187.34, "end": 4192.22, "text": " Basically we want to make sure that whatever the weights of our com1, so that's the patchify"}, {"start": 4192.22, "end": 4196.820000000001, "text": " there, whatever the type of that thing is, we want to convert our image into the same"}, {"start": 4196.820000000001, "end": 4197.820000000001, "text": " type."}, {"start": 4197.820000000001, "end": 4200.9800000000005, "text": " So whether it's float32 or float16."}, {"start": 4200.9800000000005, "end": 4204.740000000001, "text": " In this case, I'm fairly sure it's going to be float32."}, {"start": 4204.740000000001, "end": 4206.46, "text": " Not that important, but let me just kind of see."}, {"start": 4206.46, "end": 4207.820000000001, "text": " Okay, it's actually float16."}, {"start": 4207.820000000001, "end": 4216.58, "text": " Yeah, because they have during the model loading, they have this conversion into FP16 function"}, {"start": 4216.58, "end": 4217.58, "text": " being called."}, {"start": 4217.58, "end": 4218.820000000001, "text": " So that explains it."}, {"start": 4218.820000000001, "end": 4222.14, "text": " Anyway, so let me go to the forward pass."}, {"start": 4222.14, "end": 4228.34, "text": " Again, what's going to happen here is the logic I explained in my OneNote."}, {"start": 4228.34, "end": 4230.62, "text": " Let me remind you if you forgot about that."}, {"start": 4230.62, "end": 4232.820000000001, "text": " So this is the logic that's going to happen."}, {"start": 4232.820000000001, "end": 4236.5, "text": " Basically we're going to patchify and extract these vectors and we're going to end up with"}, {"start": 4236.5, "end": 4239.42, "text": " like I think 16, 16, 7, 68."}, {"start": 4239.42, "end": 4241.700000000001, "text": " That should be the tensor we end up with."}, {"start": 4241.700000000001, "end": 4250.14, "text": " So if I click F10, we're going to patchify our tensor and let's see what the shape is."}, {"start": 4250.14, "end": 4253.62, "text": " Or maybe we'll have additionally prepended one because we have a batch of one."}, {"start": 4253.62, "end": 4260.06, "text": " So yeah, so it's basically one, whoops, it's one, 7, 68, 7, 7."}, {"start": 4260.06, "end": 4266.18, "text": " So there is some basically permutation going on, but we kind of got the shape right."}, {"start": 4266.18, "end": 4269.9400000000005, "text": " Okay, so now there is the reshaping going on here."}, {"start": 4269.9400000000005, "end": 4274.820000000001, "text": " We're going to, let me just kind of step through all of this, not explain all of the details,"}, {"start": 4274.820000000001, "end": 4277.62, "text": " just some permutation and reshape going on."}, {"start": 4277.62, "end": 4281.38, "text": " So we now just flatten the 7, 7 so we end up with all of the tokens being flattened"}, {"start": 4281.38, "end": 4282.38, "text": " out."}, {"start": 4282.38, "end": 4284.66, "text": " So this is the batch size, the number of tokens."}, {"start": 4284.66, "end": 4287.46, "text": " I don't know why it's expanding this constantly."}, {"start": 4287.46, "end": 4293.82, "text": " Okay, so batch size, number of tokens, and finally the size of the embedding vector."}, {"start": 4293.82, "end": 4294.82, "text": " Cool."}, {"start": 4294.82, "end": 4296.22, "text": " So far so good."}, {"start": 4296.22, "end": 4302.38, "text": " What we do here is we now take the class token, which is a learnable embedding vector, and"}, {"start": 4302.38, "end": 4310.1, "text": " we are going to basically concatenate it with this vector X here."}, {"start": 4310.1, "end": 4313.06, "text": " Okay, so let me show you what I mean by that."}, {"start": 4313.06, "end": 4320.1, "text": " So if I take the self class embedding, this is going to be just, the shape should be 1,"}, {"start": 4320.1, "end": 4322.14, "text": " 7, 6, 8, I think."}, {"start": 4322.14, "end": 4327.38, "text": " So the shape of this is, the shape is actually just 7, 6, 8."}, {"start": 4327.38, "end": 4334.1, "text": " And then once we sum it up with zeros, with a zero vector here, this is going to convert"}, {"start": 4334.1, "end": 4341.62, "text": " it into whatever the batch size is, and then 1, and then we'll end up with 7, 6, 8 here"}, {"start": 4341.62, "end": 4343.12, "text": " as well."}, {"start": 4343.12, "end": 4346.58, "text": " I guess this is just a way to broadcast the CLS token."}, {"start": 4346.58, "end": 4349.14, "text": " I can kind of copy paste this whole thing."}, {"start": 4349.14, "end": 4352.26, "text": " I'm going to store it into a temporary variable."}, {"start": 4352.26, "end": 4355.18, "text": " Oops, did I copy paste it?"}, {"start": 4355.18, "end": 4356.18, "text": " Yeah, I did."}, {"start": 4356.18, "end": 4357.860000000001, "text": " It just kind of overflowed."}, {"start": 4357.860000000001, "end": 4361.9400000000005, "text": " I can maybe remove this part with D type and device."}, {"start": 4361.9400000000005, "end": 4365.42, "text": " Let me see where it's going to work."}, {"start": 4365.42, "end": 4366.42, "text": " Maybe or maybe not."}, {"start": 4366.42, "end": 4369.54, "text": " I think, yeah, it's important to actually preserve this part."}, {"start": 4369.54, "end": 4372.34, "text": " So I'm going to add, do it like this."}, {"start": 4372.34, "end": 4376.66, "text": " TMP and TMP shape is going to be 1, 1, 7, 6, 8."}, {"start": 4376.66, "end": 4383.3, "text": " So basically we've done broadcasting, and then once you concatenate that with the X,"}, {"start": 4383.3, "end": 4394.74, "text": " which is of shape 1, 4, 9, 7, 6, 8, what is that going to do is let me think."}, {"start": 4394.74, "end": 4405.46, "text": " Yeah, basically we're going to end up with 1, 5, 7, 6, 8, because we're going to concatenate"}, {"start": 4405.46, "end": 4409.78, "text": " the CLS to prepend the CLS to our token sequence."}, {"start": 4409.78, "end": 4410.9400000000005, "text": " That should be the case here."}, {"start": 4410.94, "end": 4417.139999999999, "text": " So if I click F10, we end up with X shape 1, 5, 7, 6, 8, as I said."}, {"start": 4417.139999999999, "end": 4421.219999999999, "text": " And then we just add the positional embeddings."}, {"start": 4421.219999999999, "end": 4424.099999999999, "text": " So that's this step here."}, {"start": 4424.099999999999, "end": 4426.139999999999, "text": " Oops, let me show you that."}, {"start": 4426.139999999999, "end": 4430.94, "text": " So that's going to be basically, let me just find it."}, {"start": 4430.94, "end": 4438.86, "text": " That's this part, adding the positional encodings onto the vectors there."}, {"start": 4438.86, "end": 4440.259999999999, "text": " Let me step over that."}, {"start": 4440.26, "end": 4445.34, "text": " Then there is some LNNorm going on, some permutation of the dimensions, because that's how this"}, {"start": 4445.34, "end": 4447.02, "text": " transformer expects it."}, {"start": 4447.02, "end": 4449.46, "text": " Then we just process basically these."}, {"start": 4449.46, "end": 4453.58, "text": " So now you can literally treat this as a simple NLP problem."}, {"start": 4453.58, "end": 4456.1, "text": " We have a bunch of tokens with the corresponding channel dimensions."}, {"start": 4456.1, "end": 4461.54, "text": " We just pass that through the transformer, and then we do the permutation again here."}, {"start": 4461.54, "end": 4462.54, "text": " And that's it."}, {"start": 4462.54, "end": 4465.34, "text": " So Vision Transformer is basically Patchify plus Transformer."}, {"start": 4465.34, "end": 4468.54, "text": " That's all there is to it."}, {"start": 4468.54, "end": 4475.78, "text": " And finally, we just want to project onto the common dimensionality of this joint multimodal"}, {"start": 4475.78, "end": 4479.04, "text": " space between the visual part and the textual part."}, {"start": 4479.04, "end": 4482.42, "text": " And that's why we have this self-projection."}, {"start": 4482.42, "end": 4486.94, "text": " So it's going to project us from basically 756 to 512."}, {"start": 4486.94, "end": 4489.74, "text": " So as you can see here, that's the corresponding dimensions."}, {"start": 4489.74, "end": 4492.66, "text": " And if we run this, we end up with what?"}, {"start": 4492.66, "end": 4497.98, "text": " We end up with a single vector that has 512 dimensions, because here we extracted some"}, {"start": 4497.98, "end": 4499.5, "text": " of the other dimensions."}, {"start": 4499.5, "end": 4505.0199999999995, "text": " So basically what happened is we took our image and we ended up with a single vector"}, {"start": 4505.0199999999995, "end": 4506.7, "text": " that has 512 dimensions."}, {"start": 4506.7, "end": 4509.0599999999995, "text": " Let's continue."}, {"start": 4509.0599999999995, "end": 4510.86, "text": " That's the encoding image function."}, {"start": 4510.86, "end": 4512.94, "text": " Now for the encoding text function."}, {"start": 4512.94, "end": 4516.299999999999, "text": " Again, let me know whether this is too much detail for you."}, {"start": 4516.299999999999, "end": 4518.5599999999995, "text": " Any feedback is super appreciated."}, {"start": 4518.5599999999995, "end": 4523.08, "text": " The first time I'm doing this, so it will be super useful to hear some feedback."}, {"start": 4523.08, "end": 4529.08, "text": " Embedding text, what's going to happen here is, so we have token embedding."}, {"start": 4529.08, "end": 4530.7, "text": " So let's see what's going to happen here."}, {"start": 4530.7, "end": 4539.42, "text": " Okay, so basically we are going to use the text tensor to index into our vocab, into"}, {"start": 4539.42, "end": 4541.22, "text": " our token embedding table."}, {"start": 4541.22, "end": 4542.22, "text": " Okay."}, {"start": 4542.22, "end": 4544.82, "text": " And let me remind you what this table is about."}, {"start": 4544.82, "end": 4548.74, "text": " So token embedding is just a vocab size transformer width."}, {"start": 4548.74, "end": 4552.34, "text": " So it's going to be just our basically embedding table."}, {"start": 4552.34, "end": 4561.34, "text": " If I enter here, if I run this code, we end up with bit size number of tokens."}, {"start": 4561.34, "end": 4565.74, "text": " Sorry, the size of the context, which is 77 and then 512."}, {"start": 4565.74, "end": 4567.76, "text": " So let me bring the shape."}, {"start": 4567.76, "end": 4570.4800000000005, "text": " So we started with this."}, {"start": 4570.4800000000005, "end": 4577.46, "text": " We started with 377 and after embedding all of those IDs, and I'm butchering the terminology"}, {"start": 4577.46, "end": 4578.860000000001, "text": " here, sorry for that."}, {"start": 4578.86, "end": 4584.62, "text": " I'm not used to explaining these types of codes out loud, but yeah."}, {"start": 4584.62, "end": 4591.38, "text": " Then we just add the self-positional embeddings, which are going to be of the shape 77, 512,"}, {"start": 4591.38, "end": 4592.66, "text": " I assume."}, {"start": 4592.66, "end": 4595.7, "text": " So let me remind myself here, 77, 512."}, {"start": 4595.7, "end": 4596.7, "text": " Yeah."}, {"start": 4596.7, "end": 4598.0, "text": " So we just add them up."}, {"start": 4598.0, "end": 4600.179999999999, "text": " So that's the step we saw again."}, {"start": 4600.179999999999, "end": 4604.62, "text": " We saw that in the one note here."}, {"start": 4604.62, "end": 4606.46, "text": " So basically that's this step."}, {"start": 4606.46, "end": 4609.9, "text": " You have this embedding table and you're going to just add it up."}, {"start": 4609.9, "end": 4611.38, "text": " You're just going to kind of put it here."}, {"start": 4611.38, "end": 4617.1, "text": " So after the embedding has happened, you just kind of align it with your tokens."}, {"start": 4617.1, "end": 4624.3, "text": " So it's going to be 77 and you just kind of add the corresponding vectors."}, {"start": 4624.3, "end": 4628.46, "text": " You just add these two, you add these two, et cetera, et cetera."}, {"start": 4628.46, "end": 4629.46, "text": " Okay."}, {"start": 4629.46, "end": 4630.46, "text": " That's what happens."}, {"start": 4630.46, "end": 4632.6, "text": " That's all."}, {"start": 4632.6, "end": 4634.02, "text": " Let me step over."}, {"start": 4634.02, "end": 4637.22, "text": " We do the, again, the permutation."}, {"start": 4637.22, "end": 4638.9400000000005, "text": " We pass through the transformer."}, {"start": 4638.9400000000005, "end": 4642.540000000001, "text": " Again, return back into the shape that's desired."}, {"start": 4642.540000000001, "end": 4645.38, "text": " Then we pass through the layer normalization."}, {"start": 4645.38, "end": 4650.06, "text": " And finally, there is this interesting piece of code."}, {"start": 4650.06, "end": 4659.02, "text": " So what's this going to do is extract those tokens that are above the end of text token."}, {"start": 4659.02, "end": 4660.5, "text": " Let me break that down for you."}, {"start": 4660.5, "end": 4662.740000000001, "text": " Again, let me show you what I mean by that."}, {"start": 4662.74, "end": 4671.86, "text": " So if I do text argmax, we're going to end up with basically a vector of threes."}, {"start": 4671.86, "end": 4672.86, "text": " Okay."}, {"start": 4672.86, "end": 4677.5599999999995, "text": " And that's because the text tensor itself."}, {"start": 4677.5599999999995, "end": 4680.099999999999, "text": " So we have zero, one, two, three."}, {"start": 4680.099999999999, "end": 4688.5599999999995, "text": " You can see that the end of text token has the highest associated ID and that's why this"}, {"start": 4688.5599999999995, "end": 4690.219999999999, "text": " argmax is going to work."}, {"start": 4690.22, "end": 4697.7, "text": " So basically what we do is we extract the location of where this end of text token is"}, {"start": 4697.7, "end": 4699.02, "text": " in our tensor."}, {"start": 4699.02, "end": 4704.26, "text": " And then we use those positions to extract the tokens from our vector x."}, {"start": 4704.26, "end": 4706.4800000000005, "text": " So let me again show you the shape of x."}, {"start": 4706.4800000000005, "end": 4709.46, "text": " So x is 3, 77, 5, 12."}, {"start": 4709.46, "end": 4716.18, "text": " And what we're going to do out of all of these 77 tokens, we extract just a single one."}, {"start": 4716.18, "end": 4717.18, "text": " And that's it."}, {"start": 4717.18, "end": 4723.1, "text": " And then we do the projection into the final 512 dimension."}, {"start": 4723.1, "end": 4728.3, "text": " So actually it's already in the correct dimension, but still they do this projection just to"}, {"start": 4728.3, "end": 4731.860000000001, "text": " increase the capacity of the model, I guess."}, {"start": 4731.860000000001, "end": 4737.02, "text": " Let me step over this and we end up with x shape being 3, 5, 12."}, {"start": 4737.02, "end": 4738.02, "text": " Okay."}, {"start": 4738.02, "end": 4739.02, "text": " So that's it."}, {"start": 4739.02, "end": 4745.820000000001, "text": " We passed in three captions and we got out three vectors that all have 512 dimensions."}, {"start": 4745.820000000001, "end": 4746.820000000001, "text": " That's everything we need."}, {"start": 4746.82, "end": 4752.62, "text": " We can just do a simple normalization and then do the dot product to get the similarity"}, {"start": 4752.62, "end": 4753.62, "text": " score."}, {"start": 4753.62, "end": 4754.62, "text": " Okay."}, {"start": 4754.62, "end": 4756.0599999999995, "text": " Let's go back here."}, {"start": 4756.0599999999995, "end": 4762.179999999999, "text": " Now this function is kind of silly because we will not be using this, but if we just"}, {"start": 4762.179999999999, "end": 4767.04, "text": " pass this model, if we just call the model, it's going to do exactly this."}, {"start": 4767.04, "end": 4768.34, "text": " So let me show you what I mean by that."}, {"start": 4768.34, "end": 4775.28, "text": " So if we go and if I click F5, so the for function again just does the encode image"}, {"start": 4775.28, "end": 4776.78, "text": " and encode text."}, {"start": 4776.78, "end": 4780.78, "text": " So I'm just going to skip over those two because we've seen it already."}, {"start": 4780.78, "end": 4782.58, "text": " So I'm just going to skip over all of that."}, {"start": 4782.58, "end": 4783.58, "text": " Whoops."}, {"start": 4783.58, "end": 4787.54, "text": " Let me just, my God, this was a mistake."}, {"start": 4787.54, "end": 4788.82, "text": " Let me go back here."}, {"start": 4788.82, "end": 4790.86, "text": " Let me go back here."}, {"start": 4790.86, "end": 4795.66, "text": " Let me just put a break point here and click F5, skip all of this."}, {"start": 4795.66, "end": 4796.66, "text": " Okay."}, {"start": 4796.66, "end": 4797.66, "text": " So we are here."}, {"start": 4797.66, "end": 4801.38, "text": " So we've done, we have, again, we have the image features, which is going to be 1, 5,"}, {"start": 4801.38, "end": 4802.38, "text": " 12."}, {"start": 4802.38, "end": 4803.38, "text": " That's our vector."}, {"start": 4803.38, "end": 4807.18, "text": " And then we're going to have text features, which is going to be 3, 5, 12."}, {"start": 4807.18, "end": 4810.4800000000005, "text": " So that's a three vectors for the three captions."}, {"start": 4810.4800000000005, "end": 4812.58, "text": " Now we do the normalization."}, {"start": 4812.58, "end": 4814.86, "text": " We've seen this in the Drupiter already."}, {"start": 4814.86, "end": 4818.3, "text": " So I'm just going to skip and assume you understand all of this."}, {"start": 4818.3, "end": 4824.18, "text": " And finally, we, we take this, this is a learnable parameter that we did."}, {"start": 4824.18, "end": 4828.7, "text": " I kind of wasn't sure what this magic number 0.7 is."}, {"start": 4828.7, "end": 4830.38, "text": " Let me show you what I mean by that."}, {"start": 4830.38, "end": 4831.9400000000005, "text": " So here is that logit scale."}, {"start": 4831.94, "end": 4835.58, "text": " They kind of make it learnable and they initialize it like this."}, {"start": 4835.58, "end": 4840.9, "text": " I'm not sure why this exact number, if anyone knows, feel free to comment down below."}, {"start": 4840.9, "end": 4849.62, "text": " I guess it doesn't matter that much, but probably knowing how these NLP things work, it probably"}, {"start": 4849.62, "end": 4850.62, "text": " matters a lot."}, {"start": 4850.62, "end": 4855.74, "text": " Like it's a, it's a, it's a difference between this, this training loss spiking or not a"}, {"start": 4855.74, "end": 4859.919999999999, "text": " single, a single change of a variable can do that."}, {"start": 4859.92, "end": 4863.1, "text": " So let's step over here."}, {"start": 4863.1, "end": 4872.38, "text": " And what we do is we find the similarity matrix between the image vectors and the text vectors."}, {"start": 4872.38, "end": 4876.84, "text": " And we just do the multiplication here because I guess they're going to pass it later on"}, {"start": 4876.84, "end": 4878.18, "text": " into soft max."}, {"start": 4878.18, "end": 4879.7, "text": " That's why they're doing this."}, {"start": 4879.7, "end": 4885.66, "text": " So in any case, we get the logits per image and logits per text, which you can get by"}, {"start": 4885.66, "end": 4891.82, "text": " just transposing so you don't have to do again, text features and do matrix multiplication"}, {"start": 4891.82, "end": 4893.26, "text": " with image features transpose."}, {"start": 4893.26, "end": 4895.26, "text": " You can just transpose result."}, {"start": 4895.26, "end": 4898.86, "text": " You can, you can kind of verify that that's going to be correct."}, {"start": 4898.86, "end": 4899.86, "text": " Okay."}, {"start": 4899.86, "end": 4901.0199999999995, "text": " Let's click F10."}, {"start": 4901.0199999999995, "end": 4902.46, "text": " So we get the logits."}, {"start": 4902.46, "end": 4907.58, "text": " And now, as I said, we do the soft max along the last dimension."}, {"start": 4907.58, "end": 4911.86, "text": " So dimension minus one, and that's going to give us the probabilities."}, {"start": 4911.86, "end": 4916.98, "text": " So logits, whoops, let me copy paste this."}, {"start": 4916.98, "end": 4920.58, "text": " This should be, I guess, one three shape or something."}, {"start": 4920.58, "end": 4921.58, "text": " Yep."}, {"start": 4921.58, "end": 4926.94, "text": " And then if I kind of just print it, so here are the results."}, {"start": 4926.94, "end": 4929.219999999999, "text": " As you can see, they are not normalized."}, {"start": 4929.219999999999, "end": 4936.58, "text": " So after doing soft max here, if I click F10 here, we end up with probabilities."}, {"start": 4936.58, "end": 4942.54, "text": " So probabilities, and we find that the caption number one or zero, depending on how you index"}, {"start": 4942.54, "end": 4948.78, "text": " your arrays, is going to be the one that corresponds to our image."}, {"start": 4948.78, "end": 4951.18, "text": " And we'll see that that makes a lot of sense."}, {"start": 4951.18, "end": 4956.78, "text": " So okay, I'm just going to exit this."}, {"start": 4956.78, "end": 4960.44, "text": " And okay, I think, yeah, this is actually the end of the program."}, {"start": 4960.44, "end": 4965.64, "text": " So we got the result that the zeroth caption is the best one."}, {"start": 4965.64, "end": 4966.64, "text": " Let's see what it is."}, {"start": 4966.64, "end": 4973.1, "text": " Well, because the zeroth caption is this caption here, it's a diagram."}, {"start": 4973.1, "end": 4975.18, "text": " And let me show you the corresponding image."}, {"start": 4975.18, "end": 4976.780000000001, "text": " I think it's somewhere here."}, {"start": 4976.780000000001, "end": 4978.9400000000005, "text": " So clip image, where are you?"}, {"start": 4978.9400000000005, "end": 4979.9400000000005, "text": " Where are you?"}, {"start": 4979.9400000000005, "end": 4981.18, "text": " Okay, clip image here."}, {"start": 4981.18, "end": 4988.26, "text": " So as you can see, it's a diagram of the clip infrastructure, of the clip architecture,"}, {"start": 4988.26, "end": 4991.18, "text": " and thus this makes a lot of sense."}, {"start": 4991.18, "end": 4994.0, "text": " Okay, guys, this was pretty much it."}, {"start": 4994.0, "end": 4996.62, "text": " This was a very, very long video."}, {"start": 4996.62, "end": 4999.4, "text": " I hope you found something useful out of it."}, {"start": 4999.4, "end": 5002.76, "text": " Before we wrap up, I just want to show you one more thing."}, {"start": 5002.76, "end": 5006.38, "text": " So there is one more cool function I found in the in the reading file, and that's the"}, {"start": 5006.38, "end": 5007.7, "text": " linear probe."}, {"start": 5007.7, "end": 5009.36, "text": " So everything is the same."}, {"start": 5009.36, "end": 5012.5, "text": " So what we do, we download the cipher 100."}, {"start": 5012.5, "end": 5018.78, "text": " What we then do is we would pass through the data set, and we encode all of the images"}, {"start": 5018.78, "end": 5020.54, "text": " into these feature vectors."}, {"start": 5020.54, "end": 5026.66, "text": " So basically, what that means is we end up with, at the end, we end up with a bunch of"}, {"start": 5026.66, "end": 5033.1, "text": " like training features and associated labels, and same for the test data set."}, {"start": 5033.1, "end": 5039.14, "text": " And then instead of doing this zero shop classifier, you can instead just train a simple logistic"}, {"start": 5039.14, "end": 5041.5, "text": " regression on top of the train feature."}, {"start": 5041.5, "end": 5044.14, "text": " So again, train features, let me show you the shape."}, {"start": 5044.14, "end": 5048.82, "text": " Shape is going to be whatever the number of data points in the training data set is, 512."}, {"start": 5048.82, "end": 5050.0199999999995, "text": " So this is the shape."}, {"start": 5050.02, "end": 5054.540000000001, "text": " And then because we have the data set and we have the associated labels, this is a supervised"}, {"start": 5054.540000000001, "end": 5059.3, "text": " data set will have basically an associated ground truth labels."}, {"start": 5059.3, "end": 5063.02, "text": " And using these, you can train a linear probe."}, {"start": 5063.02, "end": 5065.620000000001, "text": " So that's this classifier here."}, {"start": 5065.620000000001, "end": 5067.4400000000005, "text": " Instead of doing what?"}, {"start": 5067.4400000000005, "end": 5073.900000000001, "text": " Instead of doing the zero shot, where you would basically take the class, you would"}, {"start": 5073.900000000001, "end": 5077.5, "text": " embed it into the template, and then you would encode that."}, {"start": 5077.5, "end": 5079.84, "text": " And that's how you create those ad hoc classifiers."}, {"start": 5079.84, "end": 5083.6, "text": " If you recall what I showed you before here."}, {"start": 5083.6, "end": 5085.26, "text": " So that's basically this logic here."}, {"start": 5085.26, "end": 5092.900000000001, "text": " You could do zero shot, but obviously it's much better perf wise to do like a linear"}, {"start": 5092.900000000001, "end": 5093.900000000001, "text": " probe."}, {"start": 5093.900000000001, "end": 5097.38, "text": " Although again, you would you do have to train to get that."}, {"start": 5097.38, "end": 5099.5, "text": " Whereas zero shot obviously is zero shot."}, {"start": 5099.5, "end": 5101.7, "text": " Hence you don't have to do the training."}, {"start": 5101.7, "end": 5106.34, "text": " Okay, so these variables are just found on some validation set."}, {"start": 5106.34, "end": 5110.42, "text": " And then once you train the classifier, you can then see how it performs on your test"}, {"start": 5110.42, "end": 5114.34, "text": " features and report the accuracy."}, {"start": 5114.34, "end": 5115.34, "text": " That's it."}, {"start": 5115.34, "end": 5116.34, "text": " That's clip for you."}, {"start": 5116.34, "end": 5121.9400000000005, "text": " Let me try and maybe quickly recap what we've seen."}, {"start": 5121.9400000000005, "end": 5125.6, "text": " I'm going to walk you through the function here again."}, {"start": 5125.6, "end": 5130.02, "text": " We load the clip model, we saw that clip model consists out of the vision transformer and"}, {"start": 5130.02, "end": 5131.46, "text": " the transformer."}, {"start": 5131.46, "end": 5137.54, "text": " Aside from the training parameters, those models have these associated tables."}, {"start": 5137.54, "end": 5143.86, "text": " Basically it's going to the division transformer is going to have this associated position"}, {"start": 5143.86, "end": 5148.88, "text": " encoding table plus the CLS token that's going to be learnable."}, {"start": 5148.88, "end": 5155.18, "text": " Transformer is going to have this positional encoding table as well, as well as the vocab"}, {"start": 5155.18, "end": 5156.18, "text": " table."}, {"start": 5156.18, "end": 5163.9400000000005, "text": " So that's the one that we use to embed our tokens here or the well tokens or IDs."}, {"start": 5163.9400000000005, "end": 5170.42, "text": " We have the integers, so maybe better to refer to those as IDs, although people call those"}, {"start": 5170.42, "end": 5172.18, "text": " tokens as well."}, {"start": 5172.18, "end": 5174.34, "text": " And that's that part."}, {"start": 5174.34, "end": 5178.1, "text": " So after we create the clip, we can then do the tokenization."}, {"start": 5178.1, "end": 5179.9400000000005, "text": " And this is arguably the hardest part."}, {"start": 5179.9400000000005, "end": 5183.22, "text": " In my humble opinion, it's kind of dark magic."}, {"start": 5183.22, "end": 5186.56, "text": " You have to process this text and you have to decide how to kind of split it using those"}, {"start": 5186.56, "end": 5188.46, "text": " reg ex expressions."}, {"start": 5188.46, "end": 5196.14, "text": " And then after you do the reg ex expression, you use this PP to kind of split it into into"}, {"start": 5196.14, "end": 5197.360000000001, "text": " sub words."}, {"start": 5197.360000000001, "end": 5204.42, "text": " And after that, you have to do the encoding into the corresponding integers."}, {"start": 5204.42, "end": 5206.22, "text": " And that's how you get the text."}, {"start": 5206.22, "end": 5207.22, "text": " Okay."}, {"start": 5207.22, "end": 5211.860000000001, "text": " Finally, once we have those two, we can basically pass them through the vision transformer and"}, {"start": 5211.86, "end": 5216.5, "text": " we can pass them through our transformer and get associated vectors."}, {"start": 5216.5, "end": 5220.7, "text": " And then we can just do simple dot product after we normalize them to get the scores"}, {"start": 5220.7, "end": 5223.0599999999995, "text": " and thus similarity."}, {"start": 5223.0599999999995, "end": 5224.0599999999995, "text": " That's it."}, {"start": 5224.0599999999995, "end": 5225.98, "text": " I realized I want to show you one thing quickly."}, {"start": 5225.98, "end": 5228.78, "text": " As I said, this is the hardest part of this tokenization."}, {"start": 5228.78, "end": 5234.92, "text": " So I'm just going to add this emoji here and let's run the program again."}, {"start": 5234.92, "end": 5235.92, "text": " So let me focus."}, {"start": 5235.92, "end": 5239.339999999999, "text": " Let me just remove these breakpoints and let me run this again."}, {"start": 5239.34, "end": 5242.1, "text": " So let me debug the file."}, {"start": 5242.1, "end": 5245.22, "text": " Let me show you what happens when we pass the brain emoji."}, {"start": 5245.22, "end": 5246.3, "text": " Okay."}, {"start": 5246.3, "end": 5247.76, "text": " Here we are."}, {"start": 5247.76, "end": 5252.74, "text": " Let me go through the tokenize function again."}, {"start": 5252.74, "end": 5257.5, "text": " Again we encode the special tokens and this is where the magic happens."}, {"start": 5257.5, "end": 5262.02, "text": " So let me enter it into this function encode."}, {"start": 5262.02, "end": 5265.66, "text": " So this is the hardest part of the whole code base if you ask me."}, {"start": 5265.66, "end": 5269.08, "text": " I'm still trying to understand every detail here."}, {"start": 5269.08, "end": 5270.98, "text": " We do some cleaning."}, {"start": 5270.98, "end": 5271.98, "text": " Cleaning will not change this."}, {"start": 5271.98, "end": 5275.22, "text": " This captions will end up with the same caption here."}, {"start": 5275.22, "end": 5276.7, "text": " So we end up with the same caption."}, {"start": 5276.7, "end": 5278.14, "text": " Nothing changed."}, {"start": 5278.14, "end": 5281.5, "text": " Then we're going to first grab token A. So that's nothing interesting there."}, {"start": 5281.5, "end": 5282.98, "text": " I'm just going to skip that."}, {"start": 5282.98, "end": 5283.98, "text": " Finally this is the interesting part."}, {"start": 5283.98, "end": 5285.86, "text": " So now we have a token brain."}, {"start": 5285.86, "end": 5291.58, "text": " So this is emoji and this is your this is a legit Unicode character and there is a unique"}, {"start": 5291.58, "end": 5298.22, "text": " code point that Unicode assigns to each of the characters you can encounter wherever."}, {"start": 5298.22, "end": 5303.860000000001, "text": " So in whatever language you always have a Unicode is this this very nice."}, {"start": 5303.860000000001, "end": 5310.22, "text": " Well how would they call it like encoding principle to to map each of the characters Unicode characters"}, {"start": 5310.22, "end": 5313.9400000000005, "text": " onto unique code points which is just a unique integer."}, {"start": 5313.9400000000005, "end": 5315.02, "text": " Okay."}, {"start": 5315.02, "end": 5319.9400000000005, "text": " And so what happens here is once you encode this particular token you're going to end"}, {"start": 5319.9400000000005, "end": 5321.1, "text": " up with four bytes."}, {"start": 5321.1, "end": 5325.38, "text": " So this particular token once you encoded using UTF eight it's going to give you four"}, {"start": 5325.38, "end": 5326.38, "text": " bytes."}, {"start": 5326.38, "end": 5328.18, "text": " So I'll show you what I mean by that."}, {"start": 5328.18, "end": 5333.22, "text": " If I take this and just print it here we end up with four bytes."}, {"start": 5333.22, "end": 5337.9400000000005, "text": " And so that means we have four iterations here and then we're going to use this mapping"}, {"start": 5337.9400000000005, "end": 5346.42, "text": " table to fetch the associated to fetch the associated Unicode character and then we're"}, {"start": 5346.42, "end": 5347.64, "text": " going to just join them."}, {"start": 5347.64, "end": 5349.86, "text": " And so we'll end up with some very weird token."}, {"start": 5349.86, "end": 5351.52, "text": " Let me do show you that."}, {"start": 5351.52, "end": 5354.0, "text": " So here's what this token looks like."}, {"start": 5354.0, "end": 5355.0, "text": " Very weird."}, {"start": 5355.0, "end": 5358.86, "text": " This is the representation we'll use to represent our brain emoji."}, {"start": 5358.86, "end": 5363.18, "text": " And then now we have to do the BPE token part."}, {"start": 5363.18, "end": 5369.34, "text": " So this is going to split this this character into sub words."}, {"start": 5369.34, "end": 5370.9, "text": " So let me show you what we get."}, {"start": 5370.9, "end": 5377.78, "text": " So this is what how he decides to split this particular character."}, {"start": 5377.78, "end": 5380.38, "text": " And after that we just encode all of those."}, {"start": 5380.38, "end": 5382.42, "text": " So that means we'll end up with two integers."}, {"start": 5382.42, "end": 5387.84, "text": " So as you can see that the brain will be encoded using two integers."}, {"start": 5387.84, "end": 5390.0, "text": " So let me show you what the BPE tokens are."}, {"start": 5390.0, "end": 5393.7, "text": " So these two numbers eight seven nine two and five ten."}, {"start": 5393.7, "end": 5395.96, "text": " That's what we how we encode the brain."}, {"start": 5395.96, "end": 5399.1, "text": " As I said fairly fairly magic."}, {"start": 5399.1, "end": 5405.58, "text": " I think they're referring to it as a dark horse behind the NLP like success because"}, {"start": 5405.58, "end": 5410.7, "text": " nobody talks about organization but it's fairly complicated to understand and to kind"}, {"start": 5410.7, "end": 5413.179999999999, "text": " of capture all of those edge cases."}, {"start": 5413.179999999999, "end": 5415.22, "text": " Anyways enough rambling guys."}, {"start": 5415.22, "end": 5416.22, "text": " Hope you found this video useful."}, {"start": 5416.22, "end": 5418.22, "text": " If you did comment down below."}, {"start": 5418.22, "end": 5422.66, "text": " Let me know how whether I can improve something or what not."}, {"start": 5422.66, "end": 5424.139999999999, "text": " Consider subscribing to this channel."}, {"start": 5424.139999999999, "end": 5426.5, "text": " Share the video art and until next time."}, {"start": 5426.5, "end": 5441.5, "text": " Bye bye."}]