MLP-Mixer: An all-MLP Architecture for Vision | Paper explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video I explain the MLP-Mixer: An all-MLP Architecture for Vision paper, aka MLP is all you need.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ paper: https://arxiv.org/pdf/2105.01601.pdf
✅ Sutton's blog: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 We've gone the full circle
01:50 Bitter lessons by Sutton
02:50 Architecture overview
06:45 Rant, rant, rant
08:20 Results
11:00 Pareto frontier
15:10 Visualization of learned weights
18:30 Decrypting equations
21:20 No positional encodings
24:10 Dark magic, initializing weight matrices during fine-tuning

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Petar Veličković
Zvonimir Sabljic
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#mixer #mlp #allyouneed
 What's up? In this video I'm covering the MLP Mixer and all MLP architecture for vision. A newly published paper by the Google Brain team and I'd give it an alternative title which is MLP is all you need because as you'll see in the scene we went to full circle and we used to use MLPs, the multi-layer perceptrons to solve the vision tasks and we in 2012 we had the ImageNet moment with AlexNet which basically showed that it's much wiser to just use big convocional neural networks with a lot of data and compute to solve the vision tasks. Finally we started using transformers to start with the VASFANI transformer in 2017 and then the last year we had Vision Transformer which showed that with lots of data and used like the JFT300M dataset from Google, the proprietary dataset and with 300 million data points it showed that it can learn really nice representations and even achieve better results than CNNs. So finally back to 2021 this paper came out and showed that by just using these simple multi-layer perceptrons, in a clever way arguably, you can achieve similar results and compute-wise you can be even a bit better and throughput-wise as well. So yeah let's see, so basically the main thing is this paper doesn't try to achieve the new SODA, it just shows that we need to maybe investigate some other research paths which may lead us even further. So this paper is a good step towards that direction, so they showed that with simplicity they can still achieve great results, so who knows a couple of papers down the line what this may evolve to. So having said that, let's dig into the paper. So first things first, they say here the mixture relies only on basic matrix multiplication routines, changes to data layouts such as reshapes and transpositions and scalar non-linearities such as GALUs as we'll soon see. So that part associates me to Biddle Lessons by Sudden which is a blog you should check out by Richard Sudden which basically, and I'll link it down in the description, but it basically says that the model that stand the tooth of time are those which can leverage which are simple and can leverage the available computation of the time. And so this paper is super simple and you can see they even linked like a code snippet implementation of this paper at the last page which kind of tells you all about it, so it's basically 20 liner and that's awesome, so that's simple and achieves nice results, so yeah I like it. That's the part I like about this paper. The architecture wise, so they just have these channel mixing MLPs and token mixing LPs. So the token mixing MLPs basically attend across the image, so the spatial extent they attend the tokens, the patches and the channel mixing ones just focus on a single patch and they kind of process the channels, hence the mixer name. So they are mixing between processing channels and processing over the spatial extent i.e. over the tokens or patches. So here is the architecture, fairly simple, resembles the vision transformer a lot, so if you haven't checked out I've covered the vision transformer in my previous video, I'll just link it somewhere here. The architecture is very similar to the vision transformer, so you just take the input image, you split it into patches as you can see here, you flatten that out as you can see here and then you just unroll all of these matrices into vectors and you use a single fully connected layer here which is shared across all of these patches and you just project them into some new latent representation vectors here. Then they stack and of these mixer layers which we'll soon see what they are, but like they ended up with global average pooling and a linear classifier on top of it, so it's a super simple architecture. Mixer contains the part that mixes, so this is the token mixer part this year and this is the channel mixing part, this part here. So before digging into those, so just let's see what MLP is. MLP just consists out of two fully connected layers and a GalU non-linearity in between. Okay, so super simple, if we ignore the transpositions here, what they semantically do here is they just take the column here and they apply the first MLP, the MLP1, they applied on this column, so they'll project this column into some other subspace and they'll just take the same MLP, take the second column, project that one into another space and basically you arrive at the new subspace here which has the same dimension, so it's again it's C and S here as it was here. So they preserve the dimensions and that's the legacy from the transformer architecture. So basically if you take a look at the CNNs on the other hand, you usually have the spatial extent usually goes down, you have this pyramid-like structure where the spatial extent goes down, where the number of channels increases, usually two-fold. That was at least some heuristic people used to use, doesn't mean it's the best one. And here we're preserving dimensions. So the second thing they do is, again ignoring layer normalization and skip connections, they just take the row-wise, they just apply these MLP2 network to like row-wise onto these metrics and they achieve some new representation and they just repeat this n times and that's it, that's as simple as that. So skip connections are just like the the Resinopaper introduced those and the transformer used those as well as well as the layer norm. So the original was fine, transformer used layer norm. So I guess a bunch of those are just legacy artifacts. What's new here is they are not using CNNs, they're not using the convolutional kernels, they're not using the attention, they're just using MLP, they took some of the legacy from the previous art, but yeah. And they showed, as you will soon see, that they achieve comparable results. And yeah, this is here. Despite its simplicity, Mixer attains competitive results. When pre-trained on large datasets, so 300 million actually, it reaches near state-of-the-art performance previously claimed by CNNs and transformers in terms of the accuracy cost trade-offs. Okay, yeah, maybe a small rant here. It's unclear from the paper why they're using Galioos. So it seems we keep using like these exotic activation functions without having a good reason why. For their defense they have on one of the last papers, they do have the things they've tried and that didn't work, which is really super and great. But I'd like to know why they use Galioos and why not use ReLU. I think those kinds of insights would be really useful. But what probably happened is they tried a couple of these and they just took the best one, just a simple grid search over activation functions. But more probably they just took it because some other paper used it and that's it. But yeah, second thing is, I mean, we keep on using these normalizations again without any particular reason, at least I don't see it. So why are we using layer norm and not batch norm or instance normalization or group norm? There is too much tradition and legacy and taking something that worked and there is too little theoretical explanations of why we're actually using something that we're using. And that's why we get papers like this where after years of developing of CNNs and transformers, we show that, hey, we can just use MLPs, it's gonna work. Why are you complicating things? So yeah, a small rant over. Let's continue with the paper. And I'll skip some of these details for now. I just wanna give you the bigger picture of the model for now. So first things first, similar to the vision transformer, they create a family of models. So they have the small mixers with 32 means basically the patches are 32 by 32 pixels. So that means you'll have less of those patches because the patch is bigger. And then they have 16 here. So then big models, large models, and finally the huge mixture. And yeah, just a bunch of different hyperparameters depending on. So basically as you go towards a huge model, you increase everything and that's it. Again, skipping the details and focusing on the results. So as you can see, when it's pre-trained, and this is the main takeaway, when it's pre-trained on smaller data sets, it's not going to have as good performance as the better model, as the more common CNNs and transformers such as the big transfer model, the vision transformer. So as you can see, it's lagging behind a little bit here compared to BIT and VIT. But this part actually, I highlighted this part because it's actually a bit better, a lot better on VTab, which I think is more complex actually than the other ones. So that's kind of surprising. I'd like to know what this number is all about. And yeah, you can see the throughput is a bit better than the vision transformer. The compute is a bit higher, so it's hard to compare them, but it's definitely not the new state of the art, it's just comparable. You can see the perf-wise it's there, throughput is a bit better, but then compute-wise it's even, it takes more compute than the vision transformer, which is interesting. And yeah, but going to JFT300M, you can see that now we have even better performance than the big transfer model, and it's comparable to vision transfer. NFNets are decently better, it's lagging behind NFNets, both in terms of performance, both in terms of throughput, and yeah, but this NFTnet takes a bit more compute. I'd like to see comparison with the EfficientNet v2, which was recently published, and which showed that it's much better than NFNets. So yeah, I wonder why they omitted that part, probably because they were already late into the paper writing. But yeah, I'd like to see the comparison with eNet v2. So those were some tabular results. Now let's see the charts, these are really interesting. We have the chart that shows the compute accuracy trade-off here, and you can see that the mixture is directly on the Pareto frontier. And what Pareto frontier is, is just a fancy word, basically that means that if you're at this point, you cannot increase one metric of interest without decreasing the other one. So that means you can do this, because that means, oops, that means you're keeping the accuracy constant, but you're decreasing the compute. Or likewise, you can't keep the compute, and just increase the accuracy. You have to move across these lines, where if you want to increase the accuracy, you have to increase the compute. Or if you want to decrease the compute, you have to decrease the accuracy. And that's just a trade-off. And a good thing, and a really reassuring fact is that it's lying directly on that frontier, alongside with NFNets, and alongside with the division transformer. So those are some nice results. The second chart shows us that as we increase the compute, so going from 10 million data points sourced, the full data set, JFT300M, we see that basically, and the full lines are the mixer models, the other ones are the big transfer and division transformer. So we can see, and what's interesting is that for smaller models, for smaller mixer models, it plateaus really quick, and it achieves results maybe a bit inferior to those baselines. But when we go to bigger mixer models and large data regimes, we can see that it even achieves better performance at, finally, compared to, this is, I think, yeah, division transformer. So the encouraging thing is that the derivative here is positive, which means if we extrapolate, hopefully, it's going to go towards the AGI. That's the promise of this model, obviously. Yeah. And jokes aside, it does seem to have steeper slope here than compared to the division transformer, which is encouraging, which means that we can push this even further. So two papers down the line, we'll be seeing even bigger data sets and, yeah, better performance. And I haven't mentioned this part, and it's just an implementation detail. The reason they use this linear five-shot ImageNet top one instead of just using ImageNet top one is because it was really compute-intensive, even for Google Brain to train all of these, to fine-tune all these models. So as a proxy, so they've just frozen the weights that come from the images, and they trained a small linear classifier on top of it and used that as a proxy to fine-tuning. But just an implementation detail, as I said. The main thing takeaway here is that it seems that MLPs have even greater potential than division transformers, and that's exciting. Two more curves here. Again, compute versus accuracy. I can see Mixer is pretty decent on this part of the spectrum. So when we have a bunch of compute, it's pretty decent and comparable to division transformer. As we go on the lower part of the spectrum, of the compute spectrum, it seems to lag a little bit behind division transformer. But yeah, it kind of converges here. And we saw the similar behavior with division transformer compared to CNNs. Only in the big data regimes, we get really high performance out of it. Similarly here, throughput and accuracy, it's on the frontier. Here, they just have some additional tabular data, but all of the main takeaways were pretty much in those charts I already described. Regarding compute, they say here, we may scale the model in two independent ways. So increasing the model size, the number of layers, blah, blah, blah. And during the pre-training. And the second dimension is increasing the input image resolution when fine tuning. So those are the two things they had in their toolkit. And as I said, it appears that Mixer models benefit from growing pre-training dataset size even more than division transformer. Nice. Finally, nice visualizations here. And that's the end of the paper. Basically, first of all, let me explain you what these patches represent. So if you have the input image, and let's try it like this. And it's got some patches. I'll draw it like four by four. But it's usually much more. It's usually 14 by 14 or something like that. And what it does, so if you take this single patch, what it does is basically, imagine we have 14 by 14 of these patches. And so this thing here basically means that if we take this and flatten it out like this, so we have S here and C here, and that was the usual representation we used in the beginning of the paper. So this will be 14 by 14. And the number of channels doesn't matter now. So basically, this thing here is the weights which attend to all of the elements in this column. And if you remember, so these are the tokens. So that means a single pixel here will attend to a single patch here. So that means if you have something like this, let me zoom in a bit, something like this, that means that that particular fully connected layer is going to attend a lot to this part because the red, let's assume red is some positive number, so it's going to have to attend this part of the image with the positive weights and it's going to attend this part of the image with the negative weights, hence the blue part. Similarly here, you can see that the blue part, that means we're going to attend this part with negative weights and this part with the positive weight. So, and another interesting detail is that you can see there is a lot of symmetry here and they've intentionally arranged all of these like that. And also, if you take a look at the Y axis, the frequency kind of goes up. So these are the low frequency components. This is like the DC component and then we have the high frequency components here. By the way, this is the first MLP layer, this is the second token mixture layer, this is the third token mixture layer. So as we go here, deeper into the network, we can see that the patterns become much more complex, higher frequency ones than here. And the idea why they did this is because usually when we analyze CNNs, we can notice certain patterns and they mentioned the GABA filters, which is just a multi- you get those by a combination of gossians and sinusoids. And we see similar structure in the lower layers, but when we go deeper into the network, we get something that we still can't discern. And it'd be nice to kind of analyze this and do some mathematical approximations here to understand how we can model these and maybe we can take off, take some intuition further from this and develop device better models once we know what this thing is modeling. So yeah, but it's a, I guess, future work for these authors. Now let me go ahead and explain a couple of those details, which may be important to you. So first of all, let me start with this. So yeah, so I mentioned that they are sharing weights, so all patches are linearly projected with the same projection matrix. And that helps save up the memory footprint. So that's important. And they are also sharing the across the columns and across the rows. So that's the, the token mixer, the token mixing part and the channel mixing part. And let me just decrypt these equations for you. So basically we take that input matrix, which has dimensions S times C, if you remember, and they applied normalization and this weight matrix just represents the first fully connected layer of the MLP. So that's this one here. That's the network. And then they have a nonlinearity, which is GALU. So that's this one. Finally, they have the second weight matrix, which represents the second fully connected layer. So that's again, this one here. And finally, we have plus with the input, which is basically the skip connection. And then they just repeat this and apply it. The second thing is basically the, so this is the, this is across channels. And this is across, so that means it's a token mixer and this part is across basically, so let me draw the matrix here. So this is S, this is C, this goes across channels. So that means it's attending a bunch of different tokens. So it's a token mixer. And here we go, across S. So that means we're doing channel wise mixing. So these are the two equations I wanted to explain. Okay. They mentioned that tying across the channel mixing MLP, so keeping the same network here and here and here makes sense because you're basically, you want to do the positionally, that encodes the, that kind of enforces the positional invariance because whatever you learn here, you want to have a pattern that's generalizable to all of the other spatial locations. But you usually don't constrain, you usually don't take the network, like the MLP and constrain it to be the same across the columns. So that's what they mentioned and they actually tried both things. They figured out that just tying it doesn't hurt the performance, but it saves memory obviously, because you don't have to have C MLPs. You just can't, you can have just one single MLP and just apply it C times. But again, there's just a thing, like there is nothing theoretically guaranteeing that this is a better choice. They just, it's just a experiment. Important detail is that it's not using, so Mixer does not use position embeddings because the token mixing MLPs are sensitive to the order of the input tokens and therefore may learn to represent location. So, and that's it. Learn to represent location. So this is a direct, let's make a direct comparison with the Transformer. So Transformers have, as you may recall, so these are the patches embedded into the latent space. And what the Transformer does is, let's take this token for example, is going to attend all of the other tokens. So we're going to create those keys, queries and value vectors, and we're going to attend all of the other tokens, and we're going to form those alpha coefficients, basically the attention coefficients, and we're going to sum them up. And because we're summing them up, we are losing the positional information. And because of that, the original, the Vassfani paper and some, and the successor papers basically had to add additional positional information. So that's kind of going to encode somehow that this is different, so maybe like this, and then we'll maybe have the second position encoded like this, whatever. You need some unique pattern that kind of uniquely identifies each of the positions. But here, because we have MLPs, we don't need to do that. And the reason is, if you take a look at the matrix, we have S here and C here. So if you have MLP, like the token-wise mixture, applied across this column, you basically, because it's an MLP, so it's got a fully connected layer, right? So it's going to do something like this. It's going to attend all of these positions, and this weight here is some weight V1,1. And that's the first element of the output vector. Then the second element of the output vector will attend like this. Again, it's a fully connected, and it's becoming a mess really quickly, but you can understand that basically. Let me take another color. So basically, this one here is going to be some W1,2. And you'll have a collection of these weights, which are particular to this token here. So it directly learns that for that position, for that token, which corresponds to a certain position in the image. So this is the input image. Maybe this is this token here, this patch. So these weights will learn how to encode information from this particular patch, and that's why they don't need to use any positional encodings. Hopefully, that was clear enough. If not, let me know in the comments. I'll try to explain it further. Okay, so that was the detail I wanted to explain. And the final thing I want to mention is this here. Here is some dark magic here, cosine learning rate. I really wonder why are we still using these without having any theoretical justifications. Like a small footnote of why they use that particular schedule and not something else, like a simple linear schedule or even some constant learning rate schedule would be really appreciated. And again, following common practice, and I highlight common practice, so we also apply fine-tune higher resolutions with respect to those used during pre-training. So what happens here is that basically people show that when you're training for like vision benchmarks, you pre-train on certain resolution, like maybe 224 times 224, and when you want to fine-tune, you actually increase the resolution and that will help you boost your performance. So you fine-tune on 384 times 384, for example. And as I said, so I highlight the common practice, because I'm not sure if we still have an understanding of why this is making things work any better. So we are just using the legacy ideas, legacy heuristics, and keeping them and kind of keeping them in all of the present research. And I guess, yeah, we sometimes have to do that because otherwise, like the research will fall apart. You have to clench onto something. But basically, it'd be nice when we had more papers explaining why these things work, and not just combining them like black boxes. There is one more detail I want to mention, and we are done basically because they are doing this up-scaling during the fine-tune process. They somehow have to adjust the weights of the MLPs. So the reason that is, is because, if you remember, so the input image can be represented as S times C, and because the token mixer basically attends across the number of patches, when we increase the resolution, so when we increase the resolution, but we keep the number of, the size of the patch the same, that means we're going to have more of these patches. So that means S goes to S prime, which is a bigger number, and that means this thing won't fit into the previous MLP, which had certain bandwidth. So it was maybe, it was maybe, it could attend like this. It was smaller. So now we have to kind of take the weight matrix of that old MLP, and we need to initialize this new weight matrix, which is bigger somehow, and what they did is they just took these and stacked them block-wise here, and that was the initialization method they used to fine-tune the models. So yeah, just an engineering detail. I thought it's interesting and important to mention that there are a lot of these details that go into making this thing work, and the reason this works is if you think about it, if you now multiply the input vector with this new matrix, this thing is basically going to attend to the first part, and this thing is basically going to attend to the second part. So it's like you're applying these in parallel, and so that seemed like a sound way to initialize the new weight matrix. So that's the main reason they did it like this. Hopefully you found this video useful. If you did, leave a like, share the video, and see you next time.
[{"start": 0.0, "end": 5.68, "text": " What's up? In this video I'm covering the MLP Mixer and all MLP architecture for vision."}, {"start": 6.640000000000001, "end": 14.08, "text": " A newly published paper by the Google Brain team and I'd give it an alternative title which is"}, {"start": 14.08, "end": 20.88, "text": " MLP is all you need because as you'll see in the scene we went to full circle and we used to use"}, {"start": 21.44, "end": 27.6, "text": " MLPs, the multi-layer perceptrons to solve the vision tasks and we in 2012 we had the ImageNet"}, {"start": 27.6, "end": 34.800000000000004, "text": " moment with AlexNet which basically showed that it's much wiser to just use big convocional neural"}, {"start": 34.800000000000004, "end": 41.92, "text": " networks with a lot of data and compute to solve the vision tasks. Finally we started using"}, {"start": 41.92, "end": 47.44, "text": " transformers to start with the VASFANI transformer in 2017 and then the last year we had Vision"}, {"start": 47.44, "end": 54.88, "text": " Transformer which showed that with lots of data and used like the JFT300M dataset from Google,"}, {"start": 54.88, "end": 60.480000000000004, "text": " the proprietary dataset and with 300 million data points it showed that it can learn really"}, {"start": 60.480000000000004, "end": 67.68, "text": " nice representations and even achieve better results than CNNs. So finally back to 2021"}, {"start": 69.36, "end": 74.88, "text": " this paper came out and showed that by just using these simple multi-layer perceptrons,"}, {"start": 74.88, "end": 82.72, "text": " in a clever way arguably, you can achieve similar results and compute-wise you can be even a bit"}, {"start": 82.72, "end": 88.8, "text": " better and throughput-wise as well. So yeah let's see, so basically the main thing is this paper"}, {"start": 88.8, "end": 95.84, "text": " doesn't try to achieve the new SODA, it just shows that we need to maybe investigate some other"}, {"start": 95.84, "end": 102.4, "text": " research paths which may lead us even further. So this paper is a good step towards that direction,"}, {"start": 102.4, "end": 107.52, "text": " so they showed that with simplicity they can still achieve great results, so who knows a couple of"}, {"start": 107.52, "end": 114.0, "text": " papers down the line what this may evolve to. So having said that, let's dig into the paper."}, {"start": 114.56, "end": 119.44, "text": " So first things first, they say here the mixture relies only on basic matrix multiplication"}, {"start": 119.44, "end": 125.84, "text": " routines, changes to data layouts such as reshapes and transpositions and scalar non-linearities such"}, {"start": 125.84, "end": 132.72, "text": " as GALUs as we'll soon see. So that part associates me to Biddle Lessons by Sudden which is a blog you"}, {"start": 132.72, "end": 138.16, "text": " should check out by Richard Sudden which basically, and I'll link it down in the description, but it"}, {"start": 138.16, "end": 144.4, "text": " basically says that the model that stand the tooth of time are those which can leverage which are"}, {"start": 144.4, "end": 150.88, "text": " simple and can leverage the available computation of the time. And so this paper is super simple and"}, {"start": 150.88, "end": 158.24, "text": " you can see they even linked like a code snippet implementation of this paper at the last page which"}, {"start": 158.24, "end": 164.88, "text": " kind of tells you all about it, so it's basically 20 liner and that's awesome, so that's simple"}, {"start": 164.88, "end": 168.96, "text": " and achieves nice results, so yeah I like it. That's the part I like about this paper."}, {"start": 170.56, "end": 176.48000000000002, "text": " The architecture wise, so they just have these channel mixing MLPs and token mixing LPs. So the"}, {"start": 176.48000000000002, "end": 184.24, "text": " token mixing MLPs basically attend across the image, so the spatial extent they attend the tokens,"}, {"start": 184.24, "end": 190.96, "text": " the patches and the channel mixing ones just focus on a single patch and they kind of process the"}, {"start": 190.96, "end": 196.88, "text": " channels, hence the mixer name. So they are mixing between processing channels and processing"}, {"start": 196.88, "end": 204.64000000000001, "text": " over the spatial extent i.e. over the tokens or patches. So here is the architecture, fairly simple,"}, {"start": 205.84, "end": 210.0, "text": " resembles the vision transformer a lot, so if you haven't checked out I've covered the vision"}, {"start": 210.0, "end": 213.92000000000002, "text": " transformer in my previous video, I'll just link it somewhere here. The architecture is very similar"}, {"start": 213.92, "end": 218.64, "text": " to the vision transformer, so you just take the input image, you split it into patches as you can"}, {"start": 218.64, "end": 226.16, "text": " see here, you flatten that out as you can see here and then you just unroll all of these matrices into"}, {"start": 226.16, "end": 232.56, "text": " vectors and you use a single fully connected layer here which is shared across all of these"}, {"start": 233.6, "end": 240.88, "text": " patches and you just project them into some new latent representation vectors here. Then they stack"}, {"start": 240.88, "end": 247.28, "text": " and of these mixer layers which we'll soon see what they are, but like they ended up with global"}, {"start": 247.28, "end": 251.84, "text": " average pooling and a linear classifier on top of it, so it's a super simple architecture."}, {"start": 251.84, "end": 259.44, "text": " Mixer contains the part that mixes, so this is the token mixer part this year and this is the"}, {"start": 260.0, "end": 267.68, "text": " channel mixing part, this part here. So before digging into those, so just let's see what MLP is."}, {"start": 267.68, "end": 274.24, "text": " MLP just consists out of two fully connected layers and a GalU non-linearity in between. Okay,"}, {"start": 274.24, "end": 281.92, "text": " so super simple, if we ignore the transpositions here, what they semantically do here is they just"}, {"start": 281.92, "end": 290.32, "text": " take the column here and they apply the first MLP, the MLP1, they applied on this column, so"}, {"start": 290.32, "end": 295.44, "text": " they'll project this column into some other subspace and they'll just take the same MLP,"}, {"start": 295.44, "end": 302.24, "text": " take the second column, project that one into another space and basically you arrive at the"}, {"start": 302.24, "end": 310.16, "text": " new subspace here which has the same dimension, so it's again it's C and S here as it was here."}, {"start": 310.16, "end": 315.28, "text": " So they preserve the dimensions and that's the legacy from the transformer architecture. So"}, {"start": 315.28, "end": 319.28, "text": " basically if you take a look at the CNNs on the other hand, you usually have the spatial extent"}, {"start": 319.28, "end": 323.36, "text": " usually goes down, you have this pyramid-like structure where the spatial extent goes down,"}, {"start": 323.36, "end": 328.16, "text": " where the number of channels increases, usually two-fold. That was at least some"}, {"start": 328.16, "end": 334.8, "text": " heuristic people used to use, doesn't mean it's the best one. And here we're preserving dimensions."}, {"start": 334.8, "end": 340.24, "text": " So the second thing they do is, again ignoring layer normalization and skip connections,"}, {"start": 340.24, "end": 347.76, "text": " they just take the row-wise, they just apply these MLP2 network to like row-wise onto these"}, {"start": 347.76, "end": 353.28, "text": " metrics and they achieve some new representation and they just repeat this n times and that's it,"}, {"start": 353.28, "end": 359.92, "text": " that's as simple as that. So skip connections are just like the the Resinopaper introduced those"}, {"start": 359.92, "end": 364.96, "text": " and the transformer used those as well as well as the layer norm. So the original was fine,"}, {"start": 364.96, "end": 369.12, "text": " transformer used layer norm. So I guess a bunch of those are just legacy artifacts."}, {"start": 369.68, "end": 374.24, "text": " What's new here is they are not using CNNs, they're not using the convolutional kernels,"}, {"start": 374.24, "end": 378.72, "text": " they're not using the attention, they're just using MLP, they took some of the legacy from"}, {"start": 378.72, "end": 385.44, "text": " the previous art, but yeah. And they showed, as you will soon see, that they achieve comparable"}, {"start": 386.48, "end": 393.2, "text": " results. And yeah, this is here. Despite its simplicity, Mixer attains competitive results."}, {"start": 393.2, "end": 399.04, "text": " When pre-trained on large datasets, so 300 million actually, it reaches near state-of-the-art"}, {"start": 399.04, "end": 404.32, "text": " performance previously claimed by CNNs and transformers in terms of the accuracy cost"}, {"start": 404.32, "end": 412.24, "text": " trade-offs. Okay, yeah, maybe a small rant here. It's unclear from the paper why they're using"}, {"start": 412.24, "end": 422.48, "text": " Galioos. So it seems we keep using like these exotic activation functions without having a"}, {"start": 422.48, "end": 431.20000000000005, "text": " good reason why. For their defense they have on one of the last papers, they do have the"}, {"start": 431.20000000000005, "end": 436.40000000000003, "text": " things they've tried and that didn't work, which is really super and great. But I'd like to know"}, {"start": 436.40000000000003, "end": 441.92, "text": " why they use Galioos and why not use ReLU. I think those kinds of insights would be really"}, {"start": 442.56, "end": 448.88, "text": " useful. But what probably happened is they tried a couple of these and they just took the best one,"}, {"start": 448.88, "end": 455.04, "text": " just a simple grid search over activation functions. But more probably they just took it"}, {"start": 455.04, "end": 462.96, "text": " because some other paper used it and that's it. But yeah, second thing is, I mean, we keep on using"}, {"start": 462.96, "end": 471.6, "text": " these normalizations again without any particular reason, at least I don't see it. So why are we"}, {"start": 471.6, "end": 478.08, "text": " using layer norm and not batch norm or instance normalization or group norm? There is too much"}, {"start": 478.08, "end": 484.24, "text": " tradition and legacy and taking something that worked and there is too little theoretical"}, {"start": 484.24, "end": 488.47999999999996, "text": " explanations of why we're actually using something that we're using. And that's why we"}, {"start": 489.12, "end": 494.79999999999995, "text": " get papers like this where after years of developing of CNNs and transformers, we show that,"}, {"start": 494.79999999999995, "end": 500.0, "text": " hey, we can just use MLPs, it's gonna work. Why are you complicating things? So yeah,"}, {"start": 500.88, "end": 507.84, "text": " a small rant over. Let's continue with the paper. And I'll skip some of these details for now."}, {"start": 507.84, "end": 513.68, "text": " I just wanna give you the bigger picture of the model for now. So first things first,"}, {"start": 514.64, "end": 520.24, "text": " similar to the vision transformer, they create a family of models. So they have the small mixers"}, {"start": 521.04, "end": 528.8, "text": " with 32 means basically the patches are 32 by 32 pixels. So that means you'll have less of those"}, {"start": 528.8, "end": 535.04, "text": " patches because the patch is bigger. And then they have 16 here. So then big models, large models,"}, {"start": 535.04, "end": 541.1999999999999, "text": " and finally the huge mixture. And yeah, just a bunch of different hyperparameters depending on."}, {"start": 541.1999999999999, "end": 548.9599999999999, "text": " So basically as you go towards a huge model, you increase everything and that's it. Again,"}, {"start": 548.9599999999999, "end": 556.64, "text": " skipping the details and focusing on the results. So as you can see, when it's pre-trained, and this"}, {"start": 556.64, "end": 561.36, "text": " is the main takeaway, when it's pre-trained on smaller data sets, it's not going to have as good"}, {"start": 561.36, "end": 567.92, "text": " performance as the better model, as the more common CNNs and transformers such as the big"}, {"start": 567.92, "end": 573.36, "text": " transfer model, the vision transformer. So as you can see, it's lagging behind a little bit here"}, {"start": 574.72, "end": 581.36, "text": " compared to BIT and VIT. But this part actually, I highlighted this part because it's actually a"}, {"start": 581.36, "end": 588.72, "text": " bit better, a lot better on VTab, which I think is more complex actually than the other ones."}, {"start": 588.72, "end": 591.9200000000001, "text": " So that's kind of surprising. I'd like to know what this number is all about."}, {"start": 593.2, "end": 598.8000000000001, "text": " And yeah, you can see the throughput is a bit better than the vision transformer. The compute"}, {"start": 598.8000000000001, "end": 605.0400000000001, "text": " is a bit higher, so it's hard to compare them, but it's definitely not the new state of the art,"}, {"start": 605.0400000000001, "end": 609.9200000000001, "text": " it's just comparable. You can see the perf-wise it's there, throughput is a bit better, but then"}, {"start": 609.9200000000001, "end": 614.88, "text": " compute-wise it's even, it takes more compute than the vision transformer, which is interesting."}, {"start": 614.88, "end": 622.8, "text": " And yeah, but going to JFT300M, you can see that now we have even better performance than the"}, {"start": 622.8, "end": 628.8, "text": " big transfer model, and it's comparable to vision transfer. NFNets are decently better,"}, {"start": 629.4399999999999, "end": 634.72, "text": " it's lagging behind NFNets, both in terms of performance, both in terms of throughput,"}, {"start": 635.28, "end": 642.08, "text": " and yeah, but this NFTnet takes a bit more compute. I'd like to see comparison with the"}, {"start": 642.08, "end": 649.9200000000001, "text": " EfficientNet v2, which was recently published, and which showed that it's much better than NFNets."}, {"start": 649.9200000000001, "end": 654.4000000000001, "text": " So yeah, I wonder why they omitted that part, probably because they were already late into"}, {"start": 654.4000000000001, "end": 662.4000000000001, "text": " the paper writing. But yeah, I'd like to see the comparison with eNet v2. So those were some"}, {"start": 662.4000000000001, "end": 666.24, "text": " tabular results. Now let's see the charts, these are really interesting. We have the chart that"}, {"start": 666.24, "end": 674.24, "text": " shows the compute accuracy trade-off here, and you can see that the mixture is directly on the"}, {"start": 674.24, "end": 679.52, "text": " Pareto frontier. And what Pareto frontier is, is just a fancy word, basically that means that"}, {"start": 679.52, "end": 685.12, "text": " if you're at this point, you cannot increase one metric of interest without decreasing the other"}, {"start": 685.12, "end": 690.88, "text": " one. So that means you can do this, because that means, oops, that means you're keeping the"}, {"start": 690.88, "end": 696.16, "text": " accuracy constant, but you're decreasing the compute. Or likewise, you can't keep the compute,"}, {"start": 696.16, "end": 700.96, "text": " and just increase the accuracy. You have to move across these lines, where if you want to increase"}, {"start": 700.96, "end": 705.52, "text": " the accuracy, you have to increase the compute. Or if you want to decrease the compute, you have"}, {"start": 705.52, "end": 712.48, "text": " to decrease the accuracy. And that's just a trade-off. And a good thing, and a really reassuring"}, {"start": 712.48, "end": 720.32, "text": " fact is that it's lying directly on that frontier, alongside with NFNets, and alongside with the"}, {"start": 720.32, "end": 726.08, "text": " division transformer. So those are some nice results. The second chart shows us that as we"}, {"start": 726.08, "end": 732.48, "text": " increase the compute, so going from 10 million data points sourced, the full data set, JFT300M,"}, {"start": 732.48, "end": 738.8000000000001, "text": " we see that basically, and the full lines are the mixer models, the other ones are the big"}, {"start": 738.8000000000001, "end": 745.6800000000001, "text": " transfer and division transformer. So we can see, and what's interesting is that for smaller models,"}, {"start": 745.68, "end": 752.4, "text": " for smaller mixer models, it plateaus really quick, and it achieves results maybe a bit inferior to"}, {"start": 753.68, "end": 760.56, "text": " those baselines. But when we go to bigger mixer models and large data regimes, we can see that"}, {"start": 760.56, "end": 767.4399999999999, "text": " it even achieves better performance at, finally, compared to, this is, I think, yeah, division"}, {"start": 767.4399999999999, "end": 773.5999999999999, "text": " transformer. So the encouraging thing is that the derivative here is positive, which means if we"}, {"start": 773.6, "end": 780.0, "text": " extrapolate, hopefully, it's going to go towards the AGI. That's the promise of this model,"}, {"start": 780.0, "end": 790.72, "text": " obviously. Yeah. And jokes aside, it does seem to have steeper slope here than compared to the"}, {"start": 791.28, "end": 795.84, "text": " division transformer, which is encouraging, which means that we can push this even further. So"}, {"start": 795.84, "end": 801.6800000000001, "text": " two papers down the line, we'll be seeing even bigger data sets and, yeah, better performance."}, {"start": 801.68, "end": 805.92, "text": " And I haven't mentioned this part, and it's just an implementation detail. The reason they use this"}, {"start": 805.92, "end": 811.52, "text": " linear five-shot ImageNet top one instead of just using ImageNet top one is because it was really"}, {"start": 811.52, "end": 816.8, "text": " compute-intensive, even for Google Brain to train all of these, to fine-tune all these models. So"}, {"start": 816.8, "end": 822.56, "text": " as a proxy, so they've just frozen the weights that come from the images, and they trained a small"}, {"start": 822.56, "end": 828.9599999999999, "text": " linear classifier on top of it and used that as a proxy to fine-tuning. But just an implementation"}, {"start": 828.96, "end": 834.64, "text": " detail, as I said. The main thing takeaway here is that it seems that MLPs have even greater potential"}, {"start": 834.64, "end": 841.76, "text": " than division transformers, and that's exciting. Two more curves here. Again, compute versus"}, {"start": 841.76, "end": 847.2800000000001, "text": " accuracy. I can see Mixer is pretty decent on this part of the spectrum. So when we have a bunch of"}, {"start": 847.2800000000001, "end": 852.88, "text": " compute, it's pretty decent and comparable to division transformer. As we go on the lower part"}, {"start": 852.88, "end": 857.9200000000001, "text": " of the spectrum, of the compute spectrum, it seems to lag a little bit behind division transformer."}, {"start": 857.92, "end": 862.64, "text": " But yeah, it kind of converges here. And we saw the similar behavior with division transformer"}, {"start": 862.64, "end": 869.12, "text": " compared to CNNs. Only in the big data regimes, we get really high performance out of it. Similarly"}, {"start": 869.12, "end": 874.56, "text": " here, throughput and accuracy, it's on the frontier. Here, they just have some additional"}, {"start": 874.56, "end": 880.56, "text": " tabular data, but all of the main takeaways were pretty much in those charts I already described."}, {"start": 881.8399999999999, "end": 886.0799999999999, "text": " Regarding compute, they say here, we may scale the model in two independent ways. So increasing"}, {"start": 886.08, "end": 891.76, "text": " the model size, the number of layers, blah, blah, blah. And during the pre-training. And the second"}, {"start": 891.76, "end": 898.0, "text": " dimension is increasing the input image resolution when fine tuning. So those are the two things they"}, {"start": 898.0, "end": 905.6, "text": " had in their toolkit. And as I said, it appears that Mixer models benefit from growing pre-training"}, {"start": 905.6, "end": 912.48, "text": " dataset size even more than division transformer. Nice. Finally, nice visualizations here."}, {"start": 912.48, "end": 920.24, "text": " And that's the end of the paper. Basically, first of all, let me explain you what these patches"}, {"start": 920.24, "end": 927.76, "text": " represent. So if you have the input image, and let's try it like this. And it's got some patches."}, {"start": 929.2, "end": 934.96, "text": " I'll draw it like four by four. But it's usually much more. It's usually 14 by 14 or something like"}, {"start": 934.96, "end": 940.4, "text": " that. And what it does, so if you take this single patch, what it does is basically, imagine we have"}, {"start": 940.4, "end": 950.72, "text": " 14 by 14 of these patches. And so this thing here basically means that if we take this and flatten it"}, {"start": 950.72, "end": 958.64, "text": " out like this, so we have S here and C here, and that was the usual representation we used in the"}, {"start": 958.64, "end": 965.28, "text": " beginning of the paper. So this will be 14 by 14. And the number of channels doesn't matter now."}, {"start": 965.28, "end": 974.0799999999999, "text": " So basically, this thing here is the weights which attend to all of the elements in this column."}, {"start": 975.28, "end": 982.24, "text": " And if you remember, so these are the tokens. So that means a single pixel here will attend to a"}, {"start": 982.24, "end": 988.3199999999999, "text": " single patch here. So that means if you have something like this, let me zoom in a bit,"}, {"start": 988.32, "end": 995.7600000000001, "text": " something like this, that means that that particular fully connected layer is going to attend"}, {"start": 996.4000000000001, "end": 1001.6, "text": " a lot to this part because the red, let's assume red is some positive number, so it's going to have"}, {"start": 1001.6, "end": 1006.96, "text": " to attend this part of the image with the positive weights and it's going to attend this part of the"}, {"start": 1006.96, "end": 1012.72, "text": " image with the negative weights, hence the blue part. Similarly here, you can see that the blue"}, {"start": 1012.72, "end": 1016.48, "text": " part, that means we're going to attend this part with negative weights and this part with the"}, {"start": 1016.48, "end": 1022.0, "text": " positive weight. So, and another interesting detail is that you can see there is a lot of symmetry"}, {"start": 1022.0, "end": 1030.0, "text": " here and they've intentionally arranged all of these like that. And also, if you take a look at"}, {"start": 1030.0, "end": 1035.84, "text": " the Y axis, the frequency kind of goes up. So these are the low frequency components. This is like the"}, {"start": 1035.84, "end": 1045.04, "text": " DC component and then we have the high frequency components here. By the way, this is the first"}, {"start": 1045.04, "end": 1051.28, "text": " MLP layer, this is the second token mixture layer, this is the third token mixture layer. So as we go"}, {"start": 1051.28, "end": 1057.12, "text": " here, deeper into the network, we can see that the patterns become much more complex, higher"}, {"start": 1057.12, "end": 1064.1599999999999, "text": " frequency ones than here. And the idea why they did this is because usually when we analyze CNNs,"}, {"start": 1064.1599999999999, "end": 1069.36, "text": " we can notice certain patterns and they mentioned the GABA filters, which is just a multi- you get"}, {"start": 1069.36, "end": 1077.04, "text": " those by a combination of gossians and sinusoids. And we see similar structure in the lower layers,"}, {"start": 1077.04, "end": 1081.84, "text": " but when we go deeper into the network, we get something that we still can't discern."}, {"start": 1081.84, "end": 1086.3999999999999, "text": " And it'd be nice to kind of analyze this and do some mathematical approximations"}, {"start": 1086.3999999999999, "end": 1092.6399999999999, "text": " here to understand how we can model these and maybe we can take off, take some intuition"}, {"start": 1092.64, "end": 1099.0400000000002, "text": " further from this and develop device better models once we know what this thing is modeling."}, {"start": 1099.6000000000001, "end": 1105.5200000000002, "text": " So yeah, but it's a, I guess, future work for these authors. Now let me go ahead and"}, {"start": 1105.5200000000002, "end": 1112.96, "text": " explain a couple of those details, which may be important to you. So first of all,"}, {"start": 1112.96, "end": 1121.3600000000001, "text": " let me start with this. So yeah, so I mentioned that they are sharing weights, so all patches"}, {"start": 1121.36, "end": 1126.24, "text": " are linearly projected with the same projection matrix. And that helps save up the memory"}, {"start": 1126.24, "end": 1130.7199999999998, "text": " footprint. So that's important. And they are also sharing the across the columns and across the"}, {"start": 1130.7199999999998, "end": 1137.6799999999998, "text": " rows. So that's the, the token mixer, the token mixing part and the channel mixing part. And"}, {"start": 1137.6799999999998, "end": 1142.6399999999999, "text": " let me just decrypt these equations for you. So basically we take that input matrix, which has"}, {"start": 1142.6399999999999, "end": 1150.08, "text": " dimensions S times C, if you remember, and they applied normalization and this weight matrix just"}, {"start": 1150.08, "end": 1156.8, "text": " represents the first fully connected layer of the MLP. So that's this one here. That's the network."}, {"start": 1157.4399999999998, "end": 1165.52, "text": " And then they have a nonlinearity, which is GALU. So that's this one. Finally, they have"}, {"start": 1165.52, "end": 1170.72, "text": " the second weight matrix, which represents the second fully connected layer. So that's again,"}, {"start": 1170.72, "end": 1178.8799999999999, "text": " this one here. And finally, we have plus with the input, which is basically the skip connection."}, {"start": 1178.88, "end": 1185.2800000000002, "text": " And then they just repeat this and apply it. The second thing is basically the, so this is the,"}, {"start": 1185.2800000000002, "end": 1193.5200000000002, "text": " this is across channels. And this is across, so that means it's a token mixer and this part is"}, {"start": 1193.5200000000002, "end": 1201.5200000000002, "text": " across basically, so let me draw the matrix here. So this is S, this is C, this goes across channels."}, {"start": 1201.5200000000002, "end": 1207.2, "text": " So that means it's attending a bunch of different tokens. So it's a token mixer. And here we go,"}, {"start": 1207.2, "end": 1212.32, "text": " across S. So that means we're doing channel wise mixing. So these are the two equations"}, {"start": 1212.32, "end": 1218.24, "text": " I wanted to explain. Okay. They mentioned that tying across the channel mixing MLP,"}, {"start": 1218.24, "end": 1223.68, "text": " so keeping the same network here and here and here makes sense because you're basically,"}, {"start": 1223.68, "end": 1228.96, "text": " you want to do the positionally, that encodes the, that kind of enforces the positional invariance"}, {"start": 1228.96, "end": 1234.88, "text": " because whatever you learn here, you want to have a pattern that's generalizable to all of the other"}, {"start": 1234.88, "end": 1240.0, "text": " spatial locations. But you usually don't constrain, you usually don't take the"}, {"start": 1240.64, "end": 1246.3200000000002, "text": " network, like the MLP and constrain it to be the same across the columns. So that's what they"}, {"start": 1246.3200000000002, "end": 1251.6000000000001, "text": " mentioned and they actually tried both things. They figured out that just tying it doesn't"}, {"start": 1251.6000000000001, "end": 1255.3600000000001, "text": " hurt the performance, but it saves memory obviously, because you don't have to have"}, {"start": 1255.3600000000001, "end": 1261.1200000000001, "text": " C MLPs. You just can't, you can have just one single MLP and just apply it C times."}, {"start": 1261.12, "end": 1267.28, "text": " But again, there's just a thing, like there is nothing theoretically guaranteeing that this is"}, {"start": 1267.28, "end": 1275.9199999999998, "text": " a better choice. They just, it's just a experiment. Important detail is that it's not using, so"}, {"start": 1275.9199999999998, "end": 1280.8, "text": " Mixer does not use position embeddings because the token mixing MLPs are sensitive to the order"}, {"start": 1280.8, "end": 1288.1599999999999, "text": " of the input tokens and therefore may learn to represent location. So, and that's it."}, {"start": 1288.16, "end": 1298.96, "text": " Learn to represent location. So this is a direct, let's make a direct comparison with the Transformer."}, {"start": 1298.96, "end": 1306.96, "text": " So Transformers have, as you may recall, so these are the patches embedded into the latent space."}, {"start": 1307.52, "end": 1314.24, "text": " And what the Transformer does is, let's take this token for example, is going to attend all of the"}, {"start": 1314.24, "end": 1320.8, "text": " other tokens. So we're going to create those keys, queries and value vectors, and we're going to"}, {"start": 1320.8, "end": 1327.44, "text": " attend all of the other tokens, and we're going to form those alpha coefficients, basically the"}, {"start": 1327.44, "end": 1331.68, "text": " attention coefficients, and we're going to sum them up. And because we're summing them up, we are"}, {"start": 1331.68, "end": 1336.72, "text": " losing the positional information. And because of that, the original, the Vassfani paper and"}, {"start": 1336.72, "end": 1344.16, "text": " some, and the successor papers basically had to add additional positional information. So that's"}, {"start": 1344.16, "end": 1349.28, "text": " kind of going to encode somehow that this is different, so maybe like this, and then we'll"}, {"start": 1349.28, "end": 1354.72, "text": " maybe have the second position encoded like this, whatever. You need some unique pattern that kind"}, {"start": 1354.72, "end": 1360.08, "text": " of uniquely identifies each of the positions. But here, because we have MLPs, we don't need to do"}, {"start": 1360.08, "end": 1368.6399999999999, "text": " that. And the reason is, if you take a look at the matrix, we have S here and C here. So if you have"}, {"start": 1369.4399999999998, "end": 1378.56, "text": " MLP, like the token-wise mixture, applied across this column, you basically, because it's an MLP,"}, {"start": 1378.56, "end": 1383.04, "text": " so it's got a fully connected layer, right? So it's going to do something like this. It's going to"}, {"start": 1383.04, "end": 1390.8799999999999, "text": " attend all of these positions, and this weight here is some weight V1,1. And that's the first"}, {"start": 1392.24, "end": 1398.08, "text": " element of the output vector. Then the second element of the output vector will attend like this."}, {"start": 1399.04, "end": 1403.76, "text": " Again, it's a fully connected, and it's becoming a mess really quickly, but you can understand that"}, {"start": 1403.76, "end": 1415.04, "text": " basically. Let me take another color. So basically, this one here is going to be some W1,2. And you'll"}, {"start": 1415.04, "end": 1423.28, "text": " have a collection of these weights, which are particular to this token here. So it directly"}, {"start": 1423.28, "end": 1429.52, "text": " learns that for that position, for that token, which corresponds to a certain position in the"}, {"start": 1429.52, "end": 1435.76, "text": " image. So this is the input image. Maybe this is this token here, this patch. So these weights will"}, {"start": 1435.76, "end": 1441.04, "text": " learn how to encode information from this particular patch, and that's why they don't need to use any"}, {"start": 1441.04, "end": 1445.84, "text": " positional encodings. Hopefully, that was clear enough. If not, let me know in the comments. I'll"}, {"start": 1445.84, "end": 1452.08, "text": " try to explain it further. Okay, so that was the detail I wanted to explain. And the final thing"}, {"start": 1452.08, "end": 1457.44, "text": " I want to mention is this here. Here is some dark magic here, cosine learning rate. I really wonder"}, {"start": 1457.44, "end": 1461.04, "text": " why are we still using these without having any theoretical justifications."}, {"start": 1462.64, "end": 1469.28, "text": " Like a small footnote of why they use that particular schedule and not something else,"}, {"start": 1469.28, "end": 1473.76, "text": " like a simple linear schedule or even some constant learning rate schedule would be really"}, {"start": 1473.76, "end": 1479.68, "text": " appreciated. And again, following common practice, and I highlight common practice,"}, {"start": 1479.68, "end": 1484.64, "text": " so we also apply fine-tune higher resolutions with respect to those used during pre-training."}, {"start": 1484.64, "end": 1489.76, "text": " So what happens here is that basically people show that when you're training for like vision"}, {"start": 1489.76, "end": 1495.92, "text": " benchmarks, you pre-train on certain resolution, like maybe 224 times 224, and when you want to"}, {"start": 1495.92, "end": 1500.5600000000002, "text": " fine-tune, you actually increase the resolution and that will help you boost your performance."}, {"start": 1500.5600000000002, "end": 1508.72, "text": " So you fine-tune on 384 times 384, for example. And as I said, so I highlight the common practice,"}, {"start": 1508.72, "end": 1513.6000000000001, "text": " because I'm not sure if we still have an understanding of why this is making things work"}, {"start": 1513.6, "end": 1519.12, "text": " any better. So we are just using the legacy ideas, legacy heuristics, and keeping them"}, {"start": 1519.12, "end": 1525.4399999999998, "text": " and kind of keeping them in all of the present research. And I guess, yeah, we sometimes have"}, {"start": 1525.4399999999998, "end": 1530.8799999999999, "text": " to do that because otherwise, like the research will fall apart. You have to clench onto something."}, {"start": 1531.6799999999998, "end": 1537.76, "text": " But basically, it'd be nice when we had more papers explaining why these things work,"}, {"start": 1537.76, "end": 1544.64, "text": " and not just combining them like black boxes. There is one more detail I want to mention,"}, {"start": 1544.64, "end": 1552.4, "text": " and we are done basically because they are doing this up-scaling during the fine-tune process."}, {"start": 1553.2, "end": 1559.12, "text": " They somehow have to adjust the weights of the MLPs. So the reason that is, is because,"}, {"start": 1559.12, "end": 1570.32, "text": " if you remember, so the input image can be represented as S times C, and because the token"}, {"start": 1570.32, "end": 1577.84, "text": " mixer basically attends across the number of patches, when we increase the resolution,"}, {"start": 1577.84, "end": 1584.1599999999999, "text": " so when we increase the resolution, but we keep the number of, the size of the patch the same,"}, {"start": 1584.16, "end": 1590.16, "text": " that means we're going to have more of these patches. So that means S goes to S prime,"}, {"start": 1590.16, "end": 1596.3200000000002, "text": " which is a bigger number, and that means this thing won't fit into the previous MLP, which had"}, {"start": 1596.3200000000002, "end": 1602.0800000000002, "text": " certain bandwidth. So it was maybe, it was maybe, it could attend like this. It was smaller. So now"}, {"start": 1602.0800000000002, "end": 1612.3200000000002, "text": " we have to kind of take the weight matrix of that old MLP, and we need to initialize this new weight"}, {"start": 1612.32, "end": 1617.76, "text": " matrix, which is bigger somehow, and what they did is they just took these and stacked them"}, {"start": 1617.76, "end": 1626.32, "text": " block-wise here, and that was the initialization method they used to fine-tune the models."}, {"start": 1626.32, "end": 1630.96, "text": " So yeah, just an engineering detail. I thought it's interesting and important to mention"}, {"start": 1631.9199999999998, "end": 1638.56, "text": " that there are a lot of these details that go into making this thing work, and the reason this works"}, {"start": 1638.56, "end": 1646.48, "text": " is if you think about it, if you now multiply the input vector with this new matrix, this thing is"}, {"start": 1646.48, "end": 1650.8799999999999, "text": " basically going to attend to the first part, and this thing is basically going to attend to the"}, {"start": 1650.8799999999999, "end": 1658.6399999999999, "text": " second part. So it's like you're applying these in parallel, and so that seemed like a sound way to"}, {"start": 1658.6399999999999, "end": 1663.6, "text": " initialize the new weight matrix. So that's the main reason they did it like this. Hopefully you"}, {"start": 1663.6, "end": 1679.28, "text": " found this video useful. If you did, leave a like, share the video, and see you next time."}]