ConvNeXt: A ConvNet for the 2020s | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I cover the recently published "A ConvNet for the 2020s" paper. They show that ConvNets are still in the game! - by adding new design ideas and training procedures they outperform vision transformers even in big data regimes and without any attention layers.

Convolutional prior continues to stand the test of time in the field of computer vision.

Note: I also partially cover the Swin transformer paper in case you missed out on that one. :)

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2201.03545
✅ GitHub: https://github.com/facebookresearch/ConvNeXt
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro - convergence of transformers and CNNs
05:05 Main diagram explained
07:40 Main diagram corrections
10:10 Swin transformer recap
20:20 Modernizing ResNets
24:10 Diving deeper: stage ratio
27:20 Diving deeper: misc (inverted bottleneck, depthwise conv...)
34:45 Results (classification, object detection, segmentation)
37:35 RIP DanNet
38:40 Summary and outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#convnext #visiontransformers #computervision
 What's cracking guys? In this video I'm covering this new paper called A Comnet for the 2020s. The paper itself was followed by a lot of hype on social media, especially on Twitter. So yeah, let's see what it's all about. It's a paper by Facebook AI Research. I guess it's now MetaAI and UC Berkeley folks. So let's dig straight into it. The roaring 20s of visual recognition began with introduction of vision transformers or VATs, which quickly superseded Comnets as the state of the art image classification model. So the important detail here is it was very good. It was a soda, but only on image classification task. A vanilla VAT on the other hand, faces difficulties when applied to general computer vision tasks, such as object detection and semantic segmentation. So object detection and semantic segmentation. It is the hierarchical transformers like swing transformers that reintroduced several common priors, making transformers practically viable as a generic vision backbone. However, the effectiveness of such hybrid approaches is still largely credited to the intrinsic superiority of transformers rather than the inherent inductive biases of convolutions. So let me try and put this into broader historical perspective. Okay, so the story basically begins, I mean begins around 2011. That's when the first like successful CNN that has superhuman performance appeared. And the name of that CNN was not AlexNet, it was actually DanNet. So let me kind of put it here. So we have 2011 here and we have this branch of CNNs. I'm gonna draw it like this. So CNNs. And then in 2012, we have the most famous model that was AlexNet. And since then we had a bunch of different models, VGGs, Resnets, which were, Resnets were a big thing back in 2015. So maybe somewhere around this time. And all in all, that trend continued until 2020s. And then what happened is that basically in late 2020, we had Vision Transformer like published. And I've covered the paper by the way on VATs, if you're not familiar, you can check it out. I'm gonna link it somewhere here. So let me draw that as another thread in these like visual perception models. We have Transformers here and they all started with VAT. And what basically what VAT did is it petrified the image and then passed those tokens into Pure Transformer. And it just turned out to work very nicely for our image classification task. More importantly, for our story here is the appearance of Swin Transformer. So that was somewhere around. So this is like late 2020. This is early 2021. So that's like March 2021 is when the Swin Transformer appears and it stands for shifted window transformer. I think that would be the contraction. And from that point onward, basically Transformers were successful on semantic segmentation as well. And all of those more like types of visual tasks where you need dense predictions, such as semantic segmentation, instant segmentation, et cetera. So now let me draw, that's the kind of timeline. Now let me draw how these two branches of models started converging in a way. So what actually happened is that Swin Transformer kind of went this direction towards the CNN line. And now what this paper does is the same thing. It takes the most recent and best practices when it comes to training CNNs and combines those with the design ideas that Transformers introduced. So it does a similar thing. So basically what this paper does is starts going towards the transformer branch of models. So that's a mental model I have in my head at the moment basically. And now there is a boundary between these two. The reason being is that COMnext, which is the model from this paper, is still a pure CNN. So that means there is a boundary here because these are still pure CNNs. So the COMnext is still a pure CNN with some ideas from the transformer branch, whereas Swin Transformer arguably maybe even passes this virtual boundary because it has a lot of CNN priors as we are going to see very soon. So that's the basic mental model. And the whole idea is for me to show you the steps that happened here. So how did they go from ResNet-50, which is the baseline model they're using, which steps did they take, one, two, three, et cetera, to get to outperforming Vision Transformers and specifically Swin Transformer, which was the baseline they were comparing with against. So let's start with this diagram. It's one of the most important diagrams in this paper. And you can see a couple of things here. So first, we have two groups here. We have ImageNet-1K pre-trained models here. Let me just change the color here to red. And we have ImageNet-22K pre-trained models. So that's the superset of ImageNet dataset, which has around 22,000 classes. It's way bigger compared to this one. So yeah, the whole point is to see whether the models scale with additional data, which is something that VATs are notoriously famous for because they can learn the priors that the CNNs have. And you can check those out in my VAT paper, but for now, that's pretty much enough. So what we see here on the Y-axis is the top one accuracy on ImageNet. And you can see that ResNet here has lower accuracy compared to date. I'm not sure how to pronounce this one, but it's pretty much a plain VAT model. So that means little to no conf net biases, except for the patch, for the input patchify layer, which is arguably just a convolutional layer. Now we have the Swin transformer here. You can see that it outperforms the VAT. And finally, we can see the comnext here. Let me zoom in a little bit here. You can see that it outperforms Swin transformers. And not only that, so not only that we have higher performance, but you can also notice that you can also notice that by increasing the amount of computation, so by making models bigger, which you can see by the diameter of these circles here expressed in gigaflops, you can see that the performance goes up. So here for these smaller models, the accuracy was somewhere here. And then as they were increasing the model size, you can see that the center of these circles basically went up, which means they scale fairly well with model size. But the second dimension where they scale well across is also the data. And that's what we actually care arguably even more about. So here you can see that even with more data, the comnext architecture basically benefits from additional data. And that's an important finding. Again, comparisons with VAT, with Swin transformer and comnext, you can see that both the accuracy is better as well as the, I mean, scaling exists, that's what we are looking for pretty much. Now the thing is this diagram is a little bit misleading because as I said, there was a lot of hype following this paper and there were a lot of Twitter threads, let me just show you this one. So Lucas Baer, who is the co-author of the VAT paper, showed that they were actually comparing with the old VAT that still didn't have all of the modern augmentations. And if you add those additional augmentations, we see the top one accuracy translating upwards here, thus becoming almost on pair with comnext. So this convergence further supports my claims here that we have like a huge, huge like convergence happening between these two separate branches of perception models. A couple more comments here. The creator of this famous team library and the co-author of the Resonance Strikes Back paper, which I've covered in one of my previous videos, which also showed that if you train Resonance with modern training techniques and recipes, you basically get way better performance and we ought to be comparing with those novel Resonance and not with the old results from back from 2015. Yeah, he says here that while they did mention better Resonance training in the core of the paper, that headline grabbing plot used the originals. I rewrote some of the layer norm, general stuff, blah, blah, blah, and got the extra 20% boost. Basically that means that we need to shift upwards the Resonance as well and that the performance actually, like this gap here is closed down pretty much with Swin Transformers. It would be nice to see like a visualization, but like just keep that in your head. And finally, the co-author of the both efficient net paper, so both the version one and version two, says here that the auto ML models are already ahead by a couple of months, but this paper chooses to compare 2019 efficient net, we won instead of 2021 efficient net, we two, which is a paper I also covered, you can check it out if you're curious more about it. But the interesting thing here is, let's see some comparisons. Basically we have the efficient net, so let's take a look at these pre-trained only on ImageNet that has 1000 classes. You can see that efficient net, we two has higher accuracy, has much smaller amount of parameters, less computation necessary and three X bigger throughput, which means that efficient net we two is actually the common net for the 2020s. Cool, that's a necessary context before I start digging into the paper, but there is one more thing I want to cover and that's the Swin transformer, because it's arguably the model from which they were taking a lot of inspiration from, as we'll soon see, and that's why I wanted to kind of be familiar with Swin transformer, I haven't covered it before on my YouTube channel, so I'm gonna give you a quick introduction here. So what's the main idea here? I already mentioned that they are introducing certain CNN biases and now let me show you why that is. So first things first, we know that transformers scale poorly with the number of input tokens. So in the limit when a token is a single pixel, you have this many input tokens. So height times width, this is the number of tokens and we can see more importantly that we have quadratic scaling with the number of input tokens, which does not scale when you have huge resolutions, it's too computationally intensive pretty much. What this Swin transformer paper introduced is this windowed MSA, so the multi-head self-attention, plus the shifted window MSA, which is where they got the name from, so basically sliding windows, that's why they have the Swin part. And you can see here that, let me zoom in a little bit, that here they have a linear dependency on the number of input tokens, but they have a quadratic dependency on this M, which is usually kept constant around seven x seven, so seven times seven, and that's just the number of tokens inside of the window, basically. That's gonna be more clear in a couple of seconds. So all in all, this windowed MSA solves the quadratic relationship here. Okay, so let's see quickly how the thing works. And fairly simple, on the right-hand side here, you can see the original VAT, you can see that it's fairly uniform in the sense that every single, the resolutions, the spatial resolution is kept uniform across every single stage of VAT. So what happens is initially you patchify the image, and then you start processing it, processing the tokens using global self-attentions, which means every single token here, so this one here, will be able to attend all of the other tokens, so this one, and this one, and also this one. So every token can attend every single token. You basically have a fully connected graph here. Contrast that to Swin Transformer, where they introduced local computations. So that means here, you run your self-attention module, the MHA module, on these smaller parts of the input image. So basically you can see here four by four, and on each one of these windows, they'll be running, so the window is the terminology here, they'll be running a self-attention module. You can already see that this local computation is something that CNN prior is all about. I mean, you have basically kernels which you slide, you share the weights, and you have local computation, which is implicitly, you're implying that local neighborhood of pixels is more important than attending pixels which are far away. And that's something that for, at least for natural images, is usually the case, like there is a correlation, a strong correlation between neighboring pixels, and that's what CNNs are exploiting. So that understanding and human knowledge is basically ingrained, baked into the architecture itself. The second prior I can see here on top of my mind is the hierarchical approach, and that's why they call these hierarchical transformers. You can see that basically here we have self-attention acting on these fine-grained features, and then what they do is they take a pool of these windows, so let me change the color here. So let's say they take like four of these windows, and they merge all of those into this new window. So now here, we are processing the data on a coarser, like this is a coarse-grained layer, and then finally they do the same thing here, they repeat the procedure, and they have even more coarse-grained features here. So contrast that with the uniformity of the original transformers, and you'll see that a lot of this was basically, all of these ideas are stemming pretty much from the CNN branch of models. Now you may see a problem here, and that's that we are not sharing information across various spatial locations, and that can be problematic. So because you can see here, we are just processing, so these pixels here, these features, cannot learn and take information from the other, under other spatial locations here, so what they additionally do is they have this window-shifting technique. So let's understand why is this important, and how the mechanism works actually. So as I said, we need to have some mixing between various spatial locations, and how this technique does it, how it solves the problem is the following way. So in this initial, once you apply the self-attention module, that means that all of these tokens will contain information from all of the other tokens in the window by the very way how self-attention works. So that means that we'll have, so this token here will contain information from all of the other tokens, from this one, from this one, from this one, from all of the tokens. Whoops, let me draw it like this, and from this one, and that's it. Now in the next stage, once we are here, because we have a different layout, that means that the information that's contained here, so that means that this information here, which stems from this, that contains the information from all around, from all of these tokens in this quadrant, will now be kind of shared with the other tokens, and that means the following. So let me change the color here. That means that now, after we apply self-attention here, the information contained here, which is information that originally stems from here, will be shared to all of the other tokens. So this one, and this one, and all of the other tokens. Which means that now this token here contains information from these tokens here. When I say token, I mean the feature vector. And finally, once we get in the next stage, and we apply the self-attention using this layout again, you can see how this information here will be propagated to all of the other tokens here. So we'll be propagating it even to the top left. And that's the way how the information traverses this spatial extent here. That's the basic idea. Now I did mention that Swin works with these, like first on a fine-grained scale, and that means that each of these tokens is basically four by four, contains four by four pixels. So if you take a single token out here, it contains, let me change the color, it contains four times four pixels. Whereas the BAT has a more aggressive down sampling strategy, which means that this token here actually contains information from the 16 times 16 patch of the original image. So how they form the features, and this is a minor detail, I'm gonna mention it either way. So each patch is treated as a token, and its feature is set as a concatenation of the raw pixel RGB values. In our implementation, we use a patch size of four by four, as I just mentioned, and thus the feature dimension of each patch is four by four times three, and that's 48. And we have three because it's basically RGB image. So we just take the image, we take those four times four pixels, we take the RGB features and we just concatenate those, and that's what is fed into the screen transformer. So here is the architecture quickly. Again, you can see some resemblance with ComNets. After the initial patch partition layer, as they call it, we have, which hopefully makes sense now, so the spatial resolution is decreased by four because we have four times four pixels in a patch, and we have 48 because I just explained how we're forming those features. So that's why I explained those so that you're able to parse the numbers here. Then what happens is they apply a linear embedding, which effectively maps this 48 to C, even though they didn't mark that here. So what goes into this first transformer block is this, and the feature size is C, not 48. Now, why do they have two X here? And the reason is they are combining the windowed self-attention with this shifted window self-attention. And they apply those in a sequential manner, and then they do the patch merging, which I explained here. They have this merging of patches so that they have these core screened representations, and then they just keep on doing the same thing, windowed attention followed by sliding windowed attention, et cetera, et cetera. And what happens additionally is you can see here, C goes to two C, goes to four C, goes to eight C, which means we are increasing the number of channels as we are reducing the spatial dimension, which is also a pattern from CNNs. These transformer blocks themselves like are consist out of these two modules, which are your, this is your regular, this is a transformer block. The only difference is they're using, instead of the regular MSA, they're using windowed MSA and the sliding, this shift windowed MSA. I may be butchering the naming, the terminology here. I think it's, yeah, shifted windowing configuration. Got it. Cool. Okay, now that I've properly, hopefully properly, motivated the reason for this paper and some of the background knowledge like Swin Transformer and some of the, yeah, corrections to this main diagram, let's get to the rest of the paper. So our research is intended to bridge the gap between the pre-VIT and post-VIT eras for ComNets. To do this, we start with a standard ResNet, so ResNet-50 for example, trained with an improved procedure. We gradually modernize the architecture to the construction of a hierarchical vision transformer, Swin Tiny, T stands for tiny. So basically again, what they're saying here is, hey, we're taking some inspiration, design inspiration from Swin Transformers. Our exploration is directed by a quick question. How do design decisions in Transformers impact ComNet's performance? Okay, so that's the motivating statement there. Now, before I show this diagram, which is the second most important diagram, if not the most important diagram, let me just start with this. So a recent paper, which is basically this one, the ResNet Strike Spec, which I've covered in one of the previous videos, demonstrates how a set of modern training techniques can significantly enhance the performance of a simple ResNet-50 model. In our study, we use a training recipe that is close to DATES and Swin Transformers. So not quite sure why they did this, because this should have the best training procedure for ResNet-50. So I'm not sure why they're using training procedures from Transformers, probably because of the compatibility reasons, because they are later taking some design decisions from Transformers, and that's probably better. The compatibility is, I guess, better, compared to just taking the training ideas from this paper. So by itself, they say here, the enhanced training recipe increased the performance of the ResNet-50 model from 76.1 to 78.8, so plus 2.7% in top one accuracy, which is a huge boost. So with that out of the way, let's see the modernization procedure they applied. First, let me kind of dissect this diagram for you. What we can see here is in the bottom here, we have the Swin-T Transformer, and in the upper part here, we have ResNet-50. Just ignore the gray bars for now, because those correspond to these bigger alternatives, so to ResNet-200 and to Swin-B, which stands for bigger. And let's focus on these blue bars, and let's first see this orange bar. So this orange bar tells us that the top one accuracy, which is this horizontal axis here, is 81.3. And this star here, not sure if you can see it, but it says that we have 4.5 gigaflops of computation for the Swin-T model. Now, let's see what happened with ResNet. What's the progression here? So we start with 78.8, which is the numbers they got with this improved training procedure. So then they apply various techniques. They organize those across five groups. We have the macro design, we have the ResNext ideas, we have the inverted bottleneck, we have the large kernel, we have the micro design here. And you can see that all of those start, and all of those contribute to the final performance, and we can see that the final model has outperformed, so all of these outperform basically this Swin-T Transformer. And you can also see the stars here again denotes the gigaflops, the computation that's needed for a single forward pass. You can see it's varying throughout these, when they're doing these ablations, but they roughly oscillate around 4.5, which means these are comparable in that sense. Now I'm gonna focus in more detail on these first couple of techniques, and then I'm gonna slowly just scheme all of the other techniques, because there is a lot of details, and I'm gonna just kinda have to abstract some of those details away from you. So let's start with this stage ratio here. So what is stage ratio? What does that mean? They say here, Swin-T, on the other hand, follow the same principle, but with a slightly different stage compute ratio of one, one, three, one. We're gonna see what this means in a second. So for larger Swin-Transformers, the ratio is this one, and following the design, we adjust the number of blocks in each stage from three, four, six, three, in Resonant 50, to three, three, nine, three. So you can see when you divide all of this by three, you get the same ratio as in the Swin-Transformer right here. So the idea is basically they wanna have the amount of compute, the number of flops, across the model should have roughly the same ratio as the one in Swin-Transformer, because some of the other papers, they reference them here, show that that plays an important role in the design of novel architectures. So let me show you some code here, and remember these numbers here, three, four, six, three. So that's the number of blocks in each stage. So basically, if you take a look at the, let me zoom in a little bit here, at the API here, when you're constructing Resonant 50, or Resonant 101, or whatnot, you always call this underlying, this is by torch code, by the way, you call this common API function, and then you pass different number of blocks per stage, and that's what determines how big the model is, how many layers it has, and how many parameters does it has. So let me quickly show you this underscore Resonant function. So it's here, and we're passing, so that sequence of numbers is named layers, and we pass layers to this Resonant class here. So let me find the class, class Resonant. Okay, here it is. So it's passed as layers. So now let me search for layers, layers, and basically you can see here that those numbers, so three, six, four, three, or something, are passed here to these make layer functions, and they effectively, what they do is they say, take this block and repeat it this many times. So if we were to find the make layer function, I could probably just click and find, go by reference, but I'm being lazy here. We can see here that here is the blocks, so let's find the blocks, whoops, the blocks variable. Okay, blocks variable is called here, and you can see basically we have this layer append, and for the number of blocks, we're just going to be appending in this stage that amount of blocks. So that's why we have, that's how they manage to regulate the amount of computation across each of the stage of this resonant and reflect mirror the same type of ratios that are present in the Swin transformer. Cool, let me know whether you find this type of mixing of code and paper explanation useful or not. Let's continue here. So this design decision falls under this umbrella term of macro design. You can understand why that is. We are changing the macro structure of the architecture. Then they have this changing stem to patchify, and they mentioned here that a common stem cell with will aggressively down sample the input images. So that's something that's common in most of those older CNN architectures, and they say here that the stem cell in standard resonant contains a seven times seven convolution layer with try two, followed by a max pooling layer, which results in a forex down sampling of the input images, which isn't fairly aggressive down sampling. They say here, we replaced the resonant style stem cell with a patchify layer implemented using a four times four strive for convolutional layer. And that's how basically VAT like model works. What you do is you can interpret this patchifying strategy. Basically you have an image here. And what they do is they group these pixels into these patches. And then what they do is they do a linear projection of those pixels into some feature vector. And you can model that by just thinking of this as if you had, let me change the color, as if you had like a kernel, that's let's say in this case, they said four by four. So kernel that's four by four, and then you have a stride that has a size of four, which means the next step will be here, the next step will be here. And basically, yeah, that's how you implement. If you take a look at any code base, that's how you implement the patchify layer using a convolutional operation. Cool. So that's the second design decision. Then they mentioned here Resnextify. They take some ideas from the Resnext paper. So they say here, more precisely, Resnext employs grouped convolution for the three times three conv layer in a bottleneck block. As this significantly reduces the flops, the network width is expanded to compensate for the capacity loss. In our case, we use depthwise convolution, a special case of grouped convolution where the number of groups equals the number of channels. So a quick recap here, how the depthwise convolution works is the following. So let's say you have an activation volume. So that's the feature maps that comes out of your layer after you process and apply the activation functions. You get the activation volume, which looks something like this. Let me try and draw it here, something like this. And so how your regular convolutions work is that, you make a kernel, you make like a filter here, which is maybe usually three times three in most of these older CNN architectures. And now what I do here is the following. Let me just kind of draw it here roughly. So what I do is the depth of this filter is the same as the depth of your activation volume. So that's your regular convolution. Now what depthwise convolution does is the following, it slices this activation volume, like you have a single feature map here, then you have a second feature map, et cetera, et cetera. And what you do, you just take a filter that has this depth dimension is just one, and then you apply and apply that filter to this slice, and then you apply the second filter to the second slice, et cetera, et cetera. So again, let me draw that here. So the filter for the depthwise will be something like this. So again, let's say it's three times three again, but the depth component is just one. It's basically a single dimension basically. Now there is a problem in using depthwise convolution, and that's that you break the correlation between flops and the throughput, which means that basically for the same amount of compute, because of some memory issues, the throughput, i.e. the number of images you can process in a second or whatever unit of time, drops down dramatically. We're gonna address that a little bit later. And they did show that the throughput is competitive and even better than Swin Transformers, so that's cool. Now there are a lot more details here. I'm gonna skip some of those. They had the inverted block which was introduced. I'm gonna actually cover this one. So one important design in every transformer block is that it creates an inverted bottleneck. i.e. the hidden dimension of the MLP block is four times wider than the input dimension. And interestingly, this transformer design is connected to the inverted bottleneck design with an expansion ratio of four used in ComNets. The idea was popularized by MobileNet V2. So here we see again this cross-pollination from CNNs. So taking ideas from CNNs and implementing them even in the original transformer. So the idea was kinda informed by certain MobileNet architectures which came. I think they were developed prior to 2017 obviously. That's when the transformer paper originally was published. So what I said here, if you did not understand, and it's just a basic understanding of how transformers work, at least the usual architectures. So you have your MSA module here. So this is your MSA module. You have your input tokens. You have your output tokens. And then you apply an MLP. So each of these, this is a feature vector that has certain dimension like D. And what you apply MLP basically sharing the weights. So you just kinda do a for loop across these feature vectors and you apply the MLP which will map from D to 4D. So this feature vector is gonna be 4X longer. So one, two, three, four compared to this one. And then they again map it down into D dimensions. So that's what's referred to as a bottleneck layer. You can kinda imagine that you start from D and then you basically upscale it in a way to 4D and then you downscale it to D. So something like this. That's why it's called a bottleneck layer. Sorry, I meant to say inverted bottleneck the whole time. Not bottleneck layer because that one would look something like this. This would be your bottleneck layer. Okay, so this is a fun fact. I wasn't aware of that. Now here they just showed the progression of how they were additionally manipulating the modules. And I think there is a small error here. This should be I think 384 mapped down to 96 and not the other way around. But yeah, anyways, that's a small detail. Then they try and play with kernel sizes because as I said, usually you use three by three. They figure out that seven by seven works the best and that's it. Here they compare the Swin transformer block with the Resonant block with the final Comnex block. A couple more things they experimented with are the following. You can see that they are using in the Comnex block, they're using Jell-U. So that's Gaussian error linear unit or something instead of Relu. They have layer norm instead of batch norm. They have less normalization layers in general and less activation functions. You can see they apply the activation function only here. Let me change the color here. So they apply it only here, whereas here they apply it in multiple locations and all of that and they have, as you can see here, a bunch of batch norm layers, batch norm, batch norm. Here they only have a single layer norm. Okay, that's pretty much the level of detail I can tolerate for one video. So let me just slowly wrap this up. They say that our Comnex model has approximately the same number of flops, parameters, same throughput and memory use as the Swin transformer, but does not require specialized modules such as shift the window, the tension, or relative position biases, which is very cool because they get to have a simpler architecture. Finally, here are the results. They basically show that on image classification for all of various sizes, so both the small, the big one variant, for different resolutions, they outperform, as you can see here, they outperform Swin. So let's focus on a single pair here. Let's take, for example, Swin B and Comnex B working on 384 resolution. They have similar number of parameters. They have similar flops. Swin even has a bit more flops here and you can see that throughput is lower for Swin and you can see that the accuracy is better for Comnex. So basically across all of these dimensions, it outperforms the Swin model, which is very cool. They also showed that basically increasing the model, as we already mentioned, leads to increase in performance, which is something, a desirable thing. You wanna be able to scale the architecture, the size, and to get additional performance. Here, they additionally show the models pre-trained on the ImageNet 22K. And they showed that even this big data regime, where vision transformers are supposed to be superior, they are not only comparable, but they outperform. We again, focus on one of these pairs. Let me take this one. They have similar number of parameters again, similar flops. Again, actually, Swin takes more computation and it has lower throughput and it has lower accuracy. So again, it outperforms a Swin, even in the big data regime, which is fairly important. And wrapping this up, they also showed that on object detection tasks and semantic segmentation tasks, Swin transformer is again, you'll see fairly similar results as here on the image classification tests. So I don't wanna even dig into too much details. The last thing I wanna mention here is, so under similar flops, models with depth-wise convolutions are known to be slower and consume more memory than ComNets with only dense convolutions. It is natural to ask whether the design of ComNets will render it practically inefficient. As demonstrated throughout the paper, the inference throughputs of ComNets are comparable to or exceed that of Swin transformers. And that's something I was worried about, but it turns out that the throughput is actually kept on a healthy level. That's it. I wanna wrap this video by just mentioning one thing. And it's maybe a semi-rend, if that's even a word. So I just kinda skimmed through the references and I didn't see Schmidt-Huber mentioned anywhere. And it's kinda sad. I recently watched this video that Schmidt-Huber published to his YouTube channel where he explained the evolution of these earlier models. And it actually turned out, and I mentioned this in the beginning of this video, that prior to AlexNet, there was this model called DanNet. So it's spelled like this, I think, DanNet. And that model was the first convolutional neural network that showed superhuman performance on I think four or five vision benchmarks, although those benchmarks were arguably way smaller compared to ImageNet. It's kinda sad that DanNet got censored out of the AI history, even though it was such an important model back in 2011. So yeah, I encourage you to check out Schmidt-Huber's video and let me know your thoughts on this topic as well. Cool, let's do a quick summary here what we've seen in this video. So basically, we've seen this convergence going on between vision transformers on one hand side with Swin transformer being a representative that was used in this paper. And on the other hand side, we've seen ComNets being rejuvenated with novel training strategies such as the ones from the ResNet Strikes Again paper or from ComNEXT paper. We've also seen the results from the EfficientNet V2 paper, which even seem to outperform ComNEXT models. So it would be a fun comparison to compare them across multiple benchmarks and stuff like that. In a nutshell, it seems that convolutional priors are very good and hard to beat priors for natural images. And I'm very excited to see where these types of hybrid models will take us and whether we're gonna have some novel ideas that are going to boost the performance of these visual perception models even further. Anyways, if you found this video useful with all of these semi-rants, consider subscribing, sharing the video out and also do join the Discord community. I'll just link it down in the video description as always. So until next time, bye-bye. Thank you so much for watching, you
[{"start": 0.64, "end": 1.6, "text": " What's cracking guys?"}, {"start": 1.6, "end": 3.64, "text": " In this video I'm covering this new paper"}, {"start": 3.64, "end": 6.76, "text": " called A Comnet for the 2020s."}, {"start": 6.76, "end": 9.040000000000001, "text": " The paper itself was followed by a lot of hype"}, {"start": 9.040000000000001, "end": 11.56, "text": " on social media, especially on Twitter."}, {"start": 11.56, "end": 14.16, "text": " So yeah, let's see what it's all about."}, {"start": 14.16, "end": 16.76, "text": " It's a paper by Facebook AI Research."}, {"start": 16.76, "end": 21.2, "text": " I guess it's now MetaAI and UC Berkeley folks."}, {"start": 21.2, "end": 23.76, "text": " So let's dig straight into it."}, {"start": 23.76, "end": 27.14, "text": " The roaring 20s of visual recognition began"}, {"start": 27.14, "end": 30.16, "text": " with introduction of vision transformers or VATs,"}, {"start": 30.16, "end": 33.88, "text": " which quickly superseded Comnets as the state of the art"}, {"start": 33.88, "end": 35.44, "text": " image classification model."}, {"start": 35.44, "end": 39.24, "text": " So the important detail here is it was very good."}, {"start": 39.24, "end": 42.92, "text": " It was a soda, but only on image classification task."}, {"start": 42.92, "end": 45.519999999999996, "text": " A vanilla VAT on the other hand,"}, {"start": 45.519999999999996, "end": 46.96, "text": " faces difficulties when applied"}, {"start": 46.96, "end": 48.36, "text": " to general computer vision tasks,"}, {"start": 48.36, "end": 51.36, "text": " such as object detection and semantic segmentation."}, {"start": 51.36, "end": 53.74, "text": " So object detection and semantic segmentation."}, {"start": 53.74, "end": 57.56, "text": " It is the hierarchical transformers like swing transformers"}, {"start": 57.56, "end": 60.52, "text": " that reintroduced several common priors,"}, {"start": 60.52, "end": 63.0, "text": " making transformers practically viable"}, {"start": 63.0, "end": 64.64, "text": " as a generic vision backbone."}, {"start": 64.64, "end": 67.52000000000001, "text": " However, the effectiveness of such hybrid approaches"}, {"start": 67.52000000000001, "end": 71.64, "text": " is still largely credited to the intrinsic superiority"}, {"start": 71.64, "end": 75.5, "text": " of transformers rather than the inherent inductive biases"}, {"start": 75.5, "end": 76.86, "text": " of convolutions."}, {"start": 76.86, "end": 78.28, "text": " So let me try and put this"}, {"start": 78.28, "end": 81.24000000000001, "text": " into broader historical perspective."}, {"start": 81.24000000000001, "end": 83.52000000000001, "text": " Okay, so the story basically begins,"}, {"start": 83.52, "end": 86.56, "text": " I mean begins around 2011."}, {"start": 86.56, "end": 89.96, "text": " That's when the first like successful CNN"}, {"start": 89.96, "end": 92.38, "text": " that has superhuman performance appeared."}, {"start": 92.38, "end": 95.36, "text": " And the name of that CNN was not AlexNet,"}, {"start": 95.36, "end": 97.08, "text": " it was actually DanNet."}, {"start": 97.08, "end": 98.44, "text": " So let me kind of put it here."}, {"start": 98.44, "end": 102.72, "text": " So we have 2011 here and we have this branch of CNNs."}, {"start": 102.72, "end": 103.66, "text": " I'm gonna draw it like this."}, {"start": 103.66, "end": 105.16, "text": " So CNNs."}, {"start": 105.16, "end": 108.28, "text": " And then in 2012, we have the most famous model"}, {"start": 108.28, "end": 110.16, "text": " that was AlexNet."}, {"start": 110.16, "end": 112.64, "text": " And since then we had a bunch of different models,"}, {"start": 112.64, "end": 114.96, "text": " VGGs, Resnets, which were,"}, {"start": 114.96, "end": 117.92, "text": " Resnets were a big thing back in 2015."}, {"start": 117.92, "end": 120.42, "text": " So maybe somewhere around this time."}, {"start": 120.42, "end": 124.72, "text": " And all in all, that trend continued until 2020s."}, {"start": 124.72, "end": 129.12, "text": " And then what happened is that basically in late 2020,"}, {"start": 129.12, "end": 132.58, "text": " we had Vision Transformer like published."}, {"start": 132.58, "end": 135.28, "text": " And I've covered the paper by the way on VATs,"}, {"start": 135.28, "end": 136.84, "text": " if you're not familiar, you can check it out."}, {"start": 136.84, "end": 138.84, "text": " I'm gonna link it somewhere here."}, {"start": 138.84, "end": 141.64, "text": " So let me draw that as another thread"}, {"start": 141.64, "end": 144.6, "text": " in these like visual perception models."}, {"start": 144.6, "end": 149.04, "text": " We have Transformers here and they all started with VAT."}, {"start": 149.04, "end": 153.55999999999997, "text": " And what basically what VAT did is it petrified the image"}, {"start": 153.55999999999997, "end": 157.92, "text": " and then passed those tokens into Pure Transformer."}, {"start": 157.92, "end": 160.44, "text": " And it just turned out to work very nicely"}, {"start": 160.44, "end": 162.07999999999998, "text": " for our image classification task."}, {"start": 163.85999999999999, "end": 166.76, "text": " More importantly, for our story here"}, {"start": 166.76, "end": 169.27999999999997, "text": " is the appearance of Swin Transformer."}, {"start": 169.27999999999997, "end": 171.22, "text": " So that was somewhere around."}, {"start": 171.22, "end": 174.72, "text": " So this is like late 2020."}, {"start": 174.72, "end": 176.12, "text": " This is early 2021."}, {"start": 176.12, "end": 180.18, "text": " So that's like March 2021 is when the Swin Transformer"}, {"start": 180.18, "end": 184.24, "text": " appears and it stands for shifted window transformer."}, {"start": 184.24, "end": 186.44, "text": " I think that would be the contraction."}, {"start": 186.44, "end": 189.32, "text": " And from that point onward,"}, {"start": 189.32, "end": 191.64, "text": " basically Transformers were successful"}, {"start": 191.64, "end": 193.52, "text": " on semantic segmentation as well."}, {"start": 193.52, "end": 196.52, "text": " And all of those more like types of visual tasks"}, {"start": 196.52, "end": 197.8, "text": " where you need dense predictions,"}, {"start": 197.8, "end": 199.16, "text": " such as semantic segmentation,"}, {"start": 199.16, "end": 201.2, "text": " instant segmentation, et cetera."}, {"start": 201.2, "end": 204.94, "text": " So now let me draw, that's the kind of timeline."}, {"start": 204.94, "end": 208.76, "text": " Now let me draw how these two branches of models"}, {"start": 208.76, "end": 210.56, "text": " started converging in a way."}, {"start": 210.56, "end": 214.29999999999998, "text": " So what actually happened is that Swin Transformer"}, {"start": 214.29999999999998, "end": 217.5, "text": " kind of went this direction towards the CNN line."}, {"start": 218.33999999999997, "end": 223.23999999999998, "text": " And now what this paper does is the same thing."}, {"start": 223.23999999999998, "end": 227.06, "text": " It takes the most recent and best practices"}, {"start": 227.06, "end": 229.48, "text": " when it comes to training CNNs"}, {"start": 229.48, "end": 232.28, "text": " and combines those with the design ideas"}, {"start": 232.28, "end": 234.23999999999998, "text": " that Transformers introduced."}, {"start": 234.23999999999998, "end": 235.34, "text": " So it does a similar thing."}, {"start": 235.34, "end": 238.78, "text": " So basically what this paper does is"}, {"start": 238.78, "end": 243.6, "text": " starts going towards the transformer branch of models."}, {"start": 243.6, "end": 245.34, "text": " So that's a mental model I have in my head"}, {"start": 245.34, "end": 246.72, "text": " at the moment basically."}, {"start": 246.72, "end": 251.64, "text": " And now there is a boundary between these two."}, {"start": 251.64, "end": 253.72, "text": " The reason being is that COMnext,"}, {"start": 253.72, "end": 255.6, "text": " which is the model from this paper,"}, {"start": 255.6, "end": 257.56, "text": " is still a pure CNN."}, {"start": 257.56, "end": 260.4, "text": " So that means there is a boundary here"}, {"start": 260.4, "end": 262.52, "text": " because these are still pure CNNs."}, {"start": 262.52, "end": 265.12, "text": " So the COMnext is still a pure CNN"}, {"start": 265.12, "end": 267.52, "text": " with some ideas from the transformer branch,"}, {"start": 267.52, "end": 269.56, "text": " whereas Swin Transformer arguably"}, {"start": 269.56, "end": 272.68, "text": " maybe even passes this virtual boundary"}, {"start": 272.68, "end": 274.84000000000003, "text": " because it has a lot of CNN priors"}, {"start": 274.84000000000003, "end": 277.64, "text": " as we are going to see very soon."}, {"start": 277.64, "end": 280.44, "text": " So that's the basic mental model."}, {"start": 280.44, "end": 282.92, "text": " And the whole idea is for me to show you"}, {"start": 282.92, "end": 284.48, "text": " the steps that happened here."}, {"start": 284.48, "end": 287.76, "text": " So how did they go from ResNet-50,"}, {"start": 287.76, "end": 290.32, "text": " which is the baseline model they're using,"}, {"start": 290.32, "end": 293.44, "text": " which steps did they take, one, two, three, et cetera,"}, {"start": 293.44, "end": 298.24, "text": " to get to outperforming Vision Transformers"}, {"start": 298.24, "end": 300.90000000000003, "text": " and specifically Swin Transformer,"}, {"start": 300.90000000000003, "end": 304.52000000000004, "text": " which was the baseline they were comparing with against."}, {"start": 304.52000000000004, "end": 307.52000000000004, "text": " So let's start with this diagram."}, {"start": 307.52000000000004, "end": 310.02000000000004, "text": " It's one of the most important diagrams in this paper."}, {"start": 310.02000000000004, "end": 312.36, "text": " And you can see a couple of things here."}, {"start": 312.36, "end": 314.6, "text": " So first, we have two groups here."}, {"start": 314.6, "end": 317.72, "text": " We have ImageNet-1K pre-trained models here."}, {"start": 317.72, "end": 320.28000000000003, "text": " Let me just change the color here to red."}, {"start": 320.28000000000003, "end": 323.0, "text": " And we have ImageNet-22K pre-trained models."}, {"start": 323.0, "end": 325.56, "text": " So that's the superset of ImageNet dataset,"}, {"start": 325.56, "end": 328.36, "text": " which has around 22,000 classes."}, {"start": 328.36, "end": 330.04, "text": " It's way bigger compared to this one."}, {"start": 330.04, "end": 332.08000000000004, "text": " So yeah, the whole point is to see"}, {"start": 332.08000000000004, "end": 335.8, "text": " whether the models scale with additional data,"}, {"start": 335.8, "end": 339.2, "text": " which is something that VATs are notoriously famous for"}, {"start": 339.2, "end": 343.76, "text": " because they can learn the priors that the CNNs have."}, {"start": 343.76, "end": 346.59999999999997, "text": " And you can check those out in my VAT paper,"}, {"start": 346.59999999999997, "end": 350.15999999999997, "text": " but for now, that's pretty much enough."}, {"start": 350.15999999999997, "end": 352.36, "text": " So what we see here on the Y-axis"}, {"start": 352.36, "end": 355.64, "text": " is the top one accuracy on ImageNet."}, {"start": 355.64, "end": 357.68, "text": " And you can see that ResNet here"}, {"start": 359.8, "end": 362.59999999999997, "text": " has lower accuracy compared to date."}, {"start": 362.59999999999997, "end": 364.08, "text": " I'm not sure how to pronounce this one,"}, {"start": 364.08, "end": 366.38, "text": " but it's pretty much a plain VAT model."}, {"start": 366.38, "end": 371.38, "text": " So that means little to no conf net biases,"}, {"start": 371.56, "end": 374.88, "text": " except for the patch, for the input patchify layer,"}, {"start": 374.88, "end": 377.15999999999997, "text": " which is arguably just a convolutional layer."}, {"start": 378.64, "end": 380.04, "text": " Now we have the Swin transformer here."}, {"start": 380.04, "end": 383.04, "text": " You can see that it outperforms the VAT."}, {"start": 383.04, "end": 385.5, "text": " And finally, we can see the comnext here."}, {"start": 385.5, "end": 387.96, "text": " Let me zoom in a little bit here."}, {"start": 387.96, "end": 390.65999999999997, "text": " You can see that it outperforms Swin transformers."}, {"start": 390.65999999999997, "end": 393.44, "text": " And not only that, so not only that we have higher"}, {"start": 393.44, "end": 396.6, "text": " performance, but you can also notice that"}, {"start": 396.6, "end": 399.68, "text": " you can also notice that by increasing the amount"}, {"start": 399.68, "end": 402.52, "text": " of computation, so by making models bigger,"}, {"start": 402.52, "end": 405.68, "text": " which you can see by the diameter of these circles"}, {"start": 405.68, "end": 408.08, "text": " here expressed in gigaflops,"}, {"start": 408.08, "end": 409.78, "text": " you can see that the performance goes up."}, {"start": 409.78, "end": 412.88, "text": " So here for these smaller models,"}, {"start": 412.88, "end": 414.96, "text": " the accuracy was somewhere here."}, {"start": 414.96, "end": 417.46, "text": " And then as they were increasing the model size,"}, {"start": 417.46, "end": 421.2, "text": " you can see that the center of these circles"}, {"start": 421.2, "end": 424.71999999999997, "text": " basically went up, which means they scale fairly well"}, {"start": 424.71999999999997, "end": 427.24, "text": " with model size."}, {"start": 427.24, "end": 430.18, "text": " But the second dimension where they scale well across"}, {"start": 430.18, "end": 431.84, "text": " is also the data."}, {"start": 431.84, "end": 435.12, "text": " And that's what we actually care arguably even more about."}, {"start": 435.12, "end": 439.32, "text": " So here you can see that even with more data,"}, {"start": 439.32, "end": 441.4, "text": " the comnext architecture basically benefits"}, {"start": 441.4, "end": 443.36, "text": " from additional data."}, {"start": 443.36, "end": 444.91999999999996, "text": " And that's an important finding."}, {"start": 444.91999999999996, "end": 449.0, "text": " Again, comparisons with VAT, with Swin transformer"}, {"start": 449.0, "end": 453.68, "text": " and comnext, you can see that both the accuracy is better"}, {"start": 453.68, "end": 458.4, "text": " as well as the, I mean, scaling exists,"}, {"start": 458.4, "end": 460.36, "text": " that's what we are looking for pretty much."}, {"start": 460.36, "end": 463.0, "text": " Now the thing is this diagram is a little bit misleading"}, {"start": 463.0, "end": 465.24, "text": " because as I said, there was a lot of hype"}, {"start": 465.24, "end": 468.28, "text": " following this paper and there were a lot of"}, {"start": 468.28, "end": 471.12, "text": " Twitter threads, let me just show you this one."}, {"start": 471.12, "end": 475.4, "text": " So Lucas Baer, who is the co-author of the VAT paper,"}, {"start": 475.4, "end": 477.72, "text": " showed that they were actually comparing"}, {"start": 477.72, "end": 481.64000000000004, "text": " with the old VAT that still didn't have all of the modern"}, {"start": 481.64000000000004, "end": 482.6, "text": " augmentations."}, {"start": 482.6, "end": 484.76000000000005, "text": " And if you add those additional augmentations,"}, {"start": 484.76000000000005, "end": 488.48, "text": " we see the top one accuracy translating upwards here,"}, {"start": 488.48, "end": 492.16, "text": " thus becoming almost on pair with comnext."}, {"start": 492.16, "end": 495.88000000000005, "text": " So this convergence further supports my claims here"}, {"start": 495.88000000000005, "end": 499.12, "text": " that we have like a huge, huge like convergence happening"}, {"start": 499.12, "end": 504.12, "text": " between these two separate branches of perception models."}, {"start": 505.44000000000005, "end": 507.36, "text": " A couple more comments here."}, {"start": 507.36, "end": 510.36, "text": " The creator of this famous team library"}, {"start": 510.36, "end": 512.6, "text": " and the co-author of the Resonance Strikes Back paper,"}, {"start": 512.6, "end": 514.84, "text": " which I've covered in one of my previous videos,"}, {"start": 514.84, "end": 518.24, "text": " which also showed that if you train Resonance"}, {"start": 518.24, "end": 520.32, "text": " with modern training techniques and recipes,"}, {"start": 520.32, "end": 522.6, "text": " you basically get way better performance"}, {"start": 522.6, "end": 526.08, "text": " and we ought to be comparing with those novel Resonance"}, {"start": 526.08, "end": 529.0, "text": " and not with the old results from back from 2015."}, {"start": 530.08, "end": 533.2, "text": " Yeah, he says here that while they did mention"}, {"start": 533.2, "end": 535.16, "text": " better Resonance training in the core of the paper,"}, {"start": 535.16, "end": 537.36, "text": " that headline grabbing plot used the originals."}, {"start": 537.36, "end": 540.3199999999999, "text": " I rewrote some of the layer norm, general stuff,"}, {"start": 540.3199999999999, "end": 544.16, "text": " blah, blah, blah, and got the extra 20% boost."}, {"start": 544.16, "end": 546.76, "text": " Basically that means that we need to shift upwards"}, {"start": 546.76, "end": 550.28, "text": " the Resonance as well and that the performance actually,"}, {"start": 550.28, "end": 553.28, "text": " like this gap here is closed down pretty much"}, {"start": 553.28, "end": 554.3199999999999, "text": " with Swin Transformers."}, {"start": 554.3199999999999, "end": 556.16, "text": " It would be nice to see like a visualization,"}, {"start": 556.16, "end": 558.16, "text": " but like just keep that in your head."}, {"start": 558.16, "end": 561.68, "text": " And finally, the co-author of the both efficient net paper,"}, {"start": 561.68, "end": 563.64, "text": " so both the version one and version two,"}, {"start": 563.64, "end": 567.8, "text": " says here that the auto ML models are already ahead"}, {"start": 567.8, "end": 570.72, "text": " by a couple of months, but this paper chooses to compare"}, {"start": 570.72, "end": 575.4, "text": " 2019 efficient net, we won instead of 2021 efficient net,"}, {"start": 575.4, "end": 577.76, "text": " we two, which is a paper I also covered,"}, {"start": 577.76, "end": 580.64, "text": " you can check it out if you're curious more about it."}, {"start": 580.64, "end": 583.88, "text": " But the interesting thing here is,"}, {"start": 583.88, "end": 585.68, "text": " let's see some comparisons."}, {"start": 585.68, "end": 589.2, "text": " Basically we have the efficient net,"}, {"start": 589.2, "end": 591.64, "text": " so let's take a look at these pre-trained"}, {"start": 591.64, "end": 594.68, "text": " only on ImageNet that has 1000 classes."}, {"start": 594.68, "end": 598.68, "text": " You can see that efficient net, we two has higher accuracy,"}, {"start": 598.68, "end": 601.76, "text": " has much smaller amount of parameters,"}, {"start": 601.76, "end": 605.68, "text": " less computation necessary and three X bigger throughput,"}, {"start": 605.68, "end": 608.4, "text": " which means that efficient net we two"}, {"start": 608.4, "end": 611.92, "text": " is actually the common net for the 2020s."}, {"start": 611.92, "end": 616.0, "text": " Cool, that's a necessary context"}, {"start": 616.0, "end": 618.12, "text": " before I start digging into the paper,"}, {"start": 618.12, "end": 620.3199999999999, "text": " but there is one more thing I want to cover"}, {"start": 620.32, "end": 621.6, "text": " and that's the Swin transformer,"}, {"start": 621.6, "end": 626.1600000000001, "text": " because it's arguably the model from which"}, {"start": 626.1600000000001, "end": 628.2800000000001, "text": " they were taking a lot of inspiration from,"}, {"start": 628.2800000000001, "end": 631.0400000000001, "text": " as we'll soon see, and that's why I wanted to kind of"}, {"start": 631.0400000000001, "end": 632.6400000000001, "text": " be familiar with Swin transformer,"}, {"start": 632.6400000000001, "end": 634.72, "text": " I haven't covered it before on my YouTube channel,"}, {"start": 634.72, "end": 637.36, "text": " so I'm gonna give you a quick introduction here."}, {"start": 637.36, "end": 639.6, "text": " So what's the main idea here?"}, {"start": 640.8000000000001, "end": 642.6, "text": " I already mentioned that they are introducing"}, {"start": 642.6, "end": 646.2800000000001, "text": " certain CNN biases and now let me show you why that is."}, {"start": 646.2800000000001, "end": 648.96, "text": " So first things first, we know that transformers"}, {"start": 648.96, "end": 651.88, "text": " scale poorly with the number of input tokens."}, {"start": 651.88, "end": 656.0400000000001, "text": " So in the limit when a token is a single pixel,"}, {"start": 656.0400000000001, "end": 658.44, "text": " you have this many input tokens."}, {"start": 658.44, "end": 662.12, "text": " So height times width, this is the number of tokens"}, {"start": 662.12, "end": 663.8000000000001, "text": " and we can see more importantly"}, {"start": 663.8000000000001, "end": 665.6, "text": " that we have quadratic scaling"}, {"start": 666.5600000000001, "end": 668.12, "text": " with the number of input tokens,"}, {"start": 668.12, "end": 671.12, "text": " which does not scale when you have huge resolutions,"}, {"start": 671.12, "end": 673.84, "text": " it's too computationally intensive pretty much."}, {"start": 673.84, "end": 676.2800000000001, "text": " What this Swin transformer paper introduced"}, {"start": 676.28, "end": 681.28, "text": " is this windowed MSA, so the multi-head self-attention,"}, {"start": 682.16, "end": 684.76, "text": " plus the shifted window MSA,"}, {"start": 684.76, "end": 688.0, "text": " which is where they got the name from,"}, {"start": 688.0, "end": 690.64, "text": " so basically sliding windows,"}, {"start": 690.64, "end": 693.04, "text": " that's why they have the Swin part."}, {"start": 693.04, "end": 696.12, "text": " And you can see here that, let me zoom in a little bit,"}, {"start": 696.12, "end": 698.64, "text": " that here they have a linear dependency"}, {"start": 698.64, "end": 700.8399999999999, "text": " on the number of input tokens,"}, {"start": 700.8399999999999, "end": 703.8, "text": " but they have a quadratic dependency on this M,"}, {"start": 703.8, "end": 706.76, "text": " which is usually kept constant around seven x seven,"}, {"start": 706.76, "end": 710.3599999999999, "text": " so seven times seven, and that's just the number of tokens"}, {"start": 710.3599999999999, "end": 713.88, "text": " inside of the window, basically."}, {"start": 713.88, "end": 716.16, "text": " That's gonna be more clear in a couple of seconds."}, {"start": 716.16, "end": 719.56, "text": " So all in all, this windowed MSA"}, {"start": 719.56, "end": 723.3599999999999, "text": " solves the quadratic relationship here."}, {"start": 723.3599999999999, "end": 726.28, "text": " Okay, so let's see quickly how the thing works."}, {"start": 726.28, "end": 729.52, "text": " And fairly simple, on the right-hand side here,"}, {"start": 729.52, "end": 730.76, "text": " you can see the original VAT,"}, {"start": 730.76, "end": 732.4399999999999, "text": " you can see that it's fairly uniform"}, {"start": 732.44, "end": 734.24, "text": " in the sense that every single,"}, {"start": 734.24, "end": 737.32, "text": " the resolutions, the spatial resolution is kept uniform"}, {"start": 737.32, "end": 740.08, "text": " across every single stage of VAT."}, {"start": 740.08, "end": 743.24, "text": " So what happens is initially you patchify the image,"}, {"start": 743.24, "end": 745.5600000000001, "text": " and then you start processing it,"}, {"start": 745.5600000000001, "end": 748.72, "text": " processing the tokens using global self-attentions,"}, {"start": 748.72, "end": 752.12, "text": " which means every single token here, so this one here,"}, {"start": 752.12, "end": 754.44, "text": " will be able to attend all of the other tokens,"}, {"start": 754.44, "end": 757.9200000000001, "text": " so this one, and this one, and also this one."}, {"start": 757.9200000000001, "end": 760.2800000000001, "text": " So every token can attend every single token."}, {"start": 760.28, "end": 763.52, "text": " You basically have a fully connected graph here."}, {"start": 763.52, "end": 765.9599999999999, "text": " Contrast that to Swin Transformer,"}, {"start": 765.9599999999999, "end": 769.72, "text": " where they introduced local computations."}, {"start": 769.72, "end": 773.92, "text": " So that means here, you run your self-attention module,"}, {"start": 773.92, "end": 778.92, "text": " the MHA module, on these smaller parts of the input image."}, {"start": 779.72, "end": 782.3199999999999, "text": " So basically you can see here four by four,"}, {"start": 782.3199999999999, "end": 785.56, "text": " and on each one of these windows, they'll be running,"}, {"start": 785.56, "end": 787.6, "text": " so the window is the terminology here,"}, {"start": 787.6, "end": 789.4, "text": " they'll be running a self-attention module."}, {"start": 789.4, "end": 791.6, "text": " You can already see that this local computation"}, {"start": 791.6, "end": 794.92, "text": " is something that CNN prior is all about."}, {"start": 794.92, "end": 798.02, "text": " I mean, you have basically kernels which you slide,"}, {"start": 798.02, "end": 800.48, "text": " you share the weights, and you have local computation,"}, {"start": 800.48, "end": 803.4399999999999, "text": " which is implicitly, you're implying that"}, {"start": 803.4399999999999, "end": 805.8, "text": " local neighborhood of pixels is more important"}, {"start": 805.8, "end": 809.48, "text": " than attending pixels which are far away."}, {"start": 809.48, "end": 812.0799999999999, "text": " And that's something that for, at least for natural images,"}, {"start": 812.0799999999999, "end": 814.04, "text": " is usually the case, like there is a correlation,"}, {"start": 814.04, "end": 816.12, "text": " a strong correlation between neighboring pixels,"}, {"start": 816.12, "end": 818.34, "text": " and that's what CNNs are exploiting."}, {"start": 818.34, "end": 819.72, "text": " So that understanding and human knowledge"}, {"start": 819.72, "end": 823.7800000000001, "text": " is basically ingrained, baked into the architecture itself."}, {"start": 823.7800000000001, "end": 827.0, "text": " The second prior I can see here on top of my mind"}, {"start": 827.0, "end": 829.32, "text": " is the hierarchical approach,"}, {"start": 829.32, "end": 832.58, "text": " and that's why they call these hierarchical transformers."}, {"start": 832.58, "end": 835.6800000000001, "text": " You can see that basically here we have"}, {"start": 835.6800000000001, "end": 838.64, "text": " self-attention acting on these fine-grained features,"}, {"start": 838.64, "end": 842.08, "text": " and then what they do is they take a pool of these windows,"}, {"start": 842.08, "end": 843.5600000000001, "text": " so let me change the color here."}, {"start": 843.5600000000001, "end": 846.2800000000001, "text": " So let's say they take like four of these windows,"}, {"start": 846.28, "end": 849.3199999999999, "text": " and they merge all of those into this new window."}, {"start": 849.3199999999999, "end": 853.92, "text": " So now here, we are processing the data on a coarser,"}, {"start": 853.92, "end": 857.62, "text": " like this is a coarse-grained layer,"}, {"start": 857.62, "end": 859.8399999999999, "text": " and then finally they do the same thing here,"}, {"start": 859.8399999999999, "end": 861.3399999999999, "text": " they repeat the procedure,"}, {"start": 861.3399999999999, "end": 864.8199999999999, "text": " and they have even more coarse-grained features here."}, {"start": 864.8199999999999, "end": 866.8399999999999, "text": " So contrast that with the uniformity"}, {"start": 866.8399999999999, "end": 868.56, "text": " of the original transformers,"}, {"start": 868.56, "end": 872.28, "text": " and you'll see that a lot of this was basically,"}, {"start": 872.28, "end": 874.78, "text": " all of these ideas are stemming pretty much"}, {"start": 874.78, "end": 876.4599999999999, "text": " from the CNN branch of models."}, {"start": 876.4599999999999, "end": 877.9599999999999, "text": " Now you may see a problem here,"}, {"start": 879.0799999999999, "end": 881.4, "text": " and that's that we are not sharing information"}, {"start": 881.4, "end": 883.0799999999999, "text": " across various spatial locations,"}, {"start": 883.0799999999999, "end": 885.1999999999999, "text": " and that can be problematic."}, {"start": 885.1999999999999, "end": 888.28, "text": " So because you can see here, we are just processing,"}, {"start": 888.28, "end": 890.28, "text": " so these pixels here, these features,"}, {"start": 890.28, "end": 893.3199999999999, "text": " cannot learn and take information from the other,"}, {"start": 894.52, "end": 896.52, "text": " under other spatial locations here,"}, {"start": 896.52, "end": 897.52, "text": " so what they additionally do"}, {"start": 897.52, "end": 901.36, "text": " is they have this window-shifting technique."}, {"start": 901.36, "end": 903.92, "text": " So let's understand why is this important,"}, {"start": 903.92, "end": 906.9599999999999, "text": " and how the mechanism works actually."}, {"start": 906.9599999999999, "end": 909.62, "text": " So as I said, we need to have some mixing"}, {"start": 909.62, "end": 911.92, "text": " between various spatial locations,"}, {"start": 911.92, "end": 913.7199999999999, "text": " and how this technique does it,"}, {"start": 913.7199999999999, "end": 916.16, "text": " how it solves the problem is the following way."}, {"start": 916.16, "end": 920.3399999999999, "text": " So in this initial, once you apply the self-attention module,"}, {"start": 920.3399999999999, "end": 921.8399999999999, "text": " that means that all of these tokens"}, {"start": 921.8399999999999, "end": 924.3, "text": " will contain information from all of the other tokens"}, {"start": 924.3, "end": 928.56, "text": " in the window by the very way how self-attention works."}, {"start": 928.56, "end": 930.16, "text": " So that means that we'll have,"}, {"start": 930.16, "end": 932.52, "text": " so this token here will contain information"}, {"start": 932.52, "end": 935.0, "text": " from all of the other tokens, from this one, from this one,"}, {"start": 935.0, "end": 937.22, "text": " from this one, from all of the tokens."}, {"start": 937.22, "end": 940.56, "text": " Whoops, let me draw it like this, and from this one,"}, {"start": 940.56, "end": 941.56, "text": " and that's it."}, {"start": 941.56, "end": 944.3, "text": " Now in the next stage, once we are here,"}, {"start": 944.3, "end": 945.92, "text": " because we have a different layout,"}, {"start": 945.92, "end": 948.48, "text": " that means that the information that's contained here,"}, {"start": 948.48, "end": 950.56, "text": " so that means that this information here,"}, {"start": 950.56, "end": 953.26, "text": " which stems from this, that contains the information"}, {"start": 953.26, "end": 956.36, "text": " from all around, from all of these tokens in this quadrant,"}, {"start": 956.36, "end": 960.06, "text": " will now be kind of shared with the other tokens,"}, {"start": 960.06, "end": 961.3199999999999, "text": " and that means the following."}, {"start": 961.32, "end": 963.44, "text": " So let me change the color here."}, {"start": 963.44, "end": 967.1600000000001, "text": " That means that now, after we apply self-attention here,"}, {"start": 967.1600000000001, "end": 968.7600000000001, "text": " the information contained here,"}, {"start": 968.7600000000001, "end": 971.2800000000001, "text": " which is information that originally stems from here,"}, {"start": 971.2800000000001, "end": 974.62, "text": " will be shared to all of the other tokens."}, {"start": 974.62, "end": 977.5600000000001, "text": " So this one, and this one, and all of the other tokens."}, {"start": 977.5600000000001, "end": 980.4000000000001, "text": " Which means that now this token here"}, {"start": 980.4000000000001, "end": 984.44, "text": " contains information from these tokens here."}, {"start": 984.44, "end": 986.5, "text": " When I say token, I mean the feature vector."}, {"start": 986.5, "end": 990.0400000000001, "text": " And finally, once we get in the next stage,"}, {"start": 990.04, "end": 994.76, "text": " and we apply the self-attention using this layout again,"}, {"start": 994.76, "end": 998.26, "text": " you can see how this information here"}, {"start": 998.26, "end": 1002.36, "text": " will be propagated to all of the other tokens here."}, {"start": 1002.36, "end": 1005.74, "text": " So we'll be propagating it even to the top left."}, {"start": 1005.74, "end": 1008.64, "text": " And that's the way how the information traverses"}, {"start": 1008.64, "end": 1011.1999999999999, "text": " this spatial extent here."}, {"start": 1011.1999999999999, "end": 1012.88, "text": " That's the basic idea."}, {"start": 1012.88, "end": 1016.4599999999999, "text": " Now I did mention that Swin works with these,"}, {"start": 1016.4599999999999, "end": 1019.52, "text": " like first on a fine-grained scale,"}, {"start": 1019.52, "end": 1021.84, "text": " and that means that each of these tokens"}, {"start": 1021.84, "end": 1025.36, "text": " is basically four by four, contains four by four pixels."}, {"start": 1025.36, "end": 1027.72, "text": " So if you take a single token out here,"}, {"start": 1027.72, "end": 1029.96, "text": " it contains, let me change the color,"}, {"start": 1029.96, "end": 1034.92, "text": " it contains four times four pixels."}, {"start": 1034.92, "end": 1039.92, "text": " Whereas the BAT has a more aggressive down sampling strategy,"}, {"start": 1040.94, "end": 1042.6, "text": " which means that this token here"}, {"start": 1042.6, "end": 1047.6, "text": " actually contains information from the 16 times 16 patch"}, {"start": 1047.6, "end": 1050.1999999999998, "text": " of the original image."}, {"start": 1050.1999999999998, "end": 1052.36, "text": " So how they form the features,"}, {"start": 1052.36, "end": 1054.04, "text": " and this is a minor detail,"}, {"start": 1054.04, "end": 1055.8999999999999, "text": " I'm gonna mention it either way."}, {"start": 1055.8999999999999, "end": 1060.56, "text": " So each patch is treated as a token,"}, {"start": 1060.56, "end": 1063.5, "text": " and its feature is set as a concatenation"}, {"start": 1063.5, "end": 1066.34, "text": " of the raw pixel RGB values."}, {"start": 1066.34, "end": 1069.04, "text": " In our implementation, we use a patch size of four by four,"}, {"start": 1069.04, "end": 1070.48, "text": " as I just mentioned,"}, {"start": 1070.48, "end": 1072.9599999999998, "text": " and thus the feature dimension of each patch"}, {"start": 1072.9599999999998, "end": 1076.56, "text": " is four by four times three, and that's 48."}, {"start": 1076.56, "end": 1080.08, "text": " And we have three because it's basically RGB image."}, {"start": 1080.08, "end": 1081.8999999999999, "text": " So we just take the image,"}, {"start": 1081.8999999999999, "end": 1084.3999999999999, "text": " we take those four times four pixels,"}, {"start": 1084.3999999999999, "end": 1087.3799999999999, "text": " we take the RGB features and we just concatenate those,"}, {"start": 1087.3799999999999, "end": 1091.44, "text": " and that's what is fed into the screen transformer."}, {"start": 1091.44, "end": 1093.84, "text": " So here is the architecture quickly."}, {"start": 1093.84, "end": 1096.6399999999999, "text": " Again, you can see some resemblance with ComNets."}, {"start": 1096.6399999999999, "end": 1099.7, "text": " After the initial patch partition layer, as they call it,"}, {"start": 1099.7, "end": 1102.48, "text": " we have, which hopefully makes sense now,"}, {"start": 1102.48, "end": 1104.76, "text": " so the spatial resolution is decreased by four"}, {"start": 1104.76, "end": 1107.96, "text": " because we have four times four pixels in a patch,"}, {"start": 1107.96, "end": 1109.96, "text": " and we have 48 because I just explained"}, {"start": 1109.96, "end": 1111.32, "text": " how we're forming those features."}, {"start": 1111.32, "end": 1112.8, "text": " So that's why I explained those"}, {"start": 1112.8, "end": 1115.74, "text": " so that you're able to parse the numbers here."}, {"start": 1115.74, "end": 1118.24, "text": " Then what happens is they apply a linear embedding,"}, {"start": 1118.24, "end": 1123.24, "text": " which effectively maps this 48 to C,"}, {"start": 1124.18, "end": 1126.26, "text": " even though they didn't mark that here."}, {"start": 1126.26, "end": 1130.36, "text": " So what goes into this first transformer block is this,"}, {"start": 1130.36, "end": 1133.68, "text": " and the feature size is C, not 48."}, {"start": 1133.68, "end": 1136.9, "text": " Now, why do they have two X here?"}, {"start": 1136.9, "end": 1138.8200000000002, "text": " And the reason is they are combining"}, {"start": 1138.8200000000002, "end": 1140.6200000000001, "text": " the windowed self-attention"}, {"start": 1140.6200000000001, "end": 1143.88, "text": " with this shifted window self-attention."}, {"start": 1144.74, "end": 1148.66, "text": " And they apply those in a sequential manner,"}, {"start": 1148.66, "end": 1152.6200000000001, "text": " and then they do the patch merging, which I explained here."}, {"start": 1152.6200000000001, "end": 1154.54, "text": " They have this merging of patches"}, {"start": 1154.54, "end": 1158.1200000000001, "text": " so that they have these core screened representations,"}, {"start": 1158.1200000000001, "end": 1161.22, "text": " and then they just keep on doing the same thing,"}, {"start": 1161.22, "end": 1165.26, "text": " windowed attention followed by sliding windowed attention,"}, {"start": 1165.26, "end": 1166.5, "text": " et cetera, et cetera."}, {"start": 1166.5, "end": 1168.6200000000001, "text": " And what happens additionally is you can see here,"}, {"start": 1168.6200000000001, "end": 1172.06, "text": " C goes to two C, goes to four C, goes to eight C,"}, {"start": 1172.06, "end": 1176.42, "text": " which means we are increasing the number of channels"}, {"start": 1176.42, "end": 1178.42, "text": " as we are reducing the spatial dimension,"}, {"start": 1178.42, "end": 1180.46, "text": " which is also a pattern from CNNs."}, {"start": 1180.46, "end": 1182.66, "text": " These transformer blocks themselves"}, {"start": 1182.66, "end": 1185.3, "text": " like are consist out of these two modules,"}, {"start": 1185.3, "end": 1187.06, "text": " which are your, this is your regular,"}, {"start": 1187.06, "end": 1188.14, "text": " this is a transformer block."}, {"start": 1188.14, "end": 1189.42, "text": " The only difference is they're using,"}, {"start": 1189.42, "end": 1192.6200000000001, "text": " instead of the regular MSA, they're using windowed MSA"}, {"start": 1192.6200000000001, "end": 1197.0600000000002, "text": " and the sliding, this shift windowed MSA."}, {"start": 1197.0600000000002, "end": 1200.26, "text": " I may be butchering the naming, the terminology here."}, {"start": 1200.26, "end": 1202.7, "text": " I think it's, yeah, shifted windowing configuration."}, {"start": 1202.7, "end": 1204.02, "text": " Got it."}, {"start": 1204.02, "end": 1205.1000000000001, "text": " Cool."}, {"start": 1205.1000000000001, "end": 1208.14, "text": " Okay, now that I've properly, hopefully properly,"}, {"start": 1208.14, "end": 1210.5, "text": " motivated the reason for this paper"}, {"start": 1210.5, "end": 1213.0600000000002, "text": " and some of the background knowledge like Swin Transformer"}, {"start": 1213.0600000000002, "end": 1217.7, "text": " and some of the, yeah, corrections to this main diagram,"}, {"start": 1217.7, "end": 1219.82, "text": " let's get to the rest of the paper."}, {"start": 1220.6200000000001, "end": 1222.78, "text": " So our research is intended to bridge the gap"}, {"start": 1222.78, "end": 1226.82, "text": " between the pre-VIT and post-VIT eras for ComNets."}, {"start": 1226.82, "end": 1229.3400000000001, "text": " To do this, we start with a standard ResNet,"}, {"start": 1229.3400000000001, "end": 1233.5800000000002, "text": " so ResNet-50 for example, trained with an improved procedure."}, {"start": 1233.5800000000002, "end": 1236.1000000000001, "text": " We gradually modernize the architecture"}, {"start": 1236.1000000000001, "end": 1239.26, "text": " to the construction of a hierarchical vision transformer,"}, {"start": 1239.26, "end": 1241.9, "text": " Swin Tiny, T stands for tiny."}, {"start": 1241.9, "end": 1244.14, "text": " So basically again, what they're saying here is,"}, {"start": 1244.14, "end": 1246.06, "text": " hey, we're taking some inspiration,"}, {"start": 1246.06, "end": 1248.86, "text": " design inspiration from Swin Transformers."}, {"start": 1248.86, "end": 1251.4199999999998, "text": " Our exploration is directed by a quick question."}, {"start": 1251.4199999999998, "end": 1254.8999999999999, "text": " How do design decisions in Transformers"}, {"start": 1254.8999999999999, "end": 1256.98, "text": " impact ComNet's performance?"}, {"start": 1256.98, "end": 1260.74, "text": " Okay, so that's the motivating statement there."}, {"start": 1260.74, "end": 1263.1799999999998, "text": " Now, before I show this diagram,"}, {"start": 1263.1799999999998, "end": 1264.98, "text": " which is the second most important diagram,"}, {"start": 1264.98, "end": 1267.22, "text": " if not the most important diagram,"}, {"start": 1267.22, "end": 1269.1399999999999, "text": " let me just start with this."}, {"start": 1269.1399999999999, "end": 1272.4199999999998, "text": " So a recent paper, which is basically this one,"}, {"start": 1272.4199999999998, "end": 1273.6599999999999, "text": " the ResNet Strike Spec,"}, {"start": 1273.66, "end": 1276.1000000000001, "text": " which I've covered in one of the previous videos,"}, {"start": 1276.1000000000001, "end": 1278.46, "text": " demonstrates how a set of modern training techniques"}, {"start": 1278.46, "end": 1280.5400000000002, "text": " can significantly enhance the performance"}, {"start": 1280.5400000000002, "end": 1283.14, "text": " of a simple ResNet-50 model."}, {"start": 1283.14, "end": 1285.94, "text": " In our study, we use a training recipe"}, {"start": 1285.94, "end": 1290.5400000000002, "text": " that is close to DATES and Swin Transformers."}, {"start": 1290.5400000000002, "end": 1292.3000000000002, "text": " So not quite sure why they did this,"}, {"start": 1292.3000000000002, "end": 1295.38, "text": " because this should have the best training procedure"}, {"start": 1295.38, "end": 1296.5, "text": " for ResNet-50."}, {"start": 1296.5, "end": 1300.9, "text": " So I'm not sure why they're using training procedures"}, {"start": 1300.9, "end": 1302.02, "text": " from Transformers,"}, {"start": 1302.02, "end": 1304.58, "text": " probably because of the compatibility reasons,"}, {"start": 1304.58, "end": 1306.78, "text": " because they are later taking some design decisions"}, {"start": 1306.78, "end": 1308.86, "text": " from Transformers, and that's probably better."}, {"start": 1308.86, "end": 1312.42, "text": " The compatibility is, I guess, better,"}, {"start": 1312.42, "end": 1317.42, "text": " compared to just taking the training ideas from this paper."}, {"start": 1317.62, "end": 1319.22, "text": " So by itself, they say here,"}, {"start": 1319.22, "end": 1322.66, "text": " the enhanced training recipe increased the performance"}, {"start": 1322.66, "end": 1327.66, "text": " of the ResNet-50 model from 76.1 to 78.8,"}, {"start": 1328.26, "end": 1331.42, "text": " so plus 2.7% in top one accuracy,"}, {"start": 1331.42, "end": 1332.8600000000001, "text": " which is a huge boost."}, {"start": 1332.8600000000001, "end": 1334.2, "text": " So with that out of the way,"}, {"start": 1334.2, "end": 1337.7, "text": " let's see the modernization procedure they applied."}, {"start": 1337.7, "end": 1341.22, "text": " First, let me kind of dissect this diagram for you."}, {"start": 1341.22, "end": 1343.1200000000001, "text": " What we can see here is in the bottom here,"}, {"start": 1343.1200000000001, "end": 1344.9, "text": " we have the Swin-T Transformer,"}, {"start": 1344.9, "end": 1348.94, "text": " and in the upper part here, we have ResNet-50."}, {"start": 1348.94, "end": 1350.72, "text": " Just ignore the gray bars for now,"}, {"start": 1350.72, "end": 1353.98, "text": " because those correspond to these bigger alternatives,"}, {"start": 1353.98, "end": 1357.76, "text": " so to ResNet-200 and to Swin-B, which stands for bigger."}, {"start": 1358.74, "end": 1360.78, "text": " And let's focus on these blue bars,"}, {"start": 1360.78, "end": 1363.02, "text": " and let's first see this orange bar."}, {"start": 1363.02, "end": 1366.12, "text": " So this orange bar tells us that the top one accuracy,"}, {"start": 1366.12, "end": 1370.3799999999999, "text": " which is this horizontal axis here, is 81.3."}, {"start": 1370.3799999999999, "end": 1372.58, "text": " And this star here, not sure if you can see it,"}, {"start": 1372.58, "end": 1377.58, "text": " but it says that we have 4.5 gigaflops of computation"}, {"start": 1379.3, "end": 1380.7, "text": " for the Swin-T model."}, {"start": 1380.7, "end": 1384.1, "text": " Now, let's see what happened with ResNet."}, {"start": 1384.1, "end": 1385.42, "text": " What's the progression here?"}, {"start": 1385.42, "end": 1389.42, "text": " So we start with 78.8, which is the numbers they got"}, {"start": 1389.42, "end": 1391.98, "text": " with this improved training procedure."}, {"start": 1391.98, "end": 1393.94, "text": " So then they apply various techniques."}, {"start": 1393.94, "end": 1395.78, "text": " They organize those across five groups."}, {"start": 1395.78, "end": 1398.74, "text": " We have the macro design, we have the ResNext ideas,"}, {"start": 1398.74, "end": 1402.0600000000002, "text": " we have the inverted bottleneck, we have the large kernel,"}, {"start": 1402.0600000000002, "end": 1404.0600000000002, "text": " we have the micro design here."}, {"start": 1404.0600000000002, "end": 1406.3400000000001, "text": " And you can see that all of those start,"}, {"start": 1406.3400000000001, "end": 1409.46, "text": " and all of those contribute to the final performance,"}, {"start": 1409.46, "end": 1413.14, "text": " and we can see that the final model has outperformed,"}, {"start": 1413.14, "end": 1417.14, "text": " so all of these outperform basically this Swin-T Transformer."}, {"start": 1417.14, "end": 1420.42, "text": " And you can also see the stars here again denotes"}, {"start": 1420.42, "end": 1424.0600000000002, "text": " the gigaflops, the computation that's needed"}, {"start": 1424.0600000000002, "end": 1425.9, "text": " for a single forward pass."}, {"start": 1425.9, "end": 1428.18, "text": " You can see it's varying throughout these,"}, {"start": 1428.18, "end": 1430.0200000000002, "text": " when they're doing these ablations,"}, {"start": 1430.0200000000002, "end": 1434.0600000000002, "text": " but they roughly oscillate around 4.5,"}, {"start": 1434.0600000000002, "end": 1437.24, "text": " which means these are comparable in that sense."}, {"start": 1437.24, "end": 1438.8600000000001, "text": " Now I'm gonna focus in more detail"}, {"start": 1438.8600000000001, "end": 1441.0800000000002, "text": " on these first couple of techniques,"}, {"start": 1441.0800000000002, "end": 1443.8200000000002, "text": " and then I'm gonna slowly just scheme"}, {"start": 1443.8200000000002, "end": 1444.9, "text": " all of the other techniques,"}, {"start": 1444.9, "end": 1446.8200000000002, "text": " because there is a lot of details,"}, {"start": 1446.82, "end": 1448.96, "text": " and I'm gonna just kinda have to abstract"}, {"start": 1448.96, "end": 1450.72, "text": " some of those details away from you."}, {"start": 1450.72, "end": 1453.3, "text": " So let's start with this stage ratio here."}, {"start": 1453.3, "end": 1454.4399999999998, "text": " So what is stage ratio?"}, {"start": 1454.4399999999998, "end": 1455.6, "text": " What does that mean?"}, {"start": 1456.78, "end": 1458.78, "text": " They say here, Swin-T, on the other hand,"}, {"start": 1458.78, "end": 1459.96, "text": " follow the same principle,"}, {"start": 1459.96, "end": 1463.1, "text": " but with a slightly different stage compute ratio"}, {"start": 1463.1, "end": 1465.54, "text": " of one, one, three, one."}, {"start": 1465.54, "end": 1467.3799999999999, "text": " We're gonna see what this means in a second."}, {"start": 1467.3799999999999, "end": 1470.34, "text": " So for larger Swin-Transformers, the ratio is this one,"}, {"start": 1470.34, "end": 1473.28, "text": " and following the design, we adjust the number of blocks"}, {"start": 1473.28, "end": 1477.42, "text": " in each stage from three, four, six, three, in Resonant 50,"}, {"start": 1477.42, "end": 1479.06, "text": " to three, three, nine, three."}, {"start": 1479.06, "end": 1481.1399999999999, "text": " So you can see when you divide all of this by three,"}, {"start": 1481.1399999999999, "end": 1486.1399999999999, "text": " you get the same ratio as in the Swin-Transformer right here."}, {"start": 1486.1399999999999, "end": 1488.68, "text": " So the idea is basically they wanna have"}, {"start": 1488.68, "end": 1491.36, "text": " the amount of compute, the number of flops,"}, {"start": 1491.36, "end": 1496.04, "text": " across the model should have roughly the same ratio"}, {"start": 1496.04, "end": 1497.66, "text": " as the one in Swin-Transformer,"}, {"start": 1497.66, "end": 1499.2, "text": " because some of the other papers,"}, {"start": 1499.2, "end": 1500.5, "text": " they reference them here,"}, {"start": 1500.5, "end": 1503.34, "text": " show that that plays an important role"}, {"start": 1503.34, "end": 1506.56, "text": " in the design of novel architectures."}, {"start": 1506.56, "end": 1508.18, "text": " So let me show you some code here,"}, {"start": 1508.18, "end": 1510.88, "text": " and remember these numbers here, three, four, six, three."}, {"start": 1510.88, "end": 1513.66, "text": " So that's the number of blocks in each stage."}, {"start": 1513.66, "end": 1516.42, "text": " So basically, if you take a look at the,"}, {"start": 1516.42, "end": 1520.06, "text": " let me zoom in a little bit here, at the API here,"}, {"start": 1520.06, "end": 1521.56, "text": " when you're constructing Resonant 50,"}, {"start": 1521.56, "end": 1523.36, "text": " or Resonant 101, or whatnot,"}, {"start": 1523.36, "end": 1525.1, "text": " you always call this underlying,"}, {"start": 1525.1, "end": 1526.88, "text": " this is by torch code, by the way,"}, {"start": 1526.88, "end": 1530.02, "text": " you call this common API function,"}, {"start": 1530.02, "end": 1533.04, "text": " and then you pass different number of blocks per stage,"}, {"start": 1533.04, "end": 1535.46, "text": " and that's what determines how big the model is,"}, {"start": 1535.46, "end": 1538.6, "text": " how many layers it has, and how many parameters does it has."}, {"start": 1538.6, "end": 1542.54, "text": " So let me quickly show you this underscore Resonant function."}, {"start": 1542.54, "end": 1544.7, "text": " So it's here, and we're passing,"}, {"start": 1544.7, "end": 1548.96, "text": " so that sequence of numbers is named layers,"}, {"start": 1548.96, "end": 1552.96, "text": " and we pass layers to this Resonant class here."}, {"start": 1552.96, "end": 1555.98, "text": " So let me find the class, class Resonant."}, {"start": 1556.8799999999999, "end": 1557.8799999999999, "text": " Okay, here it is."}, {"start": 1557.8799999999999, "end": 1559.0, "text": " So it's passed as layers."}, {"start": 1559.0, "end": 1562.18, "text": " So now let me search for layers, layers,"}, {"start": 1562.18, "end": 1564.82, "text": " and basically you can see here that those numbers,"}, {"start": 1564.82, "end": 1567.28, "text": " so three, six, four, three, or something,"}, {"start": 1567.28, "end": 1572.28, "text": " are passed here to these make layer functions,"}, {"start": 1572.62, "end": 1577.34, "text": " and they effectively, what they do is they say,"}, {"start": 1577.34, "end": 1580.46, "text": " take this block and repeat it this many times."}, {"start": 1580.46, "end": 1582.76, "text": " So if we were to find the make layer function,"}, {"start": 1584.26, "end": 1586.1, "text": " I could probably just click and find,"}, {"start": 1586.1, "end": 1589.58, "text": " go by reference, but I'm being lazy here."}, {"start": 1589.58, "end": 1591.54, "text": " We can see here that here is the blocks,"}, {"start": 1591.54, "end": 1595.7199999999998, "text": " so let's find the blocks, whoops, the blocks variable."}, {"start": 1595.7199999999998, "end": 1598.82, "text": " Okay, blocks variable is called here,"}, {"start": 1598.82, "end": 1601.1, "text": " and you can see basically we have this layer append,"}, {"start": 1601.1, "end": 1602.7199999999998, "text": " and for the number of blocks,"}, {"start": 1602.7199999999998, "end": 1605.1399999999999, "text": " we're just going to be appending in this stage"}, {"start": 1606.02, "end": 1607.48, "text": " that amount of blocks."}, {"start": 1607.48, "end": 1611.1799999999998, "text": " So that's why we have, that's how they manage to regulate"}, {"start": 1611.1799999999998, "end": 1613.3799999999999, "text": " the amount of computation across each of the stage"}, {"start": 1613.38, "end": 1618.38, "text": " of this resonant and reflect mirror the same type of ratios"}, {"start": 1619.42, "end": 1622.3000000000002, "text": " that are present in the Swin transformer."}, {"start": 1622.3000000000002, "end": 1624.7, "text": " Cool, let me know whether you find this type of mixing"}, {"start": 1624.7, "end": 1629.16, "text": " of code and paper explanation useful or not."}, {"start": 1629.16, "end": 1630.0, "text": " Let's continue here."}, {"start": 1630.0, "end": 1633.74, "text": " So this design decision falls under this umbrella term"}, {"start": 1633.74, "end": 1635.5200000000002, "text": " of macro design."}, {"start": 1635.5200000000002, "end": 1637.0400000000002, "text": " You can understand why that is."}, {"start": 1637.0400000000002, "end": 1640.7600000000002, "text": " We are changing the macro structure of the architecture."}, {"start": 1640.7600000000002, "end": 1643.0800000000002, "text": " Then they have this changing stem to patchify,"}, {"start": 1643.08, "end": 1645.3, "text": " and they mentioned here that a common stem cell"}, {"start": 1645.3, "end": 1649.1, "text": " with will aggressively down sample the input images."}, {"start": 1649.1, "end": 1651.6, "text": " So that's something that's common in most of those older"}, {"start": 1651.6, "end": 1654.36, "text": " CNN architectures, and they say here that the stem cell"}, {"start": 1654.36, "end": 1657.26, "text": " in standard resonant contains a seven times seven"}, {"start": 1657.26, "end": 1659.8, "text": " convolution layer with try two,"}, {"start": 1659.8, "end": 1661.5, "text": " followed by a max pooling layer,"}, {"start": 1661.5, "end": 1665.54, "text": " which results in a forex down sampling of the input images,"}, {"start": 1665.54, "end": 1668.3799999999999, "text": " which isn't fairly aggressive down sampling."}, {"start": 1668.3799999999999, "end": 1671.46, "text": " They say here, we replaced the resonant style stem cell"}, {"start": 1671.46, "end": 1675.02, "text": " with a patchify layer implemented using a four times four"}, {"start": 1675.02, "end": 1677.18, "text": " strive for convolutional layer."}, {"start": 1677.18, "end": 1681.22, "text": " And that's how basically VAT like model works."}, {"start": 1681.22, "end": 1685.22, "text": " What you do is you can interpret this patchifying strategy."}, {"start": 1685.22, "end": 1686.8600000000001, "text": " Basically you have an image here."}, {"start": 1687.74, "end": 1691.5, "text": " And what they do is they group these pixels"}, {"start": 1691.5, "end": 1693.42, "text": " into these patches."}, {"start": 1693.42, "end": 1696.74, "text": " And then what they do is they do a linear projection"}, {"start": 1696.74, "end": 1700.26, "text": " of those pixels into some feature vector."}, {"start": 1700.26, "end": 1703.14, "text": " And you can model that by just thinking of this"}, {"start": 1703.14, "end": 1706.64, "text": " as if you had, let me change the color,"}, {"start": 1706.64, "end": 1708.5, "text": " as if you had like a kernel,"}, {"start": 1709.34, "end": 1713.62, "text": " that's let's say in this case, they said four by four."}, {"start": 1713.62, "end": 1715.14, "text": " So kernel that's four by four,"}, {"start": 1715.14, "end": 1719.46, "text": " and then you have a stride that has a size of four,"}, {"start": 1719.46, "end": 1721.42, "text": " which means the next step will be here,"}, {"start": 1721.42, "end": 1723.3, "text": " the next step will be here."}, {"start": 1723.3, "end": 1726.02, "text": " And basically, yeah, that's how you implement."}, {"start": 1726.02, "end": 1727.74, "text": " If you take a look at any code base,"}, {"start": 1727.74, "end": 1729.66, "text": " that's how you implement the patchify layer"}, {"start": 1729.66, "end": 1732.02, "text": " using a convolutional operation."}, {"start": 1732.02, "end": 1733.1000000000001, "text": " Cool."}, {"start": 1733.1000000000001, "end": 1734.3000000000002, "text": " So that's the second design decision."}, {"start": 1734.3000000000002, "end": 1737.1000000000001, "text": " Then they mentioned here Resnextify."}, {"start": 1738.26, "end": 1742.18, "text": " They take some ideas from the Resnext paper."}, {"start": 1742.18, "end": 1743.8600000000001, "text": " So they say here, more precisely,"}, {"start": 1743.8600000000001, "end": 1745.9, "text": " Resnext employs grouped convolution"}, {"start": 1745.9, "end": 1749.8200000000002, "text": " for the three times three conv layer in a bottleneck block."}, {"start": 1749.8200000000002, "end": 1751.7, "text": " As this significantly reduces the flops,"}, {"start": 1751.7, "end": 1753.66, "text": " the network width is expanded to compensate"}, {"start": 1753.66, "end": 1754.8200000000002, "text": " for the capacity loss."}, {"start": 1754.8200000000002, "end": 1757.16, "text": " In our case, we use depthwise convolution,"}, {"start": 1757.16, "end": 1759.1200000000001, "text": " a special case of grouped convolution"}, {"start": 1759.12, "end": 1762.02, "text": " where the number of groups equals the number of channels."}, {"start": 1762.02, "end": 1763.78, "text": " So a quick recap here,"}, {"start": 1763.78, "end": 1767.04, "text": " how the depthwise convolution works is the following."}, {"start": 1767.04, "end": 1768.76, "text": " So let's say you have an activation volume."}, {"start": 1768.76, "end": 1772.6999999999998, "text": " So that's the feature maps that comes out of your layer"}, {"start": 1772.6999999999998, "end": 1775.5, "text": " after you process and apply the activation functions."}, {"start": 1775.5, "end": 1776.6999999999998, "text": " You get the activation volume,"}, {"start": 1776.6999999999998, "end": 1778.5, "text": " which looks something like this."}, {"start": 1778.5, "end": 1782.4599999999998, "text": " Let me try and draw it here, something like this."}, {"start": 1782.46, "end": 1789.46, "text": " And so how your regular convolutions work is that,"}, {"start": 1790.26, "end": 1794.1000000000001, "text": " you make a kernel, you make like a filter here,"}, {"start": 1794.1000000000001, "end": 1797.14, "text": " which is maybe usually three times three"}, {"start": 1797.14, "end": 1800.26, "text": " in most of these older CNN architectures."}, {"start": 1800.26, "end": 1802.78, "text": " And now what I do here is the following."}, {"start": 1802.78, "end": 1805.3, "text": " Let me just kind of draw it here roughly."}, {"start": 1805.3, "end": 1810.3, "text": " So what I do is the depth of this filter"}, {"start": 1810.3, "end": 1813.3, "text": " is the same as the depth of your activation volume."}, {"start": 1813.3, "end": 1815.3, "text": " So that's your regular convolution."}, {"start": 1815.3, "end": 1818.68, "text": " Now what depthwise convolution does is the following,"}, {"start": 1818.68, "end": 1821.3999999999999, "text": " it slices this activation volume,"}, {"start": 1821.3999999999999, "end": 1823.82, "text": " like you have a single feature map here,"}, {"start": 1823.82, "end": 1827.34, "text": " then you have a second feature map, et cetera, et cetera."}, {"start": 1827.34, "end": 1831.3799999999999, "text": " And what you do, you just take a filter"}, {"start": 1831.3799999999999, "end": 1834.62, "text": " that has this depth dimension is just one,"}, {"start": 1834.62, "end": 1838.9199999999998, "text": " and then you apply and apply that filter to this slice,"}, {"start": 1838.92, "end": 1841.02, "text": " and then you apply the second filter to the second slice,"}, {"start": 1841.02, "end": 1841.8600000000001, "text": " et cetera, et cetera."}, {"start": 1841.8600000000001, "end": 1843.94, "text": " So again, let me draw that here."}, {"start": 1843.94, "end": 1846.72, "text": " So the filter for the depthwise will be something like this."}, {"start": 1846.72, "end": 1850.42, "text": " So again, let's say it's three times three again,"}, {"start": 1850.42, "end": 1852.94, "text": " but the depth component is just one."}, {"start": 1852.94, "end": 1855.72, "text": " It's basically a single dimension basically."}, {"start": 1855.72, "end": 1858.5800000000002, "text": " Now there is a problem in using depthwise convolution,"}, {"start": 1858.5800000000002, "end": 1860.5800000000002, "text": " and that's that you break the correlation"}, {"start": 1860.5800000000002, "end": 1863.0600000000002, "text": " between flops and the throughput,"}, {"start": 1863.0600000000002, "end": 1866.8600000000001, "text": " which means that basically for the same amount of compute,"}, {"start": 1866.86, "end": 1871.86, "text": " because of some memory issues, the throughput,"}, {"start": 1872.2199999999998, "end": 1874.5, "text": " i.e. the number of images you can process in a second"}, {"start": 1874.5, "end": 1877.78, "text": " or whatever unit of time, drops down dramatically."}, {"start": 1877.78, "end": 1879.74, "text": " We're gonna address that a little bit later."}, {"start": 1879.74, "end": 1882.9199999999998, "text": " And they did show that the throughput is competitive"}, {"start": 1882.9199999999998, "end": 1885.2199999999998, "text": " and even better than Swin Transformers, so that's cool."}, {"start": 1885.2199999999998, "end": 1886.62, "text": " Now there are a lot more details here."}, {"start": 1886.62, "end": 1887.9799999999998, "text": " I'm gonna skip some of those."}, {"start": 1887.9799999999998, "end": 1890.1799999999998, "text": " They had the inverted block which was introduced."}, {"start": 1890.1799999999998, "end": 1891.6999999999998, "text": " I'm gonna actually cover this one."}, {"start": 1891.6999999999998, "end": 1894.54, "text": " So one important design in every transformer block"}, {"start": 1894.54, "end": 1897.1399999999999, "text": " is that it creates an inverted bottleneck."}, {"start": 1897.1399999999999, "end": 1899.6, "text": " i.e. the hidden dimension of the MLP block"}, {"start": 1899.6, "end": 1903.7, "text": " is four times wider than the input dimension."}, {"start": 1903.7, "end": 1906.06, "text": " And interestingly, this transformer design"}, {"start": 1906.06, "end": 1909.28, "text": " is connected to the inverted bottleneck design"}, {"start": 1909.28, "end": 1912.54, "text": " with an expansion ratio of four used in ComNets."}, {"start": 1912.54, "end": 1915.42, "text": " The idea was popularized by MobileNet V2."}, {"start": 1915.42, "end": 1918.7, "text": " So here we see again this cross-pollination from CNNs."}, {"start": 1918.7, "end": 1921.5, "text": " So taking ideas from CNNs and implementing them"}, {"start": 1921.5, "end": 1923.1399999999999, "text": " even in the original transformer."}, {"start": 1923.14, "end": 1925.42, "text": " So the idea was kinda informed"}, {"start": 1925.42, "end": 1928.98, "text": " by certain MobileNet architectures which came."}, {"start": 1928.98, "end": 1932.7, "text": " I think they were developed prior to 2017 obviously."}, {"start": 1932.7, "end": 1936.5400000000002, "text": " That's when the transformer paper originally was published."}, {"start": 1936.5400000000002, "end": 1939.0600000000002, "text": " So what I said here, if you did not understand,"}, {"start": 1939.0600000000002, "end": 1940.5800000000002, "text": " and it's just a basic understanding"}, {"start": 1940.5800000000002, "end": 1944.72, "text": " of how transformers work, at least the usual architectures."}, {"start": 1944.72, "end": 1948.3000000000002, "text": " So you have your MSA module here."}, {"start": 1948.3000000000002, "end": 1950.4, "text": " So this is your MSA module."}, {"start": 1950.4, "end": 1952.0800000000002, "text": " You have your input tokens."}, {"start": 1952.08, "end": 1954.8, "text": " You have your output tokens."}, {"start": 1954.8, "end": 1956.56, "text": " And then you apply an MLP."}, {"start": 1956.56, "end": 1958.6799999999998, "text": " So each of these, this is a feature vector"}, {"start": 1958.6799999999998, "end": 1961.6799999999998, "text": " that has certain dimension like D."}, {"start": 1961.6799999999998, "end": 1966.56, "text": " And what you apply MLP basically sharing the weights."}, {"start": 1966.56, "end": 1970.4399999999998, "text": " So you just kinda do a for loop across these feature vectors"}, {"start": 1970.4399999999998, "end": 1975.4399999999998, "text": " and you apply the MLP which will map from D to 4D."}, {"start": 1976.32, "end": 1979.24, "text": " So this feature vector is gonna be 4X longer."}, {"start": 1979.24, "end": 1982.24, "text": " So one, two, three, four compared to this one."}, {"start": 1982.24, "end": 1987.24, "text": " And then they again map it down into D dimensions."}, {"start": 1987.48, "end": 1989.84, "text": " So that's what's referred to as a bottleneck layer."}, {"start": 1989.84, "end": 1992.84, "text": " You can kinda imagine that you start from D"}, {"start": 1992.84, "end": 1997.44, "text": " and then you basically upscale it in a way to 4D"}, {"start": 1997.44, "end": 1999.58, "text": " and then you downscale it to D."}, {"start": 1999.58, "end": 2000.44, "text": " So something like this."}, {"start": 2000.44, "end": 2003.2, "text": " That's why it's called a bottleneck layer."}, {"start": 2003.2, "end": 2005.88, "text": " Sorry, I meant to say inverted bottleneck the whole time."}, {"start": 2005.88, "end": 2008.08, "text": " Not bottleneck layer because that one"}, {"start": 2008.08, "end": 2010.4399999999998, "text": " would look something like this."}, {"start": 2010.4399999999998, "end": 2012.78, "text": " This would be your bottleneck layer."}, {"start": 2012.78, "end": 2016.1599999999999, "text": " Okay, so this is a fun fact."}, {"start": 2016.1599999999999, "end": 2017.8, "text": " I wasn't aware of that."}, {"start": 2017.8, "end": 2019.24, "text": " Now here they just showed the progression"}, {"start": 2019.24, "end": 2021.96, "text": " of how they were additionally manipulating the modules."}, {"start": 2021.96, "end": 2023.56, "text": " And I think there is a small error here."}, {"start": 2023.56, "end": 2028.56, "text": " This should be I think 384 mapped down to 96"}, {"start": 2030.52, "end": 2031.6399999999999, "text": " and not the other way around."}, {"start": 2031.6399999999999, "end": 2034.24, "text": " But yeah, anyways, that's a small detail."}, {"start": 2034.24, "end": 2036.58, "text": " Then they try and play with kernel sizes"}, {"start": 2036.58, "end": 2039.04, "text": " because as I said, usually you use three by three."}, {"start": 2039.04, "end": 2041.56, "text": " They figure out that seven by seven works the best"}, {"start": 2041.56, "end": 2043.0, "text": " and that's it."}, {"start": 2043.0, "end": 2044.8799999999999, "text": " Here they compare the Swin transformer block"}, {"start": 2044.8799999999999, "end": 2047.58, "text": " with the Resonant block with the final Comnex block."}, {"start": 2047.58, "end": 2049.68, "text": " A couple more things they experimented with"}, {"start": 2049.68, "end": 2051.48, "text": " are the following."}, {"start": 2051.48, "end": 2055.56, "text": " You can see that they are using in the Comnex block,"}, {"start": 2055.56, "end": 2056.96, "text": " they're using Jell-U."}, {"start": 2056.96, "end": 2060.16, "text": " So that's Gaussian error linear unit or something"}, {"start": 2060.16, "end": 2061.64, "text": " instead of Relu."}, {"start": 2061.64, "end": 2064.7599999999998, "text": " They have layer norm instead of batch norm."}, {"start": 2064.76, "end": 2067.76, "text": " They have less normalization layers in general"}, {"start": 2067.76, "end": 2069.38, "text": " and less activation functions."}, {"start": 2069.38, "end": 2072.28, "text": " You can see they apply the activation function only here."}, {"start": 2072.28, "end": 2073.76, "text": " Let me change the color here."}, {"start": 2073.76, "end": 2075.5600000000004, "text": " So they apply it only here,"}, {"start": 2075.5600000000004, "end": 2077.8, "text": " whereas here they apply it in multiple locations"}, {"start": 2077.8, "end": 2079.6800000000003, "text": " and all of that and they have, as you can see here,"}, {"start": 2079.6800000000003, "end": 2082.28, "text": " a bunch of batch norm layers, batch norm, batch norm."}, {"start": 2082.28, "end": 2084.48, "text": " Here they only have a single layer norm."}, {"start": 2084.48, "end": 2086.1200000000003, "text": " Okay, that's pretty much the level of detail"}, {"start": 2086.1200000000003, "end": 2088.48, "text": " I can tolerate for one video."}, {"start": 2088.48, "end": 2090.92, "text": " So let me just slowly wrap this up."}, {"start": 2090.92, "end": 2092.96, "text": " They say that our Comnex model"}, {"start": 2092.96, "end": 2095.96, "text": " has approximately the same number of flops,"}, {"start": 2095.96, "end": 2098.5, "text": " parameters, same throughput and memory use"}, {"start": 2098.5, "end": 2099.7200000000003, "text": " as the Swin transformer,"}, {"start": 2099.7200000000003, "end": 2101.7200000000003, "text": " but does not require specialized modules"}, {"start": 2101.7200000000003, "end": 2104.12, "text": " such as shift the window, the tension,"}, {"start": 2104.12, "end": 2105.68, "text": " or relative position biases,"}, {"start": 2105.68, "end": 2108.4, "text": " which is very cool because they get to have"}, {"start": 2108.4, "end": 2110.26, "text": " a simpler architecture."}, {"start": 2110.26, "end": 2111.76, "text": " Finally, here are the results."}, {"start": 2112.76, "end": 2116.36, "text": " They basically show that on image classification"}, {"start": 2116.36, "end": 2118.36, "text": " for all of various sizes,"}, {"start": 2118.36, "end": 2121.36, "text": " so both the small, the big one variant,"}, {"start": 2121.36, "end": 2123.48, "text": " for different resolutions,"}, {"start": 2123.48, "end": 2125.2400000000002, "text": " they outperform, as you can see here,"}, {"start": 2125.2400000000002, "end": 2126.2000000000003, "text": " they outperform Swin."}, {"start": 2126.2000000000003, "end": 2127.92, "text": " So let's focus on a single pair here."}, {"start": 2127.92, "end": 2131.48, "text": " Let's take, for example, Swin B and Comnex B"}, {"start": 2131.48, "end": 2134.48, "text": " working on 384 resolution."}, {"start": 2134.48, "end": 2136.2000000000003, "text": " They have similar number of parameters."}, {"start": 2136.2000000000003, "end": 2138.6400000000003, "text": " They have similar flops."}, {"start": 2138.6400000000003, "end": 2140.6, "text": " Swin even has a bit more flops here"}, {"start": 2140.6, "end": 2143.2400000000002, "text": " and you can see that throughput is lower for Swin"}, {"start": 2143.2400000000002, "end": 2146.1200000000003, "text": " and you can see that the accuracy is better for Comnex."}, {"start": 2146.1200000000003, "end": 2148.44, "text": " So basically across all of these dimensions,"}, {"start": 2148.44, "end": 2151.56, "text": " it outperforms the Swin model, which is very cool."}, {"start": 2152.4, "end": 2156.52, "text": " They also showed that basically increasing the model,"}, {"start": 2156.52, "end": 2158.36, "text": " as we already mentioned,"}, {"start": 2158.36, "end": 2161.12, "text": " leads to increase in performance,"}, {"start": 2161.12, "end": 2163.5, "text": " which is something, a desirable thing."}, {"start": 2163.5, "end": 2165.78, "text": " You wanna be able to scale the architecture,"}, {"start": 2165.78, "end": 2168.16, "text": " the size, and to get additional performance."}, {"start": 2169.98, "end": 2172.48, "text": " Here, they additionally show the models"}, {"start": 2172.48, "end": 2175.86, "text": " pre-trained on the ImageNet 22K."}, {"start": 2175.86, "end": 2179.2400000000002, "text": " And they showed that even this big data regime,"}, {"start": 2179.2400000000002, "end": 2184.2400000000002, "text": " where vision transformers are supposed to be superior,"}, {"start": 2185.08, "end": 2188.2000000000003, "text": " they are not only comparable, but they outperform."}, {"start": 2188.2000000000003, "end": 2190.26, "text": " We again, focus on one of these pairs."}, {"start": 2190.26, "end": 2191.58, "text": " Let me take this one."}, {"start": 2191.58, "end": 2194.1200000000003, "text": " They have similar number of parameters again,"}, {"start": 2194.1200000000003, "end": 2195.54, "text": " similar flops."}, {"start": 2195.54, "end": 2200.38, "text": " Again, actually, Swin takes more computation"}, {"start": 2200.38, "end": 2204.1200000000003, "text": " and it has lower throughput and it has lower accuracy."}, {"start": 2204.12, "end": 2205.96, "text": " So again, it outperforms a Swin,"}, {"start": 2207.12, "end": 2210.12, "text": " even in the big data regime, which is fairly important."}, {"start": 2210.12, "end": 2213.3199999999997, "text": " And wrapping this up, they also showed that"}, {"start": 2213.3199999999997, "end": 2218.16, "text": " on object detection tasks and semantic segmentation tasks,"}, {"start": 2218.16, "end": 2220.8399999999997, "text": " Swin transformer is again,"}, {"start": 2220.8399999999997, "end": 2222.3199999999997, "text": " you'll see fairly similar results"}, {"start": 2222.3199999999997, "end": 2224.3199999999997, "text": " as here on the image classification tests."}, {"start": 2224.3199999999997, "end": 2227.0, "text": " So I don't wanna even dig into too much details."}, {"start": 2227.0, "end": 2228.56, "text": " The last thing I wanna mention here is,"}, {"start": 2228.56, "end": 2229.88, "text": " so under similar flops,"}, {"start": 2229.88, "end": 2232.7599999999998, "text": " models with depth-wise convolutions are known to be slower"}, {"start": 2232.76, "end": 2234.44, "text": " and consume more memory than ComNets"}, {"start": 2234.44, "end": 2236.5200000000004, "text": " with only dense convolutions."}, {"start": 2236.5200000000004, "end": 2239.1200000000003, "text": " It is natural to ask whether the design of ComNets"}, {"start": 2239.1200000000003, "end": 2241.4, "text": " will render it practically inefficient."}, {"start": 2241.4, "end": 2242.92, "text": " As demonstrated throughout the paper,"}, {"start": 2242.92, "end": 2245.32, "text": " the inference throughputs of ComNets are comparable"}, {"start": 2245.32, "end": 2248.2400000000002, "text": " to or exceed that of Swin transformers."}, {"start": 2248.2400000000002, "end": 2249.98, "text": " And that's something I was worried about,"}, {"start": 2249.98, "end": 2253.1600000000003, "text": " but it turns out that the throughput is actually kept"}, {"start": 2253.1600000000003, "end": 2255.2400000000002, "text": " on a healthy level."}, {"start": 2256.1200000000003, "end": 2257.2400000000002, "text": " That's it."}, {"start": 2257.2400000000002, "end": 2260.92, "text": " I wanna wrap this video by just mentioning one thing."}, {"start": 2260.92, "end": 2263.98, "text": " And it's maybe a semi-rend, if that's even a word."}, {"start": 2265.88, "end": 2268.06, "text": " So I just kinda skimmed through the references"}, {"start": 2268.06, "end": 2271.92, "text": " and I didn't see Schmidt-Huber mentioned anywhere."}, {"start": 2271.92, "end": 2273.28, "text": " And it's kinda sad."}, {"start": 2273.28, "end": 2274.76, "text": " I recently watched this video"}, {"start": 2274.76, "end": 2277.44, "text": " that Schmidt-Huber published to his YouTube channel"}, {"start": 2277.44, "end": 2281.44, "text": " where he explained the evolution of these earlier models."}, {"start": 2281.44, "end": 2282.6800000000003, "text": " And it actually turned out,"}, {"start": 2282.6800000000003, "end": 2284.36, "text": " and I mentioned this in the beginning of this video,"}, {"start": 2284.36, "end": 2286.86, "text": " that prior to AlexNet,"}, {"start": 2286.86, "end": 2289.36, "text": " there was this model called DanNet."}, {"start": 2289.36, "end": 2293.08, "text": " So it's spelled like this, I think, DanNet."}, {"start": 2293.08, "end": 2296.76, "text": " And that model was the first convolutional neural network"}, {"start": 2296.76, "end": 2298.88, "text": " that showed superhuman performance"}, {"start": 2298.88, "end": 2302.2200000000003, "text": " on I think four or five vision benchmarks,"}, {"start": 2302.2200000000003, "end": 2304.96, "text": " although those benchmarks were arguably way smaller"}, {"start": 2304.96, "end": 2306.36, "text": " compared to ImageNet."}, {"start": 2306.36, "end": 2309.52, "text": " It's kinda sad that DanNet got censored out"}, {"start": 2309.52, "end": 2311.2400000000002, "text": " of the AI history,"}, {"start": 2311.2400000000002, "end": 2314.6800000000003, "text": " even though it was such an important model back in 2011."}, {"start": 2314.6800000000003, "end": 2317.1200000000003, "text": " So yeah, I encourage you to check out Schmidt-Huber's video"}, {"start": 2317.12, "end": 2319.68, "text": " and let me know your thoughts on this topic as well."}, {"start": 2319.68, "end": 2321.6, "text": " Cool, let's do a quick summary here"}, {"start": 2321.6, "end": 2323.52, "text": " what we've seen in this video."}, {"start": 2323.52, "end": 2326.04, "text": " So basically, we've seen this convergence going on"}, {"start": 2326.04, "end": 2329.0, "text": " between vision transformers on one hand side"}, {"start": 2329.0, "end": 2331.96, "text": " with Swin transformer being a representative"}, {"start": 2331.96, "end": 2333.56, "text": " that was used in this paper."}, {"start": 2333.56, "end": 2334.7599999999998, "text": " And on the other hand side,"}, {"start": 2334.7599999999998, "end": 2338.7599999999998, "text": " we've seen ComNets being rejuvenated"}, {"start": 2338.7599999999998, "end": 2340.6, "text": " with novel training strategies"}, {"start": 2340.6, "end": 2345.12, "text": " such as the ones from the ResNet Strikes Again paper"}, {"start": 2345.12, "end": 2346.88, "text": " or from ComNEXT paper."}, {"start": 2346.88, "end": 2350.0, "text": " We've also seen the results from the EfficientNet V2 paper,"}, {"start": 2350.0, "end": 2352.88, "text": " which even seem to outperform ComNEXT models."}, {"start": 2352.88, "end": 2354.48, "text": " So it would be a fun comparison"}, {"start": 2354.48, "end": 2357.12, "text": " to compare them across multiple benchmarks"}, {"start": 2357.12, "end": 2357.96, "text": " and stuff like that."}, {"start": 2357.96, "end": 2361.04, "text": " In a nutshell, it seems that convolutional priors"}, {"start": 2361.04, "end": 2365.7200000000003, "text": " are very good and hard to beat priors for natural images."}, {"start": 2365.7200000000003, "end": 2369.0, "text": " And I'm very excited to see"}, {"start": 2369.0, "end": 2372.0, "text": " where these types of hybrid models will take us"}, {"start": 2372.0, "end": 2373.92, "text": " and whether we're gonna have some novel ideas"}, {"start": 2373.92, "end": 2376.52, "text": " that are going to boost the performance"}, {"start": 2376.52, "end": 2379.92, "text": " of these visual perception models even further."}, {"start": 2379.92, "end": 2382.0, "text": " Anyways, if you found this video useful"}, {"start": 2382.0, "end": 2384.6, "text": " with all of these semi-rants,"}, {"start": 2384.6, "end": 2386.84, "text": " consider subscribing, sharing the video out"}, {"start": 2386.84, "end": 2389.6, "text": " and also do join the Discord community."}, {"start": 2389.6, "end": 2392.4, "text": " I'll just link it down in the video description as always."}, {"start": 2392.4, "end": 2394.64, "text": " So until next time, bye-bye."}, {"start": 2394.64, "end": 2401.12, "text": " Thank you so much for watching,"}, {"start": 2401.12, "end": 2403.18, "text": " you"}]