EfficientNetV2 - Smaller Models and Faster Training | Paper explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

The newest version of EfficientNet v2 achieved better results on ImageNet top-1 accuracy than recently published NFNets, Vision Transformers, etc.

You'll learn about:
✔️ What is a progressive training
✔️ Fused-MBConv layer and novel reward function for NAS

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ paper: https://arxiv.org/pdf/2104.00298.pdf
✅ code (yet to be published): https://github.com/google/automl/efficientnetv2
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 High-level overview
01:40 NAS review
07:20 Deep dive
15:35 Novel reward
17:34 Progressive training
21:10 Stochastic depth regularization
23:00 Results

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Petar Veličković
Zvonimir Sabljic
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#efficientnetv2 #neuralarchitecturesearch #rl
 What's up in this video I'm covering the newest version of efficient net paper just came out a couple of days ago The title is efficient net we to smaller models and faster training by Minxion Tan and quality Hopefully I pronounced their names correctly from Google brain and you can see the results. They got are pretty decent Taking a look at the image net accuracy like top one accuracy on the y-axis and training time on the x-axis We can see that the curves just form envelope around all of the previous models like NF nets which used the which ditch the better realization altogether and you can see the V1 version of efficient net here and some models such as vision transformer and yeah, basically they form an envelope So pretty decent results the two main ideas are basically they modify the search space by adding this new Like block called fused and become which is just a handcrafted module which was created by modifying the ambi com from the mobile net paper and We'll see those a bit later. It's not that important so that's the first thing the second thing is they updated the the objective function by adding the training step time and by adding the Number of parameters into the objective function. Finally, the the last thing is progressive training training whereby they while they're training there they are slowly and gradually increasing the Image resolution, but they're also increasing the regularization Like dropout and some random augmentations. We'll see those details a bit later. So those are the main ideas So first like the the search State the search space and the search Strategy were updated a bit and finally the progressive training. That's the second thing Before I even start digging into the paper. I want to walk you through the body of literature that came before this paper and The interesting thing is that this author the quality basically was the co-author of many of the first and Seminal neural architecture search papers including the first one the the NAS with RL paper Then we had the NAS net paper then we had the M. Ness paper and finally efficient net V1 I just want to walk you through and because I think You understanding the this previous work will help you help ground your understanding and better better better grasp the v2 version Okay. So first things first. What's neural architecture search? the basic idea is this you have you have a controller which is usually some RNN like LSTM and Basically you sample architectures from your search space you evaluate the architectures by training them and You basically evaluate them on the validation set you get the accuracy and you get you you basically give the controller the reward Which is the evaluation accuracy and this is your common RL Loop you have the agent you have the environment you get rewards and you can train the agent the controller here to Eventually sample those architectures which have high reward ie which have high validation accuracy and you can see here How the sampling happens basically the controller? Samples first maybe the number of filters then what's the height of the filter was the width? What's the stride blah blah blah in in the every single layer and that's something called global search space because you're basically For all of the end layers you have the maximum flexibility you can pick every single layer Maybe different but the the like the search space is huge and this didn't work that good I mean it did work very well, but they needed a bunch of compute like some huge number of GPU hours and Then the second paper the NAS net paper this learning transferable architectures for scalable image recognition What it did basically was scope down the search space by? By noticing that most of the handcrafted architectures in the literature like mobile Nets etc and resnets and VGG and whatnot they basically tend to Create a cell and then just replicate it and so what they decided to do is just design the cell Basically you search for the cell and then you repeat the cell n times in order to get your final Architecture and that's it that was the main idea in that paper M s net which is really closely related to efficient nets did the following so basically They introduced They they again modify the search space a little bit so now they have like a couple blocks like this Let's say maybe seven I think they had some number like that and what I do is you you now? Sample the first layer in each of these blocks, and then you just repeat them By some number of times which is also a hyper parameter Which we search for so l1 here l2 here, so we'll have l2 of these will have l3 ln of these and This is a search space and additionally they included the latency Into the final reward objective function so not only the accuracy on the validation set but also the latency which made this M nest paper suitable for finding architectures that could run on edge devices such as mobile devices etc The loop is pretty much the same the reward function is just modified. This is still Lstm the only thing they Actually a change is they are not using the reinforce algorithm. They're using PPO from open AI as the RL algorithm of choice those are the main ideas finally we get to efficient net and Efficient net basically uses the same space as M. Ness net the only difference is instead of latency They are actually optimizing for flops So they they are basically that means the floating number of floating operations per second And that's that's it they the trick they did was to find the baseline architectures called B zero by using this M. Ness space I just explained and then they found these Basically then they scale the the B zero architecture by scaling the depth scaling the width I the number of channels and the image resolution pretty much Proportionally that means they just do some great search define this alpha beta and gamma which control depth width and even resolution and then they just raise these parameters to Phi and And Phi is the thing that controls how you get novel architecture So basically by setting Phi to 2 they'll they'll sample B one and then they'll have B one through B seven depending on the Phi The nice thing about Phi is that when you when you put when it when Phi is one compared to when Phi is zero The the B one model the model with Phi equals one will have twice as much flops Compared to B zero so they can kind of intuitively increase at the Phi and they'll proportion they'll have a feeling for how much compute how much additional compute they will need and Basically the accuracy will improve as well. So those are the main ideas that you need to understand before grasping the v2 paper Hopefully this was useful if you if you like this part just comment down below So that I know whether it was useful or not Anyways, let's jump into the paper itself okay, so First things first to develop this family of models. We use a combination of training aware Mass and scaling to jointly optimize training speed and parameter efficiency. So that's the thing I mentioned basically instead of just optimizing latency or flops They're optimizing these two so the training speed and the parameter efficiency explicitly into the objective function So they they make it explicit part of objective function The models were searched from search space enriched with new ops such as fused and become so that's the layer I mentioned that we'll see that in a bit And yeah, those are some of the main ideas and this is the second idea We propose an improved method of progressive learning which adaptive adjust regularization So the dropout and data augmentation along with the image size and we'll see how that works in detail in a bit Okay, so they mentioned here that training efficiency has gained significant interest recently so they mentioned the NF nets and some other papers and What I say then is that these methods often come with expensive overhead on parameter size And if you take a look at this table, you'll see that Basically, here's the efficient that we won resonant some vision transformer You can see that the number of parameters that the V2 has is much lower Than than than these two and even the V1 has much smaller amount of parameters compared to vision transformer in this resonant derivative so usually people omit some of the dimensions and then the results are inconclusive And they actually stressed that a couple of times throughout this paper, but we'll get to that in a moment Okay, so our study shows inefficientness. So it's basically they they Dissect efficient nap V1 and try to find the bottlenecks so they can improve the version 2 which makes sense, right? So training with very large image sizes is slow. So the thing with image net V1 was that basically the the B0 model used The image resolution 2 to 4 times 2 to 4 the B7 used I think somewhere around 560 And once you have this image size, basically you you can you can you can put so much On on the GPU. So if your image size increases the batch size will have to decrease And once the batch size decreases that means your training is going to slow down So that's the first thing they're gonna tackle with and that's gonna be done with this like Progressive increase of image resolution which you can treat as a sort of a curriculum learning basically the second thing is depthwise convolutions are slow in early layers and Thirdly equally scaling up every stage is suboptimal So that's the thing I mentioned here with those alpha betas and gammas what I do is Let's take the MNAS architecture here Basically if alpha so that controls the depth is 2 that means every single layer So in every single stage is going to increase by 2 instead of maybe increasing here by I don't know 2.3 and here maybe earlier you won't need that much you'll put 1.7 you get the point basically, they want to create a non uniform scaling instead of this and Based on these observations we design a search space enriched with additional ops such as fused and become and apply training awareness And scaling to jointly optimize model accuracy training speed and parameter size So I want to stress this training speed and parameter size is something they explicitly encode into the objective function Okay, hopefully that's clear. I'm gonna skip these parts. Whoops and Let's start here While many recent books have claimed large gains in training or inference speed They often much worse. They're often much worse than efficient net in terms of parameters and flops efficiency That's something I mentioned a couple minutes ago. Basically you can see here that even though the NF net and resonant RS have Really nice top one accuracy if you take a look at these two dimensions the parameters and flops They are even worse than then the V1 version of efficient net. Okay, I mentioned this already and This thing let's let's let's dig a bit deeper into this so Depthwise convolutions have fewer parameters and flops than regular convolutions, but they cannot fully utilize modern accelerators so Here's the thing the Mb conv layer so this is the Basically a module that somebody handcrafted a couple of years ago in the mobile net paper and you can see the structure This is also called inverted residual bottleneck layer because you can see that the Number of channels actually increases here and then you have a residual connection the skip connection connecting the like the usual skip connection, okay, and The thing is these depthwise convolutions are not So modern accelerators the hover accelerators don't support this operation as much so they just kind of replaced it with calm three by three and Everything else stays the same. So these two are kind of fused together. That's hence fused and become and they said here that Basically can fuse them becomes can improve training speed with a small overhead and parameters and flops But if we replace all blocks with fused and become then it significantly increases the parameters and flops while also slowing down the training so What I want to say is that you can simply just replace the M becomes with this newly this new version the fuse version And they have some relations here. Basically when you have no fuse layers You can see the parameters the the flops and the accuracy but once you once so once you add the fused Layer the fuse modules in the in the early layers of your architecture You get small increase in parameters and flops, but you get some decent decrease in accuracy and increase in throughput Like the number of images per second per per like TPU or or or V hundred basically faster training speed but once you start adding a lot of these like In the first five layers if you if you replace the the M become by fuse one by the fuse variant in the first five layers The parameters the number of parameters and the flops just blow up Blows off and you don't get a much accuracy improvement and neither do you get training speed So there is a trade-off obviously here So they mentioned that they just want to include this thing in the as the part of the search as well Okay Let's continue. So I mentioned this already scaling up every stage is Suboptimal so you want to have non-uniform scaling and they mentioned here the non-uniform scaling strategy to gradually add More layers to later stages That's some kind of heuristic and they mentioned here says a heuristic We also gradually add more layers to later stages So as you can see this neural architecture search business is basically a little bit Handcrafting of the initial search space and some heuristics and some search so it's it's a there's a lot of engineering details But you'll hopefully get the the main points and the gist of this paper Soon, okay. So here is that additional detail. So since the search space is smaller and it's smaller because Efficient net b1 didn't use some of the options it had so this just kind of cut those options from the search space And now the search space is smaller. So they say here since it's smaller We can simply apply a random search on much larger networks. They have comparable size as efficient net b4 Which was so we had b0 to b7. So that's something in between In the middle and the the main point I want you to understand here is that The the the model size you're searching for will be the most optimal one But once you start scaling to either smaller models or bigger models If the scaling strategy is good, you'll get decent results But they still won't be as optimal is as if you directly search for them. So They just found found the trade-off that searching for the Size that's equivalent to this b4 model is the best trade-off point and Having said that this is the the the the loss function I was mentioning so the search reward combines the model accuracy The normal training step time and the parameter size using a simple way to product accuracy times the the the training step time and Parameter size you can see they're raised to the power of w which are negative. So basically you'll want to Pull this down to to zero in order to to increase the the reward and just to contrast The reward for the v2 version of efficient net with with m nes paper and enet v1 Let me show you this formula. So they previously had accuracy and the latency if you remember and w was again negative so that means you want to uh, push this down to zero you want to increase this and basically That will lead to higher rewards and that's what the controller learns to do So he finds highly accurate models with low latency constraints The enet v1 had as I mentioned flops instead of latency everything else remains the same And here we can see aside from accuracy we have These two so the the the training speed and the number of parameters the nice thing about this thing here about s Is that it correlates really nicely and they show that with the latency. So the the higher the the training speed Basically the lower the latency. So so so the the faster you're training the the lower the latency will be and that's cool. Let's continue Here is finally what they find And This is the architecture they they they they search for the optimal one They have some fused mb com layers in the initial stages as you can see here And then they have the mb coms the usual stuff and you can also see that the number of channel increases here Which is the heuristic I mentioned so compare that to the uh b0 efficient net. So that's the v1 version The previous version of efficient net they only increase the number of channels And the v1 version the previous version of efficient net they only used and becomes okay Let me let me explain you the the progressive learning part, which is arguably one of the the most important algorithmic improvements of this paper So first the motivation for this thing, uh, they notice that when the image size is small like 128 by 128 Uh, the the the smaller the regularization the smaller the augmentations they are using the higher the accuracy And as you're increasing the image resolution we can see here that basically the higher The regularization the better the accuracy and you can see the trend here So basically the higher the the resolution you want to increase the regularization in order to get the optimal the the highest performance Okay, so that's the motivation and what's the main idea? So here you can see so the the progressively as the training progresses So this is the number of epochs on the on the axis and you can see in the initial axis The images are smaller here. They are bigger and also the augmentations are increasing so you can see the shears here Uh some like geometric transformations of the image you can see them the mix up Augmentation here kicking in basically they are mixing up two images together And so that's a high level idea and here How the algorithm works like is basically they they split the training into m steps Uh where m was four I think so basically you have it something like this So this is the like the the the the training and this is the epoch zero This is the epoch n whatever and we just split this into four parts Like this, right? Okay And in this portion they use the initial Uh image size and the initial magnitude of augmentations and then they're linearly increasing those In every single stage. So here they're just going to increase it and here in this stage They're going to increase it further and finally, uh the highest The the magnitude of augmentation will be the highest as well as the image size and if we zoom in here basically So for all of these m stages They're just linearly interpolating between the target image size between the start image size just simple linear interpolation here and same goes for the regularization just uh linear interpolation between the Initial magnitude and the final magnitude and let's see what the regularizations they are using The one is dropout like classical stuff. Uh, the second one is this random random augmentation where they're basically just combining a bunch of stuff like shearing like uh, uh various uh geometric and photometric augmentations So the mix-up is just uh, you're you're basically you saw it here Uh, you're mixing up two images in the image space and you're also mixing their their labels So the the target label will be some uh, like mixture between these two So for example if we know that we so we had maybe uh, like a dog here and we had Uh cat here and basically if we want to mix up, uh with lambda equals, I don't know 0.2 Uh, you'll you'll end up with the the final um Target will be this like you'll have 02 for for I don't know dog so this is 02 and this will be For this thing for the cat you'll have 0.8 And so you'll want to train your model to predict this Distribution when you input this image here except we don't have dog and cat here But some panda bear and something some fish or whatever. Okay Uh, you get a point. Let's continue and let's see the results. Um Before that, I just want to mention one more, uh regularization they're using that's this stochastic depth So the main idea is the following so you have the the resnet block Basically, you transform the features you're probably familiar with resnets You have this residual connect the skip connection which just adds these features without transforming them You add them up and that's your residual block here. So what stochastic depth does? It has this survival probability, which is basically the probability that this block will remain here So that means with 0.2 probability you'll actually short Um connect this thing so you'll just remove this thing and basically, uh, you you'll effectively end up Uh with the shallower model. So by having stochastic depth um the It's just they just show that that improves the accuracy and it's just a variation on the dropout. Uh, like uh concept basically um, you want to make sure that the That the network becomes robust to the co-adaptation So similarly as you had dropout So the original dropout was you have maybe like three neurons here two neurons here You have a fully connected layer and you'll with some probability you'll you'll pull this Weight to zero and basically the other weights have to learn how to not rely too much on on that particular edge Or on any particular edge in in for that matter and here This is just a generalization in a sense where we are just removing the whole module And we want to make sure that the network is robust that when certain modules just disappear It will still be able to predict the the the image Like the the class in the image In this case, okay, that's the that's the main idea Let's do results. So the first result is so in particular our efficient napv2m achieves comparable accuracy to efficient napv7 while training 11x faster using the same computing resources and this was one of the objectives in the training so that makes sense, but those results are really impressive. So let's take a look here and you can see For both number of parameters Flops i.e. compute and latency you can see that the efficient napv2 forms an envelope Above all of the previous models like nfnets and vision transformers and whatnot Uh same goes for every single dimension. The interesting thing is how much better it is than the vision transformer and I mentioned it here So recently vision transformers have demonstrated impressive results on image net accuracy and training speed However here we show that properly designed com nets With improved training method can still largely outperform vision transformers in both accuracy and training efficiency and if we take a look at the numbers, uh, we can see like the The v2l which is not pre-trained on the this larger image nap which has around 21 000 classes but the version from 2012 which had 1000 classes basically the numbers are much better as you can see here compared to Uh, even the even the biggest of vision transformers which were pre-trained on the image net 21k And you can see the numbers here You can see that the accuracy is is higher. We can see that we have much lower number of parameters Uh, we have uh much less compute here And what's this? Uh, yeah inference is also smaller and the training time is also smaller So across every single dimension is just better Which means that the biases we have in convolutional neural networks are not going to go any anytime soon. So it seems Uh and vision transformer took a lot of time to train and a lot of data. I think it was uh, initially trained on the this Private data set that google has which is huge. So yeah, uh, these are some nice results, uh from from the google brain team Let's continue and see here So they also showed that the performance on when transferring to other classical Uh data sets such as cipher 10 cipher 100 flowers cars we can see that They achieved the the highest accuracy as well here So two conclusions they they derive here is scaling up data size is more effective than simply scaling up model size in high accuracy regime so basically, uh, that means that this pre-training they uh this uh additional pre-training that did on the image net 21k helped boost the the the accuracy as you can see in this column and just having a bigger model will lead to overfitting so they just Mentioned that data is What what matters and some other work supports the claim as well? So although the image 21k has 10x more data our training approach enables us to finish the pre-training of efficient NAB 2 within two days using 32 tpu cores And this is pretty impressive like in only two days you can train on this huge data set, which is a nice nice Uh nice thing to to to have obviously so because they reduce the training time. They can now train it in two days and achieve Uh great results. So they did a lot of ablations and show that uh each of the new Uh novel things they introduced like progressive training uh leads to to improvements in results um and What's interesting is maybe that here they just tried progressive learning for resonance and they showed that using progressive training you can improve the uh accuracy on the resonance and not on the efficient nets but on resonance and uh also decrease the uh training time same goes for the v1 so basically, uh this training, uh, This progressive training is not it's not it has not over fitted to the particular architecture From efficient net v2. So yeah, that's not nice ablation there. Um, that's pretty much it so basically Search is the name of the game Um this progressive training Improves the training speed it improves the accuracy A couple of heuristics here and there and that's it. That's the version two of efficient net Hopefully you found this video useful if you did leave the comment uh share that would be super helpful and Um hit that bell icon to get notified when I upload a new video until next time. See you
[{"start": 0.0, "end": 5.84, "text": " What's up in this video I'm covering the newest version of efficient net paper just came out a couple of days ago"}, {"start": 5.84, "end": 12.14, "text": " The title is efficient net we to smaller models and faster training by Minxion Tan and quality"}, {"start": 12.4, "end": 19.28, "text": " Hopefully I pronounced their names correctly from Google brain and you can see the results. They got are pretty decent"}, {"start": 20.32, "end": 25.64, "text": " Taking a look at the image net accuracy like top one accuracy on the y-axis and training time on the x-axis"}, {"start": 25.64, "end": 31.3, "text": " We can see that the curves just form envelope around all of the previous models like NF nets"}, {"start": 31.3, "end": 35.58, "text": " which used the which ditch the better realization altogether and you can see the"}, {"start": 36.06, "end": 42.82, "text": " V1 version of efficient net here and some models such as vision transformer and yeah, basically they form an envelope"}, {"start": 42.82, "end": 49.58, "text": " So pretty decent results the two main ideas are basically they modify the search space by adding this new"}, {"start": 49.58, "end": 58.62, "text": " Like block called fused and become which is just a handcrafted module which was created by modifying the ambi com from the mobile net paper and"}, {"start": 59.18, "end": 61.7, "text": " We'll see those a bit later. It's not that important"}, {"start": 61.7, "end": 68.25999999999999, "text": " so that's the first thing the second thing is they updated the the objective function by adding the training step time and by adding the"}, {"start": 69.38, "end": 75.1, "text": " Number of parameters into the objective function. Finally, the the last thing is progressive training"}, {"start": 75.1, "end": 81.0, "text": " training whereby they while they're training there they are slowly and gradually increasing the"}, {"start": 81.33999999999999, "end": 84.3, "text": " Image resolution, but they're also increasing the regularization"}, {"start": 85.02, "end": 90.94, "text": " Like dropout and some random augmentations. We'll see those details a bit later. So those are the main ideas"}, {"start": 90.94, "end": 92.94, "text": " So first like the the search"}, {"start": 93.34, "end": 95.34, "text": " State the search space and the search"}, {"start": 96.41999999999999, "end": 101.13999999999999, "text": " Strategy were updated a bit and finally the progressive training. That's the second thing"}, {"start": 101.14, "end": 107.34, "text": " Before I even start digging into the paper. I want to walk you through the body of literature that came before this paper and"}, {"start": 107.62, "end": 110.78, "text": " The interesting thing is that this author the quality"}, {"start": 111.3, "end": 115.06, "text": " basically was the co-author of many of the first and"}, {"start": 115.5, "end": 121.22, "text": " Seminal neural architecture search papers including the first one the the NAS with RL paper"}, {"start": 121.46000000000001, "end": 128.06, "text": " Then we had the NAS net paper then we had the M. Ness paper and finally efficient net V1"}, {"start": 128.06, "end": 130.06, "text": " I just want to walk you through and"}, {"start": 130.06, "end": 132.06, "text": " because I think"}, {"start": 132.26, "end": 140.02, "text": " You understanding the this previous work will help you help ground your understanding and better better better grasp the v2 version"}, {"start": 140.5, "end": 143.86, "text": " Okay. So first things first. What's neural architecture search?"}, {"start": 143.86, "end": 149.26, "text": " the basic idea is this you have you have a controller which is usually some RNN like LSTM and"}, {"start": 150.14000000000001, "end": 156.54, "text": " Basically you sample architectures from your search space you evaluate the architectures by training them and"}, {"start": 156.54, "end": 164.18, "text": " You basically evaluate them on the validation set you get the accuracy and you get you you basically give the controller the reward"}, {"start": 164.82, "end": 169.73999999999998, "text": " Which is the evaluation accuracy and this is your common RL"}, {"start": 170.26, "end": 177.06, "text": " Loop you have the agent you have the environment you get rewards and you can train the agent the controller here to"}, {"start": 177.78, "end": 184.62, "text": " Eventually sample those architectures which have high reward ie which have high validation accuracy and you can see here"}, {"start": 184.62, "end": 187.70000000000002, "text": " How the sampling happens basically the controller?"}, {"start": 188.22, "end": 192.06, "text": " Samples first maybe the number of filters then what's the height of the filter was the width?"}, {"start": 192.22, "end": 199.54, "text": " What's the stride blah blah blah in in the every single layer and that's something called global search space because you're basically"}, {"start": 200.38, "end": 204.98000000000002, "text": " For all of the end layers you have the maximum flexibility you can pick every single layer"}, {"start": 204.98000000000002, "end": 210.94, "text": " Maybe different but the the like the search space is huge and this didn't work that good"}, {"start": 210.94, "end": 216.9, "text": " I mean it did work very well, but they needed a bunch of compute like some huge number of GPU hours and"}, {"start": 217.42, "end": 222.62, "text": " Then the second paper the NAS net paper this learning transferable architectures for scalable image recognition"}, {"start": 222.82, "end": 225.72, "text": " What it did basically was scope down the search space by?"}, {"start": 226.7, "end": 231.36, "text": " By noticing that most of the handcrafted architectures in the literature like mobile"}, {"start": 232.18, "end": 237.22, "text": " Nets etc and resnets and VGG and whatnot they basically tend to"}, {"start": 237.22, "end": 243.46, "text": " Create a cell and then just replicate it and so what they decided to do is just design the cell"}, {"start": 243.86, "end": 248.98, "text": " Basically you search for the cell and then you repeat the cell n times in order to get your final"}, {"start": 249.54, "end": 252.54, "text": " Architecture and that's it that was the main idea in that paper"}, {"start": 253.18, "end": 258.82, "text": " M s net which is really closely related to efficient nets did the following so basically"}, {"start": 259.74, "end": 261.74, "text": " They introduced"}, {"start": 261.74, "end": 268.36, "text": " They they again modify the search space a little bit so now they have like a couple blocks like this"}, {"start": 269.18, "end": 270.66, "text": " Let's say maybe seven"}, {"start": 270.66, "end": 274.58, "text": " I think they had some number like that and what I do is you you now?"}, {"start": 275.14, "end": 280.14, "text": " Sample the first layer in each of these blocks, and then you just repeat them"}, {"start": 280.90000000000003, "end": 284.22, "text": " By some number of times which is also a hyper parameter"}, {"start": 284.22, "end": 291.74, "text": " Which we search for so l1 here l2 here, so we'll have l2 of these will have l3 ln of these and"}, {"start": 292.5, "end": 295.98, "text": " This is a search space and additionally they included the latency"}, {"start": 296.46000000000004, "end": 303.5, "text": " Into the final reward objective function so not only the accuracy on the validation set but also the latency which made this"}, {"start": 304.3, "end": 311.14000000000004, "text": " M nest paper suitable for finding architectures that could run on edge devices such as mobile devices etc"}, {"start": 311.14, "end": 316.9, "text": " The loop is pretty much the same the reward function is just modified. This is still"}, {"start": 317.62, "end": 319.8, "text": " Lstm the only thing they"}, {"start": 321.09999999999997, "end": 326.5, "text": " Actually a change is they are not using the reinforce algorithm. They're using PPO from open AI"}, {"start": 327.09999999999997, "end": 329.41999999999996, "text": " as the RL algorithm of choice"}, {"start": 330.38, "end": 334.14, "text": " those are the main ideas finally we get to efficient net and"}, {"start": 334.14, "end": 341.86, "text": " Efficient net basically uses the same space as M. Ness net the only difference is instead of latency"}, {"start": 341.86, "end": 344.18, "text": " They are actually optimizing for flops"}, {"start": 344.38, "end": 349.86, "text": " So they they are basically that means the floating number of floating operations per second"}, {"start": 350.3, "end": 355.24, "text": " And that's that's it they the trick they did was to find the"}, {"start": 356.06, "end": 363.21999999999997, "text": " baseline architectures called B zero by using this M. Ness space I just explained and then they found these"}, {"start": 363.22, "end": 370.06, "text": " Basically then they scale the the B zero architecture by scaling the depth scaling the width"}, {"start": 370.06, "end": 372.74, "text": " I the number of channels and the image resolution"}, {"start": 373.54, "end": 374.74, "text": " pretty much"}, {"start": 374.74, "end": 379.86, "text": " Proportionally that means they just do some great search define this alpha beta and gamma"}, {"start": 380.58000000000004, "end": 383.5, "text": " which control depth width and"}, {"start": 384.54, "end": 388.58000000000004, "text": " even resolution and then they just raise these parameters to Phi and"}, {"start": 388.58, "end": 392.9, "text": " And Phi is the thing that controls how you get novel architecture"}, {"start": 392.9, "end": 399.94, "text": " So basically by setting Phi to 2 they'll they'll sample B one and then they'll have B one through B seven depending on the Phi"}, {"start": 400.7, "end": 406.74, "text": " The nice thing about Phi is that when you when you put when it when Phi is one compared to when Phi is zero"}, {"start": 407.09999999999997, "end": 413.06, "text": " The the B one model the model with Phi equals one will have twice as much flops"}, {"start": 413.06, "end": 419.02, "text": " Compared to B zero so they can kind of intuitively increase at the Phi and they'll proportion"}, {"start": 419.02, "end": 423.38, "text": " they'll have a feeling for how much compute how much additional compute they will need and"}, {"start": 424.14, "end": 429.06, "text": " Basically the accuracy will improve as well. So those are the main ideas that you need to understand before"}, {"start": 430.22, "end": 432.22, "text": " grasping the v2 paper"}, {"start": 432.22, "end": 435.9, "text": " Hopefully this was useful if you if you like this part just comment down below"}, {"start": 436.26, "end": 438.62, "text": " So that I know whether it was useful or not"}, {"start": 439.1, "end": 441.1, "text": " Anyways, let's jump into the paper itself"}, {"start": 441.1, "end": 443.1, "text": " okay, so"}, {"start": 443.54, "end": 447.42, "text": " First things first to develop this family of models. We use a combination of training aware"}, {"start": 447.82000000000005, "end": 453.14000000000004, "text": " Mass and scaling to jointly optimize training speed and parameter efficiency. So that's the thing"}, {"start": 453.66, "end": 458.06, "text": " I mentioned basically instead of just optimizing latency or flops"}, {"start": 458.18, "end": 463.58000000000004, "text": " They're optimizing these two so the training speed and the parameter efficiency explicitly into the objective function"}, {"start": 464.3, "end": 467.38, "text": " So they they make it explicit part of objective function"}, {"start": 467.38, "end": 473.9, "text": " The models were searched from search space enriched with new ops such as fused and become so that's the layer"}, {"start": 473.9, "end": 475.9, "text": " I mentioned that we'll see that in a bit"}, {"start": 475.9, "end": 479.82, "text": " And yeah, those are some of the main ideas and this is the second idea"}, {"start": 479.82, "end": 484.14, "text": " We propose an improved method of progressive learning which adaptive adjust regularization"}, {"start": 484.14, "end": 490.82, "text": " So the dropout and data augmentation along with the image size and we'll see how that works in detail in a bit"}, {"start": 491.1, "end": 495.9, "text": " Okay, so they mentioned here that training efficiency has gained significant interest recently"}, {"start": 495.9, "end": 499.09999999999997, "text": " so they mentioned the NF nets and some other papers and"}, {"start": 499.58, "end": 504.46, "text": " What I say then is that these methods often come with expensive overhead on parameter size"}, {"start": 504.46, "end": 507.29999999999995, "text": " And if you take a look at this table, you'll see that"}, {"start": 508.14, "end": 512.42, "text": " Basically, here's the efficient that we won resonant some vision transformer"}, {"start": 512.42, "end": 516.66, "text": " You can see that the number of parameters that the V2 has is much lower"}, {"start": 517.5, "end": 524.74, "text": " Than than than these two and even the V1 has much smaller amount of parameters compared to vision transformer in this resonant"}, {"start": 524.74, "end": 526.46, "text": " derivative"}, {"start": 526.46, "end": 528.46, "text": " so usually people"}, {"start": 529.1, "end": 532.9, "text": " omit some of the dimensions and then the results are inconclusive"}, {"start": 533.58, "end": 538.38, "text": " And they actually stressed that a couple of times throughout this paper, but we'll get to that in a moment"}, {"start": 539.3, "end": 543.74, "text": " Okay, so our study shows inefficientness. So it's basically they they"}, {"start": 544.78, "end": 551.0600000000001, "text": " Dissect efficient nap V1 and try to find the bottlenecks so they can improve the version 2 which makes sense, right?"}, {"start": 551.06, "end": 559.26, "text": " So training with very large image sizes is slow. So the thing with image net V1 was that basically the the B0 model used"}, {"start": 559.9, "end": 566.2199999999999, "text": " The image resolution 2 to 4 times 2 to 4 the B7 used I think somewhere around"}, {"start": 567.42, "end": 568.54, "text": " 560"}, {"start": 568.54, "end": 574.14, "text": " And once you have this image size, basically you you can you can you can put so much"}, {"start": 574.78, "end": 580.18, "text": " On on the GPU. So if your image size increases the batch size will have to decrease"}, {"start": 580.18, "end": 584.2199999999999, "text": " And once the batch size decreases that means your training is going to slow down"}, {"start": 584.5, "end": 589.38, "text": " So that's the first thing they're gonna tackle with and that's gonna be done with this"}, {"start": 589.8599999999999, "end": 591.4599999999999, "text": " like"}, {"start": 591.4599999999999, "end": 595.7399999999999, "text": " Progressive increase of image resolution which you can treat as a sort of a curriculum"}, {"start": 596.3, "end": 601.26, "text": " learning basically the second thing is depthwise convolutions are slow in early layers and"}, {"start": 601.9399999999999, "end": 605.2199999999999, "text": " Thirdly equally scaling up every stage is suboptimal"}, {"start": 605.22, "end": 611.46, "text": " So that's the thing I mentioned here with those alpha betas and gammas what I do is"}, {"start": 611.9, "end": 613.9, "text": " Let's take the MNAS architecture here"}, {"start": 614.02, "end": 618.98, "text": " Basically if alpha so that controls the depth is 2 that means every single layer"}, {"start": 619.1, "end": 624.78, "text": " So in every single stage is going to increase by 2 instead of maybe increasing here by I don't know"}, {"start": 625.4200000000001, "end": 630.9, "text": " 2.3 and here maybe earlier you won't need that much you'll put 1.7 you get the point"}, {"start": 630.9, "end": 635.3, "text": " basically, they want to create a non uniform scaling instead of this and"}, {"start": 637.86, "end": 644.38, "text": " Based on these observations we design a search space enriched with additional ops such as fused and become and apply training awareness"}, {"start": 644.78, "end": 650.38, "text": " And scaling to jointly optimize model accuracy training speed and parameter size"}, {"start": 650.5, "end": 656.66, "text": " So I want to stress this training speed and parameter size is something they explicitly encode into the objective function"}, {"start": 656.66, "end": 660.6999999999999, "text": " Okay, hopefully that's clear. I'm gonna skip these parts. Whoops and"}, {"start": 661.4599999999999, "end": 663.4599999999999, "text": " Let's start here"}, {"start": 664.8199999999999, "end": 668.54, "text": " While many recent books have claimed large gains in training or inference speed"}, {"start": 668.54, "end": 673.78, "text": " They often much worse. They're often much worse than efficient net in terms of parameters and flops efficiency"}, {"start": 673.78, "end": 681.26, "text": " That's something I mentioned a couple minutes ago. Basically you can see here that even though the NF net and resonant RS have"}, {"start": 681.26, "end": 686.18, "text": " Really nice top one accuracy if you take a look at these two dimensions the parameters and flops"}, {"start": 686.18, "end": 691.98, "text": " They are even worse than then the V1 version of efficient net. Okay, I mentioned this already and"}, {"start": 692.78, "end": 697.54, "text": " This thing let's let's let's dig a bit deeper into this so"}, {"start": 698.18, "end": 704.98, "text": " Depthwise convolutions have fewer parameters and flops than regular convolutions, but they cannot fully utilize modern accelerators"}, {"start": 705.7, "end": 706.7, "text": " so"}, {"start": 706.7, "end": 709.18, "text": " Here's the thing the Mb conv layer"}, {"start": 709.18, "end": 711.18, "text": " so this is the"}, {"start": 711.18, "end": 717.42, "text": " Basically a module that somebody handcrafted a couple of years ago in the mobile net paper and you can see the structure"}, {"start": 717.42, "end": 719.8199999999999, "text": " This is also called inverted residual"}, {"start": 720.6999999999999, "end": 723.3399999999999, "text": " bottleneck layer because you can see that the"}, {"start": 723.9, "end": 729.06, "text": " Number of channels actually increases here and then you have a residual connection the skip connection connecting"}, {"start": 729.5799999999999, "end": 732.06, "text": " the like the usual skip connection, okay, and"}, {"start": 732.54, "end": 736.8199999999999, "text": " The thing is these depthwise convolutions are not"}, {"start": 736.82, "end": 742.6600000000001, "text": " So modern accelerators the hover accelerators don't support this operation as much so they just kind of"}, {"start": 743.0600000000001, "end": 745.0600000000001, "text": " replaced it with calm three by three and"}, {"start": 745.5400000000001, "end": 751.38, "text": " Everything else stays the same. So these two are kind of fused together. That's hence fused and become and"}, {"start": 752.1, "end": 754.1, "text": " they said here that"}, {"start": 755.3000000000001, "end": 760.6600000000001, "text": " Basically can fuse them becomes can improve training speed with a small overhead and parameters and flops"}, {"start": 760.6600000000001, "end": 765.4200000000001, "text": " But if we replace all blocks with fused and become then it significantly increases the"}, {"start": 765.42, "end": 768.02, "text": " parameters and flops while also slowing down the training so"}, {"start": 768.9, "end": 776.06, "text": " What I want to say is that you can simply just replace the M becomes with this newly this new version the fuse version"}, {"start": 776.62, "end": 780.86, "text": " And they have some relations here. Basically when you have no fuse layers"}, {"start": 780.86, "end": 787.18, "text": " You can see the parameters the the flops and the accuracy but once you once so once you add the fused"}, {"start": 787.74, "end": 791.42, "text": " Layer the fuse modules in the in the early layers of your architecture"}, {"start": 791.42, "end": 797.74, "text": " You get small increase in parameters and flops, but you get some decent decrease in accuracy and increase in throughput"}, {"start": 798.2199999999999, "end": 804.78, "text": " Like the number of images per second per per like TPU or or or V hundred basically faster training speed"}, {"start": 805.3399999999999, "end": 807.8199999999999, "text": " but once you start adding a lot of these like"}, {"start": 808.4599999999999, "end": 815.0999999999999, "text": " In the first five layers if you if you replace the the M become by fuse one by the fuse variant in the first five layers"}, {"start": 815.5799999999999, "end": 819.3399999999999, "text": " The parameters the number of parameters and the flops just blow up"}, {"start": 819.34, "end": 825.1800000000001, "text": " Blows off and you don't get a much accuracy improvement and neither do you get training speed"}, {"start": 825.1800000000001, "end": 827.1, "text": " So there is a trade-off obviously here"}, {"start": 827.1, "end": 833.58, "text": " So they mentioned that they just want to include this thing in the as the part of the search as well"}, {"start": 834.62, "end": 835.74, "text": " Okay"}, {"start": 835.74, "end": 839.4200000000001, "text": " Let's continue. So I mentioned this already scaling up every stage is"}, {"start": 840.14, "end": 847.26, "text": " Suboptimal so you want to have non-uniform scaling and they mentioned here the non-uniform scaling strategy to gradually add"}, {"start": 847.26, "end": 849.5, "text": " More layers to later stages"}, {"start": 849.5, "end": 853.26, "text": " That's some kind of heuristic and they mentioned here says a heuristic"}, {"start": 853.26, "end": 856.3, "text": " We also gradually add more layers to later stages"}, {"start": 856.3, "end": 860.86, "text": " So as you can see this neural architecture search business is basically a little bit"}, {"start": 861.34, "end": 867.5, "text": " Handcrafting of the initial search space and some heuristics and some search so it's it's a there's a lot of engineering details"}, {"start": 867.9, "end": 872.3, "text": " But you'll hopefully get the the main points and the gist of this paper"}, {"start": 872.3, "end": 879.0999999999999, "text": " Soon, okay. So here is that additional detail. So since the search space is smaller and it's smaller because"}, {"start": 879.66, "end": 886.06, "text": " Efficient net b1 didn't use some of the options it had so this just kind of cut those options from the search space"}, {"start": 886.06, "end": 889.26, "text": " And now the search space is smaller. So they say here since it's smaller"}, {"start": 889.26, "end": 894.6999999999999, "text": " We can simply apply a random search on much larger networks. They have comparable size as efficient net b4"}, {"start": 895.26, "end": 898.6999999999999, "text": " Which was so we had b0 to b7. So that's something in between"}, {"start": 898.7, "end": 902.3000000000001, "text": " In the middle and the the main point I want you to understand here is that"}, {"start": 903.1, "end": 907.1, "text": " The the the model size you're searching for will be the most optimal one"}, {"start": 907.1, "end": 910.5400000000001, "text": " But once you start scaling to either smaller models or bigger models"}, {"start": 911.6600000000001, "end": 915.6600000000001, "text": " If the scaling strategy is good, you'll get decent results"}, {"start": 915.6600000000001, "end": 921.1, "text": " But they still won't be as optimal is as if you directly search for them. So"}, {"start": 921.82, "end": 924.94, "text": " They just found found the trade-off that searching for the"}, {"start": 924.94, "end": 929.6600000000001, "text": " Size that's equivalent to this b4 model is the best trade-off point"}, {"start": 930.62, "end": 931.82, "text": " and"}, {"start": 931.82, "end": 935.1, "text": " Having said that this is the the the the loss function"}, {"start": 935.1, "end": 938.94, "text": " I was mentioning so the search reward combines the model accuracy"}, {"start": 939.1800000000001, "end": 945.9000000000001, "text": " The normal training step time and the parameter size using a simple way to product accuracy times"}, {"start": 946.86, "end": 949.4200000000001, "text": " the the the training step time and"}, {"start": 949.42, "end": 954.14, "text": " Parameter size you can see they're raised to the power of w which are negative. So basically you'll want to"}, {"start": 954.6999999999999, "end": 960.62, "text": " Pull this down to to zero in order to to increase the the reward and just to contrast"}, {"start": 961.02, "end": 967.3399999999999, "text": " The reward for the v2 version of efficient net with with m nes paper and enet v1"}, {"start": 968.2199999999999, "end": 974.78, "text": " Let me show you this formula. So they previously had accuracy and the latency if you remember and w was again negative"}, {"start": 974.78, "end": 980.38, "text": " so that means you want to uh, push this down to zero you want to increase this and basically"}, {"start": 980.9399999999999, "end": 985.98, "text": " That will lead to higher rewards and that's what the controller learns to do"}, {"start": 985.98, "end": 989.8199999999999, "text": " So he finds highly accurate models with low latency constraints"}, {"start": 990.54, "end": 995.8199999999999, "text": " The enet v1 had as I mentioned flops instead of latency everything else remains the same"}, {"start": 995.8199999999999, "end": 998.38, "text": " And here we can see aside from accuracy we have"}, {"start": 998.38, "end": 1004.78, "text": " These two so the the the training speed and the number of parameters the nice thing about this thing here about s"}, {"start": 1005.1, "end": 1011.18, "text": " Is that it correlates really nicely and they show that with the latency. So the the higher the the training speed"}, {"start": 1011.98, "end": 1017.9, "text": " Basically the lower the latency. So so so the the faster you're training the the lower the latency will be and that's cool. Let's continue"}, {"start": 1019.5, "end": 1021.74, "text": " Here is finally what they find"}, {"start": 1022.54, "end": 1023.74, "text": " And"}, {"start": 1023.74, "end": 1028.3, "text": " This is the architecture they they they they search for the optimal one"}, {"start": 1028.3, "end": 1032.94, "text": " They have some fused mb com layers in the initial stages as you can see here"}, {"start": 1032.94, "end": 1039.18, "text": " And then they have the mb coms the usual stuff and you can also see that the number of channel increases here"}, {"start": 1039.18, "end": 1046.46, "text": " Which is the heuristic I mentioned so compare that to the uh b0 efficient net. So that's the v1 version"}, {"start": 1046.78, "end": 1050.86, "text": " The previous version of efficient net they only increase the number of channels"}, {"start": 1050.86, "end": 1055.1799999999998, "text": " And the v1 version the previous version of efficient net they only used and becomes okay"}, {"start": 1055.1799999999998, "end": 1060.3, "text": " Let me let me explain you the the progressive learning part, which is arguably one of the the most important"}, {"start": 1061.08, "end": 1063.08, "text": " algorithmic improvements of this paper"}, {"start": 1063.74, "end": 1070.56, "text": " So first the motivation for this thing, uh, they notice that when the image size is small like 128 by 128"}, {"start": 1071.02, "end": 1076.62, "text": " Uh, the the the smaller the regularization the smaller the augmentations they are using the higher the accuracy"}, {"start": 1076.62, "end": 1081.58, "text": " And as you're increasing the image resolution we can see here that basically"}, {"start": 1082.3, "end": 1083.8999999999999, "text": " the higher"}, {"start": 1083.8999999999999, "end": 1087.82, "text": " The regularization the better the accuracy and you can see the trend here"}, {"start": 1087.9799999999998, "end": 1095.26, "text": " So basically the higher the the resolution you want to increase the regularization in order to get the optimal the the highest performance"}, {"start": 1095.7399999999998, "end": 1103.1799999999998, "text": " Okay, so that's the motivation and what's the main idea? So here you can see so the the progressively as the training progresses"}, {"start": 1103.18, "end": 1108.0600000000002, "text": " So this is the number of epochs on the on the axis and you can see in the initial axis"}, {"start": 1108.0600000000002, "end": 1113.5800000000002, "text": " The images are smaller here. They are bigger and also the augmentations are increasing so you can see the shears here"}, {"start": 1114.0600000000002, "end": 1118.7, "text": " Uh some like geometric transformations of the image you can see them the mix up"}, {"start": 1119.5800000000002, "end": 1124.4, "text": " Augmentation here kicking in basically they are mixing up two images together"}, {"start": 1124.8600000000001, "end": 1127.3400000000001, "text": " And so that's a high level idea and here"}, {"start": 1127.34, "end": 1132.9399999999998, "text": " How the algorithm works like is basically they they split the training into m steps"}, {"start": 1133.34, "end": 1137.1, "text": " Uh where m was four I think so basically you have it something like this"}, {"start": 1137.34, "end": 1141.6599999999999, "text": " So this is the like the the the the training and this is the epoch zero"}, {"start": 1141.74, "end": 1145.1, "text": " This is the epoch n whatever and we just split this into four parts"}, {"start": 1145.74, "end": 1147.74, "text": " Like this, right? Okay"}, {"start": 1148.22, "end": 1151.6599999999999, "text": " And in this portion they use the initial"}, {"start": 1151.66, "end": 1157.66, "text": " Uh image size and the initial magnitude of augmentations and then they're linearly increasing those"}, {"start": 1158.22, "end": 1162.78, "text": " In every single stage. So here they're just going to increase it and here in this stage"}, {"start": 1162.8600000000001, "end": 1166.3000000000002, "text": " They're going to increase it further and finally, uh the highest"}, {"start": 1167.5800000000002, "end": 1172.7, "text": " The the magnitude of augmentation will be the highest as well as the image size and if we zoom in here"}, {"start": 1173.3400000000001, "end": 1174.7, "text": " basically"}, {"start": 1174.7, "end": 1176.7, "text": " So for all of these m stages"}, {"start": 1176.7, "end": 1183.5800000000002, "text": " They're just linearly interpolating between the target image size between the start image size just simple linear interpolation here"}, {"start": 1183.9, "end": 1188.8600000000001, "text": " and same goes for the regularization just uh linear interpolation between the"}, {"start": 1189.98, "end": 1195.26, "text": " Initial magnitude and the final magnitude and let's see what the regularizations they are using"}, {"start": 1195.42, "end": 1200.88, "text": " The one is dropout like classical stuff. Uh, the second one is this random random augmentation"}, {"start": 1201.26, "end": 1205.26, "text": " where they're basically just combining a bunch of stuff like shearing like uh,"}, {"start": 1205.26, "end": 1209.02, "text": " uh various uh geometric and photometric augmentations"}, {"start": 1209.5, "end": 1212.86, "text": " So the mix-up is just uh, you're you're basically you saw it here"}, {"start": 1213.1, "end": 1218.3799999999999, "text": " Uh, you're mixing up two images in the image space and you're also mixing their their labels"}, {"start": 1218.7, "end": 1223.34, "text": " So the the target label will be some uh, like mixture between these two"}, {"start": 1223.34, "end": 1229.18, "text": " So for example if we know that we so we had maybe uh, like a dog here and we had"}, {"start": 1229.18, "end": 1236.14, "text": " Uh cat here and basically if we want to mix up, uh with lambda equals, I don't know 0.2"}, {"start": 1236.6200000000001, "end": 1238.8600000000001, "text": " Uh, you'll you'll end up with the the final"}, {"start": 1239.42, "end": 1240.3, "text": " um"}, {"start": 1240.3, "end": 1242.3, "text": " Target will be this like you'll have"}, {"start": 1243.02, "end": 1247.26, "text": " 02 for for I don't know dog so this is 02 and this will be"}, {"start": 1247.74, "end": 1250.22, "text": " For this thing for the cat you'll have 0.8"}, {"start": 1250.7, "end": 1254.22, "text": " And so you'll want to train your model to predict"}, {"start": 1254.94, "end": 1255.9, "text": " this"}, {"start": 1255.9, "end": 1261.02, "text": " Distribution when you input this image here except we don't have dog and cat here"}, {"start": 1261.02, "end": 1264.0600000000002, "text": " But some panda bear and something some fish or whatever. Okay"}, {"start": 1264.6200000000001, "end": 1269.74, "text": " Uh, you get a point. Let's continue and let's see the results. Um"}, {"start": 1271.02, "end": 1276.5400000000002, "text": " Before that, I just want to mention one more, uh regularization they're using that's this stochastic depth"}, {"start": 1277.02, "end": 1280.94, "text": " So the main idea is the following so you have the the resnet block"}, {"start": 1281.3400000000001, "end": 1284.5400000000002, "text": " Basically, you transform the features you're probably familiar with resnets"}, {"start": 1284.54, "end": 1289.1, "text": " You have this residual connect the skip connection which just adds these features without transforming them"}, {"start": 1289.58, "end": 1295.1, "text": " You add them up and that's your residual block here. So what stochastic depth does?"}, {"start": 1295.58, "end": 1300.86, "text": " It has this survival probability, which is basically the probability that this block will remain here"}, {"start": 1301.18, "end": 1305.1, "text": " So that means with 0.2 probability you'll actually short"}, {"start": 1305.58, "end": 1311.8999999999999, "text": " Um connect this thing so you'll just remove this thing and basically, uh, you you'll effectively end up"}, {"start": 1311.9, "end": 1317.02, "text": " Uh with the shallower model. So by having stochastic depth"}, {"start": 1317.5, "end": 1318.3000000000002, "text": " um"}, {"start": 1318.3000000000002, "end": 1319.18, "text": " the"}, {"start": 1319.18, "end": 1325.98, "text": " It's just they just show that that improves the accuracy and it's just a variation on the dropout. Uh, like uh"}, {"start": 1327.1000000000001, "end": 1328.5400000000002, "text": " concept basically"}, {"start": 1328.5400000000002, "end": 1331.3400000000001, "text": " um, you want to make sure that the"}, {"start": 1332.14, "end": 1335.0400000000002, "text": " That the network becomes robust to the co-adaptation"}, {"start": 1335.74, "end": 1337.9, "text": " So similarly as you had dropout"}, {"start": 1337.9, "end": 1342.6200000000001, "text": " So the original dropout was you have maybe like three neurons here two neurons here"}, {"start": 1342.94, "end": 1348.6200000000001, "text": " You have a fully connected layer and you'll with some probability you'll you'll pull this"}, {"start": 1349.18, "end": 1356.38, "text": " Weight to zero and basically the other weights have to learn how to not rely too much on on that particular edge"}, {"start": 1356.8600000000001, "end": 1360.46, "text": " Or on any particular edge in in for that matter and here"}, {"start": 1360.46, "end": 1364.94, "text": " This is just a generalization in a sense where we are just removing the whole module"}, {"start": 1364.94, "end": 1369.42, "text": " And we want to make sure that the network is robust that when certain modules just disappear"}, {"start": 1369.5800000000002, "end": 1372.8600000000001, "text": " It will still be able to predict the the the image"}, {"start": 1373.66, "end": 1375.66, "text": " Like the the class in the image"}, {"start": 1375.9, "end": 1378.6200000000001, "text": " In this case, okay, that's the that's the main idea"}, {"start": 1379.5, "end": 1386.88, "text": " Let's do results. So the first result is so in particular our efficient napv2m achieves comparable accuracy to efficient napv7"}, {"start": 1387.66, "end": 1393.8200000000002, "text": " while training 11x faster using the same computing resources and this was one of the"}, {"start": 1393.82, "end": 1401.76, "text": " objectives in the training so that makes sense, but those results are really impressive. So let's take a look here and you can see"}, {"start": 1403.6799999999998, "end": 1404.96, "text": " For both"}, {"start": 1404.96, "end": 1406.32, "text": " number of parameters"}, {"start": 1406.32, "end": 1411.84, "text": " Flops i.e. compute and latency you can see that the efficient napv2 forms an envelope"}, {"start": 1412.48, "end": 1417.84, "text": " Above all of the previous models like nfnets and vision transformers and whatnot"}, {"start": 1417.84, "end": 1424.8, "text": " Uh same goes for every single dimension. The interesting thing is how much better it is than the vision transformer and I mentioned it here"}, {"start": 1425.04, "end": 1430.08, "text": " So recently vision transformers have demonstrated impressive results on image net accuracy and training speed"}, {"start": 1430.48, "end": 1434.0, "text": " However here we show that properly designed com nets"}, {"start": 1434.48, "end": 1441.1999999999998, "text": " With improved training method can still largely outperform vision transformers in both accuracy and training efficiency"}, {"start": 1441.6799999999998, "end": 1445.6, "text": " and if we take a look at the numbers, uh, we can see like the"}, {"start": 1445.6, "end": 1452.1599999999999, "text": " The v2l which is not pre-trained on the this larger image nap which has around 21 000 classes"}, {"start": 1452.6399999999999, "end": 1453.76, "text": " but the"}, {"start": 1453.76, "end": 1456.32, "text": " version from 2012 which had 1000 classes"}, {"start": 1457.04, "end": 1458.9599999999998, "text": " basically the numbers are"}, {"start": 1458.9599999999998, "end": 1461.6, "text": " much better as you can see here compared to"}, {"start": 1462.32, "end": 1468.7199999999998, "text": " Uh, even the even the biggest of vision transformers which were pre-trained on the image net 21k"}, {"start": 1469.1999999999998, "end": 1471.1999999999998, "text": " And you can see the numbers here"}, {"start": 1471.2, "end": 1477.8400000000001, "text": " You can see that the accuracy is is higher. We can see that we have much lower number of parameters"}, {"start": 1478.32, "end": 1481.52, "text": " Uh, we have uh much less compute here"}, {"start": 1482.24, "end": 1488.48, "text": " And what's this? Uh, yeah inference is also smaller and the training time is also smaller"}, {"start": 1488.8, "end": 1491.92, "text": " So across every single dimension is just better"}, {"start": 1492.88, "end": 1498.96, "text": " Which means that the biases we have in convolutional neural networks are not going to go any anytime soon. So it seems"}, {"start": 1498.96, "end": 1506.88, "text": " Uh and vision transformer took a lot of time to train and a lot of data. I think it was uh, initially trained on the this"}, {"start": 1507.76, "end": 1515.6000000000001, "text": " Private data set that google has which is huge. So yeah, uh, these are some nice results, uh from from the google brain team"}, {"start": 1516.8, "end": 1519.28, "text": " Let's continue and see here"}, {"start": 1519.44, "end": 1524.32, "text": " So they also showed that the performance on when transferring to other classical"}, {"start": 1524.32, "end": 1529.2, "text": " Uh data sets such as cipher 10 cipher 100 flowers cars we can see that"}, {"start": 1529.84, "end": 1532.08, "text": " They achieved the the highest accuracy as well here"}, {"start": 1532.3999999999999, "end": 1540.0, "text": " So two conclusions they they derive here is scaling up data size is more effective than simply scaling up model size in high accuracy regime"}, {"start": 1540.56, "end": 1542.96, "text": " so basically, uh, that means that this"}, {"start": 1543.5, "end": 1550.96, "text": " pre-training they uh this uh additional pre-training that did on the image net 21k helped boost the the the accuracy as you can see"}, {"start": 1550.96, "end": 1552.96, "text": " in this column and"}, {"start": 1552.96, "end": 1556.8, "text": " just having a bigger model will lead to overfitting so they just"}, {"start": 1558.16, "end": 1560.16, "text": " Mentioned that data is"}, {"start": 1560.64, "end": 1564.64, "text": " What what matters and some other work supports the claim as well?"}, {"start": 1565.6000000000001, "end": 1572.08, "text": " So although the image 21k has 10x more data our training approach enables us to finish the pre-training of efficient"}, {"start": 1572.08, "end": 1575.3600000000001, "text": " NAB 2 within two days using 32 tpu cores"}, {"start": 1575.3600000000001, "end": 1581.6000000000001, "text": " And this is pretty impressive like in only two days you can train on this huge data set, which is a nice nice"}, {"start": 1581.6, "end": 1588.56, "text": " Uh nice thing to to to have obviously so because they reduce the training time. They can now train it in two days and achieve"}, {"start": 1589.36, "end": 1592.8, "text": " Uh great results. So they did a lot of ablations and show that"}, {"start": 1593.4399999999998, "end": 1595.4399999999998, "text": " uh each of the new"}, {"start": 1596.0, "end": 1598.24, "text": " Uh novel things they introduced like progressive training"}, {"start": 1598.9599999999998, "end": 1601.6799999999998, "text": " uh leads to to improvements in results"}, {"start": 1602.56, "end": 1604.0, "text": " um and"}, {"start": 1604.0, "end": 1610.9599999999998, "text": " What's interesting is maybe that here they just tried progressive learning for resonance and they showed that using progressive"}, {"start": 1610.96, "end": 1612.96, "text": " training you can improve the"}, {"start": 1613.3600000000001, "end": 1619.68, "text": " uh accuracy on the resonance and not on the efficient nets but on resonance and uh also decrease the"}, {"start": 1620.16, "end": 1626.08, "text": " uh training time same goes for the v1 so basically, uh this training, uh,"}, {"start": 1626.32, "end": 1632.42, "text": " This progressive training is not it's not it has not over fitted to the particular architecture"}, {"start": 1632.8, "end": 1638.48, "text": " From efficient net v2. So yeah, that's not nice ablation there. Um, that's pretty much it"}, {"start": 1638.48, "end": 1640.48, "text": " so basically"}, {"start": 1640.8, "end": 1643.1200000000001, "text": " Search is the name of the game"}, {"start": 1643.6, "end": 1645.6, "text": " Um this progressive training"}, {"start": 1645.84, "end": 1648.64, "text": " Improves the training speed it improves the accuracy"}, {"start": 1648.96, "end": 1653.44, "text": " A couple of heuristics here and there and that's it. That's the version two of efficient net"}, {"start": 1653.76, "end": 1656.72, "text": " Hopefully you found this video useful if you did leave the comment"}, {"start": 1657.04, "end": 1660.24, "text": " uh share that would be super helpful and"}, {"start": 1660.24, "end": 1668.5, "text": " Um hit that bell icon to get notified when I upload a new video until next time. See you"}]