Focal Transformer: Focal Self-attention for Local-Global Interactions in Vision Transformers
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I cover a new paper coming from Microsoft: "Focal Self-attention for Local-Global Interactions in Vision Transformers" where they introduce a new transformer layer called focal attention.

The main idea is to reduce the complexity but preserve the long-range dependencies. They achieve this by attending to the nearby tokens in a fine-grained manner and to the tokens that are further away they attend their coarsened representations.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2107.00641
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable:

00:00 Main idea of the paper: focal self-attention
04:55 Overview of Focal Transformer architecture
08:15 Focal Self-Attention layer
12:30 Computational complexity, overlapping regions
15:30 SOTA results but with a disclaimer
17:30 Ablations
19:50 Outro, Focal Transformer is slower than Swin

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️


Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#focaltransformer #microsoft #transformer
 What's up? In this video I'm covering focal self-attention for local global interactions in vision transformers by the Microsoft Research and Microsoft Cloud and AI teams. So they basically introduced this novel layer called focal self-attention and introduced this novel model called focal transformer. And let's see what it's all about. Basically if you focus on this diagram here, let me kind of motivate the introduction of this focal self-attention. First things first, on the left hand side we have a classical vision transformer, so that's date model from Facebook AI. And if you haven't watched my video on vision transformer, I do suggest you go ahead and watch it first, but I'll just briefly explain how it works here. So the idea is because you have a transformer and transformers were invented in the context of NLP where you introduce like a sequence of tokens as the input. So we somehow need to convert this image into that sequence of tokens. And so how we do that is basically we split the image into patches. So something like this. So we'll have a patch here, we have a patch here, etc. So the whole image will be basically built from these patches. And here you can see a query patch. So this is one particular patch. And obviously hopefully you're familiar with the self-attention like terminology, but this is a query patch. And what we can see on these attention maps here is basically that the vision transformer has both local attention as well as global attention. So let me like kind of explain what I mean by that. So if you focus on this attention map here, so what it basically says is the following. That means that as you can see this like blob here, that means that this query patch is basically attending these nearby patches a lot. And that's just one particular attention here from the first layer of this of this state transformer. And if we focus on the second head or third head, we can see that it also has so this query patch in those other heads also tends certain other areas. Like if you focus on this one, you can see that that part is these patches here. And it's also focusing pretty much on the whole cat image. So we'll be doing something like this. It's going to attend the whole cat image. And that means that those are some. So because those were learned compared to CNN's where we have like the bias of locality here, it's obvious that vision transformer found it useful to actually also attend those like parts of the image that are further away from the query patch, which tells us something about its usefulness. Right. And so the bad thing about this whole thing with with vision transformer and transformers in general is that they are they have quadratic complexity. So that means that basically you have N square complexity where N is the number of tokens. And here that's the number of patches in the image. And that's expensive. So basically what this novel focal self-attention does is the following. So the idea is let's attend to nearby patches with a fine grain structure. So as you can see here, we'll attend these closer patches with a fine resolution. But once we start getting like further and further away from the query patch, let's kind of start just attending to these course representations. So what they'll end up doing is they'll they'll basically take these four patches here and they'll summarize those into this one big patch. And the same thing here. So here we'll have like I don't know, like maybe four by four patches will be summarized into a single patch. And so that means that we both keep the local as well as the global like connections. And we also reduce the computation, obviously, because we will have less like less amount of tokens to attend to. So that's the main idea of this of this of this paper, basically. And as you can see here, so we're going from fine to coarse structure as we are going from closer closer patches towards more like the patches ever further away from from the query patch. OK, that's that's pretty much it. So now let's start digging into nitty gritty details and how this everything works like. And let's see some results. OK, so just reiterating here in this paper, we represent focal self-attention, a new mechanism that incorporates both fine grained local and coarse grained global interactions in this new mechanism. Each token attends its closest surrounding tokens at fine granularity and the tokens far away at coarse granularity. And thus can capture both short and long range visual dependencies efficiently and effectively. They additionally report that they achieve state of the art results on a couple of different vision benchmarks, such as image classification, object detection and semantic segmentation. And they report some numbers here as well as here. So this is map metric for the object classification, object detection. Sorry. And yeah, I'll get back to these results a bit later and tell you more about what I have to say about those results. OK, let's now start investigating the high level architecture. So if we if we focus here, the main difference you'll notice between this architecture and the vision transformer are these parts here. So these denominators. So as I told you, we first in order to to kind of transform. In order to parse this image by a transformer, we'll need to add these patches. Right. So we have this patch embedding layer. So the difference between vision transformer and this focal transformer is the following. So basically what they'll do in order to create these patches is in this example, what they did is they took four neighboring pixels and that's one patch. So we'll have four by four will be a single patch. And that's why you see four here. So because height and width just represent how many pixels you have, like in height and width dimensions, obviously. Divide by four and you get the number of tokens. And D is the dimension of that token after we embed it using this like linear work. I think they're using convolutional like projection layer. So now the big difference between focal transformer and the like standard vision transformer. So there are two differences. First is they're using this focal self-attention instead of your standard multi-head self-attention. And the second thing is, as you can see, they're reducing the number of these of these like number of patches, number of tokens. So as we go into this, after you apply this first layer and one times, then what they do is they reduce the number of patches here. So basically, instead of having this one patch, what you'll have end up having is you'll have this one. This will be like a single token. OK. And then going to the next layer here will have even bigger like even bigger tokens. OK. So one token will basically span this amount of an image. OK. And they are also, as you can notice here, so D goes to 2D, goes to 4D, goes to 80. And that's a very similar logic that is applied in all convolutional neural networks. So the idea is as you're reducing the spatial extent of your of your like representation, you're you're increasing the number of channels. And basically what that means is we are we are seeing like a huge merge between these transformer and CNN architectures. And it's not even clear what's a transformer, what what transform means and what a CNN means. We also saw those MLP mixers. So, yeah, there is a lot of debate. What's convolutional neural network now and what's a transformer actually? So that's that that's the high level overview of how this works. Everything else remains the same as the vision transformer. So we just have this novel layer and we have this novel logic where we are reducing the number of tokens as we go deeper and deeper. What I show here, this small diagram, let me just tell you briefly about it. Basically, as we are so for a given number of tokens, they show that they show that the receptive field of this novel layer is higher. So it's higher compared to the regular self-attention. And the reason is, as we saw, is that we are attending as we are getting further away from the query token. We're basically attending Corsair and Corsair tokens so we can attend more tokens and have like for the same amount of tokens, we can attend like a huge. We can have a huge receptive field. OK, now let me focus on on the actual meat of this paper. And that's this focal self-attention layer. So let me kind of digest how this thing goes together and how they implement this focal self-attention. So the first thing you'll notice is so imagine this is your input image and these are just patches. We have 20 times 20 patches. So that's 20 times 20 tokens. So the first thing they do is this blue blob you can see here are our query tokens. And what they do is they group them into this window. And every query inside that window will attend to the same group of value and key and keys. And let me just kind of explain what I mean by that. So imagine if we had if we didn't group them in this window fashion, what we would have to do is the following. So this token, this neighborhood would actually be something like maybe, I don't know, like this. And this tokens neighborhood here would be maybe something like this. And that means we'd have like a computational overhead. And so basically because of some computational conveniences, they group these into into a window so that it can share the neighborhood. And now I'll show you what I mean by sharing the neighborhood. So first things first, we have these three levels. So level one, level two, level three. And those correspond to those fine grained over coarse grained structure. So let me just go back here. So level one was these these fine grained like patches. And then we have these more coarse grained. And finally, these are the the coarse grained patches. OK, and let me go back here. So the first thing you'll notice is we're going to pull these tokens into coarser representations, as I said. So that means that here in level two, we're going to take this two by two, these two by two batches, and we're going to convert them into a single one. So that's the reason we have. So one, two, three, four, five, six. That's why we have six by six here after after doing this this this coarsening. And finally here, because we are on a level three, we're going to we're going to convert these four by four patches into a single into a single representation here. So that's why we have five by five. So now you can see that on this level one, what we're doing is we are attending this neighborhood here. OK, so something like this. Let me just draw it. And then on the second level, as you can see here, we're attending this neighborhood. But we have like coarsen representations. OK. And finally, the last layer, let me just draw this. OK, the last layer will attend the whole feature map. And but but the representations will be much coarser. So that's the idea. And out of those, what we do is we kind of flatten those and we concatenate those. And then we apply these like value. We apply a value projection matrices and and key projection matrices to get values and keys. And so now let me show you what I mean by this windowed queries. OK, as you can see here, this same query attends to all of these keys and values as well as this one. So all of the queries will be attending to the same keys and values. And that's the whole point here. So that means that's how they kind of save some computation. So otherwise, as I already explained, if you had if you have them separated, if you had only this one, then let me just delete this for a second. OK, whoops. Let me just delete all of these and let's focus on a single token. OK, so if we had only this token, that means that its neighborhood would actually be something like this. And then we'd had the coarser one, et cetera. Whereas for this one here, we had a totally different neighborhood. And the fact that these have different neighborhoods means that you'd have to extract all of these for all specific tokens. And that's make it more that makes it more computationally intensive to do this. And that's why they kind of bucket them into this this window. OK, that's pretty much it. Then once you have the queries, keys and values, you do your regular self-attention stuff. You basically out of queries and keys, you form those raw scores. Then you apply softmax, you get the attention coefficients and you use those to aggregate value vectors to form novel representations of these tokens here. So, again, these tokens here will have novel representations formed after we applied this layer once. OK, that's pretty much it. Hopefully that was understandable. Let me now jump further and kind of focus on this part here. So they say to perform focal self-attention, we need to first extract the surrounding tokens for each query token in the feature map. So that's the thing I just mentioned. That's the motivation behind a kind of grouping these great tokens into into these windows. And I think the Swin transformer or some of the other like previous work showed that that kind of helps. And like this paper is just copying that part. So note that the strict version of focal self-attention following figure one requires to exclude the overlapping regions across different levels. In our model, we intentionally keep them in order to capture the pyramid information for the overlap regions. So what I say there is the following thing. So let me get back to the image here. As you can see here, looking at this image, it seems like they are they don't have any overlaps. So that basically means that they only apply the fine grain structure here. And then for this course or grain structure, it seems like they're only attending these this region here. So let me just kind of. Shaded. So this part here, OK, but in reality, they are also they are also using these regions as well for the course or scale. And, yeah, they just mentioned that they do that. And I didn't see any ablations on this. But like it seems that this overlapping information on multiple scales help. And they kind of I guess it's also more computationally intensive if they wanted to exclude these regions. So they just left it like this. OK, that's that part. Additionally, they have this this learnable relative position bias that was introduced in this previous previous paper called Swin transformer. And, yeah, this bias term kind of just helps boost the performance additionally. OK, let me just focus on this one now. This is the whole point. They have less computation. So M times N is the so M times N are your like feature feature map dimension. So that means this is basically the number of tokens and and as we can see, so the complexity depends on them. And we have some like pretty much constant here. So L is the number of levels. And that was three in the examples we just saw. And they also have these S sub R L components. So let me just kind of show you what these are. And as you can see here, so that S sub R. So it's eight for this first level. And we can see why it's eight. So basically that that's basically the dimension of this pooled feature map. OK, we have six by six here. That's why we have six here and we have five here. That's why we have five of five here. So those are pretty much constants. And the whole point is that this part is constant once you set up the parameters of your architecture. And then you only depend linearly on the number of tokens, though this constant may be big. So, yeah, that's also worth mentioning. And we'll see soon like the number of flops they have and the time it takes to do an inference through this to this architecture. OK, that's pretty much it. Let me now focus on on on the actual results. Let's see what they got. So they have again they have this focal tiny focal small and focal base. They tested three of these models with. So this one is obviously the biggest one and they have smaller ones. And what they report are some like state of the art results. But as we'll see, it's not that clear cut. So if we focus on this number here, so it's eighty three point eight and it's a bit bigger than than SwinBase. But the thing is they also have more parameters. It's not a significant amount, but it's more. And they also have more flops. And as they show in the appendix as well, it takes more time for the same amount of flops. It takes more time to compute the forward prop through the focal base compared to the SwinBase. So that's something to keep in mind. If they were to kind of make those the same, I'm not sure they'd have even this big of like increasing performance. So I'd say this is more on pair than state of the art, if you ask me. But like, yeah, yeah, if you don't care about these other parameters, then you have a bit better performance. And they have the same results. They show the same results across different tasks. So this is the on the COCO. So this is basically object detection. They show some improvements. Again, I'm not sure the underlying flops and and and like number of parameters as well as the inference speed. If you don't care about those, again, they do achieve state of the art results there as well. Similarly here, so a bunch of different bunch of different benchmarks. I won't be focusing on each of those in particular. So here they show results on semantic segmentation. Again, they're a bit better compared to the baselines. And yeah, it's kind of hard to compare these because they don't have numbers for the number of parameters and flops there. But yeah, yeah, I don't want to focus too much on the results. Let me show you some ablations here. In the final model, they were actually testing. They used two levels, not three levels, as we saw in those previous diagrams. And what they tested is increasing this window size from seven to 14. And that kind of bumped up the the performance, as you can see here, but also the flops go up. So it's obviously a trade off again. And what is show here is the importance of using both those local and global like interactions. So first things first, what they show is this window attention, which basically means the following thing. So window means that you're if I go back to the diagram up there. So give me a second. So, OK, this part. So that means that these queries are actually these queries are only attending to the keys and values that are inside of this region. OK, let me just change the color. So they're only attending this part. They're not attending the other keys and values. And that basically means we don't have a nice communication across different parts of the image. And then obviously the performance drops by a lot. Let me get back here. OK. And once they add this like local part and then global part, you can see it increases and local plus global. So both using the fine grained structures. So that's the local part and the global one. So that's where you're using those windows of size seven or 14, whatever. So when you combine both of those, you get the like computational efficiency. Plus you get to kind of ingest information from the whole image. That's something that we know is useful because that's why transformers are performing better than CNN's. OK, what they additionally did here is this switch transformer had something like something called like Windows shift. And it actually hurts the performance of Swin transformer a lot if they remove that shifting. But for the for this focal transformer, it's actually not necessary. So they're just going to remove that. Here are some some depth ablations. Nothing super, super interesting. They again show they are better than Swin. Even even even when they reduced the number of as you can see here, even when they reduce depth of the focal one, they are still even better than the swing that has like like deeper layer in this particular stage of the architecture. OK. And here again, they are their own pair. So, yeah, those are just some additional claims that they are still the art on these different tasks. OK, that's pretty much it. Let me just do two more things here. OK, so what I say here is in our observation is that adding long range tokens can bring more relative improvement for image classification than object detection and vice versa for local tokens. We suspect that dense predictions like object detection more rely on fine grained local context while image classification favors more the global information. So this is an interesting statement and it kind of makes sense because if you want to understand what's in the image, if you want to classify the object in the image, you want to kind of understand what's like the whole the whole context. Whereas when you're doing object detection, it's kind of enough to kind of understand the fine grained details. But like there is also this is a double edged sword because this could also mean that the model learns the neural networks learn how to exploit those spurious signals that we we are well aware of that. Sometimes neural networks kind of exploit those spurious signals like texture in order to to infer what the object is inside of the image. And we don't want that. We want our neural networks to understand the actual content of the image itself and not those spurious signals. Second thing I want to I want to mention here is they say here, though extensive experimental results show that our focal self attention can significantly boost the performance on both image classification and dense prediction tasks. It does introduce extra computational and memory cost since each query token needs to attend to the coarsened global tokens in addition to the local tokens. So that's what something I already stressed a couple of times. I don't think this model significantly boosts upon the prior state of the art. And again, they did acknowledge it here. You have to trade off computation and so flops and also memory footprint and speed. If you want to get this small and not significant boost in performance, if you ask me and going back to the appendix here, they mentioned somewhere here. So accordingly, our focal transformer has slower running speed, though it has similar flops as wind transformers. So that's very important. This is mainly due to two reasons. We introduced the global coarse grain attention and introduces the extra computations. That's one. And two, though we conduct our focal attention on the window windows, we still observe that extracting the surrounding tokens around local windows and the global tokens across the feature map are time consuming. So just keep this in mind. If you want to actually run this in production one day, those extra parameter, those extra dimensions like computation and speed and memory do matter a lot. So that's pretty much it. Hopefully you like this paper. If you did, share it out, subscribe. And until next time, bye bye.
[{"start": 0.0, "end": 10.0, "text": " What's up? In this video I'm covering focal self-attention for local global interactions in vision transformers by the Microsoft Research and Microsoft Cloud and AI teams."}, {"start": 10.0, "end": 17.0, "text": " So they basically introduced this novel layer called focal self-attention and introduced this novel model called focal transformer."}, {"start": 17.0, "end": 19.0, "text": " And let's see what it's all about."}, {"start": 19.0, "end": 25.0, "text": " Basically if you focus on this diagram here, let me kind of motivate the introduction of this focal self-attention."}, {"start": 25.0, "end": 31.0, "text": " First things first, on the left hand side we have a classical vision transformer, so that's date model from Facebook AI."}, {"start": 31.0, "end": 39.0, "text": " And if you haven't watched my video on vision transformer, I do suggest you go ahead and watch it first, but I'll just briefly explain how it works here."}, {"start": 39.0, "end": 47.0, "text": " So the idea is because you have a transformer and transformers were invented in the context of NLP where you introduce like a sequence of tokens as the input."}, {"start": 47.0, "end": 51.0, "text": " So we somehow need to convert this image into that sequence of tokens."}, {"start": 51.0, "end": 55.0, "text": " And so how we do that is basically we split the image into patches."}, {"start": 55.0, "end": 60.0, "text": " So something like this. So we'll have a patch here, we have a patch here, etc."}, {"start": 60.0, "end": 64.0, "text": " So the whole image will be basically built from these patches."}, {"start": 64.0, "end": 68.0, "text": " And here you can see a query patch. So this is one particular patch."}, {"start": 68.0, "end": 74.0, "text": " And obviously hopefully you're familiar with the self-attention like terminology, but this is a query patch."}, {"start": 74.0, "end": 85.0, "text": " And what we can see on these attention maps here is basically that the vision transformer has both local attention as well as global attention."}, {"start": 85.0, "end": 88.0, "text": " So let me like kind of explain what I mean by that."}, {"start": 88.0, "end": 93.0, "text": " So if you focus on this attention map here, so what it basically says is the following."}, {"start": 93.0, "end": 103.0, "text": " That means that as you can see this like blob here, that means that this query patch is basically attending these nearby patches a lot."}, {"start": 103.0, "end": 109.0, "text": " And that's just one particular attention here from the first layer of this of this state transformer."}, {"start": 109.0, "end": 117.0, "text": " And if we focus on the second head or third head, we can see that it also has so this query patch in those other heads also tends certain other areas."}, {"start": 117.0, "end": 122.0, "text": " Like if you focus on this one, you can see that that part is these patches here."}, {"start": 122.0, "end": 126.0, "text": " And it's also focusing pretty much on the whole cat image. So we'll be doing something like this."}, {"start": 126.0, "end": 128.0, "text": " It's going to attend the whole cat image."}, {"start": 128.0, "end": 138.0, "text": " And that means that those are some. So because those were learned compared to CNN's where we have like the bias of locality here,"}, {"start": 138.0, "end": 147.0, "text": " it's obvious that vision transformer found it useful to actually also attend those like parts of the image that are further away from the query patch,"}, {"start": 147.0, "end": 150.0, "text": " which tells us something about its usefulness. Right."}, {"start": 150.0, "end": 158.0, "text": " And so the bad thing about this whole thing with with vision transformer and transformers in general is that they are they have quadratic complexity."}, {"start": 158.0, "end": 163.0, "text": " So that means that basically you have N square complexity where N is the number of tokens."}, {"start": 163.0, "end": 167.0, "text": " And here that's the number of patches in the image. And that's expensive."}, {"start": 167.0, "end": 172.0, "text": " So basically what this novel focal self-attention does is the following."}, {"start": 172.0, "end": 178.0, "text": " So the idea is let's attend to nearby patches with a fine grain structure."}, {"start": 178.0, "end": 182.0, "text": " So as you can see here, we'll attend these closer patches with a fine resolution."}, {"start": 182.0, "end": 190.0, "text": " But once we start getting like further and further away from the query patch, let's kind of start just attending to these course representations."}, {"start": 190.0, "end": 198.0, "text": " So what they'll end up doing is they'll they'll basically take these four patches here and they'll summarize those into this one big patch."}, {"start": 198.0, "end": 205.0, "text": " And the same thing here. So here we'll have like I don't know, like maybe four by four patches will be summarized into a single patch."}, {"start": 205.0, "end": 211.0, "text": " And so that means that we both keep the local as well as the global like connections."}, {"start": 211.0, "end": 218.0, "text": " And we also reduce the computation, obviously, because we will have less like less amount of tokens to attend to."}, {"start": 218.0, "end": 222.0, "text": " So that's the main idea of this of this of this paper, basically."}, {"start": 222.0, "end": 234.0, "text": " And as you can see here, so we're going from fine to coarse structure as we are going from closer closer patches towards more like the patches ever further away from from the query patch."}, {"start": 234.0, "end": 236.0, "text": " OK, that's that's pretty much it."}, {"start": 236.0, "end": 240.0, "text": " So now let's start digging into nitty gritty details and how this everything works like."}, {"start": 240.0, "end": 247.0, "text": " And let's see some results. OK, so just reiterating here in this paper, we represent focal self-attention,"}, {"start": 247.0, "end": 254.0, "text": " a new mechanism that incorporates both fine grained local and coarse grained global interactions in this new mechanism."}, {"start": 254.0, "end": 261.0, "text": " Each token attends its closest surrounding tokens at fine granularity and the tokens far away at coarse granularity."}, {"start": 261.0, "end": 269.0, "text": " And thus can capture both short and long range visual dependencies efficiently and effectively."}, {"start": 269.0, "end": 274.0, "text": " They additionally report that they achieve state of the art results on a couple of different vision benchmarks,"}, {"start": 274.0, "end": 278.0, "text": " such as image classification, object detection and semantic segmentation."}, {"start": 278.0, "end": 281.0, "text": " And they report some numbers here as well as here."}, {"start": 281.0, "end": 285.0, "text": " So this is map metric for the object classification, object detection."}, {"start": 285.0, "end": 292.0, "text": " Sorry. And yeah, I'll get back to these results a bit later and tell you more about what I have to say about those results."}, {"start": 292.0, "end": 298.0, "text": " OK, let's now start investigating the high level architecture."}, {"start": 298.0, "end": 307.0, "text": " So if we if we focus here, the main difference you'll notice between this architecture and the vision transformer are these parts here."}, {"start": 307.0, "end": 313.0, "text": " So these denominators. So as I told you, we first in order to to kind of transform."}, {"start": 313.0, "end": 318.0, "text": " In order to parse this image by a transformer, we'll need to add these patches. Right."}, {"start": 318.0, "end": 325.0, "text": " So we have this patch embedding layer. So the difference between vision transformer and this focal transformer is the following."}, {"start": 325.0, "end": 333.0, "text": " So basically what they'll do in order to create these patches is in this example, what they did is they took four neighboring pixels and that's one patch."}, {"start": 333.0, "end": 337.0, "text": " So we'll have four by four will be a single patch. And that's why you see four here."}, {"start": 337.0, "end": 342.0, "text": " So because height and width just represent how many pixels you have, like in height and width dimensions, obviously."}, {"start": 342.0, "end": 351.0, "text": " Divide by four and you get the number of tokens. And D is the dimension of that token after we embed it using this like linear work."}, {"start": 351.0, "end": 354.0, "text": " I think they're using convolutional like projection layer."}, {"start": 354.0, "end": 360.0, "text": " So now the big difference between focal transformer and the like standard vision transformer."}, {"start": 360.0, "end": 366.0, "text": " So there are two differences. First is they're using this focal self-attention instead of your standard multi-head self-attention."}, {"start": 366.0, "end": 374.0, "text": " And the second thing is, as you can see, they're reducing the number of these of these like number of patches, number of tokens."}, {"start": 374.0, "end": 383.0, "text": " So as we go into this, after you apply this first layer and one times, then what they do is they reduce the number of patches here."}, {"start": 383.0, "end": 389.0, "text": " So basically, instead of having this one patch, what you'll have end up having is you'll have this one."}, {"start": 389.0, "end": 398.0, "text": " This will be like a single token. OK. And then going to the next layer here will have even bigger like even bigger tokens."}, {"start": 398.0, "end": 403.0, "text": " OK. So one token will basically span this amount of an image. OK."}, {"start": 403.0, "end": 409.0, "text": " And they are also, as you can notice here, so D goes to 2D, goes to 4D, goes to 80."}, {"start": 409.0, "end": 415.0, "text": " And that's a very similar logic that is applied in all convolutional neural networks."}, {"start": 415.0, "end": 423.0, "text": " So the idea is as you're reducing the spatial extent of your of your like representation, you're you're increasing the number of channels."}, {"start": 423.0, "end": 431.0, "text": " And basically what that means is we are we are seeing like a huge merge between these transformer and CNN architectures."}, {"start": 431.0, "end": 436.0, "text": " And it's not even clear what's a transformer, what what transform means and what a CNN means."}, {"start": 436.0, "end": 440.0, "text": " We also saw those MLP mixers. So, yeah, there is a lot of debate."}, {"start": 440.0, "end": 444.0, "text": " What's convolutional neural network now and what's a transformer actually?"}, {"start": 444.0, "end": 447.0, "text": " So that's that that's the high level overview of how this works."}, {"start": 447.0, "end": 449.0, "text": " Everything else remains the same as the vision transformer."}, {"start": 449.0, "end": 458.0, "text": " So we just have this novel layer and we have this novel logic where we are reducing the number of tokens as we go deeper and deeper."}, {"start": 458.0, "end": 462.0, "text": " What I show here, this small diagram, let me just tell you briefly about it."}, {"start": 462.0, "end": 471.0, "text": " Basically, as we are so for a given number of tokens, they show that they show that the receptive field of this novel layer is higher."}, {"start": 471.0, "end": 475.0, "text": " So it's higher compared to the regular self-attention."}, {"start": 475.0, "end": 481.0, "text": " And the reason is, as we saw, is that we are attending as we are getting further away from the query token."}, {"start": 481.0, "end": 489.0, "text": " We're basically attending Corsair and Corsair tokens so we can attend more tokens and have like for the same amount of tokens, we can attend like a huge."}, {"start": 489.0, "end": 492.0, "text": " We can have a huge receptive field."}, {"start": 492.0, "end": 497.0, "text": " OK, now let me focus on on the actual meat of this paper."}, {"start": 497.0, "end": 499.0, "text": " And that's this focal self-attention layer."}, {"start": 499.0, "end": 505.0, "text": " So let me kind of digest how this thing goes together and how they implement this focal self-attention."}, {"start": 505.0, "end": 511.0, "text": " So the first thing you'll notice is so imagine this is your input image and these are just patches."}, {"start": 511.0, "end": 513.0, "text": " We have 20 times 20 patches."}, {"start": 513.0, "end": 515.0, "text": " So that's 20 times 20 tokens."}, {"start": 515.0, "end": 520.0, "text": " So the first thing they do is this blue blob you can see here are our query tokens."}, {"start": 520.0, "end": 523.0, "text": " And what they do is they group them into this window."}, {"start": 523.0, "end": 530.0, "text": " And every query inside that window will attend to the same group of value and key and keys."}, {"start": 530.0, "end": 533.0, "text": " And let me just kind of explain what I mean by that."}, {"start": 533.0, "end": 539.0, "text": " So imagine if we had if we didn't group them in this window fashion, what we would have to do is the following."}, {"start": 539.0, "end": 544.0, "text": " So this token, this neighborhood would actually be something like maybe, I don't know, like this."}, {"start": 544.0, "end": 549.0, "text": " And this tokens neighborhood here would be maybe something like this."}, {"start": 549.0, "end": 553.0, "text": " And that means we'd have like a computational overhead."}, {"start": 553.0, "end": 560.0, "text": " And so basically because of some computational conveniences, they group these into into a window so that it can share the neighborhood."}, {"start": 560.0, "end": 562.0, "text": " And now I'll show you what I mean by sharing the neighborhood."}, {"start": 562.0, "end": 564.0, "text": " So first things first, we have these three levels."}, {"start": 564.0, "end": 566.0, "text": " So level one, level two, level three."}, {"start": 566.0, "end": 570.0, "text": " And those correspond to those fine grained over coarse grained structure."}, {"start": 570.0, "end": 572.0, "text": " So let me just go back here."}, {"start": 572.0, "end": 577.0, "text": " So level one was these these fine grained like patches."}, {"start": 577.0, "end": 579.0, "text": " And then we have these more coarse grained."}, {"start": 579.0, "end": 583.0, "text": " And finally, these are the the coarse grained patches."}, {"start": 583.0, "end": 586.0, "text": " OK, and let me go back here."}, {"start": 586.0, "end": 592.0, "text": " So the first thing you'll notice is we're going to pull these tokens into coarser representations, as I said."}, {"start": 592.0, "end": 601.0, "text": " So that means that here in level two, we're going to take this two by two, these two by two batches, and we're going to convert them into a single one."}, {"start": 601.0, "end": 602.0, "text": " So that's the reason we have."}, {"start": 602.0, "end": 605.0, "text": " So one, two, three, four, five, six."}, {"start": 605.0, "end": 609.0, "text": " That's why we have six by six here after after doing this this this coarsening."}, {"start": 609.0, "end": 617.0, "text": " And finally here, because we are on a level three, we're going to we're going to convert these four by four patches into a single into a single representation here."}, {"start": 617.0, "end": 619.0, "text": " So that's why we have five by five."}, {"start": 619.0, "end": 623.0, "text": " So now you can see that on this level one, what we're doing is we are attending this neighborhood here."}, {"start": 623.0, "end": 627.0, "text": " OK, so something like this."}, {"start": 627.0, "end": 629.0, "text": " Let me just draw it."}, {"start": 629.0, "end": 634.0, "text": " And then on the second level, as you can see here, we're attending this neighborhood."}, {"start": 634.0, "end": 637.0, "text": " But we have like coarsen representations."}, {"start": 637.0, "end": 638.0, "text": " OK."}, {"start": 638.0, "end": 642.0, "text": " And finally, the last layer, let me just draw this."}, {"start": 642.0, "end": 645.0, "text": " OK, the last layer will attend the whole feature map."}, {"start": 645.0, "end": 647.0, "text": " And but but the representations will be much coarser."}, {"start": 647.0, "end": 648.0, "text": " So that's the idea."}, {"start": 648.0, "end": 653.0, "text": " And out of those, what we do is we kind of flatten those and we concatenate those."}, {"start": 653.0, "end": 656.0, "text": " And then we apply these like value."}, {"start": 656.0, "end": 661.0, "text": " We apply a value projection matrices and and key projection matrices to get values and keys."}, {"start": 661.0, "end": 666.0, "text": " And so now let me show you what I mean by this windowed queries."}, {"start": 666.0, "end": 673.0, "text": " OK, as you can see here, this same query attends to all of these keys and values as well as this one."}, {"start": 673.0, "end": 677.0, "text": " So all of the queries will be attending to the same keys and values."}, {"start": 677.0, "end": 678.0, "text": " And that's the whole point here."}, {"start": 678.0, "end": 682.0, "text": " So that means that's how they kind of save some computation."}, {"start": 682.0, "end": 690.0, "text": " So otherwise, as I already explained, if you had if you have them separated, if you had only this one, then let me just delete this for a second."}, {"start": 690.0, "end": 695.0, "text": " OK, whoops. Let me just delete all of these and let's focus on a single token."}, {"start": 695.0, "end": 701.0, "text": " OK, so if we had only this token, that means that its neighborhood would actually be something like this."}, {"start": 701.0, "end": 704.0, "text": " And then we'd had the coarser one, et cetera."}, {"start": 704.0, "end": 709.0, "text": " Whereas for this one here, we had a totally different neighborhood."}, {"start": 709.0, "end": 716.0, "text": " And the fact that these have different neighborhoods means that you'd have to extract all of these for all specific tokens."}, {"start": 716.0, "end": 719.0, "text": " And that's make it more that makes it more computationally intensive to do this."}, {"start": 719.0, "end": 722.0, "text": " And that's why they kind of bucket them into this this window."}, {"start": 722.0, "end": 724.0, "text": " OK, that's pretty much it."}, {"start": 724.0, "end": 729.0, "text": " Then once you have the queries, keys and values, you do your regular self-attention stuff."}, {"start": 729.0, "end": 733.0, "text": " You basically out of queries and keys, you form those raw scores."}, {"start": 733.0, "end": 742.0, "text": " Then you apply softmax, you get the attention coefficients and you use those to aggregate value vectors to form novel representations of these tokens here."}, {"start": 742.0, "end": 749.0, "text": " So, again, these tokens here will have novel representations formed after we applied this layer once."}, {"start": 749.0, "end": 751.0, "text": " OK, that's pretty much it."}, {"start": 751.0, "end": 753.0, "text": " Hopefully that was understandable."}, {"start": 753.0, "end": 757.0, "text": " Let me now jump further and kind of focus on this part here."}, {"start": 757.0, "end": 764.0, "text": " So they say to perform focal self-attention, we need to first extract the surrounding tokens for each query token in the feature map."}, {"start": 764.0, "end": 765.0, "text": " So that's the thing I just mentioned."}, {"start": 765.0, "end": 770.0, "text": " That's the motivation behind a kind of grouping these great tokens into into these windows."}, {"start": 770.0, "end": 776.0, "text": " And I think the Swin transformer or some of the other like previous work showed that that kind of helps."}, {"start": 776.0, "end": 779.0, "text": " And like this paper is just copying that part."}, {"start": 779.0, "end": 787.0, "text": " So note that the strict version of focal self-attention following figure one requires to exclude the overlapping regions across different levels."}, {"start": 787.0, "end": 793.0, "text": " In our model, we intentionally keep them in order to capture the pyramid information for the overlap regions."}, {"start": 793.0, "end": 795.0, "text": " So what I say there is the following thing."}, {"start": 795.0, "end": 798.0, "text": " So let me get back to the image here."}, {"start": 798.0, "end": 803.0, "text": " As you can see here, looking at this image, it seems like they are they don't have any overlaps."}, {"start": 803.0, "end": 808.0, "text": " So that basically means that they only apply the fine grain structure here."}, {"start": 808.0, "end": 815.0, "text": " And then for this course or grain structure, it seems like they're only attending these this region here."}, {"start": 815.0, "end": 818.0, "text": " So let me just kind of. Shaded."}, {"start": 818.0, "end": 826.0, "text": " So this part here, OK, but in reality, they are also they are also using these regions as well for the course or scale."}, {"start": 826.0, "end": 830.0, "text": " And, yeah, they just mentioned that they do that."}, {"start": 830.0, "end": 832.0, "text": " And I didn't see any ablations on this."}, {"start": 832.0, "end": 837.0, "text": " But like it seems that this overlapping information on multiple scales help."}, {"start": 837.0, "end": 843.0, "text": " And they kind of I guess it's also more computationally intensive if they wanted to exclude these regions."}, {"start": 843.0, "end": 846.0, "text": " So they just left it like this. OK, that's that part."}, {"start": 846.0, "end": 854.0, "text": " Additionally, they have this this learnable relative position bias that was introduced in this previous previous paper called Swin transformer."}, {"start": 854.0, "end": 858.0, "text": " And, yeah, this bias term kind of just helps boost the performance additionally."}, {"start": 858.0, "end": 860.0, "text": " OK, let me just focus on this one now."}, {"start": 860.0, "end": 864.0, "text": " This is the whole point. They have less computation."}, {"start": 864.0, "end": 869.0, "text": " So M times N is the so M times N are your like feature feature map dimension."}, {"start": 869.0, "end": 876.0, "text": " So that means this is basically the number of tokens and and as we can see, so the complexity depends on them."}, {"start": 876.0, "end": 878.0, "text": " And we have some like pretty much constant here."}, {"start": 878.0, "end": 880.0, "text": " So L is the number of levels."}, {"start": 880.0, "end": 882.0, "text": " And that was three in the examples we just saw."}, {"start": 882.0, "end": 887.0, "text": " And they also have these S sub R L components."}, {"start": 887.0, "end": 889.0, "text": " So let me just kind of show you what these are."}, {"start": 889.0, "end": 894.0, "text": " And as you can see here, so that S sub R."}, {"start": 894.0, "end": 896.0, "text": " So it's eight for this first level."}, {"start": 896.0, "end": 897.0, "text": " And we can see why it's eight."}, {"start": 897.0, "end": 901.0, "text": " So basically that that's basically the dimension of this pooled feature map."}, {"start": 901.0, "end": 903.0, "text": " OK, we have six by six here."}, {"start": 903.0, "end": 905.0, "text": " That's why we have six here and we have five here."}, {"start": 905.0, "end": 906.0, "text": " That's why we have five of five here."}, {"start": 906.0, "end": 908.0, "text": " So those are pretty much constants."}, {"start": 908.0, "end": 914.0, "text": " And the whole point is that this part is constant once you set up the parameters of your architecture."}, {"start": 914.0, "end": 919.0, "text": " And then you only depend linearly on the number of tokens, though this constant may be big."}, {"start": 919.0, "end": 922.0, "text": " So, yeah, that's also worth mentioning."}, {"start": 922.0, "end": 930.0, "text": " And we'll see soon like the number of flops they have and the time it takes to do an inference through this to this architecture."}, {"start": 930.0, "end": 932.0, "text": " OK, that's pretty much it."}, {"start": 932.0, "end": 934.0, "text": " Let me now focus on on on the actual results."}, {"start": 934.0, "end": 935.0, "text": " Let's see what they got."}, {"start": 935.0, "end": 940.0, "text": " So they have again they have this focal tiny focal small and focal base."}, {"start": 940.0, "end": 942.0, "text": " They tested three of these models with."}, {"start": 942.0, "end": 945.0, "text": " So this one is obviously the biggest one and they have smaller ones."}, {"start": 945.0, "end": 950.0, "text": " And what they report are some like state of the art results."}, {"start": 950.0, "end": 952.0, "text": " But as we'll see, it's not that clear cut."}, {"start": 952.0, "end": 958.0, "text": " So if we focus on this number here, so it's eighty three point eight and it's a bit bigger than than SwinBase."}, {"start": 958.0, "end": 961.0, "text": " But the thing is they also have more parameters."}, {"start": 961.0, "end": 963.0, "text": " It's not a significant amount, but it's more."}, {"start": 963.0, "end": 965.0, "text": " And they also have more flops."}, {"start": 965.0, "end": 971.0, "text": " And as they show in the appendix as well, it takes more time for the same amount of flops."}, {"start": 971.0, "end": 977.0, "text": " It takes more time to compute the forward prop through the focal base compared to the SwinBase."}, {"start": 977.0, "end": 979.0, "text": " So that's something to keep in mind."}, {"start": 979.0, "end": 987.0, "text": " If they were to kind of make those the same, I'm not sure they'd have even this big of like increasing performance."}, {"start": 987.0, "end": 991.0, "text": " So I'd say this is more on pair than state of the art, if you ask me."}, {"start": 991.0, "end": 998.0, "text": " But like, yeah, yeah, if you don't care about these other parameters, then you have a bit better performance."}, {"start": 998.0, "end": 1001.0, "text": " And they have the same results."}, {"start": 1001.0, "end": 1003.0, "text": " They show the same results across different tasks."}, {"start": 1003.0, "end": 1005.0, "text": " So this is the on the COCO."}, {"start": 1005.0, "end": 1007.0, "text": " So this is basically object detection."}, {"start": 1007.0, "end": 1009.0, "text": " They show some improvements."}, {"start": 1009.0, "end": 1015.0, "text": " Again, I'm not sure the underlying flops and and and like number of parameters as well as the inference speed."}, {"start": 1015.0, "end": 1020.0, "text": " If you don't care about those, again, they do achieve state of the art results there as well."}, {"start": 1020.0, "end": 1025.0, "text": " Similarly here, so a bunch of different bunch of different benchmarks."}, {"start": 1025.0, "end": 1028.0, "text": " I won't be focusing on each of those in particular."}, {"start": 1028.0, "end": 1033.0, "text": " So here they show results on semantic segmentation."}, {"start": 1033.0, "end": 1037.0, "text": " Again, they're a bit better compared to the baselines."}, {"start": 1037.0, "end": 1044.0, "text": " And yeah, it's kind of hard to compare these because they don't have numbers for the number of parameters and flops there."}, {"start": 1044.0, "end": 1047.0, "text": " But yeah, yeah, I don't want to focus too much on the results."}, {"start": 1047.0, "end": 1049.0, "text": " Let me show you some ablations here."}, {"start": 1049.0, "end": 1051.0, "text": " In the final model, they were actually testing."}, {"start": 1051.0, "end": 1055.0, "text": " They used two levels, not three levels, as we saw in those previous diagrams."}, {"start": 1055.0, "end": 1060.0, "text": " And what they tested is increasing this window size from seven to 14."}, {"start": 1060.0, "end": 1067.0, "text": " And that kind of bumped up the the performance, as you can see here, but also the flops go up."}, {"start": 1067.0, "end": 1069.0, "text": " So it's obviously a trade off again."}, {"start": 1069.0, "end": 1076.0, "text": " And what is show here is the importance of using both those local and global like interactions."}, {"start": 1076.0, "end": 1081.0, "text": " So first things first, what they show is this window attention, which basically means the following thing."}, {"start": 1081.0, "end": 1086.0, "text": " So window means that you're if I go back to the diagram up there."}, {"start": 1086.0, "end": 1089.0, "text": " So give me a second. So, OK, this part."}, {"start": 1089.0, "end": 1097.0, "text": " So that means that these queries are actually these queries are only attending to the keys and values that are inside of this region."}, {"start": 1097.0, "end": 1100.0, "text": " OK, let me just change the color. So they're only attending this part."}, {"start": 1100.0, "end": 1103.0, "text": " They're not attending the other keys and values."}, {"start": 1103.0, "end": 1108.0, "text": " And that basically means we don't have a nice communication across different parts of the image."}, {"start": 1108.0, "end": 1111.0, "text": " And then obviously the performance drops by a lot."}, {"start": 1111.0, "end": 1113.0, "text": " Let me get back here. OK."}, {"start": 1113.0, "end": 1120.0, "text": " And once they add this like local part and then global part, you can see it increases and local plus global."}, {"start": 1120.0, "end": 1122.0, "text": " So both using the fine grained structures."}, {"start": 1122.0, "end": 1124.0, "text": " So that's the local part and the global one."}, {"start": 1124.0, "end": 1129.0, "text": " So that's where you're using those windows of size seven or 14, whatever."}, {"start": 1129.0, "end": 1133.0, "text": " So when you combine both of those, you get the like computational efficiency."}, {"start": 1133.0, "end": 1137.0, "text": " Plus you get to kind of ingest information from the whole image."}, {"start": 1137.0, "end": 1142.0, "text": " That's something that we know is useful because that's why transformers are performing better than CNN's."}, {"start": 1142.0, "end": 1148.0, "text": " OK, what they additionally did here is this switch transformer had something like something called like Windows shift."}, {"start": 1148.0, "end": 1154.0, "text": " And it actually hurts the performance of Swin transformer a lot if they remove that shifting."}, {"start": 1154.0, "end": 1158.0, "text": " But for the for this focal transformer, it's actually not necessary."}, {"start": 1158.0, "end": 1160.0, "text": " So they're just going to remove that."}, {"start": 1160.0, "end": 1162.0, "text": " Here are some some depth ablations."}, {"start": 1162.0, "end": 1164.0, "text": " Nothing super, super interesting."}, {"start": 1164.0, "end": 1166.0, "text": " They again show they are better than Swin."}, {"start": 1166.0, "end": 1173.0, "text": " Even even even when they reduced the number of as you can see here, even when they reduce depth of the focal one,"}, {"start": 1173.0, "end": 1181.0, "text": " they are still even better than the swing that has like like deeper layer in this particular stage of the architecture."}, {"start": 1181.0, "end": 1184.0, "text": " OK. And here again, they are their own pair."}, {"start": 1184.0, "end": 1191.0, "text": " So, yeah, those are just some additional claims that they are still the art on these different tasks."}, {"start": 1191.0, "end": 1194.0, "text": " OK, that's pretty much it."}, {"start": 1194.0, "end": 1196.0, "text": " Let me just do two more things here."}, {"start": 1196.0, "end": 1207.0, "text": " OK, so what I say here is in our observation is that adding long range tokens can bring more relative improvement for image classification than object detection and vice versa for local tokens."}, {"start": 1207.0, "end": 1216.0, "text": " We suspect that dense predictions like object detection more rely on fine grained local context while image classification favors more the global information."}, {"start": 1216.0, "end": 1222.0, "text": " So this is an interesting statement and it kind of makes sense because if you want to understand what's in the image, if you want to classify the object in the image,"}, {"start": 1222.0, "end": 1226.0, "text": " you want to kind of understand what's like the whole the whole context."}, {"start": 1226.0, "end": 1232.0, "text": " Whereas when you're doing object detection, it's kind of enough to kind of understand the fine grained details."}, {"start": 1232.0, "end": 1243.0, "text": " But like there is also this is a double edged sword because this could also mean that the model learns the neural networks learn how to exploit those spurious signals that we we are well aware of that."}, {"start": 1243.0, "end": 1251.0, "text": " Sometimes neural networks kind of exploit those spurious signals like texture in order to to infer what the object is inside of the image."}, {"start": 1251.0, "end": 1258.0, "text": " And we don't want that. We want our neural networks to understand the actual content of the image itself and not those spurious signals."}, {"start": 1258.0, "end": 1271.0, "text": " Second thing I want to I want to mention here is they say here, though extensive experimental results show that our focal self attention can significantly boost the performance on both image classification and dense prediction tasks."}, {"start": 1271.0, "end": 1280.0, "text": " It does introduce extra computational and memory cost since each query token needs to attend to the coarsened global tokens in addition to the local tokens."}, {"start": 1280.0, "end": 1283.0, "text": " So that's what something I already stressed a couple of times."}, {"start": 1283.0, "end": 1288.0, "text": " I don't think this model significantly boosts upon the prior state of the art."}, {"start": 1288.0, "end": 1295.0, "text": " And again, they did acknowledge it here. You have to trade off computation and so flops and also memory footprint and speed."}, {"start": 1295.0, "end": 1304.0, "text": " If you want to get this small and not significant boost in performance, if you ask me and going back to the appendix here, they mentioned somewhere here."}, {"start": 1304.0, "end": 1310.0, "text": " So accordingly, our focal transformer has slower running speed, though it has similar flops as wind transformers."}, {"start": 1310.0, "end": 1313.0, "text": " So that's very important. This is mainly due to two reasons."}, {"start": 1313.0, "end": 1318.0, "text": " We introduced the global coarse grain attention and introduces the extra computations."}, {"start": 1318.0, "end": 1330.0, "text": " That's one. And two, though we conduct our focal attention on the window windows, we still observe that extracting the surrounding tokens around local windows and the global tokens across the feature map are time consuming."}, {"start": 1330.0, "end": 1341.0, "text": " So just keep this in mind. If you want to actually run this in production one day, those extra parameter, those extra dimensions like computation and speed and memory do matter a lot."}, {"start": 1341.0, "end": 1360.0, "text": " So that's pretty much it. Hopefully you like this paper. If you did, share it out, subscribe. And until next time, bye bye."}]