DeepMind Perceiver and Perceiver IO | Paper Explained
👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I cover:
* Perceiver (Perceiver: General Perception with Iterative Attention) 
* Perceiver IO (Perceiver IO: A General Architecture for Structured Inputs & Outputs)

The goal was to create a modality-agnostic, general perception architecture that could work on images, videos, audio, text, etc. alike.

The main idea is to use the cross-attention module as a bottleneck layer that will map the input modality data into the latent space - this way we avoid the quadratic curse of transformers. After that powerful latent transformers are used to refine the representation - rinse and repeat.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Perceiver: https://arxiv.org/abs/2103.03206
✅ Perceiver IO: https://arxiv.org/abs/2107.14795
✅ Code: https://github.com/deepmind/deepmind-research/tree/master/perceiver
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
02:00 Perceiver architecture explained
05:40 Comparison with Facebook DETR model
07:05 Comparison to RNNs
08:35 Algorithmic complexity of Perceiver
10:35 Positional encodings and permutation equivariance
12:00 Results - ImageNet
14:35 Pixel permutation robustness
17:40 Attention visualized
20:20 Results - AudioSet
23:30 Results - Point Cloud
25:00 Perceiver IO
26:15 Decoder explained in depth (main contribution)
28:45 GLUE results (BERT baseline)
29:50 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Bartłomiej Danek
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR ML PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#perceiver #perceiverio #deepmind
 What's cracking guys in this video I'm covering Perceiver, general perception with iterative attention and at the end of the video I'm gonna contrast it with the new newly published Perceiver IO paper. It's a work by DeepMind team Andrew Jegel, Felikski Manow, Andrew Bruch, Andrew Tissaman, Oriol Vinales and Joao Caheira. So the idea is to generate this general perception architecture and make it modality agnostic and I'm gonna break that down in a little bit but let's see the motivation behind the paper. So the perception models used in deep learning are designed for individual modalities often relying on domain specific assumptions such as the local grid structures and I guess they're referring to CNN here but basically depending on which modality you're using in your deep learning pipeline if you have like an image or if you have an audio or if you have like a point cloud you're going to use heavily specialized architectures to kind of process and understand all of those modalities. The whole idea of this paper is to kind of ditch all of those specific priors which are modality dependent like for example when you're using a CNN you're basically exploiting the fact that neighboring pixels their intensities are correlated and so you can use a kernel that's like three by three and you can kind of shift it and share the weights whereby you're also exploiting the fact that the statistics will not change across the image and so they say it here nicely in this paper we introduce the perceiver a model that builds upon transformers and hence makes few architectural assumptions about the relationship between its inputs but it also scales to hundreds of thousands of inputs like comnets so on the one hand they're reusing transformers which are like proven to be very good at modeling like arbitrary relationships between the inputs and on the other hand they kind of avoided this quadratic complexity problem that transformers have and we're going to see how they did it and that's what enables them to apply this model to hundreds of thousands of inputs so let's dig into the architecture straight away here it is it basically as you can see here we have this latent array which is n times d dimensions we have this byte array which is basically our input modality so this thing here is going to be either your image or your audio or your video or whatnot so whatever you have so let me kind of explain that using this image below here you can see various modalities we have like image here we have like a video we kind of they kind of sampled a couple frames from the video they have the audio here that's like i guess aligned with this video and we have a point cloud here so let me take one example and form the byte array like for this image so this image was going to be like i guess it's a image net image so it's 224 times 224 and that's like 50k pixels okay so basically how they're gonna flattened out and form a 50k so there will be m i guess so that's the m component of the byte array as you can see here so we had m times c and then the c is going to be because this is an rgb image it's going to be m times three and that's going to be i mean 50k times three and as you can see this is a like a pretty big input that transformers cannot handle by themselves and similarly for the video they're gonna basically concatenate the features the input features from the images as well as from the audio and they're going to form a single m time c array again so that's this part now let's see how the model works so basically you have this latent array and you use it to form these query vectors so this is your regular q k v attention pattern and what you do you form queries from this latent array and you form keys and values from your input and so you're basically in a way questioning the input for valuable information and then you're combining value vectors using those scores you formed using the q and k matrices to get like a novel representation here so this is the first component of the pipeline you have basically this cross attention module this is the first part then the second important part is once you have this like a vector that contains certain information from your input modality you're then going to apply your like basically the original transformer like block here you're gonna just kind of concatenate a couple of those blocks and form this latent transformer and that's the basically the second component of the pipeline now what I do is just kind of repeat this thing so as you can see here and now we again we form the query vectors and we again query the keys here and we we form another representation here so how you can think about this is basically the latent transformers are like kind of refining the representation so that we can ask better questions to the input I mean ask better questions to the input modality and form more informative representations and then at the end we can kind of use that for whatever specific task we are having at hand so basically classification or not so that's the the main idea I mean it's fairly simple and one detail that's worth mentioning here is that they are sharing weights between these latent transformers so as you can see here these dotted this this dashed line means that this latent transformer has the same weights as this one and they're doing the same thing for for the cross attention now there is a small and important detail I guess they are actually for this cross attention the first one is going to have its own weights and then all of the other cross attention modules are going to share the weights because otherwise they had some instability problems okay so if you watch some of my previous videos like I covered Facebook Detre model in one of my previous like videos and I think it's a nice comparison to contrast that this this perceiver with that model so we could see many basic ideas and and and like building blocks in this Detre model so as you can see here so this plays a role of basically this part of the latent array so you have this object queries and then you have a basically transformer decoder so that's the same thing as this latent transformer here and as you can see here this is if we treat this as the byte array this part this representation here basically then we have a similar structure all in all so obviously they've replaced this whole thing here so instead of having these specific cnm parts and which are obviously modality specific and then doing this intricate stuff they just kind of flatten out the inputs they do the cross attention and then they apply the transformers but yeah I thought it would be neat to kind of contrast that to to the perceiver module so again if you kind of ditch this part and you just use your input directly here and then the only other difference would be I guess interleaving these latent transformers to kind of refine the representations and here they are constantly in every single layer they're kind of consulting the input like modality so yeah I thought it would be neat to kind of mention this okay let's continue let me share a couple more interesting ideas here they mention it here because we optionally share weights between each instance of the transformer tower so that's the so that's the latent transformer and between all the all instances of the cross attention module but the first our model can be interpreted as a recurrent neural network but unrolled in depth using the same input so the same input rather than in time so unrolled in depth and using the same inputs so those are the differences so basically a quick like a recap we have a when we have rnn we have a basically nn here so we have a like a our model here we input at time step one like whatever the the the modality may be here so for example like a textual token or something what we do then is we unroll this so we apply basically the weights here are shared and then we apply another input so like a different token and then we repeat this like on and on and so the difference here is instead of using different inputs they're using constantly the same input modality so that's the byte array here so let me go back here and so secondly they are obviously not unrolling in time they are unrolling in depth so we have these powerful latent transformers where we are unrolling them in depth so i guess there is a nice mapping between these two ideas and i guess this thing but the first one so this kind of a symmetry between not sharing the weights in the weights in the first cross attention module with the other ones is kind of worth further investigation um continuing on here second thing i want to mention here and this is fairly important this is what enables us to use a perceiver directly on the input modality without doing all of those tricks like for example the vision transformer had to first do like one layer of convolution or how they kind of originally called that is to take the patches and linearly project them but you can implement that using a single convolutional layer and only then when you reduce the spatial dimensions so you basically have a much less number of tokens you can then pass it them through the you can then pass them through the transformer so in order to avoid that we need to reduce the complexity and here is the the main reason why perceiver works this results in architecture with complexity m times n plus l times n squared and this is the key so m is the size of the input modality and n is the size of the latent array and they usually use something like 512 here 512 here and if you remember we had 50k for the rgb image from image net so there is a huge difference there and we can save a lot of computation just here because we don't have a quadratic like dependency on m and this thing here is l is just a number of layers in the latent transformer and n is the again like latent dimension and so it's n squared because we are using the transformer but this time we're using the transformer and we're feeding as the input those latent variables instead of using the input modality so this is how they manage to save the computation and this is the whole trick and so by decoupling the input size and the depth we can add additional transformer layers at a cost that's independent of the input size and that's cool so we don't have m squared we have n squared so basically the the main thing to do to reduce the complexity is to have this projection of the input modality into this latent space by doing this query kqv basically attention and reducing using this as a bottleneck one final remark is let me mention the positional encodings so the reason we have to use position encodings is the same as the reason why we had to use the positional encodings in the original transformer in the first place and that's because transformers are permutation actually equivalent not invariant that means that if you have an input here and if you rearrange it so this is some tokens and if you rearrange them your output is going to be rearranged in the same fashion and that's why it's equivalent not invariant if it was invariant then that means whatever however we permute the input sequence the outputs would remain the same and if it's equivalent then it's kind of following the pattern of the input permutation at the output hopefully that's clear so basically they're using same Fourier features as in the original transformer paper and they mentioned some small detail here so in language modeling transformer inputs are typically produced by adding a position encoding to the input encoding so by adding them and we found it beneficial to instead concatenate the position and input features before passing them into the perceiver so this is just a minor detail obviously but like i just wanted to to mention that they have to add this in a way modality specific positional encodings but that's a minor like prior compared to having to actually tweak the architecture to deal with the modality okay those are pretty much all of the details you need to know about the perceiver now let's see some results they did experiments on various modalities one of them were images they also did on audio on video and on point clouds let's start with images so they have results here let me go back here and as you can see here they compare the perceiver with like the simplest baseline such as resonant 50 vat so the vision transformer here they additionally add the ff means Fourier features they basically concatenate the Fourier features along all of the pixels and then they pass that through the resonant and vat and so that's the second block and finally they have the basically the original transformer and the only thing they had to do is tweak so resize the input image from 224 to 64 times 64 because as we know quadratic complexity before i even comment the results and the drop we have here you need to be cognizant of the next fact and that's that these are the simplest baselines if we took a soda so the state-of-the-art model on image net without no pre so with no pre-training and those would be nf nets and efficient nets we too we can see that so basically the the top one accuracy is 86.5 which is way higher compared to what perceiver achieved here and then efficient nets we too additionally like with no pre-training they achieve roughly this this accuracy but they are also way more efficient obviously both parameter wise and like computation wise etc so the reason i'm mentioning this is because they are not trying to compare a perceiver with the highly specialized modality specific architectures they're just trying to show that they are as good or even a bit better than the regular baselines and so now let's see the results we can see that it's obviously better than both the resonant and the vat and we can see a significant drop here so from 77 to 73 and a smaller drop from for the vat a quick remark on this drop for resonant 50 when we had the Fourier the Fourier features basically the author is mentioned here to account for the perceivers use of features at input we train versions of the benchmark models with this input as well and found that they produce comparable if slightly worse performance to models trained solely on rgb input and i mean this slightly worse is a very arguable statement because i mean dropping from 77 to 73 four percent in top one seems like a significant drop and i would love to know why and i think i would give this another look whether there is some bug here or i don't know looks suspicious to me in any case anyways let me show you some other interesting results so these ones here so i mentioned the permutation active variants of of the transformers and as well as perceiver iode obviously and so they did a small experiment here to kind of further consolidate this idea of using modality agnostic like architecture such as perceiver i.o. and the whole point is so what it did is they took a raw image and we have scores here to top one accuracy and then they do did this permutation and you see you can see that there is a significant drop happening here as well as for vat although some somewhat smaller obviously and here we have like same results and let me just quickly explain what this permutation means so what i do is they take an image net image let me draw a simplified image there will be three by three pixels so something like this one two three one two three okay and i'm going to label these pixels as so one two three two three four five six etc all the way until now nine okay so what it did is they basically just predefined like a like a like a deterministic permutation mapping so for example one possible solution would be something like this like i would put like five here and then one and then like three or whatnot and so what it means basically is that that the transformation they they do upon the input images is they'll basically shift this pixel intensity whatever it is to this location here they'll shift this pixel intensity to this location here and they do this for every single pixel according to this permutation table and so after and they also include as you can see here the Fourier features because to make it a bit more fair but like obviously because cnns are used to exploiting the local correlations and now here it's kind of that that kind of assumption is violated because from a natural image here you end up with something that's obviously artificial and so you can see a significant drop happening for the cnn architecture vat performs somewhat better although there is a drop there as well so the reason is because the the first layer as i mentioned of the vat is basically a cnn transform and then we have transformer blocks so that's why we have because of that i guess because of the cnn part i guess that's why we we suffer the performance suffers here and because these are permutation active variants so both the original transformer as well as the perceiver like ignoring the whether the the positional encodings are learned or just Fourier ones they still kind of preserve the performance okay so that was a quick remark and experiment they did and this kind of further emphasizes the the need to have these generic architecture such as perceiver okay continuing on let me show you these they had some nice attention maps visualized here so what you see here is basically this image here you take the first cross attention block from the perceiver model and you basically take the so again you have a latent vector here so this will be n dimensional and the modality is this one here so we have m pixels here so what they do is they take a query so a single query so maybe they form from this position they form so from this feature vector latent feature vector they form a query vector and they use that very vector to query all of the keys of this image and that's how they form all of these visualizations okay so basically these are just attention scores prior to soft max so again every single pixel here will have its own key vector and we're going to do like dot product between this query and all of the keys for every pixel and that's how we form these scores and so you can see there is a qualitative change happening between this one and these ones so that's the reason is because as i mentioned they have unique weights for this first cross attention module and they share the weights for all of the other modules and they're using actually eight eight cross attention modules for the image net model that means they'll be consulting the input modality i.e. the the image eight times and then they'll have obviously the refinements using the latent transformers but you can see a qualitative change between these ones and so because these weights these two share the same weights whereas this has separate set of weights and you can see obviously qualitative change so here you can even like notice a dog image even though these are only scores i'm fairly sure this is not overlaid on top of the image i'm fairly sure this just pops up when you do the when you calculate these scores and here you can see these types of 2d sinusoids i guess some complex 2d signals which kind of resemble the Fourier features in a way and so they speculate that that's because they're using Fourier features as the positional embeddings because once they use they learn once they mentioned that they don't see these patterns anymore so i guess it's it heavily correlates with the use of Fourier features and here is just kind of like the same thing broken down for multiple query vectors so we just saw a single query vector remember there are 512 of these for the image net task and so they can have 512 of these visualizations they just have 100 here and a couple of other ones here i guess you can just see they correlate heavily these ones with these ones obviously continuing on to other modalities they did a same set of experiments on this audio set data set so basically audio set contains a bunch of data and it's videos combined with audio obviously and they have some baselines here and they compare a perceiver with those baselines using only audio as the modality using only video and using both the audio and video they basically just concatenate the data from audio and video into that m times c byte array so the details are not that important in my opinion so but yeah a couple of interesting facts here so first perceiver is kind of um um pair or better compared to all of these other baselines including resin at 50 again uh but like this c9 14 which is super specialized has a bit higher score and once they kind of drop domain specific information then they are now kind of better i mean you can always make your model perform better than the baselines if you do this kind of stuff but yeah as i mentioned previously they're trying to compare with baselines and show that this same model with some small tweaks like specific augmentations and specific positional encodings they can kind of have on-pair performance with the baselines across different modalities and that's the whole point of the paper on video we can see it performs better than these two baselines here and finally for the combination we can see that obviously using both modalities helps boost the performance so it's way better compared to using only video or comparing only audio or using only audio and again this baseline that has this specific uh like a late stage fusion has a bit better performance but again not that important for me what's interesting is that the the the difference between raw audio and male spectrogram when we're using only the audio modality is like very similar similar and usually you get like a huge boost by using these spectrograms so i guess this is something that's worth further investigation and something i found interesting yeah a small rant here they mentioned this so for video um basically a full 32 frame clip at this resolution has more than 2 million pixels and that's obviously not something that's feasible even with a model such as perceiver so they mentioned here they experimented using tiny space time patches with dimensions two times eight times eight resulting in a total of 12 000 inputs to the perceiver so this resembles the v what vat is doing a lot and so they were kind of comparing themselves with vat and here they're doing basically the same thing a hack on how to reduce the input modality into uh using domain specific knowledge into smaller modality and then input it back into perceiver so i i assume like this does not look consistent with the rest of the ideas in this paper and yeah uh just leave your comments what what are your thoughts on this and whether whether this is violating the whole idea of of the perceiver model and finally i'm just going to end it up here showing the results on the uh basically uh point cloud classification the goal here is given 3d point cloud to classify an object into certain category and again you can see vats resonance and transformer perceiver outperforms all of them but again when you have highly specialized architecture such as this point net plus bus it's way better compared to perceiver by at least six points here i'm going to wrap up this paper with this sentence so they mentioned it here while we reduced the amount of modality specific prior knowledge in the model we still employ modality specific augmentation and position encoding and we also just saw the trick they are doing to make video modality feasible and so this is something to to to have in mind it's we're still we're still definitely struggling uh with videos finally before i contrast this paper with perceiver io with the novel paper i want to mention one more thing uh that i saw in the ablations here so let me just find it so what they did here is they basically omitted as they say here performance of models built from a stack of cross attention layers with no latent transformers and now this is way more equivalent to what the debtor facebook debtor model is doing and you can see that the accuracy is way lower compared to using those latent transformers to refine the representation so that's this is a kind of argument behind the usage of latent transformers okay now let me quickly contrast this paper with perceiver io okay perceiver io a general architecture for structured inputs and outputs again the deep mind team and they say here while the perceiver supports many kinds of inputs it can only produce very simple outputs such as class scores perceiver io overcomes this limitation without sacrificing the originals appealing properties by learning to flexibly query the model's latent space to produce outputs of arbitrary size and semantics so in the original paper all of the tasks we saw so image net audio set and those point clouds all of those were classification tasks and so we had a simple uh like the output structure was very simple when you have more complex uh problems such as trying to make an agent for starcraft 2 then you have much more structured outputs so the output may be something like this like you have one array that tells you which agent you should select so which of the characters you should select and then once you select the agent you have like a matrix that tells you what's the probability of moving to all of these locations so maybe you have something like four by four matrix and basically you you have a probability of where the current character the selected character should move and so you have like complex structured output and the the thing is the original perceiver even though it was very flexible with the input it wasn't as flexible with the output and so basically what this paper does and they mentioned it here and they said here our main technical contribution is perceiver ios decoding procedure so let's see what they mean by that here is the architecture let me just kind of contrast this one with the original perceiver paper so this part is the same so we have latent array n times d we have the input byte array m times c we just do this cross attention module we do the latent transformer part so this is the latent transformer part this is the cross attention part and we just repeat that part rinse and repeat and that's the same thing as with the original perceiver so the novel thing here is this output query array so what you do is depending on your desired outputs you have like an output array that maybe has like o vectors and every vector has e features so what you would do is form o query vectors and basically again do across like attention logic and so because you have o vectors here you're going to end up with o vectors here zooming in into the decoder head because this is the main contribution of the paper let me kind of like break it down a little bit more first things first this these query vectors are going to be task dependent so depending on the structured output you want to achieve you'll be either ingraining some integrating some domain specific knowledge here or you're going to make these learnable let's see how the shapes work out here so we know we have basically all of these and their dimension the dimensionality is not that important it's going to be like x and the only important thing is that they match with like key vectors because we're going to do dot products here in order to form the attention scores now the important part is that the value vectors so these here have the dimensionality e if that's our target as you can see here so he's our target so we want to have the dimensionality of these value vectors should be e and we have we need to have all of these and that's how we make sure that this thing here is o times e and that's our desired shape so basically everything else is simple attention logic we form the weights here we use these to kind of weight these value vectors we add them up and we form via these two weights and via uh basically these two value vectors we form this vector here and we just repeat a procedure for all of the output vectors so that's basically the main contribution of the paper everything else is about trying this new perceiver io module model on various different tasks and i'm going to discover a single one and that's the language uh like uh like the glue benchmark and they compare with bird piece and you can see here depending whether they use tokenization so sentence piece is a like a famous tokenization algorithm or simply using utf-8 bytes that means no tokenization at all you just take the raw text and you encode the characters as utf-8 bytes and so you can see that they can report uh results of their own pair with bird but here's a catch i mean even though they're kind of making the the flop similar here uh the number of parameters is obviously way way bigger for perceiver io so if that's something you care about this is obviously not going to be like a fair comparison right and i guess this is where perceiver io shines the reason is we have when we have utf-8 the input size is going to be way larger and then because of that you have to kind of reduce the depth of the bird model and that means you have less parameters less expressive model and you get way bigger gap here and i find that very cool because just using utf-8 encoding instead of having to devise these intricate tokenization methods and tokenization is literally a field in and of itself so have like being able to avoid that whole step and still manage to have these results like even better than using sentence piece is pretty pretty damn amazing having said that that's it for this video if you found it useful share it out with a friend and subscribe to this channel hit the bell icon join the discord community and until next time bye bye
[{"start": 0.0, "end": 4.08, "text": " What's cracking guys in this video I'm covering Perceiver, general perception with iterative"}, {"start": 4.08, "end": 9.44, "text": " attention and at the end of the video I'm gonna contrast it with the new newly published Perceiver"}, {"start": 9.44, "end": 15.6, "text": " IO paper. It's a work by DeepMind team Andrew Jegel, Felikski Manow, Andrew Bruch, Andrew Tissaman,"}, {"start": 15.6, "end": 22.8, "text": " Oriol Vinales and Joao Caheira. So the idea is to generate this general perception architecture"}, {"start": 22.8, "end": 27.44, "text": " and make it modality agnostic and I'm gonna break that down in a little bit but let's see the"}, {"start": 27.44, "end": 33.84, "text": " motivation behind the paper. So the perception models used in deep learning are designed for"}, {"start": 33.84, "end": 38.64, "text": " individual modalities often relying on domain specific assumptions such as the local grid"}, {"start": 38.64, "end": 44.0, "text": " structures and I guess they're referring to CNN here but basically depending on which modality"}, {"start": 44.0, "end": 48.56, "text": " you're using in your deep learning pipeline if you have like an image or if you have an audio or if"}, {"start": 48.56, "end": 53.6, "text": " you have like a point cloud you're going to use heavily specialized architectures to kind of"}, {"start": 53.6, "end": 60.480000000000004, "text": " process and understand all of those modalities. The whole idea of this paper is to kind of"}, {"start": 60.480000000000004, "end": 65.68, "text": " ditch all of those specific priors which are modality dependent like for example when you're"}, {"start": 65.68, "end": 71.12, "text": " using a CNN you're basically exploiting the fact that neighboring pixels their intensities are"}, {"start": 71.12, "end": 75.36, "text": " correlated and so you can use a kernel that's like three by three and you can kind of shift it and"}, {"start": 75.36, "end": 81.04, "text": " share the weights whereby you're also exploiting the fact that the statistics will not change"}, {"start": 81.04, "end": 87.28, "text": " across the image and so they say it here nicely in this paper we introduce the perceiver a model"}, {"start": 87.28, "end": 91.76, "text": " that builds upon transformers and hence makes few architectural assumptions about the relationship"}, {"start": 91.76, "end": 97.2, "text": " between its inputs but it also scales to hundreds of thousands of inputs like comnets so on the one"}, {"start": 97.2, "end": 103.68, "text": " hand they're reusing transformers which are like proven to be very good at modeling like arbitrary"}, {"start": 103.68, "end": 109.84, "text": " relationships between the inputs and on the other hand they kind of avoided this quadratic"}, {"start": 109.84, "end": 115.12, "text": " complexity problem that transformers have and we're going to see how they did it and that's"}, {"start": 115.12, "end": 120.16, "text": " what enables them to apply this model to hundreds of thousands of inputs so let's dig into the"}, {"start": 120.16, "end": 126.96000000000001, "text": " architecture straight away here it is it basically as you can see here we have this latent array which"}, {"start": 126.96000000000001, "end": 132.96, "text": " is n times d dimensions we have this byte array which is basically our input modality so this"}, {"start": 132.96, "end": 138.0, "text": " thing here is going to be either your image or your audio or your video or whatnot so whatever you"}, {"start": 138.0, "end": 143.6, "text": " have so let me kind of explain that using this image below here you can see various modalities"}, {"start": 143.6, "end": 149.52, "text": " we have like image here we have like a video we kind of they kind of sampled a couple frames"}, {"start": 149.52, "end": 155.04, "text": " from the video they have the audio here that's like i guess aligned with this video and we have"}, {"start": 155.04, "end": 160.24, "text": " a point cloud here so let me take one example and form the byte array like for this image so this"}, {"start": 160.24, "end": 166.88, "text": " image was going to be like i guess it's a image net image so it's 224 times 224 and that's like"}, {"start": 166.88, "end": 174.96, "text": " 50k pixels okay so basically how they're gonna flattened out and form a 50k so there will be m"}, {"start": 174.96, "end": 181.04, "text": " i guess so that's the m component of the byte array as you can see here so we had m times c"}, {"start": 181.6, "end": 186.96, "text": " and then the c is going to be because this is an rgb image it's going to be m times three and that's"}, {"start": 186.96, "end": 192.24, "text": " going to be i mean 50k times three and as you can see this is a like a pretty big input that"}, {"start": 192.24, "end": 197.52, "text": " transformers cannot handle by themselves and similarly for the video they're gonna basically"}, {"start": 197.52, "end": 203.84, "text": " concatenate the features the input features from the images as well as from the audio and they're"}, {"start": 203.84, "end": 209.28, "text": " going to form a single m time c array again so that's this part now let's see how the model"}, {"start": 209.28, "end": 213.68, "text": " works so basically you have this latent array and you use it to form these query vectors so this is"}, {"start": 213.68, "end": 221.60000000000002, "text": " your regular q k v attention pattern and what you do you form queries from this latent array"}, {"start": 221.6, "end": 227.04, "text": " and you form keys and values from your input and so you're basically in a way questioning the input"}, {"start": 227.04, "end": 233.04, "text": " for valuable information and then you're combining value vectors using those scores you formed using"}, {"start": 233.04, "end": 238.79999999999998, "text": " the q and k matrices to get like a novel representation here so this is the first"}, {"start": 238.79999999999998, "end": 244.32, "text": " component of the pipeline you have basically this cross attention module this is the first part then"}, {"start": 244.32, "end": 250.07999999999998, "text": " the second important part is once you have this like a vector that contains certain information"}, {"start": 250.08, "end": 255.44000000000003, "text": " from your input modality you're then going to apply your like basically the original transformer"}, {"start": 256.16, "end": 259.92, "text": " like block here you're gonna just kind of concatenate a couple of those blocks and form"}, {"start": 259.92, "end": 266.0, "text": " this latent transformer and that's the basically the second component of the pipeline now what I"}, {"start": 266.0, "end": 270.8, "text": " do is just kind of repeat this thing so as you can see here and now we again we form the query"}, {"start": 270.8, "end": 278.72, "text": " vectors and we again query the keys here and we we form another representation here so how you can"}, {"start": 278.72, "end": 286.32000000000005, "text": " think about this is basically the latent transformers are like kind of refining the representation so"}, {"start": 286.32000000000005, "end": 291.92, "text": " that we can ask better questions to the input I mean ask better questions to the input modality"}, {"start": 291.92, "end": 297.44000000000005, "text": " and form more informative representations and then at the end we can kind of use that for whatever"}, {"start": 297.44000000000005, "end": 303.52000000000004, "text": " specific task we are having at hand so basically classification or not so that's the the main idea"}, {"start": 303.52, "end": 309.28, "text": " I mean it's fairly simple and one detail that's worth mentioning here is that they are sharing"}, {"start": 309.28, "end": 314.47999999999996, "text": " weights between these latent transformers so as you can see here these dotted this this dashed"}, {"start": 314.47999999999996, "end": 319.44, "text": " line means that this latent transformer has the same weights as this one and they're doing the"}, {"start": 319.44, "end": 325.12, "text": " same thing for for the cross attention now there is a small and important detail I guess they are"}, {"start": 325.12, "end": 329.84, "text": " actually for this cross attention the first one is going to have its own weights and then all of"}, {"start": 329.84, "end": 334.32, "text": " the other cross attention modules are going to share the weights because otherwise they had some"}, {"start": 334.32, "end": 341.44, "text": " instability problems okay so if you watch some of my previous videos like I covered Facebook"}, {"start": 341.44, "end": 347.35999999999996, "text": " Detre model in one of my previous like videos and I think it's a nice comparison to contrast"}, {"start": 347.35999999999996, "end": 353.44, "text": " that this this perceiver with that model so we could see many basic ideas and and and like building"}, {"start": 353.44, "end": 360.32, "text": " blocks in this Detre model so as you can see here so this plays a role of basically this part of the"}, {"start": 360.32, "end": 366.56, "text": " latent array so you have this object queries and then you have a basically transformer decoder so"}, {"start": 366.56, "end": 373.2, "text": " that's the same thing as this latent transformer here and as you can see here this is if we treat"}, {"start": 373.2, "end": 380.56, "text": " this as the byte array this part this representation here basically then we have a similar structure"}, {"start": 380.56, "end": 386.64, "text": " all in all so obviously they've replaced this whole thing here so instead of having these specific"}, {"start": 386.64, "end": 392.4, "text": " cnm parts and which are obviously modality specific and then doing this intricate stuff"}, {"start": 392.4, "end": 396.8, "text": " they just kind of flatten out the inputs they do the cross attention and then they apply the"}, {"start": 396.8, "end": 401.68, "text": " transformers but yeah I thought it would be neat to kind of contrast that to to the perceiver"}, {"start": 402.24, "end": 406.64, "text": " module so again if you kind of ditch this part and you just use your input directly here"}, {"start": 406.64, "end": 413.2, "text": " and then the only other difference would be I guess interleaving these latent transformers"}, {"start": 413.2, "end": 417.84, "text": " to kind of refine the representations and here they are constantly in every single layer they're"}, {"start": 417.84, "end": 425.12, "text": " kind of consulting the input like modality so yeah I thought it would be neat to kind of mention this"}, {"start": 425.12, "end": 430.64, "text": " okay let's continue let me share a couple more interesting ideas here they mention it here"}, {"start": 431.28, "end": 435.91999999999996, "text": " because we optionally share weights between each instance of the transformer tower so that's the"}, {"start": 435.92, "end": 440.64000000000004, "text": " so that's the latent transformer and between all the all instances of the cross attention module"}, {"start": 440.64000000000004, "end": 446.0, "text": " but the first our model can be interpreted as a recurrent neural network but unrolled in depth"}, {"start": 446.0, "end": 452.40000000000003, "text": " using the same input so the same input rather than in time so unrolled in depth and using the same"}, {"start": 452.40000000000003, "end": 458.32, "text": " inputs so those are the differences so basically a quick like a recap we have a when we have rnn we"}, {"start": 458.32, "end": 465.28000000000003, "text": " have a basically nn here so we have a like a our model here we input at time step one like whatever"}, {"start": 465.28, "end": 471.59999999999997, "text": " the the the modality may be here so for example like a textual token or something what we do then"}, {"start": 471.59999999999997, "end": 478.64, "text": " is we unroll this so we apply basically the weights here are shared and then we apply another input so"}, {"start": 478.64, "end": 485.59999999999997, "text": " like a different token and then we repeat this like on and on and so the difference here is instead"}, {"start": 485.59999999999997, "end": 490.4, "text": " of using different inputs they're using constantly the same input modality so that's the byte array"}, {"start": 490.4, "end": 496.08, "text": " here so let me go back here and so secondly they are obviously not unrolling in time they are"}, {"start": 496.08, "end": 502.15999999999997, "text": " unrolling in depth so we have these powerful latent transformers where we are unrolling them"}, {"start": 502.15999999999997, "end": 506.4, "text": " in depth so i guess there is a nice mapping between these two ideas and i guess this thing"}, {"start": 506.4, "end": 511.84, "text": " but the first one so this kind of a symmetry between not sharing the weights in the weights"}, {"start": 511.84, "end": 517.84, "text": " in the first cross attention module with the other ones is kind of worth further investigation"}, {"start": 517.84, "end": 522.0, "text": " um continuing on here second thing i want to mention here and this is fairly important this"}, {"start": 522.0, "end": 526.96, "text": " is what enables us to use a perceiver directly on the input modality without doing all of those"}, {"start": 526.96, "end": 533.2, "text": " tricks like for example the vision transformer had to first do like one layer of convolution"}, {"start": 533.76, "end": 539.2, "text": " or how they kind of originally called that is to take the patches and linearly project them but"}, {"start": 539.2, "end": 544.0, "text": " you can implement that using a single convolutional layer and only then when you reduce the spatial"}, {"start": 544.0, "end": 549.84, "text": " dimensions so you basically have a much less number of tokens you can then pass it them through the"}, {"start": 549.84, "end": 554.08, "text": " you can then pass them through the transformer so in order to avoid that we need to reduce the"}, {"start": 554.08, "end": 559.04, "text": " complexity and here is the the main reason why perceiver works this results in architecture"}, {"start": 559.04, "end": 566.96, "text": " with complexity m times n plus l times n squared and this is the key so m is the size of the input"}, {"start": 566.96, "end": 572.4, "text": " modality and n is the size of the latent array and they usually use something like 512 here"}, {"start": 572.4, "end": 579.84, "text": " 512 here and if you remember we had 50k for the rgb image from image net so there is a huge"}, {"start": 579.84, "end": 585.28, "text": " difference there and we can save a lot of computation just here because we don't have a quadratic like"}, {"start": 585.92, "end": 591.52, "text": " dependency on m and this thing here is l is just a number of layers in the latent transformer"}, {"start": 591.52, "end": 597.92, "text": " and n is the again like latent dimension and so it's n squared because we are using the transformer"}, {"start": 597.92, "end": 603.5999999999999, "text": " but this time we're using the transformer and we're feeding as the input those latent variables"}, {"start": 603.5999999999999, "end": 608.8, "text": " instead of using the input modality so this is how they manage to save the computation and this"}, {"start": 608.8, "end": 614.0799999999999, "text": " is the whole trick and so by decoupling the input size and the depth we can add additional"}, {"start": 614.0799999999999, "end": 618.9599999999999, "text": " transformer layers at a cost that's independent of the input size and that's cool so we don't"}, {"start": 618.9599999999999, "end": 624.3199999999999, "text": " have m squared we have n squared so basically the the main thing to do to reduce the complexity is"}, {"start": 624.32, "end": 631.9200000000001, "text": " to have this projection of the input modality into this latent space by doing this query kqv"}, {"start": 631.9200000000001, "end": 639.0400000000001, "text": " basically attention and reducing using this as a bottleneck one final remark is let me mention the"}, {"start": 639.0400000000001, "end": 644.08, "text": " positional encodings so the reason we have to use position encodings is the same as the reason why"}, {"start": 644.08, "end": 648.8800000000001, "text": " we had to use the positional encodings in the original transformer in the first place and that's"}, {"start": 648.88, "end": 654.64, "text": " because transformers are permutation actually equivalent not invariant that means that if you"}, {"start": 654.64, "end": 660.96, "text": " have an input here and if you rearrange it so this is some tokens and if you rearrange them"}, {"start": 661.6, "end": 667.04, "text": " your output is going to be rearranged in the same fashion and that's why it's equivalent not"}, {"start": 667.04, "end": 672.4, "text": " invariant if it was invariant then that means whatever however we permute the input sequence"}, {"start": 673.04, "end": 678.24, "text": " the outputs would remain the same and if it's equivalent then it's kind of following the pattern"}, {"start": 678.24, "end": 685.52, "text": " of the input permutation at the output hopefully that's clear so basically they're using same"}, {"start": 685.52, "end": 691.12, "text": " Fourier features as in the original transformer paper and they mentioned some small detail here"}, {"start": 691.12, "end": 695.52, "text": " so in language modeling transformer inputs are typically produced by adding a position encoding"}, {"start": 695.52, "end": 700.96, "text": " to the input encoding so by adding them and we found it beneficial to instead concatenate"}, {"start": 700.96, "end": 705.12, "text": " the position and input features before passing them into the perceiver so this is just a minor"}, {"start": 705.12, "end": 709.44, "text": " detail obviously but like i just wanted to to mention that they have to add this in a way"}, {"start": 709.44, "end": 716.64, "text": " modality specific positional encodings but that's a minor like prior compared to having to actually"}, {"start": 716.64, "end": 720.8, "text": " tweak the architecture to deal with the modality okay those are pretty much all of the details you"}, {"start": 720.8, "end": 725.52, "text": " need to know about the perceiver now let's see some results they did experiments on various modalities"}, {"start": 725.52, "end": 731.44, "text": " one of them were images they also did on audio on video and on point clouds let's start with images"}, {"start": 731.44, "end": 738.4000000000001, "text": " so they have results here let me go back here and as you can see here they compare the perceiver"}, {"start": 738.4000000000001, "end": 744.8000000000001, "text": " with like the simplest baseline such as resonant 50 vat so the vision transformer here they"}, {"start": 744.8000000000001, "end": 750.0, "text": " additionally add the ff means Fourier features they basically concatenate the Fourier features"}, {"start": 750.0, "end": 756.96, "text": " along all of the pixels and then they pass that through the resonant and vat and so that's the"}, {"start": 756.96, "end": 761.6, "text": " second block and finally they have the basically the original transformer and the only thing they"}, {"start": 761.6, "end": 769.6800000000001, "text": " had to do is tweak so resize the input image from 224 to 64 times 64 because as we know"}, {"start": 769.6800000000001, "end": 773.84, "text": " quadratic complexity before i even comment the results and the drop we have here"}, {"start": 774.5600000000001, "end": 780.08, "text": " you need to be cognizant of the next fact and that's that these are the simplest baselines if we"}, {"start": 780.08, "end": 786.08, "text": " took a soda so the state-of-the-art model on image net without no pre so with no pre-training and"}, {"start": 786.08, "end": 792.08, "text": " those would be nf nets and efficient nets we too we can see that so basically the the top one"}, {"start": 792.08, "end": 798.8000000000001, "text": " accuracy is 86.5 which is way higher compared to what perceiver achieved here and then efficient"}, {"start": 798.8000000000001, "end": 804.96, "text": " nets we too additionally like with no pre-training they achieve roughly this this accuracy but they"}, {"start": 804.96, "end": 809.84, "text": " are also way more efficient obviously both parameter wise and like computation wise etc so the reason"}, {"start": 809.84, "end": 815.2800000000001, "text": " i'm mentioning this is because they are not trying to compare a perceiver with the highly specialized"}, {"start": 815.28, "end": 820.9599999999999, "text": " modality specific architectures they're just trying to show that they are as good or even a bit better"}, {"start": 820.9599999999999, "end": 827.36, "text": " than the regular baselines and so now let's see the results we can see that it's obviously better"}, {"start": 827.36, "end": 834.3199999999999, "text": " than both the resonant and the vat and we can see a significant drop here so from 77 to 73 and a"}, {"start": 834.3199999999999, "end": 839.76, "text": " smaller drop from for the vat a quick remark on this drop for resonant 50 when we had the Fourier"}, {"start": 839.76, "end": 845.84, "text": " the Fourier features basically the author is mentioned here to account for the perceivers use"}, {"start": 845.84, "end": 850.3199999999999, "text": " of features at input we train versions of the benchmark models with this input as well and"}, {"start": 850.3199999999999, "end": 854.96, "text": " found that they produce comparable if slightly worse performance to models trained solely on rgb"}, {"start": 854.96, "end": 861.52, "text": " input and i mean this slightly worse is a very arguable statement because i mean dropping from"}, {"start": 861.52, "end": 868.96, "text": " 77 to 73 four percent in top one seems like a significant drop and i would love to know why"}, {"start": 868.96, "end": 873.6, "text": " and i think i would give this another look whether there is some bug here or i don't know"}, {"start": 873.6, "end": 878.72, "text": " looks suspicious to me in any case anyways let me show you some other interesting results so"}, {"start": 879.44, "end": 885.2, "text": " these ones here so i mentioned the permutation active variants of of the transformers and as"}, {"start": 885.2, "end": 891.84, "text": " well as perceiver iode obviously and so they did a small experiment here to kind of further"}, {"start": 891.84, "end": 898.64, "text": " consolidate this idea of using modality agnostic like architecture such as perceiver i.o. and the"}, {"start": 898.64, "end": 904.96, "text": " whole point is so what it did is they took a raw image and we have scores here to top one accuracy"}, {"start": 904.96, "end": 909.84, "text": " and then they do did this permutation and you see you can see that there is a significant drop"}, {"start": 909.84, "end": 916.64, "text": " happening here as well as for vat although some somewhat smaller obviously and here we have like"}, {"start": 916.64, "end": 922.16, "text": " same results and let me just quickly explain what this permutation means so what i do is they take"}, {"start": 922.16, "end": 927.12, "text": " an image net image let me draw a simplified image there will be three by three pixels so something"}, {"start": 927.12, "end": 937.04, "text": " like this one two three one two three okay and i'm going to label these pixels as so one two three"}, {"start": 937.04, "end": 947.1999999999999, "text": " two three four five six etc all the way until now nine okay so what it did is they basically just"}, {"start": 948.24, "end": 956.0799999999999, "text": " predefined like a like a like a deterministic permutation mapping so for example one possible"}, {"start": 956.7199999999999, "end": 965.04, "text": " solution would be something like this like i would put like five here and then one and then like"}, {"start": 965.04, "end": 970.88, "text": " three or whatnot and so what it means basically is that that the transformation they they do upon"}, {"start": 970.88, "end": 977.52, "text": " the input images is they'll basically shift this pixel intensity whatever it is to this location"}, {"start": 977.52, "end": 983.92, "text": " here they'll shift this pixel intensity to this location here and they do this for every single"}, {"start": 983.92, "end": 989.68, "text": " pixel according to this permutation table and so after and they also include as you can see here"}, {"start": 989.68, "end": 996.7199999999999, "text": " the Fourier features because to make it a bit more fair but like obviously because cnns are used to"}, {"start": 997.52, "end": 1003.3599999999999, "text": " exploiting the local correlations and now here it's kind of that that kind of assumption is"}, {"start": 1003.3599999999999, "end": 1008.88, "text": " violated because from a natural image here you end up with something that's obviously artificial"}, {"start": 1008.88, "end": 1014.8, "text": " and so you can see a significant drop happening for the cnn architecture vat performs somewhat"}, {"start": 1014.8, "end": 1021.28, "text": " better although there is a drop there as well so the reason is because the the first layer as i"}, {"start": 1021.28, "end": 1027.2, "text": " mentioned of the vat is basically a cnn transform and then we have transformer blocks so that's why"}, {"start": 1027.2, "end": 1033.28, "text": " we have because of that i guess because of the cnn part i guess that's why we we suffer the"}, {"start": 1033.28, "end": 1038.1599999999999, "text": " performance suffers here and because these are permutation active variants so both the original"}, {"start": 1038.1599999999999, "end": 1044.32, "text": " transformer as well as the perceiver like ignoring the whether the the positional encodings are"}, {"start": 1044.32, "end": 1050.8, "text": " learned or just Fourier ones they still kind of preserve the performance okay so that was a quick"}, {"start": 1050.8, "end": 1057.28, "text": " remark and experiment they did and this kind of further emphasizes the the need to have these"}, {"start": 1058.0, "end": 1063.28, "text": " generic architecture such as perceiver okay continuing on let me show you these they had"}, {"start": 1063.28, "end": 1070.8, "text": " some nice attention maps visualized here so what you see here is basically this image here you take"}, {"start": 1070.8, "end": 1077.28, "text": " the first cross attention block from the perceiver model and you basically take the so again you have"}, {"start": 1077.28, "end": 1085.84, "text": " a latent vector here so this will be n dimensional and the modality is this one here so we have m"}, {"start": 1085.84, "end": 1091.04, "text": " pixels here so what they do is they take a query so a single query so maybe they form from this"}, {"start": 1091.04, "end": 1095.76, "text": " position they form so from this feature vector latent feature vector they form a query vector"}, {"start": 1095.76, "end": 1101.68, "text": " and they use that very vector to query all of the keys of this image and that's how they form all"}, {"start": 1101.68, "end": 1109.52, "text": " of these visualizations okay so basically these are just attention scores prior to soft max so"}, {"start": 1109.52, "end": 1115.12, "text": " again every single pixel here will have its own key vector and we're going to do like dot product"}, {"start": 1115.12, "end": 1121.2, "text": " between this query and all of the keys for every pixel and that's how we form these scores and so"}, {"start": 1121.2, "end": 1126.4, "text": " you can see there is a qualitative change happening between this one and these ones so that's the"}, {"start": 1126.4, "end": 1132.16, "text": " reason is because as i mentioned they have unique weights for this first cross attention module and"}, {"start": 1132.16, "end": 1137.8400000000001, "text": " they share the weights for all of the other modules and they're using actually eight eight"}, {"start": 1137.8400000000001, "end": 1144.56, "text": " cross attention modules for the image net model that means they'll be consulting the input modality"}, {"start": 1144.56, "end": 1149.8400000000001, "text": " i.e. the the image eight times and then they'll have obviously the refinements using the latent"}, {"start": 1149.84, "end": 1154.8799999999999, "text": " transformers but you can see a qualitative change between these ones and so because these weights"}, {"start": 1154.8799999999999, "end": 1157.9199999999998, "text": " these two share the same weights whereas this has separate set of weights"}, {"start": 1159.84, "end": 1165.28, "text": " and you can see obviously qualitative change so here you can even like notice a dog image even"}, {"start": 1165.28, "end": 1170.48, "text": " though these are only scores i'm fairly sure this is not overlaid on top of the image i'm fairly sure"}, {"start": 1170.48, "end": 1175.04, "text": " this just pops up when you do the when you calculate these scores and here you can see these"}, {"start": 1175.04, "end": 1180.8799999999999, "text": " types of 2d sinusoids i guess some complex 2d signals which kind of resemble the Fourier"}, {"start": 1181.6, "end": 1187.44, "text": " features in a way and so they speculate that that's because they're using Fourier features"}, {"start": 1187.44, "end": 1192.32, "text": " as the positional embeddings because once they use they learn once they mentioned that they"}, {"start": 1192.32, "end": 1196.8, "text": " don't see these patterns anymore so i guess it's it heavily correlates with the use of"}, {"start": 1196.8, "end": 1204.0, "text": " Fourier features and here is just kind of like the same thing broken down for multiple"}, {"start": 1204.0, "end": 1210.72, "text": " query vectors so we just saw a single query vector remember there are 512 of these for the image net"}, {"start": 1211.36, "end": 1216.64, "text": " task and so they can have 512 of these visualizations they just have 100 here and"}, {"start": 1216.64, "end": 1221.92, "text": " a couple of other ones here i guess you can just see they correlate heavily these ones with these"}, {"start": 1221.92, "end": 1228.32, "text": " ones obviously continuing on to other modalities they did a same set of experiments on this audio"}, {"start": 1228.32, "end": 1236.1599999999999, "text": " set data set so basically audio set contains a bunch of data and it's videos combined with audio"}, {"start": 1236.1599999999999, "end": 1244.08, "text": " obviously and they have some baselines here and they compare a perceiver with those baselines"}, {"start": 1244.8, "end": 1250.72, "text": " using only audio as the modality using only video and using both the audio and video they basically"}, {"start": 1250.72, "end": 1257.4399999999998, "text": " just concatenate the data from audio and video into that m times c byte array so the details are not"}, {"start": 1257.44, "end": 1262.24, "text": " that important in my opinion so but yeah a couple of interesting facts here so first perceiver is kind"}, {"start": 1262.24, "end": 1268.88, "text": " of um um pair or better compared to all of these other baselines including resin at 50 again uh but"}, {"start": 1268.88, "end": 1274.96, "text": " like this c9 14 which is super specialized has a bit higher score and once they kind of drop domain"}, {"start": 1274.96, "end": 1280.0800000000002, "text": " specific information then they are now kind of better i mean you can always make your model"}, {"start": 1280.0800000000002, "end": 1285.1200000000001, "text": " perform better than the baselines if you do this kind of stuff but yeah as i mentioned previously"}, {"start": 1285.12, "end": 1290.6399999999999, "text": " they're trying to compare with baselines and show that this same model with some small tweaks like"}, {"start": 1290.6399999999999, "end": 1296.8799999999999, "text": " specific augmentations and specific positional encodings they can kind of have on-pair performance"}, {"start": 1296.8799999999999, "end": 1300.1599999999999, "text": " with the baselines across different modalities and that's the whole point of the paper"}, {"start": 1301.04, "end": 1307.6, "text": " on video we can see it performs better than these two baselines here and finally for the combination"}, {"start": 1307.6, "end": 1312.4799999999998, "text": " we can see that obviously using both modalities helps boost the performance so it's way better"}, {"start": 1312.48, "end": 1320.08, "text": " compared to using only video or comparing only audio or using only audio and again this baseline"}, {"start": 1320.08, "end": 1326.0, "text": " that has this specific uh like a late stage fusion has a bit better performance but again not that"}, {"start": 1326.0, "end": 1331.52, "text": " important for me what's interesting is that the the the difference between raw audio and male"}, {"start": 1331.52, "end": 1338.08, "text": " spectrogram when we're using only the audio modality is like very similar similar and usually"}, {"start": 1338.08, "end": 1342.8, "text": " you get like a huge boost by using these spectrograms so i guess this is something that's"}, {"start": 1342.8, "end": 1348.0, "text": " worth further investigation and something i found interesting yeah a small rant here"}, {"start": 1348.8799999999999, "end": 1356.6399999999999, "text": " they mentioned this so for video um basically a full 32 frame clip at this resolution has more"}, {"start": 1356.6399999999999, "end": 1362.3999999999999, "text": " than 2 million pixels and that's obviously not something that's feasible even with a model such"}, {"start": 1362.4, "end": 1368.5600000000002, "text": " as perceiver so they mentioned here they experimented using tiny space time patches with dimensions"}, {"start": 1368.5600000000002, "end": 1375.76, "text": " two times eight times eight resulting in a total of 12 000 inputs to the perceiver so this"}, {"start": 1375.76, "end": 1382.88, "text": " resembles the v what vat is doing a lot and so they were kind of comparing themselves with vat"}, {"start": 1383.44, "end": 1389.0400000000002, "text": " and here they're doing basically the same thing a hack on how to reduce the input modality into"}, {"start": 1389.04, "end": 1393.76, "text": " uh using domain specific knowledge into smaller modality and then input it back into perceiver so"}, {"start": 1393.76, "end": 1401.92, "text": " i i assume like this does not look consistent with the rest of the ideas in this paper and yeah"}, {"start": 1402.48, "end": 1407.04, "text": " uh just leave your comments what what are your thoughts on this and whether whether this is"}, {"start": 1408.0, "end": 1413.84, "text": " violating the whole idea of of the perceiver model and finally i'm just going to end it up here"}, {"start": 1413.84, "end": 1420.48, "text": " showing the results on the uh basically uh point cloud classification the goal here is given 3d"}, {"start": 1420.48, "end": 1426.0, "text": " point cloud to classify an object into certain category and again you can see vats resonance"}, {"start": 1426.0, "end": 1430.8, "text": " and transformer perceiver outperforms all of them but again when you have highly specialized"}, {"start": 1430.8, "end": 1436.32, "text": " architecture such as this point net plus bus it's way better compared to perceiver by at least six"}, {"start": 1436.32, "end": 1440.3999999999999, "text": " points here i'm going to wrap up this paper with this sentence so they mentioned it here"}, {"start": 1440.4, "end": 1444.4, "text": " while we reduced the amount of modality specific prior knowledge in the model we still employ"}, {"start": 1444.4, "end": 1450.4, "text": " modality specific augmentation and position encoding and we also just saw the trick they"}, {"start": 1450.4, "end": 1457.3600000000001, "text": " are doing to make video modality feasible and so this is something to to to have in mind it's we're"}, {"start": 1457.3600000000001, "end": 1463.52, "text": " still we're still definitely struggling uh with videos finally before i contrast this paper with"}, {"start": 1463.52, "end": 1468.64, "text": " perceiver io with the novel paper i want to mention one more thing uh that i saw in the"}, {"start": 1468.64, "end": 1473.68, "text": " ablations here so let me just find it so what they did here is they basically omitted as they say"}, {"start": 1473.68, "end": 1478.88, "text": " here performance of models built from a stack of cross attention layers with no latent transformers"}, {"start": 1478.88, "end": 1484.48, "text": " and now this is way more equivalent to what the debtor facebook debtor model is doing and you"}, {"start": 1484.48, "end": 1489.76, "text": " can see that the accuracy is way lower compared to using those latent transformers to refine the"}, {"start": 1489.76, "end": 1496.8000000000002, "text": " representation so that's this is a kind of argument behind the usage of latent transformers okay now"}, {"start": 1496.8, "end": 1501.6, "text": " let me quickly contrast this paper with perceiver io okay perceiver io a general architecture for"}, {"start": 1501.6, "end": 1507.44, "text": " structured inputs and outputs again the deep mind team and they say here while the perceiver supports"}, {"start": 1507.44, "end": 1513.36, "text": " many kinds of inputs it can only produce very simple outputs such as class scores perceiver io"}, {"start": 1513.36, "end": 1518.24, "text": " overcomes this limitation without sacrificing the originals appealing properties by learning"}, {"start": 1518.24, "end": 1523.52, "text": " to flexibly query the model's latent space to produce outputs of arbitrary size and semantics"}, {"start": 1523.52, "end": 1529.52, "text": " so in the original paper all of the tasks we saw so image net audio set and those point clouds all"}, {"start": 1529.52, "end": 1534.4, "text": " of those were classification tasks and so we had a simple uh like the output structure was very"}, {"start": 1534.4, "end": 1540.32, "text": " simple when you have more complex uh problems such as trying to make an agent for starcraft 2"}, {"start": 1540.32, "end": 1544.6399999999999, "text": " then you have much more structured outputs so the output may be something like this like you have"}, {"start": 1544.6399999999999, "end": 1549.52, "text": " one array that tells you which agent you should select so which of the characters you should"}, {"start": 1549.52, "end": 1554.56, "text": " select and then once you select the agent you have like a matrix that tells you what's the"}, {"start": 1554.56, "end": 1560.0, "text": " probability of moving to all of these locations so maybe you have something like four by four matrix"}, {"start": 1560.0, "end": 1565.84, "text": " and basically you you have a probability of where the current character the selected character should"}, {"start": 1565.84, "end": 1571.52, "text": " move and so you have like complex structured output and the the thing is the original perceiver"}, {"start": 1571.52, "end": 1576.0, "text": " even though it was very flexible with the input it wasn't as flexible with the output and so basically"}, {"start": 1576.0, "end": 1580.48, "text": " what this paper does and they mentioned it here and they said here our main technical contribution"}, {"start": 1580.48, "end": 1585.92, "text": " is perceiver ios decoding procedure so let's see what they mean by that here is the architecture"}, {"start": 1585.92, "end": 1590.08, "text": " let me just kind of contrast this one with the original perceiver paper so this part is the same"}, {"start": 1590.08, "end": 1595.76, "text": " so we have latent array n times d we have the input byte array m times c we just do this cross"}, {"start": 1595.76, "end": 1600.56, "text": " attention module we do the latent transformer part so this is the latent transformer part this is the"}, {"start": 1600.56, "end": 1606.96, "text": " cross attention part and we just repeat that part rinse and repeat and that's the same thing as with"}, {"start": 1606.96, "end": 1612.32, "text": " the original perceiver so the novel thing here is this output query array so what you do is depending"}, {"start": 1612.32, "end": 1618.96, "text": " on your desired outputs you have like an output array that maybe has like o vectors and every vector"}, {"start": 1618.96, "end": 1627.04, "text": " has e features so what you would do is form o query vectors and basically again do across like"}, {"start": 1627.04, "end": 1633.12, "text": " attention logic and so because you have o vectors here you're going to end up with o vectors here"}, {"start": 1633.12, "end": 1636.56, "text": " zooming in into the decoder head because this is the main contribution of the paper"}, {"start": 1637.36, "end": 1643.44, "text": " let me kind of like break it down a little bit more first things first this these query vectors"}, {"start": 1643.44, "end": 1648.08, "text": " are going to be task dependent so depending on the structured output you want to achieve"}, {"start": 1648.6399999999999, "end": 1654.1599999999999, "text": " you'll be either ingraining some integrating some domain specific knowledge here or you're going to"}, {"start": 1654.16, "end": 1660.5600000000002, "text": " make these learnable let's see how the shapes work out here so we know we have basically all of these"}, {"start": 1660.5600000000002, "end": 1665.0400000000002, "text": " and their dimension the dimensionality is not that important it's going to be like x and the"}, {"start": 1665.0400000000002, "end": 1670.0, "text": " only important thing is that they match with like key vectors because we're going to do dot products"}, {"start": 1670.0, "end": 1675.2, "text": " here in order to form the attention scores now the important part is that the value vectors so these"}, {"start": 1675.2, "end": 1681.28, "text": " here have the dimensionality e if that's our target as you can see here so he's our target so we want"}, {"start": 1681.28, "end": 1686.96, "text": " to have the dimensionality of these value vectors should be e and we have we need to have all of"}, {"start": 1686.96, "end": 1694.0, "text": " these and that's how we make sure that this thing here is o times e and that's our desired shape so"}, {"start": 1694.0, "end": 1700.3999999999999, "text": " basically everything else is simple attention logic we form the weights here we use these to"}, {"start": 1700.3999999999999, "end": 1706.3999999999999, "text": " kind of weight these value vectors we add them up and we form via these two weights and via"}, {"start": 1706.4, "end": 1712.5600000000002, "text": " uh basically these two value vectors we form this vector here and we just repeat a procedure for"}, {"start": 1712.5600000000002, "end": 1717.68, "text": " all of the output vectors so that's basically the main contribution of the paper everything else is"}, {"start": 1717.68, "end": 1725.52, "text": " about trying this new perceiver io module model on various different tasks and i'm going to discover"}, {"start": 1725.52, "end": 1732.96, "text": " a single one and that's the language uh like uh like the glue benchmark and they compare with"}, {"start": 1732.96, "end": 1739.04, "text": " bird piece and you can see here depending whether they use tokenization so sentence piece"}, {"start": 1739.04, "end": 1746.24, "text": " is a like a famous tokenization algorithm or simply using utf-8 bytes that means no tokenization at"}, {"start": 1746.24, "end": 1753.2, "text": " all you just take the raw text and you encode the characters as utf-8 bytes and so you can see that"}, {"start": 1753.2, "end": 1758.8, "text": " they can report uh results of their own pair with bird but here's a catch i mean even though"}, {"start": 1758.8, "end": 1765.04, "text": " they're kind of making the the flop similar here uh the number of parameters is obviously way way"}, {"start": 1765.04, "end": 1770.0, "text": " bigger for perceiver io so if that's something you care about this is obviously not going to be"}, {"start": 1771.12, "end": 1776.0, "text": " like a fair comparison right and i guess this is where perceiver io shines the reason is we have"}, {"start": 1776.0, "end": 1781.6, "text": " when we have utf-8 the input size is going to be way larger and then because of that you have to"}, {"start": 1781.6, "end": 1786.8799999999999, "text": " kind of reduce the depth of the bird model and that means you have less parameters less expressive"}, {"start": 1786.88, "end": 1792.5600000000002, "text": " model and you get way bigger gap here and i find that very cool because just using"}, {"start": 1794.16, "end": 1800.0, "text": " utf-8 encoding instead of having to devise these intricate tokenization methods and tokenization"}, {"start": 1800.0, "end": 1805.8400000000001, "text": " is literally a field in and of itself so have like being able to avoid that whole step and still"}, {"start": 1805.8400000000001, "end": 1811.5200000000002, "text": " manage to have these results like even better than using sentence piece is pretty pretty damn"}, {"start": 1811.5200000000002, "end": 1815.8400000000001, "text": " amazing having said that that's it for this video if you found it useful share it out with a friend"}, {"start": 1815.84, "end": 1822.24, "text": " and subscribe to this channel hit the bell icon join the discord community and until next time bye"}, {"start": 1822.24, "end": 1846.72, "text": " bye"}]